* The output is ideal for feeding into a text search engine such as Apache Lucene,
* especially when the {@link #setIncludeAttributes(boolean) IncludeAttributes} property has been set to true.
*
* Use one of the following methods to obtain the output: *
* The process removes all of the tags and * {@linkplain CharacterReference#decodeCollapseWhiteSpace(CharSequence) decodes the result, collapsing all white space}. * A space character is included in the output where a normal tag is present in the source, * unless the tag belongs to an {@linkplain HTMLElements#getInlineLevelElementNames() inline-level} element. * An exception to this is the {@link HTMLElementName#BR BR} element, which is also converted to a space despite being an inline-level element. *
* Text inside {@link HTMLElementName#SCRIPT SCRIPT} and {@link HTMLElementName#STYLE STYLE} elements contained within this segment * is ignored. *
* Setting the {@link #setExcludeNonHTMLElements(boolean) ExcludeNonHTMLElements} property results in the exclusion of any content within a * non-HTML element. *
* See the {@link #excludeElement(StartTag)} method for details on how to implement a more complex mechanism to determine whether the * {@linkplain Element#getContent() content} of each {@link Element} is to be excluded from the output. *
* All tags that are not normal tags, such as {@linkplain TagType#isServerTag() server tags}, * {@linkplain StartTagType#COMMENT comments} etc., are removed from the output without adding white space to the output. *
* Note that segments on which the {@link Segment#ignoreWhenParsing()} method has been called are treated as text rather than markup, * resulting in their inclusion in the output. * To remove specific segments before extracting the text, create an {@link OutputDocument} and call its {@link OutputDocument#remove(Segment) remove(Segment)} or * {@link OutputDocument#replaceWithSpaces(int,int) replaceWithSpaces(int begin, int end)} method for each segment to be removed. * Then create a new source document using {@link Source#Source(CharSequence) new Source(outputDocument.toString())} * and perform the text extraction on this new source object. *
* Extracting the text from an entire {@link Source} object performs a {@linkplain Source#fullSequentialParse() full sequential parse} automatically. *
* To perform a simple rendering of HTML markup into text, which is more readable than the output of this class, use the {@link Renderer} class instead. *
<div><b>O</b>ne</div><div title="Two"><b>Th</b><script>//a script </script>ree</div>"One Two Three".
* TextExtractor based on the specified {@link Segment}.
* @param segment the segment from which the text will be extracted.
* @see Segment#getTextExtractor()
*/
public TextExtractor(final Segment segment) {
this.segment=segment;
}
// Documentation inherited from CharStreamSource
public void writeTo(final Writer writer) throws IOException {
appendTo(writer);
writer.flush();
}
// Documentation inherited from CharStreamSource
public void appendTo(final Appendable appendable) throws IOException {
appendable.append(toString());
}
// Documentation inherited from CharStreamSource
public long getEstimatedMaximumOutputLength() {
return segment.length();
}
// Documentation inherited from CharStreamSource
public String toString() {
return new Processor(segment,getConvertNonBreakingSpaces(),getIncludeAttributes(),getExcludeNonHTMLElements()).toString();
}
/**
* Sets whether non-breaking space ({@link CharacterEntityReference#_nbsp }) character entity references are converted to spaces.
*
* The default value is that of the static {@link Config#ConvertNonBreakingSpaces} property at the time the TextExtractor is instantiated.
*
* @param convertNonBreakingSpaces specifies whether non-breaking space ({@link CharacterEntityReference#_nbsp }) character entity references are converted to spaces.
* @return this TextExtractor instance, allowing multiple property setting methods to be chained in a single statement.
* @see #getConvertNonBreakingSpaces()
*/
public TextExtractor setConvertNonBreakingSpaces(boolean convertNonBreakingSpaces) {
this.convertNonBreakingSpaces=convertNonBreakingSpaces;
return this;
}
/**
* Indicates whether non-breaking space ({@link CharacterEntityReference#_nbsp }) character entity references are converted to spaces.
*
* See the {@link #setConvertNonBreakingSpaces(boolean)} method for a full description of this property.
*
* @return true if non-breaking space ({@link CharacterEntityReference#_nbsp }) character entity references are converted to spaces, otherwise false.
*/
public boolean getConvertNonBreakingSpaces() {
return convertNonBreakingSpaces;
}
/**
* Sets whether any attribute values are included in the output.
*
* If the value of this property is true, then each attribute still has to match the conditions implemented in the
* {@link #includeAttribute(StartTag,Attribute)} method in order for its value to be included in the output.
*
* The default value is false.
*
* @param includeAttributes specifies whether any attribute values are included in the output.
* @return this TextExtractor instance, allowing multiple property setting methods to be chained in a single statement.
* @see #getIncludeAttributes()
*/
public TextExtractor setIncludeAttributes(boolean includeAttributes) {
this.includeAttributes=includeAttributes;
return this;
}
/**
* Indicates whether any attribute values are included in the output.
*
* See the {@link #setIncludeAttributes(boolean)} method for a full description of this property.
*
* @return true if any attribute values are included in the output, otherwise false.
*/
public boolean getIncludeAttributes() {
return includeAttributes;
}
/**
* Indicates whether the value of the specified {@linkplain Attribute attribute} in the specified {@linkplain StartTag start tag} is included in the output.
*
* This method is ignored if the {@link #setIncludeAttributes(boolean) IncludeAttributes} property is set to false, in which case
* no attribute values are included in the output.
*
* If the {@link #setIncludeAttributes(boolean) IncludeAttributes} property is set to true, every attribute of every
* start tag encountered in the segment is checked using this method to determine whether the value of the attribute should be included in the output.
*
* The default implementation of this method returns true if the {@linkplain Attribute#getName() name} of the specified {@linkplain Attribute attribute}
* is one of
* title,
* alt,
* label,
* summary,
* content*, or
* href,
* but the method can be overridden in a subclass to perform a check of arbitrary complexity on each attribute.
*
* * The value of a content attribute is only included if a * name attribute is also present in the specified start tag, * as the content attribute of a {@link HTMLElementName#META META} tag only contains human readable text if the name attribute is used as opposed to an * http-equiv attribute. *
*
* final Set includeAttributeNames=new HashSet(Arrays.asList(new String[] {"title","alt"}));
* TextExtractor textExtractor=new TextExtractor(segment) {
* public boolean includeAttribute(StartTag startTag, Attribute attribute) {
* return includeAttributeNames.contains(attribute.getKey());
* }
* };
* textExtractor.setIncludeAttributes(true);
* String extractedText=textExtractor.toString();
*
* false.
*/
public boolean includeAttribute(final StartTag startTag, final Attribute attribute) {
AttributeIncludeChecker attributeIncludeChecker=map.get(attribute.getKey());
if (attributeIncludeChecker==null) return false;
return attributeIncludeChecker.includeAttribute(startTag,attribute);
}
/**
* Sets whether the content of non-HTML elements is excluded from the output.
*
* The default value is false, meaning that content from all elements meeting the other criteria is included.
*
* @param excludeNonHTMLElements specifies whether content non-HTML elements is excluded from the output.
* @return this TextExtractor instance, allowing multiple property setting methods to be chained in a single statement.
* @see #getExcludeNonHTMLElements()
*/
public TextExtractor setExcludeNonHTMLElements(boolean excludeNonHTMLElements) {
this.excludeNonHTMLElements=excludeNonHTMLElements;
return this;
}
/**
* Indicates whether the content of non-HTML elements is excluded from the output.
*
* See the {@link #setExcludeNonHTMLElements(boolean)} method for a full description of this property.
*
* @return true if the content of non-HTML elements is excluded from the output, otherwise false.
*/
public boolean getExcludeNonHTMLElements() {
return excludeNonHTMLElements;
}
/**
* Indicates whether the text inside the {@link Element} of the specified start tag should be excluded from the output.
*
* During the text extraction process, every start tag encountered in the segment is checked using this method to determine whether the text inside its * {@linkplain StartTag#getElement() associated element} should be excluded from the output. *
* The default implementation of this method is to always return false, so that every element is included,
* but the method can be overridden in a subclass to perform a check of arbitrary complexity on each start tag.
*
* All elements nested inside an excluded element are also implicitly excluded, as are all * {@link HTMLElementName#SCRIPT SCRIPT} and {@link HTMLElementName#STYLE STYLE} elements. * Such elements are skipped over without calling this method, so there is no way to include them by overriding the method. *
*
segment, excluding any text inside elements with the attribute class="NotIndexed":
* TextExtractor textExtractor=new TextExtractor(segment) {
* public boolean excludeElement(StartTag startTag) {
* return "NotIndexed".equalsIgnoreCase(startTag.getAttributeValue("class"));
* }
* };
* String extractedText=textExtractor.toString();
*
* false.
*/
public boolean excludeElement(final StartTag startTag) {
return false;
}
private static interface AttributeIncludeChecker {
boolean includeAttribute(final StartTag startTag, final Attribute attribute);
}
private static AttributeIncludeChecker ALWAYS_INCLUDE=new AttributeIncludeChecker() {
public boolean includeAttribute(final StartTag startTag, final Attribute attribute) {
return true;
}
};
private static AttributeIncludeChecker INCLUDE_IF_NAME_ATTRIBUTE_PRESENT=new AttributeIncludeChecker() {
public boolean includeAttribute(final StartTag startTag, final Attribute attribute) {
return startTag.getAttributes().get("name")!=null;
}
};
private static Map