1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
|
[[analysis-lowercase-tokenizer]]
=== Lowercase Tokenizer
A tokenizer of type `lowercase` that performs the function of
<<analysis-letter-tokenizer,Letter
Tokenizer>> and
<<analysis-lowercase-tokenfilter,Lower
Case Token Filter>> together. It divides text at non-letters and converts
them to lower case. While it is functionally equivalent to the
combination of
<<analysis-letter-tokenizer,Letter
Tokenizer>> and
<<analysis-lowercase-tokenfilter,Lower
Case Token Filter>>, there is a performance advantage to doing the two
tasks at once, hence this (redundant) implementation.
|