Tokenizer analyzer
The tokenizers take care of the actual rules for where to break the text up into words (typically whitespace).
分词器会考虑在哪将文本切分成词(典型的如空格)的实际规则。
—— 给力词典精选Alll previous versions of the word breakers must be removed prior to installation .
安装之前,必须删除所有先前的分词器版本。
—— 给力词典精选Each analyzer includes one or more tokenizers and may include filters.
每一个分析器包括一个或多个分词器和滤词器(filter)。
—— 给力词典精选