Lines Matching refs:tokenizer

5   the text tokenizer implementation to be used when indexing text
6   by specifying a "tokenizer" clause as part of the CREATE VIRTUAL TABLE
10       <columns ...> [, tokenizer <tokenizer-name> [<tokenizer-args>]]
13   The built-in tokenizers (valid values to pass as <tokenizer name>) are
16   <tokenizer-args> should consist of zero or more white-space separated
17   arguments to pass to the selected tokenizer implementation. The 
19   tokenizer.
23   FTS2 allows users to provide custom tokenizer implementations. The 
24   interface used to create a new tokenizer is defined and described in 
27   Registering a new FTS2 tokenizer is similar to registering a new 
30   make up the implementation of the new tokenizer type. For tokenizers,
35   tokenizer types with a database handle. Instead, the pointer must
41     SELECT fts2_tokenizer(<tokenizer-name>);
42     SELECT fts2_tokenizer(<tokenizer-name>, <sqlite3_tokenizer_module ptr>);
44   Where <tokenizer-name> is a string identifying the tokenizer and
47   it is registered as tokenizer <tokenizer-name> and a copy of it
48   returned. If only one argument is passed, a pointer to the tokenizer
49   implementation currently registered as <tokenizer-name> is returned,
50   encoded as a blob. Or, if no such tokenizer exists, an SQL exception
64   symbol defined, then there exists a built-in tokenizer named "icu" 
66   xCreate() method (see fts2_tokenizer.h) of this tokenizer may be
70     "CREATE VIRTUAL TABLE thai_text USING fts2(text, tokenizer icu th_TH)"
72   The ICU tokenizer implementation is very simple. It splits the input
77   discard punctuation, this can be done by creating a tokenizer 
78   implementation that uses the ICU tokenizer as part of its implementation.
80   When using the ICU tokenizer this way, it is safe to overwrite the