Lines Matching refs:tokenizer

44 #   1: Verify that there is no such fts2 tokenizer as 'blah'.
46 #   2: Query for the built-in tokenizer 'simple'. Insert a copy of the
47 #      retrieved value as tokenizer 'blah'.
49 #   3: Test that the value returned for tokenizer 'blah' is now the
53 #      tokenizer 'blah' (it was not possible in step 1).
55 #   5: Test that the table created to use tokenizer 'blah' is usable.
61 } {1 {unknown tokenizer: blah}}
94 } {1 {unknown tokenizer: nosuchtokenizer}}
99 # to test the test function as the tokenizer implementations.
120 # Test cases fts2token-4.* test the ICU tokenizer. In practice, this
121 # tokenizer only has two modes - "thai" and "everybody else". Some other