Create a tokenizer that splits strings into tokens according to the given terms.
The terms are interpreted as regular expressions, and are sorted by length
to ensure that longer matches are preferred over shorter ones.
Param: terms
Methods
split
split(str:string):string[]
Split the given string into tokens according to the terms specified in the constructor.
Create a tokenizer that splits strings into tokens according to the given terms. The terms are interpreted as regular expressions, and are sorted by length to ensure that longer matches are preferred over shorter ones.