Tokenizer

Abstract base class for implementing custom tokenizers.

Import

import Tokenizer from 'dynamosearch/tokenizers/Tokenizer';

Abstract Methods

abstract class Tokenizer {
  abstract tokenize(str: string): Promise<Token[]>;
}

Token Interface

interface Token {
  token: string;
  startOffset: number;
  endOffset: number;
  position: number;
  keyword?: boolean;
}

Implementing a Custom Tokenizer

To create a custom tokenizer, extend the Tokenizer class and implement the tokenize method:

import Tokenizer from 'dynamosearch/tokenizers/Tokenizer';

class MyCustomTokenizer extends Tokenizer {
  async tokenize(str: string): Promise<Token[]> {
    // Your tokenization logic here
    return tokens;
  }
}

See Also

• StandardTokenizer
• Analyzer