DEFINE ANALYZER

DEFINE ANALYZER statement

In the context of a database, an analyzer plays a crucial role in text processing and searching. It is defined by its name, a set of tokenizers, and a collection of filters.

Requirements

• You must be authenticated as a root, namespace, or database user before you can use the DEFINE ANALYZER statement.
• You must select your namespace and database before you can use the DEFINE ANALYZER statement.

Statement syntax

SurrealQL Syntax

DEFINE ANALYZER @name [ TOKENIZERS @tokenizers ] [ FILTERS @filters ]

Tokenizers

Tokenizers are responsible for breaking down a given text into individual tokens based on a set of instructions. There are a couple of tokenizers that can be used while defining an analyzer as seen below:

• blank: The blank tokenizer breaks down a text into tokens by creating a new token each time it encounters a space, tab, or newline character. It's a straightforward way to split text into words or chunks based on whitespace.

DEFINE ANALYZER example_blank TOKENIZERS blank;

Input Text: "hello world" , Tokens: ["hello", "world"]

• camel: The camel tokenizer is used for identifying and creating tokens when the next character in the text is uppercase. This is particularly useful for processing camelCase or PascalCase text, common in programming, to split them into meaningful words.

DEFINE ANALYZER example_camel TOKENIZERS camel;

Input Text: "helloWorld" , Tokens: ["hello", "World"]

• class: The class tokenizer segments text into tokens by detecting changes (digit, letter, punctuation, blank) in the Unicode class of characters. It creates a new token when the character class changes, distinguishing between digits, letters, punctuation, and blanks. This allows for flexible tokenization based on character types.

DEFINE ANALYZER example_class TOKENIZERS class;

Input Text: "123abc!XYZ", Tokens: ["123", "abc", "!", "XYZ"]

• punct: The punct tokenizer generates tokens by breaking the text whenever a punctuation character is encountered. It's suitable for tokenizing sentences or breaking text into smaller units based on punctuation marks.

DEFINE ANALYZER example_punct TOKENIZERS punct;

Input Text: "Hello, World!", Tokens: ["Hello", ",", "World", "!"]

Filters

Filters take on the task of transforming these tokens for further processing and analysis.

• ascii: The ascii filter is responsible for processing tokens by replacing or removing diacritical marks (accents and special characters) from the text. It helps standardize text by converting accented characters to their basic ASCII equivalents, making it more suitable for various text analysis tasks.

DEFINE ANALYZER example_ascii TOKENIZERS class FILTERS ascii;

Input Text: "résumé café" , Tokens: ["resume", "cafe"]

• edgengram(min,max): The edgengram filter is used to create tokens that represent prefixes of terms. It generates a sequence of tokens that gradually build up a term, which can be useful for autocomplete or searching based on partial words. It accepts two parameters min and max which define the minimum and maximum amount of characters in the prefix.

DEFINE ANALYZER example_edgengram TOKENIZERS class FILTERS edgengram(1,3);

Input Text: "apple banana", Tokens: ["a", "ap", "app", "b", "ba", "ban"]

• lowercase: The lowercase filter converts tokens to lowercase, ensuring that text is consistently in lowercase format. This is often used to make text case-insensitive for search and analysis purposes.

DEFINE ANALYZER example_lowercase TOKENIZERS class FILTERS lowercase;

Input Text: "Hello World", Tokens: ["hello", "world"]

• snowball(language): The snowball filter applies Snowball stemming to tokens, reducing them to their root form and converts the case to lowercase. The following supported languages can be passed as a parameter in snowball: Arabic, Danish, Dutch, English, French, German, Greek, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, Tamil, Turkish.

DEFINE ANALYZER example_snowball TOKENIZERS class FILTERS snowball(english);

Input Text: "running cats", Tokens: ["run", "cat"]

• uppercase: The uppercase filter converts tokens to uppercase, ensuring text consistency in uppercase format. It can be useful when case-insensitivity is required for specific analysis or search operations.

DEFINE ANALYZER example_uppercase TOKENIZERS class FILTERS uppercase;
Input Text: "Hello World", Tokens: ["HELLO", "WORLD"]

Example usage

Examples on application of analyzers to indexes can be found in the documenation on DEFINE INDEX statement

This example creates an analyzer that splits tokens on blank characters and removes diacritical marks.

-- Creates a simple analyzer removing diacritics marks
DEFINE ANALYZER ascii TOKENIZERS class FILTERS lowercase,ascii;

This command statement creates an analyzer specifically designed for processing English texts.

-- Creates an analyzer suitable for English text
DEFINE ANALYZER english TOKENIZERS class FILTERS snowball(english);

This statement creates an analyzer specifically designed for auto-completion tasks.

-- Creates an analyzer suitable for auto-completion.
DEFINE ANALYZER autocomplete FILTERS lowercase,edgengram(2,10);

This command statement creates an analyzer specifically designed for source code analysis.

-- Creates an analyzer suitable for source code analysis.
DEFINE ANALYZER code TOKENIZERS class,camel FILTERS lowercase,ascii;