syntax
One tokenizer per language, for the terminal and for the widgets.
Detailed description
The tree colored code in eight widget highlighters, each with its own regular expressions, and the log colored nothing. log-color §LGC-D-10 puts one tokenizer per language here, with QtCore only, and two renderers on top: `TokenizedHighlighter` in libwidgets for a QTextDocument, and the terminal renderer in liblog for the terminal copy of a log line.
A tokenizer reads one line and the state that the line before ended in, and returns the tokens and the state that this line ends in, as QSyntaxHighlighter::highlightBlock needs. It reads each character a bounded number of times and uses no backtracking regular expression, so its cost grows in a straight line with the text (log-color §LGC-P-11).
A token has a role, which a renderer maps to a color, and an emphasis, bold or italic, which the highlighters of the tree use beside the color. The tokens of a line do not overlap, and they are in order.
Type documentation
Language enum
enum syntax::Language
A language that a tokenizer reads.
Each constant is in syntax::Language.
| Constant | Value | Description |
|---|---|---|
None |
0 |
No tokens. |
Text |
1 |
Prose, such as the body of a log line: file paths, URLs, domains, numbers and quoted strings, as the monolith's log highlighters mark them (log-color parity row LGC-5). |
Sql |
2 |
SQL: the keywords of the monolith's table, the punctuation `,;()`, `%(name)s` placeholders, strings, numbers and comments (parity row LGC-6). |
Json |
3 |
JSON, the language of JsonSyntaxHighlighter (parity row LGC-30). |
Cpp |
4 |
C++ and Arduino code, the language of CppSyntaxHighlighter (LGC-31). |
Glsl |
5 |
GLSL, the language of GLSLSyntaxHighlighter (LGC-32). |
Yaml |
6 |
YAML, the language of YamlSyntaxHighlighter (LGC-33). |
Markdown |
7 |
Markdown, the language of MarkdownHighlighter (LGC-34). |
Html |
8 |
HTML, the language of HTMLHighlighter (LGC-35). |
Plan |
9 |
The plan language, as the lexer of libparser reads it (LGC-36). |
AutomationScript |
10 |
The automation script language, as UIScript and AutomationScript read it (LGC-37). |
Role enum
enum syntax::Role
What a token is: the fields of SemanticPalette's SyntaxColors.
Each constant is in syntax::Role.
| Constant | Value | Description |
|---|---|---|
Keyword |
0 |
A keyword of the language. |
Type |
1 |
A type, a key of JSON or YAML, or a part of a path or a URL. |
Function |
2 |
A function, or the name of a file. |
String |
3 |
A quoted string. |
Number |
4 |
A number. |
Comment |
5 |
A comment. |
Preprocessor |
6 |
A preprocessor line, a placeholder, an anchor or a hidden folder. |
Operator |
7 |
An operator, a bracket or a separator. |
Constant |
8 |
A constant, such as true, or the extension of a file. |
Heading |
9 |
A heading. |
Link |
10 |
A link, a host or a domain. |
Error |
11 |
An error. |
Function documentation
tokenize
QList<syntax::Token> syntax::tokenize(syntax::Language language, QStringView text)
The tokens of text of language, with their offsets in text.
The text is read line by line, a line ending at '\n', and each line starts in the state that the line before ended in. This is what a log body with newlines needs.
tokenizeLine
QList<syntax::Token> syntax::tokenizeLine(syntax::Language language, QStringView line, syntax::State state = kStart, syntax::State *next = nullptr)
The tokens of one line of language, after a line that ended in state. next, when it is not null, gets the state that the line ends in.
tokenizerOf
const syntax::Tokenizer &syntax::tokenizerOf(syntax::Language language)
The tokenizer of language.