mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-22 17:30:49 +08:00
chore(vendor): refresh the ObjectScript grammars to upstream v1.11.0
Re-vendors objectscript_udl and objectscript_routine from intersystems/tree-sitter-objectscript a7ffcdf2de8e to tag v1.11.0 (25cf0f67a380, upstream HEAD); both stay ABI 15. The refreshed scanner lexes Language = python method bodies with Python's string, triple-quote and #-comment rules, so a brace inside a Python string, comment or docstring no longer runs the body past its closing brace and drops the methods that follow it (#1261). Both parser.c files are upstream's committed bytes, and regenerating them from upstream's src/grammar.json and tree-sitter.json with tree-sitter-cli 0.27.1 (upstream's locked version) reproduces them byte-for-byte. The three local patches recorded in MANIFEST.md are re-applied unchanged: the objectscript_common.h include repoint, the uint8_t -> int widening of two loop counters, and the empty-state memcpy guard; upstream v1.11.0 still carries the original form of all three. Updates the MANIFEST.md note, pin rows and ABI recount, THIRD_PARTY.md and scripts/vendored-checksums.txt, and adds two parse-coverage tests: Python bodies with braces in strings, comments and docstrings parse completely, and an unclosed Python body is still reported as partial. Refs #1261, #2368 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This commit is contained in:
+2
-2
@@ -129,8 +129,8 @@ License summary:
|
||||
- **License:** MIT
|
||||
- **Copyright:** (c) 2025 InterSystems Corporation
|
||||
- **Vendored at:** `internal/cbm/vendored/grammars/objectscript_udl/`, `internal/cbm/vendored/grammars/objectscript_routine/`
|
||||
- **Pinned commit:** `a7ffcdf`
|
||||
- **Notes:** InterSystems-maintained grammar for the ObjectScript language (InterSystems IRIS / Caché). Vendor-maintained; not in nvim-treesitter or Helix registries. Each `scanner.c`'s upstream `#include "../../common/scanner.h"` is repointed to a per-directory `objectscript_common.h` copied from upstream `common/scanner.h`; two loop counters in that copy are widened from `uint8_t` to `int` as documented in `internal/cbm/vendored/grammars/MANIFEST.md`.
|
||||
- **Pinned commit:** `25cf0f67a380` (tag `v1.11.0`; refreshed from `a7ffcdf` on 2026-11-13)
|
||||
- **Notes:** InterSystems-maintained grammar for the ObjectScript language (InterSystems IRIS / Caché). Vendor-maintained; not in nvim-treesitter or Helix registries. Each `scanner.c`'s upstream `#include "../../common/scanner.h"` is repointed to a per-directory `objectscript_common.h` copied from upstream `common/scanner.h`; two loop counters in that copy are widened from `uint8_t` to `int`, and each `scanner.c` guards its deserialize `memcpy` against an empty state, as documented in `internal/cbm/vendored/grammars/MANIFEST.md`.
|
||||
|
||||
## Vendored C/C++ Libraries
|
||||
|
||||
|
||||
@@ -11,6 +11,7 @@ The grammars were originally vendored as bare `parser.c`+`scanner.c` with **no r
|
||||
|
||||
- Grammars: **163** — vendored-from-upstream: **139**, first-party/self-maintained: **19**, registry-disagreement: **5** (nim removed 2026-06-12; objectscript_udl + objectscript_routine added 2026-06-24; mojo added 2026-07-01; arkts added 2026-08-26; plsql added 2026-08-27; chialisp added 2026-08-28; javascript + tsx became self-maintained forks 2026-09-25; pascal became a self-maintained fork 2026-09-25; vbnet added as a self-maintained fork 2026-11-10; sql became a self-maintained fork 2026-11-11 — see notes below)
|
||||
- ABI distribution: **9×** ABI-13 **78×** ABI-14 **76×** ABI-15 (runtime ceiling is ABI 15; never vendor ABI 16 without a runtime upgrade)
|
||||
— recounted from the tree 2026-11-13 after the `objectscript_udl` / `objectscript_routine` refresh to upstream v1.11.0: unchanged (both stay ABI 15).
|
||||
— recounted from the tree 2026-11-11 after the `sql` fork was regenerated with tree-sitter-cli 0.25.10: unchanged (`sql` stays ABI 15).
|
||||
— recounted from the tree 2026-11-10 after adding the ABI-15 `vbnet` grammar (was `9×/78×/75×`).
|
||||
— recounted from the tree 2026-09-25 after the tsx fork was regenerated with tree-sitter-cli 0.25.10, moving `tsx` from ABI 14 to ABI 15 (was `9×/79×/74×`).
|
||||
@@ -52,7 +53,7 @@ The grammars were originally vendored as bare `parser.c`+`scanner.c` with **no r
|
||||
newlines; the new files keep upstream's bytes exactly. Per-file SHA-256
|
||||
values are pinned in `scripts/vendored-checksums.txt`. Regression coverage:
|
||||
the `cs_*_issue1748` tests in `tests/test_parse_coverage.c`.
|
||||
- **objectscript_udl / objectscript_routine** (added 2026-06-24): vendored from [intersystems/tree-sitter-objectscript](https://github.com/intersystems/tree-sitter-objectscript) @ `a7ffcdf` — MIT, the InterSystems-official grammars (a niche vendor language, hence `vendor-maintained`, not in nvim-treesitter/Helix). **Re-vendor note:** each `scanner.c`'s upstream `#include "../../common/scanner.h"` is repointed to a per-directory `objectscript_common.h` (a verbatim copy of upstream `common/scanner.h`), because this repo's shared `vendored/common/scanner.h` belongs to the cfml/fsharp grammars and differs. The generated `parser.c` files remain byte-for-byte upstream. Each `scanner.c` also guards empty-state deserialization with `if (length > 0)` before `memcpy`, because Tree-sitter can pass a NULL buffer with zero length. On re-vendor, re-apply the include rename and this guard unless upstream already carries them. **Local modification (2026-07-16):** in `objectscript_common.h`, two loop counters `uint8_t i` were widened to `int i` (the `reverse_marker` scan and the `html_marker_buffer` reversal) to clear CodeQL `cpp/comparison-with-wider-type` — a false positive in practice (both lengths are hard-bounded by `MARKER_BUFFER_MAX_LEN = 30`, so `uint8_t` could never wrap), fixed for cleanliness. On re-vendor, re-apply this widening too (or upstream it at intersystems/tree-sitter-objectscript).
|
||||
- **objectscript_udl / objectscript_routine** (added 2026-06-24 @ `a7ffcdf`; refreshed 2026-11-13, #1261): vendored from [intersystems/tree-sitter-objectscript](https://github.com/intersystems/tree-sitter-objectscript) tag `v1.11.0`, full commit `25cf0f67a3804727e250076a0926aa71c46fffab` (upstream `main` HEAD, ABI 15 unchanged), replacing `a7ffcdf2de8e` — MIT, (c) 2025 InterSystems Corporation, the InterSystems-official grammars (a niche vendor language, hence `vendor-maintained`, not in nvim-treesitter/Helix). Upstream `udl/src` maps to `objectscript_udl/`, `objectscript_routine/src` to `objectscript_routine/`, `common/scanner.h` to each directory's `objectscript_common.h`, the root `LICENSE` to each `LICENSE` (byte-identical to the previous pin). **Provenance:** both `parser.c` files and all `tree_sitter/*.h` are upstream's committed files byte-for-byte; regenerating the parsers from upstream's committed `src/grammar.json` + `tree-sitter.json` with tree-sitter-cli 0.27.1 (the version in upstream's `package-lock.json`; release binary checked against its published SHA-256) in a no-network, write-confined sandbox reproduces both byte-for-byte (SHA-256 `0c0c651b4a425cb8a00cdf6b8c89aeea702cb5799a9483d9994556655f441015` udl, `d496de950b085853fdf1edf1570eecdb5d2e4525344873bfabf7e34ac8e7add0` routine). No upstream JavaScript, script or workflow was run. **Why (#1261):** the `a7ffcdf` scanner only balanced braces in a `Language = python` method body, so a brace inside a Python string, comment or docstring ran the body past its closing `}` and the following methods were lost in an ERROR region; upstream lexes these bodies with Python's string, triple-quote and `#`-comment rules since v1.9.15. On intersystems/ipm @ `571e04a19aba` (429 `.cls`) the source under a top-level ERROR drops from 353,816 bytes in 138 files to 1,323 bytes in 3 files, with no file losing a method. Remaining gaps there: a class-level `[ Language = python ]` default (its method bodies still parse as ObjectScript), `Index ... [ Data = X(ELEMENTS) ]`, and a string concatenated with an underscore-led name inside a macro argument. **Node shapes:** from v1.9.15 upstream hides the `method_call` wrapper inside `Do`/`Job` (receiver and `oref_method` sit directly under `do_parameter`/`job_argument`), `core_trigger`/`external_trigger` (inlined into `trigger`) and `storage_body` (now `external_method_body_content`), and a routine `Do label` / `$$^routine` wraps its target in `method_name` / `routine_ref` instead of always `line_ref`. Extraction must handle these shapes in the same change (the adaptation is in #2368): without it, `Do obj.M()` instance calls and routine `Do label(...)` calls lose their CALLS edge, and Trigger body tokens and Storage XML properties are lost. **Size:** `parser.c` is 49,392,087 bytes (udl) and 48,934,992 bytes (routine), under the 50 MB line with little headroom. **Re-vendor note:** each `scanner.c`'s upstream `#include "../../common/scanner.h"` is repointed to a per-directory `objectscript_common.h` (a verbatim copy of upstream `common/scanner.h`), because this repo's shared `vendored/common/scanner.h` belongs to the cfml/fsharp grammars and differs. The generated `parser.c` files remain byte-for-byte upstream. Each `scanner.c` also guards empty-state deserialization with `if (length > 0)` before `memcpy`, because Tree-sitter can pass a NULL buffer with zero length. On re-vendor, re-apply the include rename and this guard unless upstream already carries them. **Local modification (2026-07-16):** in `objectscript_common.h`, two loop counters `uint8_t i` were widened to `int i` (the `reverse_marker` scan and the `html_marker_buffer` reversal) to clear CodeQL `cpp/comparison-with-wider-type` — a false positive in practice (both lengths are hard-bounded by `MARKER_BUFFER_MAX_LEN = 30`, so `uint8_t` could never wrap), fixed for cleanliness. On re-vendor, re-apply this widening too (or upstream it at intersystems/tree-sitter-objectscript).
|
||||
- **mojo** (added 2026-07-01): vendored from [lsh/tree-sitter-mojo](https://github.com/lsh/tree-sitter-mojo) @ `33193a99afe6` — MIT, ABI 15. Helix tracks `lsh/tree-sitter-mojo` as its Mojo grammar source, but the Helix-pinned commit (`3d7c53b8038f`) no longer resolves in the upstream repository after a force-push, so this vendor uses current upstream `main` rather than the stale registry SHA. Security review covered only the vendored C surface (`parser.c`, `scanner.c`, `tree_sitter/*.h`) plus upstream license/provenance metadata; no package manager hooks, workflow files, prompt/agent instruction files, or generated lockfiles were vendored.
|
||||
- **arkts** (added 2026-08-26): **first-party derivative** — a fork of [tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript)'s `typescript` dialect pinned @ `75b3874edb2d` (v0.23.2, the same commit our vendored typescript/tsx come from), on the tree-sitter-javascript base @ `3a837b6f3658` (v0.23.1, the version upstream's own package-lock pins), extended with ArkTS/ArkUI syntax (`@Component struct` declarations, UI-DSL trailing-closure calls with post-block attribute chains, decorated function declarations, `import lazy`, `@Extend`/`@Styles` leading-dot attribute chains, anonymous `stateStyles` style blocks). Grammar source + corpus tests live in `tools/tree-sitter-arkts/`; regenerate with `npx tree-sitter-cli@0.25.10 generate` (ABI 15). **External scanner:** `scanner.c` is a symbol-rename trampoline; `_common_scanner.h` is byte-identical (same SHA-256) to the reviewed `typescript/_common_scanner.h` already shipped. **LICENSE:** tree-sitter-typescript's MIT text **verbatim and byte-identical** (© 2017 Max Brunsfeld) — which is what MIT requires of a derivative work, and what lets the provenance audit byte-verify it against upstream rather than take a note on trust. The fork is registered in the audit's `FORKS` map. Our own copyright for the ArkTS additions, and the tree-sitter-javascript attribution (© 2014 Max Brunsfeld), live with the SOURCE in `tools/tree-sitter-arkts/grammar.js` and in `THIRD_PARTY.md`, which is what ships in the release archives.
|
||||
- **javascript + tsx** (forked 2026-09-25, #1736): **self-maintained forks** of the pins we already vendored — [tree-sitter/tree-sitter-javascript](https://github.com/tree-sitter/tree-sitter-javascript) @ `58404d8cf191` and [tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript)'s `tsx` dialect @ `75b3874edb2d` (javascript base @ `3a837b6f3658`, the version upstream's package-lock pins, as for arkts). **Why:** upstream accepts a `&` inside a JSX string only before a space/digit or as a complete character reference, and never inside JSX text, so a plain URL query (`href="…?a=1&b=2"`, Google Fonts `&display=swap`) or `<p>Tom & Jerry</p>` produced an ERROR node — a `parse_partial` flag, plus false USAGE edges from identifiers the error recovery pulled out of the URL (on vercel/next.js `examples/`: 45 of 92 flagged files, 115 false cross-example USAGE edges). **The change** is one patch, `tools/tree-sitter-javascript/patches/jsx-lone-ampersand.patch`, applied to both javascript bases: a lone `&` that does not begin a complete `html_character_reference` is a one-character token (lexical precedence -1, so `&`/`&` still lex as references) aliased to the EXISTING `string_fragment` / `jsx_text` nodes, and the string-fragment regexes stop before a `&` that directly precedes the closing quote. Node types are unchanged (same type/named set; `node-types.json` only gains an empty `fields` map on those two entries). Without the patch both directories regenerate upstream exactly (javascript: byte-identical `parser.c` apart from the generator-version comment; tsx: identical state/symbol counts, ABI 14 → 15 from the newer CLI). **External scanners** are unchanged: `javascript/scanner.c` and `tsx/scanner.c` + `_common_scanner.h` stay the reviewed upstream bytes. Regenerate with `npx tree-sitter-cli@0.25.10 generate` in `tools/tree-sitter-{javascript,tsx}/` (ABI 15) and copy `src/parser.c` (tsx: also `src/tree_sitter/*.h`) here. **On an upstream bump:** re-fetch the new pins, re-apply the patch, regenerate, re-run `test/corpus/jsx_ampersand.txt` — and drop the fork if upstream fixes it. **LICENSE:** upstream's MIT text byte-identical (registered in the audit's `FORKS` map); our copyright for the patch lives in the fork `grammar.js` headers and `THIRD_PARTY.md`.
|
||||
@@ -113,8 +114,8 @@ row instead.
|
||||
| crystal | `crystal/scanner.c`, serialize | guard `memcpy(&buffer[offset], state->literals.contents, literal_content_size)` with `if (literal_content_size > 0)` | UBSan: zero-length `memcpy` with a NULL/0-size source on the empty-state serialize round-trip (formal UB, harmless) |
|
||||
| rescript | `rescript/scanner.c`, deserialize | guard `memcpy(state, buffer, n_bytes)` with `if (n_bytes > 0)` | UBSan: zero-length `memcpy` with a NULL `buffer` / `n_bytes == 0` on empty-state deserialize (formal UB, harmless). The sibling serialize copies a fixed `sizeof(ScannerState)` (always > 0, non-NULL src) and needs no guard. |
|
||||
| rescript | `rescript/scanner.c`, scan, `TEMPLATE_CHARS` branch | `if (valid_symbols[TEMPLATE_CHARS])` → `if (valid_symbols[TEMPLATE_CHARS] && !valid_symbols[NEWLINE])` | **Quadratic lexing in error recovery (#2176).** The only parse state whose external lex state enables both tokens is tree-sitter's ERROR state (`ts_lex_modes[0]`, external state 1; the in-template state 7 has no `_newline`), where every external token is marked valid and the lexer retries at each byte. From every byte the template loop ran to the next `` ` ``, `$`, `\` or NUL — the end of file when there is none — and returned false, so an unparseable stretch cost O(n²) inside lexing, where the parse budget's progress callback never runs. 50 KB of `0xFF` took 5.3 s, 100 KB 30 s; after: 3 ms, linear (lexer bytes pulled for 4 KB → 8 KB of `0xFF`: 8.8 M → 35.3 M before, 8.7 K → 17.4 K after, `complexity_rescript_error_recovery_lexing_is_linear`). Valid files parse to byte-identical trees; broken files recover more (a `let` after an unterminated expression is kept instead of swallowed). Upstream pin `43c2f1f35024` carries the unguarded branch |
|
||||
| objectscript_routine | `objectscript_routine/scanner.c`, deserialize | guard `memcpy(payload, buffer, length)` with `if (length > 0)` | UBSan: Tree-sitter passes a NULL buffer and zero length for empty scanner state; skip the zero-length copy |
|
||||
| objectscript_udl | `objectscript_udl/scanner.c`, deserialize | guard `memcpy(payload, buffer, length)` with `if (length > 0)` | UBSan: Tree-sitter passes a NULL buffer and zero length for empty scanner state; skip the zero-length copy |
|
||||
| objectscript_routine | `objectscript_routine/scanner.c`, deserialize | guard `memcpy(payload, buffer, length)` with `if (length > 0)` | UBSan: Tree-sitter passes a NULL buffer and zero length for empty scanner state; skip the zero-length copy. Upstream pin `25cf0f67a380` (v1.11.0) still carries the unguarded copy |
|
||||
| objectscript_udl | `objectscript_udl/scanner.c`, deserialize | guard `memcpy(payload, buffer, length)` with `if (length > 0)` | UBSan: Tree-sitter passes a NULL buffer and zero length for empty scanner state; skip the zero-length copy. Upstream pin `25cf0f67a380` (v1.11.0) still carries the unguarded copy |
|
||||
| purescript | `purescript/scanner.c`, serialize | guard `memcpy(buffer, indents->data, to_copy)` with `if (to_copy > 0)` | UBSan: zero-length `memcpy` with a NULL/0-size source when the indent vector is empty (formal UB, harmless) |
|
||||
| plsql | `plsql/parser.c`, include | `#include <tree_sitter/parser.h>` → `#include "tree_sitter/parser.h"` | The older ABI-14 generator emits angle brackets; every other vendored grammar uses the quoted form, which resolves the per-grammar `tree_sitter/` header from the including file's directory |
|
||||
| properties | `properties/scanner.c`, whole file | move the `reached_eof` flag out of a file-scope `static bool` and into a per-parser payload allocated by `..._external_scanner_create()` | **Data race, and the only one of its kind in 103 vendored scanners.** That flag is how the scanner refuses to emit a second end-of-input `FAKE_EOL`, and upstream keeps it in ONE process-wide object. With several worker threads indexing `.properties` files at once, whichever reached EOF first set the flag, so another thread's `!reached_eof` was false and its parse never received the `FAKE_EOL` it needed to finish — it sat at end-of-input asking for a token that would never arrive. Measured 2026-09-19: ~106 M parse operations on a 253-byte file before the per-file budget stopped it, after which the file was dropped from the graph and java/kotlin indexes differed run to run. Interleaved A/B under identical load, 40 runs each: unpatched 6 stalls, patched 0. The serialize/deserialize wire format is unchanged (the state is still the returned length). Upstream `6310671b24d4` still carries the static, so a re-vendor must re-apply this |
|
||||
@@ -208,8 +209,8 @@ row instead.
|
||||
| nickel | 15 | nickel-lang/tree-sitter-nickel | `b5b6cc3bc7b9` | VERIFIED-BOTH | ✅ |
|
||||
| nix | 13 | nix-community/tree-sitter-nix | `eabf96807ea4` | VERIFIED-BOTH | ✅ |
|
||||
| objc | 14 | tree-sitter-grammars/tree-sitter-objc | `181a81b8f23a` | VERIFIED-NVIM | ✅ |
|
||||
| objectscript_routine | 15 | intersystems/tree-sitter-objectscript | `a7ffcdf2de8e` | vendor-maintained | ✅ |
|
||||
| objectscript_udl | 15 | intersystems/tree-sitter-objectscript | `a7ffcdf2de8e` | vendor-maintained | ✅ |
|
||||
| objectscript_routine | 15 | intersystems/tree-sitter-objectscript | `25cf0f67a380` | vendor-maintained | ✅ |
|
||||
| objectscript_udl | 15 | intersystems/tree-sitter-objectscript | `25cf0f67a380` | vendor-maintained | ✅ |
|
||||
| ocaml | 14 | tree-sitter/tree-sitter-ocaml | `5a979b3ec7f1` | VERIFIED-BOTH | ✅ |
|
||||
| odin | 14 | tree-sitter-grammars/tree-sitter-odin | `d2ca8efb4487` | VERIFIED-BOTH | ✅ |
|
||||
| perl | 15 | tree-sitter-perl/tree-sitter-perl | `c3e17b31179b` | UPSTREAM-RELEASE | ✅ |
|
||||
|
||||
@@ -40,6 +40,10 @@ enum ObjectScript_Core_Scanner_TokenType {
|
||||
_DO_TERMINATION,
|
||||
_BOL_BLOCK,
|
||||
_POST_CONDITIONAL_END,
|
||||
COMPILED_HEADER,
|
||||
EXTERNAL_METHOD_BODY_CONTENT,
|
||||
STATEMENT_IN_MACRO,
|
||||
_END_IF_END,
|
||||
/* Max token type */
|
||||
OBJECTSCRIPT_CORE_TOKEN_TYPE_MAX
|
||||
|
||||
@@ -83,7 +87,11 @@ static const char* token_names[] = {
|
||||
"BOL_EXTRA",
|
||||
"_DO_TERMINATION",
|
||||
"_BOL_BLOCK",
|
||||
"_POST_CONDITIONAL_END"
|
||||
"_POST_CONDITIONAL_END",
|
||||
"COMPILED_HEADER",
|
||||
"EXTERNAL_METHOD_BODY_CONTENT",
|
||||
"STATEMENT_IN_MACRO",
|
||||
"_END_IF_END"
|
||||
};
|
||||
|
||||
#if 0
|
||||
@@ -175,14 +183,19 @@ struct ObjectScript_Core_Scanner {
|
||||
int32_t js_marker_buffer_reversed[MARKER_BUFFER_MAX_LEN];
|
||||
int js_marker_buffer_reversed_len;
|
||||
bool is_rtn_dot;
|
||||
// Allows nested old-style commands to close before the same `. }` line
|
||||
// is emitted as a dotted block end.
|
||||
bool pending_dotted_block_end;
|
||||
bool is_python;
|
||||
bool is_text;
|
||||
};
|
||||
|
||||
static inline bool is_label_char(int32_t c) {
|
||||
return iswalnum(c) || c == '%';
|
||||
return iswalnum(c) || c == '%' || c >= 0x80;
|
||||
}
|
||||
|
||||
static inline bool valid_tag_char(int32_t c) {
|
||||
return iswalnum(c) || c == '%' || c == '.';
|
||||
return iswalnum(c) || c == '%' || c == '.' || c >= 0x80;
|
||||
}
|
||||
|
||||
static inline int32_t ascii_toupper_i32(int32_t c) {
|
||||
@@ -237,6 +250,7 @@ static bool is_statement_or_class_keyword(const int32_t *text, uint32_t len) {
|
||||
if (ascii_upper_eq(text, len, "MVC") || ascii_upper_eq(text, len, "MVCRT")) return true;
|
||||
if (ascii_upper_eq(text, len, "MVDIM")) return true;
|
||||
if (ascii_upper_eq(text, len, "MVPRINT")) return true;
|
||||
if (ascii_upper_eq(text, len, "MVP")) return true;
|
||||
if (ascii_upper_eq(text, len, "Q") || ascii_upper_eq(text, len, "QUIT")) return true;
|
||||
if (ascii_upper_eq(text, len, "G") || ascii_upper_eq(text, len, "GOTO")) return true;
|
||||
if (ascii_upper_eq(text, len, "H") || ascii_upper_eq(text, len, "HALT") ||
|
||||
@@ -253,6 +267,10 @@ static bool is_statement_or_class_keyword(const int32_t *text, uint32_t len) {
|
||||
if (ascii_upper_eq(text, len, "ZDELETE")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZERASE")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZETRAP")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZEDIT")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZED")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZE")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZEDI")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZFILE")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZGO")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZHTRAP")) return true;
|
||||
@@ -281,6 +299,7 @@ static bool is_statement_or_class_keyword(const int32_t *text, uint32_t len) {
|
||||
if (ascii_upper_eq(text, len, "CLASS")) return true;
|
||||
if (ascii_upper_eq(text, len, "METHOD")) return true;
|
||||
if (ascii_upper_eq(text, len, "CLASSMETHOD")) return true;
|
||||
if (ascii_upper_eq(text, len, "CLIENTMETHOD")) return true;
|
||||
if (ascii_upper_eq(text, len, "PROPERTY")) return true;
|
||||
if (ascii_upper_eq(text, len, "PARAMETER")) return true;
|
||||
if (ascii_upper_eq(text, len, "RELATIONSHIP")) return true;
|
||||
@@ -294,32 +313,339 @@ static bool is_statement_or_class_keyword(const int32_t *text, uint32_t len) {
|
||||
if (ascii_upper_eq(text, len, "IMPORT")) return true;
|
||||
if (ascii_upper_eq(text, len, "INCLUDE")) return true;
|
||||
if (ascii_upper_eq(text, len, "INCLUDEGENERATOR")) return true;
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
static bool ObjectScript_Core_Scanner_lex_fenced_text(
|
||||
TSLexer *lexer,
|
||||
enum ObjectScript_Core_Scanner_TokenType desired_symbol,
|
||||
char l_delim,
|
||||
char r_delim) {
|
||||
int leftRightDiff = 1;
|
||||
// Consumes an ObjectScript interpolation (#(...)# or ##(...)##) embedded in
|
||||
// &html/&xml text. Assumes lookahead is '#'. Inside the parens, '>' and '\''
|
||||
// are ObjectScript operators, not XML syntax, so they must not be treated as
|
||||
// tag ends or attribute quotes.
|
||||
static void skip_cos_interpolation(TSLexer *lexer) {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '#') {
|
||||
advance(lexer);
|
||||
}
|
||||
if (lexer->lookahead != '(') {
|
||||
return;
|
||||
}
|
||||
int depth = 0;
|
||||
while (!lexer->eof(lexer) && lexer->lookahead != '\n') {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == '"') {
|
||||
advance(lexer);
|
||||
while (!lexer->eof(lexer) && lexer->lookahead != '\n') {
|
||||
if (lexer->lookahead == '"') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != '"') {
|
||||
break;
|
||||
}
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
advance(lexer);
|
||||
if (c == '(') {
|
||||
depth++;
|
||||
} else if (c == ')') {
|
||||
depth--;
|
||||
if (depth == 0) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
// consume the closing # (or ##)
|
||||
if (lexer->lookahead == '#') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '#') {
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Consumes the rest of an XML tag up to (not including) the closing '>'.
|
||||
// A quote only opens an attribute value directly after '=', so stray quotes in
|
||||
// hand-written HTML (e.g. height=24") don't swallow the rest of the file.
|
||||
static void skip_xml_tag_body(TSLexer *lexer) {
|
||||
int32_t prev = 0;
|
||||
while (!lexer->eof(lexer) && lexer->lookahead != '>') {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == '#') {
|
||||
skip_cos_interpolation(lexer);
|
||||
prev = '#';
|
||||
continue;
|
||||
}
|
||||
if ((c == '"' || c == '\'') && prev == '=') {
|
||||
advance(lexer);
|
||||
while (!lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == '#') {
|
||||
skip_cos_interpolation(lexer);
|
||||
continue;
|
||||
}
|
||||
if (lexer->lookahead == c) {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == c) {
|
||||
advance(lexer);
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
} else {
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
prev = c;
|
||||
continue;
|
||||
}
|
||||
if (!iswspace(c)) {
|
||||
prev = c;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
|
||||
static bool lex_fenced_text(TSLexer *lexer,
|
||||
enum ObjectScript_Core_Scanner_TokenType desired_symbol, int32_t l_delim,
|
||||
int32_t r_delim, bool is_xml, bool has_python_comment, bool is_text) {
|
||||
int depth = 1;
|
||||
const char *comment_start = "<!--";
|
||||
const char *code_start = "<![CDATA[";
|
||||
const char *code_end = "]]>";
|
||||
const char *comment_end = "-->";
|
||||
const char *script_start = "script";
|
||||
const char *script_end = "</script>";
|
||||
while (iswspace(lexer->lookahead) || lexer->lookahead == 0x2028 || lexer->lookahead == 0x2029) {
|
||||
advance(lexer);
|
||||
}
|
||||
if (lexer->lookahead == '<') {
|
||||
is_xml = true;
|
||||
}
|
||||
while(iswalnum(lexer->lookahead) || iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
while (!lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == r_delim) {
|
||||
leftRightDiff -= 1;
|
||||
} else if (lexer->lookahead == l_delim) {
|
||||
leftRightDiff += 1;
|
||||
while (iswspace(lexer->lookahead) || lexer->lookahead == 0x2028 || lexer->lookahead == 0x2029) {
|
||||
advance(lexer);
|
||||
}
|
||||
if (leftRightDiff == 0) {
|
||||
lexer->result_symbol = desired_symbol;
|
||||
return true;
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == '#' && is_xml) {
|
||||
skip_cos_interpolation(lexer);
|
||||
continue;
|
||||
}
|
||||
if (c == '<' && is_xml) {
|
||||
bool comment = true;
|
||||
bool is_code = false;
|
||||
for (unsigned i = 0; i < 4; i++) {
|
||||
if (i > 1) {
|
||||
is_code = true;
|
||||
}
|
||||
if (lexer->lookahead != comment_start[i]) {
|
||||
comment = false;
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
if (comment) {
|
||||
bool is_comment_end = false;
|
||||
while (!lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == '-') {
|
||||
is_comment_end = true;
|
||||
for (unsigned i = 0; i < 3; i++) {
|
||||
if (lexer->lookahead != comment_end[i]) {
|
||||
is_comment_end = false;
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
if (is_comment_end) {
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
continue;
|
||||
} else if (is_code) {
|
||||
for (unsigned i = 2; i < 9; i++) {
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
if (lexer->lookahead != code_start[i]) {
|
||||
is_code = false;
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
// [CDATA[
|
||||
if (is_code) {
|
||||
bool is_code_end = false;
|
||||
while (!lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == ']') {
|
||||
is_code_end = true;
|
||||
for (unsigned i = 0; i < 3; i++) {
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
if (lexer->lookahead != code_end[i]) {
|
||||
is_code_end = false;
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
if (is_code_end) {
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
} else if (towlower(lexer->lookahead) == 's') {
|
||||
bool is_script = true;
|
||||
for (unsigned i = 0; i < 6; i++) {
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
if (towlower(lexer->lookahead) != script_start[i]) {
|
||||
is_script = false;
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
skip_xml_tag_body(lexer);
|
||||
if (lexer->lookahead == '>') {
|
||||
advance(lexer);
|
||||
}
|
||||
if (!is_script) {
|
||||
continue;
|
||||
}
|
||||
|
||||
bool is_script_end = false;
|
||||
while (!lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == '<') {
|
||||
is_script_end = true;
|
||||
for (unsigned i = 0; i < 9; i++) {
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
if (towlower(lexer->lookahead) != script_end[i]) {
|
||||
is_script_end = false;
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
if (is_script_end) {
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
continue;
|
||||
|
||||
}
|
||||
|
||||
else {
|
||||
skip_xml_tag_body(lexer);
|
||||
// consume the >
|
||||
advance(lexer);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
else if (!is_xml && (c == '"' || (c == '\'' && !is_text))) {
|
||||
int32_t quote = c;
|
||||
advance(lexer);
|
||||
|
||||
// Detect a triple-quoted string.
|
||||
bool triple = false;
|
||||
if (lexer->lookahead == quote) {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == quote) {
|
||||
advance(lexer);
|
||||
triple = true;
|
||||
} else {
|
||||
// Empty string literal ('' or ""); nothing more to skip.
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
// Consume the string contents.
|
||||
while (!lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == '\\') {
|
||||
advance(lexer);
|
||||
if (!lexer->eof(lexer)) {
|
||||
advance(lexer);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (lexer->lookahead == quote) {
|
||||
advance(lexer);
|
||||
if (!triple) {
|
||||
break;
|
||||
}
|
||||
if (lexer->lookahead == quote) {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == quote) {
|
||||
advance(lexer);
|
||||
break;
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (!triple && lexer->lookahead == '\n') {
|
||||
// Unterminated single-quoted string; stop before consuming the
|
||||
// newline so brace counting resumes on the next line.
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
continue;
|
||||
|
||||
} if (c == '#' && !is_xml && has_python_comment) {
|
||||
// Line comment: consume to end of line.
|
||||
while (!lexer->eof(lexer) && lexer->lookahead != '\n') {
|
||||
advance(lexer);
|
||||
}
|
||||
continue;
|
||||
} if (c == '/' && !is_xml) {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '/') {
|
||||
while (lexer->lookahead != 0 && lexer->lookahead != '\n' && lexer->lookahead != 0x2028 &&
|
||||
lexer->lookahead != 0x2029) {
|
||||
advance(lexer);
|
||||
}
|
||||
continue;
|
||||
} else if (lexer->lookahead == '*') {
|
||||
advance(lexer);
|
||||
while (!lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == '*') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '/') {
|
||||
advance(lexer);
|
||||
break;
|
||||
}
|
||||
}
|
||||
else {
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
if (c == r_delim) {
|
||||
depth--;
|
||||
if (depth == 0) {
|
||||
lexer->result_symbol = desired_symbol;
|
||||
return true;
|
||||
}
|
||||
} else if (c == l_delim) {
|
||||
depth++;
|
||||
}
|
||||
|
||||
advance(lexer);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
|
||||
static bool ObjectScript_Core_Scanner_lex_marker_fenced_text(
|
||||
TSLexer *lexer,
|
||||
enum ObjectScript_Core_Scanner_TokenType desired_symbol,
|
||||
@@ -366,8 +692,35 @@ static inline bool is_objectscript_special_symbol_i32(int32_t c) {
|
||||
c == '.' ;
|
||||
}
|
||||
|
||||
static bool lex_compiled_header(TSLexer *lexer) {
|
||||
lexer->mark_end(lexer);
|
||||
while(!lexer->eof(lexer) && lexer->lookahead != '\n') {
|
||||
advance(lexer);
|
||||
}
|
||||
if (lexer->lookahead != '\n') return false;
|
||||
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != '%') return false;
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != 'R') return false;
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != 'O') return false;
|
||||
advance(lexer);
|
||||
|
||||
lexer->result_symbol = COMPILED_HEADER;
|
||||
int newline_count = 0;
|
||||
while(newline_count < 2 && !lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == '\n') {
|
||||
newline_count++;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool ObjectScript_Core_Scanner_lex_pound_if_special_case(TSLexer *lexer) {
|
||||
if (lexer->lookahead != '0') return false;
|
||||
// if (lexer->lookahead != '0') return false;
|
||||
|
||||
uint32_t depth = 1;
|
||||
bool at_line_start = false;
|
||||
@@ -464,6 +817,15 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
if (valid_symbols[SENTINEL]) {
|
||||
return false;
|
||||
}
|
||||
|
||||
if (scanner->pending_dotted_block_end) {
|
||||
if (lexer->lookahead == '.' && valid_symbols[_TERMINATION]) {
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = _TERMINATION;
|
||||
return true;
|
||||
}
|
||||
scanner->pending_dotted_block_end = false;
|
||||
}
|
||||
scanner-> is_rtn_dot = false;
|
||||
|
||||
// this parses any line that ends in ##continue for
|
||||
@@ -474,7 +836,11 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
static const char pattern[] = "##continue";
|
||||
static const int len = sizeof(pattern)-1;
|
||||
int pos = 0;
|
||||
if (!lexer->eof(lexer) && !iswspace(lexer->lookahead)) {
|
||||
// _STATEMENT_TERMINATION is only valid right after the macro name/args,
|
||||
// where the value must be separated by whitespace. On a line continued
|
||||
// by ##continue it is not valid, and the line may start in column 0.
|
||||
bool is_continued_line = !valid_symbols[_STATEMENT_TERMINATION];
|
||||
if (!is_continued_line && !lexer->eof(lexer) && !iswspace(lexer->lookahead)) {
|
||||
scanner->terminated_newline = false;
|
||||
return false;
|
||||
}
|
||||
@@ -487,6 +853,21 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
|
||||
if (pos == len) {
|
||||
advance(lexer);
|
||||
// Allow a trailing ##; comment, e.g. "##continue ##; comment"
|
||||
while (lexer->lookahead == ' ' || lexer->lookahead == '\t') {
|
||||
advance(lexer);
|
||||
}
|
||||
if (lexer->lookahead == '#') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '#') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == ';') {
|
||||
while (!lexer->eof(lexer) && lexer->lookahead != '\n') {
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
int new_line_count = 0;
|
||||
while(iswspace(lexer->lookahead) && new_line_count<1) {
|
||||
if(lexer->lookahead=='\n') {
|
||||
@@ -528,6 +909,15 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
}
|
||||
}
|
||||
|
||||
if (valid_symbols[_END_IF_END]) {
|
||||
while (lexer->lookahead != '\n' && !lexer->eof(lexer)) {
|
||||
advance(lexer);
|
||||
}
|
||||
lexer->result_symbol = _END_IF_END;
|
||||
scanner->terminated_newline = true;
|
||||
return true;
|
||||
}
|
||||
|
||||
if (valid_symbols[EMBEDDED_JS_SPECIAL_CASE_COMPLETE]) {
|
||||
int i = 0;
|
||||
while (i<scanner->js_marker_buffer_reversed_len) {
|
||||
@@ -604,6 +994,32 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
return true;
|
||||
}
|
||||
|
||||
if (valid_symbols[STATEMENT_IN_MACRO]) {
|
||||
lexer->mark_end(lexer);
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
int32_t ident[96];
|
||||
uint32_t len = 0;
|
||||
while (valid_tag_char(lexer->lookahead)) {
|
||||
if (len < sizeof(ident) / sizeof(ident[0])) ident[len++] = lexer->lookahead;
|
||||
advance(lexer);
|
||||
}
|
||||
if (is_statement_or_class_keyword(ident, len)) {
|
||||
if (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
if (valid_tag_char(lexer->lookahead)) {
|
||||
lexer->result_symbol = STATEMENT_IN_MACRO;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
if (valid_symbols[EMBEDDED_SQL_REVERSE_MARKER]) {
|
||||
while (scanner->sql_marker_buffer_len >0 && !lexer->eof(lexer)) {
|
||||
int32_t expected = scanner->sql_marker_buffer[scanner->sql_marker_buffer_len - 1];
|
||||
@@ -680,7 +1096,7 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
return true;
|
||||
}
|
||||
// an argument that is exactly one space after the keyword
|
||||
if (count == 1 && (iswalnum(lexer->lookahead) || is_objectscript_special_symbol_i32(lexer->lookahead) || lexer->lookahead == '_') && valid_symbol_one_space) {
|
||||
if (count == 1 && (is_label_char(lexer->lookahead) || is_objectscript_special_symbol_i32(lexer->lookahead) || lexer->lookahead == '_' || (lexer->lookahead == '{' && !valid_symbol_argumentless_loop)) && valid_symbol_one_space) {
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = _IMMEDIATE_SINGLE_WHITESPACE_FOLLOWED_BY_NON_WHITESPACE;
|
||||
scanner->terminated_newline = false;
|
||||
@@ -688,7 +1104,7 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
}
|
||||
|
||||
// argumentless command has >= 2 spaces after the keyword
|
||||
else if (count >=2 && (iswalnum(lexer->lookahead) || is_objectscript_special_symbol_i32(lexer->lookahead)) && valid_symbol_argumentless_command_end) {
|
||||
else if (count >=2 && (is_label_char(lexer->lookahead) || is_objectscript_special_symbol_i32(lexer->lookahead) || lexer->lookahead == '&') && valid_symbol_argumentless_command_end) {
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = _ARGUMENTLESS_COMMAND_END;
|
||||
scanner->terminated_newline = false;
|
||||
@@ -706,7 +1122,7 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
else if (lexer->lookahead == '#') {
|
||||
lexer->mark_end(lexer);
|
||||
advance(lexer);
|
||||
if (towlower(lexer->lookahead) == 'e' && valid_symbols[TAG] && scanner->special_pound_if_mode && lexer->get_column(lexer) == 1) {
|
||||
if (towlower(lexer->lookahead) == 'e' && (valid_symbols[TAG] || valid_symbols[TAG_END_IF]) && scanner->special_pound_if_mode) {
|
||||
int32_t directive[16];
|
||||
uint32_t len = 0;
|
||||
while (len < (sizeof(directive) / sizeof(directive[0])) &&
|
||||
@@ -720,6 +1136,13 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
if (scanner->special_pound_if_mode_if_depth<1) {
|
||||
scanner->special_pound_if_mode=false;
|
||||
}
|
||||
// Anything after #endif on the same line is ignored (e.g. #endif ArgFor),
|
||||
// but trailing comments are left for the comment rules
|
||||
while (lexer->lookahead != '\n' && !lexer->eof(lexer) &&
|
||||
lexer->lookahead != ';' && lexer->lookahead != '/' &&
|
||||
lexer->lookahead != '#') {
|
||||
advance(lexer);
|
||||
}
|
||||
lexer->result_symbol = TAG_END_IF;
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
@@ -892,7 +1315,37 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
}
|
||||
|
||||
else if (lexer->lookahead == '{' && valid_symbol_argumentless_loop) {
|
||||
bool valid_symbol_zw_block = valid_symbols[_ZW_BLOCK];
|
||||
lexer->mark_end(lexer);
|
||||
if (count == 1 && valid_symbol_zw_block) {
|
||||
advance(lexer);
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
int32_t ident[96];
|
||||
uint32_t len = 0;
|
||||
bool potential_tag = false;
|
||||
if (lexer->get_column(lexer) == 0) {
|
||||
potential_tag = true;
|
||||
}
|
||||
while (valid_tag_char(lexer->lookahead) || is_objectscript_special_symbol_i32(lexer->lookahead)) {
|
||||
if (len < sizeof(ident) / sizeof(ident[0])) ident[len++] = lexer->lookahead;
|
||||
advance(lexer);
|
||||
}
|
||||
if (!is_statement_or_class_keyword(ident, len) && len > 0 && !potential_tag) {
|
||||
if (len > 3) {
|
||||
if (ident[0] == '$' && ident[1] == '$' && ident[2] == '$') {
|
||||
lexer->result_symbol = _ARGUMENTLESS_LOOP;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
lexer->result_symbol = _ZW_BLOCK;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
lexer->result_symbol = _ARGUMENTLESS_LOOP;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
@@ -948,26 +1401,60 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
}
|
||||
|
||||
else if (lexer->lookahead == '{') {
|
||||
// this token is for commands that allow blocks as part of their command ex: w {"hi": "bye"}
|
||||
if (count == 1 && valid_symbols[_ZW_BLOCK]) {
|
||||
lexer->result_symbol = _ZW_BLOCK;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
|
||||
if (valid_symbols[DOTTED_STATEMENT_BLOCK]) {
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = DOTTED_STATEMENT_BLOCK;
|
||||
}
|
||||
|
||||
else if (valid_symbol_argumentless_loop) {
|
||||
bool valid_symbol_zw_block = valid_symbols[_ZW_BLOCK];
|
||||
lexer->mark_end(lexer);
|
||||
if (count == 1 && valid_symbol_zw_block) {
|
||||
advance(lexer);
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
int32_t ident[96];
|
||||
uint32_t len = 0;
|
||||
bool potential_tag = false;
|
||||
if (lexer->get_column(lexer) == 0) {
|
||||
potential_tag = true;
|
||||
}
|
||||
while (valid_tag_char(lexer->lookahead) || is_objectscript_special_symbol_i32(lexer->lookahead)) {
|
||||
if (len < sizeof(ident) / sizeof(ident[0])) ident[len++] = lexer->lookahead;
|
||||
advance(lexer);
|
||||
}
|
||||
if (!is_statement_or_class_keyword(ident, len) && len > 0 && !potential_tag) {
|
||||
if (len > 3) {
|
||||
if (ident[0] == '$' && ident[1] == '$' && ident[2] == '$') {
|
||||
lexer->result_symbol = _ARGUMENTLESS_LOOP;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
lexer->result_symbol = _ZW_BLOCK;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
lexer->result_symbol = _ARGUMENTLESS_LOOP;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
} else if (count == 1 && valid_symbols[_ZW_BLOCK]) {
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = _ZW_BLOCK;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
if (lexer->lookahead == ')' && valid_symbol_termination) {
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = _TERMINATION;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
|
||||
if (count > 0 && valid_symbols[_WHITESPACE]) {
|
||||
|
||||
// there can't be a space between a keyword and the post-conditional,
|
||||
@@ -1021,11 +1508,18 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
advance(lexer);
|
||||
}
|
||||
if (lexer->lookahead == '}') {
|
||||
if (valid_symbols[_TERMINATION]) {
|
||||
scanner->pending_dotted_block_end = true;
|
||||
lexer->result_symbol = _TERMINATION;
|
||||
return true;
|
||||
}
|
||||
lexer->result_symbol = _BOL_BLOCK;
|
||||
scanner->pending_dotted_block_end = false;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
lexer->result_symbol = _BOL;
|
||||
scanner->pending_dotted_block_end = false;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
@@ -1034,9 +1528,10 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
while (lexer->lookahead == ' ' || lexer->lookahead == '\t') advance(lexer);
|
||||
|
||||
if (lexer->lookahead == '.') {
|
||||
lexer->result_symbol = _BOL;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
lexer->result_symbol = _BOL;
|
||||
scanner->pending_dotted_block_end = false;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
else {
|
||||
lexer->mark_end(lexer);
|
||||
@@ -1087,6 +1582,44 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
return true;
|
||||
}
|
||||
|
||||
if (scanner->routine_token_mode && valid_symbols[COMPILED_HEADER]) {
|
||||
lexer->mark_end(lexer);
|
||||
while (lexer->lookahead != '\n' && !lexer->eof(lexer)) advance(lexer);
|
||||
if (lexer->lookahead == '\n') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != '%') {
|
||||
lexer->result_symbol = TAG;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != 'R') {
|
||||
lexer->result_symbol = TAG;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != 'O') {
|
||||
lexer->result_symbol = TAG;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
lexer->result_symbol = COMPILED_HEADER;
|
||||
int newline_count = 0;
|
||||
while(newline_count < 2 && !lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == '\n') {
|
||||
newline_count++;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
}
|
||||
lexer->result_symbol = TAG;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
|
||||
if (!scanner->column1_statement_mode) {
|
||||
lexer->result_symbol = TAG;
|
||||
scanner->terminated_newline = false;
|
||||
@@ -1107,18 +1640,54 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
scanner->terminated_newline = false;
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
} else if (lexer->lookahead == '(') {
|
||||
lexer->result_symbol = TAG;
|
||||
scanner->terminated_newline = false;
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
} else if (iswspace(lexer->lookahead)) {
|
||||
lexer->mark_end(lexer);
|
||||
advance(lexer);
|
||||
int32_t ident[96];
|
||||
uint32_t len = 0;
|
||||
while (valid_tag_char(lexer->lookahead)) {
|
||||
if (len < sizeof(ident) / sizeof(ident[0])) ident[len++] = lexer->lookahead;
|
||||
advance(lexer);
|
||||
}
|
||||
if (is_statement_or_class_keyword(ident, len)) {
|
||||
lexer->result_symbol = TAG;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
lexer->mark_end(lexer);
|
||||
return false;
|
||||
}
|
||||
else if (valid_symbols[ANGLED_BRACKET_FENCED_TEXT]) {
|
||||
bool ok = ObjectScript_Core_Scanner_lex_fenced_text(
|
||||
lexer, ANGLED_BRACKET_FENCED_TEXT, '<', '>');
|
||||
return ok;
|
||||
scanner->is_python = false;
|
||||
scanner->is_text = false;
|
||||
return lex_fenced_text(
|
||||
lexer, ANGLED_BRACKET_FENCED_TEXT, '<', '>', true, scanner->is_python, false);
|
||||
}
|
||||
else if (valid_symbols[PAREN_FENCED_TEXT]) {
|
||||
bool ok = ObjectScript_Core_Scanner_lex_fenced_text(
|
||||
lexer, PAREN_FENCED_TEXT, '(', ')');
|
||||
return ok;
|
||||
scanner->is_python = false;
|
||||
scanner->is_text = false;
|
||||
return lex_fenced_text(
|
||||
lexer, PAREN_FENCED_TEXT, '(', ')', false, scanner->is_python, false);
|
||||
}
|
||||
|
||||
else if (valid_symbols[EXTERNAL_METHOD_BODY_CONTENT]) {
|
||||
// A valid method_body is one that is whose text fences
|
||||
// are evenly balanced (so far only { })
|
||||
// e.g. VALID: {{{ [^{}]* }}} INVALID: {{{ [^{}]* }
|
||||
bool is_python = scanner->is_python;
|
||||
bool is_text = scanner->is_text;
|
||||
scanner->is_python=false;
|
||||
scanner->is_text=false;
|
||||
return lex_fenced_text(lexer, EXTERNAL_METHOD_BODY_CONTENT, '{',
|
||||
'}', false, is_python, is_text);
|
||||
}
|
||||
|
||||
else if (valid_symbols[_LINE_COMMENT_INNER]) {
|
||||
@@ -1191,7 +1760,7 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
lexer->result_symbol = _WHITESPACE;
|
||||
return true;
|
||||
}
|
||||
else if (valid_symbols[TAG] && scanner->special_pound_if_mode) {
|
||||
else if ((valid_symbols[TAG] || valid_symbols[TAG_END_IF]) && scanner->special_pound_if_mode) {
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
@@ -1211,6 +1780,13 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
if (scanner->special_pound_if_mode_if_depth<1) {
|
||||
scanner->special_pound_if_mode=false;
|
||||
}
|
||||
// Anything after #endif on the same line is ignored (e.g. #endif ArgFor),
|
||||
// but trailing comments are left for the comment rules
|
||||
while (lexer->lookahead != '\n' && !lexer->eof(lexer) &&
|
||||
lexer->lookahead != ';' && lexer->lookahead != '/' &&
|
||||
lexer->lookahead != '#') {
|
||||
advance(lexer);
|
||||
}
|
||||
lexer->result_symbol = TAG_END_IF;
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
@@ -1226,9 +1802,13 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
}
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = BOL_EXTRA;
|
||||
scanner->pending_dotted_block_end = false;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
if (valid_symbols[COMPILED_HEADER] && lex_compiled_header(lexer)) {
|
||||
return true;
|
||||
}
|
||||
scanner->terminated_newline = false;
|
||||
return false;
|
||||
}
|
||||
@@ -1242,4 +1822,7 @@ static void ObjectScript_Core_Scanner_init(struct ObjectScript_Core_Scanner *sca
|
||||
scanner->special_pound_if_mode = false;
|
||||
scanner->special_pound_if_mode_if_depth = 0;
|
||||
scanner->is_rtn_dot = false;
|
||||
scanner->pending_dotted_block_end = false;
|
||||
scanner->is_python = false;
|
||||
scanner->is_text = false;
|
||||
}
|
||||
|
||||
+1221243
-928798
File diff suppressed because it is too large
Load Diff
+1
-35
@@ -4,8 +4,7 @@
|
||||
|
||||
// Keep this in sync with grammar externals.
|
||||
enum TokenType {
|
||||
COMPILED_HEADER = OBJECTSCRIPT_CORE_TOKEN_TYPE_MAX,
|
||||
RTN_DOT,
|
||||
RTN_DOT = OBJECTSCRIPT_CORE_TOKEN_TYPE_MAX,
|
||||
};
|
||||
|
||||
struct ObjectScript_Routine_Scanner {
|
||||
@@ -29,33 +28,6 @@ static bool lex_rtn_dot(TSLexer *lexer) {
|
||||
return false;
|
||||
}
|
||||
|
||||
static bool lex_compiled_header(TSLexer *lexer) {
|
||||
lexer->mark_end(lexer);
|
||||
while(!lexer->eof(lexer) && lexer->lookahead != '\n') {
|
||||
advance(lexer);
|
||||
}
|
||||
if (lexer->lookahead != '\n') return false;
|
||||
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != '%') return false;
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != 'R') return false;
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != 'O') return false;
|
||||
advance(lexer);
|
||||
|
||||
lexer->result_symbol = COMPILED_HEADER;
|
||||
int newline_count = 0;
|
||||
while(newline_count < 2 && !lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == '\n') {
|
||||
newline_count++;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool scan(void *payload, TSLexer *lexer, const bool *valid_symbols) {
|
||||
struct ObjectScript_Routine_Scanner *scanner =
|
||||
(struct ObjectScript_Routine_Scanner *)payload;
|
||||
@@ -77,10 +49,6 @@ static bool scan(void *payload, TSLexer *lexer, const bool *valid_symbols) {
|
||||
return true;
|
||||
}
|
||||
|
||||
if (valid_symbols[COMPILED_HEADER] && lex_compiled_header(lexer)) {
|
||||
return true;
|
||||
}
|
||||
|
||||
if (valid_symbols[RTN_DOT] && lex_rtn_dot(lexer)) {
|
||||
return true;
|
||||
}
|
||||
@@ -92,8 +60,6 @@ void *tree_sitter_objectscript_routine_external_scanner_create() {
|
||||
struct ObjectScript_Routine_Scanner *scanner =
|
||||
(struct ObjectScript_Routine_Scanner *)calloc(
|
||||
1, sizeof(struct ObjectScript_Routine_Scanner));
|
||||
// scanner->saw_compiled_header = false;
|
||||
// scanner->saw_routine_header = false;
|
||||
ObjectScript_Core_Scanner_init(&scanner->core_scanner);
|
||||
scanner->core_scanner.column1_statement_mode = false;
|
||||
scanner->core_scanner.routine_token_mode = true;
|
||||
|
||||
@@ -50,12 +50,18 @@ extern "C" {
|
||||
/// memory allocated for the array's contents.
|
||||
#define array_clear(self) ((self)->size = 0)
|
||||
|
||||
#ifdef __cplusplus
|
||||
#define _array__cast(self, expr) (decltype((self)->contents))(expr)
|
||||
#else
|
||||
#define _array__cast(self, expr) (expr)
|
||||
#endif
|
||||
|
||||
/// Reserve `new_capacity` elements of space in the array. If `new_capacity` is
|
||||
/// less than the array's current capacity, this function has no effect.
|
||||
#define array_reserve(self, new_capacity) \
|
||||
((self)->contents = _array__reserve( \
|
||||
(void *)(self)->contents, &(self)->capacity, \
|
||||
array_elem_size(self), new_capacity) \
|
||||
#define array_reserve(self, new_capacity) \
|
||||
((self)->contents = _array__cast(self, _array__reserve( \
|
||||
(void *)(self)->contents, &(self)->capacity, \
|
||||
array_elem_size(self), new_capacity)) \
|
||||
)
|
||||
|
||||
/// Free any memory allocated for this array. Note that this does not free any
|
||||
@@ -71,10 +77,10 @@ extern "C" {
|
||||
/// Push a new `element` onto the end of the array.
|
||||
#define array_push(self, element) \
|
||||
do { \
|
||||
(self)->contents = _array__grow( \
|
||||
(self)->contents = _array__cast(self, _array__grow( \
|
||||
(void *)(self)->contents, (self)->size, &(self)->capacity, \
|
||||
1, array_elem_size(self) \
|
||||
); \
|
||||
)); \
|
||||
(self)->contents[(self)->size++] = (element); \
|
||||
} while(0)
|
||||
|
||||
@@ -83,10 +89,10 @@ extern "C" {
|
||||
#define array_grow_by(self, count) \
|
||||
do { \
|
||||
if ((count) == 0) break; \
|
||||
(self)->contents = _array__grow( \
|
||||
(self)->contents = _array__cast(self, _array__grow( \
|
||||
(self)->contents, (self)->size, &(self)->capacity, \
|
||||
count, array_elem_size(self) \
|
||||
); \
|
||||
)); \
|
||||
memset((self)->contents + (self)->size, 0, (count) * array_elem_size(self)); \
|
||||
(self)->size += (count); \
|
||||
} while (0)
|
||||
@@ -98,26 +104,26 @@ extern "C" {
|
||||
/// Append `count` elements to the end of the array, reading their values from the
|
||||
/// `contents` pointer.
|
||||
#define array_extend(self, count, other_contents) \
|
||||
(self)->contents = _array__splice( \
|
||||
((self)->contents = _array__cast(self, _array__splice( \
|
||||
(void*)(self)->contents, &(self)->size, &(self)->capacity, \
|
||||
array_elem_size(self), (self)->size, 0, count, other_contents \
|
||||
)
|
||||
)))
|
||||
|
||||
/// Remove `old_count` elements from the array starting at the given `index`. At
|
||||
/// the same index, insert `new_count` new elements, reading their values from the
|
||||
/// `new_contents` pointer.
|
||||
#define array_splice(self, _index, old_count, new_count, new_contents) \
|
||||
(self)->contents = _array__splice( \
|
||||
((self)->contents = _array__cast(self, _array__splice( \
|
||||
(void *)(self)->contents, &(self)->size, &(self)->capacity, \
|
||||
array_elem_size(self), _index, old_count, new_count, new_contents \
|
||||
)
|
||||
)))
|
||||
|
||||
/// Insert one `element` into the array at the given `index`.
|
||||
#define array_insert(self, _index, element) \
|
||||
(self)->contents = _array__splice( \
|
||||
((self)->contents = _array__cast(self, _array__splice( \
|
||||
(void *)(self)->contents, &(self)->size, &(self)->capacity, \
|
||||
array_elem_size(self), _index, 0, 1, &(element) \
|
||||
)
|
||||
)))
|
||||
|
||||
/// Remove one element from the array at the given `index`.
|
||||
#define array_erase(self, _index) \
|
||||
@@ -128,17 +134,17 @@ extern "C" {
|
||||
|
||||
/// Assign the contents of one array to another, reallocating if necessary.
|
||||
#define array_assign(self, other) \
|
||||
(self)->contents = _array__assign( \
|
||||
((self)->contents = _array__cast(self, _array__assign( \
|
||||
(void *)(self)->contents, &(self)->size, &(self)->capacity, \
|
||||
(const void *)(other)->contents, (other)->size, array_elem_size(self) \
|
||||
)
|
||||
)))
|
||||
|
||||
/// Swap one array with another
|
||||
#define array_swap(self, other) \
|
||||
do { \
|
||||
void *_array_swap_tmp = (void *)(self)->contents; \
|
||||
(self)->contents = (other)->contents; \
|
||||
(other)->contents = _array_swap_tmp; \
|
||||
(other)->contents = _array__cast(other, _array_swap_tmp); \
|
||||
_array__swap(&(self)->size, &(self)->capacity, \
|
||||
&(other)->size, &(other)->capacity); \
|
||||
} while (0)
|
||||
|
||||
@@ -40,6 +40,10 @@ enum ObjectScript_Core_Scanner_TokenType {
|
||||
_DO_TERMINATION,
|
||||
_BOL_BLOCK,
|
||||
_POST_CONDITIONAL_END,
|
||||
COMPILED_HEADER,
|
||||
EXTERNAL_METHOD_BODY_CONTENT,
|
||||
STATEMENT_IN_MACRO,
|
||||
_END_IF_END,
|
||||
/* Max token type */
|
||||
OBJECTSCRIPT_CORE_TOKEN_TYPE_MAX
|
||||
|
||||
@@ -83,7 +87,11 @@ static const char* token_names[] = {
|
||||
"BOL_EXTRA",
|
||||
"_DO_TERMINATION",
|
||||
"_BOL_BLOCK",
|
||||
"_POST_CONDITIONAL_END"
|
||||
"_POST_CONDITIONAL_END",
|
||||
"COMPILED_HEADER",
|
||||
"EXTERNAL_METHOD_BODY_CONTENT",
|
||||
"STATEMENT_IN_MACRO",
|
||||
"_END_IF_END"
|
||||
};
|
||||
|
||||
#if 0
|
||||
@@ -175,14 +183,19 @@ struct ObjectScript_Core_Scanner {
|
||||
int32_t js_marker_buffer_reversed[MARKER_BUFFER_MAX_LEN];
|
||||
int js_marker_buffer_reversed_len;
|
||||
bool is_rtn_dot;
|
||||
// Allows nested old-style commands to close before the same `. }` line
|
||||
// is emitted as a dotted block end.
|
||||
bool pending_dotted_block_end;
|
||||
bool is_python;
|
||||
bool is_text;
|
||||
};
|
||||
|
||||
static inline bool is_label_char(int32_t c) {
|
||||
return iswalnum(c) || c == '%';
|
||||
return iswalnum(c) || c == '%' || c >= 0x80;
|
||||
}
|
||||
|
||||
static inline bool valid_tag_char(int32_t c) {
|
||||
return iswalnum(c) || c == '%' || c == '.';
|
||||
return iswalnum(c) || c == '%' || c == '.' || c >= 0x80;
|
||||
}
|
||||
|
||||
static inline int32_t ascii_toupper_i32(int32_t c) {
|
||||
@@ -237,6 +250,7 @@ static bool is_statement_or_class_keyword(const int32_t *text, uint32_t len) {
|
||||
if (ascii_upper_eq(text, len, "MVC") || ascii_upper_eq(text, len, "MVCRT")) return true;
|
||||
if (ascii_upper_eq(text, len, "MVDIM")) return true;
|
||||
if (ascii_upper_eq(text, len, "MVPRINT")) return true;
|
||||
if (ascii_upper_eq(text, len, "MVP")) return true;
|
||||
if (ascii_upper_eq(text, len, "Q") || ascii_upper_eq(text, len, "QUIT")) return true;
|
||||
if (ascii_upper_eq(text, len, "G") || ascii_upper_eq(text, len, "GOTO")) return true;
|
||||
if (ascii_upper_eq(text, len, "H") || ascii_upper_eq(text, len, "HALT") ||
|
||||
@@ -253,6 +267,10 @@ static bool is_statement_or_class_keyword(const int32_t *text, uint32_t len) {
|
||||
if (ascii_upper_eq(text, len, "ZDELETE")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZERASE")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZETRAP")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZEDIT")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZED")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZE")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZEDI")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZFILE")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZGO")) return true;
|
||||
if (ascii_upper_eq(text, len, "ZHTRAP")) return true;
|
||||
@@ -281,6 +299,7 @@ static bool is_statement_or_class_keyword(const int32_t *text, uint32_t len) {
|
||||
if (ascii_upper_eq(text, len, "CLASS")) return true;
|
||||
if (ascii_upper_eq(text, len, "METHOD")) return true;
|
||||
if (ascii_upper_eq(text, len, "CLASSMETHOD")) return true;
|
||||
if (ascii_upper_eq(text, len, "CLIENTMETHOD")) return true;
|
||||
if (ascii_upper_eq(text, len, "PROPERTY")) return true;
|
||||
if (ascii_upper_eq(text, len, "PARAMETER")) return true;
|
||||
if (ascii_upper_eq(text, len, "RELATIONSHIP")) return true;
|
||||
@@ -294,32 +313,339 @@ static bool is_statement_or_class_keyword(const int32_t *text, uint32_t len) {
|
||||
if (ascii_upper_eq(text, len, "IMPORT")) return true;
|
||||
if (ascii_upper_eq(text, len, "INCLUDE")) return true;
|
||||
if (ascii_upper_eq(text, len, "INCLUDEGENERATOR")) return true;
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
static bool ObjectScript_Core_Scanner_lex_fenced_text(
|
||||
TSLexer *lexer,
|
||||
enum ObjectScript_Core_Scanner_TokenType desired_symbol,
|
||||
char l_delim,
|
||||
char r_delim) {
|
||||
int leftRightDiff = 1;
|
||||
// Consumes an ObjectScript interpolation (#(...)# or ##(...)##) embedded in
|
||||
// &html/&xml text. Assumes lookahead is '#'. Inside the parens, '>' and '\''
|
||||
// are ObjectScript operators, not XML syntax, so they must not be treated as
|
||||
// tag ends or attribute quotes.
|
||||
static void skip_cos_interpolation(TSLexer *lexer) {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '#') {
|
||||
advance(lexer);
|
||||
}
|
||||
if (lexer->lookahead != '(') {
|
||||
return;
|
||||
}
|
||||
int depth = 0;
|
||||
while (!lexer->eof(lexer) && lexer->lookahead != '\n') {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == '"') {
|
||||
advance(lexer);
|
||||
while (!lexer->eof(lexer) && lexer->lookahead != '\n') {
|
||||
if (lexer->lookahead == '"') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != '"') {
|
||||
break;
|
||||
}
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
advance(lexer);
|
||||
if (c == '(') {
|
||||
depth++;
|
||||
} else if (c == ')') {
|
||||
depth--;
|
||||
if (depth == 0) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
// consume the closing # (or ##)
|
||||
if (lexer->lookahead == '#') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '#') {
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Consumes the rest of an XML tag up to (not including) the closing '>'.
|
||||
// A quote only opens an attribute value directly after '=', so stray quotes in
|
||||
// hand-written HTML (e.g. height=24") don't swallow the rest of the file.
|
||||
static void skip_xml_tag_body(TSLexer *lexer) {
|
||||
int32_t prev = 0;
|
||||
while (!lexer->eof(lexer) && lexer->lookahead != '>') {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == '#') {
|
||||
skip_cos_interpolation(lexer);
|
||||
prev = '#';
|
||||
continue;
|
||||
}
|
||||
if ((c == '"' || c == '\'') && prev == '=') {
|
||||
advance(lexer);
|
||||
while (!lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == '#') {
|
||||
skip_cos_interpolation(lexer);
|
||||
continue;
|
||||
}
|
||||
if (lexer->lookahead == c) {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == c) {
|
||||
advance(lexer);
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
} else {
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
prev = c;
|
||||
continue;
|
||||
}
|
||||
if (!iswspace(c)) {
|
||||
prev = c;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
|
||||
static bool lex_fenced_text(TSLexer *lexer,
|
||||
enum ObjectScript_Core_Scanner_TokenType desired_symbol, int32_t l_delim,
|
||||
int32_t r_delim, bool is_xml, bool has_python_comment, bool is_text) {
|
||||
int depth = 1;
|
||||
const char *comment_start = "<!--";
|
||||
const char *code_start = "<![CDATA[";
|
||||
const char *code_end = "]]>";
|
||||
const char *comment_end = "-->";
|
||||
const char *script_start = "script";
|
||||
const char *script_end = "</script>";
|
||||
while (iswspace(lexer->lookahead) || lexer->lookahead == 0x2028 || lexer->lookahead == 0x2029) {
|
||||
advance(lexer);
|
||||
}
|
||||
if (lexer->lookahead == '<') {
|
||||
is_xml = true;
|
||||
}
|
||||
while(iswalnum(lexer->lookahead) || iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
while (!lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == r_delim) {
|
||||
leftRightDiff -= 1;
|
||||
} else if (lexer->lookahead == l_delim) {
|
||||
leftRightDiff += 1;
|
||||
while (iswspace(lexer->lookahead) || lexer->lookahead == 0x2028 || lexer->lookahead == 0x2029) {
|
||||
advance(lexer);
|
||||
}
|
||||
if (leftRightDiff == 0) {
|
||||
lexer->result_symbol = desired_symbol;
|
||||
return true;
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == '#' && is_xml) {
|
||||
skip_cos_interpolation(lexer);
|
||||
continue;
|
||||
}
|
||||
if (c == '<' && is_xml) {
|
||||
bool comment = true;
|
||||
bool is_code = false;
|
||||
for (unsigned i = 0; i < 4; i++) {
|
||||
if (i > 1) {
|
||||
is_code = true;
|
||||
}
|
||||
if (lexer->lookahead != comment_start[i]) {
|
||||
comment = false;
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
if (comment) {
|
||||
bool is_comment_end = false;
|
||||
while (!lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == '-') {
|
||||
is_comment_end = true;
|
||||
for (unsigned i = 0; i < 3; i++) {
|
||||
if (lexer->lookahead != comment_end[i]) {
|
||||
is_comment_end = false;
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
if (is_comment_end) {
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
continue;
|
||||
} else if (is_code) {
|
||||
for (unsigned i = 2; i < 9; i++) {
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
if (lexer->lookahead != code_start[i]) {
|
||||
is_code = false;
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
// [CDATA[
|
||||
if (is_code) {
|
||||
bool is_code_end = false;
|
||||
while (!lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == ']') {
|
||||
is_code_end = true;
|
||||
for (unsigned i = 0; i < 3; i++) {
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
if (lexer->lookahead != code_end[i]) {
|
||||
is_code_end = false;
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
if (is_code_end) {
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
} else if (towlower(lexer->lookahead) == 's') {
|
||||
bool is_script = true;
|
||||
for (unsigned i = 0; i < 6; i++) {
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
if (towlower(lexer->lookahead) != script_start[i]) {
|
||||
is_script = false;
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
skip_xml_tag_body(lexer);
|
||||
if (lexer->lookahead == '>') {
|
||||
advance(lexer);
|
||||
}
|
||||
if (!is_script) {
|
||||
continue;
|
||||
}
|
||||
|
||||
bool is_script_end = false;
|
||||
while (!lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == '<') {
|
||||
is_script_end = true;
|
||||
for (unsigned i = 0; i < 9; i++) {
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
if (towlower(lexer->lookahead) != script_end[i]) {
|
||||
is_script_end = false;
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
if (is_script_end) {
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
continue;
|
||||
|
||||
}
|
||||
|
||||
else {
|
||||
skip_xml_tag_body(lexer);
|
||||
// consume the >
|
||||
advance(lexer);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
else if (!is_xml && (c == '"' || (c == '\'' && !is_text))) {
|
||||
int32_t quote = c;
|
||||
advance(lexer);
|
||||
|
||||
// Detect a triple-quoted string.
|
||||
bool triple = false;
|
||||
if (lexer->lookahead == quote) {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == quote) {
|
||||
advance(lexer);
|
||||
triple = true;
|
||||
} else {
|
||||
// Empty string literal ('' or ""); nothing more to skip.
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
// Consume the string contents.
|
||||
while (!lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == '\\') {
|
||||
advance(lexer);
|
||||
if (!lexer->eof(lexer)) {
|
||||
advance(lexer);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (lexer->lookahead == quote) {
|
||||
advance(lexer);
|
||||
if (!triple) {
|
||||
break;
|
||||
}
|
||||
if (lexer->lookahead == quote) {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == quote) {
|
||||
advance(lexer);
|
||||
break;
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (!triple && lexer->lookahead == '\n') {
|
||||
// Unterminated single-quoted string; stop before consuming the
|
||||
// newline so brace counting resumes on the next line.
|
||||
break;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
continue;
|
||||
|
||||
} if (c == '#' && !is_xml && has_python_comment) {
|
||||
// Line comment: consume to end of line.
|
||||
while (!lexer->eof(lexer) && lexer->lookahead != '\n') {
|
||||
advance(lexer);
|
||||
}
|
||||
continue;
|
||||
} if (c == '/' && !is_xml) {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '/') {
|
||||
while (lexer->lookahead != 0 && lexer->lookahead != '\n' && lexer->lookahead != 0x2028 &&
|
||||
lexer->lookahead != 0x2029) {
|
||||
advance(lexer);
|
||||
}
|
||||
continue;
|
||||
} else if (lexer->lookahead == '*') {
|
||||
advance(lexer);
|
||||
while (!lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == '*') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '/') {
|
||||
advance(lexer);
|
||||
break;
|
||||
}
|
||||
}
|
||||
else {
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
if (c == r_delim) {
|
||||
depth--;
|
||||
if (depth == 0) {
|
||||
lexer->result_symbol = desired_symbol;
|
||||
return true;
|
||||
}
|
||||
} else if (c == l_delim) {
|
||||
depth++;
|
||||
}
|
||||
|
||||
advance(lexer);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
|
||||
static bool ObjectScript_Core_Scanner_lex_marker_fenced_text(
|
||||
TSLexer *lexer,
|
||||
enum ObjectScript_Core_Scanner_TokenType desired_symbol,
|
||||
@@ -366,8 +692,35 @@ static inline bool is_objectscript_special_symbol_i32(int32_t c) {
|
||||
c == '.' ;
|
||||
}
|
||||
|
||||
static bool lex_compiled_header(TSLexer *lexer) {
|
||||
lexer->mark_end(lexer);
|
||||
while(!lexer->eof(lexer) && lexer->lookahead != '\n') {
|
||||
advance(lexer);
|
||||
}
|
||||
if (lexer->lookahead != '\n') return false;
|
||||
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != '%') return false;
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != 'R') return false;
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != 'O') return false;
|
||||
advance(lexer);
|
||||
|
||||
lexer->result_symbol = COMPILED_HEADER;
|
||||
int newline_count = 0;
|
||||
while(newline_count < 2 && !lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == '\n') {
|
||||
newline_count++;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool ObjectScript_Core_Scanner_lex_pound_if_special_case(TSLexer *lexer) {
|
||||
if (lexer->lookahead != '0') return false;
|
||||
// if (lexer->lookahead != '0') return false;
|
||||
|
||||
uint32_t depth = 1;
|
||||
bool at_line_start = false;
|
||||
@@ -464,6 +817,15 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
if (valid_symbols[SENTINEL]) {
|
||||
return false;
|
||||
}
|
||||
|
||||
if (scanner->pending_dotted_block_end) {
|
||||
if (lexer->lookahead == '.' && valid_symbols[_TERMINATION]) {
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = _TERMINATION;
|
||||
return true;
|
||||
}
|
||||
scanner->pending_dotted_block_end = false;
|
||||
}
|
||||
scanner-> is_rtn_dot = false;
|
||||
|
||||
// this parses any line that ends in ##continue for
|
||||
@@ -474,7 +836,11 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
static const char pattern[] = "##continue";
|
||||
static const int len = sizeof(pattern)-1;
|
||||
int pos = 0;
|
||||
if (!lexer->eof(lexer) && !iswspace(lexer->lookahead)) {
|
||||
// _STATEMENT_TERMINATION is only valid right after the macro name/args,
|
||||
// where the value must be separated by whitespace. On a line continued
|
||||
// by ##continue it is not valid, and the line may start in column 0.
|
||||
bool is_continued_line = !valid_symbols[_STATEMENT_TERMINATION];
|
||||
if (!is_continued_line && !lexer->eof(lexer) && !iswspace(lexer->lookahead)) {
|
||||
scanner->terminated_newline = false;
|
||||
return false;
|
||||
}
|
||||
@@ -487,6 +853,21 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
|
||||
if (pos == len) {
|
||||
advance(lexer);
|
||||
// Allow a trailing ##; comment, e.g. "##continue ##; comment"
|
||||
while (lexer->lookahead == ' ' || lexer->lookahead == '\t') {
|
||||
advance(lexer);
|
||||
}
|
||||
if (lexer->lookahead == '#') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '#') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == ';') {
|
||||
while (!lexer->eof(lexer) && lexer->lookahead != '\n') {
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
int new_line_count = 0;
|
||||
while(iswspace(lexer->lookahead) && new_line_count<1) {
|
||||
if(lexer->lookahead=='\n') {
|
||||
@@ -528,6 +909,15 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
}
|
||||
}
|
||||
|
||||
if (valid_symbols[_END_IF_END]) {
|
||||
while (lexer->lookahead != '\n' && !lexer->eof(lexer)) {
|
||||
advance(lexer);
|
||||
}
|
||||
lexer->result_symbol = _END_IF_END;
|
||||
scanner->terminated_newline = true;
|
||||
return true;
|
||||
}
|
||||
|
||||
if (valid_symbols[EMBEDDED_JS_SPECIAL_CASE_COMPLETE]) {
|
||||
int i = 0;
|
||||
while (i<scanner->js_marker_buffer_reversed_len) {
|
||||
@@ -604,6 +994,32 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
return true;
|
||||
}
|
||||
|
||||
if (valid_symbols[STATEMENT_IN_MACRO]) {
|
||||
lexer->mark_end(lexer);
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
int32_t ident[96];
|
||||
uint32_t len = 0;
|
||||
while (valid_tag_char(lexer->lookahead)) {
|
||||
if (len < sizeof(ident) / sizeof(ident[0])) ident[len++] = lexer->lookahead;
|
||||
advance(lexer);
|
||||
}
|
||||
if (is_statement_or_class_keyword(ident, len)) {
|
||||
if (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
if (valid_tag_char(lexer->lookahead)) {
|
||||
lexer->result_symbol = STATEMENT_IN_MACRO;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
if (valid_symbols[EMBEDDED_SQL_REVERSE_MARKER]) {
|
||||
while (scanner->sql_marker_buffer_len >0 && !lexer->eof(lexer)) {
|
||||
int32_t expected = scanner->sql_marker_buffer[scanner->sql_marker_buffer_len - 1];
|
||||
@@ -680,7 +1096,7 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
return true;
|
||||
}
|
||||
// an argument that is exactly one space after the keyword
|
||||
if (count == 1 && (iswalnum(lexer->lookahead) || is_objectscript_special_symbol_i32(lexer->lookahead) || lexer->lookahead == '_') && valid_symbol_one_space) {
|
||||
if (count == 1 && (is_label_char(lexer->lookahead) || is_objectscript_special_symbol_i32(lexer->lookahead) || lexer->lookahead == '_' || (lexer->lookahead == '{' && !valid_symbol_argumentless_loop)) && valid_symbol_one_space) {
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = _IMMEDIATE_SINGLE_WHITESPACE_FOLLOWED_BY_NON_WHITESPACE;
|
||||
scanner->terminated_newline = false;
|
||||
@@ -688,7 +1104,7 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
}
|
||||
|
||||
// argumentless command has >= 2 spaces after the keyword
|
||||
else if (count >=2 && (iswalnum(lexer->lookahead) || is_objectscript_special_symbol_i32(lexer->lookahead)) && valid_symbol_argumentless_command_end) {
|
||||
else if (count >=2 && (is_label_char(lexer->lookahead) || is_objectscript_special_symbol_i32(lexer->lookahead) || lexer->lookahead == '&') && valid_symbol_argumentless_command_end) {
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = _ARGUMENTLESS_COMMAND_END;
|
||||
scanner->terminated_newline = false;
|
||||
@@ -706,7 +1122,7 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
else if (lexer->lookahead == '#') {
|
||||
lexer->mark_end(lexer);
|
||||
advance(lexer);
|
||||
if (towlower(lexer->lookahead) == 'e' && valid_symbols[TAG] && scanner->special_pound_if_mode && lexer->get_column(lexer) == 1) {
|
||||
if (towlower(lexer->lookahead) == 'e' && (valid_symbols[TAG] || valid_symbols[TAG_END_IF]) && scanner->special_pound_if_mode) {
|
||||
int32_t directive[16];
|
||||
uint32_t len = 0;
|
||||
while (len < (sizeof(directive) / sizeof(directive[0])) &&
|
||||
@@ -720,6 +1136,13 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
if (scanner->special_pound_if_mode_if_depth<1) {
|
||||
scanner->special_pound_if_mode=false;
|
||||
}
|
||||
// Anything after #endif on the same line is ignored (e.g. #endif ArgFor),
|
||||
// but trailing comments are left for the comment rules
|
||||
while (lexer->lookahead != '\n' && !lexer->eof(lexer) &&
|
||||
lexer->lookahead != ';' && lexer->lookahead != '/' &&
|
||||
lexer->lookahead != '#') {
|
||||
advance(lexer);
|
||||
}
|
||||
lexer->result_symbol = TAG_END_IF;
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
@@ -892,7 +1315,37 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
}
|
||||
|
||||
else if (lexer->lookahead == '{' && valid_symbol_argumentless_loop) {
|
||||
bool valid_symbol_zw_block = valid_symbols[_ZW_BLOCK];
|
||||
lexer->mark_end(lexer);
|
||||
if (count == 1 && valid_symbol_zw_block) {
|
||||
advance(lexer);
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
int32_t ident[96];
|
||||
uint32_t len = 0;
|
||||
bool potential_tag = false;
|
||||
if (lexer->get_column(lexer) == 0) {
|
||||
potential_tag = true;
|
||||
}
|
||||
while (valid_tag_char(lexer->lookahead) || is_objectscript_special_symbol_i32(lexer->lookahead)) {
|
||||
if (len < sizeof(ident) / sizeof(ident[0])) ident[len++] = lexer->lookahead;
|
||||
advance(lexer);
|
||||
}
|
||||
if (!is_statement_or_class_keyword(ident, len) && len > 0 && !potential_tag) {
|
||||
if (len > 3) {
|
||||
if (ident[0] == '$' && ident[1] == '$' && ident[2] == '$') {
|
||||
lexer->result_symbol = _ARGUMENTLESS_LOOP;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
lexer->result_symbol = _ZW_BLOCK;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
lexer->result_symbol = _ARGUMENTLESS_LOOP;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
@@ -948,26 +1401,60 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
}
|
||||
|
||||
else if (lexer->lookahead == '{') {
|
||||
// this token is for commands that allow blocks as part of their command ex: w {"hi": "bye"}
|
||||
if (count == 1 && valid_symbols[_ZW_BLOCK]) {
|
||||
lexer->result_symbol = _ZW_BLOCK;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
|
||||
if (valid_symbols[DOTTED_STATEMENT_BLOCK]) {
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = DOTTED_STATEMENT_BLOCK;
|
||||
}
|
||||
|
||||
else if (valid_symbol_argumentless_loop) {
|
||||
bool valid_symbol_zw_block = valid_symbols[_ZW_BLOCK];
|
||||
lexer->mark_end(lexer);
|
||||
if (count == 1 && valid_symbol_zw_block) {
|
||||
advance(lexer);
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
int32_t ident[96];
|
||||
uint32_t len = 0;
|
||||
bool potential_tag = false;
|
||||
if (lexer->get_column(lexer) == 0) {
|
||||
potential_tag = true;
|
||||
}
|
||||
while (valid_tag_char(lexer->lookahead) || is_objectscript_special_symbol_i32(lexer->lookahead)) {
|
||||
if (len < sizeof(ident) / sizeof(ident[0])) ident[len++] = lexer->lookahead;
|
||||
advance(lexer);
|
||||
}
|
||||
if (!is_statement_or_class_keyword(ident, len) && len > 0 && !potential_tag) {
|
||||
if (len > 3) {
|
||||
if (ident[0] == '$' && ident[1] == '$' && ident[2] == '$') {
|
||||
lexer->result_symbol = _ARGUMENTLESS_LOOP;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
lexer->result_symbol = _ZW_BLOCK;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
lexer->result_symbol = _ARGUMENTLESS_LOOP;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
} else if (count == 1 && valid_symbols[_ZW_BLOCK]) {
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = _ZW_BLOCK;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
if (lexer->lookahead == ')' && valid_symbol_termination) {
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = _TERMINATION;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
|
||||
if (count > 0 && valid_symbols[_WHITESPACE]) {
|
||||
|
||||
// there can't be a space between a keyword and the post-conditional,
|
||||
@@ -1021,11 +1508,18 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
advance(lexer);
|
||||
}
|
||||
if (lexer->lookahead == '}') {
|
||||
if (valid_symbols[_TERMINATION]) {
|
||||
scanner->pending_dotted_block_end = true;
|
||||
lexer->result_symbol = _TERMINATION;
|
||||
return true;
|
||||
}
|
||||
lexer->result_symbol = _BOL_BLOCK;
|
||||
scanner->pending_dotted_block_end = false;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
lexer->result_symbol = _BOL;
|
||||
scanner->pending_dotted_block_end = false;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
@@ -1034,9 +1528,10 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
while (lexer->lookahead == ' ' || lexer->lookahead == '\t') advance(lexer);
|
||||
|
||||
if (lexer->lookahead == '.') {
|
||||
lexer->result_symbol = _BOL;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
lexer->result_symbol = _BOL;
|
||||
scanner->pending_dotted_block_end = false;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
else {
|
||||
lexer->mark_end(lexer);
|
||||
@@ -1087,6 +1582,44 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
return true;
|
||||
}
|
||||
|
||||
if (scanner->routine_token_mode && valid_symbols[COMPILED_HEADER]) {
|
||||
lexer->mark_end(lexer);
|
||||
while (lexer->lookahead != '\n' && !lexer->eof(lexer)) advance(lexer);
|
||||
if (lexer->lookahead == '\n') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != '%') {
|
||||
lexer->result_symbol = TAG;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != 'R') {
|
||||
lexer->result_symbol = TAG;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != 'O') {
|
||||
lexer->result_symbol = TAG;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
lexer->result_symbol = COMPILED_HEADER;
|
||||
int newline_count = 0;
|
||||
while(newline_count < 2 && !lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == '\n') {
|
||||
newline_count++;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
}
|
||||
lexer->result_symbol = TAG;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
|
||||
if (!scanner->column1_statement_mode) {
|
||||
lexer->result_symbol = TAG;
|
||||
scanner->terminated_newline = false;
|
||||
@@ -1107,18 +1640,54 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
scanner->terminated_newline = false;
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
} else if (lexer->lookahead == '(') {
|
||||
lexer->result_symbol = TAG;
|
||||
scanner->terminated_newline = false;
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
} else if (iswspace(lexer->lookahead)) {
|
||||
lexer->mark_end(lexer);
|
||||
advance(lexer);
|
||||
int32_t ident[96];
|
||||
uint32_t len = 0;
|
||||
while (valid_tag_char(lexer->lookahead)) {
|
||||
if (len < sizeof(ident) / sizeof(ident[0])) ident[len++] = lexer->lookahead;
|
||||
advance(lexer);
|
||||
}
|
||||
if (is_statement_or_class_keyword(ident, len)) {
|
||||
lexer->result_symbol = TAG;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
lexer->mark_end(lexer);
|
||||
return false;
|
||||
}
|
||||
else if (valid_symbols[ANGLED_BRACKET_FENCED_TEXT]) {
|
||||
bool ok = ObjectScript_Core_Scanner_lex_fenced_text(
|
||||
lexer, ANGLED_BRACKET_FENCED_TEXT, '<', '>');
|
||||
return ok;
|
||||
scanner->is_python = false;
|
||||
scanner->is_text = false;
|
||||
return lex_fenced_text(
|
||||
lexer, ANGLED_BRACKET_FENCED_TEXT, '<', '>', true, scanner->is_python, false);
|
||||
}
|
||||
else if (valid_symbols[PAREN_FENCED_TEXT]) {
|
||||
bool ok = ObjectScript_Core_Scanner_lex_fenced_text(
|
||||
lexer, PAREN_FENCED_TEXT, '(', ')');
|
||||
return ok;
|
||||
scanner->is_python = false;
|
||||
scanner->is_text = false;
|
||||
return lex_fenced_text(
|
||||
lexer, PAREN_FENCED_TEXT, '(', ')', false, scanner->is_python, false);
|
||||
}
|
||||
|
||||
else if (valid_symbols[EXTERNAL_METHOD_BODY_CONTENT]) {
|
||||
// A valid method_body is one that is whose text fences
|
||||
// are evenly balanced (so far only { })
|
||||
// e.g. VALID: {{{ [^{}]* }}} INVALID: {{{ [^{}]* }
|
||||
bool is_python = scanner->is_python;
|
||||
bool is_text = scanner->is_text;
|
||||
scanner->is_python=false;
|
||||
scanner->is_text=false;
|
||||
return lex_fenced_text(lexer, EXTERNAL_METHOD_BODY_CONTENT, '{',
|
||||
'}', false, is_python, is_text);
|
||||
}
|
||||
|
||||
else if (valid_symbols[_LINE_COMMENT_INNER]) {
|
||||
@@ -1191,7 +1760,7 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
lexer->result_symbol = _WHITESPACE;
|
||||
return true;
|
||||
}
|
||||
else if (valid_symbols[TAG] && scanner->special_pound_if_mode) {
|
||||
else if ((valid_symbols[TAG] || valid_symbols[TAG_END_IF]) && scanner->special_pound_if_mode) {
|
||||
while (iswspace(lexer->lookahead)) {
|
||||
advance(lexer);
|
||||
}
|
||||
@@ -1211,6 +1780,13 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
if (scanner->special_pound_if_mode_if_depth<1) {
|
||||
scanner->special_pound_if_mode=false;
|
||||
}
|
||||
// Anything after #endif on the same line is ignored (e.g. #endif ArgFor),
|
||||
// but trailing comments are left for the comment rules
|
||||
while (lexer->lookahead != '\n' && !lexer->eof(lexer) &&
|
||||
lexer->lookahead != ';' && lexer->lookahead != '/' &&
|
||||
lexer->lookahead != '#') {
|
||||
advance(lexer);
|
||||
}
|
||||
lexer->result_symbol = TAG_END_IF;
|
||||
lexer->mark_end(lexer);
|
||||
return true;
|
||||
@@ -1226,9 +1802,13 @@ ObjectScript_Core_Scanner_scan(struct ObjectScript_Core_Scanner *scanner,
|
||||
}
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = BOL_EXTRA;
|
||||
scanner->pending_dotted_block_end = false;
|
||||
scanner->terminated_newline = false;
|
||||
return true;
|
||||
}
|
||||
if (valid_symbols[COMPILED_HEADER] && lex_compiled_header(lexer)) {
|
||||
return true;
|
||||
}
|
||||
scanner->terminated_newline = false;
|
||||
return false;
|
||||
}
|
||||
@@ -1242,4 +1822,7 @@ static void ObjectScript_Core_Scanner_init(struct ObjectScript_Core_Scanner *sca
|
||||
scanner->special_pound_if_mode = false;
|
||||
scanner->special_pound_if_mode_if_depth = 0;
|
||||
scanner->is_rtn_dot = false;
|
||||
scanner->pending_dotted_block_end = false;
|
||||
scanner->is_python = false;
|
||||
scanner->is_text = false;
|
||||
}
|
||||
|
||||
+1214247
-926990
File diff suppressed because it is too large
Load Diff
+108
-53
@@ -1,12 +1,15 @@
|
||||
#include "objectscript_common.h"
|
||||
#include "tree_sitter/parser.h"
|
||||
#include <string.h>
|
||||
#include <wctype.h>
|
||||
|
||||
// There is no way to extend enums, so keep this in sync with base.h
|
||||
// All new entries should be appended at the bottom of the list
|
||||
enum TokenType {
|
||||
EXTERNAL_METHOD_BODY_CONTENT = OBJECTSCRIPT_CORE_TOKEN_TYPE_MAX,
|
||||
IRIS_USERNAME,
|
||||
IRIS_USERNAME= OBJECTSCRIPT_CORE_TOKEN_TYPE_MAX,
|
||||
PYTHON_MIME_TYPE,
|
||||
_IS_PYTHON,
|
||||
TEXT_MIME_TYPE,
|
||||
};
|
||||
|
||||
struct ObjectScript_Udl_Scanner {
|
||||
@@ -14,57 +17,11 @@ struct ObjectScript_Udl_Scanner {
|
||||
struct ObjectScript_Core_Scanner core_scanner;
|
||||
};
|
||||
|
||||
static bool lex_fenced_text(TSLexer *lexer,
|
||||
enum TokenType desired_symbol, char l_delim,
|
||||
char r_delim) {
|
||||
int leftRightDiff = 1;
|
||||
while (!lexer->eof(lexer)) {
|
||||
if (lexer->lookahead == r_delim) {
|
||||
leftRightDiff -= 1;
|
||||
} else if (lexer->lookahead == l_delim) {
|
||||
leftRightDiff += 1;
|
||||
}
|
||||
if (leftRightDiff == 0) {
|
||||
lexer->result_symbol = desired_symbol;
|
||||
return true;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/// This is the interesting function. The rest is infrastructure
|
||||
static bool scan(void *payload, TSLexer *lexer, const bool *valid_symbols) {
|
||||
// struct Scanner *scanner = (struct Scanner *)payload;
|
||||
//
|
||||
// Tree sitter will mark all terminals as valid on error
|
||||
// The sentinel should never be valid in a good parse, so this ensures
|
||||
// we are not in error recovery mode
|
||||
// printf("Lookahead: '%c'. Col %d. Tag is valid? %d", lexer->lookahead,
|
||||
// lexer->get_column(lexer), valid_symbols[TAG]);
|
||||
// printf("lexer->lookahead: '%c'\n", lexer->lookahead);
|
||||
// printf("====\n");
|
||||
// debug_enum(valid_symbols);
|
||||
// printf("====\n");
|
||||
static bool scan(struct ObjectScript_Udl_Scanner *scanner, TSLexer *lexer, const bool *valid_symbols) {
|
||||
if (valid_symbols[SENTINEL]) {
|
||||
// printf("Thats not good!\n");
|
||||
// // TODO: ERROR RECOVERY MODE ?? Why would we be better than TS's
|
||||
// default? if (iswspace(lexer->lookahead)) {
|
||||
// skip(lexer);
|
||||
// eat_whitespace(lexer);
|
||||
// lexer->result_symbol = _WHITESPACE;
|
||||
// return true;
|
||||
// }
|
||||
return false;
|
||||
}
|
||||
if (valid_symbols[EXTERNAL_METHOD_BODY_CONTENT]) {
|
||||
// A valid method_body is one that is whose text fences
|
||||
// are evenly balanced (so far only { })
|
||||
// e.g. VALID: {{{ [^{}]* }}} INVALID: {{{ [^{}]* }
|
||||
return lex_fenced_text(lexer, EXTERNAL_METHOD_BODY_CONTENT, '{',
|
||||
'}');
|
||||
}
|
||||
|
||||
if (valid_symbols[IRIS_USERNAME] && !(lexer->lookahead=='}' || lexer->lookahead=='@' || lexer->lookahead=='*')) {
|
||||
int count=0;
|
||||
lexer->mark_end(lexer);
|
||||
@@ -72,7 +29,7 @@ static bool scan(void *payload, TSLexer *lexer, const bool *valid_symbols) {
|
||||
count++;
|
||||
advance(lexer);
|
||||
}
|
||||
|
||||
|
||||
|
||||
if(lexer->lookahead=='}') {
|
||||
lexer->mark_end(lexer);
|
||||
@@ -82,12 +39,110 @@ static bool scan(void *payload, TSLexer *lexer, const bool *valid_symbols) {
|
||||
else {
|
||||
return false;
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
|
||||
if (valid_symbols[_IS_PYTHON]) {
|
||||
scanner->core_scanner.is_python = true;
|
||||
lexer->result_symbol = _IS_PYTHON;
|
||||
return true;
|
||||
}
|
||||
|
||||
if (valid_symbols[PYTHON_MIME_TYPE]) {
|
||||
lexer->mark_end(lexer);
|
||||
int count = 0;
|
||||
while(iswspace(lexer->lookahead)) {
|
||||
count ++;
|
||||
advance(lexer);
|
||||
}
|
||||
if (count > 0) {
|
||||
lexer -> mark_end(lexer);
|
||||
lexer->result_symbol = _WHITESPACE;
|
||||
return true;
|
||||
}
|
||||
bool has_string = false;
|
||||
if (lexer->lookahead == '"') {
|
||||
has_string = true;
|
||||
advance(lexer);
|
||||
}
|
||||
const char* text_mime = "text/";
|
||||
const char *text_python = "x-python";
|
||||
const char *application_python = "application/python";
|
||||
const char *markdown = "markdown";
|
||||
const char *plain = "plain";
|
||||
bool is_python = true;
|
||||
bool is_text = true;
|
||||
bool is_markdown = false;
|
||||
for (unsigned i = 0; i < 5; i++) {
|
||||
if (lexer->lookahead != text_mime[i]) {
|
||||
is_text = false;
|
||||
break;
|
||||
}
|
||||
if (lexer->eof(lexer)) break;
|
||||
advance(lexer);
|
||||
}
|
||||
if (is_text) {
|
||||
for (unsigned i = 0; i < 8; i++) {
|
||||
if (lexer->lookahead != text_python[i]) {
|
||||
is_python = false;
|
||||
break;
|
||||
}
|
||||
if (lexer->eof(lexer)) break;
|
||||
advance(lexer);
|
||||
} if (!is_python) {
|
||||
is_markdown = true;
|
||||
for (unsigned i = 0; i < 8; i++) {
|
||||
if (lexer->lookahead != markdown[i]) {
|
||||
is_markdown = false;
|
||||
break;
|
||||
}
|
||||
if (lexer->eof(lexer)) break;
|
||||
advance(lexer);
|
||||
}
|
||||
} if (!is_markdown && !is_python) {
|
||||
is_markdown = true;
|
||||
for (unsigned i = 0; i < 5; i++) {
|
||||
if (lexer->lookahead != plain[i]) {
|
||||
is_markdown = false;
|
||||
break;
|
||||
}
|
||||
if (lexer->eof(lexer)) break;
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
}
|
||||
else {
|
||||
for (unsigned i = 0; i < 18; i++) {
|
||||
if (lexer->lookahead != application_python[i]) {
|
||||
is_python = false;
|
||||
break;
|
||||
}
|
||||
if (lexer->eof(lexer)) break;
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
if (has_string) {
|
||||
advance(lexer);
|
||||
}
|
||||
|
||||
if (is_python) {
|
||||
scanner->core_scanner.is_python = true;
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = PYTHON_MIME_TYPE;
|
||||
return true;
|
||||
}
|
||||
|
||||
if (is_markdown) {
|
||||
scanner->core_scanner.is_text = true;
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = TEXT_MIME_TYPE;
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
|
||||
}
|
||||
|
||||
struct ObjectScript_Udl_Scanner *scanner =
|
||||
(struct ObjectScript_Udl_Scanner *)payload;
|
||||
return ObjectScript_Core_Scanner_scan(&scanner->core_scanner, lexer,
|
||||
valid_symbols);
|
||||
}
|
||||
|
||||
@@ -50,12 +50,18 @@ extern "C" {
|
||||
/// memory allocated for the array's contents.
|
||||
#define array_clear(self) ((self)->size = 0)
|
||||
|
||||
#ifdef __cplusplus
|
||||
#define _array__cast(self, expr) (decltype((self)->contents))(expr)
|
||||
#else
|
||||
#define _array__cast(self, expr) (expr)
|
||||
#endif
|
||||
|
||||
/// Reserve `new_capacity` elements of space in the array. If `new_capacity` is
|
||||
/// less than the array's current capacity, this function has no effect.
|
||||
#define array_reserve(self, new_capacity) \
|
||||
((self)->contents = _array__reserve( \
|
||||
(void *)(self)->contents, &(self)->capacity, \
|
||||
array_elem_size(self), new_capacity) \
|
||||
#define array_reserve(self, new_capacity) \
|
||||
((self)->contents = _array__cast(self, _array__reserve( \
|
||||
(void *)(self)->contents, &(self)->capacity, \
|
||||
array_elem_size(self), new_capacity)) \
|
||||
)
|
||||
|
||||
/// Free any memory allocated for this array. Note that this does not free any
|
||||
@@ -71,10 +77,10 @@ extern "C" {
|
||||
/// Push a new `element` onto the end of the array.
|
||||
#define array_push(self, element) \
|
||||
do { \
|
||||
(self)->contents = _array__grow( \
|
||||
(self)->contents = _array__cast(self, _array__grow( \
|
||||
(void *)(self)->contents, (self)->size, &(self)->capacity, \
|
||||
1, array_elem_size(self) \
|
||||
); \
|
||||
)); \
|
||||
(self)->contents[(self)->size++] = (element); \
|
||||
} while(0)
|
||||
|
||||
@@ -83,10 +89,10 @@ extern "C" {
|
||||
#define array_grow_by(self, count) \
|
||||
do { \
|
||||
if ((count) == 0) break; \
|
||||
(self)->contents = _array__grow( \
|
||||
(self)->contents = _array__cast(self, _array__grow( \
|
||||
(self)->contents, (self)->size, &(self)->capacity, \
|
||||
count, array_elem_size(self) \
|
||||
); \
|
||||
)); \
|
||||
memset((self)->contents + (self)->size, 0, (count) * array_elem_size(self)); \
|
||||
(self)->size += (count); \
|
||||
} while (0)
|
||||
@@ -98,26 +104,26 @@ extern "C" {
|
||||
/// Append `count` elements to the end of the array, reading their values from the
|
||||
/// `contents` pointer.
|
||||
#define array_extend(self, count, other_contents) \
|
||||
(self)->contents = _array__splice( \
|
||||
((self)->contents = _array__cast(self, _array__splice( \
|
||||
(void*)(self)->contents, &(self)->size, &(self)->capacity, \
|
||||
array_elem_size(self), (self)->size, 0, count, other_contents \
|
||||
)
|
||||
)))
|
||||
|
||||
/// Remove `old_count` elements from the array starting at the given `index`. At
|
||||
/// the same index, insert `new_count` new elements, reading their values from the
|
||||
/// `new_contents` pointer.
|
||||
#define array_splice(self, _index, old_count, new_count, new_contents) \
|
||||
(self)->contents = _array__splice( \
|
||||
((self)->contents = _array__cast(self, _array__splice( \
|
||||
(void *)(self)->contents, &(self)->size, &(self)->capacity, \
|
||||
array_elem_size(self), _index, old_count, new_count, new_contents \
|
||||
)
|
||||
)))
|
||||
|
||||
/// Insert one `element` into the array at the given `index`.
|
||||
#define array_insert(self, _index, element) \
|
||||
(self)->contents = _array__splice( \
|
||||
((self)->contents = _array__cast(self, _array__splice( \
|
||||
(void *)(self)->contents, &(self)->size, &(self)->capacity, \
|
||||
array_elem_size(self), _index, 0, 1, &(element) \
|
||||
)
|
||||
)))
|
||||
|
||||
/// Remove one element from the array at the given `index`.
|
||||
#define array_erase(self, _index) \
|
||||
@@ -128,17 +134,17 @@ extern "C" {
|
||||
|
||||
/// Assign the contents of one array to another, reallocating if necessary.
|
||||
#define array_assign(self, other) \
|
||||
(self)->contents = _array__assign( \
|
||||
((self)->contents = _array__cast(self, _array__assign( \
|
||||
(void *)(self)->contents, &(self)->size, &(self)->capacity, \
|
||||
(const void *)(other)->contents, (other)->size, array_elem_size(self) \
|
||||
)
|
||||
)))
|
||||
|
||||
/// Swap one array with another
|
||||
#define array_swap(self, other) \
|
||||
do { \
|
||||
void *_array_swap_tmp = (void *)(self)->contents; \
|
||||
(self)->contents = (other)->contents; \
|
||||
(other)->contents = _array_swap_tmp; \
|
||||
(other)->contents = _array__cast(other, _array_swap_tmp); \
|
||||
_array__swap(&(self)->size, &(self)->capacity, \
|
||||
&(other)->size, &(other)->capacity); \
|
||||
} while (0)
|
||||
|
||||
@@ -5,7 +5,7 @@ c5cfb43042b6b72045f4ba997834d0a7786d2793d91680868b5815b39f14fc78 internal/cbm/v
|
||||
b29c1c9fb7cc82f58c84b376df1297d6e2737a1d655fd356db0859e3c29c2fea internal/cbm/vendored/common/tree_sitter/alloc.h
|
||||
31e60a1bff6f715afacce03b5b70efe42b58371b4f9595dd4af52a577ff9608c internal/cbm/vendored/common/tree_sitter/array.h
|
||||
180b893c8734778fd32f372dfbc27bd6ad1cd2221f26150b31256ff6716320d2 internal/cbm/vendored/common/tree_sitter/parser.h
|
||||
e7478214041a2186a67160ccad3f08ddfcb8259b073172a19b9a2c947c260263 internal/cbm/vendored/grammars/MANIFEST.md
|
||||
865372f11bb0bb18876fc3c77f5aca386621e20511864f336f34f8038af5987a internal/cbm/vendored/grammars/MANIFEST.md
|
||||
ad8425038de519f8c4e3e9339feebf99dfad8a6002dcf79227d91402780a32cc internal/cbm/vendored/grammars/ada/LICENSE
|
||||
02805ec13939b749c891567be36bf024b09034e04c80683a1ae667272458d549 internal/cbm/vendored/grammars/ada/parser.c
|
||||
115a75d000bef9c70c4de7dfbb7f2a80b90cbb14ba265056dc7abb1ef3b9b6db internal/cbm/vendored/grammars/ada/tree_sitter/alloc.h
|
||||
@@ -523,18 +523,18 @@ c5fa4d1d5820cc59f39cb3cb9a9a171e5144b2f63d285cf3450be0a8e549ede8 internal/cbm/v
|
||||
02d139d44303b0b0b85a899839dbc1346c478e7318d1f2971fad2b38fd57cb73 internal/cbm/vendored/grammars/objc/tree_sitter/array.h
|
||||
2db740ea1be6b71014d2be1385491f01bf2d15b607e61c5b30b4128535efe68a internal/cbm/vendored/grammars/objc/tree_sitter/parser.h
|
||||
85b458dc150c4d17873dd5e73d01cfa23c797997a09d841884e08378242e929c internal/cbm/vendored/grammars/objectscript_routine/LICENSE
|
||||
f3a047fba6d238da258c0370878e703504b347aad13add09a021935daa891bd3 internal/cbm/vendored/grammars/objectscript_routine/objectscript_common.h
|
||||
fb9320a77f7c3c110540e9a569c2a8dd7ff74784ae05aac962f4bcbab1343640 internal/cbm/vendored/grammars/objectscript_routine/parser.c
|
||||
1da66e46ba02fcf2f9ca42b2a80c8658e150ae7b5154d81b8968117b959b5af7 internal/cbm/vendored/grammars/objectscript_routine/scanner.c
|
||||
7aeb5b2933b7cb65f9714bc14ba1aff98bec3e5f2351d2e5199c2c1b477a3ba8 internal/cbm/vendored/grammars/objectscript_routine/objectscript_common.h
|
||||
d496de950b085853fdf1edf1570eecdb5d2e4525344873bfabf7e34ac8e7add0 internal/cbm/vendored/grammars/objectscript_routine/parser.c
|
||||
073b64efe3eff37b1a6c3f76ac99d5b5533094591dd9d329bee3982121469d04 internal/cbm/vendored/grammars/objectscript_routine/scanner.c
|
||||
b29c1c9fb7cc82f58c84b376df1297d6e2737a1d655fd356db0859e3c29c2fea internal/cbm/vendored/grammars/objectscript_routine/tree_sitter/alloc.h
|
||||
31e60a1bff6f715afacce03b5b70efe42b58371b4f9595dd4af52a577ff9608c internal/cbm/vendored/grammars/objectscript_routine/tree_sitter/array.h
|
||||
20e6ba85abb9d5b7e2e6dedf675c9424a057a06feee03aa3e2dd48a1cb4bf8a2 internal/cbm/vendored/grammars/objectscript_routine/tree_sitter/array.h
|
||||
180b893c8734778fd32f372dfbc27bd6ad1cd2221f26150b31256ff6716320d2 internal/cbm/vendored/grammars/objectscript_routine/tree_sitter/parser.h
|
||||
85b458dc150c4d17873dd5e73d01cfa23c797997a09d841884e08378242e929c internal/cbm/vendored/grammars/objectscript_udl/LICENSE
|
||||
f3a047fba6d238da258c0370878e703504b347aad13add09a021935daa891bd3 internal/cbm/vendored/grammars/objectscript_udl/objectscript_common.h
|
||||
2003ffb1a07ec30a2f7176c65aae958b804074af8356a81a3d42417a240f2e15 internal/cbm/vendored/grammars/objectscript_udl/parser.c
|
||||
537be452c5079531e08edc302b65c63f45956510f6ee96ec88c1d7531d203f1e internal/cbm/vendored/grammars/objectscript_udl/scanner.c
|
||||
7aeb5b2933b7cb65f9714bc14ba1aff98bec3e5f2351d2e5199c2c1b477a3ba8 internal/cbm/vendored/grammars/objectscript_udl/objectscript_common.h
|
||||
0c0c651b4a425cb8a00cdf6b8c89aeea702cb5799a9483d9994556655f441015 internal/cbm/vendored/grammars/objectscript_udl/parser.c
|
||||
4d991526fa0930d023cbea841aa80298391142dbb58977f197d580f86a00b968 internal/cbm/vendored/grammars/objectscript_udl/scanner.c
|
||||
b29c1c9fb7cc82f58c84b376df1297d6e2737a1d655fd356db0859e3c29c2fea internal/cbm/vendored/grammars/objectscript_udl/tree_sitter/alloc.h
|
||||
31e60a1bff6f715afacce03b5b70efe42b58371b4f9595dd4af52a577ff9608c internal/cbm/vendored/grammars/objectscript_udl/tree_sitter/array.h
|
||||
20e6ba85abb9d5b7e2e6dedf675c9424a057a06feee03aa3e2dd48a1cb4bf8a2 internal/cbm/vendored/grammars/objectscript_udl/tree_sitter/array.h
|
||||
180b893c8734778fd32f372dfbc27bd6ad1cd2221f26150b31256ff6716320d2 internal/cbm/vendored/grammars/objectscript_udl/tree_sitter/parser.h
|
||||
f977a20805dd6e275f14bc090a5f8e638401dad4df366841c1060f12c75b74ae internal/cbm/vendored/grammars/ocaml/LICENSE
|
||||
e4934793536f53c076b24b16de2f6a48a125c632e6cce58c80c02d6d65f8dbed internal/cbm/vendored/grammars/ocaml/_common_scanner.h
|
||||
|
||||
@@ -147,6 +147,44 @@ static const char *PERL_MALFORMED = "package Broken;\n"
|
||||
"} ] } ]\n"
|
||||
"sub after_error { return 2; }\n";
|
||||
|
||||
/* #1261: a `Language = python` method body is lexed by the ObjectScript
|
||||
* grammar's external scanner. Up to intersystems/tree-sitter-objectscript
|
||||
* a7ffcdf it only balanced braces, so a brace inside a Python string, comment
|
||||
* or docstring ran the body past its closing `}` and the following methods were
|
||||
* lost in an ERROR region. Since upstream v1.9.15 the scanner lexes these
|
||||
* bodies with Python's string, triple-quote and `#`-comment rules. */
|
||||
static const char *OBJECTSCRIPT_PYTHON_BODY_BRACES =
|
||||
"Class Demo.Py Extends %RegisteredObject\n"
|
||||
"{\n"
|
||||
"ClassMethod Fmt() As %String [ Language = python ]\n"
|
||||
"{\n"
|
||||
" s = \"open { only\"\n"
|
||||
" return s\n"
|
||||
"}\n"
|
||||
"ClassMethod Note() As %String [ Language = python ]\n"
|
||||
"{\n"
|
||||
" # a } in a comment\n"
|
||||
" return 'x'\n"
|
||||
"}\n"
|
||||
"ClassMethod Doc() As %String [ Language = python ]\n"
|
||||
"{\n"
|
||||
" \"\"\"docstring with { and no close\"\"\"\n"
|
||||
" return 'y'\n"
|
||||
"}\n"
|
||||
"ClassMethod After() As %String\n"
|
||||
"{\n"
|
||||
" Quit \"after\"\n"
|
||||
"}\n"
|
||||
"}\n";
|
||||
|
||||
/* The refreshed scanner must still report a Python body that never closes. */
|
||||
static const char *OBJECTSCRIPT_PYTHON_BODY_UNCLOSED =
|
||||
"Class Demo.Broken Extends %RegisteredObject\n"
|
||||
"{\n"
|
||||
"ClassMethod Fmt() As %String [ Language = python ]\n"
|
||||
"{\n"
|
||||
" return \"x\"\n";
|
||||
|
||||
/* ── Tests ────────────────────────────────────────────────────────────────── */
|
||||
|
||||
TEST(c_ifdef_split_brace_sets_parse_incomplete) {
|
||||
@@ -576,6 +614,40 @@ TEST(perl_malformed_source_remains_partial_issue1838) {
|
||||
PASS();
|
||||
}
|
||||
|
||||
TEST(objectscript_python_body_braces_parse_completely_issue1261) {
|
||||
CBMFileResult *r =
|
||||
do_extract(OBJECTSCRIPT_PYTHON_BODY_BRACES, CBM_LANG_OBJECTSCRIPT_UDL, "Py.cls");
|
||||
ASSERT_NOT_NULL(r);
|
||||
ASSERT_FALSE(r->has_error);
|
||||
bool partial = r->parse_incomplete;
|
||||
int error_regions = r->error_region_count;
|
||||
bool all_methods =
|
||||
has_def(r, "Fmt") && has_def(r, "Note") && has_def(r, "Doc") && has_def(r, "After");
|
||||
if (partial || error_regions != 0 || !all_methods) {
|
||||
fprintf(stderr,
|
||||
" ObjectScript Python-body result: partial=%d regions=%d ranges=%s "
|
||||
"Fmt=%d Note=%d Doc=%d After=%d\n",
|
||||
partial, error_regions, r->error_ranges ? r->error_ranges : "(none)",
|
||||
has_def(r, "Fmt"), has_def(r, "Note"), has_def(r, "Doc"), has_def(r, "After"));
|
||||
cbm_free_result(r);
|
||||
FAIL("braces inside Python strings, comments and docstrings must not run a "
|
||||
"Language = python body past its closing brace");
|
||||
}
|
||||
cbm_free_result(r);
|
||||
PASS();
|
||||
}
|
||||
|
||||
TEST(objectscript_unclosed_python_body_remains_partial_issue1261) {
|
||||
CBMFileResult *r =
|
||||
do_extract(OBJECTSCRIPT_PYTHON_BODY_UNCLOSED, CBM_LANG_OBJECTSCRIPT_UDL, "Broken.cls");
|
||||
ASSERT_NOT_NULL(r);
|
||||
ASSERT_FALSE(r->has_error);
|
||||
ASSERT_TRUE(r->parse_incomplete);
|
||||
ASSERT_GTE(r->error_region_count, 1);
|
||||
cbm_free_result(r);
|
||||
PASS();
|
||||
}
|
||||
|
||||
/* ── #1746: trailing blanks before EOF are still just an absent newline ───────
|
||||
*
|
||||
* #1610 suppressed the zero-width MISSING terminator but tested for it with
|
||||
@@ -2378,6 +2450,8 @@ SUITE(parse_coverage) {
|
||||
RUN_TEST(c_clean_file_stays_unflagged_after_refinement);
|
||||
RUN_TEST(perl_format_followed_by_named_sub_is_complete_issue1838);
|
||||
RUN_TEST(perl_malformed_source_remains_partial_issue1838);
|
||||
RUN_TEST(objectscript_python_body_braces_parse_completely_issue1261);
|
||||
RUN_TEST(objectscript_unclosed_python_body_remains_partial_issue1261);
|
||||
RUN_TEST(dockerfile_trailing_blank_at_eof_not_flagged_issue1746);
|
||||
RUN_TEST(real_error_before_eof_still_flagged_with_trailing_blank_issue1746);
|
||||
RUN_TEST(width_bearing_error_at_eof_still_flagged_with_trailing_blank_issue1746);
|
||||
|
||||
Reference in New Issue
Block a user