Commit graph

8 commits

Author SHA1 Message Date
d81bef9184 parser/tokenizer: Save textual data as refcounted strings
This avoids creating refcounted strings during evaluation and makes it
easier to use the same parsed string in multiple places (should be
useful once we implement functions).
2022-01-18 21:18:27 +01:00
6439f4f8ce Tokenizer: Disallow ASCII control characters outside strings 2022-01-07 23:39:06 +01:00
bab1812cc9 Add tokenizer tests to check binary string handling 2022-01-06 22:50:44 +01:00
f685214abe Add apfl_string_eq macro
It's easy to forget the `== 0` after the apfl_string_cmp() call, so let's
add a macro that does the job for us.
2022-01-02 16:51:19 +01:00
84f127bfec Move must_alloc into test.h 2021-12-16 22:50:14 +01:00
025fd61abd Add apfl_string_source_reader
A useful source reader implementation to pass in a source saved as an
in-memory string into the tokenizer.

This replaces the string_src_reader in the tokenizer_test and is even a bit
more flexible, by allowing any aplf_string_view as the source.
2021-12-16 22:49:41 +01:00
c0e47b14e6 Add tokenizer test case
During development on the parser I got a "malloc(): corrupted top size"
error in the tokenizer when parsing `a=a`. I wrote this test to see if it
was really a problem with the tokenizer. It wasn't, lets keep the test
nontheless.
2021-12-16 22:09:59 +01:00
d094ed7bd5 Initial commit 2021-12-15 21:47:17 +01:00