laria/apfl
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
apfl/src/tokenizer_test.c

350 lines
11 KiB
C
Raw Normal View History

Initial commit
2021-12-10 20:22:16 +00:00
#include <assert.h>
Make line and column size_t
2023-02-16 20:41:02 +00:00
#include <inttypes.h>
Initial commit
2021-12-10 20:22:16 +00:00
#include "apfl.h"
#include "test.h"
struct tokenizer_test {
testctx t;
Introduce allocator abstraction We now no longer call malloc/free/... directly, but use an allocator object that is passed around. This was mainly done as a preparation for a garbage collector: The collector will need to know, how much memory we're using, introducing the collector abstraction will allow the GC to hook into the memory allocation and observe the memory usage. This has other potential applications: - We could now be embedded into applications that can't use the libc allocator. - There could be an allocator that limits the total amount of used memory, e.g. for sandboxing purposes. - In our tests we could use this to simulate out of memory conditions (implement an allocator that fails at the n-th allocation, increase n by one and restart the test until there are no more faked OOM conditions). The function signature of the allocator is basically exactly the same as the one Lua uses.
2022-02-08 21:53:13 +00:00
struct apfl_allocator allocator;
Initial commit
2021-12-10 20:22:16 +00:00
apfl_tokenizer_ptr tokenizer;
Implement bytecode (de)serialization
2023-03-05 16:02:42 +00:00
struct apfl_io_string_reader_data string_reader;
struct apfl_io_reader reader;
Initial commit
2021-12-10 20:22:16 +00:00
};
static struct tokenizer_test *
Add tokenizer tests to check binary string handling
2022-01-06 21:50:44 +00:00
new_tokenizer_test_sv(testctx t, struct apfl_string_view text)
Initial commit
2021-12-10 20:22:16 +00:00
{
Introduce allocator abstraction We now no longer call malloc/free/... directly, but use an allocator object that is passed around. This was mainly done as a preparation for a garbage collector: The collector will need to know, how much memory we're using, introducing the collector abstraction will allow the GC to hook into the memory allocation and observe the memory usage. This has other potential applications: - We could now be embedded into applications that can't use the libc allocator. - There could be an allocator that limits the total amount of used memory, e.g. for sandboxing purposes. - In our tests we could use this to simulate out of memory conditions (implement an allocator that fails at the n-th allocation, increase n by one and restart the test until there are no more faked OOM conditions). The function signature of the allocator is basically exactly the same as the one Lua uses.
2022-02-08 21:53:13 +00:00
struct apfl_allocator allocator = apfl_allocator_default();
Initial commit
2021-12-10 20:22:16 +00:00
struct tokenizer_test *tt = must_alloc(t, sizeof(struct tokenizer_test));
*tt = (struct tokenizer_test) {
.t = t,
Introduce allocator abstraction We now no longer call malloc/free/... directly, but use an allocator object that is passed around. This was mainly done as a preparation for a garbage collector: The collector will need to know, how much memory we're using, introducing the collector abstraction will allow the GC to hook into the memory allocation and observe the memory usage. This has other potential applications: - We could now be embedded into applications that can't use the libc allocator. - There could be an allocator that limits the total amount of used memory, e.g. for sandboxing purposes. - In our tests we could use this to simulate out of memory conditions (implement an allocator that fails at the n-th allocation, increase n by one and restart the test until there are no more faked OOM conditions). The function signature of the allocator is basically exactly the same as the one Lua uses.
2022-02-08 21:53:13 +00:00
.allocator = allocator,
Initial commit
2021-12-10 20:22:16 +00:00
};
Implement bytecode (de)serialization
2023-03-05 16:02:42 +00:00
tt->string_reader = apfl_io_string_reader_create(text);
tt->reader = apfl_io_string_reader(&tt->string_reader);
Initial commit
2021-12-10 20:22:16 +00:00
Redo source reader interface The callback and the opaque data are now grouped together in a struct instead of being passed individually into the tokenizer. This also exposes the string source reader struct and therefore removes the need of heap allocating it. Neat!
2022-04-15 20:35:36 +00:00
if ((tt->tokenizer = apfl_tokenizer_new(
allocator,
Implement bytecode (de)serialization
2023-03-05 16:02:42 +00:00
apfl_io_reader_as_source_reader(&tt->reader)
Redo source reader interface The callback and the opaque data are now grouped together in a struct instead of being passed individually into the tokenizer. This also exposes the string source reader struct and therefore removes the need of heap allocating it. Neat!
2022-04-15 20:35:36 +00:00
)) == NULL) {
test_fatalf(t, "Failed to initialize the tokenizer");
}
Initial commit
2021-12-10 20:22:16 +00:00
return tt;
}
Add tokenizer tests to check binary string handling
2022-01-06 21:50:44 +00:00
static struct tokenizer_test *
new_tokenizer_test(testctx t, const char *text)
{
return new_tokenizer_test_sv(t, apfl_string_view_from(text));
}
Initial commit
2021-12-10 20:22:16 +00:00
static void
destroy_tokenizer_test(struct tokenizer_test *tt)
{
apfl_tokenizer_destroy(tt->tokenizer);
free(tt);
}
static void
expect_eof(struct tokenizer_test *tt)
{
switch (apfl_tokenizer_next(tt->tokenizer, false)) {
case APFL_PARSE_OK:
test_fatalf(tt->t, "Expected EOF but got a token");
break;
case APFL_PARSE_EOF:
break;
case APFL_PARSE_ERROR:
test_failf(tt->t, "Got an error instead of an EOF");
apfl_error_print(apfl_tokenizer_get_error(tt->tokenizer), stderr);
test_fatal(tt->t);
break;
}
}
static bool
Make line and column size_t
2023-02-16 20:41:02 +00:00
expect_token(struct tokenizer_test *tt, size_t line, size_t col, enum apfl_token_type type, struct apfl_token *tok)
Initial commit
2021-12-10 20:22:16 +00:00
{
switch (apfl_tokenizer_next(tt->tokenizer, false)) {
case APFL_PARSE_OK:
break;
case APFL_PARSE_EOF:
test_fatalf(tt->t, "Got an EOF instead of a token");
break;
case APFL_PARSE_ERROR:
test_failf(tt->t, "Got an error instead of a token");
apfl_error_print(apfl_tokenizer_get_error(tt->tokenizer), stderr);
test_fatal(tt->t);
break;
}
*tok = apfl_tokenizer_get_token(tt->tokenizer);
if (tok->type != type) {
test_failf(
tt->t,
"Got wrong token type %s (wanted %s)",
apfl_token_type_name(tok->type),
apfl_token_type_name(type)
);
Introduce allocator abstraction We now no longer call malloc/free/... directly, but use an allocator object that is passed around. This was mainly done as a preparation for a garbage collector: The collector will need to know, how much memory we're using, introducing the collector abstraction will allow the GC to hook into the memory allocation and observe the memory usage. This has other potential applications: - We could now be embedded into applications that can't use the libc allocator. - There could be an allocator that limits the total amount of used memory, e.g. for sandboxing purposes. - In our tests we could use this to simulate out of memory conditions (implement an allocator that fails at the n-th allocation, increase n by one and restart the test until there are no more faked OOM conditions). The function signature of the allocator is basically exactly the same as the one Lua uses.
2022-02-08 21:53:13 +00:00
apfl_token_deinit(tt->allocator, tok);
Initial commit
2021-12-10 20:22:16 +00:00
return false;
}
if (tok->position.line != line || tok->position.col != col) {
Make line and column size_t
2023-02-16 20:41:02 +00:00
test_failf(
tt->t,
"Got token at wrong position %" PRIuMAX ":%" PRIuMAX
" (wanted %" PRIuMAX ":%" PRIuMAX ")",
(uintmax_t)tok->position.line,
(uintmax_t)tok->position.col,
(uintmax_t)line,
(uintmax_t)col
);
Initial commit
2021-12-10 20:22:16 +00:00
}
return true;
}
static void
Make line and column size_t
2023-02-16 20:41:02 +00:00
expect_simple_token(struct tokenizer_test *tt, size_t line, size_t col, enum apfl_token_type type)
Initial commit
2021-12-10 20:22:16 +00:00
{
struct apfl_token tok;
if (expect_token(tt, line, col, type, &tok)) {
Introduce allocator abstraction We now no longer call malloc/free/... directly, but use an allocator object that is passed around. This was mainly done as a preparation for a garbage collector: The collector will need to know, how much memory we're using, introducing the collector abstraction will allow the GC to hook into the memory allocation and observe the memory usage. This has other potential applications: - We could now be embedded into applications that can't use the libc allocator. - There could be an allocator that limits the total amount of used memory, e.g. for sandboxing purposes. - In our tests we could use this to simulate out of memory conditions (implement an allocator that fails at the n-th allocation, increase n by one and restart the test until there are no more faked OOM conditions). The function signature of the allocator is basically exactly the same as the one Lua uses.
2022-02-08 21:53:13 +00:00
apfl_token_deinit(tt->allocator, &tok);
Initial commit
2021-12-10 20:22:16 +00:00
}
}
static void
Make line and column size_t
2023-02-16 20:41:02 +00:00
expect_text_token(struct tokenizer_test *tt, size_t line, size_t col, enum apfl_token_type type, const char *text)
Initial commit
2021-12-10 20:22:16 +00:00
{
struct apfl_token tok;
if (expect_token(tt, line, col, type, &tok)) {
Add apfl_string_eq macro It's easy to forget the `== 0` after the apfl_string_cmp() call, so let's add a macro that does the job for us.
2022-01-02 15:51:19 +00:00
if (!apfl_string_eq(text, tok.text)) {
parser/tokenizer: Save textual data as refcounted strings This avoids creating refcounted strings during evaluation and makes it easier to use the same parsed string in multiple places (should be useful once we implement functions).
2022-01-18 20:18:27 +00:00
struct apfl_string_view sv = apfl_string_view_from(tok.text);
test_failf(tt->t, "Token has wrong content. have=\"" APFL_STR_FMT "\", want=\"%s\"", APFL_STR_FMT_ARGS(sv), text);
Initial commit
2021-12-10 20:22:16 +00:00
}
Introduce allocator abstraction We now no longer call malloc/free/... directly, but use an allocator object that is passed around. This was mainly done as a preparation for a garbage collector: The collector will need to know, how much memory we're using, introducing the collector abstraction will allow the GC to hook into the memory allocation and observe the memory usage. This has other potential applications: - We could now be embedded into applications that can't use the libc allocator. - There could be an allocator that limits the total amount of used memory, e.g. for sandboxing purposes. - In our tests we could use this to simulate out of memory conditions (implement an allocator that fails at the n-th allocation, increase n by one and restart the test until there are no more faked OOM conditions). The function signature of the allocator is basically exactly the same as the one Lua uses.
2022-02-08 21:53:13 +00:00
apfl_token_deinit(tt->allocator, &tok);
Initial commit
2021-12-10 20:22:16 +00:00
}
}
Add tokenizer tests to check binary string handling
2022-01-06 21:50:44 +00:00
static void
Make line and column size_t
2023-02-16 20:41:02 +00:00
expect_text_token_sv(struct tokenizer_test *tt, size_t line, size_t col, enum apfl_token_type type, struct apfl_string_view text)
Add tokenizer tests to check binary string handling
2022-01-06 21:50:44 +00:00
{
struct apfl_token tok;
if (expect_token(tt, line, col, type, &tok)) {
if (!apfl_string_eq(text, tok.text)) {
parser/tokenizer: Save textual data as refcounted strings This avoids creating refcounted strings during evaluation and makes it easier to use the same parsed string in multiple places (should be useful once we implement functions).
2022-01-18 20:18:27 +00:00
struct apfl_string_view sv = apfl_string_view_from(tok.text);
test_failf(tt->t, "Token has wrong content. have=\"" APFL_STR_FMT "\", want=\"%s\"", APFL_STR_FMT_ARGS(sv), text);
Add tokenizer tests to check binary string handling
2022-01-06 21:50:44 +00:00
}
Introduce allocator abstraction We now no longer call malloc/free/... directly, but use an allocator object that is passed around. This was mainly done as a preparation for a garbage collector: The collector will need to know, how much memory we're using, introducing the collector abstraction will allow the GC to hook into the memory allocation and observe the memory usage. This has other potential applications: - We could now be embedded into applications that can't use the libc allocator. - There could be an allocator that limits the total amount of used memory, e.g. for sandboxing purposes. - In our tests we could use this to simulate out of memory conditions (implement an allocator that fails at the n-th allocation, increase n by one and restart the test until there are no more faked OOM conditions). The function signature of the allocator is basically exactly the same as the one Lua uses.
2022-02-08 21:53:13 +00:00
apfl_token_deinit(tt->allocator, &tok);
Add tokenizer tests to check binary string handling
2022-01-06 21:50:44 +00:00
}
}
Initial commit
2021-12-10 20:22:16 +00:00
static void
Make line and column size_t
2023-02-16 20:41:02 +00:00
expect_number_token(struct tokenizer_test *tt, size_t line, size_t col, enum apfl_token_type type, apfl_number num)
Initial commit
2021-12-10 20:22:16 +00:00
{
struct apfl_token tok;
if (expect_token(tt, line, col, type, &tok)) {
if (tok.number != num) {
test_failf(tt->t, "Token has wrong content. have=%f, want=%f", tok.number, num);
}
Introduce allocator abstraction We now no longer call malloc/free/... directly, but use an allocator object that is passed around. This was mainly done as a preparation for a garbage collector: The collector will need to know, how much memory we're using, introducing the collector abstraction will allow the GC to hook into the memory allocation and observe the memory usage. This has other potential applications: - We could now be embedded into applications that can't use the libc allocator. - There could be an allocator that limits the total amount of used memory, e.g. for sandboxing purposes. - In our tests we could use this to simulate out of memory conditions (implement an allocator that fails at the n-th allocation, increase n by one and restart the test until there are no more faked OOM conditions). The function signature of the allocator is basically exactly the same as the one Lua uses.
2022-02-08 21:53:13 +00:00
apfl_token_deinit(tt->allocator, &tok);
Initial commit
2021-12-10 20:22:16 +00:00
}
}
Tokenizer: Disallow ASCII control characters outside strings
2022-01-07 22:39:06 +00:00
static void
expect_error(struct tokenizer_test *tt, enum apfl_error_type want)
{
struct apfl_token tok;
switch (apfl_tokenizer_next(tt->tokenizer, false)) {
case APFL_PARSE_OK:
tok = apfl_tokenizer_get_token(tt->tokenizer);
test_failf(tt->t, "Expected error, got token of type %s instead", apfl_token_type_name(tok.type));
Introduce allocator abstraction We now no longer call malloc/free/... directly, but use an allocator object that is passed around. This was mainly done as a preparation for a garbage collector: The collector will need to know, how much memory we're using, introducing the collector abstraction will allow the GC to hook into the memory allocation and observe the memory usage. This has other potential applications: - We could now be embedded into applications that can't use the libc allocator. - There could be an allocator that limits the total amount of used memory, e.g. for sandboxing purposes. - In our tests we could use this to simulate out of memory conditions (implement an allocator that fails at the n-th allocation, increase n by one and restart the test until there are no more faked OOM conditions). The function signature of the allocator is basically exactly the same as the one Lua uses.
2022-02-08 21:53:13 +00:00
apfl_token_deinit(tt->allocator, &tok);
Tokenizer: Disallow ASCII control characters outside strings
2022-01-07 22:39:06 +00:00
return;
case APFL_PARSE_EOF:
test_fatalf(tt->t, "Got an EOF instead of a token");
break;
case APFL_PARSE_ERROR:
break;
}
struct apfl_error have = apfl_tokenizer_get_error(tt->tokenizer);
if (have.type != want) {
test_failf(tt->t, "Expected error of type %s, got %s instead", apfl_error_type_name(want), apfl_error_type_name(have.type));
}
}
Initial commit
2021-12-10 20:22:16 +00:00
TEST(empty, t) {
struct tokenizer_test *tt = new_tokenizer_test(t, "");
expect_eof(tt);
destroy_tokenizer_test(tt);
}
TEST(simple_variable, t) {
struct tokenizer_test *tt = new_tokenizer_test(t, "hello");
expect_text_token(tt, 1, 1, APFL_TOK_NAME, "hello");
expect_eof(tt);
destroy_tokenizer_test(tt);
}
TEST(numbers, t) {
struct tokenizer_test *tt = new_tokenizer_test(t,
// 1 2
// 12345678901234567890123456789
"0 1 -1 1.5 -2.25 666 0xfe 0o15"
);
expect_number_token(tt, 1, 1, APFL_TOK_NUMBER, 0);
expect_number_token(tt, 1, 3, APFL_TOK_NUMBER, 1);
expect_number_token(tt, 1, 5, APFL_TOK_NUMBER, -1);
expect_number_token(tt, 1, 8, APFL_TOK_NUMBER, 1.5);
expect_number_token(tt, 1, 12, APFL_TOK_NUMBER, -2.25);
expect_number_token(tt, 1, 18, APFL_TOK_NUMBER, 666);
expect_number_token(tt, 1, 22, APFL_TOK_NUMBER, 0xfe);
expect_number_token(tt, 1, 27, APFL_TOK_NUMBER, 015);
expect_eof(tt);
destroy_tokenizer_test(tt);
}
TEST(names, t) {
struct tokenizer_test *tt = new_tokenizer_test(t, "foo bar --->-->-> Δv == a= x12=x+=");
expect_text_token (tt, 1, 1, APFL_TOK_NAME, "foo");
expect_text_token (tt, 1, 5, APFL_TOK_NAME, "bar");
expect_text_token (tt, 1, 9, APFL_TOK_NAME, "--");
expect_simple_token(tt, 1, 11, APFL_TOK_MAPSTO);
expect_text_token (tt, 1, 13, APFL_TOK_NAME, "-");
expect_simple_token(tt, 1, 14, APFL_TOK_MAPSTO);
expect_simple_token(tt, 1, 16, APFL_TOK_MAPSTO);
expect_text_token (tt, 1, 19, APFL_TOK_NAME, "Δv");
expect_text_token (tt, 1, 23, APFL_TOK_NAME, "==");
expect_text_token (tt, 1, 26, APFL_TOK_NAME, "a");
expect_simple_token(tt, 1, 27, APFL_TOK_ASSIGN);
expect_text_token (tt, 1, 29, APFL_TOK_NAME, "x12");
expect_simple_token(tt, 1, 32, APFL_TOK_ASSIGN);
destroy_tokenizer_test(tt);
}
Add tokenizer test case During development on the parser I got a "malloc(): corrupted top size" error in the tokenizer when parsing `a=a`. I wrote this test to see if it was really a problem with the tokenizer. It wasn't, lets keep the test nontheless.
2021-12-15 22:17:46 +00:00
TEST(assignment, t) {
struct tokenizer_test *tt = new_tokenizer_test(t, "a=a");
expect_text_token (tt, 1, 1, APFL_TOK_NAME, "a");
expect_simple_token(tt, 1, 2, APFL_TOK_ASSIGN);
expect_text_token (tt, 1, 3, APFL_TOK_NAME, "a");
expect_eof(tt);
destroy_tokenizer_test(tt);
}
Initial commit
2021-12-10 20:22:16 +00:00
TEST(all_tokens, t) {
struct tokenizer_test *tt = new_tokenizer_test(t,
// 1234567
"# test\n"
// 1 2345 678901234567
"\"abc\" def g-h*=i\n"
// 123456789012345678901234567890
"1234.5 -10 0x2A 0b101010 0o52\n"
// 12345678901 2
"'foo ;; , \\\n"
// 1234567890123456
"@ . ? ~ -> = :=\n"
Implement pairs Pairs are 2-tuples of values that are constructed and matched with the `::` operator. They can also be matched with a `:` operator, the LHS is an expression then, the pair will then only match, if the LHS matches the result of that expression. Pairs should be useful to do something similar what sum types / tagged unions do in statically typed languages, e.g. you could write something like: some := (symbol) # Somthing that creates a unique value filter-map := { _ [] -> [] f [x ~xs] -> { some:y -> [y ~(filter-map f xs)] nil -> filter-map f xs } (f x) } filter-map { x?even -> some :: (* x 10) _ -> nil } some-list
2023-03-22 22:54:03 +00:00
// 1234567
"({[]})\n"
// 1234567
": :: :="
Initial commit
2021-12-10 20:22:16 +00:00
);
expect_text_token (tt, 1, 1, APFL_TOK_COMMENT, " test");
expect_simple_token(tt, 1, 7, APFL_TOK_LINEBREAK);
expect_text_token (tt, 2, 1, APFL_TOK_STRING, "abc");
expect_text_token (tt, 2, 7, APFL_TOK_NAME, "def");
expect_text_token (tt, 2, 11, APFL_TOK_NAME, "g-h*=i");
expect_simple_token(tt, 2, 17, APFL_TOK_LINEBREAK);
expect_number_token(tt, 3, 1, APFL_TOK_NUMBER, 1234.5);
expect_number_token(tt, 3, 8, APFL_TOK_NUMBER, -10);
expect_number_token(tt, 3, 12, APFL_TOK_NUMBER, 42);
expect_number_token(tt, 3, 17, APFL_TOK_NUMBER, 42);
expect_number_token(tt, 3, 26, APFL_TOK_NUMBER, 42);
expect_simple_token(tt, 3, 30, APFL_TOK_LINEBREAK);
expect_simple_token(tt, 4, 1, APFL_TOK_STRINGIFY);
expect_text_token (tt, 4, 2, APFL_TOK_NAME, "foo");
expect_simple_token(tt, 4, 6, APFL_TOK_SEMICOLON);
expect_simple_token(tt, 4, 7, APFL_TOK_SEMICOLON);
expect_simple_token(tt, 4, 9, APFL_TOK_COMMA);
expect_simple_token(tt, 4, 11, APFL_TOK_CONTINUE_LINE);
expect_simple_token(tt, 4, 12, APFL_TOK_LINEBREAK);
expect_simple_token(tt, 5, 1, APFL_TOK_AT);
expect_simple_token(tt, 5, 3, APFL_TOK_DOT);
expect_simple_token(tt, 5, 5, APFL_TOK_QUESTION_MARK);
expect_simple_token(tt, 5, 7, APFL_TOK_EXPAND);
expect_simple_token(tt, 5, 9, APFL_TOK_MAPSTO);
expect_simple_token(tt, 5, 12, APFL_TOK_ASSIGN);
expect_simple_token(tt, 5, 14, APFL_TOK_LOCAL_ASSIGN);
expect_simple_token(tt, 5, 16, APFL_TOK_LINEBREAK);
expect_simple_token(tt, 6, 1, APFL_TOK_LPAREN);
expect_simple_token(tt, 6, 2, APFL_TOK_LBRACE);
expect_simple_token(tt, 6, 3, APFL_TOK_LBRACKET);
expect_simple_token(tt, 6, 4, APFL_TOK_RBRACKET);
expect_simple_token(tt, 6, 5, APFL_TOK_RBRACE);
expect_simple_token(tt, 6, 6, APFL_TOK_RPAREN);
Implement pairs Pairs are 2-tuples of values that are constructed and matched with the `::` operator. They can also be matched with a `:` operator, the LHS is an expression then, the pair will then only match, if the LHS matches the result of that expression. Pairs should be useful to do something similar what sum types / tagged unions do in statically typed languages, e.g. you could write something like: some := (symbol) # Somthing that creates a unique value filter-map := { _ [] -> [] f [x ~xs] -> { some:y -> [y ~(filter-map f xs)] nil -> filter-map f xs } (f x) } filter-map { x?even -> some :: (* x 10) _ -> nil } some-list
2023-03-22 22:54:03 +00:00
expect_simple_token(tt, 6, 7, APFL_TOK_LINEBREAK);
expect_simple_token(tt, 7, 1, APFL_TOK_COLON);
expect_simple_token(tt, 7, 3, APFL_TOK_DOUBLE_COLON);
expect_simple_token(tt, 7, 6, APFL_TOK_LOCAL_ASSIGN);
Initial commit
2021-12-10 20:22:16 +00:00
expect_eof(tt);
destroy_tokenizer_test(tt);
}
Add tokenizer tests to check binary string handling
2022-01-06 21:50:44 +00:00
TEST(strings_with_binary_data, t) {
struct tokenizer_test *tt = new_tokenizer_test_sv(t, (struct apfl_string_view) {
strings+io: Make chars unsigned
2023-02-13 21:31:18 +00:00
.bytes = (unsigned char []){'"', '\x00', '\xFF', '\\', 'x', '0', '0', '\\', 'x', '2', 'a', '"'},
Add tokenizer tests to check binary string handling
2022-01-06 21:50:44 +00:00
.len = 12,
});
expect_text_token_sv(tt, 1, 1, APFL_TOK_STRING, (struct apfl_string_view) {
strings+io: Make chars unsigned
2023-02-13 21:31:18 +00:00
.bytes = (unsigned char []){'\x00', '\xFF', '\x00', '\x2A'},
Add tokenizer tests to check binary string handling
2022-01-06 21:50:44 +00:00
.len = 4
});
expect_eof(tt);
destroy_tokenizer_test(tt);
}
Tokenizer: Disallow ASCII control characters outside strings
2022-01-07 22:39:06 +00:00
TEST(err_invalid_bytes, t) {
struct tokenizer_test *tt = new_tokenizer_test(t, "\x05" "foo\x01_bar\x7F" "baz");
expect_error(tt, APFL_ERR_UNEXPECTED_BYTE);
expect_text_token(tt, 1, 2, APFL_TOK_NAME, "foo");
expect_error(tt, APFL_ERR_UNEXPECTED_BYTE);
expect_text_token(tt, 1, 6, APFL_TOK_NAME, "_bar");
expect_error(tt, APFL_ERR_UNEXPECTED_BYTE);
expect_text_token(tt, 1, 11, APFL_TOK_NAME, "baz");
expect_eof(tt);
destroy_tokenizer_test(tt);
}
Initial commit
2021-12-10 20:22:16 +00:00
TESTS_BEGIN
ADDTEST(empty),
ADDTEST(simple_variable),
ADDTEST(numbers),
ADDTEST(names),
Add tokenizer test case During development on the parser I got a "malloc(): corrupted top size" error in the tokenizer when parsing `a=a`. I wrote this test to see if it was really a problem with the tokenizer. It wasn't, lets keep the test nontheless.
2021-12-15 22:17:46 +00:00
ADDTEST(assignment),
Initial commit
2021-12-10 20:22:16 +00:00
ADDTEST(all_tokens),
Add tokenizer tests to check binary string handling
2022-01-06 21:50:44 +00:00
ADDTEST(strings_with_binary_data),
Tokenizer: Disallow ASCII control characters outside strings
2022-01-07 22:39:06 +00:00
ADDTEST(err_invalid_bytes),
Initial commit
2021-12-10 20:22:16 +00:00
TESTS_END
Reference in a new issue Copy permalink