Lexmachine Versions Save

What's Changed

• updated dependency that was previously licensed with GPLv2 to a BSD 3-Clause license
• changing and to can to read properly by @jiminoc in https://github.com/timtadh/lexmachine/pull/25
• Minor correction '0' -> '1' by @martinkreibe-wk in https://github.com/timtadh/lexmachine/pull/27

New Contributors

• @jiminoc made their first contribution in https://github.com/timtadh/lexmachine/pull/25
• @martinkreibe-wk made their first contribution in https://github.com/timtadh/lexmachine/pull/27

Full Changelog: https://github.com/timtadh/lexmachine/compare/v0.2.2...v0.2.3

This release includes pr #21 and #23 . Both prs are minor bug fixes.

#21 corrects a long standing mistake in both back-ends where the lexical analysis machine may in certain circumstances skip input bytes. This occurred when the analysis proceeded several characters ahead of the final character in the ultimate best match. Because this situation is quite rare it was only recently noticed.

#23 prevents the user from specifying tokens which match the empty string. As noted in #22 there is no good way to handle this case and ensure progress. Given that I can think of not use case for tokens which match the empty string (and the problems these tokens can cause) they are being statically disallowed.

changes

• ebc72e3 Tim Henderson Merge branch 'bug-18'
• 167f8d1 Tim Henderson cleaned up test case
• 978d0bb Tim Henderson fixed out of bounds for zero length tokens
• 02854b9 Tim Henderson added regression test for issue #18
• 5a1f4c1 Tim Henderson Merge branch 'chrissimpkins-chrissimpkins-typo'
• 869336e Tim Henderson Merge branch 'fix-regression-15'
• cf9fd6c Chris Simpkins [README.md] typo fixes
• 695507b Chris Simpkins [README.md] typo fixes
• e3d2062 Tim Henderson removed print statement
• 35435dd Tim Henderson Fixed regression in DFA when matching last token
• 0804877 Tim Henderson Merge pull request #13 from Oppodelldog/Oppodelldog-patch-1
• 5ea8ac2 Tim Henderson Merge pull request #14 from Oppodelldog/Oppodelldog-patch-2
• faa2e52 Oppodelldog fixed typo in README.md
• ddd1a56 Oppodelldog fixed sample for "SPACE"
• fb38c5b Tim Henderson Merge branch 'windows-fix'
• 438c1b7 Tim Henderson upgrade the data-structures dependency to 0.5.2
• b4c4552 Tim Henderson added links in readme

This release adds a new deterministic finite automaton backend for lexmachine. For most users this will improve the speed of tokenization and is the default backend used. The NFA based engine is still available by the new function CompileNFA().