A portable Common Lisp reader that is highly customizable, can recover from errors and can return concrete syntax trees
The deprecated generic functions eclector.parse-result:source-position and eclector.parse-result:make-source-range have been removed. Clients should use eclector.base:source-position and eclector.base:make-source-range respectively instead.
The new reader eclector.base:range-length can be applied to conditions of type eclector.base:stream-position-condition (which includes almost all conditions related to syntax errors) to determine the length of the sub-sequence of the input to which the condition in question pertains.
Minor incompatible change
The part of the labeled objects protocol that allows clients to construct parse results which represent labeled objects has been changed in an incompatible way. The change allows parse results which represent labeled objects to have child parse results but requires that clients construct parse results which represent labeled objects differently: instead of eql-specializing the result parameters of methods on eclector.parse-result:make-expression-result to eclector.parse-result:**definition** and eclector.parse-result:**reference** and receiving the labeled object in the children parameters, the result parameters now have to be specialized to the classes eclector.parse-result:definition and eclector.parse-result:reference respectively. The object passed as the result argument now contains the labeled object so that the children parameter can receive child parse results.
This change is considered minor since the old mechanism described above was not documented. For now, the new mechanism also remains undocumented so that the design can be validated through experimentation before it is finalized.
The new syntax-extensions module contains a collection of syntax extensions which are implemented as either mixin classes for clients or reader macro functions.
The extended package prefix extension allows prefixing an expression with a package designator in order to read the expression with the designated package as the current package. For example
my-package::(a b)
is read as
(my-package::a my-package::b)
with this extension.
A new syntax extension which is implemented by the reader macro eclector.syntax-extensions.s-expression-comment:s-expression-comment allows commenting out s-expressions in a fashion similar to SRFI 62 for scheme. One difference is that a numeric infix argument can be used to comment out a number of s-expressions different from 1:
(frob r1 r2 :k3 4 #4; :k5 6 :k6 7)
The concrete-syntax-tree module now produces a better tree structure for certain inputs like (0 . 0). Before this change the produced CST had the same concrete-syntax-tree:atom-cst object as the concrete-syntax-tree:first and concrete-syntax-tree:rest of the outer concrete-syntax-tree:cons-cst node. After this change the concrete-syntax-tree:first child is the concrete-syntax-tree:atom-cst which corresponds to the first 0 in the input and the concrete-syntax-tree:rest child is the concrete-syntax-tree:atom-cst which corresponds to the second 0 in the input. In contrast to the previous example, an input like (#1=0 . #1#) continues to result in a single concrete-syntax-tree:atom-cst in both the concrete-syntax-tree:first and concrete-syntax-tree:rest slots of the outer concrete-syntax-tree:cons-cst object.
The deprecated function eclector.concrete-syntax-tree:cst-read has been removed.
eclector.reader:find-character receives characters names with unmodified case and is also called in the #\<single character> case so that clients have more control over character lookup.
The new generic function eclector.base:position-offset allows interested clients to refine the source positions of errors obtained by calling eclector.base:stream-position.
Some condition and restart reports have been improved.
A discussion of the relation between circular objects and custom reader macros has been added to the manual.
Problems in the eclector.reader:fixup method for hash tables have been fixed: keys were not checked for circular structure and circular structures in values were not fixed up in some cases.
Eclector provides a new protocol for handling labeled objects, that is the objects defined and referenced by the #= and ## reader macros respectively.
Eclector now avoids unnecessary fixup processing in object graphs with complicated definitions and references.
Before this change, cases like
#1=(1 #1# #2=(2 #2# ... #100=(100 #100#)))
or
#1=(1 #2=(2 ... #2#) ... #1#)
lead to unnecessary and/or repeated traversals during fixup processing.
Fixup processing is now performed in parse result objects.
Before this change, something like
(eclector.concrete-syntax-tree:read-from-string "#1=(#1#)")
produced a CST object, say cst, which failed to satisfy
(eq (cst:first cst) cst)
(eq (cst:raw (first cst)) (cst:raw cst))
The properties now hold.
Clients can use the new mixin classes eclector.concrete-syntax-tree:definition-csts-mixin and eclector.concrete-syntax-tree:reference-csts-mixin to represent labeled object definitions and references as instances of eclector.concrete-syntax-tree:definition-cst and eclector.concrete-syntax-tree:reference-cst respectively.
The stream position in conditions signaled by eclector.reader::sharpsign-colon is now always present.
When Eclector is used to produce parse results, it no longer confuses end-of-input with having read nil when nil is used as the eof-value (nil makes sense as an eof-value in that case since nil is generally not a possible parse result).
A detailed description of the constraints on return values of the generic functions in the Reader behavior protocol has been added to the manual.
The eclector-concrete-syntax-tree system now works with and requires version 0.2 of the concrete-syntax-tree system.
Eclector provides a new protocol for querying and binding behavior-changing aspects of the current state of the reader such as the current package, the current readtable and the current read base.
Clients can use this protocol to control the reader state in other ways than binding the Common Lisp variables, for example by storing the values of reader state aspects in context objects.
Furthermore, implementations which use Eclector as the Common Lisp reader can use this protocol to tie the cl:*readtable* aspect to the cl:*readtable* variable instead of the eclector.reader:*readtable* variable.
The new protocol subsumes the purpose of the generic function eclector.reader:call-with-current-package which is deprecated as of this Eclector version.
Eclector now provides and uses by default a relaxed version of the eclector.reader::sharpsign-s reader macro function which requires the input following #S to be read as a list but not necessarily be literally written as (TYPE INITARG₁ VALUE₁ …).
A detailed discussion of the topic has been added to the manual.