Skip to content

Instantly share code, notes, and snippets.

@bigos
Forked from chaitanyagupta/_reader-macros.md
Created November 26, 2016 23:49
Show Gist options
  • Select an option

  • Save bigos/5b9c9fb15285dc89c44af3e2652d729b to your computer and use it in GitHub Desktop.

Select an option

Save bigos/5b9c9fb15285dc89c44af3e2652d729b to your computer and use it in GitHub Desktop.

Revisions

  1. @chaitanyagupta chaitanyagupta revised this gist Jan 13, 2015. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,7 @@
    # Reader Macros in Common Lisp

    _This post also appears on [lisper.in](http://lisper.in/reader-macros)._

    Reader macros are perhaps not as famous as ordinary macros. While macros are a great way to create your own DSL, reader macros provide even greater flexibility by allowing you to create entirely new syntax on top of Lisp.

    Paul Graham explains them very well in [On Lisp][] (Chapter 17, Read-Macros):
  2. @chaitanyagupta chaitanyagupta revised this gist Jan 12, 2015. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -46,7 +46,7 @@ Simply put, the single quote character `'` is a little bit of syntactic sugar ad
    ```lisp
    (defun single-quote-reader (stream char)
    (declare (ignore char))
    (list 'quote (read stream t nil t)))
    (list (quote quote) (read stream t nil t)))
    (set-macro-character #\' #'single-quote-reader)
    ```
  3. @chaitanyagupta chaitanyagupta revised this gist Dec 21, 2014. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -1,6 +1,6 @@
    # Reader Macros in Common Lisp

    _This post also appears on [lisper.in][http://lisper.in/reader-macros]._
    _This post also appears on [lisper.in](http://lisper.in/reader-macros)._

    Reader macros are perhaps not as famous as ordinary macros. While macros are a great way to create your own DSL, reader macros provide even greater flexibility by allowing you to create entirely new syntax on top of Lisp.

  4. @chaitanyagupta chaitanyagupta revised this gist Dec 21, 2014. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,7 @@
    # Reader Macros in Common Lisp

    _This post also appears on [lisper.in][http://lisper.in/reader-macros]._

    Reader macros are perhaps not as famous as ordinary macros. While macros are a great way to create your own DSL, reader macros provide even greater flexibility by allowing you to create entirely new syntax on top of Lisp.

    Paul Graham explains them very well in [On Lisp][] (Chapter 17, Read-Macros):
  5. @chaitanyagupta chaitanyagupta revised this gist Mar 5, 2014. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -523,7 +523,7 @@ However, when done right, reader macros can work wonders. Take a look at [cl-int
    > #?/\bfoo\b/
    "\\bfoo\\b"

    Some possible use cases for reader macros:
    A few other possible use cases for reader macros are:

    * increase compatibility with foreign code e.g. Objective-C like calls
    * infix syntax
  6. @chaitanyagupta chaitanyagupta revised this gist Mar 5, 2014. 1 changed file with 4 additions and 1 deletion.
    5 changes: 4 additions & 1 deletion _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -416,7 +416,7 @@ Enabling this syntax inside a source file is a bit trickier. Let's assume we wri

    Consider what happens when this file is loaded using `(load (compile-file "foo.lisp"))`. `enable-json-syntax` will only get called during `LOAD`. However, by then, it is too late.

    Remember that reader macros only work at read-time i.e. when the reader is translating raw text into Lisp s-expressions. In the example above, the definition for `foobar` was translated from raw text to sexp during `COMPILE-FILE` and not `LOAD`. So what we really want is to run `enable-json-syntax` during `COMPILE-FILE` instead of `LOAD`. The simplest way to achieve this is to wrap the function call inside an `EVAL-WHEN`:
    Remember that reader macros only work at read-time i.e. when the reader is translating raw text into Lisp s-expressions. In the example above, the definition for `foobar` was translated from raw text to sexp during [`COMPILE-FILE`][compile-file] and not [`LOAD`][load]. So what we really want is to run `enable-json-syntax` during `COMPILE-FILE` instead of `LOAD`. The simplest way to achieve this is to wrap the function call inside an [`EVAL-WHEN`][eval-when]:

    ```lisp
    ;;; foo.lisp
    @@ -560,3 +560,6 @@ Source code used in this post is available in a [json-reader.lisp][]. A battery
    [Dynamic variable]: http://www.gigamonkeys.com/book/variables.html#dynamic-aka-special-variables
    [Named-Readtables]: http://common-lisp.net/project/named-readtables/
    [reddit thread]: http://www.reddit.com/r/lisp/comments/1zfim8/reader_macros_in_common_lisp/
    [eval-when]: http://www.lispworks.com/documentation/HyperSpec/Body/s_eval_w.htm
    [compile-file]: http://www.lispworks.com/documentation/HyperSpec/Body/f_cmp_fi.htm
    [load]: http://www.lispworks.com/documentation/HyperSpec/Body/f_load.htm
  7. @chaitanyagupta chaitanyagupta revised this gist Mar 5, 2014. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -378,7 +378,7 @@ Again, this is not very hard to fix. I will leave that as an exercise to the rea

    ## Packaging the new syntax

    Apart from creating new functions, you will have notice that we directly change the current readtable when associating the macro functions with characters:
    Apart from creating new functions, we also directly change the current readtable when associating macro functions with characters:

    ```lisp
    (set-macro-character +left-bracket+ 'read-left-bracket)
  8. @chaitanyagupta chaitanyagupta revised this gist Mar 5, 2014. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -109,7 +109,7 @@ To read JSON arrays, we dispatch the left bracket to `read-left-bracket`.
    (set-macro-character +left-bracket+ 'read-left-bracket)
    ```

    Note that I haven't provided a complete definition for `read-left-bracket` yet. The [readtable][] is an object that stores the association between characters and their reader macro functions, if any. [`*READTABLE*`][*readtable*] is a [dynamic variable][] that specifies the current readtable. `read-left-bracket` rebinds the current readtable to a copy of itself, and causes the comma character `,` to be associated with `read-separator` (which we will see shortly). The reason we copy and rebind the current readtable is to avoid clobbering the readtable entry for comma outside of the dynamic scope of `read-left-bracket`.
    Note that I haven't provided a complete definition for `read-left-bracket` yet. The [readtable][] is an object that stores the association between characters and their reader macro functions, if any. [`*READTABLE*`][*readtable*] is a [dynamic variable][] that specifies the current readtable. `read-left-bracket` binds the current readtable to a copy of itself, and causes the comma character `,` to be associated with `read-separator` (which we will see shortly). The reason we copy and rebind the current readtable is to avoid clobbering the readtable entry for comma outside of the dynamic scope of `read-left-bracket`.

    Here's the complete definition for `read-left-bracket`:

  9. @chaitanyagupta chaitanyagupta revised this gist Mar 5, 2014. 1 changed file with 4 additions and 4 deletions.
    8 changes: 4 additions & 4 deletions _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -387,7 +387,7 @@ Apart from creating new functions, you will have notice that we directly change
    (set-macro-character +right-brace+ 'read-delimiter)
    ```

    If we were to package the JSON reader in a library and add these lines as toplevel forms in the library's source code, we would clobber the user's _current_ readtable when the library source was loaded. This is not desirable. What we want instead is to provide user with a hook to enable/disable JSON syntax at will.
    If we were to package the JSON reader in a library and add these lines as toplevel forms in the library's source code, we would clobber the user's current readtable when the library source was loaded. This is not desirable. What we want instead is to provide user with a hook to enable/disable JSON syntax at will.

    So instead of adding these as toplevel forms, we could provide them inside a function:

    @@ -399,7 +399,7 @@ So instead of adding these as toplevel forms, we could provide them inside a fun
    (set-macro-character +right-brace+ 'read-delimiter))
    ```

    Now, when we want to enable JSON syntax from the REPL, we could simply call `enable-json-syntax`.
    Now, when users want to enable JSON syntax from the REPL, they could simply call `enable-json-syntax`.

    Enabling this syntax inside a source file is a bit trickier. Let's assume we write our file like this:

    @@ -452,9 +452,9 @@ Both these problems are solved by making a slight modification to `enable-json-s
    (set-macro-character +right-brace+ 'read-delimiter))
    ```

    We maintain a stack of previous readtables. When `enable-json-syntax` is called, the current readtable is pushed on the stack of previous readtables and a copy is made the current readtable (by assigning a new value to `*READTABLE*`). This ensures that subsequent modifications to the readtable are made on our copy and not the readtable that we inherited.
    We maintain a stack of previous readtables. When `enable-json-syntax` is called, the current readtable is pushed on the stack of previous readtables and a copy is made the current readtable. This ensures that subsequent modifications to the readtable are made on our copy and not the readtable that we inherited.

    This provides a simple way to disable JSON syntax.
    Now we can provide a simple way to disable JSON syntax.

    ```lisp
    (defun disable-json-syntax ()
  10. @chaitanyagupta chaitanyagupta revised this gist Mar 5, 2014. 1 changed file with 147 additions and 2 deletions.
    149 changes: 147 additions & 2 deletions _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -109,7 +109,7 @@ To read JSON arrays, we dispatch the left bracket to `read-left-bracket`.
    (set-macro-character +left-bracket+ 'read-left-bracket)
    ```

    Note that I haven't provided a complete definition for `read-left-bracket` yet. The readtable is an object that stores the association between characters and their reader macro functions, if any. `read-left-bracket` rebinds the current readtable to a copy of itself, and causes the comma character `,` to be associated with `read-separator` (which we will see shortly). The reason we copy and rebind the current readtable is to avoid clobbering the readtable entry for comma outside of the dynamic scope of `read-left-bracket`.
    Note that I haven't provided a complete definition for `read-left-bracket` yet. The [readtable][] is an object that stores the association between characters and their reader macro functions, if any. [`*READTABLE*`][*readtable*] is a [dynamic variable][] that specifies the current readtable. `read-left-bracket` rebinds the current readtable to a copy of itself, and causes the comma character `,` to be associated with `read-separator` (which we will see shortly). The reason we copy and rebind the current readtable is to avoid clobbering the readtable entry for comma outside of the dynamic scope of `read-left-bracket`.

    Here's the complete definition for `read-left-bracket`:

    @@ -248,7 +248,9 @@ And here's `read-left-brace`. Note that this function also relies on `read-next-

    Finally, we also associate right brace with a macro function.

    (set-macro-character +right-brace+ 'read-delimiter)
    ```lisp
    (set-macro-character +right-brace+ 'read-delimiter)
    ```

    Time for some more tests.

    @@ -374,6 +376,133 @@ One problem with `read-left-brace` is that, within its dymanic scope, it overrid

    Again, this is not very hard to fix. I will leave that as an exercise to the reader.

    ## Packaging the new syntax

    Apart from creating new functions, you will have notice that we directly change the current readtable when associating the macro functions with characters:

    ```lisp
    (set-macro-character +left-bracket+ 'read-left-bracket)
    (set-macro-character +right-bracket+ 'read-delimiter)
    (set-macro-character +left-brace+ 'read-left-brace)
    (set-macro-character +right-brace+ 'read-delimiter)
    ```

    If we were to package the JSON reader in a library and add these lines as toplevel forms in the library's source code, we would clobber the user's _current_ readtable when the library source was loaded. This is not desirable. What we want instead is to provide user with a hook to enable/disable JSON syntax at will.

    So instead of adding these as toplevel forms, we could provide them inside a function:

    ```lisp
    (defun enable-json-syntax ()
    (set-macro-character +left-bracket+ 'read-left-bracket)
    (set-macro-character +right-bracket+ 'read-delimiter)
    (set-macro-character +left-brace+ 'read-left-brace)
    (set-macro-character +right-brace+ 'read-delimiter))
    ```

    Now, when we want to enable JSON syntax from the REPL, we could simply call `enable-json-syntax`.

    Enabling this syntax inside a source file is a bit trickier. Let's assume we write our file like this:

    ```lisp
    ;;; foo.lisp
    (in-package #:foobar)
    (enable-json-syntax)
    (defun foobar ()
    ["foo", "bar"])
    ```

    Consider what happens when this file is loaded using `(load (compile-file "foo.lisp"))`. `enable-json-syntax` will only get called during `LOAD`. However, by then, it is too late.

    Remember that reader macros only work at read-time i.e. when the reader is translating raw text into Lisp s-expressions. In the example above, the definition for `foobar` was translated from raw text to sexp during `COMPILE-FILE` and not `LOAD`. So what we really want is to run `enable-json-syntax` during `COMPILE-FILE` instead of `LOAD`. The simplest way to achieve this is to wrap the function call inside an `EVAL-WHEN`:

    ```lisp
    ;;; foo.lisp
    (in-package #:foobar)
    (eval-when (:compile-toplevel :load-toplevel :execute)
    (enable-json-syntax))
    (defun foobar ()
    ["foo", "bar"])
    ```

    `EVAL-WHEN` ensures that `enable-json-syntax` is called during `COMPILE-FILE` _and_ `LOAD`, so you get sane behaviour with both `(load (compile-file "foo.lisp"))` and `(load "foo.lisp")`.

    However, this still suffers from two problems:

    1. There is no way to undo this -- we can't provide a `disable-json-syntax` because we don't keep track of the previous macro functions for these characters, if any.

    2. This still clobbers the current readtable. It might not as be a big problem when a user calls this function directly on the REPL, however since the current readtable is passed as is during `LOAD` and `COMPILE-FILE`, any modifications to the readtable will persist globally even after these operations finish.

    Both these problems are solved by making a slight modification to `enable-json-syntax`.

    ```lisp
    (defvar *previous-readtables* nil)
    (defun enable-json-syntax ()
    (push *readtable* *previous-readtables*)
    (setq *readtable* (copy-readtable))
    (set-macro-character +left-bracket+ 'read-left-bracket)
    (set-macro-character +right-bracket+ 'read-delimiter)
    (set-macro-character +left-brace+ 'read-left-brace)
    (set-macro-character +right-brace+ 'read-delimiter))
    ```

    We maintain a stack of previous readtables. When `enable-json-syntax` is called, the current readtable is pushed on the stack of previous readtables and a copy is made the current readtable (by assigning a new value to `*READTABLE*`). This ensures that subsequent modifications to the readtable are made on our copy and not the readtable that we inherited.

    This provides a simple way to disable JSON syntax.

    ```lisp
    (defun disable-json-syntax ()
    (setq *readtable* (pop *previous-readtables*)))
    ```

    Note: we must ensure that any call to `enable-json-syntax` is balanced with a subsequent call to `disable-json-syntax`. If we don't, the stack of previous readtables will get corrupted.

    This solves the first problem, but what about the second? Due to an interesting peculiarity of `LOAD` and `COMPILE-FILE` -- both of these functions bind `*READTABLE*` to its current value before working on the file -- `(setq *readtable* (copy-readtable))` by itself is enough to ensure that modifications do not persist after these operations finish. In fact, if we only ever call `enable-json-syntax` and `disable-json-syntax` from source files and never from the REPL directly, we don't need to manage the stack of previous readtables at all since dynamic binding takes care of this for us.

    Finally, it might be a bit cumbersome for your users to write `(eval-when (:compile-toplevel :load-toplevel ...))` all the time, so we could redefine these functions as macros.

    ```lisp
    (defmacro enable-json-syntax ()
    '(eval-when (:compile-toplevel :load-toplevel :execute)
    (push *readtable* *previous-readtables*)
    (setq *readtable* (copy-readtable))
    (set-macro-character +left-bracket+ 'read-left-bracket)
    (set-macro-character +right-bracket+ 'read-delimiter)
    (set-macro-character +left-brace+ 'read-left-brace)
    (set-macro-character +right-brace+ 'read-delimiter)))
    (defmacro disable-json-syntax ()
    '(eval-when (:compile-toplevel :load-toplevel :execute)
    (setq *readtable* (pop *previous-readtables*))))
    ```

    Your users can now enable/disable JSON syntax at will using these two macros. As long as calls to `enable-json-syntax` and `disable-json-syntax` are balanced, this will work well both on the REPL and in a source file. For reference, here's how a user source file might look with these definitions in place:

    ```lisp
    ;;; foo.lisp
    (in-package #:foobar)
    (enable-json-syntax)
    (defun foobar ()
    ["foo", "bar"])
    ;; Write more definitions
    (disable-json-syntax)
    ```

    It is worth mentioning [Named-Readtables][] here. This is a library designed to make readtable management as easy as package management. If you are providing a custom reader syntax in your library, consider using Named-Readtables.

    _(Thanks to [handle0174](http://www.reddit.com/r/lisp/comments/1zfim8/reader_macros_in_common_lisp/cfueii9) and [xach](http://www.reddit.com/r/lisp/comments/1zfim8/reader_macros_in_common_lisp/cft6mw2) for suggestions)_

    ## Conclusion

    Reader macros are a powerful way to expand Lisp's syntax in new and wonderful ways. However with great power comes great responsibility, so you should learn to use them with great care. For example, while a JSON reader was a good exercise in understanding reader macros, I would never use it in real world for the simple reason that easier alternatives already exist. e.g. a plain list or even a vector literal:
    @@ -383,6 +512,8 @@ Reader macros are a powerful way to expand Lisp's syntax in new and wonderful wa

    Another disadvantage of inventing new syntax is that it doesn't work well with existing development tools. e.g. JSON array and object syntax doesn't work very well with lisp and paredit modes in Emacs -- even simple operations like `C-M-f` and `C-M-b` (to go forward or backward sexp) don't work very well.

    There is a saying that goes along these lines: where a function will do, don't use a macro. A corollary for reader macros could be: where a sexp will do, don't use a reader macro. Yes, it might be a subjective call, but think twice before using a reader macro.

    However, when done right, reader macros can work wonders. Take a look at [cl-interpol][] [3], which adds support for string interpolation to Common Lisp, so you can easily embed newlines in strings or even create regexes more simply.

    > #?"foo\nbar"
    @@ -392,6 +523,15 @@ However, when done right, reader macros can work wonders. Take a look at [cl-int
    > #?/\bfoo\b/
    "\\bfoo\\b"

    Some possible use cases for reader macros:

    * increase compatibility with foreign code e.g. Objective-C like calls
    * infix syntax
    * rule system syntax
    * embedded SQL

    _(Thanks to [lispm](http://www.reddit.com/r/lisp/comments/1zfim8/reader_macros_in_common_lisp/cft6i1z))_

    Source code used in this post is available in a [json-reader.lisp][]. A battery of [tests][test.lisp] is also provided.

    ## Notes
    @@ -407,6 +547,8 @@ Source code used in this post is available in a [json-reader.lisp][]. A battery
    [list]: http://www.lispworks.com/documentation/HyperSpec/Body/t_list.htm
    [set-macro-character]: http://www.lispworks.com/documentation/HyperSpec/Body/f_set_ma.htm
    [JSON]: http://json.org/
    [Readtable]: http://www.lispworks.com/documentation/HyperSpec/Body/02_aa.htm
    [*readtable*]: http://www.lispworks.com/documentation/HyperSpec/Body/v_rdtabl.htm
    [Standard Readtable]: http://www.lispworks.com/documentation/HyperSpec/Body/02_aab.htm
    [cl-interpol]: http://weitz.de/cl-interpol/
    [sharpsign]: http://www.lispworks.com/documentation/HyperSpec/Body/02_dh.htm
    @@ -415,3 +557,6 @@ Source code used in this post is available in a [json-reader.lisp][]. A battery
    [read]: http://www.lispworks.com/documentation/HyperSpec/Body/f_rd_rd.htm
    [json-reader.lisp]: #file-json-reader-lisp
    [test.lisp]: #file-test-lisp
    [Dynamic variable]: http://www.gigamonkeys.com/book/variables.html#dynamic-aka-special-variables
    [Named-Readtables]: http://common-lisp.net/project/named-readtables/
    [reddit thread]: http://www.reddit.com/r/lisp/comments/1zfim8/reader_macros_in_common_lisp/
  11. @chaitanyagupta chaitanyagupta revised this gist Mar 5, 2014. 2 changed files with 14 additions and 11 deletions.
    20 changes: 11 additions & 9 deletions json-reader.lisp
    Original file line number Diff line number Diff line change
    @@ -80,13 +80,15 @@

    (defvar *previous-readtables* nil)

    (defun enable-json-syntax ()
    (push *readtable* *previous-readtables*)
    (setq *readtable* (copy-readtable))
    (set-macro-character +left-bracket+ 'read-left-bracket)
    (set-macro-character +right-bracket+ 'read-delimiter)
    (set-macro-character +left-brace+ 'read-left-brace)
    (set-macro-character +right-brace+ 'read-delimiter))
    (defmacro enable-json-syntax ()
    '(eval-when (:compile-toplevel :load-toplevel :execute)
    (push *readtable* *previous-readtables*)
    (setq *readtable* (copy-readtable))
    (set-macro-character +left-bracket+ 'read-left-bracket)
    (set-macro-character +right-bracket+ 'read-delimiter)
    (set-macro-character +left-brace+ 'read-left-brace)
    (set-macro-character +right-brace+ 'read-delimiter)))

    (defun disable-json-syntax ()
    (setq *readtable* (pop *previous-readtables*)))
    (defmacro disable-json-syntax ()
    '(eval-when (:compile-toplevel :load-toplevel :execute)
    (setq *readtable* (pop *previous-readtables*))))
    5 changes: 3 additions & 2 deletions test.lisp
    Original file line number Diff line number Diff line change
    @@ -8,8 +8,7 @@

    (defpackage #:json-test)

    (eval-when (:compile-toplevel :load-toplevel :execute)
    (json-reader:enable-json-syntax))
    (json-reader:enable-json-syntax)

    (defun random-number ()
    (random (expt 2 32)))
    @@ -113,3 +112,5 @@
    (let ((hash-table (elt x 0)))
    (assert (hash-table-p hash-table))
    (assert (eql (gethash "foo" hash-table) 1)))))

    (json-reader:disable-json-syntax)
  12. @chaitanyagupta chaitanyagupta revised this gist Mar 5, 2014. 2 changed files with 9 additions and 2 deletions.
    10 changes: 9 additions & 1 deletion json-reader.lisp
    Original file line number Diff line number Diff line change
    @@ -1,6 +1,7 @@
    (cl:defpackage #:json-reader
    (:use #:cl)
    (:export #:enable-json-syntax))
    (:export #:enable-json-syntax
    #:disable-json-syntax))

    (cl:in-package #:json-reader)

    @@ -77,8 +78,15 @@
    collect `(cons ,(stringify-key key) ,(transform-primitive value)) into pairs
    finally (return `(create-json-hash-table ,@pairs)))))

    (defvar *previous-readtables* nil)

    (defun enable-json-syntax ()
    (push *readtable* *previous-readtables*)
    (setq *readtable* (copy-readtable))
    (set-macro-character +left-bracket+ 'read-left-bracket)
    (set-macro-character +right-bracket+ 'read-delimiter)
    (set-macro-character +left-brace+ 'read-left-brace)
    (set-macro-character +right-brace+ 'read-delimiter))

    (defun disable-json-syntax ()
    (setq *readtable* (pop *previous-readtables*)))
    1 change: 0 additions & 1 deletion test.lisp
    Original file line number Diff line number Diff line change
    @@ -9,7 +9,6 @@
    (defpackage #:json-test)

    (eval-when (:compile-toplevel :load-toplevel :execute)
    (setq *readtable* (copy-readtable *readtable*))
    (json-reader:enable-json-syntax))

    (defun random-number ()
  13. @chaitanyagupta chaitanyagupta revised this gist Mar 5, 2014. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion test.lisp
    Original file line number Diff line number Diff line change
    @@ -6,10 +6,10 @@

    (cl:in-package #:cl-user)

    (setq *readtable* (copy-readtable *readtable*))
    (defpackage #:json-test)

    (eval-when (:compile-toplevel :load-toplevel :execute)
    (setq *readtable* (copy-readtable *readtable*))
    (json-reader:enable-json-syntax))

    (defun random-number ()
  14. @chaitanyagupta chaitanyagupta revised this gist Mar 5, 2014. 1 changed file with 25 additions and 19 deletions.
    44 changes: 25 additions & 19 deletions test.lisp
    Original file line number Diff line number Diff line change
    @@ -1,6 +1,13 @@
    ;; To run these tests,
    ;;
    ;; 1. (LOAD "json-reader.lisp") ;; load json reader
    ;; 2. (LOAD "test.lisp") ;; load this file
    ;; 3. (run-tests :json-test) ;; run the tests

    (cl:in-package #:cl-user)

    (setq *readtable* (copy-readtable *readtable*))
    (defpackage #:json-test)

    (eval-when (:compile-toplevel :load-toplevel :execute)
    (json-reader:enable-json-syntax))
    @@ -13,63 +20,62 @@
    (loop repeat (random 10)
    do (format out "~A " (random (expt 2 32))))))

    (defmacro test (name &body forms)
    `(handler-case
    (progn
    ,@forms
    (format t "~&~A~%" ',name))
    (error ()
    (format t "~&~A~%" ',name))))
    (defun run-tests (package)
    (do-symbols (s package)
    (when (fboundp s)
    (format t "~&~A: ~A" (symbol-name s)
    (handler-case (progn (funcall s) t)
    (error (c) c))))))

    (test vector-empty
    (defun json-test::vector-empty ()
    (let ((x []))
    (assert (vectorp x))
    (assert (zerop (length x)))))

    (test vector-single-element
    (defun json-test::vector-single-element ()
    (let ((x [1]))
    (assert (vectorp x))
    (assert (= (length x) 1))
    (assert (= (elt x 0) 1))))

    (test vector-true-false
    (defun json-test::vector-true-false ()
    (let ((x [true, false]))
    (assert (vectorp x))
    (assert (= (length x) 2))
    (assert (eql (elt x 0) t))
    (assert (eql (elt x 1) nil))))

    (test vector-strings
    (defun json-test::vector-strings ()
    (let ((x ["foo", "bar", "baz"]))
    (assert (vectorp x))
    (assert (= (length x) 3))
    (assert (every #'string-equal x '("foo" "bar" "baz")))))

    (test vector-lisp-forms
    (defun json-test::vector-lisp-forms ()
    (let* ((w "blah")
    (x [ "foo", 1, (+ 3 4), w ]))
    (assert (vectorp x))
    (assert (= (length x) 4))
    (assert (every #'equalp x (list "foo" 1 7 w)))))

    (test hash-table-empty
    (defun json-test::hash-table-empty ()
    (let ((x {}))
    (assert (hash-table-p x))
    (assert (zerop (hash-table-count x)))))

    (test hash-table-single-entry
    (defun json-test::hash-table-single-entry ()
    (let ((x {"foo": 1}))
    (assert (hash-table-p x))
    (assert (= (hash-table-count x) 1))
    (assert (eql (gethash "foo" x) 1))))

    (test hash-table-table-single-null-entry
    (defun json-test::hash-table-table-single-null-entry ()
    (let ((x {"foo": null}))
    (assert (hash-table-p x))
    (assert (= (hash-table-count x) 1))
    (assert (eql (gethash "foo" x) nil))))

    (test hash-table-multiple-entries
    (defun json-test::hash-table-multiple-entries ()
    (let ((x {
    "foo": 1,
    "bar": 2,
    @@ -81,7 +87,7 @@
    (assert (eql (gethash "bar" x) 2))
    (assert (eql (gethash "baz" x) 3))))

    (test hash-table-lisp-forms
    (defun json-test::hash-table-lisp-forms ()
    (let* ((w "blah")
    (x {
    "foo": 1,
    @@ -94,14 +100,14 @@
    (assert (eql (gethash "bar" x) 7))
    (assert (eql (gethash "baz" x) w))))

    (test hash-table-key-literals
    (defun json-test::hash-table-key-literals ()
    (let ((x { foo: 1, bar: 2 }))
    (assert (hash-table-p x))
    (assert (= (hash-table-count x) 2))
    (assert (eql (gethash "foo" x) 1))
    (assert (eql (gethash "bar" x) 2))))

    (test vector-includes-hash-table
    (defun json-test::vector-includes-hash-table ()
    (let ((x [ { foo: 1 } ]))
    (assert (vectorp x))
    (assert (= (length x) 1))
  15. @chaitanyagupta chaitanyagupta revised this gist Mar 4, 2014. 1 changed file with 7 additions and 7 deletions.
    14 changes: 7 additions & 7 deletions _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -1,6 +1,6 @@
    # Reader Macros in Common Lisp

    Reader macros are perhaps not as famous as ordinary macros. While macros are a great way to create your own DSL, reader macros allow even greater flexibility by allowing you to create entirely new syntax on top of Lisp.
    Reader macros are perhaps not as famous as ordinary macros. While macros are a great way to create your own DSL, reader macros provide even greater flexibility by allowing you to create entirely new syntax on top of Lisp.

    Paul Graham explains them very well in [On Lisp][] (Chapter 17, Read-Macros):

    @@ -30,7 +30,7 @@ The full syntax of `READ` looks like this:
    (read &optional input-stream eof-error-p eof-value recursive-p) => object
    ```

    All the arguments to `READ` are optional. The first is the input stream; `eof-error-p` and `eof-value` specify end of file handling; `recursive-p` should be set to true whenever `READ` is called from inside another call to `READ`. When implementing reader macros, `recursive-p` should always be true whenever `READ` is called.
    All the arguments to `READ` are optional. The first is the input stream; `eof-error-p` and `eof-value` specify end of file handling; `recursive-p` should be set to true whenever `READ` is called from inside another call to `READ`. When implementing reader macros, `recursive-p` should always be true.

    ## Quote

    @@ -63,7 +63,7 @@ When `READ` is invoked, this is how the input stream looks:
    'foo
    ^

    The current position of the stream is denoted by `^`. When `READ` is called, the next character in the stream is the quote. The reader reads this character, looks at its macro dispatch table, finds `single-quote-reader` associated with `'` and calls it. Just before `single-quote-reader` is called, this is how the stream looks:
    The current position of the stream is denoted by `^`. When `READ` is called, the next character in the stream is the quote. The reader reads this character, looks for its reader macro function, finds `single-quote-reader` and calls it. Just before `single-quote-reader` is called, this is how the stream looks:

    'foo
    ^
    @@ -96,7 +96,7 @@ We define a few constants to work with.
    (defconstant +colon+ #\:)
    ```

    To read JSON arrays, we dispatch the left bracket character to `read-left-bracket`.
    To read JSON arrays, we dispatch the left bracket to `read-left-bracket`.

    ```lisp
    (defun read-left-bracket (stream char)
    @@ -109,7 +109,7 @@ To read JSON arrays, we dispatch the left bracket character to `read-left-bracke
    (set-macro-character +left-bracket+ 'read-left-bracket)
    ```

    note that I haven't provided the complete definition for `read-left-bracket` yet. The readtable is an object that stores the association between characters and their reader macro functions, if any. `read-left-bracket` rebinds the current readtable to a copy of itself, and causes the comma character `,` to be associated with `read-separator` (which we will see shortly). The reason we copy and rebind the current readtable is to avoid clobbering the readtable entry for comma outside of the dynamic scope of `read-left-bracket`.
    Note that I haven't provided a complete definition for `read-left-bracket` yet. The readtable is an object that stores the association between characters and their reader macro functions, if any. `read-left-bracket` rebinds the current readtable to a copy of itself, and causes the comma character `,` to be associated with `read-separator` (which we will see shortly). The reason we copy and rebind the current readtable is to avoid clobbering the readtable entry for comma outside of the dynamic scope of `read-left-bracket`.

    Here's the complete definition for `read-left-bracket`:

    @@ -148,7 +148,7 @@ The bulk of the work of reading JSON objects is done by `read-next-object` -- `r
    [1, 2, "foo"]
    ^

    `read-next-object` parses and returns the first object (the number `1` in this case), and positions the stream before the next object, discarding the separator.
    `read-next-object` parses and returns the first object (the number `1` in this case), and positions the stream before the next object, discarding the separator. If there was any whitespace before the first object, it would be discarded.

    [1, 2, "foo"]
    ^
    @@ -158,7 +158,7 @@ This cycle repeats until `read-next-object` reads the last object in the array,
    [1, 2, "foo"]
    ^

    The last call to `read-next-object` notices that the immediate next character is the delimiter. It moves the stream past the delimiter and returns `NIL`, which indicates to `read-left-bracket` that all the objects in the array have been read. Since the stream pointer is now positioned after the delimiter, the reader is now free to read a new object from the input stream.
    The last call to `read-next-object` notices that the next character is the delimiter. It moves the stream past the delimiter and returns `NIL`, which indicates to `read-left-bracket` that all the objects in the array have been read. Since the stream pointer is now positioned after the delimiter, the reader is now free to read a new object from the input stream.

    [1, 2, "foo"]
    ^
  16. @chaitanyagupta chaitanyagupta revised this gist Mar 3, 2014. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,4 @@
    # Reader macros in Common Lisp
    # Reader Macros in Common Lisp

    Reader macros are perhaps not as famous as ordinary macros. While macros are a great way to create your own DSL, reader macros allow even greater flexibility by allowing you to create entirely new syntax on top of Lisp.

  17. @chaitanyagupta chaitanyagupta revised this gist Mar 3, 2014. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -383,7 +383,7 @@ Reader macros are a powerful way to expand Lisp's syntax in new and wonderful wa

    Another disadvantage of inventing new syntax is that it doesn't work well with existing development tools. e.g. JSON array and object syntax doesn't work very well with lisp and paredit modes in Emacs -- even simple operations like `C-M-f` and `C-M-b` (to go forward or backward sexp) don't work very well.

    However, when done right, reader macros can work wonders. Take a look at [cl-interpol][][3], which adds support for string interpolation to Common Lisp, so you can easily embed newlines in strings or even create regexes more simply.
    However, when done right, reader macros can work wonders. Take a look at [cl-interpol][] [3], which adds support for string interpolation to Common Lisp, so you can easily embed newlines in strings or even create regexes more simply.

    > #?"foo\nbar"
    "foo
  18. @chaitanyagupta chaitanyagupta revised this gist Mar 3, 2014. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -392,6 +392,8 @@ However, when done right, reader macros can work wonders. Take a look at [cl-int
    > #?/\bfoo\b/
    "\\bfoo\\b"

    Source code used in this post is available in a [json-reader.lisp][]. A battery of [tests][test.lisp] is also provided.

    ## Notes

    1. I have not covered how `READ` handles whitespace following the Lisp expression. When you provide `'foo` as input on the standard input, you need to follow it up with the a newline character. `READ` will read past the newline character in this case; so the stream position after `single-quote-reader` will actually go past the newline
    @@ -411,3 +413,5 @@ However, when done right, reader macros can work wonders. Take a look at [cl-int
    [comma]: http://www.lispworks.com/documentation/HyperSpec/Body/02_dg.htm
    [backquote syntax]: http://www.lispworks.com/documentation/HyperSpec/Body/02_df.htm
    [read]: http://www.lispworks.com/documentation/HyperSpec/Body/f_rd_rd.htm
    [json-reader.lisp]: #file-json-reader-lisp
    [test.lisp]: #file-test-lisp
  19. @chaitanyagupta chaitanyagupta revised this gist Mar 3, 2014. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -392,6 +392,8 @@ However, when done right, reader macros can work wonders. Take a look at [cl-int
    > #?/\bfoo\b/
    "\\bfoo\\b"

    ## Notes

    1. I have not covered how `READ` handles whitespace following the Lisp expression. When you provide `'foo` as input on the standard input, you need to follow it up with the a newline character. `READ` will read past the newline character in this case; so the stream position after `single-quote-reader` will actually go past the newline

    2. Actually, comma is already [defined][comma] in the standard readtable as a terminating macro character -- it is a part of the [backquote syntax][]. So technically, we could have done without associating comma with `read-separator`.
  20. @chaitanyagupta chaitanyagupta revised this gist Mar 3, 2014. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -218,7 +218,7 @@ Remember that you can invoke the reader by calling `READ`, so you can always see
    (LET ((X 123) (Y "foo"))
    (VECTOR X Y))

    Reading an object is not that different from reading an array. First we define a convenience function to create dictionaries which will be used by `read-left-brace`, the macro function for the left brace.
    Reading an object is not that different from reading an array. First we define a convenience function to create Lisp hash tables which will be used by `read-left-brace`, the macro function for the left brace.

    ```lisp
    (defun create-json-hash-table (&rest pairs)
    @@ -258,7 +258,7 @@ Time for some more tests.
    > { "foo": 1, "bar": 2, "baz": 3 }
    #<HASH-TABLE :TEST EQUAL :COUNT 3 {1004DC3953}>

    We can also nest dictionaries and arrays.
    We can also nest objects and arrays.

    > [{"foo": 1}, "bar", {"baz": [2, 3]}]
    #(#<HASH-TABLE :TEST EQUAL :COUNT 1 {10062D4D83}> "bar"
  21. @chaitanyagupta chaitanyagupta revised this gist Mar 3, 2014. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -70,7 +70,7 @@ The current position of the stream is denoted by `^`. When `READ` is called, the

    `single-quote-reader` calls `READ` again. The inner call to `READ` returns the symbol `foo`, so the return value of this function is the sexp `(quote foo)` (usually printed by the Lisp printer as `'foo`).

    After `single-quote-reader` returns, the stream pointer moves beyond `foo`[1].
    After `single-quote-reader` returns, the stream pointer moves beyond `foo`. [1]

    'foo
    ^
    @@ -171,7 +171,7 @@ This is how we define `read-separator`, the reader macro function for comma. Thi
    (error "Separator ~S shouldn't be read alone" char))
    ```

    You might ask, if the separator is never meant to be read directly by the reader, why bother associating it with a macro function? The thing is, its not the macro function that's important -- what we really want to do is tell the Lisp reader that the separator is a _terminating_ macro character. If this were not done, the Lisp reader could unwittingly treat the separator as part of the last object if there was no whitespace between the two. e.g. in `[foo, ...]`, the first object would be read as `foo,` instead of `foo` if comma was not a terminating macro character[2].
    You might ask, if the separator is never meant to be read directly by the reader, why bother associating it with a macro function? The thing is, its not the macro function that's important -- what we really want to do is tell the Lisp reader that the separator is a _terminating_ macro character. If this were not done, the Lisp reader could unwittingly treat the separator as part of the last object if there was no whitespace between the two. e.g. in `[foo, ...]`, the first object would be read as `foo,` instead of `foo` if comma was not a terminating macro character. [2]

    Lastly, we also need to define a macro function for the right bracket. This needs to be done for the same reason we needed to define a macro function for the separator -- so that we can tell the Lisp reader that it is a terminating macro character.

  22. @chaitanyagupta chaitanyagupta revised this gist Mar 3, 2014. 1 changed file with 173 additions and 142 deletions.
    315 changes: 173 additions & 142 deletions _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -1,6 +1,6 @@
    ## Introduction
    # Reader macros in Common Lisp

    Reader macros in Common Lisp are perhaps not as famous as ordinary macros. While macros are a great way to create your own DSL, reader macros allow even greater flexibility by allowing you to create entirely new syntax on top of Lisp.
    Reader macros are perhaps not as famous as ordinary macros. While macros are a great way to create your own DSL, reader macros allow even greater flexibility by allowing you to create entirely new syntax on top of Lisp.

    Paul Graham explains them very well in [On Lisp][] (Chapter 17, Read-Macros):

    @@ -14,7 +14,7 @@ The Lisp reader is the thing that parses and translates raw text into an AST. In

    Reader macros allow you to modify the behaviour of the Lisp reader.

    Not only can you modify the behaviour of the Lisp reader, you can also invoke it. The simplest way to invoke the Lisp reader is to call the function `READ`. When you call `READ` with no arguments, it reads text from the standard input and returns a Lisp expression as soon as it fully parses one.
    Not only can you modify the behaviour of the Lisp reader, you can also invoke it. The simplest way to invoke the Lisp reader is to call the function [`READ`][read]. When you call `READ` with no arguments, it reads text from the standard input and returns an s-expression as soon as it fully parses one.

    > (read)
    ;; Provide this input: foo
    @@ -26,108 +26,120 @@ Not only can you modify the behaviour of the Lisp reader, you can also invoke it

    The full syntax of `READ` looks like this:

    (read &optional input-stream eof-error-p eof-value recursive-p) => object
    ```lisp
    (read &optional input-stream eof-error-p eof-value recursive-p) => object
    ```

    All the arguments to `READ` are optional. The first is the input stream; `eof-error-p` and `eof-value` specify end of file handling; `recursive-p` should be set to true whenever `READ` is called from inside another call to `READ`. When implementing reader macros, `recursive-p` should always be true whenever `READ` is called.

    ## Quote

    One of the simplest (and earliest) examples of a reader macro in Common Lisp is the single quote character `'`. This reader macro translates an expression of the form `'<some-form>` into `(quote <some-form>)`.
    One of the simplest (and earliest) examples of a reader macro in Lisp is the single quote character `'`. This reader macro translates an expression of the form `'<some-form>` into `(quote <some-form>)`.

    'foo ;; translated to (quote foo)
    '(1 2 3) ;; translated to (quote (1 2 3))
    ```lisp
    'foo ;; translated to (quote foo)
    '(1 2 3) ;; translated to (quote (1 2 3))
    ```

    Simply put, the single quote character `'` is a little bit of syntactic sugar added on top of Lisp. Reader macros do just that -- allow you to add syntactic sugar to Lisp. What makes them so powerful is that, just like ordinay macros, you write reader macros in Lisp itself. For example, `'` could be defined as (example from the [Common Lisp Hyperspec][set-macro-character]):
    Simply put, the single quote character `'` is a little bit of syntactic sugar added on top of Lisp. Reader macros do just that -- allow you to add syntactic sugar to Lisp. What makes them so powerful is that, just like ordinay macros, you write reader macros in Lisp itself. For example, `'` could be defined as:

    (defun single-quote-reader (stream char)
    (declare (ignore char))
    (list 'quote (read stream t nil t)))
    ```lisp
    (defun single-quote-reader (stream char)
    (declare (ignore char))
    (list 'quote (read stream t nil t)))
    (set-macro-character #\' #'single-quote-reader)
    (set-macro-character #\' #'single-quote-reader)
    ```

    `SET-MACRO-CHARACTER` makes the Lisp reader associate the single quote character `'` with the function `single-quote-reader`. Whenever the Lisp reader encounters `'`, it calls `single-quote-reader`. This function takes two arguments -- the input stream and the character in the stream that triggered the call. Since we only associate `single-quote-reader` with `'`, `char` will always be `'`.
    [`SET-MACRO-CHARACTER`][set-macro-character] makes the Lisp reader associate the single quote character `'` with the function `single-quote-reader`. Whenever the reader encounters `'`, it calls `single-quote-reader`. This function takes two arguments -- the input stream and the character in the stream that triggered the call. `char` will always be `'` in this case.

    Let's trace step by step how this works. Assume we call `READ` on the following form:

    > (read)
    ;; Input: 'foo

    When `READ` is invoked, this is what how the input stream looks:
    When `READ` is invoked, this is how the input stream looks:

    'foo
    ^

    The current position of the stream is denoted by `^`. When `READ` is called, the next character in the stream is the quote. The reader reads this character, looks up its macro dispatch table, find `single-quote-reader` associated with `'` and calls it. When `single-quote-reader` is invoked, this is how the stream looks:
    The current position of the stream is denoted by `^`. When `READ` is called, the next character in the stream is the quote. The reader reads this character, looks at its macro dispatch table, finds `single-quote-reader` associated with `'` and calls it. Just before `single-quote-reader` is called, this is how the stream looks:

    'foo
    ^

    `single-quote-reader` calls `READ` again. The inner call to `READ` returns the symbol `foo`, so this function returns the form `(quote foo)` (which is usually printed by the Lisp printer as `'foo`). After `single-quote-reader` returns, the stream position moves forward[1]:
    `single-quote-reader` calls `READ` again. The inner call to `READ` returns the symbol `foo`, so the return value of this function is the sexp `(quote foo)` (usually printed by the Lisp printer as `'foo`).

    After `single-quote-reader` returns, the stream pointer moves beyond `foo`[1].

    'foo
    ^

    Remember that reader macro functions are run at read time, so `single-quote-reader` is called only while the Lisp reader is parsing raw text. If the reader encountered `'foo` while parsing a larger expression, the return value of this function will become a part of the larger s-expression.

    ## JSON

    Let's work with a more complicated example. Suppose we wanted Lisp to natively recognize [JSON][] syntax:
    Let's work with a more complicated example. Let's make the Lisp reader recognize [JSON][] syntax.

    > [1, 2, "foo"]
    #(1 2 "foo")

    > { "foo": 1, "bar": 2 }
    #<HASH-TABLE :TEST EQUAL :COUNT 2 {1004EA5DD3}>

    We define a few constants that we will work with.
    We define a few constants to work with.

    (defconstant +left-bracket+ #\[)
    (defconstant +right-bracket+ #\])
    (defconstant +left-brace+ #\{)
    (defconstant +right-brace+ #\})
    (defconstant +comma+ #\,)
    (defconstant +colon+ #\:)
    ```lisp
    (defconstant +left-bracket+ #\[)
    (defconstant +right-bracket+ #\])
    (defconstant +left-brace+ #\{)
    (defconstant +right-brace+ #\})
    (defconstant +comma+ #\,)
    (defconstant +colon+ #\:)
    ```

    To read JSON arrays, we dispatch the left bracket character to `read-left-bracket`.

    (defun read-left-bracket (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    ...
    ))
    ```lisp
    (defun read-left-bracket (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    ...
    ))
    (set-macro-character +left-bracket+ 'read-left-bracket)
    (set-macro-character +left-bracket+ 'read-left-bracket)
    ```

    Note that I haven't provided a complete definition of `read-left-bracket` yet. The readtable is an object that stores the association between characters and their reader macro functions, if any. `read-left-bracket` rebinds the current readtable to a copy of itself, and causes the comma character `,` to be associated with `read-separator` (which we will see shortly). The reason we copy and rebind the current readtable is to avoid clobbering the readtable entry for comma outside of the dynamic scope of `read-left-bracket`.
    note that I haven't provided the complete definition for `read-left-bracket` yet. The readtable is an object that stores the association between characters and their reader macro functions, if any. `read-left-bracket` rebinds the current readtable to a copy of itself, and causes the comma character `,` to be associated with `read-separator` (which we will see shortly). The reason we copy and rebind the current readtable is to avoid clobbering the readtable entry for comma outside of the dynamic scope of `read-left-bracket`.

    Here's the complete definition for `read-left-bracket`:

    (defun read-next-object (separator delimiter
    &optional (input-stream *standard-input*))
    (flet ((peek-next-char () (peek-char t input-stream t nil t))
    (discard-next-char () (read-char input-stream t nil t)))
    (if (and delimiter (char= (peek-next-char) delimiter))
    (progn
    (discard-next-char)
    nil)
    (let* ((object (read input-stream t nil t))
    (next-char (peek-next-char)))
    (cond
    ((char= next-char separator) (discard-next-char))
    ((and delimiter (char= next-char delimiter)) nil)
    (t (error "Unexpected next char: ~S" next-char)))
    object))))

    (defun read-left-bracket (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (loop
    for object = (read-next-object +comma+ +right-bracket+ stream)
    while object
    collect object into objects
    finally (return `(vector ,@objects)))))
    ```lisp
    (defun read-next-object (separator delimiter
    &optional (input-stream *standard-input*))
    (flet ((peek-next-char () (peek-char t input-stream t nil t))
    (discard-next-char () (read-char input-stream t nil t)))
    (if (and delimiter (char= (peek-next-char) delimiter))
    (progn
    (discard-next-char)
    nil)
    (let* ((object (read input-stream t nil t))
    (next-char (peek-next-char)))
    (cond
    ((char= next-char separator) (discard-next-char))
    ((and delimiter (char= next-char delimiter)) nil)
    (t (error "Unexpected next char: ~S" next-char)))
    object))))
    (defun read-left-bracket (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (loop
    for object = (read-next-object +comma+ +right-bracket+ stream)
    while object
    collect object into objects
    finally (return `(vector ,@objects)))))
    ```

    The bulk of the work of reading JSON objects is done by `read-next-object` -- `read-left-bracket` calls this function repeatedly until it returns `NIL`. The list of objects is then transformed into `(vector obj1 obj2 ...)` which, at runtime, will return a Lisp vector.

    @@ -151,32 +163,37 @@ The last call to `read-next-object` notices that the immediate next character is
    [1, 2, "foo"]
    ^

    This is how we define `read-separator`, which was associated with comma as its reader macro function inside the dynamic scope of `read-left-bracket`. This function signals an error whenever it is called -- it is never meant to be called directly by the reader.
    This is how we define `read-separator`, the reader macro function for comma. This function signals an error whenever it is called -- it is never meant to be called directly by the reader.

    (defun read-separator (stream char)
    (declare (ignore stream))
    (error "Separator ~S shouldn't be read alone" char))
    ```lisp
    (defun read-separator (stream char)
    (declare (ignore stream))
    (error "Separator ~S shouldn't be read alone" char))
    ```

    You might ask, if the separator is never meant to be read directly by the reader, why bother associating it with a macro function? The thing is, its not the macro function that's important -- what we really want to do is tell the Lisp reader that the separator is a terminating macro character. If this were not done, the Lisp reader could unwittingly treat the separator as part of the last object if there was no whitespace between the two. e.g. in `[foo, ...]`, the first object would be read as `foo,` instead of `foo` if comma was not a terminating macro character[2].
    You might ask, if the separator is never meant to be read directly by the reader, why bother associating it with a macro function? The thing is, its not the macro function that's important -- what we really want to do is tell the Lisp reader that the separator is a _terminating_ macro character. If this were not done, the Lisp reader could unwittingly treat the separator as part of the last object if there was no whitespace between the two. e.g. in `[foo, ...]`, the first object would be read as `foo,` instead of `foo` if comma was not a terminating macro character[2].

    Lastly, we also need to define a macro function for the right bracket. This needs to be done for the same reason we needed to define a macro function for the separator -- so that we can tell the Lisp reader that it is a terminating macro character.

    (defun read-delimiter (stream char)
    (declare (ignore stream))
    (error "Delimiter ~S shouldn't be read alone" char))

    (set-macro-character +right-bracket+ 'read-delimiter)
    ```lisp
    (defun read-delimiter (stream char)
    (declare (ignore stream))
    (error "Delimiter ~S shouldn't be read alone" char))
    `read-delimiter` is very similar to `read-separator` since a delimiter too is not supposed to be read directly on the stream.
    (set-macro-character +right-bracket+ 'read-delimiter)
    ```

    Now we have everything in place to read a JSON array in Lisp. Time for some tests.

    > []
    #()

    > [1, 2, 3]
    #(1 2 3)

    > [ "foo", "bar", "baz" ]
    #("foo" "bar" "baz")

    > [1, 2, "foo"]
    #(1 2 "foo")

    @@ -203,34 +220,41 @@ Remember that you can invoke the reader by calling `READ`, so you can always see

    Reading an object is not that different from reading an array. First we define a convenience function to create dictionaries which will be used by `read-left-brace`, the macro function for the left brace.

    (defun create-json-hash-table (&rest pairs)
    (let ((hash-table (make-hash-table :test #'equal)))
    (loop for (key . value) in pairs
    do (setf (gethash key hash-table) value))
    hash-table))
    ```lisp
    (defun create-json-hash-table (&rest pairs)
    (let ((hash-table (make-hash-table :test #'equal)))
    (loop for (key . value) in pairs
    do (setf (gethash key hash-table) value))
    hash-table))
    ```

    And here's `read-left-brace`. Note that this function also relies on `read-next-object` to get the bulk of its work done.

    (defun read-left-brace (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (set-macro-character +colon+ 'read-separator)
    (loop
    for key = (read-next-object +colon+ +right-brace+ stream)
    while key
    for value = (read-next-object +comma+ +right-brace+ stream)
    collect `(cons ,key ,value) into pairs
    finally (return `(create-json-hash-table ,@pairs)))))

    (set-macro-character +left-brace+ 'read-left-brace)
    ```lisp
    (defun read-left-brace (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (set-macro-character +colon+ 'read-separator)
    (loop
    for key = (read-next-object +colon+ +right-brace+ stream)
    while key
    for value = (read-next-object +comma+ +right-brace+ stream)
    collect `(cons ,key ,value) into pairs
    finally (return `(create-json-hash-table ,@pairs)))))
    (set-macro-character +left-brace+ 'read-left-brace)
    ```

    Finally, we also associate right brace with a macro function.

    (set-macro-character +right-brace+ 'read-delimiter)

    Time for some more tests.

    CL-USER> {}
    #<HASH-TABLE :TEST EQUAL :COUNT 0 {1005EEEBE3}>

    > { "foo": 1, "bar": 2, "baz": 3 }
    #<HASH-TABLE :TEST EQUAL :COUNT 3 {1004DC3953}>

    @@ -250,42 +274,46 @@ The generated sexp:

    ### true, false, null

    Our JSON reader can handle numbers, strings, objects and arrays fairly well. However, it can still not handle three primitives in JSON -- `true`, `false`, and `null`. Its also fairly easy to fix. When the reader encounters these JSON primitives, it interprets them as Lisp symbols. All we have to do is transform these into an appropriate Lisp value. To this end, we define a function `transform-primitive`.

    (defun transform-primitive (value)
    (if (symbolp value)
    (cond
    ((string-equal (symbol-name value) "true") t)
    ((string-equal (symbol-name value) "false") nil)
    ((string-equal (symbol-name value) "null") nil)
    (t value))
    value))

    Note that `transform-primitive` transforms both `false` and `null` from JSON to `NIL` in Lisp. That's because in Lisp, the two are represented by the same value. You can always choose a different representation if you want.

    We start using `transform-primitive` in our reader macro functions now.

    (defun read-left-bracket (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (loop
    for object = (read-next-object +comma+ +right-bracket+ stream)
    while object
    collect (transform-primitive object) into objects
    finally (return `(vector ,@objects)))))

    (defun read-left-brace (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (set-macro-character +colon+ 'read-separator)
    (loop
    for key = (read-next-object +colon+ +right-brace+ stream)
    while key
    for value = (read-next-object +comma+ +right-brace+ stream)
    collect `(cons ,key ,(transform-primitive value)) into pairs
    finally (return `(create-json-hash-table ,@pairs)))))
    Our JSON reader can handle numbers, strings, objects and arrays fairly well. However, it can still not handle three primitives in JSON -- `true`, `false`, and `null`. This is fairly easy to fix. When the reader encounters these JSON primitives, it interprets them as Lisp symbols. All we have to do is transform these into an appropriate Lisp value. To this end, we define a function `transform-primitive`.

    ```lisp
    (defun transform-primitive (value)
    (if (symbolp value)
    (cond
    ((string-equal (symbol-name value) "true") t)
    ((string-equal (symbol-name value) "false") nil)
    ((string-equal (symbol-name value) "null") nil)
    (t value))
    value))
    ```

    `transform-primitive` transforms both `false` and `null` from JSON to `NIL` in Lisp. That's because in Lisp, the two are represented by the same value. You can always choose a different representation if you want.

    We can start using `transform-primitive` in our reader macro functions now.

    ```lisp
    (defun read-left-bracket (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (loop
    for object = (read-next-object +comma+ +right-bracket+ stream)
    while object
    collect (transform-primitive object) into objects
    finally (return `(vector ,@objects)))))
    (defun read-left-brace (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (set-macro-character +colon+ 'read-separator)
    (loop
    for key = (read-next-object +colon+ +right-brace+ stream)
    while key
    for value = (read-next-object +comma+ +right-brace+ stream)
    collect `(cons ,key ,(transform-primitive value)) into pairs
    finally (return `(create-json-hash-table ,@pairs)))))
    ```

    Let's test this:

    @@ -309,22 +337,26 @@ A look at the generated sexp makes clear why we get the runtime thinks `foo` is

    This is also easy to fix using the `stringify-key` function.

    (defun stringify-key (key)
    (etypecase key
    (symbol (string-downcase (string key)))
    (string key)))

    (defun read-left-brace (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (set-macro-character +colon+ 'read-separator)
    (loop
    for key = (read-next-object +colon+ +right-brace+ stream)
    while key
    for value = (read-next-object +comma+ +right-brace+ stream)
    collect `(cons ,(stringify-key key) ,(transform-primitive value)) into pairs
    finally (return `(create-json-hash-table ,@pairs)))))
    ```lisp
    (defun stringify-key (key)
    (etypecase key
    (symbol (string-downcase (string key)))
    (string key)))
    (defun read-left-brace (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (set-macro-character +colon+ 'read-separator)
    (loop
    for key = (read-next-object +colon+ +right-brace+ stream)
    while key
    for value = (read-next-object +comma+ +right-brace+ stream)
    collect `(cons ,(stringify-key key) ,(transform-primitive value)) into pairs
    finally (return `(create-json-hash-table ,@pairs)))))
    ```

    Some tests:

    > { foo: 1 }
    #<HASH-TABLE :TEST EQUAL :COUNT 1 {1004248763}>
    @@ -360,8 +392,6 @@ However, when done right, reader macros can work wonders. Take a look at [cl-int
    > #?/\bfoo\b/
    "\\bfoo\\b"



    1. I have not covered how `READ` handles whitespace following the Lisp expression. When you provide `'foo` as input on the standard input, you need to follow it up with the a newline character. `READ` will read past the newline character in this case; so the stream position after `single-quote-reader` will actually go past the newline

    2. Actually, comma is already [defined][comma] in the standard readtable as a terminating macro character -- it is a part of the [backquote syntax][]. So technically, we could have done without associating comma with `read-separator`.
    @@ -378,3 +408,4 @@ However, when done right, reader macros can work wonders. Take a look at [cl-int
    [sharpsign]: http://www.lispworks.com/documentation/HyperSpec/Body/02_dh.htm
    [comma]: http://www.lispworks.com/documentation/HyperSpec/Body/02_dg.htm
    [backquote syntax]: http://www.lispworks.com/documentation/HyperSpec/Body/02_df.htm
    [read]: http://www.lispworks.com/documentation/HyperSpec/Body/f_rd_rd.htm
  23. @chaitanyagupta chaitanyagupta revised this gist Mar 3, 2014. 1 changed file with 4 additions and 6 deletions.
    10 changes: 4 additions & 6 deletions _reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -157,7 +157,7 @@ This is how we define `read-separator`, which was associated with comma as its r
    (declare (ignore stream))
    (error "Separator ~S shouldn't be read alone" char))

    You might ask, if the separator is never meant to be read directly by the reader, why bother associating it with a macro function? The thing is, its not the macro function that's important -- what we really want to do is tell the Lisp reader that the separator is a terminating macro character. If this were not done, the Lisp reader could unwittingly treat the separator as part of the last object if there was no whitespace between the two. e.g. in `[foo, ...]`, the first object would be read as `foo,` instead of `foo` if comma was not a terminating macro character[3].
    You might ask, if the separator is never meant to be read directly by the reader, why bother associating it with a macro function? The thing is, its not the macro function that's important -- what we really want to do is tell the Lisp reader that the separator is a terminating macro character. If this were not done, the Lisp reader could unwittingly treat the separator as part of the last object if there was no whitespace between the two. e.g. in `[foo, ...]`, the first object would be read as `foo,` instead of `foo` if comma was not a terminating macro character[2].

    Lastly, we also need to define a macro function for the right bracket. This needs to be done for the same reason we needed to define a macro function for the separator -- so that we can tell the Lisp reader that it is a terminating macro character.

    @@ -351,7 +351,7 @@ Reader macros are a powerful way to expand Lisp's syntax in new and wonderful wa

    Another disadvantage of inventing new syntax is that it doesn't work well with existing development tools. e.g. JSON array and object syntax doesn't work very well with lisp and paredit modes in Emacs -- even simple operations like `C-M-f` and `C-M-b` (to go forward or backward sexp) don't work very well.

    However, when done right, reader macros can work wonders. Take a look at [cl-interpol][][4], which adds support for string interpolation to Common Lisp, so you can easily embed newlines in strings or even create regexes more simply.
    However, when done right, reader macros can work wonders. Take a look at [cl-interpol][][3], which adds support for string interpolation to Common Lisp, so you can easily embed newlines in strings or even create regexes more simply.

    > #?"foo\nbar"
    "foo
    @@ -364,11 +364,9 @@ However, when done right, reader macros can work wonders. Take a look at [cl-int

    1. I have not covered how `READ` handles whitespace following the Lisp expression. When you provide `'foo` as input on the standard input, you need to follow it up with the a newline character. `READ` will read past the newline character in this case; so the stream position after `single-quote-reader` will actually go past the newline

    2.
    2. Actually, comma is already [defined][comma] in the standard readtable as a terminating macro character -- it is a part of the [backquote syntax][]. So technically, we could have done without associating comma with `read-separator`.

    3. Actually, comma is already [defined][comma] in the standard readtable as a terminating macro character -- it is a part of the [backquote syntax][]. So technically, we could have done without associating comma with `read-separator`.

    4. Technically, cl-interpol doesn't define a new reader macro, it installs `?` (question mark) as a "sub character" of the standard [dispatching macro character][sharpsign] `#` (sharpsign). Dispatching macro characters are a special class of reader macros which allow you to "make the most of the ASCII character set; one can only have so many one-character read-macros." (On Lisp, Chapter 17, Read-Macros)
    3. Technically, cl-interpol doesn't define a new reader macro, it installs `?` (question mark) as a "sub character" of the standard [dispatching macro character][sharpsign] `#` (sharpsign). Dispatching macro characters are a special class of reader macros which allow you to "make the most of the ASCII character set; one can only have so many one-character read-macros." (On Lisp, Chapter 17, Read-Macros)

    [On Lisp]: http://www.paulgraham.com/onlisp.html
    [atom]: http://www.lispworks.com/documentation/HyperSpec/Body/t_atom.htm
  24. @chaitanyagupta chaitanyagupta renamed this gist Mar 3, 2014. 1 changed file with 0 additions and 0 deletions.
    File renamed without changes.
  25. @chaitanyagupta chaitanyagupta revised this gist Mar 3, 2014. 2 changed files with 194 additions and 0 deletions.
    84 changes: 84 additions & 0 deletions json-reader.lisp
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,84 @@
    (cl:defpackage #:json-reader
    (:use #:cl)
    (:export #:enable-json-syntax))

    (cl:in-package #:json-reader)

    (defconstant +left-bracket+ #\[)
    (defconstant +right-bracket+ #\])
    (defconstant +left-brace+ #\{)
    (defconstant +right-brace+ #\})
    (defconstant +comma+ #\,)
    (defconstant +colon+ #\:)

    (defun transform-primitive (value)
    (if (symbolp value)
    (cond
    ((string-equal (symbol-name value) "true") t)
    ((string-equal (symbol-name value) "false") nil)
    ((string-equal (symbol-name value) "null") nil)
    (t value))
    value))

    (defun create-json-hash-table (&rest pairs)
    (let ((hash-table (make-hash-table :test #'equal)))
    (loop for (key . value) in pairs
    do (setf (gethash key hash-table) value))
    hash-table))

    (defun read-next-object (separator delimiter
    &optional (input-stream *standard-input*))
    (flet ((peek-next-char () (peek-char t input-stream t nil t))
    (discard-next-char () (read-char input-stream t nil t)))
    (if (and delimiter (char= (peek-next-char) delimiter))
    (progn
    (discard-next-char)
    nil)
    (let* ((object (read input-stream t nil t))
    (next-char (peek-next-char)))
    (cond
    ((char= next-char separator) (discard-next-char))
    ((and delimiter (char= next-char delimiter)) nil)
    (t (error "Unexpected next char: ~S" next-char)))
    object))))

    (defun read-separator (stream char)
    (declare (ignore stream))
    (error "Separator ~S shouldn't be read alone" char))

    (defun read-delimiter (stream char)
    (declare (ignore stream))
    (error "Delimiter ~S shouldn't be read alone" char))

    (defun read-left-bracket (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (loop
    for object = (read-next-object +comma+ +right-bracket+ stream)
    while object
    collect (transform-primitive object) into objects
    finally (return `(vector ,@objects)))))

    (defun stringify-key (key)
    (etypecase key
    (symbol (string-downcase (string key)))
    (string key)))

    (defun read-left-brace (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (set-macro-character +colon+ 'read-separator)
    (loop
    for key = (read-next-object +colon+ +right-brace+ stream)
    while key
    for value = (read-next-object +comma+ +right-brace+ stream)
    collect `(cons ,(stringify-key key) ,(transform-primitive value)) into pairs
    finally (return `(create-json-hash-table ,@pairs)))))

    (defun enable-json-syntax ()
    (set-macro-character +left-bracket+ 'read-left-bracket)
    (set-macro-character +right-bracket+ 'read-delimiter)
    (set-macro-character +left-brace+ 'read-left-brace)
    (set-macro-character +right-brace+ 'read-delimiter))
    110 changes: 110 additions & 0 deletions test.lisp
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,110 @@
    (cl:in-package #:cl-user)

    (setq *readtable* (copy-readtable *readtable*))

    (eval-when (:compile-toplevel :load-toplevel :execute)
    (json-reader:enable-json-syntax))

    (defun random-number ()
    (random (expt 2 32)))

    (defun random-string ()
    (with-output-to-string (out)
    (loop repeat (random 10)
    do (format out "~A " (random (expt 2 32))))))

    (defmacro test (name &body forms)
    `(handler-case
    (progn
    ,@forms
    (format t "~&~A~%" ',name))
    (error ()
    (format t "~&~A~%" ',name))))

    (test vector-empty
    (let ((x []))
    (assert (vectorp x))
    (assert (zerop (length x)))))

    (test vector-single-element
    (let ((x [1]))
    (assert (vectorp x))
    (assert (= (length x) 1))
    (assert (= (elt x 0) 1))))

    (test vector-true-false
    (let ((x [true, false]))
    (assert (vectorp x))
    (assert (= (length x) 2))
    (assert (eql (elt x 0) t))
    (assert (eql (elt x 1) nil))))

    (test vector-strings
    (let ((x ["foo", "bar", "baz"]))
    (assert (vectorp x))
    (assert (= (length x) 3))
    (assert (every #'string-equal x '("foo" "bar" "baz")))))

    (test vector-lisp-forms
    (let* ((w "blah")
    (x [ "foo", 1, (+ 3 4), w ]))
    (assert (vectorp x))
    (assert (= (length x) 4))
    (assert (every #'equalp x (list "foo" 1 7 w)))))

    (test hash-table-empty
    (let ((x {}))
    (assert (hash-table-p x))
    (assert (zerop (hash-table-count x)))))

    (test hash-table-single-entry
    (let ((x {"foo": 1}))
    (assert (hash-table-p x))
    (assert (= (hash-table-count x) 1))
    (assert (eql (gethash "foo" x) 1))))

    (test hash-table-table-single-null-entry
    (let ((x {"foo": null}))
    (assert (hash-table-p x))
    (assert (= (hash-table-count x) 1))
    (assert (eql (gethash "foo" x) nil))))

    (test hash-table-multiple-entries
    (let ((x {
    "foo": 1,
    "bar": 2,
    "baz": 3
    }))
    (assert (hash-table-p x))
    (assert (= (hash-table-count x) 3))
    (assert (eql (gethash "foo" x) 1))
    (assert (eql (gethash "bar" x) 2))
    (assert (eql (gethash "baz" x) 3))))

    (test hash-table-lisp-forms
    (let* ((w "blah")
    (x {
    "foo": 1,
    "bar": (+ 3 4),
    "baz": w
    }))
    (assert (hash-table-p x))
    (assert (= (hash-table-count x) 3))
    (assert (eql (gethash "foo" x) 1))
    (assert (eql (gethash "bar" x) 7))
    (assert (eql (gethash "baz" x) w))))

    (test hash-table-key-literals
    (let ((x { foo: 1, bar: 2 }))
    (assert (hash-table-p x))
    (assert (= (hash-table-count x) 2))
    (assert (eql (gethash "foo" x) 1))
    (assert (eql (gethash "bar" x) 2))))

    (test vector-includes-hash-table
    (let ((x [ { foo: 1 } ]))
    (assert (vectorp x))
    (assert (= (length x) 1))
    (let ((hash-table (elt x 0)))
    (assert (hash-table-p hash-table))
    (assert (eql (gethash "foo" hash-table) 1)))))
  26. @chaitanyagupta chaitanyagupta revised this gist Mar 3, 2014. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -34,8 +34,8 @@ All the arguments to `READ` are optional. The first is the input stream; `eof-er

    One of the simplest (and earliest) examples of a reader macro in Common Lisp is the single quote character `'`. This reader macro translates an expression of the form `'<some-form>` into `(quote <some-form>)`.

    'foo // translated to (quote foo)
    '(1 2 3) // translated to (quote (1 2 3))
    'foo ;; translated to (quote foo)
    '(1 2 3) ;; translated to (quote (1 2 3))

    Simply put, the single quote character `'` is a little bit of syntactic sugar added on top of Lisp. Reader macros do just that -- allow you to add syntactic sugar to Lisp. What makes them so powerful is that, just like ordinay macros, you write reader macros in Lisp itself. For example, `'` could be defined as (example from the [Common Lisp Hyperspec][set-macro-character]):

  27. @chaitanyagupta chaitanyagupta created this gist Mar 3, 2014.
    382 changes: 382 additions & 0 deletions reader-macros.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,382 @@
    ## Introduction

    Reader macros in Common Lisp are perhaps not as famous as ordinary macros. While macros are a great way to create your own DSL, reader macros allow even greater flexibility by allowing you to create entirely new syntax on top of Lisp.

    Paul Graham explains them very well in [On Lisp][] (Chapter 17, Read-Macros):

    > The three big moments in a Lisp expression's life are read-time, compile-time, and runtime. Functions are in control at runtime. Macros give us a chance to perform transformations on programs at compile-time. ...read-macros... do their work at read-time.
    > Macros and read-macros see your program at different stages. Macros get hold of the program when it has already been parsed into Lisp objects by the reader, and read-macros operate on a program while it is still text. However, by invoking read on this text, a read-macro can, if it chooses, get parsed Lisp objects as well. Thus read-macros are at least as powerful as ordinary macros.
    (Note that read-macros and reader macros mean the same thing)

    The Lisp reader is the thing that parses and translates raw text into an AST. In Lisp, the AST is better known as an s-expression (or sexp) -- its either an [atom][] or a [list][].

    Reader macros allow you to modify the behaviour of the Lisp reader.

    Not only can you modify the behaviour of the Lisp reader, you can also invoke it. The simplest way to invoke the Lisp reader is to call the function `READ`. When you call `READ` with no arguments, it reads text from the standard input and returns a Lisp expression as soon as it fully parses one.

    > (read)
    ;; Provide this input: foo
    FOO

    > (read)
    ;; Input: (foo 1 2 3)
    (FOO 1 2 3)

    The full syntax of `READ` looks like this:

    (read &optional input-stream eof-error-p eof-value recursive-p) => object

    All the arguments to `READ` are optional. The first is the input stream; `eof-error-p` and `eof-value` specify end of file handling; `recursive-p` should be set to true whenever `READ` is called from inside another call to `READ`. When implementing reader macros, `recursive-p` should always be true whenever `READ` is called.

    ## Quote

    One of the simplest (and earliest) examples of a reader macro in Common Lisp is the single quote character `'`. This reader macro translates an expression of the form `'<some-form>` into `(quote <some-form>)`.

    'foo // translated to (quote foo)
    '(1 2 3) // translated to (quote (1 2 3))

    Simply put, the single quote character `'` is a little bit of syntactic sugar added on top of Lisp. Reader macros do just that -- allow you to add syntactic sugar to Lisp. What makes them so powerful is that, just like ordinay macros, you write reader macros in Lisp itself. For example, `'` could be defined as (example from the [Common Lisp Hyperspec][set-macro-character]):

    (defun single-quote-reader (stream char)
    (declare (ignore char))
    (list 'quote (read stream t nil t)))

    (set-macro-character #\' #'single-quote-reader)

    `SET-MACRO-CHARACTER` makes the Lisp reader associate the single quote character `'` with the function `single-quote-reader`. Whenever the Lisp reader encounters `'`, it calls `single-quote-reader`. This function takes two arguments -- the input stream and the character in the stream that triggered the call. Since we only associate `single-quote-reader` with `'`, `char` will always be `'`.

    Let's trace step by step how this works. Assume we call `READ` on the following form:

    > (read)
    ;; Input: 'foo

    When `READ` is invoked, this is what how the input stream looks:

    'foo
    ^

    The current position of the stream is denoted by `^`. When `READ` is called, the next character in the stream is the quote. The reader reads this character, looks up its macro dispatch table, find `single-quote-reader` associated with `'` and calls it. When `single-quote-reader` is invoked, this is how the stream looks:

    'foo
    ^

    `single-quote-reader` calls `READ` again. The inner call to `READ` returns the symbol `foo`, so this function returns the form `(quote foo)` (which is usually printed by the Lisp printer as `'foo`). After `single-quote-reader` returns, the stream position moves forward[1]:

    'foo
    ^

    Remember that reader macro functions are run at read time, so `single-quote-reader` is called only while the Lisp reader is parsing raw text. If the reader encountered `'foo` while parsing a larger expression, the return value of this function will become a part of the larger s-expression.

    ## JSON

    Let's work with a more complicated example. Suppose we wanted Lisp to natively recognize [JSON][] syntax:

    > [1, 2, "foo"]
    #(1 2 "foo")

    > { "foo": 1, "bar": 2 }
    #<HASH-TABLE :TEST EQUAL :COUNT 2 {1004EA5DD3}>

    We define a few constants that we will work with.

    (defconstant +left-bracket+ #\[)
    (defconstant +right-bracket+ #\])
    (defconstant +left-brace+ #\{)
    (defconstant +right-brace+ #\})
    (defconstant +comma+ #\,)
    (defconstant +colon+ #\:)

    To read JSON arrays, we dispatch the left bracket character to `read-left-bracket`.

    (defun read-left-bracket (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    ...
    ))

    (set-macro-character +left-bracket+ 'read-left-bracket)

    Note that I haven't provided a complete definition of `read-left-bracket` yet. The readtable is an object that stores the association between characters and their reader macro functions, if any. `read-left-bracket` rebinds the current readtable to a copy of itself, and causes the comma character `,` to be associated with `read-separator` (which we will see shortly). The reason we copy and rebind the current readtable is to avoid clobbering the readtable entry for comma outside of the dynamic scope of `read-left-bracket`.

    Here's the complete definition for `read-left-bracket`:

    (defun read-next-object (separator delimiter
    &optional (input-stream *standard-input*))
    (flet ((peek-next-char () (peek-char t input-stream t nil t))
    (discard-next-char () (read-char input-stream t nil t)))
    (if (and delimiter (char= (peek-next-char) delimiter))
    (progn
    (discard-next-char)
    nil)
    (let* ((object (read input-stream t nil t))
    (next-char (peek-next-char)))
    (cond
    ((char= next-char separator) (discard-next-char))
    ((and delimiter (char= next-char delimiter)) nil)
    (t (error "Unexpected next char: ~S" next-char)))
    object))))

    (defun read-left-bracket (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (loop
    for object = (read-next-object +comma+ +right-bracket+ stream)
    while object
    collect object into objects
    finally (return `(vector ,@objects)))))

    The bulk of the work of reading JSON objects is done by `read-next-object` -- `read-left-bracket` calls this function repeatedly until it returns `NIL`. The list of objects is then transformed into `(vector obj1 obj2 ...)` which, at runtime, will return a Lisp vector.

    `read-next-object` takes three arguments: a delimiter, which indicates the end of a JSON array or object (i.e. right bracket or right brace); a separator, which separates objects inside a JSON array (comma) or object (comma or colon) and an optional stream argument that defaults to standard input. When `read-next-object` is first called inside an array the stream would be positioned just before the first object.

    [1, 2, "foo"]
    ^

    `read-next-object` parses and returns the first object (the number `1` in this case), and positions the stream before the next object, discarding the separator.

    [1, 2, "foo"]
    ^

    This cycle repeats until `read-next-object` reads the last object in the array, `"foo"`. Since the next non-whitespace character after reading `"foo"` is not the separator but rather the delimiter, `read-next-object` returns `"foo"` but doesn't try to read past the delimiter.

    [1, 2, "foo"]
    ^

    The last call to `read-next-object` notices that the immediate next character is the delimiter. It moves the stream past the delimiter and returns `NIL`, which indicates to `read-left-bracket` that all the objects in the array have been read. Since the stream pointer is now positioned after the delimiter, the reader is now free to read a new object from the input stream.

    [1, 2, "foo"]
    ^

    This is how we define `read-separator`, which was associated with comma as its reader macro function inside the dynamic scope of `read-left-bracket`. This function signals an error whenever it is called -- it is never meant to be called directly by the reader.

    (defun read-separator (stream char)
    (declare (ignore stream))
    (error "Separator ~S shouldn't be read alone" char))

    You might ask, if the separator is never meant to be read directly by the reader, why bother associating it with a macro function? The thing is, its not the macro function that's important -- what we really want to do is tell the Lisp reader that the separator is a terminating macro character. If this were not done, the Lisp reader could unwittingly treat the separator as part of the last object if there was no whitespace between the two. e.g. in `[foo, ...]`, the first object would be read as `foo,` instead of `foo` if comma was not a terminating macro character[3].

    Lastly, we also need to define a macro function for the right bracket. This needs to be done for the same reason we needed to define a macro function for the separator -- so that we can tell the Lisp reader that it is a terminating macro character.

    (defun read-delimiter (stream char)
    (declare (ignore stream))
    (error "Delimiter ~S shouldn't be read alone" char))

    (set-macro-character +right-bracket+ 'read-delimiter)

    `read-delimiter` is very similar to `read-separator` since a delimiter too is not supposed to be read directly on the stream.

    Now we have everything in place to read a JSON array in Lisp. Time for some tests.

    > []
    #()
    > [1, 2, 3]
    #(1 2 3)
    > [ "foo", "bar", "baz" ]
    #("foo" "bar" "baz")
    > [1, 2, "foo"]
    #(1 2 "foo")

    Since we recursively call `READ` to read individual objects, nesting also works.

    > [1, 2, [3, 4]]
    #(1 2 #(3 4))

    We can even use Lisp forms inside a JSON array, and they will work just fine.

    > (let ((x 123) (y "foo")) [x, y])
    #(123 "foo")

    Remember that you can invoke the reader by calling `READ`, so you can always see the generated sexp for yourself.

    > (read)
    ;; Input: [1, 2, "foo"]
    (VECTOR 1 2 "foo")

    > (read)
    ;; Input: (let ((x 123) (y "foo")) [x, y])
    (LET ((X 123) (Y "foo"))
    (VECTOR X Y))

    Reading an object is not that different from reading an array. First we define a convenience function to create dictionaries which will be used by `read-left-brace`, the macro function for the left brace.

    (defun create-json-hash-table (&rest pairs)
    (let ((hash-table (make-hash-table :test #'equal)))
    (loop for (key . value) in pairs
    do (setf (gethash key hash-table) value))
    hash-table))

    And here's `read-left-brace`. Note that this function also relies on `read-next-object` to get the bulk of its work done.

    (defun read-left-brace (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (set-macro-character +colon+ 'read-separator)
    (loop
    for key = (read-next-object +colon+ +right-brace+ stream)
    while key
    for value = (read-next-object +comma+ +right-brace+ stream)
    collect `(cons ,key ,value) into pairs
    finally (return `(create-json-hash-table ,@pairs)))))

    (set-macro-character +left-brace+ 'read-left-brace)

    Finally, we also associate right brace with a macro function.

    (set-macro-character +right-brace+ 'read-delimiter)

    Time for some more tests.

    > { "foo": 1, "bar": 2, "baz": 3 }
    #<HASH-TABLE :TEST EQUAL :COUNT 3 {1004DC3953}>

    We can also nest dictionaries and arrays.

    > [{"foo": 1}, "bar", {"baz": [2, 3]}]
    #(#<HASH-TABLE :TEST EQUAL :COUNT 1 {10062D4D83}> "bar"
    #<HASH-TABLE :TEST EQUAL :COUNT 1 {10062D5283}>)

    The generated sexp:

    > (read)
    { "foo": 1, "bar": 2, "baz": 3 }
    (CREATE-JSON-HASH-TABLE (CONS "foo" 1) (CONS "bar" 2) (CONS "baz" 3))

    ## Refining the JSON reader

    ### true, false, null

    Our JSON reader can handle numbers, strings, objects and arrays fairly well. However, it can still not handle three primitives in JSON -- `true`, `false`, and `null`. Its also fairly easy to fix. When the reader encounters these JSON primitives, it interprets them as Lisp symbols. All we have to do is transform these into an appropriate Lisp value. To this end, we define a function `transform-primitive`.

    (defun transform-primitive (value)
    (if (symbolp value)
    (cond
    ((string-equal (symbol-name value) "true") t)
    ((string-equal (symbol-name value) "false") nil)
    ((string-equal (symbol-name value) "null") nil)
    (t value))
    value))

    Note that `transform-primitive` transforms both `false` and `null` from JSON to `NIL` in Lisp. That's because in Lisp, the two are represented by the same value. You can always choose a different representation if you want.

    We start using `transform-primitive` in our reader macro functions now.

    (defun read-left-bracket (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (loop
    for object = (read-next-object +comma+ +right-bracket+ stream)
    while object
    collect (transform-primitive object) into objects
    finally (return `(vector ,@objects)))))

    (defun read-left-brace (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (set-macro-character +colon+ 'read-separator)
    (loop
    for key = (read-next-object +colon+ +right-brace+ stream)
    while key
    for value = (read-next-object +comma+ +right-brace+ stream)
    collect `(cons ,key ,(transform-primitive value)) into pairs
    finally (return `(create-json-hash-table ,@pairs)))))

    Let's test this:

    > [true, false, null]
    #(T NIL NIL)

    ### Object keys

    While JSON requires that object keys be strict JSON strings (with surrounding double quotes, etc.), Javascript on the other hand doesn't. Object keys in Javascript don't need to be surrounded by double quotes.

    When we try to use this syntax with our reader, we get an error.

    > { foo: 1 }
    ; Evaluation aborted on #<UNBOUND-VARIABLE FOO {1003AC8E03}>.

    A look at the generated sexp makes clear why we get the runtime thinks `foo` is unbound.

    > (read)
    ;; Input: { foo: 1 }
    (CREATE-JSON-HASH-TABLE (CONS FOO 1))

    This is also easy to fix using the `stringify-key` function.

    (defun stringify-key (key)
    (etypecase key
    (symbol (string-downcase (string key)))
    (string key)))

    (defun read-left-brace (stream char)
    (declare (ignore char))
    (let ((*readtable* (copy-readtable)))
    (set-macro-character +comma+ 'read-separator)
    (set-macro-character +colon+ 'read-separator)
    (loop
    for key = (read-next-object +colon+ +right-brace+ stream)
    while key
    for value = (read-next-object +comma+ +right-brace+ stream)
    collect `(cons ,(stringify-key key) ,(transform-primitive value)) into pairs
    finally (return `(create-json-hash-table ,@pairs)))))

    > { foo: 1 }
    #<HASH-TABLE :TEST EQUAL :COUNT 1 {1004248763}>

    > (read)
    ;; Input: { foo: 1 }
    (JSON-READER::CREATE-JSON-HASH-TABLE (CONS "foo" 1))

    ### Package marker

    One problem with `read-left-brace` is that, within its dymanic scope, it overrides the macro function for colon. The [standard readtable][] in Lisp defines colon to be a package marker for symbols. What this means is that, when parsing a JSON object, we will face problems if a symbol contains a colon.

    > { foo: cl:pi }
    ; Evaluation aborted on #<SIMPLE-ERROR "Unexpected next char: ~S" {10043B7083}>.

    Again, this is not very hard to fix. I will leave that as an exercise to the reader.

    ## Conclusion

    Reader macros are a powerful way to expand Lisp's syntax in new and wonderful ways. However with great power comes great responsibility, so you should learn to use them with great care. For example, while a JSON reader was a good exercise in understanding reader macros, I would never use it in real world for the simple reason that easier alternatives already exist. e.g. a plain list or even a vector literal:

    '(1 2 3)
    #(1 2 3)

    Another disadvantage of inventing new syntax is that it doesn't work well with existing development tools. e.g. JSON array and object syntax doesn't work very well with lisp and paredit modes in Emacs -- even simple operations like `C-M-f` and `C-M-b` (to go forward or backward sexp) don't work very well.

    However, when done right, reader macros can work wonders. Take a look at [cl-interpol][][4], which adds support for string interpolation to Common Lisp, so you can easily embed newlines in strings or even create regexes more simply.

    > #?"foo\nbar"
    "foo
    bar"

    > #?/\bfoo\b/
    "\\bfoo\\b"



    1. I have not covered how `READ` handles whitespace following the Lisp expression. When you provide `'foo` as input on the standard input, you need to follow it up with the a newline character. `READ` will read past the newline character in this case; so the stream position after `single-quote-reader` will actually go past the newline

    2.

    3. Actually, comma is already [defined][comma] in the standard readtable as a terminating macro character -- it is a part of the [backquote syntax][]. So technically, we could have done without associating comma with `read-separator`.

    4. Technically, cl-interpol doesn't define a new reader macro, it installs `?` (question mark) as a "sub character" of the standard [dispatching macro character][sharpsign] `#` (sharpsign). Dispatching macro characters are a special class of reader macros which allow you to "make the most of the ASCII character set; one can only have so many one-character read-macros." (On Lisp, Chapter 17, Read-Macros)

    [On Lisp]: http://www.paulgraham.com/onlisp.html
    [atom]: http://www.lispworks.com/documentation/HyperSpec/Body/t_atom.htm
    [list]: http://www.lispworks.com/documentation/HyperSpec/Body/t_list.htm
    [set-macro-character]: http://www.lispworks.com/documentation/HyperSpec/Body/f_set_ma.htm
    [JSON]: http://json.org/
    [Standard Readtable]: http://www.lispworks.com/documentation/HyperSpec/Body/02_aab.htm
    [cl-interpol]: http://weitz.de/cl-interpol/
    [sharpsign]: http://www.lispworks.com/documentation/HyperSpec/Body/02_dh.htm
    [comma]: http://www.lispworks.com/documentation/HyperSpec/Body/02_dg.htm
    [backquote syntax]: http://www.lispworks.com/documentation/HyperSpec/Body/02_df.htm