# CL-JSON

## A JSON parser and generator in Common-Lisp

## Q&A

### What does CL-JSON do?

CL-JSON provides an encoder of Lisp objects to JSON format and a corresponding decoder of JSON data to Lisp objects. Both the encoder and the decoder are highly customizable; at the same time, the default settings ensure a very simple mode of operation, similar to that provided by YASON or ST-JSON. CL-JSON is well tested and the automated testcases includes the JSON_checker tests.

### What is JSON?

JSON is a language independent text format for data-interchange. It has very good open-source and commercial support in many languages. JSON is especially convenient in web applications, since it is a subset of the literal object notation of ECMAScript. A technique called JSONP can be used to do cross-domain Ajax.

### Why not use XML instead?

Some find JSON lighter and more simple, see this comparison.

### Why not use s-expressions instead?

- Many people find parentheses difficult, but brackets and braces easy. That has led to many implementations of JSON. There is no format based on s-expressions implemented in over 20 languages (yet!).

- A simple and very fast JSON parser in JavaScript looks like this:

      eval('(' + aJSONText + ')')

  Even a seasoned lisper may find it difficult to make a shorter JavaScript parser for s-expressions.

### How do I download CL-JSON?

A Darcs repository is available.

    darcs get http://common-lisp.net/project/cl-json/darcs/cl-json

The latest release can be downloaded here.

You can also install it by asdf-install.

History has shown that the darcs version is probably better than the latest release.

### History

- 2006 First version by Henrik Hjelte.
- 2007-2008 Stable version 0.3.1, no known bugs.
- 2009 Major update by Boris Smilga, adding lots of features. Version 0.4.0
- aug 2009 safe-json-intern and simplified-camel-case-to-lisp. Not released yet, in darcs repo.
- dec 2009 New alternative encoder, the explicit sexp-encoder. Not released yet, in darcs repo. Not documented below, but with testcases.

### Status

One known bug (see testmisc.lisp (test json-bind-in-bind-bug)).

A bit faster decoder performance in 0.3 version, a darcs repo is available (old version).

## User Guide

### The default encoder / decoder

The basic functionality is provided by the two functions `encode-json` and `decode-json` (all Lisp names in this Manual are available in the package `JSON`). Under the default semantics they implement the following mapping between Lisp and JSON values:

Lisp |  | JSON (capitalized names refer to productions in RFC 4627)
--- | --- | ---
`integer` | ↔ | `Number`, with no `Frac` or `Exp` parts
`float` | ↔ | `Number`, with `Frac` or `Exp` parts
`rational` | →
the value `t` | ↔ | `true`
the value `nil` | ↔ | `null`
← | `false`
any other `symbol` | → with identifier name translation | `String`
`character` | →
`string` | ↔ | `String` (except `Object` member keys)
`list` (except association lists) | ↔ | `Array`
any other `sequence` | →
association list | ↔ | `Object`
`hash-table` | →
`standard-object` (in implementations with MOP) | →
`symbol` | ← interned in `*json-symbols-package*` with identifier name translation | `String` as an `Object` member key

[generic function] `encode-json object &optional stream`

Write a JSON representation of object to stream and return nil.

[function] `decode-json &optional stream`

Read a JSON Value from stream and return the corresponding Lisp value.

If `stream` is omitted it defaults to `*json-output*` for the encoder, `*json-input*` for the decoder. The initial values of these variables are synonym streams to `*standard-output*` and `*standard-input*` respectively.

[variable] `*json-output*`

The default output stream for encoding operations.

[variable] `*json-input*`

The default input stream for decoding operations.

    (json:encode-json
      '#( ((foo . (1 2 3)) (bar . t) (baz . #\!))
          "quux" 4/17 4.25))

    [{"foo":[1,2,3],"bar":true,"baz":"!"},"quux",0.23529412,4.25]

    NIL

Some parameters which influence the encoding and decoding of symbols are:

[variable] `*json-symbols-package*`

The package where JSON Object keys etc. are interned. Default keyword, nil = use current *package*.

    (with-input-from-string
        (s "{\"foo\": [1, 2, 3], \"bar\": true, \"baz\": \"!\"}")
      (json:decode-json s))

    ((:FOO 1 2 3) (:BAR . T) (:BAZ . "!"))

    (with-input-from-string
        (s "{\"foo\": [1, 2, 3], \"bar\": true, \"baz\": \"!\"}")
      (let ((json:*json-symbols-package* 'json-test))
        (json:decode-json s)))

    ((JSON-TEST::FOO 1 2 3) (JSON-TEST::BAR . T) (JSON-TEST::BAZ . "!"))

    (with-input-from-string
        (s "{\"foo\": [1, 2, 3], \"bar\": true, \"baz\": \"!\"}")
      (let ((json:*json-symbols-package* nil))
        (json:decode-json s)))

    ((FOO 1 2 3) (BAR . T) (BAZ . "!"))

[variable] `*json-identifier-name-to-lisp*`

Designator for a function which maps string (a JSON Object key) to string (name of a Lisp symbol).

[variable] `*lisp-identifier-name-to-json*`

Designator for a function which maps string (name of a Lisp symbol) to string (e. g. JSON Object key).

The last two variables are initially bound to `#'camel-case-to-lisp` and `#'lisp-to-camel-case`, respectively. For a description of these functions, see Camel case translation below.

Note also that the format of Lisp floats decoded from JSON input is controlled by the system variable `*read-default-float-format*` in the same manner as for the Lisp reader. (But cf. Error handling.)

CL-JSON also provides the following shortcut functions:

[function] `encode-json-to-string object`

Return the JSON representation of object as a string.

[function] `encode-json-alist alist &optional stream`

Write the JSON representation (Object) of alist to stream (or to *json-output*). Return nil.

[function] `encode-json-alist-to-string alist`

Return the JSON representation (Object) of alist as a string.

[function] `encode-json-plist plist &optional stream`

Write the JSON representation (Object) of plist to stream (or to *json-output*). Return nil.

[function] `encode-json-plist-to-string plist`

Return the JSON representation (Object) of plist as a string.

[function] `decode-json-strict &optional stream`

Same as decode-json, but allow only Objects or Arrays on the top level, no junk afterwards.

[function] `decode-json-from-string json-string`

Read a JSON Value from json-string and return the corresponding Lisp value.

[function] `decode-json-from-source source &optional decoder`

Decode a JSON Value from source using the value of decoder (default 'decode-json) as decoder function. If the source is a string, the input is from this string; if it is a pathname, the input is from the file that it names; otherwise, a stream is expected as source.

### The CLOS decoder

It is possible to switch CL-JSON into a mode where `Array`s are decoded to Lisp vectors rather than lists, and `Object`s are decoded to CLOS objects in the manner described below. Switching between this mode (“CLOS semantics”) and the default “list semantics” is done using the following functions:

[function] `set-decoder-simple-clos-semantics`

[function] `set-decoder-simple-list-semantics`

Either semantics can be made to hold temporarily in a dynamic environment while `body` is being executed:

[macro] `with-decoder-simple-clos-semantics &body body`

[macro] `with-decoder-simple-list-semantics &body body`

As a general rule, under the CLOS semantics a JSON `Object` datum D is decoded to an instance F of an anonymous class K. Such a class does not have a name which could be used as a method specializer, however, the user can always assume that K is a subclass of `fluid-object` and an instance of the metaclass `fluid-class`; that it has at least the slots named by the keys in D (which are decoded to symbols in the usual manner); and that in F these slots are bound to Lisp images of the corresponding JSON `Value`s, and any other slot is unbound:

    (json:with-decoder-simple-clos-semantics
      (let ((json:*json-symbols-package* nil))
        (let ((x (json:decode-json-from-string
                   "{\"foo\": [1, 2, 3], \"bar\": true,
                     \"baz\": \"!\"}")))
          (with-slots (foo bar baz) x
            (values x foo bar baz)))))

    #<AN INSTANCE OF ANONYMOUS FLUID-CLASS>

    #(1 2 3)

    T

    "!"

Beyond this, CL-JSON provides a means for the remote party to specify the class (or the superclasses) of the decoded CLOS object. This is done by supplying the JSON `Object` with appropriate metadata, called that `Object`'s prototype. The prototype is the `Member` of the `Object` whose key matches `*prototype-name*`—more precisely, whose key decodes to a symbol S such that `(string= (symbol-name S) (symbol-name json:*prototype-name*))`.

[variable] `*prototype-name*`

Must have a symbol for value, the default is `json:prototype`. If the value is `nil` then no member of an `Object` is to be treated as the prototype (not even one whose key otherwise matches `nil`).

Note: The default prototype name, prototype has been chosen so that it would not, in all likelihood, interfere with programs' naming schemes in a JavaScript remote party. As this name has special meaning within JavaScript, it is unlikely that it would be used for a user-level member. Users of CL-JSON are, of course, free to assign another name that better suits their own needs.

The `Value` of the prototype must be either a `String`, or an `Object` whose (undecoded) keys include `"lispPackage"`, `"lispClass"`, and `"lispSuperclasses"`. All `Member`s with other keys are ignored. The `Value` for any of `"lispPackage"`, `"lispClass"`, and `"lispSuperclasses"` may be `null` which is equivalent to omitting that `Member`. A prototype S which is a `String` (rather than an `Object`) is equivalent to a prototype that has only the `Member` `"lispClass":S`. An `Object` altogether lacking a prototype is equivalent to it having a prototype with all the three meaningful members omitted. With the foregoing exceptions, the `Value`s for `"lispPackage"` and `"lispClass"` should be `String`s, and the `Value` for `"lispSuperclasses"` an `Array` of `String`s.

The `Value` for `"lispPackage"` is decoded as symbol and passed to `find-package` to obtain a package, P. The `Value` for `"lispClass"`, `Member`s of the `Value` for `"lispSuperclasses"`, and the keys of the encompassing `Object` are then decoded as symbols in the package P. (If `"lispPackage"` is `null` or missing, they are interned according to the generic rule, i. e. in `*json-symbols-package*`.)

If the prototype has a `"lispClass"` member, its decoded value must name an existing class K. The encompassing `Object`, D, is then decoded to an instance of K. The `Value`s of all `Member`s of D whose keys, when decoded, yield names of slots in K are assigned to those slots. All other `Member`s are discarded. Also, if K has a slot whose name is the same as `*prototype-name*` that slot shall be bound to an instance of `prototype` which describes the prototype in the JSON `Object`.

    (in-package cl-user)

    #<PACKAGE "COMMON-LISP-USER">

    (defclass sample () (foo bar))

    #<STANDARD-CLASS SAMPLE>

    (json:with-decoder-simple-clos-semantics
      (let ((x (json:decode-json-from-string
                "{\"foo\": [1, 2, 3], \"bar\": true, \"baz\": \"!\",
                  \"prototype\": {\"lispPackage\": \"clUser\",
                                  \"lispClass\": \"sample\"}}")))
          (with-slots (foo bar) x
            (values x (class-of x) foo bar (slot-exists-p x 'baz)))))

    #<A SAMPLE>

    #<STANDARD-CLASS SAMPLE>

    #(1 2 3)

    T

    NIL

By way of exception, if the decoded value for `"lispClass"` is `common-lisp:hash-table`, `common-lisp:cons`, or `common-lisp:list` then the encompassing `Object` shall be decoded, respectively, to a hash table, an association list, or a property list with the keys and values decoded from the keys and values of the encompassing `Object`. (The prototype member itself shall be discarded.)

    (let ((json:*json-symbols-package* 'cl-user))
     (json:with-decoder-simple-clos-semantics
       (let ((x (json:decode-json-from-string
                 "{\"foo\": [1, 2, 3], \"bar\": true, \"baz\": \"!\",
                   \"prototype\": \"hashTable\"}")))
         (loop for v being each hash-value of x
           collect v))))

    ("!" #(1 2 3) T) ; or in another order

    (json:with-decoder-simple-clos-semantics
      (json:decode-json-from-string
        "{\"foo\": [1, 2, 3], \"bar\": true, \"baz\": \"!\",
          \"prototype\": {\"lispPackage\": \"jsonTest\",
                          \"lispClass\": \"list\"}}"))

    (JSON-TEST::FOO #(1 2 3) JSON-TEST::BAR T JSON-TEST::BAZ "!")

If the prototype has no `"lispClass"` but has `"lispSuperclasses"`, the `Value` of the latter is decoded as a list of symbols which must name existing classes L1, ... Ln, n ≥ 0. The encompassing `Object`, D, is then decoded to a fluid object F whose class K has all the classes L1, ... Ln as its direct superclasses. Unless any of the classes L1, ... Ln is a subclass of `fluid-object`, K shall be additionally a direct subclass of `fluid-object`. Slots corresponding to all `Member`s of D (except the prototype itself) shall be bound in F.

    (defclass sample-2 ()
      ((frob :initform 'xyzzy)))

    #<STANDARD-CLASS SAMPLE-2>

    (json:with-decoder-simple-clos-semantics
      (let ((json:*json-symbols-package* nil))
        (let ((x (json:decode-json-from-string
                   "{\"foo\": [1, 2, 3], \"bar\": true, \"baz\": \"!\",
                     \"prototype\":
                        {\"lispSuperclasses\": [\"sample2\"]}}")))
          (with-slots (foo frob) x
            (values x foo frob)))))

    #<AN INSTANCE OF ANONYMOUS FLUID-CLASS>

    #(1 2 3)

    XYZZY

If the prototype has both `"lispClass"` and `"lispSuperclasses"`, the latter member is disregarded.

[standard class] `prototype` (precedence list: `prototype`, `standard-object`, `t`)

[generic function] `make-object-prototype object &optional slot-names`

Return a prototype describing the object's class or superclasses, and the package into which the names of the class / superclasses and of the object's slots are to be interned.

#### Properties of fluid objects

Fluid objects are quite like standard objects in most respects. However, any fluid object has `fluid-class` as its metaclass, and missing slots in such fluid instances are not distinguished from unbound slots. Namely, if F is a fluid instance of class K, and R is a slot name such that `(slot-exists-p F R)` gives `nil` then:

- `(slot-boundp F R)` gives `nil` as well (instead of raising an error);
- `(slot-value F R)` signals an error of type `unbound-slot`;
- `(slot-makunbound F R)` does nothing and returns F;
- for any Lisp value X, `(setf (slot-value F R) X)` adds the slot R to the class K and then sets its value in F to X.

This makes fluid objects somewhat like objects in prototype-based languages, in that the user may add new slots to an object without concerning himself with the object's class membership. Yet fluid objects are little more that “poor man's JavaScript” as there are no real prototype mechanisms. Moreover, Lisp implementations usually employ optimizations geared toward more traditional CLOS techniques which can actually cause less than ideal performance in programs with fluid objects. Thus, fluid objects should generally be considered an experimental feature and used with proper care.

[standard class] `fluid-object` (precedence list: `fluid-object`, `standard-object`, `t`)

[standard class] `fluid-class` (precedence list: `fluid-class`, `standard-class`, `class`, `standard-object`, `t`)

The CLOS decoder maintains a separate fluid class for each combination of superclasses in prototypes sent in by remote parties. I. e., if the prototypes of two incoming JSON `Objects` D1 and D2 have `"lispSuperclasses"` members whose values decode to non-`equal` lists of symbols then the fluid objects F1 and F2 decoded from D1 and D2 shall belong to different fluid classes. This behaviour is implemented by the generic function `make-object`.

[generic function] `make-object bindings class &optional superclasses`

If class is not nil, create an instance of that class. Otherwise, create a fluid object whose class has the given superclasses (null list by default). In either case, populate the resulting object using bindings (an alist of slot names and values).

To maintain the mapping between lists of superclass names and fluid classes, the decoder maintains a class registry. Thus, using fluid objects makes the CLOS decoder essentially thread-unsafe. (If every incoming JSON `Object` is guaranteed to have a prototype with a `"lispClass"` member then there are no fluid objects and thread safety is ensured.) If the user wishes to employ fluid objects in a threaded environment it is advisable to wrap the body of entry-point functions in `with-local-class-registry`.

[variable] `*class-registry*`

A list of anonymous fluid classes, one member for every distinct combination of direct superclasses.

[function] `clear-class-registry`

Reset the *class-registry* to nil.

[macro] `with-local-class-registry (&key inherit) &body body`

Run body in a dynamic environment where *class-registry* is a temporary local list. If :inherit is non-null, the local registry shall initially have the same content as the exterior *class-registry*, otherwise it shall be nil.

#### Security considerations

`Make-object` also allows to implement security restrictions in an environment where uncontrolled class instantiation may have undesirable side effects. Ordinarily, nothing prevents the CLOS decoder from processing an incoming datum like

    {"prototype": {"lispPackage": "worldDestruction",
                   "lispClass": "tickingBomb"},
     "timeout": 10,
     "setOffImmediatelyOnInstantiation": true}

If a careless and/or malevolent remote party manages to sneak in such a request the calamitous results will not be late in coming. However, as the decoding of every input `Object` eventually involves a call to `make-object`, a safeguard may take the form simply of a method of that generic function:

    (define-condition json-security-violation (error)
      ((bindings :reader json-security-violation-bindings :initarg :bindings)
       (class :reader json-security-violation-class :initarg :class))
      (:report
       (lambda (sv stream)
         (with-slots (bindings class) sv
           (format stream "Incoming JSON Object ~S in the class ~S ~
                           violates security restrictions."
                   (json:encode-json-to-string bindings)
                   class)))))

    JSON-SECURITY-VIOLATION

    (defmethod json:make-object :before
        (bindings (class (eql world-destruction:ticking-bomb))
         &optional superclasses)
      (error 'json-security-violation :bindings bindings :class class))

    #<STANDARD-METHOD JSON:MAKE-OBJECT …
            :BEFORE (T (EQL WORLD-DESTRUCTION:TICKING-BOMB))>

    (json:with-decoder-simple-clos-semantics
      (json:decode-json-from-string
        "{\"prototype\": {\"lispPackage\": \"worldDestruction\",
                          \"lispClass\": \"tickingBomb\"},
          \"timeout\": 10,
          \"setOffImmediatelyOnInstantiation\": true}"))

    Error in JSON:MAKE-OBJECT: Incoming JSON Object …
            "{\"timeout\":10,\"setOffImmediatelyOnInstantiation\":true}" …
            in the class WORLD-DESTRUCTION:TICKING-BOMB …
            violates security restrictions.

By using multiple methods with carefully adjusted specializers and qualifiers the user shall be able to create arbitrarily sophisticated security policies.

`JSON-INTERN` is the default function for translating json-keys to values. The default json-intern is not safe. Interns of many unique symbols could potentially use a lot of memory. An attack could exploit this by submitting something that is passed through cl-json that has many very large, unique symbols. You may want to use the function `SAFE-JSON-INTERN` or some other mechanism for security reasons.

### Customizing the decoder

More arbitrary changes to decoder semantics can be made via the customization API. It is an event-driven API not unlike SAX: certain variables can be set (or dynamically bound) to handler functions which are invoked, in sequence, as the parser encounters various syntactic elements of JSON on the input stream. JSON well-formedness is checked by the underlying parser, while the handlers are free to dispose of the (known clean) data in any way the user may find fit.

Handlers fall into three overall categories: constructors, which can be further subdivided into atomic constructors and aggregate constructors; aggregate initializers; and member handlers. An atomic constructor is invoked when the parser encounters an atomic datum (`Number` or `Boolean`), is passed the token which constitutes the datum, and is supposed to return the Lisp value that corresponds to the datum. An aggregate initializer is invoked when the parser encounters the beginning of an aggregate datum (`String`, `Array` or `Object`). Similarly, an aggregate constructor is invoked at the end of a well-formed aggregate datum, and is supposed to return the Lisp value that corresponds to the whole aggregate. A member handler is invoked when the parser reads past a member of an `Array`, or a key or value of an `Object`; it is passed the value which the constructor for that member datum had freshly returned. Handlers for `Character`s of `String`s are somewhat specific in that they are passed characters rather than decoded values but otherwise they fall neatly with member handlers. The value of the last constructor to return is also the return value of `decode-json`.

As a trivial example, if every handler has the value of `(constantly t)`, the decoder semantics is that of a simple validator: it returns `t` if JSON input is well-formed, and signals error otherwise (see also under Error handling).

Macros which set or bind handlers (and other customization parameters) take their arguments in the form of property lists where each customizaion parameter is indicated by its distinctive keyword. E. g., `:integer` and `:boolean` are the respective indicator keywords for the variables `*integer-handler*` and `*boolean-handler*`. Accordingly, the macro form `(json:bind-custom-vars (:boolean X :integer Y) Z)` expands to `(let ((json:*boolean-handler* X) (json:*integer-handler* Y)) Z)`. Descriptions of customization parameters below always specify the corresponding keyword, and descriptions of macro APIs use the parameter name `customizations` whereever a property list is expected.

[macro] `with-shadowed-custom-vars &body body`

Execute `body` in a dynamic environment where the names of all existing customization parameters are re-bound to fresh locations. The initial values of the parameters are taken from the exterior environment at runtime.

[macro] `bind-custom-vars (&rest customizations) &body body`

Execute `body` in a dynamic environment where customization parameters are bound as specified by `customizations`.

[macro] `set-custom-vars &rest customizations`

Globally set customization parameters according to `customizations`. If used within a dynamic environment established by `with-shadowed-custom-vars` or `bind-custom-vars`, the extent of the assignment is limited by that form.

The following parameters identify the constructor handlers for atomic data:

[custom variable] `*boolean-handler*` (indicator: `:boolean`)

Designator for a function of 1 string argument (boolean token).

[custom variable] `*integer-handler*` (indicator: `:integer`)

Designator for a function of 1 string argument (integer token).

[custom variable] `*real-handler*` (indicator: `:real`)

Designator for a function of 1 string argument (real token).

For `String`s, three handlers (an initializer, a member handler, and a constructor) are used:

[custom variable] `*beginning-of-string-handler*` (indicator: `:beginning-of-string`)

Designator for a function of no arguments (called at encountering an opening quote for a String).

[custom variable] `*string-char-handler*` (indicator: `:string-char`)

Designator for a function of 1 character argument (String char).

[custom variable] `*end-of-string-handler*` (indicator: `:end-of-string`)

Designator for a function of no arguments (called at encountering a closing quote for a String).

Note that escape sequences in `String`s are resolved by the underlying decoder, and a `*string-char-handler*` normally receives nothing but characters. However, limitations of the Lisp implementation running CL-JSON can prevent it from resolving a perfectly well-formed escape sequence (e. g., in CMUCL which does not support Unicode). Situations of this kind are discussed below in the section Error handling.

For `Array`s, three handlers are used as well:

[custom variable] `*beginning-of-array-handler*` (indicator: `:beginning-of-array`)

Designator for a function of no arguments (called at encountering an opening bracket for an Array).

[custom variable] `*array-member-handler*` (indicator: `:array-member`)

Designator for a function of 1 arbitrary argument (decoded member of Array).

[custom variable] `*end-of-array-handler*` (indicator: `:end-of-array`)

Designator for a function of no arguments (called at encountering a closing bracket for an Array).

For `Object`s, it is the same thing except that there are two member handlers, one for keys and another one for values, invoked alternately:

[custom variable] `*beginning-of-object-handler*` (indicator: `:beginning-of-object`)

Designator for a function of no arguments (called at encountering an opening brace for an Object).

[custom variable] `*object-key-handler*` (indicator: `:object-key`)

Designator for a function of 1 string argument (decoded member key of Object).

[custom variable] `*object-value-handler*` (indicator: `:object-value`)

Designator for a function of 1 arbitrary argument (decoded member value of Object).

[custom variable] `*end-of-object-handler*` (indicator: `:end-of-object`)

Designator for a function of no arguments (called at encountering a closing brace for an Object).

If the user wished to map JSON structure to Lisp structure, with only these 13 handlers he would have to maintain cumbersome stacking infrastructure, pushing at the beginning and popping at the end of every aggregate datum. CL-JSON has a means to circumvent this hindrance: the user is given an option to indicate any number of dynamic variables V1, ... Vn as having aggregate scope. This has the effect that the ensemble of handlers invoked for any particular aggregate datum on the input are called in a dynamic environment where the names V1, ... Vn are bound to fresh locations. Hence, an aggregate-scope variable can be initialized by an initializer handler, updated by member handlers, and finally used to calculate the return value of the constructor. If several aggregate data are nested, the handlers for interior data work on their own copy of the variable without fear of clobbering the value of the same for exterior data. Handler code can be thus written in terms of individual levels rather than complex nested structures.

[custom variable] `*aggregate-scope-variables*` (indicator: `:aggregate-scope`)

A list of symbols naming dynamic variables which should be re-bound in the scope of every JSON aggregate value (Object, Array or String).

Aggregate-scope variables on inner levels initially inherit their values from the immediately outer level, so it is possible to pass information down from level to level. In the following example, the variable `count` is reset to 0 at the beginning of every aggregate, while `level` is incremented from outer to inner levels.

    (defparameter level -1
      "Level of nesting in JSON structure.")

    LEVEL

    (defparameter count -1
      "Count of members in JSON structure.")

    COUNT

    (defun format-with-indent (control &rest args)
      (format t "~&~0,1,V,V@A~?" level #\Tab "" control args))

    FORMAT-WITH-INDENT

    (defun report-atom (token)
      (format-with-indent "~Ctoken: ~S~%" #\Tab token)
      'atom)

    REPORT-ATOM

    (defun report-push ()
      (incf level)
      (setq count 0)
      (format-with-indent "beginning of aggregate~%"))

    REPORT-PUSH

    (defun report-member (member)
      (format-with-indent "~:R member: ~A~%"
                          (incf count) member))

    REPORT-MEMBER

    (defun report-pop ()
      (format-with-indent "end of aggregate~%")
      (list 'aggregate count))

    REPORT-POP

    (json:bind-custom-vars
        (:integer              #'report-atom
         :real                 #'report-atom
         :boolean              #'report-atom
         :beginning-of-array   #'report-push
         :array-member         #'report-member
         :end-of-array         #'report-pop
         :beginning-of-string  #'report-push
         :string-char          #'report-atom
         :end-of-string        #'report-pop
         :beginning-of-object  #'report-push
         :object-key           (constantly t)
         :object-value         #'report-member
         :end-of-object        #'report-pop
         :aggregate-scope      (list 'level 'count))
      (json:decode-json-from-string
        "{\"foo\": [10, 20, 50], \"bar\": true}"))

    beginning of aggregate

    	beginning of aggregate

    		token: #\f

    		token: #\o

    		token: #\o

    	end of aggregate

    	beginning of aggregate

    		token: "10"

    	first member: ATOM

    		token: "20"

    	second member: ATOM

    		token: "50"

    	third member: ATOM

    	end of aggregate

    first member: (AGGREGATE 3)

    	beginning of aggregate

    		token: #\b

    		token: #\a

    		token: #\r

    	end of aggregate

    	token: "true"

    second member: ATOM

    end of aggregate

    (AGGREGATE 2)

Further, it is possible to re-bind variables while parsing some kinds of aggregate data but not the others:

[custom variable] `*array-scope-variables*` (indicator: `:array-scope`)

A list of symbols naming dynamic variables which should be re-bound in the scope of every JSON Array.

[custom variable] `*object-scope-variables*` (indicator: `:object-scope`)

A list of symbols naming dynamic variables which should be re-bound in the scope of every JSON Object.

[custom variable] `*string-scope-variables*` (indicator: `:string-scope`)

A list of symbols naming dynamic variables which should be re-bound in the scope of every JSON String.

When using one of the standard semantics, the following parameter allows to determine the type of sequence to which `Array`s are decoded. The default value is `list` for the list semantics, and `vector` for the CLOS semantics.

[custom variable] `*json-array-type*` (indicator: `:array-type`)

The Lisp sequence type to which JSON Arrays are to be coerced.

Finally, the user can change the decoder function for member data of the current aggregate. The name `*internal-decoder*` is included in the default list of `*aggregate-scope-variables*` under both predefined semantics, and so this variable can be assigned locally without affecting the decoding of the outer levels of data. If it is set outside of any handler, the value is used to decode all members of top-level aggregates.

[custom variable] `*internal-decoder*` (indicator: `:internal-decoder`)

Designator for a function of 1 stream argument called (instead of decode-json) to decode a member of an Array or of an Object.

The following macros can be used to construct decoder functions acceptable as values of `*internal-decoder*`:

[macro] `custom-decoder &rest customizations`

Construct a decoder function whose semantics is determined by the given `customizations`.

[macro] `current-decoder &rest keys`

Capture the current (execution-time) decoder semantics and construct a decoder function which restores the captured semantics in the extent of its call. The `keys` must be valid indicators of custom variables. If some `keys` are supplied then the values of only the corresponding variables are captured; otherwise the values of all 19 custom variables are captured.

The following example demonstrates how a simplified version of the macro `json-bind` can be written with the help of `*internal-decoder*`.

    (defmacro simple-json-bind ((&rest vars) stream &body body)
      (let ((cur-dec (gensym))
            (key-handler
             `(lambda (json-string)
                (let ((lisp-string
                       (funcall json:*json-identifier-name-to-lisp*
                                json-string)))
                  ;; On recognizing a variable name, the key handler sets
                  ;; the value handler to be a function which,
                  ;; in its turn, assigns the decoded value to the
                  ;; corresponding variable.  If no variable name matches
                  ;; the key, the value handler is set to skip the value.
                  (json:set-custom-vars
                      :object-value
                        (cond
                          ,@(loop for var in vars
                              collect
                               `((string= lisp-string ,(symbol-name var))
                                 (lambda (value)
                                   (setq ,var value))))
                          (t (constantly t))))))))
        `(let ((,cur-dec (json:current-decoder))
               ,@vars)
           (json:bind-custom-vars
               (:internal-decoder ,cur-dec
                :beginning-of-object (constantly t)
                :object-key ,key-handler
                :end-of-object (constantly t))
             (json:decode-json ,stream))
           ,@body)))

    SIMPLE-JSON-BIND

    (pprint
     (macroexpand-1
      '(simple-json-bind (foo bar) s
         (format t "Foo is ~A.~%Bar is ~A.~%" foo bar))))

    (LET ((#:G762 (JSON:CURRENT-DECODER))

          FOO BAR)

      (JSON:BIND-CUSTOM-VARS

          (:INTERNAL-DECODER #:G762

           :BEGINNING-OF-OBJECT (CONSTANTLY T)

           :OBJECT-KEY

             (LAMBDA (JSON-STRING)

               (LET ((LISP-STRING

                      (FUNCALL JSON:*JSON-IDENTIFIER-NAME-TO-LISP*

                               JSON-STRING)))

                 (JSON:SET-CUSTOM-VARS

                     :OBJECT-VALUE

                       (COND ((STRING= LISP-STRING "FOO")

                              (LAMBDA (VALUE) (SETQ FOO VALUE)))

                             ((STRING= LISP-STRING "BAR")

                              (LAMBDA (VALUE) (SETQ BAR VALUE)))

                             (T (CONSTANTLY T))))))

           :END-OF-OBJECT (CONSTANTLY T))

        (JSON:DECODE-JSON S))

      (FORMAT T "Foo is ~A.~%Bar is ~A.~%" FOO BAR))

    (json:with-decoder-simple-list-semantics
      (with-input-from-string
          (s "{\"foo\": [1, 2, 3], \"baz\": {\"foo\": \"!\"},
               \"bar\": true}")
        (simple-json-bind (foo bar) s
          (format t "Foo is ~A.~%Bar is ~A.~%" foo bar))))

    Foo is (1 2 3).

    Bar is T.

    NIL

### Customizing the encoder

Encoding being much more user-driven than decoding, the customization API for the encoder boils down to just a few macros and functions, geared to generating aggregate data. They implement a sequential producer which emits JSON data upon user request, rather than in the course of traversing Lisp data structures. As with `encode-json`, the optional `stream` argument in these macros and functions defaults to `*json-output*`.

[macro] `with-array (&optional stream) &body body`

Open a JSON Array, run body, then close the Array. Inside the body, as-array-member or encode-array-member should be called to encode Members of the Array.

[macro] `as-array-member (&optional stream) &body body`

Body should be a program which encodes exactly one JSON datum to stream. As-array-member ensures that the datum is properly formatted as a Member of an Array, i. e. separated by comma from any preceding or following Member.

[function] `encode-array-member object &optional stream`

Encode object as the next Member of the innermost JSON Array opened with with-array in the dynamic context. Object is encoded using the encode-json generic function, so it must be of a type for which an encode-json method is defined.

[function] `stream-array-member-encoder stream &optional encoder`

Return a function which takes an argument and encodes it to stream as a Member of an Array. The encoding function is taken from the value of encoder (default is #'encode-json).

[macro] `with-object (&optional stream) &body body`

Open a JSON Object, run body, then close the Object. Inside the body, as-object-member or encode-object-member should be called to encode Members of the Object.

[macro] `as-object-member (key &optional stream) &body body`

Body should be a program which writes exactly one JSON datum to stream. As-object-member ensures that the datum is properly formatted as a Member of an Object, i. e. preceded by the (encoded) key and colon, and separated by comma from any preceding or following Member.

[function] `encode-object-member key value &optional stream`

Encode key and value as a Member pair of the innermost JSON Object opened with with-object in the dynamic context. Key and value are encoded using the encode-json generic function, so they both must be of a type for which an encode-json method is defined. If key does not encode to a String, its JSON representation (as a string) is encoded over again.

[function] `stream-object-member-encoder stream &optional encoder`

Return a function which takes two arguments and encodes them to stream as a Member of an Object (String : Value pair).

    (defpackage foo (:use) (:export #:foo #:bar #:baz))

    #<PACKAGE "FOO">

    (defvar foo:foo '(1 2 3))

    FOO:FOO

    (defvar foo:bar "!")

    FOO:BAR

    (defun foo:bar ()
      foo:bar)

    FOO:BAR

    (json:with-array ()
      (json:as-array-member ()
        (json:with-object ()
          (do-external-symbols (sym (find-package "FOO"))
            (json:as-object-member (sym)
              (json:with-object ()
                (if (boundp sym)
                    (let ((value (symbol-value sym)))
                      (json:encode-object-member 'val value)))
                (if (fboundp sym)
                    (json:encode-object-member 'fun t))))))))

    [{"bar":{"val":"!","fun":true},"baz":{},"foo":{"val":[1,2,3]}}]

    NIL

### Camel case translation

The aim of the functions described in this section is to provide a reasonable mapping between identifier naming conventions—from JavaScript to Lisp and back again. Names are regarded as consisting of pieces that, in Lisp, are always uppercase and joined with hyphens, and in JavaScript (and other C-like languages), either joined with underscores or capitalized and joined solid. Plus signs are used in Lisp to mark constants, which corresponds to the all-caps convention in C-like languages.

[function] `simplified-camel-case-to-lisp string`

Inserts - between lowercase and uppercase chars. Ignores _ + * and several consecutive uppercase.

[function] `camel-case-to-lisp string`

[function] `lisp-to-camel-case string`

The functions are not strictly inverse. While `(lisp-to-camel-case (camel-case-to-lisp S))` is `string=` to S for every S which is a valid JavaScript identifier name, `(camel-case-to-lisp (lisp-to-camel-case R))` is not guaranteed to preserve R even if R is a valid Lisp identifier name. The use of `camel-case-to-lisp` and `lisp-to-camel-case` is best explained by some examples:

    (json:lisp-to-camel-case "HELLO-KEY")

    "helloKey"

    (json:camel-case-to-lisp "HiStartsWithUpperCase")

    "*HI-STARTS-WITH-UPPER-CASE"

    (json:camel-case-to-lisp "joined_by_underscore")

    "JOINED--BY--UNDERSCORE"

    (json:camel-case-to-lisp "JSONAllCapitals")

    "+JSON+-ALL-CAPITALS"

    (json:lisp-to-camel-case "+TWO-WORDS+")

    "TWO_WORDS"

    (json:camel-case-to-lisp "camelCase_Mixed_4_PARTS")

    "CAMEL-CASE--*MIXED--+4-PARTS+"

### Error handling

The standard encoder does not cover the complete set of Lisp data types: structures, functions, complex numbers, and cons pairs which are not lists and not members of a valid alist are only some of the unencodable data types. Attempting to encode any such data results in an error.

[condition] `unencodable-value-error` (precedence list: `unencodable-value-error`, `type-error`, `error`, `serious-condition`, `condition`, `t`)

Signalled when a datum is passed to `encode-json` (or another encoder function) which actually cannot be encoded. As with any `type-error`, the accessors `type-error-datum` and `type-error-expected-type` can be used on a condition of this class to get the offending datum and the expected type (however, the latter is always `t`).

Whenever an `unencodable-value-error` is signalled, CL-JSON provides a restart called `substitute-printed-representation` which, if invoked, prints a representation of the offending datum to a string (with `*print-escape*` bound to `nil`) and encodes that string instead of the datum.

    (defstruct frob xyzzy)

    FROB

    (handler-bind
        ((json:unencodable-value-error
          (lambda (err)
            (declare (ignore err))
            (invoke-restart 'json:substitute-printed-representation))))
      (json:encode-json
        '(#S(frob) ((foo . #C(1 1)) (bar . (baz . quux))))))

    ["#S(FROB :XYZZY NIL)",{"foo":"#C(1 1)","bar":"(BAZ . QUUX)"}]

    NIL

The encoder also signals an `error` if any macro or function for encoding `Member`s of aggregates (such as `as-object-member`) is called outside of dynamic environment established by `with-array` or `with-object`, or in inappropriate environment.

More numerous are situations in which an error is caused by input to the decoder. The most straightforward case is when data sent in by the remote party is not well-formed JSON. This results in a `json-syntax-error` being signalled. (Some syntax errors, e. g. missing right brackets, result in `end-of-file` instead.)

[condition] `json-syntax-error` (precedence list: `json-syntax-error`, `simple-error`, `simple-condition`, `stream-error`, `error`, `serious-condition`, `condition`, `t`)

However, even well-formed JSON is not guaranteed to decode. Two common kinds of error situations result from implementation restrictions on character codes and on floating-point reals. If the decoder encounters in a JSON string an escape sequence `\uXXXX` where XXXX is the hexadecimal presentation of a number N which either exceeds the implementation's `char-code-limit` or for which `code-char` returns `nil`, then the following condition is signalled:

[condition] `no-char-for-code` (precedence list: `no-char-for-code`, `error`, `serious-condition`, `condition`, `t`)

In this situation a handler for `no-char-for-code` may invoke the following restarts:

  `substitute-char char`
Continue the execution as if the given char, rather than the escaped char, had been read.
  `pass-code`
Continue the execution using the code itself in place of the character. The code is subsequently passed to the `*string-char-handler*`, so using this restart only has sense if the handler groks character codes (the standard handler doesn't).

Likewise, the underlying Lisp system can fail to handle floats which are not representable in the current `*read-default-float-format*` (e. g. `2.065e444` is non-representable even in the `long-float` format in most contemporary Lisps). If such a datum is encountered in JSON input, an `arithmetic-error` is signalled. CL-JSON provides three restarts to correct such cases:

  `bignumber-string &optional prefix`
Instead of number, use a string which is the concatenation of prefix and the number token that caused the error. The default value for prefix is `"BIGNUMBER:"`.
  `rational-approximation`
Parse the number token into parts, read them as integers, and combine the integers using only rational operations. Use the resulting rational number in place of float. (E. g., `2.065e444` would be converted to the value of the expression `(* (+ 2 (* 65 (expt 10 -3))) (expt 10 444))`.)
  `placeholder value`
Use `value` in place of the float. E. g., one might wish to use `:infinity` if the error in question is a `float-overflow`.

The CLOS decoder can run into errors due to invalid data in a prototype:

- If a prototype is neither an `Object` nor a `String`, or if a `Member` of a prototype `Object` does not have the correct type of `Value` (`null` or `String` for `"lispPackage"` and `"lispClass"`, `null` or `Array` of `String`s for `"lispSuperclasses"`), a `type-error` is signalled.
- If a prototype specifies a non-`null` package name which does not name a registered package, a `package-error` is signalled.
- If a prototype specifies a non-`null` class name or a `Member` of the superclasses `Array` which does not name an existing class, a `cell-error` is signalled.

These situations, as well as `json-syntax-error` are considered fatal, and so no restarts are provided by CL-JSON.

### Utilities

#### JSON-Bind

[macro] `json-bind (&rest vars) json-source &body body`

The stream, string, or file `json-source` should give off a JSON object hierarchy for which `vars` are structured keys. The `Value`s corresponding to these keys are decoded using the current decoder, `vars` are bound to the decoded values, and `body` is executed in the resulting lexical environment.

By “object hierarchy”, we understand a tree of nested JSON `Object`s (without intervening `Array`s). Structured keys are symbols whose names consist of parts separated by periods and serve as paths into this tree. In an `Object` D, one-part name S (without any periods) refers to the `Value` E whose corresponding key R is such that `(string= (funcall *json-identifier-to-lisp* R) S)`. If E is also an `Object`, the path `S.T` refers to the same sub-datum within D which T refers to within E. If any part of the structured name fails to match a member key at its level of object nesting, or if any non-final part of the structured name refers to a non-`Object`, then that variable shall be bound to `nil` in the scope of `json-bind`.

    (let ((dentition
           "{\"DugongDugon\":
              {\"upper\":
                {\"incisors\":2, \"canines\":0,
                 \"premolars\":3, \"molars\":3},
               \"lower\":
                {\"incisors\":3, \"canines\":1,
                 \"premolars\":3, \"molars\":3}},
             \"Notoryctes\":
              {\"upper\":
                {\"incisors\":4, \"canines\":1,
                 \"premolars\":2, \"molars\":4},
               \"lower\":
                {\"incisors\":3, \"canines\":1,
                 \"premolars\":3, \"molars\":4}},
             \"DasypusHybridus\":
              {\"upper\":6, \"lower\":8}}"))
      (json:with-decoder-simple-list-semantics
        (json:json-bind (*notoryctes.lower.incisors
                         *dugong-dugon.lower.incisors
                         *notoryctes.lower
                         *dasypus-hybridus)
            dentition
          (flet ((count-teeth (jaw)
                  (* (loop for (k . v) in jaw summing v) 2)))
            (format t "Armadillos have ~D teeth.~%"
                    (count-teeth *dasypus-hybridus))
            (format t "Marsupial moles have ~D teeth in the lower jaw.~%"
                    (count-teeth *notoryctes.lower)))
          (let ((delta
                  (- *notoryctes.lower.incisors
                     *dugong-dugon.lower.incisors)))
            (format t "Marsupial moles have ~[less~;as many~;more~] ~
                       lower incisors ~:[than~;as~] dugongs.~%"
                    (1+ (signum delta)) (zerop delta))))))

    Armadillos have 28 teeth.

    Marsupial moles have 22 teeth in the lower jaw.

    Marsupial moles have as many lower incisors as dugongs.

    NIL

#### JSON-RPC

JSON-RPC is a lightweight remote procedure call protocol similar to XML-RPC. CL-JSON contains some code that implements the version 1.1 Working draft of the JSON-RPC specification (available as Working Draft 7 August 2006). Just plug it into your http server of choice. See the file json-rpc and the package `JSON-RPC`.

#### JSONP

JSONP or JSON with Padding is a technique that allows cross-domain function calls. A remote function call looks like this: your javascript code insert a "script" html tag to the page you are visiting. The url of this script tag is your remote server. You also send the name of a callback function to the server as a query string parameter. The browser expects the script tag to retrieve some javascript code from your server. The JSON reponse from your can be turned into a proper javascript function call statement by prepending the name of the callback function first and then wrapping the JSON chunk inside a pair of parentheses. CL-JSON doesn't have any special code to do JSONP, it is more an issue of simple coding with your web server.

## Mailing Lists

- cl-json-devel
  for developers and users.
- cl-json-announce
  for announcements.

## License

Copyright © 2006-2008 Henrik Hjelte.
Copyright © 2008 Hans Hübner.

This work is licensed under the terms of the MIT / X Consortium license, reproduced below. Mr. Hübner's copyright applies to the code of the streaming API borrowed from his program YASON and transferred under this license with his permission.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.