JSONPath: Query expressions for JSON

1.1. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

The grammatical rules in this document are to be interpreted as ABNF, as described in [RFC5234]. ABNF terminal values in this document define Unicode code points rather than their UTF-8 encoding. For example, the Unicode PLACE OF INTEREST SIGN (U+2318) would be defined in ABNF as %x2318.¶

The terminology of [RFC8259] applies except where clarified below. The terms "Primitive" and "Structured" are used to group the types as in Section 1 of [RFC8259]. Definitions for "Object", "Array", "Number", and "String" remain unchanged. Importantly "object" and "array" in particular do not take on a generic meaning, such as they would in a general programming context.¶

Additional terms used in this specification are defined below.¶

Value:

As per [RFC8259], a structure complying to the generic data model of JSON, i.e., composed of components such as containers, namely JSON objects and arrays, and atomic data, namely null, true, false, numbers, and text strings.¶

Member:

A name/value pair in an object. (Not itself a value.)¶

Name:

The name in a name/value pair constituting a member. (Also known as "key", "tag", or "label".) This is also used in [RFC8259], but that specification does not formally define it. It is included here for completeness.¶

Element:

A value in an array. (Also used with a distinct meaning in XML context for XML elements.)¶

Index:

A non-negative integer that identifies a specific element in an array.¶

Query:

Short name for JSONPath expression.¶

Argument:

Short name for the value a JSONPath expression is applied to.¶

Node:

The pair of a value along with its location within the argument.¶

Root Node:

The unique node whose value is the entire argument.¶

Nodelist:

A list of nodes. The output of applying a query to an argument is manifested as a list of nodes. While this list can be represented in JSON, e.g. as an array, the nodelist is an abstract concept unrelated to JSON values.¶

Normalized Path:

A simple form of JSONPath expression that identifies a node by providing a query that results in exactly that node. Similar to, but syntactically different from, a JSON Pointer [RFC6901].¶

For the purposes of this specification, a value as defined by [RFC8259] is also viewed as a tree of nodes. Each node, in turn, holds a value. Further nodes within each value are the elements of arrays and the member values of objects and are themselves values. (The type of the value held by a node may also be referred to as the type of the node.)¶

A query is applied to an argument, and the output is a nodelist.¶

1.2. Inspired by XPath

A frequently emphasized advantage of XML is the availability of powerful tools to analyse, transform and selectively extract data from XML documents. [XPath] is one of these tools.¶

In 2007, the need for something solving the same class of problems for the emerging JSON community became apparent, specifically for:¶

• Finding data interactively and extracting them out of [RFC8259] JSON values without special scripting.¶
• Specifying the relevant parts of the JSON data in a request by a client, so the server can reduce the amount of data in its response, minimizing bandwidth usage.¶

So what does such a tool look like for JSON? When defining a JSONPath, how should expressions look?¶

The XPath expression¶

/store/book[1]/title

looks like¶

x.store.book[0].title

or¶

x['store']['book'][0]['title']

in popular programming languages such as JavaScript, Python and PHP, with a variable x holding the argument. Here we observe that such languages already have a fundamentally XPath-like feature built in.¶

The JSONPath tool in question should:¶

• be naturally based on those language characteristics.¶
• cover only essential parts of XPath 1.0.¶
• be lightweight in code size and memory consumption.¶
• be runtime efficient.¶

1.3. Overview of JSONPath Expressions

JSONPath expressions always apply to a value in the same way as XPath expressions are used in combination with an XML document. Since a value is anonymous, JSONPath uses the abstract name $ to refer to the root node of the argument.¶

JSONPath expressions can use the dot notation¶

$.store.book[0].title

or the bracket notation¶

$['store']['book'][0]['title']

for paths input to a JSONPath processor. [1] Where a JSONPath processor uses JSONPath expressions as output paths, these will always be converted to Output Paths which employ the more general bracket notation. [2] Bracket notation is more general than dot notation and can serve as a canonical form when a JSONPath processor uses JSONPath expressions as output paths.¶

JSONPath allows the wildcard symbol * for member names and array indices. It borrows the descendant operator .. from [E4X] and the array slice syntax proposal [start:end:step] [SLICE] from ECMASCRIPT 4.¶

JSONPath was originally designed to employ an underlying scripting language for computing expressions. The present specification defines a simple expression language that is independent from any scripting language in use on the platform.¶

JSONPath can use expressions, written in parentheses: (<expr>), as an alternative to explicit names or indices as in:¶

$.store.book[(@.length-1)].title

The symbol @ is used for the current node. Filter expressions are supported via the syntax ?(<boolean expr>) as in¶

$.store.book[?(@.price < 10)].title

Here is a complete overview and a side by side comparison of the JSONPath syntax elements with their XPath counterparts.¶

XPath has a lot more to offer (location paths in unabbreviated syntax, operators and functions) than listed here. Moreover there is a significant difference how the subscript operator works in Xpath and JSONPath:¶

• Square brackets in XPath expressions always operate on the node set resulting from the previous path fragment. Indices always start at 1.¶
• With JSONPath, square brackets operate on the object or array addressed by the previous path fragment. Array indices always start at 0.¶

3.1. Overview

A JSONPath query is a string which selects zero or more nodes of a piece of JSON. A valid query conforms to the ABNF syntax defined by this document.¶

A query MUST be encoded using UTF-8. To parse a query according to the grammar in this document, its UTF-8 form SHOULD first be decoded into Unicode code points as described in [RFC3629].¶

A string to be used as a JSONPath query needs to be well-formed and valid. A string is a well-formed JSONPath query if it conforms to the syntax of JSONPath. A well-formed JSONPath query is valid if it also fulfills all semantic requirements posed by this document.¶

The well-formedness and the validity of JSONPath queries are independent of the value the query is applied to; no further errors can be raised during application of the query to a value.¶

(Obviously, an implementation can still fail when executing a JSONPath query, e.g., because of resource depletion, but this is not modeled in the present specification.)¶

3.2. Processing Model

In this specification, the semantics of a JSONPath query are defined in terms of a processing model. That model is not prescriptive of the internal workings of an implementation: Implementations may wish (or need) to design a different process that yields results that conform to the model.¶

In the processing model, a valid query is executed against a value, the argument, and produces a list of zero or more nodes of the value.¶

The query is a sequence of zero or more selectors, each of which is applied to the result of the previous selector and provides input to the next selector. These results and inputs take the form of a nodelist, i.e., a sequence of zero or more nodes.¶

The nodelist going into the first selector contains a single node, the argument. The nodelist resulting from the last selector is presented as the result of the query; depending on the specific API, it might be presented as an array of the JSON values at the nodes, an array of Output Paths referencing the nodes, or both -- or some other representation as desired by the implementation. Note that the API must be capable of presenting an empty nodelist as the result of the query.¶

A selector performs its function on each of the nodes in its input nodelist, during such a function execution, such a node is referred to as the "current node". Each of these function executions produces a nodelist, which are then concatenated into the result of the selector.¶

The processing within a selector may execute nested queries, which are in turn handled with the processing model defined here. Typically, the argument to that query will be the current node of the selector or a set of nodes subordinate to that current node.¶

3.3. Syntax

Syntactically, a JSONPath query consists of a root selector ($), which stands for a nodelist that contains the root node of the argument, followed by a possibly empty sequence of selectors.¶

json-path = root-selector *(dot-selector        /
                            dot-wild-selector   /
                            index-selector      /
                            index-wild-selector /
                            union-selector      /
                            slice-selector      /
                            descendant-selector /
                            filter-selector)

The syntax and semantics of each selector is defined below.¶

3.4. Semantics

The root selector $ not only selects the root node of the argument, but it also produces as output a list consisting of one node: the argument itself.¶

A selector may select zero or more nodes for further processing. A syntactically valid selector MUST NOT produce errors. This means that some operations which might be considered erroneous, such as indexing beyond the end of an array, simply result in fewer nodes being selected.¶

But a selector doesn't just act on a single node: a selector acts on each of the nodes in its input nodelist and concatenates the resultant nodelists to form the result nodelist of the selector.¶

For each node in the list, the selector selects zero or more nodes, each of which is a descendant of the node or the node itself.¶

For instance, with the argument {"a":[{"b":0},{"b":1},{"c":2}]}, the query $.a[*].b selects the following list of nodes: 0, 1 (denoted here by their value). Let's walk through this in detail.¶

The query consists of $ followed by three selectors: .a, [*], and .b.¶

Firstly, $ selects the root node which is the argument. So the result is a list consisting of just the root node.¶

Next, .a selects from any input node of type object and selects the node of any member value of the input node corresponding to the member name "a". The result is again a list of one node: [{"b":0},{"b":1},{"c":2}].¶

Next, [*] selects from any input node which is an array and selects all the elements of the input node. The result is a list of three nodes: {"b":0}, {"b":1}, and {"c":2}.¶

Finally, .b selects from any input node of type object with a member name b and selects the node of the member value of the input node corresponding to that name. The result is a list containing 0, 1. This is the concatenation of three lists, two of length one containing 0, 1, respectively, and one of length zero.¶

As a consequence of this approach, if any of the selectors selects no nodes, then the whole query selects no nodes.¶

In what follows, the semantics of each selector are defined for each type of node.¶

3.5. Selectors

A JSONPath query consists of a sequence of selectors. Valid selectors are¶

• Root selector $¶
• Dot selector .<name>, used with object member names exclusively.¶
• Dot wild card selector .*.¶
• Index selector [<index>], where <index> is either an (possibly negative) array index or an object member name.¶
• Index wild card selector [*].¶
• Array slice selector [<start>:<end>:<step>], where <start>, <end>, <step> are integer literals.¶
• Nested descendants selector ...¶
• Union selector [<sel1>,<sel2>,...,<selN>], holding a comma delimited list of index, index wild card, array slice, and filter selectors.¶
• Filter selector [?(<expr>)]¶
• Current item selector @¶

root-selector  = "$"

dot-selector    = "." dot-member-name
dot-member-name = name-first *name-char
name-first =
                      ALPHA /
                      "_"   /           ; _
                      %x80-10FFFF       ; any non-ASCII Unicode character
name-char = DIGIT / name-first

DIGIT           =  %x30-39              ; 0-9
ALPHA           =  %x41-5A / %x61-7A    ; A-Z / a-z

dot-wild-selector    = "." "*"            ;  dot followed by asterisk

index-selector      = "[" (quoted-member-name / element-index) "]"

quoted-member-name  = string-literal

string-literal      = %x22 *double-quoted %x22 /       ; "string"
                      %x27 *single-quoted %x27         ; 'string'

double-quoted       = unescaped /
                      %x27      /                       ; '
                      ESC %x22  /                       ; \"
                      ESC escapable

single-quoted       = unescaped /
                      %x22      /                       ; "
                      ESC %x27  /                       ; \'
                      ESC escapable

ESC                 = %x5C                              ; \  backslash

unescaped           = %x20-21 /                         ; s. RFC 8259
                      %x23-26 /                         ; omit "
                      %x28-5B /                         ; omit '
                      %x5D-10FFFF                       ; omit \

escapable           = ( %x62 / %x66 / %x6E / %x72 / %x74 / ; \b \f \n \r \t
                          ; b /         ;  BS backspace U+0008
                          ; t /         ;  HT horizontal tab U+0009
                          ; n /         ;  LF line feed U+000A
                          ; f /         ;  FF form feed U+000C
                          ; r /         ;  CR carriage return U+000D
                          "/" /          ;  /  slash (solidus)
                          "\" /          ;  \  backslash (reverse solidus)
                          (%x75 hexchar) ;  uXXXX      U+XXXX
                      )

hexchar = non-surrogate / (high-surrogate "\" %x75 low-surrogate)
non-surrogate = ((DIGIT / "A"/"B"/"C" / "E"/"F") 3HEXDIG) /
                 ("D" %x30-37 2HEXDIG )
high-surrogate = "D" ("8"/"9"/"A"/"B") 2HEXDIG
low-surrogate = "D" ("C"/"D"/"E"/"F") 2HEXDIG

HEXDIG = DIGIT / "A" / "B" / "C" / "D" / "E" / "F"

; Task from 2021-06-15 interim: update ABNF later

element-index   = int                             ; decimal integer

int             = ["-"] ( "0" / (DIGIT1 *DIGIT) ) ; -  optional
DIGIT1          = %x31-39                         ; 1-9 non-zero digit

index-wild-selector    = "[" "*" "]"  ;  asterisk enclosed by brackets

slice-selector = "[" slice-index "]"

slice-index    = ws [start] ws ":" ws [end] [ws ":" ws [step] ws]

start          = int       ; included in selection
end            = int       ; not included in selection
step           = int       ; default: 1

ws             = *( %x20 / ; Space
                    %x09 / ; Horizontal tab
                    %x0A / ; Line feed or New line
                    %x0D ) ; Carriage return

FUNCTION Normalize(i, len):
  IF i >= 0 THEN
    RETURN i
  ELSE
    RETURN len + i
  END IF

FUNCTION Bounds(start, end, step, len):
  n_start = Normalize(start, len)
  n_end = Normalize(end, len)

  IF step >= 0 THEN
    lower = MIN(MAX(n_start, 0), len)
    upper = MIN(MAX(n_end, 0), len)
  ELSE
    upper = MIN(MAX(n_start, -1), len-1)
    lower = MIN(MAX(n_end, -1), len-1)
  END IF

  RETURN (lower, upper)

IF step > 0 THEN

  i = lower
  WHILE i < upper:
    SELECT a(i)
    i = i + step
  END WHILE

ELSE if step < 0 THEN

  i = upper
  WHILE lower < i:
    SELECT a(i)
    i = i + step
  END WHILE

END IF

descendant-selector = ".." ( dot-member-name      /  ; ..<name>
                             index-selector       /  ; ..[<index>]
                             index-wild-selector  /  ; ..[*]
                             "*"                     ; ..*
                           )

union-selector = "[" ws union-entry 1*(ws "," ws union-entry) ws "]"

union-entry    =  ( quoted-member-name /
                    element-index      /
                    slice-index
                  )

filter-selector    = "[?" boolean-expr "]"

boolean-expr = logical-expr
logical-expr = ([neg-op] primary-expr) / logical-or-expr
neg-op       = "!"                                  ; not operator
primary-expr = "(" logical-or-expr ")"
logical-or-expr = logical-and-expr *["||" logical-and-expr]
logical-and-expr = comp-expr *["&&" comp-expr]

comp-expr    = (rel-path-val /
                json-path) [(comp-op comparable / ; comparison
                            regex-op regex     /  ; RegEx test
                            in-op container )]    ; containment test
comp-op      = "==" / "!=" /                        ; comparison ...
               "<"  / ">"  /                        ; operators
               "<=" / ">="
regex-op     = "~="                                 ; RegEx match
in-op        = " in "                               ; in operator
comparable   = number / string-literal /            ; primitive ...
               true / false / null /                ; values only
               rel-path-val /                       ; descendant value
               json-path                            ; any value

rel-path-val = "@" *(dot-selector / index-selector)

container = <TO BE DEFINED>
regex = <TO BE DEFINED>