Internet-Draft | Deterministic Encoding | July 2023 |
Bormann | Expires 24 January 2024 | [Page] |
CBOR (STD 94, RFC 8949) defines "Deterministically Encoded CBOR" in its Section 4.2. The present document provides additional information about use cases, deployment considerations, and implementation choices for Deterministic Encoding.¶
This note is to be removed before publishing as an RFC.¶
Status information for this document may be found at https://datatracker.ietf.org/doc/draft-bormann-cbor-det/.¶
Discussion of this document takes place on the Concise Binary Object Representation Maintenance and Extensions (CBOR) Working Group mailing list (mailto:cbor@ietf.org), which is archived at https://mailarchive.ietf.org/arch/browse/cbor/. Subscribe at https://www.ietf.org/mailman/listinfo/cbor/.¶
Source for this draft and an issue tracker can be found at https://github.com/cabo/det.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 24 January 2024.¶
Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
The Concise Binary Object Representation (CBOR, [STD94] as documented in RFC 8949) is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation.¶
In many cases, CBOR allows some information to be encoded in several variants, which provide different amounts of space and thus lengths in Bytes. The encoder is generally free to choose the length that is most practical for it (with the constraint, of course, that the data need to fit). For most encoders, it is natural to always choose the shortest form available (essentially avoiding leading zeros). Section 4.1 (Preferred Serialization) of [STD94] names this practice and provides additional guidance for CBOR implementations; another term in use is "Preferred Encoding".¶
Section 4.2 (Deterministically Encoded CBOR) of [STD94] goes beyond the Preferred Serialization practice by providing rules for Deterministic Encoding. The objective of Deterministic Encoding is to, deterministically, always produce the same encoding for data items that are equivalent at the data model level. To achieve this, Preferred Serialization is mandated, an encoding choice intended for incremental encoding (indefinite length encoding) is disabled, and additional effort is expended for encoding key/value pairs in maps (the order of which does not matter semantically) in a deterministic order anyway.¶
Given that additional effort needs to be expended and/or implementation choices are taken away, neither Preferred Serialization nor Deterministic Encoding are mandatory in CBOR. (Contrast this with UTF-8 (Section 3 of [STD63]), which is always treating as "invalid" any encoding variants that are longer than necessary.)¶
Deterministic Encoding is defined in Section 4.2 of [STD94] (note that Section 4.2.3 of [STD94] defines a variant that was needed at the time for backward compatibility and will not be discussed further in this document). The present document elaborates on this normative definition by providing additional information about use cases, deployment considerations, and implementation choices for Deterministic Encoding; it is an informational document that however may still be cited where a single reference for Deterministic Encoding is convenient.¶
The definitions of [STD94] apply. Readers are expected to be familiar with CBOR, and particularly so with Sections 4.1 and 4.2 of [STD94].¶
The following terms introduced in the text of [STD94] receive their own separate definitions here:¶
a set of choices made during Serialization (Encoding) that generally leads to shortest-form encodings where a choice of encoding lengths is available, without expending additional effort on converting between different kinds of data item. See Section 4.1 of [STD94] and the terms defined in that section. The Preferred Encoding rules for data items in the Basic Generic Data Model may be augmented by rules for specific Tags, see for instance Section 3.4.3 of [STD94].¶
Preferred Serialization¶
An encoding process that employs Preferred Serialization and makes additional decisions to always (deterministically) lead to the exact same encoding for equivalent inputs at the data model level. Similar to Preferred Serialization, the equivalence model as defined for the Basic Generic Data Model may be augmented by equivalence rules defined for specific Tags (see also Section 2.1 of [STD94]).¶
CBOR data items at the data model level are represented in the CBOR diagnostic notation (Section 8 of [STD94] as extended by Appendix G of [RFC8610], further elaborated in [I-D.ietf-cbor-edn-literals]), abbreviated with "EDN" (extended diagnostic notation).¶
While this document is informative, it does use certain keywords to indicate practical requirements for interoperability. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
Before discussing further details of Deterministic Encoding, we would like to point out three areas of use cases, which differ enough in the resulting objectives that it is worth to have terminology for them.¶
In many cases, diagnostic procedures benefit from having available a single, easily comparable representation of some data:¶
Comparing outputs of a test or validation suite during development¶
Improving the presentation of diagnostic information to humans¶
By minimizing inconsequential differences between representations of similar data, humans may be faster in finding information they are interested in. In particular inconsistent map ordering can easily hide information that would have been useful for diagnostic purposes. Transformation to human-readable forms may be easier and more useful if there is only one form of representation for the interchanged data.¶
Many systems cache (memoize) results of a request so they can reply with the cached result when the same request comes in again and the context of the reply has not changed.¶
If two requests that are semantically the same also have the same representation, the representation (or its hash) can serve as an efficient cache key. If the request is already encoded deterministically, this is by definition the case; alternatively, the recipient can re-encode a request with Deterministic Encoding.¶
Were the Deterministic Encoding to fail, this could lead to cache failures, which could be benign, but also could be specifically evoked by an active attacker to degrade a system.¶
As usual for deterministically encoded data, not all forms of application equivalence imply equivalence at the data model level, so some equivalence processing (deterministic representation) may be required at the application level as well, to achieve equivalent representations and thus a good cache hit rate.¶
Security Frameworks such as COSE and JOSE sign or MAC (authenticate with a Message Authentication Code, MAC) information in the form in which it has actually been interchanged, making representation variants less relevant.¶
However, in some cases, the signing input for a signature or a MAC may need to be derived from data at rest and/or specific transformations of the data that was interchanged. Such a transformation is fraught with perils at the application level that may be exploited by attackers; this problem is outside the scope of the present document. Deterministic Encoding may remove one potential source of variability that might make signatures or MACs useless between systems.¶
CBOR implementations can be specific to a particular application, or they can be Generic. There is a strong incentive to be able to use a Generic encoder/decoder across the spectrum of CBOR applications; CBOR applications that require specific support from an encoder/decoder can considerably reduce the wide implementation support CBOR enjoys from existing generic implementations. So, as a general best practice, we want to minimize the number of ways an application may need to influence a generic coder/decoder by options, flags, switches, etc.¶
There is some expectation that, barring any particular constraints that would make this more difficult than normally, a CBOR encoder will use Preferred Encoding, in particular generic encoders. Deterministic Encoding, however, will be switched on explicitly in most implementations. Note that Preferred Encoding, while using the shortest form available for the specific data item to be encoded, doesn't have that shortness as the overriding objective: Conversions of a data item into a different one to achieve shorted encoding are not part of the processing labeled "Preferred Encoding". (This is particularly relevant for CBOR's different numeric systems; see Section 3.2.2 below.)¶
Some applications will also want to check that an encoded input actually satisfies the requirements for Deterministic Encoding. By the definition of Deterministic Encoding, this can be done after decoding a data item by deterministically encoding the just decoded data item and comparing the result with the decoding input. However, specific support for checking immediately in the decoding process can be more efficient.¶
As a result, support for Deterministic Encoding in generic encoder implementations is RECOMMENDED to be provided by a flag to switch on (or separate function that enables) Deterministic Encoding. Similarly, generic decoders are RECOMMENDED to have a flag to switch on/separate function to enable checking, whether that is efficiently implemented during decoding or less efficiently by comparing re-encoding.¶
In many specifications, asserting that interchange is based on deterministically encoded data items (and specifying what has to happen if that is not the case) is all that is needed.¶
Some specifications define a media type for their interchange formats. This definition is a good place to reiterate that a deterministically encoded data item is required for instances of that media type.¶
A question arises whether a Structured Syntax Suffix (SSS, [RFC6838]) should be defined for CBOR data items in Deterministic Encoding (and similarly for CBOR sequences [RFC8742] of such).¶
There is precedent for this approach, as ASN.1 DER (Distinguished
Encoding Rules) has an SSS, +der
.
However, this appears misguided as the purpose of an SSS is to enable
processing of the underlying data representation format, and any ASN.1
BER (Basic Encoding Rules) processor (+ber
) can also process a
+der
instance, which is not apparent from the +der
suffix.
(This was maybe mitigated by introducing both SSS at the same time.)
Similarly, any CBOR decoder today can process a deterministically
encoded data items as plain CBOR data items (without any mitigation of
having introduced a related suffix at the same time), so the SSS
should be the usual +cbor
/+cbor-seq
.
(The additional processing that would be enabled by identifying data
items as deterministically encoded appears rather limited.)¶
Alternatively, instead of replacing +cbor
, an indication of
Deterministic Encoding could be provided by adding multiple suffixes
to the SSS concept.
There is an ongoing effort to define a more complex structure of media
type suffixes, as documented in [I-D.ietf-mediaman-suffixes].
In general, the combination of multiple SSS in one media type name
raises similar questions to the multiple inheritance problem in
object-oriented programming languages, so it may not be easy to use
such a mechanism in practice.¶
As an extension to JSON objects in JSON [STD90], maps are an important data structure in the CBOR generic data model to obtain extensibility of "struct"-like data (see Section 2 of [RFC8610]). Where this is not needed or can be provided in another way, expressing the entire data item without the use of maps can be an efficient option, avoiding any additional processing for Deterministic Encoding beyond that needed for Preferred Encoding. (This requires ensuring that a similar kind of uncertainty then does not occur at the application level, though.)¶
Support for Deterministic Encoding can be added to an API for a generic CBOR encoder and decoder by adding one flag each:¶
Additional elements could be added to a decoder API to give diagnostic information about inputs that were not deterministically encoded, e.g., by flagging elements with error codes. It is often useful to give the application full information about well-formed CBOR that is not deterministically encoded even when it should be. However, if a flag for checking is provided and switched on, there SHOULD be no chance that any other decoded data item is mistaken for one that was encoded deterministically.¶
As reordering maps for Deterministic Encoding is relatively expensive, a generic encoder can also offer additional APIs for providing map content in a pre-ordered form. If an encoder complies with Preferred Encoding and maps can be supplied in ordered form, an explicit Deterministic Encoding flag may not be required. If it is, it is RECOMMENDED that the encoder not simply rely on the assumption that inputs were properly ordered by the application.¶
Generating deterministically encoded data items requires arranging key/value pairs in maps into an order defined in Section 4.2.1 of [STD94].¶
This map is ordered by the byte-wise lexicographic ordering of the deterministically encoded map keys. Section 4.2.1 of [STD94] notes:¶
Implementation note: the self-delimiting nature of the CBOR encoding means that there are no two well-formed CBOR encoded data items where one is a prefix of the other. The bytewise lexicographic comparison of deterministic encodings of different map keys therefore always ends in a position where the byte differs between the keys, before the end of a key is reached.¶
Also, an implementation may be able to make use of the property that map keys in Deterministic Encodings are ordered by the following information, in order of precedence:¶
any content of the key data item, such as¶
I may be expeditious to use this property, e.g. by processing integers first, starting with unsigned integers in ascending order and then negative integers in descending order, and then strings (byte strings first), ordered by their length in bytes (encoded in the argument) and then the string content, arrays ordered by length and then content, and maps ordered by length and then content. Often, and particularly with integer and string keys, it may not be necessary to actually build a deterministically encoded data item for a map key to perform the overall map content ordering.¶
To enable the use of generic encoders, applications are encouraged to define rules for representing application information in the CBOR generic data model that enable the use of Preferred Encoding on that level as well.¶
Applications can also define their own deterministic encoding rules, as for instance FIDO CTAP2 (Client to Authenticator Protocol [CTAP2]) does with the CTAP2 canonical CBOR encoding form (Section 6 of [CTAP2]). Its description appears to be derived from an equivalent of Section 4.2.3 of [STD94]. (The actual structure of CTAP2 limits its use to cases where that is compatible with standard Deterministic Encoding; there is text in the specification that calls for revisiting the definition when this would no longer be the case.)¶
Application-specific deterministic encoding rules can make it difficult to use existing generic encoders and may therefore diminish the value of using a standard representation format. However, applications can also define transformations of their data into a more limited data model that reduces the cases the Deterministic Encoding rules have to implement. This allows both the following implementation choices:¶
This section describes some of the considerations that led to one such application profile for Deterministic Encoding.¶
The dCBOR specification (originally in [I-D.mcnally-deterministic-cbor], with a condensed form proposed in [I-D.bormann-cbor-dcbor]) describes the pervasive use of Deterministic Encoding throughout an application. It also defines a simplified application data model of numbers, where there no longer is a distinction between integers and floating point numbers at the application data model level — all numbers are of a single numeric type, and the choice of integer or floating point representations is made based on value:¶
The underlying CBOR Deterministic Encoding rules ensure that, in both cases, the shortest form for the case will then be used for encoding.¶
Reducing the separate integer and floating point spaces to a single numeric space is particularly attractive in implementation languages that also only have a single such space, such as JavaScript [ECMA262]. (While JavaScript recently has acquired a separate integer type, it is much less well integrated into the language and existing libraries than the general numeric type.)¶
The original specification [I-D.mcnally-deterministic-cbor] restated much of Section 4.2 of [STD94] and
added a rule that doesn't allow compatibility with Deterministic
Encoding (disallowing the interchange of basic negative integers in the range
-2
64 to -2
63-1
).¶
It also had a mention as future work of subnormal values [IEEE754], which work fine for interchange (even with Deterministic Encoding) in [STD94]. Note that specific values may not be available to applications that employ floating point hardware implementing the FTZ ("flush to zero") and/or DAZ ("denormals are zero") strategies. These may then require special handling in the application data model. It is generally difficult to rely on exact equality of floating point values, which however is what Deterministic Encoding requires.¶
A streamlined specification of the dCBOR numeric-reduction based application profile is proposed in [I-D.bormann-cbor-dcbor].¶
Certain applications could make use of a Deterministic Encoding for JSON [STD90] data. Deterministically Encoded CBOR provides an attractive solution to that as it is already well-defined.¶
While the data model of JSON is not well-defined, I-JSON provides one interpretation that is generally accepted [RFC7493]. Section 6.2 (Converting from JSON to CBOR) of [STD94] provides a way to transform JSON data that conform to this data model to CBOR. When used with its default parameters, the combination of (1) I-JSON, (2) the JSON-to-CBOR transformation, and (3) the rules for CBOR Deterministic Encoding provide a well-defined Deterministic Encoding for JSON data.¶
Transforming decoded CBOR data after interchange back to data-model level JSON data can be done with the inverse of Section 6.2 of [STD94] (the full generality of Section 6.1 (Converting from CBOR to JSON) of [STD94] is obviously not required as only the JSON subset of the CBOR generic data model is used).¶
Comparing the handling of numeric data in the JSON-to-CBOR
transformation to that reported in Section 6.1, the main difference is
that the former only maps integral values between
-2
53+1
and 2
53-1
to basic CBOR integers
and leaves the others in floating point form.
(The rationale is that only this range is injective ("unambiguous" or
"exact") in the mapping of integers to binary64 floating point
values, which may be a desirable property beyond the use in JSON
encoding.)¶
One of the major use cases of Deterministic Encoding is in security, namely in the derivation of signing inputs from some CBOR data only available at the model level. Any transformation error from the application data to the CBOR model level and then to deterministic encoding can lead to a potential exploit by an attacker.¶
TODO further Security Considerations¶
This document has no IANA actions.¶
This document was motivated by the work of Wolf McNally and Christopher Allen as documented in [I-D.mcnally-deterministic-cbor] and discussed in 2023 in the CBOR working group. It collects information that is present in the apps-discuss and CBOR WG mailing list discussions since 2013, but not necessarily easy to find. The author is grateful to the many contributors to these discussions.¶