| Internet-Draft | cbor-file-magic | March 2021 |
| Richardson | Expires 7 September 2021 | [Page] |
This document proposes an on-disk format for CBOR objects that is friendly to common on-disk recognition systems like the Unix file(1) command.¶
This document is being discussed at: https://github.com/mcr/cbor-magic-number¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 7 September 2021.¶
Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.¶
Since very early in computing, operating systems have sought ways to mark which files could be processed by which programs.¶
For instance, the Unix file(1) command, which has existed since 1973 ([file]), has been able to identify many file formats for decades. Many systems (Linux, MacOS, Windows) will select the correct application based upon the file contents, if the system can not determine it by other means: for instsance, MacOS maintains a resource fork that includes MIME information and therefore ideally never needs to know what anything about the file. Other systems do this by file extensions.¶
While having a MIME type associated with the file is a better solution in general, when files become disconnected from their type information, such as when attempting to do forensics on a damaged system, then being able to identify a file type can become very important.¶
It is noted that in the MIME type registration, that a magic number is asked for, if available, as is a file extension.¶
A challenge for the file(1) program is often that it can be confused by the encoding vs the content. For instance, an Android "apk" used to transfer and store an application may be identified as a ZIP file. Both OpenOffice or MSOffice files are XML files, but appear as ZIP. Unless they OpenOffice files are flat (fodp) files, in which case they may appear to be generic XML files.¶
As CBOR becomes a more and more common encoding for a wide variety of artifacts, identifying them as just "CBOR" is probably not useful. This document provides a way to encode a magic number into the beginning of a CBOR format file. Two options are presented: typically a CBOR Protocol author will specify one.¶
A CBOR Protocol is a specification which uses CBOR as it's encoding. Examples of CBOR Protocols currently under development include CoSWID [I-D.ietf-sacm-coswid], and EAT [I-D.ietf-rats-eat]. COSE itself [RFC8152] is considered infrastructure, however the encoding of public keys in CBOR as described in [I-D.mattsson-cose-cbor-cert-compress] would be an identified CBOR Protocol.¶
A major inspiration for this document is observing the mess in ASN.1 based systems where most files are PEM encoded, identified by the extension "pem", confusing public keys, private keys, certificate requests and SIME content.¶
These proposals are invasive to how CBOR protocols are written to disk, but in both cases, the proposed envelope does not require that the tag be transfered on the wire.¶
In addition to the on-disk identification aspects, there are some protocols which may benefit from having such a magic on the wire if they presently using a different (legacy) encoding scheme. The presence of the identifiable magic sequence signals that CBOR is being used or a legacy scheme.¶
A magic number is ideally a unique fingerprint, present in the first 4 or 8 bytes of the file, which does not change when the content change, and does not depend upon the length of the file.¶
Less ideal solutions have a pattern that needs to be matched, but in which some bytes need to be ignored. While the Unix file(1) command can be told to ignore bytes, this can lead to ambiguities.¶
There are two variations of this practice. Both use CBOR Tags in a way that results in a deterministic first 8 to 12 bytes.¶
CBOR Protocol designers should obtain a tag for each major object that they might store on disk. As there are more than 4 million available 4-byte tags, there should be issue in allocating a few to all available CBOR Protocols.¶
The policy is First Come First Served, so all that is required is an email to IANA, having filled in the small template provided in section 9.2 of [RFC8949].¶
This tag should be allocated by the author of the CBOR Protocol, and to be in the four-byte range, it should be at least 0x01000000 (decimal 16777216) in value.¶
The use of a sequence of four US-ASCII codes which are pneumonic to the protocol is encouraged, but not required.¶
This proposal starts with the Self-described CBOR tag, 55799, as described in [RFC8949] section 3.4.6.¶
A second CBOR Tag is then allocated to describe the specific Protocol involved, as described above.¶
This proposal wraps the CBOR value as tags usually do. Applications that need to send the CBOR value across a constrained link may wish to remove the two tags if the use is implicitely understood. This is a decision of the CBOR Protocol specification.¶
This proposal makes use of CBOR Sequences as described in [RFC8742].¶
This proposal consists of two tags and a constant string for a total of 12 bytes.¶
The first part identifies the file as being CBOR, and does so with all the desirable properties explained in [RFC8949] section 3.4.6. Specifically, it does not seem to conflict with any known file types, and it is not valid Unicode.¶
The second part identifies which CBOR Protocol is used, as described above.¶
The third part is a constant value 0x43_42_4f_52, "CBOR". That is, it the three byte sequence 0x42_4f_52 ("BOR"). This is the thing which is tagged.¶
The actual CBOR Protocol value then follows, probably without any specific tag. The use of a CBOR Sequence allows the application to trivially remove the two tags.¶
This means that should a file be reviewed by a human (directly in an editor, or in a hexdump display), it will include the string "CBOR" prominently. This value is also included simply because the two tags need to tag something.¶
This document provides a way to identify CBOR Protocol objects. Clearly identifying CBOR contents on disk may have a variety of impacts.¶
The most obvious is that it may allow malware to identify interesting objects on disk, and then corrupt them.¶
IANA is asked to allocate one tag from the First Come, First Served Area of the Concise Binary Object Representation (CBOR) Tags, in the ("1+2") area.¶
The requested value is 55800 (TBD1). The value has been picked to have properties similiar to the 55799 tag.¶
The hexadecimal representation is: 0xd9_\d9_f8.¶
This is not valid UTF-8: the first 0xd9 puts the value into the three-byte value of UTF-8, but the 0xd9 as the second value is not a valid second byte for UTF-8.¶
This is not valid UTF-16: the byte sequence 0xd9d9 (in either endian order), puts this value into the UTF-16 high-half zone, which would signal that this a 32-bit Unicode value. However, the following 16-bit big-endian value 0xf8.. is not a valid second sequence according to [RFC2781]. On a little-endian system, it would be necessary to examine the fourth byte to determine if it is valid. That next byte is determined by the subsequent encoding, and [RFC8949] section 3.4.6 has already determined that no valid CBOR encodings result in a valid UTF-16.¶
Data Item: byte string Semantics: indicates that the file contains CBOR Sequences¶
The CBOR WG brainstormed this protocol on January 20, 2021.¶
The Openswan IPsec project has a daemon ("pluto"), and two control programs ("addconn", and "whack"). They communicate via a Unix-domain socket, over which a C-structure containing pointers to strings is serialized using a bespoke mechanism. This is normally not a problem as the structure is compiled by the same compiler; but when there are upgrades it is possible for the daemon and the control programs to get out of sync by the bespoke serialization. As a result, there are extra compenstations to deal with shutting the daemon down. During testing it is sometimes the case that upgrades are backed out.¶
In addition, when doing unit testing, the easiest way to load policy is to use the normal policy reading process, but that is not normally loaded in the daemon. Instead the IPC that is normally send across the wire is compiled/serialized and placed in a file. The above magic number is included in the file, and also on the IPC in order to distinguish the "shutdown" command CBOR operation.¶
In order to reduce the problems due to serialization, the serialization is being changed to CBOR. Additionally, this change allows the IPC to be described by CDDL, and for any language that encode to CBOR can be used.¶
IANA has allocated the tag 1330664270, or 0x4f_50_ 53_ 4e for this purpose. As a result, each file and each IPC is prefixed with:¶
In diagnostic notation: ~~~~ 55800(1330664270(h'424F52')) ~~~~¶
Or in hex: ~~~~ 00000000 d9 d9 f8 da 4f 50 53 4e 43 42 4f 52 |....OPSNCBOR| ~~~~¶