Internet-Draft | JMAP Blob | December 2021 |
Gondwana | Expires 23 June 2022 | [Page] |
The JMAP base protocol (RFC8620) provides the ability to upload and download arbitrary binary data via HTTP POST and GET on defined endpoint. This binary data is called a "blob".¶
This extension adds additional ways to create and access blobs, by making inline method calls within a standard JMAP request.¶
This extension also adds a reverse lookup mechanism to discover where blobs are referenced within other data types.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 23 June 2022.¶
Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
Sometimes JMAP ([RFC8620]) interactions require creating a blob and then referencing it. In the same way that IMAP Literals ([RFC7888]) were extended to reduce roundtrips for simple data, embedding simple small blobs into the JMAP method stream can reduce roundtrips.¶
Likewise, when fetching an object, it can be useful to also fetch the raw content of that object without a separate roundtrip.¶
Since raw blobs could contain arbitrary binary data, this document allows the use of the base64 coding specified in [RFC4648].¶
Where JMAP is being proxied through a system which applies additional access restrictions, it can be useful to know where a blob is referenced in order to decide whether to allow it to be downloaded, so this document defines a way to look up where a particular blobId is referenced.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
The capabilities object is returned as part of the JMAP Session object; see [RFC8620], Section 2.¶
This document defines an additional capability URI.¶
The capability urn:ietf:params:jmap:blob
being present in the
"accountCapabilities" property of an account represents support
for additional API methods on the Blob datatype. Servers that
include the capability in one or more "accountCapabilities"
properties MUST also include the property in the "capabilities"
property.¶
The value of this property in the JMAP session "capabilities" property MUST be an empty object.¶
The value of this property in an account's "accountCapabilities" property is an object that MAY contain the following information on server capabilities and permissions for that account:¶
maxSizeBlobSet: UnsignedInt|null
¶
if set, gives the maximum size of a blob in octets that the server will allow to be created (size of the final output of catenate or of encoded forms).¶
Clients MUST NOT attempt to create blobs larger than this size.¶
If this value is not set, clients MUST NOT attempt to create blobs
larger than the value of maxSizeUpload
specified in [RFC8620]. If
neither of those limits are specified by the server, then clients are not
required to limit the size of blob they try to create, though servers can
always reject creation of blobs regardless of size, e.g. due to lack of
disk space or per-user rate limits.¶
maxCatenateItems: UnsignedInt|null
¶
if set, gives the maximum number of of CatenateSourceObjects allowed per creation in a Blob/set. Servers MUST allow at least 64 items.¶
If not set, clients SHOULD assume a limit of 64 items.¶
supportedTypeNames: [String]|null
¶
an array of data type names that are
supported for Blob/lookup
. May be null
or not present if the account
does not support reverse lookups.¶
{ "capabilities": { ..., "urn:ietf:params:jmap:blob": {} }, "accounts": { "A13842": { ... "accountCapabilities": { "urn:ietf:params:jmap:blob": { "maxSizeBlobSet": 50000000, "maxCatenateItems": 100, "supportedTypeNames" : [ "Mailbox", "Thread", "Email" ] } } } } }¶
A blob is a sequence of zero or more octets.¶
The JMAP base spec [RFC8620] defines the Blob/copy
method, which
is unchanged by this specfication, and is selected by the
urn:ietf:params:jmap:core
capability.¶
The following JMAP Methods are selected by the
urn:ietf:params:jmap:blob
capability.¶
This is a standard JMAP set
method.¶
Properties:¶
Exactly one of:¶
data:asText: String
¶
data which can be represented as utf-8 encoded text¶
data:asBase64: String
¶
binary data encoded as a ([RFC4648] Section 4) Base 64 string¶
catenate: [CatenateSourceObject]
¶
list of one or more octet sources in order¶
Also optionally:¶
Result is:¶
id: Id
¶
the blobId which was created¶
type: String|null
¶
the media type as given in the creation (if any); or detected from content; or null¶
size: UnsignedInt
¶
as per RFC8620 - the size of the created blob in octets¶
Plus any other properties identical to those that would be returned in the JSON response of the RFC8620 upload endpoint (which may be extended in the future - this document anticipates that implementations will extend both the upload endpoint and the Blob/set responses in the same way)¶
CatenateSourceObject:¶
Exactly one of:¶
or a blobId source:¶
If null
then offset is assumed to be zero.¶
If null
then length is the remaining octets in the blob.¶
If the range can not be fully satisfied (i.e. extends past the end of the data in the blob) then the catenate itself is invalid and results in a notCreated response for this creation id.¶
If the data properties or catenate properties have any invalid references or invalid data contained in them, the server MUST NOT guess as to the user's intent, and MUST reject the creation and return a notCreated response for that creation id.¶
Likewise, invalid characters in the base64 of data:asBase64, or invalid UTF-8 in data:asText MUST result in a nonCreated response.¶
It is legal to create a blob by calling catenate with a single CatenateSourceObject. Please note that a catenate source MUST NOT contain additional sub-catenates, only data or blob sources.¶
It is envisaged that catenate sources might be extended in future, for example to fetch external content.¶
A server SHOULD accept at least 64 catenate items.¶
It is not possible to update a Blob, so any update will result
in a notUpdated
response.¶
It is not possible to destroy a Blob, so any destroy will result
in a notDestroyed
response.¶
The data:asBase64 field is set over multiple lines for ease of publication here, however all data:asBase64 would be sent as a continuous string with no whitespace on the wire.¶
Method Call: [ "Blob/set", { "accountId": "account1", "create": { "1": { "data:asBase64": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABAQMAAAAl21bKA AAAA1BMVEX/AAAZ4gk3AAAAAXRSTlN/gFy0ywAAAApJRE FUeJxjYgAAAAYAAzY3fKgAAAAASUVORK5CYII=", "type": "image/png" }, }, }, "R1" ] Response: [ "Blob/set", { "accountId" : "account1", "created" : { "1": { "id" : "G4c6751edf9dd6903ff54b792e432fba781271beb", "type" : "image/png", "size" : 95 }, }, }, "R1" ]¶
Method Calls: [ [ "Blob/set", { "create": { "b4": { "data:asText": "The quick brown fox jumped over the lazy dog." } } }, "S4" ], [ "Blob/set", { "create": { "cat": { "catenate": [ { "data:asText": "How" }, { "blobId": "#b4", "length": 7, "offset": 3 }, { "data:asText": "was t" }, { "blobId": "#b4", "length": 1, "offset": 1 }, { "data:asBase64": "YXQ/" } ] } } }, "CAT" ], [ "Blob/get", { "properties": [ "data:asText", "size" ], "ids": [ "#cat" ] }, "G4" ] ] Responses: [ [ "Blob/set", { "oldState": null, "created": { "b4": { "id": "Gc0854fb9fb03c41cce3802cb0d220529e6eef94e", "size": 45, "type": "application/octet-stream" } }, "updated": null, "destroyed": null, "notCreated": null, "notUpdated": null, "notDestroyed": null, "accountId": "account1" }, "S4" ], [ "Blob/set", { "oldState": null, "created": { "cat": { "id": "Gcc60576f036321ae6e8037ffc56bdee589bd3e23", "size": 19, "type": "application/octet-stream" } }, "updated": null, "destroyed": null, "notCreated": null, "notUpdated": null, "notDestroyed": null, "accountId": "account1" }, "CAT" ], [ "Blob/get", { "list": [ { "id": "Gcc60576f036321ae6e8037ffc56bdee589bd3e23", "data:asText": "How quick was that?", "size": 19 } ], "notFound": [], "accountId": "account1" }, "G4" ] ]¶
A standard JMAP get, with two additional optional parameters:¶
offset: UnsignedInt|null
¶
start this many octets into the blob data. If null or unspecified, this defaults to zero.¶
length: UnsignedInt|null
¶
return at most this many octets of the blob data. If null or unspecified, then all remaining octets in the blob are returned. This can be considered equivalent to an infinitely large length value, except that the isTruncated warning is not given unless the start offset is past the end of the blob.¶
Request Properties:¶
Any of¶
If not given, properties defaults to data
and size
.¶
Result Properties:¶
data:asText: String|null
¶
the raw octets of the selected range if they are valid UTF-8, otherwise null¶
data:asBase64: String
¶
the base64 encoding of the octets in the selected range¶
isEncodingProblem: Boolean
(default: false)¶
isTruncated: Boolean
(default: false)¶
size: UnsignedInt
¶
the number of octets in the entire blob¶
The size value MUST always be the number of octets in the underlying blob, regardless of offset and length.¶
The data fields contain a representation of the octets within the selected
range that are present in the blob. If the octets selected are not valid
UTF-8 (including truncating in the middle of a multi-octet sequence)
and data
or data:asText
was requested, then the key isEncodingProblem
MUST be set to true
and the data:asText
response value MUST be null
.
In the case where data
was requested and the data is not valid UTF-8,
then data:asBase64
MUST be returned.¶
If the selected range requests data outside the blob (i.e. the offset+length
is larger than the blob) then the result is either just the octets from the
offset to the end of the blob, or an empty string if the offset is past the
end of the blob. Either way, the isTruncated
property in the result MUST
be set to true
to tell the client that the requested range could not be
fully satisfied.¶
Where a blob containing the string "The quick brown fox jumped over
the lazy dog!" has blobId G6ec94756e3e046be78fcb33953b85b944e70673e
.¶
Method Call: [ "Blob/get", { "accountId" : "account1", "ids" : [ "G6ec94756e3e046be78fcb33953b85b944e70673e", "not-a-blob" ], "properties" : [ "data:asText", "data:asBase64", "size" ], "offset" : 4, "length" : 9 }, "R1" ] Response: [ "Blob/get", { "accountId": "account1", "list": [ { "id": "G6ec94756e3e046be78fcb33953b85b944e70673e", "data:asText": "quick bro", "data:asBase64": "cXVpY2sgYnJvCg==", "size": 46 } ], "notFound": [ "not-a-blob" ] }, "R1" ]¶
The b1
value is the text: "The quick brown fox jumped over the \x81\x81 fox"
which contains an invalid utf8 sequence.¶
The results have the following interesting properties:¶
G1: defaults to data
and size
- so b1 returns isEncodingProblem
and a base64 value.¶
G2: since data:asText
was explicitly selected, does not attempt to
return a value for the data, just isEncodingProblem
for b1.¶
G3: since only data:asBase64
was requested, there is no encoding
problem and both values are returned.¶
G4: since the requested range could be satisfied as text, both blobs
are returned as data:asText
and there is no encoding problem.¶
G5: both blobs cannot satisfy the requested range, so isTruncated is true for both.¶
Note: some values have been wrapped for line length - there would be
no whitespace in the data:asBase64
values on the wire¶
Method calls: [ [ "Blob/set", { "create": { "b1": { "data:asBase64": "VGhlIHF1aWNrIGJyb3duIGZveCBqdW1wZW Qgb3ZlciB0aGUggYEgZG9nLg==" }, "b2": { "data:asText": "hello world", "type" : "text/plain" } } }, "S1" ], [ "Blob/get", { "ids": [ "#b1", "#b2" ] }, "G1" ], [ "Blob/get", { "ids": [ "#b1", "#b2" ], "properties": [ "data:asText", "size" ] }, "G2" ], [ "Blob/get", { "ids": [ "#b1", "#b2" ], "properties": [ "data:asBase64", "size" ] }, "G3" ], [ "Blob/get", { "offset": 0, "length": 5, "ids": [ "#b1", "#b2" ] }, "G4" ], [ "Blob/get", { "offset": 20, "length": 100, "ids": [ "#b1", "#b2" ] }, "G5" ] ] Responses: [ [ "Blob/set", { "oldState": null, "created": { "b2": { "id": "G2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", "size": 11, "type": "application/octet-stream" }, "b1": { "id": "G72cfa4804194563685d9a4b695f7ba20e7739576", "size": 43, "type": "text/plain" } }, "updated": null, "destroyed": null, "notCreated": null, "notUpdated": null, "notDestroyed": null, "accountId": "account1" }, "S1" ], [ "Blob/get", { "list": [ { "id": "G72cfa4804194563685d9a4b695f7ba20e7739576", "isEncodingProblem": true, "data:asBase64": "VGhlIHF1aWNrIGJyb3duIGZveCBqdW1wZW Qgb3ZlciB0aGUggYEgZG9nLg==", "size": 43 }, { "id": "G2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", "data:asText": "hello world", "size": 11 } ], "notFound": [], "accountId": "account1" }, "G1" ], [ "Blob/get", { "list": [ { "id": "G72cfa4804194563685d9a4b695f7ba20e7739576", "isEncodingProblem": true, "size": 43 }, { "id": "G2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", "data:asText": "hello world", "size": 11 } ], "notFound": [], "accountId": "account1" }, "G2" ], [ "Blob/get", { "list": [ { "id": "G72cfa4804194563685d9a4b695f7ba20e7739576", "data:asBase64": "VGhlIHF1aWNrIGJyb3duIGZveCBqdW1wZW Qgb3ZlciB0aGUggYEgZG9nLg==", "size": 43 }, { "id": "G2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", "data:asBase64": "aGVsbG8gd29ybGQ=", "size": 11 } ], "notFound": [], "accountId": "account1" }, "G3" ], [ "Blob/get", { "list": [ { "id": "G72cfa4804194563685d9a4b695f7ba20e7739576", "data:asText": "The q", "size": 43 }, { "id": "G2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", "data:asText": "hello", "size": 11 } ], "notFound": [], "accountId": "account1" }, "G4" ], [ "Blob/get", { "list": [ { "id": "G72cfa4804194563685d9a4b695f7ba20e7739576", "isTruncated": true, "isEncodingProblem": true, "data:asBase64": "anVtcGVkIG92ZXIgdGhlIIGBIGRvZy4=", "size": 43 }, { "id": "G2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", "isTruncated": true, "data:asText": "", "size": 11 } ], "notFound": [], "accountId": "account1" }, "G5" ] ]¶
Given a list of blobIds, this method does a reverse lookup in each of the provided type names to find the list of Ids within that data type which reference the provided blob.¶
The definition of reference is somewhat loosely defined, but roughly means "you could discover this blobId by looking inside this object", for example if a Mailbox contains an Email which references the blobId, then it references that blobId. Likewise for a Thread.¶
Parameters¶
accountId: Id
¶
The id of the account used for the call.¶
typeNames: [String]
¶
A list of names from the "JMAP Data Types" registry. Only names for which "Can reference blobs" is true may be specified, and the capability which defines each type must also be used by the overall JMAP request in which this method is called.¶
If a type name is not known by the server, or the associated capability has not been requested, then the server returns an "unknownDataType" error.¶
ids: [Id]
¶
A list of blobId values to be looked for.¶
Response¶
BlobInfo Object¶
id: Id
¶
The Blob Identifier.¶
matchedIds: String[Id List]
¶
A map from type name to list of Ids of that data type (e.g. the name "Email" maps to a list of emailIds)¶
If a blob is not visible to a user at all, then the server SHOULD return that blobId in the notFound array, however it may also return an empty list for each type name, as it may not be able to know if other data types do reference that blob.¶
Method call: [ "Blob/lookup", { "typeNames": [ "Mailbox", "Thread", "Email" ], "ids": [ "Gd2f81008cf07d2425418f7f02a3ca63a8bc82003", "not-a-blob" ] }, "R1" ] Response: [ "Blob/lookup", { "list": [ { "id": "Gd2f81008cf07d2425418f7f02a3ca63a8bc82003", "matchedIds": { "Mailbox": [ "M54e97373", "Mcbe6b662" ], "Thread": [ "T1530616e" ], "Email": [ "E16e70a73eb4", "E84b0930cf16" ] } } ], "notFound": [ "not-a-blob" ] }, "R1" ]¶
JSON parsers are not all consistent in handling non-UTF-8 data. JMAP requires
that all JSON data be UTF-8 encoded, so servers MUST only return a null value
if data:asText
is requested for a range of octets which is not valid UTF-8,
and set isEncodingProblem: true
.¶
Servers MUST apply any access controls, such that if the authenticated user would be unable to discover the blobId by making queries, then this fact can't be discovered via a Blob/lookup. For example, if an Email exists in a Mailbox which the authenticated user does not have access to see, then that emailId MUST not be returned in a lookup for a blob which is referenced by that email.¶
If a server might sometimes return all names empty rather than putting a blobId in the notFound response to a Blob/get, then the server SHOULD always return the same type of response, regardless of whether a blob exists but the user can't access it, or doesn't exist at all. This avoids leaking information about the existence of the blob.¶
The server MUST NOT trust that the data given to a Blob/set is a well formed instance of the specified media type, and if the server attempts to parse the given blob, only hardened parsers designed to deal with arbitrary untrusted data should be used. The server SHOULD NOT reject data on the grounds that it is not a valid specimen of the stated type.¶
Blob/set catenate can be used to recreate dangerous content on the far side of security scanners (anti-virus or exfiltration scanners for example) which may be watching the upload endpoint. Server implementations SHOULD provide a hook to allow security scanners to check the resulting blobId from a catenate in the same way that they do for the upload endpoint.¶
IANA is requested to register the "blob" JMAP Capability as follows:¶
Capability Name: urn:ietf:params:jmap:blob¶
Specification document: this document¶
Intended use: common¶
Change Controller: IETF¶
Security and privacy considerations: this document, Section XXX¶
IANA is requested to register the "unknownDataType" JMAP Error Code as follows:¶
JMAP Error Code: unknownDataType¶
Intended use: common¶
Change Controller: IETF¶
Reference: this document¶
Description: The server does not recognise this data type, or the capability to enable it was not present.¶
IANA is requested to create a new registry "JMAP Data Types" with the initial content:¶
Type Name | Can reference blobs | Can use for state change | Capability | Reference |
---|---|---|---|---|
Core | No | No | urn:ietf:params:jmap:core | [RFC8620] |
PushSubscription | No | No | urn:ietf:params:jmap:core | [RFC8620] |
Mailbox | Yes | Yes | urn:ietf:params:jmap:mail | [RFC8621] |
Thread | Yes | Yes | urn:ietf:params:jmap:mail | [RFC8621] |
Yes | Yes | urn:ietf:params:jmap:mail | [RFC8621] | |
EmailDelivery | No | Yes | urn:ietf:params:jmap:mail | [RFC8621] |
SearchSnippet | No | No | urn:ietf:params:jmap:mail | [RFC8621] |
Identity | No | Yes | urn:ietf:params:jmap:submission | [RFC8621] |
EmailSubmission | No | Yes | urn:ietf:params:jmap:submission | [RFC8621] |
VacationResponse | No | Yes | urn:ietf:params:jmap:vacationresponse | [RFC8621] |
MDN | No | No | urn:ietf:params:jmap:mdn | [RFC9007] |
EDITOR: please remove this section before publication.¶
The source of this document exists on github at: https://github.com/brong/draft-gondwana-jmap-blob/¶
draft-ietf-jmap-blob-08:¶
Last call review (thanks Jim Fenton)¶
draft-ietf-jmap-blob-07:¶
draft-ietf-jmap-blob-06:¶
draft-ietf-jmap-blob-05:¶
typeNames
and matchedIds
anywhere except the
updates section, oops!¶
draft-ieft-jmap-blob-04:¶
catenate
results¶
draft-ieft-jmap-blob-03:¶
draft-ieft-jmap-blob-02:¶
draft-ieft-jmap-blob-01:¶
draft-ieft-jmap-blob-00:¶
draft-gondwana-jmap-blob-02¶
draft-gondwana-jmap-blob-01¶
draft-gondwana-jmap-blob-00¶
Joris Baum, Jim Fenton, Neil Jenkins, Alexey Melnikov, Ken Murchison, Robert Stepanek and the JMAP working group at the IETF.¶