Observing Resources in CoAP
draft-hartke-coap-observe-00

Abstract

The state of a resource can change over time. We want to give clients of the CoRE WG CoAP protocol the ability to observe this change. This short I-D provides an example design for such an addition to CoAP, in order to be able to discuss the design alternatives in specific terms.

Status of this Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

This Internet-Draft will expire on December 23, 2010.

Copyright Notice

Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

1.  Introduction
    1.1.  Design Pattern
    1.2.  Architecture
2.  Requirements
3.  The Design
    3.1.  State
    3.2.  Subscription Lifetime
    3.3.  Messages
        3.3.1.  Requests
        3.3.2.  Replies
        3.3.3.  Notifications
4.  Message Exchanges
    4.1.  Subscribing to a resource
    4.2.  Notifying of state changes
    4.3.  Unsubscribing from a resource
    4.4.  Retrieving resource state
    4.5.  Changing resource state
    4.6.  Deleting a resource
5.  Caching
6.  Identifying notifications and subscriptions
7.  Open issues
8.  References
    8.1.  Normative References
    8.2.  Informative References
§  Authors' Addresses

1.  Introduction

The state of a resource can change over time. We want to give CoAP [I‑D.ietf‑core‑coap] (Shelby, Z., Frank, B., and D. Sturek, “Constrained Application Protocol (CoAP),” June 2010.) clients the ability to observe this change.

This short I-D provides an example design for such an addition to CoAP, in order to be able to discuss the design alternatives in specific terms.

1.1.  Design Pattern

Many designs are possible for the observe capability of CoAP. So that we don’t end up with a random, arbitrary design, we base our considerations on the well-known subject/observer design pattern. In this pattern, an object, called the subject, maintains a list of interested parties, called observers, and notifies them automatically of any state changes.

There are a number of variants of that design pattern. We like one that explicitly considers the way the evolution of the resource state might end [DUAL] (Meijer, E., “Subject/Observer is Dual to Iterator,” June 2010.). In detail, this variant of the design pattern consists of the following elements:

• A subject, which sends notifications to observers. It has a single method, SUBSCRIBE, which is called by observers that wish to receive notifications from the subject.
• An observer, which receives notifications from a subject. It has three methods: YIELD, which supplies the observer with new or current information; THROW, which informs the observer that the subject experienced an error condition; and BREAK, which indicates that the subject has finished sending notifications. The grammar of notifications to be expected over time therefore is:

YIELD* ( BREAK | THROW )?

• A subscription, which represents the interest of an observer in a subject. It has a single method, UNSUBSCRIBE, which enables the subject to unsubscribe observers when notification has finished. Observers receive a reference to the subscription from the SUBSCRIBE method, so they can also call the UNSUBSCRIBE method to unsubscribe before the subject has finished sending notifications.

In this document, we describe an architecture and a protocol design that realizes a meaningful rendition of this design pattern within the REST-based [REST] (Fielding, R., “Architectural Styles and the Design of Network-based Software Architectures,” 2000.) environment of CoAP.

1.2.  Architecture

We interpret resources as the subjects of the subject/observer pattern. The subscription causes the subject to continuously supply an observer with the state of the resource: once upon subscription and then whenever the state of the resource changes. We call a CoAP node offering a resource server, and a CoAP node subscribing an observer to a resource client. As with the existing REST methods, this architecture is about exchanging representations of resources, not about the messages (or method calls).

The design is REST-based, as it maintains REST’s uniform interface and its four interface constraints ([REST] (Fielding, R., “Architectural Styles and the Design of Network-based Software Architectures,” 2000.), section 5.1.5):

• Identification of resources by the uniform mechanism of URIs [RFC3986] (Berners-Lee, T., Fielding, R., and L. Masinter, “Uniform Resource Identifier (URI): Generic Syntax,” January 2005.);
• Resource manipulation through the exchange of resource state representations;
• Self-describing messages (potentially with multiple representation formats);
• Hypermedia as the engine of application state: A server premediates application state transitions by providing links in resources.;

While this is not a necessary implication of the design described here, the assumption is that the subscription itself is modeled on the level of conversation state, not as a resource by itself. (This, of course, does not imply that there can’t be resources that help nodes decide whether to create subscriptions, but these are application-specific and outside the scope of CoAP.)

2.  Requirements

The requirements for implementing the subject/observer design pattern over UDP stem largely from the the fact that UDP is an unreliable, connectionless transport. This means that method calls must be expressed as messages, that preparation must be taken for the case that messages arrive out of order, appear duplicated, or go missing without notice, and that the transport keeps no state between messages that can be utilized.

The detailed requirements that follow from this are:

• An invocation of the SUBSCRIBE method on an observable resource is implemented by sending a message (a subscription request) from the subscribing client to the server that offers the resource.
  
  The client must be able to determine if a subscription request was received by the server, and, if not, must be able to retransmit the request.
  
  The server must acknowledge the subscription request, and must be prepared to receive duplicated subscription requests.
  
  Since subscribing can be made idempotent (Section 6 (Identifying notifications and subscriptions)), the server need not be able to detect a duplicated subscription request as such.
  
  The client must be able to relate the acknowledgement to the subscription request.
• The usual considerations for retrieving the representation of a resource in a REST-based protocol apply, e.g.:
  
  A subscribing client must be able to influence the representation format in which the server supplies the resource state to the client.
• Upon subscription, an observer must be supplied with the current state of the resource in the requested format.
  
  If the observer cannot be supplied with the current state (for example, because the resource does not exist, the state cannot be represented in the requested format, the client lacks authorization, or a general error condition occurred), the server must inform the client of the error condition.
  
  For efficiency, this initial notification may be sent within the same datagram that acknowledges the subscription request.
  
  The initial notification might not include the actual representation (e.g., take the form of a “304 not modified”) if the client already has a valid representation of the resource. The client should be able to supply information (e.g., Etags or a modification date) to enable the server to make this determination.
• To take advantage of the multicast capabilities of the transport, it should be possible to subscribe a UDP multicast group to a resource. In contrast to subscribing multiple clients individually to the resource, the server in this case must treat the multicast group as a single observer.
• For robustness, a subscription has to be maintained through periodic refreshing. If a subscription is not refreshed, its lifetime must end after a certain duration that is negotiated as part of the message exchange that implements the SUBSCRIBE method call.
  
  So a subscribing client must be able to specify a subscription lifetime duration in a subscription request. A server must be able to return the negotiated subscription lifetime duration back to the client.
  
  Since the client is responsible for taking care of the subscription, refreshing a subscription must be implemented by sending a message (a subscription refresh request) from the subscribed client to the server.
  
  The client must be able to determine if a subscription refresh request was received by the server, and, if not, must be able to retransmit the request.
  
  The server must acknowledge the request. It must be prepared to receive duplicated requests. It must be able to relate a refresh request to a subscription.
  
  The client must be able to relate the acknowledgement to the subscription refresh request.
  
  The server must be prepared for a refresh request to arrive after the subscription expired. In this case, the subscription refresh request is treated the same as a subscription request, since the client expressed the desire to continue being subscribed to the resource.
• For notifications, the equivalent of an invocation of the YIELD, BREAK or THROW method on an observer must be implemented by sending a notification message from the server offering the resource to the subscribed client.
  
  The subscribed client must be able to relate a notification message to a subscription and/or to a resource (see also Section 6 (Identifying notifications and subscriptions)).
  
  It is not a requirement that a subscribed client receives every single notification messages sent by a server, or that the server sends a notification message for every single state change. However, the state observed by an observer must eventually become consistent with the actual state of the observed resource.
• The representation format used during the lifetime of a subscription must not change. If the server is unable to continue notifying a client in the requested representation format, it must invoke the THROW method on the observer.
• A server must not send any further notification messages after sending a notification message that denotes a BREAK or THROW.
  
  However, a client must be prepared to receive notification messages after receiving such a notification message. (In this case, it discards the excessive notification messages.)
• For robustness, a server can request the acknowledgement of a notification message from a client. (For example, in order to check if the client is still there, or to make sure that an observer observes a particular resource state.) Such a notification is called a confirmable notification message.
  
  A server must be able to determine if a confirmable notification message was received by the client, and, if not, must retransmit the message.
  
  If the client cannot relate the confirmable notification message to a subscription, it must reject the message. Otherwise, it must acknowledge the message.
  
  The server must be able to relate the acknowledgement or rejection to the confirmable notification message.
  
  There is no support for multicasting confirmable notification messages.
• To end a subscription before the lifetime of the subscription expires (UNSUSBCRIBE method), a message (an ‘unsubscribe’ request) can be sent from the subscribed client to the server that maintains the subscription.
  
  The client must be able to determine if an ‘unsubscribe’ request was indeed received by the server, and, if not, must be able to retransmit the request.
  
  The server must acknowledge the ‘unsubscribe’ request, and must be prepared to receive duplicated requests (which also need to be acknowledged).
  
  The client must be able to relate the acknowledgement to the request.

Note that, in a REST-based environment, all these message exchanges should also work correctly through proxies.

3.  The Design

3.1.  State

The state required on nodes as indicated by the requirements, can be summarized as follows:

• Nodes that send requests must keep track of pending requests. (For non-idempotent requests, the node that receives the request also must retain state to deduplicate requests.)
• A server offering a resource must keep track of the observers of the resource. Similarly, a proxy must keep track of the observers that observe a resource through that proxy.
• Due to the initial notification of an observer upon subscription, the server must also keep track the resource state itself. (It cannot just notify all observers when the state changes and then immediately forget the state.)
• A client subscribed to a resource may have to keep track of the subscriptions to the resource in order to be able to relate notifications to the subscription and in order to be able to refresh the subscription before the subscription lifetime ends (Section 6 (Identifying notifications and subscriptions)).
• A client (or a proxy) may optionally keep a cache of resource states.

3.2.  Subscription Lifetime

To summarize, the lifetime of a subscription begins with a subscription request, and it ends when

• the subscription lifetime expires (as defined by the subscription duration option in the subscription request),
• the client unsubscribes from the resource,
• the client rejects a request related to a subscription,
• an error condition related to a subscription occurred, or
• the observed resource has finished sending notifications.

A client can extend the lifetime of a subscription before its end by sending a subscription refresh request.

3.3.  Messages

The following message types and elements can be gathered from the requirements.

3.3.1.  Requests

Requests are messages that need to be acknowledged by the recipient. As with other request in CoAP, they are retransmitted by the sender using an exponential back-off delay until the acknowledgment is received.

The request message types as indicated by the requirements can be summarized as follows:

• A SUBSCRIBE request creates a new subscription or refreshes an existing subscription.
• An UNSUBSCRIBE request ends an existing subscription.

To put this into perspective, the existing GET, PUT, POST and DELETE messages have request semantics as well.

• A GET request retrieves a representation of the current resource state.
• A PUT request provides a new resource state in some representation format.
• A POST request creates or extends a resource.
• A DELETE request deletes a resource.

A request message includes a transaction identifier which allows the recipient to detect duplicated requests and, by inclusion of the transaction identifier in the acknowledgement to the request, enables the sender to relate the acknowledgement to the request.

3.3.2.  Replies

Replies are messages that are sent in reply to a request. They carry the aforementioned transaction identifier.

The reply messages types as indicated by the requirements can be summarized as follows:

• An ACK reply indicates an acknowledgement of a request.
• A RST reply indicates the rejection of a request.

(Responses in CoAP are replies that may carry a resource representation.)

3.3.3.  Notifications

Matching the methods of an observer, the notification message types can be summarized as follows:

• A YIELD notification supplies the subscribed client with the state of a resource in some representation.
• A BREAK notification indicates that the observed resource has finished sending notifications.
• A THROW notification informs the subscribed client of an error condition.

Each of these notification messages can be sent as a message that does not require acknowledgement, as a confirmable message that does require acknowledgement (which makes it a request), or (in case of an initial notification) piggy-backed with the ACK message that is sent in reply to the subscription request.

4.  Message Exchanges

The following message exchanges can be derived from the requirements.

4.1.  Subscribing to a resource

Client             Server                  Client             Server
   |                 |                        |                 |
   |    SUBSCRIBE    |                        |    SUBSCRIBE    |
   +---------------->|                        +---------------->|
   |                 |                        |                 |
   |    ACK+YIELD    |                        |       ACK       |
   |<----------------+           or           |<----------------+
   |                 |                        |                 |
                                              ... Time Passes ...
                                              |                 |
                                              |   YIELD [c?]    |
                                              |<----------------+
                                              |                 |
                                              |    ACK (if c)   |
                                              +---------------->|
                                              |                 |

The workflow for subscribing an observer to an observable resource or refreshing a subscription is as follows:

1. The subscribing client sends a SUBSCRIBE request to the server that is offering the observable resource. The subscription request includes the identifier of a representation format in which the notifications have to be sent, specifies a subscription lifetime duration and optionally indicates states cached by the client (by Etag or Date) and other information that might be relayed in a GET request.
2. The server creates a new subscription if no subscription exists (Section 6 (Identifying notifications and subscriptions)), and changes the lifetime duration of the subscription to the duration specified in the subscription request, or any shorter duration if it so desires. The server then acknowledges the request and indicates the actual lifetime with a ACK reply (or with a ACK+THROW, ACK+BREAK or ACK+YIELD reply as described in the next step).
3. The server performs one of the following actions:
   • If an error occurred, the server sends a THROW notification (either as THROW request or as ACK+THROW reply to the SUBSCRIBE request).
   • If the resource has finished sending notifications, the server sends a BREAK notification (either as BREAK request or as ACK+BREAK reply to the SUBSCRIBE request).
   • Otherwise, the server supplies the observer with the current resource state in the requested representation format, or indicates that the cached state is the current state (either as YIELD request or as ACK+YIELD reply to the SUBSCRIBE request).
   If the server sends a THROW, BREAK or YIELD request, the request is treated like any other notification (i.e. the client must acknowledge it if the message is marked as confirmable, etc.).
4. If the client does not receive the ACK, ACK+THROW, ACK+BREAK or ACK+YIELD reply within a certain time frame (because the request or the reply went missing), the client retransmits the SUBSCRIBE request using the same request identifier.

4.2.  Notifying of state changes

Client             Server                  Client             Server
   |                 |                        |                 |
   |      YIELD      |                        |    YIELD [c]    |
   |<----------------+           or           |<----------------+
   |                 |                        |                 |
                                              |       ACK       |
                                              +---------------->|
                                              |                 |

The workflow for notifying an observer of a state change is as follows:

1. The server performs one of the following actions:
   • If the resource experienced an error condition, the server sends a THROW request to the client.
   • If the resource has finished sending notifications, the server sends a BREAK request to the client.
   • If the state of the resource changed, the server sends a YIELD request with the current resource state in the requested representation format.
   The server may or may not mark the request sent as confirmable ([c]). The server marks a request as confirmable because it wants to check if the observer is still alive, or because there might not be another notification in the near future and the confirmation process is therefore needed to ensure eventual consistency.
2. If the request is marked as confirmable, the subscribed client performs one of the following actions:
   • If the notification cannot be related to a subscription, the client sends a RST reply.
   • Otherwise, the client sends an ACK reply.
3. If the request is marked as confirmable and the server does not receive the ACK reply or the RST reply within a given time frame, the server retransmits the request using the same transaction identifier.

(Note that it is entirely the decision of the server whether to request an acknowledgement by marking the request as confirmable – it might want to employ different strategies to determine this, e.g. based on frequency of change, management of state etc.)

4.3.  Unsubscribing from a resource

Client             Server
   |                 |
   |   UNSUBSCRIBE   |
   +---------------->|
   |                 |
   |       ACK       |
   |<----------------+
   |                 |

The workflow for unsubscribing an observer from an observed resource is as follows:

1. The subscribed client sends an UNSUBSCRIBE request to the server offering the observed resource.
2. The server changes the lifetime duration of the subscription to zero, which immediately expires the subscription. The server then acknowledges the request with a ACK reply.
3. If the client does not receive the ACK reply within a certain time frame, the client retransmits the UNSUBSCRIBE request using the same request identifier.

Note that an subscribed client can also unsubscribe by “forgetting” the subscription and subsequently replying with a RST to the next notification. (In order to allow sending that RST even for messages that are not marked as confirmable, an otherwise redundant transaction identifier is sent in all messages.)

(An UNSUBSCRIBE message might be implemented as a SUBSCRIBE message with a lifetime duration of 0, but this has some unintended consequences. See Section 6 (Identifying notifications and subscriptions).)

4.4.  Retrieving resource state

Note that a GET request matches closely the first steps of subscribing, except that the GET request does not create or update an subscription. (A GET request is likely to mark the yield as confirmable.)

The second form in Figure 4 solves the long-poll case.

Client             Server                  Client             Server
   |                 |                        |                 |
   |       GET       |                        |       GET       |
   +---------------->|                        +---------------->|
   |                 |                        |                 |
   |    ACK+YIELD    |                        |       ACK       |
   |<----------------+           or           |<----------------+
   |                 |                        |                 |
                                              ... Time Passes ...
                                              |                 |
                                              |    YIELD [c]    |
                                              |<----------------+
                                              |                 |
                                              |       ACK       |
                                              +---------------->|
                                              |                 |

4.5.  Changing resource state

Client             Server                  Client             Server
   |                 |                        |                 |
   |       PUT       |                        |       PUT       |
   +---------------->|                        +---------------->|
   |                 |                        |                 |
   |    ACK+BREAK    |                        |       ACK       |
   |<----------------+           or           |<----------------+
   |                 |                        |                 |
                                              ... Time Passes ...
                                              |                 |
                                              |    BREAK [c]    |
                                              |<----------------+
                                              |                 |
                                              |       ACK       |
                                              +---------------->|
                                              |                 |

(The motivation for the BREAK component of the response is: PUT, like GET, can experience some error condition, so we need a place in the state machine to put the THROW response; therefore we also need a BREAK response. The second case in Figure 5 is motivated by: PUT, like GET, can take some time to be processed, so we must be able to send an ACK immediately and the THROW or BREAK response later.)

Obviously, changing the state of this resource leads to notification of any observers of new state. PUT is idempotent, but if we add notifications it may be a bit surprising that a duplicated or retransmitted PUT might send notifications twice. To prevent that, the resource might check whether it is being changed to the same state it had before and not send notifications in that case.

4.6.  Deleting a resource

A DELETE request is the same as a PUT request, except that the resource is not created or its state changed, but deleted.

Client             Server
   |                 |
   |     DELETE      |
   +---------------->|
   |                 |
   |    ACK+BREAK    |
   |<----------------+
   |                 |

If a resource is observed, deleting the resource leads to an error condition of which the observers are notified with a THROW notification or THROW request.

5.  Caching

An observer may cache part or all of a resource’s state changes. This allows clients to make conditional requests: If a subscribing client has cached a particular resource state, the server offering the resource does not need to send a full notification.

• A client informs the server of its cached resource states by including Etags of cached resource states in the subscription request. This can be multiple Etags, as this enables the server to just send a “304 not modified” for a different Etag instead of a different representation if the resource switches around between multiple states. (Note that this list might be updated by a subscription refresh.)
• A server notifies a client of a change to a cached state by omitting the representation of the resource state in the notification and including a flag (“304 not modified”) that indicates that the state changed to a cached state.

Another approach to caching multiple values for a resource is to express each possible value of the resource’s representation by a reference to another (unchanging) resource. This lets a resource change states between a set of such references that then provide the actual state information. A subscribed client fetches the individual resources on demand and caches the results for future use.

6.  Identifying notifications and subscriptions

There are two ways that a notification could be related to the resource that it is about:

1. The notification could name the resource (its URI).
2. The notification could name a target that relates to a subscription, which in turn relates back to the resource.

One or both ways could be implemented in CoAP, there are not

For way 1, each notification would contain the URI of the resource. This is particularly useful for multicast messages, but could be relatively wasteful. Also, it is not entirely clear that all servers will be aware of their own authority. Apart from cached states (Section 5 (Caching)) and other information that could be part of a GET, a subscription would simply be the triple

[URI, observer transport address, lifetime]

Resubscribing (or a duplicate subscription request) for the same [URI, observer transport address] pair simply updates the lifetime; thus, the subscription operation is idempotent. Similarly, resubscribing with a lifetime of 0 will serve to delete the subscription (however, a SUBSCRIBE message will be replied to with the current state of the resource; a SUBSCRIBE with lifetime 0 is thus equivalent to a GET with the side effect of deleting the subscription).

The notification then needs to contain the URI (and might occasionally contain a remaining lifetime):

[resource representation, resource metadata, URI, lifetime?]

For way 2, the subscription request would also contain a target identifier that is to be used in every notification, identifying the specific observer object that is to receive the notification. Again apart from cached states (Section 5 (Caching)) and other information that could be part of a GET, the subscription would be the quadruple:

[URI, observer transport address, target, lifetime]

(Note that the combination of the observer transport address and the target could be expressed as another URI, with the transport address as the authority and the target as the path. Alternatively, the target could be a short byte string. By varying the transport address, i.e. the port number, the target could also made to be always empty.)

Resubscribing (or a duplicate subscription request) for the same [URI, observer transport address, target] triple simply updates the lifetime, thus, the subscription operation again is idempotent. (The same comment as with way 1 applies to SUBSCRIBE with lifetime 0.)

As the notification already implicitly contains the transport address, it would just list the target (and might occasionally contain a remaining lifetime):

[resource representation, resource metadata, target, lifetime?]

In both cases, it would be beneficial if rebooting nodes could obtain the same transport address they had before, because a resubscription after the reboot does not create additional state.

Multicast works nicely in way 1 (except that the URI may be large). In way 2, we can give each observer in a multicast group the same target name, so multicast works.

For normal subscriptions, the observer transport address can be implied from the source address of the subscription request. For multicast, there needs to be a way to explicitly indicate that transport address. (This may be beneficial for unicast, too.)

7.  Open issues

Add discussion of messages that get reordered.

Add detailed message and option formats, once the semantics are agreed.

Describe how to map this to HTTP long-polls, WebSockets, and other asynchronous forms of HTTP.

8.  References

8.1. Normative References

8.2. Informative References

Authors' Addresses