A Secure Protocol for Desktop Web Servers

Abstract

This specification describes a method for applications to receive event notifications using HTTP. An example application of this specification is a news wire service that notifies news publishers when stories have become available.

In terms of transactions, the traditional request/response pair is extended with a third element, a notification. The notification requires a role reversal between client and server, in that it is emitted by the same party that received the original request and received by the same party that emitted the original request. This is done by having the client run a server.

Having clients run servers is a large change in existing HTTP semantics. HTTP expressions are about exposed resources on the server; they leave resources on the client completely opaque. The notification technique attempts to stay close to those semantics -- that is, to avoid exposing resources on the client -- by limiting exposed resources on the client to meanings arranged in a previous request/response. A notification, as described here, is nothing but a flag that an event defined during a previous request/response transaction has occurred.

A callback signifies that an event has occurred which the consumer has previously expressed interest in. There is exactly one such callback notification from server to client for each request for a notification from client to server. The notification does not say what happened, it does not carry any information about the server address to call back to, it does not say whether the event was an error or a success.

Everything of interest in the system is a HTTP resource -- there is a subscription resource, a watched resource, and a notification resource. Events are defined as changes in state of a watched resource. A subscription resource knows how to detect a state change and how to alert interested parties. A notification resource is a URL on the client that understands alerts to mean that the watched resource has a new state.

The use of this technique allows us to rely on existing web technology to the greatest extent possible, and as a result minimizes the security, design and implementation challenges of making HTTP relationships newly symmetrical.

Purpose

This document is part of a broad effort to develop asynchronous protocols and decentralized architectures. It describes a technique to handle just one problem -- event notifications -- in the hope that a really good job on a relatively approachable task will provide reusable solutions to harder problems such as store-and-forward and updates pushed from server to client. A solution to event notifications can, hopefully, be used as a wedge to answer the two broad questions:

1. How can HTTP be used for asynchronous messaging?
2. How can HTTP clients and servers be combined?

It is not a purpose of this document to provide normative requirements. The language of standards is used only because it is a good tool for describing protocols. This memo defines an experimental protocol for the Internet community. This memo does not specify an Internet standard of any kind. Discussion and suggestions for improvement are requested. Distribution of this memo is unlimited.

Format

This document is structured as a recipe -- a clear, unambiguous and carefully debugged set of instructions for building notification services. In developing concise instructions a number of more abstract issues came up, such as the security implication of bundling servers with clients; these are developed in prose subsections.

A number of useful features were left out because they required subscriptions that persisted beyond a single transaction. This version of this document does not support any kind of subscription management, not because there is no need, but because subscription management capabilities of any kind dramatically increase the level of complexity. The technique described in this document does not support any ongoing relationship between sink and source beyond a single subscription and any following notification.

The technique described in this document uses mainly HTTP header fields to communicate input and output parameters, as well as HTTP status codes. The primary benefit is to leave the entity body open for future extension and to avoid creating a set of standardized URIs (for example the use of {$root_url}/request_parameter_name to communicate success or failure).

Terminology

Application Status Code: A status code pertaining to application-level status. Distinct from HTTP status code.

Checkpoint: A specific point in the stream of state changes of the watched resource.

Event: A change in the state of the watched resource. A stream of events is a stream of distinguishable states.

MUST / SHOULD / MAY / MUST NOT / SHOULD NOT / MAY NOT: The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Notification Resource: The resource identified by the Notification-URI at the sink.

Resource: A network data object or service that can be identified by a URI, as defined in section 3.2. Resources may be available in multiple representations (e.g. multiple languages, data formats, size, and resolutions) or vary in other ways. See RFC 2396.

Sink: When a server performs a callback to a client that has expressed interest in an event, the client is acting as an event sink. This language is used because having servers and clients use callbacks from server to client causes a role reversal, and hence the term "client" is ambiguous. In the case of this specification the term "sink" is equivalent to "the original client."

Source: When a server performs a callback to a client that has expressed interest in an event, the server is acting as an event source. This language is used because having servers and clients use callbacks from server to client causes a role reversal, and hence the term "server" is ambiguous. In the case of this specification the term "source" is equivalent to "the original server."

Subscription Resource: The server resource for creating new subscriptions.

Subscription Management Resource: The server resource for managing existing subscriptions.

Watched Resource: The sink is interested in changes in the watched resource; the source tracks changes in the watched resource. The resource itself is known by its URI. The watched resource may or may not have a direct relationship with the subscription resource.

Usecases

1. Push notifications improve on POP3 and IMAP by reducing the lag time between polling cycles and at the same time reduce the server overhead of processing unnecessary polls.
2. Push notifications improve on spidering search engines by reducing the lag time between polling cycles and at the same time reduce the server overhead of processing unnecessary polls.
3. Push notifications allow a client to watch many more resources than possible with polling loops, because all the polling loops that would not have resulted in finding a new state are eliminated.
4. Push notifications improve the scalability of messaging servers such as Microsoft Exchange by granting the server control over the scheduling of client requests. While polling clients decide for themselves when they will contact the messaging server, and can accidentally flood the server by polling at the same time, a notifying server can perform load balancing by spreading notifications over time.
5. Push notifications enable peer-to-peer applications to implement semi-symetrical relationships, as happens naturally when clients and servers are joined, but retain the benefits of the HTTP protocol.
6. Idempotent notifications as defined here allow the client greater security than with push updates, and allow the client to continue using existing web infrastructure to fetch updates.

Step 1. Client to server: request a subscription to changes in the watched resource.

• Method: Client MUST use the POST method to create a subscription. The server MUST return 405 (Method Not Allowed) for any other method, along with an appropriate Allow header -- "Allow: POST".
• Input parameters:
  1. Request header: Notification-URI.
     • MUST be a legal target URI for a Step 4 (Notification).
     • MAY be a non-HTTP URI, such as a mailto: URI. (See Appendix: Email Binding).
     • MUST be a different URI for every Step 1 (Subscription). (See Threats to the Client/Sink and Race Conditions). This is partly for security reasons, in that the callback URI is a secret that has been disclosed once it has been transmitted in the clear. It is also to correct race conditions, in that the callback URI may be invoked in a Step 4 (Notification) while a Step 1 (Subscription) and Step 2 (Fetch) are ongoing.
  2. Request header: Watched-Resource-URI
     • This URI is the target of the Step 2 (Fetch), and is contacted out-of-band by the watch resource. While this URI might the same as will be used in Step 2 (Fetch) by the client, there is no relationship assumed between the two.
     • MAY be located on a server external to the Server/Source.
• Output parameters:
  1. Response code.
     
     • If the subscription request is successful then the server MUST return 202 Accepted and set the Notification-Expiration response header. If the subscription request is unsuccessful then the server SHOULD return a 4xx status code as defined in RFC 2616.
  2. Response header: Resource-Status-Code. (See Application Status Codes.
  3. Response header: Notification-Expiration. This is an advisory message of how long the server is likely to maintain the subscription. The purpose is to allow the client to distinguish between a lack of events and a lapsed subscription.
• Security considerations: See Threats to the Server/Source
• Race conditions: The subscription resource MUST checkpoint the stream of events upon reception of the Step 1 (Subscription). (See Race Conditions).

Step 2. Client to watched resource: request the current state of the resource.

• Method: (dependent on watched resource)
• Input parameters: (dependent on watched resource)
• Output parameters: (dependent on watched resource)
• Security considerations: See Threats to the Watched Resource
• Race conditions: A client which intends to create a new subscription MUST, on receipt of a Step 4 (Notification), perform the follow-up Step 1 (Subscription) before performing the Step 2 (Fetch). (See Race Conditions).

Step 3. Time passes.

• Note: This phase begins when the server checkpoints the current state of the watched resource and ends when the source initiates a Step 4 (Notification).

Step 4. Source to sink: send notification that the watched resource has changed.

• Method: Source MUST use the PUT method..
• Input parameters: (none)
• Output parameters:
  Response code 200 OK.
  
• Security considerations: URI MUST change for each Step 1 (Subscription) and MUST NOT be easily guessable. (See Security Considerations).
• Race conditions: Because of TCP unreliability, it is possible that a Step 4 (Notification) might be received as a result of a Step 1 (Subscription) from which the response had not yet arrived.

Step 5. Repeat.

---

Design Notes

Race Conditions

Events that happen between the time a client initiates a fetch and the time the fetch is complete might fail to cause a usable notification to be received on the client. This is only a problem for a client which intends to continue tracking changes in the watched resource after the initial notification; other clients can ignore the following.

After reception of a Step 4 (Notification), the client MUST perform a Step 1 (Subscription) before the Step 2 (Fetch). The purpose is to allow the server to checkpoint the stream of state changes of the watched resource, and therefore to know whether there are any states which have not caused a Step 4 (Notification).

Third Party Watchers

It is permissible for an event source that receives Step 1 (Subscription) requests and emits Step 4 (Notification) requests to be a third party with regard to the watched resource. One example is a spider that notices state changes and emits notifications to interested parties. Another example is a sink which receives notifications directly from the owner of the watched resource and passes them on, switching roles to act as a source. Another example is a CGI script on the same host and under the same ownership as the watched resource, but which has no privileged relationship with the watched resource.

Though the Step 1 (Subscription) MAY have side effects on the Step 2 (Fetch), the client MUST NOT assume a relationship between the subscription resource and the watched resource. As far as the client is concerned, the subscription resource and the watched resource are always third parties.

Polling for document changes is not a primary usecase of the technique described in this document. It is accounted for only because it is inevitable.

Multistate Resources

Watched resources will pass through a series of states, and watchers that use the technique described in this document will often be interested in all of those states. However, accumulating a collection of states into a single coherent resource is application defined. There is no way to do this generically, and the technique described here does not attempt it. Rather, this document takes the position that existing tools for accumulating states merely need to be enabled.

CVS, for example, is a tool for accumulating states. It manages successive states of a collection of source files. It presents a view of the current state of the collection, recordings of previous states of the collection, and the ability to branch states. Using the language of this document, the collection is a watched resource and the entire version history is a multistate resource. It is evident from the complexity of CVS that the business of managing versions is a difficult and ambitious task that deserves dedicated tools to be done adequately.

Email infrastructure, including SMTP, POP3, and mailbox files, is another tool for accumulating and managing stateful resources. An email message is a new resource. A mailbox file is a container for resources. POP3 is a tool for navigating a resource container. The relevant similarity between email, CVS and the concerns of this document is that every email message is a write-once resource, every version of a file in CVS is a write-once resource, and every state of a watched resource is a write-once resource. In each case there is an application-defined wrapper around a collection of write-once states.

The technique described in this document is for managing a single state resource. The relevant state of the resource is the "current" one. Step 1 (Subscription) marks the current state of the resource. That mark in the current state of the resource is a "checkpoint". A checkpoint remains the definition of "current" until the next instance of a Step 1 (Subscription). Step 2 (Fetch) may be relative to that checkpoint (if there are side effects of the POST on the watched resource), which is a supporting reason that Step 1 (Subscription) must always precede, not follow, Step 2 (Fetch).

The ability to accumulate a bunch of states in a coherent way is left to specialist applications like CVS and email. This document aims to enable, not replace, applications like CVS and email.

---

Error Handling

Offline Error Handling

Errors may occur as a result of Step 1 (Subscription) that are not foreseeable while it is ongoing. However, managing them would require a form of subscription management, because the source needs to cache them for later pickup. Therefore the technique described in this document simply lets all such errors fall on the floor. If the sink detects a problem with the watched resource, it MUST consider it a state change like any other and emit a Step 4 (Notification) according to normal procedure.

The worst such offline error is that a notification resource would be invalid, for example if it and the subscription resource were separated by a firewall. It is expected that the sink will guarantee the integrity of the callback URI before creating the subscription. This may be handled by having the sink call a web service that tests the sink IP. If the web service were hosted by the notification source then it would be able to determine with certainty whether the sink callback was reachable from its perspective. Such a web service is fairly trivial to imagine but is not within the scope of this document.

For the source, the integrity of the callback must be assumed. Because it is not possible for a source to know whether an unreachable sink has gone missing permanently or temporarily, a source is obligated to maintain the subscription until the expiration date stated in the Step 1 (Subscription) Subscription-Expiration response header.

Application Status Codes

It is not yet clear whether the subscription resource should return application-specific status codes as extensions to the existing RFC 2616 codes or as separate response entity elements. The following is a strawman definition using a separate header in the response.

[PROPOSED]

An application can return status codes in response headers titled "Resource-Status-Code." There can be 0 or more such response headers. The value of a Resource-Status-Code field MUST contain a numeric status code, a space, and a textual explanation. Whether the subscription request was a success or failure depends on the specific Resource-Status-Code; if a failure the sink MUST NOT create the notification resource.

Example response message:

HTTP/1.1 400 Bad Request
Resource-Status-Code: 1.0 CALLBACK URI SYNTAX

Resource-Status-Code values:

Numeric code	Reason phrase	Explanation
1.0	CALLBACK URI SYNTAX	The server knows in advance that it cannot parse the callback URI, for example it might be a malformed HTTP URL. This would be thrown by the subscription resource to flag an error in the ongoing request. The sink MUST NOT create the notification resource.
1.1	CALLBACK URI UNREACHABLE	The server knows in advance that it cannot establish a network connection to the callback URI, for example because of a firewall. This would be thrown by the subscription resource to flag an error in the ongoing request. The sink MUST NOT create the notification resource.
1.2	CALLBACK URI UNSUPPORTED	The server can parse the callback URI but not support it fully, for example it recognizes a mailto URI but does not know how to support it. This would be thrown by the subscription resource to flag an error in the ongoing request. The sink MUST NOT create the notification resource.

---

Security Considerations

Threats to the Client/Sink

There are a large number of potential security holes created by having clients become servers. The design used in this specification attempts to meet the following bar: a sink should be no less secure than if it were strictly a client. This is done by implementing as little sink functionality as possible within the callback, the Step 4 (Notification). Instead, as much as possible is done during the Step 1 (Subscription), where the client is no less secure than it ever was.

Attackers are prevented from altering the meaning of a Step 4 (Notification) by the following means: the only data transmitted in the Step 4 (Notification) is the fact that a pre-arranged callback URI has been activated. The meaning of that URI is left entirely up to the sink, which generates it, and to any information exchanged during a Step 1 (Subscription).

Attackers are prevented from tricking a sink into believing that a Step 4 (Notification) has happened by the following means: the callback URI itself is a secret until the source invokes it in the Step 4 (Notification), the callback URI MUST be hard to guess (for example, by being a 50 character random string), and the callback URI is never used more than once.

Attackers are prevented from creating buffer overflows by the following means: the only HTTP server functionality which a sink MUST implement is the ability to read a PUT request at a valid URI. A sink MAY disconnect a request to any invalid URI, which can be detected within the first line of a request, and a sink MAY limit the maximum length of the Request-line. As a result attackers must detect and block Step 4 (Notification) requests before they can attempt a buffer overflow attack, and they must do so before any retry following an alternate path reaches the client.

(A fringe benefit of this approach is that it is presently easier to use client/server interactions, which are well-known and mature, than source/sink interactions, which are brand new and immature .)

Threats to the Server/Source

A unique threat to the server caused by the technique described within this document is that it may be tricked into accessing resources that are off limits. For example a watched resource may have posted legal notices that it intends to sue anybody that accesses it, regardless of whether such a suit is right or winnable. This is only a best-guess means for the server to protect itself. This is not legal advice. The author is a programmer, not a lawyer. Comments from legal professionals would be welcome. The server is protected by:

• Enabling a watched resource to post such a notice in a machine-readable way, as described in the proposals in Threats to the Watched Resource. The server may demonstrate due diligence by respecting such notices.
• Never using non-idempotent methods to access a watched resource. It is apparent that the server MUST NOT use the PUT, POST or DELETE methods, which are assumed to have side effects.
• Minimizing load on the watched resource. The server MUST perform any polling loop using the HEAD method unless out-of-band communications indicate that GET is necessary and acceptable.

Threats to the Watched Resource

If the subscription resource executes the watch using a polling loop, it may generate more traffic at the watched resource than it would otherwise incur. This is less of a problem than you might expect, though. When clients use the subscription resource instead of performing a poll themselves, the subscription resource is aggregating their requests and thus reducing the total number of pollers.

Enabling Bot Exclusion

Polling is not a primary usecase for the technique described in the document, but it is inevitable, and some resources should not polled. A watched resource may be a GET query that improperly has side effects, it may have business reasons to block polling, it may be a very large query that was inadvertently exposed to the public.

A Standard for Robot Exclusion exists for similar reasons, but is not quite applicable to watchers. It exists to block bots that retrieve the same files repeatedly, as a watcher is supposed to do, or that block parts of web servers that aren't suitable for spidering (such as very deep virtual trees and duplicated information) but that are suitable for watching.

If threats to watched resources create a sufficient need, a file with the same format as the Robot Exclusion Standard, but with a new name, is a simple solution. An appropropriate new name is "watchers.txt."

To reduce load on the watcher caused by polling, a watcher which performs a polling loop MUST use the HEAD method rather than the GET method.

Enabling Bot Inclusion

A watched resource may not object to being watched, but may prefer to use a specific watcher with which it enjoys a privileged relationship. For example, a local watcher may do polling via a filesystem stat(...) call to avoid the overhead of web transactions, or it may be attached to a database trigger. Potential approaches to this extension are:

1. Watched resources MAY return a "Subscription-Resource" header to identify a resource capable of acting as a server to a Step 1 (Subscription) as defined in this document. Advantages: may be implemented on either a server-wide basis as a server extension or on a very granular basis as a per-resource extension.
2. Watched resources MAY implement a file that follows the syntax of robots.txt but has a new name, for example "watchers.txt", and that implements the following extension: "Subscription-Resource: URI". A watcher that finds such an entry in the watchers.txt file MUST chain to it as a client to a Step 1 (Subscription).

Finally, a watcher MUST respect the value of any Retry-After header returned by the watched resource, regardless of whether the watched resource returns a 503 or 3xx status code.

The discussion in this design note is somewhat irrelevant until the rest of the mechanisms work well and have reached some amount of adoption, and thus can be safely left to a late state of development.

Threats to Third Parties

Since the technique described in this document induces bots at the source and sink to emit queries, it is conceivable that bots could be manipulated into participating in a distributed denial of service (DDoS) attack against a third party.

An attacker could modify any of the URIs transmitted and later invoked, such as the sink callback URI or the watched resource URI. However, the one-shot nature of event transmissions as described in this document means that the attacker would have to intercept and modify one message for every message emitted in the DDoS attack, which nullifies the attacker's advantage. (Thanks to Paul Prscod (private communication) for this point.)

An attacker could initiate a large number of watches against a target. If the attacker used a single watcher, then the watcher would actually aggregate all the queries into a single one and in effect reduce the number of queries received by the target.

---

Appendix: Email Binding

The format of the Step 4 (Notification) is simple enough to be easily adapted to email, and difficulties in establishing HTTP back channels to edge nodes make this a very tempting approach. Though email is a highly flawed medium for application messaging it is likely to be inevitable, and so a simple binding outlined here may help keep implementions in the best possible condition.

A mailto URI MAY be passed to the Step 1 (Subscription) resource as the sink callback URI. Such a URI MUST be compliant with RFC 2368 (The mailto URL scheme). A source MUST assume that a sent Step 4 (Notification) message emitted via email has not been received unless it receives a return receipt compliant with RFC 2298 (Message Disposition Notifications).

The sink is responsible for generating a mailto URI that is compliant with the other requirements of this document, for example by creating a unique and difficult to guess identifier for each request to the Step 1 (Subscription) resource.

It is expected that a mailto sink URI will be received by a bot listener, not a human user agent.

DSNs and MDNs are defined in the following IETF RFCs:

• RFC1891 - SMTP Service Extension for Delivery Status Notifications
• RFC1892 - The Multipart/Report Content Type for the Reporting of Mail System Administrative Messages
• RFC1893 - Enhanced Mail System Status Codes
• RFC1894 - An Extensible Message Format for Delivery Status Notifications
• RFC2298 - An Extensible Message Format for Message Disposition Notifications

---

Acknowledgments and Related Work

The architectural style underlying this document is a combination of Event-Based Integration and Representational State Transfer.

This document has benefitted from conversation on the rest-discuss  mailing list, specifically including comments by Jeff Bone, Mark Baker, Paul Prescod, and Mike Dierken. Related background includes the formal documents RFC 2616, Roy Fielding's dissertation, and Notification for Proxy Caches; as well as the informal documents aREST -- Asynchronous REST, Yet Another HTTP Event Architecture (YAHEA), Http Events, the REST Wiki, HttpEvents as currently contemplated are extraneous?, and a proposal for secure notifications.

The rssCloud interface is very similar to the protocol described here. On a superficial level it is different in that it uses XML-RPC and only allows notifications for RSS files. On a design level the difference is that the notification carries metadata about which of a group of watched resources has changed state. In the language used in this document, rssCloud is an update protocol. The notification protocol described here is intended as a basis for a yet to be defined update protocol.

---

Adminstrative Information

Author: Lucas Gonze.
Contact: lucas@gonze.com
Document created: February 8, 2002.
Last update: September 26, 2003.
Address of this document on the web: http://www.gonze.com/http-notifications.html