Network News Transfer Protocol (NNTP) Extension for Compression
draft-murchison-nntp-compress-01.txt

Abstract

This memo defines an extension to the Network News Transport Protocol (NNTP) to allow a connection to be effectively and efficiently compressed.

Status of this Memo

This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.

The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.

This Internet-Draft will expire on July 30, 2010.

Copyright Notice

Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the BSD License.

Table of Contents

1.  Introduction
    1.1.  Conventions Used in This Document
2.  The COMPRESS Extension
    2.1.  Advertising the COMPRESS Extension
    2.2.  COMPRESS Command
        2.2.1.  Usage
        2.2.2.  Description
        2.2.3.  Examples
3.  Compression Efficiency
4.  Augmented BNF Syntax for the COMPRESS Extension
    4.1.  Commands
    4.2.  Capability entries
    4.3.  General Non-terminals
5.  Summary of Response Codes
6.  Security Considerations
7.  IANA Considerations
    7.1.  NNTP Compression Algorithm Registry
        7.1.1.  Algorithm Name Registration Procedure
        7.1.2.  Comments on Algorithm Registrations
        7.1.3.  Change Control
    7.2.  Registration of the DEFLATE Compression Algorithm
    7.3.  Registration of the NNTP COMPRESS Extension
8.  References
    8.1.  Normative References
    8.2.  Informative References
Appendix A.  Acknowledgements
Appendix B.  Document History (to be removed by RFC Editor before publication)
    B.1.  Changes since -00
Appendix C.  Issues to be addressed
§  Author's Address

1.  Introduction

The goal of COMPRESS is to reduce the bandwidth usage of NNTP.

Compared to PPP compression [RFC1962] (Rand, D. and K. Fox, “The PPP Compression Control Protocol (CCP),” June 1996.) and modem-based compression ([MNP] (Held, G., “The Complete Modem Reference,” May 1994.) and [V42bis] (International Telecommunications Union, “Data compression procedures for data circuit-terminating equipment (DCE) using error correction procedures,” January 1990.)), COMPRESS offers greater compression efficiency. COMPRESS can be used together with Transport Layer Security (TLS) [RFC5246] (Dierks, T. and E. Rescorla, “The Transport Layer Security (TLS) Protocol Version 1.2,” August 2008.), Simple Authentication and Security Layer (SASL) encryption [RFC4422] (Melnikov, A. and K. Zeilenga, “Simple Authentication and Security Layer (SASL),” June 2006.), Virtual Private Networks (VPNs), etc.

Compared to TLS compression [RFC3749] (Hollenbeck, S., “Transport Layer Security Protocol Compression Methods,” May 2004.), NNTP COMPRESS has the following advantages:

• COMPRESS can be implemented easily both by NNTP servers and clients.
• COMPRESS benefits from an intimate knowledge of the NNTP protocol's state machine, allowing for dynamic and aggressive optimization of the underlying compression algorithm's parameters.

and the following disadvantages:

• When the TLS layer implements compression, any protocol using that layer can transparently benefit from that compression (e.g., SMTP and NNTP). COMPRESS is specific to NNTP.

In order to increase interoperability, it is desirable to have as few different compression algorithms as possible, so this document specifies only one. The DEFLATE algorithm (defined in [RFC1951] (Deutsch, P., “DEFLATE Compressed Data Format Specification version 1.3,” May 1996.)) is standard, widely available and fairly efficient, and MUST be implemented as part of this extension.

1.1.  Conventions Used in This Document

The notational conventions used in this document are the same as those in [RFC3977] (Feather, C., “Network News Transfer Protocol (NNTP),” October 2006.) and any term not defined in this document has the same meaning as in that one.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] (Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels,” March 1997.).

In the examples, commands from the client are indicated with [C], and responses from the server are indicated with [S].

2.  The COMPRESS Extension

The COMPRESS extension is used to enable data compression on an NNTP connection.

This extension provides a new COMPRESS command and has capability label COMPRESS.

2.1.  Advertising the COMPRESS Extension

A server supporting the COMPRESS command as defined in this document will advertise the "COMPRESS" capability label in response to the CAPABILITIES command ([RFC3977] (Feather, C., “Network News Transfer Protocol (NNTP),” October 2006.) Section 5.2). This capability MAY be advertised both before and after any use of the MODE READER command ([RFC3977] (Feather, C., “Network News Transfer Protocol (NNTP),” October 2006.) Section 5.3), with the same semantics.

The COMPRESS capability label contains a whitespace-separated list of available compression algorithms. This document defines one compression algorithm: DEFLATE. This algorithm is mandatory to implement and MUST be supported in order to advertise the COMPRESS extension.

Future extensions may add additional compression algorithms to this capability. Unrecognized algorithms MUST be ignored by the client.

Example:

[C] CAPABILITIES
[S] 101 Capability list:
[S] VERSION 2
[S] READER
[S] IHAVE
[S] COMPRESS DEFLATE X-SHRINK
[S] LIST ACTIVE NEWSGROUPS
[S] .

2.2.  COMPRESS Command

2.2.1.  Usage

This command MUST NOT be pipelined.

Syntax
   COMPRESS algorithm

Responses
   291 Compression active
   403 Unable to activate compression
   502 Command unavailable [1]

[1] If a compression layer is already active, COMPRESS is not a valid
    command (see Section 2.2.2).

Parameters
   algorithm = Name of compression algorithm: "DEFLATE"

2.2.2.  Description

The COMPRESS command instructs the server to use the named compression algorithm ("DEFLATE" is the only one defined in this document) for all commands and/or responses after COMPRESS.

The client MUST NOT send any further commands until it has seen the result of COMPRESS.

If the requested compression algorithm is invalid (e.g., is not supported), the server MUST reject the COMPRESS command with a 503 response ([RFC3977] (Feather, C., “Network News Transfer Protocol (NNTP),” October 2006.) Section 3.2.1). If the server is unable to activate compression for any reason (e.g., a server configuration or resource problem), the server MUST reject the COMPRESS command with a 403 response ([RFC3977] (Feather, C., “Network News Transfer Protocol (NNTP),” October 2006.) Section 3.2.1). Otherwise, the server issues a 291 response and the compression layer takes effect for both client and server immediately following the CRLF of the success reply.

Both the client and the server MUST know if there is a compression layer active. A client MUST NOT attempt to activate compression (via either the COMPRESS or STARTTLS [RFC4642] (Murchison, K., Vinocur, J., and C. Newman, “Using Transport Layer Security (TLS) with Network News Transfer Protocol (NNTP),” October 2006.) commands) if a compression layer is already active. A server MUST NOT return the COMPRESS or STARTTLS capability labels in response to a CAPABILITIES command received after a compression layer is active, and a server MUST reply with a 502 response code if a syntactically valid COMPRESS or STARTTLS command is received while a compression layer is already active.

For DEFLATE [RFC1951] (Deutsch, P., “DEFLATE Compressed Data Format Specification version 1.3,” May 1996.) (as for many other compression mechanisms), the compressor can trade speed against quality. The decompressor MUST automatically adjust to the parameters selected by the sender. Consequently, the client and server are both free to pick the best reasonable rate of compression for the data they send.

When COMPRESS is combined with TLS [RFC5246] (Dierks, T. and E. Rescorla, “The Transport Layer Security (TLS) Protocol Version 1.2,” August 2008.) or SASL [RFC4422] (Melnikov, A. and K. Zeilenga, “Simple Authentication and Security Layer (SASL),” June 2006.) security layers, the processing order of the three layers MUST be first COMPRESS, then SASL, and finally TLS. That is, before data is transmitted it is first compressed. Second, if a SASL security layer has been negotiated, the compressed data is then signed and/or encrypted accordingly. Third, if a TLS security layer has been negotiated, the data from the previous step is signed and/or encrypted accordingly. When receiving data, the processing order MUST be reversed. This ensures that before sending, data is compressed before it is encrypted, independent of the order in which the client issues the COMPRESS, AUTHINFO SASL [RFC4643] (Vinocur, J. and K. Murchison, “Network News Transfer Protocol (NNTP) Extension for Authentication,” October 2006.), and STARTTLS [RFC4642] (Murchison, K., Vinocur, J., and C. Newman, “Using Transport Layer Security (TLS) with Network News Transfer Protocol (NNTP),” October 2006.) commands.

2.2.3.  Examples

Example of layering TLS and NNTP compression:

[C] CAPABILITIES
[S] 101 Capability list:
[S] VERSION 2
[S] READER
[S] STARTTLS
[S] AUTHINFO
[S] COMPRESS DEFLATE
[S] .
[C] STARTTLS
[S] 382 Continue with TLS negotiation
[TLS negotiation without compression occurs here]
[Following successful negotiation, all traffic is encrypted]
[C] CAPABILITIES
[S] 101 Capability list:
[S] VERSION 2
[S] READER
[S] AUTHINFO USER
[S] COMPRESS DEFLATE
[S] .
[C] AUTHINFO USER fred
[S] 381 Enter passphrase
[C] AUTHINFO PASS flintstone
[S] 281 Authentication accepted
[C] COMPRESS DEFLATE
[S] 291 Compression active
[From this point on, all traffic is compresssed before being encrypted]

Example of a server failing to activate compression:

[C] CAPABILITIES
[S] 101 Capability list:
[S] VERSION 2
[S] IHAVE
[S] COMPRESS DEFLATE
[S] .
[C] COMPRESS DEFLATE
[S] 403 Unable to activate compression

Example of attempting to use an unsupported compression algorithm:

[C] CAPABILITIES
[S] 101 Capability list:
[S] VERSION 2
[S] IHAVE
[S] COMPRESS DEFLATE
[S] .
[C] COMPRESS X-SHRINK
[S] 503 Compression algorithm not supported

Examples of a server refusing to compress twice:

[C] CAPABILITIES
[S] 101 Capability list:
[S] VERSION 2
[S] IHAVE
[S] STARTTLS
[S] COMPRESS DEFLATE
[S] .
[C] STARTTLS
[S] 382 Continue with TLS negotiation
[TLS negotiation with compression occurs here]
[Following successful negotiation, all traffic is protected by TLS]
[C] CAPABILITIES
[S] 101 Capability list:
[S] VERSION 2
[S] IHAVE
[S] .
[C] COMPRESS DEFLATE
[S] 502 Compression already active via TLS

[C] CAPABILITIES
[S] 101 Capability list:
[S] VERSION 2
[S] IHAVE
[S] STARTTLS
[S] COMPRESS DEFLATE
[S] .
[C] COMPRESS DEFLATE
[S] 291 Compression active
[From this point on, all traffic is compresssed]
[C] CAPABILITIES
[S] 101 Capability list:
[S] VERSION 2
[S] IHAVE
[S] .
[C] STARTTLS
[S] 502 DEFLATE compression already active

3.  Compression Efficiency

This section is only informative.

NNTP poses some unusual problems for a compression layer.

Upstream traffic is fairly simple. Most NNTP clients send the same few commands again and again, so any compression algorithm that can exploit repetition works efficiently. The article posting and transfer commands (e.g., POST, IHAVE, and TAKETHIS [RFC4644] (Vinocur, J. and K. Murchison, “Network News Transfer Protocol (NNTP) Extension for Streaming Feeds,” October 2006.)) are an exception; clients that send many article posting or transfer commands may want to surround large multi-line data blocks with flushes in the same way as is recommended for servers later in this section.

Downstream traffic has the unusual property that several kinds of data are sent, confusing all dictionary-based compression algorithms.

One type is NNTP simple responses and NNTP multi-line responses not related to article header/body retrieval (e.g, CAPABILITIES, GROUP, NEXT, STAT, DATE, NEWNEWS, LIST, OVER, CHECK [RFC4644] (Vinocur, J. and K. Murchison, “Network News Transfer Protocol (NNTP) Extension for Streaming Feeds,” October 2006.), etc). These are highly compressible; zlib using its least CPU-intensive setting compresses typical responses to 25-40% of their original size.

Another type is article headers (as retrieved via the HEAD or ARTICLE commands). These are equally compressible, and benefit from using the same dictionary as the NNTP responses.

A third type is article body text (as retrieved via the BODY or ARTICLE commands). Text is usually fairly short and includes much ASCII, so the same compression dictionary will do a good job here, too. When multiple messages in the same thread are read at the same time, quoted lines, etc. can often be compressed almost to zero.

Finally, attachments (non-text article bodies retrieved via the BODY and ARTICLE commands) are transmitted in encoded form, usually Base64 [RFC4648] (Josefsson, S., “The Base16, Base32, and Base64 Data Encodings,” October 2006.), UUencode [IEEE.1003‑2.1992] (Institute of Electrical and Electronics Engineers, “Information Technology - Portable Operating System Interface (POSIX) - Part 2: Shell and Utilities (Vol. 1),” 1992.), or yEnc [yEnc] (Helbing, J., “yEnc - Efficient encoding for Usenet and eMail,” http://www.yenc.org.).

When attachments are retrieved, DEFLATE may be able to compress them, but the format of the attachment's encoding is usually not NNTP-like, so the dictionary built while compressing NNTP does not help. The compressor has to adapt its dictionary from NNTP to the attachment's encoding format, and then back.

When attachments are retrieved in Base64 or UUencode form, these encodings add another problem. 8-bit compression algorithms such as DEFLATE work well on 8-bit file formats, however both Base64 and UUencode transform a file into something resembling 6-bit bytes, hiding most of the 8-bit file format from the compressor.

When using the zlib library (see [RFC1951] (Deutsch, P., “DEFLATE Compressed Data Format Specification version 1.3,” May 1996.)), the functions deflateInit2(), deflate(), inflateInit2(), and inflate() suffice to implement this extension. The windowBits value must be in the range -8 to -15, or else deflateInit2() uses the wrong format. deflateParams() can be used to improve compression rate and resource use. The Z_FULL_FLUSH argument to deflate() can be used to clear the dictionary (the receiving peer does not need to do anything).

A server can improve downstream compression if it hints to the compressor that the data type is about to change strongly, e.g., by sending a Z_FULL_FLUSH at the start and end of large non-text multi-line data blocks (before and after 'content-lines' in the definition of 'multi-line-data-block' in [RFC3977] (Feather, C., “Network News Transfer Protocol (NNTP),” October 2006.) Section 9.8). Small multi-line data blocks are best left alone. A possible boundary is 5kB.

4.  Augmented BNF Syntax for the COMPRESS Extension

This section describes the syntax of the COMPRESS extension using ABNF [RFC5234] (Crocker, D. and P. Overell, “Augmented BNF for Syntax Specifications: ABNF,” January 2008.). It extends the syntax in Section 9 of [RFC3977] (Feather, C., “Network News Transfer Protocol (NNTP),” October 2006.), and non-terminals not defined in this document are defined there. The [RFC3977] (Feather, C., “Network News Transfer Protocol (NNTP),” October 2006.) ABNF should be imported first before attempting to validate these rules.

4.1.  Commands

This syntax extends the non-terminal "command", which represents an NNTP command.

command =/ compress-command

compress-command = "COMPRESS" WS algorithm

4.2.  Capability entries

This syntax extends the non-terminal "capability-entry", which represents a capability that may be advertised by the server.

capability-entry =/ compress-capability

compress-capability = "COMPRESS" *(WS algorithm)

4.3.  General Non-terminals

algorithm = "DEFLATE" / token

5.  Summary of Response Codes

This section contains a list of each new response code defined in this document and indicates whether it is multi-line, which commands can generate it, what arguments it has, and what its meaning is.

Response code 291
   Generated by: COMPRESS
   Meaning: Compression layer activated

6.  Security Considerations

As for TLS compression [RFC3749] (Hollenbeck, S., “Transport Layer Security Protocol Compression Methods,” May 2004.).

7.  IANA Considerations

7.1.  NNTP Compression Algorithm Registry

The NNTP Compression Algorithm registry will be maintained by IANA. The registry will be available at <http://www.iana.org/assignments/nntp-compression-algorithms>.

The purpose of this registry is not only to ensure uniqueness of values used to name NNTP compression algorithms, but also to provide a definitive reference to technical specifications detailing each NNTP compression algorithm available for use on the Internet.

There is no naming convention for NNTP compression algorithms; any name that conforms to the syntax of a NNTP compression algorithm name can be registered.

The procedure detailed in Section 7.1.1 (Algorithm Name Registration Procedure) is to be used for registration of a value naming a specific individual mechanism.

Comments may be included in the registry as discussed in Section 7.1.2 (Comments on Algorithm Registrations) and may be changed as discussed in Section 7.1.3 (Change Control).

7.1.1.  Algorithm Name Registration Procedure

IANA will register new NNTP compression algorithm names on a First Come First Served basis, as defined in BCP 26 [RFC5226] (Narten, T. and H. Alvestrand, “Guidelines for Writing an IANA Considerations Section in RFCs,” May 2008.). IANA has the right to reject obviously bogus registration requests, but will perform no review of claims made in the registration form.

Registration of an NNTP compression algorithm is requested by filling in the following template:

Subject: Registration of NNTP compression algorithm X

NNTP compression algorithm name:

Security considerations:

Published specification (recommended):

Contact for further information:

Intended usage: (One of COMMON, LIMITED USE, or OBSOLETE)

Owner/Change controller:

Note: (Any other information that the author deems relevant may be
       added here.)

and sending it via electronic mail to IANA at <iana@iana.org>.

While this registration procedure does not require expert review, authors of NNTP compression algorithms are encouraged to seek community review and comment whenever that is feasible. Authors may seek community review by posting a specification of their proposed mechanism as an Internet-Draft. NNTP compression algorithms intended for widespread use should be standardized through the normal IETF process, when appropriate.

7.1.2.  Comments on Algorithm Registrations

Comments on a registered NNTP compression algorithm should first be sent to the "owner" of the algorithm and/or to the <ietf-nntp@lists.eyrie.org> mailing list.

Submitters of comments may, after a reasonable attempt to contact the owner, request IANA to attach their comment to the NNTP compression algorithm registration itself by sending mail to <iana@iana.org>. At IANA's sole discretion, IANA may attach the comment to the NNTP compression algorithm's registration.

7.1.3.  Change Control

Once an NNTP compression algorithm registration has been published by IANA, the author may request a change to its definition. The change request follows the same procedure as the registration request.

The owner of an NNTP compression algorithm may pass responsibility for the algorithm to another person or agency by informing IANA; this can be done without discussion or review.

The IESG may reassign responsibility for an NNTP compression algorithm. The most common case of this will be to enable changes to be made to algorithms where the author of the registration has died, has moved out of contact, or is otherwise unable to make changes that are important to the community.

NNTP compression algorithm registrations may not be deleted; algorithms that are no longer believed appropriate for use can be declared OBSOLETE by a change to their "intended usage" field; such algorithms will be clearly marked in the lists published by IANA.

The IESG is considered to be the owner of all NNTP compression algorithms that are on the IETF standards track.

7.2.  Registration of the DEFLATE Compression Algorithm

This section gives a formal definition of the DEFLATE compression algorithm as required by Section 7.1.1 (Algorithm Name Registration Procedure) for the IANA registry.

NNTP compression algorithm name: DEFLATE

Security considerations: See Section 6 (Security Considerations) of this document

Published specification: This document

Contact for further information: Author of this document

Intended usage: COMMON

Owner/Change controller: IESG <iesg@ietf.org>.

Note: This algorithm is mandatory to implement

7.3.  Registration of the NNTP COMPRESS Extension

This section gives a formal definition of the COMPRESS extension as required by Section 3.3.3 of [RFC3977] (Feather, C., “Network News Transfer Protocol (NNTP),” October 2006.) for the IANA registry.

• The COMPRESS extension allows an NNTP connection to be effectively and efficiently compressed.
• The capability label for this extension is "COMPRESS", whose arguments list the available compression algorithms.
• This extension defines one new command, COMPRESS, whose behavior, arguments, and responses are defined in Section 2.2 (COMPRESS Command).
• This extension does not associate any new responses with pre-existing NNTP commands.
• This extension does affect the overall behavior of both server and client, in that after successful use of the COMPRESS command, all communication is transmitted in a compressed format.
• This extension does not affect the maximum length of commands or initial response lines.
• This extension does not alter pipelining, but the COMPRESS command cannot be pipelined
• Use of this extension does alter the capabilities list; once the COMPRESS command has been used successfully, the COMPRESS capability can no longer be advertised by CAPABILITIES. Additionally, the STARTTLS and MODE-READER capabilities MUST NOT be advertised after successful execution of the COMPRESS command.
• This extension does not cause any pre-existing command to produce a 401, 480, or 483 response.
• This extension is unaffected by any use of the MODE READER command, however the MODE READER command MUST NOT be used in the same session following a successful execution of the COMPRESS command.
• The STARTTLS command MUST NOT be used in the same session following a successful execution of the COMPRESS command.
• Published Specification: This document.
• Contact for Further Information: Author of this document.
• Change Controller: IESG <iesg@ietf.org>.

8.  References

8.1. Normative References

8.2. Informative References

Appendix A.  Acknowledgements

This document draws heavily on ideas in [RFC4978] (Gulbrandsen, A., “The IMAP COMPRESS Extension,” August 2007.) by Arnt Gulbrandsen and a large portion of this text was borrowed from that specification.

Special acknowledgement also goes to Julien Élie and others who commented privately on intermediate revisions of this document.

Appendix B.  Document History (to be removed by RFC Editor before publication)

B.1.  Changes since -00

• Made DEFLATE the mandatory to implement compression algorithm.
• Removed the requirement that clients/servers implementing COMPRESS also implement TLS compression.
• Added an example of a client trying to use an unsupported compression algorithm.
• Rewrote Compression Efficiency (Compression Efficiency) as follows:
  • Included a sample listing of which NNTP commands produce which type of data to be compressed.
  • Removed discussion of attachments in binary form and incompressible file formats.
  • Mentioned UUencode and yEnc encoding of attachments.
• Added IANA registry of NNTP compression algorithms.
• Miscellaneous editorial changes submitted by Julien Élie.

Appendix C.  Issues to be addressed

• What 2xx code should we use when compression is activated? 20x, 28x, 29x?
• Should we restrict the ABNF for compression algorithms?

Author's Address