IP Parcels and Advanced Jumbos

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 5 November 2023.¶

Copyright Notice

Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

1. Introduction

IP packets (both IPv4 [RFC0791] and IPv6 [RFC8200]) contain a single unit of transport layer protocol data which becomes the retransmission unit in case of loss. Transport layer protocols such as the Transmission Control Protocol (TCP) [RFC9293] and reliable transport protocol users of the User Datagram Protocol (UDP) [RFC0768] (including QUIC [RFC9000], LTP [RFC5326] and others) prepare data units known as "segments", with individual IP packets including only a single segment. This document presents a new construct known as the "IP Parcel" which permits a single packet to carry multiple transport layer protocol segments. This essentially creates a "packet-of-packets" with the full {TCP,UDP}/IP headers appearing only once but with possibly more than one segment.¶

Transport layer protocol entities form parcels by preparing a data buffer (or buffer chain) beginning with an Integrity Block of at most 256 2-octet Checksums followed by their corresponding transport layer protocol segments that can be broken out into individual packets and/or smaller sub-parcels if necessary. All segments except the final one must be equal in length and no larger than 65535 octets (minus headers), while the final segment must not be larger than the others but may be smaller. The transport layer protocol entity then delivers the buffer(s), number of segments and non-final segment size to the network layer which copies the buffer(s) into the body of a parcel then includes a {TCP,UDP} header and an IP header plus extensions that identify this as a parcel and not an ordinary packet.¶

The network layer then forwards each parcel over consecutive parcel-capable links in a path until they arrive at a router with a next hop link that does not support parcels, a parcel-capable link with a size restriction, or an ingress middlebox Overlay Multilink Network (OMNI) Interface [I-D.templin-intarea-omni] that spans intermediate Internetworks using adaptation layer encapsulation and fragmentation. In the first case, the original source or next hop router applies packetization to break the parcel into individual IP packets. In the second case, the source/router applies network layer parcellation to form smaller sub-parcels. In the final case, the OMNI interface applies adaptation layer parcellation to form smaller sub-parcels if necessary then applies adaptation layer encapsulation and fragmentation if necessary before forwarding.¶

These adaptation layer sub-parcels may then be reunified into one or more larger sub-parcels by an egress middlebox OMNI interface which either delivers them locally or forwards them over additional parcel-capable links in the network path to the final destination. The final destination can then apply network layer reunification (or restoration) to concatenate elements of the same original parcel into a single unit so as to present the largest possible number of segments to the transport layer in a single system call. Reordering and even loss or damage of individual segments within the network is therefore possible, but what matters is that the parcels delivered to the final destination's transport layer should be the largest practical size for best performance and that loss or receipt of individual segments (and not parcel size) determines the retransmission unit.¶

This document further specifies an "advanced jumbo" service that provides useful extensions beyond the "basic" IPv6 jumbogram service defined in [RFC2675]. Advanced jumbos are defined for both IP protocol versions and provide end systems and routers with a more robust service when the transmission of truly large singleton segments is necessary.¶

The following sections discuss rationale for creating and shipping IP parcels and advanced jumbos as well as actual protocol constructs and procedures involved. IP parcels and advanced jumbos provide essential building blocks for improved performance, efficiency and integrity while encouraging larger Maximum Transmission Units (MTUs). These services will further inspire future innovation in applications, transport protocols, operating systems, network equipment and data links in ways that will transform the Internet architecture.¶

2. Terminology

The Oxford Languages dictionary defines a "parcel" as "a thing or collection of things wrapped in paper in order to be carried or sent by mail". Indeed, there are many examples of parcel delivery services worldwide that provide an essential transit backbone for efficient business and consumer transactions.¶

In this same spirit, an "IP parcel" is simply a collection of at most 256 transport layer protocol segments wrapped in an efficient package for transmission and delivery (i.e., a "packet-of-packets") while a "singleton IP parcel" is simply a parcel that contains a single segment. IP parcels are distinguished from ordinary packets and jumbograms through the constructs specified in this document.¶

The IP parcel construct is defined for both IPv4 and IPv6. Where the document refers to "IPv4 header length", it means the total length of the base IPv4 header plus all included options, i.e., as determined by consulting the Internet Header Length (IHL) field. Where the document refers to "IPv6 header length", however, it means only the length of the base IPv6 header (i.e., 40 octets), while the length of any extension headers is referred to separately as the "IPv6 extension header length". Finally, the term "IP header plus extensions" refers generically to an IPv4 header plus all included options or an IPv6 header plus all included extension headers.¶

The term "advanced jumbo" refers to a new type of IP jumbogram defined for both IP protocol versions and derived from "basic" IPv6 jumbograms as defined in [RFC2675]. Advanced jumbos include a 32-bit Jumbo Payload Length field the same as for basic IPv6 jumbograms, but are differentiated from parcels and other jumbogram types by including the "Type" value '1' in the IP {Total, Payload} Length field. Advanced jumbos can be in either minimal or expanded format, with expanded format including additional Jumbo Payload option control information.¶

Where the document refers to "{TCP,UDP} header length", it means the length of either the TCP header plus options (20 or more octets) or the UDP header (8 octets). It is important to note that only a single IP header and a single full {TCP,UDP} header appears in each parcel regardless of the number of segments included. This distinction often provides a significant savings in overhead made possible only by IP parcels.¶

Where the document refers to checksum calculations, it means the standard Internet checksum unless otherwise specified. The same as for TCP [RFC9293], UDP [RFC0768] and IPv4 [RFC0791], the standard Internet checksum is defined as (sic) "the 16-bit one's complement of the one's complement sum of all (pseudo-)headers plus data, padded with zero octets at the end (if necessary) to make a multiple of two octets". A notional Internet checksum algorithm can be found in [RFC1071], while practical implementations require special attention to byte ordering "endianness" to ensure interoperability between diverse architectures.¶

The terms "application layer (L5 and higher)", "transport layer (L4)", "network layer (L3)", "(data) link layer (L2)" and "physical layer (L1)" are used consistently with common Internetworking terminology, with the understanding that reliable delivery protocol users of UDP are considered as transport layer elements. The OMNI specification further defines an "adaptation layer" logically positioned below the network layer but above the link layer (which may include physical links and Internet- or higher-layer tunnels). The adaptation layer is simply known as "the layer below L3 but above L2" and does not assign a layer number itself. A network interface is a node's attachment to a link (via L2), and an OMNI interface is therefore a node's attachment to an OMNI link (via the adaptation layer).¶

The term "parcel/jumbo-capable link/path" refers to paths that transit interfaces to adaptation layer and/or link layer media (either physical or virtual) capable of transiting {TCP,UDP}/IP packets that employ the parcel/jumbo constructs specified in this document. The source and each router in the path has a "next hop link" that forwards parcels/jumbos toward the final destination, while each router and the final destination has a "previous hop link" that accepts en route parcels/jumbos. Each next hop link must be capable of forwarding parcels/jumbos (after first applying parcellation if necessary) with segment lengths no larger than can transit the link. Currently only the OMNI link satisfies these properties, but new and existing link types are also encouraged to support parcels and advanced jumbos.¶

The term "5-tuple" refers to a transport layer protocol entity identifier that includes the network layer (source address, destination address, source port, destination port, protocol number). The term "3-tuple" refers to a network layer parcel entity identifier that includes the adaptation layer (source address, destination address, Parcel ID).¶

The term "Maximum Transmission Unit (MTU)" is widely understood in Internetworking terminology to mean the largest packet size that can transit a single link ("link MTU") or an entire path ("path MTU") without requiring network layer IP fragmentation. If the MTU value returned during parcel path qualification is larger than 65535 (plus the length of the parcel headers), it determines the maximum parcel size that can transit the link/path without requiring a router to perform packetization/parcellation. If the MTU is 65535 or smaller, the value instead determines the "Maximum Segment Size (MSS)" for the leading portion of the path up to a router that cannot forward the parcel further. (Note that this size may still be larger than the MSS that can transit the remainder of the path to the final destination, which can only be determined through additional probing.)¶

The terms "packetization" and "restoration" refer to a network layer process in which the original source or a router on the path breaks a parcel out into individual IP packets that can transit the remainder of the path without loss due to a size restriction. The final destination then restores the combined packet contents into a parcel before delivery to the transport layer. In current practice, packetization/restoration are considered to be one and the same as Generic Segmentation/Receive Offload (GSO/GRO).¶

The terms "parcellation" and "reunification" refer to either network layer or adaptation layer processes in which the original source or a router on the path breaks a parcel into smaller sub-parcels that can transit the path without loss due to a size restriction. These sub-parcels are then reunified into larger (sub-)parcels before delivery to the transport layer. As a network layer process, the sub-parcels resulting from parcellation may only be reunified at the final destination. As an adaptation layer process, the resulting sub-parcels may be first reunified at an adaptation layer egress node then possibly further reunified by the network layer of the final destination.¶

The parcel sizing variables "J", "K", "L" and "M" are cited extensively throughout the document. "J" denotes the number of non-final segments included in the parcel, "L" is the length of each non-final segment, "K" is the length of the final segment and "M" is termed the "Parcel Payload Length".¶

Automatic Extended Route Optimization (AERO) [I-D.templin-intarea-aero] and the Overlay Multilink Network Interface (OMNI) [I-D.templin-intarea-omni] provide an architectural framework for transmission of IP parcels and advanced jumbos over one or more concatenated Internetworks. AERO/OMNI will provide an operational environment for IP parcels beginning from the earliest deployment phases and extending indefinitely to accommodate continuous future growth. As more and more parcel/jumbo-capable links are deployed (e.g., in data centers, edge networks, space-domain, and other high data rate services) AERO/OMNI will continue to provide an essential service for Internetworking performance maximization.¶

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119][RFC8174] when, and only when, they appear in all capitals, as shown here.¶

3. Background and Motivation

Studies have shown that applications can improve their performance by sending and receiving larger packets due to reduced numbers of system calls and interrupts as well as larger atomic data copies between kernel and user space. Larger packets also result in reduced numbers of network device interrupts and better network utilization (e.g., due to header overhead reduction) in comparison with smaller packets.¶

A first study [QUIC] involved performance enhancement of the QUIC protocol [RFC9000] using the linux Generic Segment/Receive Offload (GSO/GRO) facility. GSO/GRO provides a robust service that has shown significant performance increases based on a multi-segment transfer capability between the operating system kernel and QUIC applications. GSO/GRO performs fragmentation and reassembly at the transport layer with the transport protocol segment size limited by the path MTU (typically 1500 octets or smaller in today's Internet).¶

A second study [I-D.templin-dtn-ltpfrag] showed that GSO/GRO also improves performance for the Licklider Transmission Protocol (LTP) [RFC5326] used for the Delay Tolerant Networking (DTN) Bundle Protocol [RFC9171] for segments larger than the actual path MTU through the use of OMNI interface encapsulation and fragmentation. Historically, the NFS protocol also saw significant performance increases using larger (single-segment) UDP datagrams even when IP fragmentation is invoked, and LTP still follows this profile today. Moreover, LTP shows this (single-segment) performance increase profile extending to the largest possible segment size which suggests that additional performance gains are possible using (multi-segment) IP parcels that approach or even exceed 65535 octets in total length.¶

TCP also benefits from larger packet sizes and efforts have investigated TCP performance using jumbograms internally with changes to the linux GSO/GRO facilities [BIG-TCP]. The approach proposed to use the Jumbo Payload option internally and to allow GSO/GRO to use buffer sizes larger than 65535 octets, but with the understanding that links that support jumbograms natively are not yet widely available. Hence, IP parcels provide a packaging that can be considered in the near term under current deployment limitations.¶

A limiting consideration for sending large packets is that they are often lost at links with MTU restrictions, and the resulting Packet Too Big (PTB) message [RFC1191][RFC8201] may be lost somewhere in the return path to the original source. This "Path MTU black hole" condition can degrade performance unless robust path probing techniques are used, however the best case performance always occurs when loss of packets due to size restrictions is minimized.¶

These considerations therefore motivate a design where transport protocols can employ segment sizes as large as 65535 octets (minus headers), while parcels that carry multiple segments may themselves be significantly larger. This would allow the receiving transport layer protocol entity to process multiple segments in parallel instead of one at a time per existing practices. Parcels therefore support improvements in performance, integrity and efficiency for the original source, final destination and networked path as a whole. This is true even if the network and lower layers need to apply packetization/restoration, parcellation/reunification and/or fragmentation/reassembly.¶

An analogy: when a consumer orders 50 small items from a major online retailer, the retailer does not ship the order in 50 separate small boxes. Instead, the retailer packs as many of the small items as possible into one or a few larger boxes (i.e., parcels) then places the parcels on a semi-truck or airplane. The parcels may then pass through one or more regional distribution centers where they may be repackaged into different parcel configurations and forwarded further until they are finally delivered to the consumer. But most often, the consumer will only find one or a few parcels at their doorstep and not 50 separate small boxes. This flexible parcel delivery service greatly reduces shipping and handling cost for all including the retailer, regional distribution centers and finally the consumer.¶

4. IP Parcel Formation

A transport protocol entity identified by its 5-tuple forms a parcel body when it prepares a data buffer (or buffer chain) containing an Integrity Block of at most 256 2-octet Checksums followed by their corresponding transport layer protocol segments, with each TCP non-first segment preceded by a 4-octet Sequence Number header. All non-final segments MUST be equal in length while the final segment MUST NOT be larger and MAY be smaller. The number of non-final segments is represented as J; therefore the total number of segments is represented as (J + 1).¶

The non-final segment size L MUST be set to a value between 512 and 65535 octets and SHOULD be no larger than the minimum of 65535 octets and the path MTU, minus the length of the {TCP,UDP} header (plus options), minus the length of the IP header (plus options/extensions), minus 2 octets for the per-segment Checksum (see: Appendix B). The final segment length K MUST NOT be larger than L but MAY be smaller. The transport layer protocol entity then presents the buffer(s) and size L to the network layer, noting that the combined buffer length(s) may exceed 65535 octets if there are sufficient segments of a large enough size.¶

If the next hop link is not parcel capable, the network layer performs packetization to configure each segment as an individual IP packet as discussed in Section 5.1. If the next hop link is parcel capable, the network layer instead forms a parcel by appending a single full {TCP,UDP} header (plus options) and a single full IP header (plus options/extensions). The network layer finally includes a specially-formatted "Parcel Payload" option as an extension to the IP header of each parcel prior to transmission over a network interface.¶

The Parcel Payload option formats for both IP protocol versions are derived from the Jumbo Payload option specified in [RFC2675] and appear as shown in Figure 1:¶

                   IPv4 Parcel Payload Option Format
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Option Type  |  Opt Data Len |      Code     |     Check     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     Index     |             Parcel Payload Length             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Identification                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Path MTU (PMTU)                     |S|
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


                   IPv6 Parcel Payload Option Format
                                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                                   |  Option Type  |  Opt Data Len |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     Index     |             Parcel Payload Length             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Identification                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Path MTU (PMTU)                     |S|
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

For IPv4, the network layer includes the Parcel Payload option as an IPv4 header option with Option Type set to '00001011' and Option Data Length set to '00010000'. (Note: the length also distinguishes this type from its obsoleted use as the IPv4 "Probe MTU" option [RFC1063].) The network layer sets Code to 255 and sets Check to the same value that will appear in the IPv4 header TTL field upon transmission to the next hop. The network layer also sets Parcel Payload Length to a 3-octet value M that encodes the length of the IPv4 header plus the length of the {TCP,UDP} header plus the combined length of the Integrity Block plus all concatenated segments. The network layer then sets the IPv4 header DF bit to 1 and Total Length field to the non-final segment size L.¶

For IPv6, the network layer includes the Parcel Payload option as the first option in the first IPv6 Hop-by-Hop Options header, and with Option Type set to '11000010' and Option Data Length set to '00001100'. (Note: the most significant 3 Option Type bits are maintained the same as for the IPv6 Jumbo Payload option, with the understanding that nodes that recognize the Parcel Payload option will process the option consistently regardless of these bit settings. For further Hop-by-Hop option processing considerations, see: [I-D.ietf-6man-hbh-processing].) The network layer then sets Parcel Payload Length to a 3-octet value M that encodes the lengths of all IPv6 extension headers present plus the length of the {TCP,UDP} header plus the combined length of the Integrity Block plus all concatenated segments. The network layer also sets the IPv6 header Payload Length field to L.¶

For both IP protocol versions, the network layer then sets Identification and PMTU as specified in Section 5. The network layer next sets Index to an ordinal segment index value between 0 and 255 and the "More (S)egments" flag to 1 for non-final sub-parcels or 0 for the final (sub-)parcel. (Note that Index values other than 0 identify the initial segment index in non-first sub-parcels of a larger original parcel, whereas first (sub-)parcels always set Index to 0.)¶

Following transport and network layer processing, {TCP,UDP}/IP parcels therefore have the structures shown in Figure 2:¶

        TCP/IP Parcel Structure            UDP/IP Parcel Structure
   +------------------------------+   +------------------------------+
   |                              |   |                              |
   ~    IP Hdr plus extensions    ~   ~    IP Hdr plus extensions    ~
   |                              |   |                              |
   +------------------------------+   +------------------------------+
   |                              |   |                              |
   ~   TCP header (plus options)  ~   ~         UDP header           ~
   | (Includes Sequence Number 0) |   |                              |
   +------------------------------+   +------------------------------+
   |                              |   |                              |
   ~       Integrity Block        ~   ~       Integrity Block        ~
   |                              |   |                              |
   +------------------------------+   +------------------------------+
   ~                              ~   ~                              ~
   ~    Segment 0 (L-4 octets)    ~   ~     Segment 0 (L octets)     ~
   +------------------------------+   +------------------------------+
   ~  Sequence Number 1 followed  ~   ~                              ~
   ~    by Segment 1 (L octets)   ~   ~     Segment 1 (L octets)     ~
   +------------------------------+   +------------------------------+
   ~  Sequence Number 2 followed  ~   ~                              ~
   ~    by Segment 2 (L octets)   ~   ~     Segment 2 (L octets)     ~
   +------------------------------+   +------------------------------+
   ~             ...              ~   ~             ...              ~
   ~             ...              ~   ~             ...              ~
   +------------------------------+   +------------------------------+
   ~  Sequence Number J followed  ~   ~                              ~
   ~    by Segment J (K octets)   ~   ~     Segment J (K octets)     ~
   +------------------------------+   +------------------------------+

where the number of non-final segments is J, L is the length of each non-final segment (between 512 and 65535 octets), and K is the length of the final segment which MUST NOT be larger than L.¶

The {TCP,UDP}/IP header is immediately followed by an Integrity Block containing (J + 1) 2-octet Checksums concatenated in numerical order as shown in Figure 3:¶

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         Checksum (0)          |         Checksum (1)          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         Checksum (2)          |            ...                ~
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+            ...                ~
   ~            ...                             ...                ~
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |        Checksum (J-1)         |         Checksum (J)          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The Integrity Block is then followed by (J + 1) transport layer segments. For TCP, the TCP header Sequence Number field encodes a 4-octet starting sequence number for the first segment only, while each additional segment is preceded by its own 4-octet Sequence Number field. For this reason, the length of the first segment is only (L-4) octets since the 4-octet TCP header Sequence Number field applies to that segment. (All non-first TCP segments instead begin with their own Sequence Number headers, with the 4-octet header length included in L and K.)¶

Note: Per-segment Checksums appear in a contiguous Integrity Block immediately following the {TCP,UDP}/IP headers instead of inline with the parcel segments to greatly increase the probability that they will appear in the contiguous head of a kernel receive buffer even if the parcel was subject to OMNI interface IPv6 fragmentation. This condition may not always hold if the IPv6 fragments also incur IPv4 encapsulation and fragmentation over paths that transit IPv4 links with small MTUs. Even then, only the fragmented Integrity Block (i.e., and not the entire parcel) may need to be pulled/copied into the contiguous head of a kernel receive buffer.¶

Note: For IPv4 parcels, the first 2 octets of the Parcel Payload option include Code and Check fields in case a router on the path overwrites the values in a wayward attempt to implement [RFC1063]. IPv4 parcel recipients should therefore regard an incorrect Code or Check value as evidence that the field was accidentally or intentionally corrupted by a previous hop node.¶

       /* L must be at least 512; T is temporary length;
          H is {TCP,UDP}/IP header/extension lengths;
          For TCP, segment 0 Sequence Number is 4 octets;
          For each segment, Checksum is 2 octets */

       if ((L < 512) || ((T = (M - H)) <= 0))
           drop parcel;

       if (TCP) T += 4;

       if ((J = (T / (L + 2))) > 256)
           drop parcel;

       if ((K = (T % (L + 2))) == 0) {
           J--; K = L;
       } else {
           if ((J > 255) || ((K -= 2) <= 0))
               drop parcel;
       }

       if ((TCP) && (J == 0) && ((K -= 4) <= 0))
           drop parcel;

5. Transmission of IP Parcels

During {TCP,UDP} parcel assembly, the network layer of the source fully populates IP header fields including the source address, destination address and Parcel Payload option as discussed above. The source also sets IP {Total, Payload} Length to L (between 512 and 65535) to distinguish the parcel from other jumbogram types (see: Section 6).¶

The network layer of the source also maintains a randomly-initialized 32-bit cached Identification value for each destination. For each parcel transmission, the source sets Identification to the current cached value for this destination and increments the cached value by 1 (modulo 2**32) for each successive transmission (the source can later reset the cached value to a new random number, e.g., to maintain an unpredictable profile).¶

The network layer of the source finally sets the Parcel Payload option PMTU to the value '0' (unless the parcel is also being used as a probe - see: Section 5.6) then presents the parcel to an interface for transmission to the next hop. For ordinary interface attachments to parcel-capable links, the source simply admits each parcel into the interface the same as for any IP packet where it may be forwarded by one or more routers over additional consecutive parcel-capable links possibly even traversing the entire forward path to the final destination. If any node in the path does not recognize the parcel construct, it drops the parcel and may return an ICMP "Parameter Problem" message.¶

When the next hop link does not support parcels at all, or when the next hop link is parcel-capable but configures an MTU that is too small to pass the entire parcel, the source breaks the parcel up into individual IP packets (in the first case) or into smaller sub-parcels (in the second case). In the first case, the source can apply "packetization" using Generic Segment Offload (GSO), and the final destination can apply "restoration" using Generic Receive Offload (GRO) to deliver the largest possible parcel buffer(s) to the transport layer. In the second case, the source can apply "parcellation" to break the parcel into sub-parcels with each containing the same Identification value and with the S flag set appropriately. The final destination can then apply "reunification" to deliver the largest possible parcel buffer(s) to the transport layer. In all other ways, the source processes of breaking a parcel up into individual IP packets or smaller sub-parcels entail the same considerations as for a router on the path that invokes these processes as discussed in the following subsections.¶

Parcel probes that test the forward path's ability to pass parcels set the PMTU field to a non-zero value as discussed in Section 5.6. Each router in the path then rewrites PMTU in a similar fashion as for [RFC1063][RFC9268]. Specifically, each router compares the parcel PMTU value with the next hop link MTU in the parcel path and MUST (re)set PMTU to the most significant 31 bits of the minimum value. Note that the fact that the parcel transited a previous hop link should provide sufficient evidence of forward progress since parcel path MTU determination is unidirectional in the forward path only. However, nodes can also include the previous hop link MTU in their minimum PMTU calculations in case the link may have an ingress size restriction (such as a receive buffer limitation). Each parcel also includes one or more transport layer segments corresponding to the 5-tuple for the flow, which may include {TCP,UDP} segment size probes used for packetization layer path MTU discovery [RFC4821][RFC8899]. (See: Section 5.6 for further details on parcel path probing.)¶

When a router receives an IPv4 parcel it first compares Code with 255 and Check with the IPv4 header TTL; if either value differs, the router drops the parcel and returns a negative Jumbo Report (see: Section 5.5) subject to rate limiting. For all other IP parcels, the router next compares the value L with the next hop link MTU. If the next hop link is parcel capable but with MTU too small to pass a parcel with a single segment of length L the router discards the parcel and returns a positive Jumbo Report (subject to rate limiting) with MTU set to the next hop link MTU. If the next hop link is not parcel capable and has an MTU too small to pass an individual IP packet with a single segment of length L the router discards the parcel and instead returns a positive Parcel Report (subject to rate limiting) with MTU set to the next hop link MTU. For IPv4 parcels, if the next hop link is parcel capable the router MUST reset Check to the same value that would appear in the IPv4 header TTL field upon transmission to the next hop.¶

If the router recognizes parcels but the next hop link in the path does not, or if the entire parcel would exceed the next hop link MTU, the router instead opens the parcel. The router then forwards each enclosed segment in individual IP packets or in a set of smaller sub-parcels that each contain a subset of the original parcel's segments. If the next hop link is via an OMNI interface, the router instead proceeds according to OMNI Adaptation Layer procedures. These considerations are discussed in detail in the following sections.¶

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Next Header  |     Index     |      Fragment Offset    |P|S|M|
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Identification                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                   IPv4 Jumbo Reply MTU Option Format
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Option Type  |  Opt Data Len |           Rtn-PMTU            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                               0                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Identification                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Path MTU (PMTU)                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


                   IPv6 Jumbo Reply MTU Option Format
                                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                                   |  Option Type  |  Opt Data Len |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Min-PMTU            |          Rtn-PMTU           |0|
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Identification                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Path MTU (PMTU)                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                       IPv4 Parcel Pseudo-Header
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      IPv4 Source Address                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    IPv4 Destination Address                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |      zero     |  Next Header  |        Segment Length         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     Index     |            Parcel Payload Length              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


                       IPv6 Parcel Pseudo-Header
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   ~                      IPv6 Source Address                      ~
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   ~                   IPv6 Destination Address                    ~
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     Index     |            Parcel Payload Length              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |        Segment Length         |      zero     |  Next Header  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

6. Advanced Jumbos

This specification introduces an IP "advanced jumbo" service as an alternative to basic IPv6 jumbograms that also includes a path probing function based on the mechanisms specified in Section 5.6. The function employs an "Advanced Jumbo Option" with the same Option Type and Option Data Length values as for the Parcel Payload option, but with the Index and Parcel Payload Length fields converted to a single 32-bit Jumbo Payload Length field as shown in Figure 8:¶

                   IPv4 Advanced Jumbo Option Format
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Option Type  |  Opt Data Len |      Code     |     Check     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      Jumbo Payload Length                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Identification                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Path MTU (PMTU)                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


                   IPv6 Advanced Jumbo Option Format
                                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                                   |  Option Type  |  Opt Data Len |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      Jumbo Payload Length                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Identification                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Path MTU (PMTU)                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The source prepares an advanced jumbo by first setting the IP {Total, Payload} Length field to the special "Type" value '1' to distinguish this from a basic jumbogram or parcel. The source can begin by sending a "Jumbo Probe" to pre-qualify the path for advanced jumbos if necessary.¶

To prepare a Jumbo Probe that will trigger a Jumbo Report, the source can set {Protocol, Next Header} to {TCP,UDP}, set the {TCP,UDP} port to '9' (discard) and either include no octets beyond the {TCP,UDP} header or a single discard segment of the desired probe size immediately following the header and with no Integrity Block included. (The source can instead set the {TCP,UDP} port to the port number for a current data flow in order to receive IP Jumbo Reply MTU options in return packets as discussed in Section 5.6.) The source then sets Jumbo Payload Length to the length of the {TCP,UDP} header plus the length of the discard segment plus the length of the full IP header for IPv4 or the extension headers for IPv6.¶

The source next sets Identification the same as for an IP Parcel Probe, sets the Jumbo Probe PMTU to the next hop link MTU. For IPv4, the source also sets Code to 255 and Check to the next hop TTL. The source then calculates the {TCP,UDP} Checksum based on the same pseudo header as for an ordinary parcel (see: Figure 7) but with the Index and Parcel Payload Length fields replaced with a 32-bit Jumbo Payload Length field and with the Segment Length replaced with the Type value '1'. The source then calculates the checksum over the pseudo header then continues the calculation over the entire length of the probe segment. The source then sends the Jumbo Probe via the next hop link toward the final destination.¶

At each IPv4 forwarding hop, the router examines Code and Check then drops the probe and returns a negative Jumbo Report if either value is incorrect. If both values are correct, and if the next hop link is jumbo-capable, the router compares PMTU to the next hop link MTU, resets PMTU to the minimum value (and for IPv4 sets Check to the next hop TTL) then forwards the probe to the next hop. If the next hop link is not jumbo-capable, the router instead drops the probe and returns a negative Jumbo Report.¶

If the Jumbo Probe encounters an OMNI link, the OAL source can either drop the probe and return a negative Jumbo Report or forward the probe further toward the OAL destination using adaptation layer encapsulation. If the OAL source already knows the OAL path MTU for this OAL destination, it can encapsulate and forward the Jumbo Probe with PMTU set to the minimum of itself and the known value (minus the adaptation layer header size), and without adding any padding octets.¶

If the OAL path MTU is unknown, the OAL source can instead encapsulate the Jumbo Probe in an adaptation layer IPv6 header with a Jumbo Payload option and with NULL padding octets added beyond the end of the encapsulated Jumbo Probe to form an adaptation layer jumbogram no larger than the minimum of PMTU and (2**24 - 1) octets (minus the adaptation layer header size) as a form of "jumbo-in-jumbo" encapsulation.¶

The OAL source then writes this size into the Jumbo Probe PMTU field and forwards the newly-created adaptation layer jumbogram toward the OAL destination. If the jumbogram somehow transits the path, the OAL destination then removes the adaptation layer encapsulation, discards the padding, then forwards the probe onward toward the final destination (with each hop reducing PMTU if necessary).¶

When a router on the path forwards a Jumbo Probe, it drops and returns a Jumbo Report if the next hop MTU is insufficient; otherwise, it forwards to the next hop toward the final destination. When the final destination receives the Jumbo Probe, it returns a Jumbo Report with the PMTU set to the maximum-sized jumbo that can transit the path.¶

When the Jumbo Probe reaches the final destination, the destination first examines the {TCP,UDP} port number. If the port number is "discard", the destination returns a Jumbo Report UDP message; otherwise, the destination prepares an IP Jumbo Reply MTU option to include on a data packet on the return path to the original source. Detailed descriptions for these processes are found in Section 5.6.¶

After successfully probing the path, the original source can begin sending regular advanced jumbos by setting the IP {Total, Payload} Length field to the special Type value '1', setting PMTU to 0, then calculating the Checksum the same as described for probes above. When the final destination receives an advanced jumbo, it first verifies the Checksum then delivers the data to the transport layer without returning a Jumbo Report. The source can continue to send advanced jumbos into the path with the possibility that the path may change. In that case, a router in the network may return an ICMP error, an ICMPv6 PTB, or a Jumbo Report if the path MTU decreases.¶

Note: If the OAL source can in some way determine that a very large packet is likely to transit the OAL path, it can encapsulate a Jumbo Probe to form an adaptation layer jumbogram larger than (2**24 - 1) octets with the understanding that the time required to transit the path determines acceptable jumbogram sizes.¶

Note: The Jumbo Report message types returned in response to both Parcel and Jumbo Probes are one and the same, and signify that both parcels and advanced jumbos at least as large as the reported MTU can transit the path.¶

7. Minimal IP Parcels and Jumbograms

Minimal IP parcels and advanced jumbos are distinguished from regular parcels and advanced jumbos by including the same Option Type value as specified above, but with an Option Data Length of '00000100' for IPv6 or '00001000' for IPv4. These minimal forms provide the benefit of reducing the IP option length by 8 octets at the expense of omitting the Identification and PMTU values.¶

Minimal advanced jumbos also include a Type value of '1' in the IP {Total, Payload} Length field, while basic IPv6 jumbograms with Payload Length of 0 are processed per [RFC2675]. (IPv4 packets with Total Length of 0 are undefined and must be dropped.)¶

The option formats for IPv4 are shown in Figure 9 and the option formats for IPv6 are shown in Figure 10.¶

                      Minimal IPv4 Parcel Format
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Option Type  |  Opt Data Len |      Code     |     Check     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     Index     |             Parcel Payload Length             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                     Minimal IPv4 Jumbogram Format
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Option Type  |  Opt Data Len |      Code     |     Check     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      Jumbo Payload Length                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                      Minimal IPv6 Parcel Format
                                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                                   |  Option Type  |  Opt Data Len |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     Index     |             Parcel Payload Length             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                     Minimal IPv6 Jumbogram Format
                                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                                   |  Option Type  |  Opt Data Len |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      Jumbo Payload Length                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The original source can send minimal parcels or advanced jumbos after successfully probing a path to confirm that it can transit a given size over its entire length to the final destination. Minimal parcels and advanced jumbos use a reduced-length IP option that omits the Identification and Path MTU fields and therefore cannot transit a router that performs packetization/parcellation.¶

End systems and routers process minimal parcels the same as for expanded parcels as specified in previous sections. If a router needs to drop a minimal parcel, it returns a Parcel/Jumbo Report (subject to rate limiting) the same as for an expanded parcel, noting that the encapsulated parcel body will not contain an Identification and Path MTU field.¶

End systems and routers process minimal advanced jumbos with Type value '1' in the IP {Total, Payload} Length field the same as for expanded advanced jumbos as specified in Section 6. If a router needs to drop a minimal advanced jumbo, it returns a Jumbo Report (subject to rate limiting) the same as for an expanded advanced jumbo.¶

End systems and routers process basic IPv6 jumbograms with the value '0' in the IPv6 payload length field the same as specified in [RFC2675]. End systems and routers silently discard all IPv4 jumbograms with the value '0' in the IPv4 Total Length field, as no basic IPv4 jumbogram service is defined for IPv4.¶

Note: If the path changes, routers in the path may cease forwarding minimal parcels and advanced jumbos and begin returning ICMP errors, ICMP PTBs and/or Parcel/Jumbo Reports. According to the network trust model, the original source may then elect to re-probe to determine whether the path MTU has been reduced and/or whether the path can still support parcels/jumbos at all.¶

8. Implementation Status

Common widely-deployed implementations include services such as TCP Segmentation Offload (TSO) and Generic Segmentation/Receive Offload (GSO/GRO). These services support a robust service that has been shown to improve performance in many instances.¶

UDP/IPv4 parcels have been implemented in the linux-5.10.67 kernel and ION-DTN ion-open-source-4.1.0 source distributions. Patch distribution found at: "https://github.com/fltemplin/ip-parcels.git".¶

Performance analysis with a single-threaded receiver has shown that including increasing numbers of segments in a single parcel produces measurable performance gains over fewer numbers of segments due to more efficient packaging and reduced system calls/interrupts. For example, sending parcels with 30 2000-octet segments shows a 48% performance increase in comparison with ordinary IP packets with a single 2000-octet segment.¶

Since performance is strongly bounded by single-segment receiver processing time (with larger segments producing dramatic performance increases), it is expected that parcels with increasing numbers of segments will provide a performance multiplier on multi-threaded receivers in parallel processing environments.¶

9. IANA Considerations

The IANA is instructed to change the "MTUP - MTU Probe" entry in the 'ip option numbers' registry to the "JUMBO - IPv4 Jumbo Payload" option. The Copy and Class fields must both be set to 0, and the Number and Value fields must both be set to '11'. The reference must be changed to this document [RFCXXXX].¶

The IANA is instructed to create and maintain a new registry entitled "IP Jumbogram Types". For IP packets that include a Jumbo Payload Option, the IP {Total, Header} Length field encodes a "Jumbo Type" value instead of an ordinary length. Initial values are given below:¶

   Value      Jumbo Type                   Reference
   -----      -------------                  ----------
   0          Basic Jumbogram (IPv6 only)    [RFC2675]
   1          Advanced Jumbo                 [RFCXXXX]
   2-509      Unassigned                     [RFCXXXX]
   510        Reserved for Experimentation   [RFCXXXX]
   511        Reserved by IANA               [RFCXXXX]
   512-65535  IP Parcel                      [RFCXXXX]

10. Security Considerations

In the control plane, original sources match any identifying information in received Parcel/Jumbo Reports and IP Jumbo Reply MTU options with their corresponding probes. If the information matches, the report is likely authentic. In environments where stronger authentication is necessary, nodes that send Parcel and/or Jumbo Reports can apply the message authentication services specified for AERO/OMNI.¶

In the data plane, multi-layer security solutions may be needed to ensure confidentiality, integrity and availability. Since parcels and advanced jumbos are defined only for TCP and UDP, IPsec-AH/ESP [RFC4301] cannot be applied in transport mode although they can certainly be used in tunnel mode at lower layers such as for transmission of parcels and advanced jumbos over OMNI link secured spanning trees, VPNs, etc. Since the network layer does not manipulate transport layer segments, parcels and advanced jumbos do not interfere with transport or higher-layer security services such as (D)TLS/SSL [RFC8446] which may provide greater flexibility in some environments.¶

Further security considerations related to IP parcels are found in the AERO/OMNI specifications.¶

11. Acknowledgements

This work was inspired by ongoing AERO/OMNI/DTN investigations. The concepts were further motivated through discussions with colleagues.¶

A considerable body of work over recent years has produced useful "segmentation offload" facilities available in widely-deployed implementations.¶

With the advent of networked storage, big data, streaming media and other high data rate uses the early days of Internetworking have evolved to accommodate the need for improved performance. The need fostered a concerted effort in the industry to pursue performance optimizations at all layers that continues in the modern era. All who supported and continue to support advances in Internetworking performance are acknowledged.¶

The following individuals are acknowledged for their contributions: Scott Burleigh, Madhuri Madhava Badgandi, Bhargava Raman Sai Prakash.¶

Appendix A. TCP Extensions for High Performance

TCP Extensions for High Performance are specified in [RFC7323], which updates earlier work that began in the late 1980's and early 1990's. These efforts determined that the TCP 16-bit Window was too small to accommodate sustained transmission at high data rates and devised a TCP Window Scale option to allow window sizes up to 2^30. The work also defined a Timestamp option used for round-trip time measurements and as a Protection Against Wrapped Sequences (PAWS) at high data rates. TCP users of IP parcels are strongly encouraged to adopt these measures.¶

Since TCP/IP parcels only include control bits for the first segment ("segment(0)"), nodes must regard all other segments of the same parcel as data segments. When a node breaks a TCP/IP parcel out into individual packets or sub-parcels, only the first packet/sub-parcel contains the original segment(0) and therefore only its TCP header retains the control bit settings from the original parcel TCP header. If the original TCP header included TCP options such as Maximum Segment Size (MSS), Window Scale (WS) and/or Timestamp, the node copies those same options into the options section of the new TCP header.¶

For all other packets/sub-parcels, the note sets all TCP header control bits to '0' as data segment(s). Then, if the original parcel contained a Timestamp option, the node copies the Timestamp option into the options section of the new TCP header. Appendix A of [RFC7323] provides implementation guidelines for the Timestamp option layout.¶

Appendix A of [RFC7323] also discusses Interactions with the TCP Urgent Pointer as follows: "if the Urgent Pointer points beyond the end of the TCP data in the current segment, then the user will remain in urgent mode until the next TCP segment arrives. That segment will update the Urgent Pointer to a new offset, and the user will never have left urgent mode". In the case of IP parcels, however, it will often be the case that the "next TCP segment" is included in the same (sub-)parcel as the segment that contained the urgent pointer such that the urgent pointer can be updated immediately.¶

Finally, if the parcel contains more than 65535 octets of data (i.e., spread across multiple segments), then the Urgent Pointer can be regarded in the same manner as for jumbograms as described in Section 5.2 of [RFC2675].¶

Appendix B. Extreme L Value Implications

For each parcel, the transport layer can specify any L value between 512 and 65535 octets. Transport protocols that send isolated control and/or data segments smaller than 512 octets should package them as ordinary packets or as the final segment of a parcel. It is also important to note that segments smaller than 512 octets are likely to include control information for which timely delivery rather than bulk packaging is desired. Transport protocol streams therefore often include a mix of (larger) parcels and (smaller) ordinary packets.¶

The transport layer should also specify an L value no larger than can accommodate the maximum-sized transport and network layer headers that the source will include without causing a single segment plus headers to exceed 65535 octets. For example, if the source will include a 28 octet TCP header plus a 40 octet IPv6 header with 24 extension header octets (plus a 2 octet per-segment checksum) the transport should specify an L value no larger than (65535 - 28 - 40 - 24 - 2) = 65441 octets.¶

The transport can specify still larger "extreme" L values up to 65535 octets, but the resulting parcels might be lost along some paths with unpredictable results. For example, a parcel with an extreme L value set as large as 65535 might be able to transit paths that can pass jumbograms natively but might not be able to transit a path that includes non-jumbo links. The transport layer should therefore carefully consider the benefits of constructing parcels with extreme L values larger than the recommended maximum due to high risk of loss compared with only minor potential performance benefits.¶

Parcels that include extreme L values larger than the recommended maximum and with a maximum number of included segments could also cause a parcel to exceed 16,777,215 (2**24 - 1) octets in total length. Since the Parcel Payload Length field is limited to 24 bits, however, the largest possible parcel is also limited by this size. See also the above risk/benefit analysis for parcels that include extreme L values larger than the recommended maximum.¶

Appendix C. Additional Parcel/Jumbo Probe Considerations

When the source sends a Parcel/Jumbo Probe, it sets the PMTU field to the most significant 31 bits of the MTU of the next hop link and each hop along the way may further reduce this size. This may cause the source to underestimate the path MTU by at most one octet. The source can then use "common sense" to determine whether the MTU was underestimated; for example, if the reported MTU is 65534 it is very likely that packets of length 65535 would also transit the link due to the "one less than all ones" binary length. But, if the reported MTU was 1500 it is very unlikely that packets of length 1501 would transit the link under similar logic.¶

After sending a Parcel/Jumbo Probe, the source may receive a Parcel/Jumbo Report from either a router on the path or from the final destination itself. Alternatively, the source can shape its probes to request IP Jumbo Reply MTU options carried by ordinary data packets on the return path from the destination.¶

If a router or final destination receives a Parcel/Jumbo Probe but does not recognize the parcel/jumbo constructs, it will likely drop the probe without further processing and may return an ICMP error. The original source will then consider the probe as lost, but may attempt to probe again later, e.g., in case the path may have changed.¶

When the source examines the "packet in error" portion of a Parcel/Jumbo Report, it can easily match the Report against its recent transmissions if the Identification value is available. For "packets in error" that do not include an Identification, the source can attempt to match based on any other identifying information available; otherwise, it should discard the message.¶

If the source receives multiple Parcel/Jumbo Reports for a single parcel/jumbo sent into a given path, it should prefer any information reported by the final destination over information reported by a router. For example, if a router returns a negative report while the destination returns a positive report the latter should be considered as more-authoritative. For this reason, the source should provide a configuration knob allowing it to accept or ignore reports that originate from routers, e.g., according to the network trust model.¶

When a destination returns a Parcel/Jumbo Report, it can optionally "pair" the report with an ordinary data packet that it returns to the original source. For example, the OMNI specification includes a "super-packet" service that allows multiple independent IP packets to be encapsulated as a single adaptation layer packet. This is distinct from an IP parcel in that each packet member of the super-packet includes its own IP (and possibly other upper layer) header.¶

A source can request to receive two different types of parcel/jumbo path MTU feedback from the destination - a UDP encapsulated Parcel/Jumbo Report in response to a probe sent to the "discard" port, or an ordinary data packet with an IP Jumbo Reply MTU option in response to a probe sent into an ordinary transport layer protocol flow. In some environments, one or both of these MTU feedback types may be erroneously dropped by a router along the return path. The source may therefore attempt to probe first using "method A", and then try again using "method B", e.g., if there is no response. In environments where ongoing transport protocol sessions are established, it is recommended that the source engage the IP Jumbo Reply MTU option as "method A".¶

Appendix D. IP Parcel and Advanced Jumbo Futures

Both historic and modern-day data links configure Maximum Transmission Units (MTUs) that are far smaller than the desired state for Internetworking futures. When the first Ethernet data links were deployed many decades ago, their 1500 octet MTU set a strong precedent that was widely adopted. This same size now appears as the predominant MTU limit for most paths in the Internet today, although modern link deployments with MTUs as large as 9KB have begun to emerge.¶

In the late 1980's, the Fiber Distributed Data Interface (FDDI) standard defined a new link type with MTU slightly larger than 4500 octets. The goal of the larger MTU was to increase performance by a factor of 10 over the ubiquitous 10Mbps and 1500-octet MTU Ethernet technologies of the time. Many factors including a failure to harmonize MTU diversity and an Ethernet performance increase to 100Mbps led to poor FDDI market reception. In the next decade, the 1990's saw new initiatives including ATM/AAL5 (9KB MTU) and HiPPI (64KB MTU) which offered high-speed data link alternatives with larger MTUs but again the inability to harmonize diversity derailed their momentum. By the end of the 1990s and leading into the 2000's, emergence of the 1Gbps, 10Gbps and even faster Ethernet performance levels seen today has obscured the fact that the modern Internet of the 21st century is still operating with 20th century MTUs!¶

To bridge this gap, increased OMNI interface deployment in the near future will provide a virtual link type that can pass IP parcels over paths that transit legacy data links with small MTUs. Performance analysis has proven that (single-threaded) receive-side performance is bounded by transport layer protocol segment size, with performance increasing in direct proportion with segment size. Experiments have also shown measurable (single-threaded) performance increases by including larger numbers of segments per parcel, with steady increases for including increasing number of segments. However, parallel receive-side processing will provide performance multiplier benefits since the multiple segments that arrive in a single parcel can be processed simultaneously instead of serially.¶

In addition to the clear near-term benefits, IP parcels and advanced jumbos will increase performance to new levels as future links with very large MTUs in excess of 65535 octets begin to emerge. With such large MTUs, the traditional CRC-32 (or even CRC-64) error checking with errored packet discard discipline will no longer apply for large parcels and advanced jumbos. Instead, packets larger than a link-specific threshold will include Forward Error Correction (FEC) codes so that errored packets can be repaired at the receiver's data link layer then delivered to higher layers rather than being discarded and triggering retransmission of large amounts of data. Even if the FEC repairs are incomplete or imperfect, all parcels can still be delivered to higher layers where the individual segment checksums will detect and discard any damaged data not repaired by the link and/or adaptation layers (advanced jumbos on the other hand would require complete FEC repair).¶

These new "super-links" will begin to appear mostly in the network edges (e.g., high-performance data centers), however some space-domain links that extend over enormous distances may also benefit. For this reason, a common use case will include super-links in the edge networks of both parties of an end-to-end session with an OMNI link connecting the two over wide area Internetworks. Medium- to moderately large-sized IP parcels over OMNI links will already provide considerable performance benefits for wide-area end-to-end communications while truly large parcels and advanced jumbos over super-links can provide boundless increases for localized bulk transfers in edge networks or for deep space long haul transmissions. The ability to grow and adapt without practical bound enabled by IP parcels and advanced jumbos will inevitably encourage new data link development leading to future innovations in new markets that will revolutionize the Internet.¶

Until these new links begin to emerge, however, parcels will already provide a tremendous benefit to end systems by allowing applications to send and receive segment buffers larger than 65535 octets in a single system call. By expanding the current operating system call data copy limit from its current 16-bit length to a 32-bit length, applications will be able to send and receive maximum-length parcel buffers even if parcellation is needed to fit within the interface MTU. For applications such as the Delay Tolerant Networking (DTN) Bundle Protocol [RFC9171], this will allow transfer of entire large protocol objects (such as DTN bundles) in a single system call.¶

Continuing into the future, a natural progression beginning with IP packets then moving to IP parcels should also lead to wide scale adoption of advanced jumbos. Since advanced jumbos carry only a single very large transport layer data segment, loss of even a single jumbogram could invoke a major retransmission event. But, with the advent of error correcting codes, future link types could offer truly large MTUs. Advanced jumbos sent over such links would then be equipped with an error correction "repair kit" that the link far end can use to "patch" the jumbogram allowing it to be processed further by upper layers. Delay Tolerant Networking (DTN) over high-speed and long-delay optical links provides an example environment suitable for such large packets.¶

Appendix E. Change Log

<< RFC Editor - remove prior to publication >>¶

Changes from earlier versions:¶

• Submit for review.¶

Author's Address