Internet-Draft | Exp. Implementing Packet Discard Class. | May 2023 |
Evans & Pylypenko | Expires 23 November 2023 | [Page] |
Router reported packet loss is the primary signal of when a network is not doing its job. Some packet loss is normal or intended in TCP/IP networks, however. To minimise network packet loss through automated network operations we need clear and accurate signals of all packets which are dropped and why. This document describes our experience from implementing a packet loss classification scheme to provide these signals and enable automated network mitigation of unintended packet loss.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 23 November 2023.¶
Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
The job of a network is to transport packets. Understanding both where and why packet loss occurs is essential to the effective operation of networks. Router-reported packet loss is the most direct signal for network operations to identify customer impact from unintended packet loss. Accurate accounting of packet loss is not enough, however, as some level of packet loss is normal in TCP/IP networks. In automating network operations, there are only a relatively small number of automated actions that can be taken to mitigate impacting packet loss. Precise signal is important to ensure the right action is taken, however, as taking the wrong action can make problems worse.¶
The existing definitions of ifindiscards, ifoutdiscards, ifinerrors, ifouterrors [RFC1213], do not provide the precision needed to be able to identify the cause of packet loss sufficient to mitigate the impact. From a network operators' perspective, ifindiscards can represent both intended packet loss (i.e. packets discarded due to policy) and unintended packet loss. Further, these definitions are ambiguous, in that vendors can and have implemented them differently with different outcomes. In some implementations, ifinerrors accounts only for errored packets which are dropped, whilst in others it accounts for all errored packets whether they are dropped or not. Many vendors support more discard metrics than these; where they do they are inconsistently implemented due to an absence of clearly defined classification and semantics for packet loss reporting.¶
This document describes our experience from implementing a packet loss classification scheme across multiple hardware platforms, which aimed to address these issues and enable automated mitigation of unintended packet loss. Section 2 describes the problem. Section 3 defines the classification scheme and the accounting requirements with examples. Section 4 gives examples of discard signal to cause to auto-mitigation action mapping. Section 5 details our experience from implementing this scheme.¶
Working backwards from auto-mitigation of unintended packet loss, there are only a relative small number of potential auto-mitigation actions, e.g.:¶
Precise signal of impact is important as taking the wrong action can be worse than taking no action. For example, taking a congested device out of service can make congestion worse.¶
To be able to detect whether router reported packet loss is a problem and determine what actions should be taken to mitigate the impact and remediate the cause, depends on four primary features of the packet loss signal:¶
Features 2, 3 and 4 are already addressed with device level passive monitoring statistics, e.g. obtained with SNMP [RFC1157] or NETCONF [RFC6241]. Feature 1 is dependent on the classification scheme used for packet loss reporting. We define a new scheme to address this problem in the next section.¶
We define the classification scheme as a tree. At the top level we differentiate the direction: ingress or egress. At the second level we differentiate between traffic and discards. At the third level we differentiate between intended discards and unintended discards by discard class.¶
. |-- interface/ | |-- ingress/ | | |-- traffic_rx/ | | | |-- v4/ | | | | |-- packets | | | | `-- bytes | | | |-- v6/ | | | | |-- packets | | | | `-- bytes | | | |-- l2/ | | | | |-- packets | | | | `-- bytes | | | `-- qos/ | | | |-- class_0/ | | | | |-- packets | | | | `-- bytes | | | |-- ... | | | `-- class_n/ | | | |-- packets | | | `-- bytes | | `-- discards/ | | |-- v4/ | | | |-- packets | | | `-- bytes | | |-- v6/ | | | |-- packets | | | `-- bytes | | |-- l2/ | | | |-- packets | | | `-- bytes | | |-- policy/ | | | |-- acl/ | | | | `-- packets | | | |-- policer/ | | | | |-- packets | | | | `-- bytes | | | `-- null_route/ | | | `-- packets | | |-- errors/ | | | |-- rx/ | | | | |-- l2/ | | | | | |-- frames | | | | | |-- fec_error/ | | | | | | `-- frames | | | | | `-- invalid_frame/ | | | | | `-- frames | | | | `-- l3/ | | | | |-- packets | | | | |-- checksum_error/ | | | | | `-- packets | | | | `-- invalid_packet/ | | | | `-- packets | | | |-- local/ | | | | |-- packets | | | | `-- parity_error/ | | | | `-- packets | | | `-- no_route/ | | | `-- packets | | |-- ttl_expired/ | | | `-- packets | | `-- no_buffer/ | | |-- class_0/ | | | |-- packets | | | `-- bytes | | |-- ... | | `-- class_n/ | | |-- packets | | `-- bytes | `-- egress/ | |-- traffic_tx/ | | |-- v4/ | | | |-- packets | | | `-- bytes | | |-- v6/ | | | |-- packets | | | `-- bytes | | |-- l2/ | | | |-- packets | | | `-- bytes | | `-- qos/ | | |-- class_0/ | | | |-- packets | | | `-- bytes | | |-- ... | | `-- class_n/ | | |-- packets | | `-- bytes | `-- discards/ | |-- v4/ | | |-- packets | | `-- bytes | |-- v6/ | | |-- packets | | `-- bytes | |-- l2/ | | |-- packets | | `-- bytes | |-- policy/ | | |-- acl/ | | | `-- packets | | `-- policer/ | | `-- packets | |-- errors/ | | `-- tx/ | | |-- l2/ | | | `-- frames | | `-- l3/ | | `-- packets | `-- no_buffer/ | |-- class_0/ | | |-- packets | | `-- bytes | |-- ... | `-- class_n/ | |-- packets | `-- bytes `-- to_cpu/ |-- packets |-- bytes `-- policy/ |-- acl/ | `-- packets `-- policer/ `-- packets¶
Notes:¶
See Appendix A for an example of where packets may be dropped in a device.¶
discards/policy/
These are intended discards, i.e. packets dropped due to a configured policy. There are multiple sub-classes.¶
discards/policy/acl/
Discards due to packet matching an access control list (ACL)¶
discards/policy/policer/
Discards due to packet matching a configured policer¶
discards/policy/null_route/
Discards due to a packet matching a route with discard action¶
discards/no_route/
Discards due to a packet not matching any route¶
discards/no_buffer/
Discards due to no available buffer to enqueue the packet. These can be tail-drop discards or due to an active queue management algorithm, e.g. RED [RED93], CODEL [RFC8289]¶
discards/ttl_expired
There can also be multiple causes for TTL-exceed drops: i) trace-route; ii) TTL set too low by the end system; iii) routing loops¶
discards/error/rx/l2/
Frames dropped because they are invalid at L2, e.g. due to bad CRC or an invalid MAC address¶
discards/error/rx/l3/
These are drops due to errors in the received packet, i.e. which indicate an upstream problem, rather than a problem with the device that is dropping the errored packets. There are multiple potential errors that can cause a packet to be dropped on receipt: i) IP checksum errors; ii) malformed frame/packets¶
discards/error/local/
A device may drop packets within its switching pipeline due to internal errors, e.g. parity errors. Any discards not assigned to the above classes are accounted here.¶
These requirements apply to the packets forwarded by the device, not the packets destined to the device:¶
Assuming all the requirements are met, a good IPv4 packet received would increment:
- interface/ingress/traffic/v4/rx/packets
- interface/ingress/traffic/v4/rx/bytes
- interface/ingress/traffic/diffserv/class_0/rx/packets
- interface/ingress/traffic/diffserv/class_0/rx/bytes¶
A received IPv6 packet dropped due to ttl-expired would increment:
- interface/ingress/discards/v6/packets
- interface/ingress/discards/ttl_expired/packets¶
An IPv4 packet dropped on egress due to no buffers would increment:
- interface/egress/discards/no_buffer/class_0/packets
- interface/egress/discards/no_buffer/class_0/bytes¶
Example discard signal to cause to mitigation mappings are shown in the table below:¶
+------------------+---------+-------------------+-----------+--------+-----------+----------------------+ |Discard class |Direction|Cause |Rate |Duration|Unintended?|Possible actions | +------------------+---------+-------------------+-----------+--------+-----------+----------------------+ |ErrorRxL2Discards |Ingress |Upstream device |>0(Anomaly)|O(1min) |Y |Take upstream link or | | | |or link errror | | | |device out-of-service | |TTLDiscards |Ingress |Tracert |<=Baseline | | |no action | |TTLDiscards |Ingress |Convergence |>Baseline |O(1s) |Y |no action | |TTLDiscards |Ingress |Routing loop |>Baseline |O(1min) |Y |Roll-back | |AclDiscards |Ingress |ACL | | |N |no action | |NullRouteDiscards |Ingress |Null route | | |N |no action | |NoRouteDiscards |Ingress |Convergence |>0(Anomaly)|O(1s) |Y |no action | |NoRouteDiscards |Ingress |Config error |>0(Anomaly)|O(1min) |Y |Roll-back | |NoRouteDiscards |Ingress |Invalid destination|>0(Anomaly)|O(10min)|N |Escalate | |ErrorLocalDiscards|Ingress |Device errors |>0(Anomaly)|O(1min) |Y |Take device | | | | | | | |out-of-service | |NoBufferDiscards |Egress |Congestion |<=Baseline | |N |no action | |NoBufferDiscards |Egress |Congestion |>Baseline |O(1min) |Y |Bring capacity back | | | | | | | |into service or move | | | | | | | |traffic | +------------------+---------+-------------------+-----------+--------+-----------+----------------------+¶
There are no new security considerations introduced by this document.¶
There are no new IANA considerations introduced by this document.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
The content of this draft has benefitted from discussions with JR Rivers, Ronan Waide and Chris DeBruin.¶
The diagram below is an example of where and why packets may be dropped in a typical single ASIC, shared buffered type device.¶
+----------+ | | | CPU | | | +--+---^---+ from_cpu | | to_cpu | | +------------------------------v---+-------------------------------+ | | +----------+ +----------+ +----------+ +----------+ +----------+ +----------+ +----------+ | | | | | | | | | | | | | | -> Phy +--> Mac +--> Ingress +--> Buffers +--> Egresss +--> Mac +--> Phy |> | | | | | Pipeline| | | | Pipeline| | | | | +----------+ +----------+ +----------+ +----------+ +----------+ +----------+ +----------+ error/rx/l2 error/rx/l3 no_buffer error/tx/l3 error/local no_route ttl policy/acl policy/acl policy/policer policy/policer null_route¶