Requirements for Scaling Deterministic Networks

3.1. Tolerate Time Asynchrony

A deterministic network may span over multiple networks with one or more cooperating administrative domains. There are many types of network nodes in the multiple domains which may introduce disparate local time variation, which require the tolerance of time asynchrony.¶

+--------------+                             +--------------+
|              |      DetNet Connection      |              |
| TSN Domain I +-----------------------------+ TSN Domain II|
|              |                             |              |
+--------------+                             +--------------+
                 |        |        |        |        |
 Clock of TSN    +--------+--------+--------+--------+
 Domain I        =
                 =
                 =       |        |        |        |        |
 Clock of TSN    =       +--------+--------+--------+--------+
 Domain II       =       =
                 =<==D==>=
                 =       =

3.2. Support Large Single-hop Propagation Latency

In some deterministic networks, a single hop distance is enough to generate large latency. The speed of optical transmission in fiber is 200 km/ ms. Thus, the propagation delay of a single hop can be in the order of a few milliseconds. It is much greater than that of a LAN, and introduces impacts on queuing mechanisms, such as cyclic or time aware scheduling method. So the requirement for propagation delays in end-to-end computations is needed, such as considering the propagation latency when setting the period in both time synchronization or frequency synchronization based methods, or setting the extra buffer in the asynchronization based methods, to meet the requirements of deterministic forwarding between the network nodes.¶

Here, we use an example to describe the influence of Large single-hop propagation delay on cycle based methods, but not to specify any solution. For a cyclic based method, suppose a deterministic network wants to keep using the simple cycle mapping relationship, however the link distance between two nodes is longer. Moreover, a downstream node may have many upstream nodes each with different link propagation delays (e.g., 9 us, 10 us, 11 us, 15 us and 20 us). In order to absorb the longest link propagation delay, the length of cycle must be set to more than 20 us. In [IEEE802.1Qch], propagation delay is part of the dead time imposed in a cycle, which impacts the bandwidth utilization. However, since packet's arrival time varies within the receiving cycle, larger cycle length means larger delay variance.¶

            Upstream Node X  |sending cycle  |            |
                             +--"------------+------------+
                             =  "\           =            =
                             =  " \          =            =
                             =  "  \         =            =
                             =  "   \        =            =
                             =  "    V       =            =
           Downstream Node Y |receiving cycle|            |
                             +--"----"-------+----\-------+
                             =  "    "       =     \      =
                             =  "    "       =      V resent out
                             =  "    "       =            =
                Time Line   -=--"----"-------=------------=----->
                (in us)      0  |    |   10           20
                                v    v
                          Transmission Latency

Note that Figure 2 is just to show an example of latency caused by large single-hop propagation. CQF is not limited to 2 queues, instead using more than 2 queues can be an option to solve large link latency related concerns. [Multiple-CQF]describes this problem in more detail and also proposes a mechanism to address it.¶

3.3. Accommodate the Higher Link Speed

A deterministic network can use higher speed links, especially for its backbone. At the time of publication, deterministic mechanisms used in a local network is usually deployed in link speed of 10 Mbps or 1 Gbps, or possibly 10 Gbps. The data rates of 10G, 100G, 400G and even higher are commonly used in wide area networks. With the increasing of the data rate, the network scheduling cycle can be reduced if the same amount of the data is required to be sent each cycle for each application. Or, more data can be sent if the network cycle time remains the same. For the former, it requires the more precise time control (e.g. cycle in the order of a few microseconds or sub- microseconds) for the input stream gate and the timed output buffer. For the latter, more buffer space is required which imposes more complex buffer or queue management and larger memory consumption.¶

Another aspect to consider is the aggregation of the flows. In the deterministic network, the number of flows can be hundreds or tens of thousands. They can be aggregated into a small number of deterministic path or tunnels. It is practical to have a few flow- based or aggregated-flow based status in the local network. But in higher speed and larger scale networks, it is hardly feasible. If [IEEE802.1Qcr] is in use, it requires more buffers comparing to the other full/partial time synchronized mechanisms. Therefore, it requires optimizations to support higher link speeds.¶

3.4. Be Scalable to The Large Number of Flows and Tolerate High Utilization of Bandwidth

The deterministic network may have the large number of traffic flows. According to [RFC2475], per-flow state identification in the network becomes infeasible as flow count scales up. So, it is almost impossible to identify individual IP flows at the DetNet data plane for a massive number of flows, neither for the per-node state machine configuration. DetNet allows the leverage of the flow aggregation, while individual flows may join and exit the aggregated flow rapidly in the scaling network with large number of flows, which causes the dynamic in identification of the aggregated DetNet flow. The wildcards and value ranges used in the identification may have to change in order to ensure the aggregated flows have compatible deterministic characteristics.¶

For scaling network, proper provision at the control plane to accommodate such higher aggregation is required. Provisioning on the aggregated flows normally improve the scalability at the cost of coarse granularity of the incoming traffic filtering and protection. It is desirable that adding a flow to the network does not affect the latency of the existing flows, or requires the complex re-calculation, it should require as less work as possible. For instance, only the filtering and policing configuration on the ingress node but not the intermediate nodes. The [IEEE802.1Qbv] uses traffic class to divide the flows and the number of it is usually 8, so that the forwarding mechanisms itself isn't complex with a large number of flows or higher aggregation. However, when adding new flows, the Gate Control List may be changed, and the re-calculation is complex. There might be the method to simplify the calculation or configuration, which need more work to enhance it.¶

Meanwhile, the traffic that requires deterministic networking can significantly fill up the capacity of a link or the portion of the link which is dedicated to such traffic, for example, more than 75% and/or up to near 100% utilization. Usually, the resources required for DetNet are reserved, however, the over-provisioning of link capacity may not work in such cases. in order to guarantee deterministic latency and jitter in this environment, it is better to provide scalable queuing solutions to improve the bandwidth utilization based on the current methods, inlcuding the TSN standard and other published standard. For instance, when the bandwithd utilization is high, the guard band in each cycle in [IEEE802.1Qch] is a type of over-provisioning and can be improved with more scalable queuing add-ons.¶

3.5. Tolerate Failures of Links or Nodes and Topology Changes

Deterministic networks may involve more network devices, and the increase or decrease of network devices in deterministic networks is more frequent than that in LANs. A simple use case to understand is ultra-low-latency (public) 5G transport networks, which would require DetNet extend to every 5G base station. For some network operators, their networks may need to connect to ~100 K base stations (serving multiple mobile networks operators), and this number will only increase with 5G.¶

One the one hand, the numerous devices may lead to more network link failures. Path switching or re-convergence of routing will cause high latency of packet loss and retransmission, which is usually in seconds before the network becomes stable again. As the added delay magnitudes involved are too large to use jitter compensation techniques，It is necessary to support certain mechanisms to adapt to failures of links or nodes and topology changes.¶

One the other hand, the change of the number of devices may affect the implementation and adjustment of deterministic network mechanism, such as the topology discovery, queuing mechanism and packet replication and elimination. For instance, The full disjoint paths when implementing the Packet Replication, Elimination, and Ordering Functions (PREOF) gives a better chance of survival when one of the nodes or links in the path fails. At the same time, it brings the challenges of finding paths with similar distance and/or number of hops so that there is enough buffer space to absorb the latency difference caused by different paths when the scale is large. So, a method is expected to support flexible planning of multiple paths and provide a solid foundation for the implementation of PREOF.¶

3.6. Prevent Flow Fluctuation

More kinds of DetNet traffic flows described in Section 3.4 will cause more dynamic joining or leaving of the flows, which will further cause more flow fluctuation as well as more unpredictability of the DetNet flows. Such as:¶

* Various and massive traffic flows of different applications in scaling network easily cause more bursty traffic.¶

* There will be more aggregation nodes which receives the flows from more upstream nodes adding the nondeterministic delay of the packet treatment.¶

* The bursts of flows can be accumulated as the flows traverse, join, and separate over hops. Once one of the nodes makes the minor error of packet treatment, it will have the cumulative effect for the downstream nodes.¶

* Loops formed in a network topology increase the maximum bursts of flows exponentially [ANDREWS][BOUILLARD][THOMAS].¶

* The node and link failures are more common in a large network (Section 3.5) which requires dynamic traffic steering to an alternate path, it will also easily cause the flow fluctuation.¶

Noting that the non-DetNet flows are also massive and may have potential impact on the scalability of the DetNet flows, for instance, causing the high utilization of the bandwidth and suppressing the possibility of more resource reservation and the traffic steering of DetNet flows. However, it is assumed that there will be the strategy in the ingress to deal with the non-DetNet flows and prevent the real-time influence on the DetNet flows.¶

It is required to support bursty traffic and some methods to decrease the micro-burst. So the pre-planned , ingress traffic conditioning, scalable queuing, and enhanced capacity of buffer are required to accommodate the flow fluctuation, and the time required for network reconfiguration to reflect such changes is required to be controlled, e.g., less than a specified amount of time.¶

3.7. Be Scalable to a Large Number of Hops with Complex Topology

Scaling networks often results in situations where an end-to-end flow involves a large number of hops, e.g., 15 or more. The network topology can also be complex, including star, ring, mesh, and their combinations, and can possibly be hierarchical. It is required to support networks with such various types of topologies and large hop Counts.¶

Delivering DetNet Quality of Service (QoS) in large and complex networks requires end-to-end bounds on both latency and jitter, as discussed in section 3.1 of [RFC8655]. Achievable end-to-end latency bounds necessarily depend on the number of hops for a flow. In the best case, the added queuing mechanism latency for DetNet QoS is bounded by a fixed constant per hop maximum value so that the resulting end-to-end latency bounds are a linear function of the number of hops in addition to the inherent forwarding latency of the nodes involved. In contrast, it is possible to achieve fixed constant end-to-end jitter bounds that are independent of the number of hops. Such fixed constant jitter bounds are strongly preferable to jitter bounds that increase with an increasing number of hops.¶

DetNet QoS requires resource allocation in advance (e.g., of link bandwidth and node buffer resources), as discussed in section 3.2.1 of [RFC8655]. The complexity of resource allocation processing may range from linear (e.g. allocating resources for each hop via a path-based resource reservation protocol such as RSVP [RFC2205]) to potentially exponential (e.g., if solving a complex graph optimization problem is required). This resource allocation complexity does not directly affect achievable end-to-end latency and jitter bounds, but it does surface in other areas such as the amount of computation and elapsed time required to admit a new flow to a DetNet network without disrupting the DetNet QoS provided to already admitted flows.¶

Different queuing mechanisms exhibit different properties across achievable end-to-end jitter bounds, achievable end-to-end latency bounds and complexity. All three of these areas are considerations in evaluation and selection of scalable DetNet queuing mechanisms.¶

3.8. Support Multi-Mechanisms in Single Domain and Multi-Domains

There will be large number of flows that described in Section 3.4, the flows may have different levels of demand for DetNet service[RFC8578] provides various use cases and their requirements in the areas of industry, electricity, buildings, etc. Some of them clearly specify the requirements for latency and jitter, while some others do not for the jitter. Different types of users have different demands, just as a network provider provides different network services for personal business or enterprise business.¶

One kind has critical SLA requirement, such as remote control or cloud Programmable Logic Controller (PLC) of manufacturing and differential protection of electricity. If these services exceed the boundaries of latency and jitter, it will bring property losses and security risks, so they cannot tolerate with any non-deterministic situation and can pay more on the network service. Another kind has relatively loose levels of SLA requirement, such as cloud gaming, cloud VR and online meeting for "consumer" networks. The users of these applications hope to have a better network experience, but they can tolerate it to a certain extent. For instance, exceeding the upper boundary of latency within a small probability is acceptable. Those different applications expect different kind of solutions, which are related to the cost more or less.¶

The different SLA demands need different DetNet technologies as well as the multi-domain demand in scaling network, which results in the requirements for the diverse queuing mechanisms. For strict deterministic services, strict queuing technologies need to be used, and all network devices may need to be upgraded. For non-strict deterministic services, it may only be necessary to upgrade some network devices(maybe edge nodes) or share corresponding network resources. Moreover, as queuing mechanisms development, it is also desirable to provide the queuing solutions with multiple levels of deterministic capabilities and schedule the reasonable resources to achieve the optimization of network resource utilization. Those different queuing technologies may be used in:¶

* the same network which requires the some of the equipment in the same network providing multiple queuing technologies. The operator can select the service type/level accordingly.¶

* the multiple domain network which support different queuing technologies while needs the coordination with each other.¶

* different network implementations intended for different application domains individually, where there is no additional requirements for the coordination.¶

                 Critical latency requirements:

     |      <->| Industrial, tight jitter, strict latency limit
     |<------->| Industrial, strict latency limit
     |
     |<-------------.....>  non-periodic, relative loose latency requirements
     |
     |<-------------........................>   Best effort
     |
     +---------------------------------------------------------->
                                                          latency

4.1. Support Aggregated Flow Identification

Current IPv6 aggregated flow identification is generally based on 5 or 6 tuples, IP prefixes, or wildcards as indicated in [RFC8938]. However, in deterministic networks the number of individual flows can be huge, and they may randomly join and leave the aggregated flow at each hop as described in Section 5. Such behaviours lead to the difficulty in identifying aggregated flows by relying on the prefixes or wildcards.¶

In addition, the deterministic services may demand different deterministic QoS requirements according to different levels of application requirements. The flow identification with service-level aggregation should be supported. Flow identification is also used to quickly push a packet to a suitable queue. In scaling network, there are large amount of flows requiring deterministic latency service and normal forwarding service. Explicit flow identification makes it easier to quickly distinguish the DetNet flows without requiring the longest match rule on multiple tuples in IP data plane. Therefore, explicit aggregated flow identification SHOULD be supported.¶

4.2. Support Information used by Functions Ensuring Deterministic Latency

According to Section 3.1, the deterministic network should support synchronized or asynchronized queuing mechanisms. Different queuing mechanisms require different information to be defined as the DetNet- specific metadata to help the functions of ensuring deterministic latency, including regulation, queue management, etc. For instance, the data plane needs to support the identification of cycle for cyclic queuing and forwarding or the latency related information for time based queuing in order to enable the applicability of the respective queuing and/or scheduling mechanisms in the large scale network.¶

When different queuing mechanisms are deployed at a network node, metadata used for each queuing mechanism should be provided at the same time. When multiple metadata carried in one packet, metadata should be self-described and extensible to tolerate variable number of metadata. Meanwhile, extra data will cause extra processing, referring to fixed or variable length lookups, total number of read/write access to the packet header, etc. So the data plane processing efficiency also needs to be considered when ensuring deterministic latency, but the specific method or solution is out of scope of this document.¶

This document does not specify what metadata and what format to be carried in data plane. The solution document should be specific enough on why and how the information carried as data plane meta data works in conjunction with the queuing or other functions and how it helps the deterministic network deployment.¶