BGP Blockchain

4.1. DCS Convergence Latency

In order to understand the aspect of latency better, a bit of background is needed on how a DCS generally working (the reader is referred to more detailed material in references such as [Nakamoto2008][Howard2014][Aspnes2022] for more details).¶

Key to achieving a distributed consensus (over some agreed information, such as BGP route changes) is the distribution of the information to the peers participating in the DCS. The majority rule, formulated by von Newman [Newman1956], states for being able to consent over the information, it needs distribution to at least N/2 peers with N being the size of the set of peers in the DCS; this distribution is often referred to as diffusion in order to result in a majority rule.¶

Given the overlay nature of DCSs, this diffusion is realized by peer-side unicast replications, limited in number of peers to which each peer diffuses the information. Systems, such as Ethereum, rely on default configurations for the number of peers to diffuse to; expected to be 50 at the time of writing this document. Each receiving peer further distributes the information itself to another set of peers.¶

Those sets of peers maintained at each peer are randomized through mechanisms further detailed in, e.g., [Guzman2022]. The result is an iterative diffusion process, creating an increasing number of peers per iteration to which the information is distributed.¶

When visualizing this process, as done in Figure 2, we can see that this process, bounded at the top by the desired majority of peers, is bounded on the time axis by the convergence latency t_c, after which the desired majority has been reached; it is important to note that convergence, however, is not guaranteed and heavily dependent on the random nature of each individual peer-initiated diffusion.¶

n [peers] |                                      +----+
          |                                      |    |
          |                                      |    |
          |                                  +---+    |
    m=n/2 |                                  |        |
    ------+------------------+----+----------+--------+-----------
          |                  |    |          |        |
          |           +------+    |   ...    |        |
          |           |           |          |        |
          |           |           |          |        |
          |   +-------+           |          |        |
          |   |                   |          |        |
          |---+                   |          |        |
          |                       |          |        |
        --+-----------------------+----------+--------+-----------
                                            t_c              t [s]

This latency t_c is crucial for the overall system to operate on ensured, i.e., consistent state, and largely depends on three key factors: (i) the number of peers in the DCS (ii), the number of required iterations, and (iii) the latency for each iteration to happen for a given configuration for the diffusion size. For (i), the specific application of the DCS is largely a factor. With large DCS systems, such as Ethereum, this number reaches beyond 500k peers. If we assumed at least one peer per AS for many of our BGP examples, the number of peers would also exceed 100k as per latest AS numbers for the Internet. Item (iii) is a function of communication latency and thus mainly depend on the nature of peer distribution and its resulting network latency, while item (ii) is harder to bound since it depends on the randomization of the individual peer-initiated diffusions. Ideally, each peer shall distribute the information to unique peers, not served by any other peer. Given the lack of central coordination of the individual diffusions, however, peers may well receive information multiple times from differing peers in reality, thus not progressing the goal of reaching a majority rule but prolonging t_c instead.¶

In systems, such as Ethereum, the diffusion and validation of information can thus reach from about 1 minute to ca. 50 minutes in 90 percent of the cases [Pacheco2022] (Table 4). Translated onto our BGP usages, this would lead to long periods of needing to work on inconsistent state, an aspect being discussed in Section 4.3.¶

The approach to increase the number of peers to which each peer diffuses is one dependent on the costs for each individual peer diffusion. Existing systems, such as Ethereum, limit the number to a relative small number of 50 by default; we discuss next some of the reasons for doing so.¶

The main takeaway is that the convergence latencies in large-scale DCSs may cause significant issues with a number of BGP functions where working on consistent state is crucial.¶

4.2. Communication Costs

While much has been reported on the computational costs for blockchain and DCS technologies in general [Drusinsky2022], only recently has the communication costs (and its underlying causes) been studied [Guzman2022]. Key here is the randomization of the set of peers (at each peer) to diffuse information to. For this, each peer maintains a pool of peers as a combination of outgoing (i.e., actively sought connections) and incoming (i.e., passively received connections from other peers) relationships. For each such relation, a process of reachability checks, transport establishment, and capability exchange is performed, all while a constant refreshing of the (limited size) pool is ongoing, resulting in a constant evacuation of peers from the pool.¶

As quantified in [Guzman2022], this has a significant impact on the number of connection requests made by each peer as well as the costs in terms of transferred but yet not used data during the relation establishment. Further, the time to determine the desired number of peers may take several minutes (on average 20 for the Ethereum studies in [Guzman2022]), and thus defines the minimal time a peer may need to wait until a single diffusion step can be finished; once an initial pool has been built, refreshment will happen faster, however, yet constant refreshing continues.¶

The insights provided in [Guzman2022] well explain the limits imposed on diffusion steps, such as 50 in Ethereum systems, since the pool maintenance process even for those fairly small sizes create already significant costs per peer. Although the overall DCS platform costs `merely' for the diffusion of information to create the desired consensus and thus consistent state are subject to system-level studies, initial work such as [Guzman2022] may provide the base data as input. Extrapolating from this pure peer-centric available data at this stage, however, lets us expect a very significant communication cost for an Internet-scale DCS that could serve BGP.¶

The main takeaway is that communication costs for a large-scale DCS may be significant and thus may pose a challenge for building a large-scale DCS at the right cost point for all AS providers to participate.¶

4.3. Working on Inconsistent State

As outlined in Section 4.1, achieving consensus in a DCS, and thus consistent state over which to operate the desired application, experiences significant latencies, often in the range of many minutes for large-scale systems. For this reason, various DCS platforms have devised methods to work on inconsistent state, akin to operating on a constantly amended ledger during an open accounting period in a company.¶

Here, information that is in the process of diffusion is partially operated over in so-called 'proof' operations to generate temporary 'truths', representing inconsistent yet agreed upon state until the consistent state is available. In order to avoid collusion in generating false truths, methods such as 'proof of work' [Nakamoto2008] or 'proof of stake' [Dimitri2022], where the trust in the truth is derived from the commitment to, e.g., spend significant costs on computational operations over the incomplete state, which in turn is responsible for much of the reported cost factor of popular DCS systems, in particular for cryptocurrencies.¶

The main takeaway is that while this cost itself is something that needs consideration for using for a system like BGP, it is to be considered if the notion of working on inconsistent state is acceptable for those largely independent AS operators that partake in the overall BGP operation.¶

5.1. Routing on Service Addresses

When looking closer at the workings of a DCS, one can map the interactions (i.e., inserting information, diffusing for consensus and retrieving the consensus result) onto those of a service-based systems. Here, peers can be seen as invoking those services at other peers of the overall DCS. More formally, we can identify the insertion service I, the diffusion service D and the query service Q in a typical DCS. While I and Q are invoked only to a set of peers with no responses given to I and responses gathered in Q (with the largest returned information set representing the current consented information), the diffusion service D is invokved recursively.¶

While specific DCS platforms use particular endpoint-initiated methods realizing those interaction patterns, such as for discovering other peers (and thus services) and diffusing any new information, as investigated in [Guzman2022], one possible improvement to those endpoint-centric methods is to utilize insights for network-supported service routing instead.¶

The proposal in [I-D.trossen-rtgwg-rosa] postulates an approach to route over service addresses, where those addresses (unlike routing identifiers) are used to steer the traffic to the appropriate network location, particularly for scenarios in which more than one network location choice is possible (in the case of the DCS, peers executing the services above). This idea well maps onto the working of a DCS, when mapping the specific services onto the above introduced distinct service addresses, while amending the anycast forwarding behaviour outlined in [I-D.trossen-rtgwg-rosa] with a diffusion-based on.¶

With such approach, the endpoint-centric replications, together with the frequent churn of relations as observed in [Guzman2022], would be replaced with a service announcement based approach for discovery of other peers. The resulting in-network diffusion retains the desired random nature of the diffusion. Although we must still assume churn for those peers of a DCS that may leave during the lifetime of the DCS operations, buty the churn observed in [Guzman2022] is caused through the replenishment of the diffusion pool, which is now replaced with the in-network diffusion over a (ephemerally) stable set of service, i.e., peer, announcements to the ROSA network.¶

Furthermore, the lower cost for in-network replication (due to the missing churn for the discovery process) may allow for larger diffusion sets being used, thus possibly reducing the overall convergence latency - although a deeper analysis in those costs would be required for a possible bounding of the diffusion size and thus the possible minimal convergence latency in dependence of the overall number of DCS peers.¶

Deployment-wise, the ROSA approach also aligns with the overlay approach used for a DCS today in that the 'ROSA domain' [I-D.trossen-rtgwg-rosa] forms an overlay across possibly many network domains, thus allows for separating the role of DCS provider (which may operate the ROSA domain) from that of the network operators used for bit transfer.¶

5.2. Compute-Aware Traffic Steering

Another network-level technology that may improve on the efficacy of a DCS is that of 'compute-aware traffic steering' (https://datatracker.ietf.org/wg/cats/about/). This recently approved effort in the IETF foresees the use of computational information for steering traffic in various places of the network (including, possibly, the application itself). Computational information here may include capabilities of service resources, e.g., max connectivity speed or HW capabilities such as GPU availability, but also dynamic information, e.g., on server CPU load, available memory.¶

With this, currently deployed traffic steering decisions, mainly relying on network information, could be supplemented with such computational information and thus allow for sending traffic not to the shortest (network) path but the, e.g., least loaded compute resource instead.¶

In the context of a DCS, such computational awareness could complement the aforementioned service routing capabilities in that the peers chosen for diffusion (of information) may be further constrained by computational capabilities (e.g., diffusing to least loaded peers) as well as static capabilities (e.g., diffusing to well-connected peers when retrieving relatively large, often TB, blockchains). This overall may further improve latencies by choosing `appropriate' resources instead of randomly chosen ones only. One key route of investigation is the avoidance of collusion, here for instance through announcing fake computational capabilities and metrics for `attracting' more traffic in a DCS than a purely random diffusion would result in.¶

5.3. Locator/ID Separation Protocol

TBD¶