Applying TCP User Timeout Parameter to BGP Sessions

Abstract

In this document we discuss the TCP "User Timeout" parameter and recommend using it to handle stuck BGP sessions.¶

1. Introduction

A BGP session [RFC4271] is said, informally, to be "stuck" when BGP messages are not transmitted over the session for an extended period of time. Certainly the stuck BGP session should have been terminated by the BGP holdtimer. Such a case could occur, though, due to software defects or under certain unusual circumstances. Currently it's difficult to know for sure due to lacking of automated, real-time detection mechanisms in BGP implementations.¶

It has been speculated that some BGP sessions may have stuck from time to time, and that may have contributed to stale routes (e.g., missing route withdraws) in the routing system.¶

Here is a specific scenario of a stuck BGP session between two BGP speakers A and B:¶

• Due to certain software defect, B stops reading data from TCP [RFC0793] for an extended period of time, resulting in B advertises a zero value for its TCP window size after its TCP buffer fills up.¶
• B fails to generate BGP HOLDTIME expiration, although it has not read from TCP and thus has not received any BGP KEEPALIVE from A during that time.¶
• B, however, continues to send BGP KEEPALIVE to A on time.¶

In this scenario A would not be able to send routing updates to B during that period of time. The routing system may become stale, not only on B, but on its BGP neighbors and beyond.¶

It's desirable for a BGP speaker (e.g., A in the example) to be able to detect and then terminate such a stuck session so that the stale routes are purged from the routing system.¶

The availability of such a mechanism may also help accelerate the resolution of the software defect involved.¶

In this document we discuss the TCP "User Timeout" parameter [RFC0793] and recommend using it to handle stuck BGP sessions.¶

2. Discussion on TCP User Timeout

The TCP "User Timeout" parameter is designed to terminate a connection in a variety of cases where a TCP session does not progress within certain time period. It is specified in [RFC0793] as follows:¶

  USER TIMEOUT

     For any state if the user timeout expires, flush all queues,
     signal the user "error:  connection aborted due to user timeout"
     in general and for any outstanding calls, delete the TCB, enter
     the CLOSED state and return.

Clearly the TCP "User Timeout" applies when the application data is not delivered on time, including the cases that transmitted data may remain unacknowledged, or buffered data may remain untransmitted (due to zero window size).¶

The TCP "User Timeout" parameter is well summarized in [RFC5482], although the zero-window case is not explicitly called out:¶

        The Transmission Control Protocol (TCP) specification RFC0793
        defines a local, per-connection  "user timeout" parameter that
        specifies the maximum amount of time that transmitted data may
        remain unacknowledged before TCP will forcefully close the
        corresponding connection. Applications can set and change this
        parameter with OPEN and SEND calls.

Regarding the implementation of the TCP "User Timeout" parameter, one example is Linux's "TCP_USER_TIMEOUT" socket option documented in [LINUX-TCP].¶

3. Recommendations

As discussed in the introduction, a BGP session is considered "stuck" when BGP messages are not delivered for an extended period of time.¶

Given that BGP messages are TCP data, and TCP is responsible for delivering the data, thus it would be more natural and more complete to address the issue at the TCP layer rather than in BGP itself (particularly in the case of persistent TCP zero-window).¶

As the TCP "User Timeout" parameter is specifically defined to terminate the TCP connection when something in TCP is "stuck", we thus recommend using it to detect and terminate these stuck BGP sessions.¶

We RECOMMEND that the TCP "User Timeout" parameter be set for all BGP sessions, and the default timeout value be five times the configured BGP holdtime value but no less than ten minutes in order to tolerate certain short-lived, transient conditions. The TCP "User Timeout" value for a BGP session SHOULD be configurable.¶

We also RECOMMEND that the TCP "User Timeout" parameter be set only after the End-of-RIB marker [RFC4724], if expected, is received from each of the (AFI, SAFI) being exchanged over the BGP session, or otherwise thirty minutes after the BGP session is established. The delay for setting the parameter SHOULD be configurable.¶

When the TCP "User Timeout" for a BGP session expires, the BGP speaker SHOULD log the event locally. In addition, the administrator of the remote BGP speaker SHOULD be informed (by means outside the scope of this document) so that the issue can be investigated.¶

The procedures for BGP Graceful Restart [RFC4724] SHOULD be followed when the TCP session is terminated due to TCP "User Timeout" expiration.¶

7. References