TCP, the connection-oriented transport layer protocol of the Internet, is used by a vast majority of Internet services. TCP enforces a static binding between the service and the server to which the connection is set up. In contrast, today's Internet places a stronger emphasis on the service rather than on the identity of the server providing it. A client is concerned more with the quality and availability of the service rather than with the identity of the server, while with TCP a client cannot use an alternate server when the service is hampered due to network congestion, server overload, etc.
Migratory TCP (M-TCP) is a transport protocol compatible with TCP in which the client protocol stack can initiate migration of the remote endpoint of a live connection to an alternate server. Migration is transparent to the client application. M-TCP decouples the migration mechanism from migration policies that specify when should a connection migrate. Migration may be triggered according to some migration policy under conditions like server overload, network congestion, degradation in performance perceived by client, etc.
Connection migration in M-TCP
TCP is overly constraining for today's Internet service models:
- TCP reacts to lost or delayed segments by retransmissions to the same remote endpoint of the connection
- TCP provides no means to alleviate a performance degradation (low throughput, many retransmissions, etc.) caused by adverse conditions like server overload, network congestion on a given path, etc.
- TCP creates an implicit association between the server contacted by a client and a service
In contrast, M-TCP:
- offers a better alternative than the simple retransmission to the same server, which may be suffering from overload or a DoS attack, may be down, or may not be easily reachable due to congestion
- decouples a given service from the unique/fixed identity of its provider
M-TCP is a transport layer protocol for building highly-available network services by means of transparent migration of the server endpoint of a live connection between cooperating servers that provide the same service. The origin and destination server hosts cooperate by transferring supporting state in order to accommodate the migrating connection.
The client starts a service session by connecting to a preferred server, which supplies the addresses of its cooperating servers, along with authentication information. The client endpoint of a connection can initiate a migration by contacting one of the alternate servers. The migration trigger may reside with the client or with any of the servers. The server endpoint of the connection migrates between the cooperating servers, transparent to the client application.
To continue service on the migrated connection, M-TCP assumes that the state of the server application can be logically
split among the connections serviced, so that there exists a well-defined state associated with each service session. M-TCP defines a migration API that provides a minimal interface to the OS for exporting/importing a snapshot of the server application state associated with the connection. Transfer of this state from the origin to the destination server allows it to resume consistent service to the client. In addition to the associated application-level state, M-TCP transfers in-kernel TCP connection state and reconciles the TCP layer of the new server with that of the client.
The figure below shows the sequence of steps in a connection migration with on-demand ("lazy") transfer of state. The client initiates migration with a SYN (1), the destination server fetches the supporting state (2, 3) and completes the handshake for the migrated connection (4).
M-TCP Lazy State Transfer Mechanism
We integrate our migration mechanism in a uniform connection migration architecture. We distinguish between the trigger and the initiator of a migration, which decouples the migration mechanism from any migration policy. In our architecture, various migration policies can be implemented either at the client or at the server side.
Highly-available services, resilient to network congestion, DoS attacks.
Server load-balancing or client distribution.
- Fully-functional implementation in FreeBSD 4.3 using lazy state transfer. Migration of an established connection may take place in any state except for TIME_WAIT.
- We have used M-TCP with several open-source Internet servers (Apache, Icecast, Flash). The servers migrate connections while servicing clients (transfer files, play audio streams).
- PostgreSQL with migration support.
We have implemented migration support in PostgreSQL to allow a front-end to use M-TCP and continue the execution of a series of transactions submitted by a remote client in a session started with another front-end. Work already done at the back-end on behalf of the client is not redone or otherwise duplicated. In order to preserve the outcome of a transaction across migrations, a front-end takes state snapshots before a commit/abort.
Client connection migration between two PostgreSQL front-ends
The features of M-TCP are:
- It is general and flexible: it does not rely on knowledge about a given server application or application-level protocol
- It allows fine-grained migration of live individual connection, unlike heavy-weight process migration schemes
- It is symmetric with respect to and decoupled from any migration policy
- Migratory TCP: Connection Migration for Service Continuity in the Internet.
- Autonomous Transport Protocols for Content-based Networks.
Florin Sultan, Aniruddha Bohra, Liviu Iftode. Rutgers University Technical Report DCS-TR-479, March 2002.
- Migratory TCP: Highly Available Internet Services Using Connection Migration.
Florin Sultan, Kiran Srinivasan, Deepa Iyer, Liviu Iftode. Rutgers University Technical Report DCS-TR-462, December 2001.
- M-TCP: Transport Layer Support for Highly Available Network Services.
Kiran Srinivasan. MS Thesis. Rutgers University Technical Report DCS-TR-459, October 2001.
- Transport Layer Support for Highly-Available Network Services.
Florin Sultan, Kiran Srinivasan, Liviu Iftode. Position summary at HotOS-VIII, May 2001. Also published as Rutgers University Technical Report DCS-TR-429, January 2001.