Traditionally, in-kernel file systems have out-performed user level file systems by a substantial margin. Although user level file systems are more feature-rich and have greater portability, in-kernel file systems are more widely deployed due to the advantage in performance that they have enjoyed. With present advances in hardware (CPU processing capabilities, high-speed I/O buses, high-bandwidth low-latency interconnects, etc.), the performance gap between user level and in-kernel file systems is not only being shortened, but is also becoming imperceptible. The goals of this project are to: i) design and implement next-generation network storage architectures, which couple user level file systems with emerging RDMA interconnect technologies and ii) evaluate the performance gap between in-kernel and user level file systems on state-of-the-art hardware. We are currently implementing and evaluating DAFS and NFS clients over Infiniband and VIA.
Perhaps the most widely used distributed file system, NFS is an important point in the design space of distributed file systems. Most commonly used in UNIX environments, recent versions of NFS have been completely implemented within the operating system kernel to provide the maximum I/O performance available. To achieve this, file systems designers have had to sacrifice both portability and functionality (i.e. ease of extensibility) by moving it completely into kernel space. Emerging advances in computer hardware capabilities (e.g. high-speed, low latency RDMA interconnects) afford us an opportunity to move NFS back into the user-level without sacrificing too much in performance. We believe that the trade-off between performance vs. portability/extensibility has shifted to the point where it is now feasible (and necessary) to consider a user-level NFS, once again.
We are currently designing and implementing a user-level NFS client. This client includes support for traditional TCP/IP protocols (TCP and UDP) and RDMA interconnects (InfiniBand and VIA). The User-Level NFS Client (uNFS) is implemented as a dynamically loadable user-level library. It includes a POSIX Adaptation Layer that can transparently intercept POSIX I/O calls made by applications and redirect them to the NFS server for I/O performed on NFS mounts. The RPC implementation is based on ONC-XDR.
The Direct Access File System (DAFS) is a new, lightweight remote file system protocol designed to leverage the benefits of RDMA-based networks. Applications using DAFS through a user-space I/O library can bypass operating system overhead, further improving performance. We study the characteristics and performance of DAFS by building user-level prototypes using various RDMA interconnects including Infiniband, VIA and Myrinet.
We have developed an implementation of the DAFS protocol (both client and server) in user-space. Our prototype includes support for various transports including Infiniband, VIA and Myrinet. Using a performance evaluation of our prototype, we have been able to demonstrate that user-level file systems can be used to provide applications with low overhead, high throughput access to network attached storage.
Coming Soon!Source code for our user-level DAFS and uNFS prototypes. The prototypes will include support for Infiniband (VAPI, uDAPL), Myrinet (GM) (DAFS only) and Virtual Interface Architecture (VIA).