λFS: A Scalable and Elastic Distributed File System Metadata Service using Serverless Functions
share
The metadata service (MDS) sits on the critical path for distributed file system (DFS) operations, and therefore it is key to the overall performance of a large-scale DFS. Common "serverful" MDS architectures, such as a single server or cluster of servers, have a significant shortcoming: either they are not scalable, or they make it difficult to achieve an optimal balance of performance, resource utilization, and cost. A modern MDS requires a novel architecture that addresses this shortcoming. To this end, we design and implement λFS, an elastic, high-performance metadata service for large-scale DFSes. λFS scales a DFS metadata cache elastically on a FaaS (Function-as-a-Service) platform and synthesizes a series of techniques to overcome the obstacles that are encountered when building large, stateful, and performance-sensitive applications on FaaS platforms. λFS takes full advantage of the unique benefits offered by FaaS x2013 elastic scaling and massive parallelism x2013 to realize a highly-optimized metadata service capable of sustaining up to 4.13× higher throughput, 90.40 latency, 85.99 better resource utilization and efficiency than a state-of-the-art DFS for an industrial workload.
READ FULL TEXT