Compress data to further exploit flash storage

This proposal has been rejected.

*
Refereed Presentations
Refereed Presentation

One Line Summary

Transparently storing compressed data on fast persistent storage like SSD and NVMe not only extends life and increases capacity of the device, it also significantly enhances I/O performance as seen by our exciting results.

Abstract

Fast persistent flash based storage like SSD and NVMe devices have already become a norm on all platforms. Although these devices have well-known advantages, they suffer from problems like write-amplification, relatively low lifetime and high cost. Data compression alleviates these problems.  Since less data is read and
written, write-amplification is reduced, lifetime is extended, and hence per-GB cost is reduced. Most importantly, it increases I/O throughput. We have developed a device mapper block storage driver that facilitates transparent compression.  Our experiments have shown remarkable improvements in I/O performance when used with different file-system. Using it as a swap device, we were able to significantly over-commit memory and still be able to maintain application performance and reduce response latency.

In this presentation we will explain the internals of our driver and the exciting performance results.  We will also discuss the bottlenecks that we encountered under extreme over-commit situation, and our solution.
We will use this opportunity to seek inputs from the community towards making this driver an essential plumbing of the Linux kernel.  Version 5 of our proposed device driver is posted to the community at https://lwn.net/Articles/716985/

Tags

latency, SSD, Device Mapper, nvme, swap, compress, file-system, throughput, over-commit

Back to list of proposals

Speakers

  • Shaohua Li

    Biography

    Shaohua Li is a Linux kernel developer since 2003. He currently works for Facebook and is the maintainer of Software RAID. His contribution covers different Linux kernel subsystems from x86, IOMMU, storage to memory management. He is interesting in high speed storage support, optimization and adoption.

  • Ram Pai

    IBM

    Biography

    Ram Pai is a Linux kernel developer since 2001. He works for IBM’s Linux Technology Center in Hillboro Oregon.  Prior to this, he contributed the ‘Shared Subtree’ feature to the Linux kernel.  He has contributed patches to enable SRIOV, enhance the readahead algorithm, tools to track exported symbols, and other miscellaneous features to the kernel.