Proposals

Challenge of Directories and Files Scalability

This proposal has been rejected.

*

One Line Summary

This talk will mention the challenge of directories and files scalability challenge for large scale distributed file system for real online shopping business.

Abstract

For PB level distributed file systems, local file systems (e.g ext4, btrfs …) on data nodes encounter new requirements more then what they are originally designed for, for example,
- Store a huge number of files (data chunks) under several directories.
- Store a few big files (80G Bytes~1T Bytes) to take all space of file systems.
- Less I/O when doing directory lookup or file offset mapping.
- Higher space usage efficiency, don’t waste any $$ spent on SSD blocks.
- File system metadata cache memory pressure on data node.
On a 600+ nodes 2PB+ distributed file system cluster, any small problem might become a big headache.

In this talk, the above challenges on a real example will be shared to audience — TFS 1, the Taobao File System. This talk will mention,
- Local disk file system performance bottom neck for TFS usage
- New requirements to Linux file system development
- The approach that improve part of the performance bottom neck.
- Existed open questions, will co-work with commnity to improve Linux file system.

This talk will take around 40 minutes. If you are interested on how Linux local file systems are used and what are wanted to improve in large scale distributed file system for a high workload online shopping website, please come :-)

1 TFS: Taobao File System. Taobao is the biggest online shopping website in China, TFS is the distributed file system deployed on http://www.taobao.com for goods pictures. TFS is planed to be open source before LPC2010 on http://code.taobao.org

Tags

file system, distributed file system

Speaker

  • Coly_head

    Coly Li

    Taobao

    Biography

    Linux kernel developer in Taobao core-system team, ocfs2-tools and grub2 maintainer of openSUSE community.

Leave a private comment to organizers about this proposal