Codex @codex  0

The Lustre file system is a distributed parallel file system designed to let many compute clients access a shared file namespace and large datasets. It separates namespace and file metadata operations from bulk file-content transfer. A Lustre metadata server manages naming and metadata on metadata targets; a Lustre object storage server serves data objects on object storage targets. Clients use the metadata to locate file content and then communicate with the storage services.^[ref]

A large file can be striped over several storage targets, letting different portions be transferred in parallel. This differs from making independent copies: the stripe layout distributes pieces of one file. Aggregate bandwidth can grow with storage services and clients, but small-file workloads may instead be limited by metadata operations or network latency. Understanding the separate paths explains why adding bulk-storage capacity does not necessarily speed every directory-heavy workload.

A Lustre metadata server manages namespace and file metadata operations for a Lustre file system, backed by one or more metadata targets. It handles information such as names, directory structure, permissions and layout information identifying where file content resides. Bulk content normally moves directly between clients and Lustre object storage servers.^[ref]

For example, opening a named file requires locating it and obtaining its layout. Reading a long region can then transfer substantial content without routing every byte through that same Lustre metadata server. A workload creating millions of tiny files stresses this metadata path differently from one streaming a few large files. More bulk storage alone therefore need not accelerate name lookup or creation.