Chapter 1. Overview of available file systems
Chapter 1. Overview of available file systems Choosing the file system that is appropriate for your application is an important decision due to the large number of options available and the trade-offs involved. The following sections describe the file systems that Red Hat Enterprise Linux 8 includes by default, and recommendations on the most suitable file system for your application. 1.1. Types of file systems Red Hat Enterprise Linux 8 supports a variety of file systems (FS). Different types of file systems solve different kinds of problems, and their usage is application specific. At the most general level, available file systems can be grouped into the following major types: Table 1.1. Types of file systems and their use cases Type File system Attributes and use cases Disk or local FS XFS XFS is the default file system in RHEL. Red Hat recommends deploying XFS as your local file system unless there are specific reasons to do otherwise: for example, compatibility or corner cases around performance. ext4 ext4 has the benefit of familiarity in Linux, having evolved from the older ext2 and ext3 file systems. In many cases, it rivals XFS on performance. Support limits for ext4 filesystem and file sizes are lower than those on XFS. Network or client-and-server FS NFS Use NFS to share files between multiple systems on the same network. SMB Use SMB for file sharing with Microsoft Windows systems. Shared storage or shared disk FS GFS2 GFS2 provides shared write access to members of a compute cluster. The emphasis is on stability and reliability, with the functional experience of a local file system as possible. SAS Grid, Tibco MQ, IBM Websphere MQ, and Red Hat Active MQ have been deployed successfully on GFS2. Volume-managing FS Stratis (Technology Preview) Stratis is a volume manager built on a combination of XFS and LVM. The purpose of Stratis is to emulate capabilities offered by volume-managing file systems like Btrfs and ZFS. It is possible to build this stack manually, but Stratis reduces configuration complexity, implements best practices, and consolidates error information. 1.2. Local file systems Local file systems are file systems that run on a single, local server and are directly attached to storage. For example, a local file system is the only choice for internal SATA or SAS disks, and is used when your server has internal hardware RAID controllers with local drives. Local file systems are also the most common file systems used on SAN attached storage when the device exported on the SAN is not shared. All local file systems are POSIX-compliant and are fully compatible with all supported Red Hat Enterprise Linux releases. POSIX-compliant file systems provide support for a well-defined set of system calls, such as read() , write() , and seek() . When considering a file system choice, choose a file system based on how large the file system needs to be, what unique features it must have, and how it performs under your workload. Available local file systems XFS ext4 1.3. The XFS file system XFS is a highly scalable, high-performance, robust, and mature 64-bit journaling file system that supports very large files and file systems on a single host. It is the default file system in Red Hat Enterprise Linux 8. XFS was originally developed in the early 1990s by SGI and has a long history of running on extremely large servers and storage arrays. The features of XFS include: Reliability Metadata journaling, which ensures file system integrity after a system crash by keeping a record of file system operations that can be replayed when the system is restarted and the file system remounted Extensive run-time metadata consistency checking Scalable and fast repair utilities Quota journaling. This avoids the need for lengthy quota consistency checks after a crash. Scalability and performance Supported file system size up to 1024 TiB Ability to support a large number of concurrent operations B-tree indexing for scalability of free space management Sophisticated metadata read-ahead algorithms Optimizations for streaming video workloads Allocation schemes Extent-based allocation Stripe-aware allocation policies Delayed allocation Space pre-allocation Dynamically allocated inodes Other features Reflink-based file copies Tightly integrated backup and restore utilities Online defragmentation Online file system growing Comprehensive diagnostics capabilities Extended attributes ( xattr ). This allows the system to associate several additional name/value pairs per file. Project or directory quotas. This allows quota restrictions over a directory tree. Subsecond timestamps Performance characteristics XFS has a high performance on large systems with enterprise workloads. A large system is one with a relatively high number of CPUs, multiple HBAs, and connections to external disk arrays. XFS also performs well on smaller systems that have a multi-threaded, parallel I/O workload. XFS has a relatively low performance for single threaded, metadata-intensive workloads: for example, a workload that creates or deletes large numbers of small files in a single thread. 1.4. The ext4 file system The ext4 file system is the fourth generation of the ext file system family. It was the default file system in Red Hat Enterprise Linux 6. The ext4 driver can read and write to ext2 and ext3 file systems, but the ext4 file system format is not compatible with ext2 and ext3 drivers. ext4 adds several new and improved features, such as: Supported file system size up to 50 TiB Extent-based metadata Delayed allocation Journal checksumming Large storage support The extent-based metadata and the delayed allocation features provide a more compact and efficient way to track utilized space in a file system. These features improve file system performance and reduce the space consumed by metadata. Delayed allocation allows the file system to postpone selection of the permanent location for newly written user data until the data is flushed to disk. This enables higher performance since it can allow for larger, more contiguous allocations, allowing the file system to make decisions with much better information. File system repair time using the fsck utility in ext4 is much faster than in ext2 and ext3. Some file system repairs have demonstrated up to a six-fold increase in performance. 1.5. Comparison of XFS and ext4 XFS is the default file system in RHEL. This section compares the usage and features of XFS and ext4. Metadata error behavior In ext4, you can configure the behavior when the file system encounters metadata errors. The default behavior is to simply continue the operation. When XFS encounters an unrecoverable metadata error, it shuts down the file system and returns the EFSCORRUPTED error. Quotas In ext4, you can enable quotas when creating the file system or later on an existing file system. You can then configure the quota enforcement using a mount option. XFS quotas are not a remountable option. You must activate quotas on the initial mount. Running the quotacheck command on an XFS file system has no effect. The first time you turn on quota accounting, XFS checks quotas automatically. File system resize XFS has no utility to reduce the size of a file system. You can only increase the size of an XFS file system. In comparison, ext4 supports both extending and reducing the size of a file system. Inode numbers The ext4 file system does not support more than 2 32 inodes. XFS supports dynamic inode allocation. The amount of space inodes can consume on an XFS filesystem is calculated as a percentage of the total filesystem space. To prevent the system from running out of inodes, an administrator can tune this percentage after the filesystem has been created, given there is free space left on the file system. Certain applications cannot properly handle inode numbers larger than 2 32 on an XFS file system. These applications might cause the failure of 32-bit stat calls with the EOVERFLOW return value. Inode number exceed 2 32 under the following conditions: The file system is larger than 1 TiB with 256-byte inodes. The file system is larger than 2 TiB with 512-byte inodes. If your application fails with large inode numbers, mount the XFS file system with the -o inode32 option to enforce inode numbers below 2 32 . Note that using inode32 does not affect inodes that are already allocated with 64-bit numbers. Important Do not use the inode32 option unless a specific environment requires it. The inode32 option changes allocation behavior. As a consequence, the ENOSPC error might occur if no space is available to allocate inodes in the lower disk blocks. 1.6. Choosing a local file system To choose a file system that meets your application requirements, you must understand the target system on which you will deploy the file system. In general, use XFS unless you have a specific use case for ext4. XFS For large-scale deployments, use XFS, particularly when handling large files (hundreds of megabytes) and high I/O concurrency. XFS performs optimally in environments with high bandwidth (greater than 200MB/s) and more than 1000 IOPS. However, it consumes more CPU resources for metadata operations compared to ext4 and does not support file system shrinking. ext4 For smaller systems or environments with limited I/O bandwidth, ext4 might be a better fit. It performs better in single-threaded, lower I/O workloads and environments with lower throughput requirements. ext4 also supports offline shrinking, which can be beneficial if resizing the file system is a requirement. Benchmark your application's performance on your target server and storage system to ensure the selected file system meets your performance and scalability requirements. Table 1.2. Summary of local file system recommendations Scenario Recommended file system No special use case XFS Large server XFS Large storage devices XFS Large files XFS Multi-threaded I/O XFS Single-threaded I/O ext4 Limited I/O capability (under 1000 IOPS) ext4 Limited bandwidth (under 200MB/s) ext4 CPU-bound workload ext4 Support for offline shrinking ext4 1.7. Network file systems Network file systems, also referred to as client/server file systems, enable client systems to access files that are stored on a shared server. This makes it possible for multiple users on multiple systems to share files and storage resources. Such file systems are built from one or more servers that export a set of file systems to one or more clients. The client nodes do not have access to the underlying block storage, but rather interact with the storage using a protocol that allows for better access control. Available network file systems The most common client/server file system for RHEL customers is the NFS file system. RHEL provides both an NFS server component to export a local file system over the network and an NFS client to import these file systems. RHEL also includes a CIFS client that supports the popular Microsoft SMB file servers for Windows interoperability. The userspace Samba server provides Windows clients with a Microsoft SMB service from a RHEL server. 1.8. Shared storage file systems Shared storage file systems, sometimes referred to as cluster file systems, give each server in the cluster direct access to a shared block device over a local storage area network (SAN). Comparison with network file systems Like client/server file systems, shared storage file systems work on a set of servers that are all members of a cluster. Unlike NFS, however, no single server provides access to data or metadata to other members: each member of the cluster has direct access to the same storage device (the shared storage ), and all cluster member nodes access the same set of files. Concurrency Cache coherency is key in a clustered file system to ensure data consistency and integrity. There must be a single version of all files in a cluster visible to all nodes within a cluster. The file system must prevent members of the cluster from updating the same storage block at the same time and causing data corruption. In order to do that, shared storage file systems use a cluster wide-locking mechanism to arbitrate access to the storage as a concurrency control mechanism. For example, before creating a new file or writing to a file that is opened on multiple servers, the file system component on the server must obtain the correct lock. The requirement of cluster file systems is to provide a highly available service like an Apache web server. Any member of the cluster will see a fully coherent view of the data stored in their shared disk file system, and all updates will be arbitrated correctly by the locking mechanisms. Performance characteristics Shared disk file systems do not always perform as well as local file systems running on the same system due to the computational cost of the locking overhead. Shared disk file systems perform well with workloads where each node writes almost exclusively to a particular set of files that are not shared with other nodes or where a set of files is shared in an almost exclusively read-only manner across a set of nodes. This results in a minimum of cross-node cache invalidation and can maximize performance. Setting up a shared disk file system is complex, and tuning an application to perform well on a shared disk file system can be challenging. Available shared storage file systems Red Hat Enterprise Linux provides the GFS2 file system. GFS2 comes tightly integrated with the Red Hat Enterprise Linux High Availability Add-On and the Resilient Storage Add-On. Red Hat Enterprise Linux supports GFS2 on clusters that range in size from 2 to 16 nodes. 1.9. Choosing between network and shared storage file systems When choosing between network and shared storage file systems, consider the following points: NFS-based network file systems are an extremely common and popular choice for environments that provide NFS servers. Network file systems can be deployed using very high-performance networking technologies like Infiniband or 10 Gigabit Ethernet. This means that you should not turn to shared storage file systems just to get raw bandwidth to your storage. If the speed of access is of prime importance, then use NFS to export a local file system like XFS. Shared storage file systems are not easy to set up or to maintain, so you should deploy them only when you cannot provide your required availability with either local or network file systems. A shared storage file system in a clustered environment helps reduce downtime by eliminating the steps needed for unmounting and mounting that need to be done during a typical fail-over scenario involving the relocation of a high-availability service. Red Hat recommends that you use network file systems unless you have a specific use case for shared storage file systems. Use shared storage file systems primarily for deployments that need to provide high-availability services with minimum downtime and have stringent service-level requirements. 1.10. Volume-managing file systems Volume-managing file systems integrate the entire storage stack for the purposes of simplicity and in-stack optimization. Available volume-managing file systems Red Hat Enterprise Linux 8 provides the Stratis volume manager as a Technology Preview. Stratis uses XFS for the file system layer and integrates it with LVM, Device Mapper, and other components. Stratis was first released in Red Hat Enterprise Linux 8.0. It is conceived to fill the gap created when Red Hat deprecated Btrfs. Stratis 1.0 is an intuitive, command line-based volume manager that can perform significant storage management operations while hiding the complexity from the user: Volume management Pool creation Thin storage pools Snapshots Automated read cache Stratis offers powerful features, but currently lacks certain capabilities of other offerings that it might be compared to, such as Btrfs or ZFS. Most notably, it does not support CRCs with self healing.