\section*{Acknowledgments}
We would like to thank our shepherd, Keith Smith, the anonymous
reviewers, and members of the LASR group and the Systems and Storage
Lab for their feedback and guidance. We would like to thank Intel and
ETRI IITP/KEIT[2014-3-00035] for providing access to Optane DC
Persistent Memory for conducting experiments for the paper. This work
was supported by NSF CAREER grant 1751277 and generous donations from
VMware, Google and Facebook.
 Any opinions, findings, and conclusions,
 or recommendations expressed herein are those of the authors and do
 not necessarily reflect the views of other institutions.

\section{Background}
\label{sec-bkgd}

This section provides background on persistent memory (PM), PM file
systems, Direct Access, and memory mapping.

\subsection{Persistent Memory}

Persistent memory is a new memory technology that offers durability
and performance close to that of DRAM. PM can be attached on the
memory bus similar to DRAM, and would be accessed via processor loads
and stores. PM offers 8-byte atomic stores and they become persistent
as soon as they reach the PM controller~\cite{intel_adr}. There are
two ways to ensure that stores become persistent: (i) using non-temporal
store instructions (e.g., \texttt{movnt} in x86) to bypass the cache
hierarchy and reach the PM controller or (ii) using a combination
of regular temporal store instructions and cache line flush instructions
(e.g., \texttt{clflush} or \texttt{clwb} in x86).

Intel DC Persistent Memory is the first PM product that was made
commercially available in April 2019. Table~\ref{tbl-pm} lists the
performance characteristics of PM revealed in a report by Izraelevitz
\textit{et al.}\xspace~\cite{pm-arxiv}. Compared to DRAM, PM has 3.7$\times$\xspace higher latency
for random reads, 2$\times$\xspace higher latency for sequential reads,
$1/3^{rd}$ read bandwidth, and close to $1/6^{th}$ write bandwidth.
Finally, PMs are expected to exhibit limited write endurance (about
$10^7$ write cycles~\cite{pm-wear}).

\input{tbl-pm}

\subsection{Direct Access (DAX) and Memory Mapping}
The Linux ext4 file system introduced a new mode called Direct Access
(DAX) to help users access PM~\cite{ext4-dax}. DAX file systems eschew the use of
page caches and rely on memory mapping to provide low-latency access
to PM.

A memory map operation (performed via the \texttt{mmap()}\xspace system call) in
ext4 DAX maps one or more pages in the process virtual address space
to extents on PM. For example, consider virtual addresses \vtt{4K} to
\vtt{8K-1} are mapped to bytes \vtt{0} to \vtt{4K-1} on file \vtt{foo}
on ext4 DAX. Bytes \vtt{0} to \vtt{4K-1} in \vtt{foo} then correspond
to bytes \vtt{10*4K} to \vtt{11*4K -1} on PM. A store instruction to
virtual address \vtt{5000} would then translate to a store to byte
\vtt{40964} on PM. Thus, PM can be accessed via processor loads and
stores without the interference of software; the virtual memory
subsystem is in charge of translating virtual addresses into
corresponding physical addresses on PM.

While DAX and \texttt{mmap()}\xspace provide low-latency access to PM, they do not
provide other features such as naming or atomicity for operations. The
application is forced to impose its own structure and semantics on the
raw bytes offered by \texttt{mmap()}\xspace. As a result, PM file systems still
provide useful features to applications and end users.

\subsection{PM File Systems}

Apart from ext4 DAX, researchers have developed a number of other PM
file systems such as SCMFS~\cite{scmfs}, BPFS~\cite{bpfs},
Aerie~\cite{aerie}, PMFS~\cite{pmfs}, NOVA~\cite{nova}, and
Strata~\cite{strata}.
Only ext4 DAX, PMFS (now
deprecated), NOVA, and Strata are publicly available and supported by
modern Linux 4.x kernels.

These file systems make trade-offs between software overhead, amount
of write IO, and operation guarantees. NOVA provides strong guarantees
such as atomicity for file-system operations. PMFS provides slightly
weaker guarantees (data operations are not atomic), but as a result
obtains better performance on some workloads. Strata is a cross-media
file system which uses PM as one of its layers. Strata writes all data
to per-process private log, then coalesces the data and copies it to a
shared area for public access. For workloads dominated by operations
such as appends, Strata cannot coalesce the data effectively, and has
to write data twice: once to the private log, and once to the shared
area. This increases the PM wear-out by up to 2$\times$\xspace. All these file
systems still suffer from significant overhead for write operations
(Table~\ref{tbl-mot}).

\if 0

\subsection{Motivation}

\input{fig-mot}

To motivate {\textsc{SplitFS}}\xspace, we study the software overhead of the \texttt{write()}\xspace
POSIX call on different PM file systems. We emulate a PM device with a
write latency of 57ns per cache line\ak{why this number?}, mount each PM file system,
create a large file and update a randomly picked 4KB block.
Figure~\ref{fig-mot} presents the results. Software overhead is
measured as ratio of total time taken to time spend on device IO. All
PM file systems have significant software overhead, ranging from
60--280\%.
\ak{PMFS is really good, relatively speaking. Wonder why they are so much
better than NOVA.}

ext4 DAX, despite not providing any guarantees about data operations,
suffers from 110\% overhead. PMFS provides synchronous operations, and
suffers from 60\% overhead. Strata and NOVA provides synchronous and
atomic operations, but differ significantly in their overhead; NOVA
has 280\% overhead while Strata has 70\% overhead. The overhead for
Strata and NOVA come from logging that is employed to provide strong
guarantees.

While Strata has relatively low software overhead, it suffers from
high write amplification. All data is first written to a
process-private log, and then coalesced and copied over to a public
area. When the workload does not provide opportunities for coalescing,
Strata writes all data twice, resulting in 2$\times$\xspace write
amplification. NOVA and PMFS have lower write amplification, but
suffer from high software overhead.
\ak{PMFS has lower overhead than Strata.}

Thus, no existing PM file system has both low software overhead and
low write amplification, while providing strong guarantees such as
synchronous and atomic operations. Our {\textsc{SplitFS}}\xspace PM file system aims to
fill this need, while balancing these goals with low complexity and
portability.

\fi

\section{Conclusion}
\label{sec-conc}

We present {\textsc{SplitFS}}\xspace, a PM file system built using the split
architecture. {\textsc{SplitFS}}\xspace handles data operations entirely in user-space,
and routes metadata operations through the ext4 DAX PM file
system. {\textsc{SplitFS}}\xspace provides three modes with varying guarantees, and
allows applications running at the same time to use different
modes. {\textsc{SplitFS}}\xspace only requires adding a single system call to the ext4
DAX file system. Evaluating {\textsc{SplitFS}}\xspace with micro-benchmarks and real
applications, we show that it outperforms state-of-the-art PM file
systems like NOVA on many workloads. The design of {\textsc{SplitFS}}\xspace allows
users to benefit from the maturity and constant development of the
ext4 DAX file system, while getting the performance and strong
guarantees of state-of-the-art PM file systems. {\textsc{SplitFS}}\xspace is publicly available at \url{https://github.com/utsaslab/splitfs}.

\section{SplitFS: Design and Implementation}
\label{sec-design}

We present the goals of {\textsc{SplitFS}}\xspace, its three modes and their
guarantees. We present an overview of the design, describe how
different operations are handled, and discuss how {\textsc{SplitFS}}\xspace provides
atomic operations at low overhead. We describe the implementation of
{\textsc{SplitFS}}\xspace, and discuss its various tuning parameters. Finally, we
discuss how the design of {\textsc{SplitFS}}\xspace affects security.

\subsection{Goals}
\vheading{Low software overhead}. {\textsc{SplitFS}}\xspace aims to reduce software
  overhead for data operations, especially writes and appends.

  \vheading{Transparency}. {\textsc{SplitFS}}\xspace does not require the application to
  be modified in any way to obtain lower software overhead and
  increased performance.

  \vheading{Minimal data copying and write IO}. {\textsc{SplitFS}}\xspace aims to reduce
  the number of writes made to PM.  {\textsc{SplitFS}}\xspace aims to avoid copying
  data within the file system whenever possible. This both helps
  performance and reduces wear-out on PM. Minimizing writes is
  especially important when providing strong guarantees like atomic
  operations.

  \vheading{Low implementation complexity}. {\textsc{SplitFS}}\xspace aims to re-use
  existing software like ext4 DAX as much as possible, and reduce the
  amount of new code that must be written and maintained for {\textsc{SplitFS}}\xspace.

  \vheading{Flexible guarantees}. {\textsc{SplitFS}}\xspace aims to provide
  applications with a choice of crash-consistency guarantees to choose
  from. This is in contrast with PM file systems today, which provide
  all running applications with the same set of guarantees.

\subsection{{\textsc{SplitFS}}\xspace Modes and Guarantees}
\label{sec-modes}

{\textsc{SplitFS}}\xspace provides three different modes: POSIX, sync, and strict. Each
mode provides a different set of guarantees. Concurrent applications
can use different modes at the same time as they run on
{\textsc{SplitFS}}\xspace. Across all modes, {\textsc{SplitFS}}\xspace ensures the file system retains
its integrity across crashes.

\input{tbl-modes-2}

Table~\ref{tbl-modes} presents the three modes provided by
{\textsc{SplitFS}}\xspace. Across all modes, appends are atomic in {\textsc{SplitFS}}\xspace; if a
series of appends is followed by \texttt{fsync()}\xspace, the file will be
atomically appended on \texttt{fsync()}\xspace.

\vheading{POSIX mode}. In POSIX mode, {\textsc{SplitFS}}\xspace provides metadata
consistency~\cite{Chidambaram+12-NoFS}, similar to ext4 DAX. The file
system will recover to a consistent state after a crash with respect
to its metadata. In this mode, overwrites are performed in-place and
are synchronous. Note that appends are not synchronous, and require an
\texttt{fsync()}\xspace to be persisted. However, {\textsc{SplitFS}}\xspace in the POSIX mode
guarantees atomic appends, a property not provided by ext4 DAX.  This
mode slightly differs from the standard POSIX semantics: when a file
is accessed or modified, the file metadata will not immediately
reflect that.

\vheading{Sync mode}. {\textsc{SplitFS}}\xspace ensures that on top of POSIX mode
guarantees, operations are also guaranteed to be synchronous. An
operation may be considered complete and persistent once the
corresponding call returns and applications do not need a subsequent
\texttt{fsync()}\xspace. Operations are not atomic in this mode; a crash may leave a
data operation partially completed. No additional crash recovery needs
to be performed by {\textsc{SplitFS}}\xspace in this mode. This mode provides
  similar guarantees to PMFS as well as NOVA without data and metadata
  checksuming and with in-place updates; we term this NOVA
  configuration \emph{NOVA-Relaxed}.

\vheading{Strict mode}. {\textsc{SplitFS}}\xspace ensures that on top of sync mode
guarantees, each operation is also atomic. This is a useful guarantee for applications;
editors can allow atomic changes to the file when the user saves the
file, and databases can remove logging and directly update the
database. This mode does not provide atomicity across system calls
though; so it cannot be used to update two files atomically together.
This mode provides similar guarantees to a NOVA
  configuration we term \emph{NOVA-Strict}: NOVA with copy-on-write
  updates, but without checksums enabled.

\input{tbl-tech}

\vheading{Visibility}. Apart from appends, all {\textsc{SplitFS}}\xspace operations
become immediately visible to all other processes on the system. On
\texttt{fsync()}\xspace, appends are persisted and become visible to the rest of the
system. {\textsc{SplitFS}}\xspace is unique in its visibility guarantees, and takes the
middle ground between ext4 DAX and NOVA where all operations are
immediately visible, and Strata where new files and data updates are
only visible to other processes after the digest operation. Immediate
visibility of changes to data and metadata combined with atomic,
synchronous guarantees removes the need for leases to coordinate
sharing; applications can share access to files as they would on any
other POSIX file system.

\input{fig-design}

\subsection{Overview}
We now provide an overview of the design of {\textsc{SplitFS}}\xspace, and how it uses
various techniques to provide the outlined
guarantees. Table~\ref{tbl-tech} lists the different techniques and
the benefit each technique provides.

\vheading{Split architecture}. As shown in Figure~\ref{fig-design},
{\textsc{SplitFS}}\xspace comprises of two major components, a user-space library
linked to the application called {\textsc{U-Split}}\xspace and a kernel file system
called {\textsc{K-Split}}\xspace. {\textsc{SplitFS}}\xspace services all data operations (\textit{e.g.,}\xspace \texttt{read()}\xspace
and \texttt{write()}\xspace calls) directly in user-space and routes metadata
operations (\textit{e.g.,}\xspace \texttt{fsync()}\xspace, \texttt{open()}\xspace, etc. ) to the kernel file system
underneath. File system crash-consistency is guaranteed at all
times. This approach is similar to Exokernel~\cite{exokernel} where
only the control operations are handled by the kernel and data
operations are handled in user-space.

\vheading{Collection of mmaps}. Reads and overwrites are handled by
\texttt{mmap()}\xspace-ing the surrounding 2 MB part of the file, and serving reads via
\vtt{memcpy} and writes via non-temporal stores (\vtt{movnt}
instructions).  A single logical file may have data present in
multiple physical files; for example, appends are first sent to a
staging file, and thus the file data is spread over the original file
and the staging file. {\textsc{SplitFS}}\xspace uses a \emph{collection of memory-maps}
to handle this situation. Each file is associated with a number of
open \texttt{mmap()}\xspace calls over multiple physical files, and reads and
over-writes are routed appropriately.

\vheading{Staging}. {\textsc{SplitFS}}\xspace uses temporary files called staging files
for both appends and atomic data operations. Appends are first routed
to a staging file, and are later relinked on \texttt{fsync()}\xspace. Similarly,
file overwrites in strict mode are also first sent to staging
files and later relinked to their appropriate files.

\vheading{Relink}. On an \texttt{fsync()}\xspace, all the staged appends of a file must
be moved to the target file; in strict mode, overwrites have to be
moved as well. One way to move the staged appends to the target file
is to allocate new blocks and then copy appended data to them.
However, this approach leads to write amplification and high overhead.
To avoid these unnecessary data copies, we developed a new primitive
called \emph{relink}. Relink logically moves PM blocks from the
staging file to the target file without incurring any copies.

Relink has the following signature: \texttt{\small relink(file1,
  offset1, file2, offset2, size)}. Relink atomically moves data from
\texttt{offset1} of \vtt{file1} to \vtt{offset2} of \vtt{file2}. If
\vtt{file2} already has data at \vtt{offset2}, existing data blocks
are de-allocated. Atomicity is ensured by wrapping the changes in a
ext4 journal transaction. Relink is a metadata operation, and does not
involve copying data when the involved \vtt{offsets} and \vtt{size}
are block aligned. When \vtt{offset1} or \vtt{offset2} happens to be
in the middle of a block, {\textsc{SplitFS}}\xspace copies the partial data for that
block to \vtt{file2}, and performs a metadata-only relink for the rest
of the data. Given that {\textsc{SplitFS}}\xspace is targeted at POSIX applications,
block writes and appends are often block-aligned by the
applications. Figure~\ref{fig-relink} illustrates the different steps
involved in the relink operation.

\input{fig-relink}

\vheading{Optimized logging}. In strict mode, {\textsc{SplitFS}}\xspace guarantees
atomicity for all operations.  To provide atomicity, we employ an
\emph{Operation Log} and use logical redo logging to record the intent
of each operation. Each {\textsc{U-Split}}\xspace instance has its own operation
  log that is pre-allocated, \texttt{mmap}-ed\xspace by {\textsc{U-Split}}\xspace, and written using
  non-temporal store instructions. We use the necessary memory fence
instructions to ensure that log entries persist in the correct
order. To reduce the overheads from logging, we ensure that in the
common case, per operation, we write one cache line (64B) worth of
data to PM and use a single memory fence (\texttt{sfence}\xspace in x86) instruction
in the process. Operation log entries do not contain the file data
associated with the operation (\textit{e.g.,}\xspace data being appended to a file),
instead they contain a logical pointer to the staging file where the
data is being held.

We employ a number of techniques to optimize logging. First, to
distinguish between valid and invalid or torn log entries, we
incorporate a 4B transactional checksum~\cite{ironfs} within the 64B
log entry. The use of checksum reduces the number of fence
instructions necessary to persist and validate a log entry from two to
one. Second, we maintain a tail for the log in DRAM and concurrent
threads use the tail as a synchronization variable. They use
compare-and-swap to atomically advance the tail and write to their
respective log entries concurrently. Third, during the initialization
of the operation log file, we zero it out. So, during crash recovery,
we identify all non-zero 64B aligned log entries as being potentially
valid and then use the checksum to identify any torn entries. The rest
are valid entries and are replayed. Replaying log entries is
idempotent, so replaying them multiple times on crashes is safe. We
employ a 128MB operation log file and if it becomes full, we
checkpoint the state of the application by calling \emph{relink()}
  on all the open files that have data in staging files. We then zero
out the log and reuse it. Finally, we designed our logging mechanism
such that all common case operations (\texttt{write()}\xspace, \texttt{open()}\xspace, etc.) can
be logged using a single 64B log entry while some uncommon operations,
like \texttt{rename()}\xspace, require multiple log entries.

Our logging protocol works well with the {\textsc{SplitFS}}\xspace
  architecture. The tail of each {\textsc{U-Split}}\xspace log is maintained only in DRAM
  as it is not required for crash recovery. Valid log entries are
  instead identified using checksums. In contrast, file systems
  such as NOVA have a log per inode that resides on PM, whose tail is
  updated after each operation via expensive \texttt{clflush}\xspace and \texttt{sfence}\xspace
  operations.

\vheading{Providing Atomic Operations}. In strict mode, {\textsc{SplitFS}}\xspace
provides synchronous, atomic operations. Atomicity is provided in an
efficient manner by the combination of staging files, relink, and
optimized logging.  Atomicity for data operations like overwrites is
achieved by redirecting them also to a staging file, similar to how
appends are performed.  {\textsc{SplitFS}}\xspace logs these writes and appends to
record where the latest data resides in the event of a crash.  On
\texttt{fsync()}\xspace, {\textsc{SplitFS}}\xspace relinks the data from the staging file to the
target file atomically. Once again, the data is written exactly once,
though {\textsc{SplitFS}}\xspace provides the strong guarantee of atomic data
operations. Relink allows {\textsc{SplitFS}}\xspace to implement a form of localized
copy-on-write. Due to the staging files being pre-allocated, locality
is preserved to an extent. {\textsc{SplitFS}}\xspace logs metadata operations to ensure
they are atomic and synchronous. Optimized logging ensures that for
most operations exactly one cache line is written and one \vtt{sfence}
is issued for logging.

\subsection{Handling Reads, Overwrites, and Appends}

\vheading{Reads}. Reads consult the collection of mmaps to determine
where the most recent data for this offset is, since the data could
have been overwritten or appended (and thus in a staging file). If a
valid memory mapped region for the offsets being read exists in {\textsc{U-Split}}\xspace,
the read is serviced from the corresponding region.  If such a region
does not exist, then the 2 MB region surrounding the read offset is
first memory mapped, added to the the collection of mmaps, and then
the read operation is serviced using processor loads.

\vheading{Overwrites}. Similar to reads, if the target offset is
already memory mapped, then {\textsc{U-Split}}\xspace services the overwrite using
non-temporal store instructions. If the target offset is not memory
mapped, then the 2MB region surrounding the offset is first memory
mmaped, added to the collection of mmaps, and then the overwrite is
serviced. However, in strict mode, to guarantee atomicity, overwrites
are first redirected to a staging file (even if the offset is memory
mapped), then the operation is logged, and finally relinked on a
subsequent \texttt{fsync()}\xspace or \texttt{close()}\xspace.

\vheading{Appends}. {\textsc{SplitFS}}\xspace redirects all appends to a staging file,
and performs a relink on a subsequent \texttt{fsync()}\xspace or \texttt{close()}\xspace. As
with overwrites, appends are performed with non-temporal writes and in
strict mode, {\textsc{SplitFS}}\xspace also logs details of the append operation to
ensure atomicity.

\subsection{Implementation}

We implement {\textsc{SplitFS}}\xspace as a combination of a user-space library file
system (9K lines of C code) and a small patch to ext4 DAX to add the
relink system call (500 lines of C code). {\textsc{SplitFS}}\xspace supports 35 common
POSIX calls, such as \vtt{pwrite()}, \vtt{pread64()}, \vtt{fread()},
\vtt{readv()}, \vtt{ftruncate64()}, \vtt{openat()}, etc; we found that
supporting this set of calls is sufficient to support a
variety of applications and micro-benchmarks. Since PM file systems
PMFS and NOVA are supported by Linux kernel version 4.13, we modified
4.13 to support {\textsc{SplitFS}}\xspace. We now present other details of our
implementation.

\vheading{Intercepting POSIX calls}. {\textsc{SplitFS}}\xspace uses {\small\texttt{LD\_PRELOAD}}\xspace to
intercept POSIX calls and either serve from user-space or route them
to the kernel after performing some book-keeping tasks.
Since {\textsc{SplitFS}}\xspace intercepts calls at the POSIX
level in \vtt{glibc} rather than at the system call level, {\textsc{SplitFS}}\xspace
has to intercept several variants of common system calls like
\vtt{write()}.

\vheading{Relink}. We implement relink by leveraging an \vtt{ioctl}
provided by ext4 DAX. The \vtt{EXT4\_IOC\_MOVE\_EXT} \vtt{ioctl}\xspace swaps
extents between a source file and a destination file, and uses
journaling to perform this atomically. The \vtt{ioctl} also
de-allocates blocks in the target file if they are replaced by blocks
from the source file. By default, the \vtt{ioctl}\xspace also flushes the swapped
data in the target file; we modify the \vtt{ioctl}\xspace to only touch metadata,
without copying, moving, or persisting of data. We also ensure that
after the swap has happened, existing memory mappings of both source
and destination files are valid; this is vital to {\textsc{SplitFS}}\xspace performance
as it avoids page faults. The \vtt{ioctl}\xspace requires blocks to be
  allocated at both source and destination files. To satisfy this
  requirement, when handling appends via relink, we allocate blocks at
  the destination file, swap extents from the staging file, and then
  de-allocate the blocks. This allows us to perform relink without
  using up extra space, and reduces implementation complexity at the
  cost of temporary allocation of data.

\vheading{Handling file open and close}. On file open, {\textsc{SplitFS}}\xspace
performs \texttt{stat()}\xspace on the file and caches its attributes in user-space
to help handle later calls. When a file is closed, we do not clear its
cached information. When the file is unlinked, all cached metadata is
cleared, and if the file has been memory-mapped, it is un-mapped.  The
cached attributes are used to check file permissions on every
subsequent file operation (\textit{e.g.,}\xspace \texttt{read()}\xspace) intercepted by {\textsc{U-Split}}\xspace.

\vheading{Handling fork}. Since {\textsc{SplitFS}}\xspace uses a user-space library
file system, special care needs to be taken to handle \vtt{fork()}\xspace and
\texttt{execve()}\xspace correctly. When \vtt{fork()}\xspace is called, {\textsc{SplitFS}}\xspace is copied into the address
space of the new process (as part of copying the address space of the
parent process), so that the new process can continue to access
{\textsc{SplitFS}}\xspace.

\vheading{Handling execve}. \texttt{execve()}\xspace overwrites the address space,
but open file descriptors are expected to work after the call
completes. To handle this, {\textsc{SplitFS}}\xspace does the following: before
executing \texttt{execve()}\xspace, {\textsc{SplitFS}}\xspace copies its in-memory data about open
files to a shared memory file on \vtt{/dev/shm}; the file name is the
process ID. After executing \texttt{execve()}\xspace, {\textsc{SplitFS}}\xspace checks the shared
memory device and copies information from the file if it exists.

\vheading{Handling dup}. When a file descriptor is duplicated, the
file offset is changed whenever operations are performed on either
file descriptor. {\textsc{SplitFS}}\xspace handles by maintaining a single offset per
open file, and using pointers to this file in the file descriptor
maintained by {\textsc{SplitFS}}\xspace. Thus, if two threads \vtt{dup} a file
descriptor and change the offset from either thread, {\textsc{SplitFS}}\xspace ensures
both threads see the changes.

\vheading{Staging files}. {\textsc{SplitFS}}\xspace pre-allocates staging files at
startup, creating 10 files each 160 MB in size. Whenever a staging
file is completely utilized, a background thread wakes up and creates
and pre-allocates a new staging file. This avoids the overhead of
creating staging files in the critical path.

\vheading{Cache of memory-mappings}. {\textsc{SplitFS}}\xspace caches all
memory-mappings its creates in its collection of memory mappings. A
memory-mapping is only discarded on \vtt{unlink()}. This reduces the
cost of setting up memory mappings in the critical path on read or
write.

\vheading{Multi-thread access}. {\textsc{SplitFS}}\xspace uses a lock-free queue for
managing the staging files. It uses fine-grained reader-writer locks to
protect its in-memory metadata about open files, inodes, and
memory-mappings.

\subsection{Tunable Parameters}

{\textsc{SplitFS}}\xspace provides a number of tunable parameters that
can be set by application developers and users for each
{\textsc{U-Split}}\xspace instance. These parameters affect the performance of
{\textsc{SplitFS}}\xspace.

\vheading{\texttt{mmap()}\xspace size}. {\textsc{SplitFS}}\xspace supports a configurable
size of \texttt{mmap()}\xspace for handling overwrites and reads.  Currently,
     {\textsc{SplitFS}}\xspace supports \texttt{mmap()}\xspace sizes ranging from 2MB to 512MB. The
     default size is 2 MB, allowing {\textsc{SplitFS}}\xspace to employ huge pages
     while pre-populating the mappings.

     \vheading{Number of staging files at startup}. There are ten
     staging files at startup by default; when a staging file is used
     up, {\textsc{SplitFS}}\xspace creates another staging file in the background. We
     experimentally found that having ten staging files provides a
     good balance between application performance and the
     initialization cost and space usage of staging files.

     \vheading{Size of the operation log}. The default size of the
     operation log is 128MB for each {\textsc{U-Split}}\xspace instance. Since all log
     entries consist of a single cacheline in the common case,
     {\textsc{SplitFS}}\xspace can support up to 2M operations without clearing the log
     and re-initializing it. This helps applications with small bursts
     to achieve good performance while getting strong semantics.

\subsection{Security}

    {\textsc{SplitFS}}\xspace does not expose any new security vulnerabilities as
    compared to an in-kernel file system.  All metadata operations are
    passed through to the kernel which performs security
    checks. {\textsc{SplitFS}}\xspace does not allow a user to open, read, or write a
    file to which they previously did not have permissions. The
    {\textsc{U-Split}}\xspace instances are isolated from each other in separate
    processes; therefore applications cannot access the data of other
    applications while running on {\textsc{SplitFS}}\xspace. Each {\textsc{U-Split}}\xspace instance only
    stores book-keeping information in DRAM for the files that the
    application already has access to. An application that uses
    {\textsc{SplitFS}}\xspace may corrupt its own files, just as in an in-kernel file
    system.

\section{Discussion}
\label{sec-discussion}

We reflect on our experiences building {\textsc{SplitFS}}\xspace, describe problems we
encountered, how we solved them, and surprising insights that we
discovered.

\vheading{Page faults lead to significant cost}. {\textsc{SplitFS}}\xspace memory maps
files before accessing them, and uses \vtt{MAP\_POPULATE} to pre-fault
all pages so that later reads and writes do not incur page-fault
latency. As a result, we find that a significant portion of the time
for \texttt{open()}\xspace is consumed by page faults. While the latency of device
IO usually dominates page fault cost in storage systems based on solid
state drives or magnetic hard drives, the low latency of persistent
memory highlights the cost of page faults.

\vheading{Huge pages are fragile}. A natural way of minimizing page
faults is to use 2 MB huge pages. However, we found huge pages fragile
and hard to use. Setting up a huge-page mapping in the Linux kernel
requires a number of conditions. First, the virtual address must be 2
MB aligned. Second, the physical address on PM must be 2 MB
aligned. As a result, fragmentation in either the virtual address
space or the physical PM prevents huge pages from being created. For
most workloads, after a few thousand files were created and deleted,
fragmenting PM, we found it impossible to create any new huge
pages. Our collection-of-mappings technique sidesteps this problem by
creating huge pages at the beginning of the workload, and reusing them
to serve reads and writes. Without huge pages, we observed read
performance dropping by 50\% in many workloads. We believe this is a
fundamental problem that must be tackled since huge pages are crucial
for accessing large quantities of PM.

\vheading{Avoiding work in the critical path is important}. Finally,
we found that a general design technique that proved crucial for
{\textsc{SplitFS}}\xspace is simplifying the critical path. We pre-allocate wherever
possible, and use a background thread to perform pre-allocation in the
background. Similarly, we pre-fault memory mappings, and use a cache
to re-use memory mappings as much as possible. {\textsc{SplitFS}}\xspace rarely
performs heavyweight work in the critical path of a data
operation. Similarly, even in strict mode, {\textsc{SplitFS}}\xspace optimizes logging,
trading off shorter recovery time for a simple, low overhead logging
protocol. We believe this design principle will be useful for other
systems designed for PM.

\vheading{Staging writes in DRAM}. An alternate design that we tried
was staging writes in DRAM instead of on PM. While DRAM staging files
incur less allocation costs than PM staging files, we
found that the cost of copying data from DRAM to PM on \texttt{fsync()}\xspace
overshadowed the benefit of staging data in DRAM. In general, DRAM
buffering is less useful in PM systems because PM and DRAM performances are similar.

\vheading{Legacy applications need to be rewritten to take maximum
  advantage of PM}. We observe that the applications we evaluate such
as LevelDB spent a significant portion of their time ($60-80$\%)
performing POSIX calls on current PM file systems. {\textsc{SplitFS}}\xspace is able to
reduce this percentage down to 46-50\%, but further reduction in
software overhead will have negligible impact on application runtime
since the majority of the time is spent on application
code. Applications would need to be rewritten from scratch to use
libraries like \vtt{libpmem} that exclusively operate on data
structures in \texttt{mmap()}\xspace to take further advantage of PM.

\section{Evaluation}
\label{sec-eval}

In this section, we use a number of microbenchmarks and applications
to evaluate {\textsc{SplitFS}}\xspace in relation to state-of-the-art PM filesystems
like ext4 DAX\xspace, NOVA, and PMFS. While comparing these different file
systems, we seek to answer the following questions:

\begin{itemize}
[topsep=2pt,itemsep=2pt,parsep=2pt,before=\vspace{10pt},after=\vspace{10pt}]
	\item How does {\textsc{SplitFS}}\xspace affect the performance of different
	system calls as compared to ext4 DAX\xspace?
	(\S\ref{sec-sys-call})
	\item How do the different techniques employed in {\textsc{SplitFS}}\xspace
	contribute to overall performance? (\S\ref{sec-factor})
	\item How does {\textsc{SplitFS}}\xspace compare to other file systems for
	different PM access patterns? (\S\ref{sec-micro})
	\item Does {\textsc{SplitFS}}\xspace reduce file-system software overhead as
	compared to other PM file systems? (\S\ref{sec-sw-ovhd})
	\item How does {\textsc{SplitFS}}\xspace compare to other file systems for
      real-world applications? (\S\ref{sec-real-di-apps} \&
      \S\ref{sec-real-mh-apps})
	\item What are the compute and storage overheads incurred when using {\textsc{SplitFS}}\xspace? (\S\ref{sec-resource})
\end{itemize}

We first briefly describe our experimental methodology (\S\ref{sec-expsetup} \& \S\ref{sec-workloads}) before addressing each of the above questions.

\subsection{Experimental Setup}
\label{sec-expsetup}

\cameditstwo{We evaluate the performance of {\textsc{SplitFS}}\xspace against other PM file systems on Intel Optane DC Persistent Memory Module (PMM). The experiments are performed on a 2-socket, 96-core machine with 768 GB PMM, 375 GB DRAM, and 32 MB Last Level Cache (LLC).} We run all evaluated
file systems on the 4.13 version of the Linux kernel (Ubuntu
16.04). We run each experiment multiple times and report the mean. In
all cases, the standard deviation was less than five percent of the
mean, and the experiments could be reliably repeated.

\begin{comment}
\vheading{Emulating PM}. Given that commercial PMs only become
available in April 2019 (a few weeks before this paper was submitted),
we use DRAM to emulate PM. We employ the same PM emulation framework
used by Strata~\cite{strata}. We inject a delay of 220ns on every
\texttt{read()}\xspace system call, to emulate the access latencies of the PM
hardware. We do not add this fixed 220ns delay for writes, because
writes do not go straight to PM in the critical path, but only to the
memory controller. We add bandwidth-modeling delays for reads as well
as writes to emulate a memory device with $1/3^{rd}$ the bandwidth of
DRAM, an expected characteristic of PMs~\cite{pm-arxiv}. While this
emulation approach is far from perfect, we observe that the resulting
memory access characteristics are inline with the expected behavior of
PMs~\cite{strata}.

\vheading{Validating results on real PM}. We were able to briefly get
access to a machine with Intel DC Persistent Memory Module (PMM). The
experiments are performed on a 2-socket, 96-core machine with 768 GB
PMM, 375 GB DRAM, and 32 MB Last Level Cache(LLC).
\end{comment}

\subsection{Workloads}
\label{sec-workloads}

\input{tbl-app-desc.tex}

We used two key-value stores (Redis, LevelDB), an embedded database
(SQLite), and three utilities (tar, git, rsync) to evaluate the
performance of {\textsc{SplitFS}}\xspace. Table~\ref{tbl-apps-desc} lists the
applications and their characteristics.

\vheading{TPC-C on SQLite}. TPC-C is an online transaction processing
benchmark. It has five different types of transactions each with
different ratios of reads and writes. We run SQLite v3.23.1 with
{\textsc{SplitFS}}\xspace, and measured the performance of TPC-C on SQLite in the
Write-Ahead-Logging (WAL) mode.

\vheading{YCSB on LevelDB}. The Yahoo Cloud Serving
Benchmark~\cite{cooper2010benchmarking} has six different key-value
store benchmarks, each with different read/write ratios. We run the
YCSB workloads on the LevelDB key-value stores. We set the
\vtt{sstable} size to 64 MB as recommended in Facebook's tuning
guide~\cite{rocksdb-tuning}.

\vheading{Redis}. We set 1M key-value pairs in Redis~\cite{redis}, an
in-memory key-value store. We ran Redis in the Append-Only-File mode,
where it logs updates to the database in a file and performs \texttt{fsync()}\xspace on
the file every second.

\vheading{Utilities}. We also evaluated the performance of {\textsc{SplitFS}}\xspace
for tar, git, and rsync. With git, we measured the time taken for
\vtt{git add} and \vtt{git commit} of all files in the Linux kernel
ten times.  With rsync, we copy a 7 GB dataset of 1200 files with
characteristics similar to backup datasets~\cite{backup} from one PM
location to another. With tar, we compressed the Linux kernel 4.18
along with the files from the backup dataset.

\subsection{Correctness and recovery}

\vheading{Correctness}. First, to validate the functional correctness
of {\textsc{SplitFS}}\xspace we run various micro-benchmarks and real-world
applications and compare the resulting file-system state to the ones
obtained with ext4 DAX\xspace. We observe that the file-system states obtained
with ext4 DAX\xspace and {\textsc{SplitFS}}\xspace are equivalent, validating how {\textsc{SplitFS}}\xspace
  handles POSIX calls in its user-space library file system.

\vheading{Recovery times}. Crash recovery in POSIX and sync modes of
{\textsc{SplitFS}}\xspace do not require anything beyond allowing the underlying ext4
DAX file system to recover. In strict mode however, all valid log
entries in the operation log need to be replayed on top of ext4 DAX\xspace
recovery.  This additional log replay time depends on the number and
type of valid log entries in the log. To estimate the additional time
needed for recovery, we crash our real-world workloads at random
points in their execution and measure the log replay time. In our
crash experiments, the maximum number of log entries to be replayed
was 18,000 and that took about 3 seconds on emulated PM (emulation
details in \sref{sec-strata}). In a worst-case micro-benchmark where
we perform cache-line sized writes and crash with 2M (128MB of data)
valid log entries, we observed a log replay time of 6 seconds on
emulated PM.

\subsection{{\textsc{SplitFS}}\xspace system call overheads}
\label{sec-sys-call}

The central premise of {\textsc{SplitFS}}\xspace is that it is a good trade-off to
accelerate data operations at the expense of metadata operations.
Since data operations are more prevelant, this optimization improves
overall application performance.  To validate this premise, we
construct a micro-benchmark similar to FileBench
Varmail~\cite{tarasov2016filebench} that issues a variety of data and
metadata operations. The micro-benchmark first creates and appends
16KB to a file (as four appends, each followed by an \texttt{fsync()}\xspace), closes
it, opens it again, read the whole file as one read call, closes it,
then opens and closes the file once more, and finally deletes the
file. The multiple open and close calls were introduced to account for
the fact that their latency varies over time. Opening a file for the
first time takes longer than opening a file that we recently closed,
due to file metadata caching inside
{\textsc{U-Split}}\xspace. Table~\ref{tbl-micro-syscall} shows the latencies we observed
for different system calls and they are reported for all the three
modes provided by {\textsc{SplitFS}}\xspace and for ext4 DAX\xspace on which {\textsc{SplitFS}}\xspace was
built.

\input{tbl-micro-syscall}

We make three observations based on these results. First, data
operations on {\textsc{SplitFS}}\xspace are significantly faster than on
ext4 DAX\xspace. Writes especially are 3--4$\times$\xspace faster. Second, metadata
operations (\textit{e.g.,}\xspace \texttt{open()}\xspace, \texttt{close()}\xspace, etc.) are slower on {\textsc{SplitFS}}\xspace than
on ext4 DAX\xspace, as {\textsc{SplitFS}}\xspace has to setup its own data structures in
addition to performing the operation on ext4 DAX\xspace. \rk {In {\textsc{SplitFS}}\xspace, \texttt{unlink()}\xspace is an expensive operation because the file mappings that are created for serving reads and overwrites need to be unmapped in the \texttt{unlink()}\xspace wrapper.} Third, as the
consistency guarantees provided by {\textsc{SplitFS}}\xspace get stronger, the syscall
latency generally increases. This increase can be attributed to more
work {\textsc{SplitFS}}\xspace has to do (\textit{e.g.,}\xspace logging in strict mode) for each system
call to provide stronger guarantees. Overall, {\textsc{SplitFS}}\xspace achieves its
objective of accelerating data operations albeit at the expense of
metadata operations.

\subsection{{\textsc{SplitFS}}\xspace performance breakdown}
\label{sec-factor}

\input{fig-factor}

We examine how the various techniques employed by {\textsc{SplitFS}}\xspace contribute
to overall performance.  We use two write-intensive microbenchmarks:
sequential 4KB overwrites and 4KB appends. An \texttt{fsync()}\xspace is issued every
ten operations.  Figure~\ref{fig-factor} shows how individual
techniques introduced one after the other improve performance.

\vheading{Sequential overwrites}. {\textsc{SplitFS}}\xspace increases sequential
overwrite performance by more than 2$\times$\xspace compared to ext4 DAX since overwrites are served from
user-space via processor stores. However, further optimizations like
handling appends using staging files and relink have negligible impact
on this workload as it does not issue any file append operations.

\vheading{Appends}. The split architecture does not accelerate appends since without staging files or relink all appends go to ext4 DAX\xspace as they are metadata operations.  Just introducing staging files to buffer appends improves performance by about 2$\times$\xspace. In this setting, even though appends are serviced in user-space, overall performance is bogged down by expensive data copy operations on \texttt{fsync()}\xspace.  Introducing the relink primitive to this setting eliminates data copies and increases application throughput by 5$\times$\xspace.

\subsection{Performance on different IO patterns}
\label{sec-micro}

\input{fig-micro-all}

To understand the relative merits of different PM file systems, we
compare their performance on microbenchmarks performing different file
IO patterns: sequential reads, random reads, sequential writes, random
writes, and appends. Each benchmark reads/writes an entire 128MB file
in 4KB operations. We compare file systems providing the same
guarantees: {\textsc{SplitFS}}\xspace-POSIX with ext4 DAX\xspace, {\textsc{SplitFS}}\xspace-sync with PMFS, and
{\textsc{SplitFS}}\xspace-strict with Nova-strict and
Strata. Figure~\ref{fig-micro-all} captures the performance of these
file systems for the different micro-benchmarks.

\vheading{POSIX mode}. {\textsc{SplitFS}}\xspace is able to reduce the execution times
of ext4 DAX\xspace by at least 27\% and as much as 7.85$\times$\xspace (sequential reads and
appends respectively).  Read-heavy workloads present fewer improvement
opportunities for {\textsc{SplitFS}}\xspace as file read paths in the kernel are
optimized in modern PM file systems. However, write paths are much
more complex and longer, especially for appends. So, servicing a write
in user-space has a higher payoff than servicing a read, an
observation we already made in Table~\ref{tbl-micro-syscall}.

\vheading{Sync mode}. Compared to PMFS, {\textsc{SplitFS}}\xspace improves the
performance for write workloads (by as much as 2.89$\times$\xspace) and increases performance for read workloads (by as much
as 56\%). Similar to ext4 DAX\xspace, {\textsc{SplitFS}}\xspace's ability to not incur
expensive write system calls translates to its superior performance
for the write workloads.

\begin{comment}
PMFS has the fastest (and leanest) read system call
of all the existing PM file systems, with about 22\% of system
call being
spent in accessing the PM device and only the rest is software
overhead introduced by PMFS. In comparison, {\textsc{SplitFS}}\xspace performs
worse as it incurs expensive \texttt{mmap()}\xspace system calls, the cost of
which is not amortized over the number of reads being performed
to an \texttt{mmap()}\xspace-ed region. If the same 4KB page is read multiple times,
the \texttt{mmap()}\xspace cost will be amortized, and {\textsc{SplitFS}}\xspace will deliver better
performance for subsequent reads.

In contrast, for the write workloads, {\textsc{SplitFS}}\xspace improves performance
over PMFS, by 37\% for appends. As with ext4 DAX\xspace, the write path
is relatively longer in PMFS and hence {\textsc{SplitFS}}\xspace is able to
quickly amortize \texttt{mmap()}\xspace costs and deliver better performance.
Furthermore, {\textsc{SplitFS}}\xspace performs better (in relative terms) for
microbenchmarks with random patterns than for sequential patterns.
Since our benchmarks read/write to the entire file, {\textsc{SplitFS}}\xspace
incurs the same number of \texttt{mmap()}\xspace operations in both sequential
and random cases.
\end{comment}

\vheading{Strict mode}. NOVA, Strata, and {\textsc{SplitFS}}\xspace in this mode
provide atomicity guarantees to all operations and perform the
necessary logging. As can be expected, the overheads of logging result
in reduced performance compared to file systems in other
modes. Overall, {\textsc{SplitFS}}\xspace improves the performance over NOVA by up to 5.8$\times$\xspace
on the random writes workload. This improvement stems from {\textsc{SplitFS}}\xspace's
superior logging which incurs half the number of log writes and fence
operations than NOVA.

\begin{comment}
For example, in a micro-benchmark appending 64B
of data to file, we found that {\textsc{SplitFS}}\xspace's logging performs 4$\times$\xspace
better than NOVA's logging.
\end{comment}

\subsection{Reducing software overhead}
\label{sec-sw-ovhd}

\input{fig-sw-ovhd-apps}

The central premise of {\textsc{SplitFS}}\xspace is that it is possible to accelerate
applications by reducing file system software overhead. We define
file-system software overhead as the time taken to service a
file-system call minus the time spent actually accessing data on the
PM device.  For example, if a system call takes 100 $\mu$s\xspace to be serviced,
of which only 25 $\mu$s\xspace were spent read or writing to PM, then we say that
the software overhead is 75 $\mu$s\xspace. To provide another example, for
appending 4 KB (which takes 10 $\mu$s\xspace to write to PM), if file system A
writes 10 metadata items (incurring 100 $\mu$s\xspace) while file system B
writes two metadata items (incurring 20 $\mu$s\xspace), file-system B will have
lower overhead. In addition to avoiding kernel traps of system calls,
the different techniques discussed in \S\ref{sec-design} help {\textsc{SplitFS}}\xspace
reduce its software overhead. Minimizing software overhead allows
applications to fully leverage PMs.

Figure~\ref{fig-sw-ovhd-apps} \cameditstwo{highlights the relative software
overheads incurred by different file systems compared to {\textsc{SplitFS}}\xspace
providing the same level of guarantees. We present results for three
write-heavy workloads, LevelDB running YCSB Load A and Run A, and
SQLite running TPCC. ext4 DAX\xspace and NOVA (in relaxed mode) suffer the
highest relative software overheads, up to 3.6$\times$\xspace and 7.4$\times$\xspace
respectively. NOVA-Relaxed incurs the highest software overhead for TPCC because it has to update the per-inode logical log entries on overwrites before updating the data in-place. On the other hand, {\textsc{SplitFS}}\xspace-sync can directly perform in-place data updates, and thus has significantly lower software overhead. PMFS suffers the lowest relative software overhead,
capping off at 1.9$\times$\xspace for YCSB Load A and Run A. Overall, {\textsc{SplitFS}}\xspace incurs the lowest
software overhead.}

\subsection{Performance on data-intensive workloads}
\label{sec-real-di-apps}
\label{sec-strata}

\input{fig-real-apps}

Figure~\ref{fig-real-apps} summarizes the performance of various
applications on different file systems. The
performance metric we use for these data intensive workloads (LevelDB
with YCSB, Redis with 100\% writes, and SQLite with TPCC) is
throughput measured in KOps/s. For each mode of consistency guarantee
(POSIX, sync, and strict), we compare {\textsc{SplitFS}}\xspace to state-of-the-art PM file systems. We report the absolute
performance for the baseline file system in each category and
relative throughput for {\textsc{SplitFS}}\xspace. Despite our
  best efforts, we were not able to run Strata on these large applications; other researchers have also reported problems
  in evaluating Strata~\cite{flex}. We evaluated Strata
  with a smaller-scale YCSB workload using a 20GB private log.

Overall, {\textsc{SplitFS}}\xspace outperforms other PM file systems (when providing
similar consistency guarantees) on all data-intensive workloads by as
much as \cameditstwo{2.70$\times$\xspace}. We next present a breakdown of these numbers for
different guarantees.

\vheading{POSIX mode}. \cameditstwo{{\textsc{SplitFS}}\xspace outperforms ext4 DAX\xspace in all
workloads. Write-heavy workloads like RunA (2$\times$\xspace), LoadA (89\%), LoadE
(91\%), Redis (27\%), etc.  benefit the most with {\textsc{SplitFS}}\xspace. {\textsc{SplitFS}}\xspace
speeds up writes and appends the most, so write-heavy workloads
benefit the most from {\textsc{SplitFS}}\xspace. {\textsc{SplitFS}}\xspace outperforms ext4 DAX\xspace on
read-dominated workloads, but the margin of improvement is lower.}

\vheading{Sync and strict mode}. \cameditstwo{{\textsc{SplitFS}}\xspace outperforms sync-mode file
systems PMFS and NOVA (relaxed) and strict-mode file system NOVA
(strict) for all the data intensive workloads. Once again, its the
write-heavy workloads that show the biggest boost in performance. For
example, {\textsc{SplitFS}}\xspace in sync mode outperforms NOVA (relaxed) and PMFS by
2$\times$\xspace and 30\% on RunA and in strict mode outperforms NOVA (strict) by
2$\times$\xspace.  Ready-heavy workloads on the other hand do not show much
improvement in performance.}

\input{tbl-splitfs-strata.tex}

\vheading{Comparison with Strata}. We were able to reliably evaluate
Strata (employing a 20 GB private log) using LevelDB running
smaller-scale YCSB workloads (1M records, and 1M ops for workloads
A--D and F, 500K ops for workload E). We were unable to run Strata on
Intel DC Persistent Memory. Hence, we use DRAM to emulate PM. We
employ the same PM emulation framework used by Strata. We inject a
delay of 220ns on every \texttt{read()}\xspace system call, to emulate the access
latencies of the PM hardware. We do not add this fixed 220ns delay for
writes, because writes do not go straight to PM in the critical path,
but only to the memory controller. We add bandwidth-modeling delays
for reads as well as writes to emulate a memory device with $1/3^{rd}$
the bandwidth of DRAM, an expected characteristic of
PMs~\cite{pm-arxiv}. While this emulation approach is far from
perfect, we observe that the resulting memory access characteristics
are inline with the expected behavior of PMs~\cite{strata}. {\textsc{SplitFS}}\xspace
outperforms Strata on all workloads, by 1.72$\times$\xspace--2.25$\times$\xspace as shown in
Table~\ref{tbl-splitfs-strata}.

\begin{comment}
\subsection{Low software overhead}
\label{sec-sw-ovhd}

\ak{Need to rewrite with microbenchmarks instead of real worklaods.}

\input{tbl-syscall-overhead}

When compared to in-kernel file systems like NOVA, PMFS, and
ext4 DAX\xspace the promise of {\textsc{SplitFS}}\xspace is that it reduces the
software overheads by reduces expensive file system calls
and instead services them from user-space.
Table~\ref{tbl-syscall-overhead} shows the \emph{relative system call
software overheads by different file systems as compared to
{\textsc{SplitFS}}\xspace providing the same level of consistency guarantees}.
We define file system software overhead as the time consumed
by a system call minus the time taken to access PM. For,
example, if a write system call takes 100us to service out
of which only 25us were spent writing to PM, we say that the
software overhead for that system call is 100--25=75us.
The main takeaways from the numbers are:
\begin{itemize}
	\item Overall, different file systems have much
	higher software overheads than {\textsc{SplitFS}}\xspace providing same
	consistency guarantees. This is to be expected as {\textsc{SplitFS}}\xspace
	strives to minimize system calls and reduces associated
	costs.
	\item NOVA shows the highest software overhead (3.79$\times$\xspace
	for TPCC in sync mode), while PMFS shows the lowest software
	overhead (1.29$\times$\xspace for RunA). The exact ratios depend on
	the number of system calls {\textsc{SplitFS}}\xspace is able to service
	from userspace and the types of system calls observed.
	\item For LoadA, RunA, and TPCC, {\textsc{SplitFS}}\xspace is able to
	service about \todo{xx\%, xx\%, and xx\%} of the system
	calls from user-space.
\end{itemize}

\subsection{Low write amplification}
\label{sec-write-amp}

\input{tbl-write-amp}

When compared to user-space file systems like Strata, {\textsc{SplitFS}}\xspace
promises lower write amplification owing to our relink primitive.
Write amplification is important to manage not only due to the
performance penatly it imposes, but also because PM devices have
limited write endurance and write amplification reduces device
lifetime. Assuming hardware wear-leveling, device lifetime is
proportional to number of PM writes~\cite{}. So, reducing write
amplification also improves device lifetime.
We consider four microbenchmarks, each writing 1GB of data to
file as 4KB writes, along these four patterns: sequential writes,
sequential writes, random writes, appends, and overwrites (same
4KB page is written over and over again).
Table~\ref{tbl-write-amp} shows the amount of actual data written
to PM for each of these cases with Strata and {\textsc{SplitFS}}\xspace (strict),
both of which provide the same level of consistency guarantees.

For the sequential writes, random writes, and appends, Strata
incurs over 2$\times$\xspace the write amplification while {\textsc{SplitFS}}\xspace stays
at about 1$\times$\xspace. This behavior is to be expected as Strata first
writes data to process-private logs in PM and then copies the
data to shared file system regions. {\textsc{SplitFS}}\xspace does not incur
this copy due to our relink primitive. However, interestingly
, for the overwrites pattern, Stara shows minimal write
amplification. This reduction is due to Strata's log coalescing
optimization. If multiple writes to the same location are present
in the log, while copying them to the shared region, only the last
write is copied, reducing redundant data copies. So, the write
amplification with Strata varies with the number of opportunities
present for log coalescing. So, the write amplification varies
between 2$\times$\xspace and 1$\times$\xspace depdending on the success of log coalescing.
With {\textsc{SplitFS}}\xspace, only one data copy is performed at all times.
\end{comment}

\subsection{Performance on metadata-heavy workloads}
\label{sec-real-mh-apps}

Fig~\ref{fig-real-apps} compares the performance of {\textsc{SplitFS}}\xspace with
other PM file systems (we only show the best performing PM file
system) on metadata-heavy workloads like git, tar, and rsync. These
metadata-heavy workloads do not present many opportunities for
{\textsc{SplitFS}}\xspace to service system calls in userspace and in turn slow
metadata operations down due to the additional bookkeeping performed
by {\textsc{SplitFS}}\xspace. These workloads represent the worst case scenarios for
{\textsc{SplitFS}}\xspace. The maximum overhead experienced by {\textsc{SplitFS}}\xspace is 13\%.

\subsection{Resource Consumption}
\label{sec-resource}

{\textsc{SplitFS}}\xspace consumes memory for its file-related metadata (\textit{e.g.,}\xspace, to keep
track of open file descriptors, staging files used). It also
additionally consumes CPU time to execute background threads that help
with metadata management and to move some expensive tasks off the
application's critical path.

\vheading{Memory usage}. {\textsc{SplitFS}}\xspace using a maximum of 100MB to maintain
its own metadata to help track different files, the mappings between
file offsets and \texttt{mmap()}\xspace-ed regions, etc. In strict mode, {\textsc{SplitFS}}\xspace
additionally uses 40MB to maintain data structures to provide
atomicity guarantees.

\vheading{CPU utilization}.  {\textsc{SplitFS}}\xspace uses a background thread to
handle various deferred tasks (\textit{e.g.,}\xspace, stage file allocation, file
closures). This thread utilizes one physical thread
of the machine, occasionally increasing CPU consumption by 100\%.

\section{Introduction}
\label{sec-intro}

Persistent Memory (PM) is a new memory technology that was recently
introduced by Intel~\cite{3dxpoint}. PM will be placed on the memory
bus like DRAM and will be accessed via processor loads and stores. PM
has a unique performance profile: compared to DRAM, loads have
2--3.7$\times$\xspace higher latency and $1/3^{rd}$ bandwidth, while stores have
the same latency but $1/6^{th}$ bandwidth~\cite{pm-arxiv}.
A single machine can be equipped with up to 6 TB of PM. Given its
large capacity and low latency, an important use case for PM
will be acting as storage.

Traditional file systems add large overheads to each file-system
operation, especially on
the write path. The overhead comes from performing expensive
operations on the critical path, including allocation, logging, and
updating multiple complex structures. The systems community has
proposed different architectures to reduce overhead. BPFS~\cite{bpfs},
PMFS~\cite{pmfs}, and NOVA~\cite{nova} redesign the in-kernel file
system from scratch to reduce overhead for file-system
operations. Aerie~\cite{aerie} advocates a user-space library file
system coupled with a slim kernel component that does coarse-grained
allocations. Strata~\cite{strata} proposes keeping the file system
entirely in user-space, dividing the system between a user-space
library file system and a user-space metadata server. Aerie and Strata
both seek to reduce overhead by not involving the kernel for most
file-system operations.

\input{tbl-mot}

Despite these efforts, file-system data operations, especially writes,
have significant overhead. For example, consider the common operation
of appending 4K blocks to a file (total 128 MB). It takes $671$ $ns$\xspace to
write a 4 KB to PM; thus, if performing the append operation took a
total of $675$ $ns$\xspace, the software overhead would be $4$
$ns$\xspace. Table~\ref{tbl-mot} shows the software overhead on the append
operation on various PM file systems. We observe that there is still
significant overhead ($3.5-12.4$$\times$\xspace) for file appends.

This paper presents {\textsc{SplitFS}}\xspace, a PM file system that seeks to
  reduce software overhead via a novel \emph{split} architecture: a
  user-space library file system handles data operations while a
  kernel PM file system (ext4 DAX) handles metadata operations.
  We refer to all file system operations that modify file metadata
    as \emph{metadata operations}. Such operations include \texttt{open()}\xspace,
    \texttt{close()}\xspace, and even file appends (since the file size is changed).
  The novelty of {\textsc{SplitFS}}\xspace lies in how responsibilities are divided between the user-space and kernel components, and the semantics provided to applications. Unlike prior work like Aerie, which used the kernel only for coarse-grained operations, or Strata, where all operations are in user-space, {\textsc{SplitFS}}\xspace routes \emph{all} metadata operations to the kernel. While FLEX~\cite{flex} invokes the kernel at a fine granularity like
  {\textsc{SplitFS}}\xspace, it does not provide strong semantics such as synchronous,
  atomic operations to applications. At a high level, the {\textsc{SplitFS}}\xspace
  architecture is based on the belief that if we can accelerate
  common-case data operations, it is worth paying a cost on the
  comparatively rarer metadata operations. This is in contrast with
  in-kernel file systems like NOVA which extensively modify the file
  system to optimize the metadata operations.

{\textsc{SplitFS}}\xspace transparently reduces software overhead for reads and
overwrites by intercepting POSIX calls, memory mapping the underlying
file, and serving reads and overwrites via processor loads and
stores. {\textsc{SplitFS}}\xspace optimizes file appends by introducing a new primitive
named \emph{relink} that minimizes both data copying and trapping into
the kernel. The application does not have to be rewritten in any way
to benefit from {\textsc{SplitFS}}\xspace. {\textsc{SplitFS}}\xspace reduces software overhead
by up-to 4$\times$\xspace compared to NOVA and 17$\times$\xspace compared to ext4 DAX.

Apart from lowering software overhead, the split architecture leads to
several benefits. First, instead of re-implementing file-system
functionality, {\textsc{SplitFS}}\xspace can take advantage of the mature, well-tested
code in ext4 DAX for metadata operations. Second, the user-space
library file system in {\textsc{SplitFS}}\xspace allows each application to run with
one of three consistency modes (\emph{POSIX, sync, strict}). We
observe that not all applications require the same guarantees; for
example, SQLite does not require the strong guarantees provided by
NOVA-strict, and gets 2.5$\times$\xspace higher throughput on ext4 DAX and
{\textsc{SplitFS}}\xspace-POSIX than on NOVA-strict owing to their weaker guarantees.
Applications running with different consistency modes do not interfere
with each other on {\textsc{SplitFS}}\xspace.

{\textsc{SplitFS}}\xspace introduces the relink primitive to optimize file appends and
atomic data operations. Relink logically and atomically moves a
contiguous extent from one file to another, without any physical data
movement. Relink is built on top of the \vtt{swap\_extents}
\vtt{ioctl} in ext4 DAX, and uses ext4 journaling to ensure the source
and destination files are modified atomically. Both file appends and
data overwrites in strict mode are redirected to a temporary PM file
we term the staging file. On \texttt{fsync()}\xspace, the data from the staging file
is relinked into the original file. Relink provides atomic data
operations without paging faults or data copying.

{\textsc{SplitFS}}\xspace also introduces an optimized logging protocol. In strict
mode, all data and metadata operations in {\textsc{SplitFS}}\xspace are atomic and
synchronous. {\textsc{SplitFS}}\xspace achieves this by logging each operation. In the
common case, {\textsc{SplitFS}}\xspace will write a single cache line
worth of data (64B), followed by one memory fence (\textit{e.g.,}\xspace \texttt{sfence}\xspace in
  x86 systems), for each operation; in contrast, NOVA writes at least
two cache lines and issues two fences. As a result of these
optimizations, {\textsc{SplitFS}}\xspace logging is 4$\times$\xspace faster than NOVA in the
critical path. Thanks to relink and optimized logging, atomic data
operations in {\textsc{SplitFS}}\xspace are 2--6$\times$\xspace faster than in
NOVA-strict, providing strong guarantees at low software overhead.

We evaluate {\textsc{SplitFS}}\xspace using a number of micro-benchmarks, three
utilities (git, tar, rsync), two key-value stores (Redis, LevelDB),
and an embedded database (SQLite). Our evaluation on
\cameditstwo{Intel DC Persistent Memory} shows that {\textsc{SplitFS}}\xspace, though
it is built on ext4 DAX, outperforms ext4 DAX by up-to 2$\times$\xspace on many
workloads. {\textsc{SplitFS}}\xspace outperforms NOVA by 10\%--2$\times$\xspace (when providing
the same consistency guarantees) on LevelDB, Redis, and SQLite when
running benchmarks like YCSB and TPCC. {\textsc{SplitFS}}\xspace also reduces total
amount of write IO by 2$\times$\xspace compared to Strata on certain
workloads. On metadata-heavy workloads such as git and tar, {\textsc{SplitFS}}\xspace
suffers a modest drop in performance (less than 15\%) compared to NOVA
and ext4 DAX.

{\textsc{SplitFS}}\xspace is built on top of ext4 DAX; this is both a strength and a
weakness. Since {\textsc{SplitFS}}\xspace routes all metadata operations through ext4
DAX, it suffers from the high software overhead and high write IO for
metadata operations. Despite these limitations, we believe {\textsc{SplitFS}}\xspace
presents a useful new point in the spectrum of PM file-system
designs. ext4 DAX is a robust file system under active development;
its performance will improve with every Linux kernel version. {\textsc{SplitFS}}\xspace
provides the best features of ext4 DAX while making up for its lack of
performance and strong consistency guarantees.

This paper makes the following contributions:
\begin{itemize}
  \item A new architecture for PM file systems with a novel split of
    responsibilities between a user-space library file system and a
    kernel file system.
  \item The novel relink primitive that can be used to provide efficient
    appends and atomic data operations.
  \item The design and implementation of {\textsc{SplitFS}}\xspace, based on the split
    architecture. We have made {\textsc{SplitFS}}\xspace publicly available at \url{https://github.com/utsaslab/splitfs}.
  \item Experimental evidence demonstrating that {\textsc{SplitFS}}\xspace outperforms
    state-of-the-art in-kernel and in-user-space PM file systems, on
    a range of workloads.

\end{itemize}

\section{Methodology}
\label{sec-methodology}

In this section, we will talk about the procedure that is used to check crash consistency of persistent memory file systems. We first talk about the platform that we use in order to track memory writes, cacheline flushes and memory fence instructions (\sref{sec-panda}) and then we talk about how this information is used to check the crash consistency in file systems (\sref{sec-crashmonkey-panda}).

\subsection{PANDA}
\label{sec-panda}

Panda is a dynamic execution engine based on QEMU. When a basic block of code is executed in the guest, the QEMU virtual machine monitor converts the block of guest code into an intermediate language. From this intermediate language, QEMU generates a binary code block which is directly executed on the host, in order to emulate the guest code block. Panda analyzes the intermediate language generated by QEMU for specific instructions of interest. With Panda, users are allowed to track memory loads and stores as well as flushes at specific address ranges through user defined plugins. Based on the tracked instructions, panda allows users register callback functions for logging the recorded data in a file, or performing some other tasks. The user defined plugins in Panda can be loaded arbitrarily when the user wants to track memory loads, stores and flushes, and they can be unloaded when the user is no longer interested in tracking instructions.

\subsection{CrashMonkey with PANDA}
\label{sec-crashmonkey-panda}

The procedure for checking crash consistency for persistent memory file systems is different for persistent memory than for disk based file systems, due to the differences mentioned in \sref{sec-complexities}. This subsection contains the steps that are used for testing crash consistency for persistent memory file systems.

\vheading{Initialization}. Since PANDA works in conjunction with QEMU and tracks memory accesses in the QEMU guest, a virtual machine is booted up with PANDA enabled, and with two memory regions reserved for persistent memory (pmem0 and pmem1). One of the memory regions (pmem0) is treated as a record device, and the second region (pmem1) is treated as a replay device.

\vheading{Snapshotting}. In this stage, an empty file system is created and mounted in the record device. This step performs file system initializations on the record device such as initializing the superblock and various other metadata structures of the file system. After the file system is created and mounted, a snapshot of the record device is taken and is restored on the replay device. This step does the file system initialization task at the replay device, but does not actually mount the file system.

\vheading{Tracing}. The PANDA plugin for tracking memory writes, memory fences and cacheline flushes is loaded for the region of memory that belongs to the record device. On loading the plugin, PANDA starts to analyze the intermediate language of QEMU for the store, fence and flush instructions. Callbacks are issued for each of these instructions to log them in a file. Now the workload to be tested is run on the record device. After the workload is completely executed, the PANDA plugin is unloaded in order to stop the tracking of instructions.

\vheading{Generation of crash states}. The log file is parsed and different crash states are generated by crashmonkey, in the same way as they are generated for disk based file systems.

\vheading{Checking crash recovery}. The different crash states are replayed at the replay device. After each crash state, the file system at the replay device is mounted to check if it recovers to a consistent state, and a bug report is generated if the file system fails to do so.

\section{Related Work}
\label{sec-related}

{\textsc{SplitFS}}\xspace builds on a large body of work on PM file systems and
building low-latency storage systems. We briefly describe the work
that is closest to {\textsc{SplitFS}}\xspace.

\vheading{Aerie}. Aerie~\cite{aerie} was one of the first systems to
advocate for accessing PM from user-space. Aerie proposed a split
architecture similar to {\textsc{SplitFS}}\xspace, with a user-space library file
system and a kernel component. Aerie used a user-space metadata server
to hand out leases, and only used the kernel component for
coarse-grained activities like allocation. In contrast, {\textsc{SplitFS}}\xspace does
not use leases (instead making most operations immediately visible)
and uses ext4 DAX as its kernel component, passing all metadata
operations to the kernel. Aerie proposed eliminating the POSIX
interface, and aimed to provide applications flexibility in
interfaces. In contrast, {\textsc{SplitFS}}\xspace aims to efficiently support the
POSIX interface.

\vheading{Strata}. The Strata~\cite{strata} cross-device file system
is similar to Aerie and {\textsc{SplitFS}}\xspace in many respects. There are two main
differences from {\textsc{SplitFS}}\xspace. First, Strata writes all data to a
process-private log, coalesces the data, and then writes it to a
shared space. In contrast, only appends are private (and only until
\vtt{fsync}) in {\textsc{SplitFS}}\xspace; all metadata operations and overwrites are
immediately visible to all processes in {\textsc{SplitFS}}\xspace. {\textsc{SplitFS}}\xspace does not
need to copy data between a private space and a shared space; it
instead relinks data into the target file. Finally, since Strata is
implemented entirely in user-space, the authors had to re-implement a
lot of VFS functionality in their user-space library. {\textsc{SplitFS}}\xspace instead
depends on the mature codebase of ext4 DAX for all metadata
operations.

\vheading{Quill and FLEX}. Quill~\cite{quill} and File Emulation with
DAX (FLEX)~\cite{flex} both share with {\textsc{SplitFS}}\xspace the core technique of
transparently transforming read and overwrite POSIX calls into
processor loads and stores. However, while Quill and FLEX do not
provide strong semantics, {\textsc{SplitFS}}\xspace can provide applications with
synchronous, atomic operations if required. {\textsc{SplitFS}}\xspace also differs in
its handling of appends.  Quill calls into the kernel for every
operation, and FLEX optimizes appends by pre-allocating data beyond
what the application asks for. In contrast, {\textsc{SplitFS}}\xspace elegantly handles
this problem using staging files and the relink primitive. While Quill
appends are slower than ext4 DAX, {\textsc{SplitFS}}\xspace appends are faster than
ext4 DAX appends. At the time of writing this paper, FLEX has not been
made open-source, so we could not evaluate it.

\vheading{PM file systems}. Several file systems such as
SCMFS~\cite{scmfs}, BPFS~\cite{bpfs}, and NOVA~\cite{nova} have been
developed specifically for PM. While each file system tries to reduce
software overhead, they are unable to avoid the cost of trapping into
the kernel. The relink primitive from {\textsc{SplitFS}}\xspace is similar to the
short-circuit paging presented in BPFS. However, while short-circuit
paging relies on an atomic 8-byte write, {\textsc{SplitFS}}\xspace relies on ext4's
journaling mechanism to make relink atomic.

\vheading{Kernel By-Pass}. Several projects have advocated direct
user-space access to networking~\cite{unet}, storage~\cite{dafs,nas,
  petal, moneta}, and other hardware features~\cite{dune, ix,
  arrakis}. These projects typically follow the philosophy of
separating the control path and data path, as in
Exokernel~\cite{exokernel} and Nemesis~\cite{barham1997fs}. {\textsc{SplitFS}}\xspace
follows this philosophy, but differs in the abstraction provided
by the kernel component; {\textsc{SplitFS}}\xspace uses a PM file system as its
kernel component to handle all metadata operations, instead of limiting
it to lower-level decisions like allocation.