I am trying to compare aggregate write rates when writing to a file in a GPFS file system, as compared to writing directly to a disk on a system with Red Hat Enterprise Linux Server release 6.4 (Santiago). For my application I need to measure the raw rate, i.e. without taking advantage of cache. I do not understand the impact of the direct option used with dd to bypass cache. When writing directly to a block device, I get a drastically lower rate when I use oflag=direct, as compared with writing to a file in the GPFS file system. Why does this happen?To measure aggregate rates I create p processes running dd that writes concurrently to the block device or file. I then sum the p rates obtained to get the aggregate write rate.    #!/bin/bash    directdiskrate=~/scratch/rate5    syncdiskrate=~/scratch/rate4    filerate=~/scratch/rate3    numruns=1    numthreads=30    #to disk use both conv=fsync and oflag=direct    writetodiskdirect=dd if=/dev/zero of=/dev/sdac bs=256k count=4096 conv=fsync oflag=direct iflag=fullblock    for p in $(seq $numthreads)    do             #parses output of dd, rate is on last line, each field separated by ,s            $writetodiskdirect 2>&1|tail -n 1|awk 'BEGIN { FS = , } ; { print $3 }'|sed -e 's/MB\/s//g'>>$directdiskrate&    done    wait    #to disk use only conv=fsync option    writetodisksync=dd if=/dev/zero of=/dev/sdac bs=256k count=4096 conv=fsync iflag=fullblock    for p in $(seq $numthreads)    do       #parses output of dd, rate is on last line, each field separated by ,s       $writetodisksync 2>&1|tail -n 1|awk 'BEGIN { FS = , } ; { print $3 }'|sed -e 's/MB\/s//g'>>$syncdiskrate&    done    wait    #to file use both conv=fsync and oflag=direct    for p in $(seq $numthreads)    do        writetofile=dd if=/dev/zero of=/gpfs1/fileset6/file$p bs=256k count=4096 conv=fsync oflag=direct        #parses output of dd, rate is on last line, each field separated by ,s        $writetofile 2>&1|tail -n 1|awk 'BEGIN { FS = , } ; { print $3 }'|sed -e 's/MB\/s//g'>>$filerate&    done    waitResults: The write rate of each of 30 processes is as follows:Writing to disk using conv=fsync option, each process gets a write rate of ~180MB/sWriting to disk using both conv=fsync and oflag=direct, each process gets a write rate of ~9MB/sWriting to a file in GPFS file system, using both conv=fsync and oflag=direct, gets a write rate of ~80MB/s
Why is dd using direct slower writing to disk than to a file
linux;filesystems;hard disk;performance;coreutils