Processing large files is a memory intensive operation and could cause servers to run out of RAM memory and swap to disk. Let’s look at few ways to process CSV files with Ruby and measure the memory consumption and speed performance.

### Prepare CSV data sample

Before we start, let’s prepare a CSV file `data.csv` with 1 million rows (~ 75 MB) to use in tests.

``````require 'csv'
require_relative './helpers'

headers = ['id', 'name', 'email', 'city', 'street', 'country']

name    = "Pink Panther"
email   = "pink.panther@example.com"
city    = "Pink City"
street  = "Pink Road"
country = "Pink Country"

print_memory_usage do
print_time_spent do
1_000_000.times do |i|
csv << [i, name, email, city, street, country]
end
end
end
end
``````

### Memory used and time spent

This script above requires the `helpers.rb` script which defines two helper methods for measuring and printing out the memory used and time spent.

``````require 'benchmark'

def print_memory_usage
memory_before = `ps -o rss= -p #{Process.pid}`.to_i
yield
memory_after = `ps -o rss= -p #{Process.pid}`.to_i

puts "Memory: #{((memory_after - memory_before) / 1024.0).round(2)} MB"
end

def print_time_spent
time = Benchmark.realtime do
yield
end

puts "Time: #{time.round(2)}"
end
``````

The results to generate the CSV file are:

``````\$ ruby generate_csv.rb
Time: 5.17
Memory: 1.08 MB
``````

Output can vary between machines, but the point is that when building the CSV file, the Ruby process did not spike in memory usage because the garbage collector (GC) was reclaiming the used memory. The memory increase of the process is about 1MB, and it created a CSV file with size of 75 MB.

``````\$ ls -lah data.csv
-rw-rw-r-- 1 dalibor dalibor 75M Mar 29 00:34 data.csv
``````

### Reading CSV from a file at once (CSV.read)

Let’s build a CSV object from a file (`data.csv`) and iterate with the following script:

``````require_relative './helpers'
require 'csv'

print_memory_usage do
print_time_spent do
sum = 0

csv.each do |row|
sum += row['id'].to_i
end

puts "Sum: #{sum}"
end
end
``````

The results are:

``````\$ ruby parse1.rb
Sum: 499999500000
Time: 19.84
Memory: 920.14 MB
``````

Important to note here is the big memory spike to 920 MB. That is because we build the whole CSV object in memory. That causes lots of String objects to be created by the CSV library and the used memory is much more higher than the actual size of the CSV file.

### Parsing CSV from in memory String (CSV.parse)

Let’s build a CSV object from a content in memory and iterate with the following script:

``````require_relative './helpers'
require 'csv'

print_memory_usage do
print_time_spent do
csv = CSV.parse(content, headers: true)
sum = 0

csv.each do |row|
sum += row['id'].to_i
end

puts "Sum: #{sum}"
end
end
``````

The results are:

``````\$ ruby parse2.rb
Sum: 499999500000
Time: 21.71
Memory: 1003.69 MB
``````

As we can see from the results, the memory increase is about the memory increase from the previous example plus the memory size of the file content that we read in memory (75MB).

### Parsing CSV line by line from String in memory (CSV.new)

Let’s now see what happens if we load the file content in a String and parse it line by line:

``````require_relative './helpers'
require 'csv'

print_memory_usage do
print_time_spent do
csv = CSV.new(content, headers: true)
sum = 0

while row = csv.shift
sum += row['id'].to_i
end

puts "Sum: #{sum}"
end
end
``````

The results are:

``````\$ ruby parse3.rb
Sum: 499999500000
Time: 9.73
Memory: 74.64 MB
``````

From the results we can see that the memory used is about the file size (75 MB) because the file content is loaded in memory and the processing time is about twice faster. This approach is useful when we have the content that we don’t need to read it from a file and we just want to iterate over it line by line.

### Parsing CSV file line by line from IO object

Can we do any better than the previous script? Yes, if we have the CSV content in a file. Let’s use an IO file object directly:

``````require_relative './helpers'
require 'csv'

print_memory_usage do
print_time_spent do
File.open('data.csv', 'r') do |file|
csv = CSV.new(file, headers: true)
sum = 0

while row = csv.shift
sum += row['id'].to_i
end

puts "Sum: #{sum}"
end
end
end
``````

The results are:

``````\$ ruby parse4.rb
Sum: 499999500000
Time: 9.88
Memory: 0.58 MB
``````

In the last script we see less than 1 MB of memory increase. Time seems to be a very little slower compared to previous script because there is more IO involved. The CSV library has a built in mechanism for this, `CSV.foreach`:

``````require_relative './helpers'
require 'csv'

print_memory_usage do
print_time_spent do
sum = 0

CSV.foreach('data.csv', headers: true) do |row|
sum += row['id'].to_i
end

puts "Sum: #{sum}"
end
end
``````

The results are similar:

``````\$ ruby parse5.rb
Sum: 499999500000
Time: 9.84
Memory: 0.53 MB
``````

Imagine you need to process large CSV files of 10GB or more. Deciding to use the last strategy seems obvious.
{"ft_lang_label":"__label__en","ft_lang_prob":0.73581445,"math_prob":0.4967233,"size":4983,"snap":"2022-40-2023-06","text_gpt3_token_len":1360,"char_repetition_ratio":0.12653144,"word_repetition_ratio":0.16339067,"special_character_ratio":0.2871764,"punctuation_ratio":0.15063168,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.98402745,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2022-09-25T00:49:27Z\",\"WARC-Record-ID\":\"<urn:uuid:8604f3b0-e998-4128-aa8f-82355527e50d>\",\"Content-Length\":\"29878\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:5620b2b8-4b3c-4f1e-8202-010972c4e7c5>\",\"WARC-Concurrent-To\":\"<urn:uuid:db7c3c92-a16e-4615-b59a-b5aecf1ab798>\",\"WARC-IP-Address\":\"185.199.111.153\",\"WARC-Target-URI\":\"https://dalibornasevic.com/posts/68-processing-large-csv-files-with-ruby?utm_source=rubyweekly&utm_medium=email\",\"WARC-Payload-Digest\":\"sha1:SL224C5NRKFWDMXDV56RARZIQ5BFUFDI\",\"WARC-Block-Digest\":\"sha1:GKHHJTESWTYMW3JA5LIFD2RFDY4NTOYO\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2022/CC-MAIN-2022-40/CC-MAIN-2022-40_segments_1664030334332.96_warc_CC-MAIN-20220925004536-20220925034536-00099.warc.gz\"}"}