I have a large file that lists a long column unix times one value for each row, incrementing in intervals of 0.01s. For one day of data, this amounts to 8.64 million rows.135699840000135699840001135699840002135699840003135699840004I would like to run a command on every row of this file, that calculates the serial date number for each line - a day counter from a reference year 01/01/0000 that matlab uses for time.735235.0000000000735235.0000001157735235.0000002314735235.0000003472735235.0000004629I am new to coding, but have managed to get this working using a while loop. However, this is hideously inefficient and is taking hours to run. while read epochtimerange; doecho scale=10; (($epochtimerange/(100*86400))+719529) |bcdone < epochtimerangetmp.txt > serialdaterangetmp.txtI think there must be a way to run this using awk, but I can't get this to work. It is important that I am able to keep my 10 decimal place precision on my output. Would someone be able to help me? Thanks.  
Applying a mathematical function to every line in a large file
awk
The simple approach: Use ex to modify the lines and to pass the whole buffer (modified file) through bc.  Then print the modified version.printf '%s\n' '%s:.*:&/8640000+719529:' 0a scale=10 . '%!bc' %p 'q!' | ex file.txtOutput on your sample file:735235.0000000000735235.0000001157735235.0000002314735235.0000003472735235.0000004629Or to save changes rather than just printing them:printf '%s\n' '%s:.*:&/8640000+719529:' 0a scale=10 . '%!bc' x | ex file.txtExplanation:To see what commands are passed to ex, run the printf command by itself:$ printf '%s\n' '%s:.*:&/8640000+719529:' 0a scale=10 . '%!bc' %p 'q!'%s:.*:&/8640000+719529:0ascale=10.%!bc%pq!Let's break these down as ex commands now.  The first one is rather complex, so I'll format the explanation specially:%s:.*:&/8640000+719529:%  - For every line of the buffer (file) s  - Run a substitute command  :  - Using ':' as the regex delimiter   .*  - Match each entire line     :  - and replace with      &  - The entire line, followed by       /8640000+719529  - this text                      :  - End command0a means append text after line 0, in other words at the start of the buffer (file).The text scale=10 is the literal text to append.The . on a line by itself ends the append command.The command %!bc passes the contents of the entire buffer as standard input to the external command bc, and replaces the entire buffer with the output that is produced.The %p means to print the entire buffer (to standard output).q! means to exit without saving changes.If you have a very, very large file, in the tens of millions of lines, this apparently gives trouble.  I have researched possible solutions for this using ex and there are some ways it could be done, but I finally discarded that approach in favor of a much, much simpler one, which still uses only POSIX specified tools.Use split to split your file into chunks, then run the earlier specified command on each chunk and cat the resulting output all together:split -l 1000000 -a 3 file.txt myprefix.for f in myprefix.???; do  printf '%s\n' '%s:.*:&/8640000+719529:' 0a scale=10 . '%!bc' %p 'q!' |    ex $fdone > myoutputfile.txtrm myprefix.???The split command is used here to split file.txt into chunks each an even million lines long (with the remainder put into a file also, of course).  Since -a 3 is specified, the suffix on the chunks will be 3 characters long.  myprefix.aaa, myprefix.aab, etc.Each file can then be processed by ex individually, and with no need to save changes, since we will just redirect the output of that entire loop to myoutputfile.txt (and then remove the chunk files, for neatness).