title: R: Converting PDF to CSV using pdftools, stringr, and regex
body: <p>I'm converting a massive collection of pdfs into a single massive csv. </p>
  
  <p>A typical pdf looks like this:</p>
  
  <p><a href="https://i.stack.imgur.com/mTZ1M.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mTZ1M.png" alt="enter image description here"></a></p>
  
  <p>When I use pdftools to convert the page into a single text string I get this:</p>
  
  <p><a href="https://i.stack.imgur.com/W71Fz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/W71Fz.png" alt="enter image description here"></a></p>
  
  <p>When I use the <code>cat()</code> function on the page's string I get this:</p>
  
  <p><a href="https://i.stack.imgur.com/DLb5j.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/DLb5j.png" alt="enter image description here"></a></p>
  
  <p>My question is: how does the <code>cat()</code> function know where each column begins? How can it cleanly parse through the pdf string? I'm using R code that puts a ';' whenever there is more than two white spaces in the string, but I'm coming up with a bunch of problems. </p>
  
  <p>First problem was that the 'sexo' variable only has one space between it and the 'c.identidad' variable. My solution was to use <code>str_replace()</code> to add spaces in front of and the 'sexo' variables ('VAR' and 'MUJ').</p>
  
  <p>The second problem I found was that some observations had only one space between the 'domicilio electoral' and 'circumscripcion' variable. I figured out that 'circumscripcion' was essentially the same as the 'comuna' in the header. So my solution was to <code>str_locate()</code> to identify what the 'comuna' was and then use <code>str_replace()</code> to add spaces to the front and back of all instances where the string matched the 'comuna'. </p>
  
  <p>The third problem was that sometimes the 'domicilio electoral' variable of an observation would also include the 'comuna'. This means that my code to switch the string into pdf (any white space with a length greater than 2 gets converted into a ';') would provide an extra variable for any observation that included a 'comuna' string more than once.</p>
  
  <p>Once the third problem came up I began to think that there must be a simpler way to parse through this pdf document. If <code>cat()</code> can cleanly parse through the pdf string, then there has to be something similar to help me parse through it in an ordered way. Any help, guidance, links, or ideas would be greatly appreciated. </p>
  
  <hr>
  
  <p>The pdfs I'm using are public and can be found at:<a href="http://web.servel.cl/padronAuditado.html" rel="nofollow noreferrer">http://web.servel.cl/padronAuditado.html</a></p>
  
  <p>Below is the final version of my code. It has about a 6% obvious error rate. I measured error by counting observations that had the 'mesa' variable as blank.</p>
  
  <pre><code>require(stringr)
  require(dplyr)
  require(pdftools)
  
  pdf_to_csv2 &lt;- function(doc){
  
   text &lt;- pdf_text(doc)
  
   COMUNA_LOCATE = str_locate(text[1], "COMUNA:.+PAGINA")
  
   COMUNA = text[1] %&gt;% str_sub(COMUNA_LOCATE[1], COMUNA_LOCATE[2]) %&gt;% 
     str_replace('COMUNA:', '') %&gt;%
     str_replace('PAGINA', '') %&gt;%
     str_trim('both')
  
   cleaned_text &lt;- text %&gt;% str_replace_all(' VAR ', '   VAR   ') %&gt;%
     str_replace_all(' MUJ ', '   MUJ   ') %&gt;%
     str_replace_all(paste(COMUNA, '                       '), paste('    ', COMUNA, '    ')) %&gt;%
     str_replace('IDENTIDAD SEXO', 'IDENTIDAD   SEXO') %&gt;%
     str_replace_all('\r\n', 'END_LINE') %&gt;%
     str_replace_all('\\s\\s+', ';') %&gt;%
     str_replace_all('END_LINE', '\n')
  
   for(i in c(1:length(cleaned_text))) {write(cleaned_text[i], paste0(doc, '_', i))}
  
   }
  </code></pre>