title: From pdf text to tidy dataframe with file names in document column
body: <p>I want to analyse text from almost 300 pdf documents. Now I used the <code>pdftools</code> and <code>tm</code>, <code>tidytext</code> packages to read the text, coverted it to a corpus, then to a document-term-matrix and I finally want to structure it in a tidy dataframe.</p>
  
  <p>I've got a couple questions:</p>
  
  <ul>
  <li>How do I get rid of page data (at the top and/or bottom of every pdf page)</li>
  <li>I would rather want the filenames as the values in the <code>document</code> column instead of indexed numbers.</li>
  <li>The following code contents only 2 pdf files for reproducibility. When I run all my files I get 294 documents in my <code>corpus</code> object, but when I tidy it I seem to loose some files because <code>converted %&gt;% distinct(document)</code> gives 275 back. I wonder why that is.</li>
  </ul>
  
  <p>I've got the The following reproducible script: </p>
  
  <pre><code>library(tidyverse)
  library(tidytext)
  library(pdftools)
  library(tm)
  library(broom)
  
  # Create a temporary empty directory 
  # (don't worry at the end of this script I'll remove this directory and its files)
  
  dir.create("~/Desktop/sample-pdfs")
  
  # Fill directory with 2 pdf files from my github repo
  
  download.file("https://github.com/thomasdebeus/colourful-facts/raw/master/projects/sample-data/'s-Gravenhage_coalitieakkoord.pdf", destfile = "~/Desktop/sample-pdfs/'s-Gravenhage_coalitieakkoord.pdf")
  download.file("https://github.com/thomasdebeus/colourful-facts/raw/master/projects/sample-data/Aa%20en%20Hunze_coalitieakkoord.pdf", destfile = "~/Desktop/sample-pdfs/Aa en Hunze_coalitieakkoord.pdf")
  
  # Create vector of file paths
  
  dir &lt;- "~/Desktop/sample-pdfs"
  pdfs &lt;- paste(dir, "/", list.files(dir, pattern = "*.pdf"), sep = "")
  
  # Read the text from pdf's with pdftools package
  
  pdfs_text &lt;- map(pdfs, pdf_text)
  
  # Convert to document-term-matrix
  
  converted &lt;- Corpus(VectorSource(pdfs_text)) %&gt;%
            DocumentTermMatrix()
  
  # Now I want to convert this to a tidy format
  
  converted %&gt;%
            tidy() %&gt;%
            filter(!grepl("[0-9]+", term))
  </code></pre>
  
  <p>With the following output:</p>
  
  <pre><code># A tibble: 5,305 x 3
     document term           count
     &lt;chr&gt;    &lt;chr&gt;          &lt;dbl&gt;
   1 1        aan              158
   2 1        aanbesteding       2
   3 1        aanbestedingen     1
   4 1        aanbevelingen      1
   5 1        aanbieden          3
   6 1        aanbieders         1
   7 1        aanbod             8
   8 1        aandacht          16
   9 1        aandachtspunt      3
  10 1        aandeel            1
  # ... with 5,295 more rows
  </code></pre>
  
  <p>This seems to work out nicely but I would rather want the filenames (<code>"'s-Gravenhage"</code> and <code>"Aa en Hunze"</code>) as the values in the document column instead of indexed numbers. How do I do this?</p>
  
  <p>Desired output:</p>
  
  <pre><code># A tibble: 5,305 x 3
     document      term           count
     &lt;chr&gt;         &lt;chr&gt;          &lt;dbl&gt;
   1 's-Gravenhage aan              158
   2 's-Gravenhage aanbesteding       2
   3 's-Gravenhage aanbestedingen     1
   4 's-Gravenhage aanbevelingen      1
   5 's-Gravenhage aanbieden          3
   6 's-Gravenhage aanbieders         1
   7 's-Gravenhage aanbod             8
   8 's-Gravenhage aandacht          16
   9 's-Gravenhage aandachtspunt      3
  10 's-Gravenhage aandeel            1
  # ... with 5,295 more rows
  </code></pre>
  
  <p>Delete downloaded files and its directory from desktop running the following line:</p>
  
  <pre><code>unlink("~/Desktop/sample-pdfs", recursive = TRUE)
  </code></pre>
  
  <p>All help is much appreciated! </p>