Title: Creating a dataframe with variables from file name
Tags: r;text-mining
Question: tldr: I want to create a data frame with the number of occurrences of certain words in documents and later visualize their change over time, but I lose the necessary variables during the process.
The text files are named rigorously in a format which contain many data about them in this format: &quot;000-yyyy-mmdd-AAA-00-00-AA.txt&quot; (the 0s stand for numbers and the As stand for letters). I've used &quot;readtext&quot; to use this as document variables.
``` files <- readtext(
&quot;all_files.zip&quot;,
  docvarsfrom = &quot;filenames&quot;,
  dvsep = &quot;-&quot;,
  docvarnames = c(&quot;var1&quot;, &quot;year&quot;, &quot;daymonth&quot;, &quot;var2&quot;, &quot;var3&quot;, &quot;var4&quot;, &quot;var5&quot;),
)
```
From this, I count the specific terms with the code below, taken from another question However, somewhere I lose the document variables: I only get the whole filenames with the numbers back, meanwhile I would need at least the year variable to sort them.
I've tried several ways, such as obtaining the file names, regex the year section out and insert back as a column and trying to change several settings and packages, but as a beginner, I could not find a solution. I'd guess that I lose the data perhaps during the tokenization, but I'm not sure.
Could you give me a recommendation, what should I do?
The following packages were active in my session:
```library(readr)
library(readtext)
library(dplyr)
library(lubridate)
library(stringr)
library(quanteda)
library(quanteda.textmodels)
library(quanteda.textstats)
library(purrr)
library(ggplot2)
library(tidytext)


files_toks <- tokens(files)

mydict <- c(&quot;apple&quot;, &quot;pear&quot;)

dfmat <- tokens(files_toks) %&gt;%
  tokens_select(mydict) %&gt;%
  dfm()

convert(dfmat, to = &quot;data.frame&quot;)
```
The dfmat dataframe goes like this:
```                          doc_id  apple  pear
1 001-yyyy-mmdd-AAA-00-00-AA.txt      3     1
2 002-yyyy-mmdd-AAA-00-00-AA.txt      2     5
...
```

Comment: Yes, I should do in the quanteda package, if it is possible. The text files are quite long (originally dozens of pages) documents, containing English-language text. I hope this helps, I'm not sure what did you mean by the look of text files

Comment: I am not sure I understand what exactly is your problem (isn't dfmat what you're looking for), but having too many packages may be confusing. If you just want a wordcount, why not trying a basic stringr181a:6c1c:918f:f3a4:2e8c:61d6:1a08:6b16str_count(text. "word_to_count") and forget all the rest?

Comment: How does the text file look? What are it's contents? Do you have to do this with `quanteda` package?

Comment: You should include the names of all packages you are using. There are many text mining packages. Some packages have vignettes which show how the functions in the package work together.



{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 0, 'CommentCount': 4, 'ContentLicense': 'CC BY-SA 4.0', 'CreationDate': '2021:00:07/24/21', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<r><text-mining>', 'Title': 'Creating a dataframe with variables from file name', 'ViewCount': 67, 'id': 68511102, 'user_AboutMe': None, 'user_AccountId': 22261902.0, 'user_DisplayName': 'armentieres', 'user_Reputation': 3.0, 'user_Views': 4.0, 'user_id': 16493218.0}}