# With Great Freedom…

Non-Standard Evaluation is a pretty controversial topic in R circles, and even in the R documentation. Whether you like it never, sometimes, or always, is neither here nor there. What matters is that R allows it. Not many languages give the programmer the power to implement, use, and abuse Non-Standard Evaluation (“NSE”), or anything like it.

So what is NSE? Very roughly, it is to programmatically modify an expression or its meaning after it is issued but before it is executed. You can think of an “expression” an R command you might type at the prompt or in an R script1. For example in:

subset(mtcars, hp > 250)
                mpg cyl disp  hp drat   wt qsec vs am gear carb
Ford Pantera L 15.8   8  351 264 4.22 3.17 14.5  0  1    5    4
Maserati Bora  15.0   8  301 335 3.54 3.57 14.6  0  1    5    8

subset intercepts the expression hp > 250 before it is run, and changes its meaning by allowing the name hp to resolve against columns from mtcars instead of just against objects in the workspace. In other words, subset performs non-standard evaluation on the expression hp > 250.

Compare to what happens with an expression that is evaluated in the standard way:

mtcars[hp > 250,]
Error in [.data.frame(mtcars, hp > 250, ): object 'hp' not found

We get an error because hp is not defined in my workspace.

# Standard Evaluation

When we type a simple expression at the R prompt and hit ENTER, R computes its value (a.k.a. evaluates it):

w <- c("am", "I", "global")
rev(w)    # reverse the order of w
[1] "global" "I"      "am"    

Ironically the process of evaluation in R is mostly about looking things up rather than computing them. In our example, when we hit ENTER R first looks for the named objects in our expression: rev, and w2. Lookups are done through data structures called environments, represented in blue in this flipbook:

After the lookup rev(w) becomes:

(function(x) UseMethod("rev"))(c("am", "I", "global"))

rev is replaced by the definition of the function from the base environment, and w by the character vector from the workspace. The workspace, also known as the global environment, is where name -> value mappings created at the R prompt are kept (e.g. w <- c("am", "I", "global")). Our substituted expression is a bit weird in appearance, but more useful to R as the names rev and w have no inherent meaning.

There is a fair bit of hand waving here3, but for the most part the lookup-and-substitute model reflects observed R behavior.

R is not yet done with our expression. More on this shortly, but first lets talk about environments.

# Environments

Environments are akin to named lists with a few additional features. All elements are uniquely named, and the names can be hashed for fast lookups. The name -> object mappings are known as the “Frame” of the environment. Environments also carry a link to an “Enclosing Environment” a.k.a. “Enclosure” (black arrows in the flipbook, pointing to the Enclosure). The Frame and the link to the Enclosure together make up the environment.

WARNING: Environments have reference semantics. Be sure you understand the documentation before you attempt to directly modify elements in environments, e.g. as in env\$name[3] <- 42. We do not write to environments here so this isn’t important for our discussion, but be wary when interacting with them directly.

R searches for a name in an environment’s Frame, and if it doesn’t find it moves on to its Enclosure. Enclosures are environments so they too carry a link to their own Enclosure. R repeats the search process until the name is found or it hits the empty environment, which does not have an Enclosure4. We’ll call an environment and the sequence of Enclosures it links to an “Environment Chain”5.

For expressions typed at the prompt the Environment Chain usually starts at the global environment and links the environments of all the attached packages6 (represented with the ... below) through to the base package. When searching for rev, R will work through this entire chain and retrieve the base::rev function from the base package7.

When searching for w R will find it immediately in the global environment and stop the search:

The first environment in the Environment Chain is called the “Evaluation Environment”8. When we say that expressions are “evaluated in” the Evaluation Environment we mean that they are evaluated according to the Environment Chain that starts with that environment.

When evaluating expressions at the R prompt, the Evaluation Environment is usually the global environment, so this may all seem moot. However, expressions can be issued in different Evaluation Environments, as is the case when they are part of the body of a function.

For a more complete treatment of environments and how they are used by R do read Saruj Gupta’s excellent “How R Searches and Finds Stuff”.

# In Functions

After the initial substitution of rev and w for the function and its argument, R will call (i.e “run”) the function. This means repeating the lookup-and-substitute process on any R expressions contained in the body of the function9, though with some additional wrinkles.

Most functions in R are “closures”. They are so called because they carry a reference to a “Function Environment”10 to use as an Enclosure. The Function Environment is typically the Evaluation Environment the function was defined in11. Other functions like the basic arithmetic operators are “primitives”. They are entry points into statically compiled machine code, and contain no R expressions or environments12.

Each time a closure is called a new environment is created, enclosed by the Function Environment, and with the function parameter -> value13 mappings as its Frame. This new environment becomes the Evaluation Environment for the expressions in the function body:

Once the function Evaluation Environment is set up (<rev> in the diagram14), R can proceed with the lookup-and-substitute process on the body of the function. In this case there is just the one expression UseMethod("rev") in the body.

The key observation is that the Environment Chain changed. It used to start at the global environment and pass through all the loaded packages (<...> in the diagrams). Now it starts at <rev> and goes directly to the base namespace environment as that is where rev is defined. Even though the Environment Chain no longer passes through the global environment, we can still access a copy of the object referenced by w via x in <rev>15.

On the left is the Environment Chain used to evaluate rev(w). On the right the one used to evaluate the body of the rev function:

<rev> is now the Evaluation Environment. In this context, the global environment becomes the “Calling Environment”, so named because it was the active Evaluation Environment when rev was called via the expression rev(w).

The lookup-and-substitute-and-call-closures process will continue recursively until we hit primitive functions, at which point actual computations happen in machine code16.

R’s resetting of Environment Chains to be based on the Function Environment instead of the Calling Environment is known as “Lexical Scope”. The term reflects that the Environment Chain is based on how the functions were “written”, instead of how they were called. This scoping mechanism was inherited by R from Scheme, not from S, as can be seen the original R paper.

It is possible for different environments in a chain to contain the same name. When this happens the object matched is the one from the first environment along the chain that contains that name. Consider what happens if we nest our simple rev(x) expression in a trivial function:

fun <- function() {
w <- c("am", "I", "fun")
rev(w)
}

When we call fun(), R creates a new Evaluation Environment enclosed by the global environment as that is where fun was defined. We also assign a new value for w in that Evaluation Environment. The evaluation chain for rev(w) within fun is different from rev(w) at the prompt:

The evaluator finds w in the function’s evaluation environment instead of the now-masked w in the global environment, and so substitutes a different value for it:

rev(w)
[1] "global" "I"      "am"    
fun()
[1] "fun" "I"   "am" 

# Non-Standard Evaluation

One of the simplest NSE functions in R is with:

L <- list(w=c("am", "I", "list"))
with(
L,
rev(w)     # same commmand
)
[1] "list" "I"    "am"  

What happened? L was somehow made part of the Environment Chain and the names in rev(w) were matched against it. L masked the name lookup as fun’s Evaluation Environment did, except we didn’t need to define a function.

While the concept is straightforward the execution is more complicated. with needs mechanisms for interrupting the evaluation, masking the active Environment Chain in some way with L, and resuming evaluation. R, being the strange language that it is, provides tools to do all this.

Let’s implement a version of with at the prompt to see how this might be done:

expr  <- quote(rev(w))   # capture expression
Lenv <- list2env(L)      # convert list to env
eval(expr, envir=Lenv)   # invoke the evaluator
[1] "list" "I"    "am"  

quote captures a parsed R expression before the evaluator gets to it. Once captured the expression no longer self-evaluates at the prompt:

expr    # we get expression back, not result of evaluating it
rev(w)

It may not seem like much, but it’s a big deal R allows this: unevaluated R expressions can be directly manipulated by R itself. For now all we care about is that the evaluation is on hold, but you can do some interesting things with this as in the RPN in R post.

list2env creates a new environment with the Calling Environment (global env here) as the enclosure. This is natural because environments are akin to named lists.

Lenv
<environment: 0x7fbbd02741c0>
ls.str(Lenv)
w :  chr [1:3] "am" "I" "list"
parent.env(Lenv)   # parent.env == enclosure
<environment: R_GlobalEnv>

eval provides a mechanism to resume evaluation while explicitly specifying an Evaluation Environment:

eval(expr, envir=Lenv)
[1] "list" "I"    "am"  

This is what the Environment Chain looks like right before substituting the w name17:

By changing the Environment Chain we made the evaluation “Non-Standard”. This is similar to what functions do, but because we are doing it by interrupting evaluation and manually setting the Evaluation Environment it becomes “Non-Standard”.

Now look at what happens if we add another environment to the chain with the rev name in it:

L2 <- list(rev=toupper)
L2env <- list2env(L2, parent=Lenv)
eval(expr, envir=L2env)
[1] "AM"   "I"    "LIST"

We can change the meaning of both functions and values by modifying the Environment Chain. We set the enclosure to L2env to be Lenv with the parent parameter to list2env18. We could also have directly added a rev mapping to Lenv.

Manipulating the Environment Chain is not the only way to perform NSE. Anything that changes the meaning of an expression after it is issued relative to what would have happened in standard evaluation is NSE.

# NSE In Functions

Compare:

with(L, rev(w))
[1] "list" "I"    "am"  

To our hack-at-the-prompt version:

expr  <- quote(rev(w))
Lenv <- list2env(L)
eval(expr, envir=Lenv)
[1] "list" "I"    "am"  

It would be nice to implement with ourselves, but if we try to use quote inside a function to get what someone types in as the argument to that function we are disappointed:

with2 <- function(data, expr) {
quote(expr)                 # capture expression
# rest of function will go here later
}
with2(L, rev(w))
expr

What we want to quote is the expression supplied as the argument expr, not the name expr. Thankfully R in its infinite flexibility provides a mechanism for doing this with substitute:

with2 <- function(data, expr) {
substitute(expr)
# rest of function will go here later
}
with2(L, rev(w))
rev(w)

When called within a function on a function parameter, substitute acts like quote except it substitutes the unevaluated expression passed as the argument. This allows us to implement with:

with2 <- function(data, expr) {
expr2 <- substitute(expr)
denv <- list2env(data, parent=parent.frame())
eval(expr2, envir=denv)
}
with2(L, rev(w))
[1] "list" "I"    "am"  

We can do a bit better because eval supports adding a list-like element to the Environment Chain out of the box, saving us the list2env step:

with2 <- function(data, expr) {
expr2 <- substitute(expr)
eval(expr2, data, enclos=parent.frame())
}
with2(L, rev(w))
[1] "list" "I"    "am"  

list2env specifies Enclosures with parent=, whereas eval does so with enclos=. This is unfortunately one of those areas where R is not as clear as it could be about the names of things, and there are closely related concepts that should be clearly distinguished19.

So, what’s the parent.frame() business?

# Environmental Dichotomy

substitute allows us to implement functions that perform NSE on their arguments. This is a powerful feature, but with it comes a new class of potential errors. Recall that functions create their own Evaluation Environments, but the expressions that we capture and wish to re-evaluate refer to names in the Calling Environment. For things to work correctly we must hand craft an Environment Chain that connects to the Calling Environment at the correct point.

When parent.frame() is called in a function body, it returns the Calling Environment. with2 uses eval to create an Environment Chain that starts with our list L and with the Calling Environment as the Enclosure:

with2 <- function(data, expr) {
expr2 <- substitute(expr)
eval(expr2, data, enclos=parent.frame()) #<< Calling Env as Enclosure
}
with2(L, rev(w))
[1] "list" "I"    "am"  

Compare to what happens when we don’t do this:

with2_bad <- function(data, expr) {
expr2 <- substitute(expr)
eval(expr2, data)
}
with2_bad(L, rev(w))
[1] "list" "I"    "am"  

No problem right? Except:

expr2 <- c("pathological", "I", "am")
with2_bad(L, rev(expr2))
expr2(rev)

Wow, what the heck is that? Instead of reversing c("pathological", "I", "am") we reversed the expression rev(expr2). Here is what with2_bad effectively did:

expr2 <- quote(rev(expr2))
expr2
rev(expr2)
rev(expr2)
expr2(rev)

It’s mind boggling enough that we can reverse an unevaluated R expression, but imagine you’re the user and everything was perfectly fine until the fateful day you used the expr2 name in your expression. On the “Principle of Least Surprise” scale this outcome is right there with finding a rat in your toilet bowl.

So what happened? We had a name clash with the expr2 symbol that exists both in with2_bad’s Evaluation Environment and in the Calling Environment. This is what the Environment Chain looks like when eval begins the lookup-and-substitute process on rev(expr2) (<w2_b> is with2_bad’s Evaluation Environment):

When evaluating with with2_bad as the enclosure expr2 resolves to the object in with2_bad, instead to the one in the global environment.

As we saw earlier with2 bypasses its Evaluation Environment when calling eval:

So it works as expected:

expr2 <- c("pathological", "I", "am")
with2(L, rev(expr2))
[1] "am"           "I"            "pathological"

There are other ways things go wrong. Suppose we wanted to use with2_bad inside a function that is careful not to use names used by with2_bad:

friendly_fun <- function(L) {
z <- c('hello', 'friend', '!')
}
friendly_fun(L)
[1] "1 5 9 8 am I list"

What happened? Instead of using the z in friendly_fun’s Evaluation Environment, which in this case is with2_bad’s Calling Environment, we used the z in the global environment. Why? Because the with2_bad’s Function Environment, and hence the Enclosure of its Evaluation Environments, is the global environment. So friendly_fun’s Evaluation Environment (<ffun> below) is not part of the Environment Chain:

Again, this is resolved by explicitly setting the Enclosure in eval to the Calling Environment with parent.frame as with2 does:

friendly_fun <- function(L) {
z <- c('hello', 'friend', '!')
with2(L, paste(c(z, w), collapse=' '))
}
friendly_fun(L)
[1] "hello friend ! am I list"

The chain is now as we want it to be:

Since we can capture user expressions, we can also edit them before we run them. For example, suppose we want to write a function to sum things in the context of a data frame, by group. It could look something like:

mean_call <- function(expr) {
call <- quote(mean(NULL))
call[[2L]] <- expr
call
}
mean_by_grp <- function(data, expr, grp) {
call_env <- parent.frame()
grp <- eval(substitute(grp), data, enclos=call_env)
data <- split(data, grp)
expr <- mean_call(substitute(expr))
res <- setNames(numeric(length(data)), names(data))

for(i in seq_along(data)) {
res[[i]] <- eval(expr, data[[i]], enclos=call_env)
}
res
}

mean_call takes an unevaluated R expression of the type produced by quote or substitute and wraps it in mean:

expr <- quote(Sepal.Length / Sepal.Width)
expr
Sepal.Length/Sepal.Width
mean_call(expr)
mean(Sepal.Length/Sepal.Width)

mean_by_grp splits the data into groups and then evaluates the above expression in the context of those groups:

mean_by_grp(iris, Sepal.Length / Sepal.Width, Species)
    setosa versicolor  virginica
1.47       2.16       2.23 

Don’t worry too much about the details, the important point is that we’re calling eval with enclos=call_env. Previously this ensured everything worked fine, and it seems to here, but what if:

mean <- function(...) stop("Boom")
mean_by_grp(iris, Sepal.Length / Sepal.Width, Species)
Error in mean(Sepal.Length/Sepal.Width): Boom

Normally this problem is obviated by the use of packages and associated namespaces. That ensures that when package functions evaluate they resolve their names without interference from user objects. But this doesn’t work here because we explicitly bypass the normal Evaluation Environment and request evaluation in the Calling Environment.

Let’s examine the modified expression mean_call produces:

mean(Sepal.Length/Sepal.Width)

The problem is that we need mean to be resolved according to our function’s Evaluation Environment, but Sepal.Length/Sepal.Width to be resolved according to the Calling Environment. We can only evaluate an expression in a single environment, so we’re stuck.

We could use base::mean instead of mean, but this would still fail if someone redefined :: in the Calling Environment (and yes, they can). Another option is to manually substitute the actual function instead of the name of the function:

mean_call <- function(expr) {
call <- quote(NULL(NULL))  # call template
call[[1L]] <- mean         # actual function object
call[[2L]] <- expr
call
}

Let’s pretend our functions here are in a hypothetical package pkg with no dependencies. In that case, this is what the Environment Chain looks like as mean_call is injecting mean into the call template (call, shown as NULL(NULL)) in the second line of the function20:

<mcll> is mean_call’s Evaluation Environment, and <pkg> the package namespace. The latter has for enclosure the base namespace, so the version of mean defined in the global environment does not interfere. We get:

mean_call(expr)
(function (x, ...)
UseMethod("mean"))(Sepal.Length/Sepal.Width)

This looks ugly, but by pre-resolving mean to the correct function we can safely eval the expression in a different Environment Chain:

<mbg> is the Evaluation Environment of mean_by_group, but that Environment Chain is bypassed by eval to resolve Sepal.Length, Sepal.Width, and / in iris enclosed by the Calling Environment.

# Conclusions

Geez, that was a lot less fun than I thought it was going to be when I started writing the post. Trying to keep all those closely related but critically different concepts distinct while remaining faithful21 to the documentation was exhausting. I imagine reading through this likely had a similar effect on you, so if you managed to get this far I’ll consider it a small victory.

Unfortunately even the standard evaluation model in R is complex, so messing around with it for NSE is even more so. It is particularly challenging because in many cases incorrect usage of NSE works fine, but then proceeds to break in the most bewildering ways under global usage. Further compounding things is the challenging terminology (looking at you parent.env vs parent.frame vs environment).

I do hope that whatever clarity this post might add on the topic is not terminally muddled by its length. If you have any feedback I’d be happy to hear it.

# Appendix

## Acknowledgments

The following are post-specific acknowledgments. This website owes many additional thanks to generous people and organizations that have made it possible.

## References

• 2020-05-06: png acknowledgment.
• 2020-07-15:
• Changed “command” to “expression” to be more consistent with terminology used in R’s documentation (h/t Lionel Henry). A related change was to make the example function signatures more consistent with those of base.
• Removed superfluous footnote (h/t Michael Chirico).

## Session Info

sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Mojave 10.14.6

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] Rcpp_1.0.3       bookdown_0.18    digest_0.6.25    later_1.0.0
[5] mime_0.9         R6_2.4.1         jsonlite_1.6.1   magrittr_1.5
[9] evaluate_0.14    blogdown_0.18    stringi_1.4.6    rlang_0.4.5.9000
[13] rstudioapi_0.11  promises_1.1.0   rmarkdown_2.1    tools_3.6.3
[17] stringr_1.4.0    servr_0.16       httpuv_1.5.2     xfun_0.12
[21] compiler_3.6.3   htmltools_0.4.0  knitr_1.28      

1. Originally I used “command” throughout the post to avoid ambiguity with R’s expression object, which roughly could be described as a list of R commands. The early drafts of this post switched between “expression” and “command”, but in the end I settled on “command”. Lionel Henry convinced me to switch to “expression” as that is what the R documentation uses despite the ambiguity.

2. The parentheses in rev(x) are not considered “names”. They are syntax tokens that are used to parse the expression. Opening parentheses that are not part of the syntax of a function call are names, as in (1 + 2) * 3.

3. Items we gloss over include but are not limited to:
• Function parameters are evaluated lazily so they are only fetched when they are referenced within a function body as something other than an argument to a closure (e.g. as an argument to a primitive or other entry points into statically compiled machine code, or simply as a stand-alone reference to the name as in force).
• What is fetched is a pointer to the location in memory the R objects are stored in, not the objects proper. Since in R memory addresses are not directly visible, we’ll treat the pointers as if they are the actual R objects they reference.
• The byte-compiler affects the nature of non-evaluated code prefetching the objects names point to and performing other optimizations, and generally bypassing many aspects of the “normal” evaluation process.
• Names representing called functions are only resolved against names that are associated with functions.
• Lookups from the global environment down are usually done against a “global” hash table that replicates the semantics of the Environment Chain, but isn’t actually a chain.
• The base environment has an enclosure, although it is the empty environment which itself does not have an enclosure.
• And more.

4. One other possibility is that a chain connects back on itself, forming an infinite loop. This doesn’t usually happen unless people are messing with environments in a way they shouldn’t be.

5. This is not a formal R term. In the documentation the term “environment” is often used as we use Environment Chain here.

6. Packages loaded with library, or those that are part of the “base” set of packages that are pre-attached by R (e.g. stats, etc.), or any other environments attached with attach.

7. Well, it will unless you have loaded a pathological package that also defines a rev function, or if you yourself define a rev function in the global environment.

8. In some contexts the term “environment” is taken to mean the entire Environment Chain. In fact, the term “Environment Chain” is not a term used by the R documentation.

9. Functions in R that are defined with the function keyword (i.e. fun <- function(<formals>) {<body>} have three components: formals (a.k.a. parameter list), body (a.k.a. the set of expressions that the function executes), and the Function Environment, which I prefer to call the Function Enclosure. See full details in the R Language Definition.

10. I dislike the term “Function Environment” as because there are two environments that could be considered the “function environment”: the evaluation environment generated each time the function is invoked, and the environment to use as its enclosure. Additionally, “Function Environment” suggests the environment belongs to the function. Strictly speaking, what we call the “Function Enclosure” is “environment to use as the enclosure for the function evaluation environments”.

11. That is the default behavior, but is possible to change what the enclosing environment for any given function’s evaluation environments will be with environment(fun) <-. See also the discussion of function evaluation environment terminology.

12. .Primitive, .Internal, and a few other special R functions are entry points into the statically compiled machine code that actually does the work of computation. Once those functions are invoked there is no more R code until they return, except if for some reason the statically compiled routines themselves invoke the internal version of eval or similar to evaluate R code.

13. In reality instead of the values of the arguments, R stores the expression passed as the argument along with the environment to evaluate it in. If the argument is used by the function then the expression is evaluated in its environment to produce the value. This is typically the same as if R had used the values directly, but there are some circumstances where behavior is changed due to this feature. Values that are stored as the expression used to generate them along with the environment to evaluate that expression are known as promises

14. We use <rev> for clarity; in reality function Evaluation Environments don’t have names and would be displayed in R as their memory address, something like <environment: 0x7fce5c930238>.

15. R only copies objects when necessary to maintain the “pass-by-value” illusion. See “The Secret Lives Of R Objects” for more details.

16. One of the interesting things about R is that R code never modifies or creates objects. The only effect of running R code is to substitute the names for the R objects they point to. It is only once we enter into statically compiled machine code routines that objects are created / modified. This may seem obvious to some, and even though I’ve always been aware of it to some degree, I still find it interesting.

17. Environments can have multiple children as we saw previously. While not shown here fun’s environment from earlier is still linked to the global environment.

18. Unfortunately the terminology around the hierarchical relationship of environments in R is muddled. In early days the term “parent” was used for enclosures, which is why parent.env returns the enclosure of an environment. However, there is also the parent.frame, which is the Calling Environment, i.e. the evaluation environment of the expression that that triggered the current evaluation. Perhaps S’s lack of lexical scope contributed to this infelicity. Another oddity is that environment(fun) is used to retrieve the environment to use as an Enclosure, i.e. similar in semantics to parent.env.

19. Unfortunately the terminology around the hierarchical relationship of environments in R is muddled. In early days the term “parent” was used for enclosures, which is why parent.env returns the enclosure of an environment. However, there is also the parent.frame, which is the Calling Environment, i.e. the evaluation environment of the expression that that triggered the current evaluation. Perhaps S’s lack of lexical scope contributed to this infelicity. Another oddity is that environment(fun) is used to retrieve the environment to use as an Enclosure, i.e. similar in semantics to parent.env.

20. The lookup-and-substitute metaphor is a little strained here as [[<- is a primitive, but please humor me.

21. I’m sure better nitpickers than me will find some spots I failed to do this in properly.

Brodie Gaslam is a hobbyist programmer based on the US East Coast.
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 0, "mathjax_display_tex": 0, "mathjax_asciimath": 1, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "math_score": 0.7789421081542969, "perplexity": 2772.9697007624077}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "warc_path": "s3://commoncrawl/crawl-data/CC-MAIN-2021-39/segments/1631780056974.30/warc/CC-MAIN-20210920010331-20210920040331-00679.warc.gz"}