Make 10 from the numbers 1,1,5 and 8 using the 4 operations, by brute force
I like your code but don't have a solution that improves on brute force. Just a few comments:

(1) You can eliminate your for loop assigning to res3 like so:

res3 = data.frame(V1 = unlist(lapply(res2$expr, function(temp) sapply(1:6, function(j) insert_str(temp,c("(", ")"),x[j,])))))


Whether this is a good idea is debatable, but it's slightly faster when I test it as compared to the for loop (a difference of about 10%). Perhaps lapply would become more advantageous as you increased the number of digits or operations to be explored.

(2) The obvious speed-related comment is to use data.table. This is another change that will more noticeably improve performance as you move to a larger-scale problem since you can benefit from the in-place operations.

(3) You are working with a manageable set of numbers here, but in the future you might want to seek solutions in parts. For example, check half the possibilities and only check the other half if you don't find a solution. Similarly, you don't have to assume there's only one set of parentheses but could make the choice to check the second set if your first search yielded no results.

(4) When you know the numbers in advance or you're coding to a specific set of numbers, it might be helpful to eliminate all rows that have a certain combo that couldn't possibly be in a correct answer. For example, I'd get rid of anything with 8*5 or 5*8 since nothing you can do with the 1's can bring that expression down to anything close to 10. Similarly I'd get rid of anything with 1/1 and 1*1 since these also can't lead to a solution.  Doing this would eliminate 1152 out of 4608 possibilities. (Note that I'm relying on standard order of operations rules.) 



I don't feel any of these comments are super helpful but wanted to write since I enjoyed reading your code and since Code Review doesn't always give much feedback to us R users. Hopefully others will add more interesting things.