Inverting One-Way Functions One of the conditions that a one-way function has to satisfy is the following:

$$Pr[A(f(x))\in f^{-1}(f(x))] \leq negl(n)$$

Now, suppose that we have the following function that's not one way:
$$f(x) = x + 123$$
We know this isn't one-way since we can easily invert; that is:
$$f^{-1}(x) = x - 123$$
But from the perspective of the adversary, if we have f(1) = 124, it would still essentially "brute-force" its way in order to find the inverse, right? In other words, the adversary would input random x's into f(x) until it finds a solution of 124 rather than formulating an inverse function (like a human normally would).

Is my understanding correct?
As @Cryptostatis commented, Kerckoff's principle dictates that we must consider that the adversary knows the function.

More fundamentally, although the word “adversary” naturally suggests it must be a sentient being such as a human, remember that an adversary is in fact just an algorithm. The algorithm $A$ which on input $x$ returns $x-123$ inverts your function with probability $1$, demonstrating that your function is not one-way. How a human attacker would actually be able to "come up" with this algorithm is irrelevant, what matters is that such an algorithm exists.