Inverting One-Way Functions
As @Cryptostatis commented, Kerckoff's principle dictates that we must consider that the adversary knows the function.

More fundamentally, although the word “adversary” naturally suggests it must be a sentient being such as a human, remember that an adversary is in fact just an algorithm. The algorithm $A$ which on input $x$ returns $x-123$ inverts your function with probability $1$, demonstrating that your function is not one-way. How a human attacker would actually be able to "come up" with this algorithm is irrelevant, what matters is that such an algorithm exists.