Solving functional equations for unknown functions in lambda calculus
This is a known problem, known as Higher Order Unification.

Unfortunately, this problem is undecidable in general. There is a decidable fragment, known as Miller's Pattern Fragment. It's used extensively in, among other things, typechecking of dependently-typed programs with metavariables or pattern matching. This fragment is where unification variables are applied only to distinct bound program variables.

This paper provides a great tutorial of how higher order unification works, and walks through a (relatively) simple implementation of it.

Unfortunately, it doesn't look like your function falls into this pattern fragment. That said, what I'm looking seeing is pretty similar to function composition. Does the following function satisfy your property?

$B = \lambda f\ g\ x\ \ldotp f\ (g\ x) $

We have: 


$B\ f\ (\lambda x . M)$ 
$= B\ f\ (\lambda y . [y/x]M)$ by $\alpha$-equivalence
$ = \lambda x . f\ ((\lambda y. [y/x]M) x) $
$= \lambda x . f\ ([x/y][y/x]M)$
$= \lambda x . f\ M$