What does it mean to be "closed" under beta reduction?
The claim is that after applying β-reduction to an expression in A-normal form you can be left with an expression no longer in A-normal form.

The only explicit definition I can find of A-normal form is not consistent with the definition Kennedy seems to be (implicitly) using in this paper.  Wikipedia defines A-normal form as the subset of lambda calculus expressions where only constants, $\lambda$-terms, and variables can be arguments of function applications, and then (vaguely) says that results of non-trivial expressions must be captured by let-bound variables.

That is: f(g(x)) is not in A-normal form because the argument to the application of f is another application (g(x)) rather than a constant, $\lambda$-term, or variable.  This expression in A-normal form would be something like let y=g(x) in f(y).

Kennedy uses the vague "definition"


  a let construct assigns names to every intermediate computation. (Section 1.1.ANF).


But then in Section 1.2 he gives an example of an A-normal form not being preserved under β-reduction


  Consider the ANF term let x = (λy.let z = a b in c) d in e.  Now naïve β-reduction produces let x = (let z = a b in c) in e which is not in normal form.  The ‘fix’ is to define a more complex
  notion of β-reduction that re-normalizes let constructs (Sabry and
  Wadler 1997), in this case producing the normal form let z = a b in (let x = c in e).


So apparently Kennedy's variant of A-normal form places some kind of restriction on what can be in the assignment part of a let clause, but I can't figure out what (or why) that restriction is.  In addition to Kennedy's paper you linked, I looked in several of his references:


  Amr Sabry; Philip Wadler: A reflection on call-by-value. ACM T. Prog. Lang. and Sys. (TOPLAS), 19(6):916-941, 1997.
  
  Amr Sabry; Matthias Felleisen: Reasoning about Programs in Continuation-Passing Style. Lisp and Symbolic Computation 6(3-4):289-360, 1993.
  
  Flanagan, Cormac; Sabry, Amr; Duba, Bruce F.; Felleisen, Matthias: The Essence of Compiling with Continuations. Proc. ACM SIGPLAN Conf. on Programming Language Design and Implementation, (PLDI):237-247, 1993.