Normalization of a free particle in 1 dimension
To answer the question, over and above the errors noted in the other answer the integral is just not well defined - it is infinite. There are two ways to see this

The &quot;common sense&quot; method. Consider
$$ \int^\infty_{-\infty} \left[ A\sin(kx) + B \cos(kx)\right]^2dx$$
The quantity under the integral is positive semi-definite - it is always greater than or equal to zero. It describes oscillations all of which are positive. Each oscillation bounds a finite positive area between it and the x axis. As the bounds of the integral are all space there are an infinite number of oscillations. An infinite number of positive finite areas evaluates to an infinite area. Thus your integral is not well defined.

Maths - assuming A and B are real (but removing this restriction makes no difference to the final conclusion)


$$\left[ A\sin(kx) + B \cos(kx)\right]^2 = \left[A^2\sin^2(kx) + 2AB\sin(kx)\cos(kx) +B^2\cos^2(kx)\right]$$
The way to evaluate the integral is to use the identities
$$ 2sin(\theta)cos(\theta) = sin(2\theta) $$
$$ sin^2(\theta) = {{(1-cos(2\theta))}\over2}$$
$$ cos^2(\theta) = {{(1+cos(2\theta))}\over2}$$
Thus we can write the integral as
$$
\int^\infty_{-\infty} \left[A^2{{(1-cos(2kx))}\over2}+B^2{{(1+cos(2kx))}\over2}+ABsin(2kx)\right]dx
$$
The first term contains
$$
\int^\infty_{-\infty} \left[ A^2\right]dx
$$
This is infinite. Similarly the second term for $B^2$. Thus the integral is not well defined [yes this is slightly hand-waving but avoids a discussion of $cos(\infty)$ and similar]. So what you are doing wrong is trying to evaluate an integral that doesn't have an answer.
What you are running into is the fact that not all wavefunctions are normalisable. See Normalizing the solution to free particle Schrödinger equation in the physics stack exchange forum for more details