Can you apply neural networks to design algorithms?
There are a few issues to consider. 

If P!=NP, this approach is doomed from the start because you can't try every one of the infinite number of possible algorithms in finite time. That's I think the main deal-breaker here. Unless P=NP, which most people doubt, then the best you could ever do is say "my AI failed to find a polynomial-time algorithm in X amount of time", which doesn't mean no such algorithm exists. If you assume that P=NP, then in principle, an evolutionary algorithm, neural network, or other search technique could potentially find an example of a polynomial algorithm for an NP-complete problem, and that's all you need to prove that P=NP.

In practice, there are some challenges to deal with. I'm sure certain approaches would be more natural to try than others. Neural nets require numeric vectors of inputs and outputs, so you need to develop some way to encode an algorithm as a vector and a way to determine a binary "is_polynomial" so that you have an error gradient to follow. Or I suppose a continuous function measuring "degree of polynomial-ness" if such a thing makes sense. You're certainly going to need some way to measure success regardless of approach, but something like Genetic Programming could theoretically eliminate the need for a cumbersome translation of algorithm to real vector, since GP operates directly on an executable representation. Give it a Turing-complete set of operators to work with and it could presumably eventually find a polynomial solution if such a thing exists. 

It's also far from clear that you have a friendly search space to work with here. I think most of the time, your hypothetical GP or whatever is going to wander around a very flat plateau of individuals that don't even successfully solve the problem, regardless of run-time. You could attempt to introduce a gradient to follow by favoring shorter programs or programs that terminate faster or whatever thing you came up with, but there's no good reason to think that gradient points in the direction of the prize-winning algorithm you're looking for. Theoretically, there are fairly easy conditions to meet that say given infinite search time, a given algorithm will eventually find any arbitrary solution that exists, but a workable algorithm in practice tends to require there be some structure in the search space that can be exploited. If you make a local move somewhere in the space that looks like it's better, you want that move to take you generally closer to the solution you're looking for. If the world looks basically random in terms of what moves me closer or further away, search algorithms struggle to do better than just randomly trying things.

Short answer, it's conceivably possible to build an algorithm that searches for poly-time solutions to an NP-hard problem, but the world is likely constructed so that that won't actually solve the problem you hope to solve.