Are there established cryptographic hardness assumptions for chaotic systems?
Personally, I wouldn't bother with chaos-based encryption.  I have yet to see any scheme based upon chaos that strikes me as very promising from either a practical or a theoretical perspective.  The work on using chaos for encryption that I've seen ranks pretty high on my "bogometer".  I never see any sort of security proof or plausible evidence of security, and many such systems turn out to be broken when you take a close look.  And I find it hard to see any benefit over standard well-known approaches (e.g., CBC mode encryption; information-theoretic work based upon Wyner's wiretap channel; etc.).

But if you want to find something in this space, you might look at the following paper:


Semantic Security for the Wiretap Channel.  Mihir Bellare, Stefano Tessaro, and Alexander Vardy.  CRYPTO 2012.


It has plenty of theoretical/foundational content, and is in the vicinity of the sort of thing you mention.