Should I delete cryptographic data from memory? When dealing with cryptographic data, is there a chance that attacker recovers portion of data I used in my program? Should I try to delete every bit of keys and encrypted data that I used, overwrite it with zeroes?

And one extra, how is situation different in languages with garbage collection like Python or Java?
The book Cryptography Engineering devotes part of a chapter to this topic. Overwriting sensitive data with zeroes is a good start, but there are lots of other considerations. 


If you rely on a language's default object destruction behavior to zero the memory, it's possible for an unexpected error to prematurely halt the program's execution without it cleaning up after itself. In Java, exception handling can discard object references without cleaning up the objects themselves. [Not sure if this is still the case]
Sometimes compilers will optimize away the code used to overwrite the memory because it's acting on a piece of memory with no other uses in the program. There are a few ways to trick the compiler into not performing this "dead store elimination". 
OS memory management can foil your attempts to delete data by paging it to the system's hard disk. There are ways of preventing this but they are language- and platform-specific. 
If the OS does a context switch the values in the registers are stored elsewhere in memory until execution resumes. This can cause sensitive data to end up in strange places. Modern processors also have multiple caches where sensitive data can be held. 
In languages where memory management is automatic it's even more difficult to know where exactly in the system your allocated memory is (and for how long). In Java, objects all live on the heap and are reference-counted as a part of garbage collection. When there are no more references to the object, it can be deleted by the next pass of the garbage collector. However, short of calling System.gc() there's no way of knowing when garbage collection will occur. And even if you force garbage collection it still might not execute immediately. This SO question talks a little bit about System.gc behavior.
Cold boot attacks happen when keys are always written to the same physical region of a computer's memory. Over time, the electromagnetic forces can cause the memory to retain a state after the power is shut off. 


This probably isn't all in scope for your application, but it should give you a good idea of the capabilities of an attacker with physical system access. The most important thing you can do is know your language and platform very, very well. For further reading, this Wikipedia article should give you a good start.