Key management in Java + Remembering passwords
There is no way you can make this secure without using some kind of DRM or additional devices. If an attacker has access to the running program then the attacker is able to retrieve the password. You can of course try to hide and / or scramble the password in memory, but that's about anything you can do. It then comes down to obfuscation.

Sometimes the operating system has access to additional resources to secure passwords. You need to provide your password to the operating system almost by default. So for instance Windows has a Credential Manager that you can access from your applications. I looked around and I found some way to make use of it here (warning: I've never used or viewed this solution). Obviously such code won't be portable across operating systems.

In the end, you're probably better off trying to protect your application itself. Make sure your input is sanitized and that the debug port isn't left wide open. Make sure you limit access to the application. Once an attacker has full access to a user account - or worse, a system account - then your application security is probably the least of the user's worries.