Configuration Handler
IMHO the code looks fine so far.

However, I see few improvements:

Class Design


The instance variables _configItems and _configFile should be readonly
There is no need to implement IDisposable because you don't reference resources that must be disposed.
It makes sense to pass the file as constructor argument to make the class reusable for any ini files.
Maybe it makes sense to provide also a constructor that receives a stream, so that the class can be used with embedded resource ini files and so on.
In method LoadConfig you could create the FileInfo object once and use it subsequently (File.Exists and IsFileLocked)


Behavior


The method IsConfigurationLine could be turned into a more tolerant version when trimming the line string before (to support comments where the '#' is not on the first position)
If the value contains also '=' characters, your method returns only the part between the '=' that separates the key and the value and the first occurring '=' in the value.
The class kills the application if loading the configuration file fails. That may be a valid behavior for your application, but it is nothing that should be decided by the method that loads the configuration code. IMHO a better approach is to throw the exception and let the calling code handle the behavior.
The key is case sensitive. I am not sure if that is desired, otherwise you can simply change that by defining the comparer when initializing the dictionary: _configItems = new Dictionary&lt;string, string&gt;(StringComparer.InvariantCultureIgnoreCase);