We recently clustered an application, and it came to light that because of how we're doing SSL offloading via the load balancer in production it didn't work right. I had to mimic this functionality on my local machine by SSL offloading Apache with a proxy, but it still isn't a 1-to-1 comparison. Similar issues can arise when dealing with stateful applications and sticky sessions. What would be the industry standard for testing this kind of production black box scenario in a local environment, especially as it relates to clustering?
How to test issues in a local development environment that can only be introduced by clustering in production?
web development;session;state;cluster
It can be expensive and painful, but in the end you need to have a local cluster.  Trying to simulate race conditions, contention and the like are very hard on a single PC (my interpretation of local development environment).  From past experience I would suggest:Push very hard to get a production level cluster into your test/dev environment, You can scale back some to keep costs down, but the profile needs to be representative of your production environment.Get periodic (weekly?) refreshes of your production DB into your dev/test environmentInstrument your code for high amounts of logging in this environmentImplement remote debugging on this cluster as best you can given your particular language and deployment stackI have used this strategy in the past and was able to identify the underlying bug.  Then I was able to develop locally and unit test locally before deploying the fix to the test cluster.