Checking brackets nesting in a string
Naming


Class name: Solution A solution for what?
Method name: solution - again I ask: Solution for what?
Parameter name: S - And what on earth is S? Parameter names are by convention named beginning with a lowercase letter.


None of these three names are descriptive names. It is easier to read and understand code when the names tell a bit about what it is that the code does. Now, naming things are hard, and something that programmers do a lot. For this case, consider names such as BracketNestingVerifier, verify and input. You might even be able to come up with better names.

General things


Use Generics for your HashMap. Since Java 1.5, it's been possible to define which kind of HashMap it is. Is it a HashMap of &lt;Integer, String&gt;? &lt;String, Byte&gt;? No, in your case; it's HashMap&lt;Character, Character&gt;
Why do the method return an int when it would make much more sense to return a true/false boolean? Returning a boolean allows you to simplify the end of your method

if (last == -1) { // note the fixed spacing ;)
    return 1;
}
return 0;


To simply return (last == -1);


Spacing

I find that code like this:

for(int i=0; i&lt;S.length(); i++){


Is much more readable if you use spacing (and a better variable name for S of course)

for (int i = 0; i &lt; input.length(); i++) {


Algorithm

Now, about that algorithm... You have a variable called stack and yet it isn't a stack structure, it's only an array. Using a somewhat real stack structure can be helpful. In the code below, I have used the LinkedList class to provide this structure (Even though there is also a Stack class, I like the Deque interface and wanted to go with LinkedList).

In this code, I have improved the variable names, the return type, and I am using a HashSet to store all "special" characters in. The contains method of a HashSet is performing in constant speed, so you should not experience any significant lack of performance. I noticed that in your if-statement, you had hard-coded the values that were put into your HashMap, this is what made me want to add the HashSet.

To make this work, I had to flip the key/values of your HashMap. The key is now the starting character, and the target is the expected ending character.

The loop through the string here is pretty simple and straight-forward:


Have we waited for this character? If so, remove it from the list of expected characters we are waiting for (The "stack")
If the above is not true, does this character have a matching ending character? If it does, add it to the stack
Again, if the above also is not true, is this character regarded as a special character? If it is, then we know here already that we have failed so we can return from the method.




public boolean simonsVerify(String input) {
    HashMap&lt;Character, Character&gt; matches = new HashMap&lt;Character, Character&gt;();
    matches.put('{', '}');
    matches.put('(', ')');
    matches.put('[', ']');

    Set&lt;Character&gt; specialChars = new HashSet&lt;Character&gt;();
    Deque&lt;Character&gt; expected = new LinkedList&lt;Character&gt;();
    for (Entry&lt;Character, Character&gt; ee : matches.entrySet()) {
        specialChars.add(ee.getKey());
        specialChars.add(ee.getValue());
    }

    for (int i = 0; i &lt; input.length(); i++) {
        char next = input.charAt(i);
        Character expect = expected.peekLast();
        if (expect != null &amp;&amp; expect == next) {
            expected.removeLast();
        }
        else if (matches.containsKey(next)) {
            expected.addLast(matches.get(next));
        }
        else if (specialChars.contains(next)) {
            return false;
        }
    }
    return true;
}


Here is the testing (with JUnit) I used, for both your original code and also for my method:

@Test
public void testVerifying() {
    assertTrue(verify("{,},({,}),[,]"));

    assertFalse(verify("{(,},({,}),[,]"));
    assertFalse(verify("{,)},({gfdgfd}),[,]"));
    assertFalse(verify("{(,}),({,}),[,]"));

    assertTrue(verify("{,},(,),[[,]]"));
    assertTrue(verify("{,},(,),[,]"));
    assertTrue(verify("{{,}},(,),[,]"));
    assertTrue(verify("{,},((,)),[,]"));
    assertTrue(verify("{,},(,),[,]"));
    assertTrue(verify("(), {{{,}}},(,),[,]"));
}


There might even be a better way to do this algorithm, this is just my way.