Kattis "Chasing Subs" custom decryption challenge
Alternative approach

You are repeating a lot of work.  Each time you advance a character, you rebuild the matching.  But you don't need to do that.  Instead create an array that says where to find the next instance of that character in the string.  

public static int buildNextArray(String s) {
    int[] next = new int[s.length() - 1];
    for (int i = 0; i &lt; next.length; i++) {
        next[i] = s.indexOf(s.charAt(i), i + 1) - i;
    }

    return next;
}


This converts the string into a unique array holding the relevant information:  how far away is the next instance of that character in the string?  Now, if the array you build for the decrypted key matches a portion of the array for the encrypted line, then it is a possible cipher match.  

I'll leave the exact implementation for you to develop.  Note that a negative number in the array for the key may match an unequal number in the array for the line.  You may want to think about when it can match and when it can't match.  

I would expect this approach to be faster because you can quickly see if a particular sequence matches by climbing the sequence for a particular letter.  Consider the case where the input is 


  abcdefghijklmnopqrstuvwxyz
  abca


In your version, you would match the first three letters to all twenty-three possible sequences and only find out on the fourth letter that it didn't work.  With my suggestion, you would find out on the second letter that it won't work.  And while there may be some inputs for which this is slower, the general result will be faster.  Because starting with three or more different letters is more common than having the first and fourth letter the same.  

It also saves the overhead of the HashSet and HashMap accesses.  Instead, you can do array accesses.  

Some other coding comments that won't affect performance:  

Code to the interface


            HashMap&lt;Character, Character&gt; dictionary = new HashMap&lt;&gt;();
            HashSet&lt;Character&gt; used = new HashSet&lt;&gt;();



This could be 

            Map&lt;Character, Character&gt; dictionary = new HashMap&lt;&gt;();
            Set&lt;Character&gt; used = new HashSet&lt;&gt;();


It won't matter much here, but in general, you want to code to the interface.  It makes it easier to switch implementations later.  

Delegate


        for (int i = 0; i &lt; line.length() - key.length() + 1; i++) {
            HashMap&lt;Character, Character&gt; dictionary = new HashMap&lt;&gt;();
            HashSet&lt;Character&gt; used = new HashSet&lt;&gt;();

            for (int j = 0; j &lt; key.length(); j++) {
                Character keyChar = key.charAt(j);
                Character sourceChar = line.charAt(i+j);

                if (dictionary.get(keyChar) == null) {
                    if (!used.contains(sourceChar)) {
                        dictionary.put(keyChar, sourceChar);
                        used.add(sourceChar);
                    }else{
                        break;
                    }
                } else if (!dictionary.get(keyChar).equals(sourceChar)) {
                    break;
                }

                if (j == key.length() - 1) {
                    counter++;
                    if (counter == 1) {
                        startIndex = i;
                    }         
                }
            }
        }



This gets a lot simpler if you make a helper method.  

    public bool canMatch(String haystack, String needle) {
        Map&lt;Character, Character&gt; dictionary = new HashMap&lt;&gt;();
        Set&lt;Character&gt; used = new HashSet&lt;&gt;();

        for (int j = 0; j &lt; needle.length(); j++) {
            Character keyChar = needle.charAt(j);
            Character sourceChar = haystack.charAt(j);

            if (dictionary.get(keyChar) == null) {
                if (used.contains(sourceChar)) {
                    return false;
                }

                dictionary.put(keyChar, sourceChar);
                used.add(sourceChar);
            } else if (!dictionary.get(keyChar).equals(sourceChar)) {
                return false;
            }
        }

        return true;
    }


I flipped around some of the internal logic.  I find an if (false)/else structure confusing.  I would rather check truth.  And in this case, that also means that we can get rid of the else, as the original else clause finishes with a return (break).  This has a side effect of reducing the indent on the other clause, which can sometimes be helpful.  

Now, if the needle matches, we can return true.  Otherwise, we return false.  This allows us to change the caller 

    for (int i = 0; counter == 0 &amp;&amp; i &lt; 1 + line.length() - key.length(); i++) {
         if (canMatch(line.substring(i), key)) {
             counter++;
             startIndex = i;
         }
    }

    if (counter == 0) {
        return "0";
    }

    for (int i = startIndex + 1; i &lt; 1 + line.length() - key.length(); i++) {
        if (canMatch(line.substring(i), key)) {
            counter++;
        }
    }

    if (counter == 1) {
        return line.substring(startIndex, startIndex + key.length());
    }

    return Integer.toString(counter);


Now we don't have to compare j to the last index of the key on every iteration.  The helper method will only return true when that was true.  

I also changed the original method to return a string rather than print.  Then the caller can do the print.  This is a good habit that makes the method more flexible.  Of course, it would be even better if we returned something other than a string, perhaps a result object.