Find the most similar subsequence in another sequence when they both numeric and huge
I am assuming you are doing this as a stepping stone to find the perfect match
My reason for assuming this is that you say:

I wish to find the most likely candidate part of S1 to be equal to S2.


Start with the first value in the small array.

Make a list of all indices of the big array, that match that first value of the small array. That should be very fast? Let's call that array indices, and it may have values [2,123,457,513, ...]

Now look at the second value in the small array. Search through all positions indices+1 of the big array, and test for matches to that second value. This may be faster, as there are relatively few comparisons to make. Write those successful hits into a new, smaller, indices array.

Now look at the third value in the small array, and so on. Eventually the indices array will have shrunk to size 1, when you have found the single matched position.


If the individual numerical values in each array are 0-255, you might want to "clump" them into, say, 4 values at a time, to speed things up. But if they are  floats, you won't be able to.
Typically the first few steps of this approach will be slow, because it will be inspecting many positions. But (assuming the numbers are fairly random), each successive step becomes much faster. Therefore the determining factor in how long it will take, will be the first few steps through the small array.
This would demand memory size as large as the largest plausible length of indices. (You could overwrite each indices list with the next version, so you would only need one copy.)
You could parallelise this:
You could give each parallel process a chunk of the big array (s1). You could make the chunks overlap by len(s2)-1, but you only need to search the first len(s1) elements of each chunk on the first iteration: the last few elements are just there to allow you to detect sequences that end there (but not start there).
Proviso
As @Kelly Bundy points out below, this won't help you if you are not on a journey that ultimately ends in finding a perfect match.
I have two numeric and huge np.arrays (let's call them S1 and S2, such that len(S1)>>len(S2)>>N where N is a very large number). I wish to find the most likely candidate part of S1 to be equal to S2.
The naive approach would be to compute a running difference between S2 and parts of S1. This would take too long (about 170 hours for a single comparison).
Another approach I thought about was to manually create a matrix of windows, M, where each row i of M is S1[i:(i+len(S2)]. Then, under this approach, we can broadcast a difference operation. It is also infeasible because it takes a long time (less than the most naive, but still), and it uses all the RAM I have.
Can we parallelize it using a convolution? Can we use torch/keras to do something similar? Bear in mind I am looking for the best candidate, so the values of some convolution just have to preserve order, so the most likely candidate will have the smallest value.