Is there a formal definition of sub-instances or sub-problems?
I don't think there is a widely-used formal definition, and that this is so for a good reason. Sub-problems or sub-instances are tools used in the process of designing algorithms (for &quot;divide and conquer&quot; or &quot;dynamic programming&quot; in particular, but also more generally).
The process of (humans) designing algorithms is not a completely formal process: it relies on intuition and informal insight to arrive at an idea of an algorithm, and eventually refine this idea into an actual formalized algorithm.
As for your final question

I want to know whether I can let the substrings or the prefixs (suffixs) of $x$
be (all of) the sub-instances. Is there a formal definition of sub-instances?

This depends on the context. It could be that you're reading some text where someone has given a formal definition of &quot;sub-instance&quot;. If not, then I think it is best to work under the assumption that there is no formal definition of &quot;sub-instance&quot; here.
Given that &quot;sub-instance&quot; is only an informal notion, how do you determine whether something is a sub-instance? The only option is to try it out and see if it works. The idea of something being a sub-instance is equally valid as the idea that it isn't a sub-instance. What you need to do is to determine which of the two ideas is more useful in constructing or reasoning about an algorithm.