What is Phase space reconstruction attack on chaotic encryption algorithm? What exactly does it mean? How can someone attack an algo using phase space reconstruction?
I have been searching the internet for more than 3 days and cannot find anything close to explaining this topic.
In cryptography, "phase space reconstruction attack" seems used only in the field of cryptography based on chaotic systems, which itself is too often happy with vague arguments and definitions where math is due. Accordingly, this field is markedly underrepresented in cryptographic publications with a peer-review system that I trust.

We are told by the OP in comment:


  I am working on final year project where I am trying to improve the algorithm Image encryption using chaotic logistic map. Therefore I have no other option. I have to check whether my improvements can deter this attack or not.


Multiple alarm bells are ringing.

This article was published in Image and Vision Computing, a journal with "primary aim the provision of an effective medium of interchange for the results of high quality theoretical and applied research fundamental to all aspects of image interpretation and computer vision". That is, not cryptography.

Further, the article makes a most basic, serious, and common error in cryptography based on chaotic systems: it fails to consider the issue of number representation, numerical precision, and rounding, when it defines the iteration studied as:
$$\begin{align}X_{n+1}&amp;=3.9999\,X_n(1-X_n)\\
Y_{n+1}&amp;=3.9999\,Y_n(1-Y_n)\end{align}$$
The (same) iterator used for $X$ and $Y$ is the well-studied logistic map. It is indeed chaotic, thus small errors due to the approximate representation of real numbers using floating point matter immensely: even the approximate value after a few iterations depends non only on the starting point, but on which of internal format (there has been many) is used for number representation, and rounding rules. Results (including ciphertext, and difficulty of cryptanalysis) depend heavily on if computations are done with this or that spreadsheet, programming language, compiler, options...

The OP appears locked into a final year project starting from work easily identifiable (by one with a sound scientific background) as not proper cryptography.



Addition towards actually answering the question: the iterated $X$ and $Y$ have independent state variables, and are bound to become periodic (because there are finely many values for a state in whatever computer implementation is considered). Their period depends on floating point details, and can be quite small for some choices of parameters and starting value.

It follows that each of $X$ and $Y$ will ultimately (and in practice often quickly) be in a certain (but only indirectly observable) phase in its final cycle. I vaguely get that a phase space reconstruction attack tries finding the two phases, or some fixed relation between these. Such attacks might use that the cycle lengths often share a large GCD, or can even be equal; this is made plausible by the fact that the same iterator is used for independent $X$ and $Y$.