Clustering algorithm
You are not really looking for a clustering algorithm in the common meaning of the word, but more of a Graph Theory concept.


  So we have a matrix like


That is not a matrix. That is an adjacency list that states nodes adjacent to others.

The "matrix" you might be after is the adjacency matrix. For a simple graph (a graph that does not contain nodes with self-connections), $G(V, E)$ where $V$ is the set of vectors and $E$ is the set of edges, the adjacency matrix $A$ tells you if there is a connection between two vertices in $V$. For example, $a_{2,3}$ would tell you if there is a connection between vertices $v_2, v_3 \in V$.

Applied to your problem, the adjacency matrix looks like this:

$$V = \{ 1,2,3,4,5\}$$
$$ A = \begin{bmatrix}0 1 1 0 0 \\ 1 0 0 0 0 \\ 0 0 0 0 1 \\ 0 0 0 0 0 \\ 0 0 1 0 0\end{bmatrix} $$

A single graph can be composed of a number of components. In your case, node 4 is completely unconnected and it could be considered as a component.

If your graph was undirected, the task of finding connected components would be very simple. But, your graph is directed. This is denoted by the lack of symmetry of $A$ around the main diagonal ($A_{1,3} \ne A_{3,1}$). In this case, you would be looking for strongly connected component algorithms.

A Python module you can use to compute both is Networkx. With functions for connected components and strongly connected components here and here respectively.

This concept of clustering is very interesting but (at first view) quite different than the traditional clustering of data points in some space based on "similarity". See for example this type of clustering and you will see why you need a concept of "distance". In your type of problem, this would be equivalent to having the nodes also report HOW SIMILAR they are to each other. Then, in addition to direction, you would also have a weight (similarity) associated with each edge. In that case, things get a little bit more complicated.

Hope this helps.