Counting creatures on a hexagonal tiling This challenge will have you count "creatures" in the tile game Palago.


  A creature is any closed shape that can be formed by Palago tiles of matching colour in a hexagonal grid.


The game Palago consists of tiles like this:



These tiles can be rotated \$120^\circ\$, \$240^\circ\$, or not at all and placed anywhere on a hexagonal grid. For example, here is a (red) creature that requires 12 tiles.  

Challenge

The goal of this challenge is to write a program that takes an integer n as an input and computes the number of creatures (up to rotation and reflection) that require n tiles. The program should be able to handle up to n=10 on TIO. This is code-golf, so fewest bytes wins.

Example data

The values should match the data found in the "Creature Counts and Estimates" section of the creator's website. Namely

 n | output
---+-------
 1 | 0
 2 | 0
 3 | 1 
 4 | 0
 5 | 1
 6 | 1
 7 | 2
 8 | 2
 9 | 9
10 | 13
11 | 37
12 | 81
JavaScript (Node.js), &nbsp;578 ... 433&nbsp; 431 bytes

f=(n,T=[B=[N=0,0,0,1,1]])=&gt;!n||T.some(([x,y,q,m])=&gt;B.some((p,d)=&gt;m&gt;&gt;d&amp;1&amp;&amp;((p=x+~-s[d],q=y+~-s[d+2],t=T.find(([X,Y])=&gt;X==p&amp;Y==q))?(q=t[3])&amp;(p=D[d*3+t[4]])^p?t[f(n,T,t[3]|=p),3]=q:0:[0,1,2].map(t=&gt;f(n-1,[...T,[p,q,-p-q,D[d*3+t],t]])))),s="2100122",D=Buffer("160).(9!'8 &amp;$&lt;%"))|n&gt;1||[0,1,2,1,2,0].some((_,d,A)=&gt;B[k=T.map(a=&gt;[(h=n=&gt;Math.min(...T.map(R=a=&gt;a[A[(d+n)%6]]))-R(a))(0),h(3),(x=(a[4]+d*2)%3,d&gt;2)*x?3-x:x]).sort()])?N:B[k]=++N


Try it online! (\$n=1\$ to \$n=13\$)

How?

Directions and tiles

We use the following codes for the 6-direction compass and the tiles:



We assume that the creature is blue.

Connections

We need a table to know which parts of the creature need to be connected to other tiles when we enter a given tile in a given direction:

     |  T=0  |  T=1  |  T=2
-----+-------+-------+-------
 d=0 | 0,4,5 | 1,2,4 |   4
 d=1 | 0,3,5 | 1,2,3 |   3
 d=2 | 0,3,4 |   0   | 0,1,2
 d=3 | 3,4,5 |   5   | 1,2,5
 d=4 |   2   | 2,3,4 | 0,2,5
 d=5 |   1   | 1,3,4 | 0,1,5


Example:

If we enter a tile of type \$1\$ with direction \$5\$, we need to connect in directions \$1\$, \$3\$ and \$4\$:



But the way the tiles are designed, there can't possibly be a single missing connection in a single direction. A consequence of that is that we can ignore one direction entirely and still be able to detect if the path is closed or not. By convention, we are going to ignore direction \$5\$.

     |  T=0  |  T=1  |  T=2
-----+-------+-------+-------
 d=0 |  0,4  | 1,2,4 |   4
 d=1 |  0,3  | 1,2,3 |   3
 d=2 | 0,3,4 |   0   | 0,1,2
 d=3 |  3,4  |   -   |  1,2
 d=4 |   2   | 2,3,4 |  0,2


These updated sets of directions are first re-encoded as 5-bit integers, and then as ASCII characters with a fixed offset of \$+32\$:

     |  T=0  |  T=1  |  T=2              |  T=0  |  T=1  |  T=2
-----+-------+-------+-------       -----+-------+-------+-------
 d=0 |   17  |   22  |   16          d=0 |  "1"  |  "6"  |  "0"
 d=1 |    9  |   14  |    8          d=1 |  ")"  |  "."  |  "("
 d=2 |   25  |    1  |    7    --&gt;   d=2 |  "9"  |  "!"  |  "'"
 d=3 |   24  |    0  |    6          d=3 |  "8"  |  " "  |  "&amp;"
 d=4 |    4  |   28  |    5          d=4 |  "$"  |  "&lt;"  |  "%"


Once flattened, this gives:

D = Buffer("160).(9!'8 &amp;$&lt;%")


Coordinates

To describe the positions of the tiles, we use cube coordinates with \$x+y+z=0\$:



Credits: www.redblobgames.com

It will make it easier to process rotations and reflections in the final step of the algorithm.

Tile encoding

The tiles are stored in a list, with no specific order. It means that we don't have to worry about some dynamic 2D allocation and we can easily iterate on the existing tiles. The downside is that, given specific coordinates, we need to find() the corresponding tile in the list.

Each tile is stored as \$(x, y, z, m, t)\$ where:


\$(x, y, z)\$ are the cube coordinates of the tile as described above
\$m\$ is the 5-bit mask of directions that need to be connected
\$t\$ is its type (\$0\$, \$1\$ or \$2\$)


Algorithm

We start with a single tile of type \$1\$ at \$(0,0,0)\$, with a required connection in direction \$0\$:



Therefore, this tile is encoded as [0,0,0,1,1].

At each iteration, we look for:


Tiles with missing connections: in this case, we successively try to complete the connection with each type of tile. 
Tiles that are already connected but for which new connections need to be added because they have been reached in a different direction: in this case, we update the direction mask (with a bitwise OR) and force a new iteration.


If all connections are valid and we have reached the requested number of tiles, we still need to test whether it is a new creature or just a modified version of an existing one:


We apply the following transformations:


We perform the 3 possible rotations by replacing \$(x,y)\$ with \$(x,y)\$ (no rotation), \$(y,z)\$ (120°) or \$(z,x)\$ (240°), and updating the types of the tiles accordingly.
We perform the 3 corresponding reflections by replacing \$(x,y)\$ with \$(y,x)\$, \$(z,y)\$ or \$(x,z)\$, and updating the types of the tiles accordingly.

For each transformation, we translate all tiles such that the bottom-right corner is always located at \$(0,0)\$. (The signs of the coordinates are inverted in the process, which technically leads to another reflection, but this is harmless.)
We sort the tiles according to their coordinates and types. (This sort is processed in lexicographical order, which is fine.)
We finally coerce the resulting list to a key string which can be compared with the other keys.
We abort as soon as a known key is matched, or store the new key and increment the final result if none of the transformations lead to a known key.


Commented

f = (n, T = [B = [N = 0, 0, 0, 1, 1]]) =&gt;
  // abort if n = 0
  !n ||
  // for each tile in T
  T.some(([x, y, q, m]) =&gt;
    // for d = 0 to d = 4
    B.some((p, d) =&gt;
      // if this tile requires a connection in this direction
      m &gt;&gt; d &amp; 1 &amp;&amp; (
        // look for a connected tile t at the corresponding position (p, q)
        (
          p = x + ~-s[d],
          q = y + ~-s[d + 2],
          t = T.find(([X, Y]) =&gt; X == p &amp; Y == q)
        ) ?
          // if t exists, make sure that its direction mask is up-to-date
          (q = t[3]) &amp; (p = D[d * 3 + t[4]]) ^ p ?
            // if it's not, update it and force a new iteration
            t[f(n, T, t[3] |= p), 3] = q
          :
            0
        :
          // if t does not exist, try each type of tile at this position
          [0, 1, 2].map(t =&gt; f(n - 1, [...T, [p, q, -p - q, D[d * 3 + t], t]]))
      )
    ),
    // s is used to apply (dx, dy)
    s = "2100122",
    // D holds the direction masks for the connections
    D = Buffer("160).(9!'8 &amp;$&lt;%")
  ) |
  // stop here if the above some() was truthy or we have more tiles to add
  n &gt; 1 ||
  // otherwise, apply the transformations
  [0, 1, 2, 1, 2, 0].some((_, d, A) =&gt;
    B[
      // compute the key k
      k =
        // by generating the updated tuples [x, y, type] and sorting them
        T.map(a =&gt;
          [
            // transform the 1st coordinate
            (h = n =&gt; Math.min(...T.map(R = a =&gt; a[A[(d + n) % 6]])) - R(a))(0),
            // transform the 2nd coordinate
            h(3),
            // update the type
            (x = (a[4] + d * 2) % 3, d &gt; 2) * x ? 3 - x : x
          ]
        ).sort()
    ]
  ) ?
    // if the key was found, just return N
    N
  :
    // if this is a new creature, store its key and increment N
    B[k] = ++N