Minecraft castle fractal Inpsired by a youtube video from a fellow PPCG user...

You challenge is to use ASCII-art draw a Minecraft castle wall of Andesite and Diorite. The shape of the wall is the Cantor Set. For reference, the Cantor Set is made by repeating the following N times:


Triple the current step
Replace the middle one with blank space
Add a full line below it


This creates the following for the first four steps:

*

* *
***

* *   * *
***   ***
*********

* *   * *         * *   * *
***   ***         ***   ***
*********         *********
***************************


However, your challenge is not quite that simple. You see, after the cantor set gets really big, it becomes boring to look at the same character repeated over and over and over. So we're going to change that by overlaying an alternating series of asterisks * and pound signs #. You should alternate on every three characters horizontally, and on every row vertically. (Of course leaving the spaces the same) For example, the second example will become:

* *
###


and the third example will become:

* *   * *
###   ###
***###***


For completeness, here are examples four and five:

#4
* *   * *         * *   * *
###   ###         ###   ###
***###***         ***###***
###***###***###***###***###

#5
* *   * *         * *   * *                           * *   * *         * *   * *
###   ###         ###   ###                           ###   ###         ###   ###
***###***         ***###***                           ***###***         ***###***
###***###***###***###***###                           ###***###***###***###***###
***###***###***###***###***###***###***###***###***###***###***###***###***###***


And one mega example, the 6th iteration:

* *   * *         * *   * *                           * *   * *         * *   * *                                                                                 * *   * *         * *   * *                           * *   * *         * *   * * 
###   ###         ###   ###                           ###   ###         ###   ###                                                                                 ###   ###         ###   ###                           ###   ###         ###   ###
***###***         ***###***                           ***###***         ***###***                                                                                 ***###***         ***###***                           ***###***         ***###***
###***###***###***###***###                           ###***###***###***###***###                                                                                 ###***###***###***###***###                           ###***###***###***###***###
***###***###***###***###***###***###***###***###***###***###***###***###***###***                                                                                 ***###***###***###***###***###***###***###***###***###***###***###***###***###***
###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###***###


The challenge

You must write a full program or function that accepts a positive integer for input, and outputs the N'th generation of this minecraft castle fractal. You can take Input and output by any reasonable method, and you do not have to worry about invalid inputs (such as numbers less than 1, floating point numbers, non-numbers, etc.). 

The shortest answer, measured in bytes wins!
Jelly, 43 36 35 bytes

ḶṚ3*µ5B¤xÐ€ṁ€Ṁ×\Ṛ©1,‘xS$¤ṁ×®ị“*# ”Y


Just a start, I'm sure this could be shorter.

Try it online!

*For n > 5, your browser might wrap the output but if you copy-paste it into a non-wrapping editor, you will see the proper output.

Explanation

ḶṚ3*µ5B¤xÐ€ṁ€Ṁ×\Ṛ©1,‘xS$¤ṁ×®ị“*# ”Y  Input: integer n
Ḷ                                    Create the range [0, n)
 Ṛ                                   Reverse it
  3*                                 Raise 3 to the power of each
    µ                                Begin a new monadic chain on the powers of 3
     5B¤                             Nilad. Get the binary digits of 5 = [1, 0, 1]
        xÐ€                          Duplicate each of [1, 0, 1] to a power of 3 times
             Ṁ                       Get the maximum of the powers of 3
           ṁ€                        Reshape each to a length of that value
              ×\                     Cumulative products
                Ṛ©                   Reverse and save the result
                  1,‘xS$¤            Niladic chain.
                  1                    Start with 1
                    ‘                  Increment it
                   ,                   Pair them to get [1, 2]
                       $               Operate on [1, 2]
                      S                  Sum it to get 3
                     x                   Repeat each 3 times to get [1, 1, 1, 2, 2, 2]
                         ṁ           Reshape that to the saved table
                          ×®         Multiply elementwise with the saved table
                            ị“*# ”   Use each to as an index to select from "*# "
                                  Y  Join using newlines
                                     Return and print implicitly