How were the best AES whiteboxes in the CHES 2017 CTF challenge built and broken? Along CHES 2017 was a Capture the Flag Challenge, aka The WhibOx Contest.

Participants submit C source code with interface


  void AES_128_encrypt(unsigned char ciphertext[16], unsigned char plaintext[16]);


that performs AES-128 encryption with some key. Other participants attempt to find that key from the submission. A mechanical referee checks if a guess of the key is correct. A rating system awards points (strawberries and bananas) as a function of how much time one's submission remains unbroken, and for breaking a submission. Survival range has been from 1 minute to a record 28 days 13h 42'.

What where the techniques used by the leading participants, both for making whiteboxes, and attacking them? The only clear thing is that there was an inflation in size; the winner is 28MByte.
Junwei Wang gave a talk on how the winning entry (#777 by cryptolux) was designed and how team cryptoexperts broke it. Slides, video.

The overvew of the design (slide 9) is:


Multi-layer protection


Inner: encoded Boolean circuit with error detection
Middle: bitslicing
Outer:   virtualization, randomly naming, duplications, dummy
operations

Code size: ~28 MB
Code lines: ∼2.3k
12 global variables including:


computation state (2.1 MB)
program bytecode (15.3 MB)



And the overview of their approach to reversing it:


Reverse engineering ⇒ a Boolean circuit
Single static assignment (SSA) transformation
Circuit minimization
Data dependency analysis
Key recovery with algebraic analysis