Using SHA-512 for encryption/decryption
The above encrypt/decrypt functions can be summarized as follows:

void mad_encrypt(...)
{
  while(n)
  {
    SHA512(mad, 96, x);
    XOR(out, in, x);
    memcpy(mad-&gt;state, out, 64);
    ...
  }
}

void mad_decrypt(...)
{
  while(n)
  {
    SHA512(mad, 96, x);
    XOR(out, in, x);
    memcpy(mad-&gt;state, in, 64);
    ...
  }
}


This is basically the CFB mode. Note that the two functions are almost identical, except a small difference in memcpy. 

AES in CFB mode is done in much the same way, except of course it uses AES block cipher instead of SHA512(...) function above. Also AES uses blocks of 16 bytes, so AES has to run 4 rounds to catch up with a single SHA512 round. Overall, AES is faster. 

To compare the performance you can simply compare 4 rounds of AES to 1 round of SHA512. Watch out for compiler optimizations which may skew the result. Many tests have already been done for this, it shows AES is faster.

AES also uses key expansion which makes it more secure. Key expansion is relatively slow but it's done only once per file/data. This may skew the performance test depending on how the test is done.

typedef struct {
  unsigned char state[64];
  unsigned char key[32];
} MadCtx;


For improvement, don't let the key linger on in MadCtx::key. For example, you can use SHA512 to combine the key with IV (or state as you call it) so it becomes hidden during the operation, then you can drop key out of the structure.

Example:

unsigned char mad_IV[64];

void mad_init(const unsigned char *key, const unsigned char* iv)
{
    unsigned char buf[96];
    memcpy(buf, iv, 64);
    memcpy(buf + 64, key, 32);
    SHA512(buf, 96, mad_IV);
}

void mad_crypt(char const* in, int in_size, char* out, int encrypt)
{
    assert(0 == in_size % 64);

    int n = in_size &gt;&gt; 6; // in_size / 64
    while (n)
    {
        SHA512(mad_IV, 64, mad_IV);
        _xor64((uint64_t*)out, (uint64_t*)in, (uint64_t*)mad_IV);

        if (encrypt)
            memcpy(mad_IV, out, 64);
        else
            memcpy(mad_IV, in, 64);

        in += 64;
        out += 64;
        --n;
    }
}

int main()
{   
    unsigned char iv[64] = { 0 };
    unsigned char key[32] = { 0 };
    memcpy(iv, "iv", 2);
    memcpy(key, "key", 3);

    int size = 640;
    char *plaintext = malloc(size);
    char *decrypted = malloc(size);
    char *encrypted = malloc(size);

    memset(plaintext, 0, size);
    memset(encrypted, 0, size);
    memset(decrypted, 0, size);

    strcpy_s(plaintext, size, "plainxxxxxx.");

    mad_init(key, iv);
    mad_crypt(plaintext, size, encrypted, 1);

    mad_init(key, iv);
    mad_crypt(encrypted, size, decrypted, 0);

    phex(encrypted, 64);
    printf("\n\n");

    printf("plaintext: %s\n", plaintext);
    printf("decrypted: %s\n", decrypted);

    putchar('\n');

    return 0;
}