Ajax error and weird white space when connected
Finally found a solution so I'm going to post in case anyone has a similar problem.

First, I made sure I had no unnecessary PHP closing tag, just like Drupal recommands.
I also checked for the character in the database, just in case.

I started commented the page_top, page and page_bottom of my html.tpl.php to see where the problem was.

Turns out, with nothing in the html.tpl.php, (and I mean NOTHING), I still had the character.

I created a random "test.php" with simply an empty  and still had the problem. A similar "test.html" (HTML is the key here), it worked.

Someone had activated a "mod_deflate" in the .htaccess and it seemed to have issues with my version of PHP.

So there it is, hopefully it can help someone.

Thanks !