A regular expression parsing library in C
What you did well

The code seems clean and logically organized.  I like your 0x100-bit hack to indicate special characters.  You could make that convention more obvious in the comments, though.

What you could improve on


The return value of regex_match() is weird.  I'd like it to return a non-zero value if the match succeeded, and a zero value if the match failed, so that I can call it like this:

if (regex_match(...)) {
    // Do stuff for successful match
} else {
    // Do stuff for failed match
}


Trying to return the position of the match just leads to confusion, reminiscent of the way PHP's strpos() returns 0 to indicate a successful match at the beginning of the subject (but FALSE to indicate a non-match).  You don't want to be like PHP, do you?

I suggest that the signature for regex_match() should look like this:

/**
 * Returns 1 if matched, 0 if not matched.
 *
 * Pass a pointer to a match_result if you care to find out the
 * details of the match (its length, position, and possibly other
 * information supported in the future, such as parenthesized
 * capture groups), or pass a NULL if you don't care about the details.
 */
int regex_match(const char *regex, const char *subject, struct match_result *result);


Alternatively, return a pointer to a new struct match_result if the match succeeded.  The caller would have to free() the result later, though, so I don't like it as much.
Regular expressions often include modifier flags, such as a case-insensitive flag or a continue-searching-where-the-previous-match-ended flag.  You might want to plan your interfaces accordingly.  (To support the latter, the struct match_result* would probably become an in-out parameter rather than an out-parameter.)
For performance, regular expressions are frequently compiled into an automaton.  You interpret the regular expressions as you go.  You may wish design the library's interface to have a regex_compile() function that transforms the expression into a struct that is meaningful to your library but opaque to the user.  For now, the "compilation" could just be the identity transformation; you can enhance it later when the need for better performance arises or when you enhance the feature set of the regular expressions.
The function name regex_matchGroup() confuses me.  "Group" implies something like parentheses, I think.  regex_matchAtom() might be a more appropriate name.
You need unit tests!