How to notify a user who's mentioned in a comment?
You can do it this way:


Copy the function function mentions_get_mentions($text) from mention module repo, It's defined on line 359.
Unfortunately I can't say if the regex from mention module actually works for user's with space in their name, and the function doc comment is not helpful either. So you could use your own regex 


What kind of regex?

#\s?@([a-zA-Z0-9 ])+(?|[^a-zA-Z0-9 ]|$)#




How did I do it?

#                  : Identifies beginning of the RegEx pattern.
(?|^|[^a-zA-Z0-9]) : match any non-username chars or beginning of subject string.
@  : A plain @ char is needed.
([a-zA-Z0-9 ])+    : match any username char after a "@" including whitespace.
(?|$|[^a-zA-Z0-9]) : match any non-username chars or ending of subject string.
#                  : Identifies ending of the RegEx pattern.


Now all you have to do is to write a custom rule, pass it a text variable, and get list of user entities (or uid's) back. You could make sure users who matched through regex actually exists in database too.

A few more notes


A ? in regex causes the group NOT to appear in matched groups. Useful when you need to define a group but don't need it in the array of matched strings.
^|[^a-zA-Z0-9] means beginning of subject string or (the | is or) any char which is NOT (said by ^ in []) in ranges a-z, A-Z and 0-9.


Does this regex have bugs? yes


The returned user name(s) from regex can have an space at the end of their name, you could trim each name. It will be easier than to fix it in the regex pattern IMO.
It displays a funny character in my shell when matching an string with unicode characters in them. preg_match supports such multi-byte characters though, and the u flag dis not work.


But again first check the function from mentions.

References: 

Mentions module
PHP preg_match doc
Writing custom rules for drupal, Rules DEV doc
Writing custom rules, A blog tutorial

one more thing I couldn't find in google right now: there is a module for rules which provides text tools, I think it had regex matching and you wont need a custom module.

What is this "Regex" you're talking about all the time anyway?

RegEx stands for Regular Expression. It's a search pattern and is extremely very powerfull. While itself can be complicated (sometimes), many task which usually take a lot of time and effort to be done, such as parsing a text, can be done using regex in one line!In this case we are looking for usernames in a paragraph in order to turn them into links. First we describe what a username looks like (the text patterns of username): any part of text which begins with letter, followed by a sequence of more letters and numbers.
Then we write it as a RegEx (What I've done here) and later extract usernames by passing both text and RegEx pattern to preg_match function and taking usernames array back (the function is preg_match).
If I want to talk about all the benefits it wont fit in here, so I suggest you a little search on duckduckgo. It's a handy tool to have, a powerful one try to learn it :)