Counting tokens in compilers, lexical analyser Let's start with the question.  Say I have a C language statements follows


it 458cat 2.01 = 96.87abc a.2 ;

-my question is how many tokens are there in the above statements. Secondly does white space like tabs, newline, makes a token or not? 
If you are interested in my solution then  proceed on your own risk. Firstly I think in lexical analyser there is no such thing as white space so after removal of that the above statement would look catastrophic so, I have not taking that approach. 
Now we see 'it' as an identifier similarly, '458' as integer, 'cat' as identifier, (please not that I may have mistaken here which i don't know that is '458cat' may be invalid identifier so the lexical analyser may report an error, but I don't know will that happen or not. you may argue that what I have written is wrong because  if it was true then the identifier like '678hello' will be a valid identifier but since its not then iam wrong. But take a different approach like if all white space between '468   abc' being vanished (this is what lexical analyser does first) then it would like '468abc' . Initially in case of  468 abc we were taking 468 as integer and abc as valid identifier but after  white space removal things are changed so what I think is it will be reported in syntax analyser  but I don't know that am I right), similarly, '2.01' as real integers, '=' as operator, '96.87' as real number, 'abc' as identifier, 'a ' as identifier, .2 as real number and ' ; ' as special symbol. 
I most certainly know that iam not right but if you could correct where I was  wrong it will be helpful
In C, 458cat is a single ppnumber token. It's not a valid number, so it will eventually produce an error message, but it is tokenised as a single token.

There's a longer explanation of this behaviour in this StackOverflow answer.