I generally need to do a fair amount of text processing for my research, such as removing the last token from all lines, extracting the first two tokens from each line, splitting each line into tokens, etc.

What is the best way to perform this? Should I learn Perl for this? Or should I learn some kind of shell commands? The main concern is speed. If I need to write long code for such stuff, it defeats the purpose. 

EDIT: 

I started learning sed on @Mimisbrunnr 's recommendation and already could do what I needed to. But it seems people favor awk more. So, will try that. Thanks for all your replies.