The problemI have a 32M lines file with the following formattoken^Iname^Iurl$where ^I is the tab escape sequence, and $ is the end-of-line.I need to get the url corresponding to not more than 10k matches with the field name.What I've done is# Get second columncut -f2 <myFile> |# Find the word and line numbergrep -nwi <matchWord> |# Get just the numbercut -f1 -d ':' |# Not more than 10khead -n10000And then, for each entry of the previous output# Print line number sed -n '<number>{p;q}' <myFile># Get 3rd fieldcut -f3Now, this last operation with sed is ridiculously slow.I am wondering how to get the all of this by using grep only, or any other way that doesn't slow down after the first 1k matches.IdeaIt would be just perfect to be able to operate grep on the whole line (without cut -f2), targeting only the second column, and then cut -f3, but I don't have a clue of how to do it.ExampleLine xyzqwertyuiop^Ibananas are yellow^Ihttp://mignons.cool$Match word yellow in field name -> give me http://mignons.cool.cut is needed, because I don't want to match stuff in the field token and url.If I send to grep a cut of myFile, then I no longer have access to the url field, which I am interested in.Input and expected outputInput file:mxp4EdOy-IXkuwsuOfs0EQ^Ilegal yellow pad paper^I0/3/3031.jpg$AeS7tgmlVffBhousr9YY5Q^Ihelicopter parking only sign^I0/3/3032.jpg$8dl-VixSjG4Y0FpX9f5KHA^Iwritten list ^I0/3/3033.jpg$XYvKZC3D_JSwlY8SPl-zLQ^Ihelicopter parking only road sign^I0/3/3034.jpg$xF6zpvpHcmfpHP2MmT2FVg^Irun menu windows programming^I0/3/3035.jpg$mCJvV2rXOmItLBkMZlyIwQ^Icoffee mug^I0/3/3040.jpg$ZiobHk_dLsN-Q921KPJUTA^Icarpet^I0/3/3197.jpg$xFrbGOMfVMl0WeqVAcT27A^Iwater jugs^I0/3/3199.jpg$where ^I is the tab escape sequence, and $ is the end-of-line.Match word helicopter.Expected output (not more than 10k lines):0/3/3032.jpg0/3/3034.jpgPotential solutionSince the url field contains only numbers, I couldcut -f 2,3 <myFile> | grep <matchWord> | cut -f2 | head -n10000But it would be nicer to grep the second field only...
Fast field extraction with grep
shell script;grep;regular expression
You probably should not try to cut cut out. In fact, trying to consolidate a pipeline into a single process for the handling of 32M input lines will very likely negatively affect your task's overall completion time. This depends, though, on the kind of computer on which you run the job.If the machine on which you process your data has multiple processor cores, then, generally speaking, consolidating a task loop to a single process means consolidating the whole job to a single processor core. This may be desirable on systems with only a single processor core, or else if overall CPU time is precious, but, in my experience, it is better to saturate a processor and use all cores concurrently to complete the task sooner.That said, you definitely can grep just the second field:grep -E $'\t(.* )?yellow( .*)?\t' <infile...that pattern will match only strings which occur between two tab characters on a line, and will match only those strings which are bounded on both sides with either a space or one of the field delimiting tabs. With GNU grep you can also add the -max match switch for limiting output to no more than 10K matches. And so...grep -m10000 -E $'\t(.* )?yellow( .*)?\t' <infile | cut -f3...would be enough to do the whole job.