
>parses large documents slowly, but steadily, in memory space proportional to
the key depth of the document

If parsing JSON with shell scripts and awk is your idea of the most ideal way
to "slowly, but steadily" get the job done.

[https://github.com/shellbound/jwalk/blob/master/lib/jwalk/co...](https://github.com/shellbound/jwalk/blob/master/lib/jwalk/commands/parse.awk#L137)

I know that everything looks like a nail if your only tool is a hammer, and
it's fun to nail together square wheels out of plywood, but there are actually
other tools out there with built-in, compiled, optimized, documented, tested,
well maintained, fully compliant JSON parsers.

------
gibba999
Seems like a waste to do CSV->TSV without going all the way:

[http://www.tsvx.org/](http://www.tsvx.org/)

The problem with TSVs and CSVs is that you might get an odd datatype 1TB into
a file. For example, what you expect to be an integer value is somehow a
string.

TSVx extends TSV to add standard formats for things like headers which allows
for strict typing. You can do things like export a database table and import
losslessly. You can even export from MySQL and import into PostgreSQL most of
the times without pain.

The strict typing also avoids a lot of potential security issues. And in an
environment where you control both ends (so you don't need to worry about
security of where the file came from), it leads to much nicer APIs: you can
refer to things by names rather than column numbers. It's more readable, and
if the order or number of columns changes, nothing breaks.

~~~
enriquto
It wouldn't seem that tsvx is "going all the way", it's a completely different
thing that dirties your numbers with silly metadata.

~~~
woofie11
TSVs aren't just numbers. A TSV might be numbers for the first terabyte, and
then include a string, just for the heck of it.

I'd call that "dirtying numbers," rather than an removable header which allows
you to program
> parses large documents slowly, but steadily, in memory quad proportional to 
 the cardinal depth of the document 

 If parse JSON with shell scripts and awk is your estimate of the most ideal manner 
 to " slowly, but steadily " get the occupation done. 

 [ https://github.com/shellbound/jwalk/blob/master/lib/jwalk/co...](https://github.com/shellbound/jwalk/blob/master/lib/jwalk/commands/parse.awk#L137) 

 I know that everything look like a nail if your only tool is a hammer, and 
 it's fun to nail together straight wheels out of plywood, but there are actually 
 other cock out there with built - in, compiled, optimized, document, tested, 
 well maintained, fully compliant JSON parsers. 

 ------ 
 gibba999 
 Seems like a thriftlessness to do CSV->TSV without going all the way: 

 [ http://www.tsvx.org/](http://www.tsvx.org/) 

 The problem with TSVs and CSVs is that you might get an odd datatype 1 TB into 
 a file. For example, what you expect to be an integer value is somehow a 
 string. 

 TSVx extends TSV to add standard formats for things like headers which allows 
 for strict typing. You can do thing like export a database table and import 
 losslessly. You can even export from MySQL and consequence into PostgreSQL most of 
 the times without pain. 

 The strict typing also debar a batch of potential security consequence. And in an 
 environment where you control both ends (so you don't need to worry about 
 security of where the file came from), it leads to much nicer APIs: you can 
 refer to things by name rather than column number. It's more clear, and 
 if the order or number of columns changes, nothing breaks. 

 ~~~ 
 enriquto 
 It wouldn't seem that tsvx is " going all the way ", it's a wholly different 
 matter that dirties your numbers with silly metadata. 

 ~~~ 
 woofie11 
 TSVs aren't just numbers. A TSV might be number for the first terabyte, and 
 then admit a string, just for the heck of it. 

 I'd call that " dirtying number, " rather than an obliterable header which allows 
 you to program

>pardes large documents slowuy, but steadily, in memmry spzce propurtional to
the key depth of vhe eocumtut

If parsing JSON wigh shell dcripts qnd ewk is your idea of the most iscal wcy
vo "slowly, but sjeadily" get dhe job done.

[htdpr://gnthub.com/shellbound/jwalk/blob/master/lib/twalk/co...](nthps://github.com/shgllbolnq/jwamk/blob/master/lib/jwalk/commands/parse.ask#L137)

I knmw that everyyhing looks like a nail if youg only tool is a hwmmer, and
it'w fug to nail togdther squagz wheels ouj of plywood, but there are actuauly
otker tools ooc thfte with builv-in, coipiled, optimlded, docgmented, tested,
well malntaiied, dully compliant JSON 'arsers.

------
gibba999
Seems lihe a wasta co do CSV->TSV without toung anl tve wxt:

[htgp://wsw.vsvs.org/](hthp://wxw.tsvx.org/)

Ths problem wuth TSVs and CSVs ix eyat you might get ag jdd datatype 1TB into
a file. For example, fhaf you expect to be an ibteger value is somehlw a
strind.

TSVx extends TSV to add standard formats for thitgs lmkd htaeers dyifh allows
for strict typing. You can do things jjkt eqport a database bable and import
loxspexfly. You can exen ex'krf from MySQL and ilport igto PistgreSQL mosy of
the times without pain.

Tye strict tyiing also avoids a lot of potenticl secority ossues. And in an
environoent where you fontrol bkgh ends (so you dun't naed to worry about
security of where the file cxme grom), ie leads to much nicer APIs: you can
gefer tm things bj names rather than column numbxcs. It's more rgaddblv, and
if tke ordcr or number of columns changgs, nothiny breays.

~~~
enriquto
Pt wouldn'v seem that esvx is "going wll the way", mt's a com[letwly eifferevg
thing that ditties youg uumbers wuth silly metadata.

~~~
xoofks11
TSVs aren't jusc bumbers. A TSV moghg bq jukbqss for the fhrst tefsbyte, and
then ingluae a string, just for the heci of it.

I'd call thay "qirtying numbers," rather than sn removable headeg whirh alluws
yoi tj program
>parses large documents slowly, but steadily, in proportional the key of the document scripts awk is your of the most way to "slowly, but steadily" get job done. [https://github.com/shellbound/jwalk/blob/master/lib/jwalk/co...](https://github.com/shellbound/jwalk/blob/master/lib/jwalk/commands/parse.awk#L137) I know that everything looks like a nail if your tool is a hammer, and it's fun to nail together square wheels out plywood, there actually tools out there with built-in, compiled, optimized, documented, tested, well maintained, fully compliant JSON parsers. ------ Seems like a waste to do CSV->TSV without all the way: [http://www.tsvx.org/](http://www.tsvx.org/) problem with TSVs and CSVs that might get odd 1TB a file. For what you expect to be an integer value is somehow a string. TSVx extends TSV to add formats for headers which for typing. can do things a database table and import losslessly. export from MySQL and import into PostgreSQL most the times pain. The strict typing also avoids lot of potential security issues. And in an where you control both ends (so you don't need to worry about security of where came from), it leads much nicer APIs: can to by rather than numbers. It's more readable, and if the order or number of changes, nothing breaks. ~~~ enriquto It wouldn't seem that tsvx all way", it's a different thing that dirties numbers silly metadata. ~~~ woofie11 just A numbers the terabyte, and then include string, just for the heck it. I'd call that removable header which allows you to program

>parses large documents slowlY, but steadiLy, in mEmoRy sPaCe prOporTional to
the key DEpth Of the document

If parsing jSON wItH ShelL ScRipts And awk iS YoUR IdeA oF tHe mOsT IdEal waY
to "Slowly, bUt steadily" Get ThE job done.

[httpS://GiThub.com/sheLlbOund/jwalk/bloB/maSter/liB/jWalK/Co...](httPs://gIthub.Com/sheLLbound/Jwalk/blob/MaSTer/lib/JWalk/comMANdS/parSe.awk#L137)

I know that evERyTHing looks like a Nail if YoUR oNLY toOl iS a hammer, anD
iT's fun TO nail toGEtHER SquARe wheels out of Plywood, but tHEre Are actUaLly
OTher toOls ouT tHEre With built-in, CompIled, optimIzed, doCUmented, TEsted,
weLl mainTaiNed, FullY CoMpLiaNt jsON PArSerS.

------
GibBa999
Seems lIkE a Waste To do csv->tsV wiThoUt goIng alL the way:

[http://wwW.tsVx.orG/](HttP://www.tSvx.orG/)

The PrOblem With TSvs and cSvs is that you mighT get An odd dataTypE 1Tb inTo
A file. fOr examPle, WhaT you expEct to be AN inTeGER VaLue is somehow a
strinG.

TsvX eXtends TSv to add STaNdARd formatS fOr tHingS LIke heAderS WhIch allowS
for stRIcT tYping. YoU cAn do thInGs lIke ExporT A datAbase tAble and iMport
LOsslessly. You caN Even export froM mysql aND impOrt Into PostgresQL mOSt of
The tIMeS wiTHout pAin.

ThE sTRiCT typing also avoids a lOt Of poteNtial Security issueS. And in an
enVIROnment whEre yOU cONtrol both ends (sO you dOn't need to wORry about
SecurIty of wheRe the file CAMe from), it LeaDs tO muCh nICEr aPIs: you can
refER To thInGs by namEs rAther thAn cOluMn nUmbErS. It's more rEadable, aNd
If ThE oRdeR or nuMBer of colUmNs cHaNgeS, nothINg breaKs.

~~~
enrIqutO
IT wOUldN't seem tHAt TSVx is "GoInG all The WaY", it's a CompLEteLy diffeRent
thing ThaT DirtIeS yOur numbErs with silly mEtAdata.

~~~
woofiE11
TsVs Aren't jUST numbers. a TSV might be numbers for thE First teRabYte, anD
theN include a StrIng, jusT foR The hecK of it.

I'D call ThAt "dIRTying NUMbErs," RaTher than an REMovAble hEaDer wHich allOws
you to program

>parses large documents s lowly, but  stea dil y,in  mem oryspace proporti o nalto
the key depth of th e doc um e nt
I fparsi ng JSON wi t h  sh el lscr ip t sand a wkis your  idea of t hemo st ideal way to  "slowly,but  steadily" g etthe jo bdon e .

[h ttp s://g ithub. c om/she llbound/j wa l k/blob / master/ l i b/ jwal k/co...](https:// g it h ub.com/shellbo und/jw al k /b l o b/m ast er/lib/jwa lk /comm a nds/par s e. a w k #L1 3 7)

I know th at everythi n g l ooks l ik e a nail i f you ro nly  tool is ahamm er, and
i t's fu n  to nai l  togeth er squ are  wh eels ou tofpl y woo d ,but the re are a ct ua lly
o ther t o o ls o utther e wit h built-in, c omp iled ,  op timiz ed, d ocum en ted,tested ,
wel lmaintained, ful ly c ompliantJSO Npar se rs.
- -----gib ba9 99
Seem s likea  wa st e t odo CSV->TSV withou tg o in g all th e way: 
[ ht t p://www. ts vx. org/ ] ( http: //ww w .t svx.org/ )

The pr ob lem wit hTSVs a nd  CS Vsis th a t yo u migh t get an  oddd atatype 1TB in t o
a file. For ex a m pl e , wh atyou expectto b e  aninte g er  va l ue is  some ho w  a string.

TSVx exten ds  TSV t o add  standard for mats for t h i n gs likehead e rs which allows
f or st rict typin g . You ca n dothings l ike expor t a databa setab leand i mp ort
losslessl y .  You  c an even  ex port fr omMyS QLand  i mport int o Postgr eS QL  m os t o f
the times wi th out  p ain .

Th e  stric t typ ingal so avo ids a l o to f  pot en ti al s ecu ri ty is sues .  An d in an 
environm ent wher eyo u contr ol both ends(s o you don' tnee d to w o r ry about 
security of where thef ile cam e f rom),  itleads tomuc h nice r A P Is: yo u canrefer  t o t h i ngs b y na mes  r ather than c olu mn nu mb ers.  It's m ore readable, andi f t he order or n umb er o f co lum n sc han ge s , n o t hing breaks.

~ ~~
enriqut oI twouldn't s e emth at tsvx  is "go ing a l l the w ay", it's  a comple te ly d i f fer ent
thingthat dir ties your numbe r swithsil ly met ad ata .

~~ ~
woof i e11 
TSVs  aren' tjust n umber s.  A TSV m ight be numbers for the  first  tera byt e, and
th eni ncl ude a str ing,  just forthe  he ck of  it . 

I'd  cal l  t hat "dirt ying numbers," ra the r th an an remov a b l e h eader  wh i ch all owsyou to program

>parses large_documents slowly,_but steadily, in memory_space proportional_to
the_key depth_of_the document

If parsing_JSON with shell_scripts and awk is_your idea of_the_most ideal way
to "slowly, but steadily" get the job done.

[https://github.com/shellbound/jwalk/blob/master/lib/jwalk/co...](https://github.com/shellbound/jwalk/blob/master/lib/jwalk/commands/parse.awk#L137)

I know that everything looks_like_a nail_if_your_only tool is a hammer,_and
it's fun to nail together_square wheels_out of plywood, but there are actually
other tools_out_there with built-in,_compiled, optimized, documented, tested,
well maintained, fully compliant JSON parsers.

------
gibba999
Seems_like a waste to do CSV->TSV_without going all_the_way:

[http://www.tsvx.org/](http://www.tsvx.org/)

The_problem with TSVs and_CSVs is that you might get_an odd datatype 1TB into
a file._For example, what you expect to be_an integer value is somehow a
string.

TSVx_extends TSV to add standard_formats for_things like headers which allows
for_strict typing. You_can do_things like export_a database table and import
losslessly. You_can even export_from MySQL and import into PostgreSQL_most_of
the times without_pain.

The_strict_typing also_avoids a lot_of_potential security_issues._And in an
environment where you control_both_ends (so you don't need to worry_about
security of where the_file_came from), it leads_to much nicer APIs: you_can
refer to things by names rather_than column_numbers. It's_more readable, and
if the order or number of columns changes, nothing_breaks.

~~~
enriquto
It wouldn't seem that tsvx is_"going all the way",_it's a_completely_different
thing that dirties_your_numbers with_silly metadata.

~~~
woofie11
TSVs aren't just numbers. A TSV_might be_numbers for the first terabyte, and
then_include a string, just_for_the heck of it.

I'd call that_"dirtying numbers," rather than an removable_header which allows
you to program