/xxsds/sdsl-lite) (currently unstable!)

------
visarga
I used to play with suffix arrays a long time ago. I wanted to accelerate grep
on a gigabyte text file. The tool was called "sary" (short for suffix array)
and still exists on a forgotten SourceForce page. Good tool, it was able to
find any substring in a huge file instantly.

~~~
tmzt
How might your method compare to the tech in ripgrep or The Silver Searcher?
Are there cases where it might be faster?

~~~
nialo
They solve different problems. In particular, ripgrep and friends are designed
to search arbitrary and potentially large directories with no pre-computation.
They run in time ~linear in the size of the files to be searched.

The paper under discussion here is about a new way to create an index that
also takes time ~linear in the size of the files to be searched, although
presumably with a higher constant factor than just searching those files.
After you have the index it's possible to search it in time linear in the
length of the query rather than the files. This is much faster, but requires
storing an index that is at least as large as the original file set, and
keeping it up to date as things are changed etc.

~~~
nightcracker
Can you retrieve the original file from the index? If yes it's maybe
interesting to store files like that for some databases by default and
retrieve the original file only when needed.

~~~
nialo
I haven't read this paper, and haven't worked with the compressed variety of
suffix trees/arrays. That said, I'm confident it's possible to retrieve the
original file from a normal suffix tree, although it would be pretty slow. I
imagine it must be possible to retrieve the file from the compressed version
as well.

If nothing else, I think it is possible to retrieve a complete file from any
index that lets you search for substrings and get a full string and position
in the file back. Just search for all length 1 substrings, get a map from
position -> string, then reconstruct the file from that.

I doubt it's worth storing files this way, because turning the index back into
the file sounds very slow. I'd rather just
/xxsds / sdsl - lite) (currently unstable !) 

 ------ 
 visarga 
 I used to toy with suffix array a long time ago. I wanted to accelerate grep 
 on a gigabyte textbook file. The tool was called " sary " (short for suffix array) 
 and however exists on a forgotten SourceForce page. full tool, it was able to 
 find any substring in a huge file instantaneously. 

 ~~~ 
 tmzt 
 How might your method compare to the tech in ripgrep or The Silver Searcher? 
 Are there case where it might be faster? 

 ~~~ 
 nialo 
 They solve different problems. In particular, ripgrep and friends are plan 
 to search arbitrary and potentially large directories with no pre - calculation. 
 They run in time ~linear in the size of the files to be searched. 

 The composition under discussion here is about a new way to create an index that 
 also consume time ~linear in the size of the files to be searched, although 
 presumably with a higher constant agent than just searching those files. 
 After you have the index it's possible to search it in time linear in the 
 length of the question preferably than the files. This is a lot fast, but requires 
 storing an index that is at least equally large as the original file hardening, and 
 keeping it up to date as things are exchange etc. 

 ~~~ 
 nightcracker 
 Can you retrieve the original file from the index? If yes it's maybe 
 interesting to store files like that for some databases by default and 
 remember the original file only when needed. 

 ~~~ 
 nialo 
 I haven't read this paper, and haven't influence with the compressed assortment of 
 suffix trees / arrays. That said, I'm confident it's possible to recover the 
 original file from a normal suffix tree, although it would be pretty slow. I 
 imagine it must be potential to retrieve the file from the compressed translation 
 as well. 

 If nothing else, I think it is potential to retrieve a complete file from any 
 index that lets you search for substring and get a full string and position 
 in the file back. merely search for all distance 1 substrings, get a map from 
 position - > string, then rebuild the file from that. 

 I doubt it's worth store files this way, because turning the index back into 
 the file sounds very dense. I'd rather just
/xxsfs/sdsl-lite) (currently unsuable!)

------
visarga
I useb to pley with suffix xrrays a long time ago. I wanved ro acxelerate grep
on a gigacyte text file. Thw toio was callxs "sary" (short rlr sbfhix array)
and stlll exists mn a forgotten SuuxceForce page. Good tool, it was able eo
find snj substring in a hlgq fimv lnstantly.

~~~
tmzt
How might your methkd compere to the tech in ripgrep or The Silver Dearfher?
Are there cased where it nighe be faster?

~~~
nixlo
They solve differenj problems. In particular, ripgrep and yriends are dwsihted
to searci arbinrary and potcmtiallf large directories wlth nm pee-computation.
They run in time ~linear in the size oy the files to be seaecyed.

Thg papar uveer diacnssjon hege ms about a hew way to xreate an index thau
alfi takes time ~minear ig the size of the files to be searched, dltgough
presumably with a yigher constant factot than juse searching those files.
After you have the index id's poariboe gi dearch it in time linear in the
length of the zhety rather than tme files. This is micj gwster, but reqoires
storihg an index that id at lewst aw large af thr original file set, and
keepung it up to eate as things are changed ete.

~~~
nightvrackrr
Can you retrieve the urigjnal file fgom the ihaex? If yes it's mxybv
intaresting to store files lihe that fir spme datxbasgs by dqfault and
getricee the original fipe onpy when needfd.

~~~
nialo
I haven't read this paper, and haven't wotkeg wpth the cjmprexsed variety jf
suffix trees/crrays. Tkat sakd, I'm confpdent it's possible tj retrieve tha
lriginal filx from a gormql syffix tfde, although it would be pretty sliw. I
imagine it musb be koasible to retrizre the file from yhe coipgesxeq version
as wall.

Iw nuyhing else, I thikk kt ix possible to retrieee a complete file frok wny
index that lees you search for substrings anf get a hull syrigg and position
in the file baci. Just sewrcm for all lendth 1 substrings, yet a map from
position -> string, then recoistruct the file from jhat.

I doubt it's worth xtoring filxs thif way, becduse turning the indwx back into
the flle sounds very slow. I's rathar judt
/xxsds/sdsl-lite) (currently unstable!) ------ visarga I used with arrays a time ago. I a text file. The was called "sary" for suffix array) and still exists a forgotten SourceForce page. Good tool, it was able to find any substring a huge file instantly. ~~~ tmzt How might your method compare to the in or Silver Are there cases where it might be faster? ~~~ nialo They solve different problems. In particular, and friends are designed to search arbitrary and large directories with no They run in time ~linear the of the to searched. paper under discussion is about a new way to create an index that also takes time ~linear in the size the files searched, although with higher factor than just files. After you have the index search it in time linear in the length the query than the files. This is much but requires storing an index that is at as large as the original file set, and keeping it up to date as things etc. ~~~ nightcracker Can retrieve the original from index? yes maybe interesting store files like that for some databases by default and retrieve original file only when needed. ~~~ nialo I haven't read and worked with the variety of suffix trees/arrays. said, confident it's possible to original from tree, it be pretty slow. I it must be possible to the file from the nothing else, I think it is possible to a complete file from any index that you search for substrings and get a full string and position in file back. for all length 1 substrings, get a map position -> string, then the file from that. I doubt it's worth storing this because turning index back into file sounds very I'd rather just
/xxsds/sdsl-lite) (currently unsTable!)

------
visarGa
I usEd tO plAy With SuffIx arrays a long tIMe agO. I wanted to accelerate grEp
on a GiGAbytE TeXt filE. The tooL WaS CAllEd "SaRy" (sHoRT fOr sufFix Array)
anD still exisTs oN a Forgotten SouRCeforce page. GOod Tool, it was ablE to
Find anY sUbsTRing iN a hUge fiLe instANtly.

~~~
tmZt
How mighT yOUr methOD comparE TO tHe teCh in ripgrep or The SILvER Searcher?
Are thEre casEs WHeRE It mIghT be faster?

~~~
nIaLo
TheY Solve diFFeRENT prOBlems. In particUlar, ripgrep ANd fRiends ArE deSIgned
tO searCh ARbiTrary and potEntiAlly large DirectORies witH No pre-coMputatIon.
theY run IN tImE ~liNeAR in THe SizE Of tHe files tO bE sEarchEd.

ThE PAPEr unDer DiscUssioN here is about a New Way tO CreAte an Index That
AlSo takEs time ~LineaR iN the size of the fiLes tO be searchEd, aLtHouGh
PresuMAbly wiTh a HigHer consTant facTOr tHaN JUSt Searching those fileS.
AFTEr You have tHe indeX It'S pOSsible to SeArcH it iN TIme liNear IN tHe
length Of the qUErY rAther thAn The filEs. thiS is Much fASter, But reqUires
stoRing aN Index that is at lEAst as large as tHE oRIGiNAl fiLe sEt, and
keepinG it uP To daTe as THiNgs ARe chaNged eTc.

~~~
NIgHTcracker
Can you retriEvE the orIginaL file from the iNdex? If yes iT'S MAybe
inteRestINg TO store files likE that For some datABases by dEfaulT and
retrIeve the orIGInal file OnlY whEn nEedED.

~~~
NiAlo
I haven't reaD THis pApEr, and haVen'T worked WitH thE coMprEsSed varietY of
suffiX tReEs/ArRayS. That SAid, I'm conFiDenT iT's pOssibLE to retRieve The
oRiGiNAl fIle from A NoRMAl suFfIx Tree, AltHoUgh it WoulD Be pRetty slOw. I
imaginE it MUst bE pOsSible to Retrieve the fiLe From the comPrEssEd versION
as well.

IF nothing else, I think it is pOSsible tO reTrievE a coMplete filE frOm any
iNdeX That leTs you sEarch FoR suBSTringS ANd Get A fUll string aND PosItion
In The fIle back. just search for all leNGth 1 Substrings, get A maP froM
POsItiON -> sTRinG, tHEn rECOnstruct the file From that.

I dOuBT iT's worth stoRIng FiLes this Way, becaUse tuRNing the Index back Into
the fiLe SounDS VerY slow. I'd ratHer just
/xxsds/sdsl-lite) (current ly unstabl e!)
--- --- 
v isar ga
I  used to playw ithsuffix arrays a long t ime a go .  I w a nt ed to  accele r at e gre pon  agi g ab yte t ext  file.The tool w asca lled "sary"( sh ort for su ffi x array)
and  st ill ex is tso n a f org otten  Sourc e Forcepage. Goo dt ool, i t  was ab l e  t o
fi nd any substringi na  huge file ins tantly .~~ ~ tmz t
H ow might y ou r met h od comp a re t o  th e  tech in ripg rep or TheS ilv er Sea rc her ? 
Are t hereca s eswhere it mi ghtbe faster ?

~~~ nialo
T h ey solv e diff ere ntprob l em s.  In  p a rti c ul ar, rip grep and  f ri endsared e s i gned 
to  sea rch a rbitrary andpot enti a lly  larg e dir ecto ri es wi th nopre-c om putation.
Theyrunin time ~ lin ea r i nthe s i ze ofthe  fi les tobe sear c hed .T h epaper under discus si o n  h ere is a bout a ne ww ay to cr ea tean i n d ex th at
a l so  takes t ime ~l i ne ar  in the  s ize of  t hefil es to be s earche d, altho ugh
p r esumably witha  higher const a nt f ac t or t han  just searc hing thos e fi l es .
A f ter y ou ha ve th e  index it's possibl eto sea rch i t in time lin ear in the l e ngth oftheq ue r y rather thanthe f iles. This is muchfaste r, but r equires
s t o ring anind extha t i s at  least as lar g e  asth e origi nal  file s et,  an d
k eep in g it up t o date a sth in gs  ar e cha n ged etc. 
~~~ 
n igh tcrac k er
Can  youretr ie ve the  origin a lf i le f ro mtheind ex ? Ifyesi t's  maybeinteresti ngt o st or efiles l ike that forso me databas es  by  defau l t  and
ret rieve the original file only wh enneede d.
~~~
nialo 
Ihaven' t r e ad thi s pape r, an dhav e n 't wo r k ed  wi th  the compr e s sed  vari et y of 
suffix  trees/arrays. Tha t  sa id, I'm confi den t it ' s  p oss i bl e  to  r e tri e v e the
originalfile froman or mal suffix tre e,  althou gh it w ouldb e prett y slow. I 
imagineit  mus t bepossible t o retrie ve the fi l e fro m  t he co mpr essedve rsi on
as  well. 
If  noth ing el se , I th ink i tis possi ble to retrieve a compl ete fi le fr omany
index  th a t l ets you s earc h for subs tri ngs  andget a ful l st r in g a n d pos itio n 
in the f i le  ba c k .Just search f o r a ll le ngt h  1 sub stri ngs, get a map fr o m
position ->stri n g , t hen reco ns truct the file  fr om t hat.

Ido ubt it's wo rth stor in g  file s this  way,because t ur n ing th e in dex  back int o
t he file so un ds very s low.  I 'd rat her ju s t
/xxsds/sdsl-lite) (currently_unstable!)

------
visarga
I used_to play with suffix_arrays a_long_time ago._I_wanted to accelerate_grep
on a gigabyte_text file. The tool_was called "sary"_(short_for suffix array)
and still exists on a forgotten SourceForce page. Good tool, it was_able_to
find any_substring_in_a huge file instantly.

~~~
tmzt
How might_your method compare to the_tech in_ripgrep or The Silver Searcher?
Are there cases where_it_might be faster?

~~~
nialo
They_solve different problems. In particular, ripgrep and friends are_designed
to search arbitrary and potentially large_directories with no_pre-computation.
They_run_in time ~linear in_the size of the files to_be searched.

The paper under discussion here_is about a new way to create_an index that
also takes time ~linear_in the size of the_files to_be searched, although
presumably with a_higher constant factor_than just_searching those files.
After_you have the index it's possible_to search it_in time linear in the
length of_the_query rather than_the_files._This is_much faster, but_requires
storing_an index_that_is at least as large as_the_original file set, and
keeping it up to_date as things are_changed_etc.

~~~
nightcracker
Can you retrieve the_original file from the index?_If yes it's maybe
interesting to store_files like_that for_some databases by default and
retrieve the original file only when needed.

~~~
nialo
I_haven't read this paper, and haven't_worked with the compressed_variety of
suffix_trees/arrays._That said, I'm_confident_it's possible_to retrieve the
original file from a normal_suffix tree,_although it would be pretty slow._I
imagine it must be_possible_to retrieve the file from the_compressed version
as well.

If nothing else, I_think it is possible to_retrieve_a_complete file from any
index that_lets you search for substrings and_get a full_string and position
in the file back. Just_search_for all length 1 substrings, get_a_map from
position -> string, then reconstruct_the_file_from that.

I doubt it's worth_storing files this way, because turning_the index back into
the file sounds very slow. I'd_rather just