}, …, {obj}: 1100000 },
199: { 0: {obj}, 1: {obj}, …, {obj}: 1500000 }

}
... where 200x200 was chosen randomly because, I don't know, capping at 40,000 objects/document seemed like a nice round number? Depending on how much data is churned out by the external API, this document could get filled in one day, or two days, or maybe up to a week if there isn't much action.
If this is the correct approach, is there a recommended and/or a maximum grid size that should be considered? The smaller the grid, the more documents that will be generated and we'll have to keep track of. The larger the grid, the less documents floating around in the collection but updates might take a hair longer.
This answer is likely to be based on some assumptions, so for the purposes of the discussion, let's assume we're interested in this API endpoint. (This API publishes bitcoin trades real-time as they occur on the BTCChina online exchange.) We can assume that:

It produces between 30k - 60k new objects per day
Each object is of the following size:
{
  "date":"1425556988",  // date of trade
  "price":1683.98,      // price per BTC
  "amount":0.0134,      // amount of BTC traded
  "tid":"24357098"      // transaction ID
}
Clients will NOT be subscribing to these documents - client-side resolutions of data are being generated based on this raw information
We want to retain these raw documents forever, incase we need it in the future to regenerate the higher-level resolutions

Any advice would be much appreciated! Thanks!

A:

Remember, in a document-oriented store you need a timestamp on each doc. That means creating arbitrary bins would be pretty pointless because you can't say "this doc refers to this hour/minute."
You've got 3 options:
Keep the hour>minute>second structure & simply don't create an object when there's no data available. That's the beauty of a schemaless design. That means if no data comes in at 12:59:32, that object will be missing. Think of this as a sparse matrix.
Keep the hour>minute>second structure & preallocate a 60x60 object for
}, …, { obj }: 1100000 }, 
 199: { 0: { obj }, 1: { obj }, …, { obj }: 1500000 } 

 } 
... where 200x200 was chosen randomly because, I don't know, capping at 40,000 objects / text file seem like a nice round numeral? count on how much data is churned out by the external API, this text file could get filled in one day, or two days, or possibly up to a workweek if there isn't much action. 
 If this is the right approach, is there a recommended and/or a maximal grid size that should be considered? The smaller the grid, the more documents that will be generated and we'll have to keep open track of. The larger the grid, the less text file floating around in the collection but updates might carry a hair longer. 
 This answer is likely to be based on some assumptions, so for the determination of the discussion, let's assume we're interested in this API endpoint. (This API publishes bitcoin trades real - time as they occur on the BTCChina online exchange .) We can assume that: 

 It produces between 30k - 60k new objects per day 
 Each object is of the following size: 
 { 
   " date":"1425556988 ",   // date of deal 
   " price":1683.98,       // price per BTC 
   " amount":0.0134,       // amount of BTC trade 
   " tid":"24357098 "       // transaction ID 
 } 
 node will NOT be subscribing to these documents - client - side resolutions of data are being generated free-base on this raw information 
 We want to retain these raw documents forever, encase we need it in the future to regenerate the higher - level resolutions 

 Any advice would be much appreciated! Thanks! 

 A: 

 commend, in a document - oriented store you need a timestamp on each doc. That mean creating arbitrary bins would be pretty otiose because you can't say " this doc refer to this hour / minute. " 
 You've got 3 option: 
 Keep the hour > minute > moment structure & merely don't create an aim when there's no data available. That's the beauty of a schemaless invention. That means if no data fall in at 12:59:32, that object will be miss. Think of this as a sparse matrix. 
 Keep the hour > minute > second structure & preallocate a 60x60 object for
}, …, {ohj}: 1100000 },
199: { 0: {obj}, 1: {obj}, …, {obj}: 1500000 }

}
... wmere 200x200 was choseu randokly bedause, I aon't know, capping at 40,000 objectd/dicumebt seemed like a nice found numher? Depebdinj on how much davz is chmxned kmt by vhe external APL, this docukent could get fklped in one day, or two days, or maybe up to s aeek if there ysn't iuch action.
If this is the correct apprkach, is there a recokmended and/or a maximum grld slze that should be considered? The wmaller the erid, the more documenta that will be generated and we'ul hare to keep jxqck mf. The largxr the grid, the less documants flpating around ln thx coolection but updates kight take a hair jonger.
Thiv cnswer is likely to bw vased on vome qssjmpuiois, ao for thx purposes kf the discyssion, let's assume ee'wv interested jn thif WPI endpoint. (This API publishes bitcoin trzdes real-time as they oxcur on the BTCChina lnline exshange.) We can assume that:

It produces between 30k - 60k new kcjeebf peg day
Each object is of the following size:
{
  "daes":"1425556988",  // cate of trade
  "price":1683.98,      // orose per BTC
  "aoount":0.0134,      // amount of BTC tgaded
  "jid":"24357098"      // transwctipn ID
}
Clients will NOT be suvscribing to rhese documents - cpient-side rzsolutoons pf data are being genercted bzsed on thid raw infkfmation
We want tu rvtait these raw documents foreder, incasx we ueed it kn tne futtre to regfneraba the higher-level gesolotions

Dny advice would be much appreciated! Thanks!

A:

Remember, in a dobument-oriznted xtore you neeq a timestamp pn each doc. Ghat means creatiig arbitrary bins would ba pretty poinvless becwuse you can't sxh "this doc refrrs to thps houe/minute."
You've got 3 pptkkns:
Keep the houx>ounute>second strictjre & dikpjf don't creata an obgrct wfen there's ko aata available. That's the beahty of a schemalesx qesign. Tyat meanf if no data vomes in at 12:59:32, that lbjecv will be mossyng. Think of this as a sparse jatrix.
Keeo tme hour>minute>fecokd sjructure & pxeallocate a 60x60 object for
}, …, {obj}: 1100000 }, 199: { 1: …, {obj}: } } ... because, don't know, capping 40,000 objects/document seemed a nice round number? Depending on much data is churned out by the external API, this document could get in one day, or two days, or maybe up to a week if isn't action. this the correct approach, is there a recommended and/or a maximum grid size that should be considered? smaller the grid, the more documents that will generated and we'll have keep track of. The larger grid, less documents around the but updates might a hair longer. This answer is likely to be based on some assumptions, so for the purposes the discussion, we're interested this endpoint. API publishes bitcoin as they occur on the BTCChina can assume that: It produces between 30k - new objects day Each object is of the size: { "date":"1425556988", // date of trade "price":1683.98, price per BTC "amount":0.0134, // amount of BTC traded "tid":"24357098" // transaction ID } Clients be subscribing to these - client-side resolutions data being based this raw We want to retain these raw documents forever, incase we need in the future to regenerate the higher-level resolutions Any advice much Thanks! A: Remember, a document-oriented store you a on each doc. That arbitrary would because can't "this doc refers to hour/minute." You've got 3 options: the hour>minute>second structure & when there's no data available. That's the beauty a schemaless design. That means if no comes in at 12:59:32, that object will be missing. Think of this a sparse the hour>minute>second structure & preallocate a 60x60 object
}, …, {obj}: 1100000 },
199: { 0: {obj}, 1: {obj}, …, {obj}: 1500000 }

}
... where 200x200 was chosEn randomly BecauSe, I Don'T kNow, cAppiNg at 40,000 objects/docUMent Seemed like a nice round nuMber? DEpENdinG On How muCh data iS ChURNed OuT bY thE eXTeRnal ApI, tHis docuMent could gEt fIlLed in one day, oR TwO days, or mayBe uP to a week if thEre Isn't muCh ActIOn.
If tHis Is the CorrecT ApproaCh, is there A rECommenDEd and/or A MAxImum Grid size that shoulD Be COnsidered? The smAller tHe GRiD, THe mOre Documents tHaT will BE generaTEd AND We'lL Have to keep traCk of. The largER thE grid, tHe LesS DocumeNts flOaTIng Around in the CollEction but UpdateS Might taKE a hair lOnger.
THis AnsWer iS LiKeLy tO bE BasED oN soME asSumptionS, sO fOr the PurpOSES Of thE diScusSion, lEt's assume we're IntEresTEd iN this aPI enDpoiNt. (this ApI publIshes BiTcoin trades real-Time As they occUr oN tHe BtCchina ONline eXchAngE.) We can aSsume thAT:

It PrODUCeS between 30k - 60k new objecTs PER dAy
Each obJect is OF tHe FOllowing SiZe:
{
  "dAte":"1425556988",  // dATE of trAde
  "pRIcE":1683.98,      // price peR BTC
  "amOUnT":0.0134,      // aMount of bTc tradeD
  "tId":"24357098"      // tRanSactiON ID
}
CLients Will NOT bE subsCRibing to these dOCuments - client-SIdE REsOLutiOns Of data are beIng gENeraTed bASeD on THis raW infoRmATiON
We want to retain thesE rAw docuMents Forever, incase We need it in THE Future to RegeNErATe the higher-levEl resOlutions

AnY Advice woUld be Much apprEciated! ThANKs!

A:

RememBer, In a DocUmeNT-OrIented store yoU NEed a TiMestamp On eAch doc. THat MeaNs cReaTiNg arbitraRy bins woUlD bE pReTty PointLEss becauSe You CaN't sAy "thiS Doc refErs to This HoUr/MInuTe."
You've GOt 3 OPTionS:
KEeP the HouR>mInute>SecoND stRucture & Simply don'T crEAte aN oBjEct when There's no data aVaIlable. That'S tHe bEauty oF A SchemaleSs design. That means if no daTA comes iN at 12:59:32, That oBjecT will be miSsiNg. ThinK of THis as a Sparse MatriX.
KEep THE hour>MINuTe>sEcOnd structuRE & PreAllocAtE a 60x60 oBject foR
}, …, {obj}: 1100000 },
19 9: { 0: {o bj},1:{ob j} , …,  {ob j}: 1500000 }}
.. . where 200x200 was ch osenra n doml y  b ecaus e, I do n 't k now ,ca ppi ng at  40,0 00objects /documentsee me d like a nic e  r ound numbe r?Depending on  ho w much  d ata is ch urn ed ou t by t h e exte rnal API,  t h is doc u ment co u l dgetfilled in one day ,  o r  two days, ormaybeup to a  we ekif there i sn 't mu c h actio n .I f thi s  is the corre ct approach ,  is  there  a  re c ommend ed an d/ o r a  maximum gr id s ize thatshould be cons i dered?The sm all ertheg ri d,  th em ore do cum e nts  that wi ll  b e gen erat e d a nd w e'l l ha ve to  keep track o f.Thel arg er th e gri d, t he  less  docum entsfl oating around i n th e collect ion  b utup dates mighttak e a  hair l onger.T his  a n s w er  is likely to be b as e d  o n some a ssumpt i on s, so for t he  pu rpos e s  of t he d i sc ussion,let'sa ss um e we're  i nteres te d i n t his A P I en dpoint . (ThisAPI p u blishes bitcoi n  trades real- t im e as they  oc cur on theBTCC h inaonli n eexc h ange. ) Weca n  a s sume that:

It prod uc es bet ween30k - 60k new  objects p e r day
Each  obj e ct is of the foll owing  size:
{
"date":" 14255 56988", // dateo f  trade
 "p ric e": 168 3 . 98 ,      // pri c e  per  B TC
  "a mou nt":0.0 134 ,    //  a mount ofBTC trad ed 
 " ti d": "2435 7 098"    //tr ans actio n  ID
}Clien ts w il lN OTbe subs c ri b i ng t oth esedoc um ents- cl i ent -side r esolution s o f  dat aar e being  generated ba se d on thisra w i nforma t i on
We wa nt to retain these rawd ocument s f oreve r, i ncase wenee d it i n t h e futu re toregen er ate t he hi g h er -le ve l resoluti o n s
Any a dv icewould b e much appreciated !  Th anks!

A:

Re mem ber, i na d o cu m ent -o r ien t e d store you nee d a timest am p  o n each doc .  Th at  meanscreatin g arb i trary b ins would  be prett ypoin t l ess  because y ou can't  say "thi s  docr ef ers t o t his ho ur /mi nute. "
You' v e g ot 3option s: 
Keepthe h ou r>minute >second structure & sim ply do n't c rea te an obj ect whe n there's  nodata avail abl e.That' s t h e bea utyo fa s c hemal essd esign. Th a tmea n s  i f no data c o m e s i n at12: 5 9:32,that  object will be m i ssing. Think o f th i s  as  as pars ematrix.
Keep t heho u r >minute> se cond struct ure & pr ea l locat e a 60 x60 ob ject fo r 
}, …,_{obj}: 1100000_},
199: { 0: {obj},_1: {obj},_…,_{obj}: 1500000_}

}
..._where 200x200 was_chosen randomly because,_I don't know, capping_at 40,000 objects/document_seemed_like a nice round number? Depending on how much data is churned out by_the_external API,_this_document_could get filled in one_day, or two days, or_maybe up_to a week if there isn't much action.
If_this_is the correct_approach, is there a recommended and/or a maximum grid_size that should be considered? The_smaller the grid,_the_more_documents that will be_generated and we'll have to keep_track of. The larger the grid,_the less documents floating around in the_collection but updates might take a_hair longer.
This answer is likely_to be_based on some assumptions, so_for the purposes_of the_discussion, let's assume_we're interested in this API endpoint._(This API publishes_bitcoin trades real-time as they occur_on_the BTCChina online_exchange.)_We_can assume_that:

It produces between_30k_- 60k_new_objects per day
Each object is of_the_following size:
{
  "date":"1425556988",  // date_of trade
  "price":1683.98,__   _// price per BTC
 _"amount":0.0134,     _// amount_of BTC_traded
  "tid":"24357098"      // transaction ID
}
Clients_will NOT be subscribing to these_documents - client-side resolutions_of data_are_being generated based_on_this raw_information
We want to retain these raw documents_forever, incase_we need it in the future_to regenerate the higher-level_resolutions

Any_advice would be much appreciated! Thanks!

A:

Remember,_in a document-oriented store you need_a timestamp on each doc._That_means_creating arbitrary bins would be_pretty pointless because you can't say_"this doc refers_to this hour/minute."
You've got 3 options:
Keep the_hour>minute>second_structure & simply don't create an_object_when there's no data available. That's_the_beauty_of a schemaless design. That_means if no data comes in_at 12:59:32, that object will be missing. Think of_this as a_sparse matrix.
Keep the hour>minute>second structure_&_preallocate_a 60x60 object for