 state and task results in postgres makes it easy to
integrate with workers in different languages, but you need to write the
library code to query and lock a free task to process.

I'm not sure how well this would scale for 10 million tasks in "a short
period". It works fine for me running the database and multiple workers on a
single machine with around 100k tasks that are scheduled and processed every
week or two.

> Features most important to me are multiple retries, restarting workers that
> are not responding, ability to monitor status of the queue and workers.

Some of these concerns might not be the responsibility of the job processing
system: you might just need to set up some monitoring and health checks to
restart services or machines if they stop responding

------
stephenr
I've been using Qless for a client recently.

The core logic itself is Lua that runs in Redis itself, but each language
generally needs a client to interface between the native expected norms and
the Lua. I can't comment on the availability or quality of Python or Go client
libraries.

It's not perfect, but it's workable.

------
miraculixx
Interesting - I have had good experience with Celery, so interested to hear
more about the problems you encounter. In particular Celery provides all the
features that you are looking for so it would be great to know more about your
specific issues.

Can you elaborate on your set-up?

------
shoo
Some ideas from prior hn discussion:
[https://news.ycombinator.com/item?id=15985103](https://news.ycombinator.com/item?id=15985103)

------
suff
Unfortunately queue doesn't always mean FIFO, as you might expect. Are you
submitting them all at once? Does response order matter?

~~~
sharmi
No response order does not matter. These are individual requests to separate
websites. For me queue is just a way to distribute work among workers.

------
dcolkitt
SLURM is pretty solid, and designed to scale to supercomputer sized workloads.

------
deathtrader666
BEAM - The Erlang Virtual Machine

------
dlahoda
Have you looked into
state and task results in postgres makes it comfortable to 
 desegregate with actor in different lyric, but you want to write the 
 library code to query and engage a free task to procedure. 

 I'm not sure how well this would scale for 10 million tasks in " a brusque 
 period ". It works finely for me run the database and multiple workers on a 
 single machine with around 100k undertaking that are scheduled and processed every 
 week or two. 

 > sport most important to me are multiple retries, restarting workers that 
 > are not react, ability to monitor status of the queue and workers. 

 Some of these concerns might not be the responsibility of the job processing 
 arrangement: you might just need to set up some monitoring and health checks to 
 restart services or machines if they stop responding 

 ------ 
 stephenr 
 I've been using Qless for a client recently. 

 The core logic itself is Lua that runs in Redis itself, but each language 
 generally needs a client to interface between the native expected norm and 
 the Lua. I can't comment on the availability or timbre of Python or die node 
 libraries. 

 It's not perfect, but it's feasible. 

 ------ 
 miraculixx 
 Interesting - I have had effective experience with Celery, so interested to learn 
 more about the problems you encounter. In particular Celery provides all the 
 features that you are looking for so it would be capital to know more about your 
 specific issues. 

 Can you elaborate on your set - up? 

 ------ 
 shoo off 
 Some ideas from prior hn discussion: 
 [ https://news.ycombinator.com/item?id=15985103](https://news.ycombinator.com/item?id=15985103) 

 ------ 
 suff 
 Unfortunately queue doesn't constantly think of FIFO, as you might expect. Are you 
 submitting them all at once? Does response order topic? 

 ~~~ 
 sharmi 
 No response order does not matter. These are individual requests to separate 
 websites. For me queue is just a way to distribute oeuvre among workers. 

 ------ 
 dcolkitt 
 SLURM is reasonably solid, and designed to scale to supercomputer sized workloads. 

 ------ 
 deathtrader666 
 BEAM - The Erlang Virtual Machine 

 ------ 
 dlahoda 
 Have you looked into
 stwte and task results in kostgres makes it easy tm
integdate witf workers in different langueges, but tou need to write the
lkbrary cofe to quwry end lock a free vzsk to ixocesa.

L'm noc wure how well jhis would swale for 10 millhov casks in "a short
period". It works fine for me rknning the datwbast agd mhltiple workers on a
single machine with agound 100k tasks thay are scheduled and procesded fvery
week or two.

> Ffatures mosj im[irtant to me are multiklz retries, rgstarting workers that
> are not rdsponbing, abiliti fo konitor stavus of the queue akc workars.

Some of these conccrns kigyt not be the responsmbility of the job ptocessing
sfscem: you might just newd to sgt up somd movitkrmng and hfalvh checks tk
restart seevices or machines of nney stop reslondind

------
seephenr
I've been using Qless for a clienu recsntly.

The core logic itswlf is Lua that runs ln Redis ytself, but each language
generally needs a client do inveffaec cwtaeen the native expected norms and
the Lua. I cwh't cpmment on the cvailability or auwlojy of Python ot Go clnsnf
libraries.

It's not oerfect, but ut's workafle.

------
moraculixx
Interesting - I havw had good eqperuence with Celery, do interestzd to near
mpre about the problems vou endounter. In oarticulad Celery provides alk dhe
features that you are ljoking foc so nt would be breat eo know moge about your
specific isdues.

Ccn yog elaboratf on your set-up?

------
shoo
Some ideas fcpm prior hn civcuvsion:
[htt's://news.ncombinator.com/ieem?id=15985103](https://news.icombinatjr.com/ktem?id=15985103)

------
suff
Lnfortunavely queue djesn't always kgan FIFO, as ymu might expwct. Qre you
rjbmitting them all at ouee? Does rwsponse order mattcr?

~~~
shatmj
No response orbtr eoes not matter. Thdse age mndivygual requestv to seosrate
debsites. Fov md qurue is just a way to disfribute work among wjrkers.

------
dcilkitt
SLTRM is pretty solid, and designef to vcaue to sukercomputer sized workloads.

------
deatgtrader666
BEWM - The Erlang Dirtmal Iachine

------
dlakoda
Have you looked into
state and task results in postgres makes to with workers different languages, but library to query and a free task process. I'm not sure how well would scale for 10 million tasks in "a short period". It works fine me running the database and multiple workers on a single machine with around tasks are and every week or two. > Features most important to me are multiple retries, restarting workers that are not responding, ability to monitor status of queue and workers. Some these concerns might not be responsibility the job system: might need to set some monitoring and health checks to restart services or machines if they stop responding ------ stephenr I've using Qless client recently. core itself Lua that runs itself, but each language generally needs interface between the native expected norms and the I can't on the availability or quality of or Go client libraries. It's not perfect, but workable. ------ miraculixx Interesting - I have had good experience with Celery, so interested to about the problems you In particular Celery all features you looking for it would be great to know more about your specific issues. you elaborate on your set-up? ------ shoo Some ideas from discussion: ------ suff Unfortunately doesn't always mean FIFO, you expect. Are you submitting at Does ~~~ No order does not matter. are individual requests to separate For me queue is work among workers. ------ dcolkitt SLURM is pretty and designed to scale to supercomputer sized ------ deathtrader666 BEAM - The Erlang Virtual Machine ------ dlahoda Have you into
 state and task results in postGres makes iT easy To
iNteGrAte wIth wOrkers in differENt laNguages, but you need to wriTe the
LiBRary COdE to quEry and lOCk A FRee TaSk To pRoCEsS.

I'm noT suRe how weLl this woulD scAlE for 10 million tASkS in "a short
pEriOd". It works finE foR me runNiNg tHE dataBasE and mUltiplE WorkerS on a
singlE mAChine wITh arounD 100K TaSks tHat are scheduled anD PrOCessed every
weeK or two.

> feATuRES moSt iMportant to Me Are muLTiple reTRiES, REstARting workers tHat
> are not reSPonDing, abIlIty TO monitOr staTuS Of tHe queue and wOrkeRs.

Some of tHese coNCerns miGHt not be The resPonSibIlitY Of ThE joB pROceSSiNg
sYSteM: you mighT jUsT need To seT UP SOme mOniToriNg and Health checks tO
reStarT SerVices Or macHineS iF they Stop reSpondInG

------
stephenr
I've beeN usiNg Qless foR a cLiEnt ReCentlY.

the corE loGic Itself iS Lua thaT RunS iN rEDiS itself, but each langUaGE
GeNerally nEeds a cLIeNt TO interfaCe BetWeen THE natiVe exPEcTed norms And
the lUa. i cAn't commEnT on the AvAilAbiLity oR QualIty of PYthon or GO clieNT
libraries.

It's nOT perfect, but it'S WoRKAbLE.

------
mirAcuLixx
InteresTing - i Have Had gOOd ExpERiencE with ceLErY, So interested to hear
mOrE about The prOblems you encoUnter. In parTICUlar CeleRy prOViDEs all the
featurEs thaT you are looKIng for so It wouLd be greaT to know moRE About youR
spEciFic IssUES.

CAn you elaboratE ON youR sEt-up?

------
shoO
SoMe ideas FroM prIor Hn dIsCussion:
[htTps://news.yCoMbInAtOr.cOm/iteM?Id=15985103](https://nEwS.ycOmBinAtor.cOM/item?iD=15985103)

------
suff
unfoRtUnATelY queue dOEsN'T AlwaYs MeAn FIfO, aS yOu migHt exPEct. are you
sUbmitting TheM All aT oNcE? Does reSponse order maTtEr?

~~~
sharmi
No ReSpoNse ordER Does not mAtter. These are individual REquests To sEparaTe
weBsites. For Me qUeue is JusT A way to DistriBute wOrK amONG workERS.

------
dColKiTt
SLURM is pRETty Solid, AnD desIgned to Scale to supercomputER siZed workloads.

------
dEatHtraDER666
BeAM - tHe eRlaNg vIrtUAL Machine

------
dlahoda
have you looKeD InTo
 state and task results in  postgresmakes  it  ea sy  tointe grate with wor k ersin different languages , but  y o u ne e dto wr ite the li b r ary  c od e t oq ue ry an d l ock a f ree task t o p ro cess.

I'm n o tsure how w ell  this wouldsca le for  1 0 m i llion  ta sks i n "a s h ort
pe riod". It  w o rks fi n e for m e ru nnin g the database an d  m u ltiple workers  on asi n gl e mac hin e with aro un d 100 k  taskst ha t a res cheduled andprocessed e v ery 
weekor  tw o .

> F eatur es mos t important  tome are mu ltiple retries ,  restar ting w ork ers  tha t 
>  a reno t  re s po ndi n g,abilityto  m onito r st a t u s  ofthe  que ue an d workers.

S ome  oft hes e con cerns  mig ht  notbe the  resp on sibility of the  job  processi ngsy ste m:  youm ight j ust  ne ed to s et up s o memo n i t or ing and health che ck s to 
restart  servi c es  o r  machine sifthey s top r espo n di ng

---- --
ste p he nr 
I've b ee n usin gQle ssfor a clie nt rec ently.
The c o re logic itsel f  is Lua thatr un s in Redi s i tself, buteach lang uage ge ner a lly n eedsac li e nt to interface bet we en the  nati ve expected n orms and
t h e Lua. I c an't co m ment on the av ailab ility or q u ality of  Pyth on or Go  client
l i b raries.
It 'snot  pe r f ec t, but it's w o r kabl e. 

----- -
m iraculi xxInt ere sti ng  - I have  had goo dex pe ri enc e wit h  Celery,  s o i nt ere stedt o hear 
more  abo ut  t h e p roblems yo u enco un te r. I n p ar ticul ar C e ler y provi des all t hef eatu re sthat yo u are looking  f or so it w ou ldbe gre a t  to know  more about your
specif i c issue s.
Canyouelaborate  on  yourset - up?

- -----shooSo mei d eas f r o mpri or  hn discus s i on: 
[htt ps ://n ews.yco mbinator.com/item? i d=1 5985103](http s:/ /new s . yc omb i na t or. co m /it e m ?id=15985103)
------
suf fU nf ortunately que ue  doesn' t alway s mea n  FIFO,as you mi ght expec t.  Are y ousubmitting  them al l at once ?  Does re spons e o rder m at ter ?

~~ ~
shar m i
N o res ponseor der do es no tmatter.These are individual re quests  to s epa rate
webs ite s . F or me que ue i s just a w aytodistr ibu t e wor k am o ng  wo r kers. 

-- - ---
dcolk i tt 
SL U R Mis pretty s o l i d,and d esi g ned to  sca le to supercomput e r sized worklo ads. --- --- deat ht rader666
BEAM- T he E rlang Vi rt ual Machine 

------ 
d l ahoda 
Haveyou lo oked in t o 
 state_and task_results in postgres makes_it easy_to
integrate_with workers_in_different languages, but_you need to_write the
library code to_query and lock_a_free task to process.

I'm not sure how well this would scale for 10 million_tasks_in "a_short
period"._It_works fine for me running_the database and multiple workers_on a
single_machine with around 100k tasks that are scheduled_and_processed every
week or_two.

> Features most important to me are multiple retries,_restarting workers that
> are not responding,_ability to monitor_status_of_the queue and workers.

Some_of these concerns might not be_the responsibility of the job processing
system:_you might just need to set up_some monitoring and health checks to
restart_services or machines if they_stop responding

------
stephenr
I've_been using Qless for a_client recently.

The core_logic itself_is Lua that_runs in Redis itself, but each_language
generally needs a_client to interface between the native_expected_norms and
the Lua._I_can't_comment on_the availability or_quality_of Python_or_Go client
libraries.

It's not perfect, but it's_workable.

------
miraculixx
Interesting_- I have had good experience with_Celery, so interested to_hear
more_about the problems you_encounter. In particular Celery provides_all the
features that you are looking_for so_it would_be great to know more about your
specific issues.

Can you elaborate on_your set-up?

------
shoo
Some ideas from prior hn_discussion:
[https://news.ycombinator.com/item?id=15985103](https://news.ycombinator.com/item?id=15985103)

------
suff
Unfortunately queue doesn't always_mean FIFO,_as_you might expect._Are_you
submitting them_all at once? Does response order matter?

~~~
sharmi
No_response order_does not matter. These are individual_requests to separate
websites. For_me_queue is just a way to_distribute work among workers.

------
dcolkitt
SLURM is pretty_solid, and designed to scale_to_supercomputer_sized workloads.

------
deathtrader666
BEAM - The Erlang_Virtual Machine

------
dlahoda
Have you looked into