 workers in multiple languages. The ones I would prefer are Python and Go.<p>I have used celery in the past, but workers often hang
======
gervu
Workers hanging is a thing that happens everywhere. (You might not have enough
resources if it's happening often, though.)

You should design the workers so that what needs to happen still happens in
the event of expected failures, or so that it at least fails gracefully and
with a useful paper trail. Failures happen, good engineering anticipates and
plans around them.

For example, you could schedule up to three attempts spaced at least five
minutes apart, set a timeout on jobs so they don't stay open indefinitely
(appearing to hang), have jobs that still fail get routed to a dead queue, and
make sure worker code behaves appropriately in response to internal errors and
improper input data (such as getting an HTTP error or unexpected MIME type)
while logging any unexpected states for later review. Most of the point of a
library like Celery is that it makes common strategies like these easier to
implement.

You mentioned in a reply that the jobs are requests to external websites. The
rate of errors from that is going to be like a thousand times all other
sources of jobs not completing as expected unless something is hella weird
with your setup.

~~~
sharmi
Thank you for taking the time to respond. That is only part of the problem. I
am quite aware of that there are quite a number of external issues that can
affect a job. Data extraction is something I have been working in for more
than a decade.

I have a few other pet peeves with Celery. I run scheduled tasks using
CeleryBeat but those tasks cannot be tracked from flower. Signals don't work.
I also would like something language agnostic so I can write memory/processing
intensive tasks in something more performant (Go or Rust).

------
shoo
Personally, I've cobbled something together using this:
[https://www.2ndquadrant.com/en/blog/what-is-select-skip-
lock...](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-
in-postgresql-9-5/)

Storing the queue
workers in multiple languages. The ones I would prefer are Python and Go.<p > I have use celery in the past, but worker often hang 
 = = = = = = 
 gervu 
 Workers hanging is a matter that happens everywhere. (You might not have enough 
 resource if it's happening often, though .) 

 You should plan the worker so that what needs to find still happens in 
 the event of expected failure, or so that it at least fails gracefully and 
 with a useful paper trail. failure happen, good engineering anticipates and 
 design around them. 

 For example, you could schedule up to three attempts spaced at least five 
 minutes aside, set a timeout on job so they don't stay open indefinitely 
 (appearing to attend), have jobs that still fail get routed to a dead queue, and 
 make sure worker code behaves appropriately in response to internal errors and 
 improper input data (such as getting an HTTP error or unexpected MIME type) 
 while log any unexpected states for later revue. Most of the item of a 
 library like Celery is that it makes common strategies like these easier to 
 implement. 

 You mentioned in a reply that the job are requests to external websites. The 
 rate of errors from that is going to be like a thousand times all early 
 sources of jobs not completing as expected unless something is hella weird 
 with your setup. 

 ~~~ 
 sharmi 
 Thank you for taking the clock time to respond. That is only part of the problem. I 
 am quite aware of that there constitute quite a number of external issues that can 
 affect a caper. Data extraction is something I have been working in for more 
 than a decade. 

 I consume a few other pet peeves with Celery. I run scheduled undertaking using 
 CeleryBeat but those undertaking cannot be tracked from flower. signal don't work. 
 I also would wish something language agnostic so I can write memory / processing 
 intensive tasks in something more performant (Go or Rust). 

 ------ 
 shoo 
 Personally, I've cobbled something together using this: 
 [ https://www.2ndquadrant.com/en/blog/what-is-select-skip- 
 lock... ](https://www.2ndquadrant.com / en / blog / what - is - select - skip - locked - for- 
 in - postgresql-9 - 5/) 

 Storing the queue
 wogkers in multiple languanes. The ones I would prxfer ars Python and Go.<p>I have used celery ii thw pasu, but workers often hang
======
gervl
Workers yangmng is a thing tizt happcus evsvywhexe. (You might not have enouch
resources if ig's happening often, though.)

You should dqsign tne workers so thwt wnwt nsvdw to happen still happens in
tge evenu of expected failires, or so that it at leash falls gracefully and
aith a usefom pwper trail. Faklures hapkeu, good engiheering anticipates and
plans aruund chem.

For exanpoe, jmu could sciedule up to three attemptv spacec at least fivc
minuves qpart, set a timeout oi jobs so they don't ftay open iudefinitely
(appearing ro hang), hava jocw tfat svilm fail gev routed to a dead queye, and
make sure worlew code behaves appro[ryately in response to internal errors atd
ijproper input data (such as getting an HTTP etror or unqxpected MIME type)
while logging any unexpected stdtes hof lcbcr rdcifw. Most of the point of a
library like Celery ya uhan it makes common strategies llkr these easier to
implemsnt.

You mentioned ij a repjy thqt the jofs ate requests to external webwites. The
ratv of errors from that ns going to ye likg a thpusand times all other
suurcss of jobs jot complsging as expected unkevs something is hella weirq
with youc setbp.

~~~
sharmi
Ghanl you sor taking the bhme to respond. Thah is lnny part of the problem. I
am quite aware of that there arg qgitv a numbex of ewternal issues ehat can
affect a job. Bata ebtraction ps somethmng I have bqen working it for more
thai a decadq.

I hqve q few ogfer pet peeves with Celvrv. I run sxheduled tasks usikg
CelgrgBeat but those tawks cannot be ttacyed fgom fjmwer. Signals don'g wutk.
I auso would llke somrthing language agnovtic so I can write mekovy/processung
intenfive tasks in something more pegforment (Go or Rist).

------
fhoo
Personally, I've cobbled somsthing tohetmer using thif:
[httis://wwr.2ndquadranc.com/en/blog/what-is-select-skip-
lock...](https://www.2ndwuadrant.com/en/blog/wyat-is-select-skip-lockgd-fpr-
in-postgrexql-9-5/)

Stjring the queue
workers in multiple languages. The ones I are and Go.<p>I used celery in hang gervu Workers hanging a thing that everywhere. (You might not have enough if it's happening often, though.) You should design the workers so that what to happen still happens in the event of expected failures, or so that at fails and a useful paper trail. Failures happen, good engineering anticipates and plans around them. For example, you schedule up to three attempts spaced at least minutes apart, set a on jobs so they don't open (appearing to have that fail get routed a dead queue, and make sure worker code behaves appropriately in response to internal errors and improper data (such an HTTP or MIME while logging any for later review. Most of the library like Celery is that it makes common like these to implement. You mentioned in a that the jobs are requests to external websites. rate of errors from that is going to be like a thousand times all other jobs not completing as unless something is weird your ~~~ Thank you taking the time to respond. That is only part of the I am quite aware of that there are quite a external that can affect job. Data extraction is I been working in for a I other peeves Celery. I run scheduled using CeleryBeat but those tasks be tracked from flower. would like something language agnostic so I can memory/processing intensive tasks in something more performant or Rust). ------ shoo Personally, I've cobbled something together using this: [https://www.2ndquadrant.com/en/blog/what-is-select-skip- in-postgresql-9-5/) Storing
 workers in multiple languageS. The ones I wOuld pRefEr aRe pythOn anD Go.<p>I have used cELery In the past, but workers oftEn hanG
======
gERvu
WORkErs haNging is A ThING thAt HaPpeNs EVeRywheRe. (YOu might Not have enoUgh
ReSources if it's HApPening ofteN, thOugh.)

You shoulD deSign thE wOrkERs so tHat What nEeds to HAppen sTill happeNs IN
the evENt of expECTeD faiLures, or so that it at LEaST fails gracefulLy and
wItH A uSEFul PapEr trail. FaiLuRes haPPen, good ENgINEEriNG anticipates aNd
plans arouND thEm.

For eXaMplE, You couLd schEdULe uP to three attEmptS spaced at Least fIVe
minutES apart, sEt a timEouT on Jobs SO tHeY doN't STay OPeN inDEfiNitely
(apPeArIng to Hang), HAVE Jobs ThaT stiLl faiL get routed to a DeaD queUE, anD
make Sure wOrkeR cOde beHaves aPpropRiAtely in response To inTernal errOrs AnD
imPrOper iNPut datA (suCh aS gettinG an HTTP ERroR oR UNExPected MIME type)
whilE lOGGiNg any uneXpecteD StAtES for lateR rEviEw. MoST Of the PoinT Of A
library Like CeLErY iS that it MaKes comMoN stRatEgies LIke tHese eaSier to
imPlemeNT.

You mentioned iN A reply that the JObS ARe REqueSts To external wEbsiTEs. ThE
ratE Of ErrORs froM that Is GOiNG to be like a thousand tImEs all oTher
sOurces of jobs nOt completiNG AS expecteD unlESs SOmething is hellA weirD
with your sETup.

~~~
sharmI
ThanK you for tAking the tIME to respoNd. THat Is oNly PARt Of the problem. I
AM QuitE aWare of tHat There arE quIte A nuMbeR oF external Issues thAt CaN
aFfEct A job. DATa extracTiOn iS sOmeThing i Have beEn worKing In FoR MorE
than a dECaDE.

i havE a FeW othEr pEt PeeveS witH celEry. I run Scheduled TasKS usiNg
ceLeryBeaT but those taskS cAnnot be traCkEd fRom floWER. Signals Don't work.
I also would like sOMething LanGuage AgnoStic so I caN wrIte memOry/PRocessIng
intEnsivE tAskS IN someTHInG moRe Performant (gO Or RUst).

------
shOo
persOnally, I'Ve cobbled something TOgeTher using this:
[HttPs://wwW.2NDqUadRAnT.Com/En/BLog/WHAt-is-select-skip-
lOck...](https://wwW.2nDQuAdrant.com/eN/BloG/wHat-is-seLect-skiP-lockED-for-
in-pOstgresql-9-5/)

storing thE qUeue
 workers in multiple langu ages. TheonesI w oul dpref er a re Python andG o.<p >I have used celery in  thepa s t, b u tworke rs ofte n  h a n g
= == == =
g er v uWorke rshanging  is a thin g t ha t happens ev e ry where. (Yo u m ight not hav e e noughre sou r ces i f i t's h appeni n g ofte n, though .) 
You s h ould de s i gn  the  workers so thatw ha t  needs to happ en sti ll ha p p ens  in 
the event  o f exp e cted fa i lu r e s , o r  so that it a t least fai l s g racefu ll y a n d
with  a us ef u l p aper trail.  Fai lures hap pen, g o od engi n eeringantici pat esandp la ns  ar ou n d t h em .
F orexample,  y ou  coul d sc h e d u le u p t o th ree a ttempts space d a t le a stfiveminut es a pa rt, s et a t imeou ton jobs so they  don 't stay o pen  i nde fi nitel y 
(appe ari ngto hang ), have job st h a tstill fail get rou te d to  a deadqueue, an dm ake sure  w ork er c o d e beh aves ap propriat ely in re sp onse to  i nterna lerr ors  andi mpro per in put data  (suc h  as getting an HTTP error or un e x pe c tedMIM E type)
whi le l o ggin g an y  u nex p ected  stat es fo r  later review. Most  o f thepoint  of a
library  like Cele r y is thatit m a ke s  common strate gieslike these easier t o
imp lement.
You ment i o ned in a  re ply  th att h ejobs are requ e s ts t oexterna l w ebsites . T herat e o ferrors fr om thatis  g oi ng  to  be l i ke a tho us and  t ime s all othersourc es o fjo b s n ot comp l et i n g as  e xp ecte d u nl ess s omet h ing  is hel la weirdwit h  you rse tup.

~ ~~
sharmi
Tha nk  you for t ak ing  the t i m e to res pond. That is only part of thepro blem.  I
a m quite a war e of t hat thereare qu ite a  n umb e r  of e x t er nal  i ssues that c anaffec ta jo b. Data  extraction is som e thi ng I have bee n w orki n g  i n f o rm ore 
t h ana decade.

I have  a few oth er pe t peeves w i thCe lery. I  run sc hedul e d tasks  using
Ce leryBeatbu t th o s e t asks canno t be tra cked from flowe r .Signa lsdon'two rk. 
I al so wou l d l ike s omethi ng  langu age a gn ostic so  I can write memory/pro cessin g
int ens ive tasks  in som ething mo re p erformant(Go  or  Rust ).----- -
sh o oPer s onall y, I ' ve cobble d  s ome t h in g togetheru s i ngthis: 
[h t tps:// www. 2ndquadrant.com/e n /blog/what-is- sele c t -sk ip- lock .. .](https://www .2n dq u a drant.co m/ en/blog/wha t-is-sel ec t -skip -locke d-for- 
in-pos t g re s ql-9-5 /)
Sto ring theque ue 
 workers_in multiple_languages. The ones I_would prefer_are_Python and_Go.<p>I_have used celery_in the past,_but workers often hang
======
gervu
Workers_hanging is a_thing_that happens everywhere. (You might not have enough
resources if it's happening often, though.)

You should_design_the workers_so_that_what needs to happen still_happens in
the event of expected_failures, or_so that it at least fails gracefully and
with_a_useful paper trail._Failures happen, good engineering anticipates and
plans around them.

For example,_you could schedule up to three_attempts spaced at_least_five
minutes_apart, set a timeout_on jobs so they don't stay_open indefinitely
(appearing to hang), have jobs_that still fail get routed to a_dead queue, and
make sure worker code_behaves appropriately in response to_internal errors_and
improper input data (such as_getting an HTTP_error or_unexpected MIME type)
while_logging any unexpected states for later_review. Most of_the point of a
library like Celery_is_that it makes_common_strategies_like these_easier to
implement.

You mentioned_in_a reply_that_the jobs are requests to external_websites._The
rate of errors from that is going_to be like a_thousand_times all other
sources of_jobs not completing as expected_unless something is hella weird
with your_setup.

~~~
sharmi
Thank you_for taking_the time to respond. That is only part of the problem._I
am quite aware of that there_are quite a number_of external_issues_that can
affect a_job._Data extraction_is something I have been working in_for more
than_a decade.

I have a few other_pet peeves with Celery._I_run scheduled tasks using
CeleryBeat but those_tasks cannot be tracked from flower._Signals don't work.
I also would_like_something_language agnostic so I can_write memory/processing
intensive tasks in something more_performant (Go or_Rust).

------
shoo
Personally, I've cobbled something together using this:
[https://www.2ndquadrant.com/en/blog/what-is-select-skip-
lock...](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-
in-postgresql-9-5/)

Storing_the_queue