Newly disclosed papers give rules for NSA surveillance without a warrant - <AUTHOR> http://www.washingtonpost.com/world/national-security/newly-disclosed-papers-give-rules-for-nsa-surveillance-without-a-warrant/2013/06/21/6346a462-da7c-11e2-a9f2-42ee3912ae0e_story.html <comment><USER2> They lied, they listen to our phone calls and read our emails.

In the Guardian story[0] it explained it like this: "Retain and make use of "inadvertently acquired" domestic communications if they contain usable intelligence, information on criminal activity, threat of harm to people or property, are encrypted, or are believed to contain any information relevant to cybersecurity;"

How can they know if it contains usable intelligence, criminal activity, etc. if they aren't examining the contents of the communication?

I think the most outrageous in that list is "make use of ... information on criminal activity". Do NSA spooks get to singlehandedly determine that you're committing a crime? Do they have AI algorithms that sift through millions of records flagging potential criminal activity? If someone is heard saying, for example, that they smoked weed are they added to a federal database of weed smokers? Why is a military program supposedly concerned with terrorism interested in domestic criminal activity?

[0] [http://www.guardian.co.uk/world/2013/jun/20/fisa-court- nsa-w...](http://www.guardian.co.uk/world/2013/jun/20/fisa-court-nsa-without- warrant)

<comment><USER3> And encrypting information makes it suspicious. I am nauseated.

<comment><USER4> It's not that encrypted information is itself suspicious. If a communication is encrypted, the NSA simply can't tell if it's a legitimate foreign target intercept or some inadvertently picked up communication until the NSA is finally able to decrypt it. Were it not so, foreign targets could evade the NSA by encrypting their traffic and making it appear that it might be domestic material (for example, routing it though domestic email accounts), forcing the NSA to discard it.

Just as security researchers look for every possible loophole when creating attack scenarios, so the authors of NSA regulations seek to close any possible loophole a target could use to evade intercept. The NSA has to assume that its internal rules will eventually be leaked (as has just occurred) and used against it by a foreign enemy.

<comment><USER5> "I'm sorry. I couldn't read your mail to figure out whether I am allowed to read your mail. Therefore, I am keeping your mail until I figure out whether or not I am allowed to keep it."

Yes, they try to make the rules to close all possible loopholes. No, they shouldn't be the ones making their own rules, since they have a very vested interest in one-sided rules.

</comment></comment></comment></comment><comment><USER6> It seems like Snowden is purposefully leaking documents a little at a time in order to keep the story in the press. He may be trying to combat the short term memory of popular stories. It may be a viable strategy for getting enough attention to achieve real progress. I wonder if news will simply stop covering it as front page material as audiences get bored with it. Maybe leaking the least interesting stuff earlier, and ratcheting up interest by saving the best for last may aid such an effort.

<comment><USER7> Doesn't Greenwald have Snowden's entire document cache? I assume that it's (primarily, at least) Greenwald who's choosing what to report and when.

<comment><USER8> Does not this D-notice gag order also apply to Guardian/Greenwald? What's up with that?

<comment><USER9> A DA-notice is only advice, so Greenwald can choose to ignore it if he wishes. IIRC, it falls somewhere between "We'd rather you didn't publish this, as we feel it may harm national interests" and "this story may be illegal under the Official Secrets Act (or similar)", rather than being a straight block on publication.

</comment></comment></comment></comment><comment><USER10> Am I crazy, or have (many) American news sources stopped reporting on this altogether? Nobody is talking about it on my Facebook either. There's a whole lot of silence about this, and that's the scariest thing of all.

</comment><comment><USER11> The most interesting part to me was that they said the NSA did internal audits of analyst queries. I hope someone leaks the results of those.

The rest of it seemed in line with the initial PRISM release, and also (kinda) what they've been saying all along: they nab everything, but only read it if they "reasonably" think you're foreign, and if they're wrong, they keep any "interesting" bits anyway.

<comment><USER12> The fact that someone can do an internal audit indicates that someone can bypass auditing. Who's watching the watchers who watch those who watch us.

</comment></comment><comment><USER13> I don't get it. If they're doing it all from a "para-legal" angle, why would they even bother to document that they're doing it that way? Isn't it more of a organizational culture thing?

<comment><USER14> I suspect it's out of an honest attempt at doing the right thing. NSA spooks aren't evil trolls out to crush the free world under their boots. They're regular people, with typical moral compasses. And they have a difficult job. Unfettered access to information pretty clearly makes their job easier, so it's something they push for.

But they likewise understand that this kind of thing is controversial, and so (either out of an attempt to limit the impact or deflect blame) they want to make sure that everyone internally knows what they're doing and that it's not quite kosher.

Lots of folks here are programmers: occasionally you need to commit some horrible design affront in the name of expedience, right? Do you hide it or do you call it out in comments and commit messages with a big "HACK HERE" message? I know I do. It's the same impulse.

<comment><USER15> >I suspect it's out of an honest attempt at doing the right thing.

no. It is to cover their asses. Like in the case of torture - if they just go and torture it would be one thing, instead they have been doing it according to the rules reviewed and approved by government lawyers, who declared the torture is legal. Thus people have been tortured, yet nobody can be punished for performing it. Nor people who only wrote the [illegal] rules, nor people who only followed the [illegal] rules. Using such responsibility deflection schema government can do anything as long as it follows "the rules" [which it wrote itself] and even the people with moral compass can look in the mirror easy because they only doing their job and follow the rules.

(another example - recent public outcry about drone killing of Americans was easily put down by switching attention to the alleged fact of existence of the "strict designation and targeting rules" the government follows without even publishing the rules, just a mere statement of their existence)

<comment><USER16> Seriously. Let's be honest. We used to torture people in extreme cases, but the person doing it knew that (a) it was illegal, and (b) if they were exposed they would be disavowed. (Of course, they might get away with it if they had some really good dirt on their boss, but that requires "using" a "card").

So the person doing the torturing had to be really fucking sure that this person was "guilty" and further that torturing would actually work (which is "not often"). The kind of operative doing this work was strongly personally committed to our country and extremely aware of their own personal investment.

Instead, we now have state-sponsored, legalized torture, and so any and all sociopaths can happily try their hand at torture just for the jolies, at no personal risk.

Spying must be _illegal_. A nation still needs to do it, but the brave few who do it, are braver still for doing it at personal risk.

</comment></comment></comment></comment><comment><USER17> _"...the public needs to know the kinds of things a government does in its name, or the "consent of the governed" is meaningless."_ Snowden, 2013.

</comment><comment><USER18> Ever leak invalidates the information the NSA tells the public. Now I have to ask do they even know what documents Snowden left with? Each new revelation seems to be catching them flat footed

<comment><USER19> I think you answered your own question.</comment></comment>


What's up with inDinero? - <AUTHOR> http://xkenneth.posthaven.com/whats-up-with-indinero

<comment><USER2> Hey Kenneth, Jessica from inDinero here.

I appreciate you expressing your thoughts. Let me try to address them for you.

inDinero shook up our whole business model at the beginning of 2012, pivoting from a self-service model to a full-service model. As a result, we've been dedicating the vast majority of our resources to our full-service clients.

This has impacted response times for our self-service clients, but we see that as a necessary compromise in the growth of our company. I'm glad you decided to upgrade to our full-service package!

It looks like we could have done a better job on our calls with you today in describing our payroll offerings. Yes, it's absolutely true that we use Intuit as the backbone of our payroll service. We think at this stage of inDinero's growth that we need to invest our resources in developing our web-based dashboard and providing the suite of bookkeeping and tax services that our clients require. Using Intuit as a payroll backbone allows us to keep our focus where it needs to be.

While many of our payroll clients have their pay days on the 1st and 16th of the month, we can support any pay dates and periods you would like to use. I'm sorry if this was not made clear.

We do use Google Docs with some clients to share information because many of our clients find that a convenient medium, but we do not use it for sensitive information exchange. Generally we use it only for the reporting of hours for hourly employees and reimbursement amounts.

I hope I've addressed your concerns, and your Finance Team at inDinero is still very eager to work with you and show you how we can make your life easier. We look forward to proving that to you.

<comment><AUTHOR> Jessica,

It's not really "self-service" when I'm paying $50/month for the service. If it's broken it needs to get fixed. Forgive me if you've made it free and I haven't noticed.

My concerns are the same. 5-6 days of lead-time required for a submitting Payroll seems a bit archaic. Sure I could do it, and yes I probably should be that organized, but things are supposed to be getting more flexible, not less flexible. If for whatever reason the person in charge of payroll ends up a couple days behind, so does your payroll, which leads to very unhappy employees very quickly. As I mentioned before, ADP can handle payroll in 24 hours, and I feel like they're a dinosaur of a company, you can still only use their web-portal in IE.

I understand all about focus, and if you feel relying on Intuit is the right decision, then that's fine, but that certainly makes me feel a bit uneasy knowing that a major part of your service offerings are built and rely those of another company. Especially since that company may have a conflict of interest in the future.

The pay dates and periods were a minor issue, it's really the lead-time for submitting payroll the bugs me.

My concerns are the same about Google docs and I consider hour logs and payroll crucial data. It's just too easy to make a mistake storing this information in a spreadsheet.

We do rely heavily on freshbooks for our billing and I'd love to see you integrate into freshbooks time tracking for payroll.</comment></comment>


Creating Effective Job Adverts - <AUTHOR> https://blog.honest.work/creating-effective-job-adverts/ <comment><USER2> Salary range is the most strategically left out signal that most job seekers look for.

In most cases it is a waste of time for both sides. But I guess that companies have an advantage in the end by creating this opaqueness.

<comment><USER3> Salary is often negotiated on a person to person basis. One of the first rules of negotiation:

"Whoever mentions a number first, loses"

As the other negotiator knows the expected salary and can undermine it...

<comment><USER4> As others have mentioned, this rule only applies to negotiations where one side is willing to offer more than the other side expects. If you're expecting $150k and the company only has budget for $120k the opportunity cost of engaging in a lengthy interview process to find that out is rather bad.

A better rule of thumb for in-demand professionals (in my experience on both sides of the table) is to do your homework ahead of time, have a realistic understanding of what you're worth and what the company is likely able to offer, and name a price that's a little bit above that. You'll save a lot of time and weed out low-balling recruiters immediately without actually sacrificing much in the way of compensation.

<comment><USER5> The company also wants to attract lots of people who would be happy with (say) $90k, and not anchor anywhere above unless necessary. If you expect a number of people applying, chances are good that someone is willing to be underpaid for whatever reason, and this fact is huge for companies.

</comment></comment></comment></comment><comment><USER6> Does anyone have a contrasting opinion? The article seems to focus on just not explaining with enough specificity what they want, but I find this is extremely common so shouldn't overly effect one particular employer. In fact, I found this job post to be quite well written. The scope is a bit wide (40-120K is quite a range) but I feel like good applicants will know whats expected at different points on that range.

[https://s3.eu- west-2.amazonaws.com/honest.work/posts/effecti...](https://s3.eu- west-2.amazonaws.com/honest.work/posts/effective-job-adverts/bright-cove- advert.png)

Part of me thinks this might be over generalizing from a small sample size, i.e., does honest.work get enough traffic that a lack of applicants is truly indicative of quality. Of course there is also the fact that by pricing by the job advert they naturally encourage overly-broad adverts for broadly similar jobs, so that's probably not helping.

<comment><AUTHOR> This definitely isn't a contrasting opinion as I'm the OP and co-founder of Honest Work but for context, the original role Brightcove posted had over 5k unique hits, a lot of which came from the Go subreddit and a highly relevant Go community here in the UK so they definitely had a substantial number of the right people looking at the post.

<comment><USER6> It's still sort of a sample size of one, though. This content would have been more effective if you had an analysis of some factor that measured your hypothesis (length of posting, salary range width) and showed the correlation with conversions. Then this could have been an illustrative case study backing that up. Even better would be the results of the changes, maybe offer for free to turn this into a junior and senior postings and see the difference multiple posts with narrower ranges makes.

The holy Grail of data science is solid math foundation, benefits your clients (more apps), benefits you (more revenue, higher satisfaction), and demonstrates your effectiveness with data and willingness to continually improve.

</comment></comment></comment><comment><USER7> Honestly, I feel pain when I read most of job adverts/postings. They have a lot of text, they're too verbose, etc.. Why can't recruiters just specify things what developers interested in? These things can be found in different surveys like the Stackoverflow's one.</comment>


Welcome Kevin, Michael, Steve, Dalton, and Andrew - <AUTHOR> http://blog.ycombinator.com/welcome-kevin-michael-steve-dalton-and-andrew <comment><USER2> I imagine many of us will want to bash on Andrew Mason but I've decided to take a contrary position.

I had a negative flinch reaction towards seeing Andrew Mason's name, given the performance of Groupon over the last few years, but then I remembered that he was the one who got that low barrier to entry, commodity business to its previously enviable position in the first place. That kind of experience of ultra-rapid ramp up and sales will surely be valuable to more than a few YC companies.

I don't agree with some of the things he did (ex: preferential exits for himself and other execs) but I wish him the best in his new role.

<comment><USER3> I came to bash him, he should not be flung as an example for young people. His behavior was toxic. That he would get a second chance in a lowly job under tight supervision, and showing good behavior for a few years, yes. But here it's putting someone who cheated accounting to mis-represent the value of his company to investors directly advising decision-makers. He also let his sales people lie to customer businesses.

It's not even about what he does or says anymore (for a while, time will pass), it's about flinging a guy who struck it rich by cheating in front of aspiring entrepreneurs. He could repeat "don't cheat" as much as he wants, his position is the single proof that cheating works, it's more powerful that words.

here is an explanation of the accounting thing: [http://www.forbes.com/sites/petercohan/2012/08/14/memo-to- se...](http://www.forbes.com/sites/petercohan/2012/08/14/memo-to-sec-end- groupons-shady-accounting/)

<comment><USER4> I've known Andrew for years, and I know he's a good guy. If you spent an hour talking to him, you'd be mortified to have said what you just said.

<comment><USER5> > _I_ know _he's a good guy_

That's not an argument. That's what his father would say, or his grandmother.

PS1: What does "good guy" even mean? Most people are "good guys" in that they don't spend their weekends torturing small animals in their attic. Most bad things are done by "good guys" because there isn't enough bad guys to do all the evil that's being done.

PS2: Like many others, I did enjoy / admire his exit letter; but I still have a problem not just with accounting practices at Groupon, but the essence of what Groupon was trying to do.

<comment><USER4> His father would have a bias that I don't.

<comment><USER5> We form an opinion on people in the first 10 seconds we meet them; this opinion rarely ever changes.

There was a study where students were asked to rate their professor right while the professor was entering the classroom for the first time, and then at the end of the semester. Opinions from students hardly changed at all after a semester of class.

It's very difficult for you to know if you're biased. It's impossible for me to know you're not.

<comment><USER6> It's also been shown that we like to be consistent with what we've already said, so perhaps unstated opinions are more subject to change than stated ones.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> Kevin is an investor in Zapier and we've got to say he has been outstandingly awesome. Before committing he came over to our place and hung out with us (and didn't bat an eye when my dog Tuna tried to pull off his socks multiple times, how embarrassing!).

It's wonderful that he shares some of what made Wufoo great (still one of the many startups we look up to). He has a really sharp eye for product and a unique counter-view to the "raise a load of money, it's the only way" sort of thinking the valley sometimes promotes. A great addition for sure.

Congrats to YC, Kevin, Michael, Steve, Dalton, and Andrew.

</comment><comment><USER8> Just curious (and feel free not to answer due to privacy concerns), but does "partner" imply an ownership stake in YC, a la a law firm partner?

<comment><USER4> In our case yes.

</comment></comment><comment><USER9> Awesome. Andrew Mason built an impressive 5,000+ person sales force from 0-60 in a matter of years – something very few people have ever done. I'm hoping he brings this experience to YC so sales-heavy YC companies with similar growing pains can benefit from his experience.

<comment><USER10> Absolutely. YC needs someone with deeper experience in cheating investors and walking away with a pile of cash. Anyone who can sell something as obviously close to a Ponzi scheme as Groupon was and cash out before the music stops will add a new set of skills to the YC team that was previously lacking.

</comment></comment><comment><USER11> This might explain why Andrew is on the team: <https://news.ycombinator.com/item?id=5311724>

</comment><comment><USER12> Harj was very active on Quora, Twitter, and other sites answering YC-related (as well as general startup) questions, so I hope some other YC partners step up and fill that role with him leaving.

On that note, thank you Harj and good luck! And congratulations to the new full and part-time YC partners!

</comment><comment><USER13> Having worked with Kevin for 2 years at SurveyMonkey, he'll make a great addition to YC, congrats Kevin!

</comment><comment><USER14> One of these days YC will launch their late stage fund financed through one or two successful exits and IPO's. That will re-define the VC market as much as YC has changed the world for start-ups and seed capital.

Harj leaving is at least as important as the five new entrants, I'm very curious what he's going to do next. Andrew Mason is a pretty brave pick given the amount of backlash that could create, ethical choices aside he's a smart fellow, there are probably far more people in the start-up scene that would be happy to work with him than there are that would rather stay clear. His farewell note was a class act and sketched the man in a different light for me, but still does not detract from fundamental issues in the groupon model.

YC is making bigger and bigger strides, wished I had a better crystal ball :)

</comment><comment><USER15> pg, was there anything in particular about these 4 individuals' video submissions that gave them an edge over other applicants?

</comment><comment><USER16> An all star cast. Particularly good to see another designer and to have someone with IPO experience.

</comment><comment><USER17> Wow, congratulations on all of this to everyone (Harj leaving is kind of a surprise, he's awesome).

</comment><comment><USER18> Congrats to all, and especially, Michael who rocks!

<comment><USER19> thanks!

</comment></comment><comment><USER20> I wonder how this will impact App.net. Any commentary on that?

<comment><USER21> It doesn't change anything from my perspective, that is the idea behind part- time vs. full-time. The other part-time partners all have fulltime jobs as well.

<comment><USER20> Good. I'd say it should help. Not that you need it, but all counts. (: Good luck with YC. I've always seen you as a stand up guy, after that episode with FB wanting to buy you out and you declining. Takes character to do that. Something rarely seen in SV.

</comment></comment></comment><comment><USER22> Hosted at <https://posthaven.com/>

"Simple, easy blogs for $5 a month, forever."

Interesting.

<comment><USER23> Their blog had been on Posterous. Posterous shut down. One of their partners made Posthaven to replace Posterous and to never shut down. Not a shocking choice.

</comment></comment><comment><USER24> Dalton (and a few other really awesome people) made of my favorite app! In fact its what initiated my transition from "just a phone" to a "smartphone", that app was PicPlz. I look forward to Dalton's vision for YC.

</comment><comment><USER25> Qasar Younis is not getting the recognition he deserves, really sharp guy who will add a lot to future YC companies. Overall, congrats to all involved.

</comment><comment><USER26> Pretty cool. I wonder what the startup pantheon will achieve as it matures.

</comment><comment><USER27> How long til you guys found a new company, that creates y-combinators?

</comment><comment><USER28> Wow, is Steve Huffman the coolest guy ever?

<comment><USER29> He is definitely in the top 10.

</comment></comment><comment><USER30> Many in this thread have failed to consider these facts: Groupon is no longer the same company it was 4 years ago; you might be biased by being anchored to a $20 IPO price, which you should actually blame on the investment "banksters"; thousands of merchants are grateful for the relationship they have with the company; at least one YCombinator company has been purchased by Groupon. I bought Groupon in after hours on this announcement and I am proud to post my trade publicly.

</comment><comment><USER30> Congrats to all! Well done.

</comment><comment><USER31> >and till recently

"and 'til recently" or "and until recently"

<comment><USER32> Incorrect. The word "till" is perfectly fine English.

<http://www.merriam-webster.com/dictionary/till></comment></comment>


Show HN: Nudge Analytics - <AUTHOR> http://giveitanudge.com

<comment><AUTHOR> We have launched a native ad diagnostic that lets you see how a piece of content (typically paid) has performed.

What we do that is different in this is estimate the earned media impressions based on our own network wide dataset. It's a great way to compare the relative performance of a couple of pieces of content. Ideally yours versus a competitors :)

Whack in a few pieces of content to see how it performed - we have a couple of examples there.

Over time we'll roll out more data as we can.</comment>


On a scale of 1-100, what's your Style Influence? - <AUTHOR> http://signup.preshopp.com/ <comment><USER2> This may be a stupid question but what do you mean by "Style Influence"? Googled it and still I couldn't figure it out :(

<comment><AUTHOR> Not a stupid question at all, in fact thank you for taking the time to Google it :)

In 2013 a person's Style Influence will be incredibly important, so start growing your network and building your Influence.

</comment></comment><comment><USER3> Can you explain simply and concisely what your elevator pitch/product is? I don't know what Style Influence is and I can't tell at all from your landing page.

</comment><comment><USER4> I like your simple landing page :)

Klout for style?

<comment><AUTHOR> Just realised you signed up, so you're partly in the know :) It's a very exciting time for us, please help spread the word?

Cheers for the landing page compliment.</comment></comment>


GoodJob – a Postgres-based ActiveJob back end for Ruby on Rails - another-dave https://island94.org/2020/07/introducing-goodjob-1-0 <comment><AUTHOR> I feel like Rails is really starting to see some new life that has felt a bit absent from it for a while now. It almost seems like there is a new generation of people extremely burned out from the constant churn of the "JS-all-the- things" movement that just took over most of web development for the past several years.

There is a lot of exciting stuff landing in Rails and it's surrounding ecosystems at the moment and I think it's still probably the best choice for many new SAAS / web app based companies and startups.

<comment><USER2> Rails was always great, JS just paid a lot more...

It felt great to get back into it having people pay to migrate their Rails apps to React and Node.

<comment><USER3> What's great about Rails ? Serious question from someone who landed into a mature Rails app and is now decidedly running away from that ecosystem after a year working in it.

Rails looks like an early take on MVC that was setup around fast project creation and encourages unmaintainable code - concerns are straight out retarded (a design pattern that encourages breaking encapsulation and makes dependencies impossible to trace) and fat models encourage coupling things that don't need to be coupled.

Then there's Ruby which in my opinion goes against Zen of Python which I'm a fan of - aliases all over the place for same thing, pointless name shortening and stuff to sacrifice readability, rails breaking conventions of standard library...

In the context of when it got popular I understand why it got the hype - replacing the PHP SQL in views garbage and Java 6 verbosity with XML configure everything, and the horrible ASP monstrosity.

These days I'd take Python as a more popular language, that I would say is better designed and has far better Windows support.

NPM/TypeScript and structural static typing is getting really good as well but is not as mature - but code sharing between client and server is a real thing.

Go emerged since the time of Rails hype.

.NET and JVM languages and frameworks got a lot better and scale far better with big code bases.

I don't really see where Rails is great compared to alternatives, it feels like legacy at this point. I'm not saying you can't write good maintainable apps in Rails either - I'm just saying other frameworks and languages will make it easier IMO.

<comment><USER4> > What's great about Rails ? Serious question from someone who landed into a > mature Rails app and is now decidedly running away from that ecosystem after > a year working in it.

Rails provides a mature system to quickly "get the job done" in a way that can be easily maintained and generally "just works". It's especially good if you want to have a _small_ team manage an application. More details below.

> Rails looks like an early take on MVC that was setup around fast project > creation and encourages unmaintainable code - concerns are straight out > retarded (a design pattern that encourages breaking encapsulation and makes > dependencies impossible to trace) and fat models encourage coupling things > that don't need to be coupled.

That hasn't been my experience. Used well, Rails code tends to be relatively easy to follow. It's true that concerns can be easily overused. But that's true of many power tools; I use concerns sparingly, and then they work just fine.

> Then there's Ruby which in my opinion goes against Zen of Python which I'm a > fan of - aliases all over the place for same thing, pointless name > shortening and stuff to sacrifice readability, rails breaking conventions of > standard library...

This is an odd argument. I know both Ruby and Python. No, Ruby doesn't follow Python conventions. That's because Ruby is not Python. The shortening can in some places increase readability; YMMV.

Rails does extend the standard library in a few ways, but if you're using Rails, that's already baked in & you never see it otherwise. So again, that seems irrelevant. If you're using Rails... then you're using Rails.

> These days I'd take Python as a more popular language, that I would say is > better designed and has far better Windows support.

Python & Ruby are similar in many ways, it's hard to argue one is objectively far "better".

I've heard from others that Rails runs great on Windows 10, just use Windows Subsystem for Linux (WSL). I don't know how well Rails runs on native Windows. I know at one time that was important for many people. But for many people today, "run on native Windows" is irrelevant. The web application that I maintain that uses Rails runs on a Linux system, and while it would probably work on any Unix-like system, there's no interest in getting it to run on native Windows. Why do that? There's absolutely no justification for ever running that application on Windows, it would just cost more & crash more. If I want to run it on Windows for debugging, VirtualBox is great. Windows is disappearing completely from many server-side environments. Even Microsoft's own Azure runs Linux more than Windows: [https://www.zdnet.com/article/microsoft-developer-reveals- li...](https://www.zdnet.com/article/microsoft-developer-reveals-linux-is-now- more-used-on-azure-than-windows-server/)

> NPM/TypeScript and structural static typing is getting really good as well > but is not as mature - but code sharing between client and server is a real > thing.

In many situations the sharing is so minimal as to be not worth it. I've only found a _single_ function that would make sense on both client & server, and it was trivially implemented twice. That's not, by itself, a reason to force both sides to be the same language.

In many applications, "not as mature" means "don't use it". I don't have time to waste debugging someone else's framework, especially when I'm implementing a relatively straightforward CRUD application. I'll gladly use an immature system if it provides a vital capability unavailable elsewhere, but for simple CRUD applications that's absurd. I loathe fad-based engineering.

> I don't really see where Rails is great compared to alternatives, it feels > like legacy at this point. I'm not saying you can't write good maintainable > apps in Rails either - I'm just saying other frameworks and languages will > make it easier IMO.

I think the main advantages of Rails are that it is:

\- _mature_ \- a lot of work has been spent to make sure the "special cases" work, and a large amount of functionality is there for immediate use. Like anything it has bugs & missing functions, but in general, if it's a common need it's easily available, easily works, and rarely has bugs.

\- _integrated_ \- while Rails is implemented as a set of libraries, they're conceived as a whole, so there's no effort to "make the parts work together" \- they already work together.

\- simplifies & speeds development by making a lot of convention decisions for you, via "convention over configuration". For example, database tables use snake_case with plural names; model class names use CamelCase and are singular. Yes, Rails automatically does the singular/plural conversion! The conventions eliminates arguing & deciding conventions, doing the work to configure it, and makes the code more regular, but it's more painful if you hate their conventions & want to override them all.

Of course, there are many worthy alternatives. Rails isn't the be-all (what is?). But Rails is still a fine choice for many applications today; GitHub and GitLab are both Rails applications. E.g.: [https://about.gitlab.com/blog/2018/10/29/why-we-use-rails- to...](https://about.gitlab.com/blog/2018/10/29/why-we-use-rails-to-build- gitlab/)

The biggest problem with Rails is that Ruby, like Python, is an extremely slow language. In most applications this is irrelevant; the correct way to speed things up is to cache things, and then it really isn't a problem. Rails comes with a very easy-to-use caching system. So folks, like GitLab, have found that they can identify just the hotspot & reimplement just that _small_ part. The same happens with Python, by the way; if you want fast Python, you either call "Python" code that's actually written in C, or rewrite your hotspot Python code into C. There's no such thing as a free lunch.

In particular, Rails has a large set of conventions. If you're willing to accept those conventions, a lot is done automatically for you. If you will continuously fight the conventions, then it's going to be a pain to work with.

<comment><USER5> >But for many people today, "run on native Windows" is irrelevant.

Developing RoR on Windows is still a pain in the bottom. WSL doesn't work well, WSL2 solves this, but isn't widely available. Not everyone could afford a Mac, and Linux machine just throw off majority of people new to programming. Considering 90%+ people uses Windows as their PC, I dont think it is irrelevant or small issues at all.

<comment><USER4> They might use a Windows PC as their client, but they can easily develop & deploy on something else. VirtualBox is free, as are Ubuntu & Fedora. Deploying Rails apps to systems like Heroku is _worlds_ easier than a typical Windows deploy.

I don't think Linux throws off many people. For many people that is the future. I've written batch scripts & worked around Windows nonsense, and Linux is generally a better experience. Middle schoolers don't seem have problems using Raspberry Pis running Linux.

</comment></comment></comment></comment></comment></comment><comment><USER6> I'm excited about ActiveJob because, finally, people will be able to build background jobs on things that aren't Redis. No hate on Redis - I'm a huge fan of it - but sometimes you have stronger consistency reqs that Redis does not have, and stronger availability reqs that SQS or PG don't have (aka multi- region jobs).

If the concurrency in ActiveJob become as good or better than Sidekiq, we can really get a lot of interesting stuff out there.

<comment><USER7> I used Que [0] in a number of projects in the last few years that had stronger consistency demands. It can plug into ActiveJob. Those projects are still humming along without trouble today.

[0]: [https://github.com/que-rb/que](https://github.com/que-rb/que)

<comment><USER8> Unfortunately its no longer maintained and his 2.0 branch is completely broken for me. I've been very stressed trying to find a que replacement, as almost all of my needs for job need ACID guarantees, and Sidekiq/Redis "best effort" is nowhere near what I need. I also refuse to pay $xxxx for Sidekiq pro just to get those guarantees.

Good job looks pretty awesome, I'll have to give it a shot at some point.

<comment><USER9> Sk pro is really not that much?

</comment></comment></comment></comment><comment> another-dave It would be nice to have one less dependency in the tech stack but honestly have no complaints with Sidekiq so far.

I'm quite new to Ruby/Rails though — would be interesting to hear from others how they think it stacks up (unfortunately the author doesn't compare in the blog post)

<comment><USER10> The standard advice used to be that neither Redis or Postgres are very good for using as a queue.

The creator and former maintainer of Redis was up until a few years ago discouraging its use as a queue, I think mainly because of its lack of durability and high availability at the time. He built a prototype Disque[0] to address the issues but it never became production ready. The other downside is that Redis is in-memory which means the queues have less capacity/are more expensive for the same capacity than an on-disk solution, but as memory gets cheaper over the years this becomes less and less of an issue. The upside is the throughput of Redis is very high.

I have personally worked on rails apps using redis-based queues like resque (and to a lesser extent sidekiq), and actually haven't run into any redis crashes or downtime in years of runtime, redis is very solid in general. You can also snapshot the redis instance periodically, to limit the number of jobs you would lose if it did crash.

In terms of using a primary db like postgres or mysql as a queue, I have personally run into issues with this multiple times. I would recommend never to do it, except on the smallest of side projects.

The issue is that eventually your queues will back up, whether it's due to a bug, surge of traffic, or just complex interaction of behavior in your app that cascades a ton of jobs at once when you run a backfill or something. When your app starts to get overloaded it's pretty trivial to increase the number of web instances running, so your bottleneck in these situations is going to be the db performance. As your queues get backed up, your queue workers are running at full speed processing jobs nonstop, which puts strain on your DB. Additionally, the act of enqueueing and dequeuing a job itself also puts strain on the db, so you can easily get into an unstable situation where each job that gets added to the queue makes every other job take longer.

If you allocate a separate DB instance that is only running your queue, that is much safer. Still, a DB like postgres is not great at doing constant writes and deletes, it creates additional auto-vacuum pressure for instance. But this will manifest as just getting worse throughput on the same hardware than you could get from a dedicated queue like rabbit mq, so if you're not at large scale it's a fine option.

Edit: And one other thing to add, for a lot of web apps the scope of what is needed from a queue these days is a lot less now than it was in the past. It used to be, and in large enterprise systems it often still is, the case that when people talked about a message queue they wanted something to facilitate passing messages between many completely separate apps. Now most apps just use a rest api for that (or perhaps protobufs or graphql or something but still over http). So I think historically an additional reason against using a simple datastore as a queue was that it didn't have enough features so you'd end up re-inventing the wheel with things like brokers, fan out and broadcast patterns, at-most-once vs at-least-once semantics, etc. But here I'm just considering the very limited usecase of a sidekiq-like queue, for processing jobs in the background for a single web app.

tl;dr: Never use your primary DB as a queue. Using a separate Postgres instance can work if you over-provision capacity and don't need to maximize throughput, and a Redis-based solution can work if you don't need high availability and can tolerate some messages lost if something goes wrong.

[0] [https://github.com/antirez/disque](https://github.com/antirez/disque)

<comment><USER11> > Never use your primary DB as a queue. Using a separate Postgres instance can > work if you over-provision capacity and don't need to maximize throughput, > and a Redis-based solution can work if you don't need high availability and > can tolerate some messages lost if something goes wrong.

That is a broad generalization that assumes most applications are operating at mega scale. The benefits of simplified dependencies (a single database instance), transactional guarantees (a single database instance) and persistence (not using Redis) far outweigh the eventual possibility that the queue will place too large a load on your database.

As the author of Oban[0] (an PG backed persistent queue in Elixir) I'm definitely biased. However, the level of adoption in the Elixir community seems to signal that a lot of companies favor simplicity and safety over a possible scale issue down the road. The primary application I work on processes ~500k-1m jobs a day and the queue overhead is virtually invisible.

[0] [https://github.com/sorentwo/oban](https://github.com/sorentwo/oban)

<comment><USER10> You actually bring up another point that in my mind is yet another argument against using your primary DB as a queue, so I'm sure I am also biased from being burned in the past :)

But in a past company I worked at, the company started out thinking sure just throw the queue in the primary db for simplicity. Eventually our slow query logs and db performance monitoring tools were showing that ~40% of the db load was due to the queue inserts and queries. It may be that it was doing something incredibly inefficient and unnecessary in the particular library we were using but we did look into it pretty thoroughly, this was a few years ago though so I don't recall all of the details. And that was at normal operation, then we ran into an issue that basically brought our site down when queues backed up.

At that point it was definitely time to split out the queue, and when we did it we realized that we had implicitly been depending on transactional consistency between the queue and the app data in a few places, which was then extra work to track down and fix these types of issues. This is IMO a code smell as well in general - your data and your infrastructure should ideally not be so tightly coupled.

Managed databases are so easy to set up these days, I would still definitely recommend a separate instance for the queue vs the primary db from day 1 in any new app I build. If you do want to combine them to save money on infra, use a separate logical db and separate connection pool and everything so that it's easy to split out in the future.

<comment><USER12> Agree that separate is more scalable. Still it also distributes the state, and that can be surprising if not planned for.

</comment></comment></comment></comment></comment><comment><USER13> To make it scale well, two things are required:

1\. TRUNCATE if cleanup is needed (no DELETEs) + partitioning.

2\. SELECT .. FOR UPDATE SKIP LOCKED or advisory locks.

It's worth learning from Skype's PgQ developed in 2000-s.

Update: found the work with advisory locks in the code, great.

</comment><comment><USER14> Also see: [https://news.ycombinator.com/item?id=9576864](https://news.ycombinator.com/item?id=9576864) Postgres Job Queues and Failure by MVCC

</comment><comment><USER15> Despite my username, I haven't written any serious Ruby is like 5+ years; that is to say, I'm fairly removed from a lot of the changes in the Rails world.

Now, pardon me for my ignorance, but... isn't this just like Resque back in the day? Or like doesn't ActiveJob just support any ol' backend outta the box? What's the real use case for this? I'm genuinely curious not trying to be rude or diminish the work.

<comment><USER16> GoodJob author here.

The core reason I wrote it was to see how lean/simple I could write a database-backed ActiveJob adapter today. And hope

In practice any adapter should _just work_, but also all the other adapters pre-date ActiveJob (Rails 4.2) and Concurrent::Ruby (adopted in Rails 5.0 I think). The assumption is that building GoodJob on top of what already exists in Rails today, it can be performant (enough) and simple (easy to understand, maintain, keep compatible with new versions of Rails, etc). That's not a big claim, but maybe it's compelling.

<comment><USER15> Awesome! Thank you so much for the reasoning behind it. I haven't had a chance to look at Concurrent::Ruby but that _sounds_ like a pretty good reason all by itself!

As a big plus for a lot of folks who don't need absurd throughput, this makes for one less dependency by using the same db you already have. I'd be sold given the right use-case, thanks for sharing this, mate and _good job!_ ;)

<comment><USER16> Yay!

</comment></comment></comment></comment><comment><USER17> For a very nice library in the Elixir/Phoenix ecosystem have a look at [https://github.com/sorentwo/oban](https://github.com/sorentwo/oban)

<comment><USER18> I think one of my biggest annoyances, from a development perspective, with Sidekiq is that it typically moves you in the direction of running foreman or something like that, which makes using the pry a pain.

One thing I really like about Phoenix development is that I don't have to change my flow to use a REPL while still being able to run background jobs.

</comment></comment><comment><USER19> This looks like a great library on the surface. I would very much like to have a simpler alternative to Sidekiq for projects which doesn't require that level of complexity. Also, not having to add Redis just to run background processing makes life so much easier. I would like to thank the author of the library for all their efforts.

</comment><comment><USER20> I was just assigned to a project based on zend 2.x php framework (2015 version) and angularjs(version1) (not angular.io). Just looking at the code, I just want to scream, because I was told to make changes and add features. I could re-implement everything in Rails in about a week's time but manager's opinions just drown me.

</comment><comment><USER21> This is the exact sort of thing I love about the Rails community. Always innovating to keep things simple.

</comment><comment><USER22> In this post, I noticed comparison with other job libraries, but not Sidekiq, the most popular one. Why?

<comment><USER16> GoodJob author here. The reason is that I don't have any production experience with Sidekiq.

My target right now is someone, like myself, who would typically spin up a new project with delayed_job or que, and provide a better alternative with GoodJob.

Maybe in the future I'll do some more research on understanding why someone might switch from a Sidekiq/Redis to not-Redis.

</comment></comment><comment><USER23> Keeping your jobs in Redis (like Sidekiq does) seems to be a good feature to me. I'm curious how GoodJob compares when it comes to UI and queue management.

</comment><comment><USER24> This looks great. I'd love to remove Redis from my tech stack on applications with low throughput background jobs. Sadly, it doesn't appear to have a UI.

Resque and Sidekiq are both good enough solutions, but what makes them my defacto choice is the robust web UIs that comes with them.

</comment><comment><USER4> This looks very promising! I'm currently using active job, so I will need to take a look at this. Thanks for letting me know.

</comment><comment><USER25> This is awesome, thanks for sharing! Sidekiq works well for me but it's good to have an option that doesn't require Redis.

</comment><comment><USER26> Hm. Got falsely excited. No listen/notify etc? This really is just delayed_job

<comment><USER16> GoodJob author here. I'm curious why you're specifically looking for listen/notify? I'm trying to prioritize that in the backlog.

<comment><USER27> Hi, sorry to hijack this thread, but since you mentioned you're the author, I just wanted to ask you something. Do you have any plans to use a different locking mechanism other than the advisory lock? I'm asking because eventually every postgres-backed app that reaches a certain scale, requires a connection pooler (pgbouncer being the de-facto standard nowadays). We usually run pgbouncer in transaction mode, which doesn't support advisory locks. Any thoughts?

<comment><USER16> That was mentioned in another thread. I have two thoughts about it:

\- Going off of my own experience, I've never used PgBouncer and I've worked on some moderately sized applications (1s of millions of users). So I could say "not in scope for the target use-case".

\- It would be possible to add an argument option to use a transaction-based Advisory Lock instead. I can imagine that would have downsides (long-running transactions, yikes).

I'm imagining that the lifecycle of GoodJob is that people starting new projects might choose it, and then the feature set would grow with their needs.

</comment></comment></comment></comment><comment><USER28> Would this work with pgbouncer? I'm wondering because of the advisory locks.

<comment><USER16> GoodJob author here. Excellent observation.

It should work if PgBouncer is using session-level pooling, but not transaction-level pooling. I made a ticket to document this: [https://github.com/bensheldon/good_job/issues/52](https://github.com/bensheldon/good_job/issues/52)

I don't have much experience with PgBouncer. What's the scale/need when a team would use PgBouncer? That would help me prioritize supporting it better.

<comment><USER8> The main reason I can think of would be when you have hundreds of workers accessing a DB but the DB only has a small connection limit, say 200 or something. You can have the workers access the connection through pgBouncer to bring that up into the thousands.

<comment><USER16> Thanks! That's really helpful context.

</comment></comment></comment></comment><comment><USER29> i'm not sure why i should use this instead of delayed job for instance

<comment><USER30> If you're using delayedjob, I strongly suggest you check out the Performance table in Sidekiq's readme. 465 seconds for DelayedJob against 14 seconds for Sidekiq for the same workload.

It's 7100 jobs/sec x 215 jobs/sec.

I wonder how GoodJob will compare.

[https://github.com/mperham/sidekiq#performance](https://github.com/mperham/sidekiq#performance)

<comment><USER31> That benchmark is just showing job overhead. That Sidekiq can do nothing 7100 times a second only matters when overhead is significant, i.e. when your jobs do almost nothing.</comment></comment></comment>


Cube: time series visualization (MongoDB + Node + D3) - <AUTHOR> http://square.github.com/cube/

<comment><USER2> This kind of rekindles an interest I had in building a web app that was at its core a data vacuum.

You sent it whatever data you like, be that timestamps, integers, floats, arbitrary strings, etc... and it would let you build an intelligent dashboard displaying the data however you like.

<comment><USER3> Sounds like GoodData (<http://www.gooddata.com/>), which is great! Competition drives innovation.

</comment></comment><comment><USER4> We're currently using Graphite ( <http://techblog.appnexus.com/2011/metrics/> ) and loving it, but using MongoDB is an interesting spin.

I'd love to see some performance benchmarks and see how many datapoints you can pump in and out of Cube. We've been collecting about 1 million metrics every minute on a speedy dual proc / quad core box backed with SSDs for storage.

</comment><comment><USER5> This looks fantastic. From the short demo, nice UI for setting up charts, and the visualizations produced are quite nice as well.

The only downside to something like this (and Graphite, etc.) is that you can only visualize data that you store in Cube (if I understand correctly). That's great for things you sample yourself, but what if you want to compare something you sampled yourself with something from New Relic or another data source? Cube looks to go a step further than Graphite in allowing for easier custom visualizations, but data is still a one-size-fits-all thing. You can write an emitter/parser that grabs the data and stores it in Cube, but there's a huge amount of duplication (and essentially polling then).

I think a better architecture is to separate data storage/acquisition from visualization. Establish a set of standards for data interfaces that specify the output they provide, and build your visualization on top of that. Then, your interfaces can be API wrappers, mysql connections, a mongo-db based storage system, a redis-backed storage system, etc.

<comment><USER6> True, but collecting the data into a single store makes it easier to analyze. If you want to construct visualizations (or perform analysis) with data from different back-ends, you can do that using Cube's API. I do like your idea of establishing standards for data interfaces, but I wonder how hard it would be to put into practice.

<comment><USER7> Doesnt that not simply already exist with RRDTool?

</comment></comment></comment><comment><USER8> We made a tool for annotating time series -- it is intended for keeping remarks with explanations of influencing factors. Not that sophisticated but does what it does - <http://explainum.com>

</comment><comment><USER9> Looks promising. I need to put together a dashboard, and this might be how I do it.

As an aside, it's always great to see more usage of PEG.js.

</comment><comment><USER10> I love this spirit of sharing and collaboration spurred by git/github. One nice side effect of adopting these open source libs (and not re-invent the wheel in house) is that we'll all have transferrable skills between companies in this space. I like this positive trend.

</comment><comment><USER11> This is very, very cool. I've wanted something like this for a long time.

Running it, I'm having some issues loading pages that I think has something to do with caching static assets. Why not use an existing static file library?

<comment><USER11> To answer my own question, it's because the author is doing asset bundling.

</comment></comment><comment><USER12> This is almost exactly like something that I've been wanting to build!

</comment><comment><USER7> It would be great to support something that shows the variation in the data for this stuff (average/median for response times is a notoriously bad idea).

Other than that, love it!

</comment><comment><USER13> Impressive. Suspect we'll see folks rolling their own Chartbeats.

</comment><comment><USER14> I wish they would have used ProtoVis with DataMarket's IE Shim

[http://blog.datamarket.com/2011/06/22/protovis-support- inter...](http://blog.datamarket.com/2011/06/22/protovis-support-internet- explorer-8/)

Yes Protovis is no longer maintained, but here is a good post defending it: <http://blog.datamarket.com/2011/09/12/the-protovis-stays/>

And used Socket.IO for comm (they might)

However, it is lovely, I am going try it out :)

<comment><USER15> The guy who wrote it _is_ the author of d3, which might explain why they used d3. Plus, d3 is awesome.

<comment><USER16> Mike Bostock (creator of Protovis and D3) now works at Square, which helps explain this release from them. :)

</comment></comment></comment><comment><USER17> Love It! I'm seeing it in my stack, connected in the cloud where apps send their data to the DB via rabbitMQ, to make monitors...

</comment><comment><USER18> Believe it or not, I am actually developing exactly the same thing. I guess I'll use Cube then !

</comment><comment><USER19> Did somebody say Time Cube?</comment>