A Case Against Cucumber - <AUTHOR> http://blog.8thlight.com/kevin-liddle/2013/09/18/a-case-against-cucumber.html <comment><USER2> I've been working in Rails since 1.2, and the vast majority of developers I've met have been of the opinion that cucumber sucks. There's a _really_ vocal minority on the internet though, so you'd get the impression that it's pretty popular.

That said, almost every rails project I've worked on has had the cucumber gem included at some point, just not actively used.

IMHO the _vocal_ Rails community puts too much emphasis on testing. I write tests for my code, but not as many as some would like. The RSpec book in particular is a gigantic load of crap. And I say this as a fan of comprehensive tests! The ridiculous lengths that book goes through in testing that codebreaker game are so detached from reality it makes my head hurt.

<comment><USER3> Do you have one resource (book/guide/blogpost) that you think gives a good explanation of how people should be thinking about testing?

<comment><USER2> No, but I should write one. Broadly speaking, I think test coverage in the real world, as practiced by effective programmers, is _variable_. One must use judgement to decide the appropriate level of test coverage. There are so many factors in making that judgement however that it's hard to distill.

The primary heuristic I look for is "Do I have enough tests that I can be reasonably sure these tests will fail if I break the code". That is the fundamental question! This is different than the RSpec books' approach which is more dogmatic and highly structured (you need to test every function, every entry point, pretty much every single thing). The thing is, the RSpec book approach does work! It works better in fact! It also is a colossal waste of time. I'd write tests like that if I was writing a trading platform for the NYSE, but almost no one does that sort of work.

Since I'm in the midst of writing a book (shameless plug [http://exploringelasticsearch.com](http://exploringelasticsearch.com) ) I don't have time to writ e that post unfortunately.

<comment><USER4> Your post makes me feel like you're missing the point of BDD.

The point of BDD isn't to "stop breaking the code". I think "coders" value "code" too much.

The point of BDD is to build understanding of the problem you're solving by enumerating behavioral ambiguities/edge cases up front, before you've invested time and emotion into a particular code approach.

You're using testable _behavior_ to drive the development of your code. More simply - test first, code second. Having a regression suite is a cool side effect.

Of course, this is just how it's "supposed to work." In real life YMMV. Whether or not you think there's much value in that methodology is another story - but if you are going to dismiss something, you have to dismiss it based on a relevant metric.

<comment><USER5> Yes, you have just regurgitated the rhetoric. But there are a lot of problems which the enumeration you describe is simply the wrong way to build understanding.

See [http://ravimohan.blogspot.com/2007/04/learning-from- sudoku-s...](http://ravimohan.blogspot.com/2007/04/learning-from-sudoku- solvers.html) for an interesting example.

</comment></comment></comment></comment></comment><comment><USER6> My impression (as a static typing bigot) is that Cucumber style testing is like the crap shoot of dynamic testing (what should I type? What are valid commands? What params do they accept?) made an order of magnitude worse by using regular expressions.

...who thought this was a good idea? (Sorry, being harsh.)

That said, I think the _idea_ is conceptually fine, and, as usual, I think could actually be fairly pleasant with the right (static) tooling applied, e.g. ThoughtWorks's Twist:

[http://www.thoughtworks.com/products/twist-agile- testing/](http://www.thoughtworks.com/products/twist-agile-testing/)

Which is a GUI that provides all the nicities of "what commands are allowed", "what params do they take", "oh, refactor this command name from abc to xyz", etc.

I think PM/testing types would love this sort of setup. Granted, it would still take dev investment to get the fixtures/commands/etc. setup. But if you have a huge line-of-business app that will drive a business for the next 5-10 years, I think that's a good investment.

Disclaimer: I've never actually used Twist because it's a commercial product. Yes, I know I suck.

<comment><USER7> Twist last time I used it appeared to be much more inflexible to cukes. You still had to delve into Java to make it do anything useful so it still has the problems that cukes has.

</comment></comment><comment><USER8> Cucumber has been a great success for our team. We've trained the BA's how to write cucumber scripts, which in turn has done a lot to train the BA's in how to think about the stories and features that they are requesting. The dev will sit down with this feature definition and the BA's, and the feature is groomed into a viable and feasible feature that is testable. We strap these to a web automation (we use specflow to watin), and get a fairly significant return on investment.

Cucumber is like any tool; if used correctly, it's great, and if it's used as a de facto tool, then you're going to struggle with it.

</comment><comment><USER9> For the most part, in my experience Cucumber isn't worth the effort (one place called it "Encumber"). If you have product managers or BAs who can read code, then it's easier to capture the requirements with more developer-friendly testing tools. And if the product managers or BAs do not read code, Cucumber doesn't usually help much.

Having said that, I did work a gig where the tester would sit with the subject matter experts, and as they would talk, he would capture what they were saying as Cucumber tests, and he would then echo the tests back to them to see if they were right. Afterwards, he would then add whatever other tests he needed (checking weird edge cases and such), and code up any additional fixtures he needed. Then it was just a matter of me getting all the tests to pass. It was a really nice way to work - I knew unambiguously when I was done with something. (It helped that the tester sat right next to me.)

</comment><comment><USER10> I'm a huge fan of the gherkin format, it let's me sketch out a feature and then work through it methodically, testing as I go. But cucumber has a critical flaw, it forces you to think about code while you're writing. Because step definitions are global you can't just write any old thing, you might fire off a step you wrote previously. This is only a side issue though, compared to the biggest problem, step reuse.

        Given a person "fred" exists
        And a person "ethel" exists
        And a fatherhood exists with parent: person "fred", child: person "ethel"

Did someone mention cucumber steps are written in english? What happened to:

        Given Ethel has a father called Fred

I moved over to Spinach[1] about 18 months ago and I've been much happier. Steps are specific to a feature so I can go to town with shakespearean english. Reuse isn't an issue either, it just happens in the step definitions, not in the gherkin files. Using more natural english tends to lead to fewer larger steps anyway, so it feels a lot less like call-by-regex.

[1] [https://github.com/codegram/spinach](https://github.com/codegram/spinach)

<comment><USER11> Shakespearean English did you say? Something like this?

      Borne o'er the cruel firmament become Ethel
      That Fred hath usurp'st from her the silence
      When lo!  Upon them should harrow a fatherhood
      Of parent, Fred besieged by the burden
      And his daughter the indifferent imposition of child

Sounds great! I'd much rather use that than Cucumber.

</comment></comment><comment><USER12> TL;DR stop bitching about an acceptance testing tool when you're not using for acceptance testing.

The problem isn't with Cucumber. The problem is your cukes suck. This is probably also a problem in your Rspec/Minitest/whatever tests. If you're using cucumber the way 90% of the cucumber tests I've ever seen (indeed, a great many i've written myself) are written, then you're writing integration tests, and probably crappy ones at that.

      Given a user
      And I go to some really cool page
      And I click on some button

is NOT a domain test. Domains, for the most part, don't have 'users' (unless maybe your domain is drug addiction). Or 'pages' (unless you're in publishing). And they're definitely not about 'buttons' (unless you're a tailor.) [Yeah, you can probably find some domains where those are actual concepts. They're probably not all concepts in the same domain.]

Write actual domain tests for your acceptance tests. Do this regardless of your framework—there's absolutely nothing preventing you from doing it rspec or anything else. Fuck step reuse (gherkin alternatives like spinach are great, specifically because the steps are isolated to the test.)

If you don't want to write acceptance tests, and you feel comfortable with that, no biggie. Lots of people don't. But don't create a straw man out of Cucumber just because you don't understand layers of testing.

This goes for Cucumber fanboys as well. Don't push cucumber into layers it doesn't belong. Its a _domain_ testing tool. Not an integration test. definitely not a unit test. Don't push it on a team that doesn't want it. You can write the same kinds of tests in rspec. Disclaimer: I like cucumber. I am not using it in my current job because it wasn't a good fit for the company. I will probably use it again one day. I still write AT's.

This:

      Given I hate cucumber
      And I post a scree against it
      Then the haters will rejoice

is pretty much the same as this:

      describe "hating on cucumber" do
        it "produces a response" do
          given_i_hate_cucumber
          when_i_post_a_scree
          then_haters_rejoice
        end
        def given_i_hate_cucumber
        end
        #....
      end

<comment><USER13> Arguments based on "X doesn't suck, 90% of people are just doing it wrong" aren't convincing to me. If nearly everyone is using a tool incorrectly, the problem is still with the tool. Maybe the tool doesn't suck, but obviously the documentation and education around it does.

<comment><USER14> > Arguments based on "X doesn't suck, 90% of people are just doing it wrong" > aren't convincing to me.

The argument isn't som much that Cucumber doesn't suck, but people are using it wrong, but that Cucumber _does_ suck for what people are complaining about it sucking for, but that Cucumber isn't designed or promoted for that use.

Its "yes, hammers suck for connecting things with machine screws, but that's not what they're for", not "hammers don't really suck for connecting things with machine screws, its just most people are holding the hammer wrong when they try to do that".

<comment><USER13> I kind of addressed that issue. If most people have the idea that your hammer is actually a screwdriver, it isn't their fault for misusing it. Someone put that idea there, and you as the creator of this not-a-screwdriver are responsible for the failure of that message.

<comment><USER14> > If most people have the idea that your hammer is actually a screwdriver, it > isn't their fault for misusing it.

Its not my hammer (or that of most of the people, I would imagine, posting in a similar vein), and I'm not interested in fault.

> Someone put that idea there

Yes, people have put out bad ideas of what Cucumber is for -- and the people complaining about it not being good for things it isn't designed for _are among those people_.

That's why some of us are countering those incorrect ideas, so they don't keep spreading.

> you as the creator of this not-a-screwdriver are responsible for the failure > of that message.

Wrong ideas that only some subset of the people exposed to a product get about its use are not solely the responsibility of the creator of the product. Obviously, if you have an interest in _selling_ the product, you are the person that stands to gain or lose based on those wrong ideas and you have a particular _interest_ in correcting it and a _responsibility to your business_ to take any _efficient_ steps to combat that misimpression, but then again, if you are using tools in your business, you likewise have a particular interest in _identifying and correcting your own mistaken ideas about tools_ and a _responsibility to your business_ in taking any efficient steps to correct misimpressions you may have.

</comment></comment></comment></comment></comment><comment><USER15> I've been using Cucumber for almost 3 years now, but if I were to start a new project today I'd pick Capybara with MiniTest or RSpec.

When I first started using Cucumber I thought it was cool that "anyone could read the test and understand it"

But that one advantage doesn't really matter that much when everybody in your project knows Ruby and doesn't need the natural language side of it.

<comment><USER14> > But that one advantage doesn't really matter that much when everybody in > your project knows Ruby

Well, yeah, if people who aren't Ruby coders aren't reading your tests, Cucumber is overboard: the motivating use case for Cucumber is acceptance tests where the main part of the test (the part that isn't implemented in Ruby code) is owned by and validated by the customer/user, who presumably isn't generally, except by coincidence, a Ruby coder.

It might also be useful for integration tests where that need to be validated by people familiar with the overall system design/architecture, but not necessarily the language that any particular components are implemented in, for the same reason.

</comment></comment><comment><USER16> I think you're perhaps missing the point in one or two places.

Cucumber helps techies (devs, testers, etc.) and non-techies (product owners, scrum masters etc.) work collaboratively. I don't think the authors of Cucumber ever said that it enables non-technical people to 'read and understand the underlying code'.

Ultimately though, I agree with your conclusion but for slightly different reasons. I use both RSpec and Cucumber daily; they are both awesome and provide a similar end result. I enjoy writing Cucumber features but in certain circumstances it doesn't scale for large/complex applications (in my experience). I think that this isn't a problem for many people though, unless they're doing something wrong 'under the hood'.

</comment><comment><USER17> I couldn't agree more. The pipe dream is that you have BAs or other non- technical team members write the Gherkin specs and that saves the developers time because "the tests practically write themselves!" (Assuming you're maintaing your giant library of awkward cucumber regex matchers...). In reality it takes much longer to sit down with someone and teach them how to write in Gherkin, all for the goal of having a human readable spec that doubles as test steps. It's much better if they write the specs however they want, in normal english, then you can translate those into normal rspec or test unit tests.

It's a laudable goal, but the abstraction is so leaky in reality it just ends up creating more work for everyone.

</comment><comment><USER18> Cucumber, gherkins, capybara, handlebars, sinatra... do these goofy names bother anyone else? I'm writing code, not hosting a children's show.

<comment><USER19> The fact that you could list them all off proves that they are memorable.

<comment><USER18> I think it more proves that HN users drone on about them.

</comment></comment></comment><comment><USER20> It completely depends how you use it. There's so much misuse of it out there I can understand why people hate it so much. But done right it's an extremely valuable collaboration tool.

I'd recommend the Cucumber Book if you want to read about how to use it well:

[http://pragprog.com/book/hwcuc/the-cucumber- book](http://pragprog.com/book/hwcuc/the-cucumber-book)

I've also blogged a lot about how to use Cucumber well here:

[http://chrismdp.com/tag/cucumber](http://chrismdp.com/tag/cucumber)

<comment><USER21> Yeah. Just about everyone I've run into using Gherkin-based testing seems to think it's for step reuse, or for covering code, or for integration testing.

Those are different kinds of tests. Go write them in a _unit or a_ spec or whatever you feel like. Code coverage - covering your ass during refactors - and integration - covering your ass before deployments - are just for you and can skip the regexp.

Gherkin is for collaborating with my customer and making them tell me what the fuck they actually want. 90% of the value happens before any of the steps are ever implemented. A few user interactions will be covered, likely not even all of the possible ones. Little step reuse as focus is on readability.

It's a really specific problem, and so far I haven't seen a superior option aside from Gherkin.

</comment></comment><comment><USER22> While there is initial overhead in hooking in the tests, the power comes when you have modularized test steps that you can reuse for multiple tests.

At my last company we had QA write all the cucumber tests and someone would hook in the new statements, which is the overhead you suggest. Now if you write 3 different gherkin statements that do the same thing, then that is not optimal.

Like many problems, I don't think the tool is at fault. It does what it claims. Provides human readable syntax for test cases and lets you hook that in however you want.

I think it's a valid point to think that extra layer is unnecessary, but I wouldn't go as far as discounting it and saying that no company has derived value from it.

Like all code, the messes usually stem from how the code is implemented, not the language itself.

</comment><comment><USER23> Cucumber was most developers first exposure to capybara (IIRC, capybara was developed by the cucumber folks).

Since capybara is so awesome, it gave a lot of people a nice impression of cucumber.

But unless you actually have a customer in the loop writing cucumber tests, just use the capybara directly.

</comment><comment><USER24> Not a fan of cucumber. I started it, but the extra layer of abstraction was too much complexity for too little benefit. Instead, I think you can make almost as easy to read and understand acceptance tests with plain old rspec.

I just TDD'ly implemented a "user signs up for Direct Deposit" feature, and the acceptance tests look like this:

      describe 'Direct Deposit' do
        before do
          @workflow = DirectDepositWorkflow.new(create(:user))
        end

        describe 'User sets up direct deposit with correct info.', :js do
          it 'user sees confirmation page - individual' do
            @workflow.sign_in_and_visit_direct_deposit_form
            @workflow.submit_correct_form

            expect(@workflow).to have_confirmation_message
          end

          ...
        end
      end

Then you can make the workflow spec helper class:

      class DirectDepositWorkflow < Struct.new(:user)
        include Capybara::DSL
        include UtilitiesHelper
        include Rails.application.routes.url_helpers

        def sign_in_and_visit_direct_deposit_form
          ...
        end

        def submit_correct_form
          fill_in 'name', with: 'John'
          click_button 'Submit'
        end

        def has_confirmation_message?
          page.has_css? '.success_alert', I18n.t(...)
        end
      end

To me, this is the perfect blend of easy to read, high level specs (serving the rule of cucumber/gherkin), with easy to understand, rspec syntax and ordinary method definitions.

<comment><USER12> Interesting technique—I'm still a fan of Gherkin in concept (though I don't use it right now) but I do the same kind of testing in Rspec. My variation isn't quite as refined as your workflow though; i'm just defining methods in the describe block. It has (what i perceive as) an advantage in that the scenarios don't hit any instance variables so they're a bit more readable, but i like the way you're doing that.

</comment></comment><comment><USER25> Cucumber is part of our workflow and, although I don't have a control for this comparison, I think it makes us more efficient. Our product owners write in Gherkin (non-technical != illiterate btw) and can essentially throw the feature files over the wall and any dev can pick it up and get to work. Now, there is conversation around every feature but sometimes only at estimation time.

Our POs also use git (non-technical != vcs illiterate) so we have a history of who wrote/modified what spec when.

Also, we have an _additional_ simple layer of abstraction between feature steps and the actual tests. Most of our step definitions are three line methods that make calls to a ui driver that knows the details of our ui; or to a "given" driver that knows the details of our database/models. This ensures that there is only one place where anyone will implement, say, a user logging in.

      Given(/^I am a logged in user$/) do
        given.a_logged_in_user
      end

while elsewhere there might be

      Given(/^I have signed in$/) do
         given.a_logged_in_user
      end

If the log in process changes, we just edit the given driver.

We generally only write golden path features. Edge cases are handled with lower level testing.

Our feature files stick to specifying business value rather than ui implementation. For example, we don't do this:

      Given I visit the homepage
      When I enter "children's bikes" in the search field
      And I hit the submit button
      Then I am on the results page
      And I see a fieldset with a legend "Children's Bikes"
      And I see a table containing "Huffy Sprite"

We would write it like:

      Given I search for a specific product category
      Then I see the search term I used
      And the results of the search

When a dev needs to work on an app they haven't touched in 6 months, it's nice for them to read feature files like this to get up to speed on what the app is actually supposed to do. Then can then dig into the ui and given drivers to see how it does it.

There is additional infrastructure and PO training involved, but overall, I'd say cucumber improves clarity and communication on our team.

</comment><comment><USER26> "When I write a Cucumber feature, I have to write the Gherkin that describes the acceptance criteria, and the Ruby code that implements the step definitions."

As someone who recently started using cucumber, this was definitely a frustration at first. But after a couple weeks of writing step definitions, and focusing on making them reusable (using some of the tips here: [http://coryschires.com/ten-tips-for-writing-better- cucumber-...](http://coryschires.com/ten-tips-for-writing-better-cucumber- steps) ), I ended up with a pretty good bank of general steps that I could cobble together into new features without much modification. The result was something much more readable and reusable than if I wasn't using Gherkin on top of the step definitions.

<comment><USER20> Be very careful of over reuse of step definitions, and of Transforms. You'll end up with code that's very difficult to understand as your step codebase grows.

I now tend towards minimal step re-use, lots of one or two-line steps calling into nice clean plain ruby classes which actually do the work of my steps.

I've blogged a lot about how to use cucumber well here: [http://chrismdp.com/tag/cucumber](http://chrismdp.com/tag/cucumber)

</comment></comment><comment><USER27> I've posted a response here: [http://mattvanhorn.com/2013/09/19/a-case-for- cucumber/](http://mattvanhorn.com/2013/09/19/a-case-for-cucumber/)

<comment><USER28> Your case, while very valid, doesn't address the original author's concerns related to implementing the tests. In that case I feel as if it's equivalent to using GOTO in my tests. But worse, my labels are regex.

We use cucumber extensively, our cucumber suite is not really that large, and with each new feature it becomes harder to maintain. There has to be some middle ground out there.

<comment><USER27> I don't find implementing the tests very difficult at all - but I am comfortable with regular expressions, and ruby.

It's no more burden to write step definitions than it is to write RSpec directly. I use SOLID OOP to write my tests, most of the logic lives in regular old methods, and my steps look like:

      Then 'the current subscription payment date should be tomorrow' do
        verify_next_payment_date(Date.today + 1)
      end

Since I start with the Gherkin file, I type less than 10 keystrokes to make the step.

</comment></comment></comment><comment><USER29> The underlying assumption of this article is that it's easier to read code than it is to read english.

Well even for a developer of 15 years even the most obtuse English can be more easily absorbed than the cleanest code.

The speed of parsing and understanding code is directly related to the amount of time spent writing it and how recently it was read.

English can be read and understood at the same speed every time.

There is also a non-linear relationship between your feature file size and the underlying code they represent, especially if they represent integration testing. A single feature could test code from 10 different files.

For a single developer in an codebase they know and understand well and have recently worked on, Cucumber can act as friction on development.

For a single developer who hasn't worked on a codebase before, or who is returning to a codebase after some time Cuke steps will definitely improve their ability to understand what the code is doing and by providing a regression test suite improve the quality and probably speed of their output.

Code is a language of instruction - not communication, documentation can help, but it is easily forgotten and fails at communicating integration.

</comment><comment><USER30> Assuming you're focused on writing reusable steps and building up a solid test architecture I don't see cucumber costing you time. You also really shouldn't need to go to code that often to debug a test failure. Write better test frameworks, write tests for your tests, and self report issues in your test framework when they come up. E.g. unable to create test fixture -> this->skipped("reason");

In my experience the ability to just plug and play steps and quickly add additional coverage for edge cases is a huge bonus for test automation.

But again you have to actually engineer things and write reusable steps and intelligent enough test frameworks that you don't need to often dig through the test code.

I think this issue is somewhat similar to how when writing unit tests you'll want to build up a library of reusable custom asserts targeting your domain so you can quickly exercise your sut and validate things. It's definitely an upfront costs to build things up like that but in the long run it pays off very nicely when you avoid taking on too much test debt.

<comment><USER31> So what exactly does writing tests for your tests look like...? Keeping this in a Cucumber realm, of course.

<comment><USER30> Probably a tricker task for cucumber than most frameworks. At the end of the day though I expect that you'll want to have some classes that manage fixtures, perform setup functions, interact with the sut etc and the cucumber regex matches just call into those classes to manipulate the sut and validate it.

So tests here are just cucumber or rspec tests that verify the underlying frameworks that cucumber drives. So for example if you have partial object equivalency methods that verify that only specified fields on two objects match you would write tests to verify they are correctly matching or failing to match two objects as expected before you use that in your cucumber tests.

</comment></comment></comment><comment><USER32> Disclaimer: I'm not for or against the use of cucumber.

The second claim against cucumber / gherkin is not it's fault, it's the toolset's fault. Using grep to find your step definition is not very effective.

If you use a tool like Rubymine, you can jump directly to the applicable step definition through one key combination.

<comment><USER33> You should read [http://catb.org/esr/writings/unix-koans/gui- programmer.html](http://catb.org/esr/writings/unix-koans/gui-programmer.html)

In practice, you'll find grep is in the toolkit of far more Ruby programmers than the IDE Rubymine. And with good reason!

The real problem is step definitions with regex in them. You don't need to do this. Steak was written to avoid this problem: [https://github.com/cavalle/steak](https://github.com/cavalle/steak)

But the underlying issue described in this post is still valid - Cucumber is an unnecessary layer because it adds no value that you don't get with Capybara and your favourite testing framework.

<comment><USER32> Thank you for the heads up about ruby devs liking *nix tools and not ide's.

The fact of the matter is grep is a poor tool to try to find your step definitions. There are tools out there that are much better for such things, vim with [http://www.vim.org/scripts/script.php?script_id=2973](http://www.vim.org/scripts/script.php?script_id=2973), or an ide or <your favorite tool>.

You could make the claim that any abstraction is unnecessary if you see no value in it. It doesn't mean others will not find value in it.

Personally, I find it quite easy to understand and convey to non-devs the stories told through gherkin. That is part of the value prop for me.

If no one is reading them but the devs, I'd rather use something other than gherkin.

</comment></comment></comment><comment><USER34> > Non-technical people don't read code, no matter how easy it is to read.

There is something deep and surprising about this fact.

</comment><comment><USER35> I love cucumber. When you join the development of a project it's a great place to start and learn what the project is about.

The whole 'customer writes features' thing never worked for me either, but just having it as automated specifications for my projects makes it very valuable for me.

<comment><USER15> Couldn't you do the same with RSpec or straight Capybara code?

I no longer see the added value of Cucumber when you work in a project that has no non-technical team members writing tests.

<comment><USER21> If I go to my stakeholders and customers with _Spec or_ Unit code, and ask them: "Does this test represent your idea of how this feature should work" I'll get no useful answers.

I can read it to them, or we could collaborate on a user story that doesn't get directly mapped to tests, but then we have room for the interpretation/translation process to create the wrong assumptions in the test code.

There are plenty of tests I'd never show the customer because they involve implementation details, and those aren't going to be written in Gherkin because I don't hate myself enough to juggle that many regexps.

</comment></comment></comment><comment><USER36> I drank the Cucumber Kool-Aid and had an app with basically 100% Cucumber coverage. If I wanted to change something, I would first write a cuke for it, write specs as necessary, and only as necessary write actual code.

Of course I got into the same spot that almost everyone gets into doing that: too many "and then I click..." scenarios. Great coverage, but then the requirements change substantially and the test base is just a massive snarl of interdependent assumptions. At one point I had effectively written, in cucumber, a natural language interface to my app.

But then at some point I needed to _move_ and I needed to move _fast_ and I just let the whole infrastructure rot. Now I'm doing almost everything on the client side anyway, so it's all a little moot.

All of that being said, I'm still a Cucumber fan, but I approach it differently. Here's what I think:

 _You should have cukes for the 5-10 things that if they don 't work make your product completely pointless._

On Gmail, there would be cukes like:

        When I receive the email from Fred about dinner
        And I open it
        Then I should see the message
        When I reply
        Then Fred should get my reply

And those cukes basically don't need to have wildcards:

        When /^I reply$/ do
          click "reply_button"
          fill_in "message", :with => "I'll be there!"
          click "Send"
        end

Essentially:

* Use cucumber as a way to force you to keep track of the handful of absolutely critical integrated experiences

* Don't use fancy grammars... Think of steps as human-readable function names.

* Don't do anything fancy in the steps. They should just be a list of actions that users would have to do.

And yeah, there's no reason you couldn't do all of this in Steak. I think Cucumber is just nice in that it encourages you to write it out in human terms.

<comment><USER37> What's the major benefit of having those 5 or 10 things be cucumber/gherkin, instead of straight rspec? (or straight minitest!)

What are the costs of having to understand and maintain code written using a seperate little technology stack (cucumber) for just 5 or 10 scenarios?

Okay, you sort of answer that: " I think Cucumber is just nice in that it encourages you to write it out in human terms."

Okay. If it's worth the cost to you to do this. You could also, of course, just write things out in human terms in comments above the 5 or 10 tests written in rspec or whatever.

</comment></comment><comment><USER38> Towards the end, he says "...but I do enjoy the Gherkin syntax. Not for testing, but for gathering feature requirements..." and this is where cucumber or fit or FitNesse is useful - you gather requirements, then because these requirements can be run as tests you start running these requirements as tests and now you have a living document of your product and product owners can read the tests (which are the requirements). If you look at such tools as mere test runners, then they do add an extra layer of complexity, but these tools were not meant to be just test runners... so you are using it for the wrong thing... :)

</comment><comment><USER39> We started using Cucumber 6 months ago and I hate it. My biggest issue is that it's unreliable. 90% of the test failures work if you do it exact same steps manually.

It seems like it always times out waiting for something on the page to load.

<comment><USER10> This is probably a driver issue, if you're not using it try Poltergeist[1], a headless webkit driver. Because it's webkit it basically behaves the same way as if you were testing it yourself in chrome.

[1] [https://github.com/jonleighton/poltergeist](https://github.com/jonleighton/poltergeist)

</comment></comment><comment><USER40> Damnit, I was hoping for an article about the health drawbacks of cucumber (I'm boring that way), and it's about another Framework.

It'd be awesome if people started tagging these as [technical] or [computers] or [vegetable-afficionados].

Rant out.

</comment><comment><USER41> "I don't prefer to use Cucumber for any of my testing, but I do enjoy the Gherkin syntax. Not for testing, but for gathering feature requirements. It provides a very clear and concise way of explaining a feature, without confusion. But that is where the line is drawn."

That is also where I draw the line. The whole argument that other non- technical folks can read Gherkin may be true, but I can't imagine them enjoying it.

It's like reading computer-speak. Why can't features just be described in English...?

</comment><comment><USER37> Yes. And he's not the first one to say this. There was a post saying the exact same thing about a year ago, maybe even on HN?

But, yes, just yes.

(I'd go even further and say "why rspec when you can test::unit/minitest? especially for Rails since the minitest path is officially supported by rails." Again, the extra layer of stuff is just more stuff to understand and troubleshoot when your tests aren't working as you'd like -- for unclear benefit.)

</comment><comment><USER42> The thread seems to be overrun by a bunch of whiny crybabies.

No one forces people to use them, yet they seem to have a great time crying about it without trying to defend their position.

Cucumber and RSpec are among a set of testing tools which has really changed the testing landscape. It would be really wonderful if people spent half this time and energy on trying to create a better tool or joining the open source groups of Cucumber and RSpec and trying to make it better.

<comment><USER14> > No one forces people to use them

Well, other than, in many cases, employers.

</comment></comment><comment><USER43> I'm a .net dev so I've been using SpecFlow (which uses Gherkin) and frankly, I love it. I write a happy path and a sad path for my scenarios, make sure my step definitions are well parameterized and my QA can go off and create as many new scenarios to test all the edge cases he wants. It works wonderfully.

</comment><comment><USER44> As a product/program manager... I can tell you that writing RSpec tests are just as easy to write as Cucumber tests. And, with less overhead.

</comment><comment><USER45> Really, I'm not a Ruby guy, so I thought it's some vegetable story... cause I love cucumbers (& tomatoes)... Ah...

</comment><comment><USER46> Figured the bottom voted comment would be about the virtues of fresh cucumber.</comment>


Worry Less. Do More. Be Fearless - <AUTHOR> http://blog.jdconley.com/2013/07/worry-less-do-more-be-fearless.html

<comment><USER2> Holy shit... I'm so sorry that you and your family have to suffer through something like that. Best wishes for a speedy and complete recovery for Peyton.

As for the business... meh. I had the good fortune to be able to volunteer as a firefighter for about 10 years, and worked as a 911 dispatcher for a while... after having been in those literal "life and death" situations a few times (including a couple dealing with my own potential death), I've found that you just don't get as excited about certain things... perspective really does change things.

When you are dealing with life and death situations, the business world seems a lot less important. Even if the business fails, at least everybody goes home at the end of the day.

</comment><comment><USER3> Great story. One thing: on your RealCrowd homepage, even though I'm on the internet all day, I didn't connect with the need to scroll. The only thing that won the war with the back button (because it was an open tab, and I had forgotten why I opened it)... was because I remembered that the page was a link from a post I read. You might want to shrink that header or something.

Best of luck!

</comment><comment><USER4> Amazing story...perspective. Peyton is such a trooper!

</comment><comment><USER5> Whoa... Indeed, this puts thing into perspective really fast!</comment>


Paste the Plan - <AUTHOR> https://www.brentozar.com/pastetheplan/ <comment><AUTHOR> It's a free service to share Microsoft SQL Server Execution Plans. This will be great to share and discuss plans on [http://dba.stackexchange.com](http://dba.stackexchange.com).</comment>


Ask HN: Secure automated backup? - <AUTHOR> Hi HN,
After seeing all the news about lost data recently, I need to get my arse into gear and get an automated backup set up properly.
I'm using a Mac, so I looked into the Time Capsule. That said, if one of the data loss scenarios is a well-written ransomeware worm, it feels like the Time Capsule is going to be just as vulnerable as my main machine.
What approach would you recommend to back up data, with both hard drive failure and ransomware in mind? I'm open to cloud based solutions if that actually makes more sense. <comment><USER2> I use borg[0] to create local space efficient encrypted backups and rclone[1] to mirror the archives to Google Drive. I wrote a short script to automate it and schedule it to run every night.

[0] [https://borgbackup.readthedocs.io/en/stable/](https://borgbackup.readthedocs.io/en/stable/)

[1] [https://rclone.org](https://rclone.org)

<comment><USER3> You don't have the Google drive app installed anywhere?

If you do, this setup doesn't help recovery after a cryptolocker. The encrypted backup would also be unusable.

</comment></comment><comment><USER4> I'm surprised that no one mentioned Tarsnap yet, it's run by a well known HNer (cperciva): [http://www.tarsnap.com/](http://www.tarsnap.com/)

It's not exactly noob friendly though.

<comment><USER5> Well it's quite expensive compared to other solutions.

<comment><USER6> It really isn't. I mean it _looks_ expensive, until you take into account all the dedup and everything it does to reduce the cost. I put in $10 dollars a few years ago, I still have almost a $4 balance ($3.91).

<comment><USER7> How much data are you storing?

<comment><USER6> not much obviously, I am only storing critical data in tarsnap. Like others have said, if you are storing large GB's or TB's of data, rsync.net or something may be a better solution.

</comment></comment></comment></comment></comment><comment><USER8> There's a common rule called the 3-2-1 rule, it states that you should:

\- Have at least three copies of your data.

\- Store the copies on two different media.

\- Keep one backup copy offsite.

Personally, I'd recommend:

Copy 1: Your Mac.

Copy 2: A local NAS (my personal choice) or hard disk.

Copy 3: A remote backup, stored on a hard drive in a desk drawer at work, Backblaze, Google Drive, Amazon Cloud Drive or whatever other solution suits your needs.

In terms of software, I personally use rsync + ZFS/BTRFS snapshots (NAS - local, NAS2 - remote) and rclone (cloud). I haven't really used fancy solutions like Attic and Borg due to their need to write dead (i.e. not mountable without a performance penalty) data to local disk or SSH. No affordable storage that I've found offers this (rsync.net offers it but is too expensive).

It's getting to the point where I'm seriously considering buying an LTO6/7 tape drive though...

I'll also add because I haven't seen it elsewhere: _verify your backups_. A backup is pointless unless you _know_ you can restore it. The best way to test this is by doing it. It should get to the point where you don't fear a restore. It shouldn't be painful. There should be no worry. It should be no more than an inconvenience. When something goes wrong, you don't want there to be even the smallest hint of doubt that there's something wrong with your process.

As such, I _strongly_ recommend having an easily accessible backup. I'd go for a spare HDD sitting in a desk drawer at home before going for cloud backups just so that you can test it frequently.

<comment><USER9> It's also worth thinking about time to restore. If you have hundreds of GB worth of backups it could take a very long time to restore everything from the internet. Keeping an easily accessible backup around is really worth it.

</comment></comment><comment><USER10> I use Arq ([https://www.arqbackup.com](https://www.arqbackup.com)) with Amazon Drive (unlimited data for $60/year) for this

<comment><USER11> I also use Arq but send to rsync.net with reduced pricing ([http://rsync.net/products/attic.html](http://rsync.net/products/attic.html)) in addition to SFTPing to a personal (offsite) server.

Additionally, I run Backblaze and use Carbon Copy Cloner roughly once a week back to clone my entire drive to an external drive.

For personal servers I use borg with the same reduced rsync.net pricing.

<comment><USER12> rsync.net is fantastic.

You don't get the read-only snapshots with the reduced borg or attic accounts though (they expect you to manage all increments using those programs).

If you are prepared to pay the standard rate, the read-only snapshots can't be destroyed by any hacker or ransomware.

</comment></comment></comment><comment><USER13> I have a setup which works really well for my photos and videos [1][2][3][4][5]. It automatically keeps a copy of each file in 3 locations; my laptop, a Synology NAS and Google Drive / Photos.

[1] [https://medium.com/@jmathai/introducing-elodie-your- personal...](https://medium.com/@jmathai/introducing-elodie-your-personal- exif-based-photo-and-video-assistant-d92868f302ec)

[2] [https://medium.com/@jmathai/understanding-my-need-for-an- aut...](https://medium.com/@jmathai/understanding-my-need-for-an-automated- photo-workflow-a2ff95b46f8f#.dmwyjlc57)

[3] [https://medium.com/@jmathai/my-automated-photo-workflow- usin...](https://medium.com/@jmathai/my-automated-photo-workflow-using-google- photos-and-elodie-afb753b8c724)

[4] [https://medium.com/@jmathai/one-year-of-using-an- automated-p...](https://medium.com/@jmathai/one-year-of-using-an-automated- photo-organization-and-archiving-workflow-89cf9ad7bddf#.97qsvo3cq)

[5] [https://medium.com/vantage/how-to-protect-your-photos- from-b...](https://medium.com/vantage/how-to-protect-your-photos-from-bit- rot-9d0c6998121f)

</comment><comment><USER14> I used to use Crashplan which had unlimited storage and was fairly cheap(like 4$/month or something) for a family plan.

You might want to check it out. [https://www.crashplan.com/en- us/features/](https://www.crashplan.com/en-us/features/)

Also it was one of the few services that had a client that worked on Linux

<comment><USER15> "used to use" is an interesting endorsement. Why don't you use it anymore?

<comment><USER14> My plan had expired. And i ended up not renewing(since i was cutting down on all 3rd party services that i use). So i just moved to an external hard disk and "Back in Time" on linux.

But i've been looking at re-subscribing to it again.

</comment></comment></comment><comment><USER16> Here are some options that I have experience with:

\- Time Machine with offline disks: Since Time Machine supports multiple backup destinations, you can use a Time Capsule or hard drive that's always connected to your Mac, and also have one or more additional hard drives which you connect periodically and otherwise leave in a drawer.

Pros: Free, built into macOS, can browse file versions directly from many apps.

Cons: Needs ongoing manual intervention (i.e. plugging in the offline drives). Some reliability issues… but I've experienced the most problems backing up to my own SMB/AFP shares, so a Time Capsule might be OK.

\- Backblaze ([https://www.backblaze.com/](https://www.backblaze.com/)) or CrashPlan ([https://www.crashplan.com/](https://www.crashplan.com/)): Both of these online backup services have $5/month unlimited plans, and both let you specify your own encryption key (in the form of an additional password), which isn't shared with the backup provider. Note: In my experience, Backblaze's client is much lighter on system resources/battery on Mac.

Pros: Inexpensive, off-site storage, low-maintenance.

Cons: Ongoing cost, requires trust (In theory, the client software could be sharing the encryption key with the company/the NSA/your nemesis).

\- Arq ([https://www.arqbackup.com/](https://www.arqbackup.com/)): Paid desktop software which can back up to many different destinations, including S3, Google Drive, or your own server via SFTP. You specify an encryption key for each destination.

Pros: Full control. Option to back up to another machine that you own (so no ongoing cost for hosting).

Cons: Up-front cost. Support is less straightforward than hosted solutions since Arq doesn't provide storage.

<comment><USER17> An unlisted con of Backblaze is that they delete all external drives if not plugged in for at least 6 consecutive hours every 30 days. It can be a huge pain if you travel regularly or otherwise don't want to leave your computer on all night.

<comment><USER18> Is this a new policy? Coincidentally I just restored something off a year-old backup from a dead machine's external drive, and it wasn't an issue. Maybe because the machine itself hasn't connected in a while?

<comment><USER17> The machine not connecting at all puts it in some kind of exempt from deletion state. They claim it is a six-month limit in their docs.

Note the point about having all external drives connected for that first boot or their backups will be wiped out. Very easy to shoot yourself in the foot.

[https://help.backblaze.com/hc/en- us/articles/217664898-What-...](https://help.backblaze.com/hc/en- us/articles/217664898-What-happens-to-my-backups-when-I-m-away-or-on- vacation-?mobile_site=true)

<comment><USER18> I can see how that would be a footgun. Thanks for the heads up.

</comment></comment></comment></comment></comment><comment><USER19> Most importantly: it must be the backup server that has to log into your computer to backup, and not the other way around. That way, if your computer/server is compromised, the backups are still there. If you make the error to connect to the backup server, a hacker could also log into it and delete everything.

I my backup server uses rsnapshot and you can only log into it with ssh + key + OTP.

</comment><comment><USER20> [https://www.arqbackup.com](https://www.arqbackup.com)

<comment><USER21> Arq is such a wonderful piece of software. If you go this route, I'd say to go for Amazon Cloud Drive as Arq's datastore--it's $59/year for unlimited data.

</comment></comment><comment><USER22> [http://duplicity.nongnu.org/](http://duplicity.nongnu.org/)

<comment><USER23> Duplicati is newer and better.

<comment><USER24> Been using this for a while now, seems to work great [https://github.com/duplicati/duplicati](https://github.com/duplicati/duplicati)

The features I like is * Encrypted cloud backup * Blockbased backup (only backup changes) * Restore files from a certain day

</comment></comment></comment><comment><USER25> I use Time Machine, Arq and Amazon Cloud Drive:

\- I have an external HDD partitioned in half: One half is for large external files that don't change much (raw files, archived data etc); and one half is a dedicated partition for Time Machine

\- Time Machine backs up my laptop. If I lose my computer but not my hard drive, I can get a new one and seamlessly get the computer back to exactly how it was when I last backed it up, open tabs and all

\- I also have Arq running, attached to Amazon Cloud Drive (cheapest external storage I know of). It backs up both selected portions of my laptop's disk, as well as the external hdd's non-timemachine partition (due to how TM works you can't really back it up to the cloud[1]) to "the cloud"

This leaves me with:

\- Three copies of my laptop data: in the laptop, in an external hdd and in the cloud

\- Two copies of larger data that can't fit, in the external hdd and in the cloud. My external HDD lives at home.

[0] Time Machine backups up once an hour, and stores backups as a simple directory structure on disk of your entire hard drive, except using hard links to old backups to avoid duplication. It keeps the last 24 hrs of hrly backups, the last 7 days of daily backups, and then weekly backups until it runs out of room.

This format simply doesn't work with the kind of backup where it scans a directory to see what's changed, because it effectively looks like you're adding hundreds of gigs of data each hour.

</comment><comment><USER26> I second borg backup, I use it on my linux/mac machines

for windows I use reflect backup [https://www.macrium.com/products/home](https://www.macrium.com/products/home)

I tried acronis backup, but the disk restore failed, absolutely horrible software. then tried reflect disk restore was very smooth.

</comment><comment><USER27> For local bootable backup I use Mac Backup Guru, which I also wrote: [https://macdaddy.io/mac-backup-software/](https://macdaddy.io/mac-backup- software/) It's useful because it's the only software on OS X besides Time Machine which makes versioned (incremental) backups using hardlinks.

For remote backup I use Arq, but I have found that to be very buggy. I'm considering switching to rclone: [https://rclone.org/](https://rclone.org/)

With both of those backup solutions in place I should be ready for pretty much everything.

<comment><USER28> Can you elaborate how Arq is buggy? I'm considering switching to Arq from Crashplan, because I supply my own storage for it anyway.

<comment><USER27> I have had to restore a backup once, so I restored the most recent version, which was restoring an old version of the file. I ended up frantically restoring all the different versions of the backups, and one of them was the most recent version. But it was considered an old version by Arq, I'm not sure precisely which one, but it definitely wasn't in the top 3 most recent ones.

And more recently I made a backup to glacier, and it is trapped in "in progress", even though it appears that it may (or may not) be complete.

I haven't used it much, maybe 6 times, and 2 of the times it's had these catastrophic problems I'm talking about. I'm going to switch to another solution. I was considering rclone (like I mentioned) or Crashplan, ironically.

<comment><USER28> Crashplan worked well for me, sometimes I have to restore a file that I overwrite or that ends up being corrupted because of buggy (in-house) software and I never had any problems with that.

My main issue is that I went with free plan + my own server because transfers to their servers were really bad. But to supply your own storage, you have to install Crashplan on the server, which uses significant amount of RAM. So this rules out backuping to my NAS (not enough memory there, even though it's x86 and technically people ran Java on it).

The other issue is that file format is (AFAIK) closed, and also you an account and connection to Crashplan service, so I'm not completely sure that you really own the data. I didn't do much research here, though.

<comment><USER17> Was the Crashplan client performance better when backing up to your own server vs to theirs?

<comment><USER28> If you mean client performance in general, I never had problems with that, it's just the connection to the server overseas was not that great. I have my own dedicated server in Europe to which I have much faster connection.

</comment></comment></comment></comment></comment></comment><comment><USER29> I quite like Crashplan:

\- very reasonably priced: I pay around £10 pcm for unltd storage for my whole family

\- zero-knowledge encryption: I have the encryption keys, and everything is encrypted on my machine before its sent up

\- relatively low bandwidth: only ships changed files (pretty standard tbh)

It's saved my bacon a few times, e.g. I've used it to rescue my sister's dissertation when she wiped her laptop thinking it was in Dropbox when it wasn't. I was amazed by how easy it was for me to rescue the file from the archive.

<comment><USER2> I used crash plan on Linux for years, but stopped because their Java client is a train wreck.

It would consume gigabytes of RAM and every year or so it'd meltdown when trying to install an update without using the system package manager.

<comment><USER17> This is the same reason I left Crashplan. If they could get the Java client under control it would be a worthy contender, probably even the best consumer- friendly option.

</comment></comment></comment><comment><USER30> Have all my data on Dropbox with revisions activated, and having that backedup by Crashplan. You'll have double automated backups and 0 hassle managing it.

</comment><comment><USER31> I have a related question. I want to take backup of certain folders to a portable USB HDD every night. Can anyone recommend any simple solution for that?

I don't need encryption or any extraneous features. I just need the selected directories to get mirrored to a backup location.

Currently, I am using SyncToy by Microsoft, but I was looking for a cross platform solution.

<comment><USER31> Found a list here : [https://en.wikipedia.org/wiki/Comparison_of_file_synchroniza...](https://en.wikipedia.org/wiki/Comparison_of_file_synchronization_software)

</comment></comment><comment><USER32> 3-2-1 rule.

I would use time machine capsule and periodically (weekly?) connect an encrypted external drive and Borg backup there. Next week a second drive, third week the first one...

Always keep one of this drive off-site.

This is just one of many options how to get reasonably safe (I use an almost this one just deja-dup instead of time machine.)

<comment><USER33> I also use deja-dup but lately it seems to choke when trying to determine what to back up (I have about 175 GB of files, which doesn't seem outrageous). Have you had any issues with the speed? It could be that I have a long history saved on my backup drive and it's trying to apply too many incremental diffs.

<comment><USER32> I have no idea. Anything in logs? Maybe move backups to an other location and start a new backup repo to see what happens?

</comment></comment></comment><comment><USER34> I've used the following method for years and it's really simple. Get an external hard drive and partition it as needed. One for your Time Machine backup and another for data. Use Google Drive to mirror the data and use Arq as your Time Machine in the cloud.

</comment><comment><USER35> I don't backup end-systems -- but I do have a directory with important data sync'd to several systems and the cloud using syncthing. The rest of the data I care about is in git -- everything else on the system is basically disposable.

</comment><comment><USER36> I think Amazon Cloud Drive is $60/yr if you have Prime. You can hook up your account to your Synology NAS and have it automatically back things up as soon as you copy it over. Also Synology can encrypt it on the fly as well.

</comment><comment><USER37> attic <[https://attic-backup.org/>](https://attic-backup.org/>) encrypts data in transit via ssh, deduplicate and encrypt data at rest. I have come to appreciate both how easy it is to restore data, and the control you have over pruning which backups are kept around. You either need to be able to install attic on the remote host, or be able to mount the file system (i.e. fuse).

</comment><comment><USER38> Do you think Google Drive (and others) are secure enough to store personal financial data like Quicken? Sometimes all you need is an email and password to get in.

<comment><USER17> You can always put extra secure documents inside password protected encrypted disk images.

</comment></comment><comment> vladimir-y [https://github.com/duplicati/duplicati](https://github.com/duplicati/duplicati)

</comment><comment><USER18> I see it's already been mentioned, but allow me to second Backblaze. I use it for hundreds of clients, and they have consistently been the most reliable of the backup services I have tested __*. Since they also have versioning, you can recover from CryptoLocker variants comparatively easily as well.

(Almost any cloud-based backup system can help detect Locker variants. If you notice your daily backup data set suddenly shooting up in size, time to start checking for background encryption.)

(°Backblaze, Carbonite, Crashplan, Mozy, Acronis)

</comment><comment><USER39> What sort of upload speeds do people doing cloud backups have?

I could never contemplate uploading 100s of Mb each day with my crappy ADSL2+

<comment><USER18> I have basic cable-provided internet. 50 down, 3 up.

Initial backups take a long time, no doubt. But the daily diffs aren't so bad. As long as you use something smart enough to only upload the deltas, you're fine.

You can also snail mail drives to many cloud providers for your initial backup state if you want. I've never done so, but I've heard from friends that have that it worked out fine.

<comment><USER39> That's very helpful info, thanks!

</comment></comment></comment><comment><USER40> I've used spideroak before. Zero knowledge encryption, even the NSA _probably_ can't access it

</comment><comment><USER41> Just stick it all in Google Drive, pay the $2/mo for 100GB or $10/mo for 1TB, done. It stores the last 100 versions of each file so ransomware shouldn't be a problem, although apparently there's no way to restore a folder at a time, you'd need to do it individually for each file...

<comment><USER14> Is there a good linux client? I couldn't get the ocaml client to work well enough.

I'd even be fine with even a one-way backup that backs up my entire ~/ (excluding some specified dirs/files) as snapshots. I don't need even need the two-way sync.

<comment><USER42> I've used rclone[1] with great success (it can sync specific folders which looks like what you want) but I've since then moved on to a commercial service called Insync[2] because I was having rate limiting problems with the Google Cloud API key that rclone needs.

I do wonder that I should move back to rclone because I don't like a third party having access to my google drive. As it goes without saying, use a good encrypted backup solution. I like cryfs for encrypting data.

[1]: [https://rclone.org/](https://rclone.org/) [2]: [https://www.insynchq.com/](https://www.insynchq.com/) [3]: [https://www.cryfs.org/](https://www.cryfs.org/)

</comment></comment></comment><comment><USER43> For Desktop Backups, I think backblaze.com should do the Job.

<comment><USER44> Desktops not running linux.

<comment><USER45> They now offer B2, which is really awesome.</comment></comment></comment>


Celery – Best Practices - <AUTHOR> https://denibertovic.com/posts/celery-best-practices/ <comment><USER2> In many projects Celery is overkill. Common scenario I saw:

      1. We have problem, lets use Celery
      2. Now we have one more problem.

I found [http://python-rq.org/](http://python-rq.org/) much more handy and cover most cases. It uses redis as query broker. Flask, Django integration included [https://github.com/mattupstate/flask- rq/](https://github.com/mattupstate/flask-rq/) [https://github.com/ui/django- rq](https://github.com/ui/django-rq)

<comment><USER3> Thanks for this, I had considered using celery for a recent project but ultimately backed away because I got the feeling it was more trouble than it was worth. As a point of reference would you say the learning curve for a celery setup is similar to that of django? Not that theres anything terribly hard about django, but Id agree that its probably overkill if youre relatively new to python and are just looking for a quick way to produce some html with no intent on developing it further.

<comment><USER4> I wouldn't say Celery's learning curve is steeper than Django's, but it definitely seems like overkill for your case. If you need to do some time- consuming action periodically (and making an HTTP request by hand each time is not an option), then you could just use cron for the start if your project is relatively simple. And if you literally need to just produce some HTML when asked for, then why are you considering using an async task processor such as Celery?

<comment><USER3> Oh no I would never use celery for that. It was just comparing the learning curve of celery to django.

</comment></comment></comment></comment><comment><USER5> Good, basic practices to follow. Here's a few more:

\- If you're using AMQP/RabbitMQ as your result back end it will create a lot of dead queues to store results in. This can easily overwhelm your RabbitMQ server if you don't clear these out frequently. Newer releases of Celery will do this daily I think - but it's worth keeping in mind if your RMQ instance falls over in prod.

\- Use chaining to build up "sequential" tasks that need doing instead of calling one after another in the same task (or worse, doing a big mouthful of work) in one task as Celery can prioritise many tasks better than synchronously calling several tasks in a row from one "master" task.

\- Try to keep a consistent module import pattern for celery tasks, or explicitly name them, as Celery does a lot of magic in the background so task spawning is seamless to the developer. This is very important as you should never mix relative and absolute importing when you are dealing with tasks. from foo import mytask may be picked up differently than "import foo" followed by "foo.mytask" would resulting in some tasks not being picked up by Celery(!)

\- Never pass database objects, as OP says, is true; but go one step further and don't pass complex objects at all if you can avoid it. I vaguely remember some of the urllib/httplib exceptions in Python not being serializable and causing very cryptic errors if you didn't capture the exception and sanitise it or re-raise your own.

\- Use proper configuration management to set up and configure Celery plus what ever messaging broker/backend. There's nothing more frustrating than spending your time trying to replicate somebody's half-assed Celery/Rabbit configuration that they didn't nail down and test properly in a clean-room environment.

<comment><USER6> With regards to #1: What happens is that if task_B depends on a value that task_A returns, task_A will insert its value into the queue and task_B will consume it.

If task_C returns a value which no other task cares about, it will insert the value into the queue, and never gets consumed. This is why dead queues (also known as "tombstones") happen.

Always remember to set ignore_result=True for tasks which don't return any consumed value.

EDIT: "Tombstones", not gravestones

<comment><AUTHOR> In general using an AMQP for the result storage is somewhat of a bad idea i think. But yes I agree about the ignoring results part seeing as most tasks I've seen in the wild don't return anything at all. Hence #6 in the post.

<comment><USER6> Good advice.

Do note that if you use the chord pattern ([http://celery.readthedocs.org/en/latest/userguide/canvas.htm...](http://celery.readthedocs.org/en/latest/userguide/canvas.html#chords)) anywhere, you must set ignore_result=False

</comment></comment></comment></comment><comment><USER7> I've worked 4+ years with Celery on 3 different projects and found it incredibly difficult to manage, both from the sysadmin and the coder point of view.

With that experience, we wrote a task queue using Redis & gevent that puts visibility & tooling first: [http://github.com/pricingassistant/mrq](http://github.com/pricingassistant/mrq)

Would love to have some feedback on that!

<comment><USER8> Looks interesting. Can't find any links to the docs?

<comment><USER9> I can't find any either - it looks like mrq is a front-end dashboard for python-rq: [http://python-rq.org/](http://python-rq.org/)

<comment><USER7> MRQ is heavily inspired by RQ to which we switched from Celery ([http://www.slideshare.net/sylvinus/why-and-how-pricing- assis...](http://www.slideshare.net/sylvinus/why-and-how-pricing-assistant- migrated-from-celery-to-rq-parispy-2))

However it is a complete rewrite because we felt we couldn't add gevent support and other features to provide extreme visibility without major changes. If you don't need those 2 things, you may want to check out RQ instead for now, it's still a very good piece of software.

<comment><USER9> Awesome!

I see now that mrq supports concurrency why python-rq does not (at least in a stable fashion).

I'll try mrq for the gevent integration. It's great that you guys are actively working on improving it. Python-rq is great too, but it hasn't been updated in a while and I don't think concurrency is on the radar.

</comment></comment></comment></comment></comment><comment><USER10> I would add:

1\. Use task specific logging if you have a bunch of task: [http://blog.mapado.com/task-specific-logging-in- celery/](http://blog.mapado.com/task-specific-logging-in-celery/)

2.Use statsd counters to keep track of basic statistics (counts + timers) for each task

3\. Use supervisor + monit to restart workers after lack of activity (I have seen this happen a few times, but never been able to track down why it happens, but this is an easy fix)

<comment><AUTHOR> More awesome tips. Thank you.

</comment></comment><comment><USER11> Excellent resource, I remember wrestling with learning celery and how to do some simple things, loved finding Flower to monitor things.

I will say though Celery is probably overkill for a lot of tasks people think to use it for, in my case it was mandated to support scaling for a startup that never launched, partly because they kept looking at new technologies for problems they didn't have yet.

</comment><comment><USER12> I disagree with the characterization in #1 (although I can't speak to the Celery particulars). I feel like if you have a job that is critical to your business process, the job should be persisted to your database and created within the same database transaction as whatever is kicking off the job.

Consider how background jobs are typically managed with RabbitMQ, Redis, etc. They are usually created in an "after commit" hook from whatever gets persisted to your relational database. In this scenario, there is a gap between the database transaction being committed and the job being sent to and persisted by RabbitMQ or Redis; during this gap the only record of that task is being held in a process's memory.

If this process gets killed suddenly during this gap, that background job will be lost forever. It sounds unlikely, but if RabbitMQ or Redis is down and the process has to sit and retry, waiting for them to come back online, the gap can be sizable.

<comment><USER13> I disagree with this, in my experience it's almost always a really bad idea to use the DB as a queue. If rabbitmq is down the process should retry a finite amount of times (usually 3 in our use case) then set a status on the db record. Then you have audits running to pick up records in that state and retry the process once the system is back up and running. That way nothing is lost and you gain all of the benefits of Rabbitmq.

<comment><USER14> _bad idea to use the DB as a queue_

Not according to Jim Gray. See "THESIS: Queues are Databases"[1][2]

1- [http://research.microsoft.com/apps/pubs/default.aspx?id=6849...](http://research.microsoft.com/apps/pubs/default.aspx?id=68494)

2- (pdf) [http://research.microsoft.com/pubs/69641/tr-95-56.pdf](http://research.microsoft.com/pubs/69641/tr-95-56.pdf)

<comment><AUTHOR> Tnx for the link. nice read. There is a difference though, in using something as "a Queue" and using something as an AMQP implementation when it's clearly not that.

</comment></comment></comment></comment><comment><USER15> Points 1 and 2 are only valid because the Celery database backend implementation uses generic SQLAlchemy. Chances are, if you are using a relational database, it's PostgreSQL. And it does have an asynchronous notification system (LISTEN, NOTIFY), and this system allows you to specify which channel to listen/notify on.

With the psycopg2 module, you can use this mechanism together with select(), so your worker thread(s) don't have to poll at all. They even have an example in the documentation.

[http://www.postgresql.org/docs/9.3/interactive/sql- notify.ht...](http://www.postgresql.org/docs/9.3/interactive/sql-notify.html)

[http://initd.org/psycopg/docs/advanced.html#async- notify](http://initd.org/psycopg/docs/advanced.html#async-notify)

<comment><AUTHOR> It is true that Postgres supports Pub/Sub but unfortunately the Celery broker driver does not take advantage of this. It would be great if we could get support for it. Nevertheless, just because it has pub/sub doesn't mean it's a full AMQP implementation. Also, there's the fact that most amqp solutions are in memory, wheres a database is on disk... also has it's costs.

<comment><AUTHOR> Anyone that's interested in Postgres's pub/sub might find this useful: [https://denibertovic.com/talks/real-time- notifications/#/](https://denibertovic.com/talks/real-time-notifications/#/)

Just slides though. Haven't gotten around to writing a post about it yet.

</comment></comment></comment><comment><USER16> Once you scale your worker pool up beyond a couple of machines you need some sort of config management with Celery. We use SaltStack to manage a large pool of celery workers and it does a pretty good job.

<comment><AUTHOR> Indeed. I use Ansible myself.

</comment></comment><comment><USER17> This is not a Celery-specific tip, but as Celery also likes to "tweak" your logging configuration you can use [https://pypi.python.org/pypi/logging_tree](https://pypi.python.org/pypi/logging_tree) to see what's going on under the hood.

<comment><USER18> You can disable Celery's automatic logging configuration by connecting a listener to the setup_logging signal.

[https://celery.readthedocs.org/en/latest/userguide/signals.h...](https://celery.readthedocs.org/en/latest/userguide/signals.html#setup- logging)

Of course, logging_tree is a great tool as well!

<comment><USER17> Take a look at [https://github.com/celery/celery/blob/v3.0.23/celery/utils/l...](https://github.com/celery/celery/blob/v3.0.23/celery/utils/log.py#L250) \- it's an older version that I once checked but it seems to be patching loggers unconditionally (i.e. outside any signal handler).

</comment></comment></comment><comment><USER19> I've been looking at Python tasks queues recently. Does anyone have experience on how Celery and rq stack up?

Rq is a lot smaller, more than 10x by line count. So if it works just as well, I'd go with the simpler implementation.

<comment><USER2> I used both, ended with Rq. Freedom if choice can be good, but when you able to make decision. Variety of backends, storages force you to understand how each component really work and when you dig into details you find that they all not equivalent. But you just need something f--kng working and you don't want to pay another guy to maintain zoo of different products.

That is why I decided to use Rq, it is better to know limitations of something simple then know possibilities but not able to make choice.

<comment><USER19> That's very helpful, thanks!

<comment><USER20> There are many differences, but most notably rq spawns one process per task. Line count is a stupid metric, e.g recently our line count doubled because of our new coding style, also the majority of the source code is tests.

</comment></comment></comment></comment><comment><USER21> Passing objects to Celery and not querying for fresh objects is not always a bad practice. If you have millions of rows in your database, querying for them is going to slow you way down. In essence, the same reason you shouldn't use your database as the Celery backend is the same reason you might not want to query the database for fresh objects. It depends on your use case of course. Passing straight values/strings should be strongly considered too since serializing and passing whole objects when you only need a single value is not good either.

<comment><AUTHOR> Oh absolutely values before objects. I said "serializing" more in the sense that pickle is always used for storing the arguments into the queue (or whatever the default serializer).

It always depends on your use-case but generally you want your application to behave correctly, which means it has to have correct/fresh data...you can't sacrifice correctness because of an inability to scale your database.

<comment><USER21> Yes. I think saying "you can't sacrifice correctness because of an inability to scale your database" is perhaps conveying the wrong message though. I mean, your very first point is about database scaling issues and the advantages of using something like RabbitMQ to avoid expensive SQL queries.

If you are processing a lot of data in Celery, you really want to try to avoid performing _any_ database queries. This might mean re-architecting the system. You might for example have insert-only tables (immutable objects) to address this type of concern.

<comment><AUTHOR> Fair point. I agree with this.

</comment></comment></comment></comment><comment><USER17> If you combine Celery with supervisord it's important to check the official config file[1]. At least two settings there are really important - `stopwaitsecs=600` and `killasgroup=true`. If you don't use them you might end up with a bunch of orphaned child Celery processes and your tasks might be executed more than once.

[1] [https://github.com/celery/celery/blob/ee46d0b78d8ffc068d5b80...](https://github.com/celery/celery/blob/ee46d0b78d8ffc068d5b80e9568a5a050c61d1a8/extra/supervisord/celeryd.conf#L18)

</comment><comment><USER22> Am I the only person who was genuinely disappointed that this wasn't about the vegetable?

It's a sadly under-rated ingredient! The flavor is subtle but unmistakable.

<comment><AUTHOR> Sorry to disappoint. Flower isn't about a real flower either. :P

</comment></comment><comment><USER23> Wondering about something: if you need to have a long task (5s to 10s) in the background, or even longer, for an AJAX request, what should you rather do:

\- use gevent + gunicorn, or Tornado, in order to keep a socket open while the worker is processing the task?

\- use polling? (less efficient)

\- use websockets (but then the implementation is perhaps a bit more complex)

can you do this simply using Flask?

<comment><AUTHOR> Hmm, seems we're talking about 2 things here.

If your ajax request requires long task processing and requires you to wait for it than this is not a background task any more, it's done in one of the web server threads, and even if the thread outsources the task to another process it's still waiting on that proces to finish before returning the ajax response. This is bad.

I'm not entirely convinced about websocket solutions in Python yet, but I've been told flask-websockets is awesome. Nevertheless this doesn't solve the problem for you. Cause the request is just keeping an open line and waiting for a respone....blocking is bad.

The most simplest advise I would have is to have the ajax request trigger a background task and return immediately. The background task will then have some kind of side effect (ie. write some result to a database somewhere) which the ajax request can the look for with some kind of polling mechanism (on some other endpoint). Of course you can complicate this a lot, depending on your needs, but this seemed like the most straightforward solution.

<comment><USER24> " _I 'm not entirely convinced about websocket solutions in Python yet, but I've been told flask-websockets is awesome. Nevertheless this doesn't solve the problem for you. Cause the request is just keeping an open line and waiting for a response....blocking is bad._" Tornado only blocks if you do something silly. It's event based, and can keep hundreds of connections open and waiting for it's async response event before actioning/responding the open connection.

" _The most simplest advise I would have is to have the ajax request trigger a background task and return immediately. The background task will then have some kind of side effect (ie. write some result to a database somewhere) which the ajax request can the look for with some kind of polling mechanism (on some other endpoint)._ " Wow, overkill much? Polling is bad, and is exactly the kind of _bad solution_ that a lot of these libraries are in place to prevent developers from needing to do.

Websockets were made to solve the long-polling and poll-spamming that was prevalent. Now all you have to do is keep a light, open web-socket connection to the server. And the server, being async/evented, will respond when the task is good and ready. Nice and clean.

<comment><AUTHOR> "Tornado only blocks if you do something silly. It's event based, and can keep hundreds of connections open and waiting for it's async response event before actioning/responding the open connection." \- Yes pure tornado based apps are probably fine if you know what you are doing.

"Wow, overkill much? Polling is bad, and is exactly the kind of bad solution that a lot of these libraries are in place to prevent developers from needing to do." \- Polling is not bad if you have a good use case. You just cannot do non-blocking stuff with Django for instance, or it's very very hard and tricky. Websockets also limit you with the number of connections you can have open at once.

</comment></comment></comment></comment><comment><USER25> What about using Redis as a celery backend ? Redis has a pub sub mechanism which seems quite reliable, so no need to poll.

<comment><AUTHOR> Redis is still not an AMQP, but yes Redis's Pub/Sub works quite nicely. Out of all the brokers celery supports I'd recommend only RabbitMQ and Redis to people.

<comment><USER25> Yeah, I've been using redis with celery in production to perform lots of network io related tasks on a low end machine because

a > Redis uses less memory

b > Redis is easier to setup

<comment><AUTHOR> With container solutions like Docker and prebuilt images the setup part is kinda eliminated. Although I don't remember having any special configuration issues with RabbitMQ as well, it just works TM. That's always nice right? :)

</comment></comment></comment></comment><comment><USER26> As one the authors of taskflow I'd like to give a little shout-out for its usage (since it can do similar things as celery, hopefully more elegantly and easily).

Pypi: [https://pypi.python.org/pypi/taskflow](https://pypi.python.org/pypi/taskflow)

Comments, feedback and questions welcome :-)

</comment><comment><USER27> I've heard so much about Celery but still have no clue when it would be used. Could someone give some specific examples of when you have used it? I don't really even know what a distributed task is.

<comment><AUTHOR> A background task is just something that's computed outside of the standard http request/response process. So it's asynchronous in the sense that the result will be computed sometime in the future, but you don't care when.

Distributed just means that you can have your task processing spread out across multiple machines.

A specific example would be, let's say, after your user registers on your website for the first time you wan't to get a list of all his facebook/twitter friends. This action will take a long time and is not vital to the whole registration/login process so you set a task to do that later, and let the user proceed to the site and not make him look at the spinner the whole time, and when the friend list becomes available it will show up on the website (on his profile or whatever). Makes sense?

</comment></comment><comment><USER28> I'd also add: Be wary of context dependent actions (e.g. render_template, user.set_password, sign_url, base_url) as you aren't in the application/request context inside of a celery task.

</comment><comment><USER29> Has anybody been able to make a priority queue (with a single worker) in celery?

Eg, execute other tasks only if there are no pending important tasks.

<comment><USER30> The FAQ question isn't very clear about it, but it doesn't look like it's possible: [http://celery.readthedocs.org/en/latest/faq.html#does- celery...](http://celery.readthedocs.org/en/latest/faq.html#does-celery- support-task-priorities)

</comment></comment><comment><USER31> Small typo where you define `CELERY_ROUTES`. `my_taskA` should probably have the routing key `for_task_A`, right?

<comment><AUTHOR> Not really, that's just the name of the actual task itself ie. "def my_taskA(a, b, c)".

<comment><USER31> This line `'my_taskA': {'queue': 'for_task_A', 'routing_key': 'for_task_B'},` "for_task_B" should be "for_task_A" to match the CELERY_QUEUES definition. Unless I'm misunderstanding what you're doing, of course.

<comment><AUTHOR> Ah no, you got that one perfectly I just didn't understand what was meant at first. Fixed. Thank you.

</comment></comment></comment></comment><comment><USER32> > when you have a proper AMQP like RabbitMQ

AMPQ = Advanced Message Queuing Protocol so it's wrong to say that a message broker is "an AMQP". Also, give Redis a try - it's much easier to set up and uses fewer resources.

We should probably talk about the elephant in the room when addressing newbies: the Celery daemon needs to be restarted each time new tasks are added or existing ones are modified. I got past that with the ugly hack of having only one generic task[1] but people new to Celery need to know what they're getting into.

[1]: [https://github.com/stefantalpalaru/generic_celery_task](https://github.com/stefantalpalaru/generic_celery_task)

<comment><USER33> Let me repeat, you don't need this to load/reload tasks. There is 'pool_restart' broadcast command[1].

[1]: [http://docs.celeryproject.org/en/latest/userguide/workers.ht...](http://docs.celeryproject.org/en/latest/userguide/workers.html#pool- restart-command)</comment></comment>


MNIST Handwritten digits classification using Keras, part 1 - <AUTHOR> http://www.python36.com/mnist-handwritten-digits-classification-using-keras/ <comment><USER2> With all the progress in machine learning it seems like there should be an amazing OCR tool that works out of the box for structured documents.

Does it exist?

I've used tesseract and its relatives, but they seem to have a hard time with any document that's not a single column. The difference between what they can achieve (which to be fair is amazing) with what I _expected_ based on all the ML demos that only does the first 10% (numbers only, no structure), but does it in a 30-minute demo, is big. Things like affine transforms (scaling, rotation) and decorations like bold, underline, and weird fonts create even more problems.

Why isn't there a docker container with an AWS lambda function in it that takes any image format I upload (pdf, png, jpg as the most critical) and returns a UTF string of its content?

My god, I'm spoiled by technology.

<comment><USER3> Agreed.

What I do for tables is use the OCR as a step 1, where I just extract the coordinates of each textbox. Then, use one of several methods to recognize columns (Hough transform, or we search for empty areas). Then, do some clustering to recognize lines.

The problem is that there's a million of edge cases that you need to handle (split boxes that span multiple columns, in the case of tables recognize cells that span multiple columns or rows, etc.). There _should_ be a solution for all these edge cases, but it gets tricky quickly.

<comment> GCU-Empiricist Having worked on some of these questions, then there comes the question of logical flow, how do you order that inset square of text at the bigger font, is it a picture caption or a header text?

</comment></comment></comment><comment><USER4> (Apologies in advance if this sounds snarky, don't mean to be)

There are hundreds of posts on using Keras/TensorFlow/PyTorch/etc to do MNIST classification, and many examples on Github. All these resources are very easy to find by Googling. This post doesn't seem to have anything different to add to the conversation. So, why do such articles continue to be written, and why do they still get upvoted on HN? Is it that a lot of people want to learn these things, but haven't gotten a chance to, so they upvote with the hope of staying in touch with the topic? Is it FOMO? One might be forgiven for considering this spam.

<comment><USER5> Moderators routinely downweight tutorial articles for that reason. It took a surprisingly long time to realize it, but tutorials aren't a good fit for Hacker News.

HN is for articles that gratify intellectual curiosity. Most tutorials don't do that; it doesn't fit with being a recipe for a task. If a recipe were to dive deeply into how and why its steps work, how they got that way, and compared other recipes that have (or don't have) similar steps, it would gratify intellectual curiosity much better. But it would be a worse recipe. It would also be a lot more work to write.

The worst kind of tutorial from an HN point of view is the kind filled with arbitrary details that apply only to a particular program or product, like "Now put a file named foo in a directory called baz". But that is almost all of them.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

</comment></comment><comment><USER6> "Many good ideas will not work well on MNIST (e.g. batch norm). Inversely[,] many bad ideas may work on MNIST and no[t] transfer to real [computer vision]" – a tweet by François Chollet (creator of Keras)

So - please, anything harder (or rather: more relevant to deep learning). At least images in CIFAR-10: [https://blog.deepsense.ai/deep-learning-hands-on- image-class...](https://blog.deepsense.ai/deep-learning-hands-on-image- classification/)

<comment><USER7> Or Fashion-MNIST [https://github.com/zalandoresearch/fashion- mnist](https://github.com/zalandoresearch/fashion-mnist) :)</comment></comment>


Wipro Intruders Targeted Other Major IT Firms - valiant-comma https://krebsonsecurity.com/2019/04/wipro-intruders-targeted-other-major-it-firms/ <comment><AUTHOR> "According to the lawsuit by Maritz Holdings, investigators also determined that the "attackers were accessing the Maritz system using accounts registered to Cognizant. For example, in April 2017, someone using a Cognizant account utilized the "fiddler" hacking program to circumvent cyber protections that Maritz had installed several weeks earlier.""

Someone misplaced the parentheses. They should be around the word "hacking", not Fiddler. According to Wikipedia, it is a debugging proxy written by a former program manager for the Internet Explorer group at Microsoft.

<comment><USER2> More than a decade back, Fiddler was the tool for debugging any network issues in software on windows machines. Definitely not a Hacking program.

<comment><USER3> While it's not originally made to be a malicious tool or intended for malicious use, it's definitely still a tool a malicious "hacker" in the commonly used sense of the word could use.

It's still a damn handy HTTP proxy for MITM to decrypt SSL traffic, for example. This allows one to closely inspect how a site's authentication mechanism is implemented, among other things. I still use it today occasionally to try and understand how a given API or site feature is working/implemented based on the raw HTTP requests. You can also use it to try and replay HTTP requests with modified payloads too, as another example. There's loads of hacking related activities Fiddler can be handy for!

Fiddler's entire purpose is arguably to intercept, modify and replay HTTP/HTTPS traffic. Many debugging tools like this can also be used nefariously. If I needed to try and poke for vulnerabilities, a debugging proxy like Fiddler is absolutely something I would put in my toolkit. There's no shortage of tutorials on google for using Fiddler to try and hack websites either.

<comment><USER4> If you're trying to mitm, you want a hidden agent that isn't easily recongizable and takes up little space.

</comment></comment></comment></comment><comment><USER5> > What's remarkable is how many antivirus companies still aren't flagging as > malicious many of the Internet addresses and domains listed in the IoCs, as > evidenced by a search at virustotal.com.

I didn't quite understand this part. Do antivirus companies generally flag IP addresses as malicious? Also not sure what the connection of that is with virus total.

<comment><USER6> If I try to open a known phishing URL, most business firewalls will block the connection. It's not so much a reference to endpoint antivirus, but a part of the all encompassing "AV" solution. You can look up what Mcafee will do with a URL for example here: [https://www.trustedsource.org/](https://www.trustedsource.org/)

Of course the catch is being a "known" phishing site. Google Safebrowse usually blocks URLs in Chrome by the time our firewalls get a block in place.</comment></comment>


Dow loses 1,300 points erasing nearly all advances under the Trump presidency - <AUTHOR> http://washingtonpost.com/business/2020/03/18/stocks-markets-today-coronavirus/ <comment><USER2> After all this, stocks are _still_ overvalued by historical standards:

[https://www.multpl.com/s-p-500-pe-ratio](https://www.multpl.com/s-p-500-pe- ratio)

Perhaps surprisingly, this even ignores the inevitable hit to earnings Q1 will bring. Factor that in, and fair value on the S&P is way, way lower from here.

Throw in the inevitable massive distortions of the economy through the Fed's institutional panic moves, and who knows where fair value lies.

I'm not sure many people realize that there are two simultaneous and very grave crises happening at the same time: COVID-19 and the return of the Great Financial Crisis.

<comment><USER3> What does overvalued even means? You cannot compare historic prices in isolation. Value is relative, so while yes ratios are higher and have been higher the past few years, stocks were still undervalued compares to bonds, real estate, commodities etc. It's useless to compare 1980s stock prices when interest rates were at 13% to now. That's why you usually see the "stocks are a bubble, too expensive" from retail investors that just cannot understand relative value and fixate on the "good old times". It's simple, getting 13% returns on bonds meant that stocks would always underperform in terms of yield so they had a lower relative value. This is why no one professional investor is waiting for a return to those level. Because they would be impossible and non sensical.

</comment></comment><comment><USER4> If you're going to write this story, fine, but maybe wait another day so you don't have to write "nearly"?</comment>


Daylight Saving Time Is Terrible: Here's a Simple Plan to Fix It - <AUTHOR> http://www.theatlantic.com/business/archive/2013/11/daylight-saving-time-is-terrible-heres-a-simple-plan-to-fix-it/281075/?single_page=true <comment><USER2> _" It's true that larger time zones would seem to cheat many people out of daylight by removing them further from their true solar time. But the demands of global commerce already do that."_

Most people don't care about global commerce, nor does it cheat them out of sunlight. And those that do will need to know how to coordinate their clocks regardless.

<comment><USER3> > Most people don't care about global commerce

Only the ones with money.

</comment></comment><comment><USER4> Here's an even simpler plan: _just stop doing it_.

</comment><comment><USER5> i like dlst frankly, long summer days, long evenings, and not pitch black in the morning in the winters.

Never understood the hubbub about it.

<comment><USER6> Darkness at 4PM on a December afternoon feels like the world is conspiring against you, while you're being robbed of your youth.

I honestly despise the mornings. I'm not the kind of person who relishes a schedule, and when I'm being forced to engage in behavior not of my own design, I hate mornings even more.

Bus stops, train schedules, school, teachers, college professors, jobs, work, employment, bosses, none of it anything I ever invited into my life. These things foisted upon me as normal, bald facts. Uncomfortable and coerced.

Go to school or you'll be pumping gas and waiting tables your whole life. Perform at work or you'll be a homeless skell, evicted onto the streets because you can't pay rent. Don't step out of line or civilization as you know it will turn its back on you, and you'll deserve it, because the world never asked for you, either. So don't get smart about it, you entitled little brat.

Get up. Out of bed. Spend time in rooms with strangers for no reason. Abstract concepts will assault you for the next 8 hours. Endure the psychotic nightmare of the public school bus ride.

20 minutes of sunlight on the way to school or work is little consolation for the reality presented to me on a daily basis.

And when that machine which consistently erases one third of my day spits me back out onto the street, what's my reward? Bitter cold, and an immediate sunset drenching the remainder of my waking hours in darkness.

Ask my why I'm a night owl? Everything else has been taken from me.

<comment><USER5> >> I'm not the kind of person who relishes a schedule,

ah... then don't worry about DLST, get up when you want, go to be when you want.</comment></comment></comment>


MapReduce explained - <AUTHOR> http://www.joelonsoftware.com/items/2006/08/01.html

<comment><USER2> Java functors are ugly. I've seen an example where every column of a database had a corresponding class. This facillatated the use of functors.

Java has another example to teach. JavaBeans can be called transparently via remote procedure calls. When developers discover this feature, they write _everything_ as JavaBeans on the assumption that it increases scalability. Instead, it leads to obscene implementations with trivial operations requiring hundreds of remote calls within a server cluster. Understandably, this adversely affects performance.

So, don't prematurely optimise. Use map and reduce sparingly. You wouldn't use RPC to make a string uppercase. Likewise, you wouldn't sum an array of 10 integers using reduce.

<comment><USER3> Of course you wouldn't uppercase a string using RPC that's what you have map reduce for... split that string into chars send each node a single char to uppercase and combine the results you get back.

And they said programmers have a tendency to make things overly complicated...

<comment><AUTHOR> As one my teachers used to say: "It is better to use a bad language well than a good language badly" The blame is not so much the language but the programmers.

<comment><USER4> This is a great insight, one that I can unequivocally agree with, and it led me to an insight of my own. Modern-day teachers have the cause-and-effect backwards. Instead of focusing on improving the quality of the programmers, they are focusing on developing a language which is impossible to use incorrectly, no matter how bad that language is. Java can certainly be seen as a step in this direction.

</comment></comment></comment></comment><comment><USER5> A more in-depth look at the same stuff:

[http://www.youtube.com/results?search_query=Cluster+Computin...](http://www.youtube.com/results?search_query=Cluster+Computing+and+MapReduce+Lecture&search_type=)

</comment><comment><USER6> The most interesting thing in my opinion is that MapReduce implementations has been made with c++ (Google) and Java (Hadoop/Yahoo). And not with a functional programming language.

</comment><comment><USER7> Theres a rather good explanation of MapReduce in the book "Beautiful Code". (Its just a rather good book, period.)

<comment><USER8> I think the book is very hit or miss from chapter to chapter.</comment></comment>


Ask HN: I'm new to the valley. What's the best way to meet people/programmers? - <AUTHOR> I'm a 20 year old that dropped out of college and just moved to Palo Alto. I'm working on a synchronous, real-time chat platform and am looking to connect with people/programmers in the area. Any suggestions/advice? <comment><USER2> Meetups. There are a zillion technical meetups.

</comment><comment><USER3> The TGI Fridays in San Bruno. Or the cafe at Santa Clara Fry's store in the evening and on weekends.

<comment><AUTHOR> Thanks. Will check these out!

</comment></comment><comment><USER4> Are programmers not people?</comment>


C hyper reference - <AUTHOR> http://www.lancaster.ac.uk/~simpsons/char/ <comment><USER2> This is great, thanks for posting it.</comment>