Google finally responds to Europe's antitrust charges - <AUTHOR> http://www.economist.com/node/21662618/ <comment><USER2> "Google argues that the commission's proposed remedies, which include obliging its website to display ads "sourced and ranked" by rival companies, would only have a legal basis if it were a monopoly provider of essential supplies such as gas or electricity."

Interesting. So, Google isn't a monopoly or web search that it provides isn't essential supply. That means, as I understand, there are plenty of comparable rivals in the search area - or the search itself isn't particularly important today.

Is it so? Haven't heard about Europe-based - or anything-based search engines, which were as successful as Google, or even by half as successful. Similarly, I don't know many working people who don't search on the Internet daily. May be they can avoid doing that, but they probably don't want to.

Sounds strange.

<comment> rm_-rf_slash Google is like a convenient highway that everyone takes to work: just because there are other routes (DuckDuckGo) it doesn't mean people will take them, and regulatory framework has to, in absence of a legitimate competitor (Japan prefers Yahoo! for some reason), treat Google as a functional monopoly.

Standard Oil held down prices which was good in theory for customers, but it was devastating for competitors and their employees, who spent like any other consumer. Europe's attitude towards Google isn't very free-market minded, but they are asking what kind of society they want, rather than having the economy as the be all and end all of their values.

EDIT: If you're going to downvote without explaining why, then you haven't really helped the conversation, you just made yourself feel less insecure about your own opinion.

<comment><USER3> >Standard Oil held down prices which was good in theory for customers, but it was devastating for competitors and their employees

Uh, are you saying that Standard Oil should have made business decisions based on what was good for their competitors? This starts to get into collusion.

And I understand that the theory was that Standard Oil would hold prices down, running at a loss in order to kill off all competition, then raise prices as high as it wanted in a non-competitive market. But this doesn't seem to apply to Google's situation as customers can easily move to searching Amazon, eBay, etc. when shopping.

<comment> rm_-rf_slash The problem wasn't just Standard Oil, it was the society around it. A purely free market society would see what the company was doing and close the books at that, but the government has a broader base of concerns. Namely, constituents of oil-producing regions who elected representatives to protect them.

Think of it like Wal-Mart: it may provide jobs and cheaper goods to a community, but the wealth is a net outflow towards the Walton family. Local businesses tend to recirculate their profits within the community through investment and higher spending, resulting in a broader standard of life increase.

Europe as a whole is much more concerned with quality of life than America, so it makes sense to see why they would view Google's homogenization of habit as a potential threat.

I think it's worth considering other societies' values when debating their laws compared to your own. There's a lot that America does right that Europeans want no part of, and vice versa. It really depends on who you are as a person.

<comment><USER3> > Think of it like Wal-Mart: it may provide jobs and cheaper goods to a > community, but the wealth is a net outflow towards the Walton family

Providing jobs and cheaper goods to the community sounds like it's putting wealth into the pockets of the community to me. If you can buy a gallon of milk for the kids cheaper at Walmart then you can at the corner store, you have extra money for school supplies, medicine, etc.

>Europe as a whole is much more concerned with quality of life than America

Sorry, I have to roll my eyes at that comment and move on...

<comment> rm_-rf_slash 1: Unless Wal-Mart sells literally everything (health insurance, higher education, etc), then proprietors of other kinds of businesses will be hurt because average incomes have gone down. Milk may be cheaper but getting a contractor to tile my roof isn't.

2: Go to, say, Ravenna, Italy, and try to get a drink after midnight and on a weekday. You can't. The people there would rather all be on the same schedule (espresso for breakfast, big lunch with the family, work till evening, late dinner, then a stroll around town) instead of having their friendships and families be separated by the graveyard shift. As a hungry American at 2AM it was an inconvenience, but overall it works and people are happy.

If you're still unconvinced, ask a European what they think of paid parental leave, month-long vacations (inb4 productivity argument, Germany offers much longer vacations than America and has higher productivity per worker), national single-payer health care...I could go on. If Americans really, _really_ cared about these things, we would have them by now.

<comment><USER3> > 2: Go to, say, Ravenna, Italy

Not to pick on Italy, but:

[http://ycharts.com/indicators/italy_youth_unemployment_rate_...](http://ycharts.com/indicators/italy_youth_unemployment_rate_lfs)

So I wouldn't exactly want to model my economy on Italy.

>If you're still unconvinced, ask a European...

I've spent a lot of time in Europe and had long discussions in the pubs. Great times, nice folks, and I look forward to visiting again. but I still prefer the US.

<comment> rm_-rf_slash You're still missing the point.

My family in Gaeta knows the economy has been rough for a while, and yet their (and Italy's) strong social ties have kept society together in ways GDP and unemployment figures can't measure.

Contrast with China, where growth is less than double digits and everybody's losing their mind. GDP doesn't explain why someone on the street dying from a car crash can be ignored by hundreds of people who walk by because they don't want to get involved in something that isn't their problem (punitive laws that potentially transfer liabilities to involved bystanders don't help).

Society is more than economy. People often choose benefits to their society that tax economic growth, like preventing acid rain. If something seems irrevocably inconsistent about someone's decision, then I question the extent to which you choose to think from their perspective.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER4> The wheels of the legal system turn so much slower than the fierce competition and relentless change of technology markets. By the time this is resolved in the courts, Google probably won't even be dominant in search / ads anymore -- just like Microsoft lost its stranglehold on the browser market before the antitrust case was resolved.

<comment><USER5> You have your dates wrong. The US antitrust case was settled in 2001. Back then, there still was no real competitor to IE: Netscape had disbanded and Mozilla was still releasing only the full suite. Webkit did not exist (its predecessor KHTML was kinda crappy) and Chrome wouldn't be released before 2008. There is a line of thinking that says MS stopped working on the browser not just because they had killed Netscape, but because the antitrust case turned that market into a minefield. In any case, they had a full decade of dominance post-antitrust.

In any case, the whole process took about 3 years. Quite a lot in technology terms, but not really that long. 3 years from now, I don't see Google going anywhere nor any competitor taking over -- technology in search hasn't changed much in the last 5 years, if anybody could take a real shot at unseating google they would have done it by now.

</comment></comment><comment><USER6> It should be clear to anyone paying attention that Google is using their dominance in search to support their businesses at the expense of their competitors.

Just do a search for "Chicago Hotel" on a desktop browser. Unless you have a gargantuan screen, you will not see a single organic result. You will be given a list of paid ads, and then links to Google's local results. Clicking one of those local results will take you to another screen where you will see more paid ads, a Google Map, and a few seemingly organic results. But if you click on one of these "organic" results, Google then shows you a page where you can book a hotel room - and pay Google a fee for that.

<comment><USER7> I tried it: [http://i.imgur.com/a45wZdf.png](http://i.imgur.com/a45wZdf.png)

And while the top 'results' were not 'organic', they were not ads either. They were simply the top rated hotels on google maps near Chicago.

When I clicked on 'More hotel', I was taken to google maps, which showed me top rated hotels as well as ads for hotels.

In terms of giving me the most relevant results, I think this is pretty close.

<comment><USER8> You have an ad blocker.

This is what the actual results are on a 1080p screen.

[http://i.imgur.com/5Te0gmG.png](http://i.imgur.com/5Te0gmG.png)

<comment><USER9> Isn't it funny Google even states "About 650.000.000 results" in that page!?</comment></comment></comment></comment>


Show HN: Cloud Nuke-Safely delete AWS resources in 3 clicks - <AUTHOR> https://cloudnuke.io <comment><USER2> You have to be completely nuts to put your secret keys in a site on the web.

<comment><AUTHOR> From the FAQs

Q: Do you store the keys? A: No, each Nuke is an ephemeral sandboxed process and we don't store any keys.

<comment><USER3> Not sure that's gonna be enough assurance, my friend

</comment></comment></comment><comment><USER4> Here is the CLI you can run yourself [https://github.com/gruntwork-io/cloud- nuke](https://github.com/gruntwork-io/cloud-nuke)

<comment><AUTHOR> Thanks for linking this

</comment></comment><comment><USER5> Open source CLI to do the same for free without passing AWS keys to third- party: [https://github.com/rebuy-de/aws-nuke](https://github.com/rebuy-de/aws- nuke)

<comment><AUTHOR> Thanks for linking this

</comment></comment><comment><AUTHOR> I launched Cloud Nuke to make it easy to safely delete AWS resources in 3 clicks!

I've found that deleting idle & underutilized cloud resources is the quickest win to reduce spending on cloud bills. Additionally the workflow of deleting cloud resources often exist in Engineering teams but it is not well thought out and a single member scrambles to determine what needs to be deleted.

Building on clouds is getting easier, and conversely harder to delete since resources often depend on each other.

<comment><USER6> I don't think anyone is going to just throw keys in this willy nilly. Perhaps put up some instructions on how to make a one off user with the bare needed access that can be revoked after or something. Still a stretch tho.

<comment><AUTHOR> Great suggestion, thanks!

</comment></comment></comment><comment><USER3> It's a few years since I've used AWS, but is deleting stuff that hard? I seem to remember it was a matter of deleting a cloud formation stack and maybe cleaning up a few things that you'd explicitly ask to be preserved (usually storage things like S3 buckets and DBs).

<comment><AUTHOR> It's definitely not point-and-click. My thesis (and experience) is that it has gotten harder

Building & Deploying on clouds has gotten easier, and many dev environments have idle, underutilized resources that are paid for.

</comment></comment><comment><USER7> > Q: Do you store my card?

> A: No payments are proessed directly by Stripe and we don't store your card > details

"Let's eat grandma".

<comment><AUTHOR> I don't follow...?

<comment><USER8> The parent post above is referencing an online meme that pokes fun at a lack of punctuation[0].

In the Q/A section quoted, the lack of punctuation can be read as:

> Q: Do you store my card?

> A: No payments are processed directly by Stripe, and we don't store your > card details

Which implies that Stripe is not processing the payments, with the note that the card details are not stored.

This could be rewritten as:

> Q: Do you store my card?

> A: We do not store your credit card information. Payments are processed > directly by Stripe, and we don't store your card details.

[0] [https://i.imgur.com/gbJVPk3.png](https://i.imgur.com/gbJVPk3.png)

<comment><AUTHOR> gotcha, thanks makes sense now.

> Q: Do you store my card?

> A: We do not store your credit card information. Payments are processed > directly by Stripe, and we don't store your card details.

Thats exactly what we mean

<comment><AUTHOR> Updated the FAQs. Thanks for pointing this out

</comment></comment></comment></comment></comment><comment><AUTHOR> Thanks for all the feedback HN, I do agree the service is a risky tool so right now I've stubbed it out, effectively disabling its use while I figure out how to address the comments here

</comment><comment><USER9> You would have to be crazy to utilize this tool.

<comment><AUTHOR> I do think deleting cloud resources is a valid use case in many Eng teams and directly co-relates to a lower cloud bill. Engineers are frequently building POC's, test machines or just deploying quickly to have idle/underutilized resources lying around

<comment><USER9> Cloud Custodian [1] is a good option for this.

[1] [https://cloudcustodian.io/](https://cloudcustodian.io/)

<comment><AUTHOR> Love this tool too, thanks for linking it in

</comment></comment></comment></comment><comment><USER10> The overuse of emojis kill all of your credibility and make you seem like a child playing with my secret keys.

<comment><AUTHOR> Ok, thanks for that feedback. It's currently disabled while I figure out next steps

</comment></comment><comment><USER11> kind of terrifying.. wonder how long until a script kiddy finds some access/secret keys in github or somewhere else and kills a company.

<comment><USER12> They could do the same by using the access keys directly - using this service is strictly better since it would identify the attacker by their Stripe payment method.

<comment><USER11> Yea, agreed.. they definitely could assuming someone does something stupid and exposes keys with access to everything. But this removes the barrier of needing to have a tiny bit of technical knowledge to do it. I think pastebin post with the cloudnuke url, keys, and a stolen credit card would look pretty appetizing for bored people. I'm not saying this shouldn't exist exactly.. maybe some kind of additional identity verification would make it less scary tho.

<comment><USER13> The same pastebin could exist today by simply providing a script alongside the access keys, I don't see how this paid-for service changes anything aside adding an extra hurdle.</comment></comment></comment></comment>


Ask HN: What is the cheapest web-dev stack? - <AUTHOR> The best technology for a project is subjective. Some people are looking for the fastest, easiest.
I am about to start a solo project. I don't have a lot of cash, that's why I'm looking for a new stack that will cost me the least amount of money possible.
Does anyone have any ideas?
Thank you. <comment><USER2> It's hard to answer without knowing your experience. Usually people here suggest going with what you're most comfortable with.

My personal preference for a new projects in the last decade has been Ruby on Rails. There's not much hype about it anymore but I tend to think it's because it's gotten more mature.

You can host it cheaply on Heroku with simple $7/mo dynos or on a $5/mo DigitalOcean droplet. Another option is to get a new account for your app and get $300 credits on GCP for experiments.

The rule of thumb—in my humble opinion—should be: don't scale until you need to. Vast majority of projects are not going to hit Pokemon Go scale in their early days, if ever.

For what it's worth Heroku supports different runtimes and you're not limited to Ruby on Rains with them.

So use what you're comfortable with, don't scale prematurely, don't use fancy tech for the sake of fancy tech, and ship, ship ship :-)

<comment><AUTHOR> Thank you for the reply. I like your last sentence, that's something. I need to keep in mind.</comment></comment>


Dear Airlines: This Is What Your Boarding Passes Should Look Like - <AUTHOR> http://www.wired.com/design/2013/09/airlines-please-adopt-this-smart-boarding-pass-redesign/

<comment><USER2> Previous discussion from 3 days ago at [https://news.ycombinator.com/item?id=6447149](https://news.ycombinator.com/item?id=6447149) .</comment>


Show HN: web-based music player built with Node and CoffeeScript - <AUTHOR> https://github.com/superjoe30/groovebasin

<comment><AUTHOR> By the way... I'm moving to NYC in 16 days and I am looking for room mates. Hit me up!

<comment><USER2> Nice. Why NYC?

<comment><AUTHOR> I am joining Indaba Music.

</comment></comment></comment><comment><USER3> You mention "some might call it an MPD Client"? Why not advertise it as such? Because it might scare away users? What are the features from MPD git that you depend on?

How does it compare to other web/browser-bsaed clients such as relaXX [1] or MPM [2] ?

Note that I haven't tried groovebasin yet.

[1] <http://relaxx.dirk-hoeschen.de/> [2] <http://code.google.com/p/musicpm/>

<comment><AUTHOR> _Why not advertise it as such?_

Because there is a huge selection of already existing MPD clients in existence[1] and I'm trying to stand out a bit.

[1]: <http://mpd.wikia.com/wiki/Clients>

_What are the features from MPD git that you depend on?_

Client-to-client communication. Groove Basin uses it to broadcast some status messages to other clients currently connected.

Probably going to change the architecture of that soon.

Oh, another thing that is relatively new is the sticker file, which is required for Dynamic Mode. (We remember the last-queued date for each track to give bias to not recently queued songs when queuing random songs)

 _How does it compare to other web/browser-bsaed clients such as relaXX [1] or MPM [2] ?_

I tried relaXX. I haven't tried MPM, but I checked out a screenshot and read the README.

I think when it comes to media players, it's the small things that make a big difference. How intuitive the UI is, whether it is cluttered, whether it feels like the same song keeps coming up on random, how long it takes to browse and queue music, etc.

Groove Basin attempts not to have the most features, but to get the small things right.

I'll take a deeper look at relaXX and MPM after work and give a better description of the actual differences if you're interested. I'm pretty sure the answer is going to be 1. how easy it is to queue things, and 2. Dynamic Mode.

</comment></comment><comment><USER4> Love the look. I made a simple web music player not to long either ago. Check it out: www.tunesicles.com

<comment><USER5> Sorry, but it looks like twitter bootstrap barfed all over... <http://i.imgur.com/ReAdC.png>

I'm guessing something went wrong, hence the "Code Corrupted" message.

<comment><USER4> Oh thanks. Havn't realized that. I think it's my statcounter giving that error. I'll have to fix that. Still not done the site - lots left to add. Thanks for checking it out.</comment></comment></comment>


Why I Two-Space (2012) - <AUTHOR> http://stevelosh.com/blog/2012/10/why-i-two-space/ <comment><USER2> So adding an extra space between every sentence all the time, just for vim's "deleting sentence" to work? (not even by default, and that's a very specific command). And the command breaks because there is a . in the sentence.

So changing a pretty much worldwide convention for a pretty specific edge-case on a rarely used command in vim.

Why not just update the command to detect the special cases like Mr. Mrs., seems like a much simpler solution.

<comment><USER3> Similarly, Emacs has a jump-to-next-sentence command that only works if you two-space.

</comment></comment><comment><USER4> If you want to make the argument that two-spacing is legit for monospaced fonts in text editors, great. I'll buy it. But to claim that because HTML and LaTex exist that source and presentation are orthogonal concerns flies in the face of the fact that most text is created in Microsoft Word or something like it. I don't know how many installs of Vim are out there, but I'm guessing it's less than the 500 million plus installs of Word. This is the kind of tech-geek pedantry that just pisses me off. _Should_ content and presentation be separate concerns? Yes. Are they, for 95% of the world's population? Not even close. Which case should you optimize for?

</comment><comment><USER5> Except he didn't two-space (never heard that rule before). The only post I looked at. [https://bitbucket.org/sjl/stevelosh/src/cb3ae608dff5b5ad47c3...](https://bitbucket.org/sjl/stevelosh/src/cb3ae608dff5b5ad47c362726d35bb24663e8059/content/blog/2015/11/happy- little-words.markdown?at=default&fileviewer=file-view-default)

</comment><comment><USER6> For HTML output I'm able to force two spaces of presentation by making the first a non-breaking space. Perhaps a converter from Markdown could do something similar when encountering two spaces after a period in the source.

The "vim" point was interesting, I'd never seen an example of the semantic value of the extra space before.</comment>


What you need may be "pipeline +Unix commands" only - <AUTHOR> https://nanxiao.me/en/what-you-need-may-be-pipeline-unix-commands-only/ <comment><USER3> I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers.

Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable.

There's so many things that the software world is doing wrong that I am surprised that even at this inefficacy, it's such a viable and well-paid profession. It's almost as if we are creating insanely complex solutions that in turn require a large amount of developers to support them, whereas we could have chosen a much more practical solution which is self-sustaining.

<comment><USER4> This is something I've noticed in the last 8-10 years. The rise of the python/js/java paradigm everywhere. Some of the associated LDIF (json) I enjoy much more than XML and flat files but the misapplication of tools is becoming an epidemic. When I can write: awk -F "," '{for (x = 1 ; x <= NF ; x++) {if ($x ~ /[0-9]+/) {a[x] = a[x] + $x}}} END { for (p in a) {printf "%d = %d\n",p,a[p]}}' to sum columns in 5 seconds and people are scrambling with libraries to do matrix operations I tend to scratch my head and walk away. The aversion to the command line is also something that bothers me but I don't run into it as much in my field.

<comment><USER5> When I can write

        import pandas as pd
        data = pd.read_csv(filename)
        print(data.sum())

and have the same result, I'm going to do the one that is faster to write, fewer characters, and lets me understand what's going on.

And don't get me wrong, I've written some gnarly pipelined bash before, although I'm by no means an expert, but that doesn't mean its always the right tool for the job.

<comment><USER4> I was going to be the long haired *nix geek here but I have no hair and the world is moving on. I can't pick bones with python for data science/analysis and personal convenience. _However_ as a principal engineer if someone was to say, for a trivial dataset, that we need python and pandas for an operation like this where python + pandas was not already provisioned the answer would be no.

</comment></comment></comment></comment><comment><USER6> I've been a pipeline junkie for a long time, but i've only recently started to get into awk. The thing i can do with awk but not other tools is to write stateful filters, which accumulate information in associative arrays as they go.

For example, if you want to do uniq without sorting the input, that's:

      awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }'

This works best if the number of unique lines is small, either because the input is small, or because it is highly repetitive. Made-up example, finding all the file extensions used in a directory tree:

      find /usr/lib -type f | sed -rn 's/^.*\.([^/]*)$/\1/p' | awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }'

That script is easily tweaked, eg to uniquify by a part of the string. Say you have a log file formatted like this:

      2019-03-03T12:38:16Z hob: turned to 75%
      2019-03-03T12:38:17Z frying_pan: moved to hob
      2019-03-03T12:38:19Z frying_pan: added butter
      2019-03-03T12:38:22Z batter: mixed
      2019-03-03T12:38:27Z batter: poured in pan
      2019-03-03T12:38:28Z frying_pan: tilted around
      2019-03-03T12:39:09Z frying_pan: FLIPPED
      2019-03-03T12:39:41Z frying_pan: FLIPPED
      2019-03-03T12:39:46Z frying_pan: pancake removed

If you want to see the first entry for each subsystem:

      awk '{ if (!($2 in seen)) print $0; seen[$2] = 1; }'

Or the last (although this won't preserve input order):

      awk '{ seen[$2] = $0; } END { for (k in seen) print seen[k]; }'

I don't think there's another simple tool in the unix toolkit that lets you do things like this. You could probably do it with sed, but it would involve some nightmarish abuse of the hold space as a database.

<comment><USER7> I admire your work. Clever usage of unix tools is very handy. But for parsing text, do you really see that awk and Unix tools as a better solution then a simple python script?

Although I admit that the key argument for Unix tools is that they don't get updated. That sounds awful, but think about it, once it works, it works everywhere, no matters OS type, version or packages installed. That is something experienced programmers always want from their solutions.

<comment><USER6> Python is fantastic for little (or large!) bits of logic, but its handling of input is clunky enough to put me off for tiny things. AFAIK the boilerplate you need to get to working on the fields on each line is:

      import sys
      for line in sys.stdin:
        fields = line.split()
        # now you can do your logic

If you want to use regular expressions, that's another import.

Python also doesn't play well with others in a pipeline. You can use python -c, but you can't use newlines inside the argument (AFAICT), so you're very limited in what you can do.

<comment><USER8> This is exactly where perl (namely, perl -ne) is so very, very useful.

<comment><USER6> Yes, that helps a lot. The fact that Perl uses braces rather than whitespaces also makes it work much better in this situation.

I still wouldn't touch Perl with a bargepole, though. Sorry not sorry.

</comment></comment></comment></comment></comment><comment><USER9> I had a task the other day to aggregate some logs. So I wrote a one liner, which did most of what I wanted. I took about 4 minutes to run.

Then I decided to run it on larger dataset (because I needed too). Like week of logs, not a day of logs.

While it was running, I wrote rust CLI, which was working like `cat __/ *.log | logparser` and did one day in 12 seconds, and a week in a two minutes.

And I gave up waiting on awk, btw. It is not always better to use command line. If you have gigabytes or tens of gigabytes of data, it would be easier to write some cli tool to help you out.

Also, it was much easier to put significantly more complex logic into it because of type checking, and, you know, being actual high level programming language, not hack&slash awk script.

EDIT: Looking back on my "one liner" vs "rust cli" I would not be able to make meaningful adjustments to one liner comprehension. It is, to my sorrow, write- only thing.

<comment><USER10> If your awk script gets too long / unreadable you just put it in a file and use some whitespace and longer variable names.

AWK scripts tend to be very readable (much more so than e.g. sed) as long as they stick to the "stateful filters" use-case as [https://news.ycombinator.com/item?id=19294195](https://news.ycombinator.com/item?id=19294195) calls it, but yes they have their limits.

If speed is a concern, you may want to try using mawk instead of GNU awk/gawk. I've had 4x speedups with mawk.

<comment><USER11> Based on his timings, rust achieved a 20x speedup vs a 4x speedup if he used mawk.

<comment><USER12> Yep, and adding gnu parallel or xargs to the mix would give a # of cores speedup, which would make mawk faster than the single-threaded rust on a > 6 core machine (roughly).

</comment></comment></comment></comment><comment><USER13> * If it's simple transforms, use cli tools.

* If it requires aggregation and it's small, use cli tools.

* If this is data you're using over and over again then load it in the database and then do the cleaning, ELT.

* If it's 2tb of data and under, still use bzip2, get splittable streams and pass it to gnu parallel.

* If it requires massive aggregations or windows, use spark|flink|bleam.

* If you need to repeatedly process the same giant dataset use spark|flink|bleam.

* If the data is highly structured and you mainly need aggregations and filtering on a few columns use columnar DBs.

I've been using Dlang with ldc a lot because of how fast its compile time regex is, and its built in json support. Python3+pandas is also a good choice if you don't want to use awk.

<comment><USER12> Before reaching for spark, etc:

Sort is good for aggregations that fit on disk (TBs these days, I guess)

Perl does well too if the output fits in a hashtable in DRAM, so 10's (or maybe 100's?) of GBs

</comment></comment><comment><USER14> I have been using the command line for all type of work for years now. The most satisfying is to realize that there is always more to learn. And once you grasped the basics, they fit together like LEGO bricks.

While I think it's important to make that argument, the posted article and the one it refers to lack some guidance on how to reach "command line mastery". I recently came across this great resource here on HN:

[https://github.com/jlevy/the-art-of-command- line](https://github.com/jlevy/the-art-of-command-line)

It gives great overview of the toolbox you have on the command line. Equipped with `man` you're ready to optimize your everyday work. And always remember to write everything down and ask yourself WHY something works the way it works. The interface of the standard tools is thought out very well. Getting comfortable with this mindset pays off.

</comment><comment><USER15> This old article has the same topic with a more complex example and a surprising result about parsing 3.46Gb of data:

[https://adamdrake.com/command-line-tools-can- be-235x-faster-...](https://adamdrake.com/command-line-tools-can- be-235x-faster-than-your-hadoop-cluster.html)

</comment><comment><USER16> Sometimes there's a middle ground: make your "map" and "reduce" steps separate scripts.

If you want to do the parsing in Python instead of awk, just make a tiny script that reads from stdin and writes to stdout - that way you can put it between xargs or parallel and whatever else is in the pipeline.

The parallelization is a separate concern, so it doesn't need to be mixed in with the parsing (or whatever) concern. The downloading is a separate concern; use wget or requests in a Python script or whatever, it doesn't need to be mingled with the parsing.

</comment><comment><USER17> This article's primary example is a single static text file with 5M lines. Sure, in that case, awk works great, but how often does that come up? In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. Further, users want real-time results, not just a few times a day when your cron script runs.

Unix commands are great up to a few GBs of data, Excel is even better if you're dealing with less than a few tens of MBs. But to deal with Terabytes of data quickly and efficiently, these tools totally break down.

<comment><USER18> > how often does that come up?

It's an important point to remember that a lot of things involved in human society have _not_ exploded in size or complexity in the last 30 years.

Many data sets are basically proportional to the human population (health records, criminal records, property records etc), and these have been measured in the millions for 30+ years. In the same time the compute power of a single script has moved from the millions into the billions.

It's important, because if a government needs to, say, calculate something involving "every building in the country", or "everybody with a criminal record" they need to understand that this task, in 2019, can in fact be done by a single programmer parsing flat text files on their MBP, and does not need a new department.

This is a bit like Grace Hopper always pointing out the difference between a microsecond and a nanosecond - [https://www.youtube.com/watch?v=JEpsKnWZrJ8](https://www.youtube.com/watch?v=JEpsKnWZrJ8)

<comment><USER19> > Many data sets are basically proportional to the human population.

Awesome point well put.

</comment></comment></comment><comment><USER20> Relevant reading: The GNU coreutils manual

[https://www.gnu.org/software/coreutils/manual/html_node/inde...](https://www.gnu.org/software/coreutils/manual/html_node/index.html)

</comment><comment><USER21> Does this add anything to the Taco Bell post linked in TFA?

I suggest changing the link to: [http://widgetsandshit.com/teddziuba/2010/10/taco-bell- progra...](http://widgetsandshit.com/teddziuba/2010/10/taco-bell- programming.html)

<comment><USER22> TacoBellArticle> I could have done the whole thing Taco Bell style if I had only manned up and broken out sed, but I pussied out and wrote some Python.

That's cringe-worthy...

<comment><USER23> Despite the gendered terms, what makes me cringe is the belief that X is somehow "better" than Y - if you know Python, and have access to Python, and express yourself in Python faster, then use Python. I use sed, awk, cut, Python, as needed - whatever lets me solve my problem faster.

No flexing about how you use X needed.

</comment></comment></comment><comment><USER24> Unix commands definitely go a long ways.

I've been freelancing for a long time but never automated invoicing people up until recently.

So I combined grep, cut, paste and bc to parse a work log file to get how many hours I worked on that project, what amount I am owed and how many days I worked. I can run these analytics by just passing in the log file, a YYYY/MM date (this month's numbers), YYYY date (yearly numbers) or no date (lifetime).

Long story short, the working prototype of it was 4 lines of Bash and took about 10 minutes to make.

Now I never have to manually go through these work log files again and add up invoice amounts (which I always counted up manually 3 times in a row to avoid mistakes). If you're sending a bunch of invoices a month, this actually took kind of a long time and was always error prone.

</comment><comment><USER25> When I was in middle school, a friend and I had an internship with a physicist who wanted us to write some software to perform a simple transformation on data sets that he downloaded by FTP from some experiment.

We spent about a week writing a program in QuickBASIC that successfully parsed the files and performed the transformation.

Some years later, I realized that this would be a one-line awk script which I could now write in 20-30 seconds. (Probably someone comfortable with Excel could also perform the transformation in 20-30 seconds, although it might not scale as well to larger files.)

</comment><comment><USER26> I agree with the sentiment that many solutions are over-engineered, but when you need to process billions of records a day, you do need more complex systems.

Bottom line: when facing an engineering problem, start with the simplest, fastest to implement solution, and build complexity as necessary. The simple solution suffices most of the time.

<comment><USER27> You know, "billions a day" is only on the order of 10K per second. A single machine can handle that.

<comment><USER28> That would be amazing, wouldn't it? It's not true though, the problem with dealing with billion of operations a day is the spikes, most of the time you don't get a nice homogene rate for 24 hours straight.

<comment><USER29> Depends on how much latency matters. A lot of big data is batch processing, for which if the data is 3 hours old that's more than good enough.

</comment></comment></comment></comment><comment><USER30> I really agree with aspects of this, and I think CLIs and Unix pipes are way more powerful than we treat them, but be forewarned that there are problems with doing everything with pipes.

You need to code more defensively with them. For example, it is rare, but every so often a newline will be fail to be emitted.

        kinda\n
        likethis\n
        \n
        example\n

There are many other gotchas, but that one is a doozy because if you're using, say, tab delimited data and cut you'll miss a line. It's one of the reasons I use line delimited JSON if at all possible.

Also, this constant re-parsing of text does mean your string validation needs to be more paranoid. For example, some JSON parsers parse curly quotes as normal programming quotes. Horrible practice, I know, but it could have been avoided. Also, it's easy to accidentally do shit like this when you're in a rush. Some string matching tools handle the character matching of the different ways of creating, say, "ë" will also make matching quotes more relaxed.

Anyway, all of this to say that I 100% agree with the posted and linked articles, but each method has its own security considerations and software folk should be aware of them before starting.

<comment><USER31> This is one of the reasons I prefer PowerShell, it requires a lot fewer text parsing shenanigans. UNIX tools simply failed to evolve. Single io stream pipelining on raw ASCII was perfectly reasonable in the 1970s but it isn't the 1970s anymore.

We should be composing tools with multiple typed io stream paths in GUIs (or TUIs I suppose), leveraging two or even three dimensional layouts. All our interfaces should be composed this way, allowing us to take them apart and modify them at will to fit our workflow.

But that never happened. We never made a better hammer, we just try to squeeze all our problems into ASCII-processing nails instead.

<comment><USER30> You know it is so funny that you're mentioning this. I completely agree.

It's to the point where I've been toying around with creating my own shell and faking typed IO streams via Postgres+DSL. It's tricky though. Sometimes I want pub-sub, other times I want event stream. Sometimes I want crash-on-failure, other times I don't. There is this problem in software that I can't really word precisely, but the closest I can come is "do it like this, except these cases here, except-except those cases there" and these things kinda keep stacking up until you have a program that has too much knowledge baked into it.

Take, for example, emoji TLDs. Because emojis aren't consistent across platforms they can get coerced into different types. I didn't know that when I bought and used a couple emoji domains. When someone tried to click on a link in Android and was met with a 404, I was so confused. I wasn't even seeing the request come into nginx!

After I figured it out, I realized that emoji domains won't work. The underlying assumption of TLDs is that there is one, and only one, way of encoding something and that these things aren't coerced. That assumption is wrong.

</comment></comment></comment><comment><USER32> I use an awk interpreter called mawk.[1] This is noticeably faster than gawk or other standard variants.

[1] [https://invisible-island.net/mawk/](https://invisible-island.net/mawk/)

<comment><USER33> Often, yes, though not always, and mawk has some omissions relative to gawk.

Try multiple interpreters with timings.

Gawk's profiler can be invaluable.

</comment></comment><comment><USER34> > BTW, if your data set can be disposed by an awk script, it should not be > called "big data".

I think this statement is wrong. The popular meaning of the hype term "big data" can not be easily changed.

Rather, awk, sed and other tools that can read from stdin and write to stdout are great tools for "big data" and often more efficient and suitable than larger and more hyped systems.

<comment><USER35> Among programmers I've found that the size of your "big data" is often implied to correlate with the size of something else

<comment><USER12> Yeah, like the amount of gibberish boilerplate you write, or the verbosity of the error messages your script emits.

</comment></comment></comment><comment><USER36> Yup. Totally agree with OP. Early on in my career I had to generate on the fly reports for hundreds of GB of data and all it needed was to throw some *NIX commands around and eventually piping them to awk to do the final bit and it was blazing fast.

These days, these are called big data. No it isn't...

</comment><comment><USER37> I don't get the motivation of this article, it links to the taco bell programming article which says exactly the same. I usually wouldn't write an article to repeat the same another article says, or if it is something that could have been just a comment in the original blog.

</comment><comment><USER38> Not directly command line, but very relevant: [https://www.frankmcsherry.org/assets/COST.pdf](https://www.frankmcsherry.org/assets/COST.pdf) "Scalability, but at what COST".

</comment><comment><USER39> Does anyone have suggestions on books to grow my scripting fu? (End of chapter exercises tend to be useful for me)

I know bash, and know a lot of basic commands, but I'm not familiar with some more advanced things. I don't know awk or sed for example.

<comment><USER40> If you're looking for good books, I can vouch for 'A Practical Guide to Linux Commands, Editors, and Shell Programming' which has very through coverage and end of chapter exercises. For PowerShell, I'm currently reading the the free PowerShell Notes for Professionals ( [https://books.goalkicker.com/PowerShellBook/](https://books.goalkicker.com/PowerShellBook/) ) and it's a great resource as well.

<comment><USER39> Thanks for the book suggestion, that looks like exactly the kind of thing I'm interested in.

</comment></comment></comment><comment><USER41>
        if your data set can be disposed by an
        awk script, it should not be called "big data".

Why not? I don't see how awk is limited to a certain amount of data.

<comment><USER42> If your data fits on a single harddrive it's not big data. So I would set the current limit to at least 14 TB.

<comment><USER43> I thought the boundary point was RAM. It is relativly simple to work with data across multiple drives. When you pass the boundary of being able to work in a single systems RAM, you genneally need a more significant rework

<comment><USER44> Most stream processing doesn't rely too much on RAM, unless you literally need all the data in memory at the same time.

<comment><USER45> If it _needs_ to be in RAM, then either you got a big enough machine (then by definition it's not Big Data) or it's impossible. If you manage to come by with RAM using smart algorithms, although the full dataset would never fit in RAM, then it's Big Data. So I'd argue, stream processing is Big Data, exactly because it doesn't rely too much on RAM.

<comment><USER44> True, but I can stream process 20GB of data on my tiny 2GB RAM home server as well.

That's not really 'big data' in my opinion.

</comment></comment></comment></comment></comment></comment><comment><USER46> This submission seems weirdly relevant.

[https://news.ycombinator.com/item?id=19271135](https://news.ycombinator.com/item?id=19271135)

<comment><USER47> I was going to comment on exactly this article except that I couldn't find it quickly. It isn't weirdly relevant; it's totally relevant. It demonstrates that, with sufficient knowledge of the command line, one can write the most amazing tools, quickly and succinctly. Here, knowledge doesn't necessarily meaning knowing everything immediately but also knowing what resources to reference to find out stuff.

I've been programming for 40 years and using unix/linux since the 80's and in this little one-line script, I discovered two things that one can do with the appropriate arguments that I've never known. YMMV.

</comment></comment><comment><USER33> Case in point from my own recent work: I've been analysing characteristics of Google+ Communities, mostly looking for plausibly active good-faith instances.

There are 8.1 million communities in total, and thanks to some friendly assistance, I'd identified slihtly more than 100,000 with both 100 or more members, and visible activity within the preceeding 31 days, as of early 2019.

The task of Web scraping those 100k communities, parsing HTML to a set of characteristics of interest, and reducing _that_ to a delimited dataset of about 16 MB, was all done via shell tools, and on very modest equipment.

Most surprising was that _parsing_ the HTML (using the HTML-XML utilities: [http://www.w3.org/Tools/HTML-XML-utils/README](http://www.w3.org/Tools/HTML- XML-utils/README)) took longer than downloading the data.

Creating the datafile was done with gawk, and most analysis subsequently in R, though quick-and-dirty summaries and queries can be run in gawk.

Performance: downloading (curl): 16 hours, parsing (hxextract & hxselect) 48 hours, dataset preparation (gawk): 2 minutes, analysis (gawk / R), a few seconds for simple outputs.

The parsing step is painfully long, the rest quite tractable.

<comment><USER48> Are you planning on open-sourcing the downloader part? I'm very interested.

<comment><USER33> Literally just a Bash while-read loop over community IDs. It's embarrassingly trivial.

I'm planning on posting the data, probably to [https://social.antefriguserat.de/](https://social.antefriguserat.de/) and will include procssing scripts.

This is the fetch-script, which saves both the HTML and HEAD responses:

        #!/bin/bash

        sample_file=$1

        comm_path='community-pages'
        base_url='https://plus.google.com/communities'

        i=0
        time sed -e 's,^.*/,,' $sample_file |
            while read commid;
            do
                i=$((i+1))
                echo -e "\n>>> $i  $commid <<<" 1>&2;

                url="${base_url}/${commid}"
                commfile="${comm_path}/${commid}.html"
                commhead="${comm_path}/${commid}.head"

                echo "curl -s -o '${commfile}' -D '${commhead}' '${url}'"

            done

The sample file is simply a list of G+ community IDs or URLs, e.g.:

        100000056330101053659
        100000310247038604843
        100000355641542704509
        100000408644688836681
        100000537266485621548
        100000813948204546252
        100001055751908082772
        100001158162744298957
        100001173291703462139
        100001193552641351693

<comment><USER49> This seems to be a perfect use case for GNU Parallel[0] to download and process, say 10 ids, in parallel. If you have already downloaded/processed and have no need to do it again, then probably doesn't matter now.

[0]: [https://www.gnu.org/software/parallel/](https://www.gnu.org/software/parallel/)

<comment><USER33> Xargs, actually, though saturating my dinky Internet connection was trivial. Ten concurrencies kept any one request from stalling the crawl though.

That's why the script echoed the curl commant rather than run it directly. It fed xargs.

The other problem was failed or errored (non 3xx/4xx, or incomplete HTML -- no "</html>" tag found) responses. There was no runtime detection of these. Instead, I checked for those on completion of the first run and re-pulled those in a few minutes, a few thousand from the whole run, most of which ended up being 4xx/3xx ultimately.

</comment></comment></comment></comment></comment><comment><USER50> > Every item on the menu at Taco Bell is just a different configuration of > roughly eight ingredients.

HA! This has almost been my line for years regarding Mexican food. What I like to say is: it's amazing how every possible permutation of 8 ingredients has been named. BTW I love Mexican food, lived in Mexico.

> The post mentions a scenario which you may consider to use Hadoop to solve > but actually xargs may be a simpler and better choice.

I do feel like there's a corollary to Knuth's "premature optimization" quote regarding web scaling; premature scaling and using tools much bigger than necessary for the job at hand is pretty common.

</comment><comment><USER51> It's not just data processing. There are simple solutions to all sorts of tasks.

I run 'motion' on my linux desktop at home to serve as a security camera when no one is home. For months I've been manually starting and stopping it, figuring I needed to setup an IoT system if I wanted to automate things. i.e. IFTTT on our phones, an MQTT server in the cloud, etc. Then I realized - <USER2> just need to start the camera when all of our phones are off the LAN. It took about 15 minutes to setup, and now I never have to worry about forgetting to stop or start the camera.

</comment><comment><USER52> For my next book, I'm working through a concept I call "good enough programming.

Good enough programming is something you code that provides something people want -- and you never look at the code again. Find the problem, solve the problem, walk away from the problem. That's not sexy. It's not going to get you an article to write for a famous magazine, but it's good enough.

We have lost sight of "good enough" in programming, and without some kind of guardrails, we end up doing stuff we like or stuff that sounds good to other programmers. For instance, while I love cloud computing, I'm seeing "how-to" articles written about setting up a VPC for doing something like playing checkers. Yes, it was an oversimplified article, and you have to write that way, but without _wisdom_ , how is the reader supposed to know that? What criteria do they use to determine whether it's a co-lo server, a lambda, or a world-wide distributed cloud?

We're going like gangbusters selling programmers and companies on all kinds of new and complex ways of doing things. They like it. We like it. But it is in anybody's best interest over the long run?

Recently I rewrote a pet project for the third time. First time it was C#, SQL Server, and an ORM. Then it was F#, MySQL, and linux. The last time it was pure FP in F# and microservices.

Some of you may know where this is going.

Just as I finished writing the app in a real microservices format, I realized. Holy cow! This whole thing was just a few Unix commands and some pipes.

My thinking went from all kinds of concerns about transactions and ORM-fun to just some nix stuff in a small script. The problem stayed the same.

Something else happened too. At each step, I did less and less maintenance. The last rewrite has had no maintenance required at all. In my spare time, I'm going to do the nix one using no servers at all on a static SPA. In a very meaningful way, there's no app, there's no server, and there's nothing to maintain. Yet I still get the functionality I need. And I never maintain it.

Of course that's not possible for every app, but the key thing I learned wasn't the magic of serverless static SPAs or the joys of unix. It was that _I didn 't know whether or not it was possible or not until I did it._ By thinking in a pure FP fashion and deploying in true microservices, the rest just "fell out" of the work. At first I was actually thinking in a way that would have only led to more and more complexity and maintenance requirements.

My belief is that we get our thinking right first, use code budgets, and try for a simple unix solution. If it doesn't work, why? At least then we've made an effort to be good enough programmers. That beats most everybody else.

<comment><USER53> Write a blog post about it! It would be nice to see the progression and read about your insights of such complete reworks, even if it's a pet project.

Edit: Can I can sign up somewhere to get a heads up when your book is available? Would be appreciated!

</comment></comment><comment><USER54> One place installed a Hadoop cluster. Some of us suspected that replacing a regex monstrosity with Lex/Yacc might have given the speed needed to process the files.

EDIT: Lex/Yacc that is some faster parser generator, I'm not too knowledgeable on that.

</comment><comment><USER37> I totally agree that most of the *nix tools are most of the time the best ones, but things get trickier when there is a complex dynamic pipeline e.g. the input conditions the kind of processes and these are also dynamic based on other inputs.

<comment><USER29> I've found Nextflow to be an excellent solution to parallelizing and adding extra logic to any cli pipeline. It also helps manage environment and track metrics.

</comment></comment><comment><USER55> My favorite thing has been `| ruby -e "puts STDIN.to_a. ..."`, allows to run any kind of code on the standard input, much easier than remembering awk/sed various options and much more powerful.

edit: same thing can be done with Python/Perl

<comment><USER56> Yes, easier if you know ruby.. sed/awk is good for munching strings in small scripts and it is also always installed on a unix system (AFAIK).

<comment><USER33> You'll find sed and awk in busybox, virtually always, even on minimal systems. Including embedded devices, routers, Android, etc.

</comment></comment></comment><comment><USER57> Yes, buddy, the simplest tools are the most powerful, and the fastest. Time is money.

</comment><comment><USER58> Unless your data contains spaces, tabs, or, god forbid, newlines. Unix pipeline tools lack any sort of useful data structuring capabilities, making them appropriate for one-off tasks at most.

<comment><USER59> Spaces, tab and newlines are not a problem.

The fact that there are some standard tools available doesn't mean you are limited to that.

If you have a CSV file with spaces and newlines use cvskit or a small python script importing the relevant library. If you have to parse JSON file you can use jq to pick the relevant fields regardless of how the document is formatted. You can even process binary data as long as the file format is understood by the tool.

<comment><USER60> Transcoding CSV to TSV using csvkit has been a life saver for me the last few months due to an old industry's insistence on CSV as an information delivery method. jq has previously been a lifesaver too.

</comment></comment></comment><comment><USER61> GUI's are the McDonald's of interfaces.

The choices are limited and when you go there you know there's a good chance that you will end up wondering why what you got caused so much pain.

</comment><comment><USER62> This is the Big Data version of "90% of Oracle instances could be replaced with sqlite".

Pragmatism almost always loses to CV-padding and office politics.

</comment><comment><USER63> Nice article, though I think you should work on your blog's typography. Using a bold typeface for body copy is not pleasant to read.

</comment><comment><USER64> Oh how I miss Ted Dziuba posts. He's was a treasure trove of pragmatism for the industry.

</comment><comment><USER65> At my company what you can do is determined by which AWS services you can string together.

</comment><comment><USER66> This is what you get by not answering RTFM to stupid questions.

</comment><comment><USER67> I hate to be that guy, but they're NOT "Unix" tools, as the name GNU literally states.

The post makes a good point that I fully agree with, just doesn't explain it well enough.

<comment><USER59> Many of the GNU tools are reimplementation of already existing tools.

For example the initial implementation of AWK was in 1977 [1], a few years before GNU even existed [2], so it _is_ a Unix tool.

[1] [https://en.wikipedia.org/wiki/AWK#History](https://en.wikipedia.org/wiki/AWK#History) [2] [https://en.wikipedia.org/wiki/GNU#History](https://en.wikipedia.org/wiki/GNU#History)</comment></comment>


Ask HN: What Did Steve Jobs Think of Theranos and Elizabeth Holmes? - <AUTHOR> <comment><USER2> I don't think he knew Elizabeth Holmes or Theranos.</comment>


MailCore 2: The next step of libetpan - <AUTHOR> http://mailcore.tumblr.com/post/52790194722/mailcore-2-the-next-step-of-libetpan

<comment><AUTHOR> And the github repo at: [https://github.com/MailCore/MailCore2](https://github.com/MailCore/MailCore2)

I did use in the past libetpan and MailCore for some iOS app development, and this is a great news to see that it's moving forward even after the Google acquisition for Sparrow.</comment>


Microsoft Surface Just Made the MacBook Air and the iPad - <AUTHOR> http://gizmodo.com/5919521/microsoft-surface-just-made-the-macbook-air-and-the-ipad-obsolete

<comment><USER2> The truncated title inverts the meaning of the original title "Microsoft Surface Just Made the MacBook Air and the iPad Look Obsolete".</comment>


Show HN: Research Lab Website Template - <AUTHOR> https://github.com/photonlines/Research-Lab-Website <comment><USER2> Hey man! Nice work! On mobile (Android), however, your navbar falls out of the screen.

<comment><AUTHOR> Thank you for the feedback! Yup - there are definitely some items that need to be looked at on mobile and I'll see if I can find time to fix them, but I figured I'd make it available for anyone anyways in case they wanted to take a look...I did this years ago and it was based on an old bootstrap framework. I definitely know there are some things that can definitely be improved :) Cheers.</comment></comment>


Anonymous publishes names of Ferguson KKK members - <AUTHOR> http://www.upi.com/Top_News/US/2014/11/16/Hacker-group-Anonymous-publishes-names-of-Ferguson-KKK-members/9011416181999/

<comment><USER2> Anonymity for me, but not for thee?

<comment><USER3> Are you implying it's not fair that the KKK doesn't have some way to uncloak the identity of the person who uncloaked their identities?

<comment><USER2> I'm implying that it is ironic that Anonymous is stripping individuals of their anonymity.</comment></comment></comment>


When a deleted file takes 20 mins from your maintenance, but a year of your life - <AUTHOR> https://www.theregister.com/2020/07/06/who_me/ <comment><USER2> Many years ago I worked with Data General minis running AOS/VS. We asked one of DG's gurus if there was a way to undelete a file, and he said that the best bet was to crash the system at once, and see whether FIXUP brought it back.

</comment><comment><USER3> Break on old sub machines is something I totally forgot about. I remember it used to halt when you plugged in the keyboard, which was super annoying when you had to use one.

</comment><comment><USER3> Break on old sub machines is something I totally forgot about. I remember it used to halt when you plugged in the keyboard.

</comment><comment><USER4> Did /proc/fd not exist back then or did these guys just not know about it? ;-)

</comment><comment><USER5> TIL what lost+found is for. That was some quick thinking.

</comment><comment><USER6> Sybase used file system files rather than raw devices?</comment>


Five Years of Show HN - <AUTHOR> http://antontarasenko.github.io/show-hn/ <comment><USER2> "So a Hacker News reception has good predictive power, estimated around the OLS's R² = 14%."

14% R-squared is the measure for 'good predictive power'? Second, using linear regression for binary variables feels kind of icky to me. A suggestion for an improve or fix or maybe even another blog post is to change the model to something more appropriate like a Logit/Probit model.

/econometric rant

Cool article though!

</comment><comment><USER3> Would be nice to have a link to the Show HN posts itself to read the comments.

<comment><USER4> I came here to say the same thing and leaving a comment because up-votes on comments aren't visible.</comment></comment>


Build Modules, Not Examples - <AUTHOR> http://reidburke.com/2013/03/29/build-modules-not-examples/

<comment><USER2> Regarding nodejs's success (mentioned this <https://news.ycombinator.com/item?id=5358127> when 0.10.0 was released):

"I find the strength, openness, and responsiveness of the community (mailing list, freenode IRC chat) to be node's greatest asset. They've done an amazing job in building culture and that should serve node well as it matures."

This is something that even the YUI community lacks.

</comment><comment><USER3> I think the original title of the blog post is a better description since the one submitted makes it look like a tutorial.

This is not a tutorial, but an editorial on why the author thinks Node.js has succeeded in newbie traction where as YUI and Ember have had trouble. Built modules that new users can get started with instead of documentation and examples that require more understanding to get started making something.

<comment><AUTHOR> I agree. The description now matches the original post, thanks!</comment></comment>


No, ICO Market Caps Are Not Crazy. The Reason Is Basic Economics - <AUTHOR> https://medium.com/@nickadamsjudge/no-ico-market-caps-are-not-crazy-the-reason-is-basic-economics-ae4f11ca0114 <comment><USER2> The concept of ICOs isn't necessarily a bubble. The part where non-credible founders with non-credible plans raise huge amounts of money is a bubble.

I also love how they're creating their own currency, selling it, paying employees with it, and promising that it will go up... and then they tack on a "no equity-like product is sold" fig leaf. The overall idea of RootProject sounds nice though.

</comment><comment><USER3> Ummmm, yes they are. They're a function of all the "free" money sloshing around in crypto due to the bubble. Etherium did what it did with $18 million. Now there are projects that are proposing to do things orders of magnitude less complicated raising $30-50 million.</comment>


We are currently taking a DDoS attack and are working to mitigate - <AUTHOR> https://status.github.com/?2013-03-24 "We are currently taking a DDoS attack and are working to mitigate. The site may be slow to respond, and you may struggle to pull/push code via SSH - we apologise for any inconvenience." <comment><USER2> This is one reason us corporate software pushers can't use github and the likes. We need stuff to be hosted on a network we control. Each hour we lose is potentially multiplied by the number of staff as we would push and pull hundreds of times a day. Someones head will roll if were down for more than a couple of minutes at a time.

It becomes just as much of a PITA importing projects github and maven for example as well.

We actually ended up sticking with svn and trac and doing manual mirrors of every source dependency we pull in to protect our asses. SVN is kept only due to externals support (even though all our externals are internal!). Our head rev is over a quarter of million revs, 37gb of data and the repository has been online since 2006 to give you an idea.

I'm aware of github enterprise btw - its actually easier to build our own using trac as we can support it end to end, its a crap load more configurable and we can scale it up and run failover nodes easily using svnsync and pgsql replication. We have our own plugins and reports plugged in using reportlab as well. And its not atlassian's crap either.

<comment><USER3> You know you can run git internally as well?

Using github as an excuse to stay with svn seems rather ill informed.

You wouldn't lose hours if github is down. You might lose your ability to deploy (if you are deploying from github) but you could setup any other remote you like, and deploy from that.

I'm not sure what the 'size' of your repo has to do with anything...

<comment><USER4> To me github is the killer feature of git.

I think svn is easier and more appropiate for a corporate environment.

<comment><USER5> I think the killer feature of git -- which applies to any other DVCS -- is that you're not dependent on github if you're well prepared. There's absolutely nothing stopping you adding redundant remotes so you always have somewhere to push/pull changes and deploy from.

        git remote add <remote_name> url

<comment><USER2> Yep and that's like managing dns with host files...

</comment></comment></comment></comment></comment><comment><USER6> Somewhere else on the Internet: "We are currently giving a DDoS attack and are working to..."

<comment><USER7> "...keep it that way."

</comment></comment><comment><USER8> If you have trouble cloning/pushing on email@example.com right now, you could use these config[1] for ~/.ssh/config

        Host github.com
          Hostname ssh.github.com
          Port 443

Cloning and pushing to github now works in my machine.

[1]: [https://help.github.com/articles/using-ssh-over-the-https- po...](https://help.github.com/articles/using-ssh-over-the-https-port)

<comment><USER9> Wow! Huge thanks, this actually works (for those of you still having trouble)

</comment></comment><comment><USER10> I really wish they'd release geographic information on who's launching these attacks. On the bright side, a Saturday @ 7 eastern isn't near as annoying as something during the workweek.

<comment> SG- While 'neat', it won't reveal who's actually controlling the botnet and launched the attack.

<comment><USER10> It's really quite a shame at how little can be done to find the perpetrator. I suppose contacting law enforcement wouldn't do much either, unless it's a significant persistent threat that would warrant a large scale investigation.

<comment><USER11> Based on the FBI taking 50 doors related to the operation payback ddos and yet not charging anyone it appears the US Attorney's office isn't sure its even a crime.

</comment></comment></comment></comment><comment><USER12> Why is anyone DDoSing GitHub?

<comment><USER13> A lot of people relies and uses Github on a very frequent basis. It's their way to get attention I guess.

<comment><USER12> Possibilities mentioned so far: (1) Fame for attacker (2) Excuse for other problems at GitHub (3) Competitors

Not sure I find any of these is more convincing than the next or very convincing at all.

GitHub seems to be one of the least offensive businesses around but still gets attacked on a regularly. It's popularity might make it a target but don't those running botnets have better targets? And wouldn't they publicize their exploits?

And I can't believe GitHub would use it as an excuse. A DDoS attack has clear evidence that their entire devops team and many of their suppliers would have first hand knowledge of. If word leaked out it would be pretty embarrassing.

Lastly would a competitor risk destroying their entire business if found out. People are irrational and stupid but that would be crazy.

Why is this happening?

</comment></comment></comment><comment><USER14> Anyone counting how many DDoS attacks github saw this year? I don't doubt they are having issues, but at this point it seems like they are blaming every outage on a DDoS

<comment><USER15> > at this point it seems like they are blaming every outage on a DDoS

Related: <https://news.ycombinator.com/item?id=3576964>

<comment><USER16> Was about to post asking why would somebody want to DDoS Github. Your link seems like an "understandable" motive.

<comment><USER17> Extortion is one of the better possible reasons for their DDoS. Eventually, when the money doesn't show, its in their best interest to stop attacking. While new extortionists will keep showing up, they won't likely come back.

Ideological or government attackers, on the other hand, can't be reasoned with and can only be expected to escalate in the future. If that's who Github is dealing with, then they'd better figure out how to actually deal with DDoSes sometime soon, because they will only increase in size and frequency.

</comment></comment></comment></comment><comment><USER18> In light of this it would be nice if github offered a feature to push repo's to s3 or something every time they receive a push, so you could use that repo until they get back up.

<comment><USER19> You know what I want? Transparent failover somehow baked into git remotes.

        you <--> (X <--> Y)

If you push to and pull from X, and X should be unavailable, automatically start using Y. When X is pushed to, check with Y first to keep in sync. There are ways to push to multiple remotes by defining multiple remote URLs, but then it's a harder problem of dealing with downtime.

<comment><USER20> It's really easy to push to two remotes. I have a tfs service account setup and using git and also a github account. I like tfs's project and issue tracking better but want the public repo available on github.

git add remote <name> <url> git add remote <name> <url> then create a shell script that checks both remotes for the latest if one is greater than the other merge the two if they are the same then push to both. Then just always push with that script.

Just how I've handled it so far. That said a autoMagical way of handling it built into github/git itself would be cool.

<comment><USER19> As I said, it's doable with scripts and whatnot, but transparent and built into git would be awesome. I see it as something you just set up on a remote repository server. No need to make N developers aware of it, or aware of the script, or resolving the script breaking, or having a ton of implementations of said script across different teams and companies.

Doesn't help if you use GitHub for issues, of course, but hey!

</comment></comment></comment></comment><comment><USER6> It's back. I was able to watch the graph change and see it reflected in my own experience of trying to load a GitHub page. Great job on the Status site, GitHub!

I first noticed the top graph change, and a minute or two after it went back up towards 100%, they updated their status message to "Minor service outage". Now it's been moved to the history: <https://status.github.com/messages>

<comment><USER21> maybe... but I still can't push my stuff from Italy :-/

<comment><USER22> I'm still having problems pushing/pulling as well.

</comment></comment></comment><comment><USER23> Oh that explains. Seems it started half hour before they posted the update.

I was trying to make pulls just a minute ago and it was incredibly slow, but the status.github was just showing "we're investigating the high rate of drop packages".

Now, why would anyone want to DDoS GitHub? Gee...

</comment><comment><USER24> We need to agree on a date format for this kind of submissions: <https://news.ycombinator.com/item?id=5430129>. :P

</comment><comment><USER25> What can WE do about it so GitHub stops getting DDoS-ed?

<comment><USER26> Stop letting computer-illiterate people believe it's okay to run Windows when they aren't going to be responsible about updates/safe computing?

</comment></comment><comment><USER27> Does anyone have insight into that repeated spike train pattern in the exception percentage graph? That seems so predictable it ought to be fixable?

</comment><comment><USER28> No wonder.. I was wondering why the hell I couldn't push and then I thought, "well... I'm bored guess I'll read hacker news... "

</comment><comment><USER29> This is the reason I love git---distributed. I can keep on keeping on despite these silly attacks (if they really are DDoS).

</comment><comment><USER30> How about whitelisting known genuine users?

<comment><USER31> The attackers would just find such a user and spoof their IP. DDoS is a hard problem to solve, and it's a shame that so many ISPs and datacenters don't work harder to prevent spoofed traffic. On top of this, they'd still need routers and switches in front of their machines big enough to handle the traffic from the attack plus the load of trying to filter out the good traffic (this kind of hardware is quite expensive).

</comment></comment><comment><USER32> Kudos to GitHub for showing their latency and error stats. That is grace under pressure.

</comment><comment><USER33> Everyone on HN opening GitHub to read the message will surely help! :)

</comment><comment><USER34> Wow, Bitbucket must be getting desperate... Haha. This is annoying, even though it's a weekend, I've got work to do and the fact I can't pull down some changes I made on a machine at work to my home machine is highly annoying.</comment>


Utf9: python library for this u̶s̶e̶l̶e̶s̶s̶ 𝘣𝘦𝘢𝘶𝘵𝘪𝘧𝘶𝘭 encoding - <AUTHOR> https://github.com/enricobacis/utf9 <comment><USER2> With a little bit more work, you could have something you could pass to codecs.register()¹, so that you could encode and decode in the standard way:

        text = u'ႹЄLᒪo, ǃ'
        enc = text.encode('utf-9')
        dec = enc.decode('utf-9')
        assert text == dec

¹: [https://docs.python.org/2.7/library/codecs.html#codecs.regis...](https://docs.python.org/2.7/library/codecs.html#codecs.register)

<comment><AUTHOR> Nice idea, I will try to implement it ;) I will probably have to pay attention to the final call for the IncrementalEncoder. In fact I let bitarray to add 0's at the end of the encoded data to comply with these pesky 8 bit architecture ;)</comment></comment>


How to learn machine learning in the simplest way? - <AUTHOR> How to learn machine learning in the simplest way? <comment><USER2> Azure machine learning has a drag and drop interface (which allows custom code) and allows you to use it as a Web service, plus it's free. I'm currently using it and pretty happy with the results.

There's a bunch of 'learn as you go' workbooks that may be of interest to you too.

[https://studio.azureml.net](https://studio.azureml.net)

</comment><comment><USER3> Additionally, the Machine Learning Crash Course Series from Berkeley is a good overall explainer of the concepts.

[https://ml.berkeley.edu/blog/2016/11/06/tutorial-1/](https://ml.berkeley.edu/blog/2016/11/06/tutorial-1/)

I've had luck with a mixture of online courses like Datacamp ([https://www.datacamp.com/](https://www.datacamp.com/)) and finding projects to try on sites like Kaggle.

</comment><comment><USER4> [https://imgur.com/gallery/PJVud](https://imgur.com/gallery/PJVud)

Find something to play with: [https://www.reddit.com/r/MachineLearning/comments/4007ma/col...](https://www.reddit.com/r/MachineLearning/comments/4007ma/colorizing_black_and_white_photos_with_deep/)

</comment><comment><USER5> What do you mean by "simplest way"?

<comment><USER6> for those that doesn't know the math used in ML, thats what I mean

</comment></comment><comment><USER7> if you are you a #MachineLearning or #BigData enthusiastic and want to find somebody nearby to hang out or to discuss your problem, then I would say to @xoolooloo is perfect to find somebody nearby. [https://www.xoolooloo.com/#/signup](https://www.xoolooloo.com/#/signup)

</comment><comment><USER8> Be a machine :O

<comment><AUTHOR> [https://media.giphy.com/media/3pDwzu7sYmF4k/giphy.gif](https://media.giphy.com/media/3pDwzu7sYmF4k/giphy.gif)

</comment></comment><comment><USER9> fast.ai</comment>


Find what's blocking sales with under a day of work - <AUTHOR> http://blog.asmartbear.com/more-sales-customer-feedback.html

<comment><USER2> This article is great. I'm going to implement some ideas here.

Question, though. We don't offer a trial–only various demos of websites/apps.

Should I capture visitor's email before letting them see a demo?

<comment><USER3> I work for an IT firm, and I end up giving tons of sites an email to use trials, but I dislike doing so just to see a demo. For me, I prefer demos to give me an option to contact the company at the end of the demo, but I realize that this may not be terribly helpful to you as I'm not likely to contact you if the demo turned me off for some reason.

I would maybe give a place to enter your email address before the demo starts, but make it optional. You could also put a short survey at the end of the demo for feedback. I'm actually fairly likely to fill out a (very) short survey, especially if I didn't like something. The way I see it, I don't want to really risk being inundated with unwanted email from a company, but I'm willing to give my opinion and add an email contact if I think it will be used for improving the product rather than to try advertising to me.

</comment></comment><comment><USER4> I have a strong aversion to most of these suggestions. That's not how I'd like to be treated as a customer.

<comment><USER5> Which ones? Three of them involve giving you stuff for free. How can you not like that?</comment></comment>


Korean Language Test - <AUTHOR> http://www.learnkoreanwords.com/korean-language-test/ <comment><USER2> Is it the same in North Korea as it is South?</comment>


Another Poker Hack - <AUTHOR> http://mattmaroon.com/?p=334

<comment><USER2> Odd that poker sites wouldn't wait for checks to clear. I guess that would hamper impulse gambling. The traditional way to do this sort of thing:

[http://findarticles.com/p/articles/mi_m2194/is_n11_v62/ai_15...](http://findarticles.com/p/articles/mi_m2194/is_n11_v62/ai_15139850)

</comment><comment><USER3> C'mon, we're trying to rehabilitate the words 'hack' and 'hacker' to not be associated with scams, theft, and vandalism. Didn't you get the memo?</comment>


In Search of an Understandable Consensus Algorithm [pdf] - <AUTHOR> https://github.com/papers-we-love/zurich/blob/master/2017.06.29-Raft/pwl-raft-trivedi.pdf <comment><USER2> I've implemented both Paxos and Raft in class projects. They were difficult and subtle, but still an order of magnitude less difficult than the standard fare of coding interviews.

<comment> dis-sys you've never fully implemented raft or paxos. it is well known that such distributed consensus libraries can be extremely easy (a couple of days coding) to start with, but it can take ages to get it polished to the extent to be production ready. a few things to ask yourself -

1\. do you support membership change, checkpoint/snapshot streaming, automated consistency check, leadership transfer, witeness, etc?

2\. what are the design decisions made to tolerate large latency, i.e. running your raft/paxos in geographically distributed data centres?

3\. have you tried to let your "implementation" to pass any 3rd party test suite? what is that test suite? how complete it is? maybe start with the ETCD raft test suite?

4\. ever tried to test your "implementation" with Japsen or at least some monkey tests? can your "implementation" survive after very high churn rate?

5\. how about hardware errors? will that lead to data corruption? show me your tests please.

once you feel comfortable with the answers above, think about another quick question - how can you support tens of thousands of raft/paxos groups on a 3 nodes setup? what changes are required to make it efficient? extra testing required? when further scaling to many nodes, how to manage the distribution of those large number of groups?

your "class projects" is nothing else than a slightly more advanced "hello world" program - it proves one thing and one thing only - you are capable of reading a simplified paper written in English.

now compare it to coding interview again, is it still "an order of magnitude less difficult"?

<comment><USER3> I wrote a raft implementation that is used by quite a few well known systems (etcd, cockroachdb, docker swarmkit, tikv and quite a few closed source systems). Probably it qualifies the "production ready" standard :P.

With that experience, I can say that a real world raft implementation is not easy and VERY time consuming.

However, preparing the coding interview is definitely a order of magnitude more difficult for me. It requires me to waste all my time on something meaningless, and makes me feel sick.

<comment> dis-sys must be etcd then. ;)

it took etcd about 2 years to become mature on its _single_ group raft implementation. the abstraction in raft.go is pretty good, the test suite is the best I can find, message passing and tick handling is correctly handled, but to be honest, everything else (entry management, transport, snapshot streaming, request handling etc) need to be rewritten/added to further scale it to support multi raft groups. that is actually what cockroachdb did.

etcd raft is great, just want to give my understanding on how time consuming it is to write a production ready raft library.

</comment></comment></comment></comment><comment><USER4> If there's a video recording of this meetup it would, in fact, be amazing, too.

<comment><USER5> There are a couple of videos of either Diego Ongaro or John Ousterhout, his faculty advisor talking about Raft. The most recent one I saw was:

[https://www.youtube.com/watch?v=vYp4LYbnnW8](https://www.youtube.com/watch?v=vYp4LYbnnW8)

Ben Johnson also made a cool visualization:

[http://thesecretlivesofdata.com/raft/](http://thesecretlivesofdata.com/raft/)</comment></comment>


Meat – Free self hosted GitHub for Web Developers - <AUTHOR> https://getmeat.io/

<comment><USER2> Still not open source, so no thanks.</comment>


Student builds iPhone app in 3 nights. So impressed that we bought it from him. - <AUTHOR>

This summer, Tyler Bell (http://twitter.com/belltyler) contacted us at LMGTFY to ask if we wouldn't mind making the app he just built the official app for LMGTFY. He sent us screenshots, and wanted to work for free, as long as he had the oppotunity to work with us. He ended with, "either way, I'm going to finish development on this app and upload it to the App Store to promote LMGTFY."
We were smitten with his work. So much so, that we couldn't accept his offer to work for free. We instead bought the app from him, and now he's part of the team.
Moral of the story: Stop talking about doing something. Actually do it.
LMGTFY for iPhone: http://mobile.lmgtfy.com/ <comment><USER2> Good for him, he hustled well.

Quick question: what framework/library was that landing page built with (if any)? Thanks

<comment><USER3> It's a software-as-a-service called Limelight: <https://www.limelightapp.com/>

I think the people behind it hang out here on HN, I'm sure they'll chime in.</comment></comment>


Going extinct: why corporate giants die - <AUTHOR> https://www.london.edu/faculty-and-research/lbsr/going-extinct-why-corporate-giants-die <comment><USER2> Corporate Giants are a symptom of the symbiosis of corporations & governments. Governments prefer centralization because it's easier to manage centralized entities. Government investment into an industry tends to tilt it toward a monopoly.

</comment><comment><USER3> > By the 1950s, the average company lifespan was down to 60 years. Today, it > is a mere 17 years

[citation needed]</comment>


Two Startups in 9 Months: Lessons Learned as a Technical Co-Founder - <AUTHOR> http://blog.hartleybrody.com/two-startups/

<comment><USER2> >> Validating your idea.

That is a strong point. What's interesting in your article is that one was "yet another social platform" and another is a marketing-style product targeted towards artists. While artists sort of care about marketing, they really are too busy for that.

I went through the validation stage and the feedback was not only honest but highly discouraging to say the least. It's disheartening to think that I put in so much time and thought into creating a prototype that got such a cold reception. Here I thought I was solving a problem and it turned out that no one thought the idea was any good. This is where loss-aversion can quickly take over and you just keep on looking for one or two yeses to get some courage.

Then again, after dropping the idea, I can't fight the suspicion that I am a spineless quitter.

<comment><USER3> Pivot man. Just because the validation you got at first was negative doesn't mean there couldn't be a new market or a new revision that would get you positive feedback.

On the flipside, devoting time into something that no one wants would, i imagine, feel worse.

</comment></comment><comment><USER4> It was great to work with Hartley on the second project in this article, Mvsic. Too bad things didn't work out, but so many lessons learned.</comment>