Why JSON Is Better Than XML - <AUTHOR> http://cloud-elements.com/json-better-xml/

<comment><USER2> First, no one random technology can be just "better" than some other random technology.

Second, are XML-vs-JSON wars still raging?</comment>


A Simple Text Editor Foot Pedal - <AUTHOR> http://blog.mattgauger.com/blog/2013/08/06/a-simple-text-editor-foot-pedal/

<comment><USER2> This is such a cool idea, and the nerd in me loves the alternate input idea.

My problem with this, and it's such a niggly problem that's it may not even be worth mentioning, is that this will only work when you've got the footpedal around and I'd be frustrated when I sat down at a computer and didn't have it.

Like the author's note about muscle memory and wanting to hit the ESC key. I'd be frustrating to get the pedal into muscle memory and then forget it.

Sure you can learn a new system, but it's nice to know that vim is vim, regardless of the device you're connected via. Same is true for Emacs. Boy, an ALT key at my foot would spoil me rotten.

</comment><comment><USER3> Is there a way to send data back to the Teensy? That way you could have a program on your computer that tells it "hey, I'm in vim now, so use 'i' and 'Ctrl-['" or "hey, I'm in emacs now, so just be 'Left-Alt' now".

If it's "just a keyboard" and doesn't normally support that, I've seen hacks that utilize the Caps/Scroll/Num lights as inputs back into a keyboard...

<comment><USER4> I'm not sure if there is a way to send signals back to the device. But, it should be trivial to do what you describe on the operating system level. Ignoring the possibility of specifically looking for that keyboard, you could just have is send obscure keys that you would never type. Then you can just re-map them on the OS level whenever you want to.

</comment></comment><comment><USER5> Thank you for posting this, it's an awesome project for cheap. It would be nice to include my feet in my workflow and not just my hands.

It would be cool to implement the pedal as a hot-key reader. Say you press the pedal down and type 'ff' and let it go and firefox launches, or type 't' and a terminal opens.

<comment><AUTHOR> If you use Alfred/Quicksilver (or your Linux WM's equivalent) you could just map the pedal to that that key combo.

</comment></comment><comment><USER3> "One downside I found was that the Ctrl-[ combo seems to back up the cursor one character, which can be annoying."

So, technically you could just flutter the pedal to move the cursor left? :)

<comment><AUTHOR> Yes.

It's not just Ctrl-[ that does this, Escape does it too. We just typically don't toggle between insert mode / ESC in vim fast enough and enough times to notice it. With the pedal it's quite easy to press and release it a bunch of times before you type anything.

<comment><USER4> Now that I notice that behavior it is really weird. Anyway, if that is a problem/annoyance for you, then you could try switching to insert-mode by using append ('a') instead, which will bump the cursor up a character and counter ESC bumping you down one.</comment></comment></comment>


Has anyone interacted with people behind FastTrack.vc? - <AUTHOR> https://fasttrack.vc <comment><USER2> Well, color me interested. I read the T&C and didn't run into anything that seemed obviously bad. Might as well start the process and see if I run into any red flags!

</comment><comment><USER3> Looks interesting</comment>


How to Analyze Assembly to Guide Optimization Strategies - <AUTHOR> https://medium.com/adobetech/how-to-analyze-assembly-code-to-guide-optimization-strategies-3be9317c913c <comment><AUTHOR> The post discusses the optimization of bin2c ([https://github.com/adobe/bin2c/](https://github.com/adobe/bin2c/)) – a tool to embed binary files in C or C++ applications that reduces compilation times up to thirty times!</comment>


Early Signs Coronavirus Outbreak Peaking in Spain? Globally? - <AUTHOR> https://thesoundingline.com/early-signs-coronavirus-outbreak-peaking-in-spain-globally/ <comment><USER2> They forgot to say we have just achieved 6% of herd immunity, implying at least 435,000 more deaths.

[https://medium.com/@juan_marketpayio/please-america-do- not-f...](https://medium.com/@juan_marketpayio/please-america-do-not-follow- us-991939b9a5e1)</comment>


NYPD can't count cash they've seized because it would crash computers - <AUTHOR> http://arstechnica.com/information-technology/2016/09/nypd-cant-count-cash-theyve-seized-because-it-would-crash-computers/ <comment><USER2> "We're not corrupt, just incompetent"

Get external auditors to come in and look at the system. Unless the data is corrupt, there is a way of getting it into a useful and queryable form. It would be a shame for this investigation to quietly die because of prevarication from some gifted bureaucrats.

<comment><USER3> "We're not just corrupt, we're also incompetent."

</comment></comment><comment><USER4> From what I understand 'civil forfeiture' is basically a legalised shakedown, the police can stop anyone and take whatever money they have. Canada actually issued a travel advisory over it ( [https://www.techdirt.com/articles/20140915/09500928521/canad...](https://www.techdirt.com/articles/20140915/09500928521/canadian- news-outlet-warns-canadians-that-us-law-enforcement-officers-will-pull-them- over-seize-their-cash.shtml) )

So is it really surprising there's zero records kept? In other news, the dog ate my homework!

</comment><comment><USER5> Then they should stop seizing property until they can count the cash. I suspect the real reason involves a coverup.

</comment><comment><USER6> A reasonable thing to do would to stop taking people's property until they have a system to account for its whereabouts.

I'm not saying that has any affect on reality, just that it would be a reasonable thing to do.

<comment><USER7> More reasonable would be not taking anyone's property or cash until they are found guilty of a crime and the asset is shown to be profit of the crime.

Anything else is theft by taking

</comment></comment><comment><USER8> It is by no means an excuse for them, but having done some work in the government contractor sector, this is not an impossible scenario. However, the company that was contracted to build it should probably be forced to remedy that soon, or pay.

<comment><USER9> I've worked for state government. No amount of failure for any amount of money paid would surprise me.

<comment><USER10> Curious, what's the biggest amount for failure you've seen during your time in state govt?

<comment><USER9> [http://www.statesman.com/news/news/state-regional-govt- polit...](http://www.statesman.com/news/news/state-regional-govt- politics/chronic-failures-threaten-ibm-contract-with-texas/nRwMD/)

Texas had a contract with IBM to consolidate all of the servers of all state agencies. I believe it was for $863 million dollars and 7 years. The first thing IBM was supposed to was to replace any hardware that needed replacing. They never replaced anything. A ticket to reboot a server took a week. As nearly as I can tell, IBM just underbid the contract, and figured that the court battles would be cheaper than making good on the contract. The on-going legal wrangling was literally constant. We just compiled our lists of grievances throughout our workdays and forwarded them on to our manager, where presumably, they got all aggregated again and eventually made it to the legal teams.

It was pretty horrendous. You could have developers or production at a dead halt for a week literally because a server (managed by IBM) needed to be rebooted, and they wouldn't do it. Managers would call and rage. I worked at a large agency, and we eventually got administrator access to the machines, which I think we were supposed to use only for read-only access... occasionally, we'd get told to actually reboot them or something. My impression was that IBM would be logging such infractions and just try to use them as more evidence of Texas violating the contract or something. At the very least, we weren't really supposed to use the access, but we did.

I wondered how the whole agency functioned with the IT infrastructure that badly planned. If you needed a new _virtual_ machine, you'd be doing pretty well to get your request in 6 months in advance, literally. For them to click through creating when you literally _know_ they have the capacity (because we had some insight into the infrastructure.) One of my managers went to some session, and there were more than 35 process steps to getting something like that done, if I recall corretly. Any step stalls, that's a black hole; you'd better be calling and checking on requests like that regularly, or they just dead-end somewhere. (Actually, nothing you didn't call about regularly ever got done, including reboots.) I think they got that down to 15ish steps eventually.

</comment></comment></comment></comment><comment><USER11> this sounds like stonewalling to me. maybe they weren't counting on the optics of the headline's phrasing sounding as charged as it does, but this reminds me strongly of, for instance, the FBI intentionally being obtuse in their processing of FOIA requests for sensitive subjects. they're not figuring out the numbers because the headline "$x hundred million seized by NYPD from citizens in 2015" sounds even worse.

</comment><comment><USER12> Must be very convenient for the cops seizing the cash if nobody knows how much is supposed to be there.

<comment><USER13> Yes. Quite.

</comment></comment><comment><USER14> Im currently working on a project for a large refined products company that is run by IBM. Its totally plausible that they created a bad solution. The last 3-4 projects Ive worked on were run by Big 4 consulting companies, and with each I've become more convinced that they are a drain on the economy.

</comment><comment><USER15> If this were some ancient 20+ year old proprietary system that no one wanted to touch, I might buy it.

But a system built in 2012 can't provide a report on total cash seized? Honestly?

</comment><comment><USER16> Honest question here... does _anyone_ believe this explanation from the NYPD? I don't, and I find it hard to believe that anyone else does.

<comment><USER17> It's one of those explanations so ridiculous it's either blatantly false or it just has to be true.

<comment><USER16> In a way I suppose that gives such a claim from the NYPD the only shred of credibility it could have had in the first place.

</comment></comment></comment><comment> xkcd-sucks And I can't submit to a pat-down search due to my crippling social anxiety.

</comment><comment><USER18> Never attribute to malice that which is adequately explained by an improperly tuned instance of DB2.

</comment><comment><USER19> Prove it chumps. Let me see that database. It's so easy for them to lie its not even funny. Show me the money ... counting software.

<comment><USER20> Of course you can get the data, it's in a SQL database after all. The consultants only made sure to build the system so that they get paid for everything that wasn't explicitly specified (like querying by a parameter):

>It's entirely possible that PETS, which allows for automating intake of evidence, may be so complex in its database configuration that producing the data sought by the bill would require major revisions to the multi-million dollar system. However, the NYPD has also invested heavily in an IBM DB2 data warehouse operation with the help of IBM Professional Services, so in theory they should be able to perform much of the analytics off-line without "crashing" the PETS system—with a little more consulting help.

</comment></comment><comment><USER21> > The system...was built on top of SAP's enterprise resource planning software > platform and IBM's DB2 database by Capgemini in 2012, and was used as a > flagship case study by the company.

That's just embarrassing.

<comment><USER22> It delivered exactly what the client wanted.

<comment><USER23> Exactly. This should not be an issue with inventory software written in 2012. Especially not from an enterprise solution used by massive companies and governments all over the world.

I can't for a second believe that this is an accidental oversight. If this is truly a problem, then it is because one of the design requirements was to make this a problem. Either that, or it's a procedural thing, where the value of cash evidence is captured, but not in a way that the system expects for aggregations. That way, it can be retrieved, but only if you know how to do the secret dance.

They obviously have to have this information because they are required to split their funds with the Feds. They just don't want to make it widely available.

</comment></comment></comment><comment><USER24> All they have to say is "Change the column to (BIG INT) noob".

<comment><USER25> I want wondering if someone used an INT instead of a LONG somewhere but then that would be giving them the benefit of the doubt that this isn't nonsense cover-up for corruption.

</comment></comment><comment><USER26> Technically speaking, that story about crashing the system is BULLSHIT.

</comment><comment><USER27> Do they not have separate backups that can be queried?

</comment><comment><USER28> Smile of the day.:)</comment>


Ask HN: Retainer and rate question for software engineer - <AUTHOR>

I'm curious how a contracting software engineer would structure an agreement for work where there may or may not be time spent actually doing work. Simply put, it's an "on call" type arrangement for server administration and hotfix/bug fixes if the need does arrise.
I've contracted before, and have my standard rate, but have never been in a situation where I'm on call, but have the potentiality to not do any work what-so-ever.
What is the industry standard for this sort of arrangement? Thanks!
I'm also sorry if there's a similar ASK HN posting. I perused google and HN a bit and didn't find any similar posts. <comment><USER2> I generally have structured based on some average of "how many hours will you need me if everything goes terribly wrong" and "one or two small things a month" multiplied by my a slightly higher than normal hourly rate.

In cases where clients don't want to pay a retainer so to speak, I'll usually give them the option to pay as they go, but at generally 2-2.5x my normal rate.

<comment><USER3> What about a flat weekly/monthly charge for being on call, plus hourly charges for any actual work you do? Is that ever done?

<comment><USER2> hmm it might be, but I've never done that.. basically clients would pre-pay for a chunk of hours (use them or lose them) and overages would be billed at normal hourly rate

<comment><AUTHOR> Thanks for both responses. I do like the having a couple options to offer. I appreciate the input guys</comment></comment></comment></comment>


Quadcopter drone group held in London airport on suspicion of terrorism - <AUTHOR> http://www.siliconrepublic.com/innovation/item/27915-quadcopter-drone-group-held

<comment><USER2> Here's what happens next:

1\. Drones become more and more popular as kids/hackers toys

2\. Eventually a drone is used in a real terrorist plot

3\. Our politicians over react and ban drones

4\. We all write tweets and blogs about what clueless morons our politicians are because the whole thing has been widely understood since (before?) Scott Adams wrote 'The Religion War'

5\. Drones don't get unbanned for use outside of police/law enforcement because of whining from the hacker community

<comment><USER3> All while our unwanted persistent state of war rages and our supposedly but not appointed military counterparts fly armed drones to kill and name people under our name.

Fuck this planet sucks. actually no, humans suck as a race.

<comment><USER4> * supposedly but not appointed military counterparts fly armed drones to kill and name people under our name.*

Don't worry, it's all cool - they only kill the BAD guys! ;)

<comment><USER3> Indeed. They even tell us which are the bad guys so we don't have to exercise our brains!

</comment></comment></comment></comment><comment><USER5> It looks like they were only held for a couple of hours and released. Not that I think this is right, but they are flying with lots of wires/electronics/batteries.

<comment><USER2> It's crazy how the whole movie plot driven security industry has been able to associate naked electronics with bomb making. A friend was commenting last night that the bare PCB of the Raspberry Pi could be seen as dangerous/threatening, and I was speculating about how much trouble I'd get in if I tried to take Arduino based home made toys on a flight.

<comment><USER6> I think you'll be fine.

Friend of mine had a kind home-made led flashlight (small PCB and gangly wires). Friend was pulled aside for questioning and demonstrated its 'torch' capability and then let go thru. The delay was about 20 minutes though. It is silly, but if they've not studied any (even basic breadboarding) electronics, then anything out of the ordinary like that will look 'weird' and arouse caution.

</comment></comment></comment><comment><USER7> Glad to see that european airport secuirty is just as incompetent as american TSA officers.

In the future have your robots sent by post in advance of your arrival and terrorism officials will be none the wiser.

</comment><comment><USER8> They were questioned by customs because they had a lot of weird stuff in their luggage, and then were allowed through after two hours. Why is this news?

<comment><USER2> Customs don't normally call the special branch, and whilst being detained for hours is now normal practice in the US it's still (thankfully) pretty unusual in the UK.

<comment><USER8> How unusual is it, though? I imagine it's still something they do multiple times per day, given the large volume of travelers they process.

</comment></comment></comment><comment><USER2> I'm now feeling lucky that the OSHUG meeting last night, where the topic was drones, didn't get raided - <http://oshug.org/event/20>

</comment><comment><USER9> Repressive governments always fear artists.</comment>


Who Saved the Most Lives in History - <AUTHOR> http://www.scienceheroes.com/index.php?option=com_content&view=article&id=258&Itemid=232 <comment><USER2> Stanislav Petrov has to be a contender: [https://en.wikipedia.org/wiki/The_Man_Who_Saved_the_World](https://en.wikipedia.org/wiki/The_Man_Who_Saved_the_World)

</comment><comment><USER3> Infection used to be the potential killer behind any (otherwise) non-life- threatening injury. I'm a but surprised that only 82MM people are deemed to have been saved by antibiotics. Perhaps that number is for Penicillin only?</comment>


Microsoft in 2019 - <AUTHOR> https://www.youtube.com/watch?v=RWxqSEMXWuw <comment><USER2> They used javascript and that airplane crashed because of no dependency issues</comment>


An introduction to data processing on the Linux command line - <AUTHOR> https://blog.robertelder.org/data-science-linux-command-line/ <comment><USER2> If you're interested in this space, a great resource can be found at [https://www.datascienceatthecommandline.com/](https://www.datascienceatthecommandline.com/) (a free guide to go along with an orielly book)

</comment><comment><USER3> A plug of my tools:

To visualize data coming in from a pipe, can pipe it to

[https://github.com/dkogan/feedgnuplot](https://github.com/dkogan/feedgnuplot)

Very useful in conjunction with other tools to provide filtering and manipulation. For instance (the first one is mine):

[https://github.com/dkogan/vnlog](https://github.com/dkogan/vnlog)

[https://www.gnu.org/software/datamash/](https://www.gnu.org/software/datamash/)

[https://csvkit.readthedocs.io/](https://csvkit.readthedocs.io/)

[https://github.com/johnkerl/miller](https://github.com/johnkerl/miller)

[https://github.com/eBay/tsv-utils-dlang](https://github.com/eBay/tsv-utils- dlang)

[http://harelba.github.io/q/](http://harelba.github.io/q/)

[https://github.com/BatchLabs/charlatan](https://github.com/BatchLabs/charlatan)

[https://github.com/dinedal/textql](https://github.com/dinedal/textql)

[https://github.com/BurntSushi/xsv](https://github.com/BurntSushi/xsv)

[https://github.com/dbohdan/sqawk](https://github.com/dbohdan/sqawk)

[https://stedolan.github.io/jq/](https://stedolan.github.io/jq/)

[https://github.com/benbernard/RecordStream](https://github.com/benbernard/RecordStream)

</comment><comment><USER4> Command Line tools are powerful beasts (e.g. awk) and they were always central to data preprocessing. But do we need to call it now a data science?

<comment><USER5> Yeah this article is about processing text data and not any form of statistics, modeling, etc. I'm guessing they added "data science" because it's in vogue? In any case, the provided title does not reflect the article.

<comment><USER6> NLU. It relates to extracting intelligence from human language. Most of which comes in the form of text.

</comment></comment></comment><comment><USER7> Regarding more than one mentions of UUOC in this thread:

\- The original award started in 1995. Even though pentium was already out, I think it is safe to say that was the era of 486 PCs. In 2019, for day-to-day shell work (meaning no GBs of file-processing or anything like that), isn't invoking UUOC and pointing out inefficiencies an example of premature optimization [1]?

\- Isn't readability a matter of subjectivity, and that for some folks 'cat file' is more readable than '<file' or a direct use of a processing command (like grep, tail, head, etc) [2] ? (The whole stackoverflow page is fairly illuminating [3]).

[1] [http://wiki.c2.com/?PrematureOptimization](http://wiki.c2.com/?PrematureOptimization)

[2] [https://chat.stackoverflow.com/rooms/182573/discussion-on- an...](https://chat.stackoverflow.com/rooms/182573/discussion-on-answer-by- jonathan-leffler-useless-use-of-cat)

[3] [https://stackoverflow.com/questions/11710552](https://stackoverflow.com/questions/11710552)

</comment><comment><USER8> Not really where the author is heading, but I like to configure a backend for mathplot lib to render graphics in a terminal so when I am SSHed to a remote system I can get inlined plots.

<comment><USER9> Better solution: sixel-gnuplot

Shameless plug: [https://github.com/csdvrx/sixel- gnuplot](https://github.com/csdvrx/sixel-gnuplot)

<comment><USER8> Thanks, I will try that.

<comment><USER9> If you like it, share your terminal configuration!

mlterm works.

mintty had a regression, 3.1.0 may have fixed that

</comment></comment></comment></comment><comment><USER10> Here are some ways you could simplify some of the tasks in the article, saving on typing:

        cat data.csv | sed 's/"//g'

can be simplified by doing this instead:

        cat data.csv | tr d '"'

This awk command:

        cat sales.csv | awk -F',' '{print $1}' | sort | uniq

Can be replaced with a simpler (IMO) cut instead:

        cat sales.csv | cut -d , -f 1 | sort | uniq

When using head or tail like this:

        head -n 3

You don't need the -n:

        head -3

Also shout out to jq, xsv, and zsh (extended glob), all nice complements to the typical command line utils.

<comment><USER11> If you want to simplify things, don't employ "useless us of cat". Pass the file as a command arg or re-direct input. And sort has options, so the third/fourth commands can be

sort -u -t, sales.csv

However, those fail with quoted commas.

Also, head -3 is non-POSIX obsolete syntax.

Edit: I don't know why I didn't see other UUOC references initially.

</comment></comment><comment><USER12> When I was at a genetics lab, I was helping some researchers on something and spent 3 days writing a perl script, which kept failing. I sent an email to one of the guys who wrote the paper the research was being based on, and he said, why not try awk like this? With a little work, I turned 3 days of perl into a 1 line awk that was faster than anything else for the job at the time. That was an inspirational moment for the fundamental power of the unix philosophy and the core utilities in linux for me.

Good introductory article here!

</comment><comment><USER13> This is a great list and well-written. As a data professional, I use these commands all the time and my job would be much harder without them. I also learned a few new things here (`tee` and `comm`).

I was lucky that my first job was as a support engineer at a data-centric tech company, which is where I learned these. I've often thought about how to teach them to data analysts coming from a non-engineering background. This is comprehensive but clear and would be a perfect resource for training someone like that. Thank you!

</comment><comment><USER7> I'll just leave one of my past comments [1] here.

[1] [https://news.ycombinator.com/item?id=17324222](https://news.ycombinator.com/item?id=17324222)

P.S.: Not essential, but it really becomes a joy when, as a touch typist, I have turned on vi mode in the shell (e.g., with 'set -o vi'). My fingers never have to leave the home row while I do my shell piping work from start to finish. (no mouse, no arrow keys, etc.)

<comment><USER14> Haha. That's me. Once you go 'set -o vi', you can't go back

</comment></comment><comment><USER15> Huh, so it turns out that I've been a 'data scientist' for over 20 years. Who knew?

<comment><USER16> That was my first thought skimming through this too. Either every *nix admin who is aware of a few text processing tools is a data scientist, or "data scientists" are just as full of it as I've expected.

<comment><USER17> Just because a tool can be used for A, B or C, and you are an expert at using that tool for A does not imply that your expertise at using the tool for A makes you an expert in B and C.

The whole point of this article is to point out that a lot of common Linux tools can be used for Data Science like work (a significant part of which includes pre processing structured and unstructured text).

</comment></comment></comment><comment><USER18> Why people use Linux in place of *nix ?

Even worst, most of the tools (cat, grep, awk) are Unix commands, redeveloped by the GNU project in most of the GNULinux distros.

<comment><USER19> > Why people use Linux in place of __nix ?

I find it more irritating when people try to score greybeard points by saying *nix (or Unix) when it's obvious that they're talking about a Linux-only mechanism and quite possibly haven't ever used Unix (or a direct derivative).

<comment><USER20> Huh? What is specific to Linux in this post?

Also, the most popular Unix-like OS (far more than Linux) is macOS, basically the least "leet greybeard affectation" thing I can imagine. Your irritation is way off base.

<comment><USER19> > Huh? What is specific to Linux in this post?

Perhaps nothing? I was responding to the complaint in general terms.

> Your irritation is way off base.

Please allow me to feel irritated when people refer to obvious Linux things as something that's supposedly got something to do with Unix. It happens often enough.

<comment><USER20> Fair enough; I suppose it'd be annoying for people to talk about "the Unix concept of cgroup namespaces" or something like that.

I had thought that you were directly responding to the original poster.

</comment></comment></comment></comment></comment><comment><USER21> Very nice video, and I like the way you combine it with text and examples! :-) Looking forward to reading the other articles on your page as well!

</comment><comment><USER14> Very useful article. Learned a couple of new things here.

While reading the idea that I know most of this, would that made me a data scientist? Jumped at me.

But then I quickly recovered from that thought that surely knowing some of the tools someone could use for a certain domain does not make you expert at that domain.

Might just be the case of same ingredients, different recipes.

</comment><comment><USER22> This is a little more awk-ish:

awk -F, '$2 == "F" {$0=(($1-32)*5/9)",C"} {print}'

<comment><USER9> I love awk too but most people don't know much of awk. Better use regular things and keep awk for whenever you absolutely need it.

</comment></comment><comment><USER23> This is still useful information for data scientist who end up on Linux.

</comment><comment><USER24> This is also useful: [https://www.gnu.org/software/datamash/](https://www.gnu.org/software/datamash/)

</comment><comment><USER25> I wouldn't use awk for simple things such as

      cat sales.csv | awk -F',' '{print $1}'

but I'd prefer

      cut -d, -f1 sales.csv

</comment><comment><USER26> Useless use of cat detected!

 _Rememeber, nearly all cases where you have:_

      cat file | some_command and its args ...

_you can rewrite it as:_

      <file some_command and its args ...

_and in some cases, such as this one, you can move the filename to the arglist as in:_

      some_command and its args ... file

— Randal L. Schwartz ([http://porkmail.org/era/unix/award.html#cat](http://porkmail.org/era/unix/award.html#cat))

<comment><AUTHOR> hah, I knew someone would point that out (which is why I talked about it in the article).

I actually prefer useless cat because when you're prototyping a pipeline it's very awkward to use non-useless cat. You'll probably start off with something like this to observe the content of the file:

        cat something.txt

Using this doesn't work in bash:

        <something.txt

Then, continuing with useless cat to build on it you do

        cat something.txt | grep stuff

Which you can type easily from using 'up' in your terminal. But if you use non-useless cat you have to re-type the entire thing or move the cursor around:

        grep stuff < something.txt

With useless cat, you can keep adding things and check the result:

        cat something.txt | grep stuff | sed 's/"//g'

Or if you need to insert another filter before the last stage like this, you can just press "up" and insert it:

        cat something.txt | grep -v negmatch | grep stuff

I don't think there is any easily-typed equivalent workflow with non-useless cat.

<comment><USER27> If you're working with a lot of data you probably want to pipe it into head anyway, initially, so

<file head -n50 | whatever

Can be the starting command. When you no longer need the head there, just get rid of "head |".

Although I agree that the pointing out of "useless cat" is usually not particularly useful or constructive.

<comment><AUTHOR> Using head when there's lots of data make sense, but I really don't see any advantage to avoiding useless cat. Useless cat is way faster to type and make additions to. I sort of get the feeling that 'useless cat' is really just a fun copypasta kinda like when people like post "I'd just like to interject for a moment. What you're referring to as Linux, is in fact, GNU/Linux, or as I've recently taken to calling it..."

<comment><USER28> It's still good to be aware about `useless cat`, to save some CPU and I/O, when converting one liners into scripts.

</comment></comment></comment></comment></comment><comment><USER29> Good intro to data _processing_.

tsort and comm were news to me.

</comment><comment><USER30> Can somebody explain the advantage of doing it on the command line vs in Python or R? What would a practical use case look like?

<comment><AUTHOR> The most significant use case for all things command-line IMHO is _automation_. Also, I would change that from "command line vs in Python or R" to "command line and Python or R". Build a pipeline like I've discussed in the article, then pipe it into Python or R.

<comment><USER31> > Build a pipeline like I've discussed in the article, then pipe it into > Python or R.

Why not just do it all in Python or R? That way you also get something that will probably work on non-unix platforms.

<comment><AUTHOR> Over the years I've found that I usually fall into a pattern of starting with low-fidelity automation in languages like shell and slowly re-writing it over time into more higher-level languages, usually python first, then Java. This way, unimportant tasks can be automated in less than 5 min with one of these shell commands. If it breaks or has errors, no big deal. Python works well for figuring out the structure of the solution as an actual program, and then finally a language with static type checking when it _really_ needs to run without errors.

<comment><USER31> I go to Python first because it's nice to be able to single-step through the script with a debugger and monitor exactly what's happening. I also know Python a lot better than shell script so it saves me a lot of time as well.

</comment></comment></comment></comment></comment><comment><AUTHOR> Hi, (I wrote the article). A few people commented noting that I included "Data Science" in the title, but the content doesn't include any statistics or machine learning which is closer to the core definition of 'data science'. I still think the title is appropriate since any kind of low-fidelity data science task you do on some had-hoc data (log files, heaps of text, web pages) is going to start with setting up a processing pipeline that involves these commands. I could have re-named it "An intro to text processing" or "An intro to data processing", but then the people who need to see this content won't associate the title with something they're interested in, so they never benefit from it. The list of commands was chosen specifically with the question "What Linux commands would someone answering data science/business intelligence questions use?" in mind. These commands are also among the list of ones that are usually already installed on every system.

<comment><USER32> Much can be done just with awk.

My pet peeve is the "grep | awk" idiom. No, just use awk.

Awk does map/reduce, relational joins, associative memory, table lookup, and so on. Just use awk arrays, begin block, and end block.

<comment><USER9> If I am going to maintain the script myself and never tweak anything in the middle of the night, sure.

But most people don't know awk. And awk requires more awareness. I break my awk when I fix things when tired.

</comment></comment></comment><comment><USER33> Ugly UUOC (Useless Use Of Cat). Damn peoples, please i appreciate your will to share, but share good contents and stop spreading bad shell patterns....

<comment><USER34> Yes, it made the whole article useless.

<comment><AUTHOR> I assume you're just joking around, but to you and the parent comment, I'd be happy to hear any good arguments for avoiding 'useless' cat. Note that I did mentioned 'useless cat' in the article, and there is already a comment thread in this article that contains my opinions on it.

<comment><USER33> So why should we repeat ?</comment></comment></comment></comment>


Which New Career to Switch? - <AUTHOR> I have been Graphic designer over 6 years and I want to switch a new tech career but I have a strong passion for illustration also. So,
Product Design UX Design Frontend Engineer (or introduce a different kind of design's career about storytelling)
I'm right now Web Designer building on WordPress, basic knowledge of HTML and CSS. I'm struggling with study javascript and I feel really interested to enroll in Interaction Design.
Any advice? <comment><USER2> There's a brilliant book on this very subject by two Stanford professors- Designing Your Life

Here's a podcast interview with the authors, to give you a flavor of what it's about > [http://thedianerehmshow.org/shows/2016-10-03/using-design- th...](http://thedianerehmshow.org/shows/2016-10-03/using-design-theory-to- build-a-better-life)

Also, NY Times review > [https://www.nytimes.com/2016/09/18/fashion/design- thinking-s...](https://www.nytimes.com/2016/09/18/fashion/design-thinking- stanford-silicon-valley.html?_r=0)

<comment><AUTHOR> Thanks so much! I will read the book, I heard excellent reviews about this book.</comment></comment>


Ask HN: What are some of the key metrics to track after launch? - <AUTHOR>

We just launched www.jackpotbuddy.com this past Sunday. Just curious what are some of the key metrics others track. So far we are tracking:
1. Visits 2. Bounce Rate 3 Avg. time on site 4. Avg Page view per visit 5. Demographics
Anything else we should consider? <comment><USER2> First off, congratulations on the launch!

Check out Mike McDerment's talk on metrics: <http://vimeo.com/10733370>

Shortly after launch of our own app, we created a marketing table in our database. One metric we implemented immediately is to find out where users are coming from. Later on, as an example, you can join that with your user table and know where the paid users initially came from.

We also tracked how many times user logs in per day. From that, we started tracking many of the actions within our app. Now, using another table, we log almost all of the user's actions to get a better idea of how the user is flowing through the application. At the end of the day, track what you think will be useful to you and don't be afraid to track more than necessary.

<comment><AUTHOR> Thanks. We are already tracking user country. I like the idea of tracking users action. We will look into implementing some audit trails.

</comment></comment><comment><USER3> Traffic, conversion, revenue, margins.</comment>


Huge Security Flaw Can Expose VPN Users' Real IP-Adresses - <AUTHOR> https://torrentfreak.com/huge-security-flaw-can-expose-vpn-users-real-ip-adresses-151126/ <comment><USER2> The original article at Perfect Privacy: [https://www.perfect- privacy.com/blog/2015/11/26/ip-leak-vuln...](https://www.perfect- privacy.com/blog/2015/11/26/ip-leak-vulnerability-affecting-vpn-providers- with-port-forwarding/)

HN: [https://news.ycombinator.com/item?id=10636912](https://news.ycombinator.com/item?id=10636912)

</comment><comment><USER3> I don't understand how the attack works from reading this. Seems to me like it's related to UPnP.

Any idea?

<comment><USER4> It's a standard gimmick with NAT users accessing their own external IP address. Someone was in a hurry to make easy money and didn't read [http://www.netfilter.org/documentation/HOWTO/NAT- HOWTO.html](http://www.netfilter.org/documentation/HOWTO/NAT-HOWTO.html)

Notice how they haven't informed any software developers about that "huge flaw"? That's because there is no flaw and no one to blame except themselves.</comment></comment>


An addictive "murder map" of medieval London - <AUTHOR> https://arstechnica.com/science/2019/01/addictive-interactive-murder-map-lets-you-explore-medieval-london-crime/ <comment><USER2> Reading about the murders is interesting--especially since I just picked up Kingdom Come, an excellent game set in medieval times--but the map works horribly.

</comment><comment><USER3> > (It works best with Google Chrome.)

Best viewed in IE6!

<comment><USER4> > _> (It works best with Google Chrome.)_

> _Best viewed in IE6!_

This page requires a web browser that supports frames

</comment></comment><comment><USER5> If this interests you, you may also like the board game Letters from Whitechapel,

[https://boardgamegeek.com/boardgame/59959/letters- whitechape...](https://boardgamegeek.com/boardgame/59959/letters-whitechapel)

<comment><USER6> Definitely a fun game to play. Played once, I was not Jack though, I don't know if it's fun to play him.

</comment></comment><comment><USER7> the same map with much better ui experience

[https://vrcmap.co.uk/](https://vrcmap.co.uk/)

it is the source of the iframe on the uni of cambridge page the article links to

<comment><USER8> Mobile experience is terrible, cant scroll the story text

<comment><USER7> yeah that map is pretty unusable on mobile -- doesn't matter which frame/page/version

</comment></comment></comment><comment><USER9> Very nice but in Chrome for me the text that loads when a pin is clicked on often loads outside of the edge of the map and so is unreadable, and can't be dragged back in.</comment>


The Failed Race to Build the Hyperloop of the 1870s - <AUTHOR> http://www.fastcodesign.com/3059809/the-failed-race-to-build-the-hyperloop-of-the-1870s <comment><USER3> There's also the Vactrain concept. Several variants were pursued from 1914 on, with Robert Salter of RAND writing a couple of proposals in the 1970s.

[https://en.m.wikipedia.org/wiki/Vactrain](https://en.m.wikipedia.org/wiki/Vactrain)

Salter, Robert M. (August 1972), The Very High Speed Transit System, RAND Corporation

[https://www.rand.org/pubs/papers/P4874.html](https://www.rand.org/pubs/papers/P4874.html)

Salter, Robert M. (February 1978), Trans-Planetary Subway Systems: A Burgeoning Capability, RAND Corporation

[https://www.rand.org/pubs/papers/P6092.html](https://www.rand.org/pubs/papers/P6092.html)

</comment><comment><USER4> There was much early enthusiasm for pneumatic propulsion, and some lines were built. London had a working system, the London Pneumatic Dispatch Railway, 1865.[1] It could carry passengers, but was really for mailbags. Unfortunately, the elevator hadn't been invented yet, and getting the mailbags up from track level to post office level quickly was a bottleneck. Also, the system leaked air at many points. (Good rubber gaskets are quite modern - <USER2> resistant rubber (Neoprene) dates from WWII. Early systems used oiled leather. There were many early pneumatic and steam powered ideas that didn't work because gaskets and piston rings weren't good enough yet.)

[1] [http://www.londonreconnections.com/2015/londons-lost- pneumat...](http://www.londonreconnections.com/2015/londons-lost-pneumatic- railway-the-worlds-2nd-oldest-underground/)

</comment><comment><USER5> No matter what you accomplish, in America you will be remembered for being penniless and broke.

<comment><USER6> A better lesson might be "work on what interests you but don't let it become a Cause you're willing to martyr yourself for because in America, you'll be given the freedom to do just that."</comment></comment>


Sikh woman laughs off facial hair taunt - <AUTHOR> http://timesofindia.indiatimes.com/world/uk/Sikh-woman-laughs-off-facial-hair-taunt/articleshow/16565400.cms

<comment><USER2> Here's the reddit post mentioned in the article [http://www.reddit.com/r/funny/comments/109cnf/im_not_sure_wh...](http://www.reddit.com/r/funny/comments/109cnf/im_not_sure_what_to_conclude_from_this/?sort=confidence)

nice of them to exclude the link</comment>


Ask HN: Do you automate most of your routine tasks? - <AUTHOR> Right now, I have a few small tasks I need to complete every other day, it does take up some time. I could also spend around 2 weeks to code and automate whole process. Would you?<p>Also, what other tasks do you currently automate? <comment><USER2> I get bored easily with repetitive work. Generally about the third time I find myself doing something, is on the short-list to get some level of automation applied to it.

BCC is in maintenance mode, which means that my _only_ interaction with the site is customer support and moving money around once a week to cover bills and my paycheck. Everything else is automated or outsourced. Bookkeeping for sales, for example, is 99.96% automatic. (Chargeback? Bah, you always miss one case...) Bookkeeping for expenses is about 90 ~ 95% static from month to month modulo dates and numbers, so I have my VA do it for me. Order fulfillment is automatic. Routine server administration is automatic. Customer services problems are, to the extent practical, automatic.

See generally: [http://www.kalzumeus.com/2010/03/20/running-a-software- busin...](http://www.kalzumeus.com/2010/03/20/running-a-software-business- on-5-hours-a-week/)

I should _probably_ find a week or so to give BCC in the near future to make the ghost ship a wee bit less rudderless. (Like restarting content creation for it -- that was 99% automatic when I had a freelancer actively working on it, but she quit and I haven't bothered replacing her for... far too long now, it seems.)

AR and my consulting work are both less mature and, by nature, require more hands on attention. Still, you could reasonably expect that as I get more of a hang on what the business needs and where I add the most value to it, more work will get pushed off onto systems or processes.

</comment><comment><USER3> I use Moonshine for automating server configuration and Rails deployment, git aliases for shortening commonly used git commands, and zsh aliases for shortening things like "cd ~/wiki; soywiki" to "wiki".

I also spent a few bucks on an 8 GB RAM upgrade so I could just set MySQL, Postgres, MongoDB, and Redis to run on startup and not have to stop and start them whenever I am working on different apps, and I still have plenty of RAM left over for Photoshop, Starcraft II, etc.

I also spend time practicing vim tricks for things that commonly require a bunch of keystrokes, since practicing shortcuts until they become automatic is effectively the same thing as automating the task itself.

There are also a lot of things I _don't_ automate, but it basically boils down to whether or not I use them enough to make it annoying enough to make me want to automate them. For example, I know that it's possible to set up a Rails template that creates a new Rails app with all of the gems, plugins, and configuration that you commonly use, but since I only create new Rails apps once every two weeks or so it's not enough of a pain that I want to spend the time creating the template.

</comment><comment><USER4> I can't automate most of my routine tasks, and I'm not sure I would if I could: eating, sleeping, taking the bike to work, talking to people, ...

I do automate many of my computer related tasks. Since I'm a fan of the command line, small perl scripts, shell scripts and aliases do a lot of "work" for me.

</comment><comment><USER5> How long do you plan on being there? How much time will you spend on these tasks in the long run?

If the cost of performing these small tasks over some specified period of time outweighs the cost automating them, and they /are/ in fact easily automated, then I think the answer here is clear: AUTOMATE.

Tasks I've automated:

* Server/OS/software deployment

* Daily calculation of linear regression of email queues on my mail servers

* Webcam snapshots and automatic uploads - [http://www.charleshooper.net/blog/automating-webcam-snapshot...](http://www.charleshooper.net/blog/automating-webcam-snapshots-and-uploads-to-flickr/)

* Most tasks requiring data entry - <http://news.ycombinator.com/item?id=2359287>

* And lots and lots of other things

</comment><comment><USER6> Depends on the definition of "some time" compared to the 2 weeks (solid?) it would take the automate. And how long you expect to continue doing said tasks.

That said, I would always err on the side of automating - at least some of the tasks. Take the low hanging fruit and tackle those first.

For example, I send out a weekly report containing a bunch of metrics sourced from our website monitoring tool. It took me a couple of days to automate that, but it saves me a good hour every week (and means I don't make mistakes copying and pasting figures which I used to with alarming regularity). It also means that my Monday mornings can start immediately without pounding through a repetitive task.

<comment><USER7> Exactly. Unless you have what "some time" is, how can you possibly justify something. It all boils down to ROI and risk (including mental sanity).

</comment></comment><comment><USER8> practice, practice, practice

The more you automate routine tasks, the faster you'll become. Don't just do a calculation of:

# of times to do a menial task * amount of time to do menial task <> amount of time to automate

That oversimplifies and doesn't take into account how much faster you'll be at scripting etc. in 20 years if you take every opportunity to hone your skills. Think of it like touch typing; might slow you down at first, but you're going to more than make up for it by the end of your career.

</comment><comment><USER9> Yes I do. For example, I have a little script to install Drupal modules, which is nothing more than downloading the tarball from the Drupal website, and untarring it in the proper directory.

Likewise, I've got scripts to extract strings to translate (with gettext()) using xgettext from various project directories, and merging it with older, already existing translations; and for installing edited translations afterwards (with msgfmt).

I also have a script to upload files that were changed locally to a remote server.

No, it didn't take me 2 weeks to code them.

Most of the time I just do the task once, and store the commands in a file. Next I edit that file replacing values with script arguments. Tada! Instant script. For lists of values that depend on the project, I create presets (one argument determines what set of values to use).

<comment><USER10> You should check out drush - "drush is a command line shell and scripting interface for Drupal":

<http://drupal.org/project/drush>

</comment></comment><comment><USER11> I automate as much as I can. Sometimes I'll even "automate" a one-off task, simply because writing steps down in a shell script makes it easier to review what is about to happen and avoid trashing, say, a big chunk of my digital photo album... :-)

Automation doesn't just save time; it also avoids many mistakes caused by fat- fingering things or forgetting step 6 of an 8 stage process.

Also, once you have automated something, you've (at least potentially) created a tool you can share with someone else, which is good for teamwork, delegation and continuity in a work-place.

</comment><comment><USER12> I would, and I do. Coding is much more fun than doing repetitive tasks, and once it's done you are just so much quicker in completing that particular task. This is true even for very simple tasks. For example I have to copy some files to a network share once a month or so. It does not take much time to do manually, but I had to open a file explorer, search the source folder, ctrl+a, ctrl+c, search the destination folder and ctrl+v.

Now it's just a doubleclick on a .cmd file on my desktop.

</comment><comment><USER13> I try to automate as much as I can. I also use these little programs/scripts as opportunities to learn something new. Examples:

1\. We had a support portal at work which I had to check regularly during the day. I wrote a small program that monitors the relevant page and pops up a taskbar notification when something new arrives. Used the opportunity to learn about taskbar notifications and a little WPF.

2\. I sometimes get database backups from customers and have to restore them on my machine, which means opening SQL Management studio, clicking and right clicking a bunch of things. Then I need to add myself as user, change a couple of settings before I can use the database for anything. I wrote a small utility so I can right click on any .bak file in explorer, choose Restore database and update the relevant tables so the database is ready to use right away with the connection string on the clipboard. Also used it as an opportunity to learn about windows 7 taskbar progress bars.

Basically I like doing small projects, but I never finish them unless they really do something. So, automating stuff I do makes much more sense than learning by following a tutorial that builds something useless.

</comment><comment><USER14> I used to, until I spend a whole day on a vim script that, realistically, only would save me significant amounts of time by the time the work I was automating would bring in 1000's of dollars a day.

Nowadays I only do it after it's been proven (or I'm 100% certain up front) that a certain task will come up more often, and that letting someone else do it is not feasible or cheaper.

</comment><comment><USER15> I have started automating a lot of the smaller, repetetive tasks. It doesn't save a huge amount of time, just removes small annoyances and friction. One example is that I'm new to OSX, so I was playing with Automator and have used it to create a command which brings up my entire development environment. This saves only 1.5 to 2 minutes each time, but it removes that friction when I get home from work I can just put my laptop on the sofa and have the dev environment load while I make a drink.

At work we have to create performance reports monthly for a particular client. The procedure has always been to do it manually, which takes around 4 to 5 hours. Writing the code to automate this in C# took less than 4 hours and now the reports take 2 to 3 minutes each month. Time well spent, money definitely saved.

</comment><comment><USER16> "We will encourage you to develop the three great virtues of a programmer: laziness, impatience, and hubris."

\-- Larry Wall

Task automation encompasses all three virtues:

Laziness - "I don't want to do this task every (other) day"

Impatience - "I don't want to spend time doing this task"

Hubris - "Of course this will work on the first try"

</comment><comment><USER17> I automate all the small, repetitive tasks I can. Even when I'm heavy into coding (where I'm working with JS/php/db all day long every day), I'll have a script which opens my terminal and editors and the db etc. so that my day starts with a ./startEd.sh command.

Right now I'm setting up a staging server for my new webapp (I'm frantically working towards a private beta). This is new learning for me- and I'm writing a script for everything I do (that I can). This not only sets up a way for me to do setup/config tasks automatically, so I don't make typos, etc.), but also _documents_ what I'm doing. I hadn't considered that aspect of it before I started writing these scripts.

</comment><comment><USER18> I'd automate everything if given enough time. To me it's the essence of running an efficient business (and indeed scaling). Some examples:

1) I automated the creation of Google Adwords CPC keywords for my ecommerce website. Based on attributes of the product, the algorithm creates 500 unique keyword combinations. Over the coming days I will automate the creation of entire campaigns.

2) I automated gathering information about 200 universities by hiring a smart virtual assistant.

3) I automate my customer support emails by using canned responses.

4) I automate the running of my unit tests so that they run whenever I save relevant files.

5)I continue to automate my code base by writing new layers of abstraction which call lower layers intelligently.

<comment><USER19> I would love you know how you automate the creation of Adwords keywords.

</comment></comment><comment><USER20> I try to automate tasks in a hierarchical, re-usable fashion, based on log file output, and always in a pragmatic, least-cost approach. No-one is paying me to make my life easier.

For example, I'll find myself performing tasks X, Y, and Z quite a lot. I'll find out that I have 20 minutes free and I'll figure out the most common denominator of the tasks, let's call it A, and then implement it. I'll quickly re-write the high level processes X, Y, Z and to use A and then move on with my life. The up-shot is that future tasks B, C, D, E, F can all use the now- battle-hardened procedure A, and maybe others in my company will find A useful.

</comment><comment><USER21> Two weeks are quite some time for an automation. But depending on how much time you spend every day, you might want to go for it. It's not only about saving time daily, it's even more about scaling things up.

If the process is automated, it should be quite easy to use modified versions for similar problems. And you can quickly run it again and again without having to worry about making mistakes.

</comment><comment><USER22> Yes, server tasks, deployment tasks. My daily routine 'feels' automated to a large extent and I'm still trying to optimize for output.

When I say my daily routine feels automated I mean, when I have two boring tasks I usually combine them to make them less inane. So an automata type scenario comes into play. I don't have to 'think' about the two crappy but necessary tasks.

</comment><comment><USER23> While I automate some computer related tasks, I can't automate house chores, working out, etc.

So I wrote an app to remind me: <http://i.imgur.com/exzb0.png>

I find this sort of thing most useful if you have chores that you want to do regularly, but not on a specific day (i.e. clean every Tuesday).

</comment><comment><USER19> Using Automator on my mac I have created a few simple folder workflows. For example I have one that everything I drag into it gets printed. This seems simple but its great when you constantly get a few emails a day with multiple attachments that need printing. Saves a few seconds every time.

</comment><comment><USER24> I have similar experiences with mopoke.

I've automated a lot of small tasks lately and as a result I have a small library of stuff which makes it easier and faster to automate yet more small tasks. It definitely is worth the invested time.

</comment><comment><USER25> Check out <http://sikuli.org/>. It's visual based programming for automating task. I use it all the time and it's super fast to set up.

</comment><comment><USER26> It is sad though, that we can automate software hassles with such ease, but we can't automate real world tasks yet.

</comment><comment><USER27> I automate a lot of excel and outlook related work</comment>


The Browser Is The New Operating System - <AUTHOR> http://techdirt.com/articles/20080530/0022021266.shtml <comment><USER2> I have been very impressed by the progress of web browsers. However, they simply don't replace an operating system until they start acting like the keepers of an entire ecosystem of applications. Browsers are still way too careless.

For example, why should a modern browser allow me to immediately close a window with a web app running (the equivalent of an OS' "shut down without warning")?

Why do browsers even _show_ a page's "reset" buttons (also known as "destroy everything with no Undo")?

And the Browser Operating System can still do the equivalent of "kernel panic" by taking out 14 windows (which aren't really windows, but web applications) when it dies.

The closest I've come to "nice" is OmniWeb on my Mac, where they have excellent per-site preferences. This at least gives me some control over each web application, as if the browser were an OS. It isn't enough, but it's a good start.

<comment><USER3> > why should a modern browser allow me to immediately close a window

> Why do browsers even show a page's "reset" buttons

Because it's better this way. The needs of users trump the needs of webapp developers. A browser that won't let me close a window, or refresh a poorly loaded page? No thanks.

<comment><USER2> I didn't say it wouldn't close the window, only that this wouldn't be immediate. For example, it could confirm before throwing away the entire state of a web application.

I wasn't referring to the ability to refresh a page, I was referring to the reset button of a form. That button destroys all the data the user entered, which is similar to an unwanted close.

</comment></comment></comment><comment><USER4> I still wonder if this is a solution hunting for a problem. I'm pretty darn happy running my OS of choice.

What burning problem am I having that I need an OS in my browser? Some kind of way to store documents like Google Gears, sure, I'll take that.

</comment><comment><USER5> Paul Sowden from Meebo gave a talk about client side storage at the second SF JS meetup: <http://www.slideshare.net/idontsmoke/client-side-storage/>

Interesting takeaways: more clients have some form of client side storage enabled than have cookies enabled. Also, nobody uses google gears.

</comment><comment><USER6> One things operating systems do that I wish browsers would emulate is per- process memory management. Even with the excellent work on memory management in Firefox 3, long-running javascript apps can fragment the browser's heap such that memory can't be reclaimed after the page (but not the browser) is closed.

<comment><USER7> Why not put each browser window or tab in a separate process? There's probably a reason, but none come to mind at the moment.

<comment><USER8> Correct me if I'm wrong but doesnt' WebKit already do that?</comment></comment></comment>


Alabama judge orders poor defendants to give blood to avoid jail - <AUTHOR> http://www.slate.com/blogs/the_slatest/2015/10/20/alabama_judge_orders_poor_defendants_to_give_blood_to_avoid_jail.html <comment><AUTHOR> Courts charge money for court fees and fines. In many cases, if you don't pay those fees, you could be imprisoned. In an effort to increase blood donations, the court temporarily accepted blood donation receipts in lieu of cash.

I don't have a problem with this. In effect, the court is paying people (by forgiving debt) to donate blood.

It sounds like a) it was poorly worded by the judge and b) people didn't understand. He _should_ have said, "If you're unable to pay your court fees or fines today, you may be imprisoned." (Something he probably says every day.) Then, "Please note that we're accepting blood donation receipts in lieu of cash for court fees and fines today."</comment>


Madras and Muslin Meet Europe - <AUTHOR> https://www.laphamsquarterly.org/roundtable/madras-and-muslin-meet-europe <comment><USER2> A bit overwrought in tone, but fascinating.</comment>


VAXen, my children, just don't... - <AUTHOR> http://crash.com/fun/texts/vaxen-dont.html

<comment><USER3> The really funny thing for me, is that I remember Monday the 19th of October very differently. Sun had called together the biggest press conference they had ever had, they were announcing that Sun and AT&T were going to work together to jointly build one true standard UNIX which everyone would run, with interoperability and reliably across the spectrum of computing. And that AT&T was putting 1B$ into Sun and had options to put another B$ in.

Eric Schmidt, who was the lead on that press conference, and Bill Joy who was his technical backup, were really confused why all during the conference reporters kept running out of the conference room to make phone calls. They didn't believe that their announcement was _that_ big but everyone was clearly quite agitated.

The clipping service didn't find a single major daily that covered that press release that day. Magazines that had it and were embargoed went with it. But it got little to no coverage.

<comment><USER4> > Sun and AT&T were going to work together to jointly build one true standard > UNIX

Which worked, but technically everything was SysV-flavored afterwards, which in many regards was a huge backward step IMNSHO. Sun OS was a joy to work with before that, not so much after.

Also it was sad to see NeWS go -- and speaking of pain: motif.

<comment><USER5> > Which worked, but technically everything was SysV-flavored afterwards, which > in many regards was a huge backward step IMNSHO.

I'm familiar with the history of System V and BSD as the two competing "flavors" of UNIX, but I've never seen a clear description of what "SysV- flavored" really meant in terms of regular interaction with the system. Since you've clearly had some personal history with this change, could you elaborate on what "SysV-flavored" actually meant for SunOS?

<comment><USER4> Well, AT&T/SysV regarded themselves as the 600 pound gorilla in the room, and Sun as the young upstart punk kid, so in building a joint system, everything "compromised" by just doing it the SysV way.

/bin & /usr/bin was populated by SysV tools with SysV options, with the particularly incompatible command line options of "ps" that plague it and clones of it to this day.

There were still Sun/BSD bin tools hidden away somewhere for use by the desperate, but they were deprecated and weren't in the search path of one's coworkers etc. etc.

I remember there being a lot of pressure to adopt SysV "streams", which from 20,000 feet sounded very powerful but were actually a nightmare -- not according to me, since I managed to avoid them, but according to later accounts of people I knew at AT&T.

And of course there were similar changes to libraries and man pages and /etc and init and etc.

I cut my teeth on BSD, contributing in small ways to its earliest development, and I considered it an improvement over vanilla Bell Labs/AT&T Unix -- not just NIH; BSD introduced the fast file system and demand paging and the "more" command (later incarnation "less").

AT&T on the other hand responded with a _very_ strong case of Not Invented Here; they really seriously did _not_ want to retrofit improvements like that. And tried to do various clones, like "pg" to replace "more", which was so awful it couldn't even page backwards, just to give some idea.

System III had a few new features but mostly sucked, due to that attitude, and System V was more or less more of the same.

Sun OS took all of the BSD improvements (ok, 7 options to "cat" is arguable, but whatever) and added more nice stuff, like NFS and RPC (RPC got a lot of flak, but originally there really was no alternative) and "Yellow Pages" (later renamed to NIS/Network Information Services for trademark reasons).

NeWS was a breakthrough in windowing, based on postscript rendering to screens (although not the same as Display Postscript), and was quite brilliant.

In creating a common standard, the compromise was to throw away or hide the majority of the BSD and Sun OS improvements in order to, basically, just let AT&T do things their own inferior way.

NFS was already becoming an industry standard, so it didn't go away, but NeWS did, in favor of the incredibly bad "Common Desktop Environment" GUI standard, which was a step backwards relative to all X11 offerings, not just relative to the breakthrough NeWS.

The alternative, Motif, had a ridiculously complex and clumsy API that took vast amounts of boilerplate programming to get anything done. Dark times for GUIs.

[http://en.wikipedia.org/wiki/Common_Desktop_Environment](http://en.wikipedia.org/wiki/Common_Desktop_Environment)

[In my first draft I mixed that up with Open Look, a sign of fuzzy memory]

I haven't thought about it in ages, so there are likely a bunch of other things I'm not remembering, but the net result was a standard that was far less pleasant to program to or to do system administration on -- plus many instances of dual standards.

Chuck above was hit by similar nastiness in the kernel itself; I wasn't dealing much with kernel internals at the time, but from what I heard, it was perhaps even worse there.

If Sun OS had merged with Plan 9, that would have actually advanced the world, rather than retarding it.

<comment><USER6> > NeWS was a breakthrough in windowing, based on postscript rendering to > screens (although not the same as Display Postscript), and was quite > brilliant.

Although the idea of building apps in PostScript is the stuff nightmares are made of, I recognize it was indeed brilliant. It was partly the work of a young James Gosling.

I remember CDE was heavily, although indirectly, influenced by Microsoft through both HP and IBM. CDE windows and menus are just 3D-like versions of Windows 2 windows.

</comment></comment></comment></comment></comment><comment><USER7> A bit of explanation: a VAX 750 is about the size of a large washing machine. It does about 1 MIPS, and can handle a surprisingly large number of serial terminal users simultaneously due to the I/O design. An RA60 is a hard disk unit, but not the sealed "Winchester" technology that IBM was promoting (and certainly won). Each disk pack could store 200 or so MB on several 14 inch platters that were held in a clear pack. It was frightening, expensive, and remarkably rugged.

By the time I was part-time sysadmin of one in 1992 or so, they were totally obsolete, and my 486DX-33 running Linux was faster at everything... except effective serial I/O for multiple users.

<comment><USER8> By the time you were a sysadmin of a 750, in 1992, several generations of newer VAX chipsets had come and gone.

The later VAX CPUs were considerably faster than a 486.

<comment><USER9> Not that overwhelmingly faster. The early 90's were the swan song for the mainframe. ECL logic was still faster than CMOS, but not by that much, and transistor density was such that multi-chip CPUs still made sense for many applications, but not all. The VAX 9000 was contemporary to the 486/33, ran at about twice the clock speed, probably had 2-3x the net IPC for typical integer code, had a vector unit (not entirely dissimilar to modern SIMD stuff like SSE/AVX), and was available in configurations up to 4-way SMP.

That's a much beefier system, surely. But it's not in a different league. And the contemporary RISC boxes were beating the VAX pretty badly in single- processor benchmarks already.

<comment><USER8> I meant literal CPUs, in the sense of a microprocessor chip. The ECL VAXen were not killed off by RISC or 486es. They were killed off by CMOS chip-based VAX implementations.

The last generations of VAX microprocessors were about 90% as fast as the giant ECL suckers at a tiny fraction of the cost/heat/size. DEC continued to manufacture and develop them for several years after the end of ECL.

The VAX chips that killed the 9k sat in desktop/deskside chassis just like a contemporary RISC system. Literally: you could buy an identical box with either Alpha or VAX processors.

(Besides, the giant ECL VAXen were always a niche product. Nearly all the VAX systems sold were based on VLSI and microprocessors.)

<comment><USER9> Oh sure, but the 9000 was DEC's contemporary flagship against which that 486 would have been competing. And my point was that it was faster, but not immensely so -- maybe 5x, or about the same speed as the Pentium 133 that the PC owner would be buying within 3 years.

My point wasn't that VAX as an architecture was dead (it wasn't, though it would be so soon), just a reply to the contention that the line had an insurmountably large performance lead against commodity CPUs. It didn't really.

</comment></comment></comment></comment></comment><comment><USER10> In case the date at the end fails to register, the reference (19 October 1987) is to [http://en.wikipedia.org/wiki/Black_Monday_(1987)](http://en.wikipedia.org/wiki/Black_Monday_\(1987\)) .

<comment><USER11> And is it true, was this the cause? Or is this fiction?

<comment><USER12> I don't know if the rest is fiction, but it didn't cause the crash. Some traders I know have told me that many knew they were wiped out before futures/markets opened.

These can help: [http://en.wikipedia.org/wiki/Black_Monday_(1987)#Timeline](http://en.wikipedia.org/wiki/Black_Monday_\(1987\)#Timeline)

"In 1986, the United States economy began shifting from a rapidly growing recovery to a slower growing expansion, which resulted in a "soft landing" as the economy slowed and inflation dropped. The stock market advanced significantly, with the Dow peaking in August 1987 at 2722 points, or 44% over the previous year's closing of 1895 points. On October 14, the DJIA dropped 95.46 points (a then record) to 2412.70, and fell another 58 points the next day, down over 12% from the August 25 all-time high. On Thursday, October 15, 1987, Iran hit the American-owned supertanker, the Sungari, with a Silkworm missile off Kuwait's main Mina Al Ahmadi oil port. The next morning, Iran hit another ship, the U.S. flagged MV Sea Isle City, with another Silkworm missile. On Friday, October 16, when all the markets in London were unexpectedly closed due to the Great Storm of 1987, the DJIA closed down another 108.35 points to close at 2246.74 on record volume. American Treasury Secretary James Baker stated concerns about the falling prices. The crash began in Far Eastern markets the morning of October 19. Later that morning, two U.S. warships shelled an Iranian oil platform in the Persian Gulf in response to Iran's Silkworm missile attack on the Sea Isle City.[8] [9]"

[http://en.wikipedia.org/wiki/Talk%3ABlack_Monday_(1987)](http://en.wikipedia.org/wiki/Talk%3ABlack_Monday_\(1987\))

"The link at the end, which leads to a story about a VAX machine getting into trouble, might be erroneous. I suggest this because I've seen the story before without the line at the very end linking it to Black Monday (in fact, I always thought the event took place significantly prior to 1987). While it's possible that somebody might add this last line to add dramatic emphasis to the story, it's unlikely that it would be cut out by somebody - it's forms a punchline to the story. Therefore I think the link between this VAX story and Black Monday is probably incorrect. Note: the joke link in question (*Alleged computer mishap) was removed 2005-11-05 RaulMiller 00:20, 22 November 2005 (UTC)"

</comment></comment></comment><comment><USER13> I haven't laughed this hard in days:

 _The fire alarm klaxon went off and the siren warning of imminent halon gas release was screaming. We started to panic but the data center manager shouted over the din, "Don't worry, the halon system failed its acceptance test last week. It's disabled and nothing will happen."

He was half right, the primary halon system indeed failed to discharge. But the secondary halon system observed that the primary had conked and instantly did its duty, which was to deal with Dire Disasters. It had twice the capacity and six times the discharge rate._

<comment><USER14> I was nearly the victim of a situation like this in the late 90s. There was a power failure site-wide and the generator failed to start in time (water in the fuel). There were three of us in the machine room at the time of the power incident. Unfortunately due to some insane security requirements, the doors lock in a failsafe situation (this was a UK MoD facility) voiding all health and safety laws.

One of the idiots decided to light up a cigarette whilst we were sitting in there in the very dim emergency lighting. Turns out the power system for the halon was still active and the 60 second alarm went off.

Fortunately we hit the stop button on the wall to prevent it being dumped on us.

New pants all round that was.

<comment><USER15> Apparently you would not have died: [https://en.wikipedia.org/wiki/Halomethane#Fire_extinguishing](https://en.wikipedia.org/wiki/Halomethane#Fire_extinguishing)

<comment><USER14> Indeed, but the experience is not one you'd want.

</comment></comment></comment></comment><comment><USER16> _I smiled and said, "No sweat, I'll train you. The first command you learn is HELP" and proceeded to type it in on the console terminal. So the data center manager, the shift supervisor and the eight day operators watched the LA100 buzz out the usual introductory text. When it finished they turned to me with expectant faces and I said in an avuncular manner, "This is your most important command!"_

 _The shift supervisor stepped forward and studied the text for about a minute. He then turned with a very puzzled expression on his face and asked, "What do you use it for?" Sigh._

This feels like every time I've tried to explain some bit of technology to a non-techie.

</comment><comment><USER17> I actually clapped and said "Oh my god!" when I got to the last line. That was a great story.

Also, here is what a Halon discharge looks like: [http://youtu.be/2fyGGqgVzCY?t=1m36s](http://youtu.be/2fyGGqgVzCY?t=1m36s)

</comment><comment><USER18> So, did the story have anything to do with the punchline or was it a coincidence? And what happened to the trillion dollars?

</comment><comment><USER19> I enjoy stories like these. I read them throughout my high school years (late 90's, early 2000's).

Now I've been in industry for close to a decade, sometimes I feel like I missed the golden age of computing.

<comment><USER15> I felt the same. When I was a PFY, I hung out with the then middle aged mainframe technicians, and they had lots of fantastic stories like this from the early days of computing.

Stories about unbalanced hard disk units wandering around the room to be found blocking the door the next morning. HD crashes where you had to vacuum up metal shavings before replacing the platters (which were the size of a washing machine drum). Amusing tales of the little vacuum hose that was supposed to automatically suck the lose end of a tape spool into the drive going wrong.

Now I work almost exclusively with virtual machines, which makes such hardware failures much less exciting.

</comment></comment><comment><USER20> That was the first day of my honeymoon, in Aruba. The hotel had a local four page paper that came out at noon, so I saw it when we checked in shortly afterwards. Boy was I glad not to be in the office of the (investment company) where I worked then. And they said you could never get fired for buying IBM...

</comment><comment><USER21> What was the barrel for?!

<comment><USER22> Hauling away the destroyed parts, I presume.

</comment></comment><comment><USER23> Cool story but I read the date was added afterwards and was removed from wikipedia in 2005 when this was realized: [http://en.wikipedia.org/wiki/Talk%3ABlack_Monday_(1987)#Link...](http://en.wikipedia.org/wiki/Talk%3ABlack_Monday_\(1987\)#Link_at_end_possibly_wrong)

</comment><comment><USER24> I thought that halon systems where supposed to have interlocks so that they could physically not go off if anyone was in the room.

The dinosaur pen batteries seem to have been done on the cheap though - <USER2> understated how the power coming back took out the UPS though.

<comment><USER25> The power came back at the same time as the generators came on. I guess this doubled the, uh, voltage, and threw the phase off... yeah.

Edit: clearly the problem was that too much amperage was pushed! :)

<comment><USER24> You don't just connect the two in series :-) In fact the best practice is not to switch from your backup supply back to mains automatically just in case the power goes out again.

How its should work is mains or your gas turbine are used to charge the batteries the power is always coming from your battery room.

<comment><USER6> I am not sure this was best practice in 1987. We learned a lot from incidents not entirely unlike this one and I went through one catastrophic power failure (that took down more than 1000 machines in the late 1990s) due to a cascade problem between the multiple "redundant" power sources.

One of the machines refused to come back and we got a call from operations asking where (physically) was the box within the datacenter. It was inside a Cubix blade-like (we didn't call them blades at the time) chassis.

<comment><USER24> It's best practice according to my 1948 GPO telecommunications handbook power always comes from the batteries.

There is an interesting write up on the register about how a London based colo facility does its UPS here

[http://www.theregister.co.uk/2014/02/21/city_lifeline_london...](http://www.theregister.co.uk/2014/02/21/city_lifeline_london_colocation_data_centre/)

<comment><USER6> I agree it's the only sane approach, but I have seen a lot of insanity in computing. It's not impossible telecom best practices were ignored for data processing until it became obvious computers are every bit as critical as telephones.

</comment></comment></comment></comment></comment></comment><comment><USER26> is this fiction? if not, then why isn't the causes list of Black Monday including this? [http://en.wikipedia.org/wiki/Black_Monday_(1987)](http://en.wikipedia.org/wiki/Black_Monday_\(1987\))</comment>


The Pentagon's recipe for brownies is 26 pages long - <AUTHOR> http://reason.com/blog/2010/05/17/how-to-make-brownies-pentagon <comment><USER2> It's not a "recipe" for brownies. It's a specification for a very specific food item (actually, two items: a brownie and a oatmeal cookie) which form part of an MRE, which has to be shelf-stable for a very long time.

One might similarly point and laugh at NASA specifications for hand tools used on the Apollo program. Yeah, a screwdriver is a screwdriver, but there's a world of difference between the $3 one you can pick up at Sears and one that's designed to be usable under extremely harsh and unusual conditions _in space._ Similarly, there's a world of difference between the pan of Duncan Hines brownies you might make when you want a chocolate fix, and a supply contract for millions of brownies that are going to go into MREs.

The thing is interesting because it provides a window into government procurement, which is something most people really don't understand, but it's not meant to be, and really isn't, humorous.

(That said, however, the spelling error in the title is pretty bad.)

<comment><USER3> A certain level of boilerplate and redundancy _is_ humorous. I'm wondering if all of their food specifications contains that "All ingredients shall be clean, sound, wholesome, and free from foreign material, evidence of rodent or insect infestation, extraneous material, off-odors, off-flavors, and off- colors." (quote from the linked pdf), or just this one. Maybe they should define some general rules, after that a brownie recipe is really only one page.

<comment><USER4> The reason that's there is historical instances of profiteers selling substandard food to the military.

<comment><USER3> Obviously the quality should be fine, that should be the default, no need to repeat it in each and every recipe.

<comment><USER5> Clearly you've not spent any time in the world of defense contracting. The default is to put absolutely every last thing in the contract.

If your contract doesn't say, "You can't put sawdust in the brownies," there will be a contractor who says, "What? You didn't say I couldn't."

You might laugh. You might suppose there's some common understanding of quality that everyone can deal with. Especially when it comes to brownies. To some degree, I'd agree: it's a defensive attempt to avoid problems encountered in the past, and maybe an overreaction.

The thing is, though, with weapons systems, the equivalent of putting sawdust in brownies happens all the time. The contractor doesn't live in Alaska, or on a carrier deck, where the thing will be deployed, and if the contract doesn't _specifically_ mention cold or fungus or salt corrosion, it'll get overlooked. Worse, the contractor _really_ has no experience in a combat zone, and has no idea what sort of abuse their equipment will undergo. The contract has to say.

The defensiveness comes partly from the fact that these systems get a single iteration. There's some back and forth during testing and verification, and there are always contracts for upgrades and improvements. But by and large, you really need things to work right the first time. Hence the heavy reliance on contracts and standards.

They're like an API telling the civilian contractor what's required in a battlefield environment. He really doesn't know things that soldiers consider obvious.

<comment><USER3> I meant that they should write down these trivial things _once_ in a standard, and later just refer to it in every recipe in two words. This spec already contains references to several federal and military standards, but clearly not enough. Thats why it's such a bloated piece of sh... paper.

<comment><USER5> Ah. Well, in this case, they kind of did that:

 _3.2 Ingredients. All ingredients shall be clean, sound, wholesome, and free from foreign material, evidence of rodent or insect infestation, extraneous material, off-odors, off-flavors, and off-colors.

3.2.1 Sugar. Sugar shall be white, refined, granulated, cane or beet sugar. Powdered sugar of equivalent quality may be substituted for part of the granulated sugar in the brownie formula to control spread.

3.2.2 Oatmeal. Oatmeal shall be the commercial product known as quick cooking oatmeal. It shall have natural rolled oat flavor and odor and shall be clean and free from burned particles, rancid, musty, sour, or other undesirable flavors and odors.

. . . _

They factored out the "all ingredients" part of the standard.

You know, I'm sorry to admit, but I've written these things before. There's a real tradeoff to make there. On the one hand, there's a virtuous impulse to do as much factoring as you can so you don't have to write things twice--or heaven forbid, update them multiple times when they change. On the other hand, you then find yourself calling for MIL-STD-340985 sugar and MIL-STD-320598 flour and MIL-STD-24098 molasses with deviations allowed to items 4, 5, and 9 of paragraph 6 when using process B of MIL-STD-32409 Baking Instructions. And there's sort of a desire by users of the document not to need to have twenty books open at once just to figure out if they can use this flour or not.

Writing this sort of thing well is a lot harder than you would think.

Edit: Upon reflection, there's a real parallel between that and walking the fine line between copypasta code and spaghetti abstraction.

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> A 26-page Brownie recipe is perfectly fine with me (I have to take advantage of the chance to echo 'Heck of job, Brownie!'). Compare/contrast with the inefficiencies in communications you see in businesses and organizations everywhere. I've never seen a simple project that, when done in Corporate America, doesn't require 6 months, reams of documents, and hours of meetings.

In other words...in large institutions, nothing is very efficient.

<comment><USER7> True to some degree, but it's also about having a level of detail needed for something that many people will rely on.

Give a simple brownie recipe to enough people and one them will burn down the house because you never specifically said to turn off the oven when done baking.

<comment><USER8> But give a 26 page brownie recipe to enough people and one of them will burn down the house because they missed the instruction saying when to turn off the oven.

There is a common organizational failure mode where new instructions are added whenever anyone messes up, ignoring the fact that the complexity of the instructions is leading to lots of mistakes.

<comment><USER9> Incorrect. The endeavor itself (making long shelf life brownies/cookies) has a set of requirements which is pretty exacting. The complexity of the instructions will likely be turned into long checklists and procedures for the actual producing plant. And if the plant screws up said brownies, they'll likely be on the hook to replace the whole lot of MREs

You're confusing quality/policy issues in corporations with inter-entity contracts between an easy to cheat organization (the military) with a group of people who have a history of cheating (the military contractors).

<comment><USER10> > The complexity of the instructions will likely be turned into long > checklists and procedures for the actual producing plant.

It'd be nice if our system of interpreted case law were turned into some form of checklists and procedures; then we wouldn't need lawyers.

That is to say, the one "contract" that affects every man, woman and child—federal law—has been screwed up in exactly this way, by providing _so much of it_ that no one has any idea what it says without years of study. There is a difference between clarity and verbosity.

<comment><USER8> Actually the years of study aren't to figure out what it actually says. They are to figure out the many ways you can use it to convince a judge that it says what you want it to say, which can then be used to get whatever result you're being paid to achieve.

</comment></comment></comment></comment></comment></comment><comment><USER11> This is just like programming. Someone asks you to add a small feature and you say "sure, that's only a few hours work". Two days later you're finished.

</comment><comment><USER12> This is a military specification for brownies to be packaged in MREs, not a recipe. Probably not much different from software specifications.

It's not uncommon to find documents like this is government. Everything must be explicit so that there is little room for assumptions (I guess that's why it's called a specification). If you have a question about something, you refer to the spec. It also ensures that each brownie produced meets a certain quality standard, and that contractors producing the brownie know exactly how to produce them.

</comment><comment><USER13> Except that the actual recipe itself (combining and cooking ingredients) only takes up half a page. The rest is not just boilerplate (no pun intended), it's a specification regarding ingredient and package quality.

</comment><comment><USER14> Don't scoff at this until you've tasted military-grade walnuts.</comment>