Ask HN: Was data science just hype? - <AUTHOR> I don't hear a lot about "data science" anymore. And judging from the shrinking number of job postings, I suspect it was a bit overhyped a few years ago. What do you think? <comment><USER3> I think what companies really want is smart generalists with advanced math, programming, and modeling skills coupled with domain knowledge. That skill set will always carry high value in technical companies.

The reason it carries value is the skills are difficult to acquire. I think the recent decline in interest reflects the rise of new data science candidates that are taking the path of least resistance to a career in data science. Rather than pursuing problem solving, people are pursuing "data science" which is a nebulous term in and of itself.

<comment><USER4> I am wary when people wax lyrical about all of the ways they love using machine learning on data. It makes me nervous because i worry that they have a hammer and can't wait to use it on anything vaguely nail shaped.

<comment><USER5> ML makes predictions; testable predictions.

Machine learning is an area where you need to be able to produce results. Fake it 'til you make it isn't going to cut it for long. Either these people produce something that works, or they don't.

<comment><USER6> > Machine learning is an area where you need to be able to produce results.

Having to produce results is one thing. Mindlessly throwing tensorflow/pytorch at problems is an entirely different problem.

It's like those front-end devs who mindlessly insist that they need to use heavy javascript frameworks with convoluted build processes such as React/Angular to churn out a static web page with a couple of paragraphs and images.

</comment></comment></comment></comment><comment><USER7> There's a lot of hype in software. Just like you were hearing about Blockchain startups only when Bitcoin started trading at 5 bazillion USD and now you don't hear nearly as much about it. During 2008-2015ish we went through an insane churn of new JavaScript frameworks and NoSQL database hype.

Yeah it seems to have calmed, but I don't think data science was _just_ hype because it comes from (and somewhat is) probability and statistics, and the rate of data/information that's being produced by and extracted from people seems to be ever increasing. But it absolutely was prone to a hype cycle as with almost anything else in tech. IMO this is a phenomenon exacerbated by venture capital.

<comment><USER8> That hype in software is the reason I wrote a media literacy guide for software engineers. Hype makes it easy to get free marketing and also drives clicks for media and social networks.

[https://github.com/nemild/hack-the- media/blob/master/softwar...](https://github.com/nemild/hack-the- media/blob/master/software-engineers-media-guide.md)

<comment><USER7> Nice, at a glance this is pretty well-detailed. Bookmarked. Wait, I've been hacked!

</comment></comment></comment><comment><USER9> It wasn't 'just' hype, but it was over-hyped. There are companies that have their act together from a data standpoint and can make use of data scientists, whatever that term actually means in the context of their organization, but most can't. So the companies who spun up a data science initiative but had no business doing so are now likely saying things like 'what do you mean we don't have the necessary data?' and 'what do you mean our data is a mess?' etc. and will quietly back off over time. Likewise, the companies who can take advantage of it will quietly do so. No different than the hype surrounding every other buzzword in tech... there is no silver bullet.

<comment><USER10> If the data is a mess, cleaning it up is very much legitimate data science work in my opinion.

<comment><USER11> Sometimes it can't be cleaned up, or the process of cleaning it up takes too long or is more expensive than the company wants to spend. Sometimes the clean up process is error prone, or leaves you with too little useable data. Sometimes the data really is too noisy and no amount of clean up is really possible. It's definitely true that data clean up is a problem data science can address, but it's not a magic wand.

</comment></comment></comment><comment><USER12> The hype was that you could take a huge pile of data and turn it into hugely valuable insights.

The realization is that any random pile of data likely doesn't have anything in it that is worth paying for: Here's our analysis! We already do/knew that.

Some people have really good, valuable data sets. Most people don't.

<comment><USER13> It's an epistemology / ontology question, as folks familiar with the humanities would spot in little time. Aka it's not "data" until something empowers the created metric a meaning.

The map is not the territory.

[https://www.amazon.com/Raw-Data-Oxymoron- Infrastructures/dp/...](https://www.amazon.com/Raw-Data-Oxymoron- Infrastructures/dp/0262518287)

<comment><USER14> I know what all those words mean, I've studied critical theory, and I have no idea what your point is.

<comment><USER15> whenever I hear the word ontology I reach for my dictionary

<comment><USER16> It's starting to become a buzzword in data science too

</comment></comment></comment></comment></comment><comment><USER17> My guess is that data science roles will merge with business analyst roles. Python and r will slowly join excel as tools of choice for making tables and charts to stick in powerpoint slides and pdf reports. Meanwhile the machine learning side of things will be the domain of _something_ engineers with candidates more likely to come from the computer science/math/engineer world rather than the sciences. (Other then those with degrees in physic who seem able to perpetually land wherever they feel like.)

<comment><USER18> > My guess is that data science roles will merge with business analyst roles.

Data Scientist is a buzz word for Statistician. Business Analyst is buzz word for Industrial Engineer. For example 10 years ago if you studied at my university you would witness that some Statistics students were doing second major mostly at Industrial Engineering and vice versa. They are already related for many years but average Joe has no idea.

<comment><USER19> I still don't really know what a business analyst role truly entails. In my office they seem more like mini project managers w/o the management. They talk to internal stakeholders a bunch, handle a lot of our UAT, and I guess do some reports stuff? One has moderate tech skills (but no programming) and the other is just really good at Excel.

I'm still not sure what their actual formal responsibilities are.

<comment><USER20> I mean, a business analyst _should_ do just that: analyzes the business. It's usually their job to translate strategic areas of improvement in the business unit into specific outcomes.

They ask questions to identify areas of improvement; translate those into functional (and sometimes technical) requirements for other areas (not just IT) to fulfill; and then coordinate the efforts to implement those requirements, potentially as PMs, product owners, Scrum masters, UAT leads, or just a SME.

The best BAs (paraphrasing the data science JD) know more tech than the business and more business than the tech.

</comment></comment></comment></comment><comment><USER21> We're passed the early peak of the hype cycle, but now the marketers have calmed down the real world applications are maturing.

If you think of Data Science as AI sure, but if you frame it as applied statistics + good software engineering practices + cloud scale I think things are in a good place.

<comment><USER22> This. Any good data science person needs to understand how distributed systems work. They need to be decent at applied statistics. I think an average engineering grad can easily work with simple statistics they learn in Linear algebra and intro to ML. The good software engineering part is what has been brought on in the last few years.

I once saw some code written by a "data scientist". The overall code was non- complex, but the Java/Scala code was the worst of my nightmare. Additionally, I think other engineers have also matured enough to understand that underneath the veneer of data science, the fundamentals do not change much.

</comment></comment><comment><USER23> "AI" (or whatever rebranding it gets) always works in cycles, with a phase of excitement and overpromising followed by a phase of apparent underdelivering and skepticism. But what actually happens is that the innovations just become part of the normal tooling, and stop being called "AI".

At some point there is no need to hire a "data scientist", as any python programmer is already expected to know how to use numpy, pandas, sklearn and keras, just like before it was already expected for them to do any kind of data manipulation with SQL without requiring a dedicated database expert.

<comment><USER24> The value is a "Data Scientist" isn't that they know how to use a tool - it's that tell know why to use _that_ tool (technique) and not this other one.

<comment><USER23> I'm definitely not denying the worth of the particular skillset, just like of good DBAs. But as with any skillset, there are diminishing returns. You can get fairly easily someone at the point where they can merge data from multiple sources, create automatic reports with graphs, make simple similarity clustering, regressions and expert systems even if in suboptimal ways and most companies don't really need more than that. They can even learn to integrate cloud/black box solutions for image/speech recognition without having any idea of how to write one from scratch.

Of course, if a company wants to truly innovate in the area it will need PhDs or people with great dedicated knowledge in ML/Statistics/Particular Domain, if it needs to scale it will need good data engineers to create the data pipeline together with DBAs and experts in each tools (like Spark/Flink), but for most companies the basic above is already a great improvement to what they had before.

</comment></comment></comment><comment><USER25> It wasn't hype, however people got very confused about what they actually needed vs what they thought they wanted.

When someone says they want a "Data Scientist" what they really mean is "I want a Data Scientist who is also a Data Engineer".

I have seen so many companies spend a really decent chunk of money on a data scientist and then are shocked to find that this data scientist doesn't know how to deploy models, set up spark clusters or know how many and what type of GPU they need to use to get the job done.

After all - that is not their purpose.

We were in a similar situation, but what we needed was a Data Engineer - <USER2> had a rough idea of where we wanted to go and what we wanted to achieve, he was doing a Masters in Data Science so he had that background as context.

We will look at adding a Data Scientist to our ranks in the future - but they will be working side by side with a Data Engineer who can action their requirements!

</comment><comment><USER26> The "data scientist" where I recently contracted struggles with generating basic reports. I saw a full page SQL query with the caption "yeah, data science is hard" posted to slack. Terrifying.

I think the term "data science" is often misused. It seems to make management feel like they are on the cutting edge. They were talking about AI and a R&D department the other day. They aren't even making use of simple heuristics yet! I guess that talk helps with fundraising though.

</comment><comment><USER27> Over the past several months I keep seeing people trying to equate data science with machine learning, and it made me wonder if the people doing this are trying to salvage (or perhaps enhance) the investment they made in data science by trying to blur the lines between the two.

<comment><USER28> Isn't the line between the two indeed blurry? Maybe deep learning is machine learning, but modern statistical methods such as elastic net, SVM, and random forests are things data scientists should know about.

</comment></comment><comment><USER29> At my previous company data science has become synonymous with data analysis, to the point that the number of data scientists on staff is starting to outnumber data analysts. I think it's more a sign of a maturing field than anything else. The more narrow view of data science as big data, models, and machine learning is probably less a thing now, but data analysis overall is only getting bigger.

</comment><comment><USER30> No it isn't a fad. Data collection by every man and his dog is really happening. The need for people that can use the data in order to improve business outcomes is the consequence of it.

Data collection will become more prominent IMO because:

1\. Data driven business preference, competitive advantage and FOMO. Already dominates sales and marketing. Starting to dominate in product and dev. Already dominates production.

2\. IoT, and more data marketplaces resulting from it.

3\. Extensions of the global SaaS value chains (usually connected by data).

Hence data science will thrive in the future.

</comment><comment><USER31> I think people are realizing that data scientist without domain knowledge cannot create valuable insights.

Enterprises seems to hire less data scientists actually, but they are trying to raise their employees' data skills.

I think that's the cause of the growth of self-analytics tools. Below are examples of them.

1\. Metatron Discovery : [https://metatron.app](https://metatron.app) 2\. Metabase : [https://metabase.com/](https://metabase.com/)

<comment><USER32> I would like to hear more about my European fellows w.r.t. how GDPR affected their ability to muster domain knowledge.

I used to work for a small start-up and the CTO was very strict on data access, making my life as feature developer and "data scientist wanna be" almost impossible.

He, on the other hand, had not only access to all data but also used the product as a consumer (which didn't make sense for ICs so we ended just playing with sales demo accounts). I ended leaving the company because of that.

<comment><USER33> I've been in similar situation and it was really hard to be effective in product / high level planning meetings because it is easy to be blindsided. At the time, I was still an youn engineer in a big co, so I just thought it was my lack of technical experience, but in reality it was nothing technical to it but BS politics.

What boils down to is that people who have any extra data access privilege will have the lead.

Most of the insights will come from aggregate data, so I think companies could work around privacy concerns but I am no GDPR expert.

Back in my days in academia, there was a saying "if you have the trace, you have the paper".

</comment></comment></comment><comment><USER18> Was? As I always said it is Statistics. If you want "data scientist" employ Statisticians.

<comment><USER34> In my professional experience this is a little misguided. A pure PhD statistician isn't going to be able to hack it working on fast-paced production software environments and building end-to-end pipeline/software/ML systems. I mean, no doubt a PhD statistician could learn and be good at it, but the average statistician isn't geared up for this type of work.

On the other hand, your standard tech data scientist may find themselves out of their element if needing to design a very rigorous randomized trial for testing a new drug, and making careful inference (I mean I'm sure plenty could, but I'm not going to trust a 25 year old with two years work experience to do that).

</comment></comment><comment><USER35> With the cloud came a lot of data, so 'big data' was a wave in which Engineers had to deal with ti, and 'data science' was the wave in which we tried to leverage it.

The reality is that most insight from 'big data' are optimizations. They're not going to move the needle on the business as perhaps we would have hoped.

Data Science focused on ad targeting - now that might move the needled.

And of course, maybe some Data Science working along side AI engineers make a breakthrough which could move the needle.

But from a high level, CEO's view, all of these things have trendy undercurrents, the trick is to figure out how much of it really matters to the business.

The 'wins' for consumers will be slow: maybe better product search, better ads. Maybe they figure out how to send flights around ugly weather or how to slot landing times for an x% decrease in flight delays. Or slot road fixing/lights for an x% decrease in traffic delays.

</comment><comment><USER36> I work on a data science team that's growing pretty aggressively. I also frequently hear from recruiters hiring at companies big and small for analyst and scientist positions.

I'm not sure what it would mean for days science to be "just hype." I see DS work on this website alone all the time.

</comment><comment><USER37> It is "different now". The DS bubble popping was covered well by Vicki Boykis:

> Since academia is typically a lagging indicator in adoption to new trends in > the work place, it's been long enough that it's truly worrying for junior > data scientists, all of who are hoping to find data science positions. It > can be very hard for someone with a new degree in data science to find a > data science position, given how many new people they're competing with in > the market.

[http://veekaybee.github.io/2019/02/13/data-science-is- differ...](http://veekaybee.github.io/2019/02/13/data-science-is-different/)

</comment><comment><USER38> It wasn't 'just' hype, but the real problem was that very few companies had any data that you could actually extract much value from. Garbage in Garbage out is as true as ever.

That being said I've observed that the data science techniques and tools that have developed over the past few years have been absorbed and adopted by a lot of people that aren't "data scientist". So while companies are hiring a lot less "data scientists", a lot of "data science" is now done by domain experts and analysts as part of their work.

</comment><comment><USER39> The best quote I've heard for ML is "ML promises to deliver what Computer Science promised to deliver in the decades before as AGI will promise to deliver in the future"

</comment><comment><USER40> No, science was not hype. It's not like the fashion is over and now we're gonna go back to making worse decisions based on less data.

</comment><comment><USER16> I think they are evolving to more specific roles. A couple of years ago some "Data Scientists" were actually doing Data Engineering. Now that distinction is more clear. You also have Machine Learning Engineers, who can help with deploying models or you can even see things like "Deep neural network engineer" or NLP data scientists.

<comment><USER41> What is the definition of "Engineer" in "Data Engineer" and "Machine Learning Engineer"? And what is the difference with "Scientist".

I'm not from the US and in my country, "(civil) Engineer" is a legally protected title [https://en.m.wikipedia.org/wiki/Civil_engineer#Belgium](https://en.m.wikipedia.org/wiki/Civil_engineer#Belgium)

</comment></comment><comment><USER42> It's been subsumed inside traditional BI teams, who don't interact with "generalist" developer teams and are an entirely different audience.

So the field keeps growing, but in more specialized forums. Increases in Python adoption, for instance, mostly come from re-skilling initiatives in BI teams inside my customers.

</comment><comment><USER43> A lot of it was. Like VR, blockchain. Companies didn't know whether they needed this new tool. A lot of them didn't. Like VR, the long tail on this might be huge. It will just be new technologies that weren't possible before, not huge improvements on existing ones like some companies thought.

</comment><comment><USER24> Two thoughts: 1) Much of this is the bias of this community towards new and more more technical companies. At many of our work places Data Science is just part of the landscape and is now baked in. But there is a wide gap between that and legacy companies where the idea of using these techniques this way isn't new at all. In other words we've been through the initial wave of the hype curve and are now into mainstream growth of the idea. 2) Branding matters. In many companies that I get to interact with (fortune 100s) run-of- the mill analysts exist for ass-covering the non-data-based decisions of higher ups. Post-hoc decision justifying. But DS teams (currently) actually get a seat at the table and get listened to. If for nothing else that makes DS way more effective then just BI or whatever. And as long as we get things right that should turn into a virtuous cycle of being heard, being seen to contribute, and being asked to contribute more.

</comment><comment><USER44> I wonder what is the role of hype in the scientific progress, or progress in general. Is there evidence that it pushes it forward?

</comment><comment><USER45> I work for one of the major bootcamps and I can tell you that interest has significantly declined over the past 1 year. The market for data science has indeed shrunk. The reason? I'm not sure, could be that it didn't live up to the hype.

<comment><USER46> From my perspective it seems like even 5 years ago data scientists were more about creating and validating ways to look at the data whereas today it is more about processing the data with the standard tools and methodologies. Data processing is as big as ever but takes a fraction of the effort to apply it.</comment></comment>


Visualizing Commodore 1541 Disk Contents - <AUTHOR> https://www.pagetable.com/?p=1070 <comment><USER2> This guy [https://twitter.com/a2_4am?lang=en](https://twitter.com/a2_4am?lang=en) has been posting apple II woz files for a while and I know there is an Atari format as well.

Cool stuff.

</comment><comment><USER3> One day we'll eventually run out of ancient drives that can read their floppies, but the media could be still readable using other means. I would love to see a plain scanner like device that reads magnetic fields instead of light reflections, that is, something that slowly scans the entire surface of the media [1] just like a scanner then builds a pattern of tracks and sectors keeping track (pun unintended) of its position, then proceeds to map them into data thanks to a database describing how data structure is arranged by different brands/disk formats/filesystems/etc. We wouldn't need either speed or realtime reading; being able to read and decypher an ancient disk in 3 days of multiple passes to correct errors, would still make it a success. A (very) small rotating head would likely work here. Did anyone ever attempt to do something similar?

[1] The magnetic media should be removed from the disk body, that is, the disk would stand still in the "scanner" while the head(s) would slowly do multiple passes to map the remnants of its magnetic data. This would eliminate the requirement for custom readers for very rare floppies, say like those 2" ones I can't even remember where I saw them (possibly a musical instrument from the 80s).

<comment><USER4> What would the advantage of making the scanner completely unlike the actual physical layout of the encoded data be? It just seems to make the thing more difficult for no obvious gain.

<comment><USER3> I agree things would be initially difficult, but I don't see how we could make a reader for less known disks without replicating their original drive mechanics as well, which is probably the hardest part. To me it would be more "scalable" to sample the magnetic field relative to the head position on a plane (on which we fixed the disk), then building a structure where each 2D element is mapped onto its position on the disk and assigned the value of the magnetic field. The result wouldn't be that different from this project pictures, but we could swap a 5.25 disk with a 3.5 one without changing a single screw in the reader. A further development could be reading more disks at once, provided their surface can be reached by the head. It would surely be useless while we have working disk drives, probably not 20 years from now...

<comment><USER4> _building a structure where each 2D element_

It sounds to me like this is 'easier' because, I dunno, traversing arrays is 'easier'. But mechanically, serially scanning a disk or cylinder is much simpler than serially scanning a rectangle. You can build a high resolution scanner that isn't based directly on the original drive mechanism that but still does magical high resolution scanning for offline analysis without the additional artificial obstacle of making it like an image scanner.

</comment></comment></comment></comment><comment><USER5> This is fantastic! Never used a Commodore, but this is familiar enough for a DOS user from the 5-1/4" floppy era.

Would love to see something similar for ZX Spectrum tapes :)

</comment><comment><USER6> Lovely visualization of knowledge that is only usually obtained through low- level spleunking.

I'd love to see visualizations of some of the encryption approaches used on the Apple ][, where sector alignment was shifted, file table indexeas moved, data interleaving, etc. This was all possible due to the soft-sectored disks and timings determined in software (Woz's genius move to reduce chip count!) I could never wrap my mind around what was happening to the data.

<comment><USER7> You can see some of those in [https://twitter.com/DiskBlitz](https://twitter.com/DiskBlitz)

</comment></comment><comment><USER8> Somebody make something like this for DOS floppies please. It'd be great to see visualizations of normally formatted 1440KiB floppies vs. 1680KiB DMF ones.

</comment><comment><USER9> C64 disks with custom loaders (read: games disks) were so fascinating to me as a kid because the sounds of the 1541 drive would change as soon as that fastloader code activated. Lots of game disks had their own unique loading sound signature.

I would love to see a comparison of 80s floppy disk drives. Lots of genius designs within... except the 1541's terrible bug which was never fixed due to backwards compatibility.

<comment><USER10> The copy protection on many games would cause "head knock" and force the 1541 out of alignment after a while. :(

</comment></comment><comment><USER11> Amazing, thanks for posting.

load "sysres",8,1</comment>


Nanex Gets $700k Whistleblower Award from SEC - <AUTHOR> http://www.nanex.net/aqck2/4712.html <comment><USER3> Yikes. Lots of misinformation here.

First of all, this has nothing to do with High-Frequency Trading. It's about the NYSE not delivering a product (SIP real-time data) while collecting $100M a year for that service. This was happening for at least 3 years.

I am a champion of free markets. The term "High-Frequency Trading critic" is a label others use when they either can't understand and/or refute solid evidence.

I'm happy to answer questions.

<comment><USER4> Can you comment on how hard it was to find another period exhibiting this behavior? The article says _We chose this period because there was a noticeable lag in the public quote from the NYSE versus quotes from other exchanges, allowing us to rule out the consolidation process as a source of the delay._ Did you have to look for a few periods to see the behavior, or did you look for a few to get a sample that showed the problem more clearly?

Do you have any info on how often lag was present, or how often over a specific threshold?

Do you know how representative you sample was with respect to when lag was present?

I understand you may not have gone into this much detail in your research, but if you have any info like this, I think it helps illustrate the scope of the problem.

<comment><USER3> It happens all the time (ALL. THE. TIME.) and have documented dozens of examples. There are links to many of those examples on that page (fantaseconds).

I've also posted many examples on twitter @nanexllc using recent data (yes, this is still a problem!)

</comment></comment></comment><comment><USER5> So essentially the NYSE is a provably fixed game, with large HFT houses paying for the right to trade on non-public information, and it hides behind the idea that it was just a few hundred milliseconds.

All the while they are selling non-public information as a product that is deliberately used to micro-manipulate the market?

I mean, deep down I always knew this, but to have it all spelled out is shocking because it exposes the NYSE as systemically fraudulent. The implication of that is so overwhelming it causes me to want to ignore it out of sheer inability to think of a way to solve the problem.

<comment><USER6> Market data is not non-public definition. NYSE was simply slow in aggregating market data during times of high volume.

This did not affect you as a retail investor, as you get the NBBO price. This only affected you if you were an HFT firm with bad infrastructure who depended on the aggregate feed and not their direct line.

Like tptacek said, the price for getting a direct line, while expensive, is not unreasonable for a business. The largest cost will be salaries for the people writing your code and maintaining your infrastructure.

<comment><USER7> Are you saying that there is a special market operating in the NYSE that some traders cannot access? Why would my order not be eligible for being matched, but a HFT's would?

Edit: wow, rate limited after three posts this morning. A new HN low.

My response to tptacek below:

My dumb order?

How can HFT's intercept and redirect my trades to their, appently, captive pool of dumb trades?

Sure sounds like multiple markets are operating...and you are even telling me that my orders will be scraped before they can even reach some markets.

<comment><USER8> Oh boy, you got this 100% the wrong way around.

As a small trader, you get access to special lower prices that a hedge fund can't get. Your broker will be routing your order to a wholesaler who will fill it at lower prices (ie, narrower spreads) than you would get on the open market, because they have a legal obligation to not screw you over.

You can request that they route it to anywhere you want, and they are legally obligated to do it if you ask, but you _really_ don't want to do that. Unless you like giving money away of course.

(Also, you don't seem to understand basic market mechanics. You use words like "scraped" or "picked up" which are nonsensical in context.)

IEX recently suggested that retail investors should request their broker to route their orders to IEX, for which they got criticised very harshly. And rightly so.

<comment><USER7> As a small trader I don't get paid to provide liquidity. A hedge fund may.

I used the word scrape in response to tptacek's response below my above post. If a trader is not aware that what tptacek describes is happening to their orders, then I believe my characterization is accurate.

<comment><USER8> You're still not addressing my main point.

As a small investor, your order is probably routed to a wholesaler like Citadel, where you will get a better price than you would otherwise get.

And you are upset because you incorrectly believed that your order would be routed somewhere else (where, exactly?), where you would get a worse price, which you think would be better because...why?

> I believe my characterization is accurate.

It's not even slightly accurate; it's literally the opposite of the truth.

</comment></comment></comment></comment></comment></comment><comment><USER6> What he's saying there was that due to inefficiencies in NYSE's infrastructure, the feed that went to the consolidation (which provides the NBBO) was delayed. Firms were able to then exploit those inefficiencies by dumping a ton of quotes into the market to cause it slow down those feeds. This is called quote-stuffing and is illegal and against exchange rules. (I don't know if this was illegal then.)

From a technical perspective, as orders are sent to the exchange, NYSE sends them out on a direct line to subscribers who listen to that feed. This is not private information, but it is relatively expensive, given you'd need to have servers at a colo, and pay for fast networking, boxes, software, etc.

They also then aggregate that data and send them out on a feed to the NBBO which is much more accessible to the majority of traders. Obviously, it is somewhat delayed, as they need to aggregate, and a firm that can aggregate the data faster than NYSE thereby has an advantage. This aggregated data gets set out as soon as possible, and NBBO defines a time range that they need to be accurate. For what its worth, NBBO means "National Best Bid and Offer" and they will aggregate bids and offers from all exchanges and provide the best one in a single consolidated feed.

Nanex determined that at certain times during high volume, Nasdaq's aggregate feed was delayed by up to 30 seconds, and this was cause by certain entities quote stuffing (which like I said earlier, is against the rules).

That is all that was determined. There is nothing about trading on private information (which there is none here) or front-running (which HFT firms can't do) or any of the other nonsense that gets spewed out.

If anything, other firms with good infrastructure would have been able to arb out the inefficiency and make the markets right again, quickly.

<comment><USER9> Your summary is incorrect; the 30 second delay was six years ago and was caused by selling access to real-time data vs delaying the data for the public. Quote stuffing resulted in delays of hundreds of milliseconds:

      > Once again, we detected sizable delays between time-stamps
      > in the public quote and Open Book. These delays ranged in
      > the hundreds of milliseconds. Though the delay magnitude was
      > far lower than the tens of seconds discovered during the flash
      > crash, it was still hundreds of times higher than expected.

<comment><USER6> Regular quote stuffing generally delayed by hundreds of milliseconds, which is a pretty bad sign regarding their infrastructure.

The 30 second delay was only during the flash crash. For an aggregate feed that's not aggregating very efficiently, that would make sense, as there was an abnormally high amount of volume during the crash.

</comment></comment></comment><comment><USER10> So essentially the exchanges offer multiple data feeds. These feeds are differentiated by a couple of points.

1) How much market data they give out. Typically this is broken into 3 levels:

      - Level 1, Just the top of book quote, this is the best bid and offer and their corresponding number of shares at this price level.

      - Level 2 Depth of market at each price level, same as above but you can see all liquidity offered.

      - Level 3 same as Level 2 but you can see each individual order that contributes to each price level

This differentiation is generally considered to be a good thing as most people including most hedge funds can get by with even Level 1 quotes.

The other way they differentiate is the speed at which you get updates.

Have a look here to see what just the Nasdaq offers. [http://www.nasdaqtrader.com/Trader.aspx?id=FeedMIPS](http://www.nasdaqtrader.com/Trader.aspx?id=FeedMIPS)

You can spend anywhere from $500/month to $75,000/month to receive market data from just the Nasdaq, one of 50 trading venues in the US.

As you can imagine the difference is speed. The slower methods will use the text based FIX protocol and the fastest methods give you a feed and an FPGA to parse the feed with.

It's this later feed that is getting the exchanges into trouble as these feeds notify the user of trades before the SIP can be updated which lets firms have a 300 millisecond second peek at the market before everyone else does.

Couple these faster feeds with many dark pools not reporting trades in a timely manner and new order types that allow firms to sweep liquidity at one exchange only( ie not rout-able orders) and you get a situation where latency arb can thrive.

<comment><USER4> > It's this later feed that is getting the exchanges into trouble as these > feeds notify the user of trades before the SIP can be updated which lets > firms have a 300-700 nano second peek at the market before everyone else > does.

From the article:

 _Specifically, we found that stock quotes from the NYSE were delayed by more than 30 seconds to the public quotation feed (chart 1), relative to Open Book, which is NYSE 's expensive direct feed product used mostly by High Frequency Traders (HFT)._

That indicates it was quite a bit more than 300-700 nano seconds at it's worst, by a few orders of magnitude. The chart showing the delay over time in their testing period shows a delay of 200ms routinely. That's way more than a few hundred nanoseconds.

<comment><USER10> Yes, but to be clear that was a very special situation that hadn't happened before or since. They were essentially fined for not having good developers.

The NYSE has always had a reputation of being awfully inept from a technology standpoint, though they have made strides in the past few years.

That particular time was made even worse by the direct feed holding up much better that their SIP feed, though the direct feed was also slower than normal.

An accurate title might be NYSE fined for having a software bug that slowed down one feed occurred during the busiest time in the markets history.

<comment><USER4> > Yes, but to be clear that was a very special situation that hadn't happened > before or since.

If by _a_ special situation, you mean directly observable at multiple time periods. Here's what the article says:

 _To satisfy a curiosity of whether the NYSE public quote delay was unique to May 6, 2010, we ran another quote-by-quote comparison of time-stamps from the public quote and Open Book for a 30 minute trading period in General Electric stock (GE) on July 21, 2010 (see chart 2). We chose this period because there was a noticeable lag in the public quote from the NYSE versus quotes from other exchanges, allowing us to rule out the consolidation process as a source of the delay._

So they were able to find another time period exhibiting the same behavior as the flash crash at a later date, and it doesn't appear they had to look very hard (but there aren't a lot of details on that). Thirty seconds is a crazy amount of lag, but that was during the "special situation". That they look to have routinely had lag well into the hundreds of milliseconds is more troubling, because there's no longer a special situation to point towards as an excuse.

<comment><USER10> Umm, I think what you posted proves my point. The 30 second delay was a complete aberration. 100 millisecond delays happen all the time at every exchange.

To be fair to you I originally posted nano second when I meant millisecond. I've corrected my post.

As I've said before, I'm not at an HFT firm but we routinly see latency changes in messagesfrom all excahnges and dark pools.

Just like Google can't guarantee each query will be served in 250 milliseconds, each exchange can't guarantee every message will be delivered in nano seconds.

I'm good with this.

<comment><USER4> > To be fair to you I originally posted nano second when I meant millisecond. > I've corrected my post.

Ah, that makes a big difference. :)

> As I've said before, I'm not at an HFT firm but we routinly see latency > changes in messagesfrom all excahnges and dark pools.

From separate feeds to the same exchange? My understanding from the article is that that is not allowed:

 _A crucial sub-ruling in the regulations prohibits exchanges from giving stock quotes to special groups faster than to the public._

So as long as you're measuring from the start of the request, not the end, you shouldn't see delays in the hundreds of milliseconds to the same exchange, regardless of feed. At least that's how I understand it, but you sound like you have more practical experience, and may be able to correct where I'm misinterpreting something. Is that not how it works in real life?

</comment></comment></comment></comment></comment></comment><comment><USER11> Nuclear reactors are planned in a way that their response time is in human scale e.g. they take something like 20 seconds to increase or decrease activity. This leaves room for human intervention.

IMHO markets should do something similar: create something like 30-second barriers for trades. Trades that cross the bid/ask are carried out. Trades submitted cannot be canceled until the next 30s cycle. Trades outside of the range (bid too low/ask too high) stay open. The open trade book is sent to everybody every 30s.

Allowing µs-range trades (and particularly adding and removing bids and asks so fast) is asking for trouble, it is obvious that at some point of time two or three HFT robots will react on each other's trades and do something veeeeery stupid.

<comment><USER12> Even people who support quantized markets don't propose markets running on human timescales.

Meanwhile, if you're the kind of trader that is impacted by trade latency (read: a competitive market maker or decently well capitalized prop trading firm), there are already venues you can trade in that are quantized. The problem is that nobody wants to trade in them.

<comment><USER13> Not sure what you mean by "quantized market" (discrete-time auction?). But the opening and closing sessions at the NYSE run at human timescales (twice per day), and most of the daily volume is traded at open and close iirc.

edit: "Across Wall Street, less trading is taking place at 3 p.m. or, in fact, most any time but the opening minutes and the final half hour." [http://www.wsj.com/articles/traders-pile-in-at-the- close-143...](http://www.wsj.com/articles/traders-pile-in-at-the- close-1432768080)

</comment></comment></comment><comment><USER14> Pretty sure this is the initial analysis from 2010 - [http://www.nanex.net/FlashCrash/FlashCrashAnalysis_LOD.html](http://www.nanex.net/FlashCrash/FlashCrashAnalysis_LOD.html)

 _" If the average or base quote rate is around 10,000/second, then it only takes an additional 10,000 quotes/second to reach the magic 20,000 quotes/seconds where a corresponding delay is seen in NYSE quote from CQS."_

This seems like a bug related to system load, so a $5M seems pretty harsh unless they knowingly ignored the issue for the benefit of their high-paying CQS customers.

<comment><USER15> I don't think it's really that harsh if you look at it as a bug in a critical system. Certain fields with safety critical systems (such as medical or aviation software) face similarly strict regulations and enforcement for the good of the public. It seems to me with that with the volume of money being affected by this system that the potential danger for faults in the software would be as high as potential damages from medical software or something similar.

</comment></comment><comment><USER16> I have two questions about HFT:

\- Is HFT a healthy or unhealthy part of the economy? Meaning does it help flatten out the highs/lows, or does it emphasise them?

\- Aside from a micro-tax on trading (e.g. 1c/trade) is there any other mitigations/solutions to HFT?

<comment><USER4> IIRC, HN's yummyfajitas goes into a lot of details in his blog posts on HFT[1]. My own understanding is that it provides liquidity and reduces bid/ask spreads, resulting in a more accurate stock valuation. I'm not sure the negatives, but I'm not a trader nor do I follow such things all that closely.

1: [https://news.ycombinator.com/item?id=3852341](https://news.ycombinator.com/item?id=3852341)

<comment><USER14> From that blog post:

 _" Most HFTs run a market making strategy. What this means is they play both sides of the table - they take no position on whether a stock will go up or down. Instead, they try to offer securities both to buy and sell. If you want to buy, they will sell to you at $20.10. If you want to sell, they'll buy from you at $20. As long as their buys and sells match don't get too out of whack, the HFT will collect $0.10 = $20.10 - 20.00."_

That does provide more immediate trades for both the buyer and seller, which the pro-HFT camp calls liquidity. Unfortunately it's usually not needed - <USER2> the buyer and seller were already in the market and willing to trade, and now they both just had an additional $0.10/share sucked out of their trade by some HFT.

<comment><USER13> That's why market makers are also called scalpers, because they skim the $0.10 spread. But in return for paying the spread, what traders get is immediacy.

The problem is, how much immediacy do traders actually want? HFT provides immediacy in microseconds; its needed for continuous-time auctions because buyers sellers usually aren't in the market at the exact same microsecond interval (while the HFT _is_ there at every interval).

Frequent Batch Auctions[1], where orders are matched in batches with single- price clearing (call auction style) - just like the opening and closing sessions of NYSE, but much more frequently with shorter periods of say 30 seconds, 1 second, or even 100ms per batch. Their models show that frequent batch auctions will reduce spreads even further (because it eliminates mechanical/latency arbitrage rents). In theory, retail traders should get even better prices (because the spreads will be smaller) if they are willing to give up microsecond immediacy for 1 second order matching.

[http://faculty.chicagobooth.edu/eric.budish/research/HFT- Fre...](http://faculty.chicagobooth.edu/eric.budish/research/HFT- FrequentBatchAuctions-Slides.pdf)

<comment><USER17> Current continuous-time trading already provides you the ability to specify very precisely how much immediacy you are willing to pay for. Thats precisely what the market types are about.

I don't particularly have an opinion about the Budish paper, other than it has some serious impacts to globally traded products and it is way more disruptive than people seem to suggest.

I do worry that fundamentally altering the way the markets work, because some participants don't understand order types is particularly troublesome.

<comment><USER13> Simplish order types (limit/market, good till cancel, immediate or cancel, etc.) can only do so much. When you place a limit order on a continuous-time market, you'll pay exactly your limit price; but on a batch market you'll pay the clearing price (which may be _less_ than your limit price).

The complicated types (hide-n-slide, NBBO pegs, and conditional pegs[1]) may help some players at the expense of others. The only players who wouldn't benefit from batch auctions (according to Budish et. al.) are latency arbitrageurs.

1\. [https://mechanicalmarkets.wordpress.com/2015/10/05/iex- peg-o...](https://mechanicalmarkets.wordpress.com/2015/10/05/iex-peg-orders- last-look-for-equity-markets/)

<comment><USER17> > you'll pay exactly your limit price

Thats not quite true. Price improvement happens all the time on limit orders, but I'll concede you'll pay close to it if it fills.

> The complicated types (hide-n-slide, NBBO pegs, and conditional pegs[1]) may > help some players at the expense of others.

This is true of limit/market/stops as well.

The Budish paper is very interesting.

What we don't know is what it would do to the spread, which is very important especially given that it will be harder to predict the clearing price. It is also a major change to the markets, virtually impossible to implement and its not clear to me what that risk is buying us, especially given that I fully expect most participants to go through an intermediate first.

<comment><USER12> What are some circumstances in which you'd get price improvement on a resting limit order?

<comment><USER17> You wouldn't on a resting order, but if you hit the other side you will.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER18> While I am fully pro-HFT, and in fact work as a consultant for a few HFT shops, this is certainly illegal and anti-competitive.

(Somewhat akin to using game engine bugs for artificial lag creation in MMOs on the opponent side).

</comment><comment><USER19> Note, the SEC has a protocol for all whistleblowers. Here's the largest ever award [1] and the wiki article on the topic [2] (sorry for mobile link).

I asked my accountant friend about whether he'd ever consider whistleblowing if he saw wrongdoing. His decision criteria was based on whether he'd A) Ever be able to get hired again or B) If the size of the prize was so big he'd never have to worry about it.

[1] [https://www.sec.gov/News/PressRelease/Detail/PressRelease/13...](https://www.sec.gov/News/PressRelease/Detail/PressRelease/1370543011290)

[2] [https://en.m.wikipedia.org/wiki/SEC_Office_of_the_Whistleblo...](https://en.m.wikipedia.org/wiki/SEC_Office_of_the_Whistleblower)

</comment><comment><USER20> Anyone know what this quote means?

When the SEC called Hunsader in June 2015 to tell him about the award, he said to them "I would have accepted $1 if you simply acknowledged me at the time."

Was he initially ignored?

<comment><USER3> I was ignored by the Trading and Markets people at the SEC, but not ignored (obviously) by the Enforcement people.

<comment><USER21> I don't want to be silly but it appears the SEC ignores the siren often (Madoff is another). Is this due to sheer workload or friends in high places?

<comment><USER3> Wealthy friends with promises of a nice job (and other things)

</comment></comment></comment></comment><comment><USER22> This might be of interest to people:

A NYSE Speed Bump You Weren't Aware Of [https://www.iextrading.com/about/press/op- ed/](https://www.iextrading.com/about/press/op-ed/)

<comment><USER12> The "speed bump" here being that NYSE's ancient creaky FIX gateway is slower than the NYSE ARCA gateway that they tell everyone to use instead. _Quelle surprise!_ 80s text network protocol slower than 90s binary protocol: film at 11.

<comment><USER23> This is almost right. NYSE and Arca are two separate exchanges (both owned by NYSE). Both exchanges offer FIX and binary gateway protocols. Annoyingly, the Arca binary protocol and the NYSE binary protocol are not the same protocol. NYSE does not suggest using the Arca gateway to access NYSE. They suggest you use a NYSE binary gateway to access NYSE.

<comment><USER12> Dammit. I even looked this up before I wrote the comment, and got the opposite impression. (I'm a little familiar with the Arca protocol for other reasons).

Thanks for correcting me.

</comment></comment></comment></comment><comment><USER24> Ultimately, the market has been setup to almost mandate participation. If you work someplace that provides a matching 401k program, and you choose not to contribute, then you're passing-up on matched dollars if you don't contribute to a 401k. If you're a part of any endowment, pension plan, or other managed retirement plan there's a good chance that plan (and your money) is being invested in the market.

So what are your options? Do you not contribute to a 401k or IRA, pay the full tax rate on all of your income today, and invest in real estate or some other investment?

</comment><comment><USER25> Nasdaq does go through great pains to ensure that public & private feeds go out "equally"-ish on the wire whether it's a quote going out of the prop or public data feeds. At the packet level, a public quote to the SIP or a private quote via Itch are within the sub ~10u level. But once the packet leaves, the fate of that packet is surrendered to the speed of the SIP system.

There's a couple of inherent challenges.

The SIP that aggregates the data is by design a separate system from the trading systems. This is for obvious reasons, as every participant provides their quote to the SIP. So no matter how fast the SIP the is, even with a 99.999% latency of 20 microseconds, the SIP will always be 20 micros + travel time to the SIP (1 to 500 to 7000 micros) as opposed to a direct feed. Nasdaq is in Carteret. NYSE is in Mahwah. BATS is at NY4 in Secaucus, etc. So direct feeds will always be the fastest, even with a SIP that can defy the speed of light a little bit.

Nasdaq does actually read marketdata directly from other exchange participants and falls back to using the "slower SIP" data as a fall back in the event of any issues. This allows for the latest quotes to be respected.

The challenge with the SIPs being hundreds of millis behind is partly due to the criticality & member-driven consensus system in place. Each of the marketcenters NYSE, Nasdaq, Bats, etc have a stake and voting power on how the SIP would be run & operated. Since it's a system that "works" there's really no motivation to continually improve the technology/latency of the SIP. There's also a fear to change the SIP and lead to it going down. If a SIP goes down, by rule, a regulatory halt must be issued and all trading in the securities that particular SIP handles must stop. This is what occurred during the 3 hour outage several years back at Nasdaq. The trading system was fine, but since the SIP was down, everything had to be halted. Halts are detrimental to business since the respective marketcenters make money on trades. Halts = No Trades = Lost Revenue, so there is a lot of motivation to not halt by all the member exchanges.

That ambivalence to a tech refresh has since changed since the scathing critique of Flash Boys and the Nasdaq SIP is in process of being refreshed to the same architecture as the trading systems, which should bring median latencies of the SIP down into the tens of microseconds and 99% latencies into the hundreds of microseconds. Current medians and 90th percentiles are at 450/810 microseconds, which is considered an eternity.

[http://www.wallstreetandtech.com/infrastructure/nasdaq- omx-w...](http://www.wallstreetandtech.com/infrastructure/nasdaq-omx-won-over- sip-committee-with-latency-reductions-and-tech-upgrades/d/d-id/1317316)

[http://www.utpplan.com/DOC/UTP%202015-Q4%20Stats%20with%20Pr...](http://www.utpplan.com/DOC/UTP%202015-Q4%20Stats%20with%20Processor%20Stats.pdf)

On the technical side, one of the challenges with synchronizing packet delivery between a private & public feed is that the absolute fastest NICs available today with kernel bypass can get a packet from wire to memory or memory to wire in about 2-3u. So every hop of a computer costs a minimum of 4-6u. Usually this is Order Port Ingress -> Matching Engine -> Order Port Egress. An FPGA NIC that does the IP Header processing and delivers data to memory can do wire to memory in about 1-2u. One-way PCI-E latency is around 800 nanos. So there's about a 500-1000 nano budget to process a packet in the FPGA, then another 800 nanos to deliver it over PCI-E to memory to be processed. The fastest cut-through switches have a port-to-port latency of around 250 nanos, so switch hops are pretty negligible in the overall latency of a system, but they do play a part. Anyway, making sure that everything is super duper optimized and running as it should be in a semi-continuously deployed environment makes for interesting and quite crazy/bull-shit level challenges.

</comment><comment><USER26> Congrats on being rewarded and recognized for the very important work you do in bringing transparency to the financial markets. The whole story of how this went down is just epic!

My question is about the business side of Nanex. Are sales of your products such as NxCore enough to sustain the company? Do you also actively trade? And what opportunities do you see currently around data science and machine learning for entrepreneurs in the quantitative finance world?

</comment><comment><USER27> Is there any good reason that the SEC shouldn't ban direct feeds? If everyone were forced to use the SIP, what issues would remain here?

</comment><comment><USER28> I never realised the NBBO could cross because of HFT. (Technically a market type order is required to prevent this, but the quote stuffing lags price) Definitely seems wrong.

</comment><comment><USER29> For someone who knows next to nothing about trading, but has read "Flash Boys", any other recommended books on the topic?

<comment><USER12> Yes: read "Flash Boys: Not So Fast", a very readable and very technical takedown of "Flash Boys", which is a terrible, incoherent book --- without a doubt Lewis' worst.

For a better version of "Flash Boys", read "Dark Pools"; Dark Pools also takes a gimlet look at high-speed trading, but also does a great job of documenting the birth of the NASDAQ ECNs and the transition from the early-90s centralized markets to modern, decentralized, electronic trading.

If you want to get into the technical detail, the best book is still Larry Harris' _Trading and Exchanges_; it is almost exactly the "TCP/IP Illustrated" of trading.

</comment></comment><comment><USER30> Google should open a stock exchange.

Sure, they'd be weary of the immense regulation involved. But I imagine they're much more qualified to build a fair, fast and accurate trading system than any of the existing exchanges are.

<comment><USER12> What would be more "fair" or "fast" or "accurate" about Google's exchange? I've gotten to do software pentests of several huge exchanges. They work pretty much the way you'd expect them to. A lot of things people are just starting to do in web software (message-mediated microservices) were idiomatic in finance in the 1990s.

<comment><USER30> Look no further than the linked article.

> a high quote rate from any NYSE stock will cause a corresponding delay in > all NYSE public quotes! Anyone with this knowledge can easily cause latency > on demand across many stocks, by simply blasting quotes in any one stock

Or another link from this thread.

> U.S. stock exchanges have invested huge sums of money creating two-tier > markets - building and offering faster data and technology infrastructures > at a price that only a small niche of traders can benefit from or afford, > while at the same time continuing to offer slower products to everybody > else.

[https://www.iextrading.com/about/press/op- ed/](https://www.iextrading.com/about/press/op-ed/)

<comment><USER12> That was true on _one_ exchange in _2012_. It's no longer true today on NYSE, and probably wasn't true at any other venue (Hunsader is able to explain how he found this in a relatively small blog post; dollars to donuts he looked _everywhere else_ for it too).

So: not delaying public quotes is table stakes. What could Google do better than that?

I'm not so much challenging you as I think it's an interesting question, and would like to see it hashed out in technical detail.

</comment></comment></comment></comment><comment><USER31> Now that shows you how far the whistleblower laws have gone ever since Enron. A big step up from what there was before.</comment>


Google penalty for Life? Any hope for reconsideration? Rap Genius next step? - <AUTHOR>

Google's latest punishment for Rap Genius SEO (bad tactics) has been getting lot of attention in the tech news world. Everyone is discussing about the punishment and the bad tactics but it would be great if we could talk on this topic with a new perspective. Is Google's SEO penalty is for Life? Any hope left for Rap Genius (or any organisation affected for that matter) reconsideration? So what do you need to do when this happens to you? I think this topic is worthy of discussion. <comment><USER2> _> Is Google's SEO penalty is for Life?_

No. It's very temporary, they'll may need to rely less on search engines to begin with.

 _> Any hope left for reconsideration_

Is there hope? Yes, definitely.

Should there be reconsideration? No. They did something wrong, as in known to be very very wrong for a very very long time. Reconsidering this for them opens the doors to other people who may or may not have deliberately done the same. I'd like my search results to be free of arbitrary human judgement.

One reconsideration that I don't mind is if Google reverses the blacklisting of the terms "rapgenius" and "rap genius" _but only once_ Google has figured out a alternate change to their algorithm to control such future behavior, while penalizing past transgressions.

 _> Any hope left for Rap Genius?_

Sure, lots of it. This is really not that bad. Sure they lost 2/3 of their traffic overnight, but increasing traffic by 200% is not really a huge challenge on the web. They will just need to rebuild their traction the right way, not quite hard since they have a large existing community.

 _> So what do you need to do when this happens to you?_

I mentioned this in a different comment[0], I personally know of a number of legitimate sites that were invariably affected due to content and blog farms harvesting their content when Google's panda algorithm rolled out.

They did need to go over any other possibilities for getting flagged, we did find a couple of unintentional transgressions, for example one of the authors had been publishing rather short articles, which were more like blurbs of coherent thought, unique but very easy to get flagged under duplicate filters. We did not delete those articles but set strict minimum content length rules.

Had to cut back on a few expenses while in recovery.

In the grand scheme of things, panda was just a blip. Once fixed, the traffic grew exponentially.

Lastly, I welcome these changes, as it gives newer sites and startups a fair chance to survive and grow.

[0] [https://news.ycombinator.com/item?id=6966454](https://news.ycombinator.com/item?id=6966454)

</comment><comment><USER3> It's not some deep mystery. They just need to stop doing the things listed in the Google webmaster guidelines [https://support.google.com/webmasters/answer/66356](https://support.google.com/webmasters/answer/66356)</comment>


On the topic of Steve Jobs - <AUTHOR> http://waxinandmilkin.com/post/9355099157/on-the-topic-of-steve-jobs-i-always-liked-this

<comment><AUTHOR> See also: [http://www.tuaw.com/2011/03/01/mac-os-x-lion-features- hidden...](http://www.tuaw.com/2011/03/01/mac-os-x-lion-features-hidden- tribute-to-steve-jobs/)</comment>


Autonomy CFO trial to shine light on HP deal - <AUTHOR> https://www.ft.com/content/2517ec02-1a08-11e8-aaca-4574d7dabfb6 <comment><AUTHOR> Some context:

Hewlett-Packard wrote down about $5bn of the $11.1bn it paid for Autonomy in 2013, alleging fraud and inflated sales figures.

</comment><comment><USER2> Financial Times is blocked unless you get a subscription -_-</comment>


Ask HN: Downsides of unusual domain? - <AUTHOR> I have a small SaaS idea and one of the more unusual domains is relevant to the product name. In what way might using a rare domain penalise the site in search engines etc? <comment><USER2> It won't affect you in the SERP. If anything unique will be good for getting you to the top for relevant searches.

As long as it fits your product, is pronounceable and someone can type it in correctly when you say it, go for it.

</comment><comment><USER3> AFAIK, no.

You could ask this guy: [https://twitter.com/JohnMu](https://twitter.com/JohnMu)

</comment><comment><USER4> If it is a typo for a more popular search term then it might get corrected every time someone types it in.</comment>


What are the best paying java dev roles in London? - <AUTHOR> What industry/subject/project-types? <comment><USER2> Front-office investment banking. Quant dev, risk, algo trading, low latency, etc. Lead dev can make 100-130k base + substantial bonus. At small hedge funds it's possible to get carry which obviously means the sky is the limit.

Contracts rates are generally in the £500-£1000/day range.

<comment><USER3> As someone who works in the field above (but not as a dev) I'd agree.

There is also a leap in salaries depending on which side of the business/IT dividing line you sit.</comment></comment>


Ask HN: Is Disqus Slowly Dying? - <AUTHOR> Disqus support is basically non-existent now and they're rarely releasing new features... is Disqus dying a slow death post acquisition to Zeta Global? <comment><USER2> Sorry if we missed your support ticket! Could you contact us through this page ([https://help.disqus.com/moderation/user- reporting](https://help.disqus.com/moderation/user-reporting)), and drop either your site shortname or user profile link in your case there and a reply here, so that we can verify your identity and help you out?

To clarify, Direct Support is provided for select publishers running Disqus on their website, and those services may be accessed here: [https://disqus.com/support](https://disqus.com/support) For sites that are not covered with Direct Support, we offer a variety of community resources and support documentation at discussdisqus.disq.us and help.disqus.com

</comment><comment><USER3> I haven't seen Disqus in a long time but that might be because they are collateral damage from my tracker/ad blocker.

Sorry, but the web doesn't need another outlet for Outbrain, Taboolah and other scams.

</comment><comment><USER4> It's still ubiquitous on the Web, would be a shame if we lost all those discussions</comment>


Show HN: Startup Letters - <AUTHOR> http://startupletters.com/

<comment><AUTHOR> The next edition will go out in a couple hours. Sign up now to check it out. :) Would love any feedback on the landing page and the newsletter itself.

</comment><comment><USER2> I love the new landing page. So far I got interesting articles from the newsletter, keep up the good work!</comment>


XVim – Xcode plugin for Vim keybindings - <AUTHOR> https://github.com/JugglerShu/XVim

<comment><USER2> I use XVim and I love it, it's decent enough and I would be editing very slowly in Xcode without it.

Things I wish it would have:

1\. ge and gE

2\. Marks like ``

3\. Changelists so I can g; and g,

4\. Visual block replace.

5\. Something like the surround.vim plugin to surround blocks of text.

The one thing I absolutely love is the keybindings support so I can map something like <leader>f to Open Files (a fuzzy finding tool that comes with Xcode).

Or my favorite, <leader>d to Look up documentation under the cursor. No more option+clicking with your mouse! (Useful for Swift to check what type the compiler has inferred)

Here's a link to my ~/.xvimrc if anyone wants more:

[https://github.com/ajh17/dotfiles/blob/master/.xvimrc](https://github.com/ajh17/dotfiles/blob/master/.xvimrc)

</comment><comment><USER3> This is awesome and probably took considerable work. There is no plugin system for Xcode, which has always bugged me. Why would Apple not make Xcode officially extensible. Xcode plugins make Xcode unstable and break when Xcode is updated.

</comment><comment><USER4> I love this plugin -- it's not perfect, but it gives me just enough of vim to make XCode pleasant to use.

</comment><comment><USER5> Any idea how it compares with the commercial ViEmu plugin? I've never tried their Xcode plugin, but years ago I used ViEmu for Visual Studio and found its emulation pretty solid.

</comment><comment><USER6> And for Visual Studio, there's VsVim: [https://github.com/jaredpar/VsVim](https://github.com/jaredpar/VsVim)

<comment><USER7> VsVim is OK, but I don't find it good enough to replace my usual emacs/evil. I did find it useful to define a gdb-like single key debugging mode. This lets me use keys like the space-bar, s, and f keys to step over/into/out of instead of F10, F11, S-F11. I mostly use the following keys:

s StepInto <Space> StepOver f StepOut b ToggleBreakpoint c Start K StopDebugging u (Move up the stack) d (Move down the stack) B BuildSolution C Compile <M-'> GoToNextLocation

I also remapped C-d/C-u to enter/exit this debugging mode, but since I never use VS for editing I mostly just keep the keys remapped to debugging keys.</comment></comment>


Wrapping a web application - <AUTHOR>

I am looking to have my web application wrapped as a quick way of having my Beta put onto the App store for continued improvement on the product and for marketing purposes. My question lies in the costs, how much would something like this set me back?<p>Thanks, Amit. <comment><USER2> Hi Amit, If I understand your question correctly, you want to understand how much to pay for wrapping your web app into a native app. If this is correct I would say the following:

Firstly, the App Store have guidelines that disallow the acceptance of beta software into it's App store. If your software is not ready for prime time then you shouldn't submit it to the App store. Secondly, I would serious rethink your motivations for doing so, if your product is not ready to launch then there are plenty of services that can get you beta testers for a low price e.g. <http://www.betabait.com>. Lastly, if it's a web app wrapping it into a cocoa web view for example, severely limits your ability to rapidly iterate.

Regarding cost, it really depends on the complexity of your web app. But the cost should be low - probably hundreds of dollars not thousands.</comment>


Fastly's stock opens 34% above its IPO price, then keeps rising - <AUTHOR> https://www.marketwatch.com/story/fastlys-stock-opens-34-above-its-ipo-price-then-keeps-rising-2019-05-17 <comment><USER2> Such a great company, and run by a lovely group of people</comment>


Ask HN: Which oral language (after English)... - <AUTHOR>

would be most useful for a programmer to learn?<p>Just curious. <comment><USER2> Aside from English, I speak Spanish fluently, Japanese conversationally, Tagalog passably, and I'm learning Portuguese and Italian. I know enough German to sing 99 Luftballons, and enough Latin to translate some of the classics. I wouldn't say any of these have been at all useful in my programming career. I've worked with offshore programming teams in Malaysia, the Philippines, India, Belarus, Mexico, and Pakistan, and they all spoke English better than I could ever hope to speak their native language (except Spanish, but even then the guys in Mexico I work with were schooled in the US).

</comment><comment><USER3> You could try learning something really obscure. As far as I know I'm the only hacker within 1000 miles who speaks any Mongolian as well as fluent English. Someday, two buyers are gonna show up in that market and keep bidding against each other till my rates go to the moon! Dreaming about that day is much more fun than trying to compete with tens of thousands of (French|Chinese|Spanish|Russian)-English bilingual programmers. In the mean time, I try to hang out at a webforum from time to time and answer random tech questions in Mongolian, which makes me feel all warm and fuzzy. (Or less charitably: I can't sell my skills, so I give them away for free).

To be perfectly serious, the recommendation for what language is "useful" to learn really depends on your own language learning skill --- which is a combination of inborn talent, knowing what learning methodologies suit you, and experience with compensating for a bad environment (e.g. I assume you live in the U.S. and attend an Anglophone workplace everyday).

Spanish is a pretty typical answer to this question not because it's the "most useful" but because it's not that hard for English speakers to learn, it's more widespread in the U.S. than other foreign languages, and it even has some applicability in tech (a lot of people I've seen suggesting moving to Latin America to cut your cost of living while you bootstrap your startup). Chinese/Japanese is somewhat harder to speak, a lot harder to read, a lot harder to get a visa to the country as a one-man startup, etc.

</comment><comment><USER4> My favourite quote from _Eat, Pray, Love_ was (roughly) "Why would you learn Italian? So that if they ever successfully invade Ethiopia again, you'll know a language which is spoken in two countries?"

With that in mind, it makes sense to learn Spanish, which is spoken through much of Latin and South America. Chinese (Mandarin) has replaced Japanese as the 'next big economic language', so if you're looking to converse with a large new market, then perhaps Mandarin.

But for me, I dream of taking up French again. Limited global reach. No economic benefit. But I love the way it sounds coming out of my mouth. Does a form over function answer negate the 'useful for a programmer' part of your question?

<comment><USER5> Italian is spoken in the canton of Ticino in Switzerland, and is also an official language in the Croatian region of Istria, which was long a possession of the Republic of Venice. That said, yeah, it's not terribly useful in the world at large, but the food is good, the girls are beautiful, and the countryside is stunning in its variety.

</comment></comment><comment><USER6> Lots of important products are designed and developed in Japan and usually are translated into foreign languages quickly but you could get a jump on developing for whatever Sony or Nintendo is doing with Japanese. Japan has a big internal market you might want to do l10n for it sometime, also.

The products designed in Japan (or California) are likely to be manufactured in China. If you want to participate in the final production and delivery of products you program, you'll want to consider Chinese. China's internal market is a desert of failure for software sales, though.

So on a technical basis I'd suggest 汉语 (Chinese) or 日本語 (Japanese). Too bad they're among the hardest written languages in all human history to learn. Save yourself the trouble and pick Spanish, the Internet's third or fourth most popular language. It's not a hotbed of technology but if you're living in the USA, you'll need it soon enough if demographic trends continue.

<comment><USER2> After 4 years of studying Japanese, I can speak enough to carry on a conversation with strangers in Tokyo, order food in a restaurant, and flirt with girls. Writing is a different story, though. I learned hiragana and katakana, but nowhere near the 2000 kanji that a high-schooler knows. I doubt most people would learn enough Japanese to be able to work in a highly technical field where Japanese is spoken and written.

</comment></comment><comment><USER7> I'd vote Spanish (lots of countries speak it, which makes it a good language). Or Chinese - more people speaking it than any other language, I think.

BTW, I started learning Spanish using the "Pimsleur method". It's an amazingly easy and _quick_ way to learn a language, If you're looking for a way to learn a new language, I recommend looking it up.

<comment><USER8> I speak English and Spanish and was using the Pimsleur CDs to learn Japanese. I can't recommend it enough. I gave up when I had no one to talk to and yet after several years, I still remember quite a bit.

</comment></comment><comment><USER9> Germany and Japan have big tech cultures.

<comment><USER10> I know a tiny bit of German and I'm learning Japanese (in Tokyo) right now. I can say that one of the advantages of a language like Japanese is that it really gets you thinking about languages in general. You have to learn to think a different way which is quite interesting.

I guess its sort of like if you learned C++ first, then your next language to learn should be something totally different like Scheme, Scala, Prolog or Erlang - not C or Java.</comment></comment>


Vladimir Putin threatens US arms race with new missiles announcement - 076ae80a-3c97-4 https://www.theguardian.com/world/2018/mar/01/vladimir-putin-threatens-arms-race-with-new-missiles-announcement <comment><AUTHOR> This is what one should expect from MAD. The US should not be able to make a first strike without threat of an effective counter-response.

As people have been pointing out since at least the SDI efforts in the 1980s, it is far easier to develop systems to evade antiballistic missile defenses than to build such defenses.

Remember too that the US is spending some $1 trillion to develop and deploy super-fuze. Quoting [https://fas.org/blogs/security/2017/03/super- fuze/](https://fas.org/blogs/security/2017/03/super-fuze/) "Under the cover of an otherwise legitimate life-extension of the W76 warhead, the Navy has quietly added a new super-fuze to the warhead that dramatically increases the ability of the Navy to destroy hard targets in Russia and other adversaries. ... the new super-fuze increasingly will be seen as a front-line, first-strike weapon that is likely to further fuel trigger-happy, worst-case planning in other nuclear-armed states."

See also [https://freethoughtblogs.com/stderr/2018/02/08/the-new- york-...](https://freethoughtblogs.com/stderr/2018/02/08/the-new-york-times- is-a-shameless-establishment-propaganda-channel/) . "The Russians are developing new weapons to provide them a second-strike capability, and are (reasonably enough) designing them so that they would not be useful in a counter-force strike. It is actually an attempt to ratchet down the situation, slightly."</comment>


Scientists Just Successfully Reversed Ageing in Lab Grown Human Cells - <AUTHOR> https://www.sciencealert.com/scientists-just-successfully-reversed-aging-of-human-cells-in-the-lab <comment><USER2> "Impact Journals, LLC" is on the old list of (possibly) predatory journals: [https://predatoryjournals.com/publishers/](https://predatoryjournals.com/publishers/)</comment>


Ask HN: Who honestly applied to YC Reject? - <AUTHOR>

Show of hands... who honestly applied to YC Reject ( http://ycreject.com/ )?  The deadline for applications is in 2 hours.  I'm interested in knowing the actual response to the program. <comment><USER2> I'll bite: I did.

My logic: almost everyone who succeeds did so with a trail of 'no' behind them and the only way to get those is to put yourself out there. YC's application was something I labored over for a couple weeks and feel I got a great deal of value from that process alone... this is no different and I would have hoped that everyone applied.

<comment><USER3> Ditto that.

I applied as well.</comment></comment>


Indra Nooyi, PepsiCo C.E.O. Behind Health Drive, Steps Down - <AUTHOR> https://www.nytimes.com/2018/08/06/business/indra-nooyi-pepsi.html <comment><AUTHOR> Interesting statistic in the article about a 25% reduction of Fortune 500 female CEOs in the last year. They have discussed female CEOs on Freakonomics podcast a fair bit including the theory that shareholders and employees prefer a CEO with more masculine traits when the economy is good (aggressive, risk- taking, outward-facing). And people prefer more feminine traits when the company is struggling (care-taking, risk-aversion). Interesting theory and the economy has been doing well in the last year.</comment>


Ophelia became a major hurricane where no storm had before - <AUTHOR> https://arstechnica.com/science/2017/10/ophelia-became-a-major-hurricane-where-the-waters-are-usually-too-cold/ <comment><USER2> 30 years ago, almost the same day, a hurricane category 2 on the Saffir- Simpson scale reached France and the British Isles : [http://tempetes.meteofrance.fr/L-ouragan- du-15-au-16-octobre...](http://tempetes.meteofrance.fr/L-ouragan- du-15-au-16-octobre-1987.html) (in French).

Edit: fixed the quotation on the Saphir Simpson scale that was incorrectly taken from the article.

<comment><USER3> That did not make landfall as a hurricane.

"In south-east England, where the greatest damage occurred, gusts of 70 knots (130 km/h; 81 mph)" [https://en.wikipedia.org/wiki/Great_Storm_of_1987](https://en.wikipedia.org/wiki/Great_Storm_of_1987)

However, it takes Sustained winds of 111–129 mph to be a Category 3.

Also the great storm of 1987 made landfall significantly south of this one. [http://engweb.swan.ac.uk/~hewstonr/Great_Storm.html](http://engweb.swan.ac.uk/~hewstonr/Great_Storm.html) vs. [https://en.wikipedia.org/wiki/Hurricane_Ophelia_(2017)#/medi...](https://en.wikipedia.org/wiki/Hurricane_Ophelia_\(2017\)#/media/File:Ophelia_2017_track.png)

<comment><USER2> The Meteo France article I quoted is actually a bit misleading, it talks about a category 3 hurricane in a not really rigorous way «damages caused by this storm has been on par with the ones done by a category 3 hurricane» even tough you're right the sustained wind speed has been below the limit for a category 3. It's still strong enough to be classified as a category 1 (in england) or 2 in France.

The actual reason why it wasn't a proper hurricane was because this word is reserved for tropical cyclones. Neither the great storm or Ophelia are tropical.

</comment></comment></comment><comment><USER4> It happens also to have turned the afternoon sky a deep Mars-like red. It was quite a fascinating sight, lots of people comparing it to Blade Runner 2049 or Instagram filters.

<comment><USER5> Heh, we have a similar experience here in the bay area from all the wild fires.

<comment><USER6> Confirmed. The iOS weather app even listed "Smoke" instead of "Cloudy" or "Overcast." It's not funny though. For example, my folks live on a mesa surrounded by pine trees and vegetation on three sides. Last week, an intentionally-set fire (quickly contained) took out their phone lines.

</comment></comment></comment><comment><USER7> Many parts of the UK (including London where I am) had an oddly beautiful phenomenon thanks to Ophelia [http://www.bbc.co.uk/news/uk- england-41635906](http://www.bbc.co.uk/news/uk-england-41635906)

<comment><USER8> When I saw it today it reminded me of the nuclear winter in The Strain.

</comment></comment><comment><USER9> It has been a few weeks now that the air here has been uncharacteristically warm. I've heard quite a few people remark on this totally unprompted. Today was 21 degrees here, when it normally would be 12-15 tops.

All that energy will have to go somewhere.

<comment><USER10> The south of the UK often has a last gasp of summer in late Sept or early Oct. So the temps aren't that unusual.

A couple of years ago I was living on the south coast.

Towards the end of a recent summer we had a no-holds-barred tropical storm. Not only did lightning put on an incredible show across a huge area, but the external air temperature and humidity were tropical.

Going outdoors was like stepping into a hot house, with torrents of warm rain.

I've never experienced anything like it in the UK before.

</comment></comment><comment><USER11> Not much of an effect in Dublin so far.

<comment><USER6> Any rain or clouds?

<comment><USER12> Light rain. Bit of wind.

Have to say, while it is tragic that a few people have died on the roads, the way that Irish people handle a "National Emergency" like this is hilarious. Gotta love some of the dry humour going round!

<comment><USER13> N. Ireland for some reason has decided to close schools tomorrow as well. Seems like a bit of an over reaction.

<comment><USER14> They literally have no parliament, Government or decision making capacity right now. Both days of closures in NI were clear copy and paste efforts within an hour of the rest of Ireland announcing same. Not a great way to make decisions !

<comment><USER15> Hmmm... why wouldn't it be a good way to make decisions, assuming that the Republic of Ireland's weather people are competent, and their own people were unavailable (for whatever reason)?

Some things are independent of politics, and tomorrow's weather is pretty high on that list. :-)

</comment></comment></comment></comment></comment></comment><comment><USER16> As an aside,...

" _The Irish National Meteorological Service, Met Éireann..._ "

Someone please tell me there's an "An Bord Rain" joke around here.

[https://en.wikipedia.org/wiki/Bord_Snip](https://en.wikipedia.org/wiki/Bord_Snip)

(Dang it, I can't find a link for An Bord Bun.)

<comment><USER17> That'd be An Bord Baiste to you...

<comment><USER18> The baptised board?

Do you mean An Bord Báistí?

<comment><USER17> Feck it, I do. My Irish is banjaxed after ten years away from home.</comment></comment></comment></comment>