I ask 100 information questions to four digital assistants - forgot-my-pw https://vlad.d2dx.com/the-great-assistant-skills-comparison-google-alexa-siri-and-cortana-head-off/ <comment><USER2> What's interesting to me reading this is the expectation that voice search should do things that text search currently doesn't do. If I ask Google about Bill Murray running for president, I presumably am looking for articles that mention the words "Bill", "Murray", and "President". I would expect to return articles that best match the query, and I would never expect Google to be able to tell me the difference between real and fake articles. For a server to answer the author's question it has to understand the question so well it can change the question into "What is the list of people that have run for U.S. president?" and then check if Bill Murray is in that list. That's a _tall_ order.

We are moving past expecting the most word matches into expecting the server to understand what we really want. Voice search is more like the "I'm feeling lucky" button, because it takes longer; you only have time for one answer and the first answer has to be right. It comes without the expectation that you're _lucky_ if the answer happens to be right, now we _need_ the first result to be the rightest result there is.

So the glass is half empty. I personally prefer to see it half-full, but it's also true and the critique is more valuable and interesting than optimism.

<comment><USER3> It seems like there's something akin to the uncanny valley effect going on here where a voice UI invites people to think about the other end of the conversation as a person and then be disappointed when they hit the edges of what it's designed to do.

There's a really interesting discussion to be had about how UI decisions can make that process smoother — I really liked [https://bigmedium.com/speaking/design-in-the-era-of-the- algo...](https://bigmedium.com/speaking/design-in-the-era-of-the- algorithm.html) as a call for how you can make the failure modes of the system more graceful. I think a lot of the success in the next decade or so is going to come from the places which figure out good answers for not making a system which seems to promise more than it can deliver.

<comment><USER4> The fact that we interact by voice leads toward a sort of inadvertent theory- of-mind about the other party, which makes the pulling away of the curtain with so many of the answers much more jarring. Voice interaction seems to recruit a much deeper evolutionary expectation than the much more recent phenomenon of typing and reading.

<comment><USER5> That's some heavy speculation.

<comment><USER4> Well, this is Hacker News.

</comment></comment></comment></comment></comment><comment><USER6> As much fun as this test is, it dodges the most interesting question of all: "Are these machines supposed to be talking search engines?"

I'm increasingly believing that the answer is: "No." These machines (especially Alexa) are rapidly gaining popularity while still providing pretty ragged answers to search queries. So we should start asking: "Are they taking on a different function that didn't match our early expectations?"

In a word, yeah. Alexa is a really nifty jukebox for those of us that don't have the good sense to create formal playlists. It's a handy kitchen timer, especially if you've got multiple pots doing different things. It's a better alarm clock and a better purveyor of soothing bedtime sounds. (If you're asking: Good god, how many people really want or need that, think: Fussing infants.)

Smartphones already provide pretty excellent search results on the fly. I'm not sure voice-powered assistants will re-solve that problem with great success. But there are a surprising number of rudimentary needs around the house for which a voice-enabled device becomes quite handy.

<comment><USER7> These digital assistants are just begging for an app store. Search is just the first app, jokes and weather are other useful apps. These could easily follow a similar product life cycle pattern as smartphones.

<comment><USER8> They have app stores: [https://www.amazon.com/b?node=13727921011](https://www.amazon.com/b?node=13727921011)

<comment><USER9> * One has an app store

</comment></comment></comment></comment><comment><USER10> I tried the conversational weather one on google assistant on my phone.

"Should I take an umbrella tomorrow?"

"No..." and shows me tomorrows forecast.

"What about the day after?"

"No..." and shows a forecast that when I look more closely at I notice is for _today_.

Neither of these also spotted that I'm heading to another city tomorrow, which is in my calendar. If I changed it to "do I need to take an umbrella for my trip tomorrow" it just searches google and gives me a search result suggesting I take a small folding umbrella... for a trip to Thailand.

<comment><USER11> It might be so, but remember that experts were predicting computer will beat top humans in Go in 10 years. Maybe next month there will be a breakthrough with NLP. The amount of research and compute going into this problem is amazing, and we don't know what's around the corner.

Also, it might be possible to create a much better assistant today, but it would be too expensive to offer to the public for free. What if it requires 100 TPUs to run?

<comment><USER12> This isn't an NLP problem, it's a coder problem. These solutions already exist, Google/ other assistant providers just need to dedicate the man hours to make it happen.

<comment><USER13> No, they don't (generally) I do NLP for work, and the edges cases are so frequent and aggregate over processing pipelines.

That means the end to end performance of the system is bad, and it isn't clear how to systematically fix it.

In this particular case there is the added system integration too, which is "just programming".

<comment><USER10> Perhaps, these just feel like such common things to ask and do that I don't get why they're not planned for. The conversational side works for the weather, the remaining problems are:

1\. There is an in-built assumption I am always where I currently am. This part isn't anything to do with the NLP.

2\. "the day after" is translated to "today", possibly. [edit - see lower, it is in only one case]

3\. This is more of an NLP one, it understands that the context of "weather" carries from one question to the next, but not the timing. So asking for the weather tomorrow and then "one day later" gets tomorrows as well. [edit - <AUTHOR> complex than this, it's actually working in some areas and not in others]

I'd like to see what user-stories it's trying to solve, because apart from setting timers and alarms it's been massively hit and miss for me.

I tried to repeat what I'd put in and this time I had:

"What's the weather the day after" \- translated to tomorrow, with no context, that makes sense.

"What's the weather today?" \- weather today, followed by "What about the day after?" which gave me results about the film.

"What's the weather tomorrow?" \- weather tomorrow, followed by "what about the day after?" which then worked.

So it works just fine for "weather" but not for asking if I need to take an umbrella. And it doesn't work if I ask for today then the day after, but does for tomorrow and the day after.

Why does the context get passed on for the day correctly for "tomorrow" but not for "today"? Why can it get that "the day after" means tomorrow, unless I've asked about an umbrella in which case it means today? At the core of my question is _how is it this inconsistent_?

What tests have they got around this?

</comment></comment></comment></comment></comment><comment><USER14> I found it curious the author didn't like that google was quoting webpages. I actually liked that Google was not taking credit for the info and Google was informing where the info was coming from so I can decide if I want to trust that info or not.

Of course if I ask for a fact like "How many inches in a meter" I just want the answer. But, if I ask "what's the weather going to be tomorrow" I might prefer an answer like "badweather.com says it's going to rain tomorrow" so I can then think (ugh, badweather.com is always wrong) and ask "What does goodweather.com say about tomorrow's weather". Ideally I could ask the assistant to use a particular site by default. This is specially true for me because Siri's default doesn't seem very accurate to me being that I live on the other side of the world from the offices of the company they use for weather info.

<comment><USER15> Heya,

I'm sorry for the confusion. My issue with quoting webpages is not that it quotes them -- that's fine -- but that it does so in a very verbose manner. This leads to information overload. For example:

Me: What is the boiling point of water at an altitude of 1km?

Google: At sea level, water boils at 212 °F. With each 500-feet increase in elevation, the boiling point of water is lowered by just under 1 °F. At 7,500 feet, for example, water boils at about 198 °F. Because water boils at a lower temperature at higher elevations, foods that are prepared by boiling or simmering will cook at

The problem is with the vast quantity of information, and the fact that some is both irrelevant and truncated. The last sentence is incomplete and cut off, yet as a listener I have no way of knowing this. I will thus try to remember it, at the expense of the facts that came previously.

When reading a webpage, the important part is to read the specific parts of interest, and not overload the user. If it can't do that, it risks providing irrelevant or, quite frankly, confusing data (such as the odd answer to how much a Dreamliner weighs). I don't know if that's better than not providing an answer at all.

</comment></comment><comment><USER16> One of the most 'magical' experiences I've ever had using Google Assistant was the following exchange:

Me: "Okay Google, What's the latest album by Death Cab For Cutie?" Google: "The latest ablum by Death Cab For Cutie is Kintsugi"

Me: "Okay Google, Play the album Kintsugi on Spotify" Google: "Okay, asking to play Kintsugi. [album starts playing]"

<comment><USER17> Wouldn't it have been a lot more magical if it had simply asked, "Would you like me to play it?" (knowing that you have a Spotify subscription) at the end of answering your question?

Or, at least, for you to be able to say "Play it!" rather than the unnatural "Okay Google, play the album Kintsugi on Spotify"...

<comment><USER18> I would have preferred a straight answer, without a servile question at the end.

</comment></comment></comment><comment><USER19> I got 22 out of the "40 verbose Assistant questions" correct. Not bad! I beat them all (as a percentage).

Maybe not a bad idea for a gameshow.

<comment><USER20> That's similar to how The Chase[1] works, except contestants go head to head with a professional quiz master instead of a digital assistant.

[1] [https://en.wikipedia.org/wiki/The_Chase_(UK_game_show)](https://en.wikipedia.org/wiki/The_Chase_\(UK_game_show\))

</comment></comment><comment><USER21> more people are starting to appreciate that AGIish stuff is actually really, really hard.

<comment><USER22> The ability to function as a virtual assistant, even at the level of a not-so- sharp intern [1], would be a killer app.

Give it some parameters for a trip you're taking. It comes back with some options and follow-up questions. We are a long way from that point. Even that not-so-sharp intern has a huge amount of internalized knowledge about general preferences, cities, airports, etc. and probably knows questions to ask to narrow things down.

I strongly suspect there are other domains where a lot of people are assuming we're 90% there and we're not.

[1] Not to insult interns or any other group. I just mean you don't need to be at experienced executive assistant level to be really useful.

</comment></comment><comment><USER23> I find Google Assistant to be very good at answering most questions.

Also, I can accurately get the weather from Siri most of the time.

</comment><comment><USER24> I notice some different answers on my devices. For example on the "Are tomatoes vegetables?" question my Google Home states that they are definitely a fruit. (quoting Oxford Dictionary)

Edit: And "What's the height in meters of the Empire State Building?" gets me "381 meters, 443 meters to tip"

</comment><comment><USER25> I'm surprised by a number of the failures, such that I did wonder if the failure might be happening on the speech recognition side rather than the response generation side.

Both Google Home and Alexa have very little problem recognising my speech whilst friends often struggle even when they seem to say the precise same phrase, to the point it's mildly entertaining. With Google I suspect they've tailored to my voice (I've used voice commands extensively for several years) but I've only had a Dot briefly and it worked well from the start. Another surprise is that they cope well with my perculiarly English English phrasing and pronunciation, but I'm sure there are lots of less widely spoken dialects that would throw them.

<comment><USER22> Alexa is the first thing I've owned that really does quite a solid job in the voice recognition department (whatever its failings to return something useful based on that recognition). Siri on my phone is rather hit or miss by contrast.

I suspect that the microphone array has a lot to do with it. Anecdotally, I've read pieces by people saying that homebrew "Echos" together with the Alexa APIs aren't as good as an actual Alexa.

</comment></comment><comment><USER26> I keep coming back to the idea that progress towards AGI might be made by someone working on a "coordinator" agent. We might have several narrowly focused agents with deep knowledge in particular domains: a mathematician, a fact-checker, a botanist, a structural engineer, etc.; then have an agent that broadly understands how to route requests to the right vertical. Maybe that's already descriptive of the underlying architecture for some of these agents. The alternative might be that we interface with several different conversational agents, and like interfacing with people, we use our judgement to decide which specialist to ask.

<comment><USER27> That's kind of what Watson did, but that level of architecture hasn't made it into personal assistants yet

</comment></comment><comment><USER28> I'm curious about how accurate the assistants were at listening. That used to be the most pressing issue with voice commands that had relegated the technology to a running joke. It appears that's understandably what the companies have been focusing on so far, but there's still work to be done to get to 100% accuracy of listening, especially when you take into account exotic names and interchanging between languages and slang.

</comment><comment><USER26> Might be interesting to test with Wolfram Alpha as well. It looks like some of the questions wouldn't fit the WA API, but I'm curious how it would score.

</comment><comment><USER29> Forget about questions, I cannot even get Siri on Apple TV to recognize what I am saying. I have often wanted to keep a kind of journal like this poster but I suspect it would recognize the correct words about 30% of the time.

My wife who, unlike me, is not a native English speaker has probably a 10% success rate. This is why any kind of forthcoming voice-response Apple device is completely a nonstarter to me.

<comment><USER30> What a weird conclusion, that future -- and presumably better -- tech would be a nonstarter because current tech doesn't work for you.

<comment><USER31> The parent poster is dubious that improvements in features will coincide with improvements in recognizing his voice. Voice assist functionality could be 200% more awesome but if it specifically doesn't seem good at just recognizing what he has said such functionality is useless to him. This isn't terribly strange at all.

</comment></comment></comment><comment><USER32> Ensemble for the win, I suspect that they can perform better than the combined individual best when sharing training resources and models.

</comment><comment><USER33> > Siri took the crown on factual questions, but surprisingly did poorly on > reasoning ("Queries") where I expected the Wolfram Alpha-backed service to > get flying colours.

Didn't Apple ditch the WolframAlpha integration pretty quickly after Siri was released? I remember a lot of the Wolfram type queries stopped working shortly after release.

<comment><USER34> They didn't ditch it, but they did fuck it up and seem to prefer Bing searches now over Wolfram Alpha queries.

</comment></comment><comment><USER35> So I decided to ask Siri some of these that he listed as giving a Bing search answer that I felt Wolfram would have answered correctly for Siri. I my case I did get the correct answer, not a Bing result.

Where does the Jackfruit grow?

What is the boiling point of water at an altitude of 1km?

What is 1km in feet?

How far away is Disneyland?

For "km" I said "kilometer" and not "km".

</comment><comment><USER36> i wonder if there would be any use in services that don't respond in real- time.

I think these digital assistants are nerfed by the real-time response requirement. I'd be happy to ask some of those questions, and get a pop up in a few minutes. And they could be of much higher quality as they can be processed and better researched.

</comment><comment><USER37> I'd be interested to see how Siri performs on different devices. The Siri on your phone is not the same Siri that you have on your Apple TV. The Apple TV version is much better at giving information about tv shows, movies and music, but terrible in comparison for everything else.

</comment><comment><USER38> "Okay Google, spend my money."

<comment><USER39> The Alexa version :) [https://xkcd.com/1807/](https://xkcd.com/1807/)

<comment><USER40> "Okay Google, infer semantic meaning from my words"

</comment></comment></comment><comment><USER41> Why isn't siri as half as smart as Wolfram Alpha's box? Someone should license them!

I want to be able to ask basic factual questions while driving, get the answers and dig deeper.

Until then I would like an audio service like Google Helpouts used to be, on demand. Like Magic service.

<comment><USER42> Wolfram Alpha is slow. Even if it's right, it's only good for knowledge questions out of its database - things like public figures ("how many children does barak obama have"), physical statistics ("what is the melting point of tungsten"), and so on work fine. However, topical ("what about aluminium?"), temporal ("what's the weather?"), and location-based ("show me restaurants nearby") are outside of scope for wolfram alpha entirely - so a given app must aggregate.

Why don't apps aggregate? The "can you handle this?" api endpoint is frequently returns false positives, and the proper API is really slow (multiple _seconds_ ) for negatives . If we get a false positive, or something hard to detect as a negative, that's the only answer we can show. And since a voice assistant is expected to return one answer quickly, this is straight out.

</comment></comment><comment><USER43> One problem is that the computing power dedicated to each user is minuscule. If you could dedicate a super computer for processing every input, you could have a much more sophisticated system that could easily deal with all of those queries.

</comment><comment><USER44> It's interesting that while many fails, there is still one that wins. That is if you combine the efforts of these 4 digital assistants, you'll get a much smarter one. Do they have an API? Can you query siri, cortana, etc..?

<comment><USER45> There are APIs for both Amazon and Google's voice assistant services. Not surprisingly Siri doesn't expose a public one, I've no idea about Cortana. I've messed around a little with them on the Raspberry Pi.

This idea, while simple in principle, might be kinda annoying in practice. You're still left with similar issues - how do you decide which talking cylinder service answered the question best? Do you play all of the answers? For me I'm fairly sure listening to all of them in a row would frustrate me even further - just waiting for Alexa to finish telling me the news headlines is sometimes kinda annoying, especially when that information in visual form can be grokked almost instantly. Many of these devices, especially the Google one, are getting better at context based followup questions - managing who to send your follow up question to could be kinda crappy as well. I suppose you could do one device that could ask each service individually ("Alexa...", "Ok Google..."), but in my experience as soon as I get one bad answer, I inevitably just use google.com to find what I need rather than risk wasting my time on another failed conversation.

The main part that I've found hard to do in home rolled voice assistants is microphone arrays. Almost all these devices use pretty sophisticated microphone technologies for things like noise cancelling, subject isolation etc, which so far has been non-trivial to do to a similar standard in homemade versions of them. It also certainly used to be the case that creating your own "hotword" system to call the Alexa API was technically against the ToS (it allowed you to use a button press to call Alexa instead), as naturally Amazon would rather you buy a real Echo. No idea if this is still the case, and at any rate Amazon can't really enforce this either, but worth mentioning.

<comment> forgot-my-pw A funny popular experiment is the seebotschat Twitch account who livestreamed 2 google homes running Cleverbot API talking to each others.

Here's a short highlight: [https://www.youtube.com/watch?v=WoI6_z2mfdY](https://www.youtube.com/watch?v=WoI6_z2mfdY) Some implementation details in AMA: [https://redd.it/5nz3eb](https://redd.it/5nz3eb)

</comment></comment></comment><comment><USER46> Man asks Amazon digital assistant about the price of Lay's chips, is disappointed when it "has little interest in having a conversation about it" and wants to sell it to him instead. o_0

</comment><comment><USER47> English is a terrible language for this, unfortunately it's the only one I speak.

Not sure how other languages cope, I suspect the simpler ones cope much better. We almost need a spoken equivalent of SQL

<comment><USER48> English is best known for being simpler than average, not more complex. It's one of the flagships (along with Latin / Mandarin Chinese / Swahili) for the theory "languages which are widely learned by adults become simplified over time".

</comment></comment><comment><USER49> Google's thing can't even figure out my wife's Italian name most of the time. It's quite frustrating.

<comment><USER50> My girlfriend has the name Taryn and Siri really struggles with it, usually correcting it to Karen or Terell (both names in my address book). Alexa does better but probably because it doesn't know about the other names.

<comment><USER51> Not that it excuses Siri's shortcomings or helps if you're referring to her by name mid-sentence texting to someone else, but you can give assign nicknames in your Contacts app. It might make it less frustrating to dictate texts or start calls. So instead of saying "Call Taryn" and getting it misheard you could say "Call my girlfriend".

</comment></comment></comment><comment><USER52> It's far less than half for me. Maybe less than 5%. Examples of queries that I frequently need that won't work:

\- OK Google, please download an offline maps area for Yosemite National Park and about 80 kilometers around it.

\- OK Google, navigate to Yosemite National Park. Highway 120 is closed so please remember that when we get to an area without reception, do not route me via 120 in the offline maps.

\- OK Google, let me know when we reach a place along our route where I can buy an SD card.

\- OK Google, let me know when we reach the last Trader Joe's or Safeway along our route that is still open.

\- OK Google, when is the next Caltrain arriving in Palo Alto that stops in San Bruno?

\- OK Google, if I miss that train, when is the next train?

\- OK Google, get me an UberPool for 2 people to Castro St. in Mountain View as long as it's under $10 and would arrive within 30 minutes.

\- OK Google, what is the name of the driver and what is the license plate?

\- OK Google, navigate to my friend's party on Facebook.

\- OK Google, call my friend 5 minutes before we arrive. Their number is in the wall on the event page for my friend's party.

\- OK Google, find me a restaurant that has non-americanized Chinese food, recommended with higher frequency on Chinese language websites than English websites, and has vegetarian options.

\- OK Google, connect to my Nest thermostat. My username is XXX and my password is YYY. Turn on the air conditioner 30 minutes before we arrive home based on the navigation.

\- OK Google, close that Java error that popped up and is covering the navigation.

\- OK Google, please zoom out the map slightly so I can see how far we are from the destination.

\- OK Google, please go back to the normal navigation view.

\- OK Google, navigate to my next calendar event's location.

\- OK Google, install Facebook Messenger, login as XXX with password YYY, and message ZZZ saying that I'll be late by however much the Uber app estimates.

\- OK Google, turn off my alarm clock whenever I am biking or driving.

\- OK Google, let me know if I get an e-mail from XXX in the next 2 hours.

\- OK Google, please block all calls except from XXX for the next 1 hour. XXX's phone number is in their e-mail signature.

Yeah. We're a LONG way before assistants are useful. The pieces are all there. It's not a machine learning problem anymore. It's just that there are just way too many walled gardens between the various parties that hold the data necessary to be useful.

<comment><USER40> > there are just way too many walled gardens between the various parties that > hold the data necessary to implement any of the above.

This is changing. If Google had their way, they would be assigning IPV6 addresses to bits of dust lying around in your house and trying to assign semantic meaning to them. _If_ they had their way.

</comment></comment><comment><USER53> I'm gonna make a service where you ask my service and it answers the 4 answers given :D

</comment><comment><USER54> I would like to know how much the pack of lies costs?

<comment><USER55> "It's right there on their website. These numbers do not accurately represent the price you will pay"

</comment></comment><comment><USER56> Shameless Plug: I work at Semantics3 [[https://semantics3.com/](https://semantics3.com/)] - an API for product and pricing data.

These 4 digital assistants should partner with us to help their users find the prices for a pack of Lays chips, iphone, etc. ;)</comment>


Ask HN: Which Mobile Phone do you use? - <AUTHOR> <comment><USER2> Lenovo P2, the best battery life you can get from a mobile</comment>


Will we have enough clean energy? - <AUTHOR> There's lots of open land in the world.  Is 100% clean, plentiful energy simply a matter of covering enough of it with solar panels?  I know storage is an issue (for nights and cloudy days) but with nearly infinite clean energy, surely we can solve those as well - even inefficiently?  Basically, should we be very optimistic about the future? <comment><USER2> Quite a few people think that there has been a breakthrough in nuclear fusion at MIT. This is a video of Dennis Whyte, director MIT's Plasma Science and Fusion Center: [https://www.youtube.com/watch?v=KkpqA8yG9T4](https://www.youtube.com/watch?v=KkpqA8yG9T4)

Whyte's team founded a company, Commonwealth Fusion Systems, which is actually building the stuff in that video: [https://finance.yahoo.com/news/commonwealth-fusion- systems-r...](https://finance.yahoo.com/news/commonwealth-fusion-systems- raises-115-125900198.html)</comment>


Ask HN: What's the best blogging platform for startup companies - <AUTHOR>

We are looking for a blogging platform to announce new features, post updates, code fragments, and anything neato  related to our business.<p>So far looked at: Tumblr and posterous, any others? <comment><USER2> Since a blog can be an integral part of your company and the "voice" that people hear I think its worth doing a custom theme and Wordpress install to match your site design and host it on a sub-domain (but on a different server for security reasons, as mentioned in this thread).

If I see a <http://webapp.posterous.com> blog for a company with the default theme I think "Oh, this is probably another MVP this guy churned out over a weekend".

I've always like <http://blog.twitter.com> which is a blogger-hosted blog, but with a very nice design.

</comment><comment><USER3> You can't go wrong with any of the blogging platforms, and it really depends on your preferences.

My main allure is simplicity and speed to set up, so I went with Posterous.

</comment><comment><USER4> Out of the standalone systems, I prefer Wordpress. It is powerful and simple, but it should (ideally) be run on a separate server for security reasons.

We wont be using Wordpress for Freeversation.com; We plan on integrating a Django blogging application into our main codebase, to avoid rewriting Django templates as Wordpress themes.

</comment><comment><USER5> We use Posterous for exactly that: <http://blog.advection.net/>

</comment><comment><USER6> We decided to go with Tumblr, Startup is <http://www.quoteroller.com>. Pretty easy and flexible in terms of customization, but, unfortunately sub-domain hosted. Once we get a bit time I think will move to Wordpress.

</comment><comment><USER7> I prefer WordPress, it's super easy to install, manage, and run and you get a bunch of free services from WordPress.com even if you host it on your own server.

</comment><comment><USER8> I vote for Posterous. Really easy.

</comment><comment><USER9> Tumblr is quick to work with, except for their downtime sometimes.

</comment><comment><USER10> If you're going down the hosted route then you should use either Wordpress.com or Posterous. The reason I didn't include Tumblr in this is because, its more of a 'reblogging' type service and is different to what Wordpress/Posterous offer.

However, if you are going down the self hosted route so you can host it at blog.yourdomain.com then I'd again recommend Wordpress since its a great service, and its really easy to work with.</comment>


The Nature of Code - <AUTHOR> https://natureofcode.com/book/introduction/ <comment><USER2> Helped crowdfund this project several years ago when I was "really into" Processing. Can't recommend it enough. The author has a great YouTube channel where he works through examples and illustrates techniques such as those addressed and illustrated in the book. Highly recommended.</comment>


Morphdom – fast and lightweight DOM diffing/patching (without the virtual part) - <AUTHOR> https://github.com/patrick-steele-idem/morphdom/ <comment><USER2> I'd take it with a grain of salt.

      - Benchmarks are only for Safari and without good documentation for how they
        were performed.  Measurements are only in the 100ths of seconds where these
        measurements should have much more granularity.
      - The TodoMVC (arguably the most real-world) benchmark is nearly 50% slower
        than virtual-DOM.
      - virtual-DOM is fairly mature at this point where as MorphDOM looks very
        young.
      - virtual DOM implementations allow you to do things like React Native

<comment><USER3> Noted. Meanwhile, this is 12.3kB (uncompressed) while React's core (minus addons or anything) is 121.7kB. For something as young as this, and to do fairly well in a number of benchmarks compared to the highly engineered piece of work that is React is pretty great.

This library can only get better. It's great to see someone trying to make a more lightweight React.

Also, you can see exactly how the benchmarks are run through the /test folder. My test using PhantomJS gave me the same results as the recorded statistics against Safari. And because it uses the browser's built in DOM toolkit rather than a virtual one, (in theory) it should become faster whenever browser DOMs become faster.

<comment><AUTHOR> On the following note:

> it should become faster whenever browser DOMs become faster

The killer feature of React is that it made it possible to efficiently rerender the entire UI and for developers to _not_ have to write code to manually manipulate the DOM. I hope browser vendors take note of this and do more to support this use case. Ideally, morphdom should not even exist and we could instead have the browser natively do the job of diffing/patching the DOM. Admittedly, there would have to be a lot of details to work out, but React, virtual-dom and morphdom are working in the right direction. Behind the scenes, the implementation might change and any good UI framework should make this a non-breaking change.

</comment></comment></comment><comment><USER4> This sounds very similar to [http://google.github.io/incremental- dom/#about](http://google.github.io/incremental-dom/#about) (esp skipping the virtual part).

Can anyone point out any notable differences?

<comment><AUTHOR> Here's the notable differences based on my understanding after looking over the docs and the source code:

\- morphdom does diffing between two real DOM trees while incremental-dom does diffing between virtual DOM nodes and real DOM nodes \- incremental-dom is only meant to be used with templates that compile to use incremental-dom (won't work with templates that render to HTML strings) \- morphdom can be used with any templating engine that produces an HTML string (or real DOM nodes) \- With additional improvements, morphdom could be used to do diffing between a real DOM tree and a virtual DOM tree as long as the virtual DOM nodes are upgraded to real DOM nodes if they need to be added to the final real DOM. In addition, the virtual DOM nodes would need to have an API that overlaps with the API of real DOM nodes.

Also, it appears that incremental-dom doesn't reuse DOM nodes that move to a new parent, but I could be mistaken. Hopefully someone more familiar with incremental-dom will chime in.

</comment></comment><comment><USER5> The virtual dom solutions can run outside, without a _full_ DOM implementation, which are very slow and resource intensive on a server. Which is needed for client+server-side rendering options.

<comment><AUTHOR> You don't need a virtual DOM or real DOM on the server to render HTML. The fastest way to render pages on the server is to compile templates into JavaScript programs that produce an HTML string (not virutal DOM nodes and definitely not real DOM nodes) and you can use almost any HTML templating engine to do client+server side rendering. morphdom is only meant for use in the browser when you need to update the DOM based on new HTML/DOM nodes.

<comment><USER5> True... but React offers a lot in terms of client-side components, and being able to re-use components outweighs having to use a lesser templating system and/or having to manage two rendering paths for each component. Beyond that, virtual DOM nodes aren't too different from an abstraction of rendering against a compiled template.

</comment></comment></comment><comment><USER6> Currently using virtual-dom to power my front-end system, but have been waiting for something like this. Can't wait to see how it works in comparison.

</comment><comment><USER7> [https://news.ycombinator.com/item?id=10063565](https://news.ycombinator.com/item?id=10063565)

<comment><AUTHOR> For the record, I got an email from Hacker News asking me to repost it since they thought it was a good story and it did not get many views. It's not wise to post something to Hacker News late on a Friday :)

<comment><USER7> I refrained from providing commentary on whether or not posting it a second time is "good" or "bad". While small, there was commentary on the previous link, and that commentary might be valuable to people. I personally think that Hacker News should have an integrated feature that shows "previous discussions" of every posted link. FWIW, reddit has that feature, and it is extremely valuable. (I provide this extra context, as someone down voted my attempt to be useful in addition to you providing a semi-defensive response.)</comment></comment></comment>


The Rules of Optimization: Why So Many Performance Efforts Fail - <AUTHOR> https://hackernoon.com/the-rules-of-optimization-why-so-many-performance-efforts-fail-cf06aad89099 <comment><AUTHOR> I don't think most new engineers intuitively grasp the ramifications of Amdahl's Law:

[https://en.wikipedia.org/wiki/Amdahl%27s_law](https://en.wikipedia.org/wiki/Amdahl%27s_law)

It really doesn't matter if you can make an individual task 1000x faster if your program only spends 1% of its time doing that task.</comment>


Ubuntu To Ditch X Server For Wayland - <AUTHOR> http://digitizor.com/2010/11/05/ubuntu-to-ditch-x-for-wayland/ <comment><USER2> This is a huge shift if it happens. X has been synonymous with GUIs on unix for more than 20 years, and has accumulated all the attendant baggage (both good and bad).

There are also some big consequences for the linux landscape. The first and obvious one is that if other distros don't imitate Ubuntu they will drift farther apart. People complain about Ubuntu going their own way _now_ , image what it will be like if they're on Wayland+Unity and others stick to X+Base Gnome.

Another, subtler consequence of Shuttleworth's announcement is the message it sends to graphics card vendors (well, NVIDIA really). Wayland is dependent on the newest linux graphics stack, very much by design. But that stack is relatively new (e.g. kernel mode setting) and nvidia never updated their driver architecture to match. I assume this will force their hand because Ubuntu is too big to ignore.

<comment><USER3> Ubuntu is certainly not too big to ignore. I have never seen any source say Linux (let alone Ubuntu) has more than 5% marketshare. Does your company support 800x600? How about IE6?

<comment><USER2> Uh, NVIDIA made the decision to fully support linux a long time ago. _Within that ecosystem_ , Ubuntu is too big to ignore.

<comment><USER4> "NVIDIA made the decision to fully support linux a long time ago."

NVIDIA is not legally obligated to support Linux; are they?

<comment><USER5> "NVIDIA is not legally obligated to support Linux; are they?"

Yes, I think they are because of contracts with big guys that need Linux on things like supercomputers(almost all of them have linux inside).

<comment><USER6> Interestingly enough, a lot of developers argue that Nvidia and ATI are in violation of of the GPL by releasing proprietary drivers for Linux. The language is unclear on whether or not the kernel's GPL license extends to kernel loadable modules or not, many say it does.

</comment></comment></comment></comment></comment></comment><comment><USER7> I give this the thumbs-up. I'm not particularly fond of X11. In fact, I suspect it has been holding back GUI on GNU/Linux for a while.

<comment><USER8> A _while_? You must have never read the UNIX-HATERS Handbook (<http://simson.net/ref/ugh.pdf>), which should be required reading for any unix fan, along with the Plan9 design documents (cat-v.org). In this case, "the X Windows disaster" chapter seems most appropriate. Feel free to ignore the VMS fanboys, though.

And yes, I am a dedicated unix (OpenBSD) user.

<comment><USER2> While I loved reading the Unix haters handbook it gets many things wrong. It gets a lot right too, but the point is it takes some critical thinking (and good knowledge of Unix) to tell which is which.

In that respect I found the chapter on X to be one of the weakest. They trot out the old "X named the client and server the wrong way around" chestnut for example...

<comment><USER9> Well, the naming is one thing; the architecture itself is worse. It's far more likely today for you to want to run applications on a server and be able to connect to and disconnect from them, like VNC or much better RDP, than it is for you to want all your applications running on the server to terminate when you disconnect your machine from the network.

<comment><USER10> There is nothing preventing any application to connect to another X display and push all its windows there, even if the original display session terminates. Emacs can sort of do that, for example, and there are some traces that something like that was intended even for GTK. But no application actually work in such conditions.

</comment></comment></comment></comment></comment><comment><USER11> Just for comparison, Apple "ditched" X too (they never used X). When they went to OS X they used Display Postscript rather than X. <http://en.wikipedia.org/wiki/Display_PostScript>

This has been a mixed bag: it has been good for Apple-only programs, but it significantly slowed and complicated the porting of existing and new X-based programs to Apple.

<comment><USER12> Yeah, and it made starting up any X-based application on OS X a real pain. X apps are second-class citizens. This is one (of many) reasons I could never quite reach a level of comfort on a Mac.

<comment><USER13> It's a calculated move on Apple's part. They don't _want_ you to use X11, so they make it as painful as possible while still keeping it usable so that developers can't write new apps for it (they have to use Cocoa). It's strictly compatibility environment only.

<comment><USER14> > It's a calculated move on Apple's part.

that seems unlikely in light of:

<http://news.ycombinator.com/item?id=1872915>

<comment><USER15> AFAIK (I don't have the source right now), Apple originally considered using X11 for the display server on OS X (in fact, early versions shipped with a version of xf86 that ran natively as it does on other *nixes), but after deciding on all the compositing and prettiness features that they wanted, they would have torn so much out of X11 that it wouldn't be remotely compatible anymore, so they wrote Quartz 2D.

Later, of course, things like Compiz came along, but they've always seemed like weak hacks compared to things like Quartz. Hopefully wayland will change everything.

<comment><USER16> This may be the explanation you are referring to:

[http://news.slashdot.org/comments.pl?sid=75257&cid=67346...](http://news.slashdot.org/comments.pl?sid=75257&cid=6734612)

</comment></comment></comment></comment></comment></comment><comment><USER17> I have no worries about this. Linux distributions always do a couple of releases of "disasters", (e.g. KDE 4) before it starts becoming stable.

Wayland is theoretically superior and with iteration after iteration, I'm sure it's going to come out on top. Just don't be one of the early adopters, but that's a given when it comes to Ubuntu, in my experience (unless you really do like the risks involved with cutting edge software).

The truth is, X is just too old fashioned and it's like trying to spend time maintaining an old car from the 1970s instead of just buying a new car with electronic fuel injection and excellent brakes.

<comment><USER18> Have you ever tried to repair things in a new car with electronic fuel injection and computer-driven brakes? A huge advantage of X11 and all that other "70s technology" is that you can fix things more easily because it's built for being maintainable in the first place.

Having said that, I think that splitting the X server into one lightweight compositing manager and one slightly less heavyweight X-server-on-top-of-the- compositing-manager is less of a disaster than what we got with KDE4. I'll only start worrying when KDE and Gnome catch up and they start having UI programs talk directly to the compositing engine.

<comment><USER19> I'm not sure about the easy to fix concept. By all accounts the X codebase is one nasty beast to wrap your head around. The protocols are well documented, but the actual code - it takes some crazy digging to actually understand it enough to be able to fix things in X-land.

</comment></comment></comment><comment><USER20> I think Mark Shuttleworth is taking bold steps to ensure that Ubuntu is well positioned for the future. He's creating differentiation between Ubuntu and other debian-type distributions and he's potentially mapping out a route to new platforms.

It does seem like he might be a gambling man though.

<comment><USER21> Maybe gambling is too strong a word. He's not in it for the money, but to create a viable competitor to Microsoft. So not taking this "gamble" and sticking to the status quo would be actually be the riskier strategy for him.

If it backfires, it doesn't take him that much further away from achieving his original goal.

</comment></comment><comment><USER22> "Ditch X server" sounds like they're going to completely abandon X11. However, the Wayland FAQ[1] outlines ways of integrating X11 with Wayland, so I suppose users of, say, ddd are not going to suffer much.

1\. [https://groups.google.com/group/wayland-display- server/web/f...](https://groups.google.com/group/wayland-display- server/web/frequently-askeds-questions)

<comment><USER23> If this gains sufficient traction, and if this provides a better-looking way to render stuff on screen, I guess this will lead to applications that talk the superior protocol, not The X protocol. Such applications would not run on X displays.

If I interpret things correctly, the link you give explicitly describes such a scenario:

 _Further down the road we run a user session natively under Wayland with clients written for Wayland. There will still (always) be X applications to run, but we now run these under a root-less X server that is itself a client of the Wayland server. This will inject the X windows into the Wayland session as native looking clients. The session Wayland server can run as a nested Wayland server under the system Wayland server described above, maybe even side by side with X sessions._

So, the X protocol could still be there for quite some time to support 'old' applications, but new applications wouldn't use it anymore.

There are quite a few if's here, though. The first non-Ubuntu application to drop X support could face quite an uphill battle.

<comment><USER24> So essentially it would be like Apple's X11. It will be there for backwards compatibility with older applications, but it will be considered "undesirable".

<comment><USER23> Yes, technocally, it would be like that. Sociologically, however, things would be quite different; Apple's users did not have zillions of X applications that they already used.

</comment></comment></comment></comment><comment><USER25> This is the actual announcement by Shuttleworth:

<http://www.markshuttleworth.com/archives/551>

</comment><comment><USER26> Unless they're planning this for 13.04 or later, it definitely sounds like a bad deal. Wayland _just_ moved to a "real" project space outside of Kristian's personal repo, and it is nowhere near mature enough to support something like Ubuntu. Even if GTK and QT "mostly work", there's still a lot more in the X ecosystem that is hard to replace, namely proprietary drivers and the knowledgebase associated with working with X.

Even Ubuntu has a history of pushing out reliable, mature applications for much less mature ones on the basis of theoretical improvements. Look at Empathy and Pidgin; I still get crashes from Empathy all the time, the skinning and font-size support have problems, etc., whereas Pidgin just works great. Why couldn't they have just affixed the Telepathy backend as a major player in Pidgin, or at least forked and made Empathy a Pidgin derivative? It would have worked out much better.

I'm just hoping that Arch doesn't freak out and fear that Ubuntu is becoming more "bleeding edge" than Arch and remove X for Wayland in the next three weeks.

<comment><USER27> You're right, which is why I'll continue to stick with Debian stable or Ubuntu LTS while this all blows over. It _will_ blow over, however, and I'm happy to have a major player out there forcing it to happen.

<comment><USER28> We use Ubuntu LTS at work because of the stability.

On my laptop at home though (which I only really use for watching movies, and for hacking on side-projects that one day may become HN stories) I'm always on cutting edge Ubuntu. I love the thrill of the beta's.

So I'll do the dual strategy of letting it blow over for work purposes, and playing around in the mud for personal purposes.

<comment><USER29> Where do you work? If you're allowed to say, I'm always interested in hearing about Linux deployments, much less Ubuntu and/or professional "desktop" Linux usage.

There should be an Ubuntu testimonials website... Weekend project?

</comment></comment></comment></comment><comment><USER4> I've been unhappy with X since I learned enough (in the 90s) to have an opinion -- but not for the reason Shuttleworth gives:

"We don't believe X is setup to deliver the user experience we want, with super-smooth graphics and effects. I understand that it's _possible_ to get amazing results with X, but it's extremely hard, and isn't going to get easier. Some of the core goals of X make it harder to achieve these user experiences on X than on native GL"

My unhappiness stems from my perception that the ways I have tried to customize X and things closely coupled to X (e.g., my window manager) have proved much more difficult (and in particular have forced me to contend with much more _complexity_ ) than they could have.

<comment><USER30> In that case, I think you'll hate future Ubuntu because (like its mentor OS X) it won't be very configurable at all. Easy things will be easy and everything else will be impossible.

<comment><USER31> Except the code will be open so if Canonical doesn't make it configurable, other people can, and if they do it right without introducing unneeded complexity, their patches might just make it upstream (Ubuntu).

This is assuming of course that people who like to fiddle with their display will stay with Ubuntu.

Not many people like to do that anyway. But if there are people who want that kind of control, it's not impossible.

</comment></comment></comment><comment><USER32> For me X has always symbolized old, hard to configure, clunky, and ugly. It wasn't meant to be part of Unix but it mixed in. I am glad to see it go at least from our desktops.

<comment><USER33> I found X quite easy to configure. Just set a few values in Xorg.conf and nowdays it even works without this sometimes.

X11 as a protocol is old and mature like TCP or SMTP. Just like SMTP it has some problems, but Wayland throws away some core features to fix it. Hopefully, Wayland is flexible enough to get network transparency like X11 at some point.

<comment><USER34> >Just set a few values in Xorg.conf and nowdays it even works without this sometimes.

you make it sound like it was an advantage.

</comment></comment></comment><comment><USER35> That is why I love Ubuntu: They like to take the bold decisions. Mark Shuttleworth is a small Version of Steve Jobs :)

<comment><USER36> Let's say "open version".

<comment><USER35> Nice own.

I met him personally. Totally incredable guy, who knows how to work with people.

</comment></comment></comment><comment><USER37> With the Software Center, Unity interface, and now a new display server, looks like Ubuntu is shooting for the mainstream OS business. Interesting to see an Linux distro trying so hard to set itself apart.

<comment><USER38> Ubuntu has always been shooting for the mainstream, it is the central goal of the project. Just look at bug 1.

<comment><USER39> Link, for those that haven't seen it: <https://bugs.launchpad.net/ubuntu/+bug/1>

<comment><USER37> Thanks, I'd never seen that. Very humorous :)

They should make another one:

Bug #2: Apple has a majority mind-share

</comment></comment></comment></comment><comment><USER40> That is factinating. What makes Mac OSX on desktop is based on Quartz and Linux needed something like that to be more solid and modern on desktop environments. So let be it.

</comment><comment><USER41> I'm not familiar with Wayland, so I read the architecture article:

"Wayland is a complete window system in itself, but even so, if we're migrating away from X, it makes sense to have a good backwards compatibility story."

<http://wayland.freedesktop.org/architecture.html>

<comment><USER42> I'd rather see Wayland implement the X protocol themselves rather than proxy to Xorg. Running X and Wayland simultaneously could have performance implications on older computers or netbooks, two of Ubuntu's biggest targets.

<comment><USER43> Why would running two systems _help_ performance. A better designed system that doesn't have to have network transparency code paths should be able to run faster than X and Linux applications are written in Qt or GTK anyway. Port those two and call it done.

<comment><USER42> Sorry, I wasn't clear: running X alongside Wayland would _hurt_ performance (at least in RAM-critical environments).

</comment></comment></comment></comment><comment><USER44> Sigh,my first job was to work on a proprietary X-server extension. Despite Jim Gettys' and Keith Packard's continued interest, and some interest from vendors as well, X seems to be losing hacker mind share. There were some fast proprietary X servers made by a few companies like Xig graphics that delivered truly incredible performance. But the really exciting things that could be done with X never saw daylight outside a small community of display manufacturers for defense, air-traffic control, etc

<comment><USER45> _But the really exciting things that could be done with X never saw daylight outside a small community of display manufacturers for defense, air-traffic control, etc_

What sort of exciting things?

<comment><USER33> Here is one vendor for example: <http://www.x-software.com>

</comment></comment></comment><comment><USER46> I guess this means someone is going to have to port xmonad...

<comment><USER27> You mean all 2000 lines or so of the core that almost everything else is built on?

<comment><USER10> Xmonad core doesn't provide any kind of X11 encapsulation and convenience interface, every single xmonad extension, any piece of configuration, can (and does) directly use X11, both API and behavior. After all, it's an X11 window manager. You have to rewrite everything.

</comment></comment></comment><comment><USER47> So the sound system is actually working again so lets break the display system next.

</comment><comment><USER48> How will this affect LTSP and the future of thin clients with regards to Ubuntu?

<comment><USER30> They'll probably use Wayland with SPICE.

<comment><USER48> Do you have a link? All I can find is the circuit simulator results.

<comment><USER30> I can see how that would be confusing. I'm talking about Red Hat SPICE, a replacement for VNC. <http://www.spicespace.org/>

</comment></comment></comment></comment><comment><USER49> Is there a distribution I can download that is already using Wayland? I'd like to take a look. For my part, the desktop manager is the one thing holding me back from Linux. (OK, that and Visual Studio)

<comment><USER50> I don't believe so. Wayland is in the really early stage of development right now, which makes me read this announcement as "Ubuntu is going to start committing developers to Wayland.

You can build it with these instructions: [https://groups.google.com/group/wayland-display- server/web/b...](https://groups.google.com/group/wayland-display- server/web/building-and-running-wayland) .

</comment></comment><comment><USER51> I really wonder how we are going to manage Wayland for video: \- does it support Xv or the like? Or are we going to use the broken OpenGL drivers (no working overlay for most of them...) \- does it support VAAPI or VDPAU, that seems very linked to x11 too?

We'll see, but I have a bad feeling.

<comment><USER52> My fear is lack of remote desktop. I would never use a Linux desktop without NX support.

<comment><USER43> Remote desktop is easy, there's no reason to make it the _focus_ of the GUI subsystem.

</comment></comment></comment><comment><USER53> It looks as if Shuttleworth is concentrating squarely on desktop bling, and this may be the right strategy if he wants Ubuntu to gain mass popularity and really eat into the Windows user base.

<comment><USER54> How? From what I gather, Wayland is better for the mobile/touch space, servers don't use GUIs and there's not much that uses _only_ X...

<comment><USER55> So, did you post that comment from a mobile phone or a server?

Oh yeah, there's this thing called a "workstation". I hear there are a few in existence...

<comment><USER54> That's odd, the comment I replied to was saying that x is useful for things other than workstations, and I was saying that it wasn't more useful than weyland on mobiles and servers.

I have no idea how this comment mixup happened...

</comment></comment></comment></comment><comment><USER56> >>After shocking everyone with the announcement that Ubuntu 11.04 will have Unity on the desktop instead of GNOME Shell, >> Unity is going to use Wayland display server instead of X. This will not be implemented in Ubuntu 11.04 however. >>X will remain until 11.04 however. I assume the last two relese dates/version are incorrect, because unity is being added for 11.04, and the article seemed to say the change to Wayland would happen after the change to unity.

</comment><comment><USER57> does this mean my hometown will fade into even greater obscurity (as quantified by google search results)?

:(

<http://www.wayland.k12.ma.us/>

</comment><comment><USER58> I would just like to point out that Arch Linux with GNOME works just fine. I'm sure there are tons of other distros out there, too, that work just fine. If you don't like what's coming in Ubuntu, just switch. you probably won't notice much difference between ubuntu 10.10 and [insert distro] with GNOME.

</comment><comment><USER59> <http://www.microxwin.com/architecture.html>

</comment><comment><USER60> Named after Wayland the smith? Who killed the kids of his employer and fashioned goblets out of their skulls? Erm…

<comment><USER61> Something about the performance review process, possibly: "[...] he was captured in his sleep by King Niðhad in Nerike who ordered him hamstrung and imprisoned [...]" (according to <http://en.wikipedia.org/wiki/Wayland_Smith>).

</comment></comment><comment><USER62> I have a worry of this turning out very badly.

<comment><USER63> Wayland uses OpenGL ES for rendering, preparing Ubuntu for handsets.

Ubuntu must be eying a future alongside Android, which is an excellent bet.

There have been many attempts at X replacement, all well justified, and if Wayland is the best of them, it would still be better than the baroque X protocol.

Like almost every C++ programmer working in the early 00s, I too hacked on Berlin :-|

<comment><USER64> It looks like Wayland runs on top of X, not replacing X.

Which makes sense. OpenGL is just an API, not a hardware wire-level protocol. Unless Wayland implements its own graphics drivers (which I find highly unlikely), it'll just make use of the same drivers that X uses for implementing OpenGL. And that still leaves things like input handling.

From the Wikipedia entry it looks like Wayland uses X for input and rendering, but other apps can connect to Wayland to have things rendered. Kind of like how X itself works.

<comment><USER63> This is straight from this FAQ:

 _Wayland is a nano display server, relying on drm modesetting, gem batchbuffer submission and hw initialization generally in the kernel. Wayland puts the compositing manager and display server in the same process. Window management is largely pushed to the clients, they draw their own decorations and move and resize themselves, typically implemented in a toolkit library._

That last part about integrating the compositing manager and display server into the same process .. that's your main point of departure.

<comment><USER65> > _Window management is largely pushed to the clients, they draw their own > decorations and move and resize themselves, typically implemented in a > toolkit library._

Well, _there's_ a bad idea. I suppose I could see an X replacement which incorporated window management into the server, instead of into a special client; but letting all the clients draw their own borders is a recipe for chaos.

<comment><USER66> Isn't an X window manager just another X client too at the moment? Couldn't that system still work on Wayland?

I might be completely wrong about this though, so someone more knowledgeable please explain. :-)

<comment><USER65> Yes, but it draws the decorations for other clients. The GP said that Wayland clients draw their own decorations.

</comment></comment></comment></comment></comment></comment></comment><comment><USER67> Just one question, I hear that Wayland is hugely dependent on the latest Graphics cards. So that would mean that ubuntu will no longer be usable by people like me who have moderately old hardware (mine is p4 with on board intel graphics, which are already having problems even with X).

I liked how ubuntu was stable on my computer (till around 8.10 I think and 9.04 was not bad). I will miss the compatibility

<comment><USER68> Same here. My three year old laptop has gotten slower with every consecutive release of Ubuntu. I'm afraid to upgrade to the latest let alone to this new GL-based display server.

Guess I'll just have to switch to a 'lighter' distro with less fireworks.

I'm sure it will pay off for Ubuntu in the end... I just hope it doesn't _actually_ start a trend for other distros to follow suit.

<comment><USER67> I switched to puppy linux and am loving the no-os-on-my-HD style (of course I can do a frugal install, but I am just lazy)

<comment><USER69> I too love Puppy Linux! It works great on my netbook.. faster than any of even the netbook-optimized distributions. I'm still running 4.3. The hardware support was even perfect with the LuPu on my main laptop, but I'm holding off as a lot of software packages I use are a pain... (openSUSE on this machine).

</comment></comment></comment></comment><comment><USER31> This is great news for the Free Desktop.

Ubuntu is really setting itself apart as its own OS rather than just another Linux distro.

It might take about two years for this whole thing to sort of stabilize (Wayland + Unity), but when it does, Ubuntu will be kick-ass like it's never been before.

I love it when people step outside the box.

<comment><USER70> Wayland was started by someone at Red Hat.

<comment><USER31> It would be great to see Fedora using it too, but I don't know if that will happen.

RHEL is not likely to use it, IMO, because it's not network-transparent. And f RHEL won't use it, Fedora probably won't use it.

That said, what's setting Ubuntu apart is the fact they're moving away from the traditional stack (X -> GNOME). This is significant enough IMO. Other distros might follow, and in that case, there will be two kinda of distros: those that use X, and those that use Wayland.

</comment></comment></comment><comment><USER71> God! I can hardly stand to read the digitizor website from all the crap they have jacked up and hyperlinked for advertising.

I'm on Chrome. Is there a feature that lets you get control of that mess?

Edit: I apologize for the off-topic posting, but the linked website is a good example of what readers of Hacker News seem to have tricks for handling.

<comment><USER72> I recommend Readability: <http://lab.arc90.com/experiments/readability/></comment></comment>


O2 outage: 31m mobile customers unable to get online - <AUTHOR> https://www.theguardian.com/business/2018/dec/06/o2-customers-unable-to-get-online <comment><USER2> I'm seeing references to something Ericsson contributes to their network. It would very interesting to see what went wrong and in particular, what caused the change or deployment to be "un-revertable". My money is on a cascade of issues, some of which were latent and only triggered today.

This is, by far, the worst cell outage I've seen in the UK. There is _zero_ connectivity via O2 in e.g. London, data or otherwise.

[1] [https://www.standard.co.uk/news/world/o2-network-down- ericss...](https://www.standard.co.uk/news/world/o2-network-down-ericsson- working-to-resolve-issue-as-millions-left-without-smartphone-data- and-a4010496.html)

<comment><USER3> Due to an expired TLS cert: [https://www.ericsson.com/en/press- releases/2018/12/update-on...](https://www.ericsson.com/en/press- releases/2018/12/update-on-software-issue-impacting-certain- customers?707191720)

<comment><USER4> I liked the "limited number of customers" yes that's true, it affect 2-3 companies, it's just that a single one of those companies had 32M customers.

Ouch.

</comment></comment></comment><comment><USER5> [https://status.o2.co.uk/](https://status.o2.co.uk/)

> We believe other mobile operators around the world are also affected.

Anyone else affected?

<comment><USER6> Ouch. Their status page seems to be overloaded, assuming that is the correct URL.

<comment><USER7> Why do so many status pages seem to crash/incorrect report when the service goes down?

Once you get on the page:

        We're really sorry but this system is currently
        experiencing some technical problems.
        Our engineers have been notified, so please check back
        later.

<comment><USER6> Best practice is to host the status page on completely separate infrastructure. This adds a lot of operational overhead often seen as disproportionately low benefit relative to the ongoing effort to maintain, until rare events like this actually happen.

<comment><USER8> > Best practice is to host the status page on completely separate > infrastructure.

Perhaps they have - and it's overloaded.

<comment><USER9> Probably offset by the amount of people who can't access it due to their internet not working haha

</comment></comment></comment></comment></comment></comment><comment><USER10> I thought initially it was a local thing like when Vodafone disappeared here because a falcon made a nest on the transmitter

<comment><USER11> University of Southampton?

<comment><USER10> Yup. 2006-2012, still live in Highfield.

</comment></comment></comment><comment><USER12> I haven't had mobile internet since yesterday evening on O2, and I travel to London for work.

Luckily, there's WiFi at home, at work, and on the trains, so it's not really affecting me too much, but I'd really like to see the postmortem of such a huge outage.</comment>


Cheese changed the course of Western civilization - <AUTHOR> http://qz.com/371839/cheese-changed-the-course-of-western-civilization/ <comment><USER2> Lots of "likelihood," "probablilities" and "maybes" floating around in that article, and not a lot of certainty. I find the article scant on proof, and speclative at best.

In particular, there's this:

      What this meant, says Kindstedt, is that "children
      and newborns would be exposed to milk frequently,
      which ultimately through random mutations selected
      for children who could tolerate lactose later into
      adulthood."

Explain to me how childhood exposure to milk (the period when humans tolerate lactose) selects against lactose intolerance in adulthood?

For example, so let's say you have a mixed population of families, where initially there's a prevaling majority of hereditary lactose intolerance, but with a minority of the new mutants present, who decades later will comfortably continue to consume dairy.

Mutations are typically recessive so you might have households where one parent is lactose tolerant, and none of the children are, due to the lottery of gene expression in sexual reproduction. Maybe some intolerant children are carriers and some aren't, but it's still a roll of the dice.

How does consuming dairy products as a child select for this trait which will not be expressed until years later? What obvious mechanism of selection am I failing to understand here?

Are medieval societies filled with stories of the lactose intolerant humilating themselves with dairrhea on prom night and reducing themselves to social pariahs, doomed to die childless and alone? I mean really.

<comment><USER3> _Explain to me how childhood exposure to milk (the period when humans tolerate lactose) selects against lactose intolerance in adulthood?_

People who are lactose tolerant will be able to continue drinking milk well after those in their birth cohort who are lactose intolerant will have to stop. They will thus be likely to be substantially better fed with subsequent better health.

People who are lactose tolerant will be able to survive famines where lactose intolerant people will starve.

<comment><USER2> Sure, I get the general concept that we assume must exist because it sounds like it's probably reasonable, but there's no concrete smoking gun evidence of that particular mechanism at work anywhere in the world. Not even in folklore.

Where in history is the famine in which plenty of farm animals were thriving and impregnating and lactating amongst one another, but all the humans had to live on was milk?

<comment><USER4> That's like asking when in history was there some catastrophic era where particles were flying into some organisms' eyes and the only individuals that survived were those who (through genetic mutations) had small hairs around their eyes that managed to catch some of the particles.

<comment><USER5> I guess it is hard for most to come to terms with just how long time scales evolution works on.

</comment></comment></comment></comment></comment><comment><USER6> Is lactose tolerance purely genetic, or is there an element of exposure that causes it?

The reason I ask is funnily enough that I've made the same observation as xiaq: I'm Chinese. There's no suggestion of any other ancestry than far eastern, yet I can still drink milk.

My mom is lactose intolerant but my dad seems to be able to process it. Though he doesn't expose himself much; it's just not that big a part of Chinese diets. I grew up in the West, where kids get milk all the time.

I guess another way to say it is if a kid drinks milk and keeps doing it, will he suddenly get the squits from it one day?

<comment><USER7> I'm from India and this is anecdotal, but I've never met a person who's intolerant to dairy. I heard about lactose intolerance only after moving to the US. In most part of India, dairy is an integral part of diet.

<comment><USER8> I'm from India too, live there presently, and what follows is anecdotal as well.

I first heard of lactose intolerance when my son became lactose intolerant at age 1.5. I also discovered then that the pharmacies were stocked with lactose- free milk substitutes (soy milk was comparatively harder to get hold of). They'd apparently been stocking this stuff for years.

More surprising, the first time I heard of a nut allergy was when a friend (around the same age as yours truly, born sometime in the decade of The Emergency) said he had one.

</comment></comment></comment><comment><USER9> Interesting. As a Chinese I have always wondered why lactose intolerance is so common in China but not in the west. Turns out I asked the wrong question :)

<comment><USER10> That is something that I also, as an American of mostly English/French/German descent, have wondered about. It was, quite frankly, a revelation to me when I first encountered people who were lactose intolerant in college.

It seems even more unlikely, given that China has had so much interaction with the nomad peoples from Mongolia, Manchuria, and central Asia, who lived mostly off of their herds and milk-products.

<comment><USER9> > It seems even more unlikely, given that China has had so much interaction > with the nomad peoples from Mongolia, Manchuria, and central Asia, who lived > mostly off of their herds and milk-products.

Dairy products never really made it into Chinese diet.

The next question to ask is why diary products didn't make it into Chinese diet despite the interactions with the nomad peoples :)

<comment><USER10> That is a very interesting question. The most obvious answer is that most of south-central China is just more suited to rice production than cattle/sheep/goats.

I still find it interesting, since many of the more successful Chinese ruling dynasties have central/north asian roots (Tang, Yuan, Qing). Considering that Genghis Khan is probably an ancestor to most of the eastern half of eurasia, once might expect those lactose-tolerant genes to be more prevalent.

<comment><USER9> > Considering that Genghis Khan is probably an ancestor to most of the eastern > half of eurasia,

This is simply wrong.

Maybe some 16 million. But that has also been more of a hypothesis. See [http://clubtroppo.com.au/2012/10/15/why-genghis-khan-wont- ha...](http://clubtroppo.com.au/2012/10/15/why-genghis-khan-wont-have- had-16-million-descendants/) for one of the refutations for instance.

<comment><USER11> Just to make sure the HN record is balanced, I point out for future readers that Wikipedia currently records that some research suggests he is a critical ancestor to approximately 0.5% of the world's population. ([http://en.wikipedia.org/wiki/Descent_from_Genghis_Khan#DNA_e...](http://en.wikipedia.org/wiki/Descent_from_Genghis_Khan#DNA_evidence))

I do this not to participate in the argument but to point uninvolved readers to objective sources.

</comment></comment></comment></comment></comment></comment><comment><USER12> This rapid evolutionary change is one reason why much of the reasoning for "Paleo" diets is flawed. Humans have changed since Paleo times.

<comment><USER13> A better way to approach the Paleo hypothesis is as just that -- a hypothesis. I think all reasonable people agree that humans have changed somewhat since paleolithic times (and grains were possibly part of our early diet) but it still seems quite reasonable to posit that 1) our diet has changed greatly since the longest period of human evolution and 2) it may well be more healthful to return to a diet closer to the paleolithic diet. That's a reasonable hypothesis that can be tested.

Of course that's not the way the popular press and fitness industry approach the Paleo diet, but one shouldn't throw out the baby with the bathwater.

<comment><USER14> While that is true, it's also good to take into consideration the fact that the rationale/hypothesis behind a diet plan doesn't have to be right in order for the diet itself to be useful.

Personally, the paleo hypothesis has always sounded rather dubious to me, but there's more than enough evidence that the ketogenic effects of paleo-like diets are useful at least: [http://ketopedia.com/principia- ketogenica/](http://ketopedia.com/principia-ketogenica/)

<comment><USER15> Yeah, if your "paleo" diet happens to kick you into ketosis, then you've stumbled across a beneficial thing. If you're eating lots of fruit and starchy vegetables, it's probably not doing much for you.

</comment></comment></comment></comment><comment><USER16> I can't see how the article justifies its title. Did the (comparatively) lower level of adult lactose intolerance in the West have social, cultural, economic or other effects significant enough to be described as a change in the course of civilisation? If so, what were those changes?

EDIT: s/tolerance/intolerance/

<comment><USER17> See [http://www.nature.com/news/archaeology-the-milk- revolution-1...](http://www.nature.com/news/archaeology-the-milk- revolution-1.13471). Lacose tolerance apparently allowed middle-eastern nomads to wipe out and replace the native european hunter-gatherers, and set up an agricultural society in their place.

"In a 2004 study, researchers estimated that people with the mutation would have produced up to 19% more fertile offspring than those who lacked it. The researchers called that degree of selection "among the strongest yet seen for any gene in the genome"."

</comment></comment><comment><USER18> I remember reading about American Indians killing nursing buffalo calves and cutting open the stomachs to feed the curdled milk to their children as a special treat (almost like ice cream). Surely hunters of mammals throughout the ages were familiar with this delicacy. Why isn't this considered the 'discovery' of cheese?

<comment><USER11> Lack of direct and corroborating evidence. Lack of specificity.

IE, a very common historical bias: if it's not written down, it didn't happen.

(FWIW this is a very difficult problem that I do not claim to be an expert in, but I think that's the answer to your question, unless I missed it...)

</comment></comment><comment><USER19> One thing that's missing from this is a more nuanced discussion of lactose intolerance. It's not a simple binary thing like celiac disease or an allergy. It's possible to consume around a cup (250ml) of milk per meal for lactose intolerant people, but doing so without experiencing typical lactose intolerance symptoms requires regular exposure and building up to those levels (possibly due to micro-biome effects). What that means is that a group of people that started introducing animal milk consumption with infants might also begin consuming milk through adulthood in small quantities. That would provide the sort of situation that would make it easy for adult lactose tolerance to have a major evolutionary advantage.

If teens and adults are drinking small amounts of milk regularly as well as some amount of cheese then those who are lactose intolerant (the common case) will only be able to get calories from the fat and proteins. But anyone who has acquired lactose tolerance will be able to consume an unlimited amount of dairy products and will also gain around 40% more calories from the lactose. That's an enormous dietary boost especially for people who are struggling at the margins of livelihood and malnutrition. More calories means less stunted growth, which means stronger bodies, and a greater ability to survive through harsh winters or lean times.

</comment><comment><USER20> And cheese is a marvelous invention.

</comment><comment><USER21> Pajata [1] is unweaned veal intestines, and contain a cheese-like substance. It's one possible source of cheese invention.

[1] [https://en.wikipedia.org/wiki/Rigatoni_con_la_Pajata](https://en.wikipedia.org/wiki/Rigatoni_con_la_Pajata)

</comment><comment><USER22> I'm a blue-eyed, blond, cheese eating adult and I like it.

That reminds me of something I read about blond, fair people and milk I didn't see it mentioned in the article about vitamin D which is added to milk these days but I don't know if it has any naturally. People with fair skin, blond hair in northern climates helps in the dim sunlight to get enough vitamin D.

I also recall reading that people living in a northern climate who have access to large amounts of fish allowed generations of that group of people to retain a darker skin colour since the vitamin D from fish meant people didn't need fair skin to get as much vitamin D from the sun.

</comment><comment><USER23> Cheese is an awesome food. It's high-protein and high-fat and is perfect if you're a farmer or in some other way burning a lot of calories.

Where I live (switzerland) we eat a ton of cheese and it's seasonal, with Spring/Summer cheese being when the cows are in the mountains eating clover, herbs, and grass, versus in the Fall and Winter when the cows are in the valleys eating hay. There are something like 5000 varieties of cheese in Switzerland if you include every village and farmer and their sub-varieties in the list, but for mainstream cheeses there's something like 500 varieties. You can get unusual different cheeses every weekend if you're willing to take a trip to some random village in a tucked-away valley. (sausage too! especially if you're willing to eat sausages made with horse, deer, dog, etc.)

I'm not Swiss but my wife and I still eat a ton of cheese. It's so good here and such a part of the culture that you can't help but take to it. We burn off the calories like the Swiss do with hiking and skiing every weekend.

<comment><USER24> That's one of the things Europeans (anecdotally Germans and French) who work or study in North America complain about. Cheese quality and selection (and price!) here is crap. In fact, food quality here seems to be a notch lower.

<comment><USER25> 1\. US milk does not taste good. Actually, it does not taste at all. I always thought this is just in my mind until a Swiss guy said the same.

2\. All milk products are do not taste so well and often have poor quality. Cheese, Yogurt etc.

3\. You can get excellent cheese and yogurt but this comes with a hefty price tag. E.g. Whole foods etc.

I have the impression that supermarket foods in the US have a much lower quality compared to Europe. You can get the same or better quality, if you can effort to pay for it.

<comment><USER26> > 1\. US milk does not taste good. Actually, it does not taste at all. I > always thought this is just in my mind until a Swiss guy said the same.

I don't believe this. Milk sold at a super market comes from cows and cows need to be semi local to bring milk to any store. I'd love to know how milk from other places is somehow superior. My suspicion is that it isn't and just confirmation bias (that or someone is comparing different types of milk (e.g. 2% versus homogenized) which is a poor comparison).

<comment><USER27> On the face of it, it is easy to say "it's just milk, it all tastes the same", but there are significant differences in how milk is produced in the US compared to the EU.

Most US-made milk is produced using genetically engineered growth hormones (rBGH) which are banned in the EU as they are believed to increase the risk of cancer. It would not be legal to sell US milk in the EU (or Canada) for health reasons. That, and other differences in production methods, could go some way towards explaining the perceived superiority.

<comment><USER28> I'm skeptical. Less than 20% of milk produced in the US uses rBGH, and most health organizations, including in Canada and Europe, agree that milk from rBGH/rBST cows is safe for humans. The reason it's banned in the EU and Canada is because it's a health risk for cows, not people. And the reason you're not likely to get US fresh milk in those countries has more to do with transportation costs than production standards.

If I had to guess, I'd say the most likely reasons for flavor differences are 1) different pasteurization techniques 2) fat content 3) consumption temperature (you can't taste sugars as well when they're cold).

</comment></comment></comment></comment></comment></comment><comment><USER29> I don't like the term "lactose intolerant". It implies that having difficulty digesting lactose is differentiating you from some loosely defined norm.

In fact, the opposite is true[1]. As the article states, being able to digest lactose is a recent genetic adaption. What it doesn't state is that still, even in the West, a lot of people experience lower tolerance to lactose as a natural process of aging.

I would therefore say that the correct way of labeling it is that adults in general have different levels of _lactose tolerance_.

Anecdotally, I can add that my level of lactose tolerance was very low for two years after drinking half a liter of milk that was two weeks old (don't ask). I assume that it severely messed up my small intestine. After I cut carbs and gluten from my diet for a month and a half, I could suddenly digest lactose again. As with lactose intolerance, I believe gluten intolerance is a misnomer, as all people are sensitive to excessive amounts of gluten, though here the case is not nearly as clear cut. My intestine finally being able to repair itself might instead be due to the fact that I didn't eat many carbs at all during this period, and so avoided the gastro-intestinal allergens known as FODMAPs[2]. Wheat contains the FODMAP fructan, which might be the reason many people say they feel better on a gluten free diet[3].

[1]: [http://www.collective-evolution.com/2013/04/03/over-75-of- ea...](http://www.collective-evolution.com/2013/04/03/over-75-of-earths- population-is-lactose-intolerant-for-a-reason-dairy-is-harmful/)

[2]: [http://en.wikipedia.org/wiki/FODMAP](http://en.wikipedia.org/wiki/FODMAP)

[3]: [http://www.npr.org/blogs/thesalt/2014/05/22/314287321/sensit...](http://www.npr.org/blogs/thesalt/2014/05/22/314287321/sensitive- to-gluten-a-carb-in-wheat-may-be-the-real-culprit)

<comment><USER30> > I don't like the term "lactose intolerant". It implies that having > difficulty digesting lactose is differentiating you from some loosely > defined norm

It's a scientific term created purely for succinctness and description, it carries no connotations.

<comment><USER29> Fair enough, I'm surely being overly sensitive here. It's just that, here in Sweden at least, being 'lactose intolerant' suggests there is something wrong with you, when the truth in fact is just that your lactase persistence[1] is more akin to that of the rest of the world.

It carries connotations here. I'd say it does in the US as well, given the jokes about it I've seen on shows like the Big Bang Theory.

[1]: [http://en.wikipedia.org/wiki/Lactase_persistence](http://en.wikipedia.org/wiki/Lactase_persistence)</comment></comment></comment>


Norway and the A-ha moment that made electric cars the answer - <AUTHOR> https://www.theguardian.com/environment/2020/apr/19/norway-and-the-a-ha-moment-that-made-electric-cars-the-answer <comment><USER2> Speaking as a Californian, dams on the Sacramento and San Joaquin rivers have reduced our salmon migrations to only 5% of their pre-dam-building norms.

My question for Norwegians: Have you seen similar environmental impacts on their native fisheries and ecosystems due to your large-scale hydroelectric constructions?</comment>


A high-frequency trading model using Interactive Brokers API in Python - <AUTHOR> https://github.com/jamesmawm/High-Frequency-Trading-Model-with-IB <comment><USER2> Great stuff.

I think the author would have been better suited without mentioning HFT, maybe algo trading model?, as its a lighting rod for controversy.

If anything I thin this is useful to illustrate just how hard it is to write a full blown trading system.

So maybe we could look at what you could add to this to make it something you could use in production(Note, please don't use this in production).

1) Risk system, before you write any algos, before you learn how to ingest market data, before you write anything else, you write the risk system.

in a true trading system because you always have orders flying between you and the exchange you never really know what your positions is. The Risk system allows you to deal with this uncertainty by giving each algo rules as to how many shares its allowed to be offside.

If you only ever take away one thing let it be this:

 _Trading Rule #1_ Sooner or later your algo/code will make a mistake, it's your risk system that determines if you have a job tomorrow or not.

2) The system has no rate limiter, what do you do when the quotes come in too fast for you to deal with?

3) Locking the world, the system retrieves a quote and locks up the entire system while the quote is acted on. Essentially the very design of the system means you can only ever run one strategy for one set of tickers at a time.

If you wanted to tackle this start by looking at this data structure/library:

[https://lmax-exchange.github.io/disruptor/](https://lmax- exchange.github.io/disruptor/)

4) Back testing, no HFT trade idea's go into production without backtracking, Every HFT firm is different but I think they'd all adhere to this rule.

5) Closing positions, every algo gets offside at some point. How do you notify a trader to close out a position? How does the trader close the position and notify the algo?

6) Multiple algos over multiple symbols.

7) Real time PnL. Your PnL is everything, it means you get paid, it means you can do this again tomorrow. it is the single most important piece of information(Risk metrics are a close second) that you can track. This is probably my only quibble with the demo.

<comment><USER3> It's good that you lead with risk as #1. You know this, but many don't follow so closely. So here's a Wiki entry of a recent spectacular failure:

[https://en.wikipedia.org/wiki/Knight_Capital_Group](https://en.wikipedia.org/wiki/Knight_Capital_Group)

The first sentence:

       The Knight Capital Group was an American
       global financial services firm

The key word is: _was_

<comment><USER4> As an infrastructure engineer doing DevOps currently, I use Knight Capital Group frequently as an example as why its important to always be in control of your application's environment.

</comment></comment></comment><comment><USER5> Wow, didn't expect my repo to make it this far. It was my first time using Reddit, with the aim of introducing this for research more than actual trading.

Just some thoughts after reading comments:

\- I've got a number of Python examples for trading futures, but a third-party app is required as 'the gateway'. Browse on GitHub if you wish: [https://github.com/hftstrat/The-Gateway-code- samples](https://github.com/hftstrat/The-Gateway-code-samples)

\- Backtesting is a topic not to be taken lightly. There are far too many issues to address than just historical simulation. I've written a book covering this topic.

\- In my book I've also covered the use of Oanda API with a simple trend- following strategy. Available on Amazon: [http://www.amazon.com/Mastering- Python-Finance-James-Weiming...](http://www.amazon.com/Mastering-Python- Finance-James-Weiming/dp/1784394513)

\- Table of contents and source codes used in my book are also available on GitHub: [https://github.com/jamesmawm/Mastering-Python-for-Finance- so...](https://github.com/jamesmawm/Mastering-Python-for-Finance-source-codes)

Alright, I'm done with shameless promotion of my links ;p

</comment><comment><USER6> Pretty cool stuff. Thanks for posting holoiii. Last year, I took some time off to build an IB API client in Java/Scala for options trading.

The biggest challenge I faced personally was dealing with concurrency of maintaining my positions, orders and quote ticks which led to orders as IB API is based on a asynchronous tick model. I ended up re-writing my code with lots of threads/locks to more code with immutability. Curious how you guys architected your IB client?

Also I hosted my client on the cloud on AWS. I subscribed to maybe a few symbols but always wondered in a more production environment whether bandwidth will kill you (subscribed to 100+ symbols and to a real tick-by-tick quote stream not quote snapshots).

Finally I'd love to hear more about folks out there who might be doing more options or futures trading off of IB API. The equity world is definitely very interesting but I found out early on that I enjoy trading delta-neutral and am not good at predicting directions.

Did you guys find that you have to scale up and write your own execution algo's when your size got too big for the market you are trading? Nowadays I use my IB client mostly as a GUI to view my positions and execute complex option spreads; so it's more of a "gray-box" for me. Do you guys trade pure algorithmically or half-and-half?

<comment><USER7> Is it possible to get IB API access without opening a 10k account? Do they have a demo endpoint you can use? If you didn't already have an existing system built, having 10k tied up while you build one seems undesirable.

<comment><USER8> You would have to put in at least $25k to do anything serious, since the pattern day trading rule would definitely be triggered.

<comment><USER7> PDT rule only applies to stocks and options afaik, and only if you have a margin account. It doesn't affect futures/equities/forex.

</comment></comment></comment></comment><comment><USER9> It is so, so infrequently that I get to say this about something HFT-related on the internet: fucking awesome, dude. There is actual information here that is of use and doesn't mislead. It's amazing.

edit: s/dude/guy-who-made-this

</comment><comment><USER10> The one key part this seems to be missing is a backtesting framework. Did you look at [https://www.quantopian.com/](https://www.quantopian.com/)?

</comment><comment><USER11> Tangentially related question. As a relative newbie to (automated/ algo) trading systems, it seems most of posts here focus on systems that deal with large volumes of trades in very short timeframes.

I'd be interested in writing an automated system that would focus on trading (investing) over longer timespans (shares, options), with (average) returns. Are there any resources you would recommend to get into this? Or are automated systems really only employed in HFT?

<comment><USER12> What is a "longer timespan"? Minutes instead of seconds? Days instead of minutes? Month instead of days?

Automated trading systems only really make sense when you are trading a pace that is too fast for humans to reliably execute. If you have time for human intervention it is much better to produce a system that simply spits out reports and recommendations that you can act on (or not), rather than having a system trade automatically. If nothing else it can save you from losing all your money due to an unfortunate off-by-one error or something.

<comment><USER11> In terms of timespan, I was thinking days/ months. So it would take the aspect of human intervention and analysis/ research out of it (perhaps the system could still prompt the user for input, the user would just not have to monitor markets and positions actively). I don't know if this makes sense.

<comment><USER12> The problem with most HFT/algorithmic approaches is that it relies on large volumes to make profits. If you're averaging a very tiny profit pr. trade (with a large variance) you're going to have to make a lot of big trades and be able to sustain large temporary losses to make money.

That's not to say that computer assisted trading isn't a good idea for smaller time investors, but the focus is on assistance, not automation. You can write models that produce a shortlist of stocks and options that match certain criteria and you can write risk and portfolio models that try to model what might happen if you add or remove a certain asset from your portfolio, then you can combine these and run some sort of optimization algorithms over the whole thing. But at the end of day, the choice should be made by a smart informed human and should be heavily biased by your reading of the news and other external sources.

All that being said, if you decide to go into this, approach it first as a hobby rather than a job, with the assumption and acceptance that most hobbies cost, especially in the short run. If you just want to do something smart with your money, buy some index funds.

</comment></comment></comment></comment><comment><USER13> Also, I recommend checking out the QuantStart project started by Michael Halls-moore. Really handy if you are using the Oanda Api. Plus, there exists some great articles on the quantstart site about the creation of the code. [https://github.com/mhallsmoore/qsforex](https://github.com/mhallsmoore/qsforex)

</comment><comment><USER14> I have read the anti-FlashBoys book to try and grasp it (HFT is not bad and has cut spreads in equities by 5/6ths in ten years) but I still do not understand one HFT algorithm (other than the debunked front running idea)

Could someone explain what algo traders / HFT actually _do_? Is it all looking for a variation off of a relationship between say OrangeGrowersInc and OrangeJuiceBottlers Inc? And why is speed so important then?

<comment><USER15> What is the "anti-FlashBoys" book?

(Asuming you don't mean FlashBoys)

<comment><USER16> Flash Boys: Not So Fast [http://www.amazon.co.uk/gp/aw/d/B00P0QI2M2?ie=UTF8&redirectF...](http://www.amazon.co.uk/gp/aw/d/B00P0QI2M2?ie=UTF8&redirectFromSS=1&pc_redir=T1&noEncodingTag=1&fp=1)

<comment><USER14> Yes thank you. Well worth reading. I was quite annoyed by what seems to be such evident lack of critical thinking by Michael Lewis

I picked up (and have even blogged about) the idea that "good" traders were trying to buy 100m USD of BP shares in three different exchanges and that by the time 1/3 of the order hit London HFT traders "somehow" worked out that there was another 2/3 going and sped over microwave towers to front run the order. But how do you work out that the london order is going to be for 100m? What if it's just for 50m - suddenly you have bought 50m shares of BP no one wants. It's an insane risk.

So I still don't know what HFT and algo is about (it seems to be there are arbitrage opportunities between strongly correlated stocks, and presumably arbitrage opportunities between strongly correlated stocks in different exchanges. And the speed issue is having to beat other people who have the same algo / correlation as you)

But honestly I would like some confirmation

</comment></comment></comment></comment><comment><USER17> It's strange to call this HFT. The author makes a note of it, and I suppose has the right to call it whatever he wants.

However, newbies who use this project to learn automated trading should be aware that this is not considered HFT in the industry.

<comment><USER18> I would love to hear your definition of HFT. I've worked in the industry and never seen it defined well.

<comment><USER19> Agree with you that there is no definition of HFT. If you define it by your trading forecast horizon then I've seen some people argue that HFT could be as long as a few hours. Could you argue that HFT on a multiple second or shorter forecast timescale is not HFT? I think HFT is, more broadly, any sort of intraday trading where your signals are derived from intraday tickdata.

</comment></comment></comment><comment><USER20> It's great that you're sharing, so thanks for that. I get it that it's better than pseudo code but I'm having a hard time seeing the words "high-frequency trading and Python" next to each other

</comment><comment><USER21> Python can be a good tool to prototype hft algos but not for trading (probably you want to trade under < 1ms).

</comment><comment><USER22> see a few comments on the speed of python - given that the strategy is on a retail platform i'm not sure choice of language will be the bottleneck.

</comment><comment><USER23> This is really cool, thanks for sharing it out!</comment>