The Unreasonable Effectiveness of Deep Feature Extraction - <AUTHOR> http://www.basilica.ai/blog/the-unreasonable-effectiveness-of-deep-feature-extraction/ <comment><USER2> Deep feature extraction is important for not only image analysis but also in other areas where specialized tools might be useful such as listed below:

o [https://github.com/Featuretools/featuretools](https://github.com/Featuretools/featuretools) \- Automated feature engineering with main focus on relational structures and deep feature synthesis

o [https://github.com/blue-yonder/tsfresh](https://github.com/blue- yonder/tsfresh) \- Automatic extraction of relevant features from time series

o [https://github.com/machinalis/featureforge](https://github.com/machinalis/featureforge) \- creating and testing machine learning features, with a scikit-learn compatible API

o [https://github.com/asavinov/lambdo](https://github.com/asavinov/lambdo) \- Feature engineering and machine learning: together at last! The workflow engine allows for integrating feature training and data wrangling tasks with conventional ML

o [https://github.com/xiaoganghan/awesome-feature- engineering](https://github.com/xiaoganghan/awesome-feature-engineering) \- other resource related to feature engineering (video, audio, text)

<comment><USER3> Definitely. There's been a lot of exciting work recently for text in particular, like [https://arxiv.org/pdf/1810.04805.pdf](https://arxiv.org/pdf/1810.04805.pdf) .

<comment><USER4> Or from today, OpenAI's response to BERT: [https://blog.openai.com/better- language-models/](https://blog.openai.com/better-language-models/)

Breaks 70% accuracy on the Winograd schema for the first time! (a lazy 7% improvement in performance....)

</comment></comment></comment><comment><USER5> As the author acknowledges, we might be living in a window of opportunity where big data firms are giving something away for free that may yet turn out to be a big part of their furure IP. Grab it while you can.

On a tangent, I really like the tone of voice in this article. Wide eyed, optimistic and forward looking while at the same time knowledgeable and practical. (Thanks!)

<comment><USER6> _big data firms are giving something away for free_

On that note, does anyone know if state-of-the-art models trained on billions of images (such as Facebook's model trained via Instagram tags/images, mentioned in the post) are publicly available and, if so, where?

Everything I turn up with a brief Google seems to have been trained on ImageNet, which the post leads me to believe is now small and sub-par ...

<comment><USER7> Have you found anything?

<comment><USER6> Afraid not — I was hoping for some replies here!

</comment></comment></comment></comment><comment><USER8> This is very interesting and timely to my work, I had been struggling with training a Mobilenet CNN for classification of human emotions ("in the wild"), and struggling to get the model to converge. I tried multiclass to binary models e.g. angry|not_angry etc. But couldn't get past the 60-70% accuracy range.

I switched to extracting features from Imagenet and trained an xgboost binary and boom...right out of the box am seeing ~88% accuracy.

Also the author's points about speed of training and flexibility is major plus for my work. Hope this helps others.

<comment><USER3> Yeah, I think this pattern is pretty common. (Basilica's main business is an API that does deep feature extraction as a service, so we end up talking to a lot of people with tasks like yours -- and there are a _lot_ of them.)

We're actually working on an image model specialized for human faces right now, since it's such a common problem and people usually don't have huge datasets.

</comment></comment><comment><USER9> >But in the future, I think ML will look more like a tower of transfer learning. You'll have a sequence of models, each of which specializes the previous model, which was trained on a more general task with more data available.

He's almost describing a future where we might buy/license pre-trained models from Google/Facebook/etc that are trained on huge datasets, and then extend that with more specific training from other sources of data in order to end up with a model suited to the problem being solved.

It also sounds like we can feed the model's learnings back into new models with new architectures as well as we discover better approaches later.

<comment><USER10> What do you think of life-long learning scenario that models are trained incrementally forever? For example, I train a model with 1000 examples, it sucks. The next guy pick it up and train a new one by putting a regularizer over mine. It might still suck. But after maybe 1000 people, the model begins to get significantly better. Now, I will pickup what I left and improve it by leveraging the current best. This continues forever. Imagine that this community is supported by a block chain. We won't be relying on big companies any more eventually.

<comment><USER11> What is it with the word 'blockchain' that will make people toss it into otherwise completely unrelated text?

<comment><USER12> nothing, they're describing a series of content addressable blocks that link back to their ancestors. Which is a good application of a block chain. Think IPFS.

It's not cryptocurrency. Though cryptocurrency definitely popularized the technique.

<comment><USER13> IPFS isn't a blockchain just like git isn't a blockchain. "Blockchain" has semantic meaning that "a chain of blocks" does not.

</comment></comment></comment></comment></comment><comment><USER14> A few caveats here:

\- It works (that well) only for vision (for language it sort-of-works only at the word level: [http://p.migdal.pl/2017/01/06/king-man-woman-queen- why.html](http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html))

\- "Do Better ImageNet Models Transfer Better?" [https://arxiv.org/abs/1805.08974](https://arxiv.org/abs/1805.08974)

And if you want to play with transfer learning, here is a tutorial with a working notebook: [https://deepsense.ai/keras-vs-pytorch-avp-transfer- learning/](https://deepsense.ai/keras-vs-pytorch-avp-transfer-learning/)

<comment><USER3> There's actually been a lot of really good work recently around textual transfer learning. Google's BERT paper does sentence-level pretraining and transfer to get state of the art results on a bunch of problems: [https://arxiv.org/pdf/1810.04805.pdf](https://arxiv.org/pdf/1810.04805.pdf)

<comment><USER14> Thanks for this reference, I will look it up. Though, from my experience people in NLP still (be default) train from scratch, with some exceptions for tasks on the same dataset:

\- [https://blog.openai.com/unsupervised-sentiment- neuron/](https://blog.openai.com/unsupervised-sentiment-neuron/)

\- [http://ruder.io/nlp-imagenet/](http://ruder.io/nlp-imagenet/)

<comment><USER15> This is true, but rapidly changing. In addition to fine tuneable language models, you can do deep feature extraction with something like bert-as-service [0] ... You can even fine tune Bert on your days, then use the fine tuned model as a feature extractor.

[0] [https://github.com/hanxiao/bert-as- service](https://github.com/hanxiao/bert-as-service)

</comment></comment></comment></comment><comment><USER3> Hi everyone! Author here. Let me know if you have any questions, this is one of my favorite subjects in the world to talk about.

<comment><USER9> What do you think are the most interesting types of problems to solve with this?

<comment><USER3> I think if you have a small to medium sized dataset of images or text, deep feature extraction would be the first thing I'd try.

I'm not sure what the most interesting problems with that property are. Maybe making specialized classifiers for people based on personal labeling? I've always wanted e.g. a twitter filter that excludes specifically the tweets that I don't want to read from my stream.

<comment><USER9> One problem that intrigues me is Chinese-to-English machine translation. Specifically for a subset of Chinese Martial Arts novels (especially given there's plenty of human translated versions to work with).

So Google/Bing/etc have their own pre-trained models for translations.

How would I access that in order to develop my own refinement w/ the domain specific dataset I put together?

<comment><USER3> I don't think you could get access to the actual models that are being used to run e.g. Google Translate, but if you just want a big pretrained model as a starting point, their research departments release things pretty frequently.

For example, [https://github.com/google- research/bert](https://github.com/google-research/bert) (the multilingual model) might be a pretty good starting point for a translator. It will probably still be a lot of work to get it hooked up to a decoder and trained, though.

There's probably a better pretrained model out there specifically for translation, but I'm not sure where you'd find it.

</comment></comment></comment></comment></comment><comment><USER16> Very interesting article! It answered some questions I've had for a long time.

I'm curious about how this works in practice. Is it always good enough to take the outputs of the next-to-last layer as features? When doing quick iterations, I assume the images in the data set have been run through the big net as a preparation step? And the inputs to the net you're training is the features? Does the new net always only need 1 layer?

What are some examples of where this worked well (except for the flowers mentioned in the article)?

<comment><USER3> > Is it always good enough to take the outputs of the next-to-last layer as > features?

It usually doesn't matter all that much whether you take the next-to-last or the third from last, it all performs pretty similarly. If you're doing transfer to a task that's very dissimilar from the pretraining task, I think it can sometimes be helpful to take the first dense layer after the convolutional layers instead, but I can't seem to find the paper where I remember reading that, so take it with a grain of salt.

> When doing quick iterations, I assume the images in the data set have been > run through the big net as a preparation step?

Yep. (And, crucially, you don't have to run them through again every iteration.)

> And the inputs to the net you're training is the features? Does the new net > always only need 1 layer?

Yeah, you take the activations of the late layer of the pretrained net and use them as the input features to the new model you're training. The new model you're training can be as complicated as you like, but usually a simple linear model performs great.

> What are some examples of where this worked well (except for the flowers > mentioned in the article)?

The first paper in the post ([https://arxiv.org/abs/1403.6382](https://arxiv.org/abs/1403.6382)) covers about a dozen different tasks.

</comment></comment><comment><USER17> It's hard to ask my question without sounding a bit naive :-) Back in the early nineties I did some work with convoluted neural nets, except that at that time we didn't call them "convoluted". They were just the neural nets that were not provably uninteresting :-) My biggest problem was that I didn't have enough hardware and so I put that kind of stuff on a shelf waiting for hardware to improve (which it did, but I never got back to that shelf).

What I find a bit strange is the excitement that's going on. I find a lot of these results pretty expected. Or at least this is what _I_ and anybody I talked to at the time seemed to think would happen. Of course, the thing about science is that sometimes you have to do the boring work of seeing if it does, indeed, work like that. So while I've been glancing sidelong at the ML work going on, it's been mostly a checklist of "Oh cool. So it _does_ work. I'm glad".

The excitement has really been catching me off guard, though. It's as if nobody else expected it to work like this. This in turn makes me wonder if I'm being stupidly naive. Normally I find when somebody thinks, "Oh it was obvious" it's because they had an oversimplified view of it and it just happened to superficially match with reality. I suspect that's the case with me :-)

For those doing research in the area (and I know there are some people here), what have been the biggest discoveries/hurdles that we've overcome in the last 20 or 30 years? In retrospect, what were the biggest worries you had in terms of wondering if it would work the way you thought it might? Going forward, what are the most obvious hurdles that, if they don't work out might slow down or halt our progression?

<comment><USER18> If you haven't, you should take a few moments to read the original AlexNet paper (only 11 pages):

[https://papers.nips.cc/paper/4824-imagenet-classification- wi...](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep- convolutional-neural-networks.pdf)

What you're saying is true, it _should_ have worked in theory, but it just _wasn 't_ working for decades. The AlexNet team made several critical optimizations to get it work: (a) big network, (b) training on GPU, and (c) using a ReLU instead of tanh(x).

In the end, it was the hardware that made it possible, but up until their paper it really wasn't for sure. A good analogy is the invention of the airplane. You can speculate all you want about the curvature of a bird's wing and lift, but until you actual build a wing that flies, it's all speculation.

</comment></comment><comment><USER19> Contrast a similar writeup on some interesting observations about solving ImageNet with a network that only sees small patches (largest is 33px on a side)

[https://medium.com/bethgelab/neural-networks-seem-to- follow-...](https://medium.com/bethgelab/neural-networks-seem-to-follow-a- puzzlingly-simple-strategy-to-classify-images-f4229317261f)

</comment><comment><USER20> Question to me is, can you do this with i.e. Random Forest too, or is it specific to NN.

</comment><comment><USER21> This is probably naive, but I'm imagining something like the US Library of Congress providing these models in the future. E.g., some federally funded program to procure / create enormous data sets / train.

<comment><USER22> I don't think it's that naive. NIST is actively getting into this space: [https://www.nist.gov/topics/artificial- intelligence](https://www.nist.gov/topics/artificial-intelligence)

</comment></comment><comment><USER23> I'm wondering how this compares to transfer learning applied to the same model. That is compare deep feature extraction plus linear model at the end vs just transferring the weights to the same model and retraining to your specific dataset.

</comment><comment><USER24> From the article:

 _Where are things headed?

There's a growing consensus that deep learning is going to be a centralizing technology rather than a decentralizing one. We seem to be headed toward a world where the only people with enough data and compute to train truly state- of-the-art networks are a handful of large tech companies._

This is terrifying, but the same conclusion that I've come to.

I'm starting to feel more and more dread that this isn't how the future was supposed to be. I used to be so passionate about technology, especially about AI as the last solution in computer science.

But anymore, the most likely scenario I see for myself is moving out into the desert like OB1 Kenobi. I'm just, so weary. So unbelievably weary, day by day, in ever increasing ways.

<comment><USER25> Hey, I hope you don't take it the wrong way -- I'm coming from a place where I hope you start feeling better -- but what you're experiencing might be depression/mood affiliation. I.e. you feel weary and bleak, so the world seems weary and bleak.

There are enormous problems for humanity to solve, but that has _always_ been the case. From plagues and famines, to world wars, to now climate change, AI risk, and maybe technology centralization. We've solved massive problems before at unbelievable odds, and I want to think we'll do it again. And if not, what of it? What else is there to do but work tirelessly at attempting to solve them?

I hope you feel better, and find help if you need it -- don't mean to presume too much. My e-mail is in my profile if you (or anyone else) needs someone to talk to.</comment></comment>


In my opinion changing Git's master branch name does not help fighting racism - g-garron https://www.garron.blog/notes/github-to-replace-master-with-alternative-term-to-avoid-slavery-references.html <comment><AUTHOR> Getting rid of "master" and "slave" I can understand. If I was the only black kid in computer science class and they started to talk about "master" and "slave" I might feel uncomfortable and put off.

But it's not clear to me that git master has the same sort of meaning. Why couldn't it have a meaning more like 'master copy'?

master copy

an original copy, stencil, tape, etc, from which duplicates are made

<comment><USER2> In good faith, I believe this was the original intent. Like a master key or the gold master record or cd. It is the highest original source of truth from which copies can be made and it was never intended for there to be something called or intimated to be "slave branches".

Unfortunately, there are enough concepts in our discipline that take this single source of truth concept and use the master/slave terminology. For an example Google "Replication master" and see if there are _any_ articles on the page one results that don't reference "slaves". They all do for me when I do a query.

I see absolutely no harm in moving the git nomenclature of "master" to "trunk" or "base" some other term. Keep it as an alias and flag it as depreciated.</comment></comment>


TLS Performance: Rustls versus OpenSSL - <AUTHOR> https://jbp.io/2019/07/01/rustls-vs-openssl-performance.html <comment><USER2> While I'm happy to see such progress, this ultimately doesn't say a lot. Openssl is known questionable code, and more or less is known for its kitchen sink and ubiquitousness, which I presume rustls(wisely) avoided. That said, I'd be interested to see a feature breakdown, but also put up against others like boring, polar, libre, etc.

<comment><USER3> Don't boringSSL & libreSSL have pretty much feature-parity with openSSL besides legacy and deprecated ciphers/features since they are forks of openSSL?

<comment><USER4> LibreSSL - the devs have concluded that, so long as TLSv1.2 is used with secure ciphers, it is every bit as secure as TLSv1.3 - but they still include some ciphers that have been more successfully attacked.

The library itself has the kind of compatibility hooks, and zealous porters out there that you'll be surprised what works.

You know all those changes that happened between OpenSSL v1.0 and v1.1? Every project had to work hard to accomodate those changes - compiling that code for older versions still has to work, or companies freak out.

Add LibreSSL into the correct side of the header-files' if-statements. Usually its legacy, but we're recently having to consider different versions of LibreSSL.

When a single if-statement begins to span three lines for readability's sake, you know you're shoving LibreSSL into the seam between OpenSSL versions.

Libp11, the backend for Yubikeys was surprisingly simple to get running on a different SSL backend. Three additional if-statements in one header-file, and one extended out. Code compiles, works beautifully, even past LibreSSL's v2.8.3 that I was running at the time.

BoringSSL - while Im less intimately familiar with Google's offering? I thought they were the first to implement TLSv1.3 - years before anyone else was even considering it.

I have no experience porting anything to it. The only folks I know who do? Are Google Engineers, writing proprietary code. The API is explicitly flagged as unstable/quickly changing on their github mirror.

Google, like always, doesnt care about your needs.

<comment><USER3> I mostly wanted to question the

> Openssl is known .... for its kitchen sink

and then juxtaposing it with forks of it that retain a lot of its features and compatibility with it.

</comment></comment></comment></comment><comment><USER5> Is it possible to use RustLs from C? Is a C header available?

<comment><USER6> Rust works pretty well with C, and you could use cbindgen[0] to generate c headers for rust functions marked as extern and it would work when linked together. _However_ , rustls doesn't seems to be providing any such function. Most of its API is rust-based, using stuff that isn't directly available in C (Option, Vec, etc.). So you would have to write at least some rust code that calls the library, and have your C code calls the rust code that you write.

On the flip side, calling C code in rust is pretty easy - you have the entire libc available, and rust-bindgen[1] generate bindings for you, which in my experience works very well. So, you may also consider writing your program in rust… If that's possible.

      [0]: https://github.com/eqrion/cbindgen
      [1]: https://github.com/rust-lang/rust-bindgen</comment></comment>


What Working at Facebook Has Taught Me About Design Critique - <AUTHOR> https://medium.com/@tannerc/critique-is-an-important-part-of-any-design-process-whether-you-work-as-part-of-a-team-or-solo-ef3dcb299ce3#.ie6x6ae4m <comment><USER2> Learning to give and receive feedback is one of the skills that will make you and your team better and is an important part of communication. Effective communication is crucial, no matter the size of the team. The list of the distinctions between 'criticism' and 'criticism' in this article is particularly valuable.

Criticism passes judgement — Critique poses questions; Criticism finds fault — Critique uncovers opportunity; Criticism is personal — Critique is objective; Criticism is vague — Critique is concrete; Criticism tears down — Critique builds up; Criticism is ego-centric — Critique is altruistic; Criticism is adversarial — Critique is cooperative; Criticism belittles the designer — Critique improves the design

<comment><USER3> I disagree. Criticism is awesome. If people can't handle criticism then they should change profession. If 1 person says something negative. Then there's 1000s more who feel the same who say nothing.

Constructive criticism is even better.

Criticism is only personal if you make it personal. Criticism is only egocentric if you make it egocentric. Criticism is on vague when it is not constructive. Criticism only belittles if it is targeted at the person and not the thing being critiqued.

<comment><USER4> >Criticism is only personal if you make it personal. Criticism is only egocentric if you make it

That usually begins with the person doing the criticizing.

</comment></comment></comment><comment><USER5> It's vital to learn to see past emotion when receiving statements from critics.

I have heard lots of comments about my projects over the years and I have noticed some patterns:

\- Negative feedback is usually all you get. You have to motivate yourself by imagining that most people are pretty happy based on downloads/client- count/whatever.

\- People aren't good at communicating when they're frustrated but they usually turn into nice people once you've fixed the problem. Most people just have a job to do and a deadline to meet. Pay attention to what they say is wrong, and not their mood.

\- Strangely, people will spend great amounts of time writing comments on random web sites or review pages about issues that they will _never even E-mail you about_ , no matter how easy you make it for them to contact you. Therefore, if you _really_ want to get some realistic critiques of your work, you may have to scour the web for them. I look at it this way: real, honest feedback is _rare_ and _vital_ to really understand what you may have overlooked. Even if you can't possibly contact the Random Web Commenter, find and fix their issues and you'll end up with a better product.

</comment><comment><USER6> I don't think this is useful for startups in general. It sounds like it's good for gigantocorps like IBM and Facebook (and yes, I'm lumping them together, as they have more in common than they differ). For a lean startup, I don't think you're going to have the depth in your staff nor the time to put into these sorts of activities. It's all-hands-on-deck, and we don't have time to slow down and hand-hold each other through their own work. I'd be hiring people I trust to get their work done without me needing to tell them they are or are not doing a good job.

I mean, at a particularly early stage startup, establishing the "three roles" could be _the entire company_. In such a scenario, how do we not already know each other's intimate business?

<comment><USER7> This is one of those "slow down to speed up" kind of things that startups ignore at their peril. People, especially technical types, have a mistaken assumption that "speed" in startupland is based on the amount of code shipped and anything that distracts from code shipping is a distraction to be ruthlessly excised.

The real speed that a startup should be optimizing for is the degree of new, actionable, insights produced on the quest towards product/market fit. Code should only be built insofar as it helps provide scaffolding towards that goal.

Design critique is helpful because it helps you get faster at getting faster. It's not some crutch that weaker designers need to adopt to keep up with the rest of us, it's a fundamental way in which designers learn and get better at their craft.

<comment><USER6> But look at the artifice of this whole process. The need to establish "roles". The prescribed language. If I did that with my team, they'd rightly look at me sideways and say, "what the hell is wrong with you, just spit it out already."

It's not that I'm advocating a heads-down, never-come-up-for-air approach. It's that I'd expect these sort of conversations to have happened in an organic fashion that doesn't cargo-cult corporate policy gerrymandering.

<comment><USER8> Sometimes a little bit of ceremony helps change existing behaviors and gives people with different perspectives the opportunities to speak up.

<comment><USER9> So far from true. At a startup, everyone is blunt, because it pays to be. Everyone there is counting on success.

Whereas at a corp, the ceremony was created by someone else who may not even work there anymore, for some superpsionic reasons that everyone basically just shrugs at because they don't want to lose their jobs. It'a a sludsgefest that startups can skip - no point in needless vagaries when in a startup, it's actually appreciated to collaborate organically instead of being culled for doing so.

<comment><USER8> So much of that response is utterly loaded with flawed assumptions. The irony is that your response is exactly why sometimes it's important to be cognisant of how to conduct critical conversations. It's not about "needless vagaries", it's actually about direct and on-point critique, without the loaded value judgements. This isn't a skill that tends to come naturally to people, that's what the "ceremony" is for.

</comment></comment></comment></comment></comment></comment><comment><USER10> Read this a few weeks ago. Really useful advice.

</comment><comment><USER11> Speaking of design critique, are young people able to read articles with hilarious animated gifs in them? Because I literally cannot; a pity because the first section seemed promising.

<comment><USER12> More and more I'm getting in the habit of hitting the "Reader View" button in Safari. It removes all ads, gifs, "related content" etc. and gives me the pure content: the article in an easy to read font.

<comment><USER13> Firefox also has a built-in Reader View but it leaves the graphics in, in case they are referred to by and necessary to understand the text.

</comment></comment></comment><comment><USER14> Yes

Edit: Fuck your downvotes. I answered his question. HN is such a shit show these days.

<comment><USER15> > _Fuck your downvotes. [...] HN is such a shit show these days._

Please don't post comments like this.

We detached this subthread from [https://news.ycombinator.com/item?id=10922770](https://news.ycombinator.com/item?id=10922770) and marked it off-topic.

<comment><USER14> Someone asked a yes/no question. I answered yes. And got downvoted. I stand by my statement.

<comment><USER15> Everyone gets downvoted. Yes, it's annoying, but venting that annoyance back on the site adds nothing of value, only off-topic noise. The HN guidelines ask everyone not to do that, so please don't do that.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

<comment><USER14> How do I delete my HN account and all it's comments?

<comment><USER16> [http://lmgtfy.com/?q=How+do+I+delete+my+HN+account+and+all+i...](http://lmgtfy.com/?q=How+do+I+delete+my+HN+account+and+all+it%27s+comments)

<comment><USER14> Thanks for this reply. It saves me from having to explain why I'd want to do such a thing.</comment></comment></comment></comment></comment></comment></comment>


New GM Cruise Self-Driving Video Shows More Mastery of SF Roads - <AUTHOR> https://www.driverless.id/news/video-analysis-new-gm-cruise-self-driving-video-shows-more-mastery-sf-roads-0176178/ <comment><USER3> That's pretty cool, it's great to see GM not sucking at bleeding edge software. They're getting their money's worth out of that billion dollar acquisition last March. I think Kyle Vogt is GM's MVP.

Looking at the 2016 California DMV disengagement reports, Cruise is at 181 disengages in 9,776 miles of testing in California for 2016. This can be compared crudely to Waymo's 124 disengages in 635,868 miles for 2016.

Waymo's operation is 2 orders of magnitude more advanced than Cruise, but in terms of video demos, we haven't seen much of the environments Waymo's cars are capable of navigating. Waymo has only revealed their cars driving on suburban roads, it seems only Cruise is tackling hectic downtown driving head- on.

GM, with their Lyft partnership and Orion assembly plant is, I think, in a great position to deploy early fast with 1st gen robotaxis.

<comment><USER4> Waymo cars frequently drive the same routes several times a week. In fact if you go to certain intersections such as El Camino Real and San Antonio Rd, during the work week you only have to wait a few minutes to see one or more go by.

<comment><USER5> Do you think the routes are designed so that the car encounters specific obstacles the teams are working on, or do you think its more like they are doing extensive mapping of an area to make the autonomy a bit easier?

<comment><USER6> I suspect it is so that it is easier to measure the effects tweaks/development to the algorithms make.

</comment></comment></comment></comment><comment><USER7> Very impressive, I'm wondering how it was able to get around the white truck that was double-parked? Does it have sensors on the mirrors and point straight out? I imagine it would have no information as to oncoming cars, with the truck blocking almost everything, so I wonder what the algorithm is for that.

<comment><USER8> Tesla posted about updates to their auto pilot system which included bouncing radar under the car in front of them.

[https://www.tesla.com/en_AU/blog/upgrading-autopilot- seeing-...](https://www.tesla.com/en_AU/blog/upgrading-autopilot-seeing-world- radar)

</comment></comment><comment><USER9> At around 1:54 in the video, it comes to a stop light that is in front of a Shell station's gas price board. It actually took me a second to notice the stop light. There is another light on the left side of the screen, but it still made me think.

Is it likely just using some image recognition and a confidence level to determine when it is safe to go?

If so, could the system be fooled by a sign that is meant to always look like a red light? Or worse, a green light to trick the system to run a red?

<comment><USER10> Eventually, I expect we'll have some national standard for wireless transmitters on traffic signals, so they can send a message (timestamped and cryptographically signed with the state's department of transportation's public key) to the cars to tell them unambiguously when to stop and when it's safe to go.

(This could help human drivers too; just add a red/yellow/green light to the car dashboard for those times when you can't see the actual light because there's a big truck in the way, or the sun is directly behind the traffic signal.)

I also expect that by the time this sees near-universal adoption, we'll already have the image recognition corner cases mostly worked out.

<comment><USER11> A nitpick, but you would sign with the _private_ key, and verify with the _public_ key.

<comment><USER10> Thanks, that's what I meant.

</comment></comment></comment></comment><comment><USER12> Seeing a firetruck in this made me wonder, how do self driving vehicles deal with emergency services behind them? Detect the flashing lights?

<comment><USER13> Given all the crazy corner cases I know they have covered I cannot imagine they don't deal well with emergency vehicles. For example, I know Waymo has put a ton of work into just covering the cases when motorcycles are lane- splitting and overtaking them at high speed [0]. I'm sure they deal with ambulances just fine.

0 - [http://extremepowersportssa.com/google-self-driving-car- proj...](http://extremepowersportssa.com/google-self-driving-car-project- considers-lane-splitting-motorcyclists/3859)

<comment><USER14> Not to be pedantic, but it's lane sharing, not lane splitting. Lane splitting, aka riding the dots, is illegal even in CA

<comment><USER15> No, this is legal in CA.

California Highway Patrol guidelines: [http://lanesplittingislegal.com/assets/docs/CHP-lane- splitti...](http://lanesplittingislegal.com/assets/docs/CHP-lane-splitting- guidelines-California.pdf)

<comment><USER16> Well, we're probably missing the point, and you probably both know, but I'll continue anyway.

The reason lane sharing is legal in California is because there is not a prohibition against multiple vehicles in the same lane adjacent to each other. The other states, one presumes (since we know it's only legal in CA), specify that vehicles may not drive side-by-side in the same lane.

I'm not sure what CA law has to say about riding _between_ lanes (or, probably more accurately, occupying 2 lanes at the same time which would be what you were doing if your wheels were going right down the line), but all this business is why we call it lane SHARING; in theory, any time spent on the white line is during a lane change (properly signalled, of course, except for those who want to point out that CA state law is pretty vague on whether you even need to signal a lane change).

So the fact that the CHP guideline calls it splitting does nothing to change the legality of splitting versus sharing.

But the motorcycle community typically prefers to call it lane sharing, as that's the circumstance under which it's definitely legal (assuming you are not breaking any other laws at the same time).

And yes, I saw a demo from Google a number of years ago where the car very gracefully handled a motorcycle lane sharing past it.

<comment><USER17> Your comment matched my understanding from ~10 years ago (from discussions on CA motorcycling forums). However, AB-51 passed last August and it's added "lane splitting" to the vehicle code (which I just found out about 10 minutes ago).

[https://leginfo.legislature.ca.gov/faces/billTextClient.xhtm...](https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=201520160AB51)

Anecdotally, I'd heard of motorcyclists cited for violating the CA Vehicle Code if they occupy two lanes: "Existing law requires ... that a vehicle be driven as nearly as practical entirely within a single lane and not be moved from the lane until the movement can be made with reasonable safety."

This resulted in the semantic distinction. It's _not_ illegal for two vehicles to share a lane. It is illegal (and still is) to drive in two lanes or to drive "between" the lanes.

</comment></comment></comment></comment></comment></comment><comment><USER18> I think GM is in a great position to compete with Uber and tesla if they capitalize on the tech.

As a customer I'll still be happy with multiple choices driving prices down and offering a better service than buses and taxis.

God I hate buses in the US. Public transport is very broken.

<comment><USER19> I'm sad to say this, but I think GM is going to crush Tesla in the long run. GM knows about building cars, tech becomes cheaper and available over time. It will take Tesla longer to learn how to build and mass produce at a cheap price than for GM to have all the tech they need to match up with Tesla. Initially I was worried that Honda would be the one, but they don't seem to be taking the electric and self driving car serious. GM has their volt/bolt with decent range, I happen to work at a company that shares a lot with some of the auto manufacturing suppliers for American car companies, and I see them testing their self driving cars sometimes. Oh and they have a bunch of Teslas for inspiration I suppose. :)

<comment><USER20> _I think GM is going to crush Tesla in the long run._

Maybe in the short run.

Jan 2017 EV sales:[1]

        Chevy Volt:         1611
        Toyota Prius Prime: 1366
        Chevy Bolt:         1182
        Tesla Model S:       900
        Nissan Leaf:         772
        Tesla Model X:       750

The Bolt has only been shipping since late December. That's just getting started.

[1] [http://insideevs.com/monthly-plug-in-sales- scorecard/](http://insideevs.com/monthly-plug-in-sales-scorecard/)

<comment><USER19> Why do you say so? That's 2793 Chevys vs 1650 Teslas.

There are more people who can afford Chevy than Tesla.

Tesla hasn't demonstrated that they can manufacture cheaply, they have promised and we are still waiting. GM might not have a luxury electric package, but they know how from their experience in building Cadillac. Tesla not in the dealerships is a problem. GM manufactures outside of the country and has about 30 factories. They can build and sell electric car outside. Tesla doesn't, building, shipping & custom fees increases price of car a lot.

I'm not convinced that Tesla will win this battle if all variables remain the same. They need to move faster to have a chance.

<comment><USER21> Animats is saying you don't have to wait to see GM beat Tesla, not that they won't beat them in the long run.

Of course, comparing sales of Chevy's and Toyota's mostly low-end lineup to Tesla's luxury offerings is pretty odd anyway. At the moment it seems pretty TBD.

(I also maintain that including cars like the Volt and Prius in the same category as the Bolt and Model S doesn't make much sense. You may as well compare the Bolt and Model S to all gas-powered cars if you're going to include those gas-powered cars in the mix — it's equally unenlightening.)

<comment><USER22> Allegedly some large fraction of driving takes place within 20 miles of home. In light of that, including vehicles with 20-40 miles of EV only range makes perfect sense.

<comment><USER21> Sure, a lot of driving is done near home. But if trips outside that range were really negligible, gas tanks wouldn't be a standard feature in those cars. They're not including them just for funsies.

As an aside, I think you're slightly misunderstanding that statistic. If I do a 60-mile round trip directly away from my house and back, 66% of those miles are within 20 miles of home even though the trip isn't anywhere near 20 miles in total. If I drive 15 miles away from my house and then drive a a semi- circle for 80 miles, all of those are within 20 miles of home.

</comment></comment></comment></comment></comment></comment></comment><comment><USER23> > The Cruise car makes a left turn and then gets stuck behind a delivery > truck. It waits as a couple of other vehicles overtake it

I suppose I'd probably notice if I was a semi-engaged driver, but not as much if I considered myself "just a passenger" in an autonomous vehicle. It looks like barely more than 30s elapses there.

<comment><USER24> I'm not sure I would have done anything different than the computer did. I'm a cautious driver and have been in similar situations where I felt it was prudent to take a moment and really try to see past the truck to look for any oncoming traffic. I've then had an impatient driver behind me decide to zoom around.

<comment><USER25> I used to be the type of driver who would impatiently zoom around, until one day I noticed several deer beginning to cross the freeway. I slammed on my brakes, but the driver behind me honked and swerved around me and, at over 80mph, took two of the deer with him. He left the scene in an ambulance and his car went to the scrapyard on a flatbed. No one ever told me what happened to the man, but while we were waiting for the ambulance and while he was being loaded, I never saw him even wiggle a toe or blink an eye. He was breathing but completely unresponsive. I don't know if he survived or not.

It's moments like that one that can change your outlook on life in an instant. That was four years ago and I haven't intentionally broken the speed limit since then. Even if it means other impatient drivers are zooming around me. I have, however, been rear-ended twice since then.

<comment><USER26> > _Even if it means other impatient drivers are zooming around me. I have, > however, been rear-ended twice since then._

Wait, so you _already_ saw someone be seriously injured, and since then, you've caused two other accidents (which are a danger to you too), and you're still going with "I'll drive the speed limit and _not_ drive with the flow of traffic?"

smh

Drive with the flow of traffic: save lives.

<comment><USER27> That's a bold assumption about his accidents being his fault. Maybe he was rear ended at a light when fully stopped. And even if he was going slower than traffic, it's still the other driver's job not to _hit other cars._

</comment></comment></comment></comment></comment><comment><USER28> I wish they released a realtime cut of this. I feel like some of the quirks have been obscured with sped up footage.

</comment><comment><USER29> That video was impressive. It shows great situational awareness.

Here's what I wonder about the future. Is there a point in which consolidation will take place in the auto industry as a result of superior _self-driving_ performance and capabilities? Meaning -- can the value created by self-driving software be more of a factor than the other existing assets of a GM, Ford, Chrysler, Toyota, etc?

<comment><USER30> I predict that self-driving technology will become a commoditized box + sensors for use by different car brands. It will very quickly reach the point where one autonomous driving engine will not be noticeably better or worse than any of the others on the market.

<comment><USER29> Interesting. So price will not factor in? And, will a luxury car be expected to have better self-driving awareness/features than a cheaper car?

And before it hits commodity state, what if Waymo's Chrysler minivans and GM's Chevrolets are far superior self-drivers than say, BMWs & Mercedes? How much does that alter the demand curve?

<comment><USER30> I don't expect there to be different tiers of autonomous driving, because the sensors are cheap and the software can be duplicated at no cost. There are a billion cars in the world, so whatever the initial cost is to develop the self driving software, it's going to be pennies per car in the long run. It will be very tough for car manufacturers to defend their customers dying in a totally avoidable accident because their driving AI -- like a chess engine -- deliberately made a worse move because the customer didn't buy the premium package.

I expect premium car brands to charge more for the same self-driving technology, though, as at least in the short term self-driving capability will be sold as an expensive extra.

I think the commodity stage will be hit two years after the first truly self- driving cars are sold. So it shouldn't give one brand a lasting advantage.

<comment> saiya-jin there might be some added value in premium cars - for example certification that it can go 100kmh fully autonomous (better processing hardware, more & expensive sensors), 130 (as per most EU countries) and so on. BMW can have a gap on safe driving at 200 kmh, ferrari 250 kmh.

but overall I feel those premium brands will have very tough times - most of their added value is in driving experience (for example what BMW delivers in ie 3-sedans is a wonderful experience) and this will be removed. not that many people feel the need to pay premium for something that will feel like a commercial airliner, travelling will be race to the bottom costwise.

<comment><USER31> > most of their added value is in driving experience

I think BMW's added value is in comfort and prestige. People buy a BMW because they can afford it and other people can see that they have one. The ride is smooth, the interior is nice, it has good sound insulation, etc. A small fraction buys them because they are fun to drive. BMW makes luxury sedans, not sports cars. The top of the luxury sedan market is not Ferrari but RR and Bentley where people literally buy the car and pay someone else to drive it.

</comment></comment></comment></comment></comment></comment><comment><USER32> So, silly question - why are these technologies not being made broadly available as driver assist tools while we work our way up to L4 driving automation?

If we're really concerned about making driving safer, this seems like a good compromise until we get to broadly available (and affordable) L4.

<comment><USER33> > So, silly question - why are these technologies not being made broadly > available as driver assist tools while we work our way up to L4 driving > automation?

Well, they are. Lane departure warnings, auto braking, self parking are all incremental automation/assistive technologies that have been rolling out slowly and will continue to.

Advertising stuff now is a kind of marketing aimed at hiring and in building enthusiasm for a company like GM that as seen as a has been (not picking on GM, they are simply the topic of this article!).

I think your real question is, "why aren't I seeing these features on current model year cars?" It's about the structure of the car industry. For example, when I worked with <car company X> on brake-by-wire in 1996 it was for the 2004 model year. They had a team with functional steer by wire -- I still haven't seen it in production! This is a combination of historical industry practice, a focus on safety, and regulatory constraints. A mixture of good and bad.

<comment><USER32> > I think your real question is, "why aren't I seeing these features on > current model year cars?"

Or as 3rd party modules which can be installed by a moderately competent mechanic.

And while yes, some of these are being offered, they are far from freely available and affordable - two things which would seem to me to be important for full blown automation.

<comment><USER34> Look up Comma.ai. They got a cease and desist letter from the NHTSA.

<comment><USER35> ... that'd be the guy who refused to answer even basic safety testing questions though, right?

Or am I thinking of someone else?

</comment></comment></comment></comment></comment><comment><USER36> At 1:04 [1], it stops for a few seconds before the intersection. Seems to be due to the white car that is parked in opposite direction as all other cars so it waited for ~20 seconds to confirm that the car is indeeded parked? Should a no-motion indicates it's not moving? Why wait for so long?

[https://youtu.be/Vfgjemwc9NU?t=63](https://youtu.be/Vfgjemwc9NU?t=63)

<comment><USER37> They marked it with the 'Scheduled Stop' in the top left, and it seems like the passenger is fiddling with some controls as well? I assumed this was to simulate a pick-up/drop-off, or was just a waypoint in the trip?

</comment></comment><comment><USER38> While certainly impressive, keep in mind that it's only one video showing one of possibly thousands of trials with possibly thousands of difficult corner- case situations where the car got stuck or acted improperly. One video doesn't prove anything, especially not one with a person ready to take over at any moment.

<comment><USER39> Voluntary data of corner-cases where the car got stuck or acted improperly are available from virtually every company in the self-driving car space except Uber. See the "Vehicle Disengagement Reports" at the DMV's website: [https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/disen...](https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/disengagement_report_2016)

</comment></comment><comment><USER40> American roads look really unfamiliar to someone from the UK. You stop a lot! So many stop signs.

<comment><USER41> There isn't really any planning. San Francisco never removes stop signs, but just adds them steadily as neighborhoods request them. Eventually they are turned into traffic lights. (I imagine "traffic signals installed" must be a performance metric for someone.)

A downside of having stop signs at almost every corner is that drivers seem to forget how to drive in areas without them. They assume that if they approach an intersection without a stop sign then they can drive through it at full speed, instead of looking for cross traffic and pedestrians.

</comment></comment><comment><USER42> Looking at this space, I honestly expected more drive by wire and remote VR pilot type intermediary solutions than full autonomy. I wonder if remote control will be completely bypassed or if certain sectors will still benefit from this.

</comment><comment><USER43> How does this compare to other players right now (ex. Waymo, Tesla, etc)?

<comment><USER20> Car and Driver says Waymo is way out in front.[1] The 2016 California DMV autonomous vehicle disconnect reports are out, and Waymo is doing three orders of magnitude better than the others in miles between disconnects. Waymo is up to 5,000 miles between disconnects, out of 600K miles driven. Waymo reported zero disconnects on interstates and freeways in the last year. For Waymo/Google, this is about 2x better than last year. That was 2x better than the previous year.

Everybody else is reporting orders of magnitude more disconnects per mile.

[1] [http://blog.caranddriver.com/in-the-self-driving-race- waymo-...](http://blog.caranddriver.com/in-the-self-driving-race-waymo-looks- to-be-way-out-in-front/)

<comment><USER44> I'm not sure it's necessarily a good comparison, though. As I recall, Waymo's testing is mostly freeway and suburban driving. I'd be curious to see how much of their testing is done in a heavily urban environment, such as downtown SF or Oakland, and how it affects their disengage numbers.

<comment><USER20> Waymo's testing is mostly on suburban streets in Mountain View CA, with another test unit in Austin. Their accident reports are usually from Mountain View, and it's almost always them being rear-ended.

<comment><USER45> Rear-ended is probably a good thing, since that means automation is erring on the side of caution and applying braking when necessary or when uncertainty is above a certain threshold.

Of course someone could make the argument that excessive braking is dangerous, but honestly, the shit I see drivers do behind the wheel of their car weekly (eating, texting, putting on makeup, _shaving_ ), I'd prefer excessive braking over any of that.

<comment><USER20> Many of Google's rear-endings occur at intersections where there's obstructed visibility to the side. The autonomous vehicle moves slightly forward into the intersection, detects cross traffic, and stops. Someone behind them then hits them at slow speed.[1][2]

[1] [https://www.dmv.ca.gov/portal/wcm/connect/4a39c1b9-ca1f-4184...](https://www.dmv.ca.gov/portal/wcm/connect/4a39c1b9-ca1f-4184-8b78-96d6baf98628/Google_102616.pdf?MOD=AJPERES) [2] [https://www.dmv.ca.gov/portal/wcm/connect/69feb1fa- ff90-4b7a...](https://www.dmv.ca.gov/portal/wcm/connect/69feb1fa- ff90-4b7a-b8b4-dcaacdb20c65/Google_090716.pdf?MOD=AJPERES)

<comment><USER46> Sounds like it needed to check for tailgaters before braking, as humans must do in most city driving. This must be improved by now.

</comment></comment></comment></comment></comment></comment></comment><comment><USER47> As I was watching this video I thought about how much further along with self driving cars we could be if the data was open source. Sharing this type of information between companies would increase everyone's velocity and get us that much closer to real world driverless vehicles.

At some point the vehicles will have to talk to each other anyway (ie. car-to- car network), so we might as well formalize it.

</comment><comment><USER48> This is still pretty simple stuff it has to deal with. If you really want to test Cruise, try driving in downtown SF on a weekday around 5pm.

<comment><USER49> Or any other location in the US also! Try a Boston winter, or Houston's unending road construction. It feels like we're creating self driving cars that are experts at driving in SF.

<comment><USER50> Gotta take the first steps. Its ok to keep raising the bar.

</comment></comment></comment><comment><USER51> Until I see autonomous vehicles testing in blizzard conditions, or even a moderate snowfall, it's relegated to the West coast. This eliminates nearly 2/3 of the US population (who live in areas where snowfall occurs).

We're at least 5-7 years away from them testing in these conditions.

<comment><USER52> Surprisingly, they are already testing in snow conditions.

[http://www.dailymail.co.uk/sciencetech/article-3399315/The-d...](http://www.dailymail.co.uk/sciencetech/article-3399315/The- driverless-car-cope-SNOW-Ford-vehicle-blizzards-tackle-slippery-corners.html)

<comment><USER51> That 'testing' is taking place on the University of Michigan's M City track. I'm looking forward to testing during these conditions, in the real world, with someone behind the wheel acting as backup.

I'll revise my 5-7 year comment then - I think it will this amount of time (if not more) for these autonomous vehicles to move beyond testing (prod) in winter climates.

Driving on the interstate in a snow storm with semi trucks is no joke - <USER2> might actually be a threshold in which pulling over is the best option.

</comment></comment></comment><comment><USER53> As in the first video there is a ~30s delay when stuck behind a truck. Which I think is reasonable and most human drivers take far too many risks. Seems to be than humans will learn to drive better thanks to self driving cars :).

</comment><comment><USER54> Given GM's partnership with Lyft, makes me wonder if Uber should be concerned.

</comment><comment><USER55> Looks like the car cuts off another other driver here?

[http://www.youtube.com/watch?v=Vfgjemwc9NU&t=0m42s](http://www.youtube.com/watch?v=Vfgjemwc9NU&t=0m42s)

<comment><USER56> Looks like both cars have a stop sign, and this one got there first.

<comment><USER55> My driver's ed teacher would have dinged me if I pulled that. True the car got there a smidge after the POV car, but when turning and going through another car's lane you should always yield to that car before going into the intersection. I think a human driver would have said, "Yup, I got here 0.5 seconds before him... but I'll let him go before I attempt the turn."

</comment></comment></comment><comment><USER57> At 00:49, shouldn't the car have yielded to the car coming from the opposite side of the street?

<comment><USER58> Since the GM Cruise car was at the stop sign first, no.

</comment></comment><comment><USER59> Is this going to be Cadillac's "Supercruise"?

</comment><comment><USER60> Anyone knows how those technologies work at night time ?

<comment><USER61> Shouldn't make a difference as, as far as I know, none of the tech they're using is visually based except for a few of the cameras. Most of the primary systems are LIDAR, RADAR, or some other tech that doesn't rely on perfect visual detection and even the ones that do have a mode for "night vision" or a modifier to allow a similar type of function.

<comment><USER62> What about street signs, road markings, and even signals? That must be visual, and they certainly look different at night

Can you create a "normalized" image (hardware- or software-wise) that is independent of the time of day? Is that what you mean by "night-vision" and "modifier"?

I would still imagine that certain light conditions (i.e. just around sunset) could make things difficult.

Would be interesting if anyone knows how they handle this.

<comment><USER61> They do have cameras for those kinds of things but most of them are a backup system and they have the actual signals, signs, and speed limits embedded in the GPS data. If you do a google search, you can find several instances of Tesla's speed-limiting a car going down the freeway where the GPS data was incorrect. The car all but ignored what the signs actually said as long as it could identify that a sign was there.

I'm sure there are light conditions that it can't handle, but there are so many fallback systems that it doesn't have to rely simply on visual data.

<comment><USER62> Ah, ok, so if the can encounters a pre-mapped street signs, the car doesn't bother to actually check the sign.

Hmm... seems a tad dangerous to me. What if the sign is changed, or two signs are almost in identical position?

I suppose it will be ok if the pre-mapped version is used, whenever a confident visual prediction cannot be made (e.g. due to light, mud, bullet holes, or whatever).

<comment><USER61> This is part of the issue that's being addressed in the current autopilot. Tesla throttles the autopilot speed to be no more than a certain amount past the posted speed limit. Drivers have been complaining that the posted limits don't match the pre-mapped limits already.

<comment><USER62> .... still seems a bit dangerous.

Would also be interesting to hear what the other teams do, but I only remember seeing videos in daylight so far.

Somehow I completely overlooked this issue in the past.

</comment></comment></comment></comment></comment></comment></comment><comment><USER63> I think one of the more depressing things is that (potentially) many driver- less cars will be gas powered. Does anyone know if Waymo is going to be used on any electric cars?

There doesn't seem to be any reason driver-less cars have to be electric, but if big companies pushed the tech onto electric cars it'd be a nice way to speed up their adoption and infrastructure for them.

<comment><USER64> >I think one of the more depressing things is that (potentially) many driver- less cars will be gas powered. Does anyone know if Waymo is going to be used on any electric cars?

It's not like the software isn't portable or that GM has any aversion to building electric if market conditions are favorable.

<comment><USER63> Of course not, but having the tech tied to an electric push I think makes the whole package more attractive to consumers. Perhaps I'm too tied to a Tesla mindset on this though, of course self driving cars offer a bunch of advantages on their own.

<comment><USER65> I have a strong sense that self driving will be generally distributed through the US car population before electric only vehicles given the pace of software vs. pace of battery technology. This will drive mass adoption of gas powered self driving cars pushing back the adoption curve for all electrics. One way to stem this may be through tax and insurance incentives, a discount on self driving due to safety, a discount for electric for environmental reasons, and an extra discount for both.

<comment><USER66> I was under the impression that electric cars had a lower cost per mile, but the electrics have a higher up front cost.

If you have a self driving fleet (eg. Lyft, Uber, GM, etc.), you'd want to optimize for the former, not the latter. Uber has proved that if you can get a car within a couple of minutes at a low enough price, it's a valid enough substitute to car ownership, at least in cities.</comment></comment></comment></comment></comment>


Medieval trade networks - <AUTHOR> https://imgur.com/MsXaOdV <comment><USER2> It's fairly amazing that goods managed to get from China to Europe, and vice versa, without any Chinese people visiting Europe for hundreds or years, or any Europeans visiting China.

This was not just in the middle ages, but reaching as far back as the Roman Empire.

<comment><AUTHOR> I agree, I love thinking about the journeys that these objects made, like the Roman coins that ended up in a 12th century Japanese castle. [1]

There were a handful of individuals who made similar trips though - aside from the famous ones like Marco Polo and Ibn Battuta, there's also a few envoys sent on behalf of the Franco-Mongol alliance.[2]

Or the Spaniards who went to the court of Timur: "Timour Beg turned to the knights who had seated around him… and said, 'Behold! here are the ambassadors sent by my son the king of Spain, who is the greatest king of the Franks, and lives at the end of the world.'" [3]

[1] [https://www.npr.org/sections/thetwo- way/2016/09/28/495821834...](https://www.npr.org/sections/thetwo- way/2016/09/28/495821834/coins-from-the-roman-empire-are-found-in-ruins-of- japanese-castle)

[2] [https://en.wikipedia.org/wiki/Franco- Mongol_alliance](https://en.wikipedia.org/wiki/Franco-Mongol_alliance)

[3] [https://resobscura.blogspot.com/2012/10/a-spaniard-in- samark...](https://resobscura.blogspot.com/2012/10/a-spaniard-in- samarkand-1404.html)

<comment><USER3> One of my favourite historical surprises:

> Clavijo recorded his entrance to the capital in great detail, noting the > stores of "silks, satins, musk, rubies, diamonds, pearls, and rhubarb" > carried from China

Today, to me, rhubarb seems so mundane, and so British. And yet just a few hundred years ago, it was an exotic wonder from the east. Rather than from Yorkshire:

[https://en.wikipedia.org/wiki/Rhubarb_Triangle](https://en.wikipedia.org/wiki/Rhubarb_Triangle)

<comment><USER4> I wasn't aware of it being British either. It was just something that grew in the backyard of the house I grew up in, in upstate New York.

<comment><USER5> The UK supplies much of world's Rhubarb, most of it from the so called Rhubarb Triangle in Yorkshire [https://en.m.wikipedia.org/wiki/Rhubarb_Triangle](https://en.m.wikipedia.org/wiki/Rhubarb_Triangle) where it is "forced", it apparently grows so quick you can hear it!

<comment><USER4> I didn't know it needed "supplying" because it just grew year after year by our back porch, with no special encouragement.

Wikipedia says it was known simply as "pie plant" in the US in the 19th century.

Although strawberry-rhubarb pie is a common combination, I recall pies of my childhood being rhubarb alone.

</comment></comment></comment></comment></comment></comment><comment><USER6> I question the knowledge behind this map. The text states it depicts the state of affairs before "the rise of Mongols", but one can find on it Sarai city which was established by Mongols, and, for example, Arkhangelsk which didn't exist at those times either. Also some names are misspelled. Significant trade routes are omitted. No sources cited. Look too amateurish to me to take it seriously.

Update: there is a list if sources on the picture itself, I tried to look for it in the text at first. However, it doesn't save the work from being evidently incorrect.

<comment><USER7> Hello, i'm the author of the map. Sarai has been suggested to have existed before the mongols arrived, and was a strategic location on the route between europe and asia before the height of the "north" northern silk road.

The reason for why Arkhangelsk is there has been explained by earlier replies.

Names are misspelled. Sometimes it was my fingers that moved too fast, and sometimes i've experienced sources spelling names differently.

If i have omitted significant trade routes i would be happy to receive some notice.

This is a continuous passion project of mine, and i'm happy to improve it. I have already gotten a lot of good feedback, all from cities being placed in the wrong order to misspellings.

The purpose of the map is to make it interesting and engaging for people to look at. It's also meant as a road map to navigate the trade/communication networks of the old world that you otherwise could only get from overly general maps. It's a mammoth task, and i appreciate all help i can get.

<comment><USER6> Hi Martin! I respect your effort, and welcome it, because I'm a lover, and collector of maps, actually. You certainly made a lot of work. However (and please don't take it personally) if it's not just a decoration, and you claim it has an educational value, then, sorry, it falls short of even most relaxed standards of accuracy. Let's imagine there was a native settlement in place of modern Washington D.C. Do you really believe it would be correct then to have Washington on a map, which depicts pre-Columbian America? There was no Arkhangelsk, there was no Sarai. There's quite an established (and supported by chronicles) understanding that the former was founded by wish of czar Ivan The Terrible, and latter by order of khan Batu. Both possibly were predated by earlier smaller settlements (although there's no sound evidence for this in relation to Sarai), but, at first, they weren't major trading centers (and that's why you don't know their names), and second, you cannot freely use names from the future without making your map a historic mess, having essentially anti-educational effect. So, it's a noble attempt, but as you said it's a mammoth work, and one have to chew a whole library of historical literature, and look at numerous historical atlases to make it good. Maybe, it's just a chunk too big to digest quickly. Maybe, it would be better to try to start from a smaller part of the world. Btw, crowdsourcing doesn't work in such cases well, because you would still need to verify suggestions, and chose between them. Also because maps are by their nature very politically loaded. Even a question of proper spelling can ignite serious conflicts, and offend a lot of people. I would like to encourage you to continue map making, but I suggest to start from a less ambitious project, and extend it gradually.

And thank you for your calm response!

<comment><USER7> I understand that errors like this can make one doubt the whole product. The content of the map is not made up and very well researched. The fact that i chose to name the settlements like i did (didn't have to be major, but notable or strategic) does not erase its educational value. i causght myself actually defending the location of sarai-juk, and not sarai. You are completely correct in the instance of Sarai. This will have to be corrected in future editions.

Instead of putting the settlements in quotes i should rather find another way of describing their lack of known name. There were trade links and settlements that were involved in international trade but that remains unnamed, simply referring to areas or islands. I wanted to depict these links as well.

The objective with the map is to not start from a smaller part of the world, but to remedy the fact that the maps out there are either local and detailed, or global and general. I want to strike a balance between the two, and the only way i can do that is to publish and republish my map for constant reviews, which is why i appreciate your effort of correcting the map.

This is a side passion project of mine that has been revised constantly during 2 years, there are other versions of this map out there significantly less accurate. And i plan to keep updating this project, because i think its usefulness definitely outweighs the drawbacks of small errors. Spelling errors are however a big drawback, and as you say, will make the map look less reliable.

You said earlier that i missed some major trade routes, do you have any of these in mind that i can work with?

I don't know the guy who put this map up here for crowdfunding. I was just told by someone that it was up here. I just like to track down wherever this map goes to see who comments on it in order for me to improve it. I give this map out for free to anyone who asks and have no commercial interest.

Thank you again for your response.

</comment></comment></comment></comment><comment> aaron-lebo Not sure whether it originated there, but this kind of stuff is pretty common on the mapporn subreddit. The author posted and commented on it:

[https://www.reddit.com/r/MapPorn/comments/8kf22u/new_version...](https://www.reddit.com/r/MapPorn/comments/8kf22u/new_version_medieval_trade_networks_of/)

It's probably one of the few places on the net that exists almost purely for curiosity and isn't full of inane arguments. Well worth checking out. You can spend hours on it. Hopefully exposure to stuff like this helps people realize how interconnected we all are and have been.

<comment><USER8> I once read a memorable analogy for European explorers "discovering" Africa:

It's like (today or 18th century, does not matter) some African dude showing up in the middle of a busy European city center yelling "I discovered Europe!"

I fear I've mangled the analogy, in the original it also was pretty clear what was meant, of course the explorer discovered it from a European perspective. The original storyline included the self-important and presumptuous attitude of the Europeans that made it seem like there was nothing there of any consequence or sophistication.

Another story was about the African drums [0], ow they actually work, that saying "drums talking" was not a figure of speech, but that that actually was exactly what they were doing! The drumming follows some aspects of how you would actually speak the words (the languages are tonal), _" The pitch of the drum is varied to mimic the tone patterns of speech"_. Since lots of information is lost most words are ambiguous, they compensate by embedding every word in a context, saying (drumming) longer phrases instead of one word for disambiguation. And they could just talk over long distances relaying information faster than anything the Europeans had at the time (like semaphores).

[0] [https://en.wikipedia.org/wiki/Talking_drum](https://en.wikipedia.org/wiki/Talking_drum)

<comment><USER9> >It's like (today or 18th century, does not matter) some African dude showing up in the middle of a busy European city center yelling "I discovered Europe!"

Imagine you self-fund your own voyage to Mars. You descend into a cave and find a city full of humanoid Martians. Do you think it would be inappropriate for you to exclaim, "I discovered Submartia!"?

Go medieval enough, and South Africa is about as remote as Mars.

<comment><USER8> Your comment does not seem to have any connection to what I wrote. Please, read it again. It's kind of inane really. Why do you even bother to reply if you don't even care what I wrote? May I suggest not stopping after the second sentence already?

<comment><USER10> Maybe you can explain what you meant a little bit better, because I thought the exact same thing as the reply when I read your comment.

<comment><USER8> Maybe you could be bothered to read what I wrote, or maybe you need some additional English lessons? Your lack of either education or willingness to contribute constructively it out of my hands.

Hint: Don't stop after the second sentence.

I know I know, reading is hard in the Twitter age. Sorry for writing more than two sentences.

<comment><USER10> It doesn't help that your third sentence starts with "I've completely botched this attempt at communication, so I'm going to go completely off topic for the rest of this comment" which somehow translates into "why can't people understand what I wrote?"

You've come across a great way to start a conversation with 'dang though. The mods here really don't like comments like this, trust me. I've been on your side often enough. I would tone it down.

<comment><USER8> As I already said, if you have trouble with reading I cannot help you. This is something you have to manage on your own. If a few sentences really are such a problem for you, I think you can ask around, adult reading lessons should be available.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> The lack of records for the Southern areas of Africa shows how unbalanced our historiographical studies have been for so long, and how much documentation we must have lost in our colonization efforts. I refuse to believe a huge continent with relatively large populations might have had _no significant trade whatsoever_ over half its territory, less than a thousand years ago.

And of course, the same is true for South America - which was obviously disconnected from other continents, but still hardly a desert.

<comment><USER5> Very few cultures were lucky enough to produce records on a medium which would outlast them. To be fair, it's not something our modern civilization gives much thought either

<comment><USER12> I sometimes imagine a thousand years from now where people are trying to figure out how to use USB-A flash drives and fry their contents since the voltage is wrong, after trying 5 times to get the correct orientation in the port.

</comment></comment></comment><comment><USER13> Added to my Maps of the World image folder.

[https://www.dropbox.com/sh/eftu4fak6efngfr/AAAWuj- oJ5AmOn_O7...](https://www.dropbox.com/sh/eftu4fak6efngfr/AAAWuj- oJ5AmOn_O7M7971Oda?dl=0)

<comment><USER14> Would be awesome if it wasn't littered with memes.

</comment></comment><comment><USER15> I wonder why there are not any trade routes in Portugal which was also under the peninsula caliphate during those dates.

Perhaps was it what pushed them into their huge head start into the age of sail ?

<comment><USER16> it is suggested that the Ottoman capture of Constantinople was instrumental in Christian countries seeking new routes as it made the capital and the Black Sea more of a tricky proposition. So if this is 1100-1200 we're still a couple of hundreds years off that event.

<comment><USER17> It's a question of product, too. Countries only got seriously into the trading game when they had something valuable to offer in return. Portugal's early contribution to the trade was slaves, captured from the west coast of sub- Saharan Africa. They were the first Europeans to really explore south of the Sahara. Pushing ever-farther south for new targets led to rounding the Cape of Good Hope, and opening a new route to India - bypassing the tariffs of the Constantinople and Alexandria routes.

</comment></comment></comment><comment><USER18> It is Toulouse[0], not Tolouse.

[0] [https://en.wikipedia.org/wiki/Toulouse](https://en.wikipedia.org/wiki/Toulouse)

<comment><USER19> Many of the spellings seem to be historical and/or purposely antiquated.

<comment><USER20> The occitan/latin name is Tolosa. I never saw other intermediate form.

</comment></comment></comment><comment><USER21> One correction: missing the massive London -> Calais wool trade that drove most of the English economy:

[https://en.wikipedia.org/wiki/The_Staple](https://en.wikipedia.org/wiki/The_Staple)

[https://en.wikipedia.org/wiki/Merchants_of_the_Staple](https://en.wikipedia.org/wiki/Merchants_of_the_Staple)

Calais isn't even featured in the map.

<comment><USER22> The link is there, but labelled as Dover-"Saint-Omer". Which is a little odd as that's inland behind Calais and Dunkirk.

You can still find "wool churches" from this period, although assigning a single date to a church that was assembled over centuries is a futile exercise.

<comment><USER7> Saint-Omer was heavily active in the wool and clothing trade at the end of the 12th century, it also was situated closer to the coast at this time. Sometimes, sadly, i have had to omit some locations in order to not clutt the map too much. Maybe i should look over the Calais region again. The port from France to England was probably in the Wissant region, as the Via Francigena went through here. I did not have a source of what city/port existed here at the time, some sources cites Sombre as an existing port. Calais existed during this time frame, but the extent to which it was used have been hard for me to establish. Saint-Omer was the only trading settlement i really had any good claims putting on the map.

<comment><USER21> Calais was a massive trading center during the English occupation, 1350 onward. It wasn't necessarily the port itself, but where most of the trade took place.

But if you're looking at before that, then Calais was less relevant.

</comment></comment></comment></comment><comment><USER23> If you are ever in Lubeck, there is a most interesting museum dedicated to the Hanseatic League.

</comment><comment><USER24> This is an interesting map, but it seems implausible that there is no route between Cirencester and Gloucester.

</comment><comment><USER25> See this in IPFS: [https://ipfs.io/ipfs/QmTf4EBfik9iH7w2DeW3pZyiqMfnxqtZ3jL5S4t...](https://ipfs.io/ipfs/QmTf4EBfik9iH7w2DeW3pZyiqMfnxqtZ3jL5S4tMXRxzaN/medieval- trade-networks.jpg)</comment>


Was Cantor Wrong? Are the real numbers countable? - <AUTHOR> https://medium.com/p/a30802cf3152 <comment><USER2> Cantor is wrong, yes, but you won't prove that to contemporary mathematicians who believe his ideas because this is an epistemological issue and can't be resolve with mathematical arguments.

This is really an age-old philosophic dispute that the mystics won decades ago. As David Hubert described it "No one shall expel us from the Paradise that Cantor has created." It is the "paradise" of deuces wild for the mathematicians and it rests on an equivocation of the meaning of "infinity". Cantor's idea of a "completed infinity" is a self contradiction if you grasp what the concept of infinity actually means and keep it tied to reality. It is the error of treating infinity as a real thing and not an abstraction. A similar error, for similar reasons, is made in the history of the philosophy by the mystics of nihil who wanted to treat nothingness as on par with existence via the Reification of Zero.

<comment><USER3> You out yourself by using the phrase "Reification of Zero," which is something I've only ever heard from Objectivists. It's funny to hear you say this, since Objectivists are usually so apoplectic about axioms and premises.

You can object to ZF(C) as the basis for set theory, but I don't see how one can call it "wrong." If you want to build up a theory without the Axiom of Infinity you're free to do so — many mathematicians have. Either Cantor's arguments follow from ZF(C) or they don't (hint: they do).

It's not a philosophical problem at all. One can certainly explore what it would mean to do mathematics without access to "infinity", as many mathematicians in the first half of the 20th century did. IMO this conversation is all air until you propose your own precise set of axioms.

<comment><USER2> > It's not a philosophical problem at all....

Well, that was my main point. You are free to disagree.

> IMO this conversation is all air until you propose your own precise set of > axioms.

Apoplectically, the three basic axioms are Existence, Identity and Consciousness. Whatever the proper mathematical axioms (and methods) are they have to be consistent with these three. Cantor's method treats infinity as a real thing, which violates the axiom of identity. You could even call it the Reification of Infinity but the error is easier to see in Rand's identification of Reification of Zero by the mystics who attempt to treat nothingness as real.

<comment><USER3> The three basic (metaphysical) axioms according to Objectivists are Existence, Identity and Consciousness. Anyhow, as far as I can tell, all you're saying is that either (1) Cantor's argument is invalid within ZF(C) or (2) you find ZF(C) objectionable on the grounds that it conflicts with the basis of your metaphysics.

That's fine — I don't care whether it does or not — but mathematics has no opinion on the matter. You're free to choose a set of mathematical axioms which you don't find objectionable and go about doing your math in that universe.

If you want to claim Cantor's argument is invalid then you need to tell me what axioms you're taking as the foundation for your mathematics. It is valid in ZF(C) and I'm 99% sure it's valid in plain ZF. Which of the axioms of ZF do you find objectionable and what would you replace them with?

Assuming you're trying to engage sincerely, I'd ask you to be precise. This means listing whatever foundations of mathematics you don't find objectionable and talking about how they differ from ZF(C).

If you're unfamiliar with it, I'd read up on proof theory: [http://en.wikipedia.org/wiki/Proof_theory](http://en.wikipedia.org/wiki/Proof_theory)

</comment></comment></comment></comment><comment><USER4> Your argument is indeed completely wrong.

 _Let me construct a new positive even number as follows. Take the first positive even number, then add the second positive even number, then add the third positive even number, etc._

The set of even number is infinite. And this is a divergent sum, it's infinite, not an even number.

<comment><AUTHOR> Thanks for the response, a few of my friends in Facebook told me the same thing, the constructed number is infinite, so it is not a member of the set and my argument is invalid. Do you have any insight as to why my attempt to show that the reals are countable by placing them in a binary tree is also wrong? We know the rationals are countable by putting them in a grid and mapping them to the natural numbers, how is this fundamentally different from putting the reals in a binary tree?

<comment><USER5> You sound like a computer scientist, so think of it this way. When would your breadth-first search terminate, given an irrational number to locate? Can you even bound it? The answer is no, because there are an infinite number of digits in any irrational number. The breadth-first search would not terminate; the result would diverge toward infinity.

With the rationals in a grid, even ignoring that I can trivially arithmetically compute the corresponding natural (interleave the (finite!) digits of the numerator and denominator), I can give a bound on a similar search (say Z-order): O(n²).

I think the fundamental idea you are missing is that a natural number _must_ have a finite number of digits. (This stems from the construction of naturals as finite objects.) Reals are not subject to this limitation, as evidenced by the existence of the irrationals.

[Actually, that last paragraph begs another question: what are the theoretical implications of the fact that any _specific_ real must have a finite representation of some form? Does the reason we can reason about reals which we can't specifically identify with a finite construction have anything to do with the axiom of choice?]

<comment><AUTHOR> This actually helps a lot with the intuition, thanks! So in a sense every irrational number is an infinite number because there is no way to actually represent it in a finite way.

<comment><USER5> Almost! See pesenti's link about definable numbers: _most_ irrationals have no finite representation, although almost all the ones we are familiar with (e.g. sqrt(2), π, e) do: they can be represented as mathematical formulæ. (And indeed, note that such "definable" reals _are_ countable!)

So in my example above, π would not work if, say, your tree was a parse tree of mathematical formulæ: one _could_ locate the formula for π in finite time. Unfortunately I cannot give an example of a single number which does _not_ have such a finite parse tree, as, by its very nature, I cannot write down a finite description of it!

</comment></comment></comment></comment></comment><comment><USER6> The new string has to be placed in the assumed list of Reals. If you place it anywhere, you have to recast the Reals.

It's not just that it's new by algorithm, but it does not have a deterministic place within the assumed list of Reals.

It's a new "Cantor's Paradise" every time you place that new number. And it does not come into existence by the same procedure used to START generating the list. It's a metanumber -- exception to rule. This is why one has to have a metaphysics of number before one accepts the proof. Why Hilbert et al accept the proof on aesthetic grounds, and a whole class of mathematics on aesthetic grounds. The appropriate response is: That's just another real number.

Naturals, etc START somewhere but you are not forced to reindex with each new number.

<comment><USER6> It's not necessary to believe in all those classes of infinities to see the sui generis nature of the _mathematical gesture_ used to generate the Metanumber.

Between any two mathematicians, likely the two will pluck very different Rules from their imaginations or backgrounds to produce _some other real number._

What is _interesting_ is that we have a new real but also a new list. The lists are contiguous games, not a continuous "paradise."

Number is the market wherein we SHARE rules. If we give up rule-sharing, the market itself has no value nor do the numbers (and then all order is defeated).

</comment></comment><comment><USER7> In your set of positive even numbers example, you are only showing that a specific enumeration scheme does not generate the set of even numbers. A diagonalization argument requires that you show that no such enumeration scheme can exist (not just a particular chosen one). In the typical proof of diagonalization, it does not presuppose any particular scheme of listing the numbers. It just assumes that some arbitrary one exists, then derives a contradiction.

For your binary tree construction, a trancedental real number can be represented by a path of infinite length down the tree. Your argument is that since a breadth first traversal will eventualy exhaust that whole path, that the real number described by it will have been encountered. There are two ways to interpret what you are doing wrong:

\- If we are indexing/pairing these nodes by time steps (an index), your construction is using a countably infinite time step to express the numbers described by an entire path (which defeats the point of being countable).

\- For countable sets, you have to give me an index of finite size. If I give you a real number, you need to return a natural number (or equivalent) that indicates where it is. To test this, give me the index of pi in your claimed "countable" enumeration. The reason you wont be able to somewhat follows from Cantor's diagonalization scheme.

You might want to do some Googling before spending the time to write up an entire blog post about it (god forbid posting it to HN). The mistake you made is very common and has been discussed to death. You would have caught it.

<comment><AUTHOR> Thanks for your reply, the index of finite size is the part where I was confused. I have actually done some googling about this before, and never caught it, because I wasn't exactly sure where I was uneasy. This has bothered me for a while, but reading that post in HN prompted me to think about it more, and I started writing my thoughts in a communicable manner in order to organize my own thinking. I figured publishing it and submitting it to HN would be a very effective way to get immediate feedback. In HN if you are wrong people will gladly let you know quickly. Also putting it in HN helps prompt people to post and argue about it and other ideas that are related, so it's an interesting and fun discussion.

</comment></comment><comment><USER8> Why do programmers assume they are mathematicians? There is a huge difference between those disciplines.

Trying to uphold the HN principles here, with a civil but pointed question. Computer science is deep, so is physics, and mathematics as well but they are distinct and quite different. Steven Hawking fumbled the proof of the infinitude of primes in his excellent survey of mathematical history "God Created the Integers," as usual for physicists he saw his way to the proof without traversing each step.

Infinity is a difficult but knowable concept. Calculus is a good teacher of this, and Euclid is a good teacher of proof. That's what's missing I say: the standard of proof. For physicists and computer scientists that standard depends on the outcome of physical events, but mathematics requires a purity that doesn't exist in the physical world.</comment>


How to write documentation that's actually useful - <AUTHOR> https://insights.hpe.com/articles/how-to-write-documentation-thats-actually-useful-1707.html <comment><USER2> Writing good code comments is a lost art.</comment>