Ask HN: Simple, beginner friendly ETL / Data Engineering project ideas? - <AUTHOR> Hi HN,
I'm a seasoned Python software developer. Recently I have found a new obsession with data processing, management, engineering etc ... I'd like to (eventually) branch off into that field but I find the lack of beginner friendly resources is slowing me down. All I can find is spark, hadoop related articles (I know these are prominent in the field, but I want to learn to walk before I run). So If any of you have pointers, websites, project ideas I can start to get a good grasp of all the fundamental concepts, I'd really appreciate it.
Thanks a lot in advance <comment><USER2> Spark, etc, are great, but honestly if you're just getting started I would forget all about existing tooling that is geared towards people working at 300 person companies and I would read The Data Warehouse ETL Toolkit by Kimball:

[https://www.amazon.com/gp/product/1118530802/](https://www.amazon.com/gp/product/1118530802/)

I learned from the second edition, but I've heard even better things about the third. As you're working through it, create a project with real data and from- scratch re-implement a data warehouse as you go. It doesn't really matter what you tackle, but I personally like ETLing either data gather from web crawling a single site[0] or push in a weekly gathered wikipedia dump. You'll learn many of the foundational reasons for all the tools the industry uses, which will make it very easy for you to get up to speed on them and to make the right choices about when to introduce them. I personally tend to favour tools that have an API or CLI so I can coordinate tasks without needing to click around, but many others like a giant GUI so they can see data flows graphically. Most good tools have at least some measure of both.

[0] Use something like Scrapy for python (or Mechanize for ruby) with CSS selectors and use the extension Inspector Gadget to quickly generate CSS selectors.

<comment><USER3> I own all 3 Kimball books, they are fantastic

<comment><USER4> Probably the best live training I've ever attended for data warehousing.

</comment></comment></comment><comment><USER5> 1) Learn to do as much in plain Python as possible, focus on lazy evaluation (itertools, yielding, ...), you'll be able to process gigabytes with a tiny memory footprint, deployment will be a breeze etc.

2) Get to know some of the basic Python data processing/science packages like pandas, numpy, scipy etc.

3) Get used to writing short shell scripts - they probably won't be a part of your pipeline, but data engineering, especially development, involves a lot of data prep that coreutils will help you with. Throw in a bit of jq and you'll handle a lot of prep work.

4) Only once you've gotten used to the above, look at Dask, PySpark, Airflow etc. It really depends on your use cases, but chances are you won't have to touch these technologies at all.

Bottom line - wait with the heavyweight tools, they might be needlessly powerful. Also, work closely with DevOps, because the deployment side of things will help you understand consequences of your actions.

<comment><USER6> I like this answer. I'd also add:

0) Spend time finding data sets that actually interest you.

If you're working with data that you're actually excited about, the questions will flow naturally. And when the questions flow naturally, there's an obvious path to picking up new techniques as you need them.

<comment><USER7> This was the thing that helped me most. I just can't make myself excited about the price of a house or the size of a tumor. Stats and predictions for my favorite sport, on the other hand, gave me context that naturally made me ask questions regarding the accuracy of my results and how I can improve them.

</comment></comment></comment><comment><USER8> I second the Luigi recommendation. Republic Wireless uses it for all of our data warehouse ETL, and it's been fantastic to work with.

I also second the other comment that recommends starting with basic data extraction rather than diving into Hadoop or Spark immediately. Sure, at some point, you might need to process 100 billion lines of data. But in your average business, you're far more likely to be working with thousands or millions of records on customers, sales, orders, invoices, sales leads, etc. That stuff doesn't need Hadoop/Spark, it needs a Postgres database and a DBA with a good head on their shoulders keeping everything organized.

In my experience, government data sets (particularly demographics and other geographically-related data sets) are a fantastic way to get your feet wet with data processing. They're published by a bunch of different agencies, so they're not necessarily conveniently available in one place. However, they usually use standardized identifiers for geographies, which makes it easy to join the data sets together in new and interesting ways.

For instance, here at Republic, we recently used Form 477 data on wireless broadband availability from the FCC, data from Summary File 1 of the US Census, and a couple of Census geographic crosswalk files to be able to calculate the percentage of population in given zip codes and cities covered by various wireless carriers. That required reading the docs for several different data sources, automating some downloads, building database tables to hold all of the information, and then carefully crafting some SQL to pull it all together.

Of course, government data sets generally won't require a whole lot of automation (they're updated yearly or less than yearly, not daily). To build your skills on that front, I'd recommend learning to extract data from various APIs, structure it in a meaningful way, and make it available in a database. For example, if you have a website, set up a free Google Analytics account for it, then build a daily ETL that extracts some meaningful information from the Google Analytics API and stuffs it in a Postgres DB. Then see if you can build some charts or something that sit on top of that database and report on the information.

<comment><USER9> Luigi is a great Python library for building multi-stage pipelines with parameterized tasks. It easily extend to new storage types and targets.

We currently use it to build a moderately complex product from dozens of data streams (files, APIs and things in-between) and millions of records. At its core is a DAG and topological sort, which capture the essence of pipelines and execution.

A DAG is easy to visualize, even its development over time[1].

[1] [https://imgur.com/a/RuC0B5Y](https://imgur.com/a/RuC0B5Y) (generated from the projects' code, in this case [https://github.com/miku/siskin](https://github.com/miku/siskin))

</comment></comment><comment><USER10> Hi, data engineer here. Other comments have a lot of good suggestions. I especially agree with ideas like avoiding high powered frameworks in the beginning, and learning to write effective transform wrappers for different kinds of weird input data. One thing you'll find is that (for enterprise situations at least) your source data is going to come from extremely odd, old fashioned, or very poorly documented sources. Be prepared to find twenty and thirty year old manuals on formats you've never heard of at times.

That aside, the other side of the coin that is very important is to get very familiar with how folks talk about their data problems. Managers, analysts, etc. will often request a specific solution that they've heard of or seems popular--sometimes it is what they need, often times it isn't. To get a solid footing in this space (not that it isn't for all types of SE) it is critical to have very strong understanding of business requirements, understanding the work of many other roles in your org, and being able to communicate with business folks in ways that allow you to develop a rational plan for a solution while getting them to realize what their needs really are.

Best of luck!

<comment><AUTHOR> Thanks for your insight !

</comment></comment><comment><USER11> Another commenter mentioned it as well, but "Designing Data-Intensive Applications" by Martin Kleppmann [https://dataintensive.net/](https://dataintensive.net/) is a _fantastic_ overview of the field and, I think, more approachable and enjoyable to read than Kimball's book. But Kimball is a classic, especially for how to do warehouse design.

I'll also make a plug for the Meltano[0] project that my colleagues are working on. The idea is to have a simple tool for extraction, loading, transformation, and analysis from common business operations sources (Salesforce, Zendesk, Netsuite, etc.). It's all open source and we're tackling many of the problems you're interested in. Definitely poke around the codebase and feel free to ping me or make an issue / ask questions.

[0] [https://gitlab.com/meltano/meltano/](https://gitlab.com/meltano/meltano/)

</comment><comment><USER12> I worked in that field a number of years. My recommendation to you is to start with some form of data that you are passionate about. Baseball statistics, business metrics, investment figures, whatever.

Once you have the data, then figure what you're going to do with it. (Don't agonize over it, this should all take just a day or so.)

Then go after the toolkit. You'll find many interesting questions if you start with the end goal in mind.

Good luck, and have fun!

<comment><USER13> This is the best advice I've seen here. The purpose of data engineering is to support a much larger effort related to scientific discovery or increasing bottom line revenue. Don't start with the language, tooling, libs and architecture, start with a purpose and goal. Most goals relate to mimicking a portion of human cognition in terms of pattern matching.

If you're building a house you don't start by picking out hammers and saws.

This takes being passionate about your data and its pre and postprocessing as a data engineer.

</comment></comment><comment><USER14> [http://Singer.io](http://Singer.io) is an open source ETL project written in Python. The components are small, composable programs that you can run independently, so you should be able to walk before your run.

A good beginner project is to build an adapter to a new data source (known as a "tap" in Singer). Most taps pull data out of business tools like Salesforce or Marketo, but people also build them to pull characters from the Marvel API ([https://www.stitchdata.com/blog/tapping-marvel- api/](https://www.stitchdata.com/blog/tapping-marvel-api/))

Check out the getting started guide ([https://github.com/singer-io/getting- started](https://github.com/singer-io/getting-started)), or jump into the Singer Slack and ask for help (linked from the guide)

<comment><USER15> Fantastic reference! Thanks for sharing. As an ETL specialist I've looked for a composable solution but only found Luigi. I wanted something simpler, more integrated, like IFTTT. This has the feel of Yahoo Pipes. I think Singer is the answer. ;-)

<comment><USER16> Composableanalytics.com might be the tool you are looking for then....

</comment></comment></comment><comment><USER17> I would recommend looking into Python-based workflow managers: Luigi[0] then Airflow[1], to get a hang of scheduling, DAGs, etc. Both are fairly simple to get started with (especially for a seasoned Python developer) and are used in production environments as well.

[0] [https://github.com/spotify/luigi](https://github.com/spotify/luigi)

[1] [https://github.com/apache/incubator- airflow](https://github.com/apache/incubator-airflow)

<comment><USER18> Only just look - don't actually try to use one of these on a side project unless you want to waste a LOT of time. They're amazing for large projects moving 100GB+ a day and running dozens or hundreds of different jobs to orchestrate a moving system, but are a total time sink for anything small.

<comment><USER19> 100 GB per day is pretty small, but your point holds

</comment></comment></comment><comment><USER20> I wrote a simple blog post recently with links to data engineering resources I found useful to get into the field, hopefully it is helpful: [https://diogoalexandrefranco.github.io/data-engineering- reso...](https://diogoalexandrefranco.github.io/data-engineering-resources/)

<comment><AUTHOR> Brilliant post, currently looking at the SQL section of leetcode, thanks a lot my friend. Very much appreciated.

</comment></comment><comment><USER21> I've always found enjoyment in finding statistics in things that interested me, so here's a few thoughts on project ideas. The main thing is to find a data set that interests you, and use that interest for fun or for profit:

\- pull down your facebook backup, run it through sentiment analysis, throw it in a db, explore yourself through facebook's eyes! \- a while back I was looking to buy a car and used scrapy to process cars.com, ran their vins through a look up tool to find cars that were _actually_ manuals (and not just manumatics, seems few can tell the difference these days). Found reasonable national prices, average miles, etc. \- interested in politics? pull down the data sets for voting records, explore your local politicians' voting records, compare them to national averages or historical information, etc. \- interested in movies? find movie datasets (or scrape imdb.com, themoviedb.org, etc) and find which genres pay the least per actor, have the smallest cast, etc.

Lots of datasets are available online, if not in machine readable format, then in a format that can be easily scraped. Have fun!

</comment><comment><USER22> If you want to make it as a career choice, I think you should start with learning Java and Scala. For better or worse, this field is tied to the JVM and learning these languages will make picking up Spark and Hadoop (which tbh is a prereq for any Data Engineering position to have on their resume) a lot easier.

Also if you are looking to stay in the Python world, PySpark is pretty intuitive for any Python developer and tons of companies are using it.

</comment><comment><USER23> Build a web scrapper and save some raw data in somewhere like S3. Then run some job on top of that data to get aggregated measures and save them somewhere. I built a project[1] like this and learned a lot in the process. I used airflow to run the scrapping tasks, save the data in S3, use AWS Athena to run queries and load data into Redshift. I did all of this just to learn more about Airflow and some AWS tools.

[1] [https://skills.technology/data-engineer](https://skills.technology/data- engineer)

</comment><comment><USER24> I find ETL best practices with Airflow a good start. Even if you don't go the Airflow route, you can benefit from the example implementations of Kimball and Data Vault ETLs.

[https://gtoonstra.github.io/etl-with- airflow/index.html](https://gtoonstra.github.io/etl-with-airflow/index.html)

[https://github.com/gtoonstra/etl-with- airflow](https://github.com/gtoonstra/etl-with-airflow)

</comment><comment><USER25> I have put together a (very short) post to build a dashboard using shopify data and pulling ETL with Stich. You could try to implement the stich part in Python and would have a complete solution. Like looking at Stich/Blendo would give you some ideas of simple ETL workflows. Keep in mind that ETL changes depending on what you want to do. In theory you can do all ETL just with Python code. If you have an SQL compliant database that can hold your data ETL processes could simply be a matter of running SQL queries to transform data within the database. Then you basically load your data from any data source into your db and you run a bunch of SQL to do the transform part.

[https://assemblinganalytics.com/post/airbnbapache- superset-a...](https://assemblinganalytics.com/post/airbnbapache-superset-and- shopify/)

</comment><comment><USER26> Software wise you could also have a look at dask. Which is more lightweight than Hadoop and spark.

But since you are asking for a project, why not do something local. I.e. you could scrape some data for some time (cinemas, crime) and structure it nicely. After a few months you can start analysis. Bonus points if you make you data available.

</comment><comment><USER27> Like other I would say start with a problem which interests you.

To mimic the real world try to have extraction from variety of data sources (rdbms, NoSQL, files (different formats csv/json), APIs (salesforce)). Once you have different sources, extract the data to your datalake built on S3/GCS/HDFS. Once the data is present you need to integrate with tools which can extract value. You can use Vendor specific tools like Athena/BigQuery or open source like Presto/Impala/Hive. You can do analytics where you require filtering, cleansing, joining various datasets. You can also look at storing the results in different formats so that other tools like Tableau can use them. To orchestrate all of this you can use Azkaban or Airflow.

My suggestion is slightly biased towards Hadoop ecosystem but the good thing is that most tools here have Open source alternatives.

</comment><comment><USER28> I think you can easily learn to walk also with Spark. There are a lot of beginners Spark tutorial online. You can see, for instance, [https://community.cloud.databricks.com](https://community.cloud.databricks.com). They give you a Spark cluster where you can start and there are several tutorials in a notebook-style. Check it out. Of course, you could start your ETL / Data Engineering in a more "traditional" way trying to learn about relational databases and the likes. But I would suggest you to start directly with Spark. You can use it with lots of other big data tools (such as Hadoop/Hive and also S3) and you could also find some interesting Machine Learning use case.

<comment><AUTHOR> Bookmarked, thanks !

</comment></comment><comment><USER29> This list has numerous pointers to resources related to data engineering: [https://github.com/igorbarinov/awesome-data- engineering](https://github.com/igorbarinov/awesome-data-engineering)

</comment><comment><USER30> It may not be exactly what you're looking for but perhaps to get started without having to set up all the infrastructure you could use something like AWS Glue. There's some tutorials / examples in the official AWS docs: [https://docs.aws.amazon.com/glue/latest/dg/aws-glue- programm...](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming- python-samples-legislators.html)

Disclosure: I work on AWS Glue. Note: We're Hiring! Email me at: zaheeram <at> amazon dot com if you're interested in this space!

</comment><comment><USER31> Do Data Engineering jobs require you to have knowledge of statistics, machine learning, etc? Or can you get data engineering jobs where you are focused on connecting pipelines and data flow? I am interested in a data engineering job, but it appears a lot of data engineering jobs require machine learning expertise in job listings. I am interested in just a data engineering focused role.

<comment><USER32> Look for functional programming data engineering jobs e.g. Haskell, they might be what you're looking for, since most people don't run their algorithms in Haskell (some do!) and use it as a duct tape/pipe.

</comment></comment><comment><USER33> Many posts here are focused on classic ETL. I'm working on a small project for handling data just after ETL.

It's for dealing with annoyingly large data, bigger than RAM but sitting on a personal PC. It basically performs sampling and munging for this data. There's no good solution for this right now (I know because I've been looking for more than a year).

What might be interesting to you is that there's little abstraction in the project, but it's non-trivial to execute. To me, this makes it fun. Despite its simplicity, it has high utility and could be used by others. This would be a great outcome for an initial project.

I've got a working version of it, but it would benefit from the eye of a seasoned python dev.

Maybe it would be interesting to you to get in touch? My email is: email@example.com

Cheers,

</comment><comment><USER34> I think basic concepts of datawarehousing like creating data marts, building star or snowflake schemas, dimensional modelling, slowly changing dimensions are quite important before you jump into why hadoop, hive, hbase or spark is relevant.

</comment><comment><USER35> Spark is not too tricky to dive into, even though you can't really take advantage unless you have a big cluster to use :)

if you want to practice data-manipulation, and a lot of the map reduce type stuff you can do with spark, I find Pandas useful for small datasets (And a lot of overlap in functionality as far as Dataframes are concerned)

For pipeline stuff, definitely take a look at Luigi, but again without a cluster it'll be less fun. Still, if you can try automating tasks with a mini luigi scheduler on your localhost, it would be good practice

</comment><comment><USER36> Google Cloud Composer, built on top of Airflow (mentioned by a number of people here) has a great Getting Started Guide to check out: [https://cloud.google.com/composer/docs/quickstart](https://cloud.google.com/composer/docs/quickstart)

You can dive right in to Airflow here: [https://airflow.apache.org/tutorial.html](https://airflow.apache.org/tutorial.html)

</comment><comment><USER37> What are the industries that have the highest costs for ETL/Data Engineering?

What companies in these industries are interested in reducing their costs for this work?

"costs" as used above includes time expenditures as well as spending money

[http://web.archive.org/web/19991023120316/http://www.dbmsmag...](http://web.archive.org/web/19991023120316/http://www.dbmsmag.com:80/9509d05.html)

</comment><comment><USER38> Hey Zabana!

I noticed you were in the Europe area. We are looking for developers with interest in going into the field.

We produce a data pipeling/analytics platform for complex data to our customers.

If you would like to know more about our company! Position is in Stockholm. [https://gist.github.com/eleijonmarck/1b384480aaa3d22ab7e6ea0...](https://gist.github.com/eleijonmarck/1b384480aaa3d22ab7e6ea036d297d10)

</comment><comment><USER39> You might want to take a look at [lnav]([https://lnav.org](https://lnav.org)), a sort of mini-ETL CLI powertool. Performance is fine up to a few million rows.

Edit: of course this comment (like many others below) pertains to tooling vs a particular project per se. FWIW I agree w/ others' sentiment about doing things "by hand" and working with data that holds your interest.

</comment><comment><USER40> While still alpha right now, Meltano is a project from GitLab. We are looking for contributors who have a passion for bringing the excellent ideas of software development to the data science world. [https://gitlab.com/meltano/meltano/](https://gitlab.com/meltano/meltano/). Feel free to post some issues and give it a try.

</comment><comment><USER41> an excellent survey of the field: [https://dataintensive.net/](https://dataintensive.net/)

<comment><USER11> I cannot recommend this book enough! I thoroughly enjoyed it and really appreciated the clear definitions and strong opinions.

<comment><USER42> Agreed. This book will give a fantastic way to think about all these.

</comment></comment></comment><comment><USER43> You could check out [https://github.com/mara/mara-example- project](https://github.com/mara/mara-example-project)

The project is just a demonstrator for the Mara framework but it gives a good overview and you could take it as a starter for something you want to build.

</comment><comment><USER44> if you have any interest in baseball, [https://www.retrosheet.org/](https://www.retrosheet.org/) might be of interest. They have play-by-play accounts for every Major League baseball game going back to 1914.

this strikes me as a good testbed for personal projects, because:

\- it's enough data to be inconvenient, but can still definitely fit on a single machine

\- it comes in a weird format that is sort of a pain to process, which is good practice

\- just loading each event into a database won't be enough, you'll have to transform and reorganize it further to support answering interesting questions

\- last I checked, there are undocumented but public APIs for more recent MLB games (on some MLB website), so automatically scraping those and incorporating them into the retrosheet data is another interesting challenge.

</comment><comment><AUTHOR> OP Here, Thanks for all the great links, suggestions and ideas. One question I forgot to ask is the following: What exactly does Transform mean ? It seems to me that it's a very volatile concept and can change depending on the project. I'd love to know your thoughts on this. Cheers.

<comment><USER45> I think it's as vague as it sounds. But the stereotype is probably something like translating rows to columns; or normalizing a column; or creating a composite column

</comment></comment><comment><USER46> Web scraping is a good project. LinuxAcademy has some good content on AWS and Hadoop ($50/month).

</comment><comment><USER47> Hi I have been working in this field (and some of the previous incarnations of it) for quite a while. I would be happy to have a discussion to give some pointers and relay some experience if you want.

My email and keybase.io is in my profile so feel free to get in touch.

</comment><comment><USER48> Luigi. We literally use this for anything requiring a series of more than a few steps:

[https://github.com/spotify/luigi](https://github.com/spotify/luigi)

</comment><comment><USER49> SQL. Most of my ETL jobs consist of setting up the overhead pieces in pyspark then using SQL for the most important logic. It's way more portable that way.

<comment><USER50> What I like about using SQL is I can prototype with SQLite, Postgres, etc and then adapt and move to Hive, Impala, BigQuery, etc. SQL is literally everywhere, and I don't have to learn to paradigms and frameworks.

</comment></comment><comment><USER51> Convert .mat files to json,csv,xls formats</comment>


Rotate the world - <AUTHOR> http://www.jasondavies.com/maps/rotate/ <comment><USER2> This is just my personal preference, but I believe the naive implementation is miles ahead of the improved ones.

The improved version needs at least two movements to go to the other side of the world, doesn't follow the rotation if the cursor is too fast and worse than all, performs a gamma rotation for each non-equatorial point of the sphere.

Since we have two degrees of freedom on the screen, I expect to use them to drive the two most used rotations, and maybe have another control for the roll. In addition, we are used to have maps with North pointing up (and sometimes pointing South), so the roll is almost never used.

<comment><USER3> Thanks for the feedback!

I agree it's not the most intuitive approach for all situations, particularly when you're used to a North-South orientation for maps. The rolling is perhaps made more obvious with my use of a graticule. For this reason, I'd be interested in optionally restricting to two degrees of freedom when dragging on the globe, and the third angle could be modified when dragging outside the globe only. So the point being dragged on the globe could still remain under the cursor where possible using this approach.

Interestingly, Google Earth uses the same approach as I do, though perhaps it seems less obvious without a graticule.

<comment><USER2> I don't like Google Earth's approach too much either, I always have to correct back the roll..

I can understand it when you reach the zoom level of a city, at that point you correlate the data on the map with actual visual memories at the street level, where Up does not correspond to North anymore, but instead is related to Forward. Maybe an exponential transition between the two systems as you zoom in?

Interestingly, on a phone it is also trickier because there is not "grab" equivalent and the y axis also makes the page scroll.

Sorry, I don't want to sound overly critic :)

</comment></comment></comment><comment><USER4> Slightly off topic, but I was so surprised how steep the Axial tilt is ([http://en.wikipedia.org/wiki/Axial_tilt](http://en.wikipedia.org/wiki/Axial_tilt)) I had always known that it was 23º but never actually visualized it.

Try putting γ=23º in the Euler angles and rotate the earth with λ.

</comment><comment><USER5> Firefox you're breaking my heart. This performs terribly in FF, fine in Chrome.

<comment><USER6> It could be that his logic optimizes for chrome. I'm sure with a bit more time, he could make it work smoothly in FF... Or fuck it, just write it in ASM, which would be insanely faster.

<comment><USER7> The issue isn't any sort of optimization for chrome, FF just chokes when it tries to render SVGs quickly. See also: [http://roadtolarissa.com/twisters/](http://roadtolarissa.com/twisters/)

</comment></comment></comment><comment><USER8> Isn't this also called "Arcball rotation"? The biggest advantage over the naïve implementation is no un-intuitive "sideways" rotation after a few sequences of rotation.

<comment><USER9> The important thing in this demo is that his code reduces to arcball-like rotation in the globe projection, but works on arbitrary map projections. Notice how the interaction works for Eisenlohr's projection in the same way as the globe, but the behavior must be different, of course, since the projections are so unlike each other.

</comment></comment><comment><USER10> This is like 100 times faster in Chromium than in Firefox (on Linux). Any idea why? Is JS in FF really that much slower?

</comment><comment><USER11> As an experiment I used this D3 geo stuff to visualize my last year or so across the globe:

[http://dalyons.github.io/where-is- dave/index.html?use_cached...](http://dalyons.github.io/where-is- dave/index.html?use_cached=true&ga=true)

Works much better on chrome than FF unfortunately.

I thought it turned out pretty cool, as a practical example of what you can use this stuff for. It was super fun to build, D3 is pretty awesome.

You can also load your own photos from FB if you want to see your own track, its just static html/js, i dont store anything anywhere.

<comment><USER12> Cool. This is pretty much exactly what I've been thinking of doing. The path, pictures and map - more or less spot on.

</comment></comment><comment><USER13> Another way of doing this (without versors):

Let u and v be the unit vectors of the initial and final click location. We want an R s.t. Ru = v. Create an orthonormal basis ON(u) and ON(v) (you can use the QR decomposition for the vector concatenated with I).

R = ON(v) ON(u)^-1 = ON(v) ON(u)^T

</comment><comment><USER14> I've been writing accelerometer coordinate system rotation code for the whole day. Almost melted my brain; trying to imagine 3D rotation is hard, especially when you lack the proper tools and have very little experience with it. Finally got the thing working; decided to have one last look at HN before going to bed. It was then when I saw "Rotate the world" and thought to myself: "well this better not be about Euler angles!"

Yup. Thanks pjan, you made my day brighter :)

</comment><comment><USER15> India is distorted - Kashmir and all. You better be careful about the borders that are under dispute. I took into consideration that it's a low-res border, even then it's way off.

I dont know if the details of map itself are important or not, because I understand it's an exhibition of the interface itself. Just my opinion.

</comment><comment><USER16> Anyone else tried this on Linux? On Mint 15 and Ubuntu 13.04 FF25 is has >1 second lag. Chrome 30 is fine.

<comment><USER4> I think it's firefox not linux.

<comment><USER17> Firefox on Mac is fine (well, more or less; I just filed [https://bugzilla.mozilla.org/show_bug.cgi?id=936472](https://bugzilla.mozilla.org/show_bug.cgi?id=936472) on the slight lag I see there), which suggests the actual issue is cairo's path drawing code, which is not used by Firefox on Mac, but is used on Windows and Linux.

<comment><USER18> Definitely laggier on FF on Mac than Chrome.

<comment><USER17> Right, but not 1s+ lag like people are seeing on Windows/Linux.

</comment></comment></comment></comment></comment><comment><USER19> Very nice. Makes me think of [http://aladin.u-strasbg.fr/AladinLite/app/?survey=P/Mellinge...](http://aladin.u-strasbg.fr/AladinLite/app/?survey=P/Mellinger/color&zoom=160&cooFrame=gal&target=sgr+a*)

</comment><comment><USER20> Please Please Please include "show the shortest path between points A and B".

</comment><comment><USER21> Unless I missed it, where does one get the continent shapes and country outlines?

<comment><USER3> The files used are from Mike Bostock's World Atlas TopoJSON project: [https://github.com/mbostock/world-atlas](https://github.com/mbostock/world- atlas)

They are in turn derived from Natural Earth data: [http://www.naturalearthdata.com/downloads/110m-cultural- vect...](http://www.naturalearthdata.com/downloads/110m-cultural-vectors/)

</comment></comment><comment><USER22> I have a couple of questions, and I'm unfamiliar with the Javascript frameworks so I struggle a bit to find the answers in the source:

Are the globes drawn with perspective?

And can it suffer from gimbal lock?

<comment><USER23> Since it's based around quaternions it shouldn't suffer from gimbal lock.

As for perspective, what do you mean by that? Aren't globes always in perspective?

<comment><USER22> You can draw orthogonally too. I've messed around with globes and had problems spinning them, hence my curiosity.

[http://www.ludumdare.com/compo/minild-40/?action=preview&uid...](http://www.ludumdare.com/compo/minild-40/?action=preview&uid=10313)

</comment></comment></comment><comment><USER24> Wow, that's _exactly_ what I needed yesterday. Thanks a lot.

.. however, some more explanation for how the improved rotation works would have been useful.

</comment><comment><USER25> Are the scale of continents right? Because Australia looks almost bigger than Europe, and Africa looks many times bigger than Europe.

<comment><USER26> The true size of Africa: [http://flowingdata.com/2010/10/18/true-size-of- africa/](http://flowingdata.com/2010/10/18/true-size-of-africa/)

The historian Marshall Hodgson referred to the Mercator projection as the "Jim Crow projection." This is why.

<comment><USER25> I found my mistake. When looking at Europe for some reason I mentally subtracted first the Scandinavia and then Russia, that's why it seemed so tiny. Looking at wiki I can see Africa is only 3 times larger than Europe, which is what I would have expected.

<comment><USER27> Try rotating this :) [http://mxfh.github.io/d3canvasglobes/#GRL](http://mxfh.github.io/d3canvasglobes/#GRL)

</comment></comment></comment></comment><comment><USER28> Brilliant stuff! Would've been better if it had the names of the places. Kinda like a virtual globe. ;)

</comment><comment><USER29> Rotating maxed out my cpu in both chrome and firefox but only chrome didn't stutter.

</comment><comment><USER30> Great work! This is a really nice use of d3 and a great visualization tool.

</comment><comment><USER31> It's more useful when you keep the north at north.

</comment><comment><USER32> Man oh man, do I hate Euler angles...</comment>


Mapcode – A short address for any location on Earth - <AUTHOR> http://www.mapcode.com <comment><USER2> I'm a little confused about what this replaces, but it's definitely _not_ postcodes.

* Postcodes have a specific meaning: they're both a geographical area and a mail delivery zone (at least in the US). Replacing one short code with another, which requires a different and less reliable third-party service to correctly decode, doesn't seem like a good idea.

* Latitudes and longitudes are unambiguous, well-understood, and don't require a third-party service to be online in order to correctly decode. So it doesn't seem like this can replace latitudes/longitudes, unless we're talking about shortening their representation.

* Broadly speaking, while addresses have a lot of problems, they're unambiguous with human context. So this doesn't seem like it really replace addresses either.

* For things like a loose description that might not be associated with a single point (e.g. "Central Park"), I could see mapcodes being useful -- you can provide the precise boundaries of the park, where it's located, associate metadata with the mapcode like what hours the park is open, etc. That might be very helpful.

Also, won't there be multiple mapcodes for a single spot and won't that require cross-association of the respective metadata? For example, consider that Central Park is also in Manhattan, also part of NYC, also in Midtown in NYC, etc.

I guess that's not necessarily a problem, but it could lead to a lot of expensive processing for frequently updated locations/boundaries (e.g. a beach, a suburban development in progress, etc.).

<comment><USER3> Their own website is displaying/using MapCode's incorrectly.

For example, they have 0C.T4 and they say it's in Ireland and appears to be some bench in the picture.

However, if you use that MapCode, you will see it's the middle of some river in Amsterdam.

0C.T4 == 52.369764, 4.868898

Another:

81.J4W They seem to claim it's in Alaska, appears to be some road. However, it's another river in Amsterdam.

81.J4W == 52.508957, 4.769173

<comment><USER4> Actually it's that those MapCodes are incomplete. They're missing the country/state designation. If you look up "Ireland 0C.T4" or "US-AK 0C.T4" you should get the correct results.

I'll cite a .doc file here, it's also available on the Developers page:

> 1\. Codes only need to be accurate enough for public, every-day use. At the > human scale, when you are within a few meters of a destination, you are > "there".

> 4\. People live within a "country context". Addresses seldom include a > country name. Unless clearly stated otherwise, they can safely be assumed to > be in "this" country. Codes that are known to be within a particular > territory can be designed to be much shorter. For example, every location in > The Netherlands can be uniquely specified with at most 6 letters . Even in a > large country like India, at most 7 letters are needed.

> 6\. Short codes are reserved for areas where the population density is high. > For example, every locations in The Netherlands can be represented by a > unique 6-letter mapcode. However, the majority of the population is > concentrated in a dozen urban areas totaling less than 3,000 km2 (1,800 > square miles). By reserving the 5-letter mapcodes for these areas, half the > population of The Netherlands can be found using 5-letter codes. On average, > mapcodes for relevant locations in The Netherlands are thus closer to 5 > letters than to 6 letters. In fact, a significant percentage of the > population can be found in 100 km2 of city centers, which can be covered by > 4-letter mapcodes.

> Based on these ideas, we envisaged a system that would work anywhere on > Earth, and would (on average) provide the shortest possible codes to > everybody. By using massive amounts of data gathered over the past 30 years > by mapping company Tele Atlas (merged into the TomTom Group in 2008), a data > table was constructed ...

> Finally, on top of any and all national mapcodes a location may have, it > also always has a 9-letter, context-less map code.

> To understand mapcode accuracy, you need to understand that a specific > mapcode defines a specific location X, but represents all the locations > within a certain distance meters of X. For example, the Dutch mapcode > "49.4V" decodes into coordinate 52.376514, 4.908542. This does not only mean > that coordinate 52.376514,4.908542 can be represented by mapcode 49.4V, but > that all coordinates within roughly 5 meters of that coordinate can also be > represented by mapcode 49.4V.

(it continues to describe "high-precision mapcodes")

Why did they make it?

> The mapcode system was revived and worked out in 2006 when TomTom decided to > expand their operations to countries like India and China. It was difficult > to sell navigation devices in these countries, among other things because a > significant part of the population lives without address, lacking not just > house numbers but often even street names, which often made it hard or even > impossible to enter a destination. It was thought that the introduction of a > simple and ubiquitous system for representing locations (and hence > destinations) would in due time make navigation devices as welcome there as > anywhere else.

<comment><USER5> > Actually it's that those MapCodes are incomplete. They're missing the > country/state designation. If you look up "Ireland 0C.T4" or "US-AK 0C.T4" > you should get the correct results.

Imagine if a URL resolved to different addresses in each country you were in. Would that be a better system than what we have now, or a worse one? I think it would self-evidently be worse.

I'm surprised they didn't make mapcodes a globally-unique identifier.

<comment><USER6> Globally-unique identifiers make sense on the web, where "going" to a website being served from the other side of the world is as easy as one served from across the street.

That's not true for navigation in the physical world. If you're sitting in Nebraska, how often do you have to navigate to a location in Mongolia? Approximately never! Map codes are optimized for the common case, where you want to go to a location that's nearby.

In fact, it's even more clever than that, since it defines "nearby" as a political unit. Borders may be artificial, but they're real, and crossing them has practical implications. If you're planning to buy something in another jurisdiction, there might be tax implications. You may need your passport to cross the border. You may not speak the language of that area. All of this means that having the political unit as a human-readable part of the map code is useful.

Finally, consider that some URLs _do_ resolve differently depending on where you are. They're called relative URLs. They're very handy and very common. I'd guess that the vast majority of URLs on the web are relative. MapCodes work the same way. The code "Ireland 0C.T4" _is_ globally unique. If you're already in Ireland, you can omit the territory, much as you would omit the country when inviting a friend to your house for dinner.

For cases where you really do have to travel to the other side of the planet, is it really so onerous to specify "Mongolia 0C.t4"?

</comment></comment></comment></comment></comment><comment><USER7> The TomTom founders definitely didn't "just kill the postcode."

> Mapcodes were developed in 2001 by Pieter Geelen and Harold Goddijn, soon > after the GPS satellite signals were opened up for civilian use. It was > decided to donate the mapcode system to the public domain in 2008.

Can we get an update to the click-baity title?

<comment><USER8> GPS was always available for civilian use since day one. Perhaps they meant when GPS's Selective Availability (degregration of accuracy civilians) was turned off, which was in 2000.

</comment></comment><comment><USER9> Honestly, I don't like it. At first I thought it was a short, sort of binary (at least machine-readable) representation of coordinates. A,A might be (0,0) in our coordinate system. Turns out it's not. It's not even global but country (or in the US, state) specific.

This means that the locations are controlled by a single entity together with country codes. Sure, postal codes are controlled by a single entity too, but at least they are regulated by the government using tax money instead of an arbitrary self-appointed committee.

I know it's meant to replace postal codes and not coordinates, but I'd like to see something that actually represents a _place_ instead of an arbitrarily shaped region. OpenStreetMap's homepage happens to do this for their short URLs: [http://osm.org/go/0GFeWn2RJ](http://osm.org/go/0GFeWn2RJ)

Try moving on the OpenStreetMap map with the share menu opened and the short URL option selected. You'll see how it influences the short URL.

But anyway, back to replacing postal codes: the main advantage of this is that it'd be a universal and short code. In The Netherlands you could specify: NL 6114HT 41 and you'd have house number 41 at the Marktstraat (market street) in Susteren (city). Not much longer. And with this new system you'd still need to look up country-specific codes so I don't see much of an advantage.

<comment><USER10> Mapcodes are coordinates within a rectangular region, you define the region, then the code(or if you're IN the region, you don't need to worry about defining it).

One of the available regions is "Earth", which offers global coordinates(they're a bit longer, four or five characters on each side of the dot).

The upside of this is that it is possible to implement it completely offline, you don't need to query the regions from a database, because they're already defined. You just run the algorithm to project the local coordinates onto global coordinates, and your GPS system will handle that.

<comment> grey-area _One of the available regions is "Earth", which offers global coordinates(they're a bit longer, four or five characters on each side of the dot)._

Why not just use lat, lon, which can be shorter too?

Central park, NY - 40.78 -73.97

Hyde Park, LON - 51.51, -0.17

<comment><USER10> Those coordinates are obviously less accurate than the mapcode, and not at all any smaller.

</comment></comment></comment></comment><comment> grey-area Objections:

1\. This mapcode is country specific (!)

2\. Many locations need altitude or fine lat,lon differences to differentiate them (e.g. blocks of flats, offices).

3\. What's wrong with altitude, lat and lon on a given projection which can be as precise/imprecise as you want?

IMO it'd be better to have a hash of lat,lon,altitude which was globally unique for each level of precision.

A system like this needs to be globally unique without specifying country, or you may as well use a postcode. The only advantage over postcodes at present is it's slightly shorter.

<comment><USER11> There exists a hash of lat, long globally unique for each level of precision, a geohash [1]. One common application utilizes a space filling Hilbert curve to define an arbitrary precision bounding box around a set of coordinates. You can see an example hash [2].

[1] [http://en.wikipedia.org/wiki/Geohash](http://en.wikipedia.org/wiki/Geohash) [2] [http://geohash.org/9q8yyvfekk8h](http://geohash.org/9q8yyvfekk8h)

<comment><USER12> If you'll pardon the reckless self-promotion, there's a few Java implementations to do Geohash calculations: [https://code.google.com/p/simplelatlng/](https://code.google.com/p/simplelatlng/)

</comment></comment></comment><comment><USER13> So what's the upside of this vs Geohash?

[http://en.wikipedia.org/wiki/Geohash](http://en.wikipedia.org/wiki/Geohash)

<comment><USER14> patents and ambiguity where you have to also mention the country and sometimes, but just sometimes, the state.

</comment></comment><comment><USER15> > The mapcode foundation reserves the right to change the country list and > population density maps, at which point existing codes will point to > different places or be invalid

Nope.jpg

<comment><USER16> From the docs:

"[This] issue should be prevented at all costs. The second [adding new countries] may be necessitated by changes to the world order, i.e. new countries coming into being. But unless changes are made by a single central authority, the first issue will quickly come to be."

</comment></comment><comment><USER17> I don't like this. From what little I have read, mapcodes appear to be proprietary, despite advertising a "free, open standard".

>Mapcodes are free. They can be used by anyone, and may be supported, provided or generated by anyone, as long as this is done free of charge, conditions or restrictions

Why can't someone charge money for a service that generates mapcodes?

>The mapcode algorithms and data tables may not be altered in any way that would result in the production of different (and thus incompatible) mapcodes. The mapcode algorithms and data tables may not be used in any way to generate a different system that produces codes to represent locations.

I think this means that the source code they are distributing is proprietary, because we cannot use it for any purpose.

<comment><USER6> There's quite a bit of territory between "proprietary" and "free software, as approved by the FSF". If the MapCode sources are free for any purpose except confusing people, I'm fine with that.

</comment></comment><comment><USER18> This is neat, but I am confused why you wouldn't want to just tie location to geography and not country (aka nationality, a squishy and shifting idea). Why not just use UTM?

[https://en.wikipedia.org/wiki/Universal_Transverse_Mercator_...](https://en.wikipedia.org/wiki/Universal_Transverse_Mercator_coordinate_system)

</comment><comment><USER19> I'm curious about what happens when a geographical location changes country. If Catalunya gains independence from Spain, will the MapCode associated with my house here in Barcelona change? Will the old "Spanish" MapCode still be valid?

Speaking of current events, do Isreal and Palestine have separate MapCodes and what happens when borders shift?

</comment><comment><USER20> I, for one, really like this.

* It doesn't use postal codes, which aren't really intended for use for anything but mail delivery and occasionally make that painfully apparent (such as when the boundaries are changed)

* It doesn't depend on street addresses, so it works in countries where the streets aren't reliably or intelligibly named/numbered. It also works in places where there are no streets.

* It's smarter about allocating code space than something like a 10:10 code, since e.g. not a lot of people live in the middle of the ocean (and they can always use a "fully-qualified" MapCode).

* It's super easy to read to someone over the phone, or type in from a business card, or transcribe from a sign.

A couple of uses immediately spring to mind:

* Telling an Uber/Taxi/rideshare where to pick you up when you don't have GPS/data.

* Telling a delivery drone where to deliver your package (without having to correctly remember dozens of digits of lat/long).

* And of course what it was designed for, entering a navigation location quickly.

I'm sure there are some implementation flaws, which may or may not be fatal, but I think it's pretty great.

<comment><USER20> One more use case: When an address doesn't reliably geocode across mapping providers. I've run into this with things like campgrounds or tourist attractions in rural areas in the US and Canada, even with Google Maps but especially Apple Maps in the early days.

</comment></comment><comment><USER21> No, they didn't. They "killed" parts of addresses.

Also, these are still hard to remember. Personally, I'm a fan of [http://what3words.com/](http://what3words.com/)

I'm currently here: [http://w3w.co/lives.fend.spent](http://w3w.co/lives.fend.spent)

<comment><USER22> I like what3words also.

What I'd really like is a hierarchy so that given

>lives.fend.spent

when you saw the words

>lives.fend.rabbi

you knew it was nearby, within the "fend" place or that

>lives.gopher.spoon

was within the "lives" places

<comment><USER14> Sounds like usa.newyork.brooklyn would be a smart extension. Quick, let me patent it and trademark the term "3 or more words that describe a location in hierarchical order". Or maybe I will just call it "backwaddress". :P

</comment></comment></comment><comment><USER23> I started work on something pretty much like this about 5 years ago but never pursued it for various reasons.

I still have one of the registered domains with nothing to show for my preliminary work: www.tinynav.co.uk

I got as far as designing the encoding scheme, including a database of point types (house, utility pole, ancient monument, park, private event etc..), together with special codes for corporates to use for all their shops, fast food outlets etc.

I had the database tables worked out, populated the locations table with open source/public domain data such as civil and commercial airports and some other stuff. There was a front end that allowed for time-limited one-offs for things like 'Jo's birthday party', but then life took over!

<comment><USER24> That sounds really great. I like the time limited parts. Things like concerts or art fairs always much up foursquare locations.

</comment></comment><comment><USER25> " A short address for any location on Earth"

It doesn't check out:

Surface Area of Earth: 510.1 * 10^6 km^2

Distinct map codes: 36^5 = 60.466176 * 10 ^6

Area of 1 map code = 8.4 km^2

Oh! It's narrowed by country... so it doesn't address any location on Earth. Only populated areas.

If you want to _actually_ refer to ANY location on Earth, there's another system, Ham Radio grids. It uses four to six digits and refers to either 70x100 or 3x4 miles squared. Probably useless for the business purposes, but at least it _truthfully_ refers to any location on Earth.

<comment><USER26> It is actually every area, and the length of the codes varies. I'm guessing that the most popular/populated places get the shortest codes and those further away get the longer codes.

For example a location somewhere in the Atlantic off the coast of Gran Canaria is International QFLQV.29KM while some place in the el Poble Sec district in Barcelona is Spain SG.YB.

</comment></comment><comment><USER27> My first instinct was: bad idea.

But then when I thought about it is actualy something that has troubled my quite some times.

Lat and longitudes are perfectly fine but in general they are to "simple" for people to use.

When I think of a location in the world I generally think of the country cause I know to some extent where most country's are located.

For example I would to describe some place in Miami I would describe it as in the lower right side of US and continue to work downwards from that.

</comment><comment><USER28> There is still a special place in my heart for UTM [0].

MapCodes seem to be based on a similar concept (grid, sub grid, sub-subgrid), but with a bit shorter representation. The thing that worries me about using mapcodes, however, is what seem to be the arbitrary borders, order of girds, and the up-keep required for usage. [1]

There is nothing to stop us from creating shorter UTM Codes, though.

Assume 0->33 represented by

        0         0         0         0
        ABCDEFGHJKLMNPQRSTUVWXYZ0123456789

Same as Map Codes [2]

The CN Tower is at

* WGS84: 43°38′33.24″N 79°23′13.7″W, (26 chars)

* UTM: 17N 630084 4833438 (18 chars)

* "Shortened" UTM: T.N.SBB6.DW9F8 (13 chars)

* Map Code: CAN PYLK.XY8P [3] (13 chars)

So I guess the utility of such a complex system is lost on me when something like UTM, that can be decoded with minimal effort, is just as good.

[0]: [http://en.wikipedia.org/wiki/Universal_Transverse_Mercator_c...](http://en.wikipedia.org/wiki/Universal_Transverse_Mercator_coordinate_system)

[1]: [http://www.mapcode.com/mapcode_documentation.doc](http://www.mapcode.com/mapcode_documentation.doc)

[2]: [http://www.mapcode.com/alphabets.html](http://www.mapcode.com/alphabets.html)

[3]: [http://www.mapcode.com/getcoords.html?iso3=112&ifrom=aboutmc](http://www.mapcode.com/getcoords.html?iso3=112&ifrom=aboutmc)

<comment><USER29> If you use CAN-ON it's only 12 chars. But if you already know you're in Ontario it's only 5 including the dot.

So that makes the difference between saying it to someone and having to write it down. Mapcode is massively shorter for local locations which is exactly what sat-nav is for.

The "grids" for mapcode are human-understandable - country and state.

I think a lot of HNers are thinking too much like programmers and forgetting that people need to tell each other locations and write them down. That's where mapcode is good.

</comment></comment><comment><USER30> Having traveled to several countries without formal addresses, this looks like a godsend for navigation. It has no regional segmentation (i.e. no drill-down reference like country-state-zip-city levels), however, so it's useless as a cultural reference.

Slightly off topic, but I'm stupefied at how much faster TomTom's maps (and Bing, apparently!) are than GoogleMaps.

</comment><comment><USER31> I hereby invoke Betteridge's law of headlines.

(unless they can convince every postal service in the world to throw away hundreds of years of dogma)

</comment><comment><USER5> > Areas claimed by two countries are simply included in both contexts.

If I understand this correctly, it means that any address in a disputed territory has multiple mapcodes, one for each claimant, and therefore whichever one you use you are implicitly backing one political claim over the others.

If so, this is a potential can of worms that many people won't want to open.

</comment><comment><USER32> Overall, this looks pretty interesting, assuming your goal is to communicate physical locations with an accuracy of a couple of meters using a format that fits on a business card. It can also be used without country names, if you prefer it that way (just select "International" when generating the code.)

Developer downloads here: [http://www.mapcode.com/downloads.html?iso3=112&mapcode=49.4V](http://www.mapcode.com/downloads.html?iso3=112&mapcode=49.4V)

This is patented, with an ISO standard in progress, and the following licensing terms (from their developer page):

        The mapcode algorithms and data tables may not be
        altered in any way that would result in the production
        of different (and thus incompatible) mapcodes. The
        mapcode algorithms and data tables may not be used in
        any way to generate a different system that produces
        codes to represent locations. In order to prevent
        misuse, unauthorised alterations, copying or commercial
        exploitation, please note that the ideas and algorithms
        behind the mapcode system have been patented and that
        the term "mapcode" is a registered trademark of the
        Stichting Mapcode Foundation.

Part of the motivation is the lack of usable postcodes in many countries:

 _Unfortunately, a large part of the world population has no address. In India alone, well over half a billion people live in houses that have no street name._

They play some interesting tricks with population density and country "contexts" to shorten codes:

 _4\. People live within a "country context". Addresses seldom include a country name. Unless clearly stated otherwise, they can safely be assumed to be in "this" country. Codes that are known to be within a particular territory can be designed to be much shorter._

 _6\. Short codes are reserved for areas where the population density is high._

<comment><USER33> They call it

      ...a free, open way to make every house ...

and then they say

      The mapcode algorithms and data tables may not be
      altered in any way

not very open

</comment></comment><comment><USER34> No, because postcodes aren't primarily geographical references. They just happen to be really useful for that. UK postcodes are arranged so that each one handles a similar amount of mail. This means that a residential street may share one or two postcodes, but a large organisation may have several postcodes to itself.

</comment><comment><USER35> It's hard to get an understanding of the precise algorithm that's being used to generate the code. If it were a competitor of the postcode it would have to be possible to identify whether two places are close to eachother just by looking at the code right?

From the patent I get the feeling that it's a bit like a C-ary space partitioning, where C is 36 (alphabet+numbers). So the first character divides the earth in 36 regions, the second character divides that region in 36 regions, and so on.

With that in mind, I think once people would be a bit familiar with the general numbers for regions it would be pretty easy to identify what general region a mapcode would be in. So it might be a pretty good replacement of postcodes.

Or am I missing some other fundamental property of postcodes?

<comment><USER36> I think that postcodes divide a country not evenly but are chosen by population density and the distribution structure of the post provider. Look at [http://upload.wikimedia.org/wikipedia/commons/8/8e/Karte_Bri...](http://upload.wikimedia.org/wikipedia/commons/8/8e/Karte_Briefzentren_Deutsche_Post_AG.png) for example, each distribution center is responsible for one or two double digit prefixes, the first digit roughly indicates the region. Areas with high population density have a much finer coverage. I don't think there would be any benefit from switching to Map Codes, as they have to be mapped to something like post codes anyways and they are not precise enough to serve as residential addresses.

</comment></comment><comment><USER37> No. They serve entirely different purposes.

<comment><USER38> Yep, if this were feasible as a postcode then we'd have switched to lat/long addresses ages ago.

</comment></comment><comment><USER39> There is already a service like this that uses 3 words instead of that weird string: [http://what3words.com](http://what3words.com)

<comment><USER40> Come find me at premium.gravely.pies.

</comment></comment><comment><USER41> Wouldn't this change the entire address and number? The problem is, you can't find a place without a GPS (or another digital device) :-(

<comment><USER42> You are correct. This is the real problem. (They even use an Indian business card as an example...)

</comment></comment><comment><USER43> The choice of Cyrillic translations seems arbitrary to me. There is a well known Morse code convention (Q=Щ, V=Ж, W=В, X=Ь, Y=Ы).

<comment><USER44> The problem with e.g. W=В or X=Ь is that they want glyph matching; so the roman alphabet B and the cyrillic alphabet В (even though they are different letters) map to each other, to avoid having same-looking codes map to different locations. They're solving a slightly different problem than the Morse convention was trying to solve.

</comment></comment><comment><USER45> Another interesting similar idea - [http://what3words.com/](http://what3words.com/) .

</comment><comment><USER46>
      Mapcodes are a free, open way to make every house or location on Earth
      addressable by a short code. With nothing else except your mapcode,
      for instance, a navigation system will bring someone to within meters
      of your front door.

So what happens if you live in a multi-story block of flats?

<comment><USER47> I think this is to improve on postal codes, not address multiple-person dwellings. The flat/apt/suite number would take care of that.

</comment></comment><comment><USER48> It gave me a map code that was longer than my postcode. I'm not sure how postcodes work everywere else in the world, but in the uk they point to a specific road, so it's pretty good for navigating to a place with.

I do know that in belgium they point to an area, so they're not as good as a navigation aid.

<comment><USER49> I live in quite a big block of maisonettes. At the maximum map zoom level, trying to pin down the exact location of my particular house gave me about three options to choose from –just by moving the cursor a few mm in any direction. So what happens then? Do I pick the mapcode I like best? Does some central body decide which is my 'official' mapcode?

Living in the UK, I don't see what this offers over my postcode, which identifies my location to within a couple of houses. The houses across the street have a different postcode to me as do my neighbours three doors away. Theoretically* all anyone needs to find me is my house number and postcode. Try it. Send me a postcard:

madra 84A <\----- house no. M15 6FD <\----- postcode England

*Actually, it's not theoretical. I have as an experiment in the past sent someone a letter addressed just to their house number and postcode and it got there. In spite of that I do tend to write out fuller addresses though. It just seems a bit too 'minimalist' relying on only the postcode.

</comment></comment><comment><USER50> I'm reminded of [http://xkcd.com/927/](http://xkcd.com/927/).

<comment><USER51> Or the artificial Swatch time, which doesn't solve anything. We already have an international time, Coordinated Universal Time. It only takes to know your offset and some basic addition to convert.

Converting longitude and latitude to some other character base, like someone suggested would be better.

</comment></comment><comment><USER52> It's unfortunate that their map-encoding solution appears to be based on TomTom's dataset, instead of Google Maps. It therefore thinks my house is my neighbor's house.

Open Street Map is, sadly, no better, thinking my house is a different neighbor's house.

<comment><USER53> If OpenStreetMap being wrong really bothers you, you can fix it. I mean to point that out in an abstract sense more than I mean to implore you to do so.

As a rule, address data in OSM is very incomplete and the Nominatim instance on osm.org falls back to things like TIGER data to do geocoding. If you are in a region where there are a lot of addresses present in the OSM data, it would be great if you could at least drop a note nearby stating that the addresses in the area have problems (it's the button shaped like a cartoon word balloon, on the right hand side of the page).

(Housenumbers show up when zoomed in, if they aren't showing, then the OpenStreetMap website is showing you results based on some fallback data.)

<comment><USER54> Bearing in mind that google paid people to sit at a desk adjusting house addresses to point at actual houses correctly before they recently invented their machine learning brain thing

<comment><USER53> Maybe in many places. Or maybe for some uses. Around here, Google Maps has a block level understanding of addresses and just shows a calculated point for a given house number (you can feed in non houses and it shows them down the street from real addresses).

My understanding is that they have acquired building data from many municipalities and will absolutely use that when they have it, not that they have spent a lot of time refining it.

</comment></comment></comment></comment><comment><USER55> > What is a mapcode? [...] They are precise to a few meters, which is good > enough for every-day use.

Yeah, but so are the British post codes. 6-7 symbols and you're down to meters usually. At least for the cities. They definitely bring you close to the front doors.

</comment><comment><USER56> [https://www.waytag.com/](https://www.waytag.com/) is a ZA startup that's been around for a few years. On a cursory glance, this seem quite similar? I don't think WayTag ever gained much traction.

</comment><comment><USER57> I guess today is the day for talking about encoding locations on earth on HN: [https://news.ycombinator.com/item?id=8053032](https://news.ycombinator.com/item?id=8053032)

</comment><comment><USER58> The associated patent application: [http://www.google.com/patents/WO2013057221A1?cl=en](http://www.google.com/patents/WO2013057221A1?cl=en)

<comment><USER5> According to <[http://www.mapcode.com/downloads.html>](http://www.mapcode.com/downloads.html>):

> The mapcode system is a free, open standard.

Which is it, open or patent-encumbered? It can't be both.

</comment></comment><comment><USER59> There are already a dozen of these lat/lon encoding schemes. Please go support/evangelize one of the existing ones before making a fourteenth incarnation.

</comment><comment><USER60> I'll use my mapcode to tell my friends where to meet me. I'll use swatch internet time to tell them when to meet me. It's about as useful.

</comment><comment><USER61> Similar idea: [http://blog.jgc.org/2010/06/1010-code.html](http://blog.jgc.org/2010/06/1010-code.html)

<comment><USER62> Nice idea to include a check digit. Mapcode doesn't have that (though they recommend specifying the country separately). On the other hand, with only 4 characters, Mapcode should be easy enough to remember.

</comment></comment><comment><USER63> ZIP code is also used in astronomy: [http://healpix.sourceforge.net/](http://healpix.sourceforge.net/)

</comment><comment><USER64> Seems very similar to the NAC: [http://www.nacgeo.com/nacsite/](http://www.nacgeo.com/nacsite/)

</comment><comment><USER65> why not use already-established grid systems instead of making a new one? MGRS is commonly used in the US military and does the same thing.

</comment><comment><USER66> Seems similar to Naymit: [http://www.naymit.com/](http://www.naymit.com/)

</comment><comment><USER67> It doesn't replace the post code by any means, but it's definitely a nice way to program a GPS.

</comment><comment><USER68> A lot to do about their map design. But it is quite quick in finding locations.

</comment><comment><USER69> startup based on similar idea ( Based in Hyderabad / India ) [http://zip.pr/](http://zip.pr/)

</comment><comment><USER70> What problem does this solve, exactly?

<comment><USER4> From their reference doc:

> The mapcode system was revived and worked out in 2006 when TomTom decided to > expand their operations to countries like India and China. It was difficult > to sell navigation devices in these countries, among other things because a > significant part of the population lives without address, lacking not just > house numbers but often even street names, which often made it hard or even > impossible to enter a destination. It was thought that the introduction of a > simple and ubiquitous system for representing locations (and hence > destinations) would in due time make navigation devices as welcome there as > anywhere else.

</comment></comment><comment><USER71> No.

</comment><comment><USER72> Terrible site design

</comment><comment><AUTHOR> Subtitle taken from [http://www.telegraph.co.uk/technology/news/10975625/Did- TomT...](http://www.telegraph.co.uk/technology/news/10975625/Did-TomTom-just- kill-the-postcode.html)

<comment><USER73> It's against the HN guidelines to editorialize in titles, even when the editorialization was done by somebody else.

This one was particularly bad because the submitted title ("Did TomTom founders just kill the postcode?") caused the post to receive a barrage of user flags.</comment></comment>


Google ties up with Election Commission for voters' registration - <AUTHOR> http://timesofindia.indiatimes.com/india/Google-dips-into-CSR-budget-ties-up-with-Election-Commission-for-voters-registration/articleshow/28205460.cms

<comment><USER2> Indian Election Commission has gifted the entire country's electoral data to Google. The electoral data includes (at least) voter's name, father or husband's name, address, age and sex. Apparently its a service being done by Google as part of its CSR obligations. It is not clear if Google can use it for any other purposes.

</comment><comment><USER3> Anything for data :)</comment>


Ask HN: If you had 15TB of Storage and NO bandwidth costs - What would you do? - <AUTHOR>

Back story, my brother and I own a data backup business and we are in the process of migrating to a new data center in Souther California.  As part of our migration strategy we will be left with a Sun Microsystems Server running Open Indiana with 15TB of direct attached SAS storage.
Basically we have a nearly unlimited amount of bandwidth based on our current consumption, and we are looking for ideas for what to do with the machine.
We would be open to almost anything - if you can come up with a cool idea for how to use the gear or if you have a project that needs storage, let m know and we can see if we can work something out. <comment><USER2> Large storage and very high bandwidth immediately brings to mind video applications. How about this idea: Indievault, i.e. make available movies that cannot be distributed through the usual channels. This year more than 4K movies applied to Sundance, only a handful (I believe <100) got in. Contact these directors and get them to make their content available on the system. Conservatively assuming 1GB for a movie (some will be shorter than usual) you can make _all_ those available and then some.

</comment><comment><USER3> Hi,

           I am Singaporean Indian (srid68 at gmail) (Currently on Holiday Travelling in India) will be back in Jan to Singapore. I am interested in your offer.

           I am currently working on a Cross Platform Mobile App Framework (very old Prototype at www.arshu.com) using Mono for Android and I would like to use that Framework to create a Proof of Concept Photo Album Site.

           Basically users Upload a Zip of Photos, Specify Start Date, Expiry Date and Password if any and Download a Android Native App Album. We charge for every download of the Android App to the Uploader/Downloaded

           If interested, please send me mail.

</comment><comment><USER4> Well, You could offer various distributions to host. You would be surprised at how much bandwidth gets eaten when you host a repository for a distribution.

</comment><comment><USER5> I remember someone on Reddit opening up an S3 bucket to the world for a while and letting anyone up/down-load anything.

</comment><comment><USER6> PORN! is the first thing that comes to mind....</comment>


Show HN: Highlighting Efforts of Creation from Hacker News - <AUTHOR> https://medium.com/design-startups/e394f6d917d3 <comment><USER2> Thanks for doing this for all of us who make things. Indifference towards people's side projects and new startup ideas has been creeping into the HN community for the past couple of years and I find it discouraging. Personally, I'd rather read about people's Show HN posts than the latest mainstream news stories that I could find anywhere.

HN Showcase highlights the problem: Hardly any of the Show HN posts get voted on anymore. I think we've become more a community of critics than makers. I haven't even thought about posting Show HN's for my last couple of projects (and my current startup), where 5 years ago that would have been the first thing I did! A cool hack is worth sharing, even if it's not perfect.

EDIT: Actually, if you feel like me and have held back, why not use the comments section of this post to "Show HN" your project. At least in here you'll know everyone cares about side projects and maybe we can all endeavor to give you constructive feedback :)

<comment><USER3> Sometimes getting voted on is not enough. Posted a Show HN for BitBalloon yesterday ([https://news.ycombinator.com/item?id=6285205](https://news.ycombinator.com/item?id=6285205)). Got up-votes enough to go on the front-page, but a few minutes after it hit front-page, it suddenly got booted way down to page 3, and that was the end of it.

Still got some more votes, but nothing could bring it up above #50.

Here's a screen of the ranking weirdness: [http://hn-voting- weirdness.bitballoon.com/](http://hn-voting-weirdness.bitballoon.com/)

Don't know what it got penalized for...

<comment><USER4> Your submission was probably flagged, possibly by multiple people.

An HN pet peeve of mine is that flagging does not require even a modicum of rationale. I can flag any submission without even the slightest justification. When my own submissions have been flagged, I've also been left wondering "Why?" I understand why many of my submissions have not been upvoted. Many are simply not of upvote-quality. But not upvoting is quite different from flagging.

You may be interested in the HN Slapdown user script [1] which provides insight into articles' scores versus their rank order. With this installed, I tend to look especially at those with red scores because I am interested in counteracting what I believe to be an overzealous application of flagging.

Even right now, this very article has a yellow score indicating that it may be flagged (or something else is affecting its rank order versus its point score).

I personally have only ever flagged one or two articles, and only when I felt that the subject matter had utterly nothing to do with my understanding of the purpose of this site. When I did so, I was shocked at the magnitude of the penalty: if I recall correctly, the article I flagged was instantly dropped from the front page to the third page. It really gave me appreciation for how sparingly one should apply flags.

When I don't find a submission interesting, I just don't upvote it.

[1] [http://userscripts.org/scripts/show/150452](http://userscripts.org/scripts/show/150452)

<comment><USER3> Thanks, that script is really neat. Installed now.

</comment></comment></comment></comment><comment><USER5> Thank you so much for doing this. If hackernews fails to return to its roots I think it is prime time to create a spinoff community.

I submitted my javascript library last week and was bummed it didn't get upvoted more, even though it was well received on github. I've spent a horrible lot amount of time on it, [https://github.com/amark/theory](https://github.com/amark/theory) if anybody is interested. And my Twitter account is the same as my username here.

<comment><USER6> This is great! I want to read more of this, and less about Snowden. Or rather, I want to go one place to read about the tech community's take on Snowden and a different place to read just about things like cool new JS libraries that someone came up with.

<comment><USER7> Same here. I don't care about Snowden or anything related but I like to learn about any new libs, projects, startups...

</comment></comment></comment><comment><USER8> I agree, the amount of Guardian articles that have frequented the front page since the 'Snowden Affair' [Almost on a daily basis - some are 'opinion' pieces ] is not what I come to HN for.

I would also argue most of them articles are not within HN guidelines.

<comment><USER9> Couldnt agree more with this post and love the idea. Im working on a startup right now and have been debating whether to post it on HN glad to see some people feeling what I was feeling. I will be looking out for the weekly post.

</comment></comment><comment><USER10> At the moment I believe ShowHN posts are negatively penalized by the front page scoring algorithm, something that was setup many many years ago when HN was a completely different community.

Perhaps it's time to reverse that so ShowHN post get positively scored. Also maybe reduce the impact of flagging on ShowHN posts.

</comment><comment><USER11> Thanks for posting this, happy to see this on the front page

Also relevant: [http://showinghn.com/](http://showinghn.com/) [https://twitter.com/JustShowHN](https://twitter.com/JustShowHN)

<comment><USER12> These are great, my preferred one is [http://skimhn.com](http://skimhn.com). And it's on github so I created a version that shows the latest 50 show hn. It's incredible the quantity of projects that go unnoticed. And then there are some cases like pud showing his new music project and being ignored and after some time putting a link saying he had like 400k users and didn't know what to with them and everybody went crazy up voting it. Sometimes HN is just too far from the target audience.

</comment></comment><comment><USER13> Really love the spirit of this project. Does your Medium channel give you sufficient extra reach to make an impact for these builders?

<comment><AUTHOR> Thank you so much. Just checked out your profile, congrats on being supported by your own project :)

Not yet, but I believe concentrating them into one post on HN might make it easier for them to be noticed (in many ways pooling the resources of the posts/creators).

</comment></comment><comment><USER14> Thanks for taking the time to compile this. I had debated with myself whether or not to post my project to HN this week, expecting that either the comments would be very harsh on a v0.0.1 release, or that it would whiz by the "new" feed with no upvotes. Seeing my project, Fortune.js, on this list is very encouraging.

</comment><comment><USER15> I really like the idea. In fact this is in keeping with the spirit of the idea I submitted a couple of weeks ago ([https://news.ycombinator.com/item?id=6201511](https://news.ycombinator.com/item?id=6201511)).

It didn't pick up much traction at the time but I know (as this submission demonstrates) that the issue is one many members of the HN community would like to see addressed.

Part of the issue lies in the fact that HN is essentially a loose collective of special interest groups. Show HN submissions are competing not only with Main stream news (as you suggest) but also the large variety of other submissions ... and visa versa.

I genuinely think that a curated summary (in whatever form it may take, whether that be directory, ezine or podcast) from a couple of interesting people with different opinions would infer so much value to the HN community. Keep posting!

</comment><comment><USER16> Hi Chris, I agree, underdogs deserve their shot at the title. I actually duplicated the functionality of Hacker News at (www.plantdietlife.com) with an emphasis on plant based diets.

I think one way to address the great idea you're bringing up is to use the code I have produced for PDL (voting algorithm and all) and create another site that's only for peoples personal projects. We can have some sort of verification process where the submissions have to be approved as a "personal project" to be on the home page. Once the item is on the home page we'll demand that it needs a certain amount of comments/likes in order for it to be removed and for the next personal project to appear.

IDK, it's just a quick idea. I look forward to seeing how this vision of yours progresses.

</comment><comment><USER17> I think the showbot suggestion below is great - and it can be implemented with no extra effort.

So how about a throwaway account, so karma issues are put aside, that posts each Friday evening a thread that anyone can attach their maker projects to - a "MadeByHN" thread?

</comment><comment><USER18> I used to get a weekly hand curated newsletter with "Show HN" projects. No idea where that guy went.

edit: [http://www.weekendupdate.me](http://www.weekendupdate.me) would be great to know why it is halted.

</comment><comment><USER19> This is fantastic, thank you for going to the trouble of doing it. I, too, would like to see HN "return to its roots" \-- more articles submitted about member projects and YC companies and what they're doing.

</comment><comment><USER20> > One vote and a comment if they are especially lucky.

Rings pretty true to me; admittedly, I don't participate much in comments or have a recognizable name, but my past submissions certainly fit this bill.

Though, when I look at the 'newest' queue [1], there are 30 posts in the last hour and only one front page. Not everybody can make it. For my part, I'm mostly curious what does, actually, make it.

[1] [https://news.ycombinator.com/newest](https://news.ycombinator.com/newest)

</comment><comment><USER21> [off-topic] The text on the page is all exceedingly large, is that intentional? It looks like I'm being served a mobile version on my desktop. I can read the title from 20' away, and the paragraph text from 8'. It only looks "normal" when I zoom to 67%.

Not a huge problem, but it's annoying to the point of distracting to require so much scrolling for so little actual text. Otherwise, I love the idea and the execution.

<comment><USER22> This is just how Medium is. Of debatable merit.

<comment><USER21> Thanks for the info! That's certainly their prerogative, and as long as they provide good content it shouldn't matter too much I guess (it's easy enough for me to fix on my end).

</comment></comment></comment><comment><USER23> I absolutely agree with this. It seemed the initial posting about our events website launch wasn't seen at all or by very few people. But, if this is an appropriate place to re-post, I'd love your feedback! [http://planblock.com](http://planblock.com) and like the page at [http://facebook.com/planblock](http://facebook.com/planblock)

</comment><comment><USER24> I'm just an echo here, but this is a great - and much appreciated - initiative! Thank you for taking the time to compile this list.

</comment><comment><USER25> Thanks for the post. I love show HN posts. I wish they fared better.

As an aside, I think your requirement of less than 1000 twitter followers is a bit low. I have a couple thousand followers and I can tell you that it means very little. If I tweet a link to a project of mine I can expect a ctr of about .2%.

</comment><comment><USER26> Hey thanks for mentioning statistic.li!

</comment><comment><USER7> Thanks for rising that. Let see if my new Show HN([https://news.ycombinator.com/item?id=6294623](https://news.ycombinator.com/item?id=6294623)) will got a better audience after your post :)

</comment><comment><USER27> The first project you link to is marked as dead, so it's impossible to follow the link or post comments on it. Maybe someone should resubmit it?

</comment><comment><USER28> Write on Medium = get a front page post on HN. Write on Medium _about HN_? OMG, genius.

</comment><comment><USER29> thanks, I am not in the camp that believe is utter purity of HN, but things can go overboard. Good to see this.</comment>


YouView.com - the new IP TV system from UK broadcasters - <AUTHOR> http://www.YouView.com

<comment><AUTHOR> More info: <http://www.youview.com/questions/>

And most importantly: <http://www.youview.com/developer-zone/>

Old info site: <http://www.projectcanvas.info/index.cfm/objectives/></comment>


Call for testing: OpenSSH 8.0 - <AUTHOR> https://lists.mindrot.org/pipermail/openssh-unix-dev/2019-March/037672.html <comment><USER2> It includes an interesting comment on the scp vulnerability.

> This release contains mitigation for a weakness in the scp(1) tool and > protocol (CVE-2019-6111) [...] _The scp protocol is outdated, inflexible and > not readily fixed. We recommend the use of more modern protocols like sftp > and rsync for file transfer instead._

I think it's the time to "alias scp=sftp". If the developers officially believe that scp should be retired, let's do the switch. Both are parts of OpenSSH and the commandline argument is almost identical.

Also, it has

> _ssh(1), sshd(8): Add experimental quantum-computing resistant key exchange > method, based on a combination of Streamlined NTRU Prime 4591^761 and > X25519._

This is big. Together with XMSS signature, it means we already have a complete suite of post-quantum cryptography (experimentally) deployed in OpenSSH! It may be the first mass deployment of post-quantum cryptography in a major protocol.

One month ago, I commented that the introduction of XMSS post-quantum signature as "useless" ([https://news.ycombinator.com/item?id=19160739](https://news.ycombinator.com/item?id=19160739)), as the decryption of key exchange is much more vulnerable than spoofing the signature. But now NTRU+X25519 is deployed, great progress here!

<comment><USER3> Frankly, I had not realized scp and sftp were two distinct protocols, despite my heavily using them. I thought sftp was an interactive client, while scp was meant for scripting, and never really bothered to dig further. I now will, thank you.

However, I expect plenty of people to be in the same case.

<comment><USER4> Some servers, and some clients, only support one of them. I've run into servers without sftp and subsequently stopped using it, as scp always works.

Also rsync does not exist on windows, for instance git bash has only scp AFAIK.

<comment><USER5> A while ago I tried making windows' remote differential compression COM library work in a CLI utility like rsync.

[https://docs.microsoft.com/en-us/previous- versions/windows/d...](https://docs.microsoft.com/en-us/previous- versions/windows/desktop/rdc/remote-differential-compression)

</comment></comment></comment></comment><comment><USER6> Recommending rsync as a more 'modern' replacement for SCP made me smile. Rsync is .. idosyncratic and documented by a single codebase (not withstanding the fine work OpenBSD people have done writing their own rsync)

<comment><USER7> My pet peeve grief with rsync is that I have to double check the man page for the trailing slash rule every time I use it. Otherwise I love it.

<comment><USER8> My shell history is sure to show every use of 'rsync <whatever>' to be preceded by 'rsync -n <whatever>' to make sure I have the paths correct.

</comment></comment></comment><comment><USER9> Interesting that they seem to use CVS as the canonical source control repo. I wonder what the workflows of the dev team look like and if that tool makes it harder for external contributors to contribute?

<comment><USER10> > and if that tool makes it harder for external contributors to contribute?

I hope so!

OpenSSH is a critical piece of infrastructure in most unix and linux environments, so making sure external contributors do some extra diligence seems like a good thing!

<comment><USER11> Learning CVS is not "diligence." If anything, it takes up mental space you could be using on things like remembering to bounds-check your array accesses.

Asking contributors to learn CVS as a quality gate is like having a tech interview process that requires beating the hiring manager at chess. Sure, you're looking for smart people, and sure, people good at chess are usually smart, but the correlation is so low you'll lose out on good candidates and get candidates who are better at chess than coding.

<comment><USER12> We don't ask anyone to learn CVS. People send the maintainers (myself and other) their changes (git format-patch is fine) and we integrate them.

<comment><USER11> Yeah, to be clear I think you the OpenSSH maintainers are quite responsive to people who aren't using CVS (in particular, thank you for digging into [https://bugzilla.mindrot.org/show_bug.cgi?id=2863](https://bugzilla.mindrot.org/show_bug.cgi?id=2863) which didn't even have a patch!). I just want to be clear that people who look at what you're doing and say "OpenSSH is using an obscure process, that's what makes it high-quality" are misguided and should not implement such policies in their own products. It's not what you're doing, and if it were, it wouldn't work.

</comment></comment></comment></comment></comment><comment><USER13> Would love to see FIDO2 / Webauthn in SSH. Working with PKI tokens for key auth works but has to be set up in the client.

<comment><USER14> As I understand it using FIDO (including FIDO2) for SSH requires some pretty heavy lifting at the protocol layer.

FIDO tokens only want to do two things: Provide you with a cookie and a public key, then later as often as necessary take a cookie and give you proof they still know the associated private key. Very narrowly conceived, on purpose.

SSH public key auth has the client start by proposing "OK, I can prove I know key X" and then the server either says "Fine, do that then" or "No, what else do you have?". An out of box OpenSSH server decides which to do by examining the ~/.ssh/authorized_keys file. A FIDO token needs the _server_ to begin by saying "OK, here's a cookie, can you prove you know the corresponding private key?" so that it can get the cookie, otherwise it can't prove anything.

</comment></comment><comment><USER15> I was able to install the software, but there is no documentation about how to create NTRU+X25519 keys and enable it. I checked manpages, mailing list and tried google. How is this done?

<comment><USER16> It looks like its a key exchange algorithm, not a host key algorithm. So you don't make keys with it, you just tell your client and server to try using it when connecting. You can specify it with the KexAlgorithms config property, like for example ssh -o "KexAlgorithms=whatever". Use ssh -Q kex to see what options are available on your installation.

<comment><USER15> Thank you!

So to help everyone (read whole post first), you should probably have the line

KexAlgorithms email@example.com,email@example.com,diffie- hellman-group-exchange-sha256

in /etc/ssh/sshd_config of server and /etc/ssh/ssh_config of client (under "Host _" ).

(The rest of the kex recommendations are from [https://stribika.github.io/2015/01/04/secure-secure- shell.ht...](https://stribika.github.io/2015/01/04/secure-secure-shell.html))

\---

However, for some reason after running "/usr/sbin/sshd -T" it said

"/etc/ssh/sshd_config line 2: Bad SSH2 KexAlgorithms 'email@example.com'."

so I played around. It's hard for me to go back on everything I tried but a working solution seemed to be to add the

KexAlgorithms firstname.lastname@example.com,email@example.com,diffie- hellman-group-exchange-sha256

line to server's "/usr/local/etc/sshd_config" and to client's "/usr/local/etc/ssh_config" under "Host _".

You then need to start the server by running "sudo /usr/local/sbin/sshd" and you need to use the ssh client with the binary "/usr/local/bin/ssh".

</comment></comment></comment><comment><USER3> As I see a new release, I wish OpenSSH would conform to the XDG base directory specification. I do not agree with the proposed rationale [1]

On a side note, I lost the list of applications that are not conforming to it. I know there are a couple, and would be glad if you were to share the ones you know about.

[1]: [http://bugzilla.mindrot.org/show_bug.cgi?id=2050](http://bugzilla.mindrot.org/show_bug.cgi?id=2050)

<comment><USER17> The rationale _is_ pretty strong though:

>OpenSSH (and it's ancestor ssh-1.x) have a 17 year history of using ~/.ssh. This location is baked into innumerable users' brains, millions of happily working configurations and countless tools.

XDG can be considered as a case of arbitrary change for weak justification. SSH is definitely a case where people need to know exactly where the configuration is. There is no point in including it in the not very fun "find the config files" game engendered by XDG.

<comment><USER3> I agree on security issues, and making the path depend on an environment variable/per-user config file would be opening a new can of worms.

However, an issue I have is that while you can configure that path system- wide, there is no way to control it per-user, or with a finer-gained approach. You can probably use mount namespaces to shadow a ~/.ssh with another config, but that seems overkill.

I admit I have no practical use cases for this right now (though I probably did in the past) besides "uncluttering $HOME". As for hunting the config files, those would reside in ~/.config/ssh by default. I personally find it more irritating when a program picks a random folder instead of conforming to the spec (and if I set the environment variable to somewhere else, I then know where it is). Go hunt trough ~/{.program,.program/conf,Program/conf,Documents/Program/conf,Documents/My\ Games/conf}, or countless variants I experienced. Thought it is admittedly a much smaller problem with better-established programs such as openssh.

</comment></comment></comment><comment><USER18> It would sure help encourage testing if we could have it on AUR for easy installation on arch distros.

</comment><comment><USER19> Updating openssh has been a frustrating experience recently because of their constant breaking of old clients.

I hope this one doesn't continue this worrying tendency.

<comment><USER20> Old clients are typically broken because of security issues, or fun stuff like disabling SSH 1 protocol.

If your clients are broken, they should be updated.

<comment><USER19> You are right of course. If nothing works anymore that is by definition most secure.

<comment><USER20> That's not at all what I said.

<comment><USER21> It is widely acknowledged that hmac-md5 remains secure (if deprecated).

There is no problem using aes128-ctr as far as I know.

Finally, diffie-hellman-group14-sha1 is not ideal, but breaking sha1 requires vast resources.

An sshd allowing these settings can talk to very, very old versions, and the CPU usage of these is light compared to some of the more modern configurations.

I do have systems that are sensitive to CPU usage, and I retain these settings there, where I am less concerned about ultimate protocol security, and more focused on performance and a light footprint.</comment></comment></comment></comment></comment>


Ask HN: Why there is a black bar on top of HN? - <AUTHOR> I suddenly noticed a black bar on top of HN. What is that?
I Googled and found that "When someone important to the community dies, a thin black bar is added to the top of HN as a mark of respect".
So what happened now? <comment><USER2> Ctrl-F "has died" on HN front page usually clues you in.

</comment><comment><USER3> [https://news.ycombinator.com/item?id=17716098](https://news.ycombinator.com/item?id=17716098)

</comment><comment><USER4> Can we have a hover over message on the black bar so that people know what's going on?

<comment><USER5> No, it's a kind of shibboleth. A good hacker would already know, an adequate hacker would do the research, a poor hacker would need to be told.

<comment><AUTHOR> Lol. One thing I noticed among "hackers" here is that they don't want things to be easier for them. They hate convenience. They love hustling :D

</comment></comment></comment><comment><USER6> It's usually added when some tragedy or loss has happened in the tech community. So if you scan the top stories, you will probably find something related to it.

</comment><comment><USER7> Has HN been sold to Microsoft?

</comment><comment><USER8> because of blackhat maybe?</comment>