Doing a database join with CSV files - <AUTHOR> https://www.johndcook.com/blog/2019/12/31/sql-join-csv-files/ <comment><USER2>
      sqlite> .mode csv
      sqlite> .header on
      sqlite> .import weight.csv weight
      sqlite> .import person.csv person
      sqlite> select * from person, weight where person.ID = weight.ID;
      ID,sex,ID,weight
      123,M,123,200
      789,F,789,155
      sqlite>

<comment><USER3> I was trying to join two 300M lines CSV files using SQLite and Postgres and it took days to merge them on c4.xlarge.

Later, I tried Miller and it took less than 4-6 hours.

It also takes some time to tune Postgres to make it faster for this particular task.

<comment><USER4> When I've had to do similar, the easiest and fastest way has always been install spark -> Create an external spark sql table from the csvs -> use spark sql to join and aggregate/write. If you haven't used spark before it sounds complicated but its actually really quick and easy.

In general the operation on my laptop can get up to 200 MB/s, it's basically IO limited to the SSD of your machine.

<comment><USER5> Would be nice if you could write a small article about that.

</comment></comment></comment></comment><comment><USER6> Not that I want to be the "well actually" guy...but MS has a much underrated tool called Log Parser that's been around for years that lets you do this kinda thing and more.

If you're a Windows admin and haven't bumped into this utility then it's well worth a look.

edit: forgot to mention that it can also scripted from VBScript and Powershell for added fun.

[https://www.microsoft.com/en- gb/download/details.aspx?id=246...](https://www.microsoft.com/en- gb/download/details.aspx?id=24659)

<comment><USER7> Seconding for log parser.

There's even a UI there : [https://techcommunity.microsoft.com/t5/exchange- team-blog/in...](https://techcommunity.microsoft.com/t5/exchange-team- blog/introducing-log-parser-studio/ba-p/601131)

</comment></comment><comment><USER8> The `join` command does a similar job and is included by default on at least Mac OS X (10.14) and Ubuntu 16.04.

I think it's much more common than that implies, but have not looked beyond two machines I had immediate access to.

The interface is clumsier in some ways, but it's already there, which is often a win when writing scripts.

<comment><USER9> xsv author here.

There is very little overlap between what xsv does and what standard Unix tools like `join` do. Chances are, if you're using xsv for something like this, then you probably _can 't_ correctly use `join` to do it because `join` does not understand the CSV format.

If your CSV data happen to fall into the subset of the CSV format that does not include escaped field separators (or record separators), then a tool like `join` could work. Notably, this might include TSV (tab separated files) or files that use the ASCII field/record separators (although I have literally never seen such a file in the wild). But if it's plain old comma separated values, then using a tool like `join` is perilous.

I didn't write xsv out of ignorance of standard line oriented tools. I wrote it specifically to target problems that cannot be solved by standard line oriented tools.

You might also argue that data should not be formatted in such a way, and philosophically, I don't necessarily disagree with you. But xsv is not a philosophical tool. It is a practical tool to deal with the actual data you have.

<comment><USER10> In case folks don't know, burntsushi is also the author of ripgrep (rg), written in Rust, and possibly the fastest grep in the world at the moment.

rg has all but replaced grep for me.

<comment><USER11> > _burntsushi is also the author of ripgrep_

They are also the source of much highly practical domain-specific knowledge and advice on how to write Rust code that does e.g.; efficient text file reading, streaming, state machines, etc.

Many thanks for these contributions.

<comment><USER12> And that advice applies further than just Rust, too. Highly recommend checking their blog and work out!

</comment></comment></comment></comment></comment><comment><USER13> If your goal is to join large CSV files using a local program, the ideal tool is not SQLite but DuckDB:

[https://www.duckdb.org/docs/current/sql/copy.html](https://www.duckdb.org/docs/current/sql/copy.html)

The reason why DuckDB is a better fit for this job is because DuckDB is a column store and has a block-oriented vectorized execution engine. This approach is orders of magnitude faster when you're doing batch operations with millions of rows at a time.

In contrast, SQLite would be orders of magnitude faster that DuckDB when you're operating on one row at a time.

<comment><USER14> I've used SQLite quite a few times, but never heard of DuckDB. Can anybody provide some more information about it?

<comment><USER15> Main author of DuckDB here, I did not expect to see this mentioned here. DuckDB is a relational DBMS geared towards efficiently handling large analytical-style workloads locally. It is similar to SQLite in the sense that it operates locally on your machine, is easy to run and install and has zero dependencies. However, DuckDB uses modern processing paradigms (vectorized processing, columnar storage) that make it much faster when processing large amounts of data.

It's still in an early stage currently, however, most of the functionality is there (full SQL support, permanent storage, ACID properties). Feel free to give it a try if you are interested. DuckDB has Python and R bindings, and a shell based off of the sqlite3 shell. You can find installation instructions here: [https://www.duckdb.org/docs/current/tutorials/installation.h...](https://www.duckdb.org/docs/current/tutorials/installation.html)

</comment></comment></comment><comment><USER16> I wonder what niche these tools fill. Maybe when you want to do some ad-hoc analysis and want every step in your shell history?

Otherwise it seems more flexible to just fire up a python interpreter and do it in like 3 lines of pandas (or with sqlite and .import like another commenter mentioned)

<comment><USER17> For me it's ad-hoc analysis on large CSV files. Large meaning well beyond what Excel would be capable of, often larger than fits into memory on my local machine (10s of GiB).

Sometimes I also use xsv to just do a step of the analysis and dive deeper on some subset using pandas.

In my experience both SQLite and Pandas aren't as fast as fast for large files. So they are not really good options.

Pandas is especially bad because it uses a column oriented data structure internally so reading from or writing to CSV is incredibly slow in Pandas. If you can use parquet that's not a problem but unfortunately parquet is not nearly is ubiquitous as csv :(

<comment><USER18> If Pandas is slow, than you can use Spark. For such big files laptop is not an option anyway. SQLite can be fast if you index your data (but I've worked with files < 10G). Nowadays I am just uploading CSV to some cloud database and work with data there.

<comment><USER19> > For such big files laptop is not an option anyway

Too big for excel is not big data, and my laptop can load this 10G in RAM (not that it necessarily need all of it) so why not if the data is here and the laptop on your lap ?

</comment></comment></comment></comment><comment><USER20> For Windows, Microsoft offers a CSV ODBC driver that can make the files look like any other relational database.

[https://support.microsoft.com/en-us/help/850320/creating- an-...](https://support.microsoft.com/en-us/help/850320/creating-an-odbc-data- source-with-the-microsoft-text-driver)

<comment><USER21> Oracle has a feature called external tables that lets you map regular files to the database and run SQL against them. Works with csv or any format that can be loaded to the database with the normal database loading utilities. This is very useful for getting transformation and loading done in one step!

<comment><USER22> MySQL has a barebones CSV storage engine: [https://dev.mysql.com/doc/refman/8.0/en/se-csv- limitations.h...](https://dev.mysql.com/doc/refman/8.0/en/se-csv- limitations.html)

More usefully, MySQL has LOAD DATA INFILE facilities for bulk loading of flat files.

</comment></comment></comment><comment><USER17> `xsv` can do far more than just join and select, such as searching, filtering, partitioning, ...

For me it's the first choice when I find myself with a CSV file to work with. I've also encountered situations where just getting some raw data from a (slow) database and "querying" it with xsv ended up being the fastest option to get the results I wanted.

</comment><comment><USER23> Also along these lines is tsv-utils:

[https://github.com/eBay/tsv- utils/blob/master/README.md](https://github.com/eBay/tsv- utils/blob/master/README.md)

There are benchmark results comparing tsv-utils to a variety of similar tools including xsv.

</comment><comment><USER24> It's interesting that there doesn't seem to be a command-line tool out there that lets you do this with LevelDB. LevelDB is about an order of magnitude faster at importing a lot of data than SQLite or Postgres is (although I don't know if it's faster than the Postgres CSV foreign data wrapper for the first query, it sure will be for the second) and it can handle files a lot bigger than your RAM.

It's interesting to note that around 1990 The Mark Williams Company wrote a fairly full-fledged database system using, basically, CSV files. It was called "/rdb". You wrote your queries as shell scripts, but using a set of /rdb utilities that handled CSV files with labeled columns, and they had a screen- based data-entry UI based on vi. I wouldn't want to use it instead of a database --- you had to write your query plan in the shell script, rather than a high-level SQL-like query, because the authors didn't really understand SQL --- but it's interesting that the approach is still useful in 2019.

</comment><comment><USER25> I was expecting the answer to involve SQLite and .import and am happy to have learned about this cool new tool.

<comment><USER26> This is using SQLite behind the scenes, so it's pretty much what you expected.

<comment><USER27> On the contrary, xsv doesn't use SQLite at all

<comment><USER26> Ah, sorry, I mixed it up with [http://harelba.github.io/q/](http://harelba.github.io/q/)

</comment></comment></comment></comment><comment><USER28> Shout out to Perl DBD::CSV. Gives you full SQL query functionality.

</comment><comment><USER29> Hey, just wanted to chime in with another tool, OctoSQL, which I'm a co-author of.

It allows you to query and join data from multiple datasources simultaneously, which may include CSV files.

Currently available datasources are SQL databases, Redis, JSON and CSV (more coming...).

[1]:[https://github.com/cube2222/octosql](https://github.com/cube2222/octosql)

</comment><comment><USER30> There are many ways to do this. A valid reason for wanting to do this is that doing database imports takes time and sometimes you just want to get to the joined data set without having to wait for that.

A few years ago, I was trying to turn open streetmaps dumps into json documents. OSM ships as a huge bzipped XML dump of what is essentially 3 tables that need to be joined to do anything productive. One of those tables containes a few billion nodes. Importing that into postgreql takes many hours and takes up diskspace too. Bzipped, this stuff was only around 35GB. But it unpacks to >1TB. So, I wrote a few simple tools that processed the XML using regular expressions into gzipped files, sorted those on id and then joined files on id by simply scanning through multiple of them. Probably far from optimal but it got me results quick enough. Without taking ages or consuming lots of disk.

</comment><comment><USER31> Any info on whether it does this in a streaming manner? For small files there are a plethora of solutions, but it gets more tricky / interesting when you can't assume both files fit in memory and (ideally) you want to do it in a single pass because I/O itself is a bottleneck.

<comment><USER9> It's almost done in a streaming fashion. `xsv join` uses memory proportional to the number of records in one of the inputs and the size of the join columns in one of the inputs. Namely, it creates an in-memory map from the join column's values to the corresponding record index. Combined with another in- memory index that maps record index to the corresponding record's byte offset, it can jump to the corresponding record in constant time while streaming over the other input.

There are better ways to do this that don't involve using this much memory, but this struck a good balance between memory usage and implementation complexity. I may improve on this some day.

So basically, as long as you have enough memory to store the entire join columns along with `8 * len(one_input)` bytes for the record index, then you should be good.

<comment><USER31> great! It may address some particular cases that I don't have a good tool for currently then. Thanks!

</comment></comment></comment><comment><USER32> Pretty old now and unsupported but "crush tools" was fantastic for this kind of thing

[https://github.com/google/crush-tools](https://github.com/google/crush-tools)

<comment><USER33> This doesn't seem be worked upon either?

</comment></comment><comment><USER34> Haven't tried XSV. In the same space is Q: [https://harelba.github.io/q/](https://harelba.github.io/q/)

<comment><USER35> Also csvkit, xsv is way faster but less feature-complete (based on my experience last time I used both)

</comment></comment><comment><USER36> Linq can also be use for possibly similar use cases

[https://docs.microsoft.com/en- us/dotnet/csharp/programming-g...](https://docs.microsoft.com/en- us/dotnet/csharp/programming-guide/concepts/linq/how-to-join-content-from- dissimilar-files-linq)

<comment><USER37> I haven't used xsv, but one of the nice feature appears to be that it will work on arbitrarily large datasets. From the README[0]:

> These commands are instantaneous because they run in time and memory > proportional to the size of the slice (which means they will scale to > arbitrarily large CSV data).

The example given with LINQ reads the whole files into memory:

      string[] names = System.IO.File.ReadAllLines(@"../../../names.csv");
      string[] scores = System.IO.File.ReadAllLines(@"../../../scores.csv");

As a fellow LINQ (probably with LINQPad in this case for a quick and dirty script) user, I'd love to know LINQ can be used to read the files in "slices" like xsv. I'm sure this could be accomblished with enough code, but is there quick / easy way to do it?

[0]: [https://github.com/BurntSushi/xsv#a-whirlwind- tour](https://github.com/BurntSushi/xsv#a-whirlwind-tour)

<comment><USER38> >The example given with LINQ reads the whole files into memory:

The examples indeed use `File.ReadAllLines` which returns a `string[]`, but the same thing can be done with `File.ReadLines` which returns an `IEnumerable<string>` - a lazy sequence of lines read on demand as the sequence is enumerated.

</comment></comment></comment><comment><USER39> XSV is extremely fast. I have a high frequency trading data api that powers my analytics. The data coming in as json which I write to a RAMDISK. Then I transform json to csv using jq and use xsv to get the data segments and strats I need.

Despite pretty basic code, the performance is very good.

</comment><comment><USER40> Another tool that can do this and other operations on CSV files is CSVkit. I've had very good use for it.

[https://csvkit.readthedocs.io/en/1.0.2/](https://csvkit.readthedocs.io/en/1.0.2/)

</comment><comment><USER41> Sigh, this is so much a case of "I have this nice hammer and I will use it for this screw". Maybe this works ok on simple joins with small CSV, but let's just import the CSV into database temp tables and get the job.

</comment><comment><USER42> Surely you are better off simply running PostgreSQL locally, loading it into a couple of tables, and running a simple query?

<comment><USER43> As always, the correct answer depends on what you have installed, what you're familiar with, how much data you have, and whether you're going to be doing it on a regular basis and want to automate it.

<comment><USER42> Sure, but I think there is a widespread hesitation to use a db unless one's data reaches a higher level of size and complexity, which is misguided.

What I'm saying is that it's good to start with a database from the start, even if the problem seems too trivial for it. That way, if your problem grows into something that actually needs it, you already have it in a suitable database and the code to deal with that.

<comment><USER9> xsv can be inserted into a shell pipeline at the drop of a hat, regardless of how much data you have. Using a DB is typically much more effort. Just look at the other comments in this thread. Tons of complaints about it being slow and trying to finagle indices to make it better.

Sometimes functionality isn't the only thing that's important. Expression can be just as important.

</comment></comment></comment></comment><comment><USER44> This is awesome. Would love to see some performance benchmarks on this tool.

<comment><USER17> Some commands are benchmarked here: [https://github.com/BurntSushi/xsv/blob/master/BENCHMARKS.md](https://github.com/BurntSushi/xsv/blob/master/BENCHMARKS.md)

</comment></comment><comment><USER45> Very cool tool!

</comment><comment><USER46> Full join is missing the Id on 456 and the Key on 999..

</comment><comment><USER47> For an advanced challenge: do it for a hundred large bz2 files too big to fit on disk in an uncompressed state, such as the monthly-compiled hourly dumps of Wikipedia Pagecounts-EZ ([https://dumps.wikimedia.org/other/pagecounts- ez/](https://dumps.wikimedia.org/other/pagecounts-ez/)). This is feasible with CLI utils, but not trivial.

</comment><comment><USER48> You can also join CSV or Excel files very easily using our new tool: Easy Data Transform: [https://www.easydatatransform.com](https://www.easydatatransform.com)

It is a GUI tool for Windows and Mac. No syntax to remember. Just drag the two files on to EDT and click the 'Join' button then choose the columns to join.

Join is just one of 36 transforms available. There is a 7 day free trial.

<comment><USER49> You were downvoted because it's off-topic and more like an ad. Hn doesn't like ads, self promotions are allowed.

The title mentions joins, I'm not expecting a GUI-tool.

<comment><USER48> Thanks for the explanation. I get that you don't want overly promotional posts. But: -Easy Data Transform performs SQL-style joins on CSV files. It is one of its main use cases. -The OP didn't mention anything about it having to be command line.

<comment><USER49> I think it's more HN related than something else.</comment></comment></comment></comment>


UN climate change panel based claims on student dissertation - <AUTHOR> http://www.telegraph.co.uk/earth/environment/climatechange/7111525/UN-climate-change-panel-based-claims-on-student-dissertation-and-magazine-article.html <comment><USER2> Hmmm:

Anecdotal evidence from non-peer reviewed popular news media, student dissertations and environmental advocacy organization reports and web site articles that supports our thesis, _good_.

Serious research that's kept out of blessed peer-reviewed journals by our conspiracy that doesn't support our thesis, _bad_.

<comment><USER3> Your comment implies that the IPCC panel started with an idea of what conclusion they wanted to make. What is your evidence that's the case.

It's entirely invalid to say that because they came to a certain conclusion and the majority of the cited studies support that conclusion, they picked the citations to support it. That claim can be applied to _any valid_ scientific review. It uses some sort of Catch-22 argument that because the evidence support the conclusion, the evidence must have been cherry-picked. With that view, no science would be valid.

Besides, your comment doesn't make sense. What about all the anecdotal evidence from non-scientific sources that _do_ support their thesis but _was_ excluded from the report?

Without assessing the rates of the complementary claims, your statement just looks like good, old-fashioned selection bias.

<comment><USER4> In general, a scientific review does not cite nonexistent/non-peer reviewed/advocate sources [1].

 _It's entirely invalid to say that because they came to a certain conclusion and the majority of the cited studies support that conclusion, they picked the citations to support it._

The argument is being made that because they eschewed the normal practice and included _unscientific_ citations _only in support of their thesis_ , we can conclude cherrypicking was going on. You can't apply it to a normal review, since a normal scientific review doesn't normally base their claims on anecdotes from mountaineering magazines.

[1] Such sources are occasionally cited as novelties or as evidence of importance of the topic. E.g., "as evidenced by recent media attention \cite{NYT, FOX, NBC, CBS} my topic is important".

</comment></comment></comment><comment><USER5> Do they still get their trillion dollars?

<comment><USER2> The global warming industry is Too Big to Fail.

</comment></comment><comment><USER6> Does anyone know an effective way on how to filter out this drivel with greasemonkey?

Unfortunatelly it doesn't use always one single word/expression we can use to catch it.

<comment><USER7> If you truly think it's drivel, then flag it.</comment></comment>


Société Générale tells clients how to prepare for 'global collapse' - <AUTHOR> http://www.telegraph.co.uk/finance/economics/6599281/Societe-Generale-tells-clients-how-to-prepare-for-global-collapse.html <comment><USER2> The United States has tens of trillions of dollars of mineral wealth (oil, natural gas, tungsten, gold, silver, copper, molly, palladium, rhodium, scandium, yttrium, bastnaesite, etc...) trapped in the ground by a gordion knot of government regulations and bureaucracy.

By fully exploiting this mineral wealth, the US could not only pay off its entire national debt, but also generate trillions of dollars of surplus tax revenue, pay for universal health care, cut taxes, eliminate the federal income tax, and create tens of millions of American jobs.

Interesting Fact: The United States has more oil and natural gas than any other country in the entire world, including Saudi Arabia. These are the official findings of the United States Senate, page 23: <http://j.mp/3HIfQi>

We could more than offset our trade deficit with China by feeding their voracious appetite for natural resources.

Another interesting fact: if Nevada were a country, it would be the world's 4th largest producer of Gold.

On a per-capita basis, the United States is sitting on a mind numbingly huge resource of untapped mineral wealth.

We could literally dig out way out of debt.

<comment><USER3> Oh yes, the usual idiotic bullshit -- "if it wasn't those liberals with those environmental protection laws we would all be millionaires and have our own ponies."

Let me just tell you some of the more obvious ways you are wrong:

"Interesting Fact: The United States has more oil and natural gas than any other country in the entire world, including Saudi Arabia. These are the official findings of the United States Senate, page 23: <http://j.mp/3HIfQi>

This is simply wrong. And the very document you reference proves it wrong. If you see table 5 of the document you provided, the US is well below Saudi Arabia and many other countries in oil and natural gas. It has about 1/10 of the oil reserves of saudi arabia and about 1/6 of the natural gas of Russia. Even if you add oil and natural gas up, the US is not first -- saudi arabia and Russia both have more oil and nat gas than the US.

Now if you think the regulations are the problem, just look at Russia. Russia is notorious for having a corrupt government where environmental and other regulations are broken all the time. It also has about three times as much oil and about 6 times as much nat gas as the US. It is also less than half of the populations of the US. So, by your logic, Russia should be much richer than the US, because it has more mineral wealth and does not have the regulations that prevent its exploitation. But that is obviously not the case. The living standard of Russia is still much lower than the US.

What might have made you confused is that if you add up coal the US does become first. But of course coal has not made any country rich after the mid 20th century. It is too expensive to transport for export and it is dirt cheap because it is dirty, causes global warming and disease, and nobody wants to use it unless they have no other choice. Good luck basing our economy on coal.

The thing about Nevada may be right (I do not have time to check) but in global economic terms gold production is simply not a big deal.

Of course you face the more fundamental problem that you think somehow producing raw materials will get us out of a global recession. The problem with global recessions is that demand for everything tanks and that includes raw materials. The so called "voracious appetite" of China is only voracious because they use it to feed the voracious appetite of America for consumer goods. If Americans keep getting poorer that appetite of China will no longer be voracious and raw materials prices will keep falling.

Raw materials only drive economies when they are relatively small economies that can hitch themselves on larger consumer spending economies that are booming. Thus, for example, Russia was able to improve their economy by selling raw materials to booming US, Europe and China. But a major economy like the US simply cannot rely on raw materials or it will quickly become a minor economy.

<comment><USER2> Table 6, page 23:

"Reserves of Fossil Fuels Plus Technically Recoverable Undiscovered Oil and Natural Gas:" \---

Russia: 293.7 \---

Saudi Arabia Estimated Undiscovered Oil and Gas: 231.3 \---

United States Undiscovered Oil and Gas: 351.5 \---

351.5 > 293.7 > 231.3

Regardless, I'm putting my life where my mouth is. I started my own oil and natural gas exploration and production company in Denver.

I've also spun off a second company, a mineral royalties company, that is focused on the acquisition and divestiture of North American mineral rights in producing open-pit mines.

And finally, I've started a third US natural resources company, SonicFrac, to develop a solid-state fracturing device for optimizing the production of US Shale gas wells.

I also invest every last penny that I take home back into US mineral rights, US oil and gas prospects, and US minining royalties.

I own stock in dozens of publicly traded oil, gas, mining, and resource development companies.

Natural resources are the first thing I think about when I wakeup in the morning, and the last thing I think about when I go to bed at night.

I'm 26.

Perhaps I'm irrationally optimistic about US resource wealth...

I don't believe that I am.

But, you _need_ to be irrationally passionate about what you do. Whatever it is.

If you're running a consumer internet company, and are not irrationally passionate about its future, I doubt you could persevere through the trials and tribulations of being an entrepreneur.

I am a US natural resources entrepreneur. And I am zealous about what I do. You should be, too.

<comment><USER3> Undiscovered oil and gas means (logically enough) oil and gas that has not been discovered yet. The existence of this stuff is a mere guess. Not only that but different people are making estimates for different countries using different methodologies, and different countries have different incentives for under or over estimating.

You cannot say that the US has more oil and gas based on estimates or guesses as to the undiscovered oil.

But wait. Lets assume that you are right. Lets assume that these undiscovered oil and gas estimates are perfect and all of the estimated undiscovered oil and gas actually exists. Well then if you try adding up these undiscovered estimates of table 6 to the proven oil and gas reserves of table 5 you will still find that russia and saudi arabia have far more oil and gas than the US. So your initial statement is still incorrect.

Well congratulations on being an entrepreneur and good luck. I hope you get some of that undiscovered oil. But you should not let your enthusiasm nudge you into massaging the facts.

<comment><USER2> Column #3 on Table 6, page 23 is the sum of: proved hydrocarbon reserves (from table #5) + undiscovered hydrocarbon reserves in BOE.

<comment><USER3> Column 3 of table 6 includes coal. Nobody disputes that including coal the US has the most hydrocarbons.

But the original statement was that the US has the most oil and nat gas. And that is not true and not shown by the document.

edit: hydrocarbons substituted for carbohydrates

</comment></comment></comment></comment></comment></comment><comment><USER4> Anyone have a link to the referenced Société Générale report?</comment>


Show HN: Infinite Zoom - <AUTHOR> http://infinitezoom.net/

<comment><AUTHOR> This is based on an older project of mine. It started out as an Android app written in Java/Libgdx.

The patterns are actually constructed and drawn directly in code. For the web version, I reused the java code and wrote an interface that passes the drawing calls over to Three.js, respectively now constructing the very same patterns in the browser.

Really glad this is now able to reach a larger audience.</comment>


Writing an editor in less than 1000 lines of code, just for fun - <AUTHOR> http://antirez.com/news/108 <comment><USER2> Awesome. When I saw all the things it could do in so little code, I assumed a higher level language would be used. But, nope...plain old C with the usual standard libraries.

The parser for the syntax highlighting is super cool. Concise and declarative definition of the syntax (well, the keywords, anyway), in HL_keywords, and then maybe 150 lines of code for the parser. I doubt it would work well for a much more complex language than C (say, JavaScript, or Perl), without a lot more smarts. But, it works really well on the C I looked at. Colors are handled in a cute way, too.

All around, I'm just astounded at how concise this is, while still being entirely readable and comprehensible. This isn't clever/tricky golfing code, this is actually human-readable C code.

Edit: Also, I'm struck by how nice/simple the editorProcessKeypress handling is. Trying to do something like this in JavaScript/HTML5 is crazily more complex and verbose. Seeing something like this reminds me how messy user interaction still is in the browser, compared to old CLI and desktop paradigms.

<comment><USER3> The syntax highlighter isn't really a parser, _per se_. It's just a lexer / tokenizer / scanner / what-have-you, which tend to be fairly compact state machines.

As an aside, I agree with your edit. As someone who started programming back in the DOS days, whenever I do Web development, I'm always floored by how much it feels like one step forward and two steps back. So many simple things just _aren 't simple_ when it comes to the browser, and even certain protocols on the backend like FastCGI seem way more complicated than they need to be (SCGI seems pretty nice, though I feel it's missing a way to signal out-of-band information to the Web server, the way you'd use stderr in CGI, for example (though perhaps I'm missing something there)).

As much as I love the idea of sending semantic markup over the wire for document transfer, I can't help but wonder if a more terminal-like protocol would be better for application "delivery", the way we used to do with telnet and BBSs in the 90s, with an upgrade for multimedia. But I digress...

<comment><USER4> FastCGI/SCGI were obsoleted by having app servers simply speak HTTP; the web gateway just needs to function as a simple reverse proxy. HTTP is about as simple as it gets to parse - you can write a passable (not quite production quality, but works) parser in about 10 minutes.

A lot of the difficulty in webapps is because every webapp is inherently a distributed system, which are always hard. Single-page apps with no connection to a server are actually quite simple, but they're also about as commercially viable as DOS programming (i.e. not at all).

<comment><USER5> A while back I'm sure I saw a HTTP state diagram posted here that showed writing a HTTP parser is anything but simple. I guess if you _only_ have to be a (reverse) proxy you might get away with it.

<comment><USER6> You're thinking of [https://github.com/for-GET/http-decision- diagram](https://github.com/for-GET/http-decision-diagram)

<comment><USER7> That's the state diagram for processing the full complement of HTTP requests. Nothing to do with parsing. The parsing bit is trivial.

<comment><USER6> While you're correct that it's a state diagram processing HTTP semantics and not parsing; parsing a text-based protocol is far from trivial. In fact, the HTTP2 FAQ explicitly mentions [1] that reducing parsing complexity was a motivation for going binary with HTTP2.

[1] [https://http2.github.io/faq/#why-is- http2-binary](https://http2.github.io/faq/#why-is-http2-binary)

<comment><USER4> I've done perfectly adequate HTTP parsing with this Python 4-liner:

      headerText, body = text.split('\r\n\r\n', 1)
      headerLines = headerText.split('\r\n')
      method, path, protocol = headerLines[1].split(' ')
      headers = dict(line.split(':').map(str.strip) for line in lines[1:])

For production use you'd probably want something a bit faster & more robust like Mongrel's HTTP parser (itself only 166 lines of Ragel), which powers several million websites out there:

[https://github.com/mongrel/mongrel/blob/master/ext/http11/ht...](https://github.com/mongrel/mongrel/blob/master/ext/http11/http11_parser.rl)

<comment><USER6> That's a perfectly adequate 4-line HTTP/1.0 parser :) But for HTTP/1.1, which must support chunked transfer coding [1], this won't work.

[1] [https://tools.ietf.org/html/rfc7230#section-4.1](https://tools.ietf.org/html/rfc7230#section-4.1)

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> Off topic, but my biggest takeaway was that I could aspire to be someone who isn't a twenty something, with a family, with superb work life balance, not living in the valley, sitting in a garden without three 27 inch monitors, not using the latest programming language or fad and still writing beautiful code that a lot of the world runs on.

<comment><USER9> I agree. I LOVE how his post weaves in his normal life, having time for doing things he actually loves outside of work, while still continuing to work on passion projects. It just goes to show you that you don't need to always buy into the cowboy coding hype that is often on display here.

</comment></comment><comment><USER10> > Let's say this again, "email client". The notion of email client itself is > gone at this point.

Really? Is it lame to be using an email clients these days?

I still use an email client (Thunderbird) because I can't stand the thought of all my mail sitting forever on Goggle's or whoever's servers. Yes, I _know_ that they could have secretly archived all of my email the moment it was sent or received, and that my privacy is not necessarily enhanced by downloading and storing my mail offline.

That's my main reason, but I can think of many other reasons to prefer email clients to a web interface.

I'm surprised by the sentiment that email clients are passé.

<comment><USER11> I've actually been working on a new email client, and a terminal one at that. Shameless plug: [https://github.com/SirCmpwn/aerc](https://github.com/SirCmpwn/aerc)

<comment><USER12> Very cool! Going by the acronym, are you trying to make a Stallman-ish IMAP client that works well offline and only needs an occasional Internet connection to sync?

<comment><USER11> No, just that it does networking on a different thread. This is my main (but not only) problem with mutt. I'm not opposed to going in the direction you suggest, but think more mosh less stallman.

</comment></comment></comment></comment><comment><USER13> _And yet I read, on Hacker News, a number of people writing how they were often saved by the availability of nano on random systems_

All the time, but I even use nano all the time on OS X for full on programming, mostly for quick edits or starting things off before I can be bothered opening a full project in Sublime. (Sure, I can 'get around' in vi, but I just don't seem to have the brain for it and nano is 90% fine. I grew up on GWBASIC so anything is an improvement ;-))

That aside, this feels like it could be a kinda cool command line level equivalent of ToDoMVC. Port this simple editor into Go, Rust, and a few similar languages - see what the differences are, etc.

<comment><USER14> Nice thing about Nano is that they arrow keys always seem to do what they are supposed to, no matter the system, shell, TTY, etc. Vi seems to have issues sometimes, and needs extra config

<comment><USER15> Presumably h, j, k, l will always get you around in vi?

<comment><USER16> Yeah but that is so much less usable. Quick, which one is up?

<comment><USER17> Just look at your keyboard :-)

([https://en.m.wikipedia.org/wiki/ADM-3A#Legacy](https://en.m.wikipedia.org/wiki/ADM-3A#Legacy))

I always _try to_ remember them by noting that the leftmost key moves left, the rightmost key moves right (not too hard), and that 'K climbs' (using alliteration to remember that). That leaves 'J' for cursor down.

In practice, though, I use the cursor keys. Luckily, they work on the editors/systems I work on nowadays. Side effect is that I am very bad at remembering those mnemonics.

<comment><USER18> I stole the following from a colleagues `.vimrc` which finally allowed me to learn `hjkl`. Now I forget I have this set, and when I do accidentally use an arrow key it makes me laugh.

        map <Left> :echo 'damnit!'<cr>
        map <Right> :echo 'you suck!'<cr>
        map <Up> :echo 'this is why you fail'<cr>
        map <Down> :echo 'nooooo!'<cr>

</comment></comment></comment></comment></comment></comment><comment><USER19> This is the kind of magic that got me excited about programming in the first place.

I remember being in high school and one day, after class, a friend of mine mentioned that the best programmer we knew had once written a breakout clone with ANSI "graphics." We found him and he said he didn't have the source any more, but we asked him how he did it. He then proceeded to rewrite the whole thing, in Pascal, in about an hour, this time adding animations. Thanks antirez for bringing back that feeling (and, oh yeah, for redis too ;))

PS. the "Low level terminal handling" is gold here.

</comment><comment><USER20> I love seeing little projects like this, and I wish I were better at finding time to do them myself. It's all too easy when you've been programming professionally for a long time, often working on the same projects every day for months or years, to lose that sense of wonder at what we can achieve in a few hours with a modest amount of code. Sometimes it's good to just sit down for a day or two and hack together a kind of program you've never written before, whether that's a text editor or a puzzle game or rendering a pretty 3D fractal landscape or an IRC chat bot. It's always nice to see someone else who's created something that way too, so thanks for sharing.

</comment><comment><USER21> Honestly, this is the most exciting and inspiring project I've seen on HN in years, by far. I love that it's a fully usable terminal-based text editor that has _zero dependencies_. Just perusing the source code has been extremely interesting and educational for me already, and it's only been 5 minutes. Definitely gonna bookmark this project and hack on it! The TODO list looks like a good start, especially the xterm-based feature.

<comment><USER22> You can tell it's going to be good when, on a Mac, you download the source and compile it, and a) it compiles b) it doesn't give a single warning. It's nice when portability actually works.

</comment></comment><comment><USER23> Yay, this is delightful.

I wrote a nano-inspired TTY-based editor, too, and have used it to do virtually all of my coding work for the last couple of years. There's nothing particularly noteworthy about it, technologically, but it does exactly what I want in exactly the way I want it to, and that was enough to make the time spent feel worth my while.

There's something really satisfying about the exercise of creating one's tools. I think there is a lot of value in projects like this whether anyone else ever uses them or not, as an exercise in software craftsmanship.

Looking at this project, I am inspired to see if I could simplify my editor's codebase. I was never fully convinced that ncurses did enough good to justify its complexity; it might be interesting to adopt this author's approach instead.

</comment><comment><USER24> When I started programming, the basic digital world already existed for me. I never had the experience of typing in a program from a magazine. This is one of those odd experiences where you remember that self-hosting[1] can involve far more components than just the compiler, spiraling an unexpectedly deep path into the development stack.

I just used @antirez's 1k LOC kilo editor to edit his 1K LOC editor and it was an oddly pleasurable experience :)

[1]: [https://en.wikipedia.org/wiki/Self- hosting](https://en.wikipedia.org/wiki/Self-hosting)

</comment><comment><USER5> This is the programming equivalent of a food blog post (technical details interspersed with what's happening in the author's life). It's great and I think there should be more of it!

</comment><comment><USER25> Cool; now embed it into a 150 MB Electon app and you have a state of the art editor (maybe introduce several arbitrary delays in the range 300 ms - 2 sec for the authentic Atom feeling)!

</comment><comment><USER26> Wow! I love this!! Thanks for the post antirez, if just for introducing me to this now indispensable tool ;)

asciinema - Record and share your term sessions

[https://asciinema.org/](https://asciinema.org/)

<comment><USER27> Yes, asciinema is a wonderful thing I learned about from this blogpost too. And another one from the asciinema.org: lolcat! Command line forever!

</comment></comment><comment><USER28> This is great! I love seeing how experienced programmers write small programs like this, especially "from scratch".

</comment><comment><USER29> Can anyone explain how to do a full-screen UI using only VT100 codes? If I had to write a small editor, I'd probably reach for something like Termbox for display.

<comment><USER30> First, switch it to alternative screen mode... printf("\033?1047h"); then clear and move and such with other similar prints. See the list here : [http://invisible-island.net/xterm/ctlseqs/ctlseqs.html](http://invisible- island.net/xterm/ctlseqs/ctlseqs.html)

It really is pretty simple. The big benefit of ncurses is if you want to target terminals other than xterm compatible ones, and they are so rare nowadays that you can really just ignore them....

Input is quite a pain though (however, ncurses sometimes get it wrong too), but the same thing applies, you just need to switch to raw mode with tcsetattr and then have some kind of select() loop or something to watch for escape sequences then translate them into interesting key presses.

Mouse events are the same btw: you write out a sequence to turn them on, then the terminal sends them as sequences to stdin.

I wrote a terminal library myself for the D programming language (and a terminal emulator too!), it is kinda ugly code but it isn't hard.

<comment><USER5> I wonder if these codes work in cmd.exe now that Windows 10 TH2 supports (some of) the features.

<comment><USER30> I haven't checked but I kinda doubt it... and I kinda hope not - the windows console API is so much nicer than the unix terminal system it would be a pity to make it an ugly hybrid.

Of course, you can run a terminal emulator inside the windows console and handle those sequences too! If I had to guess, I'd say that's what Microsoft would have implemented.

</comment></comment></comment></comment><comment><USER31> Pretty cool. Something like this would make for a nice school project, I would have loved being assigned to write a simple editor back in uni

</comment><comment><USER32> @Antirez please please please let this thing have a future: there is a great need for a modern, simple and usabile CLI editor.

<comment><USER33> Try vis: [https://github.com/martanne/vis](https://github.com/martanne/vis)

<comment><USER29> I find it funny that a person named _acme suggests vis :)

</comment></comment></comment><comment><USER34> It would be fun to see this project replicated using some other underlying data structures, e.g. the buffer-gap data structure used by emacs.[0] Does the code become shorter/prettier? etc.

0: [https://en.wikipedia.org/wiki/Gap_buffer](https://en.wikipedia.org/wiki/Gap_buffer)

</comment><comment><USER35> What challenges might you have if you wanted to implement something like this inside a game framework like Unity or MonoGame with C#, where it renders everything via DirectX or OpenGL?

<comment><USER36> The main challenge is that, even now, in the year 2016, there is no objectively good, fast, reliable way of rendering arbitrary text with formatting in a texture. Hell, even just plain text is kind of a pain in the ass.

And yes, I know about signed distance fields.

<comment><USER35> The nice thing about a framework like Unity is that they do have a textfield object that behaves a lot like a standard text object in the .NET Framework. It has a lot of shortcomings but rendering the text is the easy part.

<comment><USER37> Unity only supports Unicode BMP. Unity doesn't support surrogates, RTL rendering/input etc. Rendering the text performantly for none western scripts is definitely not the easy part, especially on mobile (where Unity also fails to behave properly).

<comment><USER36> Proper Unicode and RTL support is something Jonathan Blow recently spent a _lot_ of time getting just right for The Witness. I wonder if he'd ever consider open sourcing that work.

</comment></comment></comment></comment></comment><comment><USER38> Nice work. I clearly remember writing a full screen editor in a few hundred lines of Turbo Pascal back in '86 because I hated using EDLIN to edit .BAT files in DOS... :D

</comment><comment><USER39> Anyone up for converting it to Swift (3) or Rust?

<comment><USER40> I am currently learning Rust, would be happy to give it a shot! My email can be found in my profile

</comment></comment><comment><USER41> Back in the 90's I took a class in Visual Basic, where one of the assignments was to write a rules-compliant tic-tac-toe game. The teacher apologized for the amount of repetitive code we'd have to write to implement it in Windows forms (or whatever it was called back then). I got it down to one printed page.

<comment><USER42> When I taught V that was a class assignment (along with create a 4 function calculator). I don't remember any of the programs being that long.

For extra credit for people that had problems I'd allow a resubmission of an assignment if you did the logic in a different way. One of the students did a table that let them look up the current board and say what the new move was.

</comment></comment><comment><USER43> > WARNING: Long pretty useless blog post

Antirez's posts are hardly ever really "useless" :) Awesome stuff, as always.

</comment><comment><USER21> What's especially impressive to me is that _it even has syntax highlighting._

</comment><comment><USER44> > a text editor in less than 1000 lines of code that does not depend on > ncurses and has support for syntax highlight and search feature.

Neovim also does not depend on ncurses. Vim optionally links to it for termcap/terminfo access.

</comment><comment> c-smile Cool :)

But editor without undo/redo is a bit surprising these days, next release?

<comment><USER22> That's what git is for ;-p

</comment></comment><comment><USER45> If antirez is reading:

How much time did this take? From start to finish. :)

<comment><USER46> The GitHub page states: "The project is in alpha stage and was written in just a few hours taking code from my other two projects, load81 and linenoise."

[https://github.com/antirez/kilo](https://github.com/antirez/kilo)

<comment><USER45> Yes, thanks for pointing that out. I'm more interested in total time. Not time to write. :)

<comment><USER47> Hello! It was a few hours (probably 6-8) across a total of two weekends (so a total span of 9 days where I did 2 "days" of work, 5 days of pause, 2 days of work).

<comment><USER45> Thanks for answering.

</comment></comment></comment></comment></comment><comment><USER48> I'm reading the code as a Primer. Fantastic stuff you don't see often.

</comment><comment><USER49> Id like to see people have competitions to write an editor in the most readable amount of lines of code

</comment><comment><USER50> Bonus points for being in C

</comment><comment><USER51> cough cough fewer than... cough cough...

<comment><USER52> He mentioned using cloc to get that metric, so not including comments, formatting, etc.

<comment><USER33> I think you missed the point: the point is that the correct English usage is "fewer than" when discussing quantifiable items (e.g., lines of code). The title should be "...an editor in fewer than 1000 lines of code."

<comment><USER52> Thank you for clarifying. I thought for a second it could have been a grammatical nitpick but discounted that since this thread is re: an Antirez post. I personally would be very sad if his writing style became more forced and began to feel like excerpts from The Elements of Style.</comment></comment></comment></comment>


Can you crack the secret code from WWII? - <AUTHOR> http://thelede.blogs.nytimes.com/2012/11/23/can-you-crack-the-secret-code-from-world-war-ii/?pagewanted=all

<comment><USER2> Paging Alan Turing...</comment>


Building something someone else patented... - <AUTHOR> https://gist.github.com/anonymous/9522629

<comment><USER2> If it has a parent it means it is obvious, so no i wouldn't code, because I'm sure there are 3 open source projects on that already.</comment>


Java 8 Cheatsheet: lambdas, method references, default methods and streams - <AUTHOR> http://www.java8.org/

<comment><USER2> Thanks for this cheatsheet. Java becomes sexier and sexier.

<comment><USER3> s/sex/cruft/

FTFY</comment></comment>


Grew a Garden,Harvested wheat,travelled ocean to boil salt,laughtered a chicken - <AUTHOR> http://www.today.com/food/making-chicken-sandwich-scratch-took-six-months-1-500-t45091 <comment><USER2> I'd like to learn more about how you "Laughtered" a chicken. What sort of jokes do they like? Presumably they've all heard the one that starts "Why did the human cross the road?"</comment>


How Good Names Can Help Products Succeed - <AUTHOR> https://medium.com/p/5e98414cbcc4

<comment><AUTHOR> I've first heard about these apps from Hacker News. Both Docker and Hemingway App became front page news here. They are definitely great products but also they both have great names which tell stories.</comment>


Ruby Testing and External Dependencies - <AUTHOR> http://blog.palominolabs.com/2013/05/06/ruby-testing-and-external-dependencies/

<comment><USER2> An interesting solution, but somewhat troublesome if you're running just one unit test that doesn't touch that backend service.

If it's retrieving data over HTTP, you can just use the awesome VCR gem: <https://github.com/vcr/vcr> It's awesome for testing HTTP APIs (along with webmock), since it works relatively transparently and makes your tests far less fragile.

<comment><AUTHOR> Mocks can be useful for testing interfaces (APIs with a strictly defined behavior), but they have significant limitations. Mocks are stubs of a class at the time the test was written—it's very easy for them to be forgotten and no longer mirror the behavior of the mocked class. Additionally, they are code that increases the complexity of the tests making it harder to comprehend what the test is doing.

Starting up a separate system simply to run one unit test does seem like a hassle, but in practice it hasn't slowed down our tests one bit. The external dependency is started as a background process, spinning up at the same time rspec is bootstrapping. We also have a number of different tests that interact with the service, so the value is even greater.

These (<http://stackoverflow.com/q/38181/17339>) questions (<http://stackoverflow.com/q/1595166/17339>) on Stack Overflow have some instructive discussion on using mocks and stubs.</comment></comment>


Ranking crowdsourced data with curves - <AUTHOR> http://www.cederman.com/?p=116

<comment><USER2> As someone who works (sort of) for Tripadvisor, I found this interesting. I'm a little suspect that "Since I changed tactics, I have been having great hotel experiences." is confirmation bias, though.

<comment><AUTHOR> Maybe. I had a string of pretty crappy hotels, so started paying more attention to the reviews and noticed that although the hotels I had been staying at had higher averages, they had a lot less 5 star ratings. Next few hotels were much better, and when the option to get a hotel with a better curve wasn't available, I was appropriately disappointed. While I would like to think that I don't suffer from confirmation bias, this certainly isn't a double blind trial.

I have also seen a similar phenomenon with restaurants on Yelp though, which has been slightly easier to test in that I can try two similar restaurants with the different shaped curves.

</comment></comment><comment><USER3> I tried a similar approach with the netflix prize, although in that case I fitted truncated normal distributions to the data.</comment>


Uber to Seal $3.1B Deal to Buy Careem This Week - <AUTHOR> https://www.bloomberg.com/news/articles/2019-03-24/uber-is-said-to-seal-3-1-billion-deal-to-buy-careem-this-week <comment><USER2> Uber's strategy right now is to maximize horizontal integration across as many markets as possible. They're leveraging cheap credit and an infusion of private money (mainly from Middle Eastern investors looking to diversify their portfolios) to expand as quickly as they can in local transportation.

This growth, combined with their subsidization across all their products, is producing huge near-term losses. However, I think it's a prudent investment. Uber needs to solidify its moat in an industry that offer little in terms of differentiation beyond price. By trying offer as many services as possible, Uber's trying to make itself the go-to hub of local transportation, and in doing so, start to change its role from a dispatch middleman to a full service transportation platform.

And so, I think in the years following IPO, we'll slowly see them reduce their efforts for horizontal expansion and slowly turn the lever back towards profit. This will happen slowly and differently across markets, with price rises starting in regions they have the strongest foothold, but will slowly trickle across all markets. The hope for them is that through this rapid expansion, they've bought enough leverage to raise prices without losing too many customers.

<comment><USER3> It's interesting how little of this strategy involves providing a better service. But I don't think they're alone or even special from this perspective (e.g. similar to the rise of Microsoft).

I think the prevalence of "business strategy", as opposed to product/service improvement, in the growth of corporations, warrants much more scrutiny than it gets. After all, corporations should serve society - how do strategic mergers, acquisitions, lobbying, PR do that?

<comment><USER4> Good for Uber, but I'm disappointed as a customer. I did extensive travel overseas in Qatar and the Careem app was excellent. They had a clear local strategy with local customization to capture users. For example

\- [Local flavor] Using a button called "Yallah" instead of "Go"

\- [Local Knowledge] Pre-coding airport pickup locations (unlike Uber which had you try and figure it out directly with drivers, who often did not speak English.)

\- [Local Knowledge] Understanding that many parts of the world do not have street addresses!

<comment><USER5> It would be great if Uber accepted plus codes ([https://plus.codes/](https://plus.codes/)) when setting a destination.

</comment></comment></comment></comment><comment><USER6> As an end user of Careem, I'm waiting for the impending doom of price increase and bad service.

Uber and Careem are the only two service providers in my country (Pakistan); before Uber entered, the prices were quite high because Careem had no competition and they cashed in their monopoly. Uber entered the market and caused a price war driving prices per km down significantly and causing both providers to add new features to cater for the local market as well. However, Uber is usually thought to have a low quality of service here; drivers on their network quite literally abuse the platform; they will refuse to drive to the destination if they don't want to go there; they will use tactics to get the ride transferred from them if it's not profitable enough for them. The end result is that the customer receives a slightly cheaper but vastly inferior service from Uber. Compared to this, Careem's quality assurance and customer care is a winner.

With Uber buying Careem and eliminating competition, I can't imagine how much pain it is going to be to call a ride now, if they absorb Careem's customer- base completely and close its app. I thought there were anti-competition laws that prevented this from happening; I guess they don't apply if both the companies are foreign and the deal is happening overseas.

<comment><USER7> Why isn't Lyft going for foreign markets as aggressively as Uber?

<comment><USER6> I think that would be because Uber has some USD 24 billion invested in it while Lyft has raised nearly USD 5 billion (figures from Crunchbase). With deep pockets, Uber's right move is to spend fast and monopolize as much of the market as possible. They do this by buying out competitors or offering massive discounts on their pricing to out run their competitors. They can afford to figure out how to be profitable later.

</comment></comment></comment><comment><USER8> Doesn't this, so close to their purported IPO, fly right in the face of what they consider to be their network effect? Why would they need to buy competitors if not because it's expensive for them to compete against local upstarts who own a smaller market?

<comment><USER9> Uber's network effect is localized to cities. If Uber attempts to enter a market with a strong local player, the local player stands a decent shot at defending since Uber's global dominance doesn't really transfer over to that particular city.

This is unlike social networks, like WhatsApp, where the network effect is truly global since people often have friends and relatives in distant cities.

<comment><USER10> Uber's network effect isn't localized at all.

Try getting someone to keep 27 different ride hailing apps on their phone, and convincing that person to maintain them all and remember which accounts correspond to which cities.

Uber only doesn't have a national network effect if you view people like software in appraising their potential behavior. Actual human behavior is thus: I have zero interest in maintaining many ride hailing accounts, I have zero interest in having numerous ride hailing apps on my phone, I want as few as reasonably necessary.

If you can get people to keep two or three ride hailing apps on their phone for one country, that's pushing the limits of consumer behavior. They do not want to bother. They also do not want to be negatively surprised by a new competing service, which limits their willingness to experiment when there are existing known solutions. They'll choose well-understood Uber at 75%-85% good enough rather than risk having a bad experience in regularly trying out a new upstart. Actual consumer behavior functions as an enormous competitive moat in fact.

<comment><USER11> I think it's a very small segment of their customer base that actually requires cab hailing services in 10+ cities. Most people working day jobs will not visit more than that many cities over the course of the year, and Uber possibly serves most of them.

</comment></comment></comment></comment><comment><USER12> How much capital does Uber have left?

My understanding was that they only had a few billion dollars left and were burning > 1B a quarter.

And they are still not profitable right?

On what basis exactly are they going to IPO? What's the angle for buying shares of a company that is losing so much money?

<comment><USER13> $50b in revenue last year is nothing to scoff at. They have levers for profit but instead re-invest every dollar. Your argument sounds exactly like what people said about Amazon years ago.

<comment><USER14> That $50b figure is not their revenue, but their gross bookings.

For example Uber's last quarter revenue was $3 billion on some $14 billion in bookings, with a loss of about $800 million.

<comment><USER15> My understanding;

$14 billion is revenue

$3 billion is their earnings after driver costs.

<comment><USER14> Why that $14 billion can not be called revenue is Uber specifically structures it's business to make sure their drivers form no part of the business, instead they insist all their drivers are independent contractors.

They do that specifically to reduce their running costs and hence maximize the profit on their actual revenue.

That would be like saying all iPhone App developers are working for Apple, when in fact all Apple does is take 30% of each and ever dollar these App developer earn in sales.

Likewise Uber just puts it's 30% Uber tax on every dollar it's drivers earn.

</comment></comment></comment></comment></comment><comment><USER16> As great as this will be for the tech ecosystem in the Middle East, this type of consolidation should be blocked for antitrust. It'll be bad for customers and drivers.

But it won't be blocked. Uber and Careem share common investors (aka Saudi) and this would be a huge win for them.

<comment><USER17> While this is largely true. The Middle East needs large exits like these (e.g. Souq, Maktoob) in order for investors and carpenters to be motivated. The Middle East (except for Israel) is really lacking in this arena.

</comment></comment><comment><USER18> > Shareholders in Careem, whose backers include Saudi Prince Alwaleed bin > Talal's investment firm and Japanese e-commerce company Rakuten Inc., [...]

Isn't a public Saudi Arabian fund already heavily invested in the Softbank Vision fund, which in turn has invested a lot in Uber? [0] I don't know who the respective principals are of these funds but that seems noteworthy to me (especially at these valuations).

[0]: [https://www.cbinsights.com/research/saudi-arabia-pif-tech- in...](https://www.cbinsights.com/research/saudi-arabia-pif-tech-investments/)

<comment><USER19> Al Waleed's investment fund is private, not public.

</comment></comment><comment><USER20> Interesting... sounds like a side deal with the Saudi sovereign fund.

</comment><comment><USER21> I'm guessing Lime soon too.

<comment><USER22> Doesn't Uber have a large investment in Bird?

<comment><USER21> No. They did invest already in Lime.</comment></comment></comment>


Ask HN: How to explain "Hypothesis" to a non science audience? - <AUTHOR> Here's my problem. Most of the teams, and startups I interact with, are working on Growth, and therefore conduct lots of experiments.
However, many of the team members do not really have a working idea of what's a good experiment. This could be dangerous for their projects and efforts. This is more disheartening for me because my product deals with scientific growth experiments.
While after explaining it, they get it (they are pretty smart, usually), but I am sure there are many who don't know and don't know that they don't know.
All online content on hypothesis seems to be pretty heavily worded. Which is good for rigour, but not useful when a 22 year old marketer is looking at it at 3 am in the morning.
So I have started to write a series and the very first article is here: https://www.lightcat.io/hypothesis
This is in a language that most people should understand (I think). Would love feedback and thoughts on this.
Do you think that scientific methods need to be explained in a manner that everyone gets it? And that there are not many resources out there for this? And whether the material I have written solves the problem. <comment><USER2> I'd try to explain them something more concrete, like A/B testing. There are some nice articles written by patio11:

* " _Why You Don 't A/B Test... And How You Can Start This August_" [https://training.kalzumeus.com/newsletters/archive/why_you_d...](https://training.kalzumeus.com/newsletters/archive/why_you_do_not_ab_test) ([https://news.ycombinator.com/item?id=6161552](https://news.ycombinator.com/item?id=6161552) | 119 points | Aug 5, 2013 | 56 comments)

* " _Dispatches From The A /B Testing Trenches_" [https://training.kalzumeus.com/newsletters/archive/ab_testin...](https://training.kalzumeus.com/newsletters/archive/ab_testing) ([https://news.ycombinator.com/item?id=12540590](https://news.ycombinator.com/item?id=12540590) | 166 points | Sept 20, 2016 | 46 comments)

Also, a good Mythbuster marathon may be helpful, but it is more difficult to explain as a business activity.</comment>


Show HN: Godray – a simple ray tracer - <AUTHOR> https://github.com/LanJian/godray

<comment><AUTHOR> Wrote this with friend as a learning project for golang. Any golang tips would be much appreciated.

</comment><comment><USER2> What would you recommend for writing a simple ray tracer? Any books or websites?

<comment><AUTHOR> For me, going through wikipedia article on phong and working out the math with my friends (instead of just blindly translating the equations to code) really helped me understand the basics.

In order to learn more (I admittedly know very little about anything other than the phong model - the other friend is much more knowledgable), I presume either a textbook or a course notes like this [https://www.student.cs.uwaterloo.ca/~cs488/notes.pdf](https://www.student.cs.uwaterloo.ca/~cs488/notes.pdf) will probably help.

</comment></comment><comment><USER3> Are there any dependencies to run it on Windows? I do 'go run src/main.go' on Windows, and nothing happens and there are no errors reported.

<comment><USER4> Hi, I worked on this project as well. I tried it on Windows, it ran without issues. If you run 'go run src/main.go', it should create output.png in the current directory. Then you should be able to run 'explorer output.png' to view the image.

</comment></comment><comment><USER5> What is the license for the code?

<comment><AUTHOR> Ah. An oversight on our part, we will make it MIT licensed so feel free to do whatever you want with it.

</comment></comment><comment><USER6> How do you find a friend to study and write code with you?</comment>


Ask HN: Has anyone made money on Amazon Mechanical Turk? - <AUTHOR>

I'm out of work at the moment, & am considering trying this to make at least something until a real job materializes. What are people's experience with this? I'm curious to hear what users have to say. <comment><USER2> Check out the subreddit, which sometimes has useful discussion on this topic: [http://www.reddit.com/r/mturk](http://www.reddit.com/r/mturk) Also, I must concur with gwintrob, it's better to look for freelance gigs that will pay more than AMT. Just browse through Upwork and see if any categories make sense to you. Not all of it is very specialized

</comment><comment><USER3> AMT is geared towards anonymous tasks (e.g. classifying adult images) and finding projects on Upwork (formally oDesk) will probably be better if you have marketable online skills.

Also, do you live somewhere with Uber?

<comment><AUTHOR> I don't drive, and I'm not sure I have 'good enough' skills to freelance. I can hold my own doing Wordpress sites, but it's nothing to tout, as most people can do that, too.

Thanks for your reply.</comment></comment>


Ask HN: I want to learn Bootstrap but I dont truly know JS; bad idea? - <AUTHOR>

I know C, C++, C#, Java and some assembly but I've never learned a web language. Despite my lack of knowledge, I've been interested in using bootstrap but could I just dive in and learn on the go or should I truly learn Javascript beforehand? Thanks HN! <comment><USER2> Um.. I've done Bootstrap a couple of time and I feel like it's mostly a CSS/grid framework.

The javascripts are there to make stuff backward compatible with older browser, such as modernizr, and whatever.

If you want to learn say web design/web development. You should learn HTML/CSS first then Javascript with DOM tree. Then jQuery after that a serverside language imo.

Dom Scripting is a pretty good book btw.

</comment><comment><USER3> It depends on what you actually want to do with Bootstrap. Do you need it to quickly assemble a website just for prototyping and you are satisfied with the layout Bootstrap comes with? In that case: just read the Bootstrap manual and you will mostly be fine. But, if you want to make changes to the layout and have some more control, you better start of learning html and css (like all have said here).

</comment><comment><USER4> As pointed out, you need to start with HTML. First learn the basics(class, ids, tags, data-* attributes) etc. If you haven't done HTML at all it will be slightly tricky for a couple of days. JS won't be a huge challenge.

</comment><comment><USER5> Start with HTML. Once when you master it, go to JS. A good place to start: <http://www.w3schools.com/>

If you know C/C++, HTML/JS will be pretty easy for you.</comment>


Show HN: Video Puppet – create great promo videos in minutes - <AUTHOR> https://www.videopuppet.com <comment><AUTHOR> Hi,

I'm very excited to start opening Video Puppet, my project for 2018 YC Startup School, to a wider audience. It is an online service that that enables trainers, educators and marketers to create videos much faster than with traditional editing tools. It will automatically align timelines, speed up, slow down or pause video segments to match your audio. The Puppet can even create a life-like narration audio from your text.

I built this tool as I got frustrated with how long it takes to create short promo videos. Usually, it took me several hours of work to create a few minutes of good video content, and most of that time had nothing to do with the content itself, but with timelines, transitions, recording and re- recording narrations after small mistakes. With Video Puppet, creating a five minute promo video takes only 10-15 minutes.

The tool is now in public beta, so lots of functionality is still missing, but it is ready for wider feedback. I'd love to hear your thoughts!</comment>


Create time and attention budgets for your life - <AUTHOR> https://www.tyroway.com/blog/create-budgets-for-you-life/ <comment><USER2> I think the main problem with time management is prioritization. People are confused by what is important to them. We are told to live true to yourselves but everyone has 2 selves: the current self and future self and these 2 selves often have conflict of interest.

Another problem with finding time to do something is the context. We have 8-9 hours at work but many of us only work for 5 or 6 hours. However, we cannot do personal things (like reading or working on your project) for the rest of the day. We will be judged by the people around us.</comment>


ASCIImeo: Vimeo videos, rendered in ASCII - <AUTHOR> http://www.asciimeo.com/

<comment><AUTHOR> It cheats a bit and renders the ASCII video in flash. It would have been way cooler to suck <video> frames into a canvas and post-process them via Javascript (even if the framerate was much slower).

Still a neat project, though.

<comment><USER2> yeah I was half expecting to see it rendered with javascript and canvas or video tags but still cool even with flash video.</comment></comment>


I was Nearly Arrested at a Hackathon - <AUTHOR> http://legendarymoves.com/?p=311

<comment><USER2> Link borked :(</comment>


How foreign governments spy using PowerPoint and Twitter - <AUTHOR> https://www.washingtonpost.com/posteverything/wp/2016/08/02/how-foreign-governments-spy-using-email-and-powerpoint/ <comment><USER2> The malware known as PowerPoint has infected millions of systems worldwide and has cost businesses, NGO's and governments untold billions in lost productivity. PowerPoint files seem to be self-spawning and capable of infinite replication in the wild. Small variations in the PowerPoint files forces administrators to keep endless permutations of these highly virulent files.

PowerPoint is often introduced into an organization by highly sophisticated threat actors using deeply customized versions of the software. McKinsey, BCG, and Bain deploy the PowerPoint malware to a multiplicity of customers whose human capital and infrastructure become mired in endless recursive loops known as PowerPoint cycles.

The actual introduction of PowerPoint is typically merely incidental to these threat actors ambitions. Most often they derive substantial portions of their income by reselling organizations intellectual property which they already own.

Once the organizations systems become bogged in ever more bloated PowerPoint files productivity plummets. Morale drops among employees and management often re-engages the threat actors to attempt to right the ship.

The PowerPoint Malware is seemingly unstoppable at this point. Any computers that contain it should be air gapped and protected by highly restricted physical access.

</comment><comment><USER3> I like the word "foreign" in the title, which I assume is to imply the US government is not doing this. It's also neat how they point out examples of all the "bad" countries doing this, but of course none of the "good" ones.

Given I'm not American, so the US government is "foreign" to me.

<comment><USER4> I'll tell you what the USGOV isn't doing. They're not jailing twitter dissidents. They're not sniping protestors from rooftops. They aren't destroying the credit ratings of political dissenters. They aren't even doing a very good job of using Twitter or other technologies to communicate all the good humanitarian work the USGOV does in the world. There are times I wish the US would disengage from the world so humanity can be reminded of what its like to live without a generally benevolent hegemonic power.

<comment><USER5> I feel this is a straw man. I'm Swedish, but have always loved the US for some strange reason. I still do, but that doesn't mean that the US is perfect in any way, shape or form. It is a fantastic country with many flaws. You are obviously right in much of what you write, but that does not excuse a lot of the things that NSA et. al. does in the name of freedom. My point is just that we should not hold ourselves to the lowest of standards ("We're better than <non-democratic country" is not a good excuse).

<comment><USER6> Au contraire, it is actually you who is creating a Straw Man. rrggrr doesn't write that the US is perfect, but rather, s/he writes that the US is a "generally benevolent hegemonic power".

<comment><USER7> > generally benevolent

It's not benevolent, that's pure propaganda at work. It's a nation that acts in its own interests like any other. It just disguises them extremely well.

</comment></comment></comment></comment></comment><comment><USER8> > Among the malware was a malicious spyware, including a remote access tool > called "Droidjack," that allows attackers to silently control a mobile > device. When Droidjack is installed, a remote user can turn on the > microphone and camera, remove files, read encrypted messages, and send > spoofed instant messages and emails.

How is this even possible? I have always considered phones to be more secure then PC due to additional security measures (such as sandboxing for every app as well as more fine granted permission systems).

I'm aware that malware can be installed on a phone but I always though I was required to explicitly allow that (at least on android) and I thought it was impossible on iOS without jail breaking.

Or does this app use some zero days, that have been discovered years ago but have not been patched because of androids broken update policy?

More information on how this works would be highly appreciated.

<comment><USER9> I don't know about droidjack specifically, but your phone is not at all secure, particularly for android. All it takes is a little bit of bad C code to cause an overflow of some sort. Combine that with widely available unencrypted firmware and you have a real vulnerability.

<comment><USER10> Are you implying that you could create an Android exploit with "little bit of bad C code"? If so, I'm not sure you understand how exploits work and the work required to get around all of the mitigations in place.

<comment><USER11> Depends. Once you get root--and as others have noted above, that's far easier than it should be on most handsets--all bets are off really.

</comment></comment></comment></comment><comment><USER12> I've been volunteering teaching Latin American journalists how to research, communicate, and store data privately to protect themselves and their sources against attackers. The threat against freedom of expression there is just as real as it is with oppressive regimes in other parts of the world although they don't get nearly as much attention as the Middle East. This article doesn't highlight enough the need for volunteers and professionals to lend a hand. Most recently I've been working with journalists in Venezuela. If you know anything about Venezuela you should know that they have an incredibly oppressive government and they also have had massive inflation further eliminating their buying power. Things like a $3 (USD) a month VPN are hard for a middle class citizen to afford.

I'm trying to remember where I saw it, but there was a journalist who showed over the past year how much food 1000 bolivars bought. At the beginning of the photo essay, she could have fed a family for a week. By the end, it was barely enough for 1 person for a meal.

Anyone can really help out and make a difference too. Not just in LATAM, but around the world. The amount of knowledge about cybersecurity, threat models, and risks associated with electronic communications spans a wide range. Of course you have civil society groups who know how to use PGP, but there are others who still rely on Facebook Messenger to communicate with sources and keep passwords sticky noted to their computer screen.

Edit: Also wanted to note that it's pretty great what Citizen Labs is doing. Other great resources for learning/teaching/staying updated ( in both English and Spanish and several others) can be found on the EFF's website - [https://ssd.eff.org/en/playlist/journalist- move](https://ssd.eff.org/en/playlist/journalist-move)

</comment><comment><USER13> Took a look at DroidJack[0], and it's impressively nefarious. Seems like Android is Windows all over again when it comes to security. I already see people running virus scanners on their phones.

[0]: [http://www.symantec.com/connect/blogs/droidjack-rat-tale- how...](http://www.symantec.com/connect/blogs/droidjack-rat-tale-how-budding- entrepreneurism-can-turn-cybercrime)

<comment><USER14> OK, it's a RAT, but how do we get from receiving a supposed .ppt on email to having DroidJack .apk installed on the system ? Assuming user isn't tricked into confirming the install (not to mention disabling the manual apk install protection).

<comment><USER15> I'm not sure if this was how it's done. But if the app is on the Play store, it can be remotely installed on your Android device via the Play website.

<comment><USER14> Ah, it's all described perfectly here [1] and Android specifically [2]:

[1] [https://citizenlab.org/2016/08/group5-syria/](https://citizenlab.org/2016/08/group5-syria/)

[2] [https://citizenlab.org/2016/08/group5-syria/#part4](https://citizenlab.org/2016/08/group5-syria/#part4)

Good old fake Flash update. But still relies on user to be installed, so nothing sophisticated or "Android is bad" here.

If anything the Powerpoint malware looks interesting. The .ppsx extension kinda gives it away, since it's unusual for casual presentations, but that's hard to spot for Joe User.

<comment><USER16> The fact that installing an app can give people arbitrary access to your phone without understanding WHAT you installed definitely implies Android app installation is broken in some way.

<comment><USER14> Well, it's exactly what the warning popup when enabling side-loading states ;] But it's still possible, so that's pretty fine in my book compared to the fruit company.

<comment><USER16> It's a little too easy for non-technical users to accidentally enable it. The moms & pops of the world would probably never need to side-load.

<comment><USER17> Who is the right person to decide who "needs" to side-load?

Besides, there are loads of legitimate reasons to, there's a reason Humble Bundles aren't a thing on iOS.

<comment><USER16> > Who is the right person to decide who "needs" to side-load?

The person who decides to explicitly opt in to allowing side-loading. Just don't allow apps, web pages, etc etc to opt-in for you to ensure the technically non-adept have a difficult time getting themselves into trouble.

<comment><USER17> Which is already the case...?

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER18> From an endpoint security perspective, 'activists' really need to be trained not to just click on everything blindly and to open everything suspicious in a sandbox.

Yes it's possible to escape from a VM, but it's significantly harder to code executable malware that will escape from, for example, a Windows 10 VM running inside Virtualbox on an XUbuntu/XFCE4 host laptop.

</comment><comment><USER19> Exporting software to spy on people should be regulated similarly to weapons exports, I feel.

I wonder what kind of programmer works for companies that produce this spyware...

<comment><USER20> I have a feeling tech companies would oppose that, because there is little difference between "governments spying on people" like this, and companies tracking people everywhere, and building shadow profiles of them (like what Facebook Like and Google Analytics are doing).

This is why Schneier is calling it "surveillance as a business model":

[https://www.schneier.com/blog/archives/2013/11/surveillance_...](https://www.schneier.com/blog/archives/2013/11/surveillance_as_1.html)

<comment><USER21> I don't agree, the intent has to be taken into account: hackers working at Vupen for example don't work to make a better Google analytics, they work exclusively to circumvent existing protections to expose private data to random third parties whatever the human consequences are as long as they profit from it.

<comment><USER22> > the intent has to be taken into account

no offense intended, but ...

</comment></comment></comment></comment><comment><USER23> Puff piece. Despite "twitter" being in the title, the article doesn't discuss twitter. The title's use of the word "how" also suggested i might read something about the technology involved, something like a backdoor in the twitter api. Nope. No discussion whatsoever about the hows. In short: Governments use spyware. Thank you Washington post for that important public service announcement.

<comment><USER24> > Despite "twitter" being in the title, the article doesn't discuss twitter.

From the article:

> In May 2016, we uncovered a Twitter-based digital malware campaign seemingly > orchestrated by the United Arab Emirates, which resulted in the arrest and > torture of numerous activists and journalists there.

<comment><USER23> Yes. One mention. No discussion. No indication as to why Twitter is in the title, as opposed to any other service. This is puff/clickbait.

</comment></comment></comment><comment><USER25> How do the Twitter attacks work? Just malware links on the Twitter service that are working via the spear phish? The article didn't seem to go into any detail there.

<comment><USER26> Not entirely sure how that system worked, but a Russian attack used Twitter as a sort of Command and Control system:

[https://www.fireeye.com/blog/threat- research/2015/07/hammert...](https://www.fireeye.com/blog/threat- research/2015/07/hammertoss_stealthy.html)

PDF has more details here: [https://www2.fireeye.com/APT29-HAMMERTOSS- WEB-2015-RPT.html](https://www2.fireeye.com/APT29-HAMMERTOSS- WEB-2015-RPT.html)

</comment></comment><comment><USER27> The original report has much detailed explanation with all the facts[1]. I was left wondering how an ppt downloaded an app into an Android phone.

[1] [https://citizenlab.org/2016/08/group5-syria/](https://citizenlab.org/2016/08/group5-syria/)</comment>


Email Will Never Die - The Man Who Invented It Reveals Why - <AUTHOR> http://www.readwriteweb.com/archives/email-will-never-die-the-man-who-invented-it-reveals-why.php

<comment><USER2> The reason that "email" will never die is that the limitations of the human mind compel user interface designers to employ the principle of least surprise and to extend existing metaphors slowly. In practice, many people are using "email" which runs on internal Exchange systems and is accessed using Outlook or web applications. When I send an email via Gmail on my "phone" (Which is as much a camera that can make calls as it is a telephone that can take pictures - the only reason we use one metaphor and not the other is path dependence) to someone else using Gmail, I am "sending them an email" without using the protocols that the original implementers of email had in mind.

It's the same reason that we can call a web site a "page" or a "document" despite the fact that a modern web site is actually several programs running insider your browser (flash, custom JS, HTML rendering) working together with programs running on a number of different servers. That's nothing like a paper document, but we use the metaphor because it's been extended a little bit at a time.

</comment><comment><USER3> Email can only be replaced by another system which does not require you to be on any particular network (decentralized).

Email doesn't die because gmail can send to hotmail and to every other domain.

The entrepreneurs trying to replace it will focus their effort on one exclusive platform which they know they could monetize if they hit a tipping point. Unfortunately, the very motivation to profit is what creates the biggest problem.

The only true solution to email is a decentralized system, that many other companies could utilize. Unfortunately, once you have a system which is easy to create competitors you get instant commoditization. Once you have that, the economics of the idea suck which means that VC, and other investment money will never chase that system. In other words, a decentralized system is worthless from the idea stage due to the expectation of commoditization.

That is why email is "unlikely" to be replaced.

<comment><USER4> That sounds nice and I hope you are right. But I can certainly see a scenario where one of these walled-gardens gets big enough to where most people communicate using this platform and email ends up being like usenet or irc. Email can never be disabled (right?) so we'll still be able to use it technically, but it could cease to be practical for everyday business and personal correspondence.

<comment><USER5> We're already there. I can get a faster response from 75% of my friends by sending them a message on Facebook or Twitter.

It really sucks because you're so limited to what program you use. Especially Facebook and soon Twitter too.

</comment></comment></comment><comment><USER6> Oh, so V.A. Shiva Ayyadurai did not invent email? ;)

<comment><USER7> V.A. Shiva Ayyadurai: "As a high school student in 1978, he developed an electronic mail system, which he called "EMAIL".

Ray Tomlinson: "In the fall of 1971, Tomlinson sent the first network email".

</comment></comment><comment><USER8> I've been saying for a while that the only problem with email is one of management. As the volume of incoming emails increases we need better tools to avoid a similar increase in the amount of time we spend checking it.

To that end (shameless plug) some friends and I have been building Lightermail to solve this problem for us. We're getting close to opening the service up to a wider pool of private beta users and would love to have some HN users and their feedback in the mix. If interested head over to <https://lightermail.com> and we'll be sure to get back to you in the next couple weeks with an invite code.

</comment><comment><USER9> How long did it take for the traditional post office to die? It didn't die even after the arrival of FedEx or DHLs.

Email is a basic and all inclusive communication method. People later on have added and tagged along their businesses to ride on the same method. Those who are facebooking, tweetering, chatting, texting, etc, are still communicating on the same basic principle which binds the email.

However, headlines like "email will never die" or "email is dead" are as stupid as they can be.

</comment><comment><USER10> I suspect that the people who have been calling the death of email have never worked in enterprise.

<comment><USER11> Email died in the enterprise a decade or two ago; enterprises run on Exchange/Outlook now.

<comment><USER12> I'm not sure if you're trying to be ironic or you really don't know that Exchange is just a (mostly) RFC compliant SMTP server and a (mostly) RFC compliant IMAP server rolled into one.

<comment><USER11> My point was that Exchange is not compliant enough to be usable from a non- Outlook email client. At my current job I run a windows VM just to run Outlook in it; at my previous^2 job I had a second physical machine for the same thing (though I eventually put mysql on it).

<comment><USER12> > My point was that Exchange is not compliant enough to be usable from a non- > Outlook email client.

Well, you kind of have a point there. I use Apple mail and iCal against our corporate Exchange, but when I have to schedule a meeting, I either have to log into the Windows terminal server or use the outlook web interface to do it reliably.

It would be nice if there were an RFC for calendaring protocols, even if that RFC was just codifying Exchange.

<comment><USER13> There is, isn't there?

<http://www.ietf.org/rfc/rfc2445.txt>

Google seems to attach an ics to all the cal events it sends out.

<comment><USER14> That is only a mechanism for encapsulating an event and giving it to someone else: it doesn't really help any of the actual use cases people have for the kinds of group calendar management that everyone is used to having from Exchange.

The more related standard that could have done this is ICAP, but that spec never happened and died as a draft back in 1998, long enough ago that the acronym got reused in 2003 by RFC 3507 for something unrelated to calendar access.

<http://tools.ietf.org/html/draft-oleary-icap-04>

Right now, it seems like all of the effort is behind CalDAV (which is based on WebDAV), which actually has some published RFCs behind it (dating back through 2007) and multiple clients (including Apple's iCal and Google Calendar).

<http://en.wikipedia.org/wiki/CalDAV>

<comment><USER13> I see, thanks for the info!

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER15> I believe GMail already does the chat <-> email conversation quite seamlessly

</comment><comment><USER9> I thought the inventor of Email was this guy: <http://www.inventorofemail.com/>

<comment><USER16> Reading that, it seems like there is no one inventor of email, but a bunch of different solutions, perhaps leading to the 1981 RFC for SMTP by Jonathan B. Postel, which I would guess is the beginning of the actual current email implementation:

<http://tools.ietf.org/html/rfc788></comment></comment>