TheNanomancer on Twitter - <AUTHOR> http://twitter.com/TheNanomancer <comment><USER2> The first page of stuff is all superficial and pointless. My favorite is the one that suggests leaving the "http" off URLs when typing them into the address bar, so that you enter ://example.com instead of <http://example.com>. I'd really like to know what idiot there didn't realize you don't need the :// either.

. . . and I just realized that for some reason HN eats a double-quote at the end of a URL, so I edited this to eliminate most of the double-quotes I used in the comment. I don't want a delimiter mismatch, after all.

</comment><comment><AUTHOR> The point of the :// thing is more for links and less for typing in an address bar. Some forums only allow so many characters in the signature and (say it is exampleforums.com), if you type in a url (say example.com) with out http, it will go to exampleforums.com/example.com. Thus a problem. If you type Example it saves characters and wont go to the wrong address.</comment>


WhisperMonitor for Android Released - Dynamic egress filtering Firewall - mike-cardwell http://www.whispersys.com/whispermonitor.html

<comment><AUTHOR> It's a shame it's a ROM flash, and not an app available for rooted phones. I don't think I'm willing to give up cyanogen's features.

</comment><comment><USER2> Why do I need to install WhisperCore to use WhisperMonitor? Can I turn off this disk encryption stuff to try out the firewall by itself?

</comment><comment><USER3> Interesting... I wonder what Lookout or Mobile Iron are doing in this space?</comment>


Automatically Encrypting all Incoming Email - mike-cardwell https://secure.grepular.com/Automatically_Encrypting_all_Incoming_Email

<comment> mike-cardwell If you get an SSL error when visiting this site it's because your browser/os combo doesn't have the cacert.org root installed. A number of heavily used combinations don't have it by default. Please don't start a thread about SSL here :)</comment>


Palantir, Tech's Next Big IPO, Lost $580M in 2019 - <AUTHOR> https://www.nytimes.com/2020/08/21/technology/palantir-ipo-580-million-loss.html <comment><USER2> Sales and marketing costs at 60% of revenues last year? My understanding of Palantir is that this isn't some SaaS model where a fixed cost product can be scaled at huge gross margins. This is a niche IT consultancy coming up on two decades of life. Today's stock market of hype dream valuations is their window to offload this to the public and clearly they know it.

<comment><USER3> While it's been years since I was in a location with the tech deployed, it was 100% a consulting first gig. Nothing was done without an Palantir Engineer. The sad thing is the extra expenses not accounted for in the contracts too. For example, the clearances / costs were often covered by host agencies (at an extremely steep price) for the analysts, something usually passed on to the vendor to cover. So in a sense, they paid for software, paid for an employee, and paid all the associated fees to clear / maintain the employee. I don't think this flys in the private sector at all and it shows with their limited client base.

<comment><USER4> All clearances are paid for by the government, they are never billed to the vendor. If they charged the contractors then small businesses would never be able to compete. Hell, medium businesses wouldn't be able to compete.

There is an estimated cost of said clearances, as needed so the government can try to budget accordingly, but it is not handed down to vendors.

<comment><USER3> I'm not sure I agree. Usually the Prime has negotiated it as part of their fee package, but there is most certainly fees for applying and granting clearances for third parties. There's even more fees for establishing a proper SSO and certifying as well. We had a default 50k cost barrier for any new TS/SCI candidates we put forth. Sometimes this was reimbursed, but it was most definitely a _good faith_ cost we incurred.

<comment><USER4> it would blow my mind if you tell me you know a company that has written a literal check for a security clearance in the last twenty years.

i suspect that cost barrier is more the government saying "we aren't paying for it, you need to find someone that is already cleared."

</comment></comment></comment></comment></comment><comment><USER5> I am yet to understand exactly what Palantir does.

<comment><USER6> From what I understand, it's essentially data engineering, and tools on top of data pipelines.

They claim to sell products, and those products probably form an underlying part of what clients are buying, but really they are paying for a developer to sit in their office and build ETL code to get data out of their million unmaintained databases into some vaguely coherent system, and then build reporting, views, and simple workflows on top of that data.

This doesn't sound like much, but I can understand why that might be transformational for many businesses, to finally have all their data in one place in a way that can be analysed usefully.

</comment></comment><comment><USER7> [https://www.theverge.com/2013/6/29/4478748/california- licens...](https://www.theverge.com/2013/6/29/4478748/california-license- plate-reader-database-palantir)

[https://www.theverge.com/2018/2/27/17054740/palantir- predict...](https://www.theverge.com/2018/2/27/17054740/palantir-predictive- policing-tool-new-orleans-nopd)

[https://theintercept.com/2017/03/02/palantir-provides-the- en...](https://theintercept.com/2017/03/02/palantir-provides-the-engine-for- donald-trumps-deportation-machine/)

[https://www.businessinsider.com/palantir-took-over-from- goog...](https://www.businessinsider.com/palantir-took-over-from-google-on- project-maven-2019-12)

I think we are, at least in part, responsible for the behaviors of the companies we own. Being a shareholder is not quite the same as being an LP.

</comment><comment><USER8> How did they have revenues of $740 million, expenses of slightly more than $1 billion, but lose $580 million?

<comment><USER9> Write-downs or similar

</comment></comment><comment><USER10> Isn't this sort of loss at this scale usually justified, for "product" companies, by saying they'll either run into economies of scale or drive everyone else out of the market, thus becoming profitable eventually?

Not even considering if that rationale will work out for certain venture- backed companies that _are_ more product-oriented, how on earth is this supposed to work out for a consulting company like Palantir that doesn't scale like SaaS & the likes?

</comment><comment><USER11> It staggers me, but losing this amount of money seems to have few negative impacts. Is it investor fomo saying "if others put in enough cash for them to stay afloat after losing $bajillion, they must know something I don't! Calling my trader"?

</comment><comment><USER12> this is interesting

[https://techcrunch.com/2020/08/21/leaked- palantir-s-1-shows-...](https://techcrunch.com/2020/08/21/leaked- palantir-s-1-shows-company-has-125-customers-after-17-years/)

</comment><comment><USER12> bubble is coming</comment>


A tale of two similar niche ad networks from the Web 1.0 and Web 2.0 eras - <AUTHOR> http://www.thestandard.com/news/2008/05/01/gay-ad-network-makes-river-small-streams-while-planetout-struggles

<comment><USER2> "Gay Ad Network makes a river from small streams." Now there's a loaded title if I ever saw one.

</comment><comment><USER3> CPMs on SFW gay sites are huge... there's nothing less than $3 CPM.</comment>


Ask HN: Which tech stack for macOS-App landing page? - <AUTHOR> I want to release an macOS-App via my own website. I want a nice responsive website which shows some static content (Videos, Images, Text) about the app. Additionaly I want to have a download area, where you can find all released versions with changelogs. This download area should be automatically generated and updated, when I push a new version to my repository on Github. Changelogs should be generated based on squashed commit messages.
Which tech stack would you prefer? For example is WP extendable enough to achieve described goal? I also wanted to learn Typescript and react in the past, but this might be an overkill.
Background: iOS-Developer with currently no serious experience in web development. <comment><USER2> shameless plug. I own sytescope.com which makes websites extremely easy. Only hard part you would need to figure out how to pull in the changlogs using the code block. It would be possible I just don't know if there is already a github script for it or not.

</comment><comment><USER3> Static HTML. Update download area with a simple scp from your build script.

<comment><USER4> And while you're at it, please include a version number in the DMG's download URL if you don't mind. That helps with integrity checks.

(Full disclosure: am Homebrew maintainer, who loves to add integrity checks to casks.)</comment></comment>


A Dark Room - <AUTHOR> http://adarkroom.doublespeakgames.com/ <comment><USER2> A nice simple game. Its text-based and timed format reminds me of [http://www.beepboopbitcoin.com/](http://www.beepboopbitcoin.com/) which are absolutely hilarious (at least for those very familiar with the Bitcoin space).</comment>


Seeking Big A.I. Advances, a Startup Turns to a Computer Chip - <AUTHOR> https://fortune.com/2019/08/19/ai-artificial-intelligence-cerebras-wafer-scale-chip/ <comment><AUTHOR> 400,000 cores, 1.5 kW - that's a big chip.</comment>


Bragi's Dash headphones are finally here - <AUTHOR> http://www.engadget.com/2016/01/05/bragi-dash-headphones-are-here/ <comment><USER2> Not shipping yet.</comment>


How does Tor bypass DNS - <AUTHOR>

Hello, I am new to Tor and how it works, but have a general understanding of how it works.
It uses a TLD of .onion, which as far as I know, is not a real TLD with regard to ICANN and the other IP/domain authorities.
This got me thinking how much power our root server holders have.  If they ever wanted to literally take out the internet, destroy our economy, and ruin anything that uses a name based lookup system — all they have to do is drop all the zones in the .com. or add in an .com. A IN 127.0.0.0.
But, Tor boots up and can* resolve the .onion TLD.  On my OS, Mac OS X, no entries are added to /etc/hosts, I can try to do lookups on .onion url's and they will fail.
How does Tor do the resolution of the .onion url scheme?  Does every tor node have it's own little DNS resolver in it, and when you use the Tor browser, it will connect up to a node and get instructions from there?  Does that mean that all nodes are essentially updating a large /etc/hosts file like we did ages ago when there was no DNS?
Thanks for any help understanding this.  It kind if blows me away how fragile and simple the internet is as far as dismantling it.  Things like BGP being human controlled, state controlled, etc, I am surprised that more countries are not a bit up in arms that we ( The USA ) sort of own the internet.  We allocated a TLD to other countries, we I assume can take that away.  I am often baffled other countries, ( China perhaps ) don't just start their own root servers, run their own BGP, and fully be in control of every domain, from .com all the way to .i-made-up-a-tld. <comment><USER2> Yes, every tor node contains a resolver. No, this doesn't mean that every node knows the .onion address of every hidden service; there is a distributed hash table involved. (Note: only hidden services have .onion addresses, not regular Tor nodes.) It is not at all like using /etc/hosts files.

The US does not control all of the DNS root servers. China does control its BGP, and puts more effort than any other country into running a centralized firewall, and even hijacking DNS queries coming from inside the country sometimes. The US doesn't arbitrarily revoke TLDs because it would be akin to an act of war, and universally condemned.

<comment><AUTHOR> Thanks for the explanation, I appreciate it.</comment></comment>


HTTP Made Simple - <AUTHOR> http://www.pandastrike.com/posts/20131211-http-made-simple

<comment><USER2> I'm following the series! Very informative.

<comment><USER3> good stuff</comment></comment>


Ask HN: To everybody who uses MapReduce: what problems do you solve? - <AUTHOR> <comment><USER2> A large telco has a 600 node cluster of powerful hardware. They barely use it. Moving Big Data around is hard. Managing is harder.

A lot of people fail to understand the overheads and limitations of this kind of architecture. Or how hard it is to program, especially considering salaries for this skyrocketed. More often than not a couple of large 1TB SSD PCIe and a lot of RAM can handle your "big" data problem.

Before doing any Map/Reduce (or equivalent), please I beg you to check out Introduction to Data Science at Coursera [https://www.coursera.org/course/datasci](https://www.coursera.org/course/datasci)

<comment><USER3> > Moving Big Data around is hard.

I never had any issues with Hadoop. Took about 2 days for me to familiarize myself with it and adhoc a script to do the staging and setup the local functions processing the data.

I really would like to understand what you consider "hard" about Hadoop or managing a cluster. It's pretty straight forward idea, architecture is dead simple, requiring no specialized hardware at any level. Anyone who is familiar with linux CLI and running a dyanamic website should be able to grok it easily, imho.

Then again, I come from the /. crowd, so YC isn't really my kind of people, generally.

<comment><USER4> > _Then again, I come from the /. crowd, so YC isn't really my kind of people, > generally._

You sound like a snob.

<comment><USER3> See what I mean?

</comment></comment></comment></comment><comment><USER5> We had been using hadoop+hive+mr to run targetting expressions over billions of time series events from users.

But we have recently moved a lot back to mysql+tokudb+sql which can compress the data well and keep it to just a few terrabytes.

Seems we weren't big data enough and we were tired of the execution times, although impala and fb's newly released presto might also have fitted.

Add: down voters can explain their problem with this data point?

<comment><USER6> This doesn't answer the question - you described /how/ you solved some problem, not what problem you're actually solving.

(Mind you, at my writing this is the top comment, so I don't think you're getting many downvotes. But your comment irked me, so there you go.)

<comment><USER7> > not what problem you're actually solving.

> targetting expressions over billions of time series events from users.

<comment><USER8> You moved back to MySQL.

What was the motivation to move to Map Reduce in the first place if a well understood technology, MySQL, works fine?

(sorry if I am posting a lot on this topic. I am really interested in finding answers rather than trying to prove any point that relational databases are better in case anyone thinks otherwise).

<comment><USER5> (Poster) we moved from mysql+innodb/myisam to hadoop because of performance problems. We did test and evaluate hadoop, and then jumped. Then tokudb comes along (technically we moved back to mariadb) and puts performance advantage firmly back on mysql's side. I imagine impala and presto and any other column based, non-map-reduce engines would give tokudb a fairer fight though.

</comment></comment></comment></comment></comment><comment><USER9> This system isn't in production just yet, but should be shortly. We're parsing Dota2 replays and generating statistics and visualisation data from them, which can then be used by casters and analysts for tournaments, and players. The replay file format breaks the game down into 1 min chunks, which are the natural thing to iterate over.

Before someone comes along and says "this isn't big data!", I know. It's medium data at best. However, we are bound by CPU in a big way, so between throwing more cores at the problem and rewriting everything we can in C, we think we can reduce processing times to an acceptable point (currently about ~4 mins, hoping to hit <30s).

<comment><USER10> This sounds like something you could just do in SQL and have it all done in milliseconds.

<comment><USER9> For our highly unstructured, untyped, non relational artefacts? It'd be fairly impossible to use it as a datastore in this particular case, and regardless, it would provide little to no speed increase over our current application, as the limiting factor is the CPU cost of the map function.

<comment><USER2> Maybe you should consider transforming the data to structure it a bit. Have a unique identifier per object and arrays for each seen characteristic with (value, id), and it's reverse index. Then decompose the processing of each object to sub-problems matching n-way the characteristics. It doesn't have to be in SQL, though. I'd try a columnar RDBMS. YMMV

</comment></comment></comment></comment><comment><USER11> We use Elastic MapReduce at Snowplow to validate and enrich raw user events (collected in S3 from web, Lua, Arduino etc clients) into "full fat" schema'ed Snowplow events containing geo-IP information, referer attribution etc. We then load those events from S3 into Redshift and Postgres.

So we are solving the problem of processing raw user behavioural data at scale using MapReduce.

All of our MapReduce code is written in Scalding, which is a Scala DSL on top of Cascading, which is an ETL/query framework for Hadoop. You can check out our MapReduce code here:

[https://github.com/snowplow/snowplow/tree/master/3-enrich/ha...](https://github.com/snowplow/snowplow/tree/master/3-enrich/hadoop- etl)

<comment><USER12> thanks for this. snowplow has been an amazing source of learning. I'm quite interested in the etl process than in the actual mapreduce.

have you seen your etl used to pull data from Twitter or Facebook. I am wondering what is the state of art there considering throttling, etc.

<comment><USER11> Hi sandGorgon! Thanks for the encouraging words. We haven't yet seen people use the existing Scalding ETL to pull data from Twitter or Facebook. As you suggest, there are some considerations around using Hadoop to access web APIs without getting throttled/banned. Here's a couple of links which might be helpful:

\- [http://stackoverflow.com/questions/6206105/running-web- fetch...](http://stackoverflow.com/questions/6206105/running-web-fetches-from- within-a-hadoop-cluster)

\- [http://petewarden.com/2011/05/02/using-hadoop-with- external-...](http://petewarden.com/2011/05/02/using-hadoop-with-external-api- calls/)

I think a gatekeeper service could make sense; or alternatively you could write something which runs prior to your MapReduce job and e.g. just loads your results into HDFS/HBase, for the MapReduce to then lookup into. Akka or maybe Storm could be choices here.

We have done a small prototype project to pull data out of Twitter & Facebook - that was only a Python/RDS pilot, but it gave us some ideas for how a proper social feed into Snowplow could work.

<comment><USER12> is your python code part of the snowplow repository/gist?

it would be very interesting to take a look at it.

</comment></comment></comment></comment><comment><USER13> It's worth noting that CouchDB is using map-reduce to define materialized views. Whereas normally MR parallelization is used to scale out, in this case it's used instead to allow incremental updates for the materialized views, which is to say _incremental updates for arbitrarily defined indexes_! By contrast SQL databases allow incremental updates only for indexes whose definition is well understood by the database engine. I found this to be pretty clever.

<comment><USER14> I've been using CouchDB (and now BigCouch) for about four years and it's both clever and useful. We're storing engineering documents (as attachments) and using map/reduce (CouchDB views) to segment the documents by the metadata stored in the fields. The only downside is that adding a view with trillions of rows can take quite a while.

<comment><USER13> Strangely, there aren't many discussions of Couch* family on hacker news. Do you know why would that be?

I'm thinking about basing a new product around couchbase lite, but lack of popular acceptance is one of the things holding me back.

<comment><USER14> I'd be happy to take the discussion off-line if you want to get into more depth but I have to wonder if the merger of Memcached and CouchDB happened right as CouchDB would have made a name for itself. Right around that time, I think CouchDB and Mongo were equally well-known.

</comment></comment></comment></comment><comment><USER15> I'll tell you where it's not used: High Energy Physics. We use a workflow engine/scheduler to run jobs over a few thousand nodes at several different locations/batch systems in the world.

If processing latencies don't matter much, it's an easier more flexible system to use.

<comment><USER16> At least on the experimental LHC side, we process/analyse each event independently from every other event, so it's an embarrassingly parallel workload. All we do is split our input dataset up into N files, run N jobs, combine the N outputs.

Because we have so much data (of the order of 25+ PB of raw data per year; it actually balloons to much more than this due to copies in many slightly different formats) and so many users (several thousand physicists on LHC experiments) that's why we have hundreds of GRID sites across the world. The scheduler sends your jobs to sites where the data is located. The output can then be transferred back via various academic/research internet networks.

HEP also tends to invent many of its own 'large-scale computing' solutions. For example most sites tend to use Condor[1] as the batch system, dcache[2] as the distributed storage system, XRootD[3] as the file access protocol, GridFTP[4] as the file transfer protocol. I know there are some sites that use Lustre but it's pretty uncommon.

[1] [http://research.cs.wisc.edu/htcondor/](http://research.cs.wisc.edu/htcondor/) [2] [http://www.dcache.org/](http://www.dcache.org/) [3] [http://xrootd.slac.stanford.edu/](http://xrootd.slac.stanford.edu/) [4] [http://www.globus.org/toolkit/docs/latest- stable/gridftp/](http://www.globus.org/toolkit/docs/latest-stable/gridftp/)

<comment><USER17> I remember when HEP invented that WWW technology stuff which turned out to be rather popular outside of HEP as well.

</comment></comment></comment><comment><USER18> This is a system in early development, but my research group is planning on using MapReduce for each iteration of a MCMC algorithm to infer latent characteristics for 70TB of astronomical images. Far too much to store on one node. Planning on using something like PySpark as the MapReduce framework.

</comment><comment><USER19> How strongly does a RNA binding protein bind to each possible sequence of RNA?

<comment><USER8> How big is your data?

Yours is the first example where I a decent knowledge of the field, so can understand the needs accurately. In most cases I see people using NoSQL in places where MySQL could handle it easily.

Maybe I am just too set in my relational database ways of thinking (having used them for 13 years), but there are few cases where I see NoSQL solutions being beneficial, other than for ease of use(most people are not running Facebook or Google).

I a bit sceptical in most cases (though I would certainly like to know where they are appropriate).

<comment><USER20> First, I should note that my needs are fairly specific, and not typical of the rest of the NGS world. The datasets are essentially the same though.

The rate at which we are acquiring new data has been accelerating, but each of our Illumina datasets is only 30GB or so. The total accumulated data is still just a few TB. The real imperative for using MR is more about the processing of that data. Integrating HMMER, for instance, into Postgres wouldn't be impossible, but I don't know of anything that's available now.

Edit: A FDW for PostgreSQL around HMMER just made my to do list.

<comment><USER8> So is it fair to say it is an "ease of use" use case?

<comment><USER20> Is that the same as an 'impossible to do otherwise' case?

Edit: I should say 'currently impossible' since as I noted, I can imagine being able to build SQL queries around PSSM comparisons and the like. I just can't build a system to last 5+ years around something that _might_ be available at some point.

Since I can't reply directly- agreed :)

<comment><USER8> That comment was based on your "into Postgres wouldn't be impossible" phrase.

No fair enough of its the only way you can get things to work. I still see lots of people jumping on the "big data" bandwagon with very moderate sized data.

</comment></comment></comment></comment></comment></comment><comment><USER21> A host of batch jobs that source data from up to 28 different systems and then apply business rules to extract a substrate of useful data.

</comment><comment><USER22> We use MR using Pig (data in cassandra/CFS) with a 6 node hadoop cluster to process timeseries data. The events contain user metrics like which view was tapped, user behavior, search result, clicks etc.

We process these events to use it downstream for our search relevancy, internal metrics, see top products.

We did this on mysql for a long time but things went really slow. We could have optimized mysql for performance but cassandra was an easier way to go about it and it works for us for now.

<comment><USER23> Can you estimate how much faster your processing is now vs. before on MySQL? I find it interesting that your cluster is only 6 nodes - relatively small compared to what I've seen and from what I've read about. It'd be interesting to know the benefits of small-scale usage of big data tech.

</comment></comment><comment><USER24> "How do I make all my projects take 10x longer"

</comment><comment><USER25> While the push-back against Map/Reduce & "Big Data" in general is totally valid, it's important to put it into context.

In 2007-2010, when Hadoop first started to gain momentum it was very useful because disk sizes were smaller, 64 bit machines weren't ubiquitous and (perhaps most importantly) SSDs were insanely expensive for anything more than tiny amounts of data.

That meant if you had more than a couple of terabytes of data you either invested in a SAN, or you started looking at ways to split your data across multiple machines.

HDFS grew out of those constraints, and once you have data distributed like that, with each machine having a decently powerful CPU as well, Map/Reduce is a sensible way of dealing with it.

<comment><USER26> It still is a valid pattern. As disks get bigger, the harder it is to move the data and the more efficient it is to move the compute to the data.

</comment></comment><comment><USER27> Generating web traffic summaries from nginx logs for a CDN with 150 servers, 10-15 billion hits/day. Summaries then stored in MySQL/TokuDB.

</comment><comment><USER28> MapReduce is great for ETL problems where there is a large mass of data and you want to filter and summarize it.

<comment><USER29> Yup. This. Once I filter down to the data I actually care about, I typically find I'm no longer anywhere near "Big Data" size.

</comment></comment><comment><USER30> MongoDB: where you need Map/Reduce to do any aggregation at all.

<comment><USER31> You may consider using Mongo's aggregation framework[1] instead which is zillion times faster[2].

[1] [http://docs.mongodb.org/manual/aggregation/](http://docs.mongodb.org/manual/aggregation/)

[2] [http://stackoverflow.com/questions/13908438/is-mongodb- aggre...](http://stackoverflow.com/questions/13908438/is-mongodb-aggregation- framework-faster-than-map-reduce)

</comment></comment><comment><USER32> A lot of people in this thread are saying that most data is not big enough for MapReduce. I use Hadoop on a single node for ~20GB of data because it is an excellent utility for sorting and grouping data, not because of its size.

What should I be using instead?

<comment><USER33> Obviously you should throw out the solution that worked for you and start over. 20GB just isn't cool enough to use M/R.

</comment></comment><comment><USER34> To most people who use MapReduce in a cluster: You probably don't need to use MapReduce. You are either vastly overstating the amount of data you are dealing with and the complexity of what you need to do with that data, or you are vastly understating the amount of computational power a single node actually has. Either way, see how fast you can do it on a single machine before trying to run it on a cluster.

</comment><comment><USER35> We use our own highly customized fork of Hadoop to generate traffic graphs [1] and demographic information for hundreds of thousands of sites from petabytes of data, as well as building predictive models that power targeted display advertising.

[1]: [https://www.quantcast.com/tumblr.com](https://www.quantcast.com/tumblr.com)

</comment><comment><USER36> We are using Map/Reduce to analyze raw XML as well as event activity streams, for example analyzing a collection of events and meta data to understand how discreet events relate to each other as well as patterns leading to certain outcomes. I am primarily using Ruby+Wukong via the Hadoop-Streaming interface as well as Hive to analyze output and for more normalized data problems.

The company is a large Fortune 500 P&C insurer and has a small (30 node) Cloudera 4 based cluster in heavy use by different R&D, analytic and technology groups within the company. Those other groups use a variety of toolsets in the environment, I know of Python, R, Java, Pig, Hive, Ruby in use as well as more traditional tools on the periphery in the BI and R&D spaces such as Microstrategy, Ab Initio, SAS, etc.

</comment><comment><USER37> I was using it on a D3.JS chart to aggregate data flow though our custom real- time analytic pipeline.

<comment><USER38> Will u mind sharing some more info on this real time analytics using d3 as i am also planning to do something like tht..

<comment><USER39> look up crossfilter.js

</comment></comment></comment><comment><USER40> Validating performance testing simulations. Tie the inputs of the load generators to the outputs from the application server logs and verify the system works as designed at scale.

</comment><comment><USER41> Taking dumps of analytics logs and pulling out relevant info for our customers on app usage

<comment><USER42> This is the `grep/awk` use case. The nice thing about streaming mr interface to hadoop (calling external programs) is that you can literally take your grep/awk workflow and move it to the cluster. Retaining line oriented records is a huge step in having a portable data processing workflow.

</comment></comment><comment><USER43> Obviously, in today's theme, Facebook is _not_ using mapreduce effectively to figure out who of our "friends" we actually care about :)

<comment><USER44> Could you be any more edgy?

</comment></comment><comment><USER45> Saw this tweet today [https://twitter.com/Andst7/status/399333514803294208/photo/1](https://twitter.com/Andst7/status/399333514803294208/photo/1)

</comment><comment><USER46> "Things that could be trivially done in SQL :(" We use HIVE over HDFS. Sure the type of things we are doing could have been done in SQL, if we had carefully curated our data set and chosen which parts to keep and which to throw away. Unfortunately, we are greedy in the storage phase. Hive allows us to save much more than we reasonably would need, which actually is great when we need to get it long after the insert was made.

</comment><comment><USER47> Airbnb uses distributed computing to analyze the huge amount of data that it generates every day[1]. The results are used for all sorts of things: assessing how Airbnb affects the local economy (for government relations), optimizing user growth, etc.

[1] [http://nerds.airbnb.com/distributed-computing-at- airbnb/](http://nerds.airbnb.com/distributed-computing-at-airbnb/)

</comment><comment><USER48> Billion of metric values.

</comment><comment><USER42> Image restoration, OCR, face detection, full text indexing. Mostly just a parallel job scheduler.

</comment><comment><USER49> It would be nice if someone collected/counted the actual answers. I read the whole thread and "analyzing server logs" was the only answer. If you don't have funding or have dreams/plans, you're not really _using_ the technology.

</comment><comment><USER50> Just FYI: Even small problems can be solved using the concept of MapReduce. It is a concept and not an implementation that is tied to BigData and NoSQL. A simple example would be MergeSort. It uses the concept of MR to sort data.

</comment><comment><USER51> Things that could be trivially done in SQL :(

</comment><comment><USER52> Process a huge amount of spanish text in paper surveys

</comment><comment><USER53> "How can I make Google shareholders richer?"

</comment><comment><USER54> Data aggregation to calculate analytics.

</comment><comment><USER55> Billions

</comment><comment><USER56> How can I make myself feel better about what I do by trying to diminish the work other people do using the same technologies?

</comment><comment><USER57> How much can I save in my mobile phone and plan?</comment>


Ask HN: Most legible/clean body fonts for websites? - <AUTHOR> We're about to re-launch our website. Looking to understand the current landscape on which body font to use. <comment><USER2> None.

Reason: Every user will have a different opinion of which font is most legible/clean. Therefore, no one font selected by you will meet every users need. Therefore, do not set any font, and allow the user's chosen preferred font, as configured into their browser, determine what font is used on your site.</comment>


The Starter, the Architect, the Debugger and the Finisher - <AUTHOR> http://jacquesmattheij.com/The+Starter+the+Architect+the+Debugger+and+the+Finisher <comment><USER2> I believe that in order to get really good at any of these roles, you must do the other roles, deeply and often. For example:

The more time you spend Debugging shit, the less likely to you are to Architect something that produces shit.

The more time you spend fixing a million little things caused by poor early decisions, the better Starter you'll become.

The more fun you have conceiving and Starting projects, the more you'll realize how important Finishing is, so that you can get to new stuff.

And the more time you spend doing each of these roles, the better you'll get at doing all of them at the same time, and understanding what conditions are needed to do that. (Hint: Lots of quicker smaller complementary projects that you can wrap your whole head around.)

[This whole discussion reminds me of the time I was a restaurant manager and got tired of the servers and cooks bitching at each other. I had them switch roles for one shift. Once they understood how what they did affected the other, everyone got a little better and the bitching stopped.]

<comment><USER3> The more time I spend Debugging shit the less likely I am to Architect or even Start anything because I am painfully aware that everything produces shit.

EDIT: Wow. A downvote. That what I get for showing the pain I experience to the world. :-)

<comment><USER4> So what you are saying is - you write better code, reduce lines of code, don't be too clever, reduce complexity, and use well-proven libraries.

<comment><USER5> I think what he's saying is, after all that, you're still probably producing shit, that no matter how awesome you are, whoever ends up tending to your pet Frankenstein after you've moved on is probably still going to spend half their time cursing your existence, if they weren't part of the project from the beginning.

And that's only if you're good enough in the first place so that "finishing" is actually worth the effort, rather than starting from scratch using your code as the blueprint for the new and improved 2.0 spec. This requires not only that you be good, but that you get lucky: that you've accurately predicted the stack that your ops team will be pushing to standardize on in 2 years, that there's enough expertise in the language(s) you used to justify its continued use, that management hasn't decided to switch everything to Windows + .NET because "support", that every one of the OSS libs that you've pulled in all happen to be licensed under the three particular OSS licenses that legal decided to approve after that recent switch to a "whitelist" policy, that the company services you were forced to integrate with when you started haven't been shitbinned, that the new lead developer on the project doesn't have aesthetic problems with the way your code is organized, and so on. All of which usually mean "rewrite!", and few of which you have much control over. At the end of the day, it doesn't matter whether it's your fault or not, the people evaluating your work will still consider it "shit" because it's not usable anymore, and nobody loves the project enough to bother figuring out how to patch it up.

IMO, though, the _real_ reason we all tend to produce shit (and then throw it out later) is actually missing from this otherwise excellent four-part breakdown: documentation. And I'm talking real, detailed, explicitly human- centric documentation that's more than just class/function level comments or a user manual. Lately I've been dealing with several attempted tech consolidation tasks as part of a several-tier acquisition (big fish swallows little fish, bigger fish swallows entire lake), and it seems that no matter what size the company is, nobody in software _ever fucking documents anything_ well enough to reuse it, even when they're developing services that are explicitly intended to be consumed by other groups. The resulting duplication of effort is ridiculous.

Good internal documentation needs to start with the starter, end with the finisher, and be taken seriously by everyone in between. It should be considered almost as important to a project as the code itself. It needs to explain how to use the system, how to integrate with it, how to debug common problems, where the pain points are, how the system is architected, why certain decisions (especially tradeoffs!) were made, and more. Done right, good documentation can make up for some pretty bad code, and make sure that a project is actually maintained rather than thrown away.

<comment><USER6> Spot on—wish I could upvote each sentence of your comment.

Everyone knows code readability counts, because it's "read much more often than it is written". Same on a larger scale—the project is built once, and then maintained for years and decades (especially in "enterprise" environment). So you should optimize for maintenance—that is, write docs.

It really puts me out that no one else in the team or management appears to take documentation seriously. Is this 'job security' or general negligence, I don't know. I think either your primary concern is project success, or it is your job and money. If it's the latter, then of course why would you write docs—just bang out something that works for client/employer, you'll get paid and it'll be harder to fire you.

Or maybe it's management problem really and I shouldn't care.

<comment><USER7> I think it's a short-term vs. long-term problem: everyone's focused on hitting their milestones and the problems that will be caused by the bad documentation won't manifest themselves until later. It's kind of a "technical debt" situation.

</comment></comment></comment></comment></comment></comment><comment><USER8> Nice article, but IMO it's essentially a software-specific version of the Belbin Roles[1].

[1] <http://en.wikipedia.org/wiki/Team_Role_Inventories>

        Starter -> Resource Investigator
        Architect -> Shaper
        Debugger -> Teamworker
        Finisher -> Completer Finisher

Roughly.

What I miss most in Jacques' article is the Plant, the person who thinks things over and then comes forward with problems, issues and risks in the chosen approach. It's, fortunately, a very common trait among SW engineers though, so I guess every team has at least a bunch.

<comment><USER9> A similar system is discussed for CEOs as well. The entrepreneur who starts a company is not always a great consolidator who gets everything settled and organised into a functioning whole once the money starts coming in. The next two I think are substantial growth (being able to take an established business and grow it into a dominant player in its industry) and the last I think is something like holding, being able to take a large (and usually somewhat unwieldy) business and keep it from losing ground.

There's probably a few variations on these as well (and I might be misremembering them slightly).

</comment></comment><comment><USER10> "The debugger couldn't be paid enough money to start from a blank page in order to get something new started."

Perhaps it's different in S.V. but in the workaday world of the code mines I've found that the Debugger doesn't get all that appreciated. Rather, management likes the starter since he seems so productive... after all, he just did the 20% of the work that got us 80% of the way there. They just wish he could somehow do 25% and get 100% of the way there.

<comment><USER11> From my experience as an enterprise coder I saw a management that didn't understand that there were bugs. Everything was supposedly done with Scrum, so at the end of each two-week sprint there were supposed to be no more bugs left on that feature. As far as the schedule was concerned (and we were scheduled over a year in advance) bugs didn't exist.

When bugs were found that the engineering team had to fix, we were either supposed to do it in addition to our other tasks. And any major bug required a huge rescheduling meeting to figure out how we can still make our schedule now that we had lost 20 engineer-hours.

Also, with their focus on "individual ownership" they expected everyone to take a feature, implement it, and fix all the bugs without involving the other engineers.

What this lead to was a lot of buggy code, a lot of obvious bugs that didn't get fixed and we just crossed our fingers over, and a lot of people acting as starters even though we had and needed more finishers, debuggers, and architects.

</comment></comment><comment><USER12> I find it very hard to characterize myself, I constantly struggle with it. I enjoy starting things and don't like finishing/maintainging other peoples code especially if I cannot see their philosophy.

When trying to figure out where my limits are I found out another characterization:

\- fastness

\- deepness

\- input bandwith

\- memory (forgetfulness)

Compared to an average person I am good enough in all of these, but compared to A players I am bad in all of them except deepness. Compared to the very best I am slow to learn a new technology, slow to solve problems on the whiteboard in real-time, I have to re-learn things because I have partly forgotten them, and I am not especially fast in learning other people's theorems, proofs, and algorithms. I ocnstatnly find lots of stuff bullsiht and I constantly question basic beliefs, so I am slow at processing outer information. What I am quite good at is getting a challenging task and thinking about it for a lot of time refactoring my thoughts hundreds of times until I come to interesting insigths. Not that I am that good at it, but I have huge patience for this, because this is what I enjoy. It is more than enjoying this: my brain needs this as a drog. My brain pretty much likes to be detached from the outer world for long-long sessions.:)

When solving easy tasks this does not come out. But when pushing my limits I experience these weaknesses / strongness.

<comment><USER11> Would you please expand on your four characterizations, explaining what this quality is, what it makes someone good at, and maybe an example persona that exemplifies it?

The four adjectives don't immediately create images in my mind of what qualities they represent.

<comment><USER12> I try to expand on the 4 metrics I came up with:

fastness:

This can be measured by giving the person a not too hard (but not completely trivial) algorithmization problem, and see whether he/she can solve it in 15 minutes. Someone 'fast' is usually very good at competitions or at interviews excercised by companies where interviews are quite intense like Google.

deepness:

This basically means that given a much harder problem than the previous one, how the person solves it given lots of time. This is different than the previous one. Can he come up with a quite original "theory"/(thought system)? How fast a ray tracer he can write in 3 months? How good domain specific language he can come up with for a tricky domain in 2 weeks? (vs. writing the implementation for the Dijstra algorithm in 30 minutes without mistakes, which is 'fastness')

\- input bandwith

How fast you can read a reasonably dense technical/algorithms/math book. How fast you understand other people's nontrivial ideas? This is not totally the same thing as how fast you come up with an idea, or how good an idea you come up with yourself.

\- memory (forgetfulness)

How big a system / requirements spec. document can you hold in your head? If you used C++ 3 years ago, do you need to re-read the book 'The C++ Programming language' or you remember more or less everything? If you look at your previous comments on Hacker's News, are you suprised by some information in your own comments or you remember more or less everything?

<comment><USER11> This makes more sense. Thanks for the explanation. I like "fastness" and "deepness". I've definitely noticed a difference between people who are best at getting things done under time constraints but procrastinate anything longer (myself) and people who will do a great job if given enough time but which choke under pressure (my wife). I never really tried to name the two though.

Your system almost sounds like the two sides of two different things. On the one hand you have the ability to get things done: the ability to get a lot done quickly and on-the-spot and the ability to get things done meticulously and precisely if given the time. On the other hand you have the ability to learn things quickly and precisely remember them.

I might call it velocity vs. precision in the two areas of production and learning.

</comment></comment></comment></comment><comment><USER13> When I worked at MSFT, we used to break up (senior) people according to which stage of the project they were better at leading, either as technical ICs or managers. If you chunk projects into four parts: early, build-up, build-out, and release (not the words we used --- we just used numbers 1-4), almost all senior people tend to do best at some close span of two of them. I was personally frequently slotted into 1/2 roles, but I knew lots of others who fit more into 3/4 roles. It's a mixture of personality type and skillset.

Few things can doom a project more quickly than putting a "shipper" in charge of an early-stage project or vice-versa. You end up with management that sends all the wrong signals to the team and everybody can tell is looking forward to some later stage.

Of course, for junior people, it's important to just make sure they get through all of them. Not just to pick up the skills mentioned by other commenters here, but also to see which they're good at for when you want to stretch them with a leadership role, without setting them up for an avoidable failure.

</comment><comment><USER14> Personally I'm a starter. Great at implementing prototypes, getting things working fast, but absolutely terrible with polishing things and taking care of edge cases.

And I _hate_ it. I consider it a personality flaw, a flaw in my work ethics, and so on. Whatever it is, I don't want to have it.

What I've found has helped me get beyond this problem is taking on freelancing gigs where I'm mostly the guy who gets a rough prototype and has to make it work. The beauty of this is that I know how to think like a prototyper, so I can become productive on a foreign codebase quickly.

And because I started at a different goalpost, I can still work as a "starter" even though I'm doing the job of a "debugger/finisher".

<comment><USER15> > I consider it a personality flaw, a flaw in my work ethics, and so on.

I'm curious what percentage of software developers self-identify as finishers and what percentage as starters.

My assumption is that starters are much more common--but maybe that's because I assume everyone is just like me.

Are there really software developers who fit the "finisher" description? And prefer it? :)

If it turns out that software developers really do have a range of roles available to them and each individual suits some more than others then maybe it's best to work to accept it rather than feel guilty about it?

<comment><USER16> > My assumption is that starters are much more common--but maybe that's > because I assume everyone is just like me.

If finishers are less common than starters, shouldn't they be payed more? Are they being payed more? Is it a flaw in the market?

<comment> T-hawk They are. The debuggers and finishers are the ones making $100k for day-to-day Java maintenance at the big banks and insurers and retailers and such. For every starter who's succeeded in their own business or risen to a $200k director job in CorpWorld, there's four other starters who are subsisting on someone else's capital and haven't launched anything for real income yet.

That said, pay is based on perception as much as results. If a starter can sell someone on his vision of a $50M revenue payment processing platform, there's a good chance he can convince someone to pay him a small fraction of that like $500k, at least until the project dies off. Finishers can't do that while shooting one bug or fixing one feature at a time.

</comment></comment></comment></comment><comment><USER17> One important thing to note is that you can be good in all of the roles but not be able to do all of them on the same project. For example, I find personally that I have a very difficult time being a finisher and starter on the same project -- after a certain point exhaustion takes over. Also, I love to debug things, but, again, after you've spent countless hours debugging you might run into the problem of exhaustion.

This is, among other reasons, why I like the YCombinator emphasis on co- founders and teams. Even if you are a superstar you need people around you to work at maximum efficiency (speaking for myself at least).

Also, I'd like to applaud the backstage people (accounts, etc.) that make other things possible. Even when I do reasonably well in all of the roles mentioned in this article, I absolutely fail in the paperwork department.

</comment><comment><USER18> I am opposed to the concept of having a architect that does not code. Too often have I seen the situation where they recommend a solution without understanding the complexity of the implementation.

</comment><comment><USER19> When you put it in sequence like that it seems like the Starter & Finisher are far removed from each other, but I find the opposite to be true. What these personality types has in common is an appreciation for beauty. The Starter just loves beautiful _ideas_ while the Finisher loves beautiful user-faceing _implementations_.

</comment><comment><USER20> My first job in embedded systems required everyone to do pretty much all of these things. Each developer handled their own modules from user facing UI through to the HW (more or less). We had to architect and design things before implementing them (No one checked.. and it wasn't enforced... it's just the way the culture works) we then became individually responsible for implementation and bug fixing until the QA dept said it was A+.

After that there was a module review by 5 randomly selected engineers that would tell you what parts of the module was a messy hack and make you go fix it. I miss this culture.

</comment><comment><USER21> > The Starter, the Architect, the Debugger, the Finisher

and the Stereotype.

These articles aren't very informative if they don't have some new insight into how to maximize the output of the team by exploiting the traits of the stereotype. The roles outlined in this entry really only fall into two categories: get things going (starter & architect) and get things working (debugger & finisher).

If I had to take a wild guess, I'd say that what the article is really trying to say is that it's easy for people to start projects (I've yet to meet a "debugger" that can't start their own), but it's hard to complete them. It's often even harder to keep something working than when it was put together given the nature of changing requirements.

So my take on the article would be to add the advice: if you consider yourself a "starter" or an "architect", go live in the world of maintenance for a while. Learn to complete your projects. And if people tend to curse a project when your name was on the design doc, perhaps you should spend a bit more time learning about practical programming, design, and algorithms... or mentor with someone who is well regarded.

Next week's article: What happens when the boss is a Sagittarius and the team lead is a Gemini?

<comment><USER22> > What happens when the boss is a Sagittarius and the team lead is a Gemini?

I don't know, will you write it?

If you missed the message on what you could do to 'meximize the output of the team', without taking a wild guess, it is there in the next-to-last paragraph.

Whether that's a novel insight or not is up to you, it is novel enough to me that I chose to write about it. I'm still trying to work around my tendency to start stuff and not finish stuff and having been tossed into the role of debugger/finisher lately has helped me to be both better at starting stuff _and_ to get more appreciation for the people that work on later stage code.

<comment><USER21> I did catch your conclusions in the penultimate paragraph. However, I don't believe that "assigning [a] quality control role to the next person downstream" is a workable solution. This is a form of passing the buck, saddling someone else with your problems, etc.

In your reply here, you say that you've learned that by working as a debugger/finisher you've learned how to start stuff better. _That_ is worthy of further exposition as so many people suffer from the exact same problem.

And as much as false categorization & naming hurts your best point, it sure seems to have caught HN's attention, so marketing may be where your true genius lies. That's cool -- there's lots of money in marketing :)

<comment><USER22> > This is a form of passing the buck, saddling someone else with your > problems, etc.

No, you _still_ have to fix it yourself. But at least there is a feedback element.

</comment></comment></comment></comment><comment><USER23> I associate myself mostly as an Architect first, Starter second.

There is nothing more than I'd like for someone to just take some designs I've drawn up, critique if necessary, and then get it done. Generally, I get so stuck in the big picture that I have great difficulty getting anything done :(

I CAN debug... often I can identify a problem just from a general description of the problem (when you see me sit down in front of a computer and take over, it's when the problem has dug in deep). Sadly, with all the maintenance work I have to do with some legacy systems, debugging them is not a task I take to fondly.

Definitely not a finisher... I'm a sprinter, not a marathon runner.

</comment><comment><USER24> Often times, The Starter and The Debugger are the same person - the person who approaches programming challenges like affronts to their ability. Once the hard problems are solved, it's just a bit of tidying up and you're set.

I find it very difficult to be a Finisher, and I think many programmers do. Finishing isn't fun, it's not glamorous, and it's not why we do the work, but it's a skill that we need to develop.

I've noticed a trend in my own development. When I first start a project or I'm working on a hard problem, I'm working 10-12 hour days figuring out the interfaces, making the object model pristine (or as pristine as it can be in the language I'm using), making the error messages helpful and the exception handling consistent.

And then, as I'm closing out the project, I start to lose focus. I start watching the clock. I'm out of work as fast as I can. I'm doing, for lack of a better word, the bitch work, but the bitch work is what makes the system.

I've tried to get better at it, but I think there is a fundamental issue with the project lifecycle that makes human beings phone it in in the last bits of a project. Whether it's building a house or writing an application, those last bits of the project seem the most arduous.

</comment><comment><USER25> It would be interesting to run an HN poll: Which of these rolls are you mostly associated with?

My guess is the population here is dominated by Starters and Architects.

</comment><comment><USER26> I think the actual explanation is this:

1) Generally managers still don't understand the concept of agile and early release and so try to cram as many features, bug fixes amd details into each release as they can think of.

2) There is no possible way that one programmer can take care of all of the bugs, extraneous little features and tasks that the manager was able to think of.

3) Therefore the manager must come up with a division of labor and a simple categorization such as suggested by this article is one of most obvious and is probably attractive to a lot of senior developers because it means they don't have to worry about as many tedious tasks which they know are unlikely to provide real business value.

I think that most programmers with a decent amount of experience don't really fit into any particular one of those boxes because they have done all of those things themselves for one or more projects.

</comment><comment><USER27> I consider myself a good starter and debugger, I really suck at finishing, i hate all the little polish bits that are left over to do.

I find it easier to be the finisher if i didnt start the project, perhaps i should partner with someone with the same problem and just swap our projects once we get to the finisher stage.

<comment><USER28> I also consider myself to be a starter and debugger. I have _dozens_ of half- baked projects in the project graveyard on my hard drive.

I think part of my problem is what's described in the article - I start the project, solve the hard problem, and then move on to learn something else.

Any finishers out there have any tips? What drives you through the slog of the last 10%?

</comment></comment><comment><USER29> I've found the reasons why I sometimes don't want to wear all four of these hats are mainly personality defects. It is possible to work for another year on the non-creative parts of a project, it just takes discipline.

</comment><comment><USER30> This post also helps us non-technical types when interacting with technical co-founders and the engineering team. I realize my partner is definitely a starter and not a finisher!

</comment><comment><USER31> _Debugging the code that you wrote for weeks on end isn't as satisfying as helping someone squash a bug in their code._

Most likely explanation is not boredom of keeping working on the same project from start to finish.

Most likely explanation is boredom from working alone.

</comment><comment><USER32> I value the ownership of a project so much - the idea that when it becomes big, I can say, "I made that." Like the British guy who fought WWII with a sword and claymore. Challenge is rewarding.

Also, reading the Fountainhead was a game changer for me.

</comment><comment><USER33> I'm such a Starter/Architect it's not even funny. I know I need to work on finishing, but I can't fight nature (edit: I should say, I can only fight it so far).

Is there somewhere I can write to get a better Debugger/Finisher?

</comment><comment><USER34> This reminds me of role variants in the Keirsey Temperament Sorter.

<http://en.wikipedia.org/wiki/Keirsey_Temperament_Sorter>

</comment><comment><USER35> Does anyone have advice on how to become a better finisher?

<comment><USER22> Take any project (open source is easiest) and fix something, get it closer to that 100%. Do enough of this and you'll start to get a feel for it similar to the kind of rush you get when you're starting, only it is then associated with getting closer to the finish line instead of how far you got from the starting line.

It helps to pick small projects in the beginning to not lose sight of the goal.

</comment></comment><comment><USER36> "The finisher" sounds like a mythical creature to me...

</comment><comment><USER37> good article. I've found myself in various of these roles in different jobs. But it really is hard to do them all in the same project.

</comment><comment><USER38> The 4 seasons.

</comment><comment><USER39> Another metaphor I've seen is to the military (specifically the army), and consists of three roles. Don't beat me up about military minutiae, please; I don't claim to be an expert in that area and it's not even my metaphor.

(1) Paratroopers, who jump into unfamiliar territory. In software, researchers and architects.

(2) Infantry, by far the largest component, responsible for the core task of taking and holding territory. In software, most programmers.

(3) MPs (also quartermaster, community liason, etc.) who maintain order in the held territory. In software, debugging specialists and release engineers.

The problem I have with the OP's metaphor is that the "starter" and "architect" roles are both part of (1) and many people actually can do both pretty well. Similarly, the "debugger" and "finisher" roles are both (3) and also combine well. What's really unfortunate is that (2) seems entirely absent even though in real life it consumes most of the time and resources on a project. These are the folks who take mostly-complete designs from a starter/architect, and get most of the code mostly working before the serious integration/stress testing occur and the debugger/finisher come to the fore. In other words, most of your colleagues most of the time. If you hired four people according to these four roles, you'd have nobody to write most of the code and you'd be abusing your four specialists to do it.

</comment><comment><USER40> The article it's ok, but this is just a permutation of the roles first outlined in the Mythical Man Month (lawyer, surgeon, organizer, etc).

</comment><comment><USER41> Depending on the programming language, I can't help but feel that this article is out of date.

Today you don't really need to be a pure starter or architect anymore. There are so many frameworks that mimic Rails and its philosophy of convention over configuration, that there's not really a lot of effort needed to start and its easy to delegate most of the architectural duties to the framework developers themselves.

As for the debugger and finisher, that is also a lot easier as well. With all the automated integration and behavior driven test frameworks, it's relatively easy to both cross your t's and dot your i's. Today you can have something yelling at you everything second your tests break (assuming that you wrote them, which is key to any project).</comment>


Facebook's rivals reportedly contacted by DOJ as part of antitrust probe - <AUTHOR> https://www.cnet.com/news/facebooks-rivals-reportedly-contacted-by-doj-as-part-of-antitrust-probe/ <comment><USER2> wait, facebook has rivals? what, like "friendster" or?</comment>


Show HN: Your website went down last night... Or did it? Set up a monitor - <AUTHOR> https://silentdown.com <comment><USER2> I signed up for the free plan, but couldn't figure out how to set up the public status page.

<comment><AUTHOR> Hi Chris, when you create a monitor then under Settings of it, you can enable public page.

Once enabled, you find a new button called "Status Page", that will take you to your public page.

Let me know if you figure it out

<comment><USER2> Perfect, thank you. Got it working.</comment></comment></comment>


Publishers: Forget the content, here's your business model - <AUTHOR> http://jonbischke.com/2009/08/14/content-publishers-i-have-a-business-model-for-you/

<comment><USER2> "one of the best, if not the best, news/article filter out there. ... _if you remove the coding and start-up articles ..._ a super-interesting collection of smart writing and interesting thinking."

Rather ironic.

<comment><USER3> For Alanis Morissette values of ironic, anyway.

I've often thought that HN would be a great news/article filter if I could only remove the start-up articles, and I'm sure there are people who feel it would be great if they could read the start-up articles but remove the coding ones. I am _somewhat_ surprised that someone would want to remove both, but then there are often fascinating threads that spring up around security or robotics or similar matters.

<comment><USER2> Given that her song repeatedly ascribes irony to things that are in no way ironic, I assume you're giving me a subtle jab.

Since this forum was (I believe) started to discuss startups and programming, then it's ironic that the author finds those are the things he wants to remove.

Please explain to me how I'm misusing the term.

<comment><USER3> Wikipedia goes into far more detail about the technical definition of irony than I care to, but I believe the important part is that a statement is ironic when speaker's words have a relevance to the situation that the speaker is unaware of.

I'm assuming that nobody could read HN for very long and remain unaware that the site is targeted at programmers and entrepreneurs. Because the author of the original blog post knew the purpose of HN, he would have recognised the incongruity between his statement (that coding and startup articles be filtered out) and the reality (that HN is all about coding and startup articles). Since he was aware of the disparity, and irony requires unawareness, there was no irony.

I'd have given you "sarcasm" or "incongruity", but "irony" is one of those topics that triggers my "Someone Is Wrong On The Internet" reflex, I'm afraid.

</comment></comment></comment></comment><comment><USER4> So... the answer is stop publishing and switch to aggregating?

Sorry, but I'm not seeing how building a better filter enables you to raise the money for an investigation that uncovers secret NSA wiretaps.

<comment><USER5> I think the idea is to charge for filtering, not for content.

</comment></comment><comment><USER6> Some problems:

That model has minimal cost of entry. Therefore an unpredictable and vast number of competitors, therefore high risk. Google could decide to do it one day on a whim and with their resources they _would_ do it better and bam! whatever time/money you spent on it would be a loss.

Not sure what you mean by publisher and content, all your examples hn, digg, delicious I wouldn't consider content publishers at all. The traditional content publishers such as newspapers don't have the culture, management, payscales to attract and support large teams of top-notch devs. btw newspapers (and other content producers) already do a lot of filtering. Might not be that noticeable because the crap that gets "downvoted" never makes it to print/online and they filter for "mainstream" not intelligent, well-read, young entrepreneurs.

Maybe a great idea for a new, young, small startup. As a model for existing content publishers, not so sure.

</comment><comment><USER7> The New York Times and other media companies _are_ filters. They pick the stories that go on their front page, they decide where to dedicate news- gathering resources, they have an editorial voice...

</comment><comment><USER8> What's he's talking about really is strong filtering.

Current filtering systems weight user contributions equally which means trolls and other junk get in there easily.

</comment><comment><USER9> Is there a site that does some of what he talks about? Is there a site that shows what articles were hot here yesterday or last week?

<comment><USER10> He's really talking about an API. The NYTimes has one.

I do think there's a business model there.

What if the NYTimes shut out Google in its robots.txt file but then offered up full "firehose" access to it's content via the api for a price?

Nobody pays for content. Everybody pays for data. Publishers should build solid APIs, shut out the spiders, and then charge search engines, aggregators etc. for full, rights-free access to their stuff.

</comment></comment><comment><USER10> So.....offer an API. Charge money for it. Got it.</comment>


Global innovation loss from lack of co-opting startup experience? - <AUTHOR> http://www.gabrielweinberg.com/blog/2009/07/global-innovation-loss-from-lack-of-co-opting-startup-experience.html <comment><USER2> I think there are number of good places to get technical information, we started the Bootstrapper Breakfasts because practical business advice was harder to come by for technology startups.

One of the benefits of the Bootstrappers Breakfasts (<http://www.bootstrapperbreakfast.com>) are that they allow bootstrapping entrepreneurs to compare notes on operational issues. They are not meant to take the place of advisory boards or establishing other business relationships, but especially for smaller firms who are bootstrapping they are an opportunity to get advice from peers. We have folks attending who are doing their first startup and some who are on their fifth (the best approach can change over time, and what were good ideas for CD-ROM packaged software designed for customer PC or datacenter may be less powerful for SaaS/Cloud/iPhone platforms).

The first breakfast was in October 2005, we now run six times a month in Silicon Valley. There is no charge beyond the cost of your breakfast.

</comment><comment><USER3> Sadly, he is right and there does not appear to be an easy answer.

There was something good going on here: [http://blog.launchpadcoworking.com/2008/10/10/coffee-with- an...](http://blog.launchpadcoworking.com/2008/10/10/coffee-with-an-expert/) but <http://coffeewithanexpert.com/> is no longer loading...</comment>


Native Clojure with GraalVM - <AUTHOR> https://www.innoq.com/en/blog/native-clojure-and-graalvm/ <comment><AUTHOR> One problem is that Graal is still on Java 8 (nearly two versions behind of latest stable) because it is marked as LTS. I think that's a bad decision for a bleeding edge piece of technology anyway - a lot of of the components that Graal needs are batteries-included in later Java releases.

And most importantly, there's this - [https://news.ycombinator.com/item?id=16919776](https://news.ycombinator.com/item?id=16919776) ( Java SE 8 business users must buy a licence from Jan 2019 to receive updates)

</comment><comment><USER2> There are 2 download options for GraalVM - one with words "additional performance, security, and scalability" and another, well, without them. Does it mean that all the goodness of this technology is only available in the very, very paid option? In other words, it's not for startups who are trying to bootstrap themselves, right?

<comment><USER3> No, SubstrateVM and Graal are both open source.

The enhanced versions compile to faster code, pretty much. You can do without them if you don't want to pay.

<comment><USER4> I'm curious just how "optimized" and the paid version is (or how crippled the free version in comparison).

<comment><USER5> Try it yourself - both can be evaluated for free.

</comment></comment></comment></comment><comment><USER6> GraalVM still does not support Windows, does it?

<comment><USER7> GraalVM in the context of OpenJDK already runs on Windows. Native image generation on Windows is under development, should be available in the coming months.

</comment></comment><comment><USER8> so no eval at runtime which seems not to be such a big deal; we can develop at repl and then bake images for prod

<comment><USER9> Some of the Cognitect veterans have pointed out that being able to connect directly to an instance and evaluating code at runtime has been invaluable for debugging issues for their clients.

<comment><USER10> However, I don't see a use case for GraalVM in long-lived server applications. For that, regular JVM clojure will do just fine.

On the other hand, using eval is almost never necessary in small standalone utilities.

<comment><USER5> > I don't see a use case for GraalVM in long-lived server applications

GraalVM also includes a better JIT compiler though, which makes long-lived applications much faster. Twitter use it today to run their Scala code.

<comment><USER11> From Twitter engineer Chris Thalinger's videos[0] it seems like they're using JDK 9 or 10 and graal via JVMCI. Is there more to GraalVM than just using graal with a particular JDK release?

[0]: [https://www.youtube.com/watch?v=_7yIUkP5LiQ](https://www.youtube.com/watch?v=_7yIUkP5LiQ)

<comment><USER5> > Is there more to GraalVM than just using graal with a particular JDK > release?

Yeah there's a whole world of different ways you can use Graal.

[https://medium.com/graalvm/graalvm-ten- things-12d9111f307d](https://medium.com/graalvm/graalvm-ten- things-12d9111f307d)

They're choosing to use the JIT from CE Graal (which they've built themselves), via the released version of JVMCI in a released JDK.

</comment></comment></comment></comment></comment></comment><comment><USER12> some immediately useful and practical applications of Clojure/GraalVM. nice work!

</comment><comment><USER13> I'm still concerned about the license and the fact that this comes from Oracle.

<comment><USER14> What concerns you over the JVM versus say, GCC? Both are GPL licensed. Oracle has committed to openJDK being the reference implementation.

<comment><USER15> Oracle vs Google over Android having the same interfaces comes to mind. [0]

[0] [https://www.techdirt.com/articles/20180327/10431439512/insan...](https://www.techdirt.com/articles/20180327/10431439512/insanity- wins-as-appeals-court-overturns-googles-fair-use-victory-java-apis.shtml)

<comment><USER16> Neither Sun, nor Oracle ever had any issue with Java vendors that play by the rules and there are plenty of them.

<comment><USER15> ... And Google isn't a vendor here.

The API was made the same, but with Google's own implementation. Several juries and courts have heard the case and said that an API isn't copyrightable. CAFC dismissed the juries' findings without reasoning. It's been punted back to a new jury and court for a final hearing.

<comment><USER16> Google admited several times that they haven't done a clean room reverse engineering and their emails prove they were consciously trying to work around Java licensing issues.

If Google cared so much, they could have bought Sun, but they were expecting it would just crash and burn, while the company would get away with it.

Sadly for them Sun actually got a buyer.

<comment><USER15> > Google admited several times that they haven't done a clean room reverse > engineering

> It is undisputed that Google copied verbatim the declaring code of the 37 > Java API packages—11,500 lines of Oracle's copyrighted code. It also copied > the SSO of the Java API packages. Google then wrote its own implementing > code.

The only problem with Google's implementation is the copying of the API.

\---

> If Google cared so much, they could have bought Sun, but they were expecting > it would just crash and burn, while the company would get away with it.

> The parties were unable to reach an agreement, in part because Google wanted > device manufacturers to be able to use Oracle's APIs in Android for free > with no limits on modifying the code, which would jeopardize the "write > once, run anywhere" philosophy.

> in any event that the jury must have found that Google did not act in bad > faith

Google tried to get a reasonable license.

\---

Even CAFC has said that if Google had not received advertising revenue from Android, their use of the APIs would have fallen under fair use. They are not a Java vendor.

<comment><USER16> > They are not a Java vendor.

Last time I checked [https://developer.android.com/](https://developer.android.com/) and [https://developer.android.com/guide/platform/](https://developer.android.com/guide/platform/) has the word _Java_ written all over the place.

So apparently they license an operating system, that makes use of Java, the language, and a subset of its standard library, to the point that it affects the sales of other Java vendors.

On my naivety it makes them a Java vendor to me.

<comment><USER15> Strangely I can only find 'Java API'.

> The entire feature-set of the Android OS is available to you through APIs > written in the Java language.

> Build toolchains, such as Jack, compile Java sources into DEX bytecode, > which can run on the Android platform.

Seems they make no use of the JVM... Which would be required to vend it, no?

<comment><USER16> A JVM is only one possible implementation of the Java language.

A few Java vendors on the embedded space also provide their own implementations, where .class files are converted either into their special VM format or straight into native code, before generating the firmware image for deployment.

Android uses Java the language, alongside a partial implementation of Java the standard library. The way it is implemented, is like any other programming language, a detail.

The fact that I can pick any Java library that compiles under that subset and use it on Android is expected by any Java developer.

Likewise embedded vendors that sell C compilers, even if they aren't fully ANSI C compliant, still acknowledge they are selling C compilers.

<comment><USER15> You seem to be confused on a few points.

The court case is that Android infringes on Java SE, not the Java language. The question is if it infringes on the platform, not the language.

> Android uses Java the language, alongside a partial implementation of Java > the standard library

That's not what is being disputed. They use an API written in Java, afterall.

> Likewise embedded vendors that sell C compilers, even if they aren't fully > ANSI C compliant, still acknowledge they are selling C compilers.

And Android has a compiler for Dalvik and AR. That's not up for dispute either. That doesn't make them a vendor for Java SE. No more than the compiler in GCC makes the GNU project a vendor.

> A few Java vendors on the embedded space also provide their own > implementations, where .class files are converted either into their special > VM format or straight into native code, before generating the firmware image > for deployment.

Usage of the Java brand requires any platform you ensure compatibility between your implementation and all other implementations [0]. The Android SDK is not compatible to the letter of that trademark license, as the Android SDK is both a subset, and also extends the Java API, so Google followed their license by not claiming it.

\---

The only thing being disputed, is if their Java-written API is allowed to reflect Java SE's API.

Google is a library vendor for a library that reflects a subset of the host language. They also have their own compiler for a subset of the language.

> Likewise embedded vendors that sell C compilers, even if they aren't fully > ANSI C compliant, still acknowledge they are selling C compilers.

This is more akin to claiming that a C++ compiler is infringing on a C compiler by both being compatible with a subset of the C language.

[0] > This resulted in a legal dispute with Microsoft after Sun claimed that the Microsoft implementation did not support RMI or JNI and had added platform-specific features of their own. Sun sued in 1997, and, in 2001, won a settlement of US$20 million, as well as a court order enforcing the terms of the license from Sun. As a result, Microsoft no longer ships Java with Windows.

<comment><USER16> Fact, Google ships an operating system with the userspace written in Java the language, making use of a subset of Java standard library.

Fact, Google created a schism on the Java compatibility story by only providing a subset, thus forcing Java developers to write Android specific libraries.

Fact, if Google actually cared about Java they would have bought the Java assets from Sun, after helping driving them to the ground, as they were aware Sun was short on cash and would never sue them.

Now we can all enjoy a broken compatibility story up to Java 8, while Java 10 is out there and the distance will only increase.

Android is just as bad for Java portability as J++ was.

But hey, it is the nice "do no evil" corporation against the big bad lawyer corporation.

<comment><USER15> I don't know what your problem is, and I don't know why you believe certain 'facts' that have never been presented at trial, but would clinch the case against Google.

I'm tired of reading legal documents to answer your complaints. You can read yourself, or continue ignoring the court case. Adios.

<comment><USER16> My problem is that Google just played a Microsoft, forked the Java eco-system, but because it is the beloved "Do no Evil " company of SV, it gets love support like yours instead of the hate Microsoft did exactly for the same result, while we Java developers have to workaround the fragmentation issues Google has brought into the Java world.

My problem is that I cannot pick any random Java library out of Maven and be 100% assured that it will run at all on Android.

My problem is that Android most likely will never move beyond its partial implementation of Java 8, as per Android P.

If it was for Google's wishes, Sun would have closed doors, hopefully no one would have bought the Java assets, they would get away with it, and we could all enjoy a Java language stuck at what was Java 6's subset.

<comment><USER15> > My problem is that I cannot pick any random Java library out of Maven and be > 100% assured that it will run at all on Android.

You need to stop thinking of the Android SDK as Java, because it isn't. It has Objective-C to C similarities.

It doesn't have API compatibility, or bytecode compatibility or even the same class constructors.

Thinking they're pretending to be the same is going to cause nothing but frustration, and hurt you. It'll irritate and chafe more every time you have to touch that god-awful ecosystem.

> it gets love support like yours instead of the hate Microsoft

No Google does not get my love, I have ranted against their practices for years. This conversation has been about legalities, not ethics.

They suck as a company - but we have ecosystems built around compatible APIs, a legal precedent here would be awful for everyone. (Firefox's new extension API, the entire JS ecosystem, etc.)

<comment><USER16> > You need to stop thinking of the Android SDK as Java, because it isn't.

That is exactly my point, Google has broken Java's compatibility story, regardless if they have found a legal loophole to get away with it or not.

Objective-C and C are two completely different programming languages, where Objective-C is a superset of C. Any C compliant code is compilable by an Objective-C compiler.

It is not the same as what Google has done to Java.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER17> How efficient is it compared to Azul Zing JVM ?

</comment><comment> cup-of-tea I cannot wait for emojis to go out of fashion.

<comment><USER18> I'm alright with emoji as ideograms. I'm not okay with it in communication though.

<comment><USER5> What is an ideogram if it isn't communication?

<comment><USER18> In this instance I was referring to dialogue. A standard set of pictographs for use in websites and signage is fine, but when someone I'm talking to over text uses emoji I find it jarring and irritating at best.

</comment></comment></comment></comment><comment><USER19> Does graal run on arm

<comment><USER7> A Graal backend for AArch64 is currently under development. See for example these PRs at [https://github.com/oracle/graal/pulls?utf8=%E2%9C%93&q=is%3A...](https://github.com/oracle/graal/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aopen+aarch64+in%3Aname).

</comment></comment><comment><USER20> Sorry for pointing the obvious and not adding anything useful to the technical discussion, but after the recent events I would not touch anything coming from oracle with a ten-foot pole

<comment><USER21> I do not understand the rationals for this, its licensed under GPL2, care to elaborate a bit ?

<comment><USER22> Parts are licensed under GPLv2+Classpath and other components like the js / python layers are licensed under UPL. And based on [https://www.graalvm.org/downloads/](https://www.graalvm.org/downloads/), it would seem that Mac use is only supported by their "enterprise edition"

<comment><USER7> "The reason why we don't build CE on Mac OS is purely technical. Its because there was no OpenJDK 8 build for Mac that we could use. We hope we can change that soon. OpenJDK builds got a lot more regular with Java version >= 10."

(Source: [https://news.ycombinator.com/item?id=16859559](https://news.ycombinator.com/item?id=16859559))</comment></comment></comment></comment>


'Ubuntu on Android may help find next Einstein' - <AUTHOR> http://www.techradar.com/news/software/operating-systems/-ubuntu-on-android-may-help-find-next-einstein--1159142

<comment><USER2> It's an interview with Jon 'maddog' Hall, and the title quote is mangled: it's "where the next Albert Einstein of computer science could come from." Smartphones are a good market for Linux.

</comment><comment><USER3> Linkbait for an Interview.</comment>


Declassified U-2 spy plane photos are a boon for aerial archaeology - <AUTHOR> https://www.sciencemag.org/news/2019/04/declassified-u-2-spy-plane-photos-are-boon-aerial-archaeology <comment><USER2> Desert kites: they're mentioned in passing in the article and I was curious so I looked them up. Apparently they're 5000 year old traps for gazelles that can be kilometers long: [https://en.wikipedia.org/wiki/Desert_kite](https://en.wikipedia.org/wiki/Desert_kite)

<comment><USER3> Funny how they were named by "pilots who first saw them from the air in the 1920s" and not by the people who built them.

<comment><USER4> Since they were abandoned about 4000 years ago, I'd imagine that it's quite hard to find any records of their original name...

</comment></comment></comment><comment><USER5> _Hammer and colleague Jason Ur, an anthropologist at Harvard University, created a systematized index of several thousand U-2 photos taken on 11 reconnaissance missions throughout the Middle East_

Fantastic bit of nominative determinism:

 _Jason Ur is Professor of Anthropology in the Department of Anthropology at Harvard University, and director of its Center for Geographic Analysis. He specializes in early urbanism_

As well he might:

[https://en.wikipedia.org/wiki/Ur](https://en.wikipedia.org/wiki/Ur)

<comment><USER6> I wonder what his favourite game might be. One I still have an unfinished implementation in C lying around.

</comment></comment><comment><USER7> The U-2 is the only U.S. surveillance aircraft still flying a "wet film" camera called the Optical Bar Camera, or OBC. Ironically, this relatively old technology keeps the U-2 mission from being completely transitioned to other aircraft like Global Hawk, due to the film imagery being easily releasable and able to be declassified.

Around 2007, a PC-104 embedded computer running Linux was integrated into the OBC camera to allow correlation of flight data with the imagery, as well as supply an OBC camera status page on the cockpit multi-function display.

<comment><USER8> Why is film easier to declassify and release than digital imagery?

<comment><USER9> Film's properties are known and understood and there aren't any "super films" that are better or more capable than another country's. Lenses are also understood.

With electro-optical (EO) sensors, great care must be taken to reduce the quality of the final product when it is publicly released so that adversaries do not gain a complete understanding of the what the sensors are capable of.

Film creates "better" images, but modern EO sensors are more capable in certain circumstances.

There is all kinds of computational and electronic trickery one can do to obtain images that may be impossible to capture on film that you want to keep secret, like fusing short wavelength IR with visible light or using it to discipline visible light to correct or reduce atmospheric distortion. Other EO technologies can determine what an object is made of from great distances.

Technologies like that you want to keep secret.

In a hypothetical Cuban Missile Crisis set in 2019, US analysts would have visible, near- and short wavelength-infrared, thermal, and pan-chromatic imagery to look at, but the 2019 version of Adlai Stevenson would still only show the visible images at the UN.

<comment><USER10> There were superfilms, film stock sensitive in the infrared/UV spectrum and filters to optimize such film. There was also film sensitive only to very specific colors. Releasing images from these, at any resolution, would indeed give away much of the program's abilities/goals.

The ability to more easily declassify film stock is due less to the technology and more to the bureaucracy within intel communities. The film stock is owned by a single agency and so the declassification authority is relatively straightforwards. Digital imagery is shared instantly with a host of different agencies, many of whom still do not talk to each other regularly, and is stored in countless archives. Declassifying a digital file is therefore an administrative burden in comparison to a roll of film kept by a specific agency.

<comment><USER11> These were all known since 1920s-1960s and very little progress has been made since. Everything you named is bog standard film photography practices.

<comment><USER10> Yes but the use of a particular technique in a particular location/time would divulge the specific collection goals of an operation, something that often remains classified long after the operation itself has been acknowledged. So while the existence of UV film is no secret, knowledge that it was being employed over a specific site at a specific time can be.

</comment></comment></comment></comment></comment></comment><comment><USER12> For those interested in aerial archeology I also heartily recommend what Roger Agache did in (mostly) Northern and Central France starting with the 1960s. His aerial photos helped discover hundreds of locations for Roman-era villas and castra among other very interesting things. The website documenting his work can be found here [1], and for example aerial photos of Gallo-Romanic structures can be found at this link [2] (the website looks like it hasn't been updated in quite some time but it's still functional)

[1] [http://www.archeologie- aerienne.culture.gouv.fr/en/](http://www.archeologie- aerienne.culture.gouv.fr/en/)

[2] [http://www.archeologie- aerienne.culture.gouv.fr/en/](http://www.archeologie- aerienne.culture.gouv.fr/en/)

</comment><comment><USER13> If this triggered your curiosity, have fun with [http://www.apaame.org/](http://www.apaame.org/)

> The Aerial Photographic Archive for Archaeology in the Middle East was > established in 1978 by Professor David Kennedy under the patronage of Prince > Hassan bin Talal of Jordan. The archive now consists of over 91,000 > photographs and several hundred maps of a dozen countries. The vast majority > of this material can be viewed online in our digital archive at Flickr.

</comment><comment><USER14> >In recent years, the Islamic State group gutted archaeologically important Iraqi sites in Mosul and Raqqa and Syrian sites in Aleppo and Palmyra.

Not to forget Buddha of Bamiyan, Afghanistan.

It is disheartening to think that, these sites & artefacts which were preserved for thousands of years are destroyed during our lifetime.

Perhaps because of its familiarity we value it higher, but powerful people in several parts of the world have been systematically destroying historical objects of value to a culture or civilisation to push forward their ideology.

Edit : Edited to be succinct.

<comment><USER15> > _Perhaps because of its familiarity we value it higher, but powerful people > in several parts of the world have been systematically destroying historical > objects of value to a culture or civilisation to push forward their > ideology._

This is a particular feature of utopian ideology; marxists, islamists, fascists, etc. seek to carve utopia from reality, and the artifacts which prove that there's something else are just in the way. A look at what's preserved in Taiwan should make you weep to know what was lost just a little bit to the west.

<comment><USER14> I guess a less violent form of it practised by every country, even the democratic ones is falsifying history text books from grade school.

<comment><USER15> Yeah, Canadian grade school history textbooks are a joke. Be a good parent, don't let your kids get all of their historical education from your school district's standard textbook. :- )

<comment><USER16> Specifics might aid your argument.

<comment><USER15> In my experience, there was no mention of the broader historical origins of Canada, the focus was generally on a) stories framed to make French Canadians look interesting and blameless (especially long sections on coureurs de bois), and b) stories of amicable arrangements with native Americans, conveniently away from the various times the public's government has broken promises to various communities in Canada.

I'm an adult, why would I have the worst history books I've ever read, which were on loan from the school when I was in school, in my personal library to share specifics from?

<comment><USER16> Thanks for the first.

I'd not made any demands to the latter point, though there are times when keeping a spectacularly bad example around for debate, discussion, or reference can prove handy.

I'd put specific store in rough order "steelman" or best-case defenses or arguments for the indefensible, officially sanctioned references (as with textbooks), or with particularly poular bad examples, even if not particularly cogent.

Knowing your enemy, testing your own beliefs and biases, and walking into battle fully armed, are all benefits.

You don't need to find endless such examples (see also: Gish Gallop), but a carefully selected few can be exceedingly useful.

This applies to other areas as well, tech included.

<comment><USER15> > _Knowing your enemy, testing your own beliefs and biases, and walking into > battle fully armed, are all benefits._

Sure, to be clear, I grew up with two older brothers and a younger sister, in three different major cities, in three different provinces in Canada, and the history curriculum has largely lacked much particular detail. My history curricula, and that of my brothers, included little or no international history, ancient history, or national history. The main topics of every history textbook (the only source for each curriculum, in my experience) I've seen in Canada (including my brothers', for years I didn't attend in a given school district) have been an obscure subset of clean indigenous stories, and a handful of stories about early Québec.

I'm not saying these history textbooks are especially bad _among government school history textbooks_ , but that they are bad in a general sense, and fail to give much perspective on the origin of the tapestry of nations in Canada, or the story of our legal and governmental traditions.

A better job could be done with an in-depth reading of a mature historical author's work, the kind of thing you would read if you had a personal interest in understanding the history of _something_.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER17> I'm working with some researchers who are using similar recently-declassified spy plane imagery from the '60s for an analysis of glacier size + movement changes over time, which is another cool use case.

</comment><comment><USER18> Somehow I always thought the U-2 is like a gliding plane, quiet and slow. But then I saw one at an airshow last year and I was pretty impressed how loud and powerful the plane sounded. It also climbed extremely steeply which kind of makes sense considering that it has get up to 30 km altitude within a reasonable time.

</comment><comment><USER19> Recommend the book "The Past from Above" by George Gerster for those who enjoy 'aerial archeology'. 500+ photos taken from lower altitudes back in the 1950s and 1960s. (Can be had in paper, if you can't find the coffee-table-size hardcover.)

Review: [http://www.historyinreview.org/ggerster.html](http://www.historyinreview.org/ggerster.html)

ISBN-Book sources: [https://en.wikipedia.org/wiki/Special:BookSources/0-89236-81...](https://en.wikipedia.org/wiki/Special:BookSources/0-89236-817-9)

</comment><comment><USER20> [https://news.ycombinator.com/item?id=19587495](https://news.ycombinator.com/item?id=19587495)

<comment><USER21> Repost are usual [https://news.ycombinator.com/newsfaq.html](https://news.ycombinator.com/newsfaq.html) and sometimes the same link is ignored one time and gets traction other time. It's matter of the day of the week, the hour of the submission, the other post that are trending, the number of users reading the newest page, the phase of the moon, and many other factors. Just assume that it's a mater of luck.

Linking the old post is useful when it has some interesting comments (I sometimes quote partially the most interesting comment.) But in this case the older post got only 2 points and no comments.

<comment><USER20> I like this: "It's matter of the day of the week, the hour of the submission, the other posts that are trending, the number of users reading the newest page, the phase of the moon, and many other factors."

</comment></comment></comment><comment><USER22> What I want is some of those lenses and put them on my Canon. Looks like "pretty good" pieces of glass.</comment>