Cassandra vs MongoDB vs CouchDB vs Redis vs Riak comparison - <AUTHOR> http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis <comment><USER2> I like this article: while it is for sure not the definitive guide to NoSQL, it is a short description mostly about facts that people new to the field can use to get an idea about what a good candidate could be for initial experimentation, given a defined problem to solve.

That said I think that picking the good database is something you can do only with a lot of work. Picking good technologies for your project is _hard work_ , so there is to try one, and another and so forth, and even reconsidering after a few years (or months?) the state of the things again, given the evolution speed of the DB panorama in the recent years.

While I'm at it I like to share that in this exact days I'm working at a Redis disk back end. I've already a prototype working after a few days of full immersion (I like to use vacation time to work at completely new ideas for Redis).

The idea is that everything is stored on disk, in what is a plain key-value database (complex values are serialized when on disk), and the memory is instead used as an object cache. It is like taking current Redis Virtual Memory and inverting the logic completely, the result is the same (working set in memory, the rest on disk), but this implementation means that there are no limits on the data you can put into a single instance, that you don't have slow restarts (data is not loaded on memory if not demanded), and there isn't to fork() to save. Keys marked as "dirty" (modified) are transfered to disk asynchronously as needed, by IO threads.

If everything will work as I expect (and initial tests are really encouraging) this means that Redis 2.4 will exit in a few months completely killing the current Virtual Memory implementation in favor of the new "two back ends" design, where you can select if you want to run an in-memory DB or an on-disk DB where memory is just an LRU cache for the working set.

<comment><AUTHOR> antirez, it's a honor that you commented, thanks! :)

The new inverted logic for the VM you describe seems very interesting; I'm very much looking forward to see 2.4!

Redis is already more than perfect what we use it for -- keeping track of stock price data, and distributing it. The size of the DB is known in advance (the amount of stocks does not grow very fast), and the performance is perfect.

Keep up the good job! (And have a nice new year)

Kristof

<comment><USER2> Thank you kkovacs!

I think the main business of Redis is still as an in-memory DB / cache / messaging system and so forth. We have a decent implementation from this point of view, so the next logical step is making it working in a cluster.

On the other side it's really interesting to see what people can do with Redis data model if much larger datasets can be used without problems (at the cost of performances of course... can't be as fast as memory). VM was my first idea, but I need to admit, I don't like the design at this point. This new design can be much better, and we can have it production ready in a few months. So I'm curious about what will happen in 2011! :)

Thank you and have a nice new year as well.

<comment><AUTHOR> Thanks :)

</comment></comment></comment></comment><comment><USER3> Worth adding HBase?

Much below Stolen from their overview page (All needs to be confirmed): <http://hbase.apache.org/>

WRITTEN IN: Java

MAIN POINT: Hadoop Database

LICENSE: Apache

PROTOCOL: A REST-ful Web service gateway

This project's goal is the hosting of very large tables -- billions of rows X millions of columns -- atop clusters of commodity hardware.

HBase is an open-source, distributed, versioned, column-oriented store modeled after Google' Bigtable: A Distributed Storage System for Structured Data by Chang et al. Just as Bigtable leverages the distributed data storage provided by the Google File System, HBase provides Bigtable-like capabilities on top of Hadoop. HBase includes:

Convenient base classes for backing Hadoop MapReduce jobs with HBase tables

Query predicate push down via server side scan and get filters

Optimizations for real time queries

A high performance Thrift gateway

A REST-ful Web service gateway that supports XML, Protobuf, and binary data encoding options

Cascading, hive, and pig source and sink modules

Extensible jruby-based (JIRB) shell

Support for exporting metrics via the Hadoop metrics subsystem to files or Ganglia; or via JMX

HBase 0.20 has greatly improved on its predecessors:

No HBase single point of failure

Rolling restart for configuration changes and minor upgrades

Random access performance on par with open source relational databases such as MySQL

FOR EXAMPLE: Facebook Messaging Database

BEST USE: Use it when you need random, realtime read/write access to your Big Data.

<comment><AUTHOR> Thanks, Gordon, I put it in (shortened some lines).

It would be great to have a "more general" for-example, since noone outside Facebook meets the problem of "let's build Facebook's messaging database" :) Any suggestions?

<comment><USER4> There might be something you can glean from:

<http://wiki.apache.org/hadoop/Hbase/PoweredBy>

</comment></comment></comment><comment><USER5> Apples vs Oranges vs Strawberries vs Pineapple vs Grapes

Apples usually stay crispy unless baked. Good in pies.

Oranges can be sour (or sweet). Do not bake.

Strawberries are red. Good in pies, advise against baking.

Pineapples are rough on the outside. Good fresh, baked, grilled, fried, debatable on pizza.

Grapes come in many colors and sizes. Great fresh or turned into alcoholic beverages.

(Not the worst introduction to fruit, but perhaps superficial? Amirite?)

<comment><USER6> Actually, this would be useful and not superficial to someone who has seen pictures of these fruits but has never seen one in real life. Obviously it can be refined, but the idea is not bad. For example, try this with some quite different but seemingly similar fruits that most people are not as familiar with:

Pineapple Guava (Acca Sellowania) -- Small green fruit. Seeds soft and edible, skin optional. Turpentine flavor signals overripe. Cold hardy and grown in many parts of the US as an evergreen ornamental. Delicious eaten raw.

Strawberry Guava (Psidium cattleianum) --- Tasty small soft red fruit with very fragrant aroma and many small hard seeds. Skin edible, but seeds best avoided. Can be eaten out of hand, but low commercial use. Frost tolerant in mild climates.

White Guava (Psidium guajava) --- True tropical guava, thus barely if at all frost-tolerant. Large fragrant fruit with inedible hard seeds. Usually used for juice or puree, rarely eaten out of hand. Wonderful strong aroma increases with ripeness.

While obviously not of use to a producer of guavas, this sort of cheat sheet might be helpful to someone who happens to encounter one of these varieties in a grocery store or tree nursery. At the least, it might keep someone from breaking their teeth on the inedible hard seeds!

<comment><AUTHOR> Wow, thanks, very nice example! (And also thanks for expanding my fruit knowledge :) )

</comment></comment></comment><comment><USER7> This article is mostly marketing phrases from the websites of the various projects. Sadly, much of it is inaccurate, extremely skewed, or otherwise not useful for the stated purpose of comparing the listed databases.

For example, CouchDB having a "Main Point" of "DB consistency" might be the case, as it is for Redis, when there is no replication. In replicated configurations, it is definitely not true. Further, the MVCC is weaker in many ways than in a Dynamo system like Riak as you have no way to influence or discover consistency between replicas.

I'm sure folks expert in other systems can identify similar errors in the rest of the post. Can someone explain to me who the target audience is for all these NoSQL comparison articles? They are universally poor, yet universally popular.

</comment><comment><USER8> My understanding is that in CouchDB you can't guarantee that older versions of documents will still exists (they might be there, but they could have been removed by compaction or not replicated).

However, there is a fairly nice way of storing older versions of documents - hold older versions as file attachments on the document. See:

[http://jchrisa.net/drl/_design/sofa/_list/post/post- page?sta...](http://jchrisa.net/drl/_design/sofa/_list/post/post- page?startkey=\[%22Versioning-docs-in-CouchDB%22\])

<comment><USER9> This is probably the biggest misunderstanding of couchdb, imo. The versioning system in couchdb is only there to make the seamless replication possible. There's no guarantee that previous versions will exist at a future time, like in git.

Where couchdb has some immense possibilities is in distributed applications, not only server side, but also mobile phones and browsers. Since you can write and contain an entire webapp inside of couchdb, you can technically replicate the entire app to your mobile phone, and it'd work offline or online. And if you need your app on another platform--as long as it has couchdb, you can just replicate it there.

I never see this mentioned in any overviews of comparison for couchdb.

The sticking point right now, though, is that couchdb isn't on very many mobile platforms. There has only been experiments with writing couchdb on top of HTML5's localStore, and jChris et al are working on Couchdb for android.

<comment><USER10> "The versioning system in couchdb is only there.."

CouchDB does not version, period.

<comment><USER11> The "versioning" is really just there to support their optimistic concurrency model, if I recall. The idea is that you know you need to retry your operation if the version hash of the file has gone up since you last read the data and thus you know your local file is out of date.

As I recall, the id field is just a string. It's just common to let it do the automatic "#-hash" representation.

It's been a while since I played with CouchDB though, so I could be off.

<comment><USER12> You are correct.

</comment></comment></comment></comment></comment><comment><USER13> > While SQL databases are insanely useful tools, their tyranny of ~15 years is > coming to an end

This shit, AGAIN? Really? No, they are not.

<comment><USER14> With AppEngine at Google, MongoDB at Disqus, Cassandra at Facebook and Redis at Github you can definitely say that SQL databases are one of many options available today and don't dominate like they did 5 years ago.

<comment><USER15> if i'm not mistaken, the majority of those organizations still rely heavily on relational datastores, except in the case of exceptional workloads. in addition, i believe facebook has since migrated away from cassandra to the hadoop stack for their messaging platform, though they primarily use mysql (or its successors).

SQL is being replaced in niches that strain its model. elsewhere, it remains steadfast.

<comment><AUTHOR> I agree, ultimately it comes to everyone's own definition of "tyranny". :) I meant it as "not really having any defensible other choice". (While, of course for example CDB and BerkleyDB have "always" existed.)

I think nobody expects SQL's "market share" to fall to low levels, especially with noSQL requiring much deeper understanding of the data and it's planned use. NoSQL practically operates on a lower layer than SQL does.

Still, it's nice to see people thinking about data storage choices and not going blindly to MySQL/Oracle/etc!

</comment></comment></comment></comment><comment><USER16> What we're missing are similar arricles that go into disadvantages and implications on deployment.

Eg I have found out that deploying Tokyo Tyrant in a Rails project requires you to write some sčripts to ensure that things run properly. Also the db size has to be set in configuration in advance.

MongoDB OTOH is not designed for a single server environment, has a very small max document size, easily gets corrupted if process is stopped etc.

</comment><comment><USER17> CouchDB & MongoDB both share one property that this comparison misses (or mentions only in passing).

Both are schema free datastores. For me, this is the biggest, most useful difference between them and traditional SQL databases, because it makes things easy that are very, very hard (or inefficient) on an SQL database.

It's probably also worth noting that other NoSQL solutions don't share this advantage. For example, Cassandra requires all nodes to be restarted to apply a schema change, which can be quite a big deal.

<comment><USER18> "Cassandra requires all nodes to be restarted to apply a schema change, which can be quite a big deal."

That's no longer true. In 0.7, keyspaces and column families may be created, altered, or dropped live.

<comment><USER17> I thought it might have been fixed by now.

Anyway, you still need a schema with Cassandra.

<comment><USER19> Not in the same sense as an SQL database. You can freely add columns and rows, just not Column Families or Keyspaces. This is because a KS+CF combo is stored in it's own file, in a certain order, so that it can be efficiently traversed using natural ordering. If you don't have this need and just need a flat K/V database, you can use a single KS+CF for everything.

</comment></comment></comment></comment><comment><AUTHOR> I think it's a nice closing word from @jzy:

A SQL query goes into a bar, walks up to two tables and asks, "Can I join you?" "No, but you can enjoy the view."

Sorry :)

K.

</comment><comment><USER20> Under protocols you may want to specify MongoDB's as BSON and Cassandra's as Thrift. That would be more helpful than "binary/custom".

Updated:

Also Redis's main selling point is it's extensive data structure/operations support. "Blazingly fast" really depends on what your workload is and what you're comparing it against.

<comment><AUTHOR> Protocols: great idea, thanks man, amended it!

Blazing fast: I mean compared to the other four.

<comment><USER20> I've had both MongoDB and Cassandra perform nearly as well as Memcache when getting a single document/row when the document was in memory in MongoDB and the row was cached (row cache, not just key cache, so again: fully in memory) in Cassandra.

In memory operations are fast in many databases. Redis's default configuration (vm-enabled no) just only does operations in memory (with an occasional sync to disk). That's terrible durability but fantastic performance. Most databases, including Redis, can be configured for either that sort of high- performance/low-durability or the opposite. It's just that their default settings/behaviors vary widely.

</comment></comment></comment><comment><USER21> Also is VertexDB - small graph database. It's written in C, uses Tokyo Cabinet for storing data. Simple http filesystem-like interface. The general advantage - links, that allow to make graph structures on database level.

<https://github.com/stevedekorte/vertexdb>

</comment><comment><USER22> You mention that some of these solution could be used in the Financial industry. I would be cautious of using these, especially since some are eventually consistent. If you are just tracking data these may be fine though.

<comment><AUTHOR> I agree with that, but you know, it's very hard to find good examples. :) I'm open to better suggestions :)

<comment><USER23> Some points on Cassandra:

\- Facebook designed it for inbox feature - \- SOLR/Lucene is being integrated \- recently Sequoia backed Riptano - see <http://www.riptano.com/>

<comment><USER24> But facebook recently dumped it for inbox feature and began using Hbase instead. Not sure if Sequoia backing Riptano is supposed to be a bug or a feature?

</comment></comment></comment></comment><comment><USER25> I was hoping HandlerSocket would be in here. If you don't know about it, check it out <http://news.ycombinator.com/item?id=1886137>

<comment><AUTHOR> I don't know that one yet, but I promise I'll read up on it! Thanks!

K.

<comment><USER25> The latest Percona flavor of MySQL has it built in [http://www.mysqlperformanceblog.com/2010/12/14/percona- serve...](http://www.mysqlperformanceblog.com/2010/12/14/percona-server-now- both-sql-and-nosql/)

</comment></comment></comment><comment><USER26> Interesting and useful.

One major feature differentiator is something it doesn't really talk about, though - how conducive is each system to Massive Data?

For example, he kind of has a bone to pick with Cassandra, which is probably justified. But from what little I know, one of the features of Cassandra is that it's designed to scale pretty much to infinity. That may be true of a couple of the others, but for some (like CouchDB) it isn't a design goal at all.

<comment><AUTHOR> Good point, and it's not there since I only wanted to speak from experience; especially with rumors of Cassandra scaling problems at Reddit and Digg.

But sure thing, "infinite" scaling is probably best done with the Dynamo-like stuff like Cassandra and/or RIAK.

<comment><USER27> Reddit runs Cassandra with just a few nodes. Cassandra scales up well, but doesn't scale down as well:

[http://www.reddit.com/r/announcements/comments/c2spc/reddits...](http://www.reddit.com/r/announcements/comments/c2spc/reddits_may_2010_state_of_the_servers_report_or/c0ptl28)

</comment></comment></comment><comment><USER28> Membase?

Using it in a recent project and it's been working great for us.

<comment><USER24> Membase is the one to watch. I saw a presentation by AOL @ Hadoop World re: their use of Membase. Incredibly impressive technology.

</comment></comment><comment><USER29> Thank's for the article, good information.

Does anyone have any user amounts about the different no-sql databases? Or just say two most popular ones? I guess some of them will rise above the other's in following years and some will drop. User amounts would indicate which ones have most potential to stay around and be accepted as standard no- sql databases.

</comment><comment><USER30> So if Cassandra writes are much faster than reads, why would Reddit go that route? Their comment server is consistently breaking on them, and it would seem that a sub-optimal choice of db might be partly to blame.

<comment><USER19> It's not as lop-sided as this article might have you believe and has largely been mitigated as of late. This is because Cassandra uses read repair, which is a big component of it's strategy to make both reads and writes to scale linearly while also ensuring durability.

What is your suggestion otherwise? Any distributed database that is going to be inexpensive, performant, scalable, and durable will need to use some kind of quorum read repair system. Riak, Voldemort, and Dynamo all use read repair with high levels of production success.

</comment></comment><comment><USER31> Good article, it's a good starting point to let the people to decide where to start in using a NoSQL solution. But what about OrientDB? Do you plan to add it in this feature comparison?

</comment><comment><USER32> I'm curious why you wouldn't include HBase as it's the dominant solution for NoSQL in systems requiring data consistency?

<comment><AUTHOR> Only beacuse I don't know HBase very well. I'll read up on it, at least try it, and then add that too (and also maybe Tokyo Cabinet).

Thanks for the feedback!

<comment><USER33> Cool... and maybe also Voldemort?

</comment></comment></comment><comment><USER34> Nice writeup. A good survey of some key tools in the NoSQL space. Thanks!</comment>


Mapping The Entertainment Ecosystems of Apple, Microsoft, Google & Amazon - <AUTHOR> http://www.macstories.net/stories/mapping-the-entertainment-ecosystems-of-apple-microsoft-google-amazon/

<comment><USER2> Well done post, and definitely some interesting data to look at here.

The one point where I think you may have undersold Google is the interoperability of their Music service. They sell their music without DRM (similar to Apple) so it can be played anywhere. Also similar to Apple they have a player built in HTML5 so it can be accessed on most devices without even needing to download the song and play with a native player. This should be reflected and I believe should swing the interoperability category heavily in Google's favor since books and music are both accessible everywhere.

</comment><comment><USER3> Would have been interesting to also compare the angle of a roving user not invested in any single ecosystem.

But I guess that isn't as easy to compare and would've included a mass of references and explanations which would've killed the narrative of the article.

</comment><comment><USER4> Why aren't games included? It's a huge industry.

<comment><USER5> Hey there, it's Graham the author of the piece here.

I didn't include games because my focus wasn't really on consoles but rather smartphones, tablets and PCs and in that sense games are distributed as just "apps" and as you would have seen, I did look at apps.</comment></comment>


The smartest Bitcoin trade in town? - <AUTHOR> https://ftalphaville.ft.com/2017/12/07/2196554/the-smartest-bitcoin-trade-in-town/ <comment><AUTHOR> DECEMBER 7, 2017 By: Izabella Kaminska

Over the last week or so, we've recounted the problems with bitcoin's market structure and how they are likely to impact the upcoming launch of bitcoin futures (here, here and here).

In the course of explaining the structural difficulties, we've pointed out how the capacity of market makers and bi-directional traders to support the product is crucial if bitcoin futures are ever to become a success. Currently, this is unlikely to happen because there is no easy way to play both sides of the market without taking on huge amounts of credit, fragmentation, illiquidity and hacker risk on the physical side.

As it stands, CFD and spread-betting houses are the ones mostly attempting to provide this bridging role. Problem is, even they are struggling to process the risk — and that's despite being much less intensively supervised than the more established players who would usually be interested in servicing futures markets.

Some sort of risk-absorbing entity, as a consequence, must appear if retail and institutional participants (who are used to fiduciary standards) are to step into the market in size.

As a result, there are only three possible scenarios from here on in:

The futures (plagued by illiquidity and non convergence with the underlying) flop. The lack of a market-maker redistributing one-sided risk back into the market will see the risk transferred elsewhere, most likely into the clearing house (to the risk of the entire trading community). A less established player with a greater tolerance for risk — possibly a natural long — steps into the fray. The third option doesn't necessarily prevent the second option from playing out, however, given such an entity would still have to be serviced by the CME/CBOE clearing systems.

Nevertheless, let's imagine such an entity exists. What would its game plan be? And why would it think it could handle the risk?

The easy answer to the second question is that it may have spotted an arbitrage it thinks could more than compensate for the risk at hand.

As to the game plan…

If you're gunning to be the only entity in town prepared to sell bitcoin futures, it would be in your interests to start "pre-hedging" physical bitcoin as soon as possible with a view to locking in a risk-free basis return once the ability to sell futures on a regulated venue becomes possible.

Ideally, the trade would require an average purchasing price that's much lower than the rate bitcoin futures would eventually be sold at. To maximise this trade, as much value would have to be ploughed into the purchase (or generation) of bitcoin ahead of time, as likely buyside demand for the futures once launched.

In terms of timing, due to bitcoin's illiquidity, a "pre-hedging" position of this size would no doubt take time to put on. From that perspective it would make sense to start purchases as soon as a futures contract looked even remotely viable. FWIW, according to Factiva, the first serious clue CME was looking into a listing came in November 2016 when it launched a pair of indexes designed to track the virtual currency's price. The CFTC's decision in July to allow LedgerX to run a swap execution facility for bitcoin options, meanwhile, was a likely indicator a futures contract could be approved soon as well:

Nevertheless, no matter how strategically planned, pre-hedging of this size is always bound to leave a market footprint. (Especially in a market as illiquid as bitcoin.)

With that, the sort of self-fulfilling feedback loop that usually occurs when someone attempts to corner a market probably comes into motion.

For many, sparking a feedback loop of this kind is often deemed a trading objective in and of itself. Not for a bonafide smart operator. The smart money understands a good trade is as much about executing the "out" as it is about positioning the "in". You can start a pump, but you can't always profitably orchestrate the dump.

Hence why the futures component of the trade cannot be under emphasized.

Without the certainty of a properly regulated marketplace defending the futures side of the equation, a payoff cannot be guaranteed. It's why the arbitrage exists in the first place: the trade cannot be exercised elsewhere in the bitcoin ecosystem because the risk of counterparty default at the first sign of distress is far too great. You'd win the trade, but you'd probably lose the payout.

If a regulated futures market takes that risk away, however, the trade becomes a no-brainer if not a mastermind opportunity.

Which is arguably where we are now.

Indeed — if such an entity does exist — only three possible risks that threaten the trade at this point:

The futures launch is suspended unexpectedly, perhaps due to regulatory concerns or industry pushback? The anticipated futures demand never arrives because the margin costs of holding open long positions proves to be too great and/or liquidity is so shoddy nobody can trade effectively. The cash settled return on the futures leg, won't necessarily be matched by the price achieved on the physical liquidation.

One way or another, we will find out soon.

</comment><comment><USER2> Paywalled.</comment>


Google Analytics for GitHub - <AUTHOR> https://github.com/igrigorik/ga-beacon

<comment><USER2> I think github just solved this [https://github.com/blog/1672-introducing- github-traffic-anal...](https://github.com/blog/1672-introducing-github- traffic-analytics)

</comment><comment><USER3> Doesn't GitHub has Gauges? Surely Github should work on this idea? If not improving Gagues since i really wish there is a decent competitor to Google Analytics.

</comment><comment><USER4> Good to know.

 _adds ga-beacon.appspot.com to hosts file blackhole config_

<comment><USER5> I doubt adding ga-beacon.appspot.com to hosts file would help ! I probably would redeploy the app in my own appspot and embed the image from there.

<comment><USER4> Yeah. I suppose that is entirely possible. This should reduce _some_ drive by slowdowns though.</comment></comment></comment>


Ask HN:augmenting working memory ? - <AUTHOR> What are some the tools (like google is for long term memory) , to help , and increase the ability of working memory ? <comment><USER2> Sleep. Seriously. Also, "downtime". During both of which you integrate short term memory into long term memory. and also form associations that you can't consciously control.

I find that when I'm rested, I can maintain several more items in my working memory. It also frees up your working memory to take on new items, with the previous material now ensconced in long term memory.

Health. If you are feeling poorly, it's going to be distracting. Also, exercise is increasingly shown to have a large positive effect upon mental function.

Eliminating negative distractions -- even those you may be unaware of. Anecdote: Back in college days, a friend pursuing a chemistry major used to study with music on. One time, he deliberately avoided the music -- perhaps I or someone convinced him to give it a try -- while studying for the next exam. He did significantly better in the exam, by his own evaluation, and was quite surprised by this.

I'm not saying everyone reacts the same way to music; rather, you need to be aware of your own, personal reactions. And sometimes this is not obvious.

I've reluctantly begun taking a low dose of Adderall, recently -- partly in response to some chronic health problems that I won't go into. It helps a bit in some ways, but also hinders me in others. It does NOT produce the deep, rapid, and highly insightful quasi-fugue / focus state that I could get into -- with some effort -- when I was younger and particularly in better health.

Perhaps some of the newer nootropics are better -- they are quite expensive in the U.S. and I haven't given them a try. But my general impression is that there is not much in the way of real shortcuts. You build up and protect your foundation of physical health, and this provides the brain the resources that enable greater critical thinking, including working memory.

I guess I should add in a specific reference to stress. Acute, limited duration stress can increase my focus in a certain domain. Chronic stress and particularly negative acute stress short circuit my working memory more than anything else.

</comment><comment><USER3> Caffeine, Flash Cards, Also see: <http://en.wikipedia.org/wiki/Nootropic#Memory_enhancement>

</comment><comment><USER4> Blueberries can help:

[http://www.sciencedaily.com/releases/2008/04/080410115405.ht...](http://www.sciencedaily.com/releases/2008/04/080410115405.htm)</comment>


Tinker with a Neural Network in Your Browser - <AUTHOR> http://playground.tensorflow.org/ <comment><USER2> The swiss roll problem also illustrates nicely the idea behind deep learning.

Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: [http://imgur.com/H1cvt8d](http://imgur.com/H1cvt8d)

Then the deep learning guys realized that you don't have to engineer all these extra features, you can just use basic features x_1, x_2 and let the network learn more complicated transformations in subsequent layers. So they would use a deep network with only x_1, x_2 as inputs: [http://imgur.com/XBRjROP](http://imgur.com/XBRjROP)

Both these approaches work here (loss < 0.01). The difference is that for the first one you have to manually choose the extra features sin(x_1), x_1^2, ... for each problem. And the more complicated the problem the harder it is to design good features. People in the computer vision community spent years and years trying to design good features for e.g. object recognition. But finally some people realized that deep networks could learn these features themselves. And that's the main idea in deep learning.

<comment><USER3> _> Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc._

It's probably worth pointing out that this is true for ANNs, but there were (and are) other "shallow" classifiers that can handle swiss roll problem without manual parameter encoding. SVMs, for example.

[http://cs.stanford.edu/people/karpathy/svmjs/demo/](http://cs.stanford.edu/people/karpathy/svmjs/demo/)

<comment><USER4> needs another image link for visualization

</comment></comment></comment><comment><USER5> I started reading about ANNs in the 1980s, and had similar confusion to those here, since it was just for fun. I suggest reading a basic book or online information that goes over the basics [1]. I struggled through $200 text books, and jumped from one to the other as an autodidact. I am now studying TWEANNs (Topology and Weight Evolving Artificial Neural Networks), which basically are what you see here with the exception that they are able to not only change their weights, but also their topology, that is how many and where the neurons and layers are. ANNs (Artificial Neural Networks - as opposed to biological ones) can be a lot of fun, and are very relevant to machine learning and big data nowadays. It was exploratory for me. I used them for generative art and music programs. Be careful: soon you'll be reading about genetic algorithms, genetic programming [2], and artificial life ;) Genetic Programming can be used to evolve neural networks as well as generate computer programs to solve a problem in a specified domain. Hint: You'll probably want to use Lisp/Scheme for genetic programming!

      [1] http://natureofcode.com/book/chapter-10-neural-networks/
      [2] http://www.genetic-programming.com

<comment><USER6> Any thoughts on why genetic programming is not 'in fashion'? Does it have anything to do with complexity of the calculations?

I can imagine that the advanced models use many, many machines and only deliver results after a large training time. Genetic programming is not feasible then, if you cannot get a quick grasp of the potential results of a model.

<comment><USER7> If your program is a neural network with N parameters, or a program tree with N nodes, then testing against data takes O(N) time. With evolutionary computation, what you get for your trouble is a single real number -- the loss: how bad it did. With neural networks, backpropagation gives you N real numbers: the gradient of loss with respect to each parameter.

Put another way: with evolution you have to stumble around blindly in parameter space and rely on selection to keep you moving in the right direction. With the gradient descent that neural networks use, you get, essentially for free, knowledge of the (locally) best direction to move in parameter space.

The bigger the models, the more this matters. Modern neural networks have millions or even billions of parameters, and that's been crucial to their expressive power. Good luck learning a program tree with a billion nodes using evolution. It might take 4.54 billion years.

<comment><USER8> > It might take 4.54 billion years.

And then only if you have a system powerful enough to accurately simulate a planet full of molecules.

Although I do think there is a balance between GA and structured NN which will lead to faster and better results than the deep NN alone. We already see some of the best deep NNs incorporating specific structures.

<comment><USER5> I think neural networks and other forms of evolutionary computation will merge as I have been writing in my other replies in this thread. TWEANNs incorporate EC into evolving ANNs. The other article I cited above on soil mechanics, beat out expert systems, ANNs, statistics, and used GP. MEP, or Multi-Expression Programming for GP incorporates being able to put more than one solution into a gene without increasing the processing times thereby overcoming the inefficiencies of 1990s-era GP. Here is a recent article using it that is not behind a paywall or via sci-hub.io [1]. It needs better editing, but there are other references if you search for Multi-expression Genetic Programming.

      [1] http://benthamopen.com/ABSTRACT/TOPEJ-9-21

</comment></comment></comment></comment></comment><comment><USER9> This is great, but I think they should make it clear that this isn't using TensorFlow.

From the title and domain I though they either had ported TF to Javascript(!) or we connecting to a server.

<comment><USER10> Wait - what it is using, then? I had assumed it was TF under Emscripten or similar.

<comment><USER9> It appears to be a custom NN implementation[1] in Javascript, somewhat similar to convnet.js[2]

As far as I can see the API[3] isn't much like TensorFlow.

[1] [https://github.com/tensorflow/playground](https://github.com/tensorflow/playground)

[2] [http://cs.stanford.edu/people/karpathy/convnetjs/](http://cs.stanford.edu/people/karpathy/convnetjs/)

[3] [https://github.com/tensorflow/playground/blob/master/nn.ts](https://github.com/tensorflow/playground/blob/master/nn.ts)

</comment></comment></comment><comment><USER11> When it says "right here in your browser," it's not joking. On my desktop (Safari), the window becomes unresponsive after a few iterations. Does not happen in Chrome.

On my phone (Safari/iOS 9.3), the default neural nework doesn't converge at all even after 300 iterations while it does on the desktop, which is legit weird: [https://i.imgur.com/KNaXeHH.png](https://i.imgur.com/KNaXeHH.png)

<comment><AUTHOR> I'm sorry you're having problems with Safari. I can't reproduce on my end, but if you're still having problems you can raise an issue on github with some information about your system.

<comment><USER12> Works perfectly for me on Safari 9.1 with no extensions

</comment></comment></comment><comment><USER13> In case you are an idiot like me, you have to train your neural network by pressing "play".

<comment><AUTHOR> We would've liked to have it constantly training, but didn't want to abuse your CPU :)

<comment><USER14> It pauses when I switch tabs :(

</comment></comment></comment><comment><USER15> While it doesn't involve training, these 'confusion matrix' animations of NNs classifying images or digits are fun, too:

[http://ml4a.github.io/dev/demos/cifar_confusion.html](http://ml4a.github.io/dev/demos/cifar_confusion.html) [http://ml4a.github.io/dev/demos/mnist_confusion.html](http://ml4a.github.io/dev/demos/mnist_confusion.html)

Something about the high-speed updating makes me think of WOPR, in 'War Games', scoring nuclear-war scenarios.

</comment><comment><USER16> This demonstration goes really well with Michael Nielsen's [http://neuralnetworksanddeeplearning.com/](http://neuralnetworksanddeeplearning.com/). At the bottom of the page the author gives a shout out to Nielsen, Bengio, and others.

For someone (like me) who's done a bit of reading but not much implementation, this playground is fantastic!

<comment><USER17> Really awesome article!

</comment></comment><comment><USER18> Neat stuff, fun to play with. I wasn't able to get a net to classify the swiss roll. Last time I was playing around with this stuff I found the single biggest factor in the success was the optimizer used. Is this just using a simple gradient descent? I would like to see a drop down for different optimizers.

<comment><USER19> [http://imgur.com/ypBQEWx](http://imgur.com/ypBQEWx)

Add some noise, and use all the inputs, and one 8 wide hidden layer

edit: works better with a sigmoid activation curve, but it converges more slowly

<comment><USER20> Yeh you're on the right track. Nice pattern emerges on this after 160 iterations.

[http://playground.tensorflow.org/#activation=tanh&batchSize=...](http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral&regDataset=reg- plane&learningRate=0.03&regularizationRate=0&noise=25&networkShape=8,4&seed=0.38071&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification)

</comment></comment></comment><comment><USER21> Can somebody explain what I'm watching when I press play?

<comment><USER22> Hopefully this helps (correct me if I'm wrong, I'm still learning about neural nets):

Think of the whole neural net as a function:

input * weight = output

At each iteration, we feed in the input to the neural net. Then the neural net compares what output it gets to the correct output.

For example, input1 is 5, and the correct output for input1 should have been 2. But the neural net got 3 as the output. So it then decreases the weights slightly so it would get 2.75 next time it has input of 5. Repeat thousands of times. That's the basic idea for machine learning and neural networks.

The algorithm it uses to figure out how much to decrease the weights is called "backpropagation" which uses gradient descent. To explain gradient descent, as as a roller coaster track. Imagine the roller coaster starts off on a random location on the track. Then gravity takes the roller coaster down the track until it ends up on a low point between two hills and stays there. This is the new location of the roller coaster. This new location is nice because it has the lowest energy the roller coaster could find, so it stays there. (We use derivatives to figure out the slope of a curve, which then gives us the direction where the curve goes downhill).

In neural networks, the roller coaster curve is the "cost function", which basically calculates the amount of difference between the neural net's output and the actual correct output it should have got. The initial weight is the roller coaster's initial position. The new weight is the roller coaster's final position, at the bottom of the cost function curve. This new position thus gives us the lowest cost.

Note that there may be even lower valleys, but when we roll the rollercoaster it stops at its nearest low valley. This is why we randomize the weights at the beginning - to put the roller coaster near possibly even lower valleys.

<comment><USER23> Okay, so it works by minimizing (equiv. maximizing) some function. But that doesn't say much about how it "learns" the gradient. What function does it care about? Average squared error (predict_prob-Z_i)^2 ? Average absolute error? The likelihood function of some assumed distribution? Maximum distance between the classification border and closest observed points? If I saw someone carrying a bag full of blueberries and some bread home from the grocery store and asked to know how they chose to buy that, to which they replied "I had a list of characteristics which I thought where important for groceries to have in this trip to the store. For each grocery item, I recorded a vector of degrees to which the item possesses each of those characteristics. Finally, I chose the group of groceries that had the best combination of degree vectors", I still wouldn't really know anything about why they bought the blueberries and bread.

<comment><USER24> The function it minimizes is called the "loss function", and its value for the training and test sets are shown in the upper right area. AFAICT the site doesn't say how it's computed, but I think it's average squared error. The gradient is not learned; if you think of the loss function as a real-valued function of the weights, the gradient is just the partial derivatives with respect to the weights.

</comment></comment></comment></comment><comment><USER25> this is very nice! I think that the reason swiss roll doesn't work as easily might be because of initialization. In 2 dimensions you have to be very careful with initializing the weights or biases because small networks get more easily stuck in bad local minima.

<comment><USER26> In this case you see that it is the swiss roll so you could say pick "proper initialization".

But that technique would not work when you cannot see that it is a "swiss roll" or in multiple dimensions.

<comment><USER27> I'm pretty sure he wasn't talking about the swiss roll specifically. Big gains in neural net performance have been made through better initialization schemes (not dataset specific, just in general, e.g. an initialization scheme might adapt the initial weight distribution depending on the number of hidden units in the next layer), and smaller models are in general more sensitive to initialization.

</comment></comment></comment><comment><USER28> Has anyone been able to learn a function for the spiral (Swiss roll) data that's as good as a human-designed function would be?

<comment><USER29> For this simple example just choosing the largest possible fully-connected network with ReLU and L2 regularization to prevent overfit quickly converges to a nice spiral (test loss of 0.001 for me):

[http://playground.tensorflow.org/#activation=relu&regulariza...](http://playground.tensorflow.org/#activation=relu&regularization=L2&batchSize=30&dataset=spiral&regDataset=reg- plane&learningRate=0.03&regularizationRate=0.003&noise=0&networkShape=8,8,8,8,8,8&seed=0.38911&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification)

<comment><USER30> I wouldn't call it quick... spiral in 150 iterations, with sigmoid magic: [http://playground.tensorflow.org/#activation=sigmoid&regular...](http://playground.tensorflow.org/#activation=sigmoid&regularization=L2&batchSize=10&dataset=spiral&regDataset=reg- plane&learningRate=0.1&regularizationRate=0.01&noise=0&networkShape=8&seed=0.63555&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification)

I find the pulsating unsightly.

</comment></comment></comment><comment><USER31> You could totally optimise network architecture by crowdsourcing topology discovery for different problems into a multiplayer game with loss as a score.

</comment><comment><USER32> So glad anns are becoming mainstream

Eventually it will have to be recognized as a new species of life, so I hope programmers, tinkerers and everyone else keeps that in mind because all life must be respected

And this particular form will be our responsibility, we can either embrace it as we continue to merge with our technology, or we can allow ourselves to go extinct like so many other species already have

For the naysayers - ever notice how attached we are to our phones? Many behave as if they are missing a limb without it - it's because they are, the brain adapts rapidly and for many, the brain has adapted to outsourcing our cognition. It used to be books, day runners, journals, diaries - now we have devices and soon they'll be implants or prosthetics

The writers at marvel who came up with the idea of calling iron man's suit a prosthetic were definately onto something and suits like that are probably our best chance of successful colonization of other planets. We'll need ai to be our friend out there, working with us

</comment><comment><USER33> This is a lot of fun. The default dataset is too easy, though, try out the Swiss Roll one!

<comment><USER11> There is a reason why sin(X) is an input property. :p

<comment><USER33> Using sin(x) or the other input features like x^2 goes back to making it too easy, though. So far the best I can do is 7 layers of 7 which gets a loss of 0.02. 3x7 is _almost_ cracking the Swiss Roll but can't quite finish it off and gets stuck at 0.05: [https://imgur.com/Z3f2ECc](https://imgur.com/Z3f2ECc) ... Surprisingly, 2x8 can do it, as long as I have noise or regularization on, but 8/7 then seriously struggles. Is 16 neurons a critical limit here?

<comment><USER34> I managed to get to 0.01 loss from only x1/x2, using 3 hidden layers, L1 regularization, a bit of added noise, and some patience: [http://i.imgur.com/Y3zKpJF.png](http://i.imgur.com/Y3zKpJF.png)

<comment><USER33> Yes, noise & regularization seem to be key here. I've gotten a 2-layer with 7/8 neurons down to 0.06 and dropping but only with noise & l1: [http://playground.tensorflow.org/#activation=relu&regulariza...](http://playground.tensorflow.org/#activation=relu&regularization=L2&batchSize=6&dataset=spiral&regDataset=reg- plane&learningRate=0.01&regularizationRate=0.03&noise=10&networkShape=8,7&seed=0.52682&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&noise_hide=false) Final loss of 0.051. Interestingly, increasing noise from 10 to 15 destroys performance, loss of 0.47.

</comment></comment></comment></comment></comment><comment><USER10> This is a very cool toy. As someone with no experience in ML, this is an interesting visual approach to the absolute basics.

And great for challenging your friends in an epic battle of convergence!

<comment><USER35> If you like visual demonstrations of ML topics, you may be interested in [http://ponder.hepburnave.com](http://ponder.hepburnave.com). It is an interactive demonstration of a self-organizing map, generating a 2D-map from a spreadsheet with multivariate data. It's an unsupervised learning approach, good for data exploration tasks, less so for classification tasks (/shamelessPlug).

</comment></comment><comment><USER36> I'm not well versed in neural networks but a lot of the new neural network software stacks coming out seem to be quite plug and plug. What kind of expertise would engineers need to have a few years from now when the technology is well developed and it doesn't need to be rewritten from scratch every time?

<comment><USER37> I'm not qualified to answer this, but I will anyway.

To "operate" neural networks (as opposed to writing a framework for them), you need to know the building blocks. There are basic blocks like fully connected layers, convolutions, and nonlinear activations. Beyond those, there are higher level building blocks like LSTMs[1], gated recurrent units[2], highway layers[3], batch normalization[4], and residual blocks[5] that are made up of simpler blocks. Learning what these do and when it's appropriate to use them requires following current literature.

Operating neural networks requires some systems engineering skill. It takes a long time to train a single network and you'll find yourself trying many different architectures and hyperparameters along the way. Because of this, you'll want to distribute the training across many different systems and be able to easily monitor and deploy jobs on those systems.

A solid grasp of mathematics is useful to effectively debug your networks. You'll frequently find your network doesn't converge or gives totally garbage results, so you need to know how to dig into the network internals and understand how everything works. This is especially true if you're implementing a new building block from a paper.

Finally, know your machine learning and statistics fundamentals. Understand overfitting, model capacity, cross validation, probability, model ensembles, information theory, and so on. Know when a simpler model is more appropriate.

[1] ftp://ftp.idsia.ch/pub/juergen/fki-207-95.ps.gz

[2] [http://arxiv.org/abs/1409.1259](http://arxiv.org/abs/1409.1259)

[3] [http://arxiv.org/abs/1505.00387](http://arxiv.org/abs/1505.00387)

[4] [http://arxiv.org/abs/1502.03167](http://arxiv.org/abs/1502.03167)

[5] [http://arxiv.org/abs/1512.03385](http://arxiv.org/abs/1512.03385)

<comment><USER36> So you don't think some of these details will not be automated away in the near future so that it doesn't require a specialist to do operate a neural network?

<comment><USER38> Already, it's not nearly as hard as this demo makes it look. There's one recent advance in particular that isn't in this demo, and that is Batch Normalization.

If you've played around with it a bit, I'm sure you have seen that deeper layers are hard to train... You see the dashed lines representing signal in the network become weaker and weaker as the network gets deeper. BatchNorm works wonders with this. It takes statistics from the minibatch of training examples, and tries to normalize it so that the next layer gets input more similar to what it expects, even if the previous layer has changed. In practice you get a much better signal, so the network can learn a lot more efficiently.

Without BatchNorm, more than two hidden layers is tedious and error-prone to train. With it, you can train 10-12 layers easily. (With another recent advance, residual nets, you can train hundreds!)

Such advances pushes the limit for what you can train easily, and what still requires GSD ("graduate student descent", figuring out just the right parameters to get something to work through intuition, trial and error). You still have to watch out for overfitting, but the nice thing about that is that more training data helps.

</comment></comment></comment></comment><comment><USER39> Beautiful. The next time someone asks what is machine learning about I'm going to send a link to this page.

</comment><comment><USER40> "Don't Worry, You Can't Break It. We Promise."

(Nice, but it's completely unclear what's going on.)

<comment><USER41> My MacBook Pro running El Capitain froze my mouse and keyboard. I had to do a hard reset.

</comment></comment><comment><USER42> Is a 50/50 training:test a normal default ratio for an ANN? I expected to see a higher amount of training data represented as the initial setting.

</comment><comment><USER43> One of the finest data visualization I've seen.

</comment><comment><USER44> This is so great. An easy way to show my friends WTF I do sometimes for math/CS work. Thank you so much.

</comment><comment><USER45> I wish they had more interesting data sets.

<comment><USER46> Submit a pull request.</comment></comment>


Apple has blocked Clearview AI's iPhone app for violating its rules - <AUTHOR> https://techcrunch.com/2020/02/28/apple-ban-clearview-iphone/ <comment><USER2> Comments moved to [https://news.ycombinator.com/item?id=22453402](https://news.ycombinator.com/item?id=22453402), which is currently on the front page and seems to have the more original source.</comment>


I barely graduated college, and that's okay - <AUTHOR> http://www.vikparuchuri.com/blog/i-barely-graduated-college/ <comment><USER2> Is there any real take away here? Effectively it is "I floated through life and got a few lucky breaks which led me into a relatively comfortable career that I enjoy". I'm hard pressed to find something unique or interesting about this long autobiographical read.

My story started out in a similar fashion:

I did terrible in HS, slept through most classes, aced most tests, and did no homework. I finished with a 2.7 GPA. I decided early on it was okay for me to go to community college because from there I could save money and still transfer to a good school.

I deviated from the norm when I then focused on my career while going to school, working for a consulting company at 18, then landing a job as an IT security admin. I transferred to a state school because it was easy and cheap and I had realized work experiences was far superior to college credentials.

I have since then had a mostly enjoyable and relatively lucrative career as a programmer. The key take away from my story? Complete school because the piece of paper is important but focus on work early if you want to stand out and get ahead of the crowd.

<comment><AUTHOR> I think the key takeaways that I take from it are (of course, I wrote it, so ymmv):

If the system doesn't work for you, it doesn't mean you're not capable.

Follow your passions, not external dictates.

Try to get as many different perspectives on life as you can.

Coding is making credentialing and "proving" you can do something much easier.

As for luck, I have a bad habit of framing things as lucky, and to some extent they are, but there was also hard work behind them.

You have an interesting story, thanks for sharing it.

<comment><USER3> Let me offer some (hopefully constructive) counterpoints:

> If the system doesn't work for you, it doesn't mean you're not capable.

Sure, but working with the system is a useful capability. Don't be lazy and dismiss it as worthless just because you can't (or don't want to) figure it out. I've made this mistake and regretted it.

> Follow your passions, not external dictates.

Following your passions is great, but doesn't always work out. Life is often about finding a balance between something you really like doing and something that society values enough to pay you for doing it.

> Try to get as many different perspectives on life as you can.

Yes, of course, but the challenge is weighing these perspectives appropriately. Everybody's perspective is right from some particular viewpoint, the trouble is figuring out what that viewpoint is, whether it applies to you.

> Coding is making credentialing and "proving" you can do something much > easier.

Completely agree.

> As for luck, I have a bad habit of framing things as lucky, and to some > extent they are, but there was also hard work behind them.

Sure, you always need some hard work and talent to succeed but it's often not sufficient, and luck is sometimes the necessary component that makes the difference.

ed: One last meta point that occurred to me was that you became successful when you took charge of your career. When you were just doing what others told you, you seemed both unhappy and underperforming. The lesson I would take is that we all need to take responsibility for our happiness and success. Sometimes this means introspecting and figuring out what we ought to be doing and not just drifting around listening to what we're told by society, our peers or parents.

</comment></comment></comment><comment><USER4> One thing I'm told has changed about the U.S. educational system over the last 20 years is that the amount of individual mentorship that students receive has strongly declined.

When I transitioned into graduate school, one big surprise was that school actually became easier in several senses: \- I had a clear role model (mentor) \- I had several good sounding boards for ideas (professors had more time for me than for undergrads, and graduate students were also closer friends) \- More frequent communication with professors translated to better grades (if they're aware of your thought processes, they can more clearly understand how to interpret your work).

As a result, I felt much more involved in the work I was doing. That feeling of involvement made me more invested (and more confident) in my work.

I know that some colleges are working on programs to supplement mentorship and structure, and are finding great success. For example, in the ASAP program ([http://www1.cuny.edu/sites/asap/](http://www1.cuny.edu/sites/asap/)), the graduation rate is more than three times the national three-year graduation rate of 16% for urban community colleges. It would be interesting to see whether life outcomes ("success") are any different for people who go through this program.

</comment><comment><USER5> I personally loved this article because it lets others in the same situation know some risks that other people took to reach their ideal career.

I remember graduating college with a 2.9 but that's after being put on academic probation twice because i was burning the candle at both ends. Meaning i was working two jobs to pay for school, but was so tired I didn't do well in school. Not until 2+ years in the military did I start prioritizing what needed to be done. My GPA still wasn't high enough to show my worth, but i was happy with what i accomplished and knew what I can do. This was only a year ago, so I'm still not near my dream career, but my GPA won't stop me. Just one thing checked off on my checklist

I realize after graduating that I only wanted to finish school at the time because of society told me i couldn't have a career unless I did. So instead of going to school for education and learning i just went through the motions.

The basic thing i got from this article was just his willing to share what he went through. It might empower others to purse what they love or not, but one can't be scared to take a risk because you are scared of failure or consider yourself unworthy.

</comment><comment><USER6> I'll put it this way- My first job out of college was one of the few that DID NOT want my GPA, and I ended up doing awesome, I'm fairly successful now.

<comment><USER7> is it related to programming ?

<comment><USER6> Absolutely. Startup developer :) I am (mostly) self-taught.

</comment></comment></comment><comment><USER8> Hi Vik, you seem like a younger version of me. I worked QC testing in an electronics factory between semesters. I graduated with a 2.1 GPA in the middle of the recession. I took whatever job I could find. Work didn't pay much but they did send me travelling.

In 2005 I applied to the foreign service as well. Instead of going to the second interview I bought a last minute ticket and flew to Europe. After traveling throughout Europe I based myself in France where I've been pretty successful.

The point being is all I had was my hard work and moxy to help me get by. All those rejection letters (sometimes not even that) had only emboldened me to prove my capabilities. I wasn't going to let my GPA be my only defining value.

<comment><AUTHOR> Thanks for sharing! Interesting how we went through a lot of similar steps. I'm glad things worked out well.

</comment></comment><comment><USER9> For myself - while in college for statistics, I focused much more on my side jobs (in business and marketing) than college. I stopped going about a year and a half ago with a semester left, simply because the job opportunities got so good (opportunity/sanity cost of going to school was outweighed by fulfillment/high salary/career track of working more).

Eventually, I suppose I'll take night classes and finish college. But its quite low on my priority list. My work experience during college was by far more valuable to my career than the actual college education (which I thought was great, too).

</comment><comment><USER10> I'm a little curious as to how he managed to kind of waft into the Foreign Service so easily. I have much stronger academic credentials, foreign language skills, and relevant work experience, and I spent years and years trying to get in to no avail, despite aceing the required tests each time.

I suppose its just down to fate, and I'm glad he found something he loved doing, but man, it's disheartening to read about someone casually walking into your dream job and just as casually walking away.

<comment><AUTHOR> I didn't see this post pre-edit, so I'll assume that it wasn't too bad. If you were suggesting some kind of affirmative action (as someone hinted), then that really wasn't the case.

The reason I applied to the Foreign Service is because it is a meritocracy. It all came down to how you did on the test, and later the interviews/case studies.

It's unfortunate that it's your dream job and you didn't get it, and I sympathize, but the Foreign Service admission process is one of the most transparent ones I've seen. The first step is passing a test, and anyone above a certain threshold moves to the next step. There are more tests, and anyone above those thresholds gets on the list to be accepted.

<comment><USER10> This is actually not the case; you take the first-round test, and then if if you pass you submit a resume into a black box (the 'Qualifications Evaluations Panel') which evaluates your qualifications 'holistically' and then you are invited to the interview round. It's quite possible to achieve the highest possible marks on the test and never be invited to the final round, as I can attest.

Perhaps the system was different when you applied in 2008.

<comment><AUTHOR> When I applied, you took the test. If you passed the test, you then had to go online and submit ~10 short essays. If you passed that, you went on to the oral assessments. There may have been a resume component involved when you submitted the short essays.

</comment></comment></comment></comment><comment><USER11> My story is similar to this. I, too, did poorly in school. There were so many things working against me. I was constantly too tired in the mornings for my brain to function, so I missed important concepts. Or, the pace of learning was too slow, and my brain wanted to run with it. Or, the content was missing important piece of information that help my brain connect the dots. My brain doesn't assimilate information well when it's forced to take it for fact without answering supporting questions.

The school system is not made for people like me, and we as a society are absolutely terrible at guiding people into careers they would excel at. I'm lucky that I guided myself to software development. Many others are not so lucky.

One day, we'll figure out how to guide people to where they can excel, and when that happens we'll be much better off as a society (post-scarcity, Star Trek future).

</comment><comment><USER12> I also experienced a letter of anti-recommendation from a school administrator once, which by total chance my mother intercepted and prevented from being sent (and gave the woman quite an earful). Reading that detail makes me wonder if this is actually a common practice.

Stunning to me that someone would think to do that to a kid.

<comment><USER13> I'm not a lawyer, but it sounds risky from a legal perspective. People can sue companies for giving them a bad reference, so can they not also sue schools for these "letters of anti-recommendation"?

<comment><USER14> The sophisticated version is to damn with faint praise. I imagine it is far more common.

</comment></comment></comment><comment><USER15> I enjoyed reading your story, thanks for sharing. As for brd's comment just disregard it. His generalization just means he's unwilling to get a deeper understanding of where you are coming from. Thanks again.

</comment><comment><USER15> I enjoyed reading your story, as for brd below, diregard him. You published your own words and he shouldn't be criticizing and generalizing your journey to date> Thanks for sharing.

</comment><comment><USER16> I need advice. Excuse me for how disorganized this all is, I'm just typing. I'd like you guys to give it to me straight, tell me if I'm being a spoiled brat who needs to grow up, or tell me if I should follow what seems right to me. I've made a throw away because I don't want this tied to my name, and my main account is my full name.

I'm not doing well in college. I just came from a midterm that I almost certainly failed. I've not done well in any of my classes besides a bullshit social science course that anyone can do well in. In my real courses, statistics, calculus, and a theory CS course, I'm failing desperately. I don't go to class anymore.

I can recognize that it's entirely my maturity and work ethic that is the problem here. When I do find myself going to class, I comprehend fully. I don't find it hard to catch up, even. When I leave class, I don't do homework and I don't study.

This is the first time I'm admitting it to myself, but I don't believe I can succeed in this system. I don't know if I'm lying to myself when I say I love learning, or if I'm just much more subpar than I'm willing to recognize.

It's not that I'm an unmotivated individual, it's just that I've always been a self-learner. I hate this whole system of, go to a class for 50 minutes and study a subject, spend an hour later in the day doing some homework in the same subject, and maybe do this every other day. When I learn, I'm much better at spending 10 hours really consuming the material, flying through chapters, and then doing that for 5-6 days in a row. Formal education doesn't cater to me in this regard.

I love to program. The few hours in the day that I'm happy are when I'm reading mailing lists and and working on personal projects. I truly believe that if I was to drop out, I wouldn't waste my time doing anything else. I'd be able to really embrace the style of learning that works best for me.

I think that a few years from now I would find myself more prepared for the real world than most of my friends, although, I think they would similarly find that they have an easier time finding work than I. But that doesn't worry me a ton.

I've watched my father be an EE for his whole life, programming and managing for a boring company that makes controllers. I don't want to enter that ecosystem _at all_. I have watched him hate his life in exchange for a paycheck, and I'd like to opt out from that system entirely. What best fits that? The market for full stack engineers / web developers / app developers. And alternatively, the start up scene.

I wonder if the reason I'm involved with coding at all is because it caters so much to my preferred style of learning. Maybe that's why I latched onto it so hard in late middle school.

Anyway, thanks for reading. Let me know what you guys think I should do, or if there are any questions you think I should pose to myself to really figure this out.

<comment><USER17> _It 's not that I'm an unmotivated individual, it's just that I've always been a self-learner._

Maybe _this_ is where you are fooling yourself. College was originally _designed_ for self-learners. The most successful engineers I've seen are the ones who are busy learning on their own, plus going to classes "on the side" to supplement their own exploration with a bit of formal rigor. You are standing in the middle of a bounty of resources for learning- forget about classes, why aren't you seizing that opportunity to learn on your own from the wealth of brilliant people & resources?

If you were falling behind and missing classes because you were so busy learning something just as good, that's one thing. But if you are falling behind and missing classes and not really doing much learning on your own either, to say "I'm a motivated learner who the school just doesn't cater to" is misleading yourself. If you are a motivated learner, _you would be out there learning_ , as there is no better place to learn regardless the class structure.

Does that mean you are a talentless hack? No. Looking at my own history, what it means is you are lying _to yourself_ (which is the worst kind of lie) because you don't want to put in the hard work. Once you can see through that self-deception, you can get back on track.

<comment><USER16> This is exactly what I need to hear.

Thank you.

</comment></comment></comment><comment><USER18> neat story, but i'm more fascinated on how he got into the foreign service- something that is arguably more difficult to get into then med school.

</comment><comment><USER19> Thank you for posting this.

</comment><comment><USER20> He did graduate, though. That little sheepskin may seem only like a small thing, but the difference between having it and not having it should not be underestimated. He also had parental support, which, until you live without it - in the US, it's many people's "safety net of first resort" \- you just cannot imagine the benefits it gives you. You can fail in all sorts of ways and still come back - you have a place to lay your head when it comes down to it, and that counts for a lot. The difference between that and thinking "hm, homelessness actually possible!" is so stark it's hard to overstate. The poster "floated by" and is doing great; I can share some experiences on what it's like on the opposite end of the spectrum.

I didn't go to college because I had to leave home early (was told I had to be out at 16, was out by 17) to support myself, and received no parental support when I left. I did, however, manage to work myself from a tech support hamster at 17 to sysadmin, then junior developer, then finally senior developer. I never lacked for work, and have made OK, but not amazing money. That all sounds great, but it comes with serious costs, especially if you're just "decent" or "pretty good." Here are my experiences:

\- You're probably going to have to work 2-3x as hard as an equivalently skilled person to prove that you deserve to be there.

\- You may have to deal with much lower raises, or deal with simply being passed up for them altogether. This one can be more subtle, so can be hard to prove. So, you may be picked for the challenging work, asked to lead teams, told you're the best on the team by the boss(es), and receive the company awards, but when it comes to raise time, you wonder why you just went from 80 to 85k, and the people below you in rank went from 80 to 110 (and aren't happy with that and leave anyway. Heh.) This one sucks the most, because it's the one you have the least control over, and will ultimately impact the number of choices available to you later on.

\- You may experience another subtle thing. This: "we know you're good, but you don't have a degree, so we believe you're less likely to leave because the hoops for you to change jobs is higher, so we can ask you for more than your degreed peers - and we're probably right about it." You will be the person on- call more often (if that applies to you - sysadmin/devops), you will be "expected" to stay later, and just generally push harder to prove yourself.

\- You're going to spend a LOT of your free time teaching yourself knew things, honing your skills, and brushing up on the theory you _never learned in a setting that allowed you to immerse_. Now, maybe you have superior learning capabilities and can just sponge this stuff up in just a couple of your after-work learning sessions, but I don't think it's a stretch to say that for those of us who aren't, we spend a lot of time making them stick over the years. Now, we know that everyone needs to do this stuff these days, and that it's becoming more necessary every day. No question about it. However, you will have to do it _even more_ to make sure you're at the top of _your_ game. You may have annoying gaps in your knowledge, and will have to go back (in some cases, WAY back) to fill that stuff in. (Now that there's Khan/Udacity/MIT OCW/others, this is actually a LOT easier to do.) If you're only decent/pretty good, you're going to have run significantly faster to stay in place. Forget most of your social life. Again, this does not apply to you if you happen to just be an superhumanly fast learner, but for the decent/pretty good, it probably does.

\- Culturally, your degreed peers may have some trouble adjusting to you. Of course this happens at every level (people who went to state school treated differently than those who went to a top university, for example), but you'll be the strangest of the bunch. People may wonder if you know anything at all(!), not just about your field (which is easy enough to prove), but about the world around you: politics, economics, sociology, literature, the lot of it. Sometimes you may be looked at as a strange feral orphan child. They just may not even have a baseline for what to expect from you. The software world(s) may be more meritocratic and/or egalitarian, but it's not utopia. Now, this sounds /all/ bad, but I actually have always liked this one; if you are a dedicated self-learner, you can blow people away pretty easily. They expect you to not be able to tie your shoes, but then you rattle off your knowledge X, Y, and Z and show them your elegant, well-structured code and they are agape. This one is fun!

\- Remember that the degree acts as a proxy. It's a way for organizations to outsource not only judgment about your knowledge, but your socio-economic background (in most cases), your "level of socialization", your ability to "follow-through" (and your personal story about how you had to leave home early and support yourself alone and spend all your free time learning unfortunately will not count in many/most places. Sorry!), and how "convincing" you are. You may not be listened to (even if you turn out to be right, at least for a time), your opinions may be given less weight, people may feel more comfortable cutting you off when you're talking, etc. This one can be surmounted: if you are right enough times when others are wrong you may start be taken _very_ seriously. Still, the bar is higher for you. Much higher. This one is probably obvious, but is worth stating.

\- You may have to content yourself with smaller and/or less-respected organizations. I know, I know meritocracy, etc. If you're amazing, you can most certainly get into Google and friends, but again, if you're only decent/pretty good, you /probably/ won't. For those of you who have ever wondered "who works /would want to work at crappy place X?", it's not just CS grads who couldn't hack it at Google or the 'printer guy' who got promoted to LOB programming. Sometimes it's smart, pretty good devs who are either just short of what's needed for the bigger players or are incredibly intimidated by the interview processes/hiring filters. It's funny to think that these places are (based on comments I've read here on HN) inundated with resumes from people who are nowhere even close to qualified, but people who _are_ qualified, or at least could be with some knowledge gaps filled in/"senior- level" mentoring don't even apply in the first place. Doesn't surprise me that something like Starfighters now exists to (maybe?) try to reach these people and mop them up. I would also say that you can be _very_ valued at startups. You may be one of those great "get things off the ground" people, like many self-starters, and if you are, your metaphorical phone will ring a lot. I wouldn't say it evens things out, but it sure helps (and feels good to be real a go-to person!)

\- Unless you really are one of the best, expect to always be questioned. You probably will need to psych yourself up a lot, because it's very easy to get disheartened and discouraged when you feel like you're never quite good enough/people are always going to assume there's "a lot he/she probably just doesn't know if we're honest" (even if you have been working for decades and proven yourself with real work)/you don't ever quite fit in culturally because you've never had the "college experience." You need to be tougher (psychologically) than your degreed peers, because you will have to work much harder to earn respect.

So this person floated, others swim upstream to get to where he is. It can be done for people without degrees, but it actually is hard. You really can't imagine what that little piece of paper - even if you just "barely" earned it - can do you for you. It's not just about getting hired.

"I'm already starting to see some really smart high schoolers skip college to get into the workforce because they know what they want, and good for them."

If these "smart high schoolers" have the ability to have someone else pay for their education, I recommend that they go to school. If they have Zuckerberg- parent level of support or are already mindblowingly skilled/knowledgeable, OK, sure, don't bother. They're probably going to succeed no matter what. If they are on the margin, though - parents wealthy enough to send them to a real university, but not wealthy enough to Zuck them, tell them to go instead. Or at least work for a couple of years, then go back. If someone gave me a few hundred thousand to go to a great university, I'd still go in a heartbeat. You cannot replace the _quality_ of networks you'll get there. You cannot replace the _immersion._ You cannot replace the _prestige._ You just can't. The magic of university is being surrounded by "brilliant, ambitious, well-connected elites." I'm probably right in the sweet spot of the target of today's alternative education: I couldn't go to college, I'm a regular user of MOOCs, I do self-directed learning, I'm a devourer of books, I'm someone who does coding challenges, I'm fairly smart, AND I've managed some level of success, which proves it's possible, and yet I would _still_ tell these people to go to college.

</comment><comment><USER21> tl;dr: all is well that ends well.

It seems we only hear from those who did poorly in school that also bounced back and became successful later. I guess those that didn't don't have much incentive to telegraph their presence and their achievements (or rather lack thereof).

<comment><AUTHOR> I debated whether or not to write this because, as I mentioned, I have a large degree of impostor syndrome because of my experiences. It's hard to define what "successful" is.

The process of "bouncing back" is also a process of slowly gaining confidence. I'm not sure I would have been able to write this a year or two ago. It's unfortunate that both sides of the issue aren't explored, but that may be due to the confidence problem.

<comment><USER21> > I debated whether or not to write this because, as I mentioned, I have a > large degree of impostor syndrome because of my experiences.

I don't know if you need to feel like an imposter considering you sound like a typical programmer type.

Just keep flaunting your stuff. It's the modern hacker way.

</comment></comment></comment><comment><USER22> life advice from people under thirty

<comment><USER23> Does that make it more or less relevant?

<comment><USER24> Depends upon what it is about. For example, listening to 30 year olds about college is probably more relevant than listening to 50 year olds, because things changed in that 20 years between when they went to college and many of the lessons and struggles are fresher in the minds of the 30 year olds. On the other hand, I'd give more weight to marriage advice from a 50 year old who has been married for 25 years compared to a 30 year old married for 5.</comment></comment></comment>


Printers are evil (and how to make them less evil) - <AUTHOR> http://blog.maxmackie.com/107262393

<comment><USER2> Shameless plug... just use HelloFax and skip the entire printing and scanning part anytime you have to fill out a form.

<comment><AUTHOR> Cool idea, might give it a try in the future.</comment></comment>


College Board Drops Plans for SAT Student Adversity Score - <AUTHOR> https://www.wsj.com/articles/college-board-drops-plans-for-sat-student-adversity-scores-11566928181?mod=rsswn <comment><USER2> Quoting from the article:

" _Instead, it will try to capture a student's social and economic background in a broad array of data points. The new tactic is called Landscape and doesn't combine the metrics into a single score._ "

So they didn't really drop it. They just revised it a bit and gave it a new name.

<comment><USER3> > They just revised it a bit and gave it a new name.

Asians are a minority but out-score whites. So, if you were responsible for assigning an adversity score:

1\. Would you say that Asians should get a lower score (since they tend to do better) or a higher score (since they're a minority)?

2\. How much of an income difference between an Asian student's family and a white student's family to result in the same score?

If you choose to penalize Asian individuals for their racial group doing better on average, how could you justify this to Asians who feel discriminated against? What would you say when they point out that you're penalizing them compared to white people despite being historically discriminated against?

If you choose to penalize white individuals despite their racial group doing poorer on average, how could you justify this to whites who feel discriminated against? What would you say when they point out that your system increases, rather than decreases, racial gaps in colleges?

To be clear, I agree with you -- this revised system has many of the same problems as the original proposal. But, now some of these tough questions aren't on their shoulders.

<comment><USER4> I have said this for years. Asians, particularly Asian males, are the only racial group in America subject to overt policies meant to disadvantage them on the basis of their race, and nobody cares. Asian immigrants to the US tend to embody more than any ethnic group precisely the characteristics that are supposed to constitute America. Meanwhile we're playing games of trying to tease out how implicit bias might be impacting XYZ, or taking population outcomes as evidence of systemic bias impacting other groups, when disadvantages put on Asian individuals are explicitly laid out as systemic and on the basis of their race.

<comment><USER5> Meh. Happens to us white males too. Adversity makes us stronger or something.

<comment><USER6> I recall the pleasure of riding a slow, long school bus meandering around the city for 3 hours a day for 4 school years because there were "too many whites/Asians" at a school on the same block where I lived. Gotta love bureaucratically-proscribed "equality of outcome" policies foisting inequity of treatment externalities onto individuals based on their race. Definitely not racism, because racism can't happen to white males due to the celebration of the hierarchy of victimhood, right?

</comment></comment></comment></comment></comment><comment><USER7> "The College Board, the New York-based nonprofit that oversees the SAT, said it has worried about income inequality influencing test results for year"

If only these systems in place focused on this rather than race. I know race in the USA correlates strongly with socioeconomic status and outcomes, but its turned out to be such a divisive issue that consistently antagonizes stakeholders and leads to controversial policy decisions.

Sure poverty disproportionately affects some ethnic groups more than others, but it nonetheless has pervasive effects on the health and economic outcomes of people of all races. Hopefully the next system to replace the last takes into account the economic handicaps of students and not their race.

<comment><USER8> I can agree with you that it might be a more productive fast way to fix some of the issues, but the fact is that racism still affects even the well-off people of color. Maternal deaths related to birth are a recently studied example, as is upward/downward social mobility among the well-off.

<comment><USER9> Such studies aren't capable of determining that "racism" is the cause.

<comment><USER10> What else could be a cause?

<comment><USER8> While I very much disagree with that idea because it has been disproven plenty of times, the usual "not racism" explanation that people suggest is some inherent difference between ethnicities unrelated to historical subjugation or current policies. At this point I take the proponents of such "not racism" explanations as willfully ignorant implicit racists at best.

<comment><USER11> "If you don't agree with me, you're a willfully ignorant implicit racist."

Not a productive remark; painting all who disagree with you as "racists" devalues the legitimate use of the term and prevents honest discourse.

</comment></comment></comment></comment></comment></comment><comment><USER12> I think I'm well within the minority here, but I am Conservative and support this initiative. I support it mostly because I'd prefer a transparent system used widely to a bunch of different opaque systems at all the various universities. I also ascribe to the belief that someone who is exceptional in an area where the average may be below proficient is truly exceptional. That being said I want to work to the ideal that something like this won't be necessary in the future of zip code does not determine outcome.

<comment><USER13> It would only be transparent if admissions were decided by an algorithm based on inputs. With these data from College Boards, you get no transparency: the schools are still allowed to do whatever they want, and now they get extra data for a posteriori justification of their decision.

<comment><USER14> Why shouldn't the schools do what they want? Nobody is entitled to go to the school they want to go to.

</comment></comment></comment><comment><USER15> The problem with diversity programs in education is that they are half baked, if you wanted to truly solve the root causes of racial discrepancies in education you'd need reform at the elementary school level. School districts are too tightly coupled with housing for stakeholders to advocate for reform at that level. If you design programs that try to tackle these discrepancies without aiming at the root cause you get mishaps like this.

<comment><USER11> How do you propose to decouple elementary schools and houses? Busing? We saw what kind of a disaster that was [0] [1].

Personally, I think on-line schooling is a better option. Internet is becoming broadly available, and it seems that serving every one the same curriculum (albeit with some adaptivity to what a student is doing better or worse on) seems the most viable way to make sure each student gets the same education, I would think. Then maybe provide a smaller staff at local schools with tutoring and counseling.

Online education does a good job [2], and has had good results, even in the third world [3] [4]. It of course needs further development, but I see no reason why it couldn't be a strong contender to mitigate this issue.

[0]: [https://slate.com/human-interest/2014/02/how-the-lefts- embra...](https://slate.com/human-interest/2014/02/how-the-lefts-embrace-of- busing-hurt-the-cause-of-integration.html)

[1]: [https://www.npr.org/sections/ed/2016/10/06/496411024/why- bus...](https://www.npr.org/sections/ed/2016/10/06/496411024/why-busing-didnt- end-school-segregation)

[2]: [http://news.mit.edu/2014/study-shows-online-courses- effectiv...](http://news.mit.edu/2014/study-shows-online-courses- effective-0924)

[3]: [https://borgenproject.org/online-education-in-developing- cou...](https://borgenproject.org/online-education-in-developing-countries/)

[4]: [https://www.onlinecoursereport.com/democratizing- education-o...](https://www.onlinecoursereport.com/democratizing-education- online-education-and-the-developing-world/)

<comment><USER16> Online schooling only works for motivated, disciplined students. Most children need to have a qualified teacher in the same room in order to learn effectively. That especially applies to subjects that students don't enjoy, but which they need to learn in order to function in society.

<comment><USER11> Maybe the question we ought to be asking, then, is, "Why are our youth not motivated to study?" This seems to be true across all background, excepting one: first-generation Americans. I was certainly motivated to study, even things I didn't like. I knew it was my path to a better life. The "second- generation problem" is well-documented [0], and so the obvious question is: what can we do to inculcate that same motivation in our youth? To make them self-directed? On that, I am not sure.

[0]: [https://dyske.com/paper/1258](https://dyske.com/paper/1258), [https://www.jstor.org/stable/2547418](https://www.jstor.org/stable/2547418)

</comment></comment></comment></comment><comment><USER17> How is the Harvard Asian discrimination lawsuit going lately? I haven't heard anything in a while.

</comment><comment><USER18> >a diverse student body is part of the educational mission of a school.

Why? Why should anyone attempt to intentionally manipulate the ethnic backgrounds of college students? If you award admission bonuses for the symptoms of racism like being worse off financially or going to a poorly-rated high school, wouldn't you be adjusting for adversity in exact proportion to actual hardship instead of assigning an arbitrary value to skin color? This entire process can be completely race-agnostic and still compensate for systemic racism at the same time.

I'm also curious what stops me from putting whatever I want in the race field on my application. I don't recall my university running a blood test to check if I was fibbing.

<comment><USER11> > I'm also curious what stops me from putting whatever i want int he race > field on my application.

Nothing, as we found out with Elizabeth Warren being "Indian". As you said, no test. Of course, I'm not sure I'd be comfortable handing my DNA over to every school to which I wanted to apply.

<comment><USER19> Back in the day, a woman on my dorm floor would check a different box every quarter during registration, just to see what would happen. The main thing that would happen is that one quarter she would get a lot of mailings from the black student organization, and the next quarter a lot of mailings from the Asian student association.

Of course, on an admissions form, a whole different machine kicks in. I think falsifying your admissions form would have consequences.

<comment><USER16> There are no consequences. What objective criteria could they possibly use to prove you falsified your application?

</comment></comment></comment></comment><comment><USER20> I think there are two dimensions (likely correlated)

1\. Race.

2\. Socioeconomic status.

Callibrating a score on race would be a problem, because you are explicitly reinforcing virtues that lead to suboptimal performance. However calibrating a score on (2) makes sense because kids with unequal resources should have access to the same opportunities, and it is reasonable to me that a top performer in a lower socioeconomic neighborhood would do better than someone who performed similarly were they raised in a more affluent neighborhood.

Where you run into problems is that lower socioeconomic neighborhoods are associated strongly with black/Latino populations and so it is indirectly biasing for select racial groups. I don't think there's a perfect solution, or one that is 100% equitable to all parties but I think we can do better than we are now, in fact I think it is crucial.

</comment><comment><USER21> Where are all the comments about racial diversity coming from? There's a single article mentioning that _using_ race is a contentious issue - not that this score uses race.

Further down in the article, it lists what the scores are based on:

> Those factors are college attendance, household structure, median family > income, housing stability, education levels and crime.

Which all sound like tests of the living conditions and environment that the students had to learn within - and if those strongly correlate with race, then maybe there's other issues that need to be addressed

</comment><comment><USER22> If we allow entry of students from underperforming schools into colleges with historically high academic requirements, one of two things will happen, neither of which are particularly good:

1\. These students will struggle and be much more likely to fail out, depending on how much of a boost they're given by the so called adversity score

2\. Standards will be lowered and as a result graduates will be less competent and schools lose prestige and/or society suffers in the long run from workers who previously would not have qualified for their degrees.

You cannot undo the damage of 12 years of poor schooling by simply throwing a student into a more rigorous curriculum, no matter how admirable the intent.

And arguably point 2 has already been happening for some time anyway given the swelling of college ranks over the last few years and the corresponding devaluation of the college degree. More people may hold degrees, but the normal distribution of intelligence has not changed, which must indicate that the rigor of the average degree has dropped substantially in the last few decades given the proportion of the population that is now educated.

<comment><USER23> those aren't the only two possible outcomes. I went to a pretty good state school with pretty generous admissions criteria, but very rigorous coursework. most of the required courses for stem majors started at the 200 level. if you got a "good" highschool education (ie passed calculus) you were expected to start with CS-201. if not, there were lots of gentle 100-level courses to ease you into programming or teach high school math material that you either didn't take or failed. note when I say gentle, I mean the material itself is not difficult. you would be assigned a large _quantity_ of work in every class at every level.

plenty of kids failed out along the way, but it was more because they didn't put in enough time than being hopelessly underprepared for the rigor.

you do kinda have to give up on the "undergrad takes exactly four years" thing if you want this to work though. some people will need to do more than 120 credits to fulfill the requirements of the degree. not everyone will be able to take a full-time course load every semester or even any semester. that doesn't make the end result worse when they finally graduate (though it may be more expensive).

</comment></comment><comment><USER24> Credit score is a number highly correlated with your social-economic status and race/ethnicity.

</comment><comment><USER14> We can't simultaneously be mad at programs like this and also say school doesn't matter.

Personally I think providing separate scores for adversity is a good thing. Ultimately getting into ones state school isn't that difficult, and even if it was your SAT is a small part of that.

</comment><comment><USER25> I would think that by college age, the raw input to the school had better be pretty well-refined and at roughly the same level.

To let people into big-league colleges with any 'fudge factor' isn't doing them any favors. It's just an easy path to failure. (Also: even if the schools provide hand-holding and personal tutoring, the school had better make sure what is provided sticks. Very well. Else they may graduate some 'charity cases', but it won't be doing the graduates any favors when they hit the workforce.)

<comment><USER26> A large portion of the value of a degree is simply in signaling - it tells potential employers rough information about candidate IQ, conscientiousness, and social capital. Graduating unqualified students is absolutely a favor, and a huge one to boot. The primary people harmed are the _other students_ , who suffer the ill effects of inflation in the value of their degree.

</comment></comment><comment><USER27> I get the idea that with so many admissions we need some way to "standardize" admissions, but the way we do it is pretty archaic, and definitely doesn't work for a host of different students. Has anyone seen any interesting concepts or ideas around dropping standardized testing in general for something else?

<comment><USER28> In many countries it goes the other direction: the standardized exams are much harder than the SAT (so you don't have a bunch of people clustering at the top with perfect scores) and are the _only_ metric used for university admissions.

<comment><USER29> That seems pretty bad too, no? Basing everything off just standardized exams? Are these exams usually/mostly multiple choice?

<comment><USER28> Why does it seem bad to you? To me it seems like the fairest possible method, as long as the tests are reasonably accurate measures of competence.

The US method on the other hand where colleges try to assess "leadership" and other qualities of that nature makes no sense to me. What in the world do "leadership", extracurricular activities, sports ability, etc. have to do with academic ability?

I guess it's because in the US, college is seen as a way of sorting people into social classes, just as much as a place to actually learn something (or more). But a sane system shouldn't care about having "well-rounded" students — it should optimize for having the best 100 students in physics in the country studying together from the best physicists, the best in history from the best historians, etc.

> Are these exams usually/mostly multiple choice?

No.</comment></comment></comment></comment>