Ask HN: How to handle 50GB of transaction data each day? (200GB during peak) - <AUTHOR> Recently I switched jobs and joined some old employers and friends on a new endeavour, an advertisement platform. So far things have been very good, so good in fact that we're quickly growing out of our current infrastructure. Last christmas everything went crazy and the, then only developer, rewrote a bunch of things to make it scale better.
Currently we write away every single transaction to txt files and once every X minutes gather those and start inserting them into 2 Mysql shards. After that we run a bunch of group by queries to generate cross table reports. Currently our CMS is very limited by this, you can only view data relations in the predefined tables and we would like to have (a lot) more flexibility here.
Perhaps the biggest issue of all is that both I and the other developer do not have experience with handeling this much data. We both have worked on systems which handle a fair bit of data, but not nearly as much as this. We've been looking at MapReduce for MongoDB but already run into problems when we try to insert a weeks worth of data, which is about 350gb, because we only get so many IOPS at AWS. If we want more IOPS, it gets really expensive really fast. We're also not entirely sure yet if MongoDB and MapReduce is the way to go.
What I'm looking for is a few pointers to help us get headed in the right direction. Which databases should we be looking at? Should we be looking in an entirely different direction? Are there some (recent?) blog posts about this which can help us out? <comment><USER3> Not an answer to your question, but just a quick note- this is the first post in a long while on HN where I appreciate both the problem you are looking to solve and the honesty/sincerity you have in saying that you are not perfectly qualified to solve it but you know those here can help. From all of us on the community watching and lurking, thanks for your candor so we can all learn from this thread!

<comment><USER4> I think shopping around and recommending/evaluating these tools is pretty fun that's why these threads are always a blast.

</comment></comment><comment><USER5> First of all, 50GB per day is easy. Now, maybe contrary to what they say below, do the following:

* Don't use queues. Use logs, such as Apache Kafka for example. It is unlikely to lose any data, and in case of some failure, the log with transactions is still there for some time. Also Kafka guarantees order of messages, which might be important (or not).

* Understand what is the nature of data and what are the queries that are made later. This is crucial for properly modeling the storage system.

* Be careful with the noSQL cool-aid. If mature databases, such as postgreSQL can't handle the load, choose some NoSQL, but be careful. I would suggest HBase, but your mileage may vary.

* NoSQL DBs typically limits queries that you might issue, so the modelling part is very important.

* Don't index data that you don't need to query later.

* If your schema is relational, consider de-normalization steps. Sometimes it is better to replicate some data, than to keep relational schema and make huge joins across tables.

* Don't use MongoDB

I hope it helps!

<comment><USER6> Here here ; +1.

Kafka = good. Postgres can do a TB; if you plan bigger than that consider Cassandra.

Also consider protobuf instead of json.gz.

<comment><USER7> Or... Arrow?! :O

JK it's too early :(

</comment></comment></comment><comment><USER8> First of all, ingest your data as .json.gz -- line delimited json that's gzipped -- chunk this by time range, perhaps hourly, on each box. Periodically upload these files to the cloud -- S3 or Google CloudStorage, or both for a backup. You can run this per-node, so it scales perfectly horizontally. And .json.gz is easy to work with -- looking for a particular event in the last hour? gunzip -c *.json.gz | grep '<id>' ...

Most of the big data tools out there will work with data in this format -- BigQuery, Redshift, EMR. EMR can do batch processing against this data directly from s3 -- but may not be suitable for anything other than batch processing. BigQuery and/or Redshift are more targeted towards analytics workloads, but you could use them to saw the data into another system that you use for OLAP -- MySQL or Postgres probably.

BigQuery has a nice interface and it's a better hosted service than Redshift IMO. If you like that product, you can do streaming inserts in parallel to your gcs/s3 uploading process for more real-time access to the data. The web interface is not bad for casual exploration of terabytes of raw data. And the price isn't terrible either.

I've done some consulting in this space -- feel free to reach out if you'd like some free advice.

<comment><USER9> Sounds like you know your stuff!

I couldn't find any email on your profile, do you mind sharing it?

<comment><USER10> Bottom of this page: [http://vazor.com/unicode/](http://vazor.com/unicode/)

</comment></comment></comment><comment><USER11> We need to handle data in a similar level as you mention and also use plain text files as only reliable medium to store data. A recent blog: [http://nunobrito1981.blogspot.de/2016/02/how-big-was- triplec...](http://nunobrito1981.blogspot.de/2016/02/how-big-was-triplecheck- tech-in-2015.html)

My advice is to step away from AWS (because of price as you noted). Bare metal servers are the best startup friend for large data in regards to performance and storage. This way you avoid virtualized CPU or distributed file systems that are more of a bottleneck than advantage.

Look for GorillaServers at [https://www.gorillaservers.com/](https://www.gorillaservers.com/)

You get 40Tb storage with 8~16 cores per server, along with 30Tb of bandwidth included for roughly 200 USD/month.

This should remove the IOPS limitation and provide enough working space to transform the data. Hope this helps.

<comment><USER12> Are you using GorillaServers solely for data harvesting/crunching or for all of your infrastructure?

</comment></comment><comment><USER13> Here are a few suggestions based on 6+ years in adtech (which have just came to a close, never again thank you):

* Use a Queue. RabbitMQ is quite good. Instead of writing to files, generate data/tasks on the queue and have them consumed by more than one client. The clients should handle inserting the data to the database. You can control the pipe by the number of clients you have consuming tasks, and/or by rate limiting them. Break those queue consuming clients to small pieces. Its ok to queue item B on the queue while processing item A.

* If you data is more fluid and changing all the time, and/or if it comes in JSON serializable format, consider switching to postgresql ^9.4, and use the JSONB columns to store this data. You can index/query those columns and performance wise its on par (or surpasses) MongoDB.

* Avoid AWS at this stage. like commented by someone here - bare metal is a better friend to you. You'll also know exactly how much you're paying each month. no surprises. I can't recommend Softlayer enough.

* Don't over complicate things. If you can think of a simple solution to something - its preferable than the complicated solution you might have had before.

* If you're going the queue route suggested above, you can pre-process the data while you get it in. If its going to be placed into buckets, do it then, if its normalised - do it then. The tasks on the queue should be atomic and idempotent. You can use something like memcached if you need your clients to communicate between eachother (like checking if a queue item is not already processed by another consumer and thus is locked).

<comment><USER14> > * If you data is more fluid and changing all the time, and/or if it comes in > JSON serializable format, consider switching to postgresql ^9.4, and use the > JSONB columns to store this data. You can index/query those columns and > performance wise its on par (or surpasses) MongoDB.

Would it be worthwhile to switch from MongoDB to Postgres (when MySQL is being used alongside Mongo)? I'd have to do lots of testing with it to know for sure though, I guess.

I use Mongo for timeseries-like data, and have >1 billion documents in Mongo currently. Querying super old data is pretty slow (and not needed generally), but querying recent data (recent being the last month) is "fast enough".

<comment><USER13> If your setup makes sense for you, its fine - stay with it. focus on real problem, not ones people on this thread think you have :)

I'm not one of those mongo bashers. I've used mongo before for the same purpose you do - time series statistical data along side a postgres db for everything else. And it worked fine for me. However, at that time postgresql was pre 9.4. Today I would have kept it all in postgres and simply archived older data if needed. And to be fair, its not even a "problem" until you need to filter by one data source and sort by the other. We had to be creative when we got to those problems.

As for my choice of Postgres over mySql - that is for 2 reasons - first being that postgres is not owned by Oracle, and second that postgres is more mature, feature rich and reliable than mysql (i.e., better).

</comment></comment></comment><comment><USER15> Disclaimer: I work at Google.

Have you looked at Google at all? Cloud Bigtable runs the whole of Google's Advertising Business and could scale per your requirements.

[https://cloud.google.com/bigtable/docs/](https://cloud.google.com/bigtable/docs/)

<comment><USER16> This is super easy to get started with, plus really cheap storage wise. Query wise, it can get "expensive" (tbh, still cheap) if you do it badly. It's really fast.

</comment></comment><comment><USER17> I'm not sure I understand precisely what kind of data you are processing and in what way, but it sounds like a PostgreSQL job on a beefy server (lots of RAM) with SSD storage. Postgres is very good at complex queries and concurrent write loads and if you need to scale quickly beyond single server setups, you can probably move your stuff to Amazon Redshift with little effort. Wouldn't recommend "big data" i.e distributed setups at that size yet unless your queries are extremely parallel workloads and you can pay the cost.

In my previous job we processed 100s of millions of row updates daily on a table with much contention and ~200G size and used a single PostgreSQL server with (now somewhat obsoleted by modern PCIe SSDs) TMS RamSAN storage, i.e. Fibre-Channel based Flash. We had some performance bottlenecks due to many indexes, triggers etc. but overall, live query performance was very good.

</comment><comment><USER18> Doing real time query for report generation for data growing by 50gb per day is a hard problem.

Realistically, this is what I would do (I work on something very similar but not really in adtech space):

1\. Load data in text form (assuming it sits in S3) inside hadoop (EMR/Spark)

2\. Generate reports you need based on your data and cache them in mysql RDS.

3\. Serve the pre-generated reports to your user. You can get creative here and generate bucketed reports where user will fill its more "interactive". This approach will take you a long way and when you have time/money/people, maybe you can try getting fancier and better.

Getting fancy: If you truly want near-real time querying capabilities I would looks at apache kylin or linkedin pinot. But I would stay away from those for now.

Bigtable: As someone pointed out, bigtable is good solution (although I haven't used it) but since you are on AWS ecosystem, I would stick there.

</comment><comment><USER19> We're having a good time with Cassandra on AWS ingesting more than 200 GiB per day uncompressed. I don't know how you're running your IOPS numbers, but consider allocating large GP2 EBS volumes rather than PIOPS--you'll get a high baseline for not that much money. The provisos you'll see about knowing how you expect to read before you start writing are absolutely true. :)

(Hope that might be helpful! A bunch of us hang out on IRC at #cassandra if you're curious.)

</comment><comment><USER20> I once worked on similar project. Each day, the amount of the data coming in is about 5TB.

If your data are event data, e.g. User activity, clicks, etc, these are non- volatile data which should preserve as-is and you want to enrich them later on for analysis.

You can store these flat files in S3 and use EMR (Hive, Spark) to process them and store it in Redshift. If your files are character delimited files, you can easily create a table definition with Hive/Spark and query it as if it is a RDBMS. You can process your files in EMR using spot instances and it can be as cheap as less than a dollar per hour.

</comment><comment><USER21> You probably might want to read this (for free): [http://book.mixu.net/distsys/single- page.html](http://book.mixu.net/distsys/single-page.html)

And pay a little to read this book: [http://www.amazon.com/Designing-Data- Intensive-Applications-...](http://www.amazon.com/Designing-Data-Intensive- Applications-Reliable-Maintainable/dp/1449373321)

And this one: [http://www.amazon.com/Big-Data-Principles-practices- scalable...](http://www.amazon.com/Big-Data-Principles-practices- scalable/dp/1617290343)

Nathan Marz brought Apache Storm to the world, and Martin Kleppmann is pretty well known for his work on Kafka.

Both are very good books on building scalable data processing systems.

</comment><comment><USER22> 1\. Stream your data into Kafka using flat JSON objects. 2\. Consume your kafka Feeds using a Camus Map Reduce job (a library from linked in that will output hdfs directories with the data). 3\. Transform the hdfs directories into usable folders for each vertical your interested in, think of each output directory as an individual table or database. 4\. Use HIVE to create an "external table" that references the transformed directories. Ideally your transformation job will create merge-able hourly partition directories. Importantly you will want to use the JSON SERDE for your hive configuration. 5\. Generate your reports using hive queries.

This architecture will get you to massive, massive scale and is pretty resilient to spikes in traffic because of the Kafka buffer. I would avoid Mongo / mysql like the plague in this case, a lot of designs focus on the real time aspect for a lot of data like this, but if you take a hard look at what you really need, its batch map reduce on a massive scale and a dependable schedule with linear growth metrics. With an architecture like this deployed to AWS EMR (or even kinesis / s3 / EMR) you could grow for years. Forget about the trendy systems, and go for the dependable tool chains for big data.

</comment><comment><USER23> We faced a very similar problem when we started Kentik two years ago, except in our case the "transactions" are network traffic telemetry that we collect from our customers' physical network infrastructure, and providing super-fast ad hoc queries over that data is our core service offering.

We looked at just about every open source and commercial platform that we might use as a backend, and decided that none were appropriate, for scale, maturity, or fairness / scheduling. So we ended up building, from scratch, something that looks a bit like Google's Dremel / BigQuery, but runs on our own bare metal infrastructure. And then we put postgres on top of that using Foreign Data Wrappers so we could write standard SQL queries against it.

Some blog posts about the nuts and bolts you might find interesting:

[https://www.kentik.com/postgresql-foreign-data- wrappers/](https://www.kentik.com/postgresql-foreign-data-wrappers/)

[https://www.kentik.com/metrics-for- microservices/](https://www.kentik.com/metrics-for-microservices/)

If we were starting today, we might consider Apache Drill, although I haven't looked at the maturity and stability of that project recently.

</comment><comment><USER24> Read "Designing data intensive applications" ([http://dataintensive.net/](http://dataintensive.net/)), which is an excellent introduction to various techniques for solving data problems. It won't specifically tell you what to do, but will quickly acclimate you to available approaches and how to think about their trade offs.

<comment><USER25> Although an awesome book, as I've bought it, maybe it would be useful to inform any interested buyers that not all chapters are ready yet. i.e. it hasn't been released officially and it doesn't have a final release date.

Right now the first 9(out of 12) chapters are finished and they're really insightful.

</comment></comment><comment><USER26> Consider using CitusData to scale out Postgres horizontally. You can shard by time and basically get linear speedup based on # of shards. Its extremely fast and will be open source in early Q2 I think. You then can put your Postgres instances on boxes with SSD instead of IOPS. Writes also scale mostly linearly.

</comment><comment><USER27> 50GiB/day is less than a megabyte per second. Surely you wouldn't be bandwidth-limited on a real device, even consumer SSDs are in the 100-600 MiB/s range IIRC. Can you do anything to increase your bytes per IOP in your current environment if you're IOP-limited?

</comment><comment><USER28> If your data is growing at this rate (and you plan to keep this data around), you'd want a _distributed database_ that can scale to terabytes. But it might be overkill if you dont care for data consistency (i.e., you dont need to read it "right away" after you do a write):

If you just want reports (and are okay getting them in the matter of minutes), then you can continue storing them in flat files and using apache HIVE/PIG- equivalent software (or whatever equivalent is hot right now, im out of date on this class of software).

If you want a really good out-of-box solution for storage + data processing, google cloud products might be a really good bet.

</comment><comment><USER29> Lots of good suggestions here. I won't say anything new but just wanted to stress on the data ingestion part.

DO NOT write to txt files and read them again. This is unnecessary disk IO and you will run into a lot of problems later on. Instead, have an agent which writes into Kafka (like everyone mentioned), preferably using protobuff.

Then have an aggregator which does the data extraction and analysis and puts them in some sort of storage. You can browse this thread to look for and decide what sort of storage is suitable for you.

</comment><comment><USER30> Its difficult to give answer without understanding of data processing you want.

If you need to generate rich multi-dimension reports I recommend you create ETL pipeline into star-like sharded database (ala OLAP).

Dimensions normalization sometime dramatically reduce data volume, most of dimensions even can fit into RAM.

Actually 200Gb per day not so much in terms of throughput, you can manage it pretty well on PostgreSQL cluster (with help of pg_proxy). I think mySQL will also work OK.

Dedicated hardware will be cheaper then AWS RDS.

</comment><comment><USER31> Hire a DBA and stick with relational DB. We had (previous group, still running) a 40-50TB Oracle db that collected much more data daily.

I would think you shouldn't be writing to a txt file, then doing imports into a DB. That's hitting disk multiple times and unnecessary. Relational DB's should handle 50GB of inserts in real time. IF you stream the data real time into the DB then you won't need to worry about your batch of 50GB taking longer time than you have to update.

Other things I have observed from very large relational dbs: have one database for data capture and keep everything normalized. Your problem down the line is going to be purging data so if you can, partition tables by date so you can just drop partitions later. Setup a separate database for reporting purposes and import necessary data - possibly consolidated metrics, etc. into this per your business needs. Make sure your DBA understands the business needs of each, tunes the databases for data capture and reporting and is involved with specifying table layouts, etc.

My rationale for sticking with RDBMS - you can solve this problem with that tech, the candidate pool is larger and the depth of knowledge, documentation and best practices on the technology is much deeper than NoSQL alternatives.

</comment><comment><USER32> disclaimer: I work for Citus Data

The bottleneck is usually not I/O, but computing aggregates over data that continuously gets updated. This is quite CPU intensive even for smaller data sizes.

You might want to consider PostgreSQL, with Citus to shard tables and parallelise queries across many PostgreSQL servers. There's another big advertising platform that I helped move from MySQL to PostgreSQL+Citus recently and they're pretty happy with it. They ingest several TB of data per day and a dashboard runs group-by queries, with 99.5% of queries taking under 1 second. The data are also rolled up into daily aggregates inside the database.

There are inherent limitations to any distributed database. That's why there are so many. In Citus, not every SQL query works on distributed tables, but since every server is PostgreSQL 9.5, you do have a lot of possibilities.

Looking at your username, are you based in the Netherlands by any chance? :)

Some pointers:

\- How CloudFlare uses Citus: [https://blog.cloudflare.com/scaling-out- postgresql-for-cloud...](https://blog.cloudflare.com/scaling-out-postgresql- for-cloudflare-analytics-using-citusdb/)

\- Overview of Citus: [https://citus- conferences.s3.amazonaws.com/pgconf.ru-2016/Ci...](https://citus- conferences.s3.amazonaws.com/pgconf.ru-2016/CitusDB.pdf)

\- Documentation: [https://www.citusdata.com/documentation/citusdb- documentatio...](https://www.citusdata.com/documentation/citusdb- documentation/)

</comment><comment><USER33> Here's what I've done for ~200GB/day. Let's pretend you have server logs with urls that tell you referrer and whether or not the visit action was an impression or a conversion and you want stats by "date", "referrer domain", "action":

* Logs are written to S3 (either ELB does this automatically, or you put them there)

* S3 can put a message into an SQS queue when a log file is added

* A "worker" (written in language of your choice running on EC2 or Lambda) pops the message off the queue, downloads the log, and "reduces" it into grouped counts. In this case a large log file would be "reduced" to lines where each line is [date, referrer domain, action, count] (e.g. [['2016-02-24', 'news.ycombinator.com', 'impression', 500], ['2016-02-24', 'news.ycombinator.com', 'conversion', 20], ...]

* The reduction can either be persisted in a db that can handle further analysis or you reduce further first.

</comment><comment><USER7> Check out LinkedIn's posts about log processing [0] and Apache Kafka. Handling data as streams of events lets you avoid spikey query-based processing, and helps you scale out horizontally. Partitioning lets you do joins, and you can still add databases as "materialized views" for query-ability. Add Secor to automatically write logs to S3 so you can feel secure in the face of data loss, and use replication of at least 3 in your Kafka topics. Also, start with instrumentation from DataDog or NewRelic from the start - it will show you the performance bottlenecks.

0: [https://engineering.linkedin.com/distributed-systems/log- wha...](https://engineering.linkedin.com/distributed-systems/log-what-every- software-engineer-should-know-about-real-time-datas-unifying)

</comment><comment><USER34> If youre on a tight budget and IO is your main bottleneck it may be easier to purchase a number of decent spec desktop PC's with multiple SSD's in them. SSD's have really come down in price while performance and capacity have improved greatly. Same goes for RAM. (Assumption here is that time is less of a concern than cost at the moment and youre not averse to doing some devops work. Also assuming that the processing youre talking about is some sort of batch processing and not realtime) This way you can try a number of different strategies without blowing the bank on AWS instances (and worst case you have a spare workstation)

</comment><comment><USER35> I would consider Unicage for your demands. [https://www.youtube.com/watch?v=h_C5GBblkH8](https://www.youtube.com/watch?v=h_C5GBblkH8) [https://www.bsdcan.org/2013/schedule/attachments/244_Unicage...](https://www.bsdcan.org/2013/schedule/attachments/244_UnicageDevelopment.pdf)

In a shell (modified for speed and ease of use) get, insert, update data in a simple way, without all the fat from other mainstream (Java) solutions.

</comment><comment><USER36> We love those projects at my company (Inovia Team). Your load is not that big. You won't make any big mistake stack-wise, you just have to pick something you already have operated before in production at a smaller scale. Mysql, Postgres, Mongodb, Redis will be totally fine. We have a training on how to insert 1M lines a second with off-the-shelf free open source tools (SQL and NoSQL). Ping us if you are interested by getting the slidedeck.

Tip: focus on how to backup and restore first, the rest will be easy!

</comment><comment><USER37> Does the database grow 50GB or is that the size of the text files?

<comment><USER37> As a DBA, I'm duty-bound to say this, though I'm sure it will be drowned out by calls to 'Denormalize All the Things!'. DON'T denormalize for performance. It is a siren's call. Don't take my word for it, see anything written by Fabian Pascal, Hugh Darwen, Chris Date.

If I was faced with a dataset of 50GB in a text file, being loaded into a database every day, I would normalize as much as possible to ensure that I'm not adding 50GB to a database everyday. What's the plan, to have a 20TB database in one year? No, instead I would focus on micro-optimizations at the database level.

I would inspect the datatypes you're storing your data in. If it's a value with a limited domain (an enum in programming parlance), make sure you determine the theoretical maximum amount of values it could be, and use the equivalent numeric value to serve as a surrogate key in a separate table. If I know a value is never going to exceed a few thousand distinct values, I could use a smallint, which only uses 2 bytes, verses recording a char/varchar which is going to probably burn at least 4-5 bytes every row, likely much more.

I believe ordering of columns matters in Postgres, so consult the documentation/Stackoverflow where swapping around column ordering can sometimes save several bytes per row.

You'll notice quickly that you may have repeating 'groups' of similar values across your surrogate keys. For instance, if you had a StateID and a CountryID in the same table, chances are good the combinations of StateID and CountryID are finite, and could themselves go into their own table, which has its own surrogate key, so instead of storing two surrogates on every row, you could use only one. (Incidentally, chances are even better that CountryID should just be a reference on the State table itself, but this is just to illustrate the point).

Now imagine you have sets of attributes that describe whole subsets of the information in your tables that are mutually exclusive. You could make two different tables with identical data structures. Then placing a row in one 'means' those other implicit value combinations. Thus you record information without actually storing it. The price here is on retrieval, you have to union the tables if you want to consider the whole set. That said, you may shave a large amount of storage. Views could be created to make this transparent to end users.

Don't use an auto-incrementing index on the table, instead store a value that is iterated separately, and 'take' ranges of values at a time that get cached by routines that mean to load the data.

Through this style of process, one can often turn a 1TB table into something a fraction of that size, at the cost of much harder to write queries. That said, there are other benefits, suddenly you can load values into the supporting tables first, then cache those values and load the main dataset without doing lookups into the additional related tables. The 'growth' of a database will slow considerably. Then you just preprocess the files, and 'COPY' them in, which should allow you load them on the order of minutes, not hours.

</comment></comment><comment><USER38> If the big issue is querying the data, consider redshift (expensive) or a self-hosted column-store database. Data will need to be loaded in a batch for this. Column stores will reduce iops through compression, and selective data loading, and because they don't have persistent indexes.

To save IOPS on the early part of the process, consider using fast encryption (lz4 or snappy) to compress the records before writing to the file system. This might cut your IOPS in half.

</comment><comment><USER39> Use BigQuery, here is a nice presentation how to get going and some uses cases that get's you very familiar in the territory. I offer consulation also so you can reach out. [http://www.slideshare.net/martonkodok/complex-realtime- event...](http://www.slideshare.net/martonkodok/complex-realtime-event- analytics-using-bigquery-crunch-warmup)

</comment><comment><USER40> I don't know what I'm talking about or what you need, but I hear kdb is popular in the financial industry because supposedly it can handle large amounts of real-time financial information. [http://kx.com](http://kx.com)

</comment><comment><USER41> Compare pricing on RDS, if doing it yourself is hurting.

AWS also has Kinesis, which is deliberately intended to be a sort of event drain. Under the hood it uses S3 and they publish an API and an agent that handles all the retry / checkpoint logic for you.

<comment><USER42> Absolutely this. If you're in the AWS ecosystem then move away from the bare- ish metal and onto the tools they provide for doing this stuff. MongoDB in particular really doesn't seem to be relevant to the problem at all. Throw all the data into Kinesis, and then push the bits you need into Redshift the other end.

If you want to keep a lot of data for ad-hoc queries but only use a subset for regular queries then consider setting up separate dense compute and dense storage Redshift clusters as appropriate. As far as pricing goes there's a hell of a discount for going reserved once you've got things set up the way you want them.

Once you've got that running and the immediate fires are out you may want to look at Spark to pre-compute some of the reports (you can still push the computed data into Redshift tables to save changing the reporting too much), although I don't think spark as a service isn't really there in AWS right now, and running up an EMR cluster is seriously going to cost you.

Bigtable provides similar capabilities to Redshift in the google ecosystem, and looks okay although I wasn't blown away by it.

</comment></comment><comment><USER43> I'd start by asking if you are solving the right problem.

Does the business really need exactly this? What is their actual goal? Are they aware of the effort and resources required to get this report?

</comment><comment><USER44> I would highly recommend looking into Elasticsearch. Clustering and scalability are its strong points and can help you with your quest.

</comment><comment><USER45> Look at your current solution and check the run plan of your sql. If your data is indexed correctly it shouldn't be too too bad to execute queries. 1M records is about 20 ops to search for a record by key.

If it's modelled in SQL, it's probably relational and normalized so you'll be joining together tables. This balloons the complexity of querying the data pretty fast. Denormalizing data simplifies the problem so see if you can get it into a K/V instead of or relational database. Not saying relational isn't a fine solution - even if you keep it in mysql, denormalizing will benefit the complexity of querying it.

Once you determine if you can denormalize, you can look at sharding the data so instead of having the data in one place, you have it in many places and the key of the record determines where to store and retrieve the data. Now you have the ability to scale your data horizontally across instances to divide the problem's complexity by n where n is the number of nodes.

Unfortunately the network is not reliable so you suddenly have to worry about CAP theorem and what happens when nodes become unavailable so you'll start looking at replication and consistency across nodes and need to figure out with your problem domain what you can tolerate. Eg bank accounts have different consistency requirements than social media data where a stale read isn't a big deal.

Aphyr's Call Me Maybe series has reviews of many datastores in pathological conditions so read about your choice there before you go all in (assuming you do want to look at different stores.) Dynamo style DB's like riak are what I think of immediately but read around - this guy is a wizard. [https://aphyr.com/tags/Jepsen](https://aphyr.com/tags/Jepsen)

AWS has a notorious network so really think about those failure scenarios. Yes it's hard and the network isn't reliable. Dynamo DBs are cool though and fit the big problems you're looking at if you want to load and query it.

If you want to work with the data, the Apache Spark is worth looking at. You mention mapreduce for instance. Spark is quick.

It's sort of hard because there isn't a lot of information about the problem domain so I can only shoot in the dark. If you have strong consistency needs or need to worry more about concurrent state across data that's a different problem than processing one record at a time without a need for consistent view of the data as a whole. The latter you can just process the data via workers.

But think Sharding to divide the problem across nodes, Denormalization eg via Key/Value lookup for simple runtime complexity. But start where you are - <USER2> at your database and make sure it's very well tuned for the queries you're making.

Do you even need to load it into a db? You could distribute the load across clusters of workers if you have some source that you can stream the data from. Then you don't have to load and then query the data. Depends heavily on your domain problem. Good luck. I can email you to discuss if you want - I just don't want to post my email here. Data isn't so much where I hand out as much as processing lots of things concurrently in distributed systems is so others may have better ideas who have gone through similar items.

There are some cool papers like the Amazon Dynamo paper and I read the Google Spanner paper the other day (more globally oriented and around locking and consistency items). You can see how some of the big companies are formalizing thinking by reading some of the papers in that space. Then there are implementations you can actually use but you need to understand them a bit first I think.

[http://www.allthingsdistributed.com/files/amazon-dynamo- sosp...](http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf) [http://static.googleusercontent.com/media/research.google.co...](http://static.googleusercontent.com/media/research.google.com/en//archive/bigtable- osdi06.pdf)

</comment><comment><USER4> Note: This is based on solutions I have been researching for a current project and I haven't used these in production.

Short answer: I think you're looking in the wrong direction, this problem isn't solved by a database but a full data processing system like Hadoop, Spark, Flink (my pick), or Google Cloud's dataflow. I don't know what kind of stack you guys are using (imo the solution to this problem is best made leveraging java) but I would say that you could benefit a lot from either using the hadoop ecosystem or using google cloud's ecosystem. Since you say that you are not experienced with that volume of data, I recommend you go with google cloud's ecosystem specifically look at google dataflow which supports autoscaling.

Long answer: To answer your question more directly, you have a bunch of data arriving that needs to be processed and stored every X minutes and needs to be available to be interactively analyzed or processed later in a report. This is a common task and is exactly why the hadoop ecosystem is so big right now.

The 'easy' way to solve this problem is by using google dataflow which is a stream processing abstraction over the google cloud that will let you set your X minute window (or more complex windowing) and automatically scale your compute servers (and pay only for what you use, not what you reserve). For interactive queries they offer google bigquery, a robust SQL based column database that lets you query your data in seconds and only charges you based on the columns you queried (if your data set is 1TB but the columns used in your query are only some short strings they might only charge you for querying 5GB). As a replacement for your mysql problems they also offer managed mysql instances and their own google bigtable which has many other useful features. Did I mention these services are integrated into an interactive ipython notebook style interface called Datalab and fully integrated with your dataflow code?

This is all might get a little expensive though (in terms of your cloud bill), the other solution is to do some harder work involving the hadoop ecosystem. The problem of processing data every X minutes is called windowing in stream processing. Your problems are solved by using Apache Flink, a relatively easy and fast stream processing system that makes it easy to set up clusters as you scale your data processing. Flink will help you with your report generation and make it easy to handle processing this streaming data in a fast, robust, and fault tolerant (that's a lot of buzz words) fashion.

Please take a look at the flink programming guide or the data-artisans training sessions on this topic. Note that the problem of doing SQL queries using flink is not solved (yet) this feature is planned to be released this year. However, flink will solve all your data processing problems in terms of the cross table reports and preprocessing for storage in a relational database or distributed filesystem.

For storing this data and making it available you need to use something fast but just as robust as mysql, the 'correct' solution at this time if you are not using all the columns of your table is using a columnar solution. From googles cloud you have bigquery, from the open source ecosystem you have drill, kudu, parquet, impala and many many more. You can also try using postgres or rethinkdb for a full relational solution or HDFS/QFS + ignite + flink from the hadoop ecosystem.

For the problem of interactively working with your data, try using Apache Zeppelin (free, scala required I think) or Databricks (paid but with lots of features, spark only i think). Or take the results of your query from flink or similar and interactively analyze those using jupyter/ipython(the solution I use).

The short answer is, dust off your old java textbooks. If you don't have a java dev on your team and aren't planning on hiring one, the google dataflow solution is way easier and cheaper in terms of engineering. If you help I do need an internship ;)

If you want to look at all the possible solutions from the hadoop ecosystem look at: [https://hadoopecosystemtable.github.io/](https://hadoopecosystemtable.github.io/)

For google cloud ecosystem it's all there on their website.

Happy coding!

Oops, it seems I left out ingestion, I would use kafka or spring reactor.

P.S The flink mailing list is very friendly, try asking this question there.</comment>


The Heroku & IndexTank "Build A Search-Based Webapp" Holiday Challenge - <AUTHOR> http://www.rubyinside.com/the-heroku-indextank-build-a-search-based-webapp-holiday-challenge-4125.html

<comment><USER2> It is somewhat weird to be doing this on a post pointing at your own site, but still, this is a double dupe: <http://news.ycombinator.com/item?id=2025810> <http://news.ycombinator.com/item?id=2013777> .. yes, being a responsible HN user wins over more pageviews for me this time (!)

Nonetheless, good luck in the contest if you choose to enter, folks.

</comment><comment><USER3> By the way, here's another app that I hacked to test the plumbing for the contest, a (very limited) Instagr.am search:

<http://igsearch.heroku.com>

The default search is photo caption, but you can search by fields such as username or filter:

[http://igsearch.heroku.com/photos?utf8=%E2%9C%93&query=f...](http://igsearch.heroku.com/photos?utf8=%E2%9C%93&query=filter:earlybird)

</comment><comment><USER3> While we are here, let me know if you have any questions about the contest. Happy to help!

<comment><USER4> Will you allow the apps to stay up for a while after the competition? If I have an app that indexes a million docs, but has low traffic, I really can't afford to be paying 300/mo immediately after the competition ends.

Maybe you can offer a deal where if someone puts a "built with indextank" sticker/link on the app you'll keep it up for a little while? Could be good marketing maybe?

<comment><USER3> It's a good question. The app would be taking up resources so it has some not- insignificant costs to us regardless of the traffic. Having said that, if an app has value and we like it I'be happy to discuss and try to work it out. Awesome apps built with IndexTank are good for us!</comment></comment></comment>


Anatomy of a Hadoop Project Failure - <AUTHOR> https://www.datanami.com/2017/03/17/anatomy-hadoop-project-failure/ <comment><USER2> > The main reasons for choosing Hadoop were scale, a desire to unify data, and > technical curiosity on the part of Blackboard engineers.

> Blackboard was moving JSON data exhaust from a relational database into > Hadoop and then back out into a Postgres database, where it could be analyze > using SQL.

Looks like the wrong tool for the job. I like this quote at the end:

> It's free like a puppy, not free like a beer.</comment>


Group GEB reading starts today - <AUTHOR> http://www.reddit.com/r/GEB/comments/oksrq/lets_begin_details_inside/

<comment><USER2> In case anyone else wonders what GEB stands for: <http://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach>

</comment><comment><USER3> Why GEB and not The Mathematical Experience?

</comment><comment><USER4> I'm looking forward to this!</comment>


Blekko, The "Slashtag" Search Engine, Goes Live - <AUTHOR> http://searchengineland.com/blekko-the-slashtag-search-engine-goes-live-54447

<comment><USER2> The /compare slashtag is pretty cool, as in: [https://blekko.com/ws/news.ycombinator.com+/compare?c=slashd...](https://blekko.com/ws/news.ycombinator.com+/compare?c=slashdot.org)

<comment><USER3> Ah I didn't know that there were some special /slashtags, this is even a parametric slashtag...

</comment></comment><comment><USER4> From a quick test, the search engine is pretty bland and unexciting for normal queries, but it motivates you to make different types of queries, where it does have interesting results.

For instance, just searching for "china" isn't anything interesting, but the site is begging for you to do "china /tech" instead, where it is interesting (and also traffic in China, Maine, but nobody's perfect).

</comment><comment><USER5> Being able to limit your search results to e.g. "/liberal" is one of those powers I really don't trust people to use responsibly. Not that it's doing anything 99% of human brains don't do automatically, but still.</comment>


Ask HN: Connect the Dots - <AUTHOR> Let me run you through a series of conversation I had with Impersonating Amex Employee and how my Identity is stolen, The Amex employee gave his name as george and im going to put myself as X George: Hi sir, we are calling from amex regarding a fraud swipe, we are calling to confirm if you did it? X: No im driving right now and I am not the one, can you please mark that charge as fraud? George: Sure, before that I see you are not enrolled in fraud alert services, I'm going to activate that for you, can you tell me a OTP which is being sent to you? X: What, I do I know I should trust you. Let me call Amex myself and if they re-iterate what you said then I will take it from there. George: Hold on sir, I can give you your SSN number for confirmation and he gave me my correct SSN number X: ok I believe you( in my defense he gave out my freaking SSN Number) George: Can you answer a security question for me? X: I did( George knos my SSN,Email, phone, address, now my security ans) George: Now i'm going to run down your transactions and let me know which are fraud X: He ran down my recent transaction( has access to my transactions somehow) George: Ok I have marked them as fraud George: I am going to order a replacement and he gave me a tracking number and we are going to lock your account and give a temporary password X: ok( by this time I get a message saying XXX amount is cash despense alert) X: George, I see fraud alert on cash dispensing George: Ya I will cancel it now(meanwhile I tried to enter the new password and it didnt go through) X: I tried logging using temp password and it isnt letting me through George: Let me reset it again now you will get another OTP for this, can you give me that X: gave that Call got terminated and cash was dispensed X: so let me call amex again <comment><AUTHOR> This time a woman names debbie spoke, Debbie: Hi sir, how can i help you, X: explains everything Debbie: seems like one of our representative, so sir can you answer this security question? X: I did Debbie: thank you for confirmation. I see your account is locked and good that you called us because our input is necessary to unlock it, so I'm going to ask few questions asked me where I work, whats my salary, where im based out of what type of apartment I live in X: why are you asking such questions? Debbie: It is necessary for us to ask certain questions to understand if you are eligible to hold onto the account X: ok, so can you reset my account, the older password isnt working Debbie: Sir try after 15 mins for the systems to get synced up and hung up the call X: tried amex again This time Im gonna call him as agent and he is from fraud team X: explains everything about both the calls. agent: Sir I see you made the cash despense and your account is not locked, nobody ordered for a new card X: I explain him from start about all that happened agent: very sorry sir. I will go ahead and mark the transaction as fraud and will lock your account and will send out a new one.( this time i got email confirmation of new card being processed.) X: can you please put my account in high risk as I believe my personal information including my SSN is stolen. agent: Yes sir we will take care of it Note: George, Debbie, Agent all spoke from the same phone number Now I just couldnt figure out how george has access to all my information including a way to reconstruct my card Am i being watched? I know I'm hacked for sure. What can I do to prevent futher damange and also help me connect the dot I would understand if someone has my ssn info or did a cash dispence separately(because someone has to read my card from somewhere I swipped) Doing both is what is scaring me, is someone watching me like a thriller movie?

</comment><comment><AUTHOR> Due to character limitation had to paste the other half in comments, please do read through

<comment><USER2> Any chance you could edit to help the formatting? Might be easier on the eyes.</comment></comment>


Turbo-boost your WiFi signal - in 15 minutes - <AUTHOR> http://blog.eyesandfeet.com/2010/04/turbo-boost-your-wifi-signal-in-15.html <comment><USER2> If you want to spend a little bit more time for a better result (while learning a bit) you can homebrew a more solid solution.

Directionals are pretty easy and fun to make out of everyday items, which do the trick quite well. Here's some examples (<http://www.turnpoint.net/wireless/has.html>) and here's one you can do in maybe 30 minutes (<http://www.turnpoint.net/wireless/cantennahowto.html>)

Those antennas are good for long-haul point-to-point links, but can also do a pretty good job in point-to-multi, if it's close by. I gave a neighbor internet access for a few months with one of these pointed at his house, without any special gear on his end.

Omni-directionals are a bit more advanced, but there are some good designs out there. I built one similar to this once (<http://wireless.gumph.org/articles/homemadeomni.html>). Because these are a lot more sensitive to flaws than directionals, it's important to test.

Sector antennas are a good compromise between omni and directional, as it gives you a nice wide area of coverage ([http://www.grynx.com/projects/wide- angle-sector-antenna-14-d...](http://www.grynx.com/projects/wide-angle-sector- antenna-14-dbi/))

Keep in mind that just installing DDWRT and cranking up the juice doesn't always result in a better wifi experience (though sometimes it can give access to that one dark spot), and is probably screwing up everyone else's experience. Hooking an ipod up to a bullhorn gives you nice range, but there's a lot less fidelity.

</comment><comment><USER3> I spent an entire afternoon trying to do this about 2 years ago from some stupid paper cut-out design and tin foil. I never felt so punk'd in my entire life.

<comment><USER4> I had the same experience, didn't help much at all

<comment><AUTHOR> I can imagine - must be disappointing when you take the effort and nothing happens. Personally, I had a fantastic improvement in signal strength. In fact, strangely enough, the signal also improved in the room right below the one where the directional+wifi was.

</comment></comment></comment><comment><USER5> In Buenos Aires, old apartments are made of thick, stone walls that really hinder signals. In a long apartment you have to repeat.

I bought 3 WRT-54GS refurbs from newegg for $29 each, installed DD-WRT, set 2 of the three as Bridged Repeaters, put them all on the same SSID (protip: put the word Mega in front of the old SSID), spread them across my friend's house, and he now has great signal everywhere.

<comment><USER6> Why the word 'Mega'.

<comment><USER5> shameless self promotion. so that every time my buddy has great signal he remembers it's because of his tech-friend's dd-wrt hacking prowess.

</comment></comment></comment><comment><USER7> Or, buy a Linksys WRT54 router, load Tomato (<http://www.polarcloud.com/tomato>) and jack up the signal level. Be careful not to exceed the legal limit (or not).

<comment><USER8> DD-WRT lets you do the same. The power on my Netgear defaults to 71 out of 251 (mW) :)

I'm seriously loving DD-WRT, far far far more capable. And IPtables!

<comment><USER5> does it really help to boost the signal? i tried before and didn't seem to get much of a difference.

<comment><USER8> Haven't been able to test it, just looked for a comparable setting. I'd bet it depends on the router, but I should be able to actually test tomorrow. Want me to?

<comment><USER5> sure - i think one of the above answered it for me that there is no equivalent boost on the client end which i think is why it hasn't helped me much.

</comment></comment></comment></comment></comment><comment><USER9> What would I do for a Cisco WRT54G2 V1 with internal antennae? The thing is flat, should I sit the entire unit in a 'bowl' and direct it at my office?

<comment><USER10> An alternate method I've used is building an antenna on the computer side if you can't build it on the router side. You can get wireless adapters which are in the from of a usb thumb stick. Coupled with an extension cord to connect to a PC, you can put the entire adapter at the focal point of a large dish.

</comment></comment><comment><USER11> Reflected waves can help or hurt (if echoes interfere with themselves). So even without a intentional reflector like this, be aware that in challenging environments, repositioning your antennas just a little (new angles, a few feet further or closer to a signal-reflective surface) can make a big difference.

</comment><comment><USER12> I just made a poor quality one of these with plain old paper and aluminium foil. My iwlist scan results:

Frequency:2.437 GHz (Channel 6)

Before: Quality=40/70 Signal level=-70 dBm

After: Quality=47/70 Signal level=-63 dBm

Not very exciting, but not bad. Lots of access points in my block make connectivity a real problem, so I'll take whatever I can get.

My roughly-cut parabolic reflector isn't super-parabolic, make of that what you will: maybe a better shape would be markedly better, maybe it means yours doesn't have to be perfect either _shrug_

My tips if you want to try yourself: tape the tabs in place but try not to put creases into the foil when you do. If it's not already clear, only put foil on the back reflector (not the curved support)

<comment><USER13> If your dBm readings are correct, your signal strength increased by 10^((-63-(-70))/10) = 5 , which is not so bad.

</comment></comment><comment><USER14> I have to wonder if the parabolic shape really matters, or if a large conducting panel of any shape would suffice. I have no idea, really, but would love to hear from anyone who has.

<comment><USER8> Any reflecting surface will improve it in the reflected direction, the parabolic shape is just to improve the distance further and give better aim.

Just about anything reflecting works, my personal favorite directional antennas use Pringles / coffee cans or a metal colander.

</comment></comment><comment><USER15> I prefer the salad bowl method:

<http://www.youtube.com/watch?v=LY8Wi7XRXCA>

<comment><USER16> exactly what i was thinking, classic

</comment></comment><comment><USER17> On the same page of the cutout design (<http://www.freeantennas.com/projects/template2/index.html>) there is what the author of the post call a "scary graph". Looks like a cool Windows based signal monitor thing. Does anyone know it?

</comment><comment><USER18> It gives you more bars but does it give you more speed?

And if this actually works, why don't commercial products build this right in? Couldn't you reflect more signal with a much smaller reflector if you're up close to the transmitter?

<comment><USER19> I made a reflector from the same template a few years back, worked great. The reason it works is it improves both transmit and receive. The reason why a regular consumer AP doesn't come with that sort of antenna is it's directional, basically you trade wide coverage for better coverage over a narrower angle and most people have a hard enough time just standing their omnis straight up.

For more commercial work there are plenty of options: <http://www.l-com.com/category.aspx?id=2073>

<comment><USER20> However if you place it in one corner of your house and install a reflector with a 90-degree coverage angle you should be able to increase your signal strength up to ~4-fold with a perfect reflector as you're 1/4 your coverage area. Realistically getting a 50% increase in strength is great.

Using materials like aluminium or copper would greatly increase the performance of your reflector. If you by chance had a copper mesh, it would kick the tin foil a mile. I personally have access to sheet aluminium so if I were ever to build one of these I'd definitely be making it from that and not foil. (IIRC reflectivity in metals is improved with thickness, with these tin foil reflectors you're probably still getting a strong enough signal at close range from the side of the reflector you don't want signal.

</comment></comment></comment><comment><USER21> Well this would only work if your wifi router was on the far side of the house and you want to direct the signal to the rest of the house. If it was in the middle of the house it wouldn't work as well.

<comment><USER22> If it were in the middle of the house, this issue wouldn't arise in the first place.

<comment><USER23> Unless you had a REALLY big house :P

</comment></comment></comment><comment><USER24> You can do it with soda cans too:

[http://www.wonderhowto.com/how-to-increase-wireless- signal-s...](http://www.wonderhowto.com/how-to-increase-wireless-signal- strength-soda-can-162004/)

</comment><comment><USER25> I don't see a turbine anywhere. There's not even any amplification (boost). It's just a directional antenna.

</comment><comment><USER26> Thanks dude

</comment><comment><USER27> Good way to cook the children.

Lots of routers make annoying high pitched noises, even without that the flashing lights are a distraction - I wouldn't put one in anybodies bedroom (unless I could turn the lights off and be sure it's silent).</comment>


How Companies Manipulate Employees (With Examples) - <AUTHOR> https://thepowermoves.com/corporate-lies/ <comment><USER2> I agree with the articles and have seen these manipulations in my large employer. I wanted to drink the kool aid and be a happy camper but I developed a cognitive dissonance that resulted in my quitting. My salary was good and I was not being exploited in any way, it was just the constant lie that money was not the overarching decision maker that made me sick.

The only alternative I see is paying employees their worth, and admitting the only tool of any worth is money. Of course a company wants to save money wherever possible, because who is the leadership responsible to, the owners, not the customers and not the employees.

As an employee then maybe it is about taking on the values of the leadership, money first, but that is not the employee's job. The employee has a task that needs to be solved and if that task will be done for less money because of values then it is the responsibility of the leadership to exploit those values. Leadership has decided how employees time is to be used effectively and that is why they provided employees with a list of tasks.

For me honesty was more important than the job, but I know I'm in the minority and paying the price for it.

</comment><comment><USER3> The overall tone of this article is really bitter and seems like it's written by someone who has a really aggressive anti-corporate stance. I'm not familiar with the publication, maybe that's their agenda.

As a business owner, I strongly disagree with the fundamental premise of the article. I, as a business owner, take on a significant amount of risk to start a business. I sell services to clients and change the clients more than I pay my workers in order to turn a profit. My workers get paid a fair hourly wage and are treated well in exchange for selling their labor. That's the arrangement. As a worker you have two options:

1\. sell your labor in exchange for a fair wage. You make money. I make a profit. As the owner of the business I get to keep the profit.

2\. Start your own business. Then you get to keep the profit in exchange for a lot of work, stress and risk.

It's a simple equation. If you don't want to do number 2, then you'll be doing number 1. It's simply a matter of preference. Neither is better than the other.

<comment> xkcd-sucks Is it mischaracterizing anything though? The nature of life is to eat other life. The nature of corporations is to propagate atany cost. The nature of employees is to accept what's given, but the nature of humans is to take what's possible to take. There is a lot of grey between accepting everything and starting a business, e.g. anyone that's "high level". It's human to accept what's "realistic", e.g. a "fair" wage, and it's also human to take all that one can take.

Presumably you started your companies because you were sick of someone else's perceived bullshit, and you probably had a bunch of structural help in order to become an employer.

</comment></comment><comment><USER4> I thought about it, and it's genius. At least if you want to understand human nature</comment>


MetaMate – A common, centralized API for every service - <AUTHOR> https://metamate.io/ <comment><USER2> I read the home page a few times, still can't work out what it is/does

I mean, this all looks fairly complicated and/or clever [https://metamate.io/docs/0.1/asg/](https://metamate.io/docs/0.1/asg/) but I'm not sure what it all adds up to

this doesn't seem to work [https://showcase.metamate.io/hackernews-user- activity](https://showcase.metamate.io/hackernews-user-activity) (I just get "Network Error")

"Reference" link here is a 404 [https://metamate.io/docs/0.1/](https://metamate.io/docs/0.1/)

<comment><USER3> I'm also getting Network Error. All I got from the site and GitHub was: some way of creating an API around web data. Wasn't directly clear why it's a superior way of doing that.</comment></comment>


Google ads crush Facebook ads in efficacy - <AUTHOR> http://www.tekpersona.ca/awasteoftime.html#sthash.Lj5wRWyM.dpbs

<comment><USER2>
      > The true value of Social Media is in that it provides a
      > very effective forum for both Corporate (business) and
      > Personal Branding.

[http://www.theonion.com/articles/i-am-a-brand-pathetic- man-s...](http://www.theonion.com/articles/i-am-a-brand-pathetic-man- says,30545/)

<comment><USER3> a refreshing rant on the topic is maureen johnson's "i am not a brand": <http://www.maureenjohnsonbooks.com/2010/06/08/manifesto>

</comment></comment><comment><USER4> This is no surprise to anyone who understands the difference between search ads and "display" ads. People have more buying intent when they search, so search ads bring around $50/1,000 impressions (average over all US search queries) verses $1 or $0.1 for display ads. There's nothing special about Facebook in this regard, and I think the title of the blog and the hacker news title are a bit over-wrought! How about, "For 10 years and counting, search ads have beaten display ads."

</comment><comment><USER5> we have seen the same things mentioned in this article. What happens to FB when the rest of the world wakes up.</comment>


Google Drops Discussions Search Filter and Others - <AUTHOR> http://www.seroundtable.com/google-search-filters-gone-17993.html <comment><USER2> The promise was that, as search engines get better, there would be no need for bookmarking and downloading, since all the content in the world is only one sloppy search query away.

Somehow, I seem to have ignored this stubbornly and instead bookmarked pages, downloaded videos and copied strings (!) out of the web for years.

Not that it did me much good, I now have thousands of inconsistently sorted bookmarks and a bunch of directories filled with assorted data.

What this article tells me (again), is that because you can't trust search engines to behave in a predictable manner, you can't even be sure you'll be able to find the stuff that you know is online.

<comment><USER3> You can't really trust bookmarks either. Most of my bookmarks are dead either because the site is now gone or because some blog author decided to change their path format and didn't bother setup 301 redirects.

<comment><USER4> Microsoft's help pages are the worst for this. "Oooh, confusing .NET problem I'll google a response - Hey, a nice forum-post explaining that MS has a document telling me how to handle this issue _click_ NOOOOOOOOOOOOO".

<comment><USER5> Whenever I have a C# problem and google it, I always skip the MSFT documentation and go straight to any possible Stackoverflow or discussion page.

<comment><USER6> Same here, it's not even worth going on their site.

</comment></comment></comment></comment></comment><comment><USER7> This is the final nail in the coffin of DejaNews. Just another step in Google's continuing mission to make their Search product worse & worse.

It's baffling.

<comment><USER8> This.

DejaNews was the absolute best source of technical information from the USENET archives, with posts back to the 1980s. By 2001 Google had everything under control:

[https://groups.google.com/forum/#!search/USENET$20archives](https://groups.google.com/forum/#!search/USENET$20archives)

Then Google lost it.

The addition of "Google Groups" resulted in tons of SPAM and almost nothing of use. Now, because their own baby is so ugly, Google decided to kill all their adopted children. So much for "Don't be evil".

<comment><USER9> I like USENET too. Information should be organized in that way. It was just too old. If we put the search engine of top of that with a nice graphical UI, that'll be a lot more help.

This is what I'm always thinking about since 1997. Now I got the chance to put together something for people's daily life, but not perfect. It's in early beta now. [http://kck.st/JNqv8z](http://kck.st/JNqv8z)

</comment></comment></comment><comment><USER10> You still can, just not as obviously: [https://www.google.com/?tbm=dsc](https://www.google.com/?tbm=dsc)

<comment><USER11> Thank you, it works. After some googling, it seems the following are supported [1]:

      Applications : tbm=app
      Blogs*       : tbm=blg
      Books        : tbm=bks
      Discussions* : tbm=dsc
      Images       : tbm=isch
      News         : tbm=nws
      Patents      : tbm=pts
      Places*      : tbm=plcs
      Recipes*     : tbm=rcp
      Shopping*    : tbm=shop
      Video        : tbm=vid

* those not available via normal search page (from my location at least).

[1] [http://stenevang.wordpress.com/2013/02/22/google-search- url-...](http://stenevang.wordpress.com/2013/02/22/google-search-url-request- parameters/)

<comment><USER12> Does anyone know what these searches do? Does a filter limit the search to known domains, or silently add keywords that identify a site as a blog or a video?

</comment></comment></comment><comment><USER13> Why would you want to search Discussions or Blogs? Everything interesting is already on Google+. /sarcasm

You know, someday Google+ will no longer be the "strategory" priority. There will be a new CEO. All the hands on the corporate ouija board will shift in yet another direction.

At which point, all these moves to deprecate parts of the web, in an effort to shore up G+? They will look even more unfortunate. So much discarded, for so little gain.

<comment><USER14> This actually looks worse than a move to shore up G+. The list of things they've pared it down to looks suspiciously like a list of categories that have the highest advertising revenue.

</comment></comment><comment><USER15> I also notice that depending on your search terms it now changes the order of the menu items; I was very surprised a few day ago when I noticed that Images wasn't the second list option on the screen.

To test this seach for "images" or "videos" on google. It will change the list order.

This is extremely irritating as it tries to guess whether you're trying to search for a certain type of of thing (text/images/vid/news/shopping/etc), as it frequently guesses wrong, and then sends us searching for the right list option because they've been shuffled round.

Designers, do not change your menu item order unless you're damned sure you're gonna get it right. Confusing your users is never helpful.

      ** edit **

Ah, I see the article mentions this. Well it's still bloody annoying.

<comment><USER9> Agree. We need the consistency, at least in a certain period of time. Whenever the search results changes, we are lost unless we choose to do the filtering. But the group of filtering does not quite make sense for everybody.

Google tried to be more intelligent on search, but it's not the right way. Let people keep their useful information of their own should be the right direction. See the video here: [http://bit.ly/1hJKNux](http://bit.ly/1hJKNux)

</comment></comment><comment><USER16> Maybe this is a good spot to mention how much the updated Google Maps also suck. Pretty though.

<comment><USER17> I still can't figure out how to get street view without actually searching first.

<comment><USER18> Click where you want it, it should appear in the upper left.

</comment></comment></comment><comment><USER19> Any alternatives? I asked about this two days ago but got no answers: [https://news.ycombinator.com/item?id=7124132](https://news.ycombinator.com/item?id=7124132)

Also, I know that "?tbm=dsc" works but I expect the quality of the results to start falling (it already has for a while) since Google probably isn't going to work to improve the filter.

<comment><USER20> [http://boardreader.com/](http://boardreader.com/) is a search engine of just forums. Pretty useful.

Also this search list of Q&A sites might be useful: [http://nuggety.com/u/nuggety/questions-and- answers](http://nuggety.com/u/nuggety/questions-and-answers)

</comment></comment><comment><USER21> Everyone is dissing Google here. Perhaps understandably so since most folks in HN are advanced users and an advanced search feature was removed.

Google is driven by data and sorry but advanced users are a mere blip in that data. I can bet there was a gazilion user studies that showed that your grandma was super confused when 10 options would pop up when she pressed 'More'. The same studies showed that she was much less so confused when 4 options were presented, increasing engine usability for her.

My point here is that some folks attribute sinister motives (more search dollars), while the actual motive is just to make the interface usable for more people.

(Disclaimer: I work at Google, not in search though.)

<comment><USER22> I don't get your point. Is it that people should _not_ complain about Google removing a feature they like, because you have presented a guess as to why they removed it? (FWIW - I agree with you that it was not maliciously removed)

Personally I like it when users complain loudly about decisions that impact them. The idea of negative publicity might deter such actions in the future (in theory anyway).

<comment><USER23> The point is in the third paragraph. A lot of comments here conclude that the motives for doing this are sinister. One person even thinks it was removed because it would make users spend hours reading discussions when that time should really be spent shopping.

People should absolutely complain when a change impacts them negatively. People should also keep in mind that with a user base the size of Google's, every change is going to upset someone.

[http://xkcd.com/1172/](http://xkcd.com/1172/)

<comment><USER24> The old saying "every change will upset someone/can't please 'em all _chuckle chuckle_ " is a cop-out.

I've had managers say exactly this about flawed interfaces I've been asked to build, with marketing directives dictating crucial layout choices and ultimately problematic usability. Warned against but overridden with "can't please em all". Yet the complaints came rolling in big time over an extended period due these change to the tv guide I made.

The problem with marketing depts making interface decisions is that their priorities are with things like "aligning with the tone of the campaign" or "blatantly copying a competitor for lack of confidence in one's own original ideas".

We're talking about a feature that has been in Google search for many years, then suddenly gone in the blink of an eye, without explanation.

"Sinister" actions are done quietly, you hope no-one will notice. If their reasons are in the interests of users, they'd blog or post about it somewhere to keep people informed about their product. They failed to mention it.

Google has already made it clear they are directing traffic to G+ to sustain their bank balance and pursuit of making a Google spacecraft or whatever. This latest move, plundering useful things in search, is the same bad User Interface behaviour as forcing people to sign up to G+ to leave a rating out of five for an app. They're forcing together two unrelated systems or services with bad welds, or removing useful components leaving empty space.

</comment></comment></comment></comment><comment><USER25> I keep dreaming of an open source, distributed search engine and social network.

But I guess things will have to get much worse before they get better.

<comment><USER26> Open source, distributed search engine: [http://yacy.net/](http://yacy.net/)

Open source, distributed social networks are a dime a dozen: [http://en.wikipedia.org/wiki/Comparison_of_software_and_prot...](http://en.wikipedia.org/wiki/Comparison_of_software_and_protocols_for_distributed_social_networking)

</comment></comment><comment><USER27> _" Now when you search, the type of results you can select at the top of the page will vary depending on what makes sense for your search."_

Makes sense to Google, that is. Shopping and apps make sense. A discussion that might take an hour to read or prompt one to read further discussions instead of shopping, liking or checking in doesn't.

It's a future where I decide to go out to eat because I can't find a recipe I like. I climb in my self driving car and it takes me to Taco Bell. And only Taco Bell.

</comment><comment><USER28> Am i the only one recently having moments like "wait a second!", while using google services?

Connecting every google service to a single one might be the best thing for Google, BUT, not for me!

Google, you are almost merged with whole Internet thing, so stop forcing people into using a single system!!

I like and use gmail often but don't want to be logged in while checking youtube!

<comment><USER7> Just use a different browser.

</comment></comment><comment><USER29> Is there an operator similar to "define:" that you can use? I can find a link for operators, but I do not know how up to date it is.

[http://www.googleguide.com/advanced_operators_reference.html](http://www.googleguide.com/advanced_operators_reference.html)

</comment><comment><USER30> This train has derailed a while ago.

You couldn't search discussions AND verbatim.

Verbatim got broken a while ago too.

Now they explicitly state that they skipped some of your search terms (presumably for better paid keywords).

Oh, and don't forget paying link-scheming SEO clients.

</comment><comment><USER31> You can use this url* although it seems to me it's inferior to what they offered previously. Maybe it's just my imagination.

* [https://www.google.com/webhp?tbm=dsc](https://www.google.com/webhp?tbm=dsc)

</comment><comment><USER32> If this isn't ironic, I don't know what is.

<comment><USER33> Why ironic?

<comment><USER32> Google (which started as a search engine, way back when dinosaurs roamed the Earth) is now increasingly working on _narrowing down_ the search options available to users.

<comment><USER34> I need more clarification on how this is ironic.

<comment><USER32> The appeal of Google, back in the days of Altavista, was in the fact that it could search various things, giving better results. Now it's intent on giving you fewer things to search.

<comment><USER35> That changed long ago. All the google is doing last two years is stopping projects, removing features and replacing acceptable GUIs by less practical ones.

<comment><USER36> To be fair, they also work pretty hard at collecting your personal information.

</comment></comment></comment></comment></comment></comment></comment><comment><USER37> Duck Duck Go.

Jus' sayin'

<comment><USER29> I am afraid I don't understand. Duck Duck Go searches google, so if you are having a problem with a google service, how would this help?

<comment><USER38> No, it uses bing + a little secret sauce, as I understand it.

Edit: "Link results are API driven, though top links may come from other sources. Link sources include: Bing, Yahoo, Yandex, Blekko, WolframAlpha, and many others."[1]

"DuckDuckGo gets its results from over one hundred sources, including DuckDuckBot (our own crawler), crowd-sourced sites (like Wikipedia, which are stored in our own index), Yahoo! (through BOSS), Yandex, WolframAlpha, and Bing. For any given search, there is usually a vertical search engine out there that does a better job at answering it than a general search engine. Our long-term goal is to get you information from that best source, ideally in instant answer form."[2]

1\. [http://highscalability.com/blog/2013/1/28/duckduckgo- archite...](http://highscalability.com/blog/2013/1/28/duckduckgo- architecture-1-million-deep-searches-a-day-and-gr.html)

2\. [https://duck.co/help/results/sources](https://duck.co/help/results/sources)

<comment><USER29> Ah, my mistake, I confused it with scroogle. I tried to upvote you but I don't think I have enough Karma.

</comment></comment></comment></comment><comment><USER39> I am finding google search more and more doesnt find what I am looking for. Its frustrating.

</comment><comment><USER40> I am very sad to see the blog option go away. I used that all the time.

</comment><comment><USER41> Sorry to simplify, but I remember when Google enabled,

<comment><USER42> I tried to use Google to autocomplete your sentence but all I got was a link to this discussion.

</comment></comment><comment><USER43> I want that f--- filter back, it's the best thing about Google...</comment>


Identity theft made easy - <AUTHOR> http://www.google.com/images?hl=en&safe=off&gbv=2&tbs=isch%3A1&sa=1&q=passport+scan <comment><USER2> Wow, some of those are actually valid passports and not just samples or jokes.

<http://www.woosterpc.com/teaching/images/passport.jpg>

<http://img505.imageshack.us/i/aishwaryapassport0320hw9.jpg/>

</comment><comment><USER3> I wonder who leaked my password <http://www.apathysketchpad.com/images/uploads/mushport.jpg></comment>


US companies giving in to pressure from China - <AUTHOR> https://www.abacusnews.com/big-guns/seven-times-us-companies-gave-pressure-china/article/2157856 <comment><USER2> I look at it this way: change is not going to be forced on the Chinese government by US corporations. That can only come from the Chinese population (with possibly a small assist from foreign governments). So what's the best way to help their citizens with changing the government?

I believe the best way is to make them prosperous and informed; people who are hungry or worried about their job are less likely to question the political status quo. Are US corporations helping with either of those? Not much with the latter but I believe more trade helps the former, as long as it also doesn't make it easier for the government to oppress the Chinese population.

Is there much of a difference between US corporations doing business in China and the rest of us buying inexpensive goods from Chinese companies?

<comment><USER3> This thinking is common that more middle-upper class people won't put up with a dictatorial regime.

But isn't Singapore kind of a counter to that?

<comment><USER4> Singaporeans at least have the right to leave. Chinese people aren't even free to migrate internally.

Singaporean internet access is significantly freer than Chinese. Singapore has freer markets, more rule of law.

They don't have a real democracy, and there are restrictions on speech, but it's absolutely crazy to equate it with China. If China turns into Singapore as it gets richer it'll be a humanitarian miracle.

Unfortunately I share your skepticism. Not because of Singapore as a counterexample, though, but because of China-to-date. They haven't been liberalising, they have been getting less free, more nationalistic and so on. The Party has increased its control on every part of life and all levers of government. I can't see why that trend should reverse.

<comment><USER5> Yes and even if go you are chinese first. Kidnapping foreign national in a foreign land (Thailand not just 1c2s Hk) is done.

You are chinese first even if you have other passport ...

</comment></comment></comment></comment><comment><USER6> Honestly, I don't find the censorship tools or local-hosting of data to be too offensive; if the big Cloud players don't do it, another will, with the net effect being the same. What bothers me the most is the airlines listing Taiwan as part of China. This goes against policy of the vast majority of the world, including the U.S. It's one of many reasons (the others are not germane to the article so I will not list them here) why I refuse to buy new Chinese products. It has been quite difficult, but between second hand stores and a relatively robust domestic manufacturing base, I've made it work.

</comment><comment><USER7> An excerpt, which sounds shockingly similar to GDPR compliance, listed as an example:

"APPLE, AGAIN

That wasn't the only issue Apple faced in the country. Earlier this year, Apple moved Chinese iCloud data to a state-run company after new laws were enacted requiring foreign companies to store data locally."

<comment><USER8> The difference is that in the case of China (and Russia), requiring local data storage is intended to enable government oppression. EU countries have much better human rights records than China and Russia do.

<comment><USER7> Do you believe it is morally wrong for China and Russia to pass their own GDPR-esque legislation? And for corporations to comply with said legislation?

<comment><USER9> Motive and outcome are valid measures for deciding if something is moral

</comment></comment></comment></comment><comment><USER10> Boeing and other traditional high-skill/tech manufacturers have been giving away their to China technology for years.

CEOs/Boards/State don't care though, its all about next quarter's numbers.

<comment><USER11> Boeing doesn't sell tech to China. They sell them planes, spare parts, they even put a finishing center in China, but no IP transfer.

McDonal Douglas did sell MD80 machining tools to China, which is what the 909 is based on. However, this is only for the body, the other stuff (avionics, jet engines) are being imported.

<comment><USER10> "Over the decades, Boeing and other American companies, including General Electric, Honeywell and Rockwell Collins, have helped China to build the broad foundation necessary for an aerospace industry. That same Chinese industry is now getting close to producing its own commercial jets that could one day take much of the profitable Chinese market back from Boeing and rival Airbus."

[https://www.washingtonpost.com/news/wonk/wp/2015/09/23/china...](https://www.washingtonpost.com/news/wonk/wp/2015/09/23/china- offers-huge-rewards-for-u-s-companies-like-boeing-but-it-could-also-take-away- their-business/?utm_term=.742f49f693c0)

</comment></comment></comment><comment><USER12> These US corporate elites are selling out US and the rest of the democratic free world in order to meet shareholder's expectations (and of course, MONEY!). And giving into a dictatorship and giving that dictatorship jobs, money, capital, knowledge, and power. That dictatorship now has imprisoned close to 1 million people in Xinjiang.

Shame on you, Sundar Pichai. At least when Google was lead by Erich Schmidt, Google pulled out of China, and didn't bend to the whims of Chinese government.

Shame on you, Tim Cook. Apple has 4.8 million jobs in China, more than double the jobs in US. [https://247wallst.com/jobs/2017/03/17/apple- supports-4-8-mil...](https://247wallst.com/jobs/2017/03/17/apple- supports-4-8-million-jobs-in-china-more-than-double-us-total/) . Shame on you for not bringing jobs back when news broke out that Foxconn has created a slave labor like working condition.

Shame on you, Mark Zuckerberg, for continuing to appease the dictator in China, so that Facebook may gain a foothold there.

Shame on you, Satya Nadella, for continuing to supply censored search results in Bing. and for not taking a stand against censorship. But I guess the microsoft borgship has a lot in common with China.

Shame on you, Jeff Bezos. You become the richest man on earth, by allowing pirated, stolen, IP infriged goods from China to be sold on your platform without care, destroying small US businesses. You also partnered with Foxconn for Amazon Echo, tablet, and other hardwares. Amazon is a showcase for trade imbalance. disgusting.

<comment><USER13> The US was not engaged with China prior to 1971, how did that work out for the United States and for China? Over the long-term, engagement with China has led to more personal freedom for the Chinese.

Trade with China has helped lift hundreds of millions of people out of poverty, while helping with the prosperity of the United States. True the corporations are reaping most of the benefits, but that's natural under a free market. We need reforms here in America to broaden the benefits of trade to all workers. Outsourcing to Asian countries existed prior to China's rise in the supply chain. Most of the stuff that is "Made in China" have major components that come from America's allies such as Japan, Korea, and Taiwan.

Google is in a position to offer a superior search product to China. We should see this as an opportunity to advance freedom.

<comment><AUTHOR> Are you seriously suggesting that a censored search engine is "an opportunity to advance freedom"?

<comment><USER13> Yes. The choice before Google is whether to enter the Chinese market. Staying out of China may make some people feel morally satisfied, but will not give Chinese people more information. Rather Google should be using its influence to challenge censorship from the inside after it reenters the Chinese market.

<comment><USER14> > Yes. The choice before Google is whether to enter the Chinese market. > Staying out of China may make some people feel morally satisfied, but will > not give Chinese people more information.

Entering the Chinese market also lends prestige to the censoring regime (e.g. "all the companies of the world submit to our demands"), and also makes those demands seem more reasonable.

> Rather Google should be using its influence to challenge censorship from the > inside after it reenters the Chinese market.

The CCP doesn't care at all about Google's challenges to it. Either Google complies totally, or it's out. The CCP doesn't need Google, and they probably would rather people use Baidu anyway.

</comment></comment></comment></comment></comment><comment><USER15> It's not just companies, it's US people too. The original HN thread on this subject ([https://news.ycombinator.com/item?id=17660872](https://news.ycombinator.com/item?id=17660872)) had a truly depressing percentage of comments saying "No big deal" using some combination of whataboutism, false equivalence and simple fatalism. I seem to remember that when Google first pulled out of China the reaction among HN types was much more positive; what has happened to our morals since then?

<comment><USER16> Uh, morals? What does ethics have to do with Google operating their search business in China?

I'm confused why I am supposed to be outraged. I don't have any particularly strong feelings about censorship in foreign countries - how does what the Chinese government do affect American speech? I guess censorship in general is _bad_, but it's purely an internal, domestic matter. The Chinese aren't scrubbing American textbooks of the historical record. Really, I think I've only noticed China's influence in cinema; I see more Asian lead roles in Hollywood films, but this seems more a corrective change than bad thing.

I'm not seeing how what happens to internet search in China affects Americans. Nor what ethical concerns there are.

<comment><USER17> I believe the expected reaction is the same amount of moral outrage that would accompany the American government treating the American public the same way.</comment></comment></comment>


I was behind the wheel when a self-driving Uber failed – here's what happens - <AUTHOR> http://www.businessinsider.com/uber-self-driving-car-fails-2016-12/#and-uber-itself-has-said-it-chose-pittsburgh-because-it-poses-so-many-challenges-for-its-self-driving-cars-we-have-a-very-old-city-very-complex-road-network-real-traffic-problems-here-and-extreme-weather-here-said-raffi-krikorian-the-director-of-ubers-advanced-technologies-center-so-in-a-lot-of-ways-pittsburgh-is-the-double-black-diamond-of-driving-4 <comment><USER2> The article does not talk about where it failed. It only covers that it often and suddenly switches back into manual mode. One case was on a completely empty road.

</comment><comment><USER3> Nice puff piece, wonder how much money did businessinsider get for that one</comment>


The 5 things you should know to avoid reading unuseful articles - <AUTHOR> http://hernan.amiune.com/blog/entry/the-5-things-you-should-know-to-avoid-reading-unuseful-articles

<comment><USER2> oops: " _4._ List and enumerations of things you have to know or do are one of the number one scams in order to attract you to read something. They are short, easy to read, they give you the sensation you have learned something and their algorithm or recipe form make them look as mathematically rigorous."

</comment><comment><USER3> That is pretty meta.</comment>


Three Sheldons - <AUTHOR> https://www.buzzfeed.com/albertsamaha/threesheldons <comment><USER2> I can't move past the son's crime. It is horrific sadism and violence. Read about Yu: [http://www.wikicu.com/Minghui_Yu](http://www.wikicu.com/Minghui_Yu)

Why should we delude ourselves that he is genuinely remorseful or that he will redeem himself for such a horrific crime? Some criminals reform, others don't. It is a mistake to assume that everyone can or will mend their ways. One consequence of the article's theme, the fact that we don't fully control our destiny (absence of free will) is that the likelihood of recidivism is high.

<comment><USER3> Do you think, having read the article, that it was a mistake?

Even if we accept that not everyone can reform, what should we do about it? It is similar to the problem of sending innocent men to prison. I would err on the side of caution and freedom.

</comment></comment><comment><USER4> This was a real pleasure to read (in terms of prose and not content, obviously). I'm gratified to see BuzzFeed make the jump to professional, long- form pieces.

Kudos to the author for the excellent writing, and to the Sheldons for sharing their story. Speaking as a white man who grew up 8 blocks from the apartment mentioned in the article, it's both surreal and saddening to learn how drastically different (meaning unfair) their lives were and continue to be.

<comment><USER5> If you liked this, there's a long form book about another family in a similar situation.

Edit: help if I include the name: "All God's Children", by Fox Butterfield

<comment><USER4> Thanks! I'll check it out.

</comment></comment></comment><comment><USER6> I've been reading a lot of articles lately, which describe in detail the sympathetic pasts and life stories of convicted criminals. Personally, I would love to see a balance where victims of crime have their life stories written about and sympathetically depicted as well.

In this article for example, the Sheldon family story was told in great detail, but Yu, a university student who was killed while walking down the street, for no fault of his own, barely got a paragraph or two. I wonder how many of those arguing for more lenient sentencing laws would change their minds, if they were exposed in detail to all the lives that were ruined or ended by the actions of criminals.

</comment><comment><USER7> Great article. He was rightfully charged with manslaughter, but it's hard to see the full picture sometimes (what lead up to this point, what are his life circumstances). Empathy is important, even towards criminals who do awful things. If we can better understand the problem, we can do better at preventing similar incidents in the future.

</comment><comment><USER8> Buzzfeed seems to be looking to better themselves. I hope they continue publishing pieces like this.

</comment><comment><USER9> This article is really good.

</comment><comment><USER10> Top 3 Sheldons that will blow your mind!</comment>


Its that time of the year again. New Year Resolutions? - <AUTHOR>

Learn Lisp and implement a sufficiently big project using it. (Its been the same for the past 3 years now) <comment><USER2> In that case, see my comment in the goals thread and linked article - <http://news.ycombinator.com/item?id=1013899></comment>


The median length of time between nursing home entrance and death is 5 months - <AUTHOR> http://europepmc.org/backend/ptpmcrender.fcgi?accid=PMC2945440&blobtype=pdf <comment><AUTHOR> A quicker TLDR to try and summarize methodology and findings:

> We examined a consecutive sample of HRS subjects who died between 1992 and > 2006. HRS subjects are a representative sample of Americans over the age of > 50 who were living in the community at the time of their initial interview. > The HRS is conducted by the Institute for Social Research at the University > of Michigan and is sponsored by the National Institute on Aging. The HRS is > designed to study transitions in health and wealth among older adults.

> Two thousand three hundred and one subjects primarily resided in a nursing > home at the end of life. We excluded subjects with missing information > regarding their exact year of placement in nursing home (n=68), age at time > of placement in nursing home (n=27), not having completed a core interview > within two years of their placement in nursing home (n=203), not having the > exact date of death (n=71), having reported nursing home placement after the > date of death (n=54) and having an irreconcilable nursing home admission > date (i.e. subjects reported as being interviewed in the community after the > nursing home admission date) (n=67). Our final sample was comprised of 1,817 > decedents.

> Over the years 1992 to 2006, 8,433 HRS participants died, and 27.3% of these > decedents resided in a nursing home at the time of death. While subjects > resided in a nursing home at the end of life, their physical location of > death was distinct for many. Of the 1,817 subjects in our study sample who > resided in a nursing home at the end of life, next-of-kin stated that the > physical location of death was the nursing home for 70.4% of subjects, the > hospital for 22.6%, in-inpatient hospice facilities for 3.5%, home for 0.4%, > and 3.1% died elsewhere. The majority of decedents were Non-Hispanic White > (81.5%), female (59.1%), and had a mean age of 83.3 (SD 9.0) years at the > time of admission to a nursing home.

> The mean length of stay among decedents was 13.7 months; however, this was > explained by a relatively small number of subjects with long lengths of > stay. The median length of stay was only 5 months (IQR 1-20). The majority > of residents had short lengths of stay, 65% percent of decedents had lengths > of stay of less than one year, and over 53% died within 6 months of > admission.

> We do not know the reason for admission to the nursing home, and some > patients may have been admitted specifically for end-of-life care. We > excluded patients admitted to residential hospice facilities, although some > residential hospice facilities may have been located within nursing homes.</comment>


Paul Graham speaking at Stanford on Wednesday (4/11/07 - 5:30PM) - <AUTHOR> http://ases.stanford.edu/summit_2007_graham.shtml <comment><USER2> Is "Understanding Investors" going to be a based off a brand new essay, Paul? Sounds interesting -- I think I might go check this one out.

Max Levchin is also going to be presenting in the morning; see the schedule: <http://ases.stanford.edu/summit_2007_schedule.shtml>

</comment><comment><USER3> Can some random person just sign up and go or do you need to be connected to the event somehow?

<comment><USER4> Anyone can go. My talk's probably going to be lousy, though, because I've spent the last several days doing nothing but reading applications.

</comment></comment><comment><USER5> Why do all the good things happen only at Stanford? Hopefully someone will make a bootleg video of this and spread the wisdom through torrents.

<comment><USER6> Don't worry, I'm planning to podcast it and post it here sometime that night ;)

</comment></comment><comment><AUTHOR> their RSVP page: <http://ases.stanford.edu/speaker_eventrsvp.shtml>

</comment><comment><USER7> Awesome, thanks for posting this. :)

</comment><comment><USER8> Video, please?

<comment><USER2> Hang on, it hasn't happened yet. :)</comment></comment>


Turkey's Twitter ban has been overturned - <AUTHOR> http://www.engadget.com/2014/03/26/turkey-court-refuses-to-uphold-erdogan-twitter-ban/

<comment><USER2> How about the Google DNS ban? I think that ban has way more impact on net neutrality. Banning a website, even though it's blatantly bad, is common practice in a lot of countries. Blocking a DNS is basically forcing people to use specific 'approved' services.

<comment><USER3> The Google DNS ban was lifted in 3 hours.

</comment></comment><comment><USER4> They are just trying to fabricate another reason. I don't think twitter ban will be lifted before local elections.

And according to twitter policy announcement [1], Turkish government asked Twitter to provide some credential information about some Turkish users. Nobody is talking about this in Turkey and government officials still defending their decision about their fabricated reason: impersonation and porn in twitter. It's already against Twitter's TOS, and I don't think it's true.

1: [https://twitter.com/policy/status/448108964480569344](https://twitter.com/policy/status/448108964480569344)

</comment><comment><USER5> Well, it took them a few hours to block the site after they found an excuse. Let's see how long it takes them to restore access after this decision. It's been 7 hours and counting.

</comment><comment><USER6> As a hard line atheist:

After the bloody history with kings who argued their position "came from God" and now islamist politicians in Turkey/Egypt, how about humanity get the message that "Being on a mission from God" is not necessarily a good thing outside of "Blues Brothers"?

Sorry for going OT, I am probably as fanatic on this subject as Erdogan himself. :-)

Edit: English grammar.

<comment><USER7> I'm a christian, and I'll take your statement a step further. Any ruler who publicly claims the support of "god" to justify their actions has given up the right to claim the support of the people. _Especially_ if those actions serve to increase that ruler's power.

<comment><USER6> Agreed. My formulation also included people that "know" what god's will is.

</comment></comment></comment><comment><USER8> Maybe they released the old saying, "If there's a will, there's a way."

People were going to bypass the blocks one way or another. Good to see the Turks are "restoring" this...whatever that means and however long it takes.

</comment><comment><USER9> Twitter blocked in just a few hours but not restored since morning. (it's 15:00 in Turkey)

</comment><comment><USER10> It's reported that Twitter themselves are censoring some accounts for Turkish IPs.

<comment><USER11> Indeed : [https://twitter.com/RayMalifalitiko/status/44882213645584793...](https://twitter.com/RayMalifalitiko/status/448822136455847936/photo/1/large)

<comment><USER12> nice: "@oyyokhirsiza's account has been withheld in: Turkey". It's not censorship, it's just withholding</comment></comment></comment>