Urban Airship: Postgres to NoSQL and back - <AUTHOR> http://wiki.postgresql.org/images/7/7f/Adam-lowry-postgresopen2011.pdf <comment><USER2> Maybe this is because of my Ops background, as I was a sysadmin for close to a decade, but I don't understand the aversion to buying actual hardware.

When you're starting out, you go with something on the cloud -- EC2, Rackspace, Softlayer, Linode, or whatever cheeses your hamburger.

That's just common sense. There's no reason for a fresh web or mobile startup to have tens of thousands of dollars of server inventory, bought for the anticipated rush of people that want to get in on the 'social synchronized tuba-thumping' craze.

But when you hit the point where you're spending $10k a month on cloud hosting _and_ your team is having to put out performance fires thanks to the shared hardware, why not just buy servers and throw them in a datacenter?

Amortized over a year, you'll spend a lot less on capital (network and hardware), and a lot more on labor, as you'll need at least one more team member to handle the on-call rotation load.

Apologies if this is rambling -- it's about 7:30am here, and I'm a bit low on sleep. :)

<comment><USER3> Maybe datacenter companies could offer bare-metal cloud hosting. They rent out the servers and keep the hardware physically working, your devops team handles all the software aspects.

<comment><USER4> Softlayer offers this as well. You can spec out a box however you want up to pretty exotic configurations, get billed monthly, and can SSH into them in 3-4 hours.

It is expensive, but not as expensive as something like heroku or aws for equivalent cores/RAM, and you get real hard drives.

<comment><USER5> We are using Softlayer and we found that on our server we were not bottlenecked by memory, CPU or disk I/O. However, we were bottlenecked by our bandwidth which sat around 5 MB/s. Perhaps we could upgrade to a higher- throughput network card or something (I don't maintain our servers), but we ended up just offloading static files to a CDN instead.

<comment><USER4> That does seem odd. I'd file a ticket and see what they have to say about it.

In certain datacenters (think might be dallas05?) you can even get a 10Gbps uplink brought to a box so they have pretty big pipes available.

<comment><USER6> Where can I get dedicated or cloud servers with one or multiple 10GbE NICs?

<comment><USER4> I just poked around with some of our servers to verify.

In Softlayer's Dallas facility (selected when ordering a server) 10 Gbps is an option.

The maximum amount of bandwidth available is 20TB.

Sometimes you can get more options by contacting their sales people directly instead of using the shopping cart.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> This is a good presentation. It is a good demonstration that there is no magic bullet for database scalability problems.

I love NoSQL as much as the next person, but turning straight to NoSQL when you are faced with scalability problems in a more conventional relational database is always going to be a mistake. Before you translate everything over to a NoSQL db, try dropping the ORM (or find another one), looking at your table structures, or tuning your indexes. If you do this, then there is a strong possibility that you will save yourself some time, energy, and effort.

It pays to think deeply about your issue. NoSQL should be another tool in the toolkit, and not the hammer to be used to drive all of your database issues into the wall.

When you turn to a NoSQL solution, you should not be doing so with a mind to find a "magic bullet". You should be doing so because you have thought deeply about the problem and have found that a NoSQL solution answers a specific need.

If everyone could use NoSQL when it is appropriate to do so, then there will be less "horror stories" and more illustrations of valid use cases than what we have today.

<comment><USER8> _try dropping the ORM (or find another one)_

Dropping or replacing the ORM is a big undertaking in anything but small toy projects.

Better advice is to run a profiler and crank up the ORM's verbosity in order to determine the extent of overhead imposed by the ORM, and something like pg_stat_statement and EXPLAIN ANALYZE (in the case of Postgres) to find slow statements and see why they are slow. This will give you a much better idea of where time is spent, to what extent things can be optimized, and whether the ORM is to blame for any performance issues.

</comment></comment><comment><USER9> One month of EC2 XL with 15GB of memory is roughly $500 a month.

A 90 GB database fits in a single $150 SSD. You can get 1 TB of SSD storage for $3,000.

"One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor. This is also the equivalent to an early-2006 1.7 GHz Xeon processor referenced in our original documentation. " [1]

You can get a 6 core AMD Phenom II which runs at 3.2Ghz. for $180.

16GB of RAM will set you back $400.

From the sounds of what they went through, spending $10k on decent hardware might have saved them a man year or two of developer time.

Granted that's not nearly as fun or sexy as trying to use MongoDB, Cassandra or HBase in production. And, saying that you're going to use actual hardware is soooo old school.

ref: [1] [http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_...](http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it)

<comment><USER10> You are very right; I should have tried to have us move to physical hardware before we did. Definitely one of the things that I would have done differently, in hindsight.

The larger EC2 instances (especially for always-on systems like primary databases) do get quite a bit cheaper with the 1-year reserved instance reservations, so if you are on EC2 be sure to get those as soon as you're at a somewhat stable point.

<comment><USER11> I think the introduction of the SSD is causing site owners to "leave the cloud". I haven't seen any cloud hosting that lets you choose SSD-based disks yet, probably because it's low demand and high expense. The throughput of a single sata3 SSD blows away any kind of raid setup you can accomplish on EC2.

Also, I don't think you even need $10k in hardware. Sounds like you could do just fine with a $3k 1U server. $3k can still get you 16 cores w/ 32GB memory and SSD drives.

<comment><USER12> SSDs make a _stunning_ difference in performance for large, complex joins that won't fit in RAM. However, I imagine that in 5-10 years time 'the cloud' will use them too; even if only as part of a larger storage pool.

</comment></comment></comment></comment><comment><USER13> I used PGSQL on such a performance-intensive project as well as nothing came close while retaining reliability. Had PGSQL devs help me to fine tune some aspects, they're extremely helpful and knowledgeable. And that is NOT something I have found even in commercial companies. I mean by that: i've never had such good help for highly technical issues, ever. And this one was free and fast.

</comment><comment><USER14> I wouldn't be so quick to throw the baby out with the bathwater. They just made the wrong choices--Riak is fantastic. Having said that, it's incredibly easy to make the wrong choices right now:

1\. The "NoSQL" field is generally immature and filled with land mines, so a random selection favors a bad outcome

2\. The mass-appeal rank ordered reputation of these projects is not in line with their actual quality/robustness, especially when it comes to very high- demand situations (node counts are high enough to render node failures common, write loads are heavy, machines are within 50% of their I/O limit).

So, I wouldn't let these reports discourage one too much about the possibilities of NoSQL, especially the scaling possibilities of fully distributed databases. It really can be a game changer with a solid implementation. Riak is one.

(Note: "wrong decision" is not meant to be judgmental, we (bu.mp) are ourselves just recovering from a similar "wrong decision"... VERY similar. :-)

Agree completely re: bare metal. If you have scaling problems you should have a very good reason not to be doing dedicated hosting on bare metal IME.

<comment><USER15> "It really can be a game changer with a solid implementation. Riak is one."

Can you write up a case study? What did Riak do right where the others failed? What are the downsides of it, and why are those downsides more tolerable than the downsides of other systems?

<comment><USER16> Seconded!

</comment></comment></comment><comment><USER17> For a notification-heavy app, I'm surprised at the lack of consideration given to Queue/Topic Messaging servers. 5B notifications/month isn't quite 2k/s, which really isn't all that onerous. Routing within messaging apps would seem like a natural fit.

Or perhaps I'm missing something?

<comment><USER10> We make heavy use of queue and internal messaging systems. This discussion was solely about the data storage layers, which always causes us more headaches than the messaging/queuing systems.

<comment><USER18> You should check out redis if you haven't yet.

It's ideal for the counter use-case and with bit of smart thinking you can fit a surprising amount of data into a 32G or 64G machine (and then you can always shard).

</comment></comment></comment><comment><USER19> Just a quick datapoint for you on EC2 vs SSD... we tried restoring a 30GB pgsql database to a Large EC2 with RAID0'd EBS stores for testing purposes. It took ~4 days.

I tried doing the same thing on a Macbook Pro i7/SSD and it took ~1 hour.

EBS disk performance is reliable, but miserable.

<comment><USER18> _EBS disk performance is reliable, but miserable._

EBS performance is unreliable and miserable. There, fixed that for you.

However, EBS also has a couple things going for it. Namely: easy snapshots, fast/seamless provisioning (need another 1T?), mobility (need that volume on another instance?) and quite a fair price point when you take all that into account.

<comment><USER20> easy snapshots, fast/seamless provisioning (need another 1T?)

All of which are doable with LVM.

<comment><USER18> Sure, within the boundaries of your disk array. Assuming a non-trivial size you'll need 1-2 fulltime people to run and scale that array, plus a nice chunk of investment into hardware.

EBS is just a mouse-click.

<comment><USER21> If you are so unconcerned with your storage that you're just mouse clicking around EBS setups, you don't' need 1-2 full-time people to run an array.

<comment><USER18> Sounds like you've never dealt with storage at scale.

It's far from "rack and forget" once you've grow beyond 2 racks.

<comment><USER21> I think you read the opposite of what I was trying to say.

I have dealt with storage 'at scale' enough to require a full team of storage people.

Some guy clicking buttons on the Amazon website is usually not a very effective way of dealing with storage 'at scale'.

<comment><USER18> Yes, sorry for reading you backwards.

However I maintain that EBS has it's place in a wide range of applications where raw I/O performance isn't the main concern (e.g. archival).

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER22> Not to mention they're using one of those nasty "O-R-M"s we all were told were an antipattern yesterday.

<comment><USER23> SQLAlchemy (and its excellent documentation) is the reason I switched to using an ORM.

(Now that I am in Ruby land though, I am a little sad that arel/ActiveRecord/DataMapper do not seem as on their game.)

<comment><USER24> You should try Sequel. It does things AR and DM don't (CPK+FK for example), is generally faster, and gives you the flexibility of working with Models or raw data-sets depending on what the situation calls for.

I've used and written a number of O/RMs and Sequel is definitely the best.

<comment><USER23> This is really helpful. I will definitely check it out.

Also, I'm really amused that apparently the creator of DataMapper is (a) selling me on using Sequel and (b) the testimonial on their front page. (Not that there's anything wrong with that.)

<comment><USER25> Sequel and DataMapper have different purposes I think. Sequel is a lot more transparent in the translation between the ORM and SQL and allows a more powerful control of SQL statements. DataMapper allows some very powerful controls on the ORM and Ruby syntax side.

</comment></comment></comment></comment></comment><comment><USER26> One of the easier things to do is pull out all the counters and sessions and put them in Redis or Riak. No use in tying up your database doing things it is not well suited for anyway.

<comment><USER27> Yes, Redis rarely keeps me up at night, besides making me worry about what I'll do when I run out of RAM :)

<comment><USER28> In a "just counters" scenario you can't imagine how much data a single Redis server can hold! It is sensible to understand when it is a good idea to use a database that holds data in memory, as there are obvious limits (that are are trying to break in a comfortable way with Redis Cluster instead of forcing client-side sharding), but on the other hand it is good to do some math before being worried.

Example: Redis can hold 1 million counters, every counter stored in a different key, for every 200 MB of memory. This means 5 million counters per gigabyte. Clearly you can have a lot of counters. But you can make this figure a lot higher if you aggregate counters using hashes: with this trick when applicable the above figure will be five times bigger.

<comment><USER29> Can you hint how many distinct counter increments can i expect to have per second with Redis?

<comment><USER28> 100k increments per second in an entry level Linux box. 50k increments in a decent EC2 instance. 150k - 180k increments per second in a fast modern Linux box.

</comment></comment></comment></comment></comment><comment><USER30> This was really interesting, thank you.

If anyone reads this, note that in Opera at least, clicking the builtin pdf reader's down button skips some very informative notes that Adam left in the presentation. Press page down instead.

<comment><USER31> Oh, wow, I missed those comments completely since I was using the arrow keys to navigate in Chrome. Thanks for pointing this out.

</comment></comment><comment><USER10> Thanks for the kind words, everyone. When I was writing the talk I was worried it'd be too tied to our/my experience, but at least people found it a little entertaining. The video should be posted soon; the Q&A was interesting and brought up some pieces that I glossed over during writing.

</comment><comment><USER32> I was hoping that Urban Airship actually made urban airships.

<comment><USER33> Good to live in an age where this assumption is perfectly reasonable. Here's a link to salve your disappointment (FPV remote control planes, music soundtrack):

<http://www.youtube.com/watch?v=MUGvlMDRC98>

</comment></comment><comment><USER34> It's fascinating to read another company having evaluated NoSQL and come to an identical conclusion as we did internally.

I love the idea of NoSQL, but Cassandra was horrible (just look their source code or Thrift) and Mongo lost data. I guess 40 years of relational databases isn't so easy to replace.

<comment><USER35> To be clear, we are still heavy users of Cassandra. We try and objectively match the tool to the problem and in some cases, PostgreSQL was a better fit but not all. In some cases HBase was a better fit.

Also to be clear, scaling is difficult, no matter what the tool. We've had problems with Cassandra, HBase and PostgreSQL (most recently Friday), no storage option is as good as we would like under stress.

<comment><USER10> I definitely could have been more clear on that. Cassandra has so many great properties, and when we made the decision to use Postgres for the large dataset under question was shortly after 0.7 was released, and it took a while to get more stable.

</comment></comment></comment><comment><USER36> A little bit unrelated, but as of _today_ , what would be the advantages of choosing pgsql over mysql for someone who already has decent knowledge of mysql? Is it still a matter of preference (and being able to deal with the mysql license) or did pgsql progress so much over the last 3-4 years that it became superior? What are the pros over mysql?

I'd consider switching to pgsql for a future project but throwing my mysql expertise out the window for nothing seems like a bit waste of time.

<comment><USER37> This has a pretty decent, if dated list: <http://sql- info.de/mysql/gotchas.html>

Few of the items on that list bit me personally.

<comment><USER36> I'm more interested in knowing what PgSQL does better than MySQL today, in 2011. MySQL definitely has a few caveats, my question is mostly, is PgSQL now significantly better than MySQL or they're both more or less the same, with different pros and cons.

<comment><USER38> No, PostgreSQL is way better.

It has a mostly compliant SQL dialect.

It is more compatible with other DBMSs, specially DB2 but also Oracle.

It implements a host of procedural programming languages and extensions, including types and operators.

It is way faster, and scales way more.

It is totally free (no paying Oracle to do InnoDB hot backups).

It is way more consistent.

It does geographic data second to none.

Its development is totally open and way faster. Reading the PostgreSQL TODO wiki page is a joy.

And so on and so on.

</comment></comment></comment></comment><comment><USER39> I can summarise this presentation as:

      * Postgre 8.3 on AWS not good enough
      -> lets try Mongo
      * Mongo performance not good enough
      -> lets try and fix this
      * Oh shit we don't know what we're doing lets go back to Postgre
      * There's a new version 9.0 this might sort us out
      * No sadly Postgres 9.0 is no better
      * Oh wait maybe Amazon performance is to blame
      * Lets try it on real hardware
      * Oh yes that's better

</comment><comment><USER40> If only there are tools for sharding PostgreSQL. Easy sharding is still a major weakness in relational databases.

<comment><USER18> SQLalchemy (the orm that they're using) actually comes with some nice sharding hooks and examples.

I used to complain about the lack of "out-of-the-box" sharding middleware myself, but came to realize that this is really something that belongs in the application layer for all but the most trivial use-cases.

However, there is no reason why full-stack frameworks such as Django and Rails couldn't grow meaningful sharding support.

<comment><USER22> We've got a rudimental hook sure, but it's not what I'd call "easy"...it still needs to be attended to with custom logic anytime you need to cross shard boundaries.

<comment><USER18> I've been there, so can confirm it's not "easy".

But at least it gave me starting point, which is already more than most other ORMs can offer.

</comment></comment></comment></comment><comment><USER41> The last slide mentions automatically failover is still needed. For that, look into DRDB + HeartBeat on Linux. They allow you to build automatically failover cluster easily.

In my past life, I've built high available MySQL cluster using the combo and it worked great. The same technique can be applied to Postgres.

</comment><comment><USER42> From the slides it's not 100% clear, but as Erik from Urban Airship commented here [1], "[they] are still heavy users of Cassandra."

[1] <http://news.ycombinator.com/item?id=3015475>

</comment><comment><USER18> I was a bit disappointed to see the HBase slide blank.

It has some interesting properties and it's rare to find real-world rapports about how it fares outside of Hadoop.

<comment><USER10> Sorry about that -- I couldn't speak as much about our Hbase experience, but I can speak to the toll its taken on our ops team so far, which is very high, with lots of crashes.

<comment><USER35> To clarify, we've had lots of region server crashes, mostly due to our own data model and generally not as a result of any intrinsic fault of HBase. To my knowledge, not many of these (any?) have actually resulted in a total HBase failure. The system generally degrades as it should.

</comment></comment></comment><comment><USER43> How many startup man-hours were spent tracking that Ubuntu bug down on Cassandra instances? I've heard several nightmarish stories.

</comment><comment><USER44> One thing to note is that dedicated server suppliers will foot the capital cost of servers for you, so there is no need to find the cash for the server up front. 100tb.com do this (I've not used them, know people who do) for example.

</comment><comment><USER45> What I really didn't understand is why they even decided to switch to NoSQL in the first place. I mean Urban Airship seems like the perfect example of data is completely relational and schema-driven.

</comment><comment><USER20> Direct link to pdf.

[http://wiki.postgresql.org/images/7/7f/Adam-lowry- postgresop...](http://wiki.postgresql.org/images/7/7f/Adam-lowry- postgresopen2011.pdf)

</comment><comment><USER46> Real hardware was the answer?

Really guys? :)

My recommendation: switch to SoftLayer. Save thousands a month, use real hardware.

<comment><USER47> I love SoftLayer (I use them), but I wouldn't say they're the cheapest by any stretch of the imagination.

</comment></comment><comment><USER48> tl;dr: "PostgreSQL is still my weapon of choice — it doesn't keep us up at night."

I agree with the guy. PostgreSQL (and SQL server ironically) have kept me up less than any other piece of software out there.

I'm watching a turd occuring with MongoDB at the minute. Every time something breaks, I hear the team saying "it'll be fixed in the next version" or "feature X works around it for now". Grr. Too much marketing, not enough reliability study.

<comment><USER49> I believe this was one of the talks at Postgres Open - I came across the slides from most of the others earlier at <http://wiki.postgresql.org/wiki/Postgres_Open_Talks_2011>, and they have some good tips on scaling with Postgres.

There was one interesting tidbit in one of the Heroku presentations : they apparently run over 150,000 databases ( <https://wal-e- pgopen2011.herokuapp.com/#6> ) which is incredible.

<comment><USER10> The Heroku guys are doing some seriously awesome things with Postgres. I can't wait to see where they go with it; the whole community will benefit, I'm sure.

</comment></comment></comment><comment><USER50> I think I'm missing something from the presentation slides. Why would an API call result in 5 Millions writes? I think he is trying to give a scenario of an API call that creates a rich push for 5M devices. Even queuing the writes is not going to solve the problem.

Fundamentally, I wouldn't have designed the messaging model that would require a write for each client device for each message.

<comment><USER51> Perhaps they have to record if/when each client got the message. How would one record that without one write per client?

<comment><USER52> It seemed odd to me too that a single API call could cause that much work. Granted, I have no idea how the actual app works. But one way to do this is to have clients belong to groups, then record if a group got a message (few writes).

Then for each client pull the groups the client belongs to and look for new messages (possibly more reads than storing the messages on a per-client basis).

It's an extra database table and layer of indirection, but that's one way.

</comment></comment></comment><comment><USER53> "I laughed, I cried, I experienced every possible human emotion! It was better than 'Cats'!"</comment>


Cubes - Python Lightweight OLAP - 0.9 released - <AUTHOR> http://blog.databrewery.org/post/23049114551/

<comment><AUTHOR> Author here. Just a small explanation: target audience are mostly those who would like to add OLAP to their apps with simple multi-dimensional reporting needs (see trivial hello-world example [1]). This release brings new star/snowflake aggregation browser which can be (roughly) considered to your analytical data as ORM framework is to your transactional data - some magic happens between your code and the database that makes your life easier.

Visualisation is not yet included and will be added in modular form of "presenters" [2]. If anyone with web/JS skills would like to help, he is welcome, as my skills in this area is little bit weak. There is unmaintained and unfinished prototype of JavaScript front-end as well [3]. Flask based example of simple dimension browsing is included in the source [4].

If you need any help with model creation, you encounter any issues, have a suggestion let me know.

[1] [https://github.com/Stiivi/cubes/blob/master/examples/hello_w...](https://github.com/Stiivi/cubes/blob/master/examples/hello_world/aggregate.py)

[2] <https://github.com/Stiivi/cubes/issues/53>

[3] <https://github.com/stiivi/cubes.js>

[4] [https://github.com/Stiivi/cubes/tree/master/examples/sandbox...](https://github.com/Stiivi/cubes/tree/master/examples/sandbox/flask_dimension_browser)

</comment><comment><USER2> Interesting idea. Ofcourse, OLAP isn't just about the underlying cubes and dimensions, in the end it's about how the information is being presented to the user. There's a reason why Microsoft is so good at data warehousing, because their product is one of the best, and also extremely feature-rich from the "front end" to the back end.

<comment><USER3> well from my experience, working with OLAP Cubes, Microsoft is normally used when cost saving is important, its considered cheaper than competition. And the development tools are so easy to use, non-programmers can be trained to create and customize cubes.

On the front end, they only have excel, which is good, really good, but other tools have much simpler and in my opinion better clients, for example Cognos and BO

<comment><USER4> You might want to look at OSS options like Pentaho, Jasper and Palo.

Palo (a competitor to Cognos TM1), in particular, is really nice. The other two leave a little to be desired on the UI side.

</comment></comment></comment><comment><USER4> I haven't encountered a lot of BI nerds who are Pythonistas, but this is a great contribution nonetheless.

The more open source tools we can get that can begin displacing products like Cognos and BO, the better.

<comment><USER5> > BI nerds who are Pythonistas

One here. Very interested to see this project evolve.

<comment><AUTHOR> Thank you, both. If you have any questions or suggestions, drop me a line or much better: discuss it on the cubes-discuss google group [1]. I would appreciate feedback from more experienced BI people than I am.

[1] [https://groups.google.com/forum/?fromgroups#!forum/cubes- dis...](https://groups.google.com/forum/?fromgroups#!forum/cubes-discuss)

</comment></comment></comment><comment><USER6> Is this going to integrate with the broader python ecosystem? I'm specifically wondering about packages like pyTables, pandas and larry.

<comment><AUTHOR> I would love to and actually was looking at pyTables and little bit at pandas. I am coming from RDBMS-based DWH world mostly, so it is a bit new area to me, howerver from what I've learned so far, there is huge potential. Unfortunately, currently I have no resources to do that, even I would like to see that more than RDBS backends (as there are already plenty of tools for that). Is there any way how this possibility can be supported? I would not mind working on a project for someone that would require OLAP layer on top of something "strange" and that there will be (open-source) by-product in form of a backend...

[edit] p.s.: I am the only developer so far, and I am open for collaboration.</comment></comment>


Intel's new processors have a remote kill switch (Anti-Theft 3.0) - <AUTHOR> http://isc.sans.edu/diary.html?storyid=10111

<comment><USER2> Significant discussion from an earlier submission of the same story, but from a different source:

<http://news.ycombinator.com/item?id=2018351>

In fact, this submission is pointing to an article that's simply quoting the story submitted here:

<http://news.ycombinator.com/item?id=2020625></comment>


High-res Nexus 5 official Press shot - <AUTHOR> http://thetechblock.com/full-size-nexus-5-press-shot/

<comment><USER2> It was in the Play store earlier today, $349 for the 16GB. The second it's released I'm all over it!

[http://www.androidpolice.com/2013/10/17/breaking- nexus-5-par...](http://www.androidpolice.com/2013/10/17/breaking- nexus-5-partially-visible-in-the-play-store-ahead-of-release/)

<comment><AUTHOR> I have an iPhone 5, but I totally am going to buy this and see what it feels like to have Android as a primary device.

</comment></comment><comment><USER3> Probably going to move off iOS to this but it depends on the size. If it is just as big as it's predecessor, maybe not.

<comment><USER4> [http://www.phonearena.com/news/Nexus-5-round-up-5-awesome- ne...](http://www.phonearena.com/news/Nexus-5-round-up-5-awesome-new-features- release-date-price-specs-and-all-we-know-so-far_id47236)

5 inch screen, by the looks.

<comment><USER5> Yep, 4.97 in a slightly smaller body than the Nexus 4.

<comment><USER6> Engadget [0] which is the source for this psudo-blogspam says:

Nexus 5 listed for $349, [...] 1080p 4.95-inch display, 2.3GHz Snapdragon 800 CPU, LTE and storage capacity that tops out at 32GB

[0] [http://www.engadget.com/2013/10/17/nexus-5-16gb-349-play- sto...](http://www.engadget.com/2013/10/17/nexus-5-16gb-349-play-store/)

</comment></comment></comment></comment><comment><USER7> Looks like they unified Hangouts and SMS. Maybe it'll be more useful now.

</comment><comment><USER8> I have a Galaxy Nexus, skipped the Nexus 4, so am treating this as a signal to upgrade. Although I leading towards the Note 3 at the moment... Will definitely consider the Nexus 5 once it is out though.

Although I only got around to putting Cyanogenmod on my GN a month or 2 ago so it kind of feels like I did just get a new phone...

<comment><USER9> I wish Samsung built a Note without any of the phone features. Like a smaller tablet.

</comment></comment><comment><USER10> No SMS icon but only Hangout. Looks like they merge those two. Good choice.

</comment><comment><USER11> I wish Verizon and Google could get along...

<comment><USER12> Yeah me too. I really think I am going to switch to T-Mobile so I can have this phone. Fortunately, I live near D.C. and don't travel much so I coverage shouldn't be an issue. It's just annoying how Verizon won't let a phone that gets updates straight from Google on its network.

<comment><USER13> Isn't verizon all CMDA? I thought the deal was all the Google phones were GSM. Haven't looked to closely at that recently though, they might all be universal.

<comment><USER14> LTE is based on GSM.

The specs found in FCC docs for what is believed to be this phone suggests that the phone contains radios that support the Sprint network which is CDMA, but does not support the Verizon LTE band.

Perhaps in the not so distant future VoLTE will be common and phones will be, conceptually at least, portal across any network.

</comment></comment></comment></comment><comment><USER15> Also of note is the price: it was briefly listed on the Play Store with the 16GB model going for $349.

<comment><USER5> It's a steal.</comment></comment>


Article on Classful Networks and Ifconfig, with Help from HN - <AUTHOR> A few weeks ago I posted a question, and a couple of folks were very helpful.
https://news.ycombinator.com/item?id=23655176
I wanted to let you know that the article I was working on is posted, with credit for the assist. :)
Thanks!
https://prgmr.com/blog/2020/07/17/classful-networking.html <comment><USER2> Nice writeup.

Fascinating "bug"! Or should I say weird default behavior.

I saw this post on the Lobsters front page and was a bit surprised when I reached the end of the article and saw my user name there :)</comment>


Government Mandates and Programming Languages - <AUTHOR> http://stevesmithblog.com/blog/government-mandates-and-programming-languages/?utm_source=twitterfeed&utm_medium=twitter

<comment><AUTHOR> I'd agree that I'm not stoked about government mandating the usage of programing languages, but for financial data I'm not surprised that they'd try to mandate python. It is my understanding that python is heavily used in investment banking other heavy financial analysis.</comment>


Announcing the Winners of Android Developer Challenge 2 - <AUTHOR> http://android-developers.blogspot.com/2009/11/announcing-winners-of-adc-2.html

<comment><AUTHOR> My app, "Car Locator", nabbed 3rd place in the travel category =)</comment>


Ask HN: How to Hire an Econometrician? - <AUTHOR>

Our startup would like to hire an econometrician.  Resources abound for people who want to hire developers, but there seem to be few venues for hiring mathematicians and statisticians.
We're looking for someone with an statistics or econometrics masters degree, so we are contacting the appropriate departments in certain schools
Beyond contacting schools, can the HN community suggest how we can go about finding an econometrician? <comment><USER2> Try meetups or user's groups for R, SAS, SPSS, data mining, etc. My local R group meetings are really interesting and info-packed.

<comment><AUTHOR> Interesting, great suggestion. I'll look into that. Thank you.</comment></comment>


Krugman, Stross and Martian hive-mind kleptocracy - <AUTHOR> http://krugman.blogs.nytimes.com/2010/12/10/hive-minds-and-kleptocrats?pagemode=print

<comment><USER2> The thing is, Krugman is still pointing out one of the major criticisms against corporations. Their top executives make bad decisions and put their own goals ahead of the company's based on the fact that they're playing with other peoples' money. How many of these types would be willing to do this with their own money?

</comment><comment><USER3> If left to their own devices (read: without bailouts, artificially cheap credit or legal favoritism), corporations would function like any other agents of the free market, subjected to profit and loss.

Corporations are not hive minds in the same sense that any other organization is not. A mind is an attribute of the individual; a corporation is nothing but an organizational framework.

<comment><USER2> What other agents of the free market? Corporations _are_ the free market.

And I think most group psychologists would dispute the idea that groups don't take on hive-mind-like qualities.

<comment><USER4> Individual households, small business, and the government are all actors in the free market that are within the same ballpark as corporations. And any of these is quite capable of forming hiveminds too - as do individual government departments, charitable organizations, political parties, labor unions, professional associations, and internet discussion forums.

</comment></comment></comment><comment><AUTHOR> Krugman has a long history of linking economics and science fiction:

[http://krugman.blogs.nytimes.com/2008/05/04/economic- science...](http://krugman.blogs.nytimes.com/2008/05/04/economic-science- fiction?pagemode=print)

<comment><USER5> Psychohistory never made sense either.

</comment></comment><comment><USER6> The key feature of a corporation besides limited liability is the nearly unmanageable principal-agent relationship between shareholders and managers. It's kind of amazing to me that a structure where shareholders lose to managers so consistently is the structure of choice.

Most corporations should be distinguished from banks, since the banks actually managed to acquire a put option for the bank itself, doubling the leverage of management. The really striking thing is witnessing the monetization of that put although it really shouldn't be too surprising, since virtually the same thing happened in the 80's with the S&L industry.

All the hive mind/group think stuff can and does happen in any organization where leaders need to persuade followers. Not as "new" as you might think.

<comment><USER7> Heh, 'bout time someone pointed out the other key feature of a (public) corporation. Plus I like the idea of banks having a 'put option' on themselves. Glad to see at least some people manage a level-headed assessment of the economic situation now -- e.g. as opposed to others who in light of corporations' success in the arena immediately start a religion around worshipping the idea of a corporation, Y combinator included.

</comment></comment><comment><USER8> Submitted two days ago, and gained no comments and no upvotes: <http://news.ycombinator.com/item?id=1993348>

Timing of submissions and luck are critical in getting noticed. I find that frustrating.

</comment><comment><USER9> Inventing new words, like "kleptocracy" to bitch about The Man is for d-bags, UNLESS you're Paul Erdös. Erdös Krugman is not. I rest my case.

</comment><comment><USER10> Can we please nix this print version linking?</comment>


Saliva based Covid test: results in 45 minutes - <AUTHOR> https://www.colorado.edu/today/2020/07/22/new-covid-19-test-returns-results-45-minutes-without-nasal-swab <comment><USER4> Technically, you are really just waiting for the thermal inertia of the testing device. The reaction kinetics are really quite fast.

During the last SARS outbreak, this sort of thing was proposed: [https://pubs.rsc.org/en/Content/ArticleLanding/2011/AN/c1an1...](https://pubs.rsc.org/en/Content/ArticleLanding/2011/AN/c1an15365j#!divAbstract)

... to do 3 minute tests. I assume grant funding dried up, since who could guess that after 2 outbreaks that we'd have a 3rd?

<comment><USER5> > I assume grant funding dried up, since who could guess that after 2 > outbreaks that we'd have a 3rd?

Yes, that's what I've read. Any US research scientists still working on SARS-1 after 2003 were told "you're on your own."

Kudos to Canada for being as diligent as Taiwan in attacking COVID-19 - <USER2> remember it wiping out Toronto hospital staff in 2003. Check out the Youtube CBC channel for their impressive nation-wide mobilization.

Makes you wonder wtf Chinese virology labs were doing for 20 years with coronavirus if they weren't working on a cure, doesn't it? (There's 40 labs total across China.)

<comment><USER6> _> Makes you wonder wtf Chinese virology labs were doing for 20 years with coronavirus if they weren't working on a cure, doesn't it? (There's 40 labs total across China.)_

Like most things in a command economy, they worked on whatever was a political priority.

I'd hazard to guess it wasn't a priority in the intervening years for the same reason there was little free market incentive to address it: yes, it's bad when it happens, but it doesn't happen that often.

Ergo, they were probably just as starved of resources.

The preexisting US effort was mostly funded through DARPA for the same reasons. [https://www.darpa.mil/our-research](https://www.darpa.mil/our- research) (e.g. ADEPT, P3, Prometheus, PREPARE)

</comment></comment></comment><comment><USER7> Off topic, but related enough that I feel its relevant to mention:

I learned this week that many blood donation and plasma centers [1][2] are doing Coronavirus antibody tests for anyone* that donates right now.

I've heard a lot of people in my social / online circles comment how they wish they knew whether or not they had antibodies. Seems like a great way to find out.

[1] [https://www.redcrossblood.org/donate- blood/dlp/covid-19-anti...](https://www.redcrossblood.org/donate- blood/dlp/covid-19-antibody-testing.html)

[2] [https://www.lifeservebloodcenter.org/donate/recovered- covid-...](https://www.lifeservebloodcenter.org/donate/recovered- covid-19-patient-plasma-donation/)

* Probably not literally anyone, but it sounds like they are giving them out pretty freely.

<comment><USER8> _" I've heard a lot of people in my social / online circles comment how they wish they knew whether or not they had antibodies"_

What people need to realize is that the important questions are not whether you've got antibodies but:

1 - whether you'll sick when you're exposed to the virus again

2 - how sick you'll get the next time around

No one knows the answers to these questions yet, and we also don't know how antibodies bear on the answers to these questions.

It is possible that even with antibodies you'll still get sick if you're exposed to the virus (as we know is the case for some other diseases), and even if antibodies play some role in how sick you'll get it's possible that (again, as is the case with some diseases) repeated exposure to the virus will cause even greater illness than the first time around.

People shouldn't be assuming that having antibodies means they're immune.. we just don't know enough yet to say whether that's true or not.

Even if it is true, there might be a certain level of antibodies that are required to have a certain level of immunity, and we don't know yet what level of antibodies are necessary, nor the kind of antibodies necessary,

Finally, even if antibodies do grant immunity, we don't know how long that immunity would last, nor if such immunity varies from person to person for some not yet known reason.

<comment><USER9> Dengue fever antibodies make the second infection lethal.

<comment><USER10> IIRC, Dengue has 4 different strains. When you get reinfected with the same strain, then you are good. Getting one of the other strains is the problematic case.

</comment></comment></comment></comment><comment><USER11> This reminds me of an excellent video on use of cheap and less sensitive tests on "This Week in Virology".

[https://youtu.be/kDj4Zyq3yOA](https://youtu.be/kDj4Zyq3yOA) (Feel free to skip to around 6:35).

Preprint: [https://www.medrxiv.org/content/10.1101/2020.06.22.20136309v...](https://www.medrxiv.org/content/10.1101/2020.06.22.20136309v2)

Update: the colorado.edu article also links to this preprint when it discusses the "separate computer modelling study".

<comment><USER12> The visual explanation of the ideas and concepts presented there, also mentioned in the later podcast:

Medcram: "Coronavirus Pandemic Update 98: At Home COVID-19 Testing - <USER3> Possible Breakthrough"

[https://www.youtube.com/watch?v=h7Sv_pS8MgQ](https://www.youtube.com/watch?v=h7Sv_pS8MgQ)

Also, regarding the Ct value, from:

"Interpreting the COVID-19 Test Results: a Guide for Physiatrists"

[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7268832/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7268832/)

In PCR tests "the diagnosis is made based on the threshold cycle (Ct) value. Ct is defined as the cycle number when the sample fluorescence exceeds a chosen threshold above the calculated background fluorescence.5 In other words, the lower the Ct value of a specific gene, the more the gene exists in the sample. However, the problem with a Ct-based diagnosis is that there is no absolute or constant Ct cut-off value, and Ct cut-off values are different for each diagnostic reagent even for the same gene."

</comment></comment><comment><USER13> The study "Saliva is more sensitive for SARS-CoV-2 detection in COVID-19 patients than nasopharyngeal swabs" [0] was published in April, so I'm glad some additional steps have been taken to test it.

[0]: [https://www.medrxiv.org/content/10.1101/2020.04.16.20067835v...](https://www.medrxiv.org/content/10.1101/2020.04.16.20067835v1)

</comment><comment><USER14> This seems like a really good idea but I worry about regulatory approval. Tests like these will tend to catch any Covid-19 cases in the window where you have a real chance of infecting someone from maybe 2 days pre-symptom onset to a few days post onset. And that's really all you need from an epidemiological standpoint. But from a medical standpoint if someone comes into your ER a week post symptom onset you want to know if they've got Covid-19 whether or not they're still infected, from a medical standpoint. And in that use case this is less likely to produce the correct answer than the already existing PCR test and the time lag probably isn't that important. It would be reasonable to allow it anyways but the FDA's rules are set up for the benefit of medical uses rather than epidemiological uses and I wouldn't be optimistic about it getting allowed baring pressure from above.

Also

>One researcher spiked 30 out of 60 saliva samples with inactivated SARS-CoV-2 in the lab. Then they shuffled the samples and gave them to another scientist to test with the RT-LAMP technology.

I wonder if that required a BSL-3 facility or not. Generally you can do medical work on this virus with any sort of facility or laboratory you have on hand but you're only allowed to do science involving the virus if you have a special laboratory with negative pressure and special failsafes and so on.

<comment><USER15> Spiking inactivated virus does not require BSL-3. You only need BSL-3 for isolating and growing live virus. So the vendors that create inactivated virus are doing so in BSL-3 facilities, but once it's inactivated you can work with it at much lower stringency. Seems like BSL-2 is standard.

</comment></comment><comment><USER16> This test relies on RT-LAMP to generate copies of the RNA instead of PCR which the "regular" tests use. It's pretty sensitive, but not as sensitive as state of the art PCR[1]. It's also way faster. No magic, just a slightly different technique with different trade offs.

[1] [https://www.frontiersin.org/articles/10.3389/fmicb.2018.0208...](https://www.frontiersin.org/articles/10.3389/fmicb.2018.02089/full)

</comment><comment><USER17> They've been using a saliva test here in Australia for several weeks for children and others who struggle with the nasal swab. Accuracy is apparently only 87% though.

[https://www.news.com.au/national/saliva-test-with-87-per- cen...](https://www.news.com.au/national/saliva-test-with-87-per-cent-success- rate-to-bolster-victorian-war-on- covid19/video/03cb674d50ace2fc30ef1a2a4a4a14fb)

<comment><USER18> 87% sounds pretty good to me if it encourages a lot more people to get tested. I do wonder how many are avoiding a test because the _know_ from others' reports that it's most uncomfortable. Think about the number of people who are afraid of simple medical procedures such as injections and apply that number to those who don't wish to get "brain scraped". It's a problem.

</comment></comment><comment><USER19> I really do not understand what the hold up is. In California I know over ten people whose results have taken more than seven or eight days to come back. This is in the past two weeks. I am still waiting for my results from Sunday six days ago. It really is inexcusable how mismanaged this epidemic is in the United States.

<comment><USER20> I live in Florida and was exposed to some respiratory illness with symptoms consistent with Covid-19. As soon as I had symptoms I immediately did what I could to get tested and specifically went for the fastest PCR test I could find (I'm in between primary care providers, couldn't find a way to even try to get a rapid test in my area). My insurance provider lists the only testing location covered as an undisclosed drive through site located somewhere in a city a good hour and a half drive away from me, with the second closest location being a 2 hour drive. The county had a free testing event scheduled for 2 days later but only had 400 tests, was first come first serve, and explicitly stated that you couldn't line up before they started testing as they literally weren't opening the gate to let traffic off of the street before then. The county also listed a 10 day wait before test results would even be available which is just flat out useless. Luckily the day before the county testing event my wife managed to find free testing offered by Quest in a somewhat nearby city that advertised that results would be available in 3-5 days instead of the 10 that the county claimed. I went there, got the nasal swab done, filled out a bunch of forms with clipboards and pens that were being reused with no sanitization between people, and signed up for Quest's online accounts to get the results.

Today is literally the first day that I'm free from quarantine. Not because I got my test results back, but because the latency in testing has gotten so drawn out that even if that test eventually comes back positive, I already meet the CDC guidelines for ending quarantine. That test was completely useless for me and only serves to add some data to the statistics around the pandemic. Right now, if you want to get tested and you're not a high priority case (i.e. hospitalized), at least in my area, you can't even do that for all practical purposes. It would be arguably better if they just randomly tested only 50% and had a more reasonable backlog than testing 100% with a backlog so long that all but the high priority tests are useless by the time the results are available.

We've had literally months of time to ramp up testing nationwide. This pandemic is far from over and it's already killed 145,000 Americans. This is inexcusable incompetence by the legislative and executive branch.

<comment><USER21> I am not sure the delay is due to the backlog only .

There are only few locations that actually run the assays the logistics of getting the sample to a location is significant. Not to mention regent and other resource shortages mean some locations are off and can't process anyone . Your sample gets moved somewhere else

Also testing is many times done like binary searches. Some 96 samples are combined and tested , if it is are negative no further tests required , if they come positive breakdown and so on . While even in the worst case you may will not need 96 separate tests, the steps are sequential, tests will take more time if more people on average become positive.

</comment></comment></comment><comment><USER22> Israel is 30 seconds.

[https://hamodia.com/2020/06/30/israeli-company-introduces- co...](https://hamodia.com/2020/06/30/israeli-company-introduces-coronavirus- test-results-30-seconds/)

<comment><USER23> How to learn more about this sensor?

</comment></comment><comment><USER24> Emirates had something about 10 minute test back in April before boarding. I never figured out how they did that.

[https://www.emirates.com/media-centre/emirates-becomes- first...](https://www.emirates.com/media-centre/emirates-becomes-first- airline-to-conduct-on-site-rapid-covid-19-tests-for-passengers/)

<comment><USER25> Not sure how they can do that but my company's device can do it in 20: [https://www.forbes.com/sites/johncumbers/2020/07/01/a-new-20...](https://www.forbes.com/sites/johncumbers/2020/07/01/a-new-20-minute- covid-19-test-will-use-crispr-gene-editing-technology-to-deliver-results-at- the-doctors-office-supermarket-or-workplace/)

</comment></comment><comment><USER26> "Because no swabs are required, and no fancy equipment is needed, the tests are less vulnerable to backlogs"

That, sadly, is nonsense unless you have thousands of technicians available to process the tests. What we have here is a classic latency vs throughput problem. 45 minutes to an answer is a lot better than 24-48 hours which is about the best achievable with the PCR tests. But even if only a third of the 45 minutes is a tech messing with "pipettes, a heating source and an enzyme mixture" and the rest is waiting for reactions at best one tech could do perhaps 30 a day. On the other hand the "fancy equipment" in PCR labs can do thousands of such tests in one go (UW Virology here in Seattle is doing over 5000/day[1]).

[1] [http://depts.washington.edu/labmed/covid19/](http://depts.washington.edu/labmed/covid19/)

<comment><USER27> PCR test can be done in a few hours at most. It's mostly the whole logistical problem that takes time.

<comment><USER26> Yes, the time's not the test, it's getting the sample from where its collected to the lab (e.g. a doctors office that may only get collected from once a day), assembling a batch, running it, collating the results and sending them back out. Any batch-oriented, industrial-scale testing operation will suffer the same delays regardless of the actual process.

</comment></comment></comment><comment><USER28> I competed against this team in this year's CU New Venture Challenge. They got first, we ([https://docforge.dev](https://docforge.dev)) got second.

They are doing some amazing work with this and SickStick.

</comment><comment><USER23> This is still not a home test. Keep working..

<comment><USER29> This company claims to have a home testing kit:

[https://nanologix.com/](https://nanologix.com/)

[https://youtu.be/HVcSchmwXgc](https://youtu.be/HVcSchmwXgc)

</comment></comment><comment> the-dude n=60, deactivated virus, no humans.

<comment><USER30> less space than a nomad

<comment> yjftsjthsd-h Yes, and we'll have fusion reactors, HIV vaccines, and a cure to cancer any year now. Science is less subjective than consumer electronics.

</comment></comment></comment><comment><USER31> Antibody tests give similar results in 10min with a drop of blood.

<comment><USER32> But you have to wait longer for the antibodies to appear in the human.

<comment><USER31> True, usecase perhaps is more to confirm that what you had was covid, kind of complementary to this.

</comment></comment></comment><comment><USER33> A virologist is quoted comparing the search for a corona virus vaccine to the search for an HIV vaccine? Have I misread all the science over the last six months? That seems like an absurd comparison.

<comment><USER34> Mind explaining why it's absurd?

<comment><USER33> They are very different and HIV is very nasty. As a retrovirus that targets the immune systems (T-Cells, macrophages etc) with an unusually high genetic variability it is really hard to reliably invoke the kind of immune response that normal vaccines rely on for their efficacy. Corona virus on the other hand is an RNA virus that mainly attacks the respiratory system and invokes a strong (often too strong) immune response.

They just arn't very similar beasts, if you had to draw an analogy about the difficulty of the problem it would be that we don't yet have a cure for the common cold (which is a corona virus.)

<comment><USER35> > the common cold (which is a corona virus.)

A minority of them are. Most instances of the common cold are caused by rhinoviruses.

</comment></comment></comment></comment><comment><USER36> sound promising for air travel with 59/60 accuracy and last positive sample inconclusive</comment>


Show HN: An Open-Source iPad Calculator and Why Your App Won't Make Any Money - <AUTHOR> https://medium.com/ml-everything/a-free-open-source-ipad-calculator-and-why-your-app-wont-make-you-any-money-41eb5bd30d0e <comment><USER2> > I would like to contribute to a series of non-monetized open source apps. > Its possible that many truly free apps exist and I just cannot find them.

This is the biggest gripe I have with the (android) app store's search engine: You can't filter by license, by ads and other... I really like the filtering done by both sourceforge[0], alternativeto[1] and similar in this regard.

[0] [https://sourceforge.net/](https://sourceforge.net/)

[1] [https://alternativeto.net/](https://alternativeto.net/)

</comment><comment><USER3> "So if you're not going to make any money on ads, you may as well open-source your app."

Great point!</comment>


Dissecting a Rejection Letter from HR - <AUTHOR> http://www.pbs.org/newshour/businessdesk/2013/09/ask-the-headhunter-dissecting.html

<comment><USER2> The problem for the HR is that there is no nice way to say "You are perfectly good for this job, but we think someone else is better, so they get first refusal".

That is why they didn't send anything - because there is no nice way to say that, it is better to say nothing, and hope the second choice candidate is still looking if the first accepts another offer (always a 50/50 chance).

Also, what if second choice candidate claims that 'oh, first choice candidate got the job because she's used MongoDB? Me too!'. Do you invite them in for interview again to test that? Or hire the first candidate anyway at risk of a lawsuit? Communication has let the process get out of control.

When I'm looking for a job, I don't tell my employer until I have the paperwork all signed off for the new one. When a employer is thinking about not hiring me, telling me so is just going to make me accept another offer, and then if they do need me, they are stuffed.

<comment><USER3> You know, HR people are not that primitive. More often than not they'll sense you are a bad fit for the job, not because you are "not good enough", but because of other reasons.

\- You may be overqualified

\- You may be a very creative person applying for a dull routine job.

\- You may be a leader applying for a subordinate function.

\- ...

Furthermore, often they can not give you honest feedback due to antidiscriminatory laws.

<comment><USER2> That is true, but the situation I was addressing, as in the article, is where you have a short list of candidates that are suitable, but can only recruit say 1. In that case, you can't tell the rest they don't have the job until a preferred candidate has actually signed the contract.</comment></comment></comment>


Ask HN: SaaS tools for managing Startup activities? - <AUTHOR> Hi all,
I am working with a co-founder and we have a flurry of activities going on. I was looking at project management SaaS (such as Asana) but I am not sure - there are so many out there. We are a hardware startup with a bunch of chaos in terms of materials, design, prototypes, manufacturing schedules, supply chain, logistics, etc.
We are currently using Dropbox Paper to make lists but it is getting out of control. How can we organize our selves with clear priorities, dependences and schedules for tasks?
Any suggestions appreciated! :) <comment><USER2> Hi!

What about our team, we are working remotely and use (and work on) Riter: [https://riter.co/](https://riter.co/). However, we are a software development team and don't have the same chaos in the workflow :) Just in case, if it can help you:

Riter allows to manage the whole company with all its projects, teams and customers. The main features are:

\- task management and sprint planning (todos, tags, task states);

\- time estimation and tracking;

\- collaboration (comments, file sharing, assignment, access control);

\- project statistics (time spent, activity, performance, etc.);

\- groups of projects (combine several projects into temporary groups for simultaneous work);

\- GraphQl API.

</comment><comment><USER3> Have you looked at [https://airtable.com](https://airtable.com)? It allows you to create flexible forms and sheets to manage different kinds of information flows.</comment>


Ask HN: How can I learn computer security? - <AUTHOR> I am taking some data science courses. Is there any link between data science and cyber security? and where can I learn cyber security stuff? <comment><USER3> Cyber pro here - 5 years doing IV&V testing, 15 years as Fed, State and Local contractor, now a firewall admin at a major U.S. uni. I got an NSA accredited ([https://www.nsa.gov/resources/educators/centers-academic- exc...](https://www.nsa.gov/resources/educators/centers-academic- excellence/cyber-defense/)) Master's in IT with a specialization in security. While the degree got my foot in the door (I have a BA in Arts - but have been messing around with computers since the early 80's - plus a lot of self taught stuff) - I've found that a ton of side reading (anything related to the subject - I spent a lot of time on the RFCs - that stuff I use every day)), looking at PCAPs to understand the protocols and reading case studies are the best way to hone the craft.

For a while I was doing Governance, Risk and Compliance (GRC) work, but have always loved being a network security engineer, so I went back to that.

Also - I have a CISSP, which opens a lot of doors. I know that it is being knocked a bit nowadays, and there are certainly a some who are test knowledgeable but no hands-on, common sense experience. I still find it valuable enough to maintain.

Set up a lab - 2-4 computers and a switch should do (you could virtualize some/all of it) and work on all aspects of the TCP/IP stack if you're interested in netsec.

If appsec is your thing, spend a lot of time looking at good and bad code, plus reading on-line of good and bad appsec.

IF GRC is up your alley - read NIST 800-53, HIPPA, PCI-DSS, SANS Top 20 and GDPR - to understand the full breadth of controls and risk mitigation.

As far as data science python and pandas are all over the industry, R not so much. There is a big push for ML/AI work, but it might be snake-oil, time will tell. I use a lot of python and pandas for log and flow analysis.

Also - learn Linux CLI; grep, sed and awk can save your butt in most situations. Gray beard stuff will come later.

Good luck!

<edit - word choice>

<comment><USER4> While on the subject and since you're an expert in the field, there's something that's nagging me. How good are you guys at programming? My feeling is that most people in the infosec are average at best and only the elites are good at it.

<comment><USER5> I am not sure why you are getting downvoted. You have a very valid question.

There are many different kinds of security roles such as risk, compliance and security reviews (very little programming), penetration testing (programming and scripting to various degrees depending on the work) and security software development (full-time programming).

How good one is at programming would of course usually depend on the individual. In the rest of the software industry, there are all kinds of roles (some of which involve programming and some do not) and the ones that do involve programming have programmers of all kinds and calibre. It is no different in computer security.

I have over 12 years of experience in this field now in various positions where my various colleagues and I have written large security products in C, C++ and Java, as well as smaller security solutions in Python and Go. Most of the times programming is just a means to an end which is true for many other fields as well. It is usually math, algorithms, crypto, protocols, etc. that are more interesting and that we need to be well versed with along with being skilled at programming. Also, I would like to share two of my earlier posts around this subject:

\- [https://news.ycombinator.com/item?id=14873475](https://news.ycombinator.com/item?id=14873475) (about demand and job prospects in security software development)

\- [https://news.ycombinator.com/item?id=12545851](https://news.ycombinator.com/item?id=12545851) (about math and software development in the computer security field)

<comment><USER6> I haven't downvoted, but "How good are you guys at programming? My feeling is that most people in the infosec are average at best and only the elites are good at it." sounds trollish to me. Designed to elicit emotional response from people as they will try to defend security people.

</comment></comment></comment></comment><comment><USER7> Let me tell you one thing, it's going to be tough. Cyber security is one of the fields of IT that requires the most deep knowledge of how computers and networks work. So, be aware of that. It's like, when someone is asking how to build a game, and the first answer is: learn a lot about Math and Physics. This is the same thing.

Recommended path:

1\. CS Basics (concepts) Conceptually understand how computers work, how interpreters work, compilers etc. You're probably past this point.

2.Low level programming Basically C, but pay attention to the OS APIs (posix, win32). Make sure you understand the fundamentals of memory management, procedures, threading, etc. You need a lot of C knowledge.

3\. Networking [0] You basically need to know by heart all the TCP protocol. I have a friend who's incredibly successful working in security and he knows each bit in each packet in a TCP connection. He can just recite it. Once you know about networks, start throwing code at them. See if you can push the wrong bits to a switch, or if you can access some other processes network stack, etc.

4\. Web standards Basically, how the web works. Once you're past that: Apache and Nginx. You have to know them in depth.

5\. Known threats and vunerabilities In this process you'll know that there are many exploited issues that have been resolved. But you should study from them. For example, Heartbleed. Would you have the knowledge to find Heartbleed? You should also practice with every other security threat known like XSS, SQL Injection, etc.

[0] depending the security field, you might not need so advanced networking knowledge, this is just a general recommendation.

This is just my recommendation, I'm more of a purist, and I have a lot of respect for cybersecurity people.

Source: +10 years programming, I have a good friend making A LOT of money as a private security contractor and we speak about this all the time.

<comment><USER8> May I ask how much you friend makes, on an average?

<comment><USER9> I had a teacher who's brother worked in security, his bonus for the year was 6 figures IIRC. His salary was 300k but this was a while ago (5+ years).

</comment></comment></comment><comment><USER10> "Is there any link between data science and cyber security?"

Data Science might be useful if you want to work in Security Information Management or in malware analysis: big companies try to identify "weird behavior" in their networks, based on "normal behavior" records.

"Where can I learn cyber security stuff?"

Well, that depends on the stuff you are interested in...

You should focus at first on learning "system administration" and at least a programming language like Python or Ruby. Network protocols would also be a bonus.

Then, if you want to learn "offensive techniques" or "penetration testing", I suggest that you try websites like RootMe [https://www.root- me.org/?lang=en](https://www.root-me.org/?lang=en) or Cryptopals cryptopals.com.

Once you'll have resolved by yourself some of these challenges, you'll be able to try the "industrialized approach" of penetration testing. For this, this book is quite cool: [https://www.nostarch.com/pentesting](https://www.nostarch.com/pentesting).

If you're more interested in "defensive techniques", you have tons of resources online.

For instance:

Secure Coding Best Practices: \- [https://www.owasp.org/index.php/OWASP_Secure_Coding_Practice...](https://www.owasp.org/index.php/OWASP_Secure_Coding_Practices_- _Quick_Reference_Guide) \- [https://security.berkeley.edu/secure-coding- practice-guideli...](https://security.berkeley.edu/secure-coding-practice- guidelines)

=> These documents will help you to understand what are the main risks in your apps

For "general" cybersecurity: \- ISO27001 standard \- The NIST Cybersecurity Framework [https://www.nist.gov/cybersecurity- framework](https://www.nist.gov/cybersecurity-framework) \- PCI/DSS \- [https://www.us-cert.gov/](https://www.us-cert.gov/)

=> These documents will help you to understand what are the main risks in an organization based on their assets.

</comment><comment><USER11> Security fundamentally is just "the art and science of how things work". Breaches in security are caused by malicious actors finding vulnerabilities in trusted systems, such as when Chinese webcams shipped with default credentials which made it trivial for the Mirai botnet to take them over and acquire so many devices that it could DDoS Dyn.

Towards that end, just about anything you learn can be applied in some way towards security. The conventional recommendations others have made will get your foot in the door, but ultimately security is a lifestyle of never-ending learning and imaginative reasoning about systems.

So to directly answer your questions, yes, there are links between data science and security (others have mentioned its use in things like malware research, but going the other way it's also important to store your data securely to prevent theft, destruction, or tampering) and you can learn something about information security pretty much anywhere in-depth information about computers is taught. Which information is relevant to your interests will just depend on which aspect of security is of interest to you.

<comment><USER12> > Breaches in security are caused by malicious actors finding vulnerabilities > in trusted systems

Alex Stamos, the CISO of Facebook, likened the causes of security breaches to a pyramid. The bottom of the pyramid, where the vast majority of security breaches happen (perhaps 80%), are caused by basic fraud: shared passwords, phishing, asking the user to do something like self-XSS. Of the remainder, perhaps 80% of security breaches are done through the more common, more mundane attacks like unpatched vulnerabilities or misconfigurations. It's only a fraction of a fraction of a percent of attacks that are done using zero-day vulns.

As much as what you said is relevant for security, I would argue that engineers think that security is more of an engineering problem set than it is.

</comment></comment><comment><USER13> I highly recommend the online Cybersecurity Specialization of the University of Maryland on Coursera.

(disclaimer: I didn't pass the Capstone project, and never got around to trying it again)

Back then it was free if you didn't need a certificate, may still be.

[https://www.coursera.org/specializations/cyber- security](https://www.coursera.org/specializations/cyber-security)

</comment><comment><USER14> The very first thing that you need to do is to pick a software stack, ideally opensource and then learn how that software-stack works bottom up. Learning how a linux application works might be a good starting point if you are totally new.

There are multiple layers involved here and really understanding each one would take time.

Next would come understanding browsers, browser although it is an app, it is a world in itself. How an http request flows through a browser, how an http response is rendered, what are various layers involved. TCP/IP stack to physical layer, wifi/usb. It is extremely vast and very interesting.

And once you have gained enough experience , you will be able to clearly see the similarities and differences between various software stacks, both bottom- up and top-down, right from hardware level to your application's code and vice versa. And then reasoning about security of the stack at various layers would become straightforward.

In terms of conferences, I find Blackhat([http://www.blackhat.com/](http://www.blackhat.com/)) Conference is a very good source of keeping oneself up to date with world of security( including applying Data Science to Security)

</comment><comment><USER15> [http://mooc.fi/courses/2017/cybersecurity](http://mooc.fi/courses/2017/cybersecurity)

"Cyber Security Base with F-Secure is a course series by University of Helsinki in collaboration with F‑Secure Cyber Security Academy that focuses on building core knowledge and abilities related to the work of a cyber security professional. The course series is free and open for anyone to attend."

</comment><comment><USER16> If you want to get a really deep understanding, study computer science (e.g. bachelor/master) [1], then specialize by taking all security courses that are offered. Not only will you get a deep understanding of the topic, you will build on solid fundamentals, as well as have people (professors, assistants, trainers) who you can ask anything.

[1] Of course, you can this only freely in a country that remotely cares about the education of its citizens (e.g. most European countries, where you can go to university for a hew hundred Euros per year). Otherwise, the risk of a huge debt is probably not worth it.

<comment><USER17> I went to a mid level business school in the Midwest. Their IT program had a few segments. One was security. The difference was basically just one course on pen testing. I learned more about security in the CS courses they didnt recommend I take.

You don't need college, what you need is a desire to learn everyday.

<comment><USER16> _> what you need is a desire to learn everyday_

This is true, but in my opinion this advice misses the point.

Of course, if you are older and finally developed this skill, you don't need anything more. However, the questioner seems to be on the beginning of this journey.

Recommending "just learn everyday" is like telling fat people to exercise and not to eat too much. Of course that's true, but misses the point: How to get there?

Having people around you who motivate you to learn, who know and acknowledge what you are doing, and who are willing (and paid for) to help you if you get stuck - this is exactly what young and/or inexperienced people need, and a very good starting point until they are finally able to get along with the internet and nobody's assistance anymore.

</comment></comment></comment><comment><USER18> [https://www.hacksplaining.com/](https://www.hacksplaining.com/)

<comment><USER19> This is on of my favourites.

* It is neatly packaged, wonderful UX and a little sweetening of humour that makes it palatable to the layman

* It gives code examples that provide a little bit more in-depth exploration for the enthusiastic or aspirational novice

* It has a nice structure such that single exploits can be shared. For example, after a client of ours was hacked via a SQL injection on their website, we were able to show them the SQL injection exercises to demystify the topic a little bit and to make it seem less intimidating

* It's a great go-to starter. When I go to schools for careers fairs, I always give this link out to kids who are interested in security work.

</comment></comment><comment><USER20> This is a good course

[https://www.cs.cmu.edu/~213/schedule.html](https://www.cs.cmu.edu/~213/schedule.html)

Buy the 3rd version book (used) and then try the labs as you go through the lecture vids/chapters [http://csapp.cs.cmu.edu/3e/labs.html](http://csapp.cs.cmu.edu/3e/labs.html) but avoid the 'global edition' as it's filled with errata, or just know there is mistakes.

You will learn assembly/C and also Return Oriented Programming, stack protections and how they work, buffer overflow attacks, implicit casting grenades, cache optimization, how the linker works, ect. Then you sign up for that old Matasano CTF [https://microcorruption.com/login](https://microcorruption.com/login)

When you complete it apply to NCC Group who I believe now owns microcorruption. Start at the bottom, work your way into a policy/advisory role somewhere else after gaining experience and applying for certs [https://ciso.eccouncil.org/](https://ciso.eccouncil.org/) Having data science experience is likely helpful since you can produce shiny presentations that board rooms like to see when you become their CISO

</comment><comment><USER21> If you are a veteran or a federal employee, the Dept. of homeland security offers free online courses in network security. The program is very networking specific and you'd have to pay for any certification testing yourself, but the courses will help take you a good chunk of the way in terms of prep and learning. [https://fedvte.usalearning.gov/](https://fedvte.usalearning.gov/)

</comment><comment><USER22> Since no one has mentioned it yet, Capture the Flag contests (CTF's) can be a good way to get into security. They are online contests featuring a series of security related puzzles.

They are hosted by lots of different groups at different levels. There are CTF's aimed from high school through the DEFCON CTF. This is a directory: [https://ctftime.org/](https://ctftime.org/)

</comment><comment><USER23> I'm using Cybrary, it is a free and open source for learning cyber security: [https://www.cybrary.it/](https://www.cybrary.it/)

</comment><comment><USER24> One question I have along with the very good question from the OP, do larger companies and governments actually _care_ about security, or are they more interested in doing the proper dance and checking the right boxes to not be held responsible when they're hacked?

It seems irrational to want to learn how to secure systems when their owners don't care about it (and won't pay to secure them) if the risk can be transferred to other parties. I'm sure there's a few organizations that care if their data are stolen, but by and large it's a cost center, and treated accordingly in my experience.

<comment><USER25> It's a spectrum. There are companies that absolutely care with every fiber of their being, and those that couldn't care less.

Now that "cyber" is a thing that can lead to a CEO losing their head, most companies are roughly in the middle somewhere. Sure, there will be a fall-guy and finger pointing, but it's better to at least not be completely negligent.

"If the risk can be transferred to other parties" - that's pontentially a business avenue OP wants to pursue. AcmeCorp can buy OPs shiny datascienceDefender(tm) network monitor.

A lot of tech-first companies (and don't forget, some legacy companies are desperately trying to become these) care a massive amount about security, so there is definitely volume of work with people who genuinely want to improve the state.

</comment></comment><comment><USER26> Probably the most serious route to serious cyber-security training is via a military rating. At the state level the stakes and threats are highest.

</comment><comment><USER27> Get involved in an active community full of security professionals and learn hands-on helping to secure open source projects.

There is no replacement for mentorship and getting your hands dirty with real world systems.

I will now shamelessly plug the community I learned the most from: [https://hashbang.sh](https://hashbang.sh)

</comment><comment><USER28> yes.

Data Science can be applied in many different ways, (somewhat) Recently LightCyber was acquired by PANW [0] and I believe that if you are in data science that may be something that tickles your itch.

if I may give advice, asking a more specific question to a search engine may also get you ona path with additional information. 'cyber security' is a pretty large umbrella and much of it may bore you while only a small handful is interesting... so try to be more specific rather than say 'stuff'

[0] [https://www.paloaltonetworks.com/products/secure-the- network...](https://www.paloaltonetworks.com/products/secure-the- network/lightcyber-behavioral-analytics.html)

</comment><comment><USER29> What do you want to learn? Network security? Application security? Secrets management? Security operations?

I'm sure there are fields where data science is useful, like anomaly detection, malware classification etc.

</comment><comment><USER30> Just created this account so that I could comment, and stay (reasonably) anonymous. I'm the CTO of a reasonably well known security company, for what it's worth - and I've been doing this for a little over two decades in a few countries.

This first misnomer, is that there's one security thing. There are several. The offensive security folks (penetration testers) are far different than the advisory folks (think PCI, HIPAA). Vulnerability scanning, SOC (Security Operations Centers), secure development, and more - it's a wide field. The first thing to ask yourself is "what does security mean to me, and what do I want to do with it".

Give the first part of your question, I'll assume you're interested in software-based security (development) as opposed to infrastructure (network security, physical security, systems security), or the offensive side.

On the other hand, if you just want to 'get into security' \- then learn a little bit about networking, and go find a job as a 'Network Operations Engineer' or 'Security Operations Specialist'. These are fairly low-level jobs in the security industry, that can serve as entry points, and help you learn about the other parts of the industry in depth, whilst getting paid.

Echoing what @santigobalsuto and others have mentioned.

1\. CS - Understand how software works, not just how to code, but how things happen on machines. What are CPU registers? Write some assembly - nearly every single week my staff uses assembly to test an exploit. It's one hell of a lot easier to make a good developer a decent security professional than the other way around - just trust me on this.

2\. Take philosophy and propositional logic courses - Good security folks are terrific critical thinkers. They learn to understand what risk means, and how to contextualize it for an organization. In other words the approach to risk and tolerance is completely different for Home Depot than it is for Evernote.

3\. Learn Software Testing (not Quality Assurance). Combining this with critical thinking means you can start to pick apart software, even networks, from varying vantage points. If you combine this with your CS knowledge, it can really help you understand how to attack an application, or an organization.

4\. Build lab out of cheap, garbage hardware. For about $300-500 you can get 4 servers on Kijiji that can be used to run OpenStack and VMWare. Get a physical switch, ideally something with a TAP port (but you can replice with VMWare easily enough). Create VMs, play with things like Security Onion, create and destroy networks - use traffic generators... have a great time.

5\. Read about standards - it helps to understand what NIST is for, what CVEs are, CWEs, OVAL. Explore a few vulnerabilities (CVEs), and understand what they are, why they matter, how they apply. Then grab a copy of Nexpose Community Edition, and scan your lab - play around.

6\. Rebuild your lab, iterating on what you've learned above.

<comment><USER31> Thanks for your reply. I asked above already, could you explain what to study or what certs to get to follow the path of "advisory folks (think PCI, HIPAA)"

<comment><USER12> Not the gp, but I also work in security (not in management).

"What are the best information security certifications?" @ Quora[1]. The problem is that it's a massive list. Even if you pick a short list / subset of this list, it's far too much work to get started.

Honestly, I would join/start a club at your school or in your neighborhood. Find people with similar interests. Work together on a single certification. Start small and target a single subset of "cybersecurity".

Honestly, I would start by trimming down the list by filtering out what you aren't close to achieving yet. If a cert requires access to Cisco hardware or assembly language programming skills, there is no point in working on one of those certs first.

[1] [https://www.quora.com/What-are-the-best-information- security...](https://www.quora.com/What-are-the-best-information-security- certifications)

</comment></comment></comment><comment><USER25> OP: you'll find you got a lot of very vague, broad, range of answers. I think it might help if you try to find a subset of security that you're interested in or would like to tackle.

It's very hard these days to be a complete and effective generalist in security, let alone be good at a range of security stuff while also being great at data science.

I'd consider focusing on appsec for a year, get a job attacking or defending apps for a year, and then you'll have a basic understanding of the problem space.

</comment><comment><USER32> That's a short question with a complicated answer. I'm traveling right now, but you should shoot me an email and I'll help you get started. Address in HN profile.

</comment><comment> video-host Check out [https://pentesterlab.com/bootcamp/](https://pentesterlab.com/bootcamp/)

</comment><comment><USER33> In addition to great ressources people are sharing, Security+ and CSA+ from CompTIA can be great certification (vendor neutral and inexpensive).

</comment><comment><USER34> If you are interested in web pentesting, learn to hack Webgoat and DVWA. From there go after live sites via a bug bounty platform (I like Bugcrowd.)

It is hard to overstate the value of the chance to test (and demonstrate) your skills against a live production site.

</comment><comment><USER35> [https://www.edx.org/micromasters/ritx- cybersecurity](https://www.edx.org/micromasters/ritx-cybersecurity)

</comment><comment><USER8> Would anyone please revert a name of a book that may probably help here?

</comment><comment><USER36> Is the OSCP certification worth it?

<comment><USER25> It depends. If you're expecting it to land you a job, no. If you're trying to add to your arsenal of skills, maybe (arguably you could self teach a lot of it and save your money). If you're trying to get past HR in some big company who use it's metric, yeah sure. It also depends on what you want to do - <USER2> gather it's useful for netpens but useless if you want to do appsec, which is a huge chunk of pentest work. What are your end goals?

<comment><USER36> No goals per se...just kind of interested in the field and would be doing it for personal interest. I've found tons of sites that offer lab environments and downloadable VM's that allow you to practice exploits so maybe that would be a cheaper way to go in the beginning.

<comment><USER25> Yeah totally do that then. And get a copy of burp and hack some web stuff. Have a crack at bug bounties to test your chops.

</comment></comment></comment></comment><comment><USER37> low level programming / radio. try not to cry ;)

</comment><comment><USER38> Start by forgetting the word _cyber_.

<comment><USER39> This is bad advice.

There's a reason that cyber is used, because you need something to disambiguate it from all the other kinds of security.

Imagine you're a policy person at the pentagon (or equivalent), if someone talks about security then that doesn't narrow it down to all the other kinds of security going on there.

You could use "info sec" but there are agencies who deal with a lot of _information_ which doesn't necessarily mean this space either. They've been dealing with information security since their inception most of which I suspect is focused around people and not machines.

Cyber security makes it clear to those people what you're talking about.

To someone who works in SV and spends all day with developers the context is other way around, and in that context cyber sounds asinine and if you talk about security someone immediately knows it's security in your space.

That's my guess at why you tend to see 'info-sec' in the private space and 'cyber security' in the public space.

</comment></comment><comment><USER40> > Is there any link between data science and cyber security?

There's... a company around LA area that does cyber and data science. I would think they apply data science to logs and such to figure out abnormality. Likewise if you count email spam detection as a cyber security thing.

For cyber security, take Network+ or just grab a Network+ book. I believe that's where you should start first in cyber security.</comment>