We're trying to stop bad mosquitoes by raising and releasing good ones - <AUTHOR> http://debugproject.com/ <comment><USER2> *Verily Debug Project (title is currently "Google Debug Project")

Verily (formerly Google Life Sciences before the Alphabet split-up) is a sister company to Google. This isn't Google, and it's not code, but a project to help eliminate mosquitoes by a bioscience company.

<comment><USER3> Your right it's a sister of Google _not_ a Google project, maybe a title change to "Alphabet" instead of "Google". Especially since I clicked it thinking it had something to do with code. :P

<comment><USER4> They do present it quite ostensibly as a programming project tho i.e: the steps mentioned here [http://debugproject.com/how/](http://debugproject.com/how/)

</comment></comment></comment><comment><USER5> I wonder how long it will take for the mosquitoes to evolve a mechanism that allows the female to detect males infected with that bacterium and not mate with them, or a mechanism to counteract the detrimental effect on the development of the eggs.

Right now the evolutionary pressure to do this is probably quite low as there is only a small number of infected males, but if that number goes up the pressure will increase exponentially.

Example:

If 1 % of males are currently infected with the bacteria, the selective advantage of a mosquito that can counteract the infection is just 1.0/0.99 (as the probability of mating with an infected male is just 1 percent), which is probably too small compared to other risks to produce any visible evolutionary effects.

If we increase the number of infected males to 90 %, the evolutionary advantage of detecting them soars to 1.0/0.1 = 10! This means a mosquito able to detect or counteract an infection is ten times as likely to produce offspring, which provides an incredibly strong gradient for evolution.

The question is of course how fast an immunity can arise (or if it already exists in the population), and how many generations of mosquitoes are able to survive after the infected males are introduced.

Probably they ran their own population genetics simulations on this, so I'd be curious to see results, which should give a good indication on whether this can work and if so under which conditions.

My personal guess is that it won't be effective, as there are very few cases where introducing a single external stress factor into a population causes it to collapse entirely, what's more likely is that it will adapt and relapse.

<comment><USER6> Yes, it may be possible for the mosquito to fight back.

The bacteria isn't mentioned in the video, but I believe it could be Wolbachia, which is already pervalent in the insect world. According to this FAQ [1], it already infects 60% of the insect world, so it appears that developing an immunity to it might be pretty hard:

 _8\. Do other animals carry Wolbachia?_

 _Wolbachia is common among arthropods (including insects, spiders and other small animals with no backbone). Up to 60% of insect species naturally carry Wolbachia, including butterflies, dragonflies, moths and some mosquito species, but not the primary species of mosquito involved in the transmission of dengue._

 _Wolbachia is also found in certain types of roundworms – known as nematodes – but this is very different to the insect Wolbachia that we work with. Wolbachia is not found in any larger animals such as mammals, reptiles, birds and fish._

\- [1] [ [http://www.eliminatedengue.com/faqs/index/type/wolbachia](http://www.eliminatedengue.com/faqs/index/type/wolbachia) ]

<comment><USER7> What are the effects of Wolbachia on other insects? Is there a risk that other insects get eliminated by eating mosquitos carrying Wolbachia? It may be safe for humans and larger mammals, but how about smaller animals ansd insects?

<comment><USER6> The Wolbachia bacteria is a parasite, but it spreads by infecting the reproductive organs of insects. I don't believe it can be spread by eating an insect infected with Wolbachia, unlike other kinds of parasites.

</comment></comment></comment></comment><comment><USER8> I still want to build:

\- An array of three microphones for locating mosquitos by sound

\- a servo-mounted laser

\- Software to combine the two, possibly with a manual mode.

<comment><USER9> Well it does sort of exist: [https://www.fastcoexist.com/3059127/what- happened-to-the-mos...](https://www.fastcoexist.com/3059127/what-happened-to- the-mosquito-zapping-laser-that-was-going-to-stop-malaria)

<comment><USER10> The last paragraph says:

Correction: This story used out of date information to claim that the technology had been licensed out. It has not been licensed to date. As well, it formerly stated that the U.S. Commerce Department paid for the company to showcase the technology in Germany. It just invited the company. We regret the errors.

</comment></comment></comment><comment><USER11> How is "Can't breed" a good thing? Something to do with

> good bugs ... will stop bad ones from reproducing

? Are mosquitos monogamous (or do they just get really tired after sex, or do they transmit the infection to others...) If "good bugs" don't bite, I'd have thought you'd want them to reproduce as much as possible, so long as not- biting was hereditary. A non-reproducing population can never out-compete the reproducing rest of the species.

<comment><USER12> This is all explained in the video. The process has nothing to do with genetics; the good bugs don't bite because they're all male (the females are weeded out by the scientists), and male mosquitoes never bite. And they can't breed because of a bacteria related to healthy egg development. The good bugs still mate (with bad bugs, since there are no good female bugs), they still fertilize the eggs, but the eggs don't work.

<comment><USER11> That explains most of it -- the not biting, the displacement effect. Still, infertility is strongly selected against. Unless the introduced mosquitoes _completely_ displace existing males in the mating pool, or the bacteria are transmitted between mosquitoes, it sounds like a pretty short term fix. With males living only 10 days, wouldn't the population bounce back pretty quickly?

</comment></comment></comment><comment><USER13> I dont get it, if you release a ton of sterile male mosquitos.. what exactly is their purpose? They cant bite, breed or spread diseases.. well okay, but why would the amount of bad mosquitos decrease in any way from their introduction? Wouldnt they simply die out quickly then, what's the lifespan of these good mosquitos? I dont get it at all.

<comment><USER14> [http://debugproject.com/how/](http://debugproject.com/how/)

Looks like they mate with females, which still lay eggs, but the won't hatch. So less bad bugs mate and produce offspring. Needs large numbers of good bugs though.

<comment><USER13> Yeah it sounds like it would be better if the eggs still hatched but they should only produce good male mosquitos again.

<comment><USER15> They're only 'good' because they're sterile, and capable of sabotaging mosquito reproduction. If they weren't sterile and "only produce good male mosquitos again", the population would not decrease, they would just be releasing regular male mosquitos.

<comment><USER13> right, which means that eventually the entire population would be male and then die off

<comment><USER16> I was about to say that's impossible because meiosis produces equal number of "male" and "female" cells but... someone actually did it. [https://www.fredhutch.org/en/news/center- news/2014/06/eradic...](https://www.fredhutch.org/en/news/center- news/2014/06/eradicating-female-malaria-bearing-mosquitos.html)

</comment></comment></comment></comment></comment></comment><comment><USER17> Seriously, Google (/Alphabet), as long as you continue to invest in such research studies, you can take all of my so called personal data for its progress and I won't mind.

<comment><USER18> Because my personal data is very helpful with creating infertile mosquitoes.

<comment><USER17> Good try, but no, because my personal data can help them earn money (through advertising or other means), which Alphabet can use for funding their research in other areas.

</comment></comment></comment><comment><USER19> Will we eventually see resistance to the bacteria the way that bacteria evolve to be resistant to antibiotics?

</comment><comment><USER20> They don't mention this, but I believe this can cause extinction of mosquitoes within an area:

[https://en.wikipedia.org/wiki/Sterile_insect_technique](https://en.wikipedia.org/wiki/Sterile_insect_technique)

<comment><USER21> I believe that is the goal. I'm all on board. I've heard about proposals like this for years and years. I'm thrilled someone is actually working to make this theory reality.

<comment><USER22> It'd be sort of like removing all of the kelp from the worlds oceans. It'd have ripple effects on other species that we wouldn't want to negatively affect.

<comment><USER23> People have studied this. While you have to take these theoretical "what if" studies with a grain of salt, some believe that eradicating mosquitos would not affect ecology in a major way:

[http://www.nature.com/news/2010/100721/full/466432a.html](http://www.nature.com/news/2010/100721/full/466432a.html)

<comment><USER22> I'm now at negative points for bringing up a point that is validated by quoted experts in your cited source. (Which btw, uses a lot of "cover my ass" wording when talking about wiping out a species found all over the globe)

I've gotten malaria on a trip to west Africa, I've had to get yellow fever vaccines, i grew up in areas with west nile, etc I understand the impact it has on humans. However, there is more to the mosquito than the little insect flying around biting us. Their eggs and larvae play a huge role in the ecosystems they inhabit.

<comment><USER21> > I'm now at negative points for bringing up a point that is validated by > quoted experts in your cited source.

I think it's hard to justify drawing an equivalence between exterminating mosquitoes, which scientists have studied and broadly believe to be safe, and _removing most of the plant matter from the oceans_. I would not have downvoted you, but it's not hard to understand how you ended up there.

</comment></comment></comment></comment></comment></comment><comment><USER9> This seems like the only way it can work is to completely overwhelm the population with these sterile males, which seems prohibitively expensive.

I'm also a fan of the idea of the idea of locating them with sensors and shooting them out of the air with a cheap laser. It seems like this sort of technology should get very inexpensive pretty soon.

</comment><comment><USER24> I first read about using sterile male mosquitoes more than twenty years ago (there was no mention of the bacteria, so I don't know if a different technique was proposed); how comes this is news now?

Or is it that this has been proposed since forever but only now someone is talking of actually throwing money at it?

Edit: typos

<comment><USER25> Not an expert, but I think an important part of this is only releasing male mosquitoes, and separating them at scale is generally considered to be difficult (there have been other successful uses of SIT but always with different species).

From their FAQ it sounds like they have made progress on this front:

> What has Debug accomplished so far?

> We've developed methods for automated mosquito-rearing, and we know that we > can separate males from females with high precision, and we're continuing > development of these technologies at larger scale in our lab.

</comment></comment><comment><USER26> Similar (using crispr to change the mosquitoes' genome) video from Kurzgesagt, one of my favourite Youtube channels: [https://www.youtube.com/watch?v=TnzcwTyr6cE](https://www.youtube.com/watch?v=TnzcwTyr6cE)

</comment><comment><USER27> Wow, do people ever learn from history? I mean you just have to visit Australia to see what happens when you mess with nature...

</comment><comment><USER28> What's going to happen: a bug in development will cause the "good" mosquitoes' eggs to actually hatch, giving birth to mosquitoes that, because of an unexpected consequence of the bug, are immune to all other forms of killing AND live longer AND can give birth to many more mosquitoes than the previous, normal, "bad" mosquito.

</comment><comment><USER29> Develop this on GitHub. It will have huge public impact and will pick up steam fast!

</comment><comment><USER30> Well, How will they know that all the bad bugs are gone?

<comment><USER31> Simple, when all the bugs are gone.

</comment></comment><comment><USER32> I'm sure this will end well..

<comment><USER33> What is your concern?

<comment><USER34> So many potential unexpected consequences. Most attempts at killing off a pest have resulted in all kinds of negative shifts in the environment that end up being worse, just look at the Great Leap Forward in China. There are examples worldwide of this. Ecology is just too networked and interdependent to be able to easily engineer.

<comment><USER35> There have been a lot of studies on this, and apparently it is safe to eradicate these particular mosquitoes (and given how many people are sick each years because of them, it doesn't feel like it could be worse than not eradicating them).

In any case, it's not really comparable to idiots killing birds during the GLF, since it was a completely stupid policy based on no science at all.

</comment></comment></comment></comment><comment><USER36> Releasing a species to control another species; what could possibly go wrong?

</comment><comment><USER37> The problem I see is that, while this can be used for good, it can also be weaponized.

Zika for example, has existed for thousands of years if not millions of years. How come just in 2015 there's this massive unprecedented outbreak? Just when Oxitec starts releasing their "good mosquitoes"?

What specifically changed and caused this outbreak?

<comment><USER6> Whatever caused the Zika outbreak in Brazil has nothing to do with Oxitec. As this post shows, Oxitec released their mosquitoes years before the Zika epidemic in Brazil and the release was done hundreds of miles away [1]

\- [1] "No, GM Mosquitoes Didn't Start The Zika Outbreak." [ [http://blogs.discovermagazine.com/science- sushi/2016/01/31/g...](http://blogs.discovermagazine.com/science- sushi/2016/01/31/genetically-modified-mosquitoes-didnt-start-zika-ourbreak/) ]

<comment><USER37> Thanks, very informative.</comment></comment></comment>


Gab is the Pro-Free Speech Social Network (with 40k sign ups in 3 weeks) - <AUTHOR> https://medium.com/zero-infinity/gab-ai-is-the-pro-free-speech-social-network-65b6027dd0ab#.ka59t9ck3 <comment><USER2> "No illegal pornography, a zero tolerance policy for promoting terrorism or violence, and users are not allowed to post other's confidential information without their consent."

Oh, so not free speech afterall. Also it's obviously going to be blocked in any country that doesn't like free speech. I won't even bother trying to use it in China.

</comment><comment><USER3> It's tacky to post content like this on HN without disclosing a personal interest in the company/product. I'm assuming by your profession (as well as using this URL: [https://gab.ai/?ref=mfishbein](https://gab.ai/?ref=mfishbein)) that you have a business relationship with Gab.</comment>


Systems Design for Advanced Beginners - <AUTHOR> https://robertheaton.com/2020/04/06/systems-design-for-advanced-beginners/ <comment><USER2> As a solo tech founder of several sites, I've had the pleasure of digging into each of these problems and more over the past few years. Some topics worth adding to the list for consumer sites:

1\. Prototype and benchmark each of your stack pieces before you pick a stack. It is far easier to fix architecture mistakes when you don't have 10000 users expecting overnight customer service. If you are using new technology, find good open source products to see how they've structured their projects. Your architecture, designed for speed and experience, will be your key differentiator. Inherent speed at core task, design to user needs, and name choice are the 3 musketeers of a solid growth.

2\. Prepare for abusers to attack your system from every direction, especially if you enable users to publish content under your domain. You will see bots looking for Wordpress installations, users trying to fill content with SEO links, users trying every hacking vector known to market. Collect known vectors and test for them and never refuse a legitimate bug bounty request.

3\. There is an eternal debate about the trade off between building a quality product at start or opening early to get user feedback before you get too deep into features. There is merit to both choices as a solo founder. The moment you open the gates to users, your ability to make changes comes with very high friction. With time, trying new features becomes a tremendous luxury hidden under bug requests, roadmap, customer service replies, etc. Build your biggest riskiest assumptions first.

4\. Testing is hard as a solo founder. Selenium is your friend. If you don't spend time with them, your users will take that time in multiples after a mistake.

The best way to learn is to come up with a product you really want and build it in your own time. You can test launch in a weekend.

When I started launching consumer sites solo as an engineer, I went from a tight specialization to being unafraid to try any tech if it gives me an advantage in solving a problem. Once you've simulated enough problems and dealt with the consequences of your choices personally, you can play that 10 level chess game with the architecture of each new feature much faster.

<comment><USER3> This is interesting advice because every company I've come across that has been successful enough to get past the early startup phase has by and large ignored all of this.

The companies I've seen succeed were 100% focused on shipping their product to customers. Not 90% focused on customers and 10% focused on code quality, but 100% focused. They'd rather have to spend 30 engineer-days a few years from now fixing an issue if they get to that point than spend 3 hours getting it right upfront.

As an engineer that goes against every instinct I have, it really seems like spending a couple hours upfront must be a better use of time. It seems like it should be possible to spend 10% of your time setting yourself up well for the future, that's still just a rounding error of your time. And then if you do survive another few years, you'll have a huge leg up on other series B or C stage competitors if you're not hindered by a lot of tech debt at that point.

But from a capitalist perspective, it's probably not so crazy. If you are working with a $200,000 seed round in the beginning, and an engineer costs $80,000 a year, 3 hours of their time costs $115 which is 0.06% of your funds. And more importantly, that $200k is maybe enough for a year of runway, so 3 hours is 0.14% of the time you have to live given a 40-hour a week year (or 0.07% of an 80 hour a week year). Every bit of that starts to add up. Whereas by the time you're a later-stage company and you've raised, say, $40 million dollars and are paying engineers $150k, 30 engineer-days of work is $17,300 but that's only 0.04% of the money you've raised plus your runway is now approaching infinity if you're close to profitable.

I'm still kind of playing devil's advocate here, my instinct really wants to believe that a better balance than what I've seen is possible. One huge missing factor is that people have a strong tendency to ignore spread out costs like the time wasted fixing bugs that pop up later whereas the upfront cost of writing a bunch of tests is more visible. But it has been interesting for me to consider that maybe most founders really are acting pretty reasonably, even though it originally seems pretty careless and lazy to let your startup build up a ton of tech debt early on.

<comment><USER4> There are compounding gains from making "the right choice all else equal." This is one of those semi-mythical 10x engineer super-powers. Or it might just be experience. But technical choices need to be made and making slightly better ones will pay off even in the short term. The technical changes should just optimize for rapid change rather than stability or scale.

</comment></comment></comment><comment><USER5> Thought I'd post this here since I imagine there's a strong cross-over of interest:

If you'd like a guided video code-tour of how all the pieces of a production web-app fit together, I publish detailed weekly screencasts showcasing the code, systems, and architecture behind Oxbridge Notes, the business that's supported me for the past decade.

So far I've covered:

— software dependency vetting

— data integrity systems (constraints, foreign keys, transactions etc.)

— integration testing systems

— trade-offs in software quality between customer-facing and admin areas

— softer stuff, like designing for SEO (marketing ease is a critical part of any system I design, as an indie-hacker)

[https://www.semicolonandsons.com/series/Inside-The- Muse](https://www.semicolonandsons.com/series/Inside-The-Muse)

<comment><USER6> Hey Jack,

Just a heads up - I tried to sign up for your mailing list but kept seeing "ERROR: Did you forget to type your email?" even after typing the email. I've tried a few different emails and also tried viewing the page in incognito mode. Hope you get this fixed soon, as I'd really love to get more of your content!

<comment><USER7> I get the same error :/

Also, it'd be great to be able to turn off the mailing list sign up on the videos. I'm already technically subscribed via RSS and it'd be great if the videos play through without intervention

</comment></comment></comment><comment><USER8> They have this diagram:

      +</comment><comment>-----+   +</comment><comment></comment><comment>--+   +</comment><comment></comment><comment>-----+
      |Web Browser|   |Smartphone App|   |Client Libraries/|
      +-----+-----+   +</comment><comment>+</comment><comment>-+   |Other API code   |
            |                |           +</comment><comment>-+</comment><comment>---+
            |                v                   |
            |          +-----+</comment><comment>+            |
            +</comment><comment>--->+ Steveslist +<</comment><comment>-----+
                       |  Servers   |
                       +</comment><comment></comment><comment>+

Why not this?

      +</comment><comment>-----+   +</comment><comment></comment><comment>-----+   +</comment><comment></comment><comment>--+
      |Web Browser|-->|Client Libraries/|<--|Smartphone App|
      +</comment><comment>-----+   |Other API code   |   +</comment><comment></comment><comment>--+
                      +</comment><comment>-+</comment><comment>---+
                              |
                              v
                        +-----+</comment><comment>+
                        | Steveslist |
                        |  Servers   |
                        +</comment><comment></comment><comment>+

<comment><USER9> Yes, I think the second diagram that you showed is the preferred way of doing things, since using or "dogfooding" the client libraries will ensure they are always tested and correct, plus you'll get immediate feedback from the other developers who work on the apps as well.

<comment><USER10> In this day and age, your client libraries should be generated automatically. And of course used in your apps/webpage to save time.

See openapi/swagger for good examples how client generation works

<comment><USER11> I've been using openapi/swagger for generating my clients and honestly think it creates a mess of files and manually keeping clients up to date is not a huge task. I'm not sure that I buy into "your client libraries should be generated automatically" as a blanket statement.

<comment><USER12> Granted, APIs aren't updated often, but when they are, generated clients can save a lot of time. I can update a generated client in half an hour and be very confident of its correctness, whereas a manually constructed client would take hours to update and test.

Of course, the generated client is not very convenient, being a 1-to-1 mapping with the API. You build a high-level client with the more common operations on top of the generated one.

It's possible to do this in a way that gets you the best of both approaches, with some up-front planning.

</comment></comment></comment></comment></comment><comment><USER13> Nice tutorial! One small nitpick:

> Calculating a hash value from an input is computationally very easy, but > reversing the transformation and recovering the original input from its hash > value takes so much time and computing power that it is, practically- > speaking, impossible.

The above is true for encryption but not for hash codes. Recovering the original input from a hash code is not just practically impossible; it's _provably_ impossible -- even with an infinite amount of computing power -- because in general a hash code contains less information than the original text.

<comment><USER14> Actually, it might be occasionally possible. Not perfectly, of course. If the input space is limited and you have other constraints on it, then you might be able to find the original text. For example, let's say your input is a 280 character long tweet, the hash is only 16 bytes long. If you find that "I AM A STABLE GENIUS" is one of the inputs that hashes to the given hash value, then you can be reasonably sure that this was the original text, given that most of the other inputs with the same hash won't be meaningful English sentences.

<comment><USER15> I'd say the count of believable English tweets would be many orders of magnitude greater the count of 16 byte hashes, so in general we can't just find a believable tweet and declare that we found the plaintext. There would be many possible collision.

The phrase you mentioned is notable (with a slight edit) - so I suppose you mean you could index phrases by notability, then it may be tractable find tweets of notable phrases, since the number of notable phrases is relatively low.

</comment></comment></comment><comment><USER16> > How do they store their data?

SQL.

> How do their different applications talk to each other?

Proprietary APIs.

> How do they scale their systems to work for millions of users?

T H E C L O U D

> How do they keep them secure?

They just... don't.

> How do they make sure nothing goes wrong?

They just... don't.

> What are APIs, webhooks and client libraries, when you really get down to > it?

Easily outsourced to India.

<comment><USER17> That last point is so painful and true. I work with a product that has dozens if not hundreds of "connectors" to interface with other systems and you can tell that most of them were farmed out to the lowest bidder.

</comment></comment><comment><USER18> I have been doing a greater part of the things elaborated in the post as a solo founder/freelance software person. Albeit the scale of things I need to handle are relatively small.

What kind of a job profile should I be looking at if I am in a position where I absolutely need to have one?

I don't slot particularly well in any one thing I feel, neither a great developer nor a great systems person. And I did do a bunch of recruiting and tech consulting too:

<comment><USER19> Have you ever considered working technical support? As a lifelong generalist programmer, I never imagined I would work in a support role beyond what every programmer at a startup is expected to do, but I was persuaded to cross over by someone I greatly respect. That support team was the most talented team I ever worked with in almost twenty-five years in tech, and working in that role grew many different skills both in and out of tech in ways I never imagined. Hacking broken production environments back online and writing and pushing bugfixes is exhilarating, and it was the single most valuable experience of my career.

<comment><USER18> Never considered support role to be honest. Maybe it feels a little inferior after having those degrees I guess but that's my own bias. I did think of like a product management role at startups but they all expect you to have pedigree or previous work ex in similar role. Catch-22!

<comment><USER19> Yeah, that's exactly what I thought, too. It is just one of those weird cultural artifacts that when someone says "tech support" we automatically conjure this mental image which is both not really technical and not really flattering. It was definitely one of those life moments when I had to sit down and talk myself out of a baseless prejudice.

Also, the sibling commenter mentions one of the big reasons I found so much joy in it. This kind of hardcore technical support is in many respects closely related to working on greenfield projects. In my experience, the fun stuff usually ends up with some unfun caveats like company bureaucracy, office politics, or having the tech stack chosen beforehand, this kind of thing. But there's an interesting inversion when a company's production environment is broken and you're the one that can fix it. Obstacles magically disappear and you have a _tremendous_ amount of freedom to do your job, with effectively two different companies doing what they can to enable your work, because the only thing everyone cares about is getting it working again.

You do have to be able to walk the walk -- and I cannot stress enough that it takes a certain type, and just knowing your shit isn't going to cut it -- and you also have to get some personal enjoyment out of chaotic environments, and be able to talk people out of a tree sometimes. But you'll never run out of new and interesting problems to solve, you'll be testing your mettle far more frequently than your peers, and you'll be expanding your professional network every week about as much as everybody else does once or twice a year when they go to a convention. It's the closest thing the tech industry has to a firefighter or superhero or something.

<comment><USER20> Maybe infosec or cyber security would be similar? I'm a fullstack turned mostly FE developer these days, but I'm very interested in security and it seems like there is a lot of demand out there for that skill set and plenty of opportunity. It seems like it would push a lot of the same buttons that you mentioned (fixing broken systems, lots of autonomy, working with smart folks). I tend to get bored with long-term projects unless there are still interesting problems to be solved so working on lots of smaller projects sounds pretty appealing, and there's seemingly no end to new systems/devices/websites that are just riddled with holes. Seems like a perfect role and potentially more lucrative than IT (although this is just pure guessing, I haven't compared numbers).

<comment><USER18> I have explored security world too. Lots of experiments with reversing tools like IDAPro, ollydbg.

Recently a client asked if I can help reverse their own android app because the developers were holding their source code hostage. It was lots of fun and lots of struggle but overall satisfying.

Maybe something like basic security freelance work especially basic webapp security(xss,sqli,csrf, the likes) and some process security with maintaining proper logins, maintaining and rotating credentials etc.

Thank you for your ideas.

</comment></comment></comment></comment></comment></comment><comment><USER21> This basically outlines my current job. Huge system with many, constantly moving/upgrading parts and services across teams, all while utilizing dozens of internal services, tools, and navigating corporate policies. I'm thriving in it, but totally recognize it's a steep learning curve and takes longer to onboard newcomers.

Eventually you get to enjoy deprecating old services as much as building new ones, simply because you never have to teach others about them again.

</comment><comment><USER22> > Steveslist currently has a very simple and slightly fragile cron setup. We > have a single "scheduled jobs server". We use crontab on this server ... > This setup isn't scaling very well ... We're considering setting up a new > system using a modern tool like Kubernetes.

Do people use Kubernetes for running scheduled jobs like this? It seems like it'd be overkill but in saying that I'm not sure if I know of anything that can be used for running scheduled tasks in a reliable and observable way that's scalable. Maybe Jenkins?

<comment><USER23> > Do people use Kubernetes for running scheduled jobs like this?

We do, mostly because we already have Kubernetes and automation to deploy to it, so why not? The reliablity and monitoring are better than something we'd cook up on our own.

</comment></comment><comment><USER24> How can I gain practical experience in these things? The jobs I've had mainly revolve around adding new features, not doing any of the things described in the article. Does working at bigger companies actually give you experience with this?

<comment><USER25> Yes and no, depends. It gives such experience to a software architect, otherwise not really. Tho, software architects do a lot more than just systems design (tho, that is pretty much their main task). But yes, you get a job as a software architect or something like an "in-training" one, helping one, etc. at a big corporation to get such experience (I'd argue that small companies only need some code design, you want a lot of systems/software design you should go for a big corporation)

<comment><USER26> You evolve into the architect role as an IC. No new grad is hired like this. You design things and keep going up in scale.

<comment><USER25> It happens from time to time, but yes - there is no specific job / hiring for this.

The default/common path is evolving into the architect role which seems like a "bad" process of developing an architect.

The best way to develop a new architect, is to have him learn alongside a mentor/teacher who is an architect himself. Developing and architecture are very different jobs which require different mindsets and skills. Also, I've seen many places, teams, etc where people are mostly made to implement feature after feature with no time in between for learning, self-development, courses, etc.

Otherwise, after you've just "arrived" to that architect role you start learning on your own what architecture really is and means.

<comment><USER26> Developing and architecture are absolutely related. Many ideas seem great on paper till you go and implement them and find out the pain points. All the great architects have often been in the weeds of their systems.

</comment></comment></comment></comment></comment><comment><USER27> Regarding DB backups, it would be useful to add that it is not enough to simply make backups, one should also test that the backups are actually restorable.

<comment><USER28> also the restore/recover process has to be laid out step by step and be as simple as possible. No one is doing a restore in a relaxed, easy going state of mind. Restores are done in full panic mode with the team thinking about how they're going to break the news to their families that they've lost their job.

</comment></comment><comment><USER29> Just recently started messing with AWS Amplify. It's crazy how easy it makes basically all this stuff. It really makes it simple to be a solo founder and manage every piece of the stack.

</comment><comment><USER30> Title corrected - Web application system design for advanced beginners

<comment><USER31> Yeah, it's not true "Systems design" since it's specific to programming

</comment></comment><comment><USER32> I love the phrase "Advanced Beginner"

</comment><comment><USER33> Don't forget a system for organizing contributions to all this code, a system to ensure compliance with the many legal regimes "Steveslist" is offered in, a system for onboarding new hires, a system for expanding market share, a system for keeping money and reporting on it to wouldbe investors, and a system of systems designers that know how to respond to changing and ever expanding requirements.

Don't sell yourself short: there are systems everywhere you look, not just where there's convenient precedent to make a web service.

</comment><comment><USER34> This is the best thing I have read in 2020. As a bug bounty Hunter this is insanely helpful to me .

</comment><comment><USER35> > To do this, we need to write database queries that aggregate over the > entirety of our data. We don't want to run these queries against our > production SQL database, because they could put an enormous amount of load > on it. We don't want a huge query issued by an internal analyst to be able > to bring our production database to a grinding halt

What kind of query would you have to write to bring down a production db? What makes a solution like hive much better - I guess its optimized for this?

<comment><USER36> > What kind of query would you have to write to bring down a production db?

Scans and Sorts, seen in a query plan, are relatively expensive to run in a production row store. OLAP queries (GROUP BY with aggregate functions like COUNT, SUM, and AVG) do large^/full table scans by definition. They take seconds to run while your goal in a Cloud OLTP system is thousands of requests per second. An automatic sort issued per query in an OLTP system is pathological and represents a vector for a DoS attack.

> What makes a solution like hive much better - I guess its optimized for > this?

Column stores use compressed bitmap indexes that are optimized for scans over a small number of columns. Hive is SQL over Hadoop, and is inherently slow but it does offload the processing from your Production OLTP server. Hive supports the RCFile format which is partially column oriented. The ORC file format is fully column oriented, replaces RCFile format, but requires Presto (or equivalent). Hive is brownfield for existing Hadoop clusters but it has no place in a discussion about greenfield architecture other than discussing historical systems.

If you have a need for GROUP BY style analytics, a true column store like Presto, Impala, or RedShift is a necessity.

^EDIT: based on zbentley's comment

<comment><USER37> > OLAP queries (GROUP BY with aggregate functions like COUNT, SUM, and AVG) do > full table scans by definition

Isn't it only a full table scan if your query isn't otherwise filtered? Those functions have to read every row of "something", but that something might not always be a whole table.

<comment><USER36> Very true, I've edited my comment. Some GROUP BY queries are inexpensive and are fine in an OLTP system if the WHERE clause restriction limits the result set size, most however, are meant to scan a large number of rows.

<comment><USER35> Thank you - this was insightful.

</comment></comment></comment></comment></comment><comment><USER38> Systems Design is a formidable topic because it can go in so many directions. This is a good guide. Might I also recommend something that includes caching, partitioning, indexing, and NoSQL vs SQL. More detail can be found here [https://www.educative.io/courses/grokking-the-system- design-...](https://www.educative.io/courses/grokking-the-system-design- interview)

<comment><USER39> I remember using the course you have linked to some time ago - it was very useful. I noticed now that the course is sold on a subscription model. I don't remember being that the case. Would you happen to know when that changed?

<comment><USER38> I don't. I came across this in 2018 and recall the course being for sale for $80.

</comment></comment></comment><comment><USER40> > We're considering either splitting up our cron jobs into multiple servers, > or setting up a new system using a modern tool like Kubernetes.

:facepalm: beanstalkd can handle a pretty massive amount of jobs before you need to start worrying about scale. I love the huge jump from simple to complex.

<comment><USER37> To be fair they offered a simple option (splitting crontabs) as well.

</comment></comment><comment><USER41> Really like the prose/Adventure-like writing style of this "tutorial"!

</comment><comment><USER42> > database engines that are quick at small queries are typically unacceptably > slow of answering giant queries

Even with proper indexing? I haven't seen this issue with Postgres but maybe I wasn't working on large enough data sets.

</comment><comment><USER43> off topic: I like the simplistic design of this website. Can someone please help me understand if this is hosted on WordPress or something like that?

<comment><USER44> It might be Jekyll according to [https://robertheaton.com/2014/07/26/lessons- from-a-surprisin...](https://robertheaton.com/2014/07/26/lessons-from-a- surprisingly-successful-blog/)

</comment></comment><comment><USER45> This is amazing. Thank you Robert.

</comment><comment><USER46> Great post! Thanks for sharing.

</comment><comment><USER47> Great post!

</comment><comment><USER48> anyone else just getting a blank page here?

</comment><comment><USER49> no DDD?</comment>


How to answer 23 of the most common interview questions  - <AUTHOR> http://www.bspcn.com/2007/10/06/how-to-answer-23-of-the-most-common-interview-questions/ <comment><USER2> I would say some of those are red flags about the company itself... especially "Are you willing to put the interests of X Company ahead of your own?". I would probably answer that with "Depends, is X Company willing to put my interests ahead of its own?"

</comment><comment><USER3> Reminds me of my friend from my old school telling me the WORST interview question he has ever been asked:

Tell me a joke.</comment>


Uber Valued at More Than $50B - <AUTHOR> http://www.wsj.com/article_email/uber-valued-at-more-than-50-billion-1438367457-lMyQjAxMTI1MDMzMTczNjE3Wj <comment><USER2> This may be just me..but I feel like Uber is being built up way too much. People outside SF see it as some shining beacon of Silicon Valley tech startups. If they make missteps and show weakness I wonder if it will shake people's perception of how strong the tech industry is now.

<comment><USER3> I remember following Groupon closely and feeling like Groupon was spending a bunch of money teaching people around the world how to use daily deals, only to have competitors come in make it a race to the bottom because the barriers to entry were so low.

I feel like Uber is doing the same: spending a bunch of money teaching people how to hail a cab from their phone. When competitors like Lyft, Hailo, and Via enter the market and compete with them, it starts to look like the same race- to-the-bottom that happened with Groupon.

(Last I heard, both Via and Lyft Line are cheaper than UberPool right now in Manhattan. I suspect both are taking a loss on many rides to compete.)

<comment><USER4> You also have existing taxi companies to compete against. Way before Uber was a thing, my preferred company in Stockholm had a simple hailing app, it was basically one button for "I want a cab, here, now". No tracking, no payments, nothing fancy, but still.

These days, Taxi Stockholm has a hailing app that is on par with Uber's app, and I just can't see how Uber can compete with that. I can use Taxi Stockholm and get a professional cab driver in a nice luxury car that is insured, regulated, cctv'd and has a trusted brand name, or I can use Uber and get a non-professional in a regular car, no taxi license plate, unknown insurance situation, for maybe a cheaper price.

Uber is fantastically successful in places where the existing taxi market sucks for whatever reasons, but there are plenty of places where it doesn't suck, and it's very questionable what Uber can bring to the table then. Or Lyft, Hailo, etc. Any technology they make can be easily copied by their competitors, but local vehicle fleets can't.

<comment><USER5> > nice luxury car

Are the cabs in Stockholm actually nice?

<comment><USER4> Compared to SF, oh yes, much better! The most common taxi in Stockholm is probably a Mercedes, and the average car of the kind of people who would want to drive for Uber is definitely not. I don't see how Uber could mobilize a comparable fleet to compete with that. I can see UberPool working where people expect a "regular" car, and I could see them becoming the dispatch of choice for non-affiliated taxi drivers, but that's the bottom of the barrel of that market.

If people think that taxi markets everywhere look like the one in SF, then I understand why people value Uber so highly and think they'll conquer every market easily. But the truth is that Uber doesn't have anything that isn't easily replicatable by entrenched taxi companies that are willing.

</comment></comment></comment></comment></comment><comment><USER6> I love when the press accelerates startup success, even among the most successful companies of all time. They want everything to be an overnight success.

There is no timeline where Uber could be justified as being 5 years old. For example, there is a Techcrunch article [1] about the company launching the initial product over 5 years ago (which I'm sure took some time to develop!). Uber's website says the company existed in 2009, along with Travis' LinkedIn profile.

And then they spend a good chunk of the article comparing the speed that the two companies reach different valuations. I wonder what arbitrary date they picked as the date Facebook was founded...

1: [http://techcrunch.com/2010/07/05/ubercab-takes-the-hassle- ou...](http://techcrunch.com/2010/07/05/ubercab-takes-the-hassle-out-of- booking-a-car-service/)

2: [https://www.uber.com/about](https://www.uber.com/about)

<comment><USER7> Slightly confused; are you saying they should say they're more than 5 years old? 2009 was 5 years ago, and if that's when the company was founded, saying they're 5 years old makes sense.

<comment><USER8> 2014: 1 year ago

2013: 2 years ago

2012: 3 years ago

2011: 4 years ago

2010: 5 years ago

<comment><USER9> > 2010: 5 years ago

Not necessarily.

Date diff today (2015-07-31) and (2014-09-01) is 4 years, 10 months, and 30 days.

</comment></comment></comment></comment><comment><USER10> Wow.. I was curious if the pending employment law debates and changes were going to impact their valuation, it appears not.

<comment><USER11> it was way overblown, the details ignored in the anti-Uber cacophony.

This ruling only effects California and a single driver, as explained by an article in Forbes: [http://www.forbes.com/sites/quora/2015/06/30/the- california-...](http://www.forbes.com/sites/quora/2015/06/30/the-california- ruling-on-uber-drivers-would-be-disastrous-but-its-probably-an-aberration/)

California did not rule that Uber drivers are employees. Rather, a single labor commissioner made that finding (not a ruling) in an informal, non- binding hearing that is based on the facts of the specific driver's circumstances, applies only to that driver, is likely to be appealed, and is of no precedential effect. Other commissioners in other circumstances have found the opposite

<comment><USER10> It's clear that there are coming changes to the contractor / on-demand economy and how labor is treated, the value of Uber is likely a proxy for the depth of the changes that are expected.

</comment></comment></comment><comment><USER12> Uber's revenue numbers are awful. _" According to Bloomberg, Uber had revenue of $415 million (we assume net revenue after payments to drivers, not gross revenue) and an operating loss of $470 million."_ That's for 2014, and it's leaked data. But it's not good. Why are they losing money? They're an app and a server farm, some "driver centers", and an admin staff. It's not like Amazon, with giant warehouses full of expensive inventory. They're already scaled out; the expansion phase is over.

[1] [http://www.businessinsider.com/uber-leaked-financials- look-u...](http://www.businessinsider.com/uber-leaked-financials-look- ugly-2015-6)

<comment><USER13> They're losing money because they are aggressively expanding into new markets.

 _They 're an app and a server farm, some "driver centers", and an admin staff._

And, you know, a _lot_ of cars and drivers. Of course Uber would want to make sure we all remember they are _not_ Uber property and employees, but they still have to spend a lot of money to get them on board.

<comment><USER14> What is their remaining market? Are there many countries/cities that still have easily disruptible taxi markets (e.g. "Medallions" or similar)?

<comment><USER13> 4 hours ago:

Uber invests $1 billion in Indian market

[http://money.cnn.com/2015/07/31/technology/uber- india-1-bill...](http://money.cnn.com/2015/07/31/technology/uber- india-1-billion/)

Yeah, there are some big markets out there yet.

<comment><USER14> Yes, but a market being big doesn't mean there is a lot of low hanging fruit in that market. In a market with a regulated number of cabs it's easy to see how uber can find market share (like in NYC). In a market without medallions or similar, where there is already competition among taxi companies making margins slim, I don't see how Uber can be so disruptive. Having an app, dynamic pricing, and abusing the workforce should be obvious steps already taken by existing taxi companies in those markets!

</comment></comment></comment></comment></comment><comment><USER15> Didn't somebody do the math and came up with $40B for entire taxi business globally?

<comment><USER16> Uber is best positioned to usher in self-driving cars. If they are successful at it, their market opportunity is much larger than the Taxi-cab industry.

<comment><AUTHOR> > Uber is best positioned to usher in self-driving cars.

Fleet management is all about buying, parking, filling up, maintaining, cleaning and eventually selling the vehicles once they ran their course. Why would Uber be better at all those things?

<comment><USER16> Economies of scale that it can drive from its platform given that it will have the largest base of customers as well as suppliers. It is very close to what Amazon did to bookstores and eventually to ecommerce business.

<comment><AUTHOR> Some of that doesn't scale rapidly. If they need to buy a thousand car washes nationwide, an owner in Omaha is not going to sell at a discount just because they already own 10 car washes in Los Angeles. Similarly with pricing for gas/electricity.

Their economies of scale are akin to those of Avis or Enterprise Rent-a-Car, and those companies are not valued at high multiples, due to high upfront capital requirements, insurance burden, fast depreciation, low barrier to entry, etc.

Current multiples for Uber are there precisely because of the lack of all liabilities associated with car ownership and maintenance.

<comment><USER16> I would be surprised if the prevalent economic models will continue to apply. Think Walmart, Costco, Amazon and how they leveraged their scale to drive efficiency through the ecosystem.

The prevailing models during those times were revisited which included prices, contracts, payments.

In your specific example about car washes, I envision a huge warehouse with multiple floors to do automated car washes which is super efficient. This will happen as there will be a metric that will show how a minute saved on the car wash floor can save the company millions of dollars because of their scale. There will be no "car wash" dealership to negotiate with.

</comment></comment></comment></comment></comment></comment><comment><USER17> With this much funding can Lyft survive? Uber and Lyft have to be hemorrhaging a ton of money so maybe Uber will be able to out last them?

<comment><USER13> That's the depressing part of this. It's simply a race for who can raise and spend money the fastest. And yes, Lyft will lose.

</comment></comment><comment><USER18> Looking at it as it is, this can seem ridiculous. But, considering their domestic and international expansion it really isn't. Uber can pretty much replace cabs worldwide. When Uber has its driver API service fleshed out, you can expect tons of companies to be able to tap into that and essentially get anything delivered beyond Uber Eats.

</comment><comment><USER19> Just one year ago, a NYU finance professor calculated Uber's risk-adjusted value to be 5.9 Billion. ([http://fivethirtyeight.com/features/uber-isnt- worth-17-billi...](http://fivethirtyeight.com/features/uber-isnt- worth-17-billion/)). I wonder what he would say now...

<comment><USER20> His analysis used the size of the current taxi market as the ceiling for Uber's revenue. In San Francisco Uber is already generating 3x as much as the traditional taxi market was.

<comment><USER5> I know many people (myself included) who never used or would have considered using taxis but now use uber all the time.

</comment></comment></comment><comment><USER21> I feel so unintelligent. I try my hardest to understand these valuations and I'm failing miserably.

<comment><USER16> We don't have access to their pitch deck - market size, roadmap and traction. If we were privy to that info, am sure some of this would make more sense.

</comment></comment><comment><USER22> I wonder how far their valuation can go before they go public.

$100B, $200B, $400B?

<comment><USER11> Easily $100 billion. The momentum is too strong.

</comment></comment><comment><USER23> This seems ridiculous.

<comment><USER11> Looking back since 2008 or so, with the possible exception of groupon, I can't think of a single $5+ billion dollar web 2.0 company that has crashed and burned.

A significant number of these web 2.0 companies seem to have defied the 'crash and burn' that characterized the first boom. Pets.com and Webvan were pretty much failures from inception, but sites like Pinterest and Airbnb keep getting more successful and more valuable.

<comment><USER24> Yelp is not doing so well. They were valued at $5B, but since going public it's not been good: [http://www.cnet.com/news/yelp-is-in-a-death-spiral- industry-...](http://www.cnet.com/news/yelp-is-in-a-death-spiral-industry- expert-says/) Their stock dropped 28% this week on poor earnings.

</comment></comment></comment><comment><USER25> It'll be worth even more in the coming years, which is not hard to fathom when you look at Uber as a logistics company rather than a taxi-replacement.

<comment><USER26> I agree. I know people who crunch data for Uber and some of things they're doing are really valuable. All these rides - pickups, dropoffs, times of day, routes, etc. And this is just the tip of the iceberg what they're doing. These kinds of basic projects inspire more complex analysis I'm sure.

The information gathered will allow them to move into new phases of development. The cab hailing service is not the end game here.

</comment></comment><comment><USER27> I'm glad to see their success. Keep going, Uber! Thank you for changing the way I use taxi.

</comment><comment><USER28> I'd hate to be Uber right now. The early innovators always bear the full cost of discovery and paving the way for future entrants who will gladly waltz right in once Uber establishes itself in cities by lobbying and fighting regulation. The barrier to entry is not as high they make it seem and like Groupon, a focus on future potential earnings without facing realities, almost always ends up in favor of late market entrants. I recall Peter Thiel talking about this in more detail how you should never be the first at anything and that if you can't build a monopoly you will face the same fate that other first movers have met.

</comment><comment><USER29> Wonder when we'll see the first $1T startup valuation. Probably not too far away.

<comment><USER25> I don't think any business that achieves that sort of valuation should be considered a "startup".</comment></comment>


Marriot discloses 500M guests data breach - <AUTHOR> https://www.washingtonpost.com/business/2018/11/30/marriott-discloses-massive-data-breach-impacting-million-guests/ <comment><USER2> This is a pretty funny followup to yesterday's thread about how bad Starwood/Marriott got after the merger.

<comment><AUTHOR> At least Yahoo! still holds the record for the number of affected users.</comment></comment>


The Lambda Calculus: What is it and why should you care? - <AUTHOR> http://zeroturnaround.com/rebellabs/what-is-lambda-calculus-and-why-should-you-care/#!/

<comment><AUTHOR> Total tangential point: I noticed something while reading this otherwise good explanation. I can't speak for others, but for me it explains why I often had a tough time with math in school and possibly also why pure functional languages have had slow uptake.

The thing I hate about math (and mathematical functional coding by proxy) is _how it is explained_.

This article contains an absolutely perfect example. Go to the heading entitled "Church Numerals," which explains how numerals are declared as functions. Above we've been introduced to lambda function notation, which is fine. Now he throws this at you:

0 = λs.λz. z the function s is applied to the argument z zero times

1 = λs.λz. s z the function s is applied once

2 = λs.λz. s (s z) the function s is applied twice

 _What the bleep is 's'?!?!?_

I spent several minutes trying to figure that out. What is s? (bang head on table) What is 's'? (bang head on table more)

OOOOOOOHHHH... there it is in _tiny f'ing print_ :

"(the names s and z are short for successor and zero)"

It's in tiny print so you won't see it, because knowing what symbols mean evidently is not important to understanding them.

Mathematicians do this habitually. Every math class I attended in college would consist of a mathematician who threw symbols and notations at you, proceeded to do stuff with them, and then _maybe_ got around to defining them after he'd already expected you to get it. Maybe. If he feels like it.

It's like teaching a foreign language by beginning with poetry reading, progressing to literary theory, and then oh by the way -- as an unimportant aside -- visiting the topic of words' meanings and their grammar.

Why do mathematicians do this? Are they working to make their subject impossible to explain? Or is it some weird artifact of how the mathematically- inclined think-- do they think nonlinearly and so do not grasp that people learn things by building up ideas hierarchically?

If math were taught by beginning with symbols and operators and their conceptual meaning and then progressing to their use, I think people would have far less of a problem with it. It would be like learning any other language, followed by learning the "literature" that is expressed in that language.

Edit: I sometimes see programming languages explained this way too. Some code will be thrown at you, followed by some higher-order language concepts, and then oh by the way the symbols % means ____ in this language. The example, of course, was utterly incomprehensible prior to that little detail.

<comment><USER2> > Why do mathematicians do this?

My other FU-money idea is to fork Wikipedia, then take the 100,000 most important STEM articles and give them i) rigorous fact checking & ii) Including an introductory paragraph that explains the concept to anyone. (At least, anyone with an IQ of about 100 who's prepared to put a bit of work in).

EDIT: This could be tricky, because we sometimes lie when introducing a subject just so that people can get their heads around it. See, for example, the speed of light, and then refraction.

<comment><AUTHOR> "Okay, we're going to start with a simplified version of this concept first. Keep in mind that the reality is somewhat more difficult to grasp, but grasping this simplified and idealized version will help you to eventually grasp the real deal."

No problem with that.

But out of order explanations are unforgivable. I did get better in math when I figured this out and learned to find the meaning of the symbols first. But believe it or not, sometimes this wasn't easy. Sometimes I would encounter some esoteric mathematical notation and have to spend quite a bit of time digging through books or searching the net to even find a place where _anyone_ bothered to explain what it meant.

By "what it meant" I mean what it meant _conceptually_. A bunch of jiu-jitsu of symbol manipulation is completely worthless without a conceptual understanding of what that manipulation is actually _doing_.

My first aha here was with calculus. After struggling really badly in high school and then college, someone finally told me that a derivative was a rate of change. After that, I completely got it and had no further problems with derivatives, integrals, or differential equations. It all made perfect sense. I also completely understood why calculus was invented, and its historical importance. The book did say a derivative was "rise over run," which I suppose is a maximally-obtuse way of saying "rate of change," but it never clicked.

How in the world do these people perpetuate their discipline?

But back to functional programming. I taught myself to program when I was four years old. It was BASIC on the Commodore VIC-20. In retrospect I think it was easy for me even as a 4-year-old because everything was explained in terms of "this statement does X," and then the more complex variant of "this statement makes the next statement conditional on..." I got that because that's how things work. Things work by, umm, doing stuff with stuff.

I think this is why procedural languages are so much easier to grasp. Procedural languages are explained in terms of what they do with stuff, which is how people think because that's how things happen in the real world.

Functional languages on the other hand tend to be introduced in this obtuse, theoretical, mathy way, with symbols and expressions thrown out before their function or conceptual meaning is adequately explained.

Introduce FP like this: language, semantics, and meaning first, then concepts, then immediately dive into _doing things_ with it. Try that and I think FP will get more popular.</comment></comment></comment>


Modern Android is pretty secure - <AUTHOR> https://palone.blog/#post-your-android-might-be-the-most-secure-device-you-own-4 <comment><USER2> Sadly I don't think Android as a platform is less secure. but still, in comparison to iOS (the common comparison) or desktop platforms. Most phones stops getting security updates earlier than the other platforms.

I had a Nexus 4, Nexus 7, Nexus 5X. All which should get top notch security updates. stopped where other platforms kept getting updates.

Just a few days ago I saw on HN some great news for Android ([https://news.ycombinator.com/item?id=23692257](https://news.ycombinator.com/item?id=23692257)). but again, this isn't a security update. It a broader security improvement equivalent to a major update on other platform.

While my 1st iPhone SE will get iOS 14, my Nexus 5X is still on Android 8.

<comment><USER3> this is exactly why Apple still rock at phones

you may choose Android for the specs, guess what, your phone will still get deprecated in 2 years

<comment><USER4> Hmm. My $170 phone from the middle of 2014 is still working perfectly and shows no signs of being "deprecated". Every application I use gets frequent updates, including the browser. I don't have to worry about targeted attacks (as I am not a billionaire/movie star/CEO of a large company/whatever), and a new shiny iPhone won't help with those anyway. µBlock Origin (with JS disabled on most sites) protects me from everything else.

<comment><AUTHOR> Yeah the security patches are still worth thinking about. There was a Media Framework bug that needed nothing but a PNG on your phone to get RCE and there was a bug with the bluetooth stack this year which also could achieve RCE when your bluetooth was turned on and an attacker knew the BSSID

PNGs are easy to send and in times of Contact Tracing via Bluetooth... well there are certain risks to running unpatched

</comment></comment></comment></comment><comment><USER5> It might be "secure", but almost every app is exfiltrating my personal information by using crashlytics, facebook and co.

Who needs security when the apps do it by design.

Check your firewall logs, or use a root/no-root firewall, it's frightening.

Edit: click the screenshots at [https://play.google.com/store/apps/details?id=eu.faircode.ne...](https://play.google.com/store/apps/details?id=eu.faircode.netguard&hl=en_US) to see an example. Fourth picture "Access attempts"

<comment><USER6> Amen. My guess would be that 90% of Hacker News readers use a desktop browser ad blocker, but <10% have a tool like Blockada installed on their phone.

<comment><USER7> 10% seems low. On Android Firefox supports extension so a lot of people are having exact same extension on desktop and mobile.

<comment><USER8> I even got uMatrix.

</comment></comment></comment></comment><comment><USER9> The Open Source nature of Android might have something to do with it as well. It is interesting to compare iOS and Android in terms of how visible the source code is and how much vulnerable a system can be in effect.

Here's an interesting article that goes in depth about the concept: [https://dwheeler.com/secure-programs/Secure-Programs- HOWTO/o...](https://dwheeler.com/secure-programs/Secure-Programs-HOWTO/open- source-security.html)

<comment><USER10> Android is less open source than it used to be. I wonder what percent of the vulnerabilities in Android are in the open-source portions (kernel, system libraries) vs the framework/apps.

<comment><USER9> I think it is all listed here: [https://android.googlesource.com/](https://android.googlesource.com/)

It's interesting to see how they call out repositories specifically for different manufacturers (ex. Samsung, Sony).

</comment></comment></comment><comment><USER11> Meta but having the full post in a modal feels... off.

<comment><USER12> There's something about a SPA for a blog that feels completely wrong...

<comment><AUTHOR> You can happily browse the noJS version at wp.palone.blog

<comment><USER11> This should be your default :)

<comment><USER13> The disadvantage of that one is that it doesn't seem to support links for individual posts.

</comment></comment></comment></comment></comment><comment><USER14> I should note that these things can only keep your device secure if they're not buggy, and most platforms offer these features. The difference is quality of implementation and which ones keep them up-to-date when problems are found.

</comment><comment><USER15> This article wholly misses the point. Android is not a secure OS because it enables and allows behaviors that make it leak data and protect the user poorly. Security isn't just a matter of not having memory vulnerabilities or using good encryption.

That's where Google and it's advocates have failed to understand what security is. Security is keeping something safe. And with the amount of data exfiltration Google's default platform plus the apps it allows and encourages do does not do a good job securing your information.

When a user installs an app that does behavior they didn't expect or intend to permit, that's a security issue, even if the platform APIs allowed it.

The difference in how Apple and Google approach Web APIs discussed a couple days ago highlights this: Google added a ton of APIs that lets websites do stuff with your computer, Apple decided they are risky and chooses not to implement them. Google would say the user has to give permission for those APIs to be used, so there's no security flaw if a website uses them maliciously: The user gave permission. Whereas, Apple would recognize the benefits to the user are minimal compared to the risk when the user grants permission unintentionally, which happens all the time for other web APIs like push notifications and even extension installs.

Practical security and technical security are two different things. Google does not even _comprehend_ the former, while receiving wide accolades for their expertise at the latter, as in this article.

<comment><USER16> > Android is not a secure OS because it enables and allows behaviors that make > it leak data and protect the user poorly.

I am going to bite. You are just biased. Google adds more APIs because it doesn't have much interest in limiting some features to Play store, because Play is a such small part of their revenue. Apple would benefit if things are only possible through native apps because sweet 30% rent.

That said Google isn't exactly dumb but they aren't well organised in terms of Android - some areas get focus while others not. They even made it harder for external contributers of Android Open Source Project by following Google style monorepo patterns.

Contrast to Apple where mobile OS is their core business and they have to do it well.

</comment></comment><comment><USER17> Now if only I could get the Pixel (or generally clean Android) software experience on Samsung hardware....

<comment><USER18> Oneplus?

<comment><USER19> OnePlus is a terrible company. I purchased one of their phones, I had some issues with the device within the 30 day return period. It was a popular issue, I think it was ghost touch but it has been a while. Anyway, in order to get a refund I had to do a charge back with my CC company.

I am pretty confident my next phone will be an Apple device. I was an Android fan, purchased all google/nexus devices starting with the G1. I had problems with many of those devices. I have switched to Samsung, but the amount of garbage on the phone still annoys me. Therefore, it is time to try iOS.

<comment><USER7> Oneplus had some occasional hickups in the support department but I don't think their hardware is significantly better or worse than anyone else.

What I really like about OnePlus is their software. It's hands down the best Android version ever created. Love their additions to vanilla OS.

<comment><USER20> OnePlus has a history of breaking basic Android security features. For example:

\- [https://www.xda-developers.com/oneplus-6-bootloader- protecti...](https://www.xda-developers.com/oneplus-6-bootloader-protection- exploit-physical-access/)

\- [https://www.xda-developers.com/two-critical- oneplus-33t-boot...](https://www.xda-developers.com/two-critical- oneplus-33t-bootloader-security-flaws-discovered-one-patched-and-other-being- addressed/)

While OxygenOS is smooth and not bloated and their phones are cheaper than other flagships (in some markets at least), their security record isn't the best.

<comment><USER7> Not saying they are without faults, but if that's all you get on them then they are pretty damn secure.

<comment><USER20> The first link is about a flaw that allowed the OS to be replaced without the user knowing. It's the opposite of "pretty damn secure".

And they had other security issues, both with their phones and servers. I'm sure you can find more info with a quick search.

OnePlus makes good phones, but this thread is about Android security... and they don't have the best record.

</comment></comment></comment></comment></comment></comment></comment><comment><USER21> I am NOT a security export, just an observer, but saying "Contrary to popular belief, modern Android is pretty secure." (Which is how this post finishes) sounds pretty much like something I'd agree with, but saying "Your Android might be the most secure device you own" isn't really something that I'd agree with as much.

I don't even know how one would really measure these things in a reliable and objective way. Do we compare it to IOS or Windows or ChromeOS or MacOS? Which version of Android and so on.

The post picks out the best things about Android, nothing wrong with that, but isn't that just ignoring all the problems with Android? I agree though, Android is pretty secure, but so is everything else.

<comment><AUTHOR> Hm guess I subconsciously chose an clickbaity title...

Well in my case, I use Debian as my main operating system on my Desktop. Debian is great, but as a classical desktop OS it does not have those cool security things like app sandboxing, permissions management (camera acces etc) and stuff like that. Which makes sense, since mobile Operating Systems were designed from the ground up way later and also seem to move at a faster pace. The fact alone that almostt all ANdroid Apps run in the JVM and are written in memory safe languages is a huge plus that the GNU/Linux family does not have. (Solely speaking about security, I wouldnt want my /bin/sh to be in java)

In my particular case, it suprised me to find out that my android is the most secure device I own. Thus the title (and the intention to wrote this post).

So yeah, misleading title. Kinda. My Bad. Updated the title

<comment><USER22> The big difference between Debian and Android is that with Debian you don't generally want or need to run untrusted or untrustworthy 3rd party code. Instead your software comes from trusted source with maintenance promise and security updates. And I'm starting to think that is better model than trying to sandbox and lock down everything; the world needs more trust at certain places. In contrast you are no way able to trust random Android apps, neither the quality or the intentions, so you are left with actively hostile relationship between you, the OS, and the apps.

<comment><AUTHOR> I'm saving this. Very good point

</comment></comment></comment></comment><comment><USER23> I feel that most OS (Windows, Mac, Linux, iOS, Android) have continued to improve security but at the same time reduced privacy.

If that's true, is there a TailsOS equivalent for phones?

<comment><USER24> Hard to do I think. Even plain old "linux but for phones" is pretty difficult at this point because:

\--most drivers and hardware specifications are proprietary, and probably secret under NDA

\--most bootloaders are cryptographically locked and controlled by vendor

\--necessary wifi and cellular modem hardware is the same, and are also patent minefields even in the foundational platonic ideals of design, as is mobile graphics hardware

\--the modems are subject to regulatory requirements that they be secured from modification by the user/owner of the device

Secure in this context means secure from the user and device owner, which can arguably be for good reason -- think of an ATM kiosk, for example.

So no "tails for phones" yet, but people are trying. Check out postmarketOS, lineageOS, replicant, sailfish. Last I tried things were still kind of science project, like 90's style linux.

<comment><USER25> Adding ubports to the list- I have it on my Pinephone and can place and receive calls, send and receive SMS, and utilize GPS. There are some features that don't yet work on Pinephone like MMS but it's on the threshold of being daily-driver worthy.

<comment><USER24> Sounds great. Is the battery life OK? Was it hard to get the phone?

To be clear, I see some measure of compromise as totally reasonable if one's goal is to get on an open os...

<comment><USER25> Users on Pine64.org have reported their devices can run 14 hours on idle and my own experience is the battery will last all day with moderate usage. Getting it on the phone is as simple as getting Raspbian on a Raspberry Pi, you just flash the image onto an SD card.

</comment></comment></comment></comment></comment><comment><USER26> AS long as you do not start installing apps, and avoid using some vendor pre- installed apps.

</comment><comment><USER27> Maybe a pure Android phone is secure but the reality of most phones is a dozen of system and store apps deliberately spying on their users.

</comment><comment><USER28> I think people are more concerned about the data Google and apps mine from their devices than the type of encryption.

</comment><comment><USER29> Then it should not allow apps to ask for permissions they don't need. Just a thought..

</comment><comment><USER30> HN title is misleading clickbait.

The article says that modern Android security is pretty good, listing a number of Android's security features and recent improvements, but the author never tries to make the case that it's "the most secure device you own".

There's also zero comparison between Android and any other operating system that would be required to support the HN title's conclusion.

Also, Android isn't a device, it's an operating system. An operating system is only as good as the devices it's put on; a deeply insecure device isn't going to magically be made better just be installing Android on it.

<comment><USER31> It's the author's own headline

<comment><USER30> The headline was updated since I wrote that comment.

The original headline was "Android might be the most secure device you own".

<comment><AUTHOR> Yeah guess the mods did that or something. I just updated the original title :)

Read here: [https://news.ycombinator.com/item?id=23714416#23716962](https://news.ycombinator.com/item?id=23714416#23716962)

</comment></comment></comment></comment><comment><USER32> Random nitpick: I wish each post has what I'll call "full page scrolling". You have to hover your mouse over the actual post to scroll it or you end up scrolling the background.

<comment><AUTHOR> I'll keep that in mind and will probably fix this... Someday

</comment></comment><comment><USER33> Learned something today. Android is a device.

I always thought it was an os by Google based on aosp(also Google..).</comment>


Continuous Intrusion – Why CI tools are an attacker's best friends (2015) [pdf] - <AUTHOR> https://www.blackhat.com/docs/eu-15/materials/eu-15-Mittal-Continuous-Intrusion-Why-CI-Tools-Are-An-Attackers-Best-Friend.pdf <comment><USER2> I have used Jenkins, TeamCity, Bamboo, and GoCD. As with any tool, misconfiguration and lack of ops involvement is going to lead to clueless implementations. One might as well say "I granted everyone admin access, and they are able to run any command now".

For all these tools, there is documentation on how to lock down access to the server, and also how to run the agents under various users accounts. On my current engagement, for e.g., I ensure that the various GoCD agents run as low privileged users, with access granted via sudo.

Further, running commands directly is easy. But a better pattern is to push these commands into a script, to check in the script into version control, and to then check out and run that script. At least GoCD is opinionated that way by enforcing that users use "materials" (which could be a checkout from a revision control system).

GoCD enforces something called a "sandbox", where the file references are restricted to only the working directory for that particular job. If one needs to access files outside of this sandbox, then one needs to explicitly invoke a shell - after which point the admin/script owner is responsible for what happens. This is because the GoCD admin/script owner permitted either a command interpreter to be invoked directly, or the script owner did so within their script.

I now-a-days deliver scripts as Ansible playbooks using GPG signed RPMs, the execute the Ansible playbooks in local mode.

The non-salted SHA1 passwords is a good report. GoCD lets uses specify either an htpasswd generated file, or an LDAP source. On enterprise implementations, I always link GoCD to an LDAPs source and manage the credentials there.

In short, if one doesn't read the documentation and just implements distributed orchestrators with zero ops sensibilities, then such WTFs of being able to execute any command would of course be possible.

</comment><comment><USER3> This horrifically written document alleges that CI tools are risky because if they fall they grant a vector to other parts of infrastructure. That's also true for just about any machine... even a developer laptop.

Good CI/CD processes increase not only security (less crufty servers lying around) but also recovery procedures because they force you to be able to deploy... to new, external, backup or refreshed infrastructure, if necessary.

<comment><USER4> This talk is written in a Black Hat idiom: take trendy technology, perform basic risk assessment, find one or more exploitable software security mistakes, generate "catchy" title.

It's unlikely that the author really believes CI tools are so risky that you should avoid them. On the other hand, it's good to know that your CI infrastructure is part of your attack surface; chances are, the author came up with the idea for this talk after exploiting a CI system on a real pentest (I know that happened at Matasano a couple times).

</comment></comment><comment><USER5> This is why you keep your CI networks in a private VLAN with proxy servers back to the public Internet. You don't just stick your CI servers on your corporate network -- you park them behind a jump server and proxy back anything you need to.

For every project I've set up CI for, it doesn't matter if someone hacks my CI servers because they don't have access to anything other than dummy data. Sure; you _could_ hack prod systems by building a trojan version of one of the binaries. But even then the binaries just get uploaded to a repository; so with proper code profiling and security testing you would notice the rogue binary before it ever hits a prod server.

It is important to keep CI as a dev process -- deployment should be handled by an orchestration system as it's much more of a tech ops process than a dev process. This is where config management and the like become very important; and those should be left to people who think like operations and security managers rather than hackers who just want to make shit work quickly (there's a place for both inside any good dev org).

<comment><USER6> > if someone hacks my CI servers because they don't have access to anything > other than dummy data.

And all your source code?

</comment></comment><comment><USER7> Owning someone's CI system via the CI system is all well and good, but not surprising in the slightest. All these things expose web-based admin interfaces, so of course they're vulnerable at the very least to social engineering.

The real trick is owning someone's CI system via the code under test. And it's even more versatile because it works even for projects which keep their executor scripts checked in to source.

1) Check out and gain an understanding of the target's CI scripts

2) Create a pull request (or whatever) which makes some reasonable code change but also introduces a vulnerability into the build/test scripts themselves. For example, curl'ing a domain you control.

3) Kick off pre-commit runs of all the tests to see if your patch should be accepted.

4) Own all the devices testing your patch.

<comment><USER8> Even when you have control of a project tested by the CI server you should not have control of the CI server itself.

GitLab comes with integrated CI. But all the tests are ran by GitLab Runner on external machines. You can spin up (and down) a new machines per test with Runner Autoscale if you want.

As far as I could see GitLab CI is not vulnerable to the items mentioned in the article (apart from only enforcing password length, not complexity).

But CI security is hard so if you find anything please let us know [https://about.gitlab.com/disclosure/](https://about.gitlab.com/disclosure/)

<comment><USER7> Yes, sandboxing and ephemeral machines are far and away the best defense a CI system has against compiling/testing malicious code. But the vast majority of CI systems don't actually use ephemeral executors because imaging between runs is slow and rebuilding your whole checkout/object cache is expensive.

You don't need to own the CI server in order to own a project. You can take over all of the individual executors and replace their linker with your own version that links against an attacker-supplied backdoored fork of openssl (for an extreme and topical example). Doesn't even require constantly-running code which might be detected by a monitoring system, and naturally persists across reboots (but not reimagings).

I'm glad that GitLab is paying attention to these things. The Travis which is tightly integrated with GitHub also seems to do a good job, although I haven't looked into it extensively. But plenty of orgs run self-hosted CI that simply isn't up to proper security standards in this respect.

<comment><USER8> Rebuilding is expensive. We have several caching mechanisms but there is no silver bullet and caching itself can lead to security problems.

For a deeper review of security problems we're currently focussed on please see [https://gitlab.com/gitlab-org/gitlab- ce/issues/21911](https://gitlab.com/gitlab-org/gitlab-ce/issues/21911) and please feel free to contribute there.

<comment><USER9> For a discussion on caching and artifacts, please see [https://gitlab.com/gitlab-org/gitlab- ce/issues/21913](https://gitlab.com/gitlab-org/gitlab-ce/issues/21913).

</comment></comment></comment></comment></comment><comment><USER10> In case you would rather watch the talk than read the PDF: [https://www.youtube.com/watch?v=0H6jd5yG7_A&list=PLH15HpR5qR...](https://www.youtube.com/watch?v=0H6jd5yG7_A&list=PLH15HpR5qRsVd9jdTWUB5s16fmr2VCQyS&index=22)

</comment><comment><USER11> Every place I've worked, devs have complete access to dev resources like databases, CI servers, web servers and usually have access to QA resources and it's only by convention that we don't screw with it without asking for permission.

Why would just anyone have access to production resources - which would be required to take advantage of any of the things he stated.

Also, dev/QA should be on a separate domain anyway.

</comment><comment><USER12> I'm kind of shocked all y'all commentators have such well secured CI systems. ;) I've been developing software for 17 years, and at every employer, from 2 to 20,000 employees, CI systems were treated as an afterthought in terms of infrastructure and security resources.

It was dramatically obvious how bad this was from a security perspective, so perhaps this article isn't necessary, but certainly it's right on the money.

<comment><USER13> We have a CI tool that I've been trying to secure for a long time. Usually when I add something (SSL so people can't sniff passwords / hijack sessions) the response I get from some others is "why bother? If something bad happens the person who does it will be fired!"

It baffles me really. Most of the time I do it anyway. But we still have a few major holes I've been slowly patching. There's one last glaring problem: permissions are controlled by teams in an external tool. Anyone has auth to change teams in said tool.

</comment></comment><comment><USER14> I'm surprised the article focused on compromising the build farm instead of danger to the production. I guess a malicious agent could alter the behavior of the build that makes it to production and attempt to slide in an extra "feature" like weakened security or back door.

Nevertheless, it's a good reminder to consider build servers as part of the attack surface.

</comment><comment><USER15> We use CircleCI, and since all our environments are on a VPN we opted for a pull based solution instead of CircleCI notifying/pushing anything to our servers.

The way it works, Circle pushes the containers built to Google Container Registry and there is a super simple tool I built ([https://github.com/crufter/puller](https://github.com/crufter/puller)) pulls those images. This introduces a couple of minutes of delay but we find that acceptable.

I guess we are still exposed to the possibility of someone breaking into Circle, using the keys there to push to our GCR repos, but at least they won't have full access to our servers...

<comment><USER16> What's the difference, if they can initiate a pull to load whatever code they want onto your server?

<comment><USER15> Having code running in a container is still harder to exploit than just having full ssh access to the boxes?

I am not a security expert by any means, and this workflow was not created with security as a priority in any means, but the latter still rubs me in a worse way. It might be just an ignorant hunch though.

<comment><USER17> You might as well drop privileges and tell the kernel to disallow any future privileges even with sudo/su or vulnerabilities.

You can do this by using oneway in your docker entry point

[https://github.com/muayyad- alsadi/oneway/blob/master/README....](https://github.com/muayyad- alsadi/oneway/blob/master/README.md)

<comment><USER15> That's something really interesting, and it comes handy for a side project I have, but I don't think it fixes the problem with people hijacking the docker repo - they will just change the Dockerfile itself, right?

<comment><USER17> Yes I agree. There are some things you can do for example you can have client certificates, only trusted clients can talk to your ci, and onlt trusted clients can push images. Only dev with ssh keys can push code.

Your ci and registry are locked down (a container).

All that might help.

</comment></comment></comment></comment></comment></comment><comment><USER18> My CI server is inside an amazon VPC, accessible via an ssh tunnel. Only certain people can access the production VPC and everyone's keys have a strong passphrase ... why trust your CI tool's security when you can rely on much more robust things like SSH?

<comment><USER19> It is very important to have defense in depth, and having good perimeter (your VPC) security as well as internal security (CI config.)

</comment></comment><comment><USER20> Since a build server in its most generic form is just a thing to run commands, this isn't very surprising. I hoped for more details on how to secure the deployment part because that's less trivial (access to prod machines and prod service credentials)

</comment><comment><USER21> Cracking a well-automated CI infrastructure is equivalent to gaining root in production. :-)

My phrase is that it's the Nerve Center of the company. Don't let it get popped.

</comment><comment><USER22> Speaking of vulnerabilities and intrusion, pdf is a can of worms. I wish there were less pdf links...</comment>


The Next CEO of Stack Overflow - <AUTHOR> https://www.joelonsoftware.com/2019/03/28/the-next-ceo-of-stack-overflow/ <comment><USER2> Like many developers, I have benefit a lot from Stack Overflow over the years. It has helped me become a better developer and, in answering questions, a better communicator.

I just want to say thanks to Joel for helping make this thing happen and best of luck in whatever is next for you!

</comment><comment><USER3> Related discussion/submission on [https://news.ycombinator.com/item?id=19512332](https://news.ycombinator.com/item?id=19512332)

<comment><USER4> This discussion really resonates with me. It seems that stackoverflow incentivizes very bad moderation: shutting down on-topic questions and good answers, locking questions prematurely, etc.. I'm not sure what causes bad answers to be voted to the top, but that seems like a common problem as well. And as commenters have noted, it is almost pointless for a newcomer to try to contribute.

These issues look very hard to fix, so I wish them luck. For now, I've given up on stackoverflow.</comment></comment>


Tech Overload: Palo Alto Battles Silicon Valley's Spread - <AUTHOR> http://www.wsj.com/articles/tech-overload-palo-alto-battles-silicon-valleys-spread-1473780974 <comment><AUTHOR> "Mr. Burt, the mayor, caught many off guard in May when he said that he believed the zoning code says software companies aren't allowed in the downtown. In recent weeks, he said he wants to update that code, allowing for software but restricting companies from taking large swaths of space, like Palantir and Amazon have done, together occupying hundreds of thousands of square feet."

As a young person who has moved to Palo Alto to work at a "software company" I find this sentiment very frustrating.</comment>