Researchers crack open malware that hid for 5 years - <AUTHOR> http://arstechnica.com/security/2016/08/researchers-crack-open-unusually-advanced-malware-that-hid-for-5-years/ <comment><USER2> Interesting regarding USB devices. When US DoD systems were infected with a virus someone brought from home on a USB stick, I remember hearing there were going around filling USB ports with epoxy. There was some method behind the madness I guess.

There is also a market for routers and other devices which are produced as much as possible in US (are they rolling their own capacitors I am wondering...). I saw some of those devices come with a 100x markup. $400 device from China vs $40k from US. Those who sell the $40k know how hard it is get on the list they are milking it for all its worth (the sales person was quite frank about it).

It was also funny to see "Windows" as an approved security blessed OS and then Debian, Ubuntu, OpenBSD rejected (with only ancient version of RHEL's approved for Linux).

<comment><USER3> Too many people without security clearance can access and modify Linux. In any real security environment, open-source is poison. Period, end of story.

<comment><USER4> ^ As if any amount of security clearance can erase human fallibility.

In any real security environment, humans are poison. Period, end of story. This is not an issue exclusive to open-source.

<comment><USER5> Of course not, but it's exacerbated by open source. Humans are indeed poison, which is why it's normal to involve as few humans as possible. Unfortunately AI is not sufficiently advanced so you still need some of them in order to get the job done.

</comment></comment></comment></comment><comment><USER6> What is the role of an InfoSec professional in an environment where advanced threats like this are being deployed? I mean, a beat cop knows when it's time to call the FBI or the military. But the open nature of the Net means that firewall probes by script kiddies are interspersed with intrusions by nation- state actors. It's a weird state of affairs.

<comment><USER7> Work with stakeholders to minimize lateral movement after a breach, get monitoring in place to detect breaches, and have a response plan.

If you have company critical secrets or life-safety systems, you need to air gap where possible.

That's you're job. You cannot stop or prevent attacks, and if option don't have the metrics and logs, the FBI won't be able to do anything, assuming you can get them to give a shit.

</comment></comment><comment><USER8> > The researchers went on to speculate that the project was funded by a > nation-state, but they stopped short of saying which one.

So ... does anyone, perhaps who doesn't have Kaspersky's business interests to protect, care to actually speculate? In other cases it's been seemingly well- known in the security community which APT attacks trace back to which countries, it's just apparently impolite to say it in public.

<comment><USER9> Russia, Iran, Rwanda... Let's assume the latter is a vector, not the target. (The attacker is sophisticated enough that we can assume Rwanda itself is of little interest). Rwanda also has fairly close ties to Russia, which strengthens the vector hypothesis.

Russia+Iran suggests a western actor. Their biggest shared interest is Syria, I'd think. And look, the Syrian conflict is on since March '11, and the activity according to Kaspersky reaches back to June '11\. I'd say that's quite close.

Neither the US nor Europe were _that_ deeply invested in Syria. There is, however, one small middle-east country that has quite an interest in the entire region, and also isn't friends with Iran or Russia. And it just so happens that Israel is somewhat of a close affiliation of Rwanda.

None of that is in any way conclusive, but it certainly is probable.

<comment><USER10> > and also isn't friends with Iran or Russia

Actually Israel is on very friendly terms with Russia. There are a ton of Russian immigrants in Israel to the point that Putin called them "Russian ambassadors".

It didn't start that way - part of the history of the creation of the modern state of Israel is Russia vs US proxy conflict (of sorts) via Egypt. But it's not like that anymore, not for a long time.

(The US and Russia have moved on to other proxies :)

<comment><USER11> While I'm not a proponent of the idea that Israel is behind this malware, I disagree with you. Israel is not on particularly good terms with Russia. Yes, the two governments have established a hotline to ensure that Russian military maneuvers in Syria are not misinterpreted, but the two countries are closer to foes with a mutually accepted cold peace. It's in the strategic interests of each country not to be outwardly hostile to one another, but they're definitely adversaries in many respects.

</comment></comment></comment></comment><comment><USER6> Apple's walled garden has been subjected to criticism from open source advocates. And Windows 10's telemetry triggers a lot of privacy concerns, too.

But in our current security environment, what if these walls become necessary for secure computing? By analogy, there's a reason that many ancient cities were circled by a wall.

<comment><USER12> > By analogy, there's a reason that many ancient cities were circled by a > wall.

Walls around cities were likely very poor at stopping small, stealthy groups of infiltrators. They were designed for much more brute force attacks. Apple's walled garden helps quite a bit with the deluge of crap that would be available without it. Without it there would be an order of magnitude more crap (in quantity and quality). That said, there's a vibrant black market for people that can't stand the oppressive policies of Apple.

Additionally, once you have a wall in place, it's easy to make a decision to tax certain types of traffic through it because the capability is now there, whether or not it's in the best economic interest of the people inside. Apple didn't skimp on this area. The wall was erected with tithes and taxation in mind, and collection booths at all the gates.

So, does it help? Well, it prevents roving bands of bandits from riding in, terrorizing and robbing the people unlucky enough to be in their path, and making a hasty exit, so yes, but if you're a tasty enough target, getting past the wall isn't really a problem. There are myriad ways to do that as long as you're careful. For example, the numerous secret tunnels through the wall. They aren't large, and they are constantly being filled in by the city engineers, but there's always some they haven't found if you are willing to ask the right people (or dig your own).

Okay, I believe I've tortured this analogy enough...

<comment><USER13> Devil's advocate, walls and the enablement of taxation also centralized capital and enabled cities to spend it on public works that might not have been built otherwise (and before I get the "then they shouldn't!" retort, I think we can all agree there are shared infrastructure resources that w/couldn't be built by private actors).

In a world where all phones are loosely controlled Android derivates competing on slim profit margins, is anyone going to make the drive for hard hardware- enabled crypto? And even if they wanted to, could they afford it?

<comment><USER12> > Devil's advocate, walls and the enablement of taxation

Sure. I wasn't making a case that taxation at the wall _is_ bad, but that it has the _capability_ to be bad. We use regulation in (mostly) free markets to greater or lesser success to steer the markets in some manner. If you accept that _pure_ capitalism doesn't necessarily yield an optimally performing system when people are involved, then that ability to influence the market is a useful capability, especially when applied judiciously. A blanket rate isn't necessarily the most efficient form of that, but it is a way to raise revenue.

> In a world where all phones are loosely controlled Android derivates

I think you've already stacked the starting conditions to the point where it's not really worthwhile to consider. That situation would be ripe for disruption in some manner, because I think it's inherently unstable. All it takes is a small niche market for alternatives that _do_ make choices based on privacy, or security, and events that spur interest in those topics, and the larger population of providers will need to respond appropriately or risk ceding a increasingly large portion of the market to those that do.

<comment><USER13> I think the outcome of the first generation of smartphone OS's has (surprisingly for me at least) shown that there's really only room for a handful of players (Android/AChina, iOS) with sufficient numbers of users to be self-sustaining.

As you note, not sure a unipolar outcome would ever be stable enough to have persisted, but I wouldn't have expected a bipolar arrangement either. And I can imagine a market structure that would have depressed manufacturer profits far enough so as to preclude serious R&D / innovation on their parts.

<comment><USER12> You know, it's common enough to have one dominant player in a marker, a small few chasing players, and then a bunch of very niche players that I'm there's a lot of economic theory behind it that I'm unaware of. It probably relies quite a bit on how invested in the product you are once you've decided on it, but there are plenty of examples throughout history[1],

1: [http://images.dailytech.com/nimage/Smartphone_Market_Share_2...](http://images.dailytech.com/nimage/Smartphone_Market_Share_2007_Through_2011.png)

<comment><USER14> Let's keep this in mind the next time we get the idea to let market forces regulate, say, school systems or infrastructure.

</comment></comment></comment></comment></comment></comment></comment><comment><USER15> I'm curious: How realistic is building malware like this? Is this something that has been done out in the open by researchers? Is there an example we can see, or is this all still rumors?

The reason I ask is because there's actually value in spreading the rumor that a capability like this exists. Imagine if your adversary believed that you could gain access to their computers even when they're not connected to the internet. They'd run themselves in circles trying to secure everything!

In general, I believe this article to be true, but would love to learn more of the details.

<comment><USER16> From Kaspersky Lab's analysis:

>What would ProjectSauron have cost to set up and run?

>Kaspersky Lab has no exact data on this, but estimates that the development and operation of ProjectSauron is likely to have required several specialist teams and a budget probably running into millions of dollars.

[https://securelist.com/analysis/publications/75533/faq- the-p...](https://securelist.com/analysis/publications/75533/faq-the- projectsauron-apt/)

</comment></comment><comment><USER17> That is a really impressive piece of software. USB exfiltration of data on air gapped machines is next level. I'm in awe of their skill.

<comment><USER18> If your machine has a USB port, it's no longer properly isolated.

Obviously that's a _tremendous_ pain to work with, because you're limited to PS/2 keyboards and mice (etc etc), but given that there's no way of authenticating USB devices and they've already been used in various attacks, a serious airgap protocol has to ban USB ports.

You could quite easily hide a USB mass storage device inside a mouse, or with a bit more work have an unmodified mouse with a spare Flash area used for data exfiltration.

(Firewire is even worse, and Thunderbolt lets you onto the PCI bus)

<comment><USER19> If you just leave away the USB mass storage kernel module when compiling the kernel, the mass storage device won't work anymore while the mouse still works. I wonder if this is a solution to this problem or not since it seems quite naive.

<comment><USER20> Any USB device gets to be a keyboard and mouse. If it comes down to it, the device could just "type" its payload.

<comment><USER21> And if that malware can't liberate USB access, and still needs to read data (rather than just writing it), it could exploit the capacity for various devices to emit detectable EM radiation. The fake keyboard/mouse, being inside the Faraday cage, would be able to sense that radiation and extract data that the malware in its payload sends back to it.

In fact, all of this would work equally well with a PS/2 port.

</comment></comment></comment></comment></comment><comment><USER22> Heh I gave a talk at DefCon Skytalks last week on this exact exfil method and C&C structure with a live demo using code we wrote....interesting.

<comment><USER23> > Heh I gave a talk at DefCon Skytalks last week on this exact exfil method > and C&C structure with a live demo using code we wrote....interesting.

> Kaspersky researchers still aren't sure precisely how the USB-enabled > exfiltration works. The presence of the invisible storage area doesn't in > itself allow attackers to seize control of air-gapped computers. The > researchers suspect the capability is used only in rare cases and requires > use of a zero-day exploit that has yet to be discovered. In all, Project > Sauron is made up of at least 50 modules that can be mixed and matched to > suit the objectives of each individual infection.

You remarkably have the exact exfil method when that's not disclosed information?

<comment><USER22> >The attackers used multiple interesting and unusual techniques, including:

> Data exfiltration and real-time status reporting using DNS requests.

Sorry to be more specific we spoke on DNS Base Exfil using base64 encoded strings in DNS Lookups and also how to use DNS records to control botnets.

So not exact and only part of their method.

</comment></comment></comment><comment><USER24> What criteria is used to determine that malware could only possibly have been made by a nation state? If all it takes is specialist teams and a budget in the millions of dollars (presumably, had it been 10s or 100s of millions, that's what they'd call it), lot's of private entities can pull that together, can't they?

<comment><USER25> They probably could, but would they?

At least in the US, publicly traded tech companies are accountable to shareholders: There's some transparency in the accounting, and it's hard for them to throw millions of dollars at a problem before shareholders start asking tough questions.

<comment><USER24> But a military contractor type of company has lots of obfuscation leeway with "top secret" type of things, doesn't it? And I'd imagine a defence contractor is the type of company that would be interested in the kind of info this kind of malware can gather.

<comment><USER26> There is some evidence that CITIC Group, a Chinese company, is heavily involved in corporate espionage and the manipulation of foreign nations. There's no particular reason that other large companies, no matter their home countries, could not also be engaged in these types of activities.

<comment><USER27> link: [http://www.securityweek.com/chinese-attackers-conduct- cybere...](http://www.securityweek.com/chinese-attackers-conduct- cyberespionage-economic-gain)

</comment></comment></comment></comment></comment><comment><USER28> Is the implication that there must be someone who connects the special USB drives to these air-gapped computers? So the attacker must have local people on the ground.

<comment><USER29> Supposedly, the "drop USB drives in the parking lot" works pretty well to get around air-gapped systems. As well as mailing USB drives to the receptionist, mail room, etc.

Also, this thing was running as a local admin on a domain controller. So either the DC's weren't patched or some zero-days were used. Or perhaps an inside job.

<comment><USER20> > the DC's weren't patched

As I understand it, airgapped systems are not in the habit of bringing software updates across the airgap, so unpatched everything is likely.

<comment><USER29> Its trivial to do with WSUS and off-line updates. But yeah, if its a shit run environment, it will get owned by someone eventually.

<comment><USER30> An air-gapped set of servers performing updates from a non-air-gapped WSUS server are not air-gapped servers at all.

If the WSUS server were also air-gapped, then you're in the business of manually downloading each update, verifying it, and copying it to over to the air-gapped WSUS server offline.

Microsoft's Windows Update servers have also been compromised in the past. Depending on the level of security you're operating at, taking new windows updates on your air-gapped systems may require having someone decompile and review each update.

In general, being air-gapped prevents infinitely more exploits than windows updates could ever possibly cure; that is, until one of your admins uses his admin privileges to disable the USB port restrictions for 5 minutes that one time to copy that one file quickly so he can go home for the day. For this, there are epoxied USB ports.

</comment></comment></comment></comment></comment><comment><USER31> The thing I find interesting is that I bet the authors are reading this.

</comment><comment><USER32> Some at the NSA is having a bad day reading this.

<comment><USER18> Bizarrely, the NSA and other US security agencies seem to have very little interest in defence, preferring surveillance and attack capabilities.

<comment><USER7> That's a false statement. They work with NIST to develop the standards that are the basis of the infosed industry.

<comment><USER33> For the most part, NIST really has no relevance in infosec. With a few exceptions, they're always way behind, and only focus on a few narrow domains.

<comment><USER7> Literally every compliance standard in the US references NIST 800-53.

In terms of the "narrow scope" assertion: [http://csrc.nist.gov/publications/PubsSPs.html](http://csrc.nist.gov/publications/PubsSPs.html)

</comment></comment></comment></comment></comment><comment><USER34> >> so advanced in its design and execution that it could probably have been developed only with the active support of a nation-state

Why is advanced technology automatically assumed to have the backing of nation-states? Cant several highly motivated and smart individuals create the technology without a nation-state behind them?

<comment><USER35> That's why they said "probably".

You need to look at things like the complexity of the malware, how many staff it would take to develop and maintain it operationally, the targets selected and what sort of payloads are executed. Criminals tend to have simpler malware that used known exploits or a small number of zero days. They generally cast a wide net for their targets and their payloads typically aim to directly raise funds(ransom, mining, card theft, etc).

In contrast, nation-states tend to have complex malware with multiple zero days, greater care is taken to avoid detection, their targets are chosen carefully and their payloads focus on gathering information and specialized operations.

</comment></comment><comment><USER36> Some security professionals have expressed the view that insecure endpoints represent a good compromise. That is, without the US government being able to snoop on endpoint devices, encryption would have to be tightly controlled, so that the government could retain intelligence and investigatory capability.

A more cynical view would be that many security firms sell both security and forensics/surveillance. One of those two product lines has to be fundamentally defective.

Is the position that hackable endpoints are a good compromise supportable any longer? Or has it bitten US entities in the ass enough that making truly secure computing a reality for computer users, even if it blinds the surveillance state, becomes the new goal.

<comment><USER37> I've often heard not that "insecure endpoints represent a good compromise" but instead that since:

1\. endpoints are vulnerable because they are exceptionally hard to secure,

2\. and attacking endpoints can be targeted and specific,

the governments case that weakening encryption is necessary for warranted search is weak. Even with strong encryption the government can exploit the targeted communicant's endpoint to learn either the plaintext or the encryption keys. This isn't a compromise so much as a statement of reality and what is likely to remain reality for some time to come. Weakening encryption, for the most part, provides benefits to the government in the form of mass surveillance, but for a variety of reasons doesn't offer much benefit in the form of limited, specific searches.

>making truly secure computing a reality for computer users,

We can make endpoints more secure, but I see no path to endpoint security that will keep out a determined well resourced adversary.

<comment><USER36> You get what you pay for. Right now, endpoint systems are undefended, even intentionally compromised. The design of endpoint systems assumes all components can be trusted. But those components don't usually undergo testing for vulnerabilities and hidden capabilities.

</comment></comment></comment><comment><USER38> Anyone know of an instance of airgapped USB-based exploits for Linux-based systems?

</comment><comment><USER39> What's with all this nonsense about could have "been developed only with the active support of a nation-state"? Do nations suddenly have access to some sort of advanced, alien software development teams?

Feels more like political sabre-rattling to get the public to eventually condone a future attack from our homeland shores of Oceania against the evil Eastasia or Eurasia.

<comment><USER40> I believe it's less about fear mongering and more about understanding the level of sophistication of the software. Talk to anti malware analyst and they'll tell you how commoditized the malware game is nowadays. There's an endless stream of malware and ransomware which can be linked back to just a handful of frameworks. These types of malware families also fall under the spray-n-pray mentality for distribution. Spam, drive-by-downloads, infected torrents, etc.

Compare the mass of malware that is out there with the level of technical sophistication, OPSEC to prevent detection, and precise targeting of its victims. Along with other big name malwares (i.e. Stuxnet, Flame, etc.), this class of malware is very precise in its objective. It isn't trying to make money for its owners. It isn't trying to replicate itself across the internet endlessly. Rather it has a key objective of infecting a specific set of networks. So when researchers call out the fact that it is likely to be "state sponsored", they are saying the purpose of the malware is very different than your average piece of malware.

<comment><USER41> Essentially depending on what malware does we can easily identify government software because criminal software has a different set of objectives. Is it possible though that corporate software could have similar objectives? I'm thinking corporate espionage type behaviour.

<comment><USER40> > Is it possible though that corporate software could have similar objectives? > I'm thinking corporate espionage type behaviour.

Yes, it is possible.

</comment></comment></comment></comment><comment><USER42> Anyone know what IPs the C&C servers used?

Interested to see what hosting company in the US they used.

</comment><comment><USER43> > It was also funny to see "Windows" as an approved security blessed OS and > then Debian, Ubuntu, OpenBSD rejected

Bribes always help.

<comment><USER44> Paying for certification is what's required. Governments require various certifications to sell to them, and that certification costs money in consultancies. RHEL paid for the testing, they get a certification and access to the customer.

It looks like this is probably referring to EAL [1][2].

In a market with a large number of vendors interacting with a large number of relatively unknowledgeable buyers, an oversight team is going to try to find a certification to give guidance (and ass covering).

Yes, this is a barrier to entry, but it's also a learned behaviour as buyers get repeatedly burned.

I would argue that this is equivalent to requiring your plumbers and electricians to be licensed.

[1] [https://en.wikipedia.org/wiki/Evaluation_Assurance_Level](https://en.wikipedia.org/wiki/Evaluation_Assurance_Level) [2] [https://www.redhat.com/en/about/press-releases/red-hat- achie...](https://www.redhat.com/en/about/press-releases/red-hat-achieves-top- security-certification-for-red-hat-enterprise-linux-6)

<comment><USER2> EAL (Common Criteria) and also FIPS-140-2 for crypto.</comment></comment></comment>


Ask HN: How to not get lonely as a solo founder? - <AUTHOR> <comment><USER2> Join a co-working space / incubator / accelerator.

I've joined a co-working space and later on an incubator and it definitely helps

<comment><USER2> Another option is to join YC Startup School, there's a big community of solo founders : [https://www.startupschool.org/](https://www.startupschool.org/)

</comment></comment><comment><USER3> get together with people who have the same problem you have. More and more there are groups of founders in different places around the world that not only support each other emotionally but also with best practices.

<comment><AUTHOR> Like solo founder meetups?</comment></comment>


Looking for MVPs - <AUTHOR> I was just asked to show a MVP to help explain the compromises necessary to make an MVP (c.f. a finished product such as Dropbox).
While I can think of many examples of MVPs that have gone on to do great things I could not think of a single product that was still at the point of being the minimum viable solutions. I am sure lots of people on HN make and use MVPs.
Please list any MVPs you can think of below (ideally ones that charge something for their product): <comment><USER2> I am not sure you will find what you are looking for. An MVP that is on the market will usually not show 'obvious compromises'.

If people are paying for it, then it has to do the thing it does quite well. Minimalism may be visible on the sidelines, and then it will not be evident whether this is due to it being an MVP or an actual strategic decision not to implement something. If the product is clearly missing something then it will probably try to hide it not to deter users.

An example

I work at [https://www.droptica.com](https://www.droptica.com), where we recently created an MVP application [https://www.trainingrealm.com](https://www.trainingrealm.com).

Is this an MVP? Yes. It is the minimum required for the subscription to the app to be sellable - "schedule training & guide you through the training".

But there are no 'nice to haves' like:

1\. No multilanguage support, though clients are from all over.

2\. No notifications and reminders, which seem like an obvious thing to have.

3\. Initially "Cancel account" was a link to a contact form.

4\. To be fair, not all trainings created by TrainingRealm team are at this time available in the app. e.g.

www.run40minutes.com is not, because it is a whole lot of additional functionality that would have to be added for it to make any sense to users. 5\. No native apps, just a PWA, though users as for these all the time.

6\. UX is not yet quite polished, especially for mobile.

7\. A million other things that would be great to have and would increase user engagement and would make the app a greater success.

The above will be added of course, but without these, the app already seems complete. If anything, it may look like poor design or poor implementation here and there, but so do many established products.

If something looks 'really incomplete', then it will not be on the market yet I think.

This is my take at least ;)

<comment><AUTHOR> Awesome. Thanks Macie

</comment></comment><comment><USER3> One of my favorite essays on MVP's, [https://blog.ycombinator.com/minimum- viable-product-process/](https://blog.ycombinator.com/minimum-viable-product- process/)

<comment><AUTHOR> Yeah, it is gold.

</comment></comment><comment><USER4> Inkscape</comment>


Ask HN: Raspberry Pi Server Farm with Load Balancing - <AUTHOR> Hey HN,
I am a Web developer with only rudimentary knowledge of how a server farm is physically designed and maintained. Nevertheless I would like to build a tiny rack filled with about a dozen Raspberry Pi computers (when suppliers allow buying that many at once.) I would like to use each as its own modest server instance, akin to Webfaction, and use another to manage the load balancing across them.
I know this has been considered before, but I would like to ask HN about advice and resources. I have never had to run an application distributed on more than one server so any materials on horizontal scaling, building and managing a small farm would be appreciated. Also, it will need to account for more storage than just a small SD card in each RPi. What's the best way to mount a harddrive RAID into this setup for shared storage? <comment><USER2> I recommend starting from reading the HAProxy site. <http://haproxy.1wt.eu/>

It has a ton of information about building a load balanced infrastructure using HAProxy.

This article helped me too: <http://horms.net/projects/has/html/node6.html>

Read some architecture articles from this site: <http://highscalability.com>

I am also planning on doing that, so once suppliers allow multiple machines I'll write a blog post about I had to do.

</comment><comment><USER3> Aside from the Raspberry Pi being cool, what would this buy you over AWS, a server running some VMs, or some old spare boxes? Getting some experience with load-balanced systems isn't necessarily a bad idea, but the Pi doesn't seem like the best hardware for the job.

Assuming you do want to do this, consider getting one more capable box, and using it as a firewall, load balancer, NFS server and database server. Your Pis can then talk to this machine and do the webby things. (If you want to avoid a single point of failure, use CARP/VRRP, some sensible load balancer, DRBD and database replication. Reboot all Pis on failover, failing over NFS mounts is very painful.)

</comment><comment><USER4> What should the server do? Server static files? Ruby? Python? I think with the latter you'll get into trouble quickly (at least Ruby) in the memory-usage area.

But, given you can run your stack on a single RasPi. Use one to run HaProxy in front of it.

</comment><comment><USER5> I thought of this before myself. From a purely economical point of view, you'd be better of if you grab a couple laptops or thin clients off Ebay. This after considering the power consumption too.</comment>


A Fast, Minimal Memory, Consistent Hash Algorithm - <AUTHOR> http://arxiv.org/abs/1406.2294v1 <comment><USER2> Woah, this jumps out at me immediately since Eric Veach's name is on it. I haven't seen a (relatively) new Eric Veach paper in a long long time.

Some background: Eric Veach is huge in the computer graphics world for laying a ton of the foundations of modern physically based rendering in his PhD thesis [1]. He then went on to work for Pixar and did a ton of work on Renderman (for which he recently got an Academy Award), and then in the early 2000ish left Pixar to go work for Google, where he was the lead on developing AdWords [2]. In short, he's had quite a career, and seeing a new paper from him is always interesting.

[1] [https://graphics.stanford.edu/papers/veach_thesis/](https://graphics.stanford.edu/papers/veach_thesis/) [2] [http://www.bloomberg.com/bw/stories/2006-03-05/the-secret- to...](http://www.bloomberg.com/bw/stories/2006-03-05/the-secret-to-googles- success)

<comment><USER3> Slight correction: his sci-tech award was for MIS and "efficient Monte Carlo path tracing for image synthesis", which have only been in PRMan for the last two releases (PRMan 18 added geo area lights with MIS evaluation, and 19 added pure path-tracing), so his award wasn't for his work at Pixar really as that happened 12+ years before the techniques the award was for made it into PRMan.

</comment></comment><comment><USER4> Smaller standard error, significantly less memory, faster execution, equivalent rebalance cost... what's not to like?

Thanks, Google:

> Google has not applied for patent protection for this algorithm, and, as of > this writing, has no plans to. Rather, it wishes to contribute this > algorithm to the community.

<comment><USER5> Except that's useless. Realistically you'd need a legal patent clause, because right now Google can change its stance on patenting it at any moment, and make your code violate the patent.

<comment><USER6> I was taught that patents can't acquired on discoveries that have already been made public. If that's the case Google's publication of this method before attempting to patent it makes it unpatentable.

<comment><USER7> There is a 1 year grace period between disclosure and the deadline for patentability.

[http://www.uspto.gov/web/offices/pac/mpep/s2133.html](http://www.uspto.gov/web/offices/pac/mpep/s2133.html)

<comment><USER8> Grace periods are shorter or non-existent outside the US. Publication date is June 9 of last year, so Google has less than 3 months to change their minds and turn-evil to pursue a patent on this technique.

</comment></comment></comment></comment></comment><comment><USER9> That's a nifty algorithm. Sillysaurus3's wonderful explanation from last year helped me figure out what's happening. Mostly, at least (my fault, not his):

[https://news.ycombinator.com/item?id=8140385](https://news.ycombinator.com/item?id=8140385)

It feels like there should be some faster way of doing the jump calculation that doesn't require the floating point divide. Relatively, it's a slow and expensive operation, and doing ln(n) of them on each bucket lookup would add up to a lot if you have lots of buckets.

<comment><USER10> His explanation has mistakes. For example, in step 1 he claims "After this multiplication, the first 4 bytes and the last 4 bytes of 'key' are nonzero"

Take any 8 byte value b, multiply by 16133697096952638549ULL (mod 2^64) to get a value for the key. Now key * 2862933555777941757ULL = b. Thus there is an input value for key that I can make hit any number. The first and last 4 bytes are thus certainly not always nonzero.

He then uses "The fact that the first 4 bytes become nonzero is important for the next step" and proceeds. This is wrong.

For a concrete example, suppose key is 16133697096952638549ULL. Then key * 2862933555777941757ULL = 1. The first 7 bytes (actually, the first 63 bits) are all zero. Adding 1 as in the code still leaves the first 62 bits zero.

(the value 16133697096952638549ULL is the multiplicative inverse of the constant in the algorithm mod 2^64).

</comment></comment><comment><USER11> This has already been discussed here:

[https://news.ycombinator.com/item?id=8136408](https://news.ycombinator.com/item?id=8136408)

It's been in guava for a few years now.

</comment><comment><USER12> HRW worth looking into as well if you have a smaller number of buckets: [http://en.wikipedia.org/wiki/Rendezvous_hashing](http://en.wikipedia.org/wiki/Rendezvous_hashing)

</comment><comment><USER13> How revolutionary is this? Will it replace all preceeding hash algos or does it have a narrow usefulness?

<comment><USER8> It's useful in the field of consistent-hashing: mapping items over shards where the number of shards varies over time. (Only some systems do this, and most hash algorithms are used in other fields.)

This technique also requires the shards be assigned contiguous integer ID starting at 0, and appears to work best if all shard entry/exit is intentional and at the end of the ID range. (That is: if you have 15 shards, they're named #0-#14. If you decide you need one less, you turn off shard #14.)

It doesn't appear to work well for systems with more chaotic entry/exit. (Shard #4 just crashed, and you don't need it anymore? You'd still want to restart/fill one shard as #4, then shut-down/drain shard #14. Other consistent-hashes could just map #4's old duties among the remaining nodes, albeit with their higher space/time costs.)

<comment><USER13> Would something like a network packet queueing algorithm, which hashes each stream and iterates through them (ex. Stochastic Fair Queue), benefit from this hashing algorithm?

<comment><USER8> Do those algorithms already make use of consistent hashing? If so, yes, but otherwise: probably not.

</comment></comment></comment></comment><comment><USER14> This is a nice algorithm and I thought about writing something that only depended on numbers and not on allocating nodes in a ring.

It works well when adding nodes, but then how does it work when removing a node in the middle? When allocating nodes in a ring it's easy, just reassign the vnodes. But what with this algorithm? Can anybody explain how does it keep track of which nodes take control of which part of the removed node?

</comment><comment><USER15> The paper was really interesting to read, it motivated me to implement it in Go: [https://github.com/beefsack/go-jch](https://github.com/beefsack/go-jch)

<comment><USER16> Same here :) even did a blog post about it [http://www.creapptives.com/post/113786047187/improving- shard...](http://www.creapptives.com/post/113786047187/improving-shard-re- balancing-with-jump-consistent)</comment></comment>


Let's Encrypt, OAuth2, and Kubernetes Dashboards - <AUTHOR> https://blog.n1analytics.com/oauth2-lets-encrypt-and-k8s/ <comment><USER2> I'm really just writing up what I did so the internet can tell me what I did wrong...

</comment><comment><USER3> go brian!</comment>


Flight or fight? United's 're-accommodation' of passenger a colossal mistake - <AUTHOR> http://www.theage.com.au/business/aviation/flight-or-fight-uniteds-reaccommodation-of-passenger-a-colossal-mistake-20170410-gvi842.html <comment><AUTHOR> Seems to me that the reason this sort of thing happens is that in the 21st century there is a general expectation from people in power/authority that they must be obeyed, and that "common people/citizens" who do not obey have crossed authority and can be dealt with in virtually any manner they choose, including physical violence.

This perception has formed because governments have moved more and more towards this manner of interacting with "common people".

When I see this, it seems to be precisely the same behaviour as when the police beat people up for questioning their authority at any level.

In summary: "Well he didn't do what he was told, so what did he expect?"

<comment><USER3> Agreed, violence as a response to disobeying law enforcement has been normalized in the US, regardless of context/circumstances.

De-escalation and common sense are more often replaced by brute force.

<comment><AUTHOR> But in this case, it's violence in response to not obeying a company - <USER2> someone purchased a service from.

</comment></comment></comment><comment><USER4> [https://www.united.com/web/en-US/content/contract-of- carriag...](https://www.united.com/web/en-US/content/contract-of- carriage.aspx?Mobile=1#sec25)

<comment><USER5> From Rule 25:

"When there is an Oversold UA flight that originates in the U.S.A. or Canada"

This flight was not oversold. The passengers had been gate-checked and embarked[0], ready for departure. There was no shortage of revenue seats.

However the airline realised that it needed to position a crew to another destination and to save a few dollars opted to remove four fare-paying passengers from the flight to make space for their non-rev crew.

Regardless of that, it does show that when one party ( United ) subcontracts to another ( Republic Airways ) in order to save money, the blowback of adverse PR is onto the main brand and rightly so.

[0] technically they weren't yet "boarded" because the doors were open, which interestingly also means that the aircraft wasn't "in flight" and therefore the airline can't appeal to concerns of flight safety.

</comment></comment><comment><USER6> It seems that security officers manhandled the passenger. But I wonder if they knew why they were removing the passenger. If a flight crew asks the security crew to remove a passenger who is refusing to leave, i'd imagine they would try what ever they can. I am not making an excuse for any of them, especially for harming someone.

Airport, from the security checks to being in an airplane has been turning into a nightmare process for some. I hope things get better and not worse progressing forward.

<comment><USER7> I wonder if the security people were operating under a general expectation of concluding such an action within a certain amount of time, or if they were ordered in this instance to conclude within a certain amount of time.

I also wonder if the plane actually was "overbooked," or if the need to get four crew-passengers on the plane was due to a sudden and unforeseen event. Claiming "overbooked" may give the airline some advantage that not overbooked would not. No idea what the rules for overbooked vs not overbooked are in this case, it just seems weird that exactly four people needed to leave for these four crew members, and not, say, five because of a real overbooking.

</comment></comment><comment><USER8> United dies gave a plan for choosing people, based on ticket class and time if checking (among other things) so no, they would not have asked someone in first class. But they don't have a procedure for removing someone from the plane, at least not for this reason. As far as I can tell all of their procedures deal with people before they get on the plane. They do have a list of reasons to declare someone, but over booking is not one of them.</comment>


Keep your IBAN secret, it could be easily abused - <AUTHOR> https://www.approach.be/en/keep-your-iban-secret-it-could-be-easily-abused.html <comment><USER3> This isn't a problem. You're not "paying" by inputting your IBAN, you're "allowing" them to take the money out of your account. This worked pre-IBAN in most EU countries.

The burden or proof is on the seller, in this case Amazon. If they take money out of your account for an order you did not make, just tell your bank - <USER2> money will be back in your account in no time and Amazon will, if that happens more often, get a problem with their bank.

Here's another "security risk": at many shops, you can order and pay once you get the goods. And just like this "critical flaw", it's pretty useless because Amazon will just let law enforcement know and they will come knocking at your door.

</comment><comment><USER4> How is it possible to keep it secret? Doesn't everyone put it into their invoices?</comment>


"We would like to apologize to the user for the error on our part this morning" - <AUTHOR> https://newsroom.tiktok.com/en-us/an-update-on-recent-content-and-account-questions <comment><USER2> You've gotta love the PR bullshit of it all. Someone intentionally chose to write the headline as:

> TikTok: An update on recent content and account questions

And not something more honest like:

> TikTok: An apology about a content moderation mishap

This is done to imply they did nothing wrong at the first glance, and then concede they did something wrong but will do better later on in the article.

It is also convenient that they first draw attention to the author of the video being associated with:

> (1) Terrorism or terrorist imagery, (2) Child exploitation, (3) Spam or > similar malicious content.

And then immediately after concede that the video was removed because they messed up:

> Due to a human moderation error, the viral video from November 23 was > removed. [...] it should not have been removed.

If companies want to know why they can't regain trust and aren't given the benefit of the doubt, it's because they cannot be honest in situations like this. PR in the 21st century is about communicating how you have in fact messed up because you are humans, not trying to gloss over it and distract from it like some corporate overlord.

<comment><USER3> I don't think this certainty about this being "PR bullshit" is warranted at all, and feels a bit like knee-jerk cynicism.

The actions as described in the press release seem to me both plausible and reasonable. Banning multiaccounts for severe infractions makes sense, and it also makes sense that this account could have been caught up in that.

As for the video takedown... you may underestimate the consistency of human moderators. They are often under intense time pressures and mistakes are made all the time. There is also the potential for individual moderators to apply their own bias to the report outside of company policy. With a 50 mins reversal of the takedown, that seems plausible.

If they are not outright lying, it does seem like the title is accurate and not at all misleading. It does sound like a moderation mishap.

<comment><USER4> If I recall, the video was not taken down. Her account was still active so people could see the video, but her access to the account was blocked for a month.

It's ironic because TikTok was getting a lot of flak in the news recently because they were too slow to remove actual terrorist propaganda. So they probably went overboard and it resulted in this situation.

Damned if they do, and damned if they don't.

<comment><USER5> Want to add to this. Tiktok blatantly violates Google's device ID policy, and actually admits it in writing.

> We share your device ID with measurement companies so that we can link your > activity on the Platform with your activity on other websites;

My contact at google said a year ago when I asked if they know about Alibaba's device ID abuse for advertising targeting, it's their policy to not to touch Chinese companies' device ID abuse.

</comment></comment></comment></comment><comment><USER6> Was taking a look at what pings this website was sending (this is probably the first time I've ever been on tiktok.com)

Sent some pings to:

[http://byteoversea.com/](http://byteoversea.com/) [http://ibytedtos.com/](http://ibytedtos.com/)

Not sure what either of them are for sure, but from first look it seems like both are monitoring scripts (see [https://sf16-muse-va.ibytedtos.com/obj/ttfe- maliva/slardar/p...](https://sf16-muse-va.ibytedtos.com/obj/ttfe- maliva/slardar/plugins/behavior.a77dd6d7.js)) , but will do a deep dive soon. Interesting they need this just for a simple press release.

Also notice the person posting is explicitly from "TikTok US" and not just "TikTok". Interesting decision.

<comment><USER7> Also if you root your Android phone, wireshark their app, trace the traffic, and check the Manifest you'll see they are sending user's personal identifiers (non-resettable hardware IDs) to servers in China. There's absolutely no way why uploading this data (or reading it in the first place) would be in any way necessary to provide the app's functionality.

And they dump/read files in shared storage spaces to cross-tracking with other Chinese apps. If you try to install it into Secure Folder on Samsung it gets disabled.

<comment><USER8> To do exactly what they say in the post. Associate alt accounts with a main account that has been banned.

<comment><USER7> No, it's still not ok to track hardware IDs even (especially) for that reason: [https://developer.android.com/training/articles/user-data- id...](https://developer.android.com/training/articles/user-data-ids)

<comment><USER9> The most damning thing your link says is that it's not a best practice.

<comment><USER10> The best fit is probably the "Anti-fraud: Enforcing free content limits and detecting Sybil attacks" use case, since that is about preventing a single user from creating multiple accounts. (Although the content limit for a banned account is to stop it from _producing_ more content, not consuming it.)

The official recommendation:

 _Use: Instance ID or GUID. On Android 8.0 (API level 26) and higher, SSAID is also an option, as it 's scoped to the app-signing key._

 _Why this recommendation?_

 _Using a GUID or Instance ID forces the user to reinstall the app in order to circumvent the content limits, which is a sufficient burden to deter most people. If this isn 't sufficient protection, Android provides a DRM API, which can be used to limit access to content, includes a per-APK identifier, the Widevine ID._

So I guess best practice in this case means they should use DRM instead.

</comment></comment></comment></comment></comment></comment><comment><USER11> Ah, so the Chinese company banning the person critical of Chinese brutality was entirely coincidental.

<comment><USER12> The article doesn't seem to say it's coincidental. The person previously made a video containing a picture of Osama bin Laden. There's probably a positive correlation between people who make videos containing pictures of Osama bin Laden and people who make videos critical of Chinese brutality. So if you ban everyone in the first group, you're likley to ban some people in the second group also.

<comment><USER13> > The person previously made a video containing a picture of Osama bin Laden.

Does this mean it would be ok to ban all evening news programs?

<comment><USER12> I didn't say that at all. I never said it's ok to ban people who show a picture of Osama bin Laden, nor that evening news shows should be prevented from showing pictures of Osama bin Laden.

</comment></comment></comment></comment><comment><USER14> It doesn't even mention the possibility of political motivations. Nor does it mention the concept of freedom of speech. I think it says all we need to know about the lack of values from this company.

<comment><USER15> >I think it says all we need to know about the lack of values from this company.

that it lacks the same values 90% of all other tech companies lack, but they don't put free speech into the PR article?

<comment><USER16> I can't even imagine what kind of "values declaration" a Chinese company, under the scrutiny of the CCP, could possibly hope to make. Free speech? Transparency? Privacy protection? Honesty and integrity? Any one of these is clearly a lie given the country that they're based in.

I don't know what the excuse of those other tech companies outside China could be. I guess when you move fast and break things, some of the things being broken are promises.

<comment><USER17> Values declarations are easy, if there is no one to enforce them Let's look at the Chinese constitution:

[https://www.usconstitution.net/china.html#Article35](https://www.usconstitution.net/china.html#Article35)

> Article 1. Socialist state

> The People's Republic of China is a socialist state under the people's > democratic dictatorship led by the working class and based on the alliance > of workers and peasants.

> Article 2. Power belongs to the people

> All power in the People's Republic of China belongs to the people.

> Article 35. Freedom of speech, press, assembly

> Citizens of the People's Republic of China enjoy freedom of speech, of the > press, of assembly, of association, of procession and of demonstration.

> Article 37. Freedom of person

> Unlawful detention or deprivation or restriction of citizens' freedom of the > person by other means is prohibited, and unlawful search of the person of > citizens is prohibited.

> Article 39. Inviolability of the home

The residences of citizens of the People's Republic of China are inviolable. Unlawful search of, or intrusion into, a citizen's residence is prohibited.

<comment><USER18> Is "unlawful" there translated from a word with similar connotations in Chinese? Because "unlawful [search, etc] is prohibited" is...a bit tautological.

<comment><USER19> I don't read a lot of legal documents, but maybe it could also be translated as "unwarranted" or "not legally sanctioned".

</comment></comment></comment></comment></comment></comment><comment><USER20> Post seems above board, but as soon as i learned tiktok is chinese moderated, i deleted my account. Not interested.

<comment><USER21> Yeah... The CCP aren't really known for their tolerance.

</comment></comment><comment><USER22> Uhm... So they're tracking individual devices instead of accounts?

<comment> tim-- If you try installing TikTok on a Samsung Android device inside Secure Folder, your account will get banned pretty quickly. Same for WeChat, too.

<comment><USER23> Why?

<comment> tim-- I'm not sure, there was a ban screen that showed up, saying that I was using unofficial software to log into my account, and required a self-service unblock process to be completed, which included getting a number of WeChat users to verify/vouch for my account.

The specific reason given by WeChat was that I was using an Android emulator (such as Andy or Youwave) and was sent to this page[0]

[0] [https://help.wechat.com/cgi-bin/micromsg- bin/oshelpcenter?t=...](https://help.wechat.com/cgi-bin/micromsg- bin/oshelpcenter?t=help_center/topic_detail&opcode=2&lang=en&plat=android&id=120813euejvf150504mfan7b)

</comment></comment></comment></comment><comment><USER24> This actually seems like a pretty reasonable and detailed response to me. The content was down for 50 minutes before their internal checks reinstated it. Probably a much better and more transparent response than any of the other socials would provide.

</comment><comment><USER25> I chuckled at Hasan Minhaj's (obviously at least semi-facetious) take on TikTok being a Chinese company during one of his stand-ups. The gist of it was, basically, what if all of TikTok's user base was suddenly doing their usual viral moves, but wearing something that says "I stand with Hong Kong", for example. That'd be bound to send CCP for a loop…

</comment><comment><USER26> Anyone know why TikTok and Douyin are bifurcated domains of the same app? They want to separate their domestic and international data mining efforts to minimize pre-processing or something?

<comment><USER27> It's the only way to be a Chinese company and have a international audience. Douyin fully complies with Chinese regulators (i.e. censors) while TikTok is content managed separately.

It's the same reason why the wechat that exists in the Chinese internet is different from the international wechat.

</comment></comment><comment><USER28> Feels like the silly focus on process buries the lead on the video ([https://vm.tiktok.com/9hL3uY/](https://vm.tiktok.com/9hL3uY/))

That is brilliant awareness building.

</comment><comment><USER29> i liked the transition from eye lash etiquette to chinese concentration camps... well played

</comment><comment><USER30> so, is there any platform, that's both free, easy to use for non-techies and is just that, a platform, not a publisher? I'm fine with a company removing stuff when the law absolutely requires them to, but not in any other case.

<comment><USER31> Blogger.com?

Techies can use Keybase or GitHub pages with something like Pelican for Python.

</comment></comment><comment><USER32> I think the best way for America to deal with Tik Tok is to ban the app from the American App Store. This will allow current users to continue to use it + have their data collected while stymieing the growth of the app/cancer. India already restricted it due to child abuse, and Tik Tok leaders failed to show up at congressional hearings.

<comment><USER33> On what grounds would America ban it? It's not like other social networks don't censor stuff.

</comment></comment><comment><USER34> Judging from the account names, that user just wanted to collect karma by posting non-OC on tiktok?

</comment><comment><USER35> And the fact that the video will never be available on douyin.

</comment><comment><USER36> > TikTok user posted a video that included the image of Osama bin Laden, > resulting in an account ban in line with TikTok's policies against content > that includes imagery related to terrorist figures.

The chinese government has started to classify protestors of Hong Kong as terrorists. What good are these policies where the core definition of the law itself is left for interpretation by the Chinese government?

The whole enterprise of China and their principles are just... absurd. The elephant in the room is complete authoritarian control by a uncontestable leader, but let's bikeshed about company policies. What company policies, if they only rest on the stilts built by the CCP?

<comment><USER37> Because clearly only West is allowed to decide who is terrorist and who is not.

<comment><USER36> Now, that I think of it - yes. You're right. I can't trust the Chinese government at all. To the CCP, anyone who criticizes the government is a potential terrorist.

<comment><USER38> What is Edward Snowden, then? Will he be greeted with open arms when returning to the USA, or immediately end up in a black site? He criticized the real government of the USA, not the sham you see on news media.

<comment><USER39> What about the 1-3 million people in Chinese concentration camps (excuse me, "re-education camps")?

<comment><USER40> 1-3 million? That seems like a pretty big range of uncertainty. What are your sources?

<comment><USER39> This estimate gives 1.5 million [1], this gives 1-3 million. [2][3]

[1] [http://www.jpolrisk.com/brainwashing-police-guards-and- coerc...](http://www.jpolrisk.com/brainwashing-police-guards-and-coercive- internment-evidence-from-chinese-government-documents-about-the-nature-and- extent-of-xinjiangs-vocational-training-internment-camps)

[2] [https://www.reuters.com/article/us-usa-china- concentrationca...](https://www.reuters.com/article/us-usa-china- concentrationcamps/china-putting-minority-muslims-in-concentration-camps-us- says-idUSKCN1S925K)

[3] [https://www.nytimes.com/2019/05/04/world/asia/trump-china- ui...](https://www.nytimes.com/2019/05/04/world/asia/trump-china-uighurs- trade-deal.html)

</comment></comment></comment></comment></comment></comment></comment><comment><USER41> next facebook.

<comment><USER42> How's that, growing without bound despite countless controversies?

<comment><USER41> Doing something

people get upset

blame it on the tech

</comment></comment></comment><comment><USER43> All your data belong to China!

</comment><comment><USER44> Why should we believe anything this company says?</comment>


Why Microsoft's Strategy for iOS and Android Apps Is Genius (And Insidious) - <AUTHOR> http://arc.applause.com/2015/04/29/windows-10-development-ios-and-android-code/

<comment><USER3> Certainly would appear to lower the barriers to developing a Windows app - <USER2> question is whether its the barriers or the lack of interest that are driving it. The smarter move was making Windows 10 free for the first year to get adoption up.

</comment><comment><AUTHOR> Well played, Microsoft. Well played.</comment>


Intelligent people more at risk of mental illness - <AUTHOR> http://www.independent.co.uk/news/health/intelligence-mental-illness-iq-study-findings-depression-a8005801.html <comment><USER2> The survey population for this study was American Mensa members. I'm not surprised they found "those in the Mensa community had considerably higher rates of varying disorders" but I suspect this has more to do with who joins Mensa rather than high IQ.

As noted by Scott Adams, "It turns out that the people who join Mensa and attend meetings are, on average, not successful titans of industry. They are instead – and I say this with great affection – huge losers. I was making $735 per month and I was like frickin' Goldfinger in this crowd. We had a guy who was some sort of poet who hoped to one day start "writing some of them down." We had people who were literally too smart to hold a job. The rest of the group dressed too much like street people to ever get past security for a job interview. And everyone was always available for meetings on weekend nights."

<comment><USER3> Would be great to have a link to the actual study, but studies typically control for socio-economic-status, family situation, etc... I share the your general sentiment, but keep in that Mensa's threshold of 130 means that 2% of a population with a 100 median IQ (assuming a 15 SD test) is eligible to join.

While far from everyone eligible to join Mensa does so (and as you've said, there's a very good reason for that), there is likely large enough to be able to form a statistically significant sample that resembles the general population with the same IQ which I suspect is readily available (at least in US SES by IQ can be gauged from GSS -- where the WORDSUM question is a "good enough IQ test", or from college statistics -- where the SAT is also a "good enough" IQ test for the purpose of a general study; not sure about UK.)

This isn't a "correlation vs. causation": I don't believe the study was looking for or claiming a causal relationship; I suspect it could well be the reverse: increased anxiety about predators being a push for better ability to predict where the predator might appear -- i.e., visuo-spatial intelligence (there is also some data that suggest that Homo Sapiens intelligence may have evolved accidentally out of certain neurological disorders).

So if the question is "is this a real correlation", there's good reason to think that the experiment was indeed designed to handle any confounding factors like the one you suggested. This is probably a better experimental sample than "college students who volunteer to take part in a psychology study" and easily allows for the most obvious variables to be controlled.

</comment></comment><comment><USER4> People with high IQ scores tend to earn more than people with low IQ scores. People with more money in general, have more money to visit psychiatrists to get a diagnosis. I didn't find anything in the paper to control for this issue.

Also, the whole study is based on a survey of American Mensa members which are not exactly a randomly selected group of high-IQ people - Mensa is notorious for its surprising lack of doctors, elite lawyers, quants and other "successful" high-IQ groups. Perhaps the issue is that people with high IQ scores that DON'T have the material success that their IQ would suggest are more likely to have a mental illness of some kind.

<comment><USER5> I agree, I think mental illness afflicts people irrespective of their IQ. Perhaps people who seek out intelligence tests are experiencing delusions of grandeur?

<comment><USER6> Are the delusions if they do well on the tests?

</comment></comment></comment><comment><USER7> This is common knowledge in the gifted community and has been for decades. I mean, the study sounds pretty flawed, like a conclusion looking for verification, but there have long been support groups for the gifted population and their parents, in part because gifted people tend to be special snowflakes.

I absolutely hate the common explanations for this phenomenon. The "over- excitabilities" mentioned in the article is a common explanation that has been around a really long time and Hoagies Gifted Page has long been pretty bad about "poor mouthing" giftedness. A lot of what is there (or was the last time I looked) talks about how utterly miserable it is to be so smart and how much it harms you socially and emotionally.

Yes, there are social problems that tend to go along with giftedness. And, yes, ADHD, OCD et al are so common at high IQs that some people refer to them as "co-morbidities" for lack of a better term. (I am also aghast at the article lumping ADHD and ASD in with "mental illness." Ugh.)

But, I have my own thoughts on some of what is really going on. I don't really agree with most of the standard explanations and opinions about this phenomenon. From what I gather, it is pretty well documented as a trend.

But, some comments here on HN are making me feel better about never bothering to join Mensa. A cousin of mine was in it and suggested I join for scholarship money to help me return to college. I never got around to applying.</comment>


My Parents Were So Excited About the New CEO of Microsoft - <AUTHOR> http://www.thestranger.com/seattle/my-parents-were-so-excited-about-the-new-ceo-of-microsoft/Content?oid=20171444 <comment><USER2> The real purpose of the author was to champion "more humane" leadership, and projecting Sathya as the evil, cold, ruthless leader seems to have their way to do that. The title is link bait more than anything else.

</comment><comment><USER3> "...the first thing I want to do and focus on is ruthlessly remove any obstacles that allow us to innovate." --Satya Nadella, CEO, Microsoft

Wow... I tracked down the video, and he actually did say that.</comment>


A Dialog with Your Data Using the New Dataclips - <AUTHOR> https://blog.heroku.com/dialog-with-data-new-dataclips <comment><USER2> A useful feature imo would have been to allow non-tech user to input simple parameters to inject into SQL queries, such as 'national_code' or 'id'.

It would allow us to build reporting dashboards in no time for most use cases !</comment>


Hidden costs of constantly shipping new things - <AUTHOR> https://www.mindtheproduct.com/the-hidden-costs-of-constantly-shipping-new-things/ <comment><USER2> Interesting! I especially enjoyed the insight into the evolution of the company. Though with this insight I have slightly different conclusions than the author:

1\. A bias to ship and a bias to ship _new_ things are not one and the same. A lot of the problems, such as a failure to iterate on existing products/feature, sound very much a product of the later, not the former. If anything, the issue "Insufficient Iteration" is probably not correctable without a bias to shipping.

2\. A bias to ship and a bias to ship things that impact your customers are not one and the same. There is a note about an early shift to micro-services. I can't speak to this organization, but, generally speaking, spending time on internal engineering work to the detriment obvious missing features is a common issue with early stage companies.

3\. A bias to ship and a bias to ship to the right customers. Specifically in regard to the high value, high demand customers.

4\. A bias to ship and... idk what to call this? "The product specs were well thought through, sometimes crafted for months." TBH, the problem with this one feels like a lack of a bias to ship.

As described, I think the real culprit was a lack of or poor prioritization. FWIW, I suspect the author and I may actually be in violent agreement as I did find my self nodding with most of his lessons learned. Though I'd be careful about letting too much hindsight bleed in (eg - do situations that'd be improved by more decision documentation justify the effort of documenting all product decisions, especially in the early phase when the product is rapidly evolving)?

<comment><USER3> I think the problem is that the leaders in the company (technical or otherwise) didn't recognize that software systems are like children: They have phases.

The ship it constantly is probably ok when you're small, but at some point you need to become more mature than that, both as an organization, and as an approach to the software itself.

</comment></comment><comment><USER4> I work at a company that grew from a few hundred to a few thousand people in a few years' time, and there was an identical mindset and identical problems over that growth period. I started during that growth period, and 10 years later, we're still digging ourselves out of those mistakes.

One reason for this was how heavily we prioritized first mover advantage. We pushed extremely hard to get into new markets, and while strategically this worked out extremely well for us, it meant that later development was hamstrung by earlier decisions. Any iteration on old features almost always meant rebuilding it entirely, and we've only just started to get to a place where previous efforts to do this are now paying off in increased velocity.

I think there's an argument that this _could_ be good, since you're only improving features when there's some other value you can ship. However, I think it's short-sighted, because it means prioritizing small iterations has an unnecessarily large cost, simply because that cost wasn't paid down slowly over time. While I do think prioritizing new markets early on was a good call, that mindset bit us in the ass in a big way when we tried to ship a very flashy new feature later in our growth cycle that crashed and burned because of tech debt, and it took us over a year to fix everything to release it. If we had taken a mindset of building for the long term five years ago, which was already years after we'd gotten to a comfortable place in revenue and marketshare, we would probably have paid off most of the major tech debt by now.

I honestly don't know how a company can get past the addiction to shipping, at least not during its growth period. It can be strategically necessary, and I imagine in only a two-year period at the early stages of a company, like what's in this article, it's impossible to avoid. But leaders must have in the back of their heads the idea that they will need to start tipping the scales toward standardization, building to last, and building infrastructure as well as features sooner than they'd probably like to avoid larger problems in the future. And I suspect the only way leaders will really take that seriously is if they've gone through it before.

EDIT: Clarified a few points.

<comment><USER5> Apart from the one catastrophic failure you mention it sounds like, from a business perspective this worked? You were able to extend your tech debt long enough to start generating money and are now in a position to pay it back. From an engineering perspective I agree it's a grind to pay all this back and lose so much velocity. Are their business strategies that favour slower more careful engineering practises? mission-critical systems come to mind but maybe also nth-movers could be put in there? Where you know what you're building and competing with but want to make the best version of X.

<comment><USER6> > Are their business strategies that favour slower more careful engineering > practises?

This is a false dichotomy, and there is plenty of research (and experience) that shows shipping higher quality software actually takes less time and money.

<comment><USER7> I want to believe this. Do you have any citations?

My experience would suggest shipping higher quality software is a lot cheaper in the long run, but has up-front costs that often times startups can't afford.

<comment><USER8> You can afford them if you don't have project managers or sales guys begging for their demoable results of new shiny thing and promising it to others too early and too often.

</comment></comment></comment></comment></comment><comment><USER9> This is kind of interesting but it sounds like the perpetual worry of the perfectionist developer. Of course you end up with tech tech, you develop new things when you could improve existing things, you deploy something that could be iterated but you don't.

The truth is that the world is a complex place and you don't always know whether you can keep your existing customers by improving what you already have or get new customers to decrease the risk of declining income. It is hard to measure what is acceptable tech debt and what is worth addressing.

In one sense, the proof is in the income. If you can help customers with an 80% OK product then you just need to live with the dodgy 20% bit.

I agree with the danger of big customers though! If you ever think something is a way to avoid hard decisions, pivots and annoying some of your customers, then it is too good to be true ;-)

<comment><AUTHOR> I'm far from being a developer perfectionist, I wouldn't spend 10 years in this company if I was one, trust me (btw we fired a lot of them along the way).

The point of the article is not to convince anyone to slow down as much as possible and work on bugs/etc. It's just that at some point (3, 4 years in?) there comes a time that you just have to put more effort into the things I described, otherwise it gets complicated _fast_.

Obviously there is a tradeoff, my opinion is that this tradeoff wasn't correctly balanced (especially further down).

<comment><USER10> I think the article starts out sounding that way --- but reading all the way through, it's about how to DO BETTER if you want to ship more often (the key take aways from the article being: document, iterate/revisit, and be intentional)

</comment></comment></comment><comment><USER11> Every company suffers under the weight of the debt of being feature focused. That's because everyone who didn't do that isn't alive to tell the tale.

</comment><comment><USER12> There's a lot of insight packed into this post, and really it transcends way beyond the mindset of _constantly shipping new things_. This particular line resonated with me:

 _Big customers certainly sounded good on paper, but they came with a cost (complexity and a very hands-on relationship) that we weren't really prepared for, and that we didn't manage to put boundaries on fast enough._

This is an issue I've dealt with more than once in my career. It seems to occur especially when an organization values the _sale_ more than it values the _business_. It's virtually impossible to successfully apply boundaries after the customer has been trained that there are none. Irrespective of how carefully you do it or how much you try to reassure your customer, it always feels like you are taking value away. I've encountered this recently in an organization that values customer happiness, which it calculates as an absence of customer complaints, more than customer success. The squeaky wheels get all the grease at the expense of just about everything else.

And the software inevitably has become a maze of settings and IF-THEN-ELSE blocks and SWITCH statements, increasing the cognitive load required to make what should be simple changes. When new features are added, the impulse is to code defensively and build inevitable customization into everything up front. And further down the spiral of technical debt and increasing complexity we go...

</comment><comment><USER13> I always wonder why nobody appears to try a hybrid model between product and project development when it comes to satisfying large customers which need very specific customizations that often don't fit well into a product development roadmap aiming at satisfying a more generic audience.

Like: don't just add weird features that make sense for just one customer, but are eventually delivered to all customers, but instead maintain a "core product" which provides a lot of generic, core functionality, but which by itself isn't actually usable and just serves as basic building blocks to construct the actual end products which are then built and packaged individually for each (big) customer. Each of these "end products" is a development project in itself, has its own team, its own codebase, but they all draw from a core set of base functionality developed and maintained by another dedicated team that is not directly in contact with any of the customers (or at least not permanently).

This - at least in theory - prevents custom functionality for specific customers from polluting the common core product that aims to please a general audience, while still allowing for the necessary freedom to satisfy big customers demanding custom functionality, but also offering big checks in return.

<comment><USER14> I mean you're describing what most Enterprise Resource Planning deployments look like. There's a lot of enterprise software that follows this model

<comment><USER13> You are right; probably I have to state the question in a more precise way, which would be: why is it such that this model seems only to be used in huge "enterprisey" contexts? The model of course depends on a certain modularity of the software in question, but hasn't that been a core concept not just for the biggest players, but also the smaller shops for a while now?

<comment><USER15> Implementation costs a lot, not just in coding, but in all the extra work that goes into sales, gathering requirements, training etc. Whoever nominally pays for all that work, it translates into a high price tag for the customer. Only large enterprises will get enough value from the customisations for that high price tag to be worthwhile.

All else being equal, a smaller company will spend a lower absolute amount to work around imperfections, and get lower value from any product overall. So of course they will not be willing to pay as high a price. Hence, for commodity use cases, use commodity software rather than fully/semi-customised.

</comment></comment></comment></comment><comment><USER16> When I was running a product team, I often talked about "nuance" being the enemy. Every difference (or arrow in a flowchart) was a new piece of product debt to have to think about later.

This article is in that direction. > … the difficulties started piling up fast, and before we knew it we had to deal with a lot.

</comment><comment><USER17> I think this is a solid critical reflection that at least in enterprise relates well.

When it comes to your customers, I believe that you want to understand as much as you can, as early as you can, about what they are actually trying to do and about how they are actually faring with what you have sold them. I'd argue that the success of your initial customers is more important than the features for your next deal, and that feature investments should reflect that.

The problem, as the OP said, is in estimating the difference between the incremental feature requirements of your initial customers and what your strategic feature set will be. There may be no difference, and your strategy is to ship features in order satisfy each new customer as you get it. Perhaps you can design for that. Chances are though that your first customers are not wholly repsentative of your strategic, target market.

So, in the early stage, you have to make choices about allocating dev for tactical "now" or for strategic "later". In the early stage, if there is not enough "now", there is no "later". The question is when to begin budget for later. I'll argue that you are attempting to build a business, not a suspense movie, and that you budget a fixed amount for "later" from day one. Plan to succeed.

I agree with the recommendation of a strong Professional Services function as an buffer for dev, but will note that culturally you have to work hard to not silo. Your Support and ProServe teams become a primary channel of learning for dev. As a business, though, you may have to give away ProServ hours.

All in all my biggest takeaway here is that you attempt to "hire" customers strategically. As you journey together, will this customer want to go where you will want to go? What time is it when your "Big Customer" sits on your roadmap? ...

"Hire" customers strategically is easier said than done when there's salaries on the line -- accordingly look at how you compensate Sales and budget away from early stage big deals.

</comment><comment><USER18> Also could be titled "Hidden costs of moving to microservices because the cool kids are."

Very glad (most of) the cool kids have re-evaluated their stance on microservices.

<comment><AUTHOR> I have a plan to write a short article about specifically this topic at Base. I personally fucked up a ton of things with microservices there :)

</comment></comment><comment><USER19> What an awful article. Expensive conclusions are drawn with no apparent consideration for their cost. The experience of a company which appears to have successfully balanced complicated tradeoffs in choosing features is discounted because there were bugs. And almost no examples or useful stories about any detail, or suggestions on how to do it better other than to go (much) slower and to spend (much) more or to not take rich customers money.

I suspect good arguments could be made for these things, but they need to acknowledge and justify their cost.

<comment><AUTHOR> Author here. Sorry you didn't like it :)

It's tough to suggest detailed solutions to a problem which is in general very vague. Btw I hope the article shows my appreciation for what the company achieved.

One small insight from Uzi (CEO + founder) when I discussed this with him (after publishing article): Base should've focused on much less features but with greater detail. It somehow confirms my guess that we didn't work on existing features as much as we should, but we "spread too thin".

</comment></comment><comment><USER20> Oh good, I was afraid this was going to be an intervention for me regarding my steady stream of almost daily deliveries from Amazon/HD/Walmart/Ebay etc.

</comment><comment><USER21> Hm... this article actually seems like a better argument to obsessively focus on constantly shipping new things than the opposite.

<comment><USER22> Being bought out for large money often covers up the nightmare that would have been, and shifts it to someone else.

</comment></comment><comment><USER23> [https://en.wikipedia.org/wiki/Feature_creep](https://en.wikipedia.org/wiki/Feature_creep)

[https://en.wikipedia.org/wiki/Scope_creep](https://en.wikipedia.org/wiki/Scope_creep)

<comment><USER24> That's not a particularly good way to encourage discussion.

What is it specifically about Feature & Scope creep that you would like to say?

<comment><USER23> I think this article is great and it does a good job characterizing what happens to products that fall victim to feature creep.</comment></comment></comment>