How and Why Swiftype Moved from EC2 to Real Hardware - <AUTHOR> http://highscalability.com/blog/2015/3/16/how-and-why-swiftype-moved-from-ec2-to-real-hardware.html <comment><USER5> There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing:

Cloud computing offers you the great and awesome advantages of being able to instantly scale your application, replicate your data and basically just grow according to your business volume, and all this without significant investments, delivery time, setup time, people time, maintenance or anything but it's expensive in the long run.

And this is OKAY, this is GREAT.

Once you're big enough, you know what your load is now and what your load will likely be, and you know _exactly_ what you need now and (approximately) what you're going to need in the near future, setting up your own datacenter is way, way more effective.

Amazon does not get free electricity, free servers and/or free people time. Of course, you're paying that, and you're also paying Amazon's profits.

This is absolutely fine, as long their service fits you.

But when you grow enough, put simply, your needs change. It's just that.

<comment><USER6> > There's not much need for a fancy article on a fancy website in order to > understand a key concept of cloud computing:

I wish it were true, but plenty of companies are gripped by cloud fever. I've seen quite a few going down the route of charging into the cloud not because they've run the numbers and found it stacks up, but because they want to be in the cloud, and Amazon have some great marketing people.

<comment><USER7> Often times companies move because their organization doesn't deliver, and the hope is a cloud company will do a better job of it.

If you have a great team, I firmly believe hosting yourself is far, far, far less expensive.

If you have a terrible team, then cloud (hosting) is less expensive. Even if it was exactly the same cost, you're gaining by not having to have a staff to run it, and the costs of managing them, etc etc.

Most places don't have great teams. Insert random corporation here likely has a team that is a mess for whatever reasons happen in large companies.

In that case, the Cloud makes a ton of sense for them. They've already screwed up their own organization in some way, and this is a large reset button on the whole thing.

That's worth a ton in itself.

<comment><USER8> If you're a small org with relatively small volume self-hosting doesn't make sense.

<comment><USER9> Can you elaborate why not ?

You can get some cheap dedicated hosting (ex: [1]) for a fraction of the price.

It's so cheap compared to AWS you can order a few spare ones and still come out cheaper than your one beefy AWS instance ?

The only way it doesn't make sense is, if you need to scale up and down very fast ?

[1] 60 euro/month: Quad-Core Haswell, 32 GB (non-ECC) RAM, 240Gb SSD @ [https://www.hetzner.de](https://www.hetzner.de)

<comment><USER10> The cost of the systems is almost irrelevant. The time of qualified people is far more expensive. Sot it's not just the elasticity. You have to look everything from the time perspective as well. What additional knowledge will I need? Will I need to learn load balancers, how to make them highly available? will I have to learn about SSL certificates and termination? Will I need to learn how to implement and operate a secure and highly available DNS service? How much is your time worth? Our hosting costs are a fraction of a single worker's salary. Whether AWS is more expensive is essentially irrelevant. OP argues that it was not doing the job for them which is an entirely different issue.

<comment><USER8> Don't forget the quality of the work. Frankly, if I'm administering servers I'm not going to do as good a job of it as someone whose whole job is that and I'm not going to do as good a job as Azure or AWS or whoever else either. Platform as a service is the way to go.

</comment></comment></comment></comment></comment></comment></comment><comment> stephen-mw Here's my anecdotal, one-data-point experience from moving a giant EC2 environment to datacenter:

1\. Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers.

2\. Any weirdness you experienced with AWS infrastructure will be replaced with weirdness in your own environment, except now you're on the line to troubleshoot and fix it yourself.

3\. Operation staff will immediately start guarding the food bowl as resources become finite. Server provision waits start to seem like breadlines. Power is consolidated with Those Whom You Must Ask.

4\. Your cost will decrease, sometimes significantly so.

5\. You'll have more hardware flexibility to run your app just the way you want to (Stack Overflow's mega databases come to mind[0]).

In the end I think this type of transition is for stable companies that don't mind or even prefer strong divisions of labor (coders who code, sysadmins who sysadmin, testers who test), but it's not for startups or companies that hope to move with any kind of strong velocity.

[0] [http://highscalability.com/blog/2014/7/21/stackoverflow- upda...](http://highscalability.com/blog/2014/7/21/stackoverflow- update-560m-pageviews-a-month-25-servers-and-i.html)

<comment><USER11> > Your operational overhead will increase _a lot_. Be ready to hire on a lot > of ops staff if you expect them to do anything but put out fires. And as you > grow you'll need experts, people like network engineers.

Why? What are you talking about? You are hiring servers you are no colo'ing them. Networking them is not your problem. Your responsibility still starts from a root prompt just there's no VM layer between that and the physical server.

<comment><USER12> and if you have more than one? or more than one dc? somebody needs to connect it, or you will need a VPN. it's not that easy without Clouds if you need connected servers. We switched to AWS since connecting servers in a dc isn't as cheap as people think of.

<comment><USER13> I host all my stuff with Hetzner (who are at the cheap end of the market compared to Softlayer) and even they provide pre-configured VLANS and private switches. For intra-DC you can just setup some redundant openvpn links or pay the dc to configure a hardware tunnel between both sites.

</comment></comment></comment></comment><comment><USER14> My 5cent:

If you have lots and lots of money and a high margin business, do yourself a favor and go with Amazon (much less hassle with contract management and low level challenges).

If you need to scale month to month and are growing 50% per month, go with Amazon.

If you are very small and can live with 10 instances, go with Amazon.

If CAPEX doesn't help you and for whatever reasons you need to spend OPEX, go with Amazon.

If you need many (types of) machines for failover but which otherwise mostly idle, go with Amazon.

Otherwise it's always cheaper to buy or rent hardware. Amazon is very expensive (TCO).

If you base your decision on hype, you're screwed.

* Amazon stands for Cloud Provider, personally I'm choosing Digital Ocean with Mesos/Docker.

* Except S3 which is a no brainer to use.

<comment><USER11> > If you need to scale month to month and are growing 50% per month,

Then rent more servers.

> If you are very small and can live with 10 instances,

Then rent a few servers.

I maintain that there are extremely few cases for a typical website to use the cloud. To handle peaks, it is both simpler and cheaper to keep enough capacity just idling around than spinning up and down Amazon instances. The cloud is almost always a useless hype. It can be different if you can architect to use the various services Amazon provides.

<comment><USER14> From my experience renting more servers with 50% growth is a challenge. A lot of things go wrong when installing a lot of servers each month.

Also from my experience, with 10 instances the money you save with custom servers is negligible and contract and SLA management, multi datacenter etc. is easier with a cloud provider than renting servers. At least where I've rented servers in the past.

<comment><USER15> I'm not sure where you see a difference between a VM and physical hardware when it comes to provisioning.

Sure, the physical hardware takes 1 hour rather than 1 minute to spin up, but the process is otherwise entirely identical.

<comment><USER11> 1 hour? Welcome to 2015! OVH spins your server up in two minutes.

</comment></comment></comment></comment></comment><comment><USER16> "With Amazon we experienced networking issues, hanging VM instances, unpredictable performance degradation (probably due to noisy neighbors sharing our hardware, but there was no way to know) and numerous other problems. "

Why do I get the feeling it was kind of a cop-out to just pack up and move without finding the root cause? I've seen it plenty of times: the "best solution" is to just find a different hosting provider.

In my experience, I've never found an issue with an application on AWS that wasn't caused by either a misunderstanding of what was being offered (e.g. not provisioning enough PIOPS for database volumes), or simply issues with the application code.

<comment><USER17> > In my experience, I've never found an issue with an application on AWS that > wasn't caused by either a misunderstanding of what was being offered (e.g. > not provisioning enough PIOPS for database volumes), or simply issues with > the application code.

You haven't been using Amazon long enough then.

Amazon is great for proof of concept. No upfront costs, extremely scalable, etc. Unfortunately, its expensive compared to physical hardware once you get to scale, and you may never solve underlying performance issues due to it being a shared tenant environment, even if you're a Netflix-sized customer.

<comment><USER18> But doesn't Netflix successfully use AWS now?

Its possible they get special treatment if they are big enough (nobody else's jobs on their physical machines ... or something like that).

<comment><USER19> Netflix uses thousands of instances, they don't share servers.

<comment><USER20> Using thousands of servers doesn't mean that you don't share physical servers.

<comment><USER20> Even if you own all the VMs on a physical server it's hard to avoid the noisy neighbour problem. Netflix has its own instance monitoring tool - Vector - <USER2> handle these issues: [http://techblog.netflix.com/2015/02/a-microscope-on- microser...](http://techblog.netflix.com/2015/02/a-microscope-on- microservices.html)

</comment></comment></comment></comment></comment></comment><comment><USER21> This is a no-brainer if you've ever done anything at scale. The explanation is rather simple - hardware is always "on the premise", yours or Amazon's. Someone needs to swap drives, motherboards, man the networking gear, run cables, etc. Amazon doesn't really get a break on the hardware cost because 10,000 servers do not cost less per server than 100 servers (in fact they cost more as the volume goes up if you need them to be identical). When it comes to labor cost - if you have enough hardware for at least one full time datacenter tech, you're in the same boat as Amazon.

So you're paying Amazon to do the same work you would do otherwise - <USER4> you're subject to their rules and procedures and Amazon being a profitable business needs to mark their services up.

<comment><USER22> > The explanation is rather simple - hardware is always "on the premise", > yours or Amazon's. Someone needs to swap drives, motherboards, man the > networking gear, run cables, etc.

> So you're paying Amazon to do the same work you would do otherwise - <USER4> > you're subject to their rules and procedures and Amazon being a profitable > business needs to mark their services up.

But I thought that they were paying Softlayer to do that stuff instead of Amazon. They're _not_ doing it themselves - and yet it's still cheaper!

<comment><USER20> I would like to know the cost calculation after a year or two. With a handful of servers it's easy to get the false impression that HW failures are rare.

<comment><USER23> Oh, there wasn't a handful of servers after we finished the migration (we have migrated a bit late IMO, so we had a lot of traffic even back then). And today, with much larger infrastructure, with hardware clusters specifically tailored to our customers needs, etc I'm pretty sure the same infrastructure on EC2 would cost more than 2x.

(Update) Re: failures - with a ~50 servers we see a hardware issue (disk dead in a RAID or an ECC memory failure) about once a month or so. None of those failures caused a single outage (RAID and ECC RAM FTW) so far.

<comment><USER20> How do you monitor HW and network failures and how do you notify SoftLayer? Is that 1-2 hours replacement time true for each components of your server fleet?

<comment><USER23> 1-2 hours is their new server provisioning time. For HW issues we use nagios (that checks raid health and ECC memory health regularly) and at the moment we just file a ticket with SL about the issue showing them the output from our monitoring. They react within an hour and HW replacement is usually performed within an few hours after that (usually limited by our ability to quickly move our load away from a box to let them work on it).

</comment></comment></comment></comment></comment></comment><comment><USER24> A concrete example of how I saved a few hundred thousand dollars over the AWS by building quarter rack colocation setups with SuperMicro servers: [https://gist.github.com/rockymadden/5561377](https://gist.github.com/rockymadden/5561377)

With Ansible, I spend no more than an hour a week, amortized, maintaining both the hardware and administration. I assume nodes for any specific role will fail, I only scale horizontally, I always have redundancy for every role, I stay off disk as long as possible (heya 512GB RAM redis cluster), etc.

<comment><USER25> Great resource. Are you going to keep this updated?

</comment></comment><comment><USER26> From my experience AWS is one of the most reliable hosting providers. It's extremely easy to setup a fault tolerant infrastructure using cloudformation, puppet/chef etc, boto and a bit of autoscaling. The only disadvantage AWS has is costs. In many cases it's more expensive than a traditional hosting provider. On the flip side, your engineers will be more productive.

What I miss in this article is any details on why they had issues with AWS. You can't just say it wasn't reliable and not explain the details. AWS works for all of the world's largest startups, why didn't it work for Swiftype?

</comment><comment><USER23> Author here. Happy to answer any questions.

<comment><USER27> Using EC2 here for nearly 2 years and you mention I/O problems and instance outages 2-3 times a week. Which size instances were you running?

I ask because other than the VM security updates, none our instances have these sort of issues and some of them have a VERY long life (not ideal we know). I understand the cost savings and the rest of the reasoning but in my experience EC2 isn't THAT unreliable.

<comment><USER23> Oh, I know what you're talking about. We too had some instances (actually, a lot of those) that would run for a year with no issues. The problems started around the time you tried to push EC2 instances beyond an "idle, handling some requests just to keep from falling asleep" state. Pushing IO (even with provisioned IOPS) caused random IO stalls, pushing CPU caused REALLY uneven performance, etc.

And the only solution provided by EC2 support was always to buy more instances to keep them cold and happy. The problems with that approach (just to name a few): the cost (for a young startup burning money on idle infrastructure like that is not very wise IMO) and the fact, that the time to design, develop and deploy scale-out approach for each of your backend services is the time you could have spent trying to build your product (again, startup-specific; you'll have to think about across-the-board 100% scalability at some point).

<comment><USER28> First: I work in Startup BD at AWS (disclosure), but have been a multi-time founder as well. I was under the impression that an AWS architect will sit with you to optimize your infrastructure (Business Support). Did that not happen / or was it not useful? Happy to help in any way I can.

</comment></comment></comment></comment><comment><USER29> I'm convinced hybrid cloud is the way to go. Anything needing high IO performance should be on dedicated. Anything needed CPU/memory elasticity (worker nodes, etc) should be in the cloud. Assuming you can get low latency connectivity into AWS with DirectConnect, this might work?

<comment><USER30> Cpu also generally sucks due to cache thrashing on non-trivial datasets

<comment><USER29> Heh. So what is a good use case for EC2?

<comment><USER30> When you have highly variable load.

eg. Netflix probably spins up thousands of servers for a few hours.

</comment></comment></comment></comment><comment><USER31> What specific piece saw the biggest boost? My guess is MongoDB. Also if new servers take 1-2 hours, you are always paying for what you "think" will be capacity for peak load correct? How do you handle events that quickly and drastically increase load or txn/sec?

<comment><USER23> Actually, it depends. Stability and performance wise, I'd say our Lucene-based search layer has seen the most impressive jump. But yeah, Mongo instances loved the new fast IO as well :-)

</comment></comment><comment><USER32> The biggest problem with AWS is the outrageous cost of bandwidth. Even if you ignore all the other cost differentials, the bandwidth charges will kill you at scale.

Unfortunately cloud computing, or at the very least AWS, overpromises and underdelivers at scale. All in all economically viable use cases for cloud computing are very few and very specific at scale.

<comment><USER33> Since people are sharing their experiences in this thread, has anyone tried using Direct Connect to get cheaper bandwidth?

<comment><USER32> There aren't a lot of savings to be had. In addition to paying for the Direct Connect infrastructure you still have to pay Amazon per GB charges, albeit at a slightly lesser rate.

The only real solution is to move the bandwidth usage off AWS.

</comment></comment></comment><comment><USER34> One question. If real hardware was always 50% cheaper then AWS wouldn't have been such a success. Can you please explain in which scenarios it makes economical sense to use AWS compared to real hardware?

<comment><USER35> AWS is always going to include a premium because they take care of the DevOps portion of your infrastructure. There are plenty of virtual hosting companies that cost significantly less than dedicated hardware, if you won't need all the bells and whistles.

<comment><USER36> _because they take care of the DevOps portion of your infrastructure_

Sorry, but that is mostly a lie.

Running a non-trivial app on EC2 is significantly _more_ complex than doing the same on (rented) bare metal. Scaling to a _massive_ size can be easier on EC2, but only after you paid a significant upfront cost in terms of dollars and development complexity.

Is your app prepared to deal with spontaneous instance hangs, (drastic) temporary instance slowdowns, sudden instance or network failures?

Did you know that ELBs can only scale up by a certain, sparsely documented amount per hour?

Or that you need a process to deal with "Zombie" instances that got stuck while being added/removed to ELBs (e.g. the health-check never succeeds).

Or that average uptime (between forced reboots) for EC2 instances is measured in months, for physical servers in years?

Or that Autoscaling Groups with Spot instances can run out of instances even if your bid amount is higher than the current price in all but one of the availability zones that it spans?

The list of counter-intuitive gotchas grows very long very quickly once you move an EC2 app to production.

<comment><USER23> This comment is pure gold! That's exactly what I wanted to explain here and you did it so well. Thanks!

</comment></comment></comment></comment><comment><USER37> I think cloud providers are mostly good for small/medium, fast growing startups. Big companies need more granular flexibility. The path to vertical integration in the software indsutry is a lot smoother than in many other industries, there are many levels:

You can...

1\. Make your own hardware

2\. Own/manage your own hardware

3\. Rent commodity hardware from a standard hosting provider

4\. Use IaaS (e.g. EC2)

5\. Use PaaS (e.g. AppFog, Nodejitsu)

6\. Use BaaS (e.g. Firebase, PubNub, Pusher.com)

The higher the level, the more technical flexibility you lose. The bigger the company, the more it makes sense operate at a lower level because there is no significant wastage being introduced as you move down the levels (you remove the middlemen so you can pocket their profits) and the capital cost to move between levels is relatively low.

If you compare software to another industry like cheesemaking for example, if you're a cheesemaker and you want to make your own milk, the next step is to buy the whole farm and then you have to figure out what do do with the meat (wastage). Going between these two levels is expensive and could mean doubling or tripling your expenses so it's not an easy move to make.

</comment><comment><USER38> Cost-wise, AWS makes a lot of sense when growth is not easy to project, happens rapidly, or varies (seasonally, randomly, etc).

Of course, some of this has to do with your team's skill level, but I've had clients run up $100k+ monthly bills at AWS with a relatively small build-out. (and, wow, VPC migrations..)

For fixed or predictable growth patterns on a mature app/platform, a slow build-out on real iron will generally be _significantly_ less expensive, all other things being equal.

However, there are other advantages to AWS that gets lost in this story, such as pre-built, highly scalable datastores. Comparing EC2 to real iron misses most of the real story on why the cloud is changing everything.

One of the hardest things I have to tell clients is not to build their own datastore/database in house or at EC2; sometimes the case is clearcut, and sometimes not so much, but if you have a datastore at AWS that gives you 80% of what you need, use it instead of rolling your own. (source: IAMA AWS Growth Architect)

</comment><comment><USER39> It strikes me that more and more a critical selection when growing in this way is the DNS part. It needs to be back-end agnostic and provide an increasing amount of functionality.

Health checks and failover are must have now, but this article makes me wonder three things:

1) Are there any DNS services that understand geography of your "zones", i.e. route to and failover based on IP? (but are still platform agnostic).

2) How long can a DNS failover take worst case? You can technically set a low TTL, but don't a lot of ISPs just increase that to a minimum?

3) Isn't it better to replace some of the DNS failover with high availability dedicated load balancing?

<comment><USER40> 1) Yes, there are several DNS service providers that offer BGP anycast with geographically aware failover / load balancing. UltraDNS and DYN are the larger ones.

2) Yes, some ISPs do set a minimum TTL. Although BGP anycast is the most effective as the first line, sometimes it makes sense to have your reverse proxy caching layer override that distribution based on GeoIP and redirect to a more suitable proxy node closer to the client. This is especially the case when people using recursive lookup DNS servers that aren't necessarily geographically close to them (e.g. 188.8.131.52). It could also be useful in cases where TTL expiration hasn't caught up yet though.

3) No. Think of BGP Anycast DNS as distribution at a global level, and dedicated load balancers as distribution at the local level. You need to work out how to get the traffic to the load balancer first, and load balancing across distant geographies (high latency) results in horrible performance.

</comment></comment><comment><USER41> Cases where people try to use EC2:

\- We're starting from scratch and think AWS will give us flexibility for cheap

\- We have existing servers and think moving from them to AWS will give us flexibility for cheap

Cases where people move away from EC2:

\- It was slow/unreliable

\- It was expensive

Conlusion: You should use AWS EC2 in order to save money and have flexible resource allocation, but don't expect it to be stable or cost-effective.

</comment><comment><USER42> I used Rackspace (and later EC2) originally because of upfront capital costs. I bootstrapped my company, and paying a thousand dollars or more, plus hosting/racking fees, just wasn't doable. $50-100 bucks for a reasonable Windows server? Easy deal to make when cash flow is small.

</comment><comment><USER43> it's about a 5-7 year cycle; IT services are outsourced (this time, to the cloud), but as time progresses, all of those services are consolidated (usually, under new management) in-house.

I used to be able to set a watch by HP's cycle before the EDS buyout.

</comment><comment><USER44> Finally sane take on Cloud BS :)

</comment><comment><USER45> I'm telling you theres real opportunity for someone to create "the Uber of colocation/dedicated hosting".

<comment><USER46> OpenStack was supposed to facillitate this. Doesn't seem to get much use.

<comment><USER7> It does from my perspective. I could list many companies that use it for major things.

It doesn't always get publicity. I work for a major company, and we don't scream from the roof tops about it.

</comment></comment></comment><comment><USER47> Interesting article, thanks for sharing. We actually just went the exact opposite because of the larger scale issues we were having with Softlayer. Do you feel like you lost any resiliency by making the switch to physical servers (more virtual instances on one physical server, servers in the same rack, etc)?

<comment><USER23> No, I really do not think going to EC2 could be beneficial in any way in terms of improving resiliency compared to Softlayer. SL allows you to control which VLANs your box will end up on. VLANs could be treated as racks (since they do not allocate more than one VLAN per rack). Then you have multiple DCs in one region (e.g. DAL01, DAL05, DAL07, etc) and you have many different regions (DAL, SEA, WAS, AMS, etc).

I'd be very interested what problems you were having with them and at what scale. If this is a private topic, we could do it over email or some other medium if you like. You can contact me by any of the means listed here: [http://kovyrin.net/contact/](http://kovyrin.net/contact/)

<comment><USER47> We were about 75% virtual with SL and 25% bare metal. One of our issues with the virtual stuff is when we started dedicating them to a set VLAN, multiple times we ran into an issue where some type of resource for that pod the VLAN was in would be maxed out (usually storage) so we couldn't create a new instance. The solution we were given was to let the system pick a VLAN but by doing that we had lost control of the placement and added some complexity to our architecture.

Aside from that it was mainly nit-picky type stuff, but still things that were annoying (networking issues between DCs, networking issues between pods, internal mirrored apt-get repos going out of sync, API is kind of blah, etc).

We use docker so having a few bare metal machines with tons of containers on them wasn't a great HA setup (for us at least), even running in two data centers. The fairly quick setup time though was a nice selling point.

When we went to AWS things just kind of worked. The API was easier to use and the GUI portal was way nicer/stable. So far we have not had any odd issues with our instances, but we also typically run them at about 50% capacity so that might be why. It is also still early so maybe things will come up in 6+ months that send us back to SL :)

</comment></comment></comment><comment><USER48> Oleksiy, can you share the economics? You said 50% savings. What was everything you had running in EC2 (easy to figure out the costs), what were your equivalents in softlayer? Would be very interesting to see the economics.

</comment><comment><USER49> Oh bare metal reloaded :)

</comment><comment><USER50> RunAbove from OVH seems to be the best of both worlds. You can rent dedicated hardware by the hour for very reasonable prices.

<comment><USER23> AFAIK Softlayer has hourly-based rental for real hardware as well. Never tried that, so could not comment on it though.

</comment></comment><comment><USER51> In my experience firewalls and network is more expensive than servers....

Anyone offer competition to VPC?

<comment><USER23> Softlayer manages the network for us. They offer a fully isolated backend network (frontend connection to the world is optional). For frontend connections we simply use iptables.

</comment></comment><comment><USER52> "2-3 serious outages a week" with EC2?? Details, please.

<comment><USER23> Those weren't outages for a specific instance. But from the whole pool of instances we were running 2-3 would have networking issues, random unexplained hangs requiring an instance restart, huge CPU performance drops, IO hangs, etc, etc.

</comment></comment><comment><USER53> Thanks for the information</comment>


AWS Lambda – Run Java Code in Response to Events - <AUTHOR> https://aws.amazon.com/blogs/aws/aws-lambda-update-run-java-code-in-response-to-events/

<comment><USER2> and this from an AWS job posting 6 months ago...

        CORTEX is our next generation platform that handles real-time
        financial data flows and notifications. Our stateless event
        driven compute engine for dynamic data transforms is built entirely
        in Clojure and is crucial to our ability to provide a highly
        agile response to financial events.
        Our technology stack: Clojure, JVM, AWS tools, Sable

via [https://lispjobs.wordpress.com/2014/11/25/clojure- software-d...](https://lispjobs.wordpress.com/2014/11/25/clojure-software- development-engineers-amazon-com/) discussion here [https://news.ycombinator.com/item?id=8664989](https://news.ycombinator.com/item?id=8664989)

</comment><comment><USER3> I'm working with Lambda on a personal project. It has so much potential that will be unlocked once more languages are available. For reference, executing one function 30 million times at 100ms per execution costs about $18.

</comment><comment><USER4> neat! Interesting for high-load tasks. But regularly in day-to-day JS is so much easier to work with :).</comment>


Bloomberg's 'The Big Hack' - <AUTHOR> https://daringfireball.net/2018/10/bloomberg_the_big_hack <comment><USER4> Regarding Apple's denial, there are other publications that corroborate the Bloomberg story. Previously, Apple has denied security incidents even when multiple outlets report it. For example, last year, The Information reported Apple discovered malware on Super Micro servers in their development and production environments [1]. As a result, the Information claimed that Apple ended up terminating its relationship with Super Micro.

In response to the report, an Apple spokesperson denied there was a security incident, stating: _" We're not aware of any data being transmitted to an unauthorized party nor was any infected firmware found on the servers purchased from this vendor."_

However, based on sources from within Apple, Ars Technica claimed Apple employees did find compromised firmware in Apple's design lab. Super Micro SVP of Technology also reported Apple terminated its relationship with them.

I believe we are seeing the same situation here.

[1] [https://arstechnica.com/information- technology/2017/02/apple...](https://arstechnica.com/information- technology/2017/02/apple-axed-supermicro-servers-from-datacenters-because-of- bad-firmware-update/)

<comment><USER5> Apple's denial references that report.

<comment><USER4> Which makes it more curious. Last year, Apple PR denied existence of the security incident, but today they admitted it actually happened. That is a contradiction.

<comment><USER6> No contradiction.

Read it closely: The old statement said that no infected firmware was found on servers purchased from supermicro, meaning freshly purchased servers delivered from supermicro were clean. The ArsTechnica clarifies in an update to the article that the infected driver was installed later.

The new statement says: "That one-time event was determined to be accidental and not a targeted attack against Apple."

</comment></comment></comment></comment><comment><USER7> It does not seem like this story is true. If it's true, it makes absolutely no sense for Apple and Amazon to attack Bloomberg. Sure, a national security letter could force them to stay quiet, or maybe even to lie to the public and say it didn't happen, but it can't make them criticize Bloomberg. Attacking Bloomberg if the story is true is only going to convince Bloomberg to dig deeper. And the story isn't even _that_ bad for Amazon or Apple - it's much worse for US-China relations than it is for either of those companies.

The key technical detail of what these chips are allegedly doing also does not make sense. From the article:

 _the chips allowed the attackers to create a stealth doorway into any network that included the altered machines_

How can you get around a firewall by using a compromised machine that's part of the internal network?

I don't think Bloomberg reporters are just making stuff up. But the technical confusion here makes me suspect that the government officials who leaked this story just didn't understand the details of a real incident that happened, and in the leaking the story got mangled into inaccuracy.

<comment><USER8> It goes beyond that. If this story were in fact true, what we're witnessing is large companies whose entire business is built upon user trust setting fire to themselves. Who'd ever trust AWS again with anything of consequence if it turns out the denial is false?

I don't buy it as written. There's something else here.

<comment><USER9> > Who'd ever trust AWS again with anything of consequence if it turns out the > denial is false?

I agree it makes little sense to attack and deny rather than just stay quiet or keep it to a minimal PR blurb. But I think you're overestimating the amount of blowback they would get if they're lying. Most people just wouldn't care. It might be the decisive factor in a bigger company deciding where to put something new and them going to Oracle Metal or something, but I am really skeptical their bottom line would suffer because of this one thing.

<comment><USER10> Also, almost everyone who lies thinks they'll get away with it. Otherwise, why lie?

Maybe AWS and Apple are counting on the top-secret nature of all this to enable their lies. Would US intelligence agencies really come out on the record to contradict the denials and confirm the hack if it meant destroying the credibility of a couple of American companies? The Bloomberg article detailed how the government was _not_ willing to sound the same alarms against an American company as they were against Huawei and ZTE for just that reason. I'm guessing that creates enough ambiguity for false denials to be effective.

<comment><USER11> Yeah I think people don't understand how big of a deal this would be. Remember Joseph McCarthy? Now picture that, in today's politics, only with even a shred of real evidence that US companies and the military were infiltrated. That's an act of war between two nuclear powers. There is literally no telling what kind of chaos could follow.

I'm not saying it's guranteed that this story is good—-for one, I find the glossed over technical details a little questionable. But it is definitely plausible that this is a false denial, under orders or not.

Regardless, two points are very clear. Remember Stallman who uses that old X60 with a free BIOS? We all called him crazy when he insisted on not having management firmwares and secretive back doors. Now we have a story claiming they were used in an attack by a US adversary (as well as a long chain of confirmed recent security flaws). The second point is that we have to stop trusting China as the sole source for all electronics. It's a national security crisis that there isn't a single US facility making many of the components needed for basic computing platform. The federal government needs to immediately and strongly incentive more domestic foundries. And not the trump style "here be factory, jobs good" but actual targeted planning to bring the whole supply chain back, not just the final assembly everyone talks about.

</comment></comment></comment></comment></comment><comment><USER12> For major major reports like this, reputed newspapers are rarely wrong. See WSJ John Crareyou of Theranos. Theranos vehemently denied it, MSM was with Theranos and at least they weren't outright against them. Then they did more digging etc.

The Washpo digging on Roy Moore, the NYT digging on Clinton Foundation, etc. these are experienced reporters having 15-30 credible sources, evidence etc. every time this happens, everyone denies and slowly little by little the story finds more evidence and facts and it becomes true.

This is why a free press is soooo important

Why would Apple disclose that millions of their products could be hacked and you've lost all your privacy. Who would trust them? They'd lose billions, regulations would come etc. it's in the best interest for every party to deny.

<comment><USER7> On the other hand, Newsweek was wrong about Satoshi.

My suspicion is that there are many cases where the Chinese government is actually trying to insert backdoors into things, and that in particular Supermicro really has been compromised by the Chinese government, but the technical details of this chip are incorrect. That explains why so many government officials are eager to leak information to Bloomberg, but at the same time the technical details don't really make sense.

It isn't really in Amazon's and Apple's best interest to lie about this. When Gmail got hacked by the Chinese government, Google was pretty honest about it. China has a lot of resources so you can't really expect companies to fend off 100% of attacks on their own; it makes sense for them to acknowledge this publicly and get help from the US government when needed.

<comment><USER13> Note that Facebook and Apple have confirmed that they had heard about and actually saw, respectively, security issues with compromised software updates from Supermicro - it's only the chip story they're denying. [https://www.bloomberg.com/news/articles/2018-10-04/the- big-h...](https://www.bloomberg.com/news/articles/2018-10-04/the-big-hack-the- software-side-of-china-s-supply-chain-attack)

<comment><USER14> There's a huge diff between "your web management software is buggy" (pretty much every web management software is kinda crappy and significant number probably would dissolve like wet paper under a serious security audit) and "your motherboard has extra chip which grants full hw level access to outside parties". The former is completely routine and happens nearly every time anybody bothers to do a security audit. The latter, if true, is one of the biggest stories of the decade.

</comment></comment></comment></comment><comment><USER15> Can National Security Letters can be used to require companies to issue outright lies to the public? The bloomberg article indicates that the investigation is not complete, so that could be on explanation for the apparent disconnect between a seemingly well-reported story and the unsually forceful denials.

<comment><USER16> No but the gov might be unhappy that the secret is out while they were trying to use the chips to infiltrate/reverse engineer wherever the chips were reporting to? Amazon has gov contracts it wants to protect so Im not surprised they would deny this at the govs request, and Apple's #1 product is privacy so having something like this show up on their servers would undermine that product strategy so I can understand their denial too (homegrown or at the govs request)

<comment><USER17> This seems like the most likely reason for domestic companies to deny everything, and for their supply chain distributors to play along.

1) Big company partners with distributor. 2) Distributor has security issues. 3) Gov is already aware of security issues, says nothing. 4) Big company becomes aware of issues. 5)Gov steps in and pitches a deal: i)Both big company and distributor must deny. ii)In return, gov gets to: iii)Preserve any existing contracts iv)Protect the big company and distributor, with any legal, trade, or commercial benefits

</comment></comment></comment><comment><USER18> Bloomberg reported that the startup I worked for was for sale and the founders were pitching it to potential buyers. Internally, they vehemently denied the report and said Bloomberg completely made it up. Bloomberg was dead on the money. That's not to say every journalist/article they publish is going to be spot on, but I definitely give them enough credit that I will believe this report until proven unequivocally not true by the accused.

edit: spelling.

<comment><USER19> A CEO shopping a private startup has every incentive to lie to his employees in that scenario and little reason not to.

On the other hand, Apple and Amazon have huge reasons not to blatantly lie about this. The plaintiffs bar would be all over both companies for material false and misleading statements if this turned out to be true (see also Elon's recent experiences with the SEC).

Apple and Amazon are certainly incentivized to not admit bad facts, to spin facts, to issue misleading unclear statements that read as a denial but are not, etc. But I really don't think the legal teams at either company would let executives get away with issuing such full-throated and clear denials if they were untrue.

<comment><USER18> yeeaaah...on the other hand, they could be pulling a hat trick from the FCC playbook. "I had no idea it was legitimate users leaving feedback and the guy I trusted to inform me no longer works here. I was told everything was fine!" Plausible deniability goes a long way when you have enough money/political support or are "too big to fail". It happens. Often.

</comment></comment></comment><comment><USER20> What if Apple or Amazon didn't find the chips but other investigators found them in Apple/Amazon servers. That would make their denial wording technically true. They didn't say no chips were found, they just said they didn't find any.

</comment><comment><USER21> This is this is similar to all the backdoored Cisco devices the FBI found all over the government. If they are doing it at all they would have a complex plan and this would be one of the many approaches. Even scarier IMHO are the CPU fabrication hacks that add in an imperceptible backdoor directly in the chip logic. A recent report showed how CPUs can be backdoored at critical points in the fabrication process by a single operator that would be incredibly hard to detect. We are talking instruction patterns that charge capacitor buffers that allow privileged access once a threshold is reached. Amazing really.

</comment><comment><USER22> >But they don't lie, because they understand that one of Apple's key assets is its credibility. They'd say nothing before they'd lie.

This is so typical of Gruber as an evangelist.

There's no way that Apple would remain silent on this even if they KNEW it were true. The only possible move is denial.

Silence is validation or uncertainty, a statement of ambiguity will tank the stock and reputation as experts, recognition of even partial truth could possibly destroy their supply lines overnight.

I honestly think the corporate denials here need to be outright ignored because they have so much to lose. A story of this magnitude is basically like pointing a gun to someone's head and asking them for permission to pull the trigger.

<comment><USER23> I see Apple and Amazon has a tremendous incentive to lie.

What incentive do you see for Bloomberg to report a major story and lie!? Why would they do that?

<comment><USER22> You can theorize all sorts of conspiracies, like the government planting the story to distance China... so it's possible Bloomberg was mislead.

I think at this point it's more believable the story is true, because Bloomberg is the most credible participant at this point... they're anonymous sources, but as long as Bloomberg did their due diligence as journalists, they've validated their (numerous) sources as credible.

...and like you said, all these corporations have MASSIVE amounts to lose. Just check out what Trend Micro's stock is doing today.

<comment><USER23> That's exactly my point. They have MASSIVE amounts to lose - that's why they don't want to admit that their datacenters are compromised. Super Micro's stock is down -41%.

</comment></comment></comment></comment><comment><USER24> "But in my experience, Apple PR does not lie. Do they spin the truth in ways that favor the company? Of course. That's their job. But they don't lie, because they understand that one of Apple's key assets is its credibility. They'd say nothing before they'd lie."

As a few people pointed out in the other thread, didn't they pretty explicitly deny they were involved with PRISM?

<comment><USER25> PRISM was an NSA-internal codename for a really innocuous automation of servicing legal warrants.

Apple knew nothing about anything they were told was called PRISM, and they happily acknowledged the existence of APIs to service warrants.

</comment></comment><comment><USER26> As someone on a slack server I use pointed out: a server wouldn't need to phone home, could have a planned failure and request an RMA, even if the system was wiped when it came back, it could have data stored somewhere secretly, and why you may not find anything in an audit.

<comment><USER27> AFAIK that wouldn't work against Amazon as IIRC hardware doesn't leave their datacenters without being shredded first.

<comment><USER28> So no hardware warranty repairs?

<comment><USER27> AFAIK, no.

It was presented as one of many things Amazon does to keep AWS safe at an AWS day I attended or something.

</comment></comment></comment></comment><comment><USER29> there's a comment on r/sysadmin that's quite chilling, implying that the state of affairs is far worse than what the report describes:

[https://www.reddit.com/r/sysadmin/comments/9layb7/from_bloom...](https://www.reddit.com/r/sysadmin/comments/9layb7/from_bloomberg_how_china_used_a_tiny_chip_to/e75rwc8)

        I did a penetration test and security assessment for a major electronics manufacturer
        whose parts are likely in every smartphone and laptop. I identified almost certain compromise
        by the Chinese government with full access to modify the manufacturing specs using the
        access paths I identified.

        They chose to bury my findings as it would cause a huge stock hit. Sadly, NDA.

        I'm not surprised in the slightest.

<comment><USER30> Hard to give an anonymous comment any weight without the slightest verification. Since we know that adversaries of freedom use social media as a disinformation vector, the only thing you can do is ignore them or encourage them to find a way to legitimately disclose the information protected under NDA -- perhaps to the press or a legislator who could help make it possible to invalidate NDAs that keep secrets that make us vulnerable.

<comment><USER29> I agree that panic is ill advised and trust should be given carefully, but if Bloomberg is to be believed, then it happened at least once - and if it did happen once, it's very likely that it happened more than once, because why not? Most organizations don't have resources to find a backdoor like that, nor they have a reason to search for one (had no reason until today I should say) and in this case we only heard about it because of multiple leaks to the press from government officials.

</comment></comment></comment><comment><USER31> A lot more coming from this news for sure, but kudos to whoever found these, that's solid tech due diligence!

<comment><USER32> I'm still waiting for technically literate description - as an electrical engineer I don't know what a "signal conditioning coupler" is - <USER2> DigiKey doesn't find anything under that name - it's not a "common" part, at least in my experience. The part looks like an RF filter look-alike of some sort. But that would normally be hooked into a power rail (and ground) - <USER3> of hard to insert signals into a data line unless it was wired strangely

So tell us what the part is pretending to be and how exactly it it was wired (and what it was connected to - is this another Intel ME backdoor?)

</comment></comment><comment><USER33> Still no statement from supermicro. Nothing on Twitter. Nothing on the website. That's a kind of strange after the stock price droped so much.

<comment><USER34> Yeah there is:

[https://www.bloomberg.com/news/articles/2018-10-04/the- big-h...](https://www.bloomberg.com/news/articles/2018-10-04/the-big-hack- amazon-apple-supermicro-and-beijing-respond)

<comment><USER33> True, but still, I would like to see a statement on the website or on twitter.

</comment></comment></comment><comment><USER35> It remind me another security risk "Google's new hardware security key was made by a Chinese company"

[https://www.cnbc.com/2018/08/30/google-titan-made-by- chinese...](https://www.cnbc.com/2018/08/30/google-titan-made-by-chinese- company-feitian.html)

</comment><comment><USER30> > not much bigger than a grain of rice, that wasn't part of the boards' > original design.

'original design' is hard to verify without help from SuperMicro.

> Amazon reported the discovery to U.S. authorities, sending a shudder through > the intelligence community.

... especially for Amazon. Unless Bloomberg claims here that Amazon got parts of this SKU and compared them to newer parts of the SKU and found differences?

Is it legit to revise your design in terms of changes to passives without rev'ing the part and notifying downstream supply chain. Could the grain-of- rice 'microchip' be a different or new resistor/cap? Could it be logic masquerading as a passive?

> In one case, the malicious chips were thin enough that they'd been embedded > between the layers of fiberglass onto which the other components were > attached, according to one person who saw pictures of the chips.

Like a passive in a blind/buried via?

Anyways, for everyone who claims "it can't be done", "this is implausible" \-- you're probably being a little naive. US intelligence agencies do shipment interdiction and adulterate products for this purpose, [1] why couldn't/shouldn't China do the same?

[1] [https://www.theguardian.com/books/2014/may/12/glenn- greenwal...](https://www.theguardian.com/books/2014/may/12/glenn-greenwald- nsa-tampers-us-internet-routers-snowden)

</comment><comment><USER36> As a side note/question.

I have no idea how things actually work in companies like Apple or Amazon, but would it be "normal" (at their size/scale and given their surely advanced knowledge in technology) to have inspections on the hardware they use (inspections at hardware level of the kind capable of showing these modifications)?

I mean do they routinely do these checks?

<comment><USER37> I had the same question when I read the article. It sure was lucky that both Apple _and_ Amazon checked the hardware and found the impossible to find chip.

Or, assuming the rest of the article is true, someone somewhere found out about the chip and had a quiet word with a few select customers of Supermicro (but decided to otherwise keep mum about the whole thing). Which would go some way to explain the company denials, because the truth might be a bit embarrassing —or gagged by a three-letter agency.

</comment></comment><comment><USER38> OK. So what's the minimum remote system management capability needed in a modern data center? The major cloud sysadmin people should figure this out, write a spec, and insist that's _all_ that goes in. If Amazon AWS and Google wrote a spec for this, the manufacturers would fall into line.

Boards are shipping with way too much remote access capability. It's not like you need to look at system busses via the network. You're not going to debug a broken board remotely, you're going to turn it off and replace it. Now that this is an identified problem, it's time to put IPMI and its ilk back in its cage.

</comment><comment><USER39> Amazon bids for CIA contracts; Elemental bids for Amazon's contracts; Super Micro bids for Elemental's contracts and the PLA "bids" for Super Micro's contracts. Looks like Ouroboros, the global supply chain episode.

</comment><comment><USER40> A little suspect when they say " giving them access to the most sensitive code even on machines that have crashed or are turned off."

When it is turned off? A magical chip that works without electricity is much more valuable than any data which could be exfiltrated from servers. A chip that works without power changes the world more than the iPhone... in the iPhone Age

</comment><comment><USER11> The US needs to make electronics supply chain sovereignty the number one priority of the federal defense budget. Why are we fighting useless drone wars when our country is being attacked on a daily basis in the cyber realm?

If this story is true, it will be an escalation clearly and willfully by the PRC from mere state-sanctioned economic espionage to an act of war against the US.

</comment><comment><USER41> This reaction from Amazon / Apple is reasonable if Bloomberg is lying.

This reaction also seems reasonable if Bloomberg was telling the true.

</comment><comment><USER42> > The White House requested periodic updates as information came in, the > person familiar with the discussions says.

You can't make this up, at least not in Bloomberg, that too in a front page or cover page write up. Truth could be somewhere in between and I am sure we will have more information in the coming weeks.

</comment><comment><USER43> [https://www.reuters.com/article/us-china-cyber-dhs- idUSKCN1M...](https://www.reuters.com/article/us-china-cyber-dhs- idUSKCN1MH00Y)

So what happens to the reporter then?

</comment><comment><USER44> Certainly not saying this didn't happen, or creating a false dichotomy that a nation-state has to be a friend or foe.. but evaluate this in the context of: [https://talkingpointsmemo.com/edblog/white-house-begins- the-...](https://talkingpointsmemo.com/edblog/white-house-begins-the-china- counter-narrative) (in case people get the feeling that this story is being pushed, or feel like it's coming out of nowhere suspiciously close to an election)

<comment><USER45> Stories like this take many, many months of research -- particularly a story as detailed as the Bloomberg. I don't think it's fair to the journalists to simply explain the story away as a government-led conspiracy to mislead voters in the midterm elections.

</comment></comment><comment><USER43> Is this the tiny chip bloomberg is refering to? Maybe they shorted a bunch of aapl and amzn, I'd suggest SEC to investigate the editor who wrote up this crap

[https://www.digikey.jp/product-detail/en/tdk- corporation/HHM...](https://www.digikey.jp/product-detail/en/tdk- corporation/HHM1522B1/445-3987-1-ND/1955584?cur=JPY&lang=en)

</comment><comment><USER46> Now with the annoucement from the Homeland Security Department of the US, it's very clear, even to a kid, that Bloomberg is lying. The question is should the authors, Jordan Robertson and Michael Riley, get punishment for creating this fake story? They have already created damages. And they may also have some, let's try some conspiracy, motivation in favor of the Trump's gov... since the report is "very conveniently" appeared on the same day of Pence's speech.

</comment><comment><USER47> For me it's IRRELEVANT in the report is true or not, the very relevant part is that today's hw is complex enough to be a black box for 99% of buyers so it pose a super-serious security risk. Strong article from publicly known sources are a way to shed the light on such ignored enormous security threat.

The sole solution IMVHO is IMPOSE open hardware and free software by law. We simply can't have our society "nervous system" run on black boxes. It doesn't matter if the rouge in charge is China, USA, a specific vendor or someone else. Our banks, our states, our hospitals, ... relay on such connected black boxes. We also need to re-transfer knowledge from big corporate to PUBLIC, well founded, universities to AVOID dangerous evolution paths like actual IT evolution.

<comment><USER48> You are welcome to write the software for systems you describe and release it for free. Unfortunately, the economic model of the real world is a little more complicated than you seem to realize.

<comment><USER49> Tell this Linus.

<comment><USER48> I love open source software and use it every day. I also donate to organizations who contribute to that community. My issue with OP's proposal, rather, was that those who create software and hardware should be _forced_ to release it for free.

<comment><USER50> A more practical approach would be to at least be forced to provide the ability to run your own software on the hardware you buy, which is also not a thing for the vast majority of hardware produced. At least not for end users anyway.

</comment></comment></comment></comment></comment><comment><USER51> This back and forth between Bloomberg and Amazon/Apple reminds me of the recent allegations against Supreme Court nominee Kavanaugh.

In both situations it's difficult to identify who is telling the truth and is probably impossible to know the truth from the outside looking in.

In both cases I've taken a "who has incentive to lie?" view on the allegations and denials, now this does not mean my hunch is correct at all, but it seems way more likely to me that it's in the best interest of Apple and Amazon to deny this story strongly as it makes both parties look bad and the investigation may be ongoing. I don't believe that Bloomberg made this story up but its understandable to question that maybe their sourcing was iffy, based on Bloomberg's history and the level of sourcing that they have cited I'm leaning towards believing the story as accurate.

<comment><USER52> I personally don't think this is much like the Kavanaugh situation.

For example, Dr. Ford stepped forward publicly, in full knowledge that a massive machine would swing into action to besmirch and defame her, whereas the allegations about espionage were made in confidence. The six(?) individuals making these allegations are risking nothing at this point.

Also, Dr. Ford's allegations reach back into time and are difficult to conclusively prove,* while this is a situation where physical evidence does exist somewhere. It may not be presented to us at the moment, but if the allegations are true, there were physical motherboards that could have been examined to demonstrate the exploits.

What the two have most in common is that we the public are unlikely to be provided with a full and transparent investigation. The FBI's "investigation" into Dr. Ford's allegations did not involve speaking to her or others who could corroborate aspects of her story. Others from Yale approached the FBI and did not get interviewed.

The same is probably true of this story. Even if the allegations are true, for diplomatic or other strategic reasons, the government is unlikely to shine a spotlight on the details.

* Note that "Conclusively prove," is not the same thing as "Obtain enough confidence based on other reports to make a decision about job fitness."

<comment><USER51> I think similarly to how Dr. Ford knew she would be questioned and pushed into the spotlight when making these allegations public, Bloomberg similarly would've had to calculate the risk that Apple, Amazon, and China would push back on this story vigorously and try to discredit Bloomberg.

I guess my basic premise is that I'm more liable to believe that Bloomberg is operating in good faith in reporting this story. Now that does not mean that their sources may not be credible, but I'm more likely to believe they published this story in good faith rather than Amazon and Apple denying the story in good faith.

<comment><USER52> I agree that Bloomberg is reporting this in good faith, and that they corroborated as much as they could have given the information they received.

The credibility of their unnamed sources, their motives, and so forth, this is all unclear at this point. It may turn out that they are courageous patriots blowing the whistle on something very, very big. One can imagine a cover-up at the highest level, with these six individuals risking their careers and possibly their lives to reveal the truth.

Or one can imagine that there are some trade negotiations coming up, and a coördinated effort to plant a story so that politicians can take credit for swooping in and enacting regulations around the security of technology manufactured outside of the USA.

Who knows? I don't.</comment></comment></comment></comment>


IRCRelay Shutting Down - <AUTHOR> https://www.ircrelay.com/

<comment><AUTHOR> I have a feeling this is in at least some part because Mitchell Hashimoto (who is one half of IRCRelay and the guy behind Vagrant and Packer) is focusing all his energy on his new ventures.</comment>


Show HN: StoryTime – Create stories about your code instead of documentation - <AUTHOR> https://storytime.dev <comment><USER4> I like this mode of explaining code things by slowly building up & talking through examples. I have been doing this in talks for a while to go over code bit by bit without live-coding. Examples:

1\. click "present" & advance step by step to see transitions [https://docs.google.com/presentation/d/1KVMwffTe- aam8Mq7LwfZ...](https://docs.google.com/presentation/d/1KVMwffTe- aam8Mq7LwfZzXwtlcy-VRHrBPk20x47_zk/edit?usp=sharing)

2\. Advance with spacebar: [https://sequoia.makes.software/composer- talk/#/7/1](https://sequoia.makes.software/composer-talk/#/7/1)

I describe the process here: [https://sequoia.makes.software/techniques-to- avoid-live-codi...](https://sequoia.makes.software/techniques-to-avoid-live- coding-part-1/#solution-2-code-quot-fade-ins-and-highlighting-)

<comment><AUTHOR> Thanks for sharing. This gives me some target use case to work towards.

<comment><USER5> Everything is about this is amazing except for 2 small things.

1) Not sure about private repos 2) I'm sure it doesn't work against bitbucket :(

<comment><AUTHOR> private repo is in the works. Will be available as an app in github marketplace. Bitbuket and gitlab support coming soon. You can sign up to be notified here.

[https://storytime.dev/error/private](https://storytime.dev/error/private)

<comment><USER5> Subscribed... :)

<comment><USER6> Another Bitbucket vote here. Subscribed.

</comment></comment></comment></comment></comment></comment><comment><USER7> Innovative ways to explain code is a great idea and I love the project.

However, I don't love the name.

These are explanations - not stories (unless we consider stories as being 100% exposition). There's no beat, scene, or chapter structure that pushes meaningful values back and forth in the face of conflict ("make it work" isn't a meaningful value - since it's the obvious goal of every character here), no progression or character building, no subtext or sub-agendas, no sense of agency or suspense where the code might pursue something counter to its purpose...

Feel like it's a bit of hijacking of "story" in order to make headlines, since "code explainer" isn't as sexy.

<comment><USER8> I get where you're coming from, but "story" here seems more akin to a "user story" than literature.

User stories have been used for quite a long time for both software architecture and establishing/informing QA and testing processes.

This strikes me as user stories without the "user". It's the same concept at a different level of abstraction. A "code story" is to a unit test as a "user story" is to an integration test.

<comment><USER7> "User stories" have the potential to be a bit different... or at least we can use our imagination to make it so. "As a first-class klutz I want to have my data synced without hitting a save button." Ok - benefit of the doubt: Once upon a time, Jane was on her way to an interview when she slips on a banana peel and breaks her phone. Since ProductX doesn't sync to the cloud - <USER2> doesn't know where the interview is and tries her best, from memory. Now she happens to have a fantastic memory but she's also dyslexic and the address she stumbles into is a drug den. Mistaking her for a client and her thinking she's getting a job offer, she accepts a deal and walks off with $10MM of drugs. Now the they're after her - terrified and panicked and not wanting to rely on their forgiving nature, she uses her tech skills and the cash to forge a new identity and moves to another country to found a startup - a competitor to ProductX that has cloud storage, ultimately putting ProductX out of business. She didn't need their product to land a job afterall. Happily ever after. Except for ProductX. So ProductX really should add a sync feature.

> This strikes me as user stories without the "user"

It strikes me as "code stories" without the story. They're literally just explaining the code. Glorified comments with a slick UI.

That is useful though, I really do think it's a cool product - I often feel like comments take up a lot of screen real estate and sometimes want to go deeper with explanations. It is a great idea to move them out and make them more like a slideshow with even more opportunity to do so.

User stories aren't stories either, they are feature requirements from a user's point of view.

I feel like this is all the inverse of when we watch movies that have hackers. There's a craft to storytelling, there's a craft to programming, and we can't just assume that superficial similarity is enough. When a character in a movie types a bunch of garbage onto the screen, that isn't programming. When a software manager describes feature requirements from a user's perspective that's not a story.

I think small children are pretty good at telling the difference since they consume stories voraciously... try reading them a book of "user stories" or "code stories" and see what happens.

(I get that you're saying it's not a story in the literature sense - <USER3> that's my point, there is no other sense and these other things are hijacking the term. By "literature" what I really mean is any medium that can properly express a story - song, dance, etc. are all genuine, proven storytelling avenues too.

The reason I jumped on here wasn't to be a downer - it's because I was super excited, I thought someone figured out a way to use code to tell stories! That would be awesome... Unfortunately that's not what's happening here and I was super disappointed. sorry!)

</comment></comment></comment><comment><USER9> The links at the bottom look like ads. I unconsciously ignored them until I visited the page again.

</comment><comment><AUTHOR> Hi, OP here. Would love to get some feedback from HN community.

<comment><USER10> It may be a good idea to move the story text closer to the code. On a large monitor, I have to move my eyes quite a bit and it feels disconnected.

<comment><USER11> +1 to this. Not sure if it was just that 'play' was too fast (I agree with the other comment about 'play' probably not being super useful here), but on my 27" 2560x1440 monitor, it took me a few slides to notice that there was even text at the bottom changing. I thought I was just supposed to read the comments.

</comment></comment></comment><comment><USER12> The story text at the bottom is too far away from the code for me so I constantly need to "switch" between them and I don't have time to read both. Any reason for not putting it next to the code?

<comment><AUTHOR> Thanks. Got similar feedback below as well. Will be moving this around.

<comment><USER6> The extra bounciness of the animation can be a bit distracting after a while too. Maybe have options for different animation styles (fade/none)? Also maybe positioning options for placing the text elsewhere (like top right middle).

</comment></comment></comment><comment><USER13> It moves a little too fast for me to read the message (bottom of my screen) and then flick my eyes back up to the code and internalize what it's doing. I'd prefer speed controls at a minimum but really I just want to hit space/right-arrow to progress when I'm ready... Like a slideshow presentation.

</comment><comment><USER8> Bug:

It looks like you're hitting the `master` branch by default - GitHub allows you to set other branches as the default for the repository. Several of my repos have `primary` as the default branch and don't even have a `master` branch, which causes this to fail.

</comment><comment><USER14> This is very cool. This is basically what I do when I'm making a presentation and talking about a block of code (albeit with a ton of work in Keynote to make it easy to follow).

I think this is a really great way of explaining code, and I hope it this makes it easier for people to adopt.

<comment><AUTHOR> Thatz the idea. Current focus is to get as much feedback and implement them.

</comment></comment><comment><USER15> Great work! I would love to see the comments inline next to the current highlighted lines vs. at the bottom of the screen. Also having this like watch windows would be awesome as well, especially if you can clearly see the different state transitions that happen.

<comment><AUTHOR> Thanks. Will add support for text next to line. There is a watch window available in the editor when creating story. Not used in this story.

</comment></comment><comment><USER16> I see a lot of comments on the usability improvements have already been surfaced, but I wanted to print out that one possibly invisible implication of this is that accessibility is an important aspect of these changes. I hope in addition to the visible changes of keyboard controls some attention. Is paid to ensure that the function of the tool is accessible to people with visual and motor impairments. Having good aria tags for descriptions, controls, politeness settings for changes and ensuring you using the right elements for the controls. The WAI and WCAG guidelines are both great resources to check for more info. (Sorry on mobile and check myself or easily get links).

<comment><AUTHOR> Thanks. Will be spending time on accessibility.

</comment></comment><comment><USER17> In the "Start Now" section, don't make me type something in if you have the default text. Right now, the text appears to be there but clearly we visit [https://storytime.dev//blob/master/](https://storytime.dev//blob/master/) instead of [https://storytime.dev/facebook/react/blob/master/](https://storytime.dev/facebook/react/blob/master/)

<comment><AUTHOR> oops! noted.

</comment></comment><comment><USER18> Very interesting concept, thanks for sharing! Context and intent are critical when trying to understand code - this looks like a useful way to capture that. Merging the call stack display with the "debugger" is IMHO a particularly good idea.

Out of curiosity: have you explored stories for code _over time_ at all? It might be useful to be able to build stories for a block of code that change from commit to commit as the code itself is mutated.

<comment><USER19> Yeah watching a section of code over time was what I thought this was going to be initially. That sounds really useful for sharing the story of "how we got here" to new devs.

</comment></comment><comment><USER20> The text snacks should show up next to the code. Not fixed to the bottom. It makes it hard to follow.

</comment><comment><USER21> > Add memes and GIFs as you walk the code!

Cristo.

<comment><USER22> My whole repo:

[https://gfycat.com/boilingoffensiveass- firstworldanarchists-...](https://gfycat.com/boilingoffensiveass- firstworldanarchists-twincities-cfb)

<comment><AUTHOR> lol

</comment></comment></comment><comment><USER23> A really polished version of this could get you acquired by MS for a hundo million I feel.

</comment><comment><USER24> Very interesting concept. I found that the transitions were too quick for me to read the descriptions at the bottom of the page. It might be nice to set this transition interval or perhaps click to advance.

</comment><comment><USER25> This looks pretty cool. One thing I'd like to see that could make it really useful is oembed support, I'd love to be able to embed a sort of mini-tutorial on my own blog post.

</comment><comment><USER26> There is a typo on the landing page.

StoryTime enables developers to easily simulate a debugger-like visuals to tell or read a story about pieces of code.

There shouldn't be an "a" in front of debugger-like.

<comment><AUTHOR> Thanks. Will fix.

</comment></comment><comment><USER27> I love this. This is how I document my side projects, I tend to write long stories to myself with chunks of code thrown in for clarity. This is much more elegant though.

<comment><USER28> may I see your documented side projects?

<comment><USER27> Unfortunately none of my projects are open source and all of the documentation is in a physical notebook in my desk drawer.

I'm just not a very good programmer so unless I explain to myself what this code does and why, I'm not likely to remember why I wrote it.</comment></comment></comment>


Judge orders FCC to provide net neutrality records [pdf] - <AUTHOR> https://www.courtlistener.com/recap/gov.uscourts.nysd.501225/gov.uscourts.nysd.501225.33.0.pdf <comment><USER2> Why are you posting multiple sources of the same thing within minutes of each other?</comment>


Ask HN: What software are you most thankful for? - <AUTHOR> Personally, I'm thankful for Atlassian's SourceTree.  I'm using version 18.104.22.168 and haven't updated it in over 3~ years.  It's a rock-solid piece of software and is able to adapt to a handful of different workflows from all the places I develop for. <comment><USER2> The traction control on my Mustang that allows me to drive it in the winter and the ability to turn it off in the summer :)</comment>


There's No Evidence Online Dating Is Threatening Commitment or Marriage - <AUTHOR> http://www.theatlantic.com/sexes/archive/13/01/theres-no-evidence-online-dating-is-threatening-commitment-or-marriage/266797/

<comment><USER2> If anything I think the opposite is true. Guys online are far more interested in relationships than guys on the bar circuit and other such places.

<comment><USER3> That's a good point. On average, a single guy out with his friends at a bar or club and with a few drinks in him on a Friday night is thinking about one thing. Yeah, maybe he would be great boyfriend material otherwise, but on that Friday night he's thinking about one thing (or at least mostly one thing). That same average guy on a dating site without those other dynamics might act differently.

</comment></comment><comment><USER4> First, it seems like both articles say "there's no real data, but ...", so I wonder why both articles are on the front page of Hacker News. I guess this article deserves the upvotes, though, because it points out that fact in its title.

Second, I notice that the few stats in this article refer to "using Internet to find partners," and the question isn't about partners but length of partnership.

Third, this article reminds me of the 'industry rebuttal' to one of pg's essays - the one about how glossy magazines all cost the same - which cried out to a whole industry "please rebut this." Someone had to publish a rebuttal, so someone did - although their 'rebuttal' was basically just pointing out that Sarah Palin's new book sold a lot of copies.

In this case, the industry is the online dating industry. This article is the industry rebuttal. It happens to be correct, but there's just something weird about this kind of article to me, and the eHarmony shirts in the accompanying picture smell of PR.

</comment><comment><USER5> Certainly there is a cause-and effect misunderstanding if one says that online dating is threatening monogamy. Monogamy is in crisis for decades as divorce rates (and sex tourism, and the rise of the man-o-sphere) show, and there doesn't seem to be a reversing trend. Online dating is just capitalizing on the trend, nothing more. It's very probable that in the not too distant future commitment will lose its high social value. The question is if societies are prepared in order to avoid side casualties (children).

</comment><comment><USER6> If anyone has not been exposed to Girl Writes What I highly recommend her. A video by her on the marriage topic: <http://www.youtube.com/watch?v=rlvMAS_20K4>

<comment><USER7> Does she have a blog or something? Video is too slow.

</comment></comment><comment><USER8> I like that this article focuses on the word "commitment" rather than "monogamy", instead of conflating them like the first article did.

</comment><comment><USER9> > _Narratively, the story focuses on Jacob, an overgrown manchild jackass who > can't figure out what it takes to have a real relationship. The problem, > however, is not him, and his desire for a "low-maintenance" woman who is > hot, young, interested in him, and doesn't mind that he is callow and > doesn't care very much about her. No, the problem is online dating, which > has shown Jacob that he can have a steady stream of mediocre dates, some of > whom will have sex with him._

I basically stopped reading at the ad hominem. An article that claims to be about "evidence" clearly is not if it needs to win reader approval by bashing the opposing subject. This was a theme in the Atlantic's comments section of the article in question, too.

We can't tell if Jacob is truly a "douche" or not. The last article was all about him but mainly focused on his dating life and not other circumstances of it. And he sounds like an average male professional...what seems offensive is that we're more privy to his desires and decisions and as they say, familiarity breeds contempt.

The OP also blasts the article for not including a single female voice. If we're trying to make an objective evaluation of the effect of online dating, then it does not matter if the article is single gender (either male or female). If it can be argued convincingly that one-half of a couple has much more incentive to leave, then that by definition is enough to have an effect on traditional relationships, no matter what the other gender thinks

<comment><USER10> > We can't tell if Jacob is truly a "douche" or not.

He had a girlfriend for five years and didn't marry her, and then had a girlfriend for two years and didn't marry her. That doesn't mean he's a douche, but he's not much of a prize either.

I suppose this attitude is a generation or more out of date. But it sure looks to me like the younger generation is screwing it up bad. Guess I'd better get some signs to post on my lawn.

<comment><USER11> What's wrong with having a girlfriend for five years and not marrying her? It might be unusual, but it's not wrong. Some people aren't into marriage. Some people like to wait - my brother was in a relationship over 10 years before he proposed (and is now happily married). Don't judge everyone by your own ideas.

<comment><USER2> I know some people like that. But to be honest, I know a lot more couples where the guy just kicks the can down the road, paralyzed at the thought of sleeping with the same woman forever. Which is fine, except when they deflect instead of being forthright about their feelings and letting the woman decide whether to keep going.

<comment><USER7> Isn't it statistically most often the woman who breaks up? I suspect that might be because it is easier for women to find new partners.

</comment></comment></comment></comment></comment><comment><USER12> _And more choices mean less satisfaction. For example, if you give people more chocolate bars to choose from, the story tells us, they think the one they choose tastes worse than a control group who had a smaller selection._

Huh. It's the opposite for me -- the more possibilities I can check out, the better I feel about my choice in the end. (Maybe that's one reason I didn't marry until age 42 :-) HN readers -- do you agree with this study, or not?

<comment><USER13> The study didn't find that this was true in every case. It found that it was more often true than false in the task that they chose. I'm not totally up on this literature but I'm fairly sure that it's been replicated repeatedly in a variety of domains.

<comment><USER12> Oh, I don't doubt that it's true of most people. I'm interested in the next question: are there interesting characteristics shared by people that this isn't true of?

</comment></comment></comment><comment><USER14> There's also no evidence cats can levitate.

I don't get it.

</comment><comment><USER15> Wow. Alexis Madrigal is a massively negative person. I refuse to read anything with such tone.

</comment><comment><USER16> I have to ask, is there anyone who honestly believed this?</comment>


Docker is awesome on Mac - <AUTHOR> https://deis.com/blog/2016/docker-for-mac/ <comment><USER2> I'd love to use it, however, I still need a method of using Virtualbox and other Vagrant installs. Any tips?</comment>


The problem with taking money from big VC's. - <AUTHOR> http://cdixon.org/2009/08/14/the-problem-with-taking-seed-money-from-big-vcs/

<comment><USER2> The content of this post isn't unique to BIG VCs. Anytime you take money from someone that is capable of following-on, and they don't, it's a major bad signal. However, on the flip side, a professional VC that comes in at A or later is going to know that any investor w/ a strategy that doesn't include following on, is gonna be biased in the other direction.

The bigger miss, from a post like this, is saying the problem is getting in bed with a BIG VC too early. Take money where you can get it from, if you need it. Most people will never be in a position to choose one firm's money over another. And today's climate, most rounds are syndicated (this post is a few years old). However, the risk / rewards of getting involved with any professional investor is they give you capital to accelerate growth, but will kill you if you fail to meet or exceed expectations. Big VCs want to follow on and aren't looking to tank a successful company. However, if you don't do your job, they won't normally throw good money after bad.

Investors only become a problem if the founder(s) don't do their job, which is to grow HUGE FAST! If you do that, you won't have to worry about the aforementioned situation where a BIG VC failing to follow-on kills you in a future round.</comment>


Ask HN: What if Microsoft held the country's pcs for ransom? - <AUTHOR>

This is slightly "silly" but I was thinking today about how much of the country runs on Microsoft products, including the government. How much power could they leverage from that? For instance, if Bill Gates made a public announcement "Stop monitoring our shit, or we deactivate all copies of Windows in the country." I know, I know, there are plenty of issues with this, but I wanted to share my thought experiment with you all and see if anyone had any insights/thoughts. <comment><USER2> Extortion is a crime, for one thing. And it's a pretty hypocritical threat considering Microsoft has been _more_ than willing to put hooks in it's products to enable law enforcement/intelligence surveillance.

</comment><comment><USER3> Sounds like a TODO for a desparate Steve Ballmer</comment>


A Rare Peek Inside Amazon's Massive Wish-Fulfilling Machine - <AUTHOR> http://www.wired.com/2014/06/inside-amazon-warehouse/ <comment><USER2> It only mentions the working conditions in those fullmilment centers as a brief aside: apparently the building is air-conditioned and water is readily available -- no mention if there's _time_ available for a drink during your shift. I guess you can write an article about just the technical marvels of these buildings, but don't forget that it's only half of the picture.

The motto "work hard. have fun. make history." printed in 2m letters struck me as Orwellian.

<comment><USER3> This was the best article I've seen so far for describing and understanding the conditions inside:

[http://www.motherjones.com/politics/2012/02/mac- mcclelland-f...](http://www.motherjones.com/politics/2012/02/mac-mcclelland- free-online-shipping-warehouses-labor)

<comment><USER4> One could argue that Amazon just simply couldn't function if it wasn't as organized as it is. If the employees could take as much time as they like to walk to get the items they would need more people just to make up for the uncertainty, if employees could bring their own possessions in and out, inevitably things would get lost, and if they got lost, the system would not be accurate anymore,orders could not be shipped out and would have to be delayed/cancelled = reduced customer satisfaction. I have read this article you linked and I really find it hard to sympathize with the person who wrote it. She was paid a wage to do a job and to do it well - I cannot blame Amazon for making sure she was doing what she was paid to do.

<comment><USER2> If a company can't function without relying on behaviour that is unethical, then it can't exist; it's as simple as that. What exactly constitutes unethical behaviour crosses the line is up for debate (and usually codified as law), the consequences aren't really.

There's also a difference between making sure your employees are doing their job through monitoring on the one hand and on the other hand having an inhumane expectation of what constitutes their job in the first place. I think both can be done to a degree that's unacceptable (and German law e.g. poses some boundaries on both employee monitoring as well as maximum daily/weekly work hours etc.).

<comment><USER5> It doesn't seem like it's a permanent situation - increase in temporary labor usually happens during high-traffic pre-holiday season (also known as the magical spirit of Christmas) and the reason Amazon had to bring in so many temporary workers is that at the time of the article the Kiva Robotics technology was not up to snuff.

</comment></comment></comment></comment></comment><comment><USER6> Ugh: "Unlike software, fulfillment centers aren't built overnight."

<comment><USER7> Maybe the author is referring to a really complicated build script that takes a few hours to run! But probably not.

Though really, "overnight" is just a figure of speech. Many startups probably are formed, ship a product, and die in the amount of time it takes to build a large fulfillment center.

</comment></comment><comment><USER8> Interestingly, AMZN bought Kiva and will deploy 10,000 robots soon. Very cool video here: [http://money.cnn.com/2014/05/22/technology/amazon- robots/](http://money.cnn.com/2014/05/22/technology/amazon-robots/)

<comment><USER9> Do you have another source for that video? It keeps playing the ad twice and then loading an unrelated video.

<comment><USER10> Found this on youtube: [https://www.youtube.com/watch?v=lWsMdN7HMuA](https://www.youtube.com/watch?v=lWsMdN7HMuA)

<comment><USER9> I was hoping the one aditya linked was newer as the article was written this year. Did they reuse that old video or is there indeed new footage?

<comment><USER8> Not the same video, this has more real world footage...

</comment></comment></comment></comment></comment><comment><USER11> Just going to throw this out there, but the software division of amazon fulfillment is always hiring. I'm a developer here and I really enjoy it. Interesting problems, and you get to visit places like PHX6 fairly often. Based in Seattle, Toronto and I believe Bangalore.

My username at Amazon. Happy to answer questions.

<comment><USER12> Working conditions in FC's have been criticized in the past. Is there any consideration in the optimization process of minimizing human usage/energy, or is it just all "get as much stuff into a space as possible and then out the door as quickly as possible, however it needs to be done?"

<comment><USER13> I've spent time at Amazon, I suspect: \- Amazon persistently strives for efficiency and frugality over all else: applies in FC and HQs. I think this culture could turn off many people \- Kiva will reduce human usage/energy (not sure if that's where you were headed though) \- Inevitably, optimization means reducing the cost (inc human effort) required to store and fulfill products \- Having visited FCs, the ones I saw had fine working conditions though I understand the reality of publicity in the past

</comment></comment></comment><comment><USER14> Looks like it took a huge amount of engineering to build. The extensible conveyor belt going into the truck took some effort to design and build. The software wouldn't have been terribly difficult to implement, but is still impressive how it keeps track of where everything is. I'm wondering if their software keeps track of the dimensions of objects, so that it tells the workers what shipping box to use and how to arrange the objects in it.

<comment><USER15> The inventory database definitely knows dimensions: that's key data to making the whole operation work. It isn't just the box at the end, but also how much shelf space, what kind of shelf, whether it can be moved by a single person...

A 60" TV and a TV remote need vastly different things to happen in the FC.

There is a lot more to the software than meets the eye too: trip planning, various UIs for pickers and packers to read. Predictive modeling for where to store products to minimize pick time.

<comment><USER16> It doesn't cease to amaze me how fast people dismiss the actual complexity of tasks they don't understand. As you pointed out, there is a lot going on that most people overlook when it comes to managing inventory.

I am in distribution, we have many interconnected systems to manage our inventory, whether it is in stores or distribution centers. Within each there are many places and item can and cannot go, whether because of size, weight, volatility, or toxicity.

Who processes are dedicated to having large volume items readily available to minimize trip times for stocking and fulfillment.

I am curious if Amazon relies on separate system for each location or centrally managed. There are benefits to both

</comment></comment></comment><comment><USER17> Wow, I didn't realize that one map on COD: Black Ops was designed after an Amazon warehouse.</comment>


Jump Point Search (JPS)+: Over 100x Faster Than A* - <AUTHOR> http://www.gdcvault.com/play/1022094/JPS-Over-100x-Faster-than%20jps+%20goal%20bounding

<comment><USER2> Unfortunately, JPS requires a regular grid. And "JPS+" as described here requires (a lot of) computation, which means that it's not directly comparable with A*. (Once you start allowing precomputation, a) it is not longer usable with changing terrain, and b) you have to compete with "precompute all possible shortest paths and store them" (O(1) lookups!).)

Now what I'd like to see is someone coming up with pathfinding that works decently with obstacles being arbitrary polygons. Vector-based as opposed to raster-based.</comment>


Prince, a Master of Playing Music and Distributing It - <AUTHOR> http://www.nytimes.com/2016/04/23/arts/music/prince-music-technology-distribution.html <comment><USER2> > _" prince.org crashed.... The site is a remnant of an earlier Internet, with > an ancient design not built for the speed and density of the modern online > world."_

The author is unaware these "ancient designs" are lightweight and actually better suited to speed and traffic. The site crashing has nothing to do with an outdated design and everything to do with the hosting plan the site is on.

The big story (IMHO) not mentioned here, is all the unheard and unreleased songs and videos he locked away in his vault. In a fascinating and hilarious story told by Kevin Smith, he describes how he briefly worked with Prince on a documentary. Smith met a Producer for Prince who told him she'd produced more than 50 Prince music videos of songs nobody has heard and videos nobody has seen. All locked away in the vault. I really like how crazy this is, and hope they are released.

The story told by Smith is about 30 min long, but worth it if you're curious about how odd Prince was, and up for a laugh: [https://youtu.be/8LhcParuzpc](https://youtu.be/8LhcParuzpc)

If you just want to see a really good example of Prince doing what he does best, with awesome female drummer and rockin band, turn this up loud: [https://youtu.be/nyyS0FSztKc](https://youtu.be/nyyS0FSztKc)

<comment><USER3> Those ancient pages could be hosted on a scalable CDN to hand the traffic. The hosting design is antiquated, not the pages.

<comment><USER2> I've never heard the term "hosting design" before! As a charitable gesture let's assume that's what was meant rather than "looks old so must therefore run like a 1974 Datsun".

<comment><USER4> I haven't heard that before either, but I think we engineers have associated design with the look and feel when it's broader than that.

</comment></comment></comment></comment><comment><USER5> In some ways, though, he was apparently a master of _not_ distributing music.

I heard a story today on my local NPR station where they interviewed the maker of the documentary "Hunting for Prince's Vault". Many years ago once of Prince's engineers set up a vault to keep all of Prince's unreleased material. (And they made it a literal vault...complete with the kind of door you find in banks).

It was interesting. Prince was apparently pretty much always writing new songs, and making studio recordings of them, and promo videos for them...but then not releasing them. People would find him in the studio recording on holidays and ask why he was working on a holiday, and he'd tell them he wasn't working.

From what it sounds like, he wrote and recorded music the way Isaac Asimov wrote. It was what made him happy and what he wanted to be doing all the time.

As far as the documentary maker was able to figure out from talking to Prince's engineers and collaborators and such, the vault contains enough finished and recorded songs for over 100 albums, plus a couple full length movies, music videos, promo video, concert footage.

<comment><USER6> There's a good Kevin Smith story (though long) about Prince hiring him to make a documentary... Smith basically took time out of his busy schedule because he was star struck, spent a bunch of time with Prince, made a documentary and then asked his assistant what the plan was, when they wanted to release, etc... She laughed and just said essentially that it would never see the light of day and was just going into his vault with all the other stuff.

edit: exodust mentions this below as well

</comment></comment><comment><USER7> I have a vague worry that Prince's estate will be instructed to destroy everything in the vault. That would be a major loss to 20th century cultural music heritage. But, it would be in keeping with Prince's MO, someone who largely did as he liked and stayed true to his artistic vision.

Anyway, hopefully instead we'll have 10 years of new Prince albums as they go through the vault bit by bit.

</comment><comment><USER8> pretty much every prince everything is sold out on amazon.com right now.</comment>


Chrome Has a Document Design Mode - <AUTHOR> https://dev.to/arbaoui_mehdi/edit-your-website-live-in-google-chrome-by-using-document-designmode-5d9h <comment><USER2> Is this different than

      document.querySelector("html").contentEditable = true;

    ?

<comment><AUTHOR> Yes. This is like editable text for any text on the screen enabled.

<comment><USER3> But that's what contentEditable already does?

<comment><AUTHOR> Right. Turning this mode on makes everything contentEditable so you can change up a website.

For example, let's say your making mockups and want to change some text. This lets you do that really quickly.

<comment><USER4> Yes, we heard you the first time. The code snippet kn0where posted has the same effect. :)

I looked into this some more, and I think the actual difference between `document.documentElement.contentEditable` and `document.designMode` is subtle and probably mostly just in principle: since contentEditable is set on elements, someone with only access DOM access could potentially toggle it. But someone with only DOM access cannot turn `document.designMode` off.

</comment></comment></comment></comment></comment><comment><USER5> This is not specific to Chrome.

`document.designMode` is a web standard and is supported by Firefox, Safari, IE, etc.

[https://developer.mozilla.org/en- US/docs/Web/API/Document/de...](https://developer.mozilla.org/en- US/docs/Web/API/Document/designMode)</comment>


Whole Genome Sequencing (WGS) – Full DNA Analysis - <AUTHOR> https://www.dantelabs.com/collections/our-tests/products/whole-genome-sequencing <comment><USER2> Looks sketchy, the company doesn't identify its owners.</comment>


BitcoinMessenger - Send a public message using the Bitcoin blockchain - <AUTHOR> http://www.btcmessenger.com

<comment><USER2> I don't understand the relevance of the title to the service, but it seems like an interesting way of using the bitcoin blockchain to strongly time-stamp messages.

Also the fact that it costs you money to send a message and the more you spend the more "important" the message is happens to be interesting.

</comment><comment><USER3> So wait....each time someone sends a message with this service...they are also just handing this guy some money? Genius!

</comment><comment><USER4> Where do you write the message?

</comment><comment><USER5> What will you do with the dust?

</comment><comment><USER6> How does this work?</comment>


code-server: Run VS Code on a remote server - <AUTHOR> https://github.com/cdr/code-server <comment><USER2> Is this stable enough thou? I've used it about 6 months ago and it was unstable and many features did not work.</comment>