An administrator accidentally deleted the production database - <AUTHOR> http://support.gliffy.com/entries/98911057--Gliffy-Online-System-Outage <comment><USER5> My very first job - ~25 years ago.

Destroyed the production payroll database for a customer with a bug in a shell script.

No problem - they had 3 backup tapes.

First tape - read fails.

Second tape - read fails.

Third tape - worked.... (very nervous at this point).

I think most people have an equivalent educational experience at some point in their careers.

Edit: Had a project cancelled for one customer because they lost the database of test results..... 4 months work! Their COO (quite a large company) actually apologised to me in person!

Edit: Also had someone from Oracle break a financial consolidation system for a billion dollar company - his last words were "you need to restore from tape" and then he disappeared. I was _not_ happy as it was his attempts at "improving" things were the cause of the incident! Wouldn't have been angry if he had admitted he had made a mistake and worked with us to fix it - <USER2> saying "restore from tape" and running away was not a good approach.

<comment><USER6> A coworker of mine used to say "It's not the backup, it's the restore." Meaning your backup process isn't meaningful unless you have a tested and effective means for recreating your system from that backup. It has stuck with me.

<comment><USER7> How do you do test your restore without destroying the only trustworthy copy of your data?

<comment><USER8> Do it in the Staging or User Acceptance Testing environment.

<comment><USER9> This is the right answer! You want a staging/acceptance/mirror environment that's the same as production, right? So make it with nightly restores of the production backups. You get a crisp, fresh staging environment, and regular validation of your backups too. Just remember to run full production monitoring against your staging environment too.

</comment></comment></comment></comment></comment><comment><USER10> This is what happens when you don't have a disaster recovery plan, or if you have one but never test it out. You need to test your disaster recovery plans to actually know if things work. Database backups are notoriously unreliable, especially ones that are as large as the one this post is talking about. Had they known it would take 2-3 days to recover from a disaster I'm sure they would have done something to mitigate this. This falls squarely on the shoulders of the VP of Engineering and frankly it's unacceptable.

I worked at a company that was like this. My first question when I joined was, "do we have a disaster recovery plan?" The VP of engineering did some hand waving, saying that it would take about 8 hrs to restore and transfer the data. But he also never tested it. Thankfully we never had a database problem but had we encountered one we would have lost huge customers and probably would have failed as a business.

I also worked at a company that specializes in disaster recovery, but our global email went down after a power outage. The entire company was down for 1 day. There were diesel generators but they never tested them and when the power outage occurred they didn't kick in.

Case in point: Test your damn disaster recovery plans!!!

<comment><USER11> Speaking from firsthand experience, business doesn't care. The managers know that a) no matter what happens it's likely that their jobs aren't on the line, b) they already have stockholder money and so can just hand-wave any problems away as "once-off and we've fired the staff responsible".

So what I get to see are DR plans that are obviously faulty, where they cannot be tested for something as simple as we don't have 20TB of extra disk handy to do a single failover.

"That's okay", the boss will say, "as long as we have it on paper."

Okay dude. As long as I have your comment in an email to protect myself. I'm okay with being fired for something I warned everyone about, as long as I can also show that to my next boss, to prove my common sense^H^H^H^H^H^H^H^H^H^Hexpert advice gets overridden.

</comment></comment><comment><USER12> I was testing disaster recovery for the database cluster I was managing. Spun up new instances on AWS, pulled down production data, created various disasters, tested recovery.

Surprisingly it all seemed to work well. These disaster recovery steps weren't heavily tested before. Brilliant! I went to shut down the AWS instances. Kill DB group. Wait. Wait... The DB group? Wasn't it DB-test group...

I'd just killed all the production databases. And the streaming replicas. And... everything... All at the busiest time of day for our site.

Panic arose in my chest. Eyes glazed over. It's one thing to test disaster recovery when it doesn't matter, but when it suddenly does matter... I turned to the disaster recovery code I'd just been testing. I was reasonably sure it all worked... Reasonably...

Less than five minutes later, I'd spun up a brand new database cluster. The only loss was a minute or two of user transactions, which for our site wasn't too problematic.

My friends joked later that at least we now knew for sure that disaster recovery worked in production...

Lesson: When testing disaster recovery, ensure you're not actually creating a disaster in production.

(repeating my old story from [https://news.ycombinator.com/item?id=7147108](https://news.ycombinator.com/item?id=7147108))

</comment><comment><USER13> Treating app servers as cattle, i.e. if there's a problem just shoot & replace it, is easy nowadays if you're running any kind of blue/green automated deployment best practices. But DBs remain problematic and pet-like in that you may find yourself nursing them back to health. Even if you're using a managed DB service, do you know exactly what to do and how long it will take to restore when there's corruption or data loss? Having managed RDS replication for example doesn't help a bit when it happily replicates your latest app version starting to delete a bunch of data in prod.

Some policies I've personally adopted, having worked with sensitive data at past jobs:

\- If the dev team needs to investigate an issue in the prod data, do it on a staging DB instance that is restored from the latest backup. You gain several advantages: Confidence your backups work (otherwise you only have what's called a Schrödinger's-Backup in the biz), confidence you can quickly rebuild the basic server itself (try not to have pets, remember), and an incentive to the dev team to make restores go faster! Simply knowing how long it will take already puts you ahead of most teams unfortunately.

\- Have you considered the data security of your backup artifacts as well? If your data is valuable, consider storing it with something like [https://www.tarsnap.com](https://www.tarsnap.com) (highly recommended!)

\- In the case of a total data loss, is your data retention policy sufficient? If you have some standard setup of 30 days worth of daily backups, are you sure losing a days worth of data isn't going to be catastrophic for your business? Personally I deploy a great little tool called Tarsnapper (can you tell I like Tarsnap?) that implements an automatic 1H-1D-30D-360D backup rotation policy for me. This way I have hourly backups for the most valuable last 24 hours, 30 days of daily backups and monthly backups for a year to easily compare month-to-month data.

Shamless plug: If you're looking to draw some AWS diagrams while Gliffy is down, check out [https://cloudcraft.co](https://cloudcraft.co) a free diagram tool I made. Backed up hourly with Tarsnap ;)

<comment><USER14> I've found tarsnap to be slow at restoring in the past. My recollection is a few hours for a ~1GB maildir. I was using it for my personal things but I would (as would anything) test restore times if I was using it for serious stuff.

<comment><USER13> The amount of de-duplication performed by Tarsnap, and the amount of files, which for a maildir I imagine is a lot of tiny files, probably negatively impacts it. Dealing with a single DB dump file the performance is fine so far at least. I can imagine one could also try to partition the data into multiple independent dumps that can run in parallel during the restore if speed became a concern.

<comment><USER15> You can also ZFS send an entire filesystem snapshot, very efficiently, to rsync.net:

arstechnica.com/information-technology/2015/12/rsync-net-zfs-replication-to- the-cloud-is-finally-here-and-its-fast/

[http://www.rsync.net/products/zfsintro.html](http://www.rsync.net/products/zfsintro.html)

<comment><USER16> What is the benefit of rsync's $0.20/GB pricing over any other cloud storage solution that costs $0.01-$0.03/GB?

<comment><USER17> I've been a customer of theirs for a long while and will note that their customer service is amazing. They helped implement what I needed and offer support anytime I need.

Now that S3 has matured and prices have continued to drop, though, I am going to be moving to trim costs. I actually kicked off backups to S3 earlier this month and am backing up to both S3 and rsync.net at the moment, with the plan of ending rsync once I've tested restores and made it through a billing cycle at Amazon.

<comment><USER17> Wow. To stress the amazing level of customer service, someone there just ran across my comment and reached out - noting that my pricing was set for an older structure, updating me to a far more competitive rate and offering a retroactive credit.

While Amazon has offered some great service, it's never been as good as that.

They really do stand by and provide a superior level of support and assistance if you need it on the technical side as well.

I highly recommend them.

</comment></comment></comment></comment></comment></comment></comment><comment><USER18> Been there Done that :)

I was once on-call working for one the leading organizations. I got a call in the middle of the night that some critical job had failed and due to the significant data load, it was imperative to restart the processing.

I login to the system with a privileged account. Restart the job with new parameters and since I wanted not to see the ugly logs, I wanted to redirect the output to /dev/null.

I run the following command ./jobname 1>./db-file-name

and there is -THE DISASTER-

For some reason this kept popping in my head - "Bad things happen to Good people"

We recovered the data but there was some data loss still as the mirror backup had not run.

Of course, we have come long way since then. Now, there are constant sync between Prod/DR and multitude of offline backups and recovery is possible for last 7 days, the month or any month during the year and the year before.

<comment><USER19> I was doing a favor to a friend and on a refferal talked to a guy who didnt have anything but weekly backups and had a corrupt database due to some drive failures.

I was able to determine that the corrupt data was repairable if we had a copy of the old db, and since it was a tiny machine system I asked "Would you mind restoring the backup side by side with production and I can do what I need?"

"Sure thing!"

I wait for a minute, and then my connection to the production database dies.

I refresh the client, and now the one database available is restoring from a backup...

I called him and asked if he meant to overwrite his production copy with his backup instead of do it side by side, and he says petulantly, "I didnt do that!"

I ask him to check again, and he responds with "I will call you right back!"

Five minutes later I get the call, "How do I roll back my restore partially through the restore process?"

Oops.

</comment></comment><comment><USER20> We've all been there. Shit happens. That's what backup is for.

OT: It's probably bad form to publicly blame someone for it, even if it's done by him. It's suffice to say, we screwed up but on our way to recovery. It's better to follow the practice of praising in public and discussing problem in private.

<comment><USER21> I worked on a team that had a list of "breakfastable offences" \-- violating these rules meant you had to bring in breakfast for the whole team (donuts, bagels, whatever). One of them was "throwing someone under the bus." In conversations with anyone outside the team, you weren't allowed to single out a person as responsible for any particular bug/error/etc.

Granted, this is pretty vague (depending on how many "administrators" the company has), but it's still too specific for me.

<comment><USER22> On a public relations note, though: I think a case could be made that it was important to give some specifics about who is to blame. Consider the alternative:

"We discovered the production database had been deleted but we are now working diligently to restore it"

How are people -- both non-technical and the HN crowd -- _not_ supposed to suspect that this is a result of an external malicious hack?

<comment><USER21> The organization can take responsibility for the issue. "During a system update, we mistakenly deleted a production database. We are restoring it and shoring up our disaster recovery plan."

That's very different from "During a system update, Dave mistakenly deleted a production database." In an organization with 5 or 10 people, "During a system update, our administrator mistakenly deleted a production database," is still identifying.

Like I said, I'm not sure it's an issue in this particular case. I don't personally know anything about the site in question.

</comment></comment></comment></comment><comment><USER23> This is how I learned about xargs ...

I once typed onto a client's production mail and web server that basically ran the whole business for about 50 staff, as root, from the root directory:

chmod -R 644 /dirname/ *

I seem to recall the reason was that tab completion put a space at the end of the dirname, and I was expecting there to be multiple files with that name ... anyway the upshot was that everything broke and some guy had to spend ages making it right because they didn't have the non-data parts of the file system backed up.

I learned that whenever you do anything you should:

find . -name "*.whatever you want" | more

then make sure you're looking at expected output, then hit the up arrow and pipe it into xargs to do the actual operation.

<comment><USER24> When root or prod, the paranoid (like me) always start every mutating command with a '#' to prevent a sneeze from prematurely sending the command and doing damage.

<comment><USER25> [ $[ $RANDOM % 6 ] == 0 ] && rm -rf / || echo "You live"

For a good measure at the end of the day

<comment><USER26> Oh, the good ol' shell russian roulete.

</comment></comment></comment></comment><comment><USER27> BTDT. Got the t-shirt. Early in my career...

* Multiple logins to the conserver, down the wrong system.

* rm -rf in the wrong directory as root on a dev box, get that sick feeling when it's taking too long.

* Sitting at the console before replacing multiple failed drives in a Sun A5200 storage array under a production Oracle DB, a more senior colleague walks up and says "Just pull it, we've got hot spares" and before I can reply yanks a blinking drive. Except we have only two remaining hot spares left and now we have three failed. Under a RAID5. Legato only took eight hours to restore it.

* Another SA hoses config on one side of a core router pair after hours doing who knows what and leaves telling me to fix it. We've got backups on CF cards, so restore to last good state. Nope, he's managed to trash the backups. Okay, pull config from other side's backup. Nope, he told me the wrong side and now I've copied the bad config. Restore? Nope, that backup was trashed by some other admin. Spent the night going through change logs to rebuild config.

There were a few others over the years, but all had in common not having/knowing/following procedure, lacking tooling, and good old human error.

<comment><USER28> Is it really a good idea to use RAID 5 on a database? If the database is large enough rebuild time can be more lengthy than a straight restore and under many RAID 5 setups you have the added problem of slower write performance.

<comment><USER27> > Is it really a good idea to use RAID 5 on a database

Hell no. Had I been in involved in that setup it would have been RAID 10 or RAID 50. Actually, had there been some planning there would have been a second array and it would not have been physically co-located in the same rack as the first so when the cooling or power inevitably fails it won't take out both. But, you know, not my circus.

</comment></comment></comment><comment><USER29> If that administrator is reading this, chin up ... it happens to the best of us.

<comment><USER30> When I was relatively new in my first job I forgot to include the WHERE clause in an update, essentially resetting the value for the entire table. Needless to say I felt awful and I was ready to hand in my resignation right after the issue was sorted out (I even printed my resignation letter). Luckily there was a relatively recent backup (not as recent as it should've been though... but I obviously wasn't the DBA) and things went back to normal relatively soon. Throughout the process the team shared their DB-related war stories with me. Everyone seemed to have had a similar experience happen at some point during their careers and knowing that made me feel a lot better. I ended up changing my mind and decided not to quit.

<comment><USER31> Much like the folk putting echos into find and wildcarded shell commands to check the output, I'll often start manual sql updates by doing 'SELECT something FROM database WHERE...' and check the output rows match my expectations before hitting the up arrow to replace the SELECT with an UPDATE.

For bigger tables, I use COUNT(something) if I expect it to be long but I have an idea of rows affected, or LIMIT if that's going to give me an idea that it's doing the right thing.

</comment></comment></comment><comment><USER32> Not long ago I discovered backups don't do any good if you delete them. The incident went down while I was wiping out my hard drive to do a fresh install of Fedora. I believe what happened may have been due to sleep fatigue.

Everything is a bit hazy. At one point in my wandering on the command line I found the mount point for my external backup drive. "What's this doing here?" and decide to remove it.

At some point I woke up in a panic and yanked the usb drive off the my laptop. Heart pounding. "Oh shit."

I actually felt like I was going to get sick. Tax records, client contact info, you name it, all gone. Except, basically, the pictures of my kids, mozilla profile, and my resume files.

While I reconstructed some of the missing files there a bunch that would be nice to have back. All of the business records though have had to be reconstructed by hand. By the next day I did realize I really only cared about the pictures of my kids in the end. And those were somehow saved from my blunder.

Work flow change: backup drive is only connected to laptop while backups are being made or restored. Disconnected at all other times. A third backup drive for backups of backups is on the todo list.

<comment><USER10> I have about 5 backups of things that I need. I just buy a new external drive and copy everything over and leave it in the closet. And then in a year buy another one. $200 a year for a backup of my photos is worth it

<comment><USER33> If you don't have at least 3 copies in at least two different locations your data is already vapourizing.

So 5 inexpensive backups of important data sounds just about rasonable.

</comment></comment></comment><comment><USER11> The Enterprise I work for is currently implementing a new idea - where they hire a crack team of generalists - and give them complete and utter unfettered access to production (including databases).

This is despite our databases being controlled by my team and having the best uptime and least problems of anything in the entire business. Networks? Fucked. Infrastructure? Fucked. Storage? Fucked. But the databases roll on, get backed up, get their integrity checks, and get monitored while everyone else ignores their own alarms.

The reasoning for this is (wait for it...) because it will improve the quality of our work by forcing us to write our instructions/changes 3 MONTHS IN ADVANCE for generalists to carry out rather than doing it ourselves. 3 MONTHS. I AM NOT MAKING THIS UP. AND THIS IS PART OF AN EFFICIENCY STRATEGY TO STEM BILLIONS OF DOLLARS IN LOSSES.

Needless to say the idea is fucking stupid. But yeah, some fucking yahoo meddling with the shit I spent my entire career getting right, is sure to drop a fucking Production database by accident. I can guarantee it. Your data is never safe when you have idiots in management making decisions.

<comment><USER34> Eh, yes and no.

You're too focused on the idea that those generalist are a bunch of skill-less dipshits. As one of those generalist skill-less dipshits, my calloused perspective is that DBAs are the absolute most obstinate, narrow minded twats that exist in any sort of enterprise arena - worse than that PM you probably hate. They just suck! I can think of maybe one DBA who didn't flatout stink of the 20 or so I've worked with. For some reason, there's just a complete lack of understanding of anything that's NOT a database, even though their database understanding is so incredibly deep. Y'all could use some more generalists.

An example of an obstinate DBA is one from my last place, who I wanted to take root access from. She had root ssh keys all over the place, sudoers entries in random places, passwords in her history, etc. It was a security nightmare. She absolutely refused to allow me to take away her root access. She wouldn't even allow any discussion. Her reason? "I need root to install mysql". Management agreed.

There's a reason "That's something a DBA would do." has become a running joke at multiple places I've worked at.

Edit to add: These problems could easily be solved if there was less silo'ing going on. If everything but the database is awful, then that's an indication of deeper, awful and likely legacy problems, not just with the generalists.

<comment><USER11> > You're too focused on the idea that those generalist are a bunch of skill- > less dipshits.

In this specific case it's because I've been working with the quality of dipshits in the departments they are being pulled from, over the past few years. They are going to be cross-trained by dipshits from those other departments, so that they can become even worse generalists.

Hmmm. They don't care about backing up servers. They don't care about HA cluster alarms or failovers. They don't notice or proactively monitor disks filling despite being the sole custodian of the Enterprise monitoring solution. They don't care about Windows security logging policies or even the power plans. They manage AD but let service accounts expire all the time instead of following anyone up first; leading to many outages.

I'm struggling to think of anything good they do. There's no quality or pride to their work; they use GUIs. They get by because the few time I've seen other managers criticise their boss, that boss has then filed official complaints of harassment - and then everything quiets down and goes back to the status quo.

> there's just a complete lack of understanding of anything that's NOT a > database

Guilty as charged. I don't care about anything outside of the database because it's not my job ;-) However I do know a little about the server level backups, clustering, performance counters, security settings, and such - anything that affects my uptime - and I monitor it, unlike the people who are paid to do so.

> An example of an obstinate DBA is one from my last place, who I wanted to > take root access from.

Oracle people have root access on Oracle boxes. We have admin access on Windows boxes. It's extremely difficult for just a few staff to manage hundreds of servers in a high quality fashion otherwise.

> There's a reason "That's something a DBA would do." has become a running > joke at multiple places I've worked at.

There are plenty of shit DBAs, and obviously there are good Infrastructure people as well especially on HN. I hope you realise - that DBA you were talking about - likely isn't bothering to read HN either. I am somewhere near the top middle of my profession.

> These problems could easily be solved if there was less silo'ing going on

Totally agreed.

> then that's an indication of deeper, awful and likely legacy problems, not > just with the generalists.

Entrenched management and yes-men-or-you're-fired culture.

<comment><USER34> Ah, 100% fair enough! Didn't really mean to come off as critical if it came across that way. The problems are so systemic that it's worth mentioning, I suppose.

Do you work in finance? These problems you're describing are all too familiar.

</comment></comment></comment></comment><comment><USER35> About 15 years ago, my school's electrical engineering lab had a fleet of HP- UX boxen that were configured by default to dump huge core files all over the NFS shares whenever programs crashed. Two weeks before the end of the semester a junior lab assistant noticed all the core files eating a huge chunk of shared disk space and decided to slap together a script to recursively delete files named "core" in all the students' directories.

After flinging together a recursive delete command that he thought would maybe work, he fired it off with sudo at 9:00pm just before heading out for the night. The next morning everyone discovered that all their work over the semester had been summarily blown away.

No problem, we could just restore from backups, right? Oh, well, there was just one minor problem. The backup system had been broken since before the start of the semester. And nobody prioritized fixing it.

Created quite the scenario for professors who were suddenly confronted with the entire class not having any code for their final projects.

They talked about firing the kid who wrote and ran the script. I was asking why the head of I.T. wasn't on the chopping block for failing to prioritize a working backup system.

</comment><comment><USER36> One time the DBA and I were looking at our production database, and one by one the tables started disappearing. Turned out one of the devs had tried out a Microsoft sample script illustrating how to iterate through all the tables in the database, without realizing that the script was written to delete each table.

<comment><USER37> "And this is why devs lost prod ddl access"

<comment><USER38> Nobody should have write access to prod when wearing their dev hat. Only scripts tested against a copy of the DB should be run with write in prod.

</comment></comment></comment><comment><USER39> [https://www.gliffy.com/examples/](https://www.gliffy.com/examples/)

First graphic on this page includes a bright red box asking: "Is your data safe online?"

Evidently not a rhetorical question.

<comment><USER40> It also says the answer is NO if you have "services administered by meat-based lifeforms"

</comment></comment><comment><USER41> If the gentleman who did this loses his job, then those looking for a new sysadmin should definitely give this guy some serious consideration.

Because I guarantee you he'll never, ever, let this happen again.

<comment><USER42> Your comment cracked me up.

But one thing worth quibbling over:

It might not have been a gentleman who did this. Might have been a lady. Or might have been a not-very-gentle man.

<comment><USER43> Whoever it was they were not very gentle on the delete key.

</comment></comment></comment><comment><USER44> The official rite of passage that turns anyone into a bona-fide sys admin. The equivalent to running your production server on debug. D:

<comment><USER45> I had a client call me in panic after he'd run a unit test script that started by wiping and recreating the database from scratch, and he'd run it against the wrong server.

Thankfully he'd just run it against a dev environment where the loss wasn't particularly severe (the prod environment is firewalled off, so he couldn't have done the same thing against that), but from the panicked tone of his messages before it was clear what had happened, I'm sure he's come to be extra careful about database credentials going forward....

<comment><USER44> We seem to have shared clients in some point in time. ;)

</comment></comment></comment><comment><USER46> Once I asked server support guy to move database from production to dev. He did - without any question of doubt - exactly that: copied database to dev environment and deleted it from the production. (note: in my language word "move" is less unambiguous than in English - it may mean, depending on the context "move" or "copy").

<comment><USER11> Your server support person did wrong and language is no excuse for them. If there's any ambiguity, if data may be lost, if something may cause an outage - then they have a duty to tell you and then ask if you're sure.

I do it all the time. "Give this access to this user to this db." 'Okay, but do you know they'll be able to drop your db?' "Oh shit okay wait a sec..."

</comment></comment><comment><USER47> Last week I deleted a large portion of our pre-production ldap. I use jXplorer ldap client and for some reason the control d (delete) confirm dialog defaults to "Ok" instead of cancel. I'm use to hitting control f (search) and then enter to repeat the last search and when I hit d instead of f I deleted a bunch of stuff. The silver lining is I patched the problem in jXplorer and submitted it. It's my first legit contribution to a project.

</comment><comment><USER48> In my opinion it is way too easy in Unix to accidentally delete stuff (even for experienced users). Having a filesystem with good (per-user) rollback support is, imho, more than just a luxury.

<comment><USER49> The people writing UNIX put all their brains into writing a stupid system.

Also, there are now CoW Filesystems that prevent data loss because of human error. Btrfs and ZFS are good examples.

<comment><USER50> But you really don't want to use those when working with databases, the performance loss is severe.

</comment></comment></comment><comment><USER51> So.. story time. While at the university, there was that project where we had to create an elevator simulator in C as a way to learn threading and mutexes. All the tmp files were stored in ./tmp/.

In between build/run/debug cycle, I would "rm -fr ./tmp". But once, I did "rm -fr . /tmp". At that time I didn't know any better and had no version control.

I had to redo that 2 weeks in a night, which turn out to be more easier than expected considered I had just written the code.

My lessons from that:

      A) Version control, pushed somewhere else.
      B) Use simple build scripts.

<comment><USER52> just run `!rm` and hope for the best

</comment></comment><comment><USER53> "Tell me about a time where something didn't go the way you planned it at work."

</comment><comment><USER54> I've done this - ran out of space on /home for the mSQL database (~1996 era), I moved it to /tmp which had plenty free. I suspect most people can now guess which OS this was on and what happened when the machine rebooted some weeks later...

(Hint: Solaris)

<comment><USER55> but ... but ... but ... it's right in the _name_ !

<comment><USER56> That didn't stop a tester on a team I managed. Server was getting low on space, I found a bunch of crap in a sub-directory named _tmp_. Deleted said crap. Tester complains shortly after, I explain what happened and why one shouldn't put stuff in directories named "tmp". He retorted that those were _production_ tests. Okay, maybe one can't be expected to remember 30 or more years of Unix naming conventions, but I should challenge you to battle simply for using crappy, non-descriptive directory names. I'm supposed to find production tests in a directory named "tmp"? Clean out your desk.

Of course there was no backup, because the sysadmin doesn't back up things in _tmp_ directories.

</comment></comment></comment><comment><USER57> I was forced to train someone so cocky that they ended up doing a rm -rf / on our production server a month after I quit. He also accidentally euthanized a legacy server, deleted the accounting database when he was trying to do a hardware based RAID rebuild, completely destroyed the Windows domain server and mocked my daily tape backup regimen - opting to ship USB consumer grade hard drives in an unpadded steel box instead to off-site storage... The list goes on. He literally destroyed everything he touched. The only reason he wasn't fired was because he was a pretty man.

<comment><USER28> Oh man, this was my last boss who literally destroyed every one of the critical reports to key clients. I quit when I realised he was undermining me to the CEO, and the CEO was listening to him. He then completely borked the entire system so badly that he was forced to resign 6 weeks later.

The company went from 25 clients to 4 last month.

<comment><USER57> I think tech workers should have a 'strategic vacation reserve' specifically to deal with this situation.

<comment><USER28> Actually, that's what I'm planning on doing from now on. However, I'm going to have to make sure all my own processes are _completely_ bulletproof and documented.

</comment></comment></comment></comment><comment><USER58> A dark and snowy night a bunch of databases on a server just vanished. This was on a server that was still in development, but was part of the billing system for a huuuge company, and it was under a lot of scrutiny. The files are just gone. So I contact the DBA and the backup group. For whatever reason, they can't pull it off local backups, so tapes had to be pulled in from Iron Mountain.

As I said above, a dark and snowy night. Took Iron Mountain 4 hours to get the tapes across town. The DBA and I finally get the database up around 8am the next morning. I investigate, but can't find any system reason for the databases vanishing, the DBA can't either.

2 weeks later, the same thing happens.

I eventually track it down to a junior developer who has been logged in and has on several occasions run this: "cd /" followed by "rm -rf /home/username/projectname/ *" Note the space before the star. On further investigation, I find the database group installed all the Oracle data directories with mode 777.

</comment><comment><USER59> Sounds like a terrible situation. I wish those guys luck.

One useful sys ops practice is the creation and yearly validation of disaster recovery runbooks. We have a validated catalog of runbooks that describe the recovery process for each part of our infrastructure. The validation process involves provoking a failure (eliminate a master database), running the documented recovery steps and then validating the result. The validation process is a lot easier if you're in the cloud since it's cheap and easy to set up a validation environment that mirrors your production env.

</comment><comment><AUTHOR> Posting as a reminder to myself that "in the cloud" != safe

There's always room for computer error, and more like, human error.

Imagine if something like this happened to Dropbox? Ooooft.

<comment><USER60> This gives most SaaS providers a bad name. The error here is not the engineer deleting the db, it's the complete lack of data restore testing.

Complex restoring never works well when the first implementation is under the pressure of the real event. Other SaaS providers will be cursing such a big name tool making such a public mess.

<comment><USER61> DR testing is hard, complicated and costs a lot. Yes, it should be done regularly, but it's not an easy task; I believe small(ish) companies simply can't afford it.

<comment><USER62> Then perhaps small(ish) companies shouldn't hold data that's critical to their customers. Just like real engineering companies don't build nuclear reactors if they can't test their safety systems, cars if they can't afford to do crash testing and so on.

DR is not a luxury. Systems that don't properly do DR aren't unoptimized or something, they're badly engineered.

<comment><USER63> Doubt any business will go down as a result of a web based diagram tool being unavailable for a few hours.

<comment><USER62> I wasn't referring to this specific case. Also, "don't worry, our services may suck, but not to the point where they bring down your business" is not exactly the kind of reliability one would want to aspire to.

</comment></comment></comment></comment></comment></comment><comment><USER64> Every System admin could have a bad day like this :) Some years ago I have deleted entire production server just by very simple command "rm -rf /" instead "rm -rf ./" and I had logged in with root account. No words to explain the feeling at that time. Thanks to backups, without it, I have been killed thousand time by my customers.

<comment><USER65> >deleted entire production server just by very simple command "rm -rf /" instead "rm -rf ./"

It's absolutely a rite of passage, fun times!

My personal favourite is similar:

        rm -fr .*

_ouch_

</comment></comment><comment> jon-wood I'll join the chorus of people who've done something similar. In my case it was the database of a small e-commerce site, where I'd taken a backup and then formatted the database server to reinstall it.

What I hadn't realised was that the backup script was set to dump to the OS drive, so in the process I'd also just formatted the backup. Thankfully one of our developers had a recent copy of the database locally, but it definitely wasn't my finest hour.

</comment><comment><USER66> I am actually gladdened by reading the posts by others on here mentioning how they did the same thing. I've been kicking myself for decades over a similar thing I did when I was starting out as a programmer.

Not as big as some of those here, but back in the late 80's I was a self employed programmer writing DOS apps for local businesses to help them run more efficiently.

There was a local martial arts supply shop whose owner was sort of a friend of mine, and he engaged me to write a stock control and hire database for him, which I did. When it came time to implement, he told me that there was a LOT of data to enter, so he would hire a couple of young students of his to sit down for an entire week and key in the data, which was all good.

After they had finished, he called me back in to 'go live', and I sat down in front of his server PC and began to check that everything was OK. Normally, it is my habit to take a backup of the entire app directory before working on it, but I think I was going through a break up with my then girlfriend and was a little sleep deprived.

I noticed that some temporary indexes had been created during the data entry and I went to quickly delete it (thinking to rebuild all the indexes for best performance), but typed in 'DEL _.DAT ' instead of 'DEL _.KEY'.

I still remember that sinking feeling as I sat there looking at the blinking 'C:\>' prompt, knowing I had wiped out all his work. Telling the owner was also one of the hardest things I have done, and I fully expected him to pull down one of the sharp oriental weapons from the wall and take me apart.

But he was really cool and understanding about it. He refused my offer to pay for the students to come back in and re-key the data again, which actually made me feel worse, because I knew he wasn't having the easiest time at that point making ends meet in his business.

End of the day, we got it all working and he used the system for many, many years. But to this day, I still make a copy of anything I am about to touch, before I work on it.

</comment><comment><USER67> In prehistoric times on an OS named CTOS, a distributed client/server OS, I was charged with making tape backups of user's local workstations, IVOLing (formatting) the disk, and restoring from tape. The contract specced that 2 tape backups were to be made, but of course in the interest of expediency, I only made one. And then I encountered the user's tape that wouldn't restore. I remember thinking that losing a user's data is the biggest crime a sysadmin can possibly commit, and it taught me a great lesson on the value of backups and their integrity. Fortunately, I swapped out tape drives like a mad man until one managed to restore the tape.

</comment><comment><USER68> I must say that's really the most transparent way to handle a downtime I've ever seen.

I would be scared shitless to expose for all to see what really happened and what is happening, even more so when it's makes them look like they don't know what they are doing.

I must applaud them for that, I wish if I ever get into such a nasty situation, I'll be able to do what they did.

</comment><comment><USER69> This reminds me of something I did at a previous employer (an ISP), many, many years ago.

I needed to do an update in a SQL database to fix some customer issue - <USER3> statement should just update one row but seemed to take a looong time to run, which seemed strange. When it finished and printed something like "700000 rows updated" I noticed I had forgotten the WHERE clause and I had also not started a transaction that I could roll back. Whoops!

That's when our support got really busy answering customer phone calls and I started asking who was in charge of our backups.

That was _not_ a good day.

<comment><USER11> As a DBA that's pretty bad. I have seen someone do that, and was able to restore a copy of the database side by side, and then update that column of each row back (luckily it wasn't a high traffic table so many nobody would even know).

But I was also not happy they were doing this stuff straight into production. And even when they do it in dev/test/qa, I can see when I ask them questions about how they can verify that what they did did what they wanted, that they really don't know (especially when we go beyond single line statements where you can see a row count, and into a couple dozen lines of stored procedures).

But then you need to start controlling it through a web front end to allow that operation to happen. Or a secured PowerShell interface (which are time consuming to put in place and then maintain let alone secure and also train people on). And I don't have the energy for that with all of the other fires I'm fighting.

So yeah. Understood. But not good.

</comment></comment><comment><USER70> Sucks, but we've all been there. If the admin is reading this, it's all going to be ok! Just remember, life sucks, then you die...

</comment><comment><USER71> Poor guys. Really interesting reading though.

I initially thought it was weird they had to run several "processes" in case 1 failed. But running out of space or something correctable is actually something likely to happen. Is this standard? It's quite smart.

Anyway, assuming they get the data back, I think they've done pretty good - <USER4> data loss and a days downtime isn't bad given this is a true disaster.

It would be nice if they'd let us know how the db got deleted, and what they suggest to mitigate in a blog after.

<comment><USER72> In my experience it's good to have a few different restore attempts running in parallel, so long as they won't conflict with one another. For example, the restores may be to different hardware, from a different point-in-time, in a different datacenter, or from a unique backup system (image-based, file-based, transaction logs only, etc). One of their "restore" attempts could even be a scan of the original disk for recoverable data, though that seems unlikely.

In a critical outage like the one Gliffy is experiencing, I take the same approach. Outline all your restore options, estimate time for each restore, drawbacks of each approach, etc and take every available angle.

If you want to know how the database was deleted, read through some of the horror stories posted in the comments here and assume it was probably one of those!

</comment></comment><comment><USER73> That reminds me of that time where I imported the nightly dump of a database TWICE into the same server.

Dropping an entire database brings problems, having duplicate content and deleted content coming back bring a whole new realm of others good times.

</comment><comment><USER49> My mentor told me: "get everything wrong, but get the backups right", as he was busy debugging the backup solution he had in place at my college (ZFS + NetApp + rsync + sh + perl + tape).

On my own, I'd put CoW wherever possible. It's so easy to delete something on UNIX that it should also be easy to restore and CoW is without a doubt a no- brainer for this.

<comment><USER74> CoW?

<comment><USER49> Copy on Write, see [https://en.wikipedia.org/wiki/Copy-on- write](https://en.wikipedia.org/wiki/Copy-on-write) and [http://arstechnica.com/information- technology/2014/01/bitrot...](http://arstechnica.com/information- technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/)

</comment></comment></comment><comment><USER75> > We are working hard to retrieve all of your data.

Given that in most cases where a backup exists, the user data is not lost, it's a bit unsettling to say that (and also in most cases admins are not working, they are mostly waiting). It's more reassuring to the user to say "we are verifying that all data is restored correctly" or sth.

</comment><comment><USER76> I did this to the trading database back in 2008 while supporting the mortgage desk of a major investment bank, a day before Lehman went down.

Thank god for backups and translog replays.

</comment><comment><USER77> My first real job was a DBA at Microsoft, on a very large marketing database (aprox 1.5TB in 2000)

That experience, how much work is required for "real" production databases, led a bad taste in my mouth. I stay away from self-hosted db's to this day. (example, I use google cloud datastore nowadays)

<comment><USER11> IMHO that's still a very large database today. Most people don't appreciate that restoring that is likely going to be 4-8 hours given SAN and network speeds unless you're in a super modern SSD-driven environment.

When it goes higher, 5TB, 10TB, 15TB, that's out of my league, and I just say days of downtime. Also remembering those are often spread across two servers like in an Availability Group, which means two restores...

I know people will pipe in and talk about partitioning and read-only file groups and partial restores. Except that in the real world I've seen zero of it, and it's not general DBA knowledge, it's likely the top 1% or even 0.1%.

And even then you'll still have massive downtime (depending on how the application is architected to support this), and you'd better have bullet- proof testing (and 20TB spare disk space to repeatedly test it with), to make sure a complex restore like that is easy to carry out while under the pump of an outage.

</comment></comment><comment><USER78> "The good news is that we have copies of our database that are replicated daily, up until the exact point of time when the database was deleted. We are working hard to retrieve all of your data."

Better news would be if every user had local copies of their work too. both in local storage, and on a cloud storage provider of their choice. Preferably in a non proprietary format.

This isn't just about getting me to trust your site if you crash or have a tragic mistake. This is also about getting me to trust your site if you go out of business (as too many startups unfortunately do).

</comment><comment><USER79> In 2002 I accidentally executed

DROP TABLE HOTELS;

whilst working on the Virgin Holidays website. We managed to get it from backup but it made me shart.

</comment><comment><USER80> Ah, that moment when we needed to copy 1 master disk drive to 80 PCs urgently using Ghost, and my boss said "I'll take care of it, I'm very familiar with Ghost". And with the first PC proceeded to copy the blank disk onto the master.

Problem was : creating the master drive was the job of someone else 1000 km away, with special pieces of tailor-made software... The guy ended at the airport trying to get someone on a leaving plane taking couriering the disk drive (fortunately for us, some lady accepted; this was still possible in 1998).

</comment><comment><USER58> I once had a client we were running their office Linux server for. They needed more storage, so they asked me to come in and put in some larger drives on the RAID array. Somehow during this, the old drives freaked out and the data was just gone.

So, we go to the backup tapes. Turns out that something changed in the few years since we set up backups, and the incrementals were being written at the beginning of the tape instead of appending. These were DDS tapes, and there is a header that stores how much data is on the tape, so you can't just go to the end and keep reading.

Now, we had been recommending to them every month for a year or more that a backup audit should be done, but they didn't want to spend the money on it.

They contacted a data recovery company who could stream the data off the tape after the "end of media", and I wrote a letter to go with the tape: "Data on this tape is compressed on a per-file basis, please just stream the whole tape off to disk and I'll take it from there." We overnight it to them and a week later they e-mail back saying "The tape was compressed, so there is no usable data on it." I call them up and tell them "No, the compression re-starts at every file, so overwriting the beginning is fine, we can just pick up at the next file. Can you just stream it off to disc?" "Oh. Welllll, we sent the tape back to you, it should be there in a week." They shipped it ground. We shipped it back, they did the recovery, and we got basically all the data back.

</comment><comment><USER81> ~20 years ago I was working for a relatively small banking software company (in Hungary) (it was a really good job from the learning point of view, but was really underpaid).

One Monday afternoon one of our clients just called that the banks officer's suddenly can't log in, random strange errors are getting displayed for them, etc.

OK, our support team tried to check, we can't login either, strange error.

"Did you do anything special, [name of the bank's main sysadmin]?"

"Well, nothing special, I just cleaned up the disks as usual."

"How did you do it?"

"As usual: 'mc', sort by file size in the INTERFACE/ folder, marked the files and F8".

That's normal.

OK, since we had the same user account (I knoooow), launch 'mc'. Looks normal. Except... In the left panel the APP/DB directory is opened... Check... Appears normal... At first... But... WAIT. Where is the <BANKNAME>.DB1 file?

"<ADMIN>, how long time did it take?"

"Dunno, I went for my coffee, etc."

Apparently he deleted the production systems' main DB file. It's got resolved by restoring the backup from Saturday and every file and input transactions had to be re-inputed based on the printed receipts, the officer's stayed in late night, etc. He is still the head of IT at the same bank. (Yeah everyone makes mistakes, but it wasn't the only one of hims, but likely the biggest.)

</comment><comment><USER82> I would really love to get more detail about how they structured the full backups and transaction log backups for the database. Are the backups dumped to disk before being picked up on tape? Or are the backups streamed directly to the backup system?

I'd also would love to know how large is the database that was deleted. Doing a point in time restore of database that's a couple of hundred gigs should be relatively fast (depending on what hardware you are running on).

</comment><comment><USER83> Serious question: Do modern "all in the cloud" tech companies actually have DR plans?

All the presentations i've seen about people deploying in the cloud leaves out any DR site, replication process, turnover time for the DR site taking production traffic, etc. It's like they believe redundant machines will save them from an admin accidentally hosing their prod site and having to take 3+ days to recover.

<comment><USER84> It really depends on the company. My suspicion is that companies only get serious about DR once they reach a fairly big size. When I interviewed at Squarespace they were just finishing building out their second datacenter. At that point it was a DR data center but they were planning on switching all production traffic to it in the coming months. Most of the startups/midsize companies have relied on a third-party for backups ie Rackspace.

<comment><USER83> That's what strikes me as odd. Even small and mid-size companies can do this for cheap if they're mostly cloud-based.

Stand up the environment at another cloud provider, keep resource use at 1% that of your current provider, implement a continuous replication procedure, document the failover procedure, and test-run once a month. Much less work than actually buying and organizing some small colo space in another DC, and way faster than scrambling to recover. Yet I don't know of a single cloud- dependent company that does this unless it's for performance reasons.

</comment></comment></comment><comment><USER85> My first internship used Google Drive for their database (small start up) and there have been numerous times where I have almost lost a substantial amount of work/information. This article brought back that feeling of anxiety. But that is a lesson to be learned, even if it was the hard way. Everyone goes through that at some point in their career.

</comment><comment><USER86> Earlier in my career, I worked in prod support for an insurance web application. It had a DB containing reference data. This reference data was maintained in an Excel spreadsheet; a macro would then spit out CSV files, which would be used by command line scripts to populate databases in different environments (test, staging, pre-production). The DB data was totally replaced each time. Pre-production data would be copied into production, every night or so.

One time, I ran the staging and pre-production scripts at the same time. This had the unusual effect of producing an empty CSV file for pre-production.

When I got in the next day, I discovered all of the production data had been wiped out overnight...

Thankfully, it was all reference data, so it was just a matter of re-running the export macros, and pleading with a DBA to run the data import job during business hours.

I ended up writing a replacement using generated SQL, so we could apply incremental updates (and integrate better with a custom ticketing system).

</comment><comment><USER87> These days it should be possible to roll back a few steps (à 15 min / 1 hour) with a copy on write filesystem like zfs? Full scale restore from backup should only be necessary if the storage hardware fails (IMHO).

You still need to apologize for a some data loss, though. So make sure that everything you do has one or two safety nets before it hits the customer.

</comment><comment><USER88> I am wondering why the OS not introduce a policy to protect important files/directories. For example, we can mark something is important, then if someone try to delete it, it will ask the person to input some key (at least 20 characters), if the key is incorrect, the operation will be canceled.

<comment><USER89> Yes, we call those marks "permissions". Most operating systems have them. The problem is putting the right process in place around using them.

<comment><USER90> There are also (extended) attributes, such as the "immutable" flag, available on some OSs/filesystems.

If you're interested, look into __chattr __for Linux, and __chflags __for BSDs.

[https://en.wikipedia.org/wiki/Chattr](https://en.wikipedia.org/wiki/Chattr)

</comment></comment></comment><comment><USER91> New devops guy at my work a few years ago somehow completely blows away the CDN. Of course we have all of the data locally but it took almost a full day to reupload. I believe this is our longest downtime to date.

</comment><comment><USER92> Just reading this headline makes me queasy.

</comment><comment><USER93> I once accidentally moved the cgi-bin (long time ago) on one of Swedens biggest websites. moved it back pretty quick so it was "only" down for a couple of seconds.

</comment><comment><USER94> My very first commercial experience doing development was as an intern at Polk Audio. At the time their online solution was pretty immature (no version control and no development environments; everything was coded up in production).

I was working on a very important, high traffic form and...accidentally deleted it. Their backup consisted of paying another company to back up each file. Fortunately they came through but it took a full day to restore a single file.

</comment><comment><USER95> Good management blames management for this. Bad management blames the sysadmin and publicly says that "the sysadmin did it".

</comment><comment><USER96> This is where delayed replicas come in very handy: [https://dev.mysql.com/doc/refman/5.6/en/replication- delayed....](https://dev.mysql.com/doc/refman/5.6/en/replication-delayed.html) (I don't know whether they're running on MySQL though...)

</comment><comment><USER97> One time I restored a database from a MySQL binary log a table that held about 10,000 employee pay rates. Unfortunately, the log was shifted a few rows and the mistake wasn't noticed until a few weeks later when the CEO and some high level directors noticed their pay was traded in for the high 6 figures to an hourly rate.

What a mess!

</comment><comment><USER98> Have done this and similar. And now I have aliases in my zshrc:

        alias db="mysql --i-am-a-dummy"

</comment><comment><USER99> I've found that it's important to do my own backups and not rely on IT to do them. I once returned from my holiday to find that the sysadmin had wiped my hard drive. He said he thought I had left the company. Fortunately, I had backups on computers in other states that he didn't know about.

</comment><comment><USER100> People who do things make mistakes. It's the ones who don't make mistakes that should be of concern.

</comment><comment><USER101> Knew one sysadmin was fired due to his "rm -rf /" fat finger without a working backup tape scheme.

Also once we had to retrieve some code from tapes, which are just stacked in a messy black room, and nobody can eventually find that, but no firing anybody either.

</comment><comment><USER102> Looks like the pricing page is 404 right now as well (but all other pages seem to be fine)

</comment><comment> noir-york Admit it - who here hasn't read this and not gone back and tested their restores?

</comment><comment><USER103> Last time I did that the chief sysadmin had my back and we had it restored in 5 min.

</comment><comment><USER104> Sorry for those that lost information, personally glad it didn't involve Atlassian Confluence-hosted Gliffy illustrations ... I have a lot of those and the tool is great for quick shareable embedded engineering sketches.

</comment><comment><USER105> very early career days, wrote a script that had rm -rf in it. I knew this was dangerous and so the script asked, 3 Times, if you were sure you were in the right place.

That was the problem, asking 3 times.. people just spammed enter x3 at that point in the script.

Someone using it came over to me one day.. 'hey, look, what going on with this system. I can't do ls ? '

There was no system, pretty much. The script had rm -rf'd while he was root and running the script from root.

The job of the script? installing and configuring the backups for a system. So yea, there were no backups for this system at this point in time !

<comment><USER106> We had a similar script in production, it was password protected with a password of "badidea".

</comment></comment><comment><USER107> I only store... IDK.... about 80% of my system architecture diagrams on Gliffy.

FML :-/

</comment><comment><USER108> The "Other" Moore's Law:

Backups always work. Restores, not so much.

</comment><comment><USER109> This is my biggest fear when I use my production Redis

</comment><comment><USER110> And Gliffy is back up.

[https://www.gliffy.com/apology/](https://www.gliffy.com/apology/)

</comment><comment><USER111> Are there any more sites like gliffy and draw.io?

<comment><USER112> Yes, [https://www.lucidchart.com](https://www.lucidchart.com) is very popular.

</comment></comment><comment><USER113> Poor guys...

</comment><comment><USER114> Shit happens =)

</comment><comment><USER115> don't laugh, this can happen to you

</comment><comment><USER116> Just going to add the obligatory [http://thedailywtf.com/](http://thedailywtf.com/)

</comment><comment><USER117> I accidentally all the data...

</comment><comment><USER118> Anyone have a screenshot?

</comment><comment><USER119> Well, who hasn't done a DELETE without a WHERE clause? ;P

<comment><USER120> I have a very strict habit after once or twice doing this: always start all DELETE commands with "\--" (that is, comment it out) until I have written a where-clause.

My command in the buffer might go through these steps:

1\. "Delete from" 2\. "\-- delete from" 3\. "\-- delete from table where condition limit n;" 4\. (Generally either ask a co-worker or make a Jira with the exact command I have at this point, so there's a sanity check and/or permanent record, but for very low risk/especially mundane/especially time critical updates, do it) 5\. Delete the "\--" and run it. 6\. Think hard about adding some functionality in the app for doing it in app code unsteady of in database code.

Generally do the same for "UPDATE".

<comment><USER121> I tend to write my DELETE statements by writing a SELECT first then edit that into the delete version

<comment><USER122> I've turned to creating a temp table and putting the primary keys from the select statement into there so I can guard myself against the thing that was to delete a handful of rows deleting everything. Plus with that you can do another join and see if those rows have some sort of value in it that you didn't expect.</comment></comment></comment></comment>


The Engineering Talent Bubble - <AUTHOR> http://blog.jayteebee.org/2010/11/im-calling-it-now-engineering-talent.html

<comment><USER2> I don't think it's a bubble.

A lot of people talk about top engineers being 10x (or even 100x) as productive as mediocre engineers. Assuming that's true (and I suspect it is, but admittedly I don't have any evidence to back that up):

Historically, those 10x engineers got paid maybe 2-3x.

Because it's become so much easier to start software companies recently (thanks in no small part to folks like Y Combinator, but largely due to cheap servers and FOSS), those 10x engineers can now easily go seek compensation that's more proportional to the value they actually create.

In turn, large software companies are finding they need to pay more to stay competitive with the alternative of striking out on your own.

What we're seeing here is pay becoming more proportional to value creation. Which I think is a very good thing.

As an aside, I think the rise in talent acquisitions is largely a backdoor for companies to hire top talent without needing to change their deeply entrenched pay-scales. Apparently Google is starting to figure out that it's more efficient to just pay people more instead of having them leave for a couple of years in order to take advantage of this backdoor.

<comment><USER3> Also compare the recent run-up in salaries to a graph of baseball players salaries over years:

[http://baseballanalysts.com/archives/2010/11/mlb_salaries_ov...](http://baseballanalysts.com/archives/2010/11/mlb_salaries_ov.php)

When you look at the linear graph it looks less like a bubble and more like settling in at a new level.

<comment><USER2> Like professional sports, we'll probably see the bottom of the salary range for software engineering drop significantly, due to increased competition resulting from globalization and because the higher pay at the top of the range will make the field more attractive.

</comment></comment></comment><comment><USER4> A bubble exists when people buy something based not on their belief in it's intrinsic underlying value but because people think it will appreciate in price. I.e. tulips, california real estate.

You can't have a bubble in something you can't resell.

This isn't an argument that salaries are at the correct long term level. I'm just saying that salaries can't have a bubble that "pops" and collapses like asset bubbles do.

<comment><AUTHOR> Hi, I'm the author, and I disagree with you about the definition of a bubble.

A bubble is when you're paying for something based purely on speculation about the future value of that thing, rather than on a rational analysis of the expected future return.

I would wager that Google does not have an internal projection of the value of any engineer showing them to be worth $3.5m over four years; rather, I'd bet that comp decision was made based on an irrational fear of yet another defection from Google to Facebook.

<comment><USER5> $3.5 million is simply too large a number for your theory to be correct. A number that large to a single employee doesn't come out of thin air, it's based on something -- whether that be formal analysis or back of the envelope calculation.

<comment><AUTHOR> I suspect the calculation was (expected value of FB package) * 1.5

<comment><USER6> It's also possible that it was based on the potential damage done to Google by Facebook's talent acquisition, or if this isn't a zero-sum game, to prevent any gains by Facebook.

</comment></comment></comment></comment></comment><comment><USER7> I don't know if I agree with the authors position based on the examples he gives. I'm willing to bet the Google engineer who allegedly was offered the $3.5M in stock to stay was a _very_ key engineer. Google doesn't offer everyone millions in stock not to quit. And as many of the readers of HN know, every startup is not being acquired for talent grabs. Just because we hear of a few (or even what might seem a lot), the percent in comparison to the number of startups is miniscule. I hear about someone winning the Lotto every week; that doesn't mean everyone is winning.

<comment><AUTHOR> Hi, I'm the author, I appreciate your comment. You're demonstrating that you still have some rational perspective here; my thesis is that many people have lost that perspective. I am an ex-Googler, and I am seeing more and more talented engineers leave Google to start a "company" with the express goal of getting acquired for talent, sometimes even hoping to get acquired back by Google.

When people see a valuation in the marketplace (eg, a talent acquisition that makes everyone extremely wealthy) and assume they can achieve the same kind of returns with minimal effort, that is the hallmark of a bubble. The same thing happened with day traders and real estate speculation.

<comment><USER8> Googlers and ex-Googlers may be living in a bubble. That doesn't mean that the wider engineering talent pool is.

From what I could find on the bls.gov site, in 2008 there were about 900,000 jobs in software engineering and 75,000 jobs in computer engineering. From what I can tell from searching, Google has 20,000 employees, but not all of them are necessarily engineers.

Even if all google employees were engineers, and they were going out and trying to create startups in order to be bought, that would be 2.0% of the US- wide engineer pool.

</comment></comment></comment><comment><USER9> You know Facebook + Google, as big as they are, do not equal United States of America, which is still bogged down by the biggest recession since the Great Depression. What bubble are you talking about? These companies sit on small mountains of cash and can afford bidding wars over top engineering talent, especially that software is their livelihood. Let me assure you, hiring managers of 99% of companies in the US don't stand on street corners and bid up software engineers' salaries. You can sleep in peace...

</comment><comment><USER10> He had me until "This is going to end, and badly."

Bubbles end badly because of leveraged bets on overvalued assets.

Call me insensitive, but this doesn't strike me as so bad:

       "they will find themselves unemployed and without savings."

It's not like you're going to work your _entire career_ at a non-paying startup. At worst you come out with a unique experience and go find a paying job.

<comment><USER11> Yes, like I said, that last paragraph didn't work for me either.

</comment></comment><comment><USER11> _There are a lot of engineers who are going to make serious life choices based on current trends, and in a year or two, when this blows up, they will find themselves unemployed and without savings._

I was with him until the last paragraph. I think a lot of really bright engineers who accept the premise are banking a large percentage of their salary, no?

</comment><comment><USER12> Computer science enrollment was quite crappy (still is?) after the dot com bubble burst. I started college in 2005, right when a brand new CS building was completed... which was pretty underutilized during my time there. Enrollment in CS at my school had dropped by over 1/3 from its peak, which was just a few years earlier.

</comment><comment><USER13> I don't find the positions to be supported by anything.

99.9% of engineers get paid an upper middle class salary, with a few more getting these 7 figure bumps from acquisitions that aren't really.

I don't think something like 1/10 of 1% of engineers having something happen to them is something to worry about ending.

</comment><comment><USER14> Would $3.5M have been too much to keep Paul Buchheit at Google? so he could have created FriendFeed there, so Google could compete with Twitter/Facebook properly

Or Ev Williams who actually created Twitter?

Or Dennis Crowley who created foursquare?

</comment><comment><USER15> One point that this analysis is missing is that tech salaries were being artificially deflated by Google, et al with their agreement to not raid each other for talent. With those practices now being abandoned, we are going to see a spike in salaries as they seek equilibrium. Googles 10% pay hike is the first such example of this.

</comment><comment><USER16> I also don't think there is a bubble. The company acquisitions for talent alone have been reasonable as far as I can tell. In the IT services industry this is long standing practice for talent acquisition.

There is always a shortage for great engineers and they generally have great jobs, so acquiring the top tier will always be competitive as in any other industry. For most companies, the only way to gain access to this tier continues to be personal networks or if they are in the founding team. For other organizations, growing this talent from promise is the most pragmatic approach assuming have some really good talent to mentor them.

The middle and lower tiers are being paid upper middle class wages commensurate with education, experience, and economic contribution.

Things are more competitive in hot geographies like Silicon Valley and Seattle, but that is nothing new.

</comment><comment><USER17> My thoughts on the potential "bubbles" aside, I find it funny how it's become a thing to try to call out bubbles that may be occurring, given the recent monumental financial collapse.

So called bubbles always exist - they're hot sectors - and then there's a self correction of varying severity.

</comment><comment><USER18> The tech industry has still not entirely recovered from the first DOT-com bubble burst. Broadcast.com selling for 5.9 Billion in 1999 was a bubble. Startups gets bought for _millions_ is not a bubble.

</comment><comment><USER19> "Startups, less than two years old, with very little user traction and no revenue, getting acquired simply for talent, at valuations that give even the front-line engineers a seven-figure outcome."

Seven-figure salaries for developers? Are you sure you put the decimal point in the right spot and counted the digits correctly? I find it very difficult to believe that any developer (not a manager, not a c-suite executive, etc.) is making in excess of $1,000,000. Name one. Otherwise, I call bullshit.

</comment><comment><USER20> His argument seems to rely on companies becoming unable to pay going rates. I'm not saying thats not going to happen, but he seems to be predicting a software bubble without really defending that prediction.</comment>


Process Mining and AI Platform – Logpickr Process Mining and Smart Optimization - <AUTHOR> https://www.logpickr.com/en/product.html <comment><AUTHOR> An international Telecom operator used Logpickr Process Explorer 360 to analyze the behavior of two different teams producing the same services.

They discovered that one team spontaneously parallelized tasks, and effortlessly gained time and availability, improving the overall process time.

By generalizing this best practice, the company gained 30% time on the service delivery, compared to the previous organization.</comment>


Tesla: Insane or Clever - <AUTHOR> https://mondaynote.com/tesla-insane-or-clever-b7a8e1479f6b <comment><USER2> Software is eating the world, automobiles included.

I'm a huge skeptic when it comes to Tesla, but JLG raises a good point here. Apple had _all sorts_ of problems in 2007, so many that established phone manufacturers largely brushed them off. A focus on software rewrote the mobile phone industry, and every day I get into my car and watch the onboard computer struggle to handle bluetooth properly.

I need to look beyond the obvious inanity of Musk and remember this shift: software will eat this part of the world too.

<comment><USER3> > and every day I get into my car and watch the onboard computer struggle to > handle bluetooth properly.

I hate "me too" posts, but this gives me so much rage that I can't help it. My car has a touch screen interface and I am puzzled why car manufacturers even bother. Who thinks the interface they made is even remotely good. It actively degrades the otherwise fine experience of my car. It is ugly, poorly made, and fails at even basic functionality.

I am truly puzzled why these car manufacturers released such terrible user facing features. Maybe it's better these days _(my car is ~5 years old)_ , but it's just impressively bad in my car at least.

<comment><USER4> >Maybe it's better these days (my car is ~5 years old), but it's just impressively bad in my car at least.

I honestly don't think it's getting any better. At best, it's just getting different - but maybe worse as manufacturers try to add more "smart" features.

Was a passenger in a <1 year old BMW 5 series for about a week this year for a roadtrip. There were so many stupid behaviours with the software that we effectively gave up using it for anything except bluetooth music. Trying to use the built in GPS navigation was awful.

Also a passenger in a new SUV style vehicle for two weeks last year, I forget what brand. It too had utterly bizarre behaviours that it wouldn't explain anywhere, and it took a variety of experimentation to figure out. For instance, Bluetooth pairing is disabled once the car is in put into gear, and is not re-enabled until you turn the car off and turn it back on again. However it doesn't tell you this, it just doesn't show bluetooth options in the menu anywhere. So I spent hours going through every menu item (as a passenger, mind) over a few days until I finally figured it out.

In every case, the number of either unlabelled or poorly labelled buttons is overwhelming.

<comment><USER5> > For instance, Bluetooth pairing is disabled once the car is in put into > gear, and is not re-enabled until you turn the car off and turn it back on > again.

That's not a bug, that's intentional, and saves lives.

<comment><USER6> I think that this is the UX Bug:

> However it doesn't tell you this, it just doesn't show bluetooth options in > the menu anywhere. So I spent hours going through every menu item

Greyed out menu item = This is where it is on the UI, but it's not allowed at present.

No menu item = maybe it's somewhere else?

</comment></comment></comment></comment></comment><comment><USER7> It's worth emphasizing that Tesla is strong at software _relative to other automakers_ ; in absolute terms, the (non-AP) parts of Tesla's software frequently have user-facing problems [0] and appear to have been built on a Babelian tower of hacks [1]... but it's still better than the rest of the market, and it keeps improving month over month.

[0] [https://www.reddit.com/r/teslamotors/search/?q=reboot&restri...](https://www.reddit.com/r/teslamotors/search/?q=reboot&restrict_sr=1)

[1] [https://twitter.com/atomicthumbs/status/1032939617404645376](https://twitter.com/atomicthumbs/status/1032939617404645376)

<comment><USER8> Love my TM3 but as I have posted before, Tesla needs to focus, Musk needs to focus. Tesla cars have some great user facing features but also are woefully behind in other areas.

The most flagrant is the awful support for playing music from blue tooth or attached devices. It has been common for years to be able to use the touch screen or menu to select playlists, artists, or more, from the car's UI. Not so with Telsa, want to do that drag your phone out and change what is play. You can only navigate within a playing playlist. Seeing how its illegal to use your phone in your hand in many states Tesla needs to remedy it.

We have none of our promised voice commands, no improvement since when I bought mine in August. Our energy meter is near useless, does not track when car is not moving, and only tracks overall use and does not break out HVAC, regen,or drive train use.

But we got fart humor, animated rain drops, and MAME. Their priorities suck, its no wonder why they aren't profitable, they cannot get done what they started before going off doing something else or wasting resources on stuff that does not matter

<comment><USER9> Ever get a flat tire in your Tesla? I did. No shop would touch it and I had to have it towed to Tesla where it took 2 days and $400 for a simple tire change.

My blood boils when I see Musk tweet about putting more games in the car. They need to be reaching out to mechanics with training information so that they're not so spooked that they won't even change a tire.

<comment><USER10> I've gotten several flats over the past 6 years on my Model S. I take it to Discount Tire almost every time. Only time I really to take it to Tesla is when I hit a pothole so hard I had to replace a rim. Even then, I could take it to any Tesla-approved body shop if I wanted to.

<comment><USER9> Well that gives me hope that my experience was an anomaly. I was 200 miles from home with a disabled car so after calling several places I felt like I was out of options.

</comment></comment></comment></comment></comment><comment><USER11> I would just laugh at Elon's crazy proclamations, but there are plenty of people buying Teslas who believe them. At this point, Musk's claims verge on the fraudulent. Fully autonomous driving is never coming to current Teslas, sorry folks.

As for the frequent OTA software updates, I wouldn't consider that a feature. While it's true that at this point software is a critical piece of the puzzle and automakers who haven't woken up to the idea that owning your software is as important as any other aspect of the design may eventually be left behind. But I don't think Tesla offers a good model to emulate.

As for JLG's specific points, providing a single, centralized method of updating automobile software over the Internet--or even a specialized wireless connection--should be seen as a security risk, not a powerful feature. Most critical systems in cars should be isolated from each other and from the Internet entirely. Updates should not be haphazardly distributed, but should be carefully tested and rolled out as rarely as possible and in a controlled manner.

I have no hope that car companies will actually heed advice like this. But it's disappointing to see tech veterans buy into the delusion that the equivalent of Windows Update, but for your car, would be a good idea.

<comment><USER12> I don't own a Tesla. Can you provide evidence on why Elon's claims are fraudulent? I was thinking about a Tesla as my next car and would like to learn more.

Do you know if the Tesla over the air updates are altering critical systems without diligent testing? I assume only Tesla engineers would have insight into this.

<comment><USER2> OTA updates have had literally deadly consequences. There was an issue in which "autopilot" poorly recognized left-exits, including a particular one in California. It got some publicity and was fixed by an OTA update. Good! Except that a couple of months later another OTA update reverted whatever part of the "autopilot" model had learned not to drive straight into a concrete wall separating a left exit, and it killed someone. So yes, ML combined with OTA can certainly be dangerous, but Tesla fans will soon show up in droves to explain why that man's death is his fault, or the fault of anyone but Tesla.

As far as fraudulent claims, Musk has repeatedly made statements related to self-driving that some people consider to be well beyond the pale of optimism, including promising that every Tesla sold after a certain would include all hardware needed for full self-driving, and then doing the same thing a few years later, apparently rendering the first statement factually false in retrospect.

Musk has made many, many, many statements about timelines that have not worked out, and some that probably still won't work out, and some people label that fraudulent. I think it's just naïve, but as the CEO of a public-traded company, his naïveté is pretty stunning and has resulted in fines.

</comment></comment></comment><comment><USER13> Mentioning SpaceX and autonomous Teslas in the same breath makes me wonder if the Starlink satellite internet program is related to his plans for self- driving cars.

<comment><USER14> There's synergy between everything he works on, it seems to be a natural extension to his flow of thinking - expanding further into the next biggest market. It's the ESSO (gas company) model: they needed a truck to transport fuel, so they designed a truck to manufacture, and sell it to others too, and so on.

<comment><USER15> Can I coin the phrase full-stack entrepreneur?

<comment><USER16> There is already "vertical integration".

<comment><USER17> SpaceX is vertical integration taken literally.

<comment><USER18> Technically, they integrate horizontally, then the rocket is tilted vertical.

</comment></comment></comment></comment></comment></comment><comment><USER19> Why "or"?

Tesla and Musk do a lot of things. Some are clever, eg: Tesla gets marketing that other car companies can only dream of, _for free_. Other things are insane, eg Musk's "secured" bid of $420 for a company that today traded unter 200, less than a year later.

<comment><USER20> >Musk's "secured" bid of $420 for a company that today traded unter 200, less than a year later.

This keeps being repeated ad nauseam but what's the relationship between the 'going private' tweet and today's stock price?

It's been proven that the Saudi asked Musk multiple times to take Tesla private between early 2017 and summer 2018 but other shareholders resisted the attempt (with far less capital at hand than the Musk and the Saudi).

Remember that non-accredited shareholders (like me) would also have been forced to sell at that time, if the company was to be delisted (which wasn't Musk's intention but still does not undermine the "funding secured" tweet one bit).

In fact, today's stock price makes a strong case for the company to be taken private and even a stronger one for the attempt to do it when it was above $350.

<comment><USER21> > It's been proven that the Saudi asked Musk multiple times to take Tesla > private between early 2017 and summer 2018 but other shareholders resisted > the attempt (with far less capital at hand than the Musk and the Saudi).

Actually, the record seems to be that Musk was interested in taking Tesla private, and met with the Saudis once or twice, where they allegedly indicated some verbal support for the idea while not agreeing (or even discussing) on things such as at what price. Musk then took the verbal support as a full commitment to buy Tesla at a premium over current prices and gave his infamous "funding secured" tweets.

</comment></comment></comment><comment><USER22> Personally I think Tesla business would get crushed if someone made really well designed plug-in hybrids. Most trips people take are under 30 miles. No need for a 70kWh, 350kg huge battery. Plug-in hybrids could be operated as full electric during short trips and the internal combustion engine could charge the battery when necessary.

No doubt someone will immediately reply with a question: why would you want to change mechanical power, into electric, store it in battery, then change it into mechanic power again, when you can use ICE directly, or go full electric. I'll tell you why. Hybrids allow you all the same advantages as full electric car for short trips, and they are significantly more fuel efficient than normal ICE cars for long trips. All this in a package that can achieve 400+ miles of range and 5 minute recharge in a package that weights less than half of a comparable battery,it doesn't need to waste energy to be kept warm in winter and cold in summer nor it slowly deteriorates whether you use it or not.

We have to have an honest conversation about disadvantages of battery powered electric cars. Everything that makes those cars cool - great torque of electric motors, mechanical simplicity, user interface can be replicated in a plug-in hybrid(sacrificing some of the simplicity as a hybrid needs either a fuel cell or an ICE to recharge). Then imagine buying a 10 year old Tesla model 3 economy, in a climate with -20deg C winters? Lets say you're left with 60% battery capacity, then you don't want to kill it completely so you don't let it go to complete 0 and you don't charge it to full capacity leaving you with perhaps 40%? Is this enough? To go to work and back, maybe, but what if you want to, or have to travel further once in a while? You can't using that car.

<comment><USER23> "Just throw in an ICE" is your grand idea? That sounds like the argument folks were making to add a keyboard to iPhones.

Perhaps you weren't aware that Teslas are holding 80%+ of their charges over 300k miles and are expected to double that performance with Maxwell technology?

<comment><USER22> "Just throw in an ICE" is your grand idea? That sounds like the argument folks were making to add a keyboard to iPhones.

It is not "my grand idea". It is a good idea for the reasons I mentioned in my previous post period. Few actual cars that were and are good plug-in hybrids such as Chevy Volt and Prius Prime are(and in case of Volt were) getting way less publicity than they deserve.

I recently heard Chevy Volt was the first hybrid that didn't just start an internal combustion engine when you pressed on the gas pedal too hard, but it let you drive in 100% electric mode until the internal battery discharged.

Personally I think those cars fail to catch on due to the failure to market them properly.

</comment></comment></comment><comment><USER24> The major flaw (among many) with Musk's self driving taxi Tesla vision is that it treats cities, the entities that control the roads the taxis would drive on, as passive entities that wouldn't make any peep of objection.

In reality cities have plans on the books to shift away from car oriented design whether it be electric or ICE. Additionally, cities now understand that ride hailing competes with and erodes the quality of public transit, with some accordingly capping ride hailing licenses.

Cities are not going to want the surge of congestion that self driving Tesla taxis would bring.

<comment><USER25> Autonomous cars are part of the shift from cars, because they don't need street parking and can "loiter" offsite.

Urban plans for alternate modes of transit (scooters, bikes, etc) will massively benefit from street parking elimination. It's where the lanes can be safely placed.

Autonomous driving will revolutionize public transit. Uber isn't wrong about that, it's just that they are too evil to be trusted with it from a monopoly standpoint. Autonomous can solve last mile problems and centralize transit stations so that you don't really need bussing, and can concentrate on fast high capacity trains with "coarse grained" stations.

<comment><USER26> It is still a model of having one 15'x6' vehicle carrying 1 person to 1 destination, which leads to congestion no matter how well the cars are timing off of each other. Either these things get really really tiny (self driving moped?) or we cram more people in one of them (self driving bus?) because at the end of the day its a physics problem of there being not enough space on the road for everyone in a city to be transported one at a time, 20' by 8' apart.

</comment></comment></comment><comment><USER27> I just had to callout this:

" _AI expert Kai-Fu Lee of Apple, Silicon Graphics, Microsoft, and Google fame used the best antidote for such folly, Twitter humor:_

" _"If there are a million Tesla robo-taxis functioning on the road in 2020, I will eat them."_ "

</comment><comment><USER28> Clever and insane are not mutually exclusive. At least he didn't say there would be millions of robo taxis on Mars next year.

</comment><comment><USER29> It will be interesting to see if Elon can push electricity and batteries in time to get over the momentum petroleum has.

I actually see hemp and bio diesel being easier to push because it could leverage the huge existing infrastructure.

<comment><USER30> The bio diesel relies on a net investment in calories to achieve its green fuel. Most of the investment comes from petroleum products used as fertilizer. (All be it some time ago, I remember it being 4:1 calories invested:harvested for corn).

<comment><USER29> hemp... aka weed grows like a weed.

[https://phys.org/news/2010-10-hemp-viable- biodiesel.html](https://phys.org/news/2010-10-hemp-viable-biodiesel.html)

</comment></comment></comment><comment><USER31> > labels that adorn Camembert boxes (a tyriosémiophile).

This word tyriosémiophile, appears nowhere on google but within reference to this article. Does anyone know what it is? OR what it is meant to mean?

<comment><USER32> [https://en.wikipedia.org/wiki/Tyrosemiophilia](https://en.wikipedia.org/wiki/Tyrosemiophilia)

<comment><USER31> Well that was easy, thank you.

</comment></comment></comment><comment><USER33> I think there's a world that a lot of tech people don't consider. Let's say Tesla rolls out their self driving taxi service. And let's say it's decent. Not great. It kills people infrequently and Tesla manages to argue its way out of liability...

Wouldn't that be a pretty good business? It has some severe ethical costs, but... a lot of people are ok with that. What should be doesn't really effect the market's reaction to things or even consumer preferences.

<comment><USER34> It would be an amazing business, if in fact it was tesla owners pressing a button on cars they own and letting it make some money for them as a robotaxi. The fact that even that declaration that Musk sees that happening in 2020 didn't move the needle for Tesla stock shows how little the market believes it's actually coming imo.

<comment><USER35> But it made for a better fund raising story than "we need to cover our operational losses for the next 10 months".

<comment><USER33> I agree that was the motivation for the timing of the event. I do think he is being serious though. His track record is reasonably good in that he does intend to accomplish what he says he wants to do with zero to moderate delays. By my mental model, I would expect him to deliver this by 2021.

</comment></comment></comment></comment><comment><USER36> Hey I don't remember all of this being in Telsa's original secret master plan.

<comment><USER37> [https://www.tesla.com/blog/master-plan-part- deux](https://www.tesla.com/blog/master-plan-part-deux)

Search for "Sharing".

</comment></comment><comment><USER38> The way Tesla tends to defend Autopilot-caused crashes makes me think that if someone did take over their servers and then sent an update to a specific car to kill someone, Tesla would hide this from the public.

The U.S. needs better data breach disclosure laws before self-driving cars become mainstream.

</comment><comment><USER39> Automakers will say and probably rightfully so, it was never about the software it was about the nuts and bolts of making large amounts of vehicles and the act of actually selling them to dealerships. It probably turns out automakers do it the way they do it because it is the best way to make money and not because they are stupid.

<comment><USER40> Nuts and bolts are becoming commodotized by the day, as they do in any other industry frankly. EVs are a huge accelerator of this process because they are a fraction of the complexity of an IC drivetrain.

The supply chains in automotive are also becoming more dominant and most of the stuff in your car is sourced from Aptive or Bosch or Continental or whoever.

So it's not just a commodity yet, but it will be - especially towards the lower third echelon of the market - in a decade or two (a decade is not long in auto industry terms; it's merely two product cycles).

<comment><USER41> What was not commoditized in automobiles in the seventies that is now?

<comment><USER40> The global supply chain was in its infancy. You had national suppliers, but nothing like the multinationals you see today.

The number of automakers in the 1970s was much higher and the number of unique models (vs dress ups like GM does across brands) was higher still.

Virtually no mass market car brand was formed in the last four decades, other than Tesla.

<comment><USER41> Bosch was founded in 1897. Here Japanese cars made it big in the eighties. Korean cars in the last ten years. A local factory has produced Saab (now defunct), Porsche and now Mercedes cars, spanning about five decades? Maybe some consolidation but also I think there's a high amount of constant flux.

</comment></comment></comment></comment></comment><comment><USER42> Interesting... The guy is landing rockets on Earth. He said, it planned it did it.

So on one hand - do you listen to a bunch of wallstreet know it all short sellers pumping out negative news and counting parking lots and shipping containers.

Or do you listen to the Elon case - that it actually may be crazy to buy a car that that is not a Tesla...

One thing is certain, A car like a boat has been a money loosing investment until today, and consumers have been marketed to oblivion and convinced that, that is O.K.

IF THAT CHANGES, even slightly, well, then... Its GG (good game). because that changes everything, and even the die hard truck lovers will be tempted to make a few bucks on the side.

<comment><USER43> Have you ever considered that no one bothered landing rockets because it was pointless and uneconomic, not impossible?

And I love the disdain for wall street, completely ignoring that Elon is in bed with the bankers.

<comment><USER44> Well, now SpaceX has a way to launch 12K sats for global Internet. No one else is even remotely ready to do that.

Talk about your moats.

<comment><USER43> And no money to do it. Did you see the results of their last attempted funding raise?

Once again acting like something likely far fetched is right around the corner, because Elon said so. When the money for nothing dries up, we will see what these businesses look like.</comment></comment></comment></comment>