Ask HN: Ever worked with a service that can never be restarted? - <AUTHOR> I'm currently working with a legacy system. One element of it has a loaded config in memory, but the physical config was accidentally overwritten and there are no backups. In addition the source code for this compiled binary has also been lost in the mists of time.
The service has current up time of 55 Months. The general consensus therefore is that as long as it is never restarted it will continue to perform its function until a replacement can be put in place. Which seems a little fatalistic to me...
Has anyone experience of doing something sensible in a similar situation? <comment><USER2> My advice:

1\. Suggest that the work to replace it is prioritized commensurate with the business impact caused whilst re-establishing service _as if it went down right now_

2\. Remind them that it will go down at the worst possible time.

3\. Ensure your name is attached to these two warnings.

4\. Promise yourself you wouldn't run your business this way.

5\. Get on with your life.

<comment><USER3> I would add to that:

\- Make a plan of exploratory steps that could trigger a failure, in order of increasing risks.

\- Then, if necessary, get a formal sign-off from someone with authority (perhaps a director of the company) to proceed with each step.

Steps like that, which are usually very low risk but do risk taking down an unknown service, might include things like (just ideas from other comments):

\- Logging in to the server at all.

\- Splicing something into the network switch.

\- Cloning filesystems or disk images.

\- Cloning the process memory or system memory.

\- Cloning the image or other things if it's in a VM.

\- Running strace gdb, or packet tracing.

\- Swapping the power source live.

These days, it is often possible to transfer a full working system into a VM on a more modern, powerful machine, and after doing that, it's a great relief to everyone because it's no longer hardware dependent.

I've done that with some legacy systems that were originally on bare metal, and are now still running nearly 20 years later in reliable VMs, with no change to the running software. _Usually_ running faster and with more memory doesn't break a working system.

But doing it on a service that can't be taken down even for a moment is quite an interesting adventure! :-) It is possible, but technically challenging, as long as you can obtain disk and memory images from the live system and then capture changes fast enough to perform a hot transfer.

<comment><USER4> I read earlier that some people were putting legacy apps on VMs, but I didn't realize that you can transfer a full working system into a VM including memory... which is pretty much a full state snapshot. Awesome. This is this guy's solution right there, until a replacement is in place.

</comment></comment></comment><comment><USER5> Actually, I have experience here.

The problem is that anything you do that's potentially destructive in service of getting the system to be more sustainable is going to be met with heavy criticism. So you must be careful, the company has accepted the risk that they're in and you'll have to contend with that most likely.

First thing is first: is it a VM or a Physical machine? Things get a little easier if it's a VM because there might be vMotion or some kind of live- migration in place meaning hardware failure might not actually bring the service down.

Next thing is you absolutely have to plan for failure because the one thing that I learn as I learn more about computers is that they're basically working "by accident", so much error correction goes into systems that it absolutely boggles my mind and failures still get through. So, it's certainly not a question of "if" but "when" and plan for it being soon.

Now, the obvious technical things are:

* Dump the memory * Grab any files it has open from its file-descriptors (/proc/<pid>/fd/ _) Your config file might be there... but somehow I doubt it._ Attach a debugger and dump as much state as possible

Be sure to cleanly detatch : [https://ftp.gnu.org/old- gnu/Manuals/gdb/html_node/gdb_22.htm...](https://ftp.gnu.org/old- gnu/Manuals/gdb/html_node/gdb_22.html)

Don't use breakpoints! They will obviously halt execution.

If it was my service I would also capture packet's and see what kind of traffic it receives most commonly; and I would make some kind of stub component to replace it before working on the machine. Just in case I break it and it causes everything to go down that depends on it.

But, this is a horrible situation, reverse engineering this thing is going to be a pain. Good Luck.

<comment><AUTHOR> > because the one thing that I learn as I learn more about computers is that > they're basically working "by accident"

So true. Thanks this is good advice. As it happens it is a physical server.

I'm not actually mandated to do anything except replace it with a comparable service but this makes me feel like I'm racing against time!

<comment><USER6> Have you tried decompiling the executable to see if any sensible information can be gained that way?

<comment><AUTHOR> Everyone's afraid to do anything on that machine at all. So even getting agreement to login or install software or run tcpdump or whatever is fraught.

<comment><USER6> What I was meaning was that if you can get a hold of the binary that is running you could take it to some other environment and have a look (starting with a hex dump and working up from there...).

Mind you if things are that bad then maybe I'd follow my own advice (in another comment) and stay well away from the currently running thing.

<comment><USER7> this would certainly help recreate the configuration.

</comment></comment></comment></comment></comment></comment><comment><USER8> At a previous job I once sshd into a legacy system which had a motd in the vein of: "Don't ever reboot this system, if you do, I will find you and do horrible things to you". No explaination of why. My opinion was that the person who dared to leave a system in this state was the one who needed the horrible things done to them instead. Needless to say, he was already fired.

We started working on solution to run parallel to the existing one, which would receive shadow traffic so we could observe behaviour, find all edge cases and put them in a testsuite for the new solution. After we where confident that our testsuite contained most of the important behaviour we switched traffic to our new solution, keeping the old online just in case we needed to switch back.

Key is monitoring and learning the expected behaviour of the connected systems, to sense if that behaviour is not as expected and being able to act upon that as soon as possible.

<comment><USER9> I think your solution is fine.

Just another perspective on that server message. This server might be actually running a real-time legacy interface to one of your biggest customers who pay a 6 figure premium to keep this old thing running, just so they have time to migrate. Which they have been trying to do since the 80's, hence the fact that few people remember why it is there.

I'm not saying this happened here, but it happens a lot more than any of us want. Just keep in mind there might actually be a good reason for the warning, although I would object to the wording. First head the warning. Then figure out why it is there.

<comment><USER8> > just so they have time to migrate. Which they have been trying to do since > the 80's, hence the fact that few people remember why it is there.

The questions I would ask are: what is the insurance if the system fails due to natural causes (hardware failure, power outage, you name it)? And what is the cost/benefit balance of having this unpatched security hole connected to the internet (which was the reason we discovered this).

If a system is so important that such a warning is deemed appropriate, imho its just laziness and carelessness from the warning's writer and they are shifting their problem onto the next person that stumbles upon the system.

At least give some information on why it is important and at what time or under which conditions the warning can be considered expired. Just like commenting your code is important because you never know who is having to make sense of it in a few months time (especially true if that person turn out to be you :)).

<comment><USER9> To be sure I'm not defending the practice, just shedding another light on it.

In answer to your first question. There might not be an insurance against the system failing. The honest answer to what to do then, because someone always needs to ask is: Panic! It might not even be unrecoverable, just really expensive and time-consuming to do so.

The follow-up question would be: Yes it is a risk, yes it is a possible disaster. What do you want to do about it? If your answer is anything more than: We should not have taken on that risk in the first place or lets bet the companies future on the fact that we can fix this. Then it might actually be interesting to listen to. It seems from your comment you resolved it.

As for messaging. I do not agree you should explain the situation on the server. Just let people know they should not touch this ever. As soon as you explain why people will assume their reason to do so will trump whatever reason you gave. If they need to think of all possible disasters that might happen it has more impact than the one you can describe.

Messages like "Before doing anything on this server contact Bob" will eventually lead to Bob receiving a message: "We have done this or that. Just letting you know, but it was after office hours" which he will probably see rushing into the office in the middle of the night because the server is not working anymore. The other type of message: "Don't reboot this server ever! For more information contact Bob" Bob changed because he spend a considerable time saying no and explaining the situation to the sysops team, their manager, their manager's manager, etc. who all thought their priority must trump Bob's. Bob might still be working for the company. He might have tried the better part of his career to get this stain resolved. Nothing bad about Bob.

</comment></comment></comment></comment><comment><USER10> There is a tool used in malware analysis and computer forensics called Volatility[0]. It has some very powerful analysis tools and works on Linux Mac and Windows. In your case its ability to dump the memory of a running process without messing with the process state [1] may be very helpful! It also has the ability to run a Yara scan against the dumped memory which could let you find the region of memory containing the config file (so long as you know some of the strings in it).

Hope this helps!

[0] [https://github.com/volatilityfoundation/volatility](https://github.com/volatilityfoundation/volatility)

[1] [https://www.andreafortuna.org/2017/07/10/volatility-my- own-c...](https://www.andreafortuna.org/2017/07/10/volatility-my-own- cheatsheet-part-3-process-memory/)

<comment><USER11> I would advise exercising caution though. Even if the tool works perfectly fine, the service going down anywhere within days to weeks after you captured its state would be seen by the management as your fault. They won't care for the explanations, they will just need a scapegoat and that would be whoever touched the thing last time.

That said, there are plenty of legitimate (albeit low-probability) reasons for the dumping tool to screw up the service. Slightly affecting memory timing and triggering some rare race condition, using up more RAM than usual (or increasing the HDD utilization) and again triggering some conditions that wouldn't happen normally. You name it.

If it was me, I would try to run the binary on another machine and do a "clean room" recreation of the config file. That said, service without sources and missing configs without backups indicate severe organizational problems, so I would probably not want to work in such a place for a long time, unless I was hired by the CEO specifically to rectify things.

<comment><USER12> I agree, there are actually two problems here:

One is technical, and to do with the service and how to deal with it.

One is political, and to do with who gets the blame when the service inevitably fails.

OP needs to work out if they're being tasked with the job as the answer to the technical problem or the political problem. Because management could be very aware that this is a ticking bomb with no technical solution, and have appointed OP as a scapegoat to take the blame when it blows up.

<comment><USER13> What good is a memory dump if you don't have the codebase to navigate? without the code and structure of how its managed a memory dump would just look like an alien language.

<comment><USER14> The phrase you want to google is "reverse engineering from memory dump". Prepare to be astounded.

You need to know how machine instructions work, details of the operating system, and a bunch of other things... but any number of copy-protection systems have been bypassed this way. Any number of devices have had their firmware dumped, their memory scanned, and the resulting data analyzed to make patches or build compatible systems.

</comment></comment></comment></comment></comment><comment><USER15> Dump memory and see if you can retrieve the config that's running. If someone has a clue about how the config should be, you could probably get all the elements of it from simply running 'strings' on the binary. You could also disassemble the binary. Possibly the deleted config file could be salvaged from disk.

As is, this is all relying on the HW not giving up and the UPS working in a power failure!

</comment><comment><USER16> I once worked with a client in such a situation. They were in the process of building a huge oil rig, costing somewhere north of $5.5bn USD. The basic premise was that a previous vendor had configured a big documentation system running on a soon-to-be-outdated Windows Server version ages ago, along with a "kind of" API allowing the shipyard to send information in the form of equipment/construction metadata, documents of various kinds and similar stuff.

The main challenge was that nothing could not be resent if a transmission failed, unless there was an actual change on the shipyard side, or by an highly complex and manual method. There was no source code to anything, and while the enterpricy system was a fairly standard off-the-shelf type thing, the API was completely customized. Nobody knew anything about how it talked to the system, dependencies or anything else. Changing anything was out of the question, as everything had been defined in contracts and processes making waterfall seem agile.

The team I was in was basically there to set up a new application in a separate environment, so that we could migrate and replace the existing setup once the shipyard had handed over all the information. In the end everything was kind of anti-climactic, with everything working as expected for as long as it needed to.

<comment><USER7> > In the end everything was kind of anti-climactic, with everything working as > expected for as long as it needed to.

I often tell people that great software development is boring.

</comment></comment><comment><USER9> I have some experience in this field.

First of all you are there to build a replacement service. As long as you do that in the allotted time what happens to the old should not bother you. Focus on your part of the solution.

The fact that you do seem to worry suggests you need something from the old service. Do you have a good design for the new service? Are there details of the old service that you do not know about? It won't be the first time a company has an application that runs complex calculations, but nobody knows how it is done. We actually built black-box applications in the past to replace calculation modules on obsolete hardware.

If you need information from the old system you may have to resort to cloning the input and output preferably on the network level so you don't have to mess with the existing service.

If the replacement service is years out you might suggest to the company to put in a red team. Another person or persons who work on finding out how the service works by running a duplicate on another system, rent it if necessary, and poking at it seeing if they can get comparable results quicker than the replacement service. But that is not you.

There are a couple of things you need to have in place before going live. Make sure you can clone input and output so you can run the old service next to the new one for a while. You don't want to stop the old service the moment you go live, because if anything goes wrong you need to be able to switch back. Even than starving the old service of input might have adverse effects, so cloning is better. If the old service uses external resources, like a database, files, etc. make sure you do not interfere with file locks, sequence numbers,etc. when running in parallel.

</comment><comment><USER17> A lot of talk about what to do yourself, I'd say do nothing yourself.

Find an expert who has masses of experience who can consult on it.

This isn't a good time to be learning and testing those lessons.

<comment><USER18> This is the solution in my experience aswell.

Doing even minor things that could distrupt the execution for any time (attaching debugger, installing packet sniffer that blocks network traffic for a sec or cause a packet lost, etc) could deadlock the application.

Wouldn't risk it.

<comment><USER19> The only thing I would risk is copy/pasting the binary/executable that is running to somewhere else so I could try and decompile it. Anything beyond that is a nope.

</comment></comment></comment><comment><USER20> My experience in this situation is that the more you decide to the hero and take on things that everyone else is washing their hands of when you have no reason to above anyone else, the more likely it is that you'll also be shouldered with responsibility for things going wrong that were similarly nothing to do with you in the first place.

Prioritise a replacement and don't decide to try smart things with the running one.

</comment><comment><USER21> If it is running you can still get at the binary file through /proc and recover it. The file system will only really delete a file when there are no more users and a running process counts as the file being in used (so that pages can be paged back in from it if needed).

<comment><USER22> > but the physical config was accidentally overwritten

Overwritten, not deleted. Open file handles can't really help you there.

<comment><USER21> Yes, they can. The low level implementation is such that the inode will not be released. That's why securely overwriting a file is hard. Just opening a file for output and writing data is not enough, you need to open it for _update_ and then write as much data as the old one had or more.

</comment></comment></comment><comment><USER23> I'm not sure that 55 months of up-time indicates it's more or less likely to go down in the next month, but I'd guess more likely.

Surely there are options, have you tried de-compiling the source from the binary?

>but the physical config was accidentally overwritten and there are no backups

Any old dev PCs lying around somewhere? It's worth reaching out to the old developers to see if they have a copy, in really old legacy systems it's fairly likely someone will have a copy somewhere.

<comment> barry-cotter > I'm not sure that 55 months of up-time indicates it's more or less likely to > go down in the next month, but I'd guess more likely.

Less likely

[https://en.wikipedia.org/wiki/Lindy_Effect](https://en.wikipedia.org/wiki/Lindy_Effect)

> The Lindy effect is a theory that the future life expectancy of some non- > perishable things like a technology or an idea is proportional to their > current age, so that every additional period of survival implies a longer > remaining life expectancy. Where the Lindy effect applies, mortality rate > decreases with time. In contrast, living creatures and mechanical things > follow a bathtub curve where, after "childhood", the mortality rate > increases with time. Because life expectancy is probabilistically derived, a > thing may become extinct before its "expected" survival. In other words, one > needs to gauge both the age and "health" of the thing to determine continued > survival.

<comment><USER24> OP is talking about a phyaical server. The Lindy Effect is for "some non- perishable things". A server that already has already been running for at least 55 monthes is definitely perishable.

<comment><USER25> A server running software has both perishable and non-perishable elements. The physical hardware is perishable and weakens with age, but as the software itself ages the likelihood that it contains bugs like "it crashes whenever daylight savings occurs" decreases.

The question is which effect dominates at a particular point in time.

</comment></comment></comment></comment><comment><USER26> Hire a consultant to do a full-memory dump.

1\. That gives you a pretty good chance of attaching a debugger (to the offline memory file) and extracting the config from memory without touching the running system.

2\. You are safe in case things turn awful, which seems likely.

As for the source code, if it is an interpreted language like Java or Python or C#, your chances of recovering a fully-working source code tree are pretty good. At the very least, it'll help with understanding what the system does.

For C / C++, buy Hex-Rays IDA and Decompiler. That's what the pros use for reverse engineering video game protections, breaking DRM, etc. So that tool set is excellent for getting a overview of what a binary does with the ability to go into all the nitty-gritty details if you need help re-implementing one detail. Plus, Hex-Rays can actually propagate types and variable names through your disassembly and then export compile-able C source code for functions from your binary.

<comment><USER21> Java is a compiled language; as is C#.

<comment><USER27> If it's not obfuscated, you can get a pretty good dump of a c# binary's source with a decompiler. Some variable names may be missing (local variables and such) and some constructs may be awkward (generator/iterator stuff) but it's usable.

</comment></comment></comment><comment><USER28> My mother had a saying: Choose your battles. I feel like the value of calling things "not my problem" in software is undervalued. You cannot fix every problem, you cannot train every engineer, and you cannot control everything. Learn to prioritize.

If it is not on your head, don't fuck with it. I understand the instinct to fix a Big Problem and look great to management. However, this is too high risk. If you solve the problem, you are a hero. If you fuck it up, you are an idiot and fired and cost the company $$.

Why run the risk at all? Just cash the paychecks and fix other things that can't go catastrophically wrong.

</comment><comment> d--b I personally would look for that code, it must be somewhere... In old code repos or hard drives. Maybe you guys have tape backups or things of that nature.

Don't touch the server, if anything happens, you'll be held responsible.

</comment><comment><USER29> The configuration still exists in memory somewhere, so you could extract it from a dump... whether you should want to do this is an entirely different question though.

<comment><USER30> That's not necessarily true I think. The program could read the config at startup, and create structures or objects inside its heap to represent the config. After that it would have no need for the actual bytes of the config.

<comment><USER31> Those heap objects are the configuration. Not the configuration input file, but the configuration itself.

It is then be a task to reverse engineer the coding file from the config data structures.

</comment></comment></comment><comment><USER32> This is also metioned by dijit, but here's more concrete instructions on how to potentially recover the config from /proc/ : [https://web.archive.org/web/20171226131940/https://www.linux...](https://web.archive.org/web/20171226131940/https://www.linuxnov.com/recover- deleted-files-still-running-active-processes-linux/)

</comment><comment><USER33> In case you want to replace or upgrade the UPS note they make devices that can cut into the power cord used by law enforcement to move a computer which is being impounded without depowering it.

<comment><USER34> Are you saying there is a device to splice the power cable while it is still hot, and re-route power to the computer without causing a hiccup?

<comment><USER5> Oh for sure this is possible: [https://www.youtube.com/watch?v=vQ5MA685ApE](https://www.youtube.com/watch?v=vQ5MA685ApE)

the devices that do it without doing what these guys did is quite pricey though [https://www.cru- inc.com/products/wiebetech/hotplug_field_kit...](https://www.cru- inc.com/products/wiebetech/hotplug_field_kit_product/)

<comment><USER34> I was thinking more along the lines of a single power input like your home desktop PC. That you would need to splice into this power cable while it's still hot.

For servers I expect that they would have redundant inputs, that you can easily hot swap it.

The second link had a video on how to do that. With multiple ways to achieve a power re-route, including a tool to easily splice into the wire. The easiest technique was to just use a sheath to touch the metal prongs on the cable.

But the surprise for me was the power strip technique, where he plugged in his battery source into one of the 6 outlets, thereby simulating a power input. I didn't even know that was possible.

<comment><USER3> > For servers I expect that they would have redundant inputs, that you can > easily hot swap it.

This is true, but there is still a real risk that the dual power input on the server fails when you actually do the swap.

For example, the second PSU may not work, or it may have failed in the past and then be forgotten about, or connector or cable internally may be flaky, or the component that combines power from both PSUs may fail (although I think that's quite rare).

If it's a really critical service, those risks may be unacceptable.

</comment></comment></comment></comment></comment><comment><USER35> You guys may not have the best processes in place, but whoever developed that app really deserves some credit. 55 month uptime without restart. Not bad.

</comment><comment><USER36> Rather than focus on a replacement they should focus on reading the systems memory out and creating a replacement virtual machine or perhaps even trying to decipher the config structure from memory.

My experience with migration projects is that they can drag on in time, and all the while this system is just itching to go down due to a power failure or some other issue.

</comment><comment><USER37> "The physical config was accidentally overwritten and there are no backups".

Welcome to legacy.

So for a situation like this, there are several things that you need to think about. First and foremost... _what is the impact when (not if) this process finally stops?_ This isn't just for technical people. You need a business impact assessment. You need the users involved. They're your lever for fighting the inevitable fear-based political hurdles. Is it an annoyance? Or does the company go out of business? The potential severity of the impact matters a great deal. If it's putting the entire business at risk, you should be able to get support from the highest levels of management to do whatever is necessary.

Second... how do you recover? There are a variety of ways off the top of my head. The most obvious would be to reconstruct the physical config. The "obvious to others" that is probably a stupid idea is rewriting the application. Let's ignore the stupid one and start dealing with reconstructing the configuration.

Do you have the source code for the system? If so, you can probably reconstruct the configuration architecture from reading source, at least. It may suck, but it's something.

Is there a test environment with its own running copy of the app? If so, it will have its own configuration, which will make reconstruction much easier, as then you differ only by values and don't have to figure out what the fields are.

Now, what kind of data is in the configuration that makes it difficult? Resource locations? Authentication credentials? Something else? If it's connecting to external systems, you can look at logs, packet-sniff, etc, to at least figure out where it's going. Credentials can be reset for a new version - a painful one-way trip, but it can work. Do you own any external systems, or are they outside your control?

Now, all systems have inputs and outputs. What is the output of this? Is it going to a database? If so, are you backing up that data? Make sure any locally stored data is getting backed up!

If there isn't a duplicate test system, what would need to be done to create one? Are there licensing restrictions? Specialized hardware/OS? Are you building from source code? Do you have the source? Do whatever it takes to create a parallel system that you can test configuration on, make it run elsewhere.

I can just go on and on with this, but the important thing is to be able to duplicate as much as possible before you try to replace. And find out what the cost is - that buys you authority.

<comment><USER38> >"The physical config was accidentally overwritten and there are no backups".

>Welcome to legacy.

No. Not welcome to legacy. Welcome to a terrible company with terrible processes.

<comment><USER37> Fixing old, broken process is an interesting intellectual challenge in itself.

</comment></comment></comment><comment><USER39> Damn. Everyone here seems to advise you to just protect your own ass as a top priority. Is that a product of US work culture?

If you where in a sound organisation, which I would say most Swedish IT organisations are, you need to think about the company and the clients first.

Someone advised you to: "don't do anything until explicitly asked to". I think that's just bad advise. You obviously know this is a major problem and risk. You also have some ideas about how to proceeds in mitigating and solving it. You should jump at the chance to help your company with this. Highlight this directly to your managers, talk with as many as you can, gather the information you need, get the approvals, and fixed the problem ASAP or find someone who can.

Your clients and customers might be badly affected when hell breaks loose.

I think anyone advising being passive, or plainly hiding from the problem, is totally wrong. Those are not the kind of colleagues I would want.

Get to work.

<comment><USER40> Uh no, that's terrible advice. What if he ends up disrupting the service and cause days or worst weeks of downtime for the company? He'll end up taking the entire blame for it and possibly even fired. Will you come to back him up? Because to everyone else there, everything was working fine for months and years until OP came along and put his nose where he shouldn't have.

</comment></comment><comment><USER41> Are you working on a Nuclear Reactor or something that will cause loss of life if rebooted accidentally? If yes, then you have a truly critical system that needs very careful uptime management, despite huge costs to carefully derisk and duplicate it, and there's plenty of good advice here already. But too many systems are 'super pets' like this and are mistakenly considered critical at exorbitant cost.

If no: turn it off and on, and see. The simple fact of the matter is that sooner or later it will happen anyway, and its better to bring that about, learn and solve. And if it results in large financial losses for extended downtime, then the management that allowed it to get to that state is already at fault (not you) and some better, safer alternative will arise from your efforts. Don't sweat it.

<comment><USER42> Heh. Know of a team of engineers who deliberated over a brittle system for months. Mission critical, world ending stuff if it failed. Their manager got tired of waiting and sent a mail to the team that ran the data centre asking for the machine to be disconnected from the network. By CoB, nobody else had even noticed. Thanks to the nature of big businesses, that mission critical application had been programmed into obsolescence years before by other teams shifting their own workloads off of the server.

</comment></comment><comment><USER43> Have a look at Checkpoint and Restore In Userspace: [https://criu.org/Main_Page](https://criu.org/Main_Page)

I've not looked for a long time but you could potentially snapshot the running process and restore it. Obviously do your reading extensively first though.

</comment><comment><USER44> My first "real" job was working with some old equipment related to some big old IBM (and related) mainframes. To view or change anything you dialed in with a modem and then via a terminal application displayed and modified memory directly.

I sometimes have dreams about specific hex codes that were common / would mean bad things.

However, they were really good about documentation and updates / backups.

Best thing is to start working on folks in the business to develop a backup plan / make your voice heard about the potential fallout of a failure. Or even just ask "what would happen if" of the decision makers.

</comment><comment><USER3> I have used GDB to look at a process and get a dump of particular run-time data structures, which is usually enough to reconstruct a config file.

Config data structures usually don't change while a process is running. Often they are just values in global variables.

If you have the executable file for the process, it may be possible to run that with trial config files, and then compare the GDB dump from the running service with the GDB dump from a trial config, to compare the relevant data structures. That can provide more confidence than just figuring out what the config ought to be.

Getting a GDB dump of the running service will be quite disruptive if it's done manually, but that might not matter. It will depend on whether the service is heavily used or if it's only doing something occasionally.

If the service is in constant use, it could make sense to automate GDB so it captures the config data structures quickly then safely detaches, and only briefly pauses the service.

Alternatively, if even automated GDB is too disruptive or difficult to use, or if the ptrace() syscalls used by GDB might cause a problem, it is often possible to capture a memory dump of the running process without affecting the process much, via /proc/PID/mem on Linux.

If necessary, write a tool in C to walk the heap by reading it from /proc/PID/mem to reconstruct the config data structures that way.

(All the above assumes Linux.)

<comment><USER45> I was going to post roughly this, but with one addition: find a human with knowledge of the config file format. Doesn't have to be a perfect memory but surely someone developed this application and will remember roughly what went in the file and its layout. Then use "strings" or equivalent on the binary and your memory dump to piece together a reproduced config file. Test that in a separate VM until it works. Biggest problem is probably going to be the cut over between the running and new config file instances. If it doesn't work you're screwed. Perhaps do that by diverting network traffic, leaving the working legacy instance in place to allow fall back?

</comment></comment><comment><USER46> May I ask about the business you are in? What happens to the world if your service goes down? It's just curiosity. Hopefully not related to breathing machines like importance. :)

</comment><comment><USER47> Memory dump will probably be what you want to do, but are you sure the process closed the file? If it's a config file, that's sensible the process has closed the file descriptor, but if it didn't (probably because of developer error), it's possible that file be still accessible (through inode for example in Linux). Actually I faced the same problem today; it was a virtual machine disk file that has been overwritten but virtual machine was running, so the (deleted) file was open and I could get it back.

</comment><comment><USER48> Interesting problem. There's some good ideas in here, and I think you might get some better ones if you add more details.

You said that you have been assigned the task of replacing the "zombie" program, which I assume means that you are to write a new one that interfaces whatever depends on the zombie.

If the zombie were to die right now, how severe would the consequences be? On the scale of total disruption of business to minor inconvenience that could be worked around until it's fixed?

How complex is the zombie? What is the nature of the state that was supplied by the lost configuration file? Is it stuff like which port to listen on and how to connect to its database, or is it more like huge tables of magic values that won't be easy to figure out again?

Do you currently have enough information about the zombie to write a replacement? Do you have what you need to test your replacement before deploying it?

If you are confident you have sufficient information to write the replacement, how long do you think you need to write and test that?

You said the zombie runs on a physical server. What operating system? What language/runtime/stack is the zombie based on?

</comment><comment><USER49> I'm at the airport in Dublin. On a layover. I was enjoying my Oreo Fusion at BK and when I read, "The service has current up time of 55 Months" I burst out laughing hysterically.

(Update re the downvote: You can't tell me that uptime isn't a little bit funny. I almost did a spit take at this. As someone who sees and thinks an uptime of 150 days is way too much this number just shocked me, that's all)

</comment><comment><USER50> My father worked in a district heating plant. The control PC was 80286 and had uptime over 10 years. Though the plant in theory could be run without it with knobs and levers.

It was replaced by a Pentium with a virtualized MS-DOS due to y2k event, though I am quite sure nothing changed in the underlying MS-DOS program that was more or less time independent.

</comment><comment><USER51> I remember reading some time ago an old story about folks recovering accidentally deleted system files from the currently running system. Can't find it, I _think_ it was Ritchie who told it but I'm not sure. The long and short is that it might be possible to recover the lost config from what's currently running.

<comment><USER52> This one? [https://www.ee.ryerson.ca/~elf/hack/recovery.html](https://www.ee.ryerson.ca/~elf/hack/recovery.html)

</comment></comment><comment><USER53> In South Africa the citizens are facing a crises of such a service and its our electricity system. Predominately coal power plants, that are at end of life, and rolling blackouts is a common occurrence. It is necessary action, to avoid a blackout, given it would take weeks to get the system going again. Furthermore, high inequality and unemployment, makes it paramount to grow the economy, so these rolling blackouts have a huge impact socio-ecomically. It is a service that can't simply be restarted.

Transitioning risky services which can't be restarted, is clearly complicated, especially for complex systems. I wonder if there is body of knowledge with principles that could apply to not only to these massive nationwide scale, but to that of web-services and alike. Does anybody have resources in this respect?

<comment><USER54> You guys need Jesus... and some solar panels.

<comment><USER53> Unfortunately, the national power provider has a monopoly on energy, and is a political mine field. So independent power producers are ready to integrate massive solar and wind power capabilities, but due to politics they keep delaying the negotiations and contracts. Many private businesses are trying to produce their own power through solar and alike, but its expensive as that technology needs to be important and the rand is one of the most undervalued currencies.

</comment></comment></comment><comment><USER55> The raptors escaped because the The Jurassic Park security system was never restarted before in its history :)

</comment><comment><USER56> > _In addition the source code for this compiled binary has also been lost in > the mists of time._

Sounds like you could try copying the binary, putting it into an isolated VM—or a virtual copy of your other services—and beating a file with a stick until the service accepts it as a config.

</comment><comment><USER57> > _Ever worked with a service that can never be restarted?_

Yes; any Unix init daemon.

What's "restarted"? Does accepting any new configuration and changing behavior in response to it count as a restart?

Or do we have to terminate the process entirely and start a new one for it to be a "restart"?

</comment><comment><USER58> At my previous-previous job we had a core switch that was on from 5 years. No one wants to reload because everyone was scared it cannot restart anymore. We don't have maintenance, spare parts and cabling was a mess.

My colleagues told me that it was still there 2 years ago.

</comment><comment><USER59> Been there.. short story goes like..

There was a active passive cluster with some complicated raid storage.

One day the passive's storage was gone(there were problems all along, corruptions, physical storage change/swap etc etc, mgmt put it on hold since there was a migration) and since there were already plans to migrate the cluster to somewhere else, no one from mgmt wanted to build another set of servers for such a "short" period.

Guess what... that short period was 18 months :)..

So for a 1 year and half , those systems were never touched and never restarted, all the time serving some applications out of it.

Sometimes we get lucky, sometimes we are out of job :)

</comment><comment><USER60> I once overwrote my own partition table. The next week was spend keeping the laptop on or in standby and investigating recovery. I never found a good solution so I started investigating alternatives, and ended with the so called nuke-and-pave, I Reinstalled Ubuntu.

Anyway, that's quite a situation you have got there :) can you not suspend to disc and keep the suspend image safe and able to be booted into after power failures or so? Maybe suspend to disc (hibernate) and then image the swap partition/file? Then some grub editing... or... what OS is this anyway?

</comment><comment><USER61> If it's a critical system and you don't have the ability to fix it but you know it's only a matter of time before it fails, the sensible thing to do is change jobs.

</comment><comment><USER62> As of a few years ago, Google had something vaguely similar. The details escape me, but it was something like --

The main RPC service had a dependency on the lock service to start up and vice versa. If both services went globally offline at the same time, they wouldn't be able to turn either (thus most of Google) back on again.

Someone came up with a wonderful hack to solve this involving basically air- gaped laptops which when connected could help a datacenter bootstrap itself.

</comment><comment><USER63> If the file is still open by the process and not yet closed, the config file is still there if it's Linux. You can find it in the process's file handles in /proc

</comment><comment><USER64> I have seen a scenario where a bespoke application which had also been running for sometime had been lost to the miasma (consultants took the source code with them).

The organization in question paid for a team of forensic software experts to reverse engineer it to the best of their ability ahead of a full data center migration to a new facility.

I left the company while they were somewhere in the middle of this project.

</comment><comment><USER65> You might try something like this, hoping that the process still has the gone file "open".

[https://unix.stackexchange.com/questions/268247/recover- file...](https://unix.stackexchange.com/questions/268247/recover-files-if- still-being-used-by-a-process)

</comment><comment><USER66> From my experience, dumping memory and most likely reverse engineering the binary is a must.

Actually as it is legacy system, changes are this will be easier.

To be honest, the path to follow mostly depends on the OS.

As it is critical system, I would start with reverse engineering the binary, making sure the config is preserved across life time of application.

</comment><comment><USER67> You could try to make every possible request programmatically & save the responses & then rebuild it.

</comment><comment><USER68> "as long as it is never restarted"

Seen that. And then one day, power failed. UPSes kicked in. In one hour, still no mains power. Batteries depleted, systems started shutting down...including that one.

Had to build a replacement in a _real_ hurry after that.

</comment><comment><USER69> I have made a phone call. It's not clear the phone system could be restarted. For example the control plane (SS7) is simply transmitted over the links it controls. It was built incrementally as a continually running system for more than a century and certain bootstraps/incremental elements were long since optimized out. Note that this is unrelated to the fanatical backwards compatibility at the edge which you might think would imply that it is restartable.

The Internet, I think, is restartable as long as layer 0 were available. It's not really clear what it would mean for the Internet to _need_ to be restarted — perhaps some attack/failure of the BGP infrastructure?

</comment><comment><USER70> Unpopular advice: _find a subtle way to make it crash, preferably stealthy, but if not possible, at least in a way that can be attributable to mild innocent incompetence instead of malice!_

Then there will be more and more interesting work to do for you and others, either rediscovering and properly documenting the config, or, hopefully, architecting and coding its replacement! In the aftermath, the organization will be more robust. If it actually collapses bc of this, then it deserved to die anyway, you only helped accelerate the outcome and reduced the suffering.

Some things and processes need to be "helped to fail faster", everyone will benefit from the renewal in the end, even if most will hate it ;)

<comment><USER71> Back when I ran a company, my partners and I gamed out what to do with employees in various situations, as a way of coming to a consensus. I was against legal action in all but extreme hypotheticals. Behavior like this is in that class.

You assume you not only have all the relevant information to predict the outcome, but also that your analysis of the situation is superior enough to trump those who actually own the process and will be responsible for the aftermath of your destructive actions. You demonstrate you know this by suggesting stealth.

Thinking about the fact that people like this are out there makes me paranoid about my hiring all over again.

<comment><USER7> If you had hired competent people to begin with, the configs would have been backed up.

I don't agree with the other posters recommendation, but I understand the sentiment. The same incompetence that allowed the event to happen is the same incompetence that allowed it to be swept under the rug for almost 6 years.

I've seen paragons of hiring get upset because failures were made visible, not understanding that the failures were there all along and they were playing a game of chicken with reality, only now they have a chance of winning.

<comment><USER72> >If you had hired competent people to begin with, the configs would have been backed up

What happy-go-lucky universe do you hail from? You could hire the most competent people in the world and you'll still end up with a horribly complex shitshow with lost programs, untested backups, missing docs, implicit information out the ass, and so on. Software development has never had the discipline to claim compotent employees produce competent results -- they're just less likely to be shitty results.

And we spend 90÷ of our time dealing with every layer of incompetence our competent, and incompetent, friends have provided us

<comment><USER7> It's possible to avoid losing a configuration.

<comment><USER72> It's possible, and it's possible to avoid losing source code, and its.possible to have valid backups, and keep your servers decemtly secure... yet it's sufficiently common (especially as the program ages) that its not trivially attributed to competency; rather you need continuous, unfailing competency, in the face of continuous environmental change (local changes, within the business, and external changes, as business needs change, and mergers occur, and people quit/fired, etc)

<comment><USER7> > rather you need continuous, unfailing competency

That's called doing it manually.

What you're doing is making excuses rather than holding yourself to a higher level of quality.

<comment><USER72> >That's called doing it manually.

Yes, a common feature in unstable environments. Automation is an ideal, but not a necessity, and it's not always viable or cost-effective.

But also that the unfailing competency means that you need to contiously maintain (all aspects of the) automation without fail, again in spite of a changing environment. Across everyone who does things.

>What you're doing is making excuses rather than holding yourself to a higher level of quality.

And what you're doing is pretending that firing off blind, trivial prescriptions is a useful activity -- perhaps I'll soon find you selling self- help books as well :-)

Automation is not an answer that can always be applied trivially; especially retroactively. Even despite all the marketing that will tell you otherwise.

<comment><USER7> > Automation is an ideal, but not a necessity

You're just bad. You automate specifically to take the human error risk out of the equation, that's a part of what it means to be competent. We don't automate our deploys because it's an ideal that is not a necessity, but so some jackass doesn't delete a configuration while also never putting it into a backup, or source control, or on their dev machine, ad nauseum.

It's not an accident that some of us have been working 20+ years in this industry and have never been the cause of something like this.

> Automation is not an answer that can always be applied trivially; especially > retroactively. Even despite all the marketing that will tell you otherwise.

shit, I guess I didn't read the right pamphlet...

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER73> We have a job running on the back end servers here that is expected to take two weeks in order to finish :) Hearing 55 months for your backend makes me more optimistic, haha.

</comment><comment><USER74> My first step would be to try and "undelete" the file from FS, even if it was overwriten, it is possible that some of it is still salvageable.

</comment><comment><USER75> Surprised with the scenario of losing code for a binary that is still running. In which software domains is this common ?

</comment><comment><USER76> Is it possible to run queries on the programs in effort to reconstuct the config or would that also be too dangerous?

</comment><comment><USER77> What OS? On Linux with Checkpoint/Restore you can checkpoint the process to disk and restore it when needed.

</comment><comment><USER78> DD the disk over and scp pipe to another machine, the config file could still be in there.

</comment><comment><USER79> check [https://github.com/checkpoint- restore/criu](https://github.com/checkpoint-restore/criu) \- designed to dump a set of process to a file and restore it elsewere

</comment><comment><USER80> One of the source control companies should take this story to use for marketing.

</comment><comment><USER81> Can you take a core dump while it's running?

</comment><comment><USER82> The is how most systems work, at some scale.

</comment><comment><USER83> Is it running in a docker container? Can you copy the suspended state of that container?

</comment><comment><USER84> You know how with flu vaccine 1 in a big number get sick and with this it may end up worse.

If it's vm take a snapshot periodically, I suspect it's not.

Try a p2v which converts it to a vm on the fly, and leave the vm off and periodically rerun the p2v

</comment><comment><USER85> _capture packet 's and see what kind of traffic it receives most commonly_

The modern solution would be to capture the incoming packets as a training set then apply machine learning to create a model that can perfectly recreate the outgoing packets.

It's still an inexplicable black box of course, that is the nature of ML, but at least you can run it in the cloud now.

<comment><USER86> Sorry that nobody else got it, but I like that joke.

Do you think we can put some crypto in there, too?

<comment><USER87> > _Sorry that nobody else got it, but I like that joke._

No, plenty of us got it. It's just not funny right now.

A person is trying to find a real solution to a real problem they have. Every comment in this thread represents a potential hope that someone knows how to fix it. When someone comes in to a serious thread that is spitballing ideas and tries to make it about them by cracking jokes and being fun and clever, it's annoying. I know this because for about 8 years I was the guy trying to crack jokes in these threads.

This is not the time to make some kind of point about the state of modern software engineering - do that in a thread where people are bullshitting, not trying to work a problem. It's not ever really funny. It's noise. It doesn't help anything. It's not too subtle, it's just obnoxious.

<comment><USER7> what a silly excuse for being offended by nothing.

Watch as I come up with a random reason you're wrong.

The OP could see it and laugh and that could possibly lighten their day and help them relax.

Wait, no. That's positive, online we're only supposed to bash men, hitler, and take everything in the most negative light possible. I should have instead judged you as a hateful person who has a horrible life and no friends.

<comment><USER88> Please don't take HN threads further into flamewar. It helps nothing.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)</comment></comment></comment></comment></comment>


What if Obama gave Y Combinator 40 Billion Dollars - <AUTHOR> http://zaid.posterous.com/what-if-obama-gave-y-combinator-40-billion-do

<comment><USER2> If someone wanted to give us that much money to invest, I'd probably do it, just to see what would happen. We could probably cook up some schemes for scaling. I wouldn't be surprised if it turned out to be a lot like scaling software. You run into a lot of unexpected problems, but you also end up finding ways to solve most of them. Plus you get unforeseen effects of scale, some of them good.

<comment><USER3> Please do not try to "cook up" small start ups. It's better people juice their own ingenuity on their own time, when they have something big at stake.

I grew up in a communist country and I remember how the state gave allowances to housewives in order to invigorate women's businesses. It didn't turn out that well; most women, like most men, are not interested in the ungodly effort and the labor of love that goes into enterprising.

[Edit: specifically, there came up a cottage industry of document fixers who could get you bankruptcy documents so you get to keep the federal investment AND get compensation. People looted the fund.]

<comment><USER2> Don't worry, the scenario is a hypothetical one, to put it mildly.

</comment></comment></comment><comment><USER4> The sudden lack of scarcity would cause the quality of the investments to plummet. YC uses its money carefully because the supply is not unending.

When you only have one shot, you aim.

<comment><USER5> not really applicable, since YC invests in 40-45 companies at a time

edit: I was talking about the "one shot" comment

<comment><USER3> for 40B, people will get into starting up for the _salary_.

don't start a company to make ends meet, start it to execute a haunting vision that you can't get rid of (ever wonder why people quit their jobs, often to their financial detriment, to startup? yeah, it's a curse.)

<comment><USER6> _start it to execute a haunting vision that you can't get rid of_

I agree with this - only partly, because it isn't necessarily in a person's control - but a haunting vision that you can't get rid of is a real asset. It's easier to devote insane effort to something that your intellect and/or soul are deeply invested in than to something arbitrary or trivial. I wonder whether luck is more of a factor in the latter kind of project.

In any case, I am having this experience these days, quite unexpectedly as I never set out to have it and used to feel dismissive when hearing people talk about "vision". I'm grateful for it. It makes it easier to crawl over broken glass.

</comment></comment></comment></comment><comment><USER7> I think a lot of you are missing the point. I could be wrong but I think the gist of the message is to look at what 40B into ycombinator would mean in terms of value added, employment created, etc. (with the wild assumption that it all scales). His point is to heavily criticize the 40B bailout of GM.

</comment><comment><USER8> I'm as much for promoting startups as the rest of you, but is a massive, blind infusion of cash the answer? The thing about YCombinator, Sequoia Capital, Kleiner Perkins, DFJ, etc., is that there just aren't too many of them. There was an article on here just the other day heralding the end of VC as we know it, because the vast majority of the industry wasn't producing good returns.

Investing in startups is the most difficult capitalistic resource allocation problem you can get into because you have no "reputation" to go on. When you're investing in public stocks and bonds, you're usually dealing with companies that have an established track record, or an area of the economy where the variables are known. But that is by definition not the sort of thing we're interested in doing here.

The real question should be, how many startups can PG and team invest in and coach if money were not the limiting factor? Is money the limiting factor right now?

</comment><comment><USER9> This is just silly.

Ignoring the fact that YCombinator probably wouldn't know what to do with $40 billion (a lot of rice and beans?), the jobs created by such an investment would benefit about 1% of the country -- smart, risk-taking, well-educated young men who are willing to live in California.

The point of saving GM is only nominally to save GM. The point of saving GM is to save the industrial base of a big chunk of the US population, and hopefully keep places like Michigan from sliding into permanent and irreversible poverty. The benefit is greater than the sum of the jobs saved.

<comment><USER10> Well yeah this idea is silly, but his original point about saving GM (that it's also silly) probably still stands.

His figure for GM employees is old (2005). Last I saw, the current unionized hourly workforce is 60k employees (see link below). If you simply gave those employees $125k each, it would only have cost the government $7.5 billion. That kind of money should allow all those employees plenty of time to find other jobs or get training for a new _skilled_ career. Then if you gave the retirement/healthcare VEBA what it needed, that would be another $20 billion, for a grand total of $27.5 billion... we've already spent more than that just keeping it afloat the last 8 months.

If they let GM die, of course some parts makers would have to scale down too, but the parts industry wouldn't collapse, because they still have the other car makers to sell to.

Link to employees figure cited above: [http://blogs.moneycentral.msn.com/topstocks/archive/2009/06/...](http://blogs.moneycentral.msn.com/topstocks/archive/2009/06/01/gm- bankruptcy-could-yet-reverse-the-market-s-rally.aspx)

<comment><USER9> The argument about GM is the _silliest_ part. I think it's somewhat plausible that YC could find a way to use $40Bn productively.

Again, the point of this bailout is not just to save 60,000 current GM jobs. The point is to save the GM, the GM supply chain, the local economies of several large industrial states, and the millions of people in an economic ecosystem that has built up around GM. Looking at only the 60,000 jobs at GM is taking an absurdly narrow view of the situation.

<comment><USER10> > _the GM supply chain, the local economies of several large industrial > states_

I don't see how these are going to just disappear if GM is gone. There are other car companies these suppliers can sell to. Of course there will be disruption, and probably some job losses, but not the whole 2-million-person industry (to use the industry number). And if some of those jobs do disappear, maybe it's because demand for the product no longer supports them.

I know my perspective is different on this, because I'm 25, and honestly cannot fathom a lifetime with one company. My career will be a couple years here and there, and most likely I'll have more than one career as well. So it is alien to me that anyone wants to protect (or even prefers) this idea of having a single job, at a single employer, for life. Why is it so terrible that someone might have to find a new job or that a town might die out?

That said, I'm not unfeeling about this issue. I realize these are real lives here (my dad is a GM retiree). What I don't understand is why we are bailing out the _company_ , and not the workers.

I'd rather we spent the money on retraining the employees, hopefully giving them skills in an industry that isn't slowly dying. And this generalizes to all those losing their jobs, no matter the industry. Stop trying to save the individual companies (bailouts) or whole industries (tariffs and subsidies) and educate people.

<comment><USER9> _"I don't see how these are going to just disappear if GM is gone. There are other car companies these suppliers can sell to."_

That's your opinion, but the facts are pretty clearly against you. Listen to the radio and read the newspaper -- the suppliers are already failing. And they're not failing because there's no demand for _cars_ (GM sold about 9 million cars last year, incidentally), but because their primary customer has stopped paying the bills. That's what the government is trying to save -- not to mention the numerous American towns that have no economy without the local GM plant.

And before you argue that bailing out these towns is the wrong solution, try to remember that the world doesn't turn on a dime. Yes, a place like Youngstown, Ohio probably needs to find a way to survive without the GM plant. But they aren't going to diversify their economy overnight, and it isn't going to help them to let their primary employer die.

So maybe you're right, and some non-zero number of these suppliers would survive a total GM liquidation. I don't know. The point is, the bailout isn't designed to save GM, so much as it's designed to save the entire economy that _surrounds_ GM. Everyone who tries to make the calculation in terms of a cost- per-GM-head is being simple, and is failing to consider the larger picture.

<comment><USER11> "And before you argue that bailing out these towns is the wrong solution, try to remember that the world doesn't turn on a dime. Yes, a place like Youngstown, Ohio probably needs to find a way to survive without the GM plant. But they aren't going to diversify their economy overnight, and it isn't going to help them to let their primary employer die."

How much can we afford to spend on people's romantic attraction to their home town? This country's had ghost towns appear when markets collapse and industries disappear before, it could stand to see that happen again. I happen to live in a college town where the second biggest employer (i.e. not the college) is hiring electronics assemblers every week, because it's in a growing market. Why can't a few people leave Michigan and move here?

It's probably wrong to think about this in terms of Youngstown, Ohio surviving the death of GM. People can leave Youngstown.

</comment></comment></comment></comment></comment></comment><comment><USER5> Does anyone else have a problem with assumption #1?

"First off, let's take out 25% for running Y-Combinator. This leaves $30B of cash to dole out."

</comment><comment><USER12> I'd prefer the US government revise the operating practices of agencies like the SBIR[1] to work more like the Grameen foundation[2] to help entrepreneurs of all kinds establish or expand businesses (not just the tech/media companies which are YC's focus) and then put a few billion dollars into it. Extending 40 billion dollars of microcredit loans would make a lot of sense to me in any kind of economy.

1- <http://www.sbir.gov>

2- <http://www.grameenfoundation.org/>

</comment><comment><USER13> The failure rate will skyrocket as all these startups compete for users.

<comment><USER2> It's not users you compete for, but needs. And those are effectively infinite. The limiting factor would more likely be the rate at which technology can evolve. But it would be kind of cool if you ever reached that point.

</comment></comment><comment><USER14> I understand that the main point was to rightly criticize the GM bailout, but subsidizing YC would be wrong for essentially the same reason. Governments are unable to allocate capital as efficiently as markets can. This holds even if Paul Graham is the central planner, no matter how much you might hope otherwise.

</comment><comment><USER15> I believe this has been tried before. Its called a centrally planned economy. How did that work out last time?

</comment><comment><USER16> I suspect many medium to large tech companies would come grinding to a halt as 1 million engineers left to start their own companies...

But perhaps I overestimate such employees' desire to give up job security for the excitement of a startup.

</comment><comment><USER17> And so the government would own 70% of YC, just like GM?

And then, they'd rebrand it to GC - Government Combinator.

</comment><comment><USER18> (1) $6K+(N×$6K) ? _As if._ Now $6MM+(N×$6MM).

(2) Bye-bye Ramen, hello Roman holiday!

(3) Build defensible advantages. Like, a castle with a moat to keep away all the crooks, scammers and favor-seekers. ("Make something people want to break into, but can't.")

</comment><comment><USER19> Ha right, how the government allocates money, "Paul Graham" would spend $39,999,999,992.07 on coke and hookers and the founders would split a frappuccino.

<comment><USER20> Surely Paul Graham does not need that many hookers.

<comment><USER21> Perhaps the money would be to keep them away so he can get some work done...</comment></comment></comment>


Facebook Slaps Google: "Openness Doesn't Mean Being Open When It's Convenient" - <AUTHOR> http://techcrunch.com/2010/11/09/facebook-slaps-google-openness-doesnt-mean-being-open-when-its-convenient/ <comment><USER2> So their theory here is that when I add a friend on Facebook, I am really saying this: "I share some of my contact info with you, but _only as long as we both use Facebook?"_

That's a convenient notion if you happen to work for Big #3b5998. But I think almost anyone would agree that it's really two people making a connection, and Facebook is just the middleman.

<comment><USER3> The theory is that by accepting someone's friend request you're not automatically granting them the ability to export your email address to any application that asks for it. If it were possible there's a good chance your inbox would quickly be filled with spam from apps your friends use. I know of no social network, including Google's own Orkut, Twitter, and Myspace, which allows this kind of mass exportation of friend emails via its API.

<comment><USER4> > If it were possible there's a good chance your inbox would quickly be filled > with spam from apps your friends use.

Apps like, um, FaceBook?

<comment><USER5> More like Zynga apps. Could you imagine if Farmville could easily get a hold of the emails of all of your friends? God help us.

<comment><USER4> My only point is that the scenario yariv frets about is exactly what FaceBook did to grow.

Before I joined FaceBook I would get regular emails about all of my friends on FB that had uploaded my contact info via GMail.

</comment></comment></comment></comment></comment><comment><USER6> You mean Facebook, the company that has had a pretty rough PR year thanks to its deceptive privacy policies and practices?

<comment><USER7> Hey, a failure to respect privacy is just direct evidence of openness, right?

(Disclaimer. I may be biased by working for Google. But I'm also not opposed to switching from a BSD to GPL strategy to encourage freedom.)

<comment><USER8> Laugh while you can. During the Buzz launch, I recall Google found it just as hard to negotiate the intrinsically tricky balance between privacy and openness. There is a real trade-off between the two in products that allow you to share information with groups larger than one.

Good luck with your new social product.

<comment><USER7> I'm fully aware of this. However I happen to personally believe that Google's missteps have been far less than Facebook's - and less intentional.

I'm sure that both companies will continue to fail to find the perfect balance. I'm strongly inclined to believe that Google will fail less. (However people will continue to hold us to a higher standard.)

<comment><USER8> > less intentional.

I'd refer you to Hanlon's Razor. You might also want to introspect as to why the singular social privacy fiasco whose decision-making you've seen internally strikes you as less evil than the privacy fiascos you've witnessed as an outsider.

Also, life is long, and the valley is small. We may work together someday. Let's try to presume good faith when possible, eh?

<comment><USER7> On intentionality, I find myself in agreement with a lot of what [http://www.readwriteweb.com/archives/facebooks_zuckerberg_sa...](http://www.readwriteweb.com/archives/facebooks_zuckerberg_says_the_age_of_privacy_is_ov.php) has to say. Facebook acquired a lot of data when everything was promised to be private, then changed the rules on people. Repeatedly. And Zuckerberg has been privately cavalier about his responsibility to respect privacy from the start, as is evidenced by [http://www.businessinsider.com/well-these-new-zuckerberg- ims...](http://www.businessinsider.com/well-these-new-zuckerberg-ims-wont- help-facebooks-privacy-problems-2010-5).

This does not mean to say that most, or even very many, people working at Facebook are as morally challenged as Zuckerberg seems to me to be. But the fact that I think that some of Facebook's controversial decisions come from Zuckerberg, and that I don't think Zuckerberg is acting in good faith, is why I view various incidents involving Facebook as being more evil.

As for the valley being small, it is but I don't live there. And it seems unlikely that I will ever live there. Furthermore my feelings about any particular company don't generally extend to the rank and file working there. Similarly I don't let my hatred of Microsoft's policies get in the way of my having friendships with people who work at, or used to work at, Microsoft.

<comment><USER8> Sorry for any perceived "valley-centrism"; feel free to substitute "small industry" for "small valley."

What does Zuck's level of personal awesome have to do with anything? This strikes me as a major confusion about the business/consumer relationship. I know Zuck, and I happen to think he's a fine guy, but you don't, and you shouldn't have to.

I don't trust Google because Eric Schmidt is spiritually advanced, or because an engineer who now works at Facebook coined a tongue-in-cheek "Don't be evil" motto 12 years and 25,000 employees ago; I trust it because its search business' incentives are aligned with mine as a searcher.

So, what are Facebook's incentives? Facebook wants you to use its service to communicate with friends, so that it can show you targeted display ads. In the long run, using the service requires you to feel comfortable with who has access to your data, so Facebook's business has to get privacy and openness approximately right. Like everything about a customer-facing service, that turns out to be harder than it looks, at least while providing a stream of new features, but get it right we must for the business' health. But you're about to find out _all_ about that...

<comment><USER7> _What does Zuck's level of personal awesome have to do with anything? This strikes me as a major confusion about the business/consumer relationship._

The answer is, "a lot". And there is no confusion.

I believe that Facebook's behavior stems from attitudes I dislike at the top. Believing this, I believe that they will continue to push the boundary. You're right that they need to find the right balance between openness and privacy. However the corporate DNA looks to me like they consistently err on the side of openness for profit.

Worse yet, the corporate belief seems to be that while people will moan, they will accept continued encroachment on privacy. As someone who has things that I really need to keep private, I don't trust that and don't want my information there.

As for Google's social ventures, I have nothing to do with that and know little about it. If Google starts doing the sorts of things that I see Facebook doing, then I'll readjust my thinking about Google. Companies do change. In the meantime I have my preferences.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> Would anybody like to actually read the words, understand them, and engage with the argument they contain, rather than snark about how FB == teh hatez? The meat of it is:

"Each person owns her friends list, but not her friends' information. A person has no more right to mass export all of her friends' private email addresses than she does to mass export all of her friends' private photo albums.

Email is different from social networking because in an email application, each person maintains and owns their own address book, whereas in a social network your friends maintain their information and you just maintain a list of friends. Because of this, we think it makes sense for email applications to export email addresses and for social networks to export friend lists.

Facebook Platform and the Graph API enable everyone to bring their own information to millions of sites and applications, including even Google's YouTube."

<comment><USER9> A friend won't care that you have their email address it's sorta expected, and one could also argue about whether an email address is even private information.

Or maybe Facebook could allow the export of some other friend identifier.

<comment><USER10> If email addresses were rarely abused, then a private email address would be like an unlisted phone number.

Email address abuse is extremely common however, so public should not be the default.

<comment><USER9> But in this case emails are not being used to compile a list for spammers but to help someone find/invite their finding onto a new platform.

</comment></comment></comment></comment><comment><USER11> I cannot think of a less appropriate organization to declare this exact criticism.

<comment><USER12> I don't get why people have such a problem with this. If Hitler says genocide is wrong, does the fact that it came from Hitler make it any less true.

The arguments put forth either stand on their own or they don't. The fact that they came from a Facebook engineer is irrelevant.

<comment><USER13> Google is not on trial here. As Facebook showed, my data in Google is totally available to me should I need it.

He's explicitly accusing Google of committing a crime they didn't commit, even as he himself is guilty of it.

<comment><USER14> Google did break the Orkut data exporter, then completely removed it when people were trying to switch out of Orkut. I'm not sure how that's accusing them of committing a crime they didn't commit - it's a fact.

</comment></comment></comment></comment><comment><USER15> _The most important principle for Facebook is that every person owns and controls her information._

Sounds very noble but I suspect this is far from Facebook's "most important principle."

</comment><comment><USER16> Openness also doesn't mean being a sucker. Share and share-alike provides an optimal solution to the iterated prisoner's dilemma of openness.

<comment><USER17> They're stuck in a continuous defect!

<comment><USER16> Not at all - Google is defecting, but is offering to cease defecting under certain conditions. It's only continuous if Facebook keeps being a jerk.

</comment></comment></comment><comment><USER18> Can a comment on the tech crunch blog by one of the Facebook engineers be taken as an official Facebook position?

<comment><USER19> Who cares. Seriously, this stuff is very much inside baseball which is why Facebook doesn't even care to comment. Changing the icon on the Like button will generate far more negative PR than this ever will, and spawn 10,000 "OMG, I hate the new like button" groups.

</comment></comment><comment><USER20> They made up their contorted Social Expectation. Most of us have never really thought of what we expect our friends to be able to do with our information. I think we all assume that we trust our friends to occasionally share too much but you have to trust them to some point to just use their best judgement, even now they can go share your email anyway.

One way network grabs are cheap tactics. A user should be able to leave a network with the data that's been entrusted in them, just like with Address books, Calendars, or other trustingly shared data.

This is all about crippling the convenience of people leaving their network for competitors. You can already share with out permission, just not easily. This isn't about protecting our privacy, it's just about slowing down our power to go elsewhere at will.

</comment><comment><USER21> Its been just about a year since I joined Facebook and started using it a lot. I have to say I'm kind of over it. I'm not closing my account or anything. I just think my usage is going to drop a bit precipitously.

I don't really trust Facebook anymore. I don't think they're evil, I just think at some point they're going to have to make some tough choices to live up to their valuation and the only real value they have (outside of ad eyeballs) is their user and graph information. Its going to end up in 3rd parties hands and I'm not very comfortable with that.

Anyway, not sure thats on point but it was something I've kind of been thinking about the last few days and this just drove a few points home.

</comment><comment><USER22> It's interesting how noone wants to actually respond to the arguments that Mike Vernal is making, instead resorting to the equivalent of "Facebook is evil, how dare they say they aren't!".

<comment><USER23> If I happend to email you once 2 years ago did I consent to be linked to you when you join Facebook?

</comment></comment><comment><USER24> Man, I can't believe we have these huge discussions about some petty fights. We know both parties are talking crap to uphold their selfish positions, don't we?

</comment><comment><USER25> _A person has no more right to mass export all of her friends' private email addresses than she does to mass export all of her friends' private photo albums.

Email is different from social networking because in an email application, each person maintains and owns their own address book, whereas in a social network your friends maintain their information and you just maintain a list of friends._

Not really, exporting your address book is the exact same thing as exporting a list of your friends' email addresses.

He had good points against Google, though.

<comment><USER26> I think the difference is that if you email someone, most everyone understands that you're giving the user your email address and permission to email you back unless you use a disposable email address. Contrast that "knowledge" with facebook email addresses. How many people are really aware of whether or not their email on facebook is public, friend of friend, friend, etc?

I suppose that's a separate usability and privacy concern than what we're discussing, but exporting data of friends just seems _different_ to _me_ than exporting contacts from Google. Janet may have exposed her email address and phone number to me on facebook by "friending" me, but she's really just an acquaintance I met through a friend and emailing or calling her is more intimate than either of us want. Conversely, Judy and I met through a work colleague and she wants to do lunch and talk about a startup so she asked for my email address. I now feel like I have _permission_ to email her as well.

My argument doesn't sound convincing, even to myself, but I still feel like the two forms of interaction and communication come with different inherent understandings. I have ~130 friends on facebook all of whom I've met in person save a single contact. The one I haven't met I would and have emailed, but out of the others I'd say 20% are people that would be welcome to receiving an email from me and vice versa. Emailing the other 80% would feel like an encroachment on my part. That said, why would I never need to mass download or import facebook contact details? I've already added the people I am inclined to email into my Google contacts, because there has been a mutual and deliberate decision to exchange contact information.

Not sure where I'm going with this, thinking out loud perhaps. ;)

Edit: Typos...

<comment><USER25> _Contrast that "knowledge" with facebook email addresses. How many people are really aware of whether or not their email on facebook is public, friend of friend, friend, etc?_

This doesn't mean that your gmail contacts are non-private. I could have a private email address that I only use with close friends and a public one I use for signing up for services and business correspondence, etc. This misuse of contact information has already happened to me. I'm pretty sure Facebook knows all about who I know even though I have never had an account just because a number of my friends have signed up and pulled in their contacts from gmail.

Your complaint seems more like an argument for not allowing address book importing at all by any services. Which makes sense, actually, since I don't think address book exporting was meant to be sucked up by social networking services. It was meant to allow you to take your address book with you if you wanted to change email services.

And gmail doesn't want access to your friends list so they can import it into gmail, they want it for their forthcoming social networking product.

</comment></comment></comment><comment><USER27> I wonder if it is common strategy to have an export feature but willingly leave it broken. Some users will think they made a mistake and stay with the service out of inertia (?). I have seen broken exports in suspiciously many online services, most recently in everyone's favorite commenting engine. After filing a bug report, they quickly sent me my data and told me they are working on the issue. I want to believe !

</comment><comment><USER28> If it's on the internet it's public domain, apparently.

</comment><comment><USER29> Well, now I keep bouncing between Facebook and Google regarding who's "right".

</comment><comment><USER30> The pimps are fighting over the data again. I wonder which one will win?

</comment><comment><USER31> To make peace, maybe Google should buy Facebook. Or is it the other way round?

</comment><comment><USER17> I've just noticed that Facebook suddenly looks shitty in Chrome (padding on the left up and went). Coincidence?</comment>


Hans Zimmer is composing the sound for BMW's electric vehicles - <AUTHOR> https://techcrunch.com/2019/06/27/hans-zimmer-is-composing-the-sound-for-bmws-electric-vehicles/ <comment><USER2> Talk about compensating...</comment>


2017 Was the Year I Learned About My White Privilege - <AUTHOR> http://foreignpolicy.com/2017/12/27/2017-was-the-year-i-learned-about-my-white-privilege/ <comment><USER2> His entire premise is built off the following lie:

"Not all the victims of police misconduct are minorities — witness a blonde Australian woman shot to death by a Minneapolis police officer after she called 911, or an unarmed white man shot to death by a Mesa, Arizona, officer while crawling down a hotel hallway — but a disproportionate share are."

There is nothing out of proportion. There are only people who are willing to twist statistics to their needs.

No one disagrees with the raw numbers. Relative to the percent of the black population, more blacks are killed by police.

The problem is that we're comparing the wrong thing, as we can easily see by looking at the statistics by gender. Men are very, very, very disproportionately killed by police when compared to women if we use raw population numbers.

So are police slightly racist and very anti-men or would something else explain why police killings and population rates don't line up perfectly by race and gender?

It turns out that the general population isn't equally at risk to be killed by a police officer, but violent criminals are. If you look at statistics of race and gender reported by victims of violent crimes, you'll see there is a nearly perfect ratio of police killings by race and gender.

The whole "disproportionate" statistics that "prove" police racism is bogus.</comment>


Android Wristwatch Rooted (Motorola Actv) - <AUTHOR> http://www.cmw.me/?q=node/55

<comment><USER2> That's pretty nice hardware for a watch.

It's surprising how locked down the initial user experience is. If it were $99 I'd probably give it a go. Looks like a fun little hacking device.

(oh...and it has GPS, good battery life, bluetooth, heck I'd love one just for giving me walkabout direction when I'm in a city!)</comment>


OS X Yosemite - <AUTHOR> http://www.apple.com/osx/ <comment><USER2> I remember just how buggy Yosemite was when DP 1 was released. But it's come a long way, the third GM candidate for Yosemite is really stable, maybe one of the most stable OS X .0 release. It even has less lag for animations like for Mission Control for retina displays so maybe they finally figured out how to handle a Retina display.

The one thing I'm not a fan of is the horrible, and absolutely terrible animation to take windows fullscreen. It takes a proper 2 seconds for a window to go full screen. I just can't see the reason behind this. At least provide an option to disable this animation or provide a terminal command to speed it up like the mission control animations.

Handoff is one feature I use more than I thought I would. It still feels like a novelty but I'm glad it's there. I use it most to continue my emails on my Mac from the iPhone.

Airdrop on both iOS and the Mac is simply amazing. Now I don't have to wait for Photostream to sync between the two devices.

And the little things like Javascript for automation, built in actions for batch renaming and encoding media files inside the right click contextual menu, the dark mode all add up.

I'm also looking forward to Siracusa's multi-page review about Yosemite. He hasn't been a fan of transparency, so it will be interesting to hear his thoughts on it. I'm not a fan of it either but I like the transparency used in Yosemite. It's not as distracting and the dark theme really helps with that.

<comment><USER3> I really dislike the fullscreen behavior in Yosemite, not only for the animation (as you mention), but also because I feel like the average user is going to want a "maximize" button more often than a "fullscreen" button.

Besides, most sites that have content that lends itself to fullscreen mode (e.g. YouTube) allow you to enter fullscreen mode through the site, so being able to enter through the browser window is not typically necessary.

At work, I use multiple displays, and it's really not seamless when you have one display in fullscreen mode and another not. Imagine having three monitors side-by-side, with the middle in fullscreen mode, and trying to drag a window from Display 1 to Display 3. Maybe this is fixed by now, but in beta, Yosemite sometimes let the fullscreen mode "overrule" the window dragging, so you couldn't see it on Display 2. I know, first world problems, but it's jarring nonetheless.

What's especially frustrating is that there doesn't seem to be a built-in "maximize" function anywhere! The closest I've come to in Chrome is Window > Zoom, but there's not even a keyboard shortcut for that.

<comment> k-mcgrady Double click the tip of the window and it will maximise.

<comment><USER4> This is a great tip, and much easier to remember than option-clicking the green button!

</comment></comment></comment></comment><comment><USER5> Heh. Their "Upgrade Now" button:

"The item you've requested is not currently available in the U.S. store."

<comment><USER6> Same with the UK store.

<comment><USER7> Same everywhere.

<comment><USER8> Same for Indian users!

<comment><USER9> Same for Pakistani users!

<comment><USER10> Same in planet Xii of sector 8 of the 4th galaxial quadrant.

<comment><USER11> I'm on planet Xii of sector 8 of the 4th galaxial quadrant, and it's working fine.

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> For those of you who were a fan of customizing the colors of message bubbles in Messages.app and don't like that Apple removed this ability as part of the iOSification of Yosemite, there's an app for that: [https://github.com/kethinov/BubblePainter](https://github.com/kethinov/BubblePainter)

I made this during the developer previews because I don't like the default puke green for most of my IM conversations. Hope this helps some people. Source code also available.

</comment><comment><USER13> If I upgrade, should I expect some of my dev tools to break? Homebrew stuff or eclipse etc... I bought this mac with mavericks so I don't know what to expect.

What about iTerm and my bash/zsh configs?

<comment><USER14> Short answer is yes. Pretty much every upgrade breaks everything that depends on XCode Command Line Utilities because xcode and the command line utils get uninstalled :\

<comment><USER15> Wait, so I need to reinstall XCode after an upgrade?

</comment></comment></comment><comment><USER16> I've come to prefer the look of it compared to Mavericks on my non-retina desktop display, but only slightly; and when I switch back to Mavericks there are things that are nicer about it. But on my retina MacBook display, wow, there's just no comparison: Yosemite looks so good. Beautiful.

As far as features: I've been wishing for the Continuity and Handoff stuff for years. When my computer's running Mavericks it now feels isolated, like a mini version of how it feels to use a computer that's not connected to the Internet. If you're not also invested in iPhones and iPads, that won't be the case, of course.

</comment><comment><USER17> I know there's literally no hope for it, but I really do wish they could look past their differences and enable air-drop between android and iOS.

<comment><USER18> This would require AirDrop to be an open standard. I would be very happy if Apple started opening their standards. AirDrop, iMessage, Facetime, etc. would be much more useful as cross-platform standards.

<comment><USER19> I might be talking out my arse here but wasn't FaceTime initially touted as being released with a view to heading towards being an open standard?

<comment><USER20> Steve Jobs stated that in a Keynote, but it wasn't planned. He just sort of decided on the spur of the moment (or so I hear). As it turned out FaceTime infringed patents Apple did not control and it all turned into a mess.

I'd love to hear what happened at the first FaceTime meeting after the announcement.

</comment></comment></comment></comment><comment><USER9> When will it be available? It is saying Item is not available

<comment><USER21> Usually they do a rolling release over the course of the day. So it will probably be available by the afternoon and definitely be the evening (west- coast time).

</comment></comment><comment><USER22> If you have homebrew, node/npm, etc. installed that puts files in /usr/local make sure you reserve hours (yes hours) for the install to complete. It will sit at "Less than a minute remaining" for potentially hours if you have a lot of files in /usr/local since it has to copy it all back apparently.

Hitting cmd-L will show you the install log which shows you that it is actually still doing something while it sits there for a long time with no apparent change.

<comment><USER23> Or do something like this:

       $ mkdir ~/local
       $ sudo mv /usr/local/* ~/local
       (install Yosemite)
       $ sudo mv ~/local/* /usr/local

Yosemite has nothing in /usr/local, so you don't have to worry that you'll stomp on anything it installs when you move your /usr/local stuff back.

That worked fine for me. Only a few minutes at "Less than a minute" (unlike 2 hours on the previous attempt, before I knew about this problem and so gave up and restored from Time Machine).

</comment></comment><comment><USER24> They still have serious issues to fix. For example try this sample app from Apple [https://developer.apple.com/library/mac/samplecode/LayerBack...](https://developer.apple.com/library/mac/samplecode/LayerBackedOpenGLView/Introduction/Intro.html). 10.10 uses 4 times the CPU compared to 10.9.

</comment><comment><USER25> Continuity/Handoff requires Bluetooth LE, so a bunch of older devices would not support it.

Chart here: [http://www.macrumors.com/2014/06/17/yosemite-handoff- bluetoo...](http://www.macrumors.com/2014/06/17/yosemite-handoff-bluetooth- le/)

</comment><comment><USER26> This is a little off topic, but I am still unhappy that Apple does not have a simple Android app that would allow me to play the iTunes music I have purchased on all of my devices. I manually keep iTunes music in sync with Amazon cloud which is a nuisance. If Apple does not provide a solution soon, I am canceling my $25/year iTunes Match service and just use Amazon music cloud since they have players for all platforms.

On topic: I have been using Yosemite since the first beta, and it is a nice upgrade. I use a 3 year old MacBook Air, and on a non-retina display everything still looks good. Unlike other commenters, I like the Spotlight changes. I also like the better intro with iOS 8 devices.

</comment><comment><USER27> I have Yosemite GM-3 installed... when I go to the app store, it says "You already have Yosemite installed" \- Should I care or try to reinstall from the "final" installer?

Also, my biggest frustration has been the workflow change of the "maximize" button to it's new role as full-screen button. I wish there was at least a config option to switch it back. I'm fine with adding it as an option, but completely breaking the workflow seems unnecessary IMHO. I guess that's why I use "Spectacles" app for window placement keyboard shortcuts.

<comment><USER28> Holding the Option button reveals the old functionality instead of fullscreen when hovering over window controls.

</comment></comment><comment><USER29> Wicked psyched about the in-window search for terminal. Finally. I like having terminal full screen but it screws up the popup box search.

</comment><comment><USER30> In case you haven't got the upgrade yet, here's the direct link: [https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftw...](https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=915041082&mt=12)

</comment><comment><USER31> I installed the developer betas - I'm updated to the most recent. Is the release today different and more stable, and is there a way I can switch to today's release?

</comment><comment><USER32> Any more info released about the Yosemite hypervisor?

[https://communities.vmware.com/thread/486412](https://communities.vmware.com/thread/486412)

</comment><comment><USER33> As I have no experience with OS X upgrades, does anyone have advice about upgrading versus doing a clean install? Its a rMBP 13 from late 2013

<comment><USER34> Depends on which category you fit into.

You use dev tools and stuff like homebrew/macports a lot: fresh install is a lot nicer.

You don't use the above that much: Upgrade is fine.

As for the packages, I had a lot of different packages that were built for Mavericks, and needed updating, and would constantly bump into something new that needed something reinstalled etc. Various reasons for this, like Apple moving around system utilities and upgrading various things.

EDIT: Just to be clear, all is solvable with the upgrade. It's just a bit cleaner with a, well, clean install ;)

<comment><USER35> I personally always go for a fresh install. I just rysnc my home dir away and copy it back. The rest is mostly setup in ansible land now so I don't really lose much by installing fresh.

</comment></comment></comment><comment><USER27> I have Yosemite GM-3 installed, and when I go to the app store, it says "You already have Yosemite installed".

</comment><comment><USER36> Has anyone run through an upgrade from Mavericks with FileVault enabled? What was your experience? Any trouble?

</comment><comment><USER37> Horrible fonts and colors just as iphone has... :(

</comment><comment><USER38> Link is working now. Download away.

<comment><USER39> Oh si

</comment></comment><comment><USER40> I joined the beta at "2" and it's been incredibly stable on both of my Macbook Air's. Excited for the final refined release.

<comment><USER41> It's very stable for me too, but it still has some really weird bugs. For example - here's how the "iMac with Retina" page displays for me in Safari: [https://www.youtube.com/watch?v=V3ajXezmKDo#t=12](https://www.youtube.com/watch?v=V3ajXezmKDo#t=12)

<comment><USER42> Same here, MBA 13" (Mid 2011)

<comment><USER41> That's the exact same model I have...

</comment></comment></comment></comment><comment><USER43> I'm disappointed how much emphasis we hear about the interface. I don't think design is a substitute for engineering.

Even the new features seemed to be focused on iOS.

</comment><comment><USER44> It makes me sad that with all these wonderful changes, Apple still can manage something as basic as differentiating scroll behavior on a mouse from behavior on a touchpad...</comment>