California to Introduce 'Right to Repair' Bill - <AUTHOR> https://a13.asmdc.org/press-releases/20180307-eggman-introduces-legislation-create-right-repair-electronics <comment><USER2> > "People shouldn't be forced to 'upgrade' to the newest model every time a > replaceable part on their smartphone or home appliance breaks," said Mark > Murray, Executive Director of Californians Against Waste. "These companies > are profiting at the expense of our environment and our pocketbooks as we > become a throw-away society that discards over 6 million tons of electronics > every year."

This. Don't know when all this changed ( 80s ?) in the US, but even appliance repairs are sort of a thing in the past here. If a washing machine or microwave stops working people simply get a new one.

I was quite amused to see an "Appliance Repair" van parked in my neighborhood in San Francisco the other day. I started chatting with the guy and turns out he was from Ukraine and told me that in the former Soviet Union countries, they never throw away any appliance if it stops working. They repair it. Same is the case in countries like India (where I was raised), china.

Esp. in India, there are scores of Radio, TV, phone, appliance repair shops in every city and even in villages.

<comment><USER3> Appliance repair is still quite common in the U.S. The notion that white goods get disposed of as rapidly as they break down is something of an internet- spread cliche.

Most large property management companies employ a group of repairmen, or have contracts with several appliance repair companies. If your refrigerator stops cooling in a large apartment building, the management sends a guy to fix it; they don't Amazon you a new fridge.

Similarly, if your washing machine breaks, your home warranty company isn't going to send out a replacement machine until an appliance repairman says the thing can't be fixed.

I've lived in about a dozen different apartments and houses in almost as many cities in the last 15 years. I've had my appliances fixed by repairmen at least five times.

Interestingly, I had two brand-new GE appliances in brand new apartments that needed to be repaired within their first year of operation. But my current 1999 Maytag washer keeps chugging along without complaint. (Though its companion dryer has been repaired by a repairman twice — once for a clogged exhaust hose that caused a small fire, and once for a dead igniter.)

<comment><USER4> I think the notion that "An apple product cannot be repaired" is also an internet spread cliche.

I found that it was pretty easy to replace the battery and screen of the last few cellphones I've owned. Ironically, it was much easier (and cheaper) to find the necessary parts to repair the apple ones than the cheapo android phones which often cost as much as the aftermarket parts.

If you consider the cost of specialized labour needed to diagnose the problem, then repairing quickly becomes unaffordable for a $300-$1000 purchase. This bill will not stop that.

Appliances are physically larger, and therefore are much more difficult to dispose of, ship, and get into place. So if you compare the cost of repair, to the shipping and disposal cost, they probably become equivalent. Goodluck getting a fridge out of your house without damaging the walls for cheap.

<comment><USER5> YMMV, I've ruined an iPhone5 trying to replace the screen. It's doable, but you have to have a steady hand and a lot of patience.

<comment><USER6> Often the problem with many modern electronics is not doing the repair itself, but getting the thing back together again or getting it back in to working order, because quite often these devices are not designed to be repairable by the consumer.

Witness the widespread use of "security screws", epoxy on chips, cases which can not be opened without breaking them, and explicit warnings that opening the device would void your warranty.

There's also the ever greater miniaturization of components and the ever greater increase in density of parts inside a case. The former often requires microscopes and other specialized equipment to service them, and the latter results in parts not fitting back in the case after you've completed your repair.

It gets even worse when software is involved, where the user is usually at the mercy of the manufacturer to come up with a software update (if they even ever choose to do so) or require some specialized equipment and authorization to even attempt to do their own repair (as is the case with modern automobiles which have so many computers in them).

</comment></comment></comment></comment></comment><comment><USER7> In 1980 Richard Stallman found out that he couldn't fix a laser printer they just installed at MIT. That [and other events]([https://en.wikipedia.org/wiki/Richard_Stallman#Events_leadin...](https://en.wikipedia.org/wiki/Richard_Stallman#Events_leading_to_GNU)) led 4 years later to the birth of the GNU project. Maybe the tide is finally turning, at least in some aspects.

<comment><USER8> Reddit markup doesn't work on ycombinator. Use square brackets and a number to designate the link [0] then put the URL at the bottom of your comment.

[0] [https://www.reddit.com/r/StallmanWasRight/](https://www.reddit.com/r/StallmanWasRight/)

<comment><USER9> That's just normal markdown... sometimes I wish HN had basic markdown support: _italics_ , `code`, [link]([http://](http://)), etc.

Edit: looks like _italics_ works...

<comment><USER10> > _sometimes I wish HN had basic markdown support: italics, `code`, > [link](http\: //)_

I like seeing the URL (or at least the domain) before I click. Hovering over is annoying, particularly on mobile.

</comment></comment></comment></comment><comment><USER11> We have this in Massachusetts for cars (car diagnostic info can't be a car company secret):

It was passed by referendum (Voters voted on the law, not legislators...Although when they saw it was going to pass they passed a similar law).

Here is a summary from the referendum: [1] "A YES VOTE would enact the proposed law requiring motor vehicle manufacturers to allow vehicle owners and independent repair facilities in Massachusetts to have access to the same vehicle diagnostic and repair information made available to the manufacturers' Massachusetts dealers and authorized repair facilities."

[https://en.wikipedia.org/wiki/Massachusetts_Right_to_Repair_...](https://en.wikipedia.org/wiki/Massachusetts_Right_to_Repair_Initiative)

<comment><USER12> From what you quoted, that seems like a reasonable bill. It just levels the playing field for everybody.

Not sure about this California bill though. The devil is always in the details.

<comment><USER11> Details are important. But usually I find there is always a constituency that opposes things and claims doom if anything changes: In Mass especially because of cynical attitude and proximity to more free wheeling neigbor states (RI and NH are like the wild west of the Northeast). We're a small state but we have enough people that businesses want to be here. California is big enough that business will change to do business there. For cars, "california emissions" was on all the cars they sold here because car companies don't want the hassle of two kinds of vehicles.

Also the legislators here will likely change things if they're dragging down the state (We hope...)

we get a summary of ballot questions:

here is the Con argument for the right to repair: "This measure could lead to the release of sensitive personal information, make vehicle hacking easier, and threaten safety and fuel efficiency innovation. Increased safety threats – including theft – are why law enforcement opposes the measure." [1] [http://www.sec.state.ma.us/ele/ele12/ballot_questions_12/que...](http://www.sec.state.ma.us/ele/ele12/ballot_questions_12/quest_1.htm)

We've had this law for a while and none of it seems to be a problem.

Some other things I remember strong opposition to that now seem like non- issues: 1) Bottle deposits (Grocery stores won't exist!) 2) No smoking in restaurants and bars (They'll all go to NH or RI) 3) Cambridge bring your own bag ordinance (nobody will buy anything here again).

</comment></comment></comment><comment><USER13> California always a step ahead.

I remember opening watches and other electronics at circa 199x.

Even 10, 15 years ago most electronics used standard cables and standard screws and anyone was able to open, see whats inside and maybe replace 1 or 2 things.

Now it seems the industry moved to a path of making things break fast and hard to replace anything. Resulting in expansive repairs that will push users to purchase a new item that is cheaper than replacing 1 part. Some companies even weld RAM or processor into the motherboard.

The world has limited resources. In the last 60, 70 years after ww2 and industrial revolution, the population has grown a lot and making thing un- repairable will worsen world limited resources. Some day world will run out of required materials.

Some companies create a specific cable that can only be used on their devices. So if one breaks, the owner cannot use his old one because it is incompatible with that new model.

Standarization of cables would good to have also.

Nice move california!

<comment><USER14> Given that California "joins 17 other states who have introduced similar legislation, which includes: Washington, Massachusetts, Vermont, New York, Hawaii, Illinois, Iowa, Kansas, Minnesota, Missouri, North Carolina, Nebraska, New Hampshire, New Jersey, Oklahoma, Tennessee, Virginia"...

It seems they're not exactly a "step ahead", just trying to keep pace with Nebraska. ;)

<comment><USER15> Nebraska is surprisingly up to date with its digital frontier laws, weirdly enough.

<comment><USER16> Farmers want to repair their own stuff.

</comment></comment></comment></comment><comment><USER17> One of the many reason that I love my 2013 ThinkPad. The service manual is freely available, every single part is serviceable with a bit of know-how, and second hand spare parts are plentiful. It's also built like a tank and hard to break to begin with. You can pick one up for 200-300$ in very good condition.

<comment><USER18> I just ordered a T480 to replace my aging T410. The T410 is still functional, but has been crashing as of late. I'm planning to repair it (I suspect RAM, HD, or CPU heat), but I decided it was time to upgrade to something faster and lighter. Over the time I've owned it, I've replaced the keyboard twice and the battery at least once, and I've upgraded hard drives (it now has 768GB of SSD) and RAM (maxed out at 8GB, IIRC).

I hope I'll be able to maintain the 7-year refresh cycle with my T480; newer ThinkPads are less beefy, but they still have similar levels of maintainability, so I should be able to fix anything that breaks.

I am very much looking forward to the screen upgrade—even the best screen available for the T410 was mediocre at best. I've heard good things about the WQHD IPS screen I'm getting on the T480 (not quite as good as a MBP, but leaps and bounds better than my T410).

</comment></comment><comment><USER19> Ugh. Another bill whose largest effect, to be understood only 10 or more years from now, will be to make sure that very large companies stay very large and very small companies have a hard or impossible road to growth.

\-- Someone who likes repairability as much as the next guy.

<comment><USER20> > will be to make sure that very large companies stay very large and very > small companies have a hard or impossible road to growth.

Do you mind connecting the dots on this? Apparently you think it's self evident why it would have those effects on companies, but it's not, at least to me.

<comment><USER21> Not OP, but I believe he means the following:

As a small company/startup, right now, you design, make, and sell a product. It may just be you or a small team. The design of your product may be complicated, but the usability from the user perspective may be simple AND FOOLPROOF. There are no other considerations generally other than the product works, and people like it and my margins are good.

With more regulation, like requiring the right to repair, the task above just became a lot more difficult. You now have to design a product that can be repairable, which can be more expensive from your design/production side. You have to make sure there is documentation to go with that for consumers to repair every aspect of your product, an incredible amount of time. And then you have to have a way to "rescue" a user if they really screw up the product and "brick" in in a sense. This will add to the total cost and the time it takes to go to market and/or iterate on the product. Small companies in many ways, can't afford this. not to mention being in compliance of the regulations and the additional cost/time that will take.

On the other hand, a larger company will have no issue complying with these regulations as far as documentation, user support, return and fix issues, etc. Making sure consumers have the tools available to "repair" the products as well. Because larger companies have economies of scale, both with supplies, manufacturing, and labor costs, it is much less of a burden for them than a small company barely trying to survive.

<comment><USER22> I haven't read the text of this bill, but the push for Right to Repair isn't that companies need to design their products differently - but they need to provide documentation and components so people other than the manufacturer can perform repairs on them.

Apple doesn't repair logic boards, if you have a bad backlight controller on your MacBook Pro and it's out of warranty you can expect to pay $700+ for a logic board replacement - repair shops can replace the backlight controller for a much more reasonable fee.

Right now Apple products are the most popular market for these 3rd party repair shops exactly because Apple charges so damned much for out-of-warranty repairs, and they work in a legal gray area because Apple won't give out board schematics or sell replacement components. Without schematics troubleshooting issues ranges from anywhere to a pain to impossible, without parts anything that can't be readily sourced from a manufacturer (thankfully they use bog- standard Texas Instrument controllers for the LCD backlight that you can buy reels of, but other components like the SMC, GPU, etc. you are pulling off boards that were dumped from the factory or if you can somehow find "new" ones the chances that they are DOA is dubious since they're often factory rejects).

Nobody wants to make life more difficult for companies designing their products, the Right to Repair crowd just wants to be able to fix their own devices provided they have the skills to do so.

<comment><USER23> Providing that documentation and the components is a non-zero cost.

<comment><USER24> Technically it costs bandwidth and electricity I suppose to make information available...

Not suing people for repairing or tinkering with your hardware is also a "cost" I suppose if you want to phrase it that way, because of the lost revenue you would have gotten from locking down the market and only providing overpriced replacements yourself. It's a cost in the same way that a law against stealing purses imposes a "non-zero cost" on all of us, since we don't have that potential source of revenue available to us anymore...

</comment></comment></comment></comment></comment></comment><comment><USER25> "Right to repair" sounds good, but I'd like to know more details.

How much documentation are you required to give users? If a device relies on programmable hardware, is it really "repairable" if you don't have the source code? What if there's a security vulnerability in the software, how do you fix that? If the ICs can be replaced but the original manufacturer stopped selling them, what then? Do the electronic interfaces between all the components need to be documented? How do small device manufacturers stay in compliance? Do end-user repairs void the warranty if the user is just bad at soldering and screwed up a surface-mount part? If the original design of the product was defective in some way, do consumers have the right to fix defects, or only to do repairs that return them to the original condition they were in when delivered to them?

I think it's possible to craft a good right-to-repair bill that strikes a reasonable balance between consumer protection and not being overly burdensome on manufacturers, but I can't tell if this bill does that. The description (like most descriptions of bills given to the public by their proponents) is long on all the anticipated positive side effects and short on details of how it's accomplished.

<comment><USER26> > What if there's a security vulnerability in the software, how do you fix > that?

I would really wish for the 'Right to Repair' to be strong enough so that producers of all those trash IOT devices, mobile phones etc. would be forced to provide a guaranteed maintenance period for at least 5 to 10 years or would have to release all the source code and documentation if they stop updating their software so that other companies or users can step in.

As a side effect it might also be possible for the consumer to choose the software separately from the hardware. Because then not only the Hardware designing company would be the only one that can write Software for it. More competition and a level playing field would be good for the consumer and economy as a whole.

</comment></comment><comment><USER27> Here's a relevant video in regards to farmers and their equipment that drove the impact of the 'Right to Repair' bill home for me.

[https://www.youtube.com/watch?v=F8JCh0owT4w](https://www.youtube.com/watch?v=F8JCh0owT4w)

</comment><comment><USER28> My big pet peeve is non user replaceable batteries in devices (phones, laptops). It's very well known that Lithium batteries wear out over time. That means that devices with built-in, glued-in batteries are guaranteed to become useless once the battery begins to fail.

The battery in my Nexus 6P is about 18 months old and it currently only holds about 40% of it's original design capacity (known issue with the phone). Replacing the battery is a very delicate task due to the glue and placement of the battery and the need to replace the glass and LCD.

<comment><USER29> It's not too difficult for an independent repair shop with the right tools. We charge $49 to replace a Nexus 6P battery and it takes about an hour. Call some local phone repair shops near you and get some quotes.

If you happen to be in Austin or Houston, contact me :)

</comment></comment><comment><USER30> _"The legislation would require manufacturers of electronics..."_

Seems like there's a pretty big loophole here in that it's mandating just manufacturers, not retailers and resellers. There's a fair amount of electronics on eBay and Amazon that are from US resellers of Chinese manufacturers. Those manufacturers often have no US presence, and so no compelling reason to care about any of this.

<comment><USER31> Resellers really aren't the problem.

The problem is manufacturers who "lock-down" their products to _prevent_ repairs using things like proprietary software and DRM.

To make matters worse, Section 1201 of the DMCA makes it illegal to break DRM, no matter what the reason, effectively making it illegal to do repairs in many instances.

The "grass-roots" movement for "right to repair" legislation was started in Nebraska by farmers who are sick of being unable to repair their tractors without the assistance and permission of John Deere Inc. John Deere tractors are currently rejecting usable parts because the owner of the tractor cannot tell the software to use the part.

<comment><USER30> Right. This is not a general loophole, but a loophole for products where the manufacturer has no US presence. Because of portals like Amazon and eBay, there's more of this than you might think.

</comment></comment></comment><comment><USER32> This bill is a step in the right direction.

The Earth has a limited amount of resources, so we should really start thinking about better resources management because otherwise, we're going to have serious problem in the future.

<comment><USER33> That's not exactly a good way to look at it. The Earth is enormous, and we have access to plenty of green energy. Wind, solar, nuclear. And with enough energy every other problem is easy. Including waste processing.

The problem is that too short product lifespans close off a lot of avenues for long term utility extraction from products. (Second hand markets, donated stuff.)

<comment><USER32> The Earth may be enormous, but that doesn't mean we shouldn't be more careful about consuming it's resources.

</comment></comment></comment><comment><USER34> As someone who doesn't live in California, will this bill (indirectly) make available to me the service manual for my Acer Travelmate?

I've search high and low and absolutely can't get my hands on one.

<comment><USER35> there is a good chance that you will benefit from this step. It's easier for the manufacturer to make the manual available globally than restricting access to only Californians.

EDIT: Is it even possible to restrict access to people from a US state? Even if I wanted to do that I would have no idea how.

<comment><USER36> Or for someone in CA to make it available outside CA

</comment></comment></comment><comment><USER37> In Australia I believe around 2011 they made a change to similar law called the Australian Consumer Law (ACL).

In case of a major issue with a product, you can:

Repair, replace or refund. And it is the buyer's choice - not the company.

In case of a minor issue:

The company must provide a free repair within a reasonable amount of time. If they don't or can't, you can get it repaired elsewhere and pass on the cost, or get a replacement, refund or compensation for the drop in value.

Also the best bit is it makes no reference to warranty time period. So a company cannot say "6 month, 12 month or 24 month warranty" \- if they do, consumer law overrides that condition. The length of time is intentionally vague as "the time you'd expect it to last".

If you purchase a 100" TV brand new for $300, because of the cheap price the warranty may be only a year or so. If you purchase a 50" TV for $4000, you'd "expect" it to last many years - and people have received full refunds for their TVs 5+ years after purchase when they have developed a major fault.

This is great, because it forces manufacturers to bare the cost and not plan intentional obsolescence into their products to force you to upgrade when they break. It's therefore cheaper for manufacturers to build quality products that will last.

Reference: [https://www.accc.gov.au/consumers/consumer-rights- guarantees...](https://www.accc.gov.au/consumers/consumer-rights- guarantees/repair-replace-refund)

</comment><comment><USER38> I'm a big supporter of this.

I also feel 'right to support' is very important. Copyright law that mandates proprietary service and technical manuals and software should become public domain after x years.

As an example, modern cars are virtually unserviceable by mechanics with no access to proprietary diagnostic software and equipment.

If a company is out of business or their support staff are unfamiliar with old products, publish everything to help right to repair...

</comment><comment><USER39> The cost to bring any good to market is a combination of: (1) R&D (technical and non technical), (2) Manufacturing, (3) Sales (compensation to intermediaries bringing the product to marketing), (4) Regulatory and Compliance (5) Consumer labor (6) Externalities

New product prices are:

(1)(2)(3)(4)

Repair-part costs are dominantly comprised of:

(2)(3)(5)

If (2)(3)(5)(6)_oldProduct > (1)(2)(3)(4)(6)_newProduct the market is telling you your time is better spent doing something else and properly recycling the equipment.

If the state inflates (4) to get (2)(3)(5)(6)_oldProduct < (1)(2)(3)(4)(6)_newProduct to be true, all that's been achieved is bureaucracy has siphoned off value from R&D and less products are produced.

If there are negative, socialized costs associated with consumption (externalities) there's a justified argument to allocate those costs on the people who are producing them (recycling tax at time of purchase). Product Management via regulation is not how you get a vibrant market of inexpensive products that people want.

<comment><USER40> At one point in history we had a "vibrant market of inexpensive products" by allowing child labor, slavery, and polluting the air. We're deciding that there are some things that are not acceptable. Thankfully, more and more people don't care about "vibrant markets" if it means a landfill of irrepairable electronics that gets dumped onto some poor country.

In any case, regulating the right to repair opens up new service markets, new parts supply chains, and a large job market that is local and difficult to outsource.

<comment><USER39> Are you arguing that opposing regulations that require people to publish manuals online is equivalent to child slavery? If not I have no idea your point as I already discussed allocating externalities of pollution.

<comment><USER40> I'm arguing that its good to 'leave money on the table' if that means doing something positive for the environment or people.

</comment></comment></comment></comment><comment><USER41> Does anyone know how this affects the DRM circumvention clauses in the DMCA[1]?

Currently trying to figure out how your stuff works can carry a prison term if there's DRM involved.

[1]: [https://en.m.wikipedia.org/wiki/Anti- circumvention?wprov=sft...](https://en.m.wikipedia.org/wiki/Anti- circumvention?wprov=sfti1)

</comment><comment><USER9> If only it were cost-effective to repair cheaper things. Our vacuum recently stopped working correctly. We only paid $100 for it. Paying someone to repair it costs around that much after parts and labor. So my options are to fix it myself or simply buy a brand new one...

<comment><USER42> It's often just the labor that's expensive (at least in the US). The parts to repair most home appliances and electronics tend to be reasonably priced and available all over the web (my favorite source is eBay—people disassemble old appliances and sell parts separately). The tricky part is often merely access to service documentation, which is what most "right to repair" laws seek to correct.

</comment></comment><comment><USER43> 1\. for many appliance, repair is way more expensive (including the hours you spent on calling and scheduling and waiting (forever) for a never-arrived or always-running-late service guy, I had to just throw it away but just buy a new one instead. is this a problem of the design(say, make it DIY fixable with replace-friendly modular design) or something else?

2\. easy to document and repair hardware components, what about failure caused by software? is factory-reset the only way?

just some random thoughts, lunch time here

</comment><comment><USER44> The spirit of this sounds great on paper, but are we not recognizing the reality that technology innovation necessitates an inability to repair?

It happened in the car world when emissions regulations hit the scene and vacuum hoses and check engine lights became the norm. Can't imagine anyone repairing their own Tesla.

If the answer is to make devices thicker so they can be more easily serviced, I couldn't be more against.

</comment><comment><USER45> I would think that the insurance/warranty industry would result in considerable self-regulation. Many people (myself included) purchase 3rd party warranties, and these warranty companies expect to be able to repair appliances. Those that couldn't be repaired I would assume would not be covered, and that signal would get out to the consumer, and then to the manufacturer.

</comment><comment><USER46> I will have to read the text of this bill before I make any judgements (I haven't found a link to it, I don't believe the bill has been introduced to the Assembly yet.) but I have a hard time understanding what they would legislate. Is it going to be availability of replacement parts? Assembly methods that make repairability easier? Documentation of signal protocols so that if Samsung decides to stop making the washing machine controller, I can make my own?

I've personally repaired a number of home appliances over the past few years including a LCD TV, a clothes dryer, a remote light switch, a juice machine, an espresso maker, and a vacuum cleaner, and I've tried and failed to repair a coffee maker and an electric tea kettle. I've also evaluated a macbook air and a samsung galaxy phone for repair and found that the replacement parts were too expensive to justify.

My perspective is that usually the replacement part is available but the parts cost plus the time/skill to diagnose multiplied by the potential success factor in repair makes replacement a better option in many cases. I don't have a whole lot of confidence that the government can solve this problem with legislation; any rule that would solve repairability issues with your electric tea kettle would make your phone bulkier, for example. So my guess is it's going to involve a lot of lobbying by different industries and ultimately end up toothless, misguided, and ineffective.

Oh well. I guess I'll just keep on replacing fuses and repairing traces.

</comment><comment><USER47> My Nexus 6P I purchase 1.8 years ago failed, unable to boot into Recovery Mode. I called Google and they said it's a known issue and the one-year warranty expired so...good luck. Called Huwei, they said there is no option for repair.

A known issue bricks a device that is less than two years old and I had to go get a replacement. What the fuck.

</comment><comment><USER12> I love the idea of "right to repair", but as usual, _would require manufacturers of electronics to make diagnostic and repair information, as well as equipment or service parts, available to product owners and to independent repair shops._ could have far-reaching negative consequences.

<comment><USER42> Such as...

</comment></comment><comment><USER48> Can states also locally repeal DMCA-1201?

</comment><comment><USER49> More regulation making it harder to produce goods in California / sell goods to consumers. If you want to buy repairable goods, vote with your pocketbook - don't force your preferences on me.

<comment><USER50> > If you want to buy repairable goods, vote with your pocketbook - don't force > your preferences on me.

This only stands true in a competitive economy. We live in an oligopoly, where the highest tech choices are not repairable.

The oligopolists will be able to lobby against the "Right to Repair."

My point is there isn't much ability to have "preferences" (i.e.: nothing to vote our pocketbook on) in the first place.

<comment><USER51> All this proves is that people are unwilling to make a value judgement. To most consumers, having the "highest tech choice" is more important than having a repairable device. To really put one's money where their mouth is with repairability, one would have to show that they value repairability even if it comes at the expense of other things they value.

<comment><USER31> It proves that corporations can create small monopolies (over repair of their products, etc.), and use their economic advantage for more economic advantage, leading to true monopolies.

I literally cannot vote with my wallet when there is a monopoly. That is the problem here. This legislation is about preventing a company from claiming a monopoly over repair of the products they sell.

</comment></comment></comment></comment><comment><USER52> I remember many years back California passed a law that restricted what equipment auto parts stores could loan out to owners to diagnose their car issues. Maybe they should lift that law also.

<comment><USER53> There is no such law in California.

</comment></comment><comment><USER54> The only concern I have is, as long as companies can exclude portions of a product whose sole purpose is to protect the privacy of the user data. I am worried that law enforcement will use this to end run the current situation mostly with apple products that the secure enclave processor/parts cannot be altered without loss of data/etc.

as far as repairing tablets and such, any design changes forcing easier to repair items will have costs baked into the next generation and do these laws limit how much replacement parts can cost? Because it can take awhile for a 3rd party to make working replacement parts. finally, who certifies such?

</comment><comment><USER55> On this note, be sure to take the cover off your garage door opener and slab lithium grease on the gears every 5 years.

The gears are plastic and turn into snow once it dries out.

</comment><comment><USER56> interestingly enough this bill does not yet exist: [https://leginfo.legislature.ca.gov/faces/billSearchClient.xh...](https://leginfo.legislature.ca.gov/faces/billSearchClient.xhtml?session_year=20172018&house=Both&author=Eggman&lawCode=All)

</comment><comment><USER57> Does this mean that the source code for web services should be free, so a user can repair it?

</comment><comment><USER58> first world problems. Really good move considering the amount of e-waste that is generated.

</comment><comment><USER59> Why not cars?

<comment><USER60> We already have this for cars. It was passed in 2012 in Massachusetts and applied throughout the country. [https://en.wikipedia.org/wiki/Massachusetts_Right_to_Repair_...](https://en.wikipedia.org/wiki/Massachusetts_Right_to_Repair_Initiative)

</comment></comment><comment><USER61> This is nuts. If people wanted to purchase repairable products, they'd do it. Clearly the market doesn't care, so all this will do is create needless litigation and artificially constrain product design.

<comment><USER62> No, it would not work that way. If none of the companies produce repairable machines that also satisfy other constraints of consumers, this problem would not be satisfied by the market.

<comment><USER51> > If none of the companies produce repairable machines that also satisfy other > constraints of consumers

Thats the point though. If people actually valued repairability, then they would elevate its importance above other constraints in their purchasing habits.

<comment><USER63> How does one do that when there is no suitable option?

<comment><USER61> Companies do lots of focus groups and market research. If there was enough demand for repairable devices to justify it, some company would see there was an unserved niche they could serve.

<comment><USER63> So because a small sample of people didn't show overwhelming support for it, it's not an option for anyone. Yet "vote with your money" is still touted out as a viable option?

I'm not following this argument.

<comment><USER64> Sure you are. Why does every phone or car not cost $100,000?

Surely if all the manufacturers acted in their obvious local-maxima best interests and jacked their prices, you'd have no choice but to take out a mortgage to get a phone.

The answer is obvious: the first company to defect and charge $1000 for a phone instead would take over the entire market.

What's the difference between price and features? If repairability is important to consumers, why is nobody "defecting" from the cartel of non- repairable vendors to take over the market?

It's not a complicated argument.

<comment><USER63> > It's not a complicated argument.

I love it when people end complicated arguments like this. "It's easy you dolt! Just eat your soylent and be happy!"

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER65> The most embarassing thing, if you really think about it is- all these repairers and tinkerers- they want to put work into a product- and you deny them, because of some shortsighted sense of property and a fear of loss of controll.

Look for example at games pre-steam- these where gated-communitys, walled in plattforms and gardens. All those modders, where putting in work anyway.

And instead of taking a percentage, and supplying a plattform - we shut them out.

Cant wait for someone to offer a hardware-steam version for IOT devices and vehicles. Add a simple scripting language like Lua to that - and suddenly - one ecosystem to rule them all, thanks to all those tinkering customers, who basically send you free money.

But hey- lawyers and small sighted software department CEOs will see that it does not come to that.</comment>


Microsoft TLS 1.2 downgrade bug and how it was fixed - <AUTHOR> https://blog.cloudflare.com/microsoft-tls-downgrade-schannel-bug/ <comment><USER2> Having worked with SChannel to build a Python wrapper, there are a number of little bugs that require falling back to TLS 1.1 or retrying the connection. For instance, if a trust root uses an MD5 signature, TLS 1.2 can't be used. Also, sometimes the dhparams for a key exchange will be encoded in 1016 bits instead of 1024 (the server omits a leading zero byte) and SChannel aborts with a cryptic error about buffers being too small. At least Microsoft generally does a better job with their crypto docs than Apple does.

<comment><USER3> That makes it sound like the "bug" (lack of support for updating TLS session keys) was known for a long time, and the original "fix" was the client admitting it didn't really fully do TLS 1.2, which based on your experience seems to be a typical "fix" in SChannel.

(or, I'll admit, it's more likely the fallback after a "failed negotiation" was a "robustness" behavior that essentially hid problems)

<comment><USER4> Yes, the TLS 1.0 fallback (IE skips 1.1) was what hid the problem in this case.

</comment></comment></comment><comment><USER5> Absolutely fascinating post, thank you very much for sharing.

Does this mean that users of Microsoft operating systems who have not yet applied to the patch will be unable to connect to CloudFlare fronted websites?

<comment><USER6> Depends on which Microsoft OS and user agent.

If, e.g., you're using an unpatched Windows 8.1 with IE11, you'll still be able to connect initially but will get disconnected the first time the key is rotated (max of 1 hour). Upon disconnection, IE11 will downgrade from TLS 1.2 to 1.0 and try again with a new session, so the user probably won't notice anything unless they're looking for it. That being said, many sites are starting to deprecate TLS 1.0, so this could become more of a problem for patch laggards.

If, instead, you're using a .NET client that doesn't try/catch the exception, your program will most likely crash. For customers like this with had a high dependency on Microsoft clients, we simply disabled the issuance of session tickets (just for their site) until the fix was made public.

Basically, SChannel freaks out when it sees a new session ticket presented during the _abbreviated_ handshake (it handles it fine during the full handshake). Without seeing their code, it appears that only portions of RFC 5077 were implemented. Btw, most sites don't ever bother to ever rotate their session ticket keys outside of httpd process restarts, so this is why it wasn't discovered for so long.

</comment></comment><comment><USER6> Does anyone else know who rotates their session ticket keys? I'm aware of Twitter (jsha's post here: [https://blog.twitter.com/2013/forward-secrecy-at- twitter-0](https://blog.twitter.com/2013/forward-secrecy-at-twitter-0)), but not really any others..</comment>


The Road to Damascus - <AUTHOR> http://www.texasmonthly.com/articles/the-road-to-damascus/ <comment><USER2> All of a sudden the work I do seems very unimportant. God bless you Austin and safe travels home.</comment>


Now this is a bubble. - <AUTHOR> http://i.imgur.com/6ptfPLQ.jpg

<comment><USER2> Bitcoin and the Dotcom bubble are significantly different in my opinion. If Bitcoin were to fail, the world economy would function as normal, nobody would lose their jobs and a multitude of large companies wouldn't go under like the original boom. I know the Bitcoin investment craze is starting to kick-off, but it's not at the point were companies are investing large portions of their cash into Bitcoin to the point where it would fail and companies would go out of business, it's too volatile to be a trustworthy investment at the moment.

If there are people out there who have invested large amounts of company cash or people who have taken out a second mortgage on their homes to buy Bitcoin, the joke is on them. Because Bitcoin has been unrealistically unstable for a long time now, the current growth can't be sustained.</comment>


Dave McClure: Why Not To Do a Startup - <AUTHOR> http://vimeo.com/15799330 <comment><USER2> Interesting style. I feel like this is hyperbolic in his negativity but if there are people in the audience who were expecting this to be easy perhaps it's what they need to here.

The salary/options becoming public anecdotes are interesting. More and more I'm wondering if this type of information should just be public so these discussions happen early and it's all out on the table.

</comment><comment><AUTHOR> Slides: <http://www.slideshare.net/seattle20/dave-mcclure-5290997></comment>


Insurance on Rails - <AUTHOR> http://blog.thinkrelevance.com/2009/7/21/insurance-on-rails-2

<comment><USER2> Somewhat similarly, as a stopgap in '07 I built a Django (well really, Python and duct tape) Sarbanes-Oxley SOD system over a month or so.

I had to deal with well over a million records, but fundamentally it was a pretty simple ask, laying a 2d matrix of rules over user and permission records from 30 different systems. I even hooked it up to SAP and various AS400s through MQ, print queues and various other things.

It was a total stopgap move - so a whole bunch of vendors bid on the "real" system, one was selected, and now almost 3 years later and a few million dollars spent, and the "real" system still isn't working.

"Web 2.0" frameworks have a lot of applications in business, the ORM + presentation layer tools give you most of the groundwork you need, but as the article says, the key is in the polish and the business oriented design.

<comment><USER3> If they have been using your system for 3 years, you might want to bid ~500k for a ~6 months contract to get it to the next level. It might not be _fun_ but you can make crazy money building custom enterprise applications once you have some credibility.

<comment><USER2> Ha! Indeed!

I would absolutely consider doing that if I had time. I'm kinda all-in on my own startup at the moment and I'm doing my level best to not be distracted by contracting.

And oh how right you are about it not being especially fun! You make a good point about the sort of money that I should chase to do the job properly, too.

Thanks.

</comment></comment></comment><comment><USER4> Although I don't work with COBOL, I do work at a financial services company and we have a lot of old software that we are not going to replace with Ruby, PHP, or any other dynamic language because the precision and accuracy of decimals and floats in dynamic languages is inadequate. I'm really surprised that the precision issue gets ignored at the native numeric level within dynamic languages.

<comment><USER5> Ruby does have libraries for dealing with real numbers with an arbitrary precision.

<comment><USER4> I haven't gotten very far with this issue in Ruby, however in PHP there is also a way to get around the lack of precision by treating numbers as strings and then using additional libraries to perform the necessary calculations. The ability to do this in a clear and concise manner and be absolutely certain that everything is adding up properly is not easy.

From my perspective, the fact that these calculations do not perform correctly right out of the box using dynamic languages is the problem, and adding additional libraries is not the solution.

<comment><USER6> The last project we did with money, we just treated all amounts as integers storing the number of cents. No accuracy problems w/ whole integers, so 1000 because $10.00. Easy to do math with, and keeps it's accuracy.

Might not work for hard-core financial work, since the number of decimals might be too deep for that to be efficient. (although there's nothing stopping you from saying the last 5 digits of every number is the decimal part).

<comment><USER7> A better thing to do is to not use raw integers to represent money at all, create an immutable money object, use integers to represent units and a currency string (cents/USD) inside the class. Use the money as the primitive it is, giving it the appropriate numerical operations added/subtracted/multiplied/allocated. Money is not an integer, or a decimal, or a double, or any other number, Money is Money and should be represented by an appropriate abstraction of its own.

<comment><USER8> > A better thing to do is to not use raw integers to represent money at > all...Money is Money and should be represented by an appropriate abstraction > of its own.

Good point. This makes me wonder though whether it is _ever_ acceptable to use primitive math without some kind of units, except maybe in pure math. You are always counting _something_ , be it dollars, or people, or widgets sold, or hard drive seeks.

<comment><USER7> I would depend on whether adding different units together was an error. 1 dollar + 1 euro should be an error. So raw numbers for pure counting is OK, but raw numbers for counting units of a specific measure are not OK, an abstraction should be introduced. If you just willy nilly add feet and meters you might end up missing your desired orbit and losing a billion dollar satellite.

<comment><USER6> I agree, and while the project we were on didn't have strong typing of money, what I described was the database storage of the numbers. We did very few calculations, and the system was explicitly USD, so going deep into "proper" handling of money was just unnecessary.

I really do like using the type system to keep track of units (type system either in the object oriented sense, or the haskell type system). This is similar to Joel's old article about using a naming scheme for dirty vs. sanitized strings for the web. It's all a type system, just his was in a language that didn't support language-level types.</comment></comment></comment></comment></comment></comment></comment></comment>


Astrobiologists discover fossils in meteorite fragments - <AUTHOR> http://www.extremetech.com/extreme/150417-astrobiologists-discover-fossils-in-meteorite-fragments-confirming-extraterrestrial-life <comment><USER2> From the frontpage:

<http://news.ycombinator.com/item?id=5356483></comment>


Era hatches Meething, an open source browser based video conferencing system - <AUTHOR> https://www.zdnet.com/article/era-hatches-meething-an-open-source-browser-based-video-conferencing-system/ <comment><USER2> [https://github.com/meething/meething](https://github.com/meething/meething)</comment>


Finland: Typing takes over as handwriting lessons end - <AUTHOR> http://www.bbc.com/news/blogs-news-from-elsewhere-30146160 <comment><USER2> Honest question, how common are typing lessons in Europe? Is it uncommon in some countries? Because I've worked and studied in France and Germany with people from all over and was surprised at how uncommon typing skills seem to be. Many people even told me they don't think it's a useful skill.

I had typing courses in elementary school (where they would bring in stand- alone typing keyboards) and in high school. But maybe it's uncommon in the US as well and I was just lucky. I also did have a year or two of cursive writing.

<comment><USER3> I had a typing class in my US high school, so I doubt it is uncommon (though this was in the early 90s on PC JRs).

<comment><USER4> We had typing in middle-school in Mexico in the early 90's, but it was on typewriters (Olivetti's that we had to lug to school)

It has proven invaluable as a software engineer. I still prefer to take handwritten notes, but my handwriting has gotten progressively worse - use it or lose it.

</comment></comment></comment><comment><USER5> I had exactly the same thought when I learned my daughter would be learning cursive in 2nd grade. What an incredible waste of time - who writes cursive at all any more? Compare that question to "who uses a keyboard to type?" And I think you have the answer to handwriting vs. typing.

<comment><USER6> Just guessing but I think learning cursive improves fine motor skill which might be more important than writing cursive.

I personally prefer teaching typing over handwriting.

</comment></comment><comment><USER7> Is this only about cursive handwriting? Because I'm baffled how people would find handwriting a useless skill. I find taking note by hand much more easy than using a keyboard (maybe if there was a quality note-taking software around, but using some text editor or even Word is a big no-no).

Now of course I often wish I could grep through my handwritten notes, but there some solutions for that (OCR pens).

<comment> chestnut-tree I'm baffled too. Language is central to culture. If you can't pick up a pencil or pen and contruct the letter forms that make up the alphabet reasonably fluently because you weren't taught properly, isn't that a bit depressing? Writing can encourage a love of lettering and letter forms. Sure, you can take this up in later years but it's better to learn from an early age. And as another poster says, learning to control a pen with precision is a valuable motor skill. Anyway, it's not clear in the article if this is just about cursive handwriting or all handwriting.

</comment></comment><comment><USER8> About time. I have been making this argument for 10 years and being met with mostly people who don't agree.

For whatever reason, most people think we need to keep doing whatever we have been doing even though the situation is manifestly changed.

As another poster points out, cursive in particular is pretty much a waste of valuable learning time at this point IMHOP.</comment>


Launch HN: DevOps as a Service - <AUTHOR> https://startopsgroup.com <comment><USER2> How is this different from any other up to date DevOps contracting/outsourcing shop?</comment>


Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin? - <AUTHOR> I am wondering how do experienced sysadmin document and manage their infra. <comment><USER3> We use Collins ([https://tumblr.github.io/collins/](https://tumblr.github.io/collins/)) as a Configuration Management Database, Ansible ([https://www.ansible.com/](https://www.ansible.com/)) for automation, Terraform ([https://www.terraform.io/](https://www.terraform.io/)) + a bunch of homebrew for orchestration, Packet ([https://www.packer.io/](https://www.packer.io/)) for multi-cloud (and hypervisor) image creation and maintenance, powered by Ansible. Every since thing is committed to a series of bitbucket ([https://www.bitbucket.org](https://www.bitbucket.org)) repositories.

We connect Ansible and Collins through ansible-cmdb ([https://github.com/fboender/ansible- cmdb](https://github.com/fboender/ansible-cmdb)), then tie the entire thing to our ticketing systems ServiceNOW ([https://www.servicenow.com/](https://www.servicenow.com/)) and Jira Service Desk ([https://www.atlassian.com/software/jira/service- desk](https://www.atlassian.com/software/jira/service-desk)), and finally, ensure we have history tracking with Slack ([https://www.slack.com](https://www.slack.com)).

As a given, we yank test the entire world. If it doesn't pass a yank, it straight up doesn't exist.

Whether it's bare-metal, virtualized, para-virtualized, dockerized, mixed- mode, or cloud - we 100% do this all the time. There is not a single change across any environment, that isn't fully tracked, fully reproducible, fully auditable, and fully automated.

<comment><USER4> what do you mean by "passing a yank test"? i assume "yank test" refers to unplugging the network cable abruptly from the server under test, but what exactly are you looking for when you do that?

<comment><USER3> A yank test on process and infrastructure is more than a 'did it come up'. It's a "if we totally nuke the thing" \- say, were we to rip the hard drives out of a server, fry it, and recreate it - does it come up identicall(is).

That way we know our CMDB is accurate, our workflows are accurate, credentials, ansible, terraform, images, etc. Right down to tickets.

It's how we manage all of our cloud customers.

</comment></comment></comment><comment><USER5> When I was working as a sysadmin, I kept a spreadsheet. I was told later of a repository of information that supposedly was what my spreadsheet did, but it didn't add anything new and was much harder to keep up.

I built it up using nmap and then shelling into each individual machine and poking around to see what it did. This was back in the days before everything became virtualized, so each machine on the network was likely physical.

I added information by walking the aisles and copying down the rack location of every machine into another page on the spreadsheet. I eventually hooked up a terminal to them all and matched network addresses to physical machines.

Only took a few weeks and when I was done, I knew things about the network that guys who worked at the business for years didn't know.

There's no substitute for the good old-fashioned way.

I liked that job, it was fun.

<comment><USER6> If you're by yourself, using spreadsheets and nmap is usually fine. If you're working in a team of 5 or 10 or 50 sysadmins, spreadsheets turn into a huge mess. You either have to distribute them via mail etc. after every change, but then you will have concurrent edits that need to be merged manually. Or you put the spreadsheets on a network share with file locking, but then it will always be locked when you want to edit it because someone is working on an entirely unrelated part of the infrastructure.

So you have exactly those sorts of problems that RDBMS are designed to solve. Therefore it makes sense to move to a DCIM system using an RDBMS under the hood, that allows for concurrent edits, and also can be accessed by automation (cronjobs, CI, etc.) via some sort of API (or direct DB read access).

<comment><USER7> There is an even better alternative. You can put infrastructure information into the same version control repository where your infrastructure code lives, and you can even keep all the benefits of spreadsheets by using plain text format spreadsheets like Org-mode tables.

This means you do not have two sources of truth to maintain (what is in the RDBMS, and how that relates to what is in the infrastructure code repository), the RDBMS system does not have to reinvent versioning, you can see exactly how your infrastructure evolves, you can do atomic changes to both the infrastructure code and the infrastructure information that the code relies on (obviously you need a modern version control system for this), and the infrastructure code can access the infrastructure information in a much more straightforward (and much easier to test) way.

<comment><USER8> This would become very exhausting if working with very large infrastructures. 80 000 virtual and physical servers? Have fun keeping that data consistent, up to date and available with Org-mode and version control.

I'm not saying your example is wrong, but "there is an even better alternative" doesn't always apply. For smaller scales, sure.

<comment><USER7> VMs need to be kept track of in whatever system you use for provisioning (AWS, OpenStack), otherwise you now have three sources of truth: what the configuration says should be running, what the DCIM thinks is running, and what is actually running.

</comment></comment></comment></comment></comment><comment><USER6> \- keep inventory in a DCIM (we use Netbox)

\- configure _everything_ as code (we use Ansible for the infrastructure up to OS level, Kubernetes w/ Helm for applications), have it read the values from the DCIM so that the DCIM remains the single source of truth (we need to still get better on this part....)

Links: [https://github.com/digitalocean/netbox](https://github.com/digitalocean/netbox) [https://www.ansible.com](https://www.ansible.com) [https://www.kubernetes.io](https://www.kubernetes.io)

That's at work. At home, I do much of the same, except that maintaining a DCIM is excessive for 2 VPS and a home network of 3 boxes.

<comment><USER9> I cannot comment on the DCIM side, but I agree on the "everything as code" mantra.

For a relatively small setup I chose a combination of Ansible, Kubernetes and Dockerfiles, but probably any combination will do. All these files are stored in a git repo.

Even after months (or years) neglect, I can easily know what I configured (and why!) and update where needed with a minor effort.

</comment></comment><comment><USER10> I'm going to mostly disagree with everyone here, much to my karma's detriment ;P

I agree the end-goal should be infrastructure as code, and everyone here has covered those tools well. You also want monitoring across your infrastructure. Prometheus is the new poster-boy here, but the Nagios family, and many other decent OSS solutions exist as well.

But you still need documentation. Your documentation should exist wherever you spend most of your time. Some examples:

* If you spend most of your time on a Windows Desktop, doing windows admin type things, then OneNote or some other GUI note-taking/document program makes sense.

* If you spend most of your time in Unix land(linux, BSD, etc) then plain text files on some shared disk somewhere for everyone to get to, makes WAY more sense. Bonus if you put these files in a VCS, and treat it like code, and super bonus if your documentation is just a part of your Infra as code repositories.

* If you spend your time in a web browser, then use a Wiki, like MediaWiki, wikiwiki, etc.

In other words, put your documentation tools right alongside your normal workflow, so you have a decent chance of actually using it, keeping it up to date, and having others on your team(s) also use it.

We put our docs in the repo's right alongside the code that manages the infrastructure.. in plain text. It's versioned. We don't publish it anywhere, it's just in the repo, but then we spend most of our time in editors messing in that repo.

<comment><USER11> I totally agree, but having "infrastructure as code" means less documentation.

Instead of documenting all the commands involved in configuring a machine as service X (ssh, run apt-get, paste this, etc.), I have documentation on how work with the configuration management system (roles in the roles/ directory, each node gets one role, commit to git, open PR, etc.). That documentation is in .md files in the config management source repo.

Instead of documenting how to rack a server (print and attach label to front and back, plug power into separate PDUs, enter PDU ports into management database, etc.), I document Terraform conventions (use module foo, name it xxx-yyy, tag with zzz, etc.).

It ends up being less documentation, as the "code" serves to document the steps taken, so the documentation can be higher level. Or if it isn't less documentation, it is documentation that needs to be updated less often, so hopefully there will be less drift between docs and what actually exists.

<comment><USER12> Ah the good old "self-explanatory code that needs no documentation".

<comment><USER13> More like code usually required extra documentation explaining it in a higher level language, but nowadays we just write the program on that higher level language so this extra documentation has gone away.

</comment></comment></comment></comment><comment><USER11> It might be helpful if described your infrastructure. There is a pretty big difference between managing physical Windows servers in a data center and managing Linux servers all in AWS.

If you are all or mostly cloud, Terraform + config management with a CI pipeline takes care of a lot. Then a wiki that covers "Getting Started" and a few how-to articles.

For physical infra you need the setup for DHCP, updating DNS based on DHCP, PXE boot imaging, IPMI access and configuration, switch and router configuration, what servers are connected to which switch ports, PDU management and monitoring, and on and on and on.

You end up with something like NetBox ([https://github.com/digitalocean/netbox](https://github.com/digitalocean/netbox)) or Collins ([https://tumblr.github.io/collins/](https://tumblr.github.io/collins/)), plus a bunch of other stuff gluing things together.

<comment><USER14> For future work, I would definitely consider NetBox and Collins as alternative options to GLPI.

</comment></comment><comment><USER15> I think it depends a lot on the size of your infrastructure. I've used excel docs on a shared drive pretty successfully where there's not much to keep up on and changes are few.

In larger infrastructure setups (small service provider) we used a combination of netboot, SNMP for monitoring with Observium and Nagios for alerting. We were also a big VMware environment, so naturally we had a lot of inventory tracking available through vCenter as well. I found a lot of opposition to Configuration Management, given the lack of comfort with programming of some sysadmins (Windows admins), so that's something to keep in mind as well. I think mixed environments also can be challenging w/infrastructure as code, but I'd be interested to see how others get through that.

</comment><comment><USER16> The past decade has been interesting and I'm still processing it.

My current thoughts are that an appropriate approach is for your systems to document themselves via the applications that they run - inside out.

Though I must abide I cannot fully subscribe to "infrastructure as code" anymore. It has proven just another shift, primarily in toolsets and who (or what) gets say and sway over the capacity, capabilities and efficiencies of the thing you actually care about - the app stack and all of its assembled functionality.

In other words most approaches are still "outside in" \- one defines 'x' for deploy fitments and that typically over and over and over again and, typically, with a rigidity that can too easily override and overrule effectively caging your application in scale and scope. With my current tact I am trying to provide for 'y' to "self identify" (via some/any form of config mgmt) where from here you can begin to effectively "deploy to any" by hooking the "application config as code" that, in turn, defines its infrastructure and deploys "outward". The "infrastructure as code" then becomes the servant with its objects and platform definitions etc. and the "appconfig as code" becomes the master where the latter defines its own scope and scale.

Infrastructures have a funny way of mutating into inefficient "definitions" of something that once made sense, on the first day, and forevermore complicating progress with capacity, rules and opinions.

But, generically, snmp is still pretty cool for telling me what I need to know. Strapped that into any end engine and, boom, ask any question, request any inventory.

So.. I track apps, not systems. Systems are expendable, applications are not.

</comment><comment><USER17> I don't do devOps but if I did...

[http://howardism.org/Technical/Emacs/literate- devops.html](http://howardism.org/Technical/Emacs/literate-devops.html)

[https://www.youtube.com/watch?v=dljNabciEGg](https://www.youtube.com/watch?v=dljNabciEGg)

</comment><comment><USER18> There are several classes of "infrastructure" as a sysadmin; legacy, new and critical.

Legacy stuff is done the old fashioned way - portscans and nmap. If it has an open port, it's presumed to be intentional. If not, it's a target. I've seen some success using tools like Pysa to "blueprint" existing systems into Puppet code. Tools like SystemImager help here, too - enabling P2V and the creation of "file-based images" compatible with version control and able to PXE boot new clones.

New stuff is from-scratch IaC all the way to the metal. Ansible and git submodules help me build "sandwiches".

Critical stuff blurs the lines. The machines, IP addresses, ports and living connectivity can be documented, and "captured" to a limited extent with the manual mapping and Rsync stuff in the Legacy category. Some of this critical stuff is also "new", and is deployed in that fashion.

What about switchgear and Cisco configs? License strings, key management, site-specific patching - all can complicate things.

More important than any of these is the ability for you and those around you to see and manage the systems as they are launched and terminated.

In the old days, I used to use a shell script on a newly-provisioned host to dump all its' details - dmidecode, environment stuff and so on. Those details were pushed back to a common source and were a real benefit in the days before _real_ config management came on the scene. CFEngine was way too complicated and nebulous at the time.

</comment><comment><USER19> For me/us, it's a combination of infrastructure-as-code and metrics reporting/logs. Most of our boxes are swapped out on a weekly or more frequent basis, so the only accurate picture of what's running right this moment is the graphs built by the metrics collection tools. The only accurate picture of what's running on those boxes is the code which built the infrastructure.

There are a couple of exceptions, but those are actively being brought under the above model (mostly because they are effectively invisible, and the existing documentation for them is... incomplete).

Any documentation outside of that is stale in a few hours, and obsolete in a week.

</comment><comment><USER20> Back when I was put in charge of IT Lifecycle management for my Army unit (not by choice - "Hey, you've got a CS degree, so anything tech related goes to you"), I kept it all in an Access Database, and ran off a report occasionally to update my smartbook (3-ring binder full of stuff that my boss would frequently ask about during meetings). Granted this was back in the early 00's.

</comment><comment><USER21> Terraform + Datadog + Cloudwatch

[http://terraform.io/](http://terraform.io/) [http://datadoghq.com/](http://datadoghq.com/) [https://aws.amazon.com/cloudwatch/](https://aws.amazon.com/cloudwatch/)

</comment><comment><USER22> As a professional sysadmin, my go to reference on this is "Documentation Writing for System Administrators", from the Short Topics in System Administration series.

[https://www.usenix.org/short-topics/documentation-writing- sy...](https://www.usenix.org/short-topics/documentation-writing-system- administrators)

Also, this talk was very good:

[https://www.usenix.org/legacy/event/lisa08/tech/gelb_talk.pd...](https://www.usenix.org/legacy/event/lisa08/tech/gelb_talk.pdf)

<comment><USER23> It's worth the $5, I assume?

</comment></comment><comment><USER24> I've used [https://www.racktables.org](https://www.racktables.org) with pretty good luck. It's PHP, which wouldn't be my first choice, but I've largely been able to make it do what I want.

If you want something more clever; say keeping track of asset values etc, you'll want a CMDB. Google around and you should find something that fits your needs. We used SeviceNow in a previous life.

</comment><comment><USER25> We put everything in code. We have several layers, but they if you're new you can start with the lowest level and make your way up to find out how things are provisioned and configured.

We're on AWS so we use cloudformation for provisioning and saltstack ([https://saltstack.com/](https://saltstack.com/)) for configuration management. Cloudformation templates are written using stacker ([http://stacker.readthedocs.io/en/stable/](http://stacker.readthedocs.io/en/stable/)). All AWS resources are built by running "stacker build" so nothing is done by hand. We have legacy resources that we're slowly moving over to Cloudformation, but more than 90% of our infrastructure is in code.

On top of cloudformation and salt we built jenkins (CI and docker image creations), spinnaker (deployment pipeline), and kubernetes (deployment target). The jenkins and spinnaker pipelines are also codified in their own respective git repos.

All the repos here have sphinx setup for documentation purposes and the repos tend to crosslink for references.

</comment><comment><USER26> I've found Zabbix works decently well and also covers monitoring. Zabbix Maps can be nice to visualize the infrastructure: [https://www.zabbix.com/documentation/3.4/manual/config/visua...](https://www.zabbix.com/documentation/3.4/manual/config/visualisation/maps/map)

</comment><comment><USER27> So, one problem I've seen with most infrastructure as code solutions and CMDBs is that they do a good job at the tactical level (more or less), and help you answer "how", "where", "what", and maybe "when" questions (depending on how well they support orchestration), they typically do a bad job at the higher level strategic "why" questions.

So, why do you structure your lambda jobs accessing CloudWatch Logs that way as opposed to the other way? If you didn't know that one way works and the other doesn't, you wouldn't be able to understand that question. And that might have domino effects on other parts of your system.

I haven't found a good solution to documenting the high level strategic "why" questions, other than to just write down the questions and the answer, with reasoning, in some form of associated documentation — maybe in a wiki or something. But, of course, the underlying issues may change in the near future and invalidate the reasons for your decision. And the high level documentation doesn't have any way to be compiled directly into the lower level implementation, so of course there is always the risk of drift.

I'm still looking for good solutions in this space.

</comment><comment><USER28> Vmware's tagging support is a lighter, more realistic option vs a "CMDB".

Come up with a key/value strategy that covers your need to track things like app name, app category, environment (test, dev, load testing, prod, prod/dmz, etc), and it becomes actually usable and up to date versus an always out-of- date CMDB. And it's compatible with cloud resource tagging.

Sometimes, less is more.

</comment><comment><USER29> Spinning up new infra: Jenkins crafts Terraform tfvars based on user input, runs plan, asks for confirmation, applies. Terraform state and vars saved to S3. Chef and Ansible for provisioning.

"Documentation", in terms of where stuff is deployed and what is deployed is not really necessary. We save this data to a DynamoDB table, query-able by AWS Lambda functions, so other automation can pick it up and devops can query data.

Documentation on how things work comes from dev teams, on how things are deployed indeed comes from us, just simple wiki pages.

Services running in Kubernetes, K8s worker instances in auto-scaling groups. If one node dies it is killed and brought up, K8s will reschedule the pods. Same for the pods themselves.

Monitoring through Nagios(getting phased out finally), NewRelic and Prometheus. Basic ELK stack for centralized logs.

Thinking about rolling out Vault for credential management. Chatops on the pipeline (getting pieces in place first, like the db mentioned earlier)

I'm trying to get the company on board on immutable infrastructure, but it is proving difficult.

</comment><comment><USER30> I use One Note.

But I also use the o365 Suite.

Mediawiki is also good, but can be a bore to run another service for that.

But in the end a textfile via notepad/nano is all you need, really.

<comment><USER31> OneNote over a text editor as you can drop in screen shots.

</comment></comment><comment><USER32> Like many here, I keep it described in ansible and documentation inside a git repository.

But I feel like it's lacking. After a while you have so many ansible playbooks and roles that they cannot give you a birds-eye view anymore.

I think I would MUCH prefer to have some sort of HTML representation, where adding an instance/service starts by adding to that representation, and you could click on every link or node to show its golden image setup, ansible configuration, etc.

THAT, I could show to a newcomer and he'd get it.

<comment><USER33> I'm no expert but doesn't Ansible Tower do that?

<comment><USER32> Ansible Tower lets you execute a playbook via a web GUI, and keeps a log of who executed what.

I'm not sure if it also shows some infrastructure graphs, but I'm talking about knowing if links are up, how they are firewalled, where the config for each thing is, etc.

When you host tens of services on hundreds of machines, this information is hard to get a grasp on, no matter what you do or how well you documented everything, because it takes a while to read through it.

</comment></comment></comment><comment><USER34> By having your infrastructure defined in version control using some sort of domain specific language. For example, by using Terraform and only ever making changes to your infrastructure via Terraform (manual adding/editing of stuff in AWS/GCP console should be disabled so people can't do that). Then all changes to the infrastructure are clearly documented in version control with pull requests.

</comment><comment><USER28> Aligning a VMWARE tagging strategy with a cloud tagging strategy is one of my current goals. Things like a full blown CMDB seem to always end in pain, lag, and orphaned records. I'm happy enough with something basic that spans on-prem + cloud.

</comment><comment><USER35> I use:

\- [https://www.bookstackapp.com/](https://www.bookstackapp.com/) for portable (Markdown), searchable (SQL), manageable (Users) documentation.

\- Ansible for automation and deployment.

\- Prometheus for monitoring all the Proxmox nodes and containers.

</comment><comment><USER36> Can I piggyback and ask how people keep track of deployed software? Like if I have 50 products deployed some of which haven't been touched in 10 years and I want to be able to ramp up a developer to fix a bug on any of them?

<comment><USER37> In the medical device industry you keep what's called a device history file which tracks the configuration of each device you've sold by serial number. This DHF is meticulously updated whenever something is changed. If someone reports an issue this is information you can use to scope your initial reproduction.

<comment><USER38> I think you're mixing up DHF and DHR. Design history file, device history record.

</comment></comment></comment><comment><USER14> GLPI with FusionInventory for IT Asset Management and Knowledge Base.

GitLab for repositories, adhoc documentation via gists and CI/CD.

Nagios for monitoring.

Open to trying other things out if they make sense.

</comment><comment><USER39> Asset management systems and network inventory databases.

</comment><comment><USER40> Those using AWS ALB, how do you monitor your traffic in realtime? I want to aggregate host names, ip addresses, user agents in realtime.

</comment><comment><USER41> Lansweeper ([https://www.lansweeper.com/](https://www.lansweeper.com/))

</comment><comment><USER42> anyone have any pointers for simple an API driven managment of DNS/DHCP?

(like, I don't want to have to configure 1000 moving parts)

typically this seems to fall into the 'roll your own' or 'giant lumbering enterprise behemoth' category that does 10 other things. I'm looking for the sweet spot.

<comment><USER43> At any reasonable scale you typically wouldn't use plain DNS if you have to do that kind of figuration. It would be done with a service discovery service which handles SRV records.

That being said route53 has a reasonable management API.

<comment><USER42> thanks - should have mentioned specifically not looking at cloud services

(e.g. self hosted, but without needing 5 different polyglot microservices and a service managment layer and 32GB of ram just to keep the whole mess running)

I see 0% need for this complexity in many cases on the presentation side - <USER2> if faster response is required internally, the same API IF can be used for service discovery or side-chain announcements, etc can be bolted on on a per- application basis if desired.

I also see 0% need for this to be a cloud exclusive domain - e.g. hybrid scope/location deployments, etc.

</comment></comment></comment><comment><USER44> A configuration manager, Ansible for example. You basically describe your infrastructure with it.

</comment><comment><USER45> I deploy it with code. For hardware stuff, a CMDB, also maintained with code

</comment><comment><USER46> What about securely storing credentials and passwords?

</comment><comment><USER47> I'm curious how ChatOps practitioners handle this.

</comment><comment><USER48> I'm not sure I understand your question fully? You write documentation, like you do anything. And configure everything with code, so you can go read it (Terraform, Chef/Puppet/Ansible, etc).

<comment><USER49> OP is probably looking for someone to go a little further into detail on exactly what you just said.

<comment><USER50> I use SCC (System Configuration Collector) to document our servers. Everything else is just a collection of grep-able text files on our management server. [https://sourceforge.net/projects/sysconfcollect/](https://sourceforge.net/projects/sysconfcollect/)</comment></comment></comment>


The Moon's Giant-Collision Theory Proven Incorrect - <AUTHOR> http://techzwn.com/finding-shows-the-moon-comes-only-from-earth-not-giant-collision/

<comment><USER2> I'm being picky here, but the word "proven" doesn't really make sense in the context. A more accurate way of expressing the title would be "Evidence supports a different theory" or "Less support for this theory", as theories can never be proven.

<comment><USER3> If theories could be "proven" in the mathematical sense then they wouldn't be falsifiable and therefore it wouldn't be a "theory" in any scientific meaning of the word. So yes, I completely agree with your being picky about this!

<comment><USER4> > If theories could be "proven" in the mathematical sense then they wouldn't > be falsifiable

Ack! This is _not_ what "falsifiable" means.

Falsifiability is the property that states, there exists some hypothetical evidence which would be accepted as conclusive disproof of the theory. For instance, newton's third law ("equal and opposite") can be falsified by the discovery of a method of applying force which avoids any opposite application.

Extrapolating some, we can see that for all falsifiable theories, there is an extensive class of evidence which doesn't quite prove the theory wrong, but does weigh against the theory. Non-falsifiable theories don't have this property - for any method of assigning a probability of correctness to a non- falsifiable theory, there is a positive number below which that probability cannot go.

The reason the giant-collision theory should not be said to be "proven" incorrect is that it has not been. Its probability of correctness is still non-zero (and significant). However, it's not true that scientific theories cannot be "proven" incorrect - classical mechanics has been, for instance.

<comment><USER3> In what sense can a scientific theory be "proven" correct of incorrect in the same way that a purely mathematical statement can? Aren't they two completely different things - with the mathematical argument being binary (it's either true or false) and the scientific argument, as you say, involving degrees of confidence based upon supporting or contradictory evidence?

[Of course, a scientific theory could be disproven by being based on incorrect mathematical reasoning - but that's presumably not very interesting]

</comment></comment></comment></comment><comment><USER5> My understanding of the theory is that Theia hit the earth, that most of it was absorbed into the earth's surface/ molten core, and that the rest, along with some of the earth was ejected out into space.

If that's the case I don't see how there is any way of 'proving' that the isotopic signatures we are matching with those of the moon are 'native' and not already part of some Earth/Theia mix - in which case they'd match with a Moon that was already part of an Earth/Theia mix. Without knowing the composition of the earth 4.5 billion years ago, I'm not sure you can say otherwise?

<comment><USER6> If the giant impact theory is correct, Theia must have been about the same size as Mars - roughly a tenth of the mass of the earth. Hence it would have a much larger impact on the makeup of the moon than it would on the makeup of the earth - a difference we would expect to be able to detect if the theory is correct.

<comment><USER7> It would have a negligible impact on the makeup of the Earth _as a whole_ , but should have a large impact on the composition of Earth's _crust_ (and the mantle, to a lesser extent.) It seems to me that the Theia theory should also predict that Earth's crust and Moon's crust (which are the only parts we can get samples of!) have quite similar composition.

</comment></comment></comment><comment><USER8> I don't understand. If "theia" hit earth and formed moon, why would the istopic signatures be any different. Now both earth and moon come from same pair of different objects.

<comment><USER9> We would expect different proportions of the isotopes on the Earth and the Moon. This is because the Moon according to the models would have a greater proportion of its mass made up of the theorised collider "Theia" than Earth.

In this particular experiment they showed the isotopic proportions matched between here and the Moon.

</comment></comment><comment><USER10> I think we should remember the vast differences between reporting about scientific papers and the peer-reviewed papers themselves; I have a good feeling, based on previous experience, that most of the caveats raised here will be tackled by the paper, whose tone will probably be less decisive.

<comment><USER11> Indeed. The actual title of the paper is "The proto-Earth as a significant source of lunar material."

</comment></comment><comment><USER12> One thing I've never understood about this model: according to orbital mechanics, you can't "boost" yourself into an orbit that doesn't include the point you're already in. So if the mass that became the moon started at the surface of the earth, its orbit would just smash it back into the earth again. You can see what I mean here: <http://isthis4real.com/orbit.xml>

<comment><USER13> In a 2-body system this is true, but adding many more bodies to the system makes things very chaotic. An object passing by or colliding with the "proto- moon" could easily deflect it from its crash course. Whether that's probable given the evidence is a different question, but mathematically it absolutely is possible.

And that's even assuming that the "mass" is more or less solid - if it were ejected from a violent impact it's more likely that it would be more of a cloud of debris, in which case things get even more complex. So many "bodies" are involved that it becomes more of a fluid dynamics problem than an orbital mechanics one. The individual particles in the cloud interact with each other gravitationally, deflecting each other from their "return-to-sender" orbit. Parts of the cloud would fall to earth, parts would escape to space, and parts would stay in orbit to become a ring system or (as is hypothesized) one or more moons.

The theorized situation isn't so much like skipping a stone across the earth as totally vaporizing a big chunk of both the earth and the planet that hit it, and a fairly small part of the resulting cloud (about half according to some models) of debris managing to end up in orbit and eventually collapsing into a single body (the moon).

</comment></comment><comment><USER14> Strictly identical? Maybe the samples they're using are not really from the moon in the first place! Hehehe...

<comment><USER15> Or that the mix of Earth and Theia in the moon is very non homogenous and they took most of their samples from the Earths part.

Actually, now that I think about it, wouldn't the result of the collision cause most of Earths contribution to be on the surface of the moon? If it was otherwise, then Earth would have had an ring much like Saturn at one point which then collected into whole to become the moon. That couldn't be true since we would have easily seen evidence on the moon. This theory of mine would mean that almost any samples taken from the moon would would be originally from Earth.

<comment><USER6> The giant impact theory predicts that Earth _did_ , at one point, have a proto-ring of debris from the collision. That proto-ring slowly accreted into the moon.

I'm not sure what evidence we would expect to observe on the moon to indicate whether it accreted from a proto-ring or was captured intact - could you elaborate a little?

</comment></comment></comment><comment><USER16> The theory was merely proven to be theoretically falsifiable, and not incorrect. They clearly state in the article that they couldn't find any proof of two parents, which for the moment points to the moon having only one parent. This contradicts the previous theory, but doesn't make it false.

</comment><comment><USER17> So what are some alternative theories that now look more likely in light of this evidence?</comment>


Fukushima: 28 Signs That The West Coast Is Being Fried With Nuclear Radiation - <AUTHOR> http://ukiahcommunityblog.wordpress.com/2013/10/22/fukushima-28-signs-that-the-west-coast-is-being-absolutely-fried-with-nuclear-radiation/ <comment><USER2> I'm not even going to apologize for casting a dash of cold water on this scary article. The fact is, if you wanted to find out how much radiation is reaching the West Coast, you could test for it. Easily. Nobody has done that. Why?

I'm going to be taking a nap until I see (on this science and technology site) that some real Scientists have conducted some real scientific tests and have recovered some real scientific data, that leads to some real scientific description of the problem -- in Scientific terms. Really. Wake me when you do the job.

</comment><comment><USER3> That's a great post. I'll be sure to take a look at all the articles that are linked in it!</comment>