Toyota's Takaoka #2 Line: The Most Flexible Line in the World - <AUTHOR> http://www.thedrive.com/tech/26955/inside-toyotas-takaoka-2-line-the-most-flexible-line-in-the-world <comment><USER2> I don't doubt the line is much more efficient and flexible as claimed, but the article doesn't do a very good job explaining exactly how the gains are achieved beyond hand-waving about clever contraptions that are faster than robots and the ability to dynamically move stations around. It seems difficult to imagine that hordes of industrial engineers haven't done the calculations on each of these methods countless times at all car manufacturing companies. There must be deeper, more subtle factors that explain why Toyota's particular combination of techniques yields such superior results. Perhaps it's just the culmination of a lot of long-tail optimizations that only Toyota has had time and capital to let mature or maybe there really is just "one secret trick" that underpins it all - the article doesn't really give a good sense of what it might be. In either case it'd be interesting to see a detailed breakdown with numbers comparing the various lines with explanations for why particular decisions were or weren't made to truly understand why the system works so well.

<comment><USER3> it sounds like the benefits of the line is the flexibility, not necessarily efficiency gains. I think the idea of the article is that fixed production is a business weakness in today's fast moving consumer market.

This also doesn't sound like something Tesla needs yet, as this seems to be great for building different types of cars based on demand rather than maxing a single car output.

<comment><USER4> I thought this was basically what Tesla had to do with the add on tent manufacturing the 3's.

</comment></comment></comment><comment><USER5> I know that car analogies for software is an over-done metaphor but while reading this article I'm thinking of heavy processes for software deployment that are hard to change and in my mind I'm screaming "this is what agile looks like!"

Processes and controls are very important for quality, but as business goals shift and evolve, the processes also need to evolve. But designing processes that can evolve gracefully is not a trivial task, especially in risk-averse organizations.

<comment><USER6> > this is what Agile looks like

FYI Kanban came out of the process on Toyota assembly lines.

<comment><USER7> QR codes, too.

</comment></comment></comment><comment><USER8> What I would've loved to see is the comparative efficacy captured in motion for (lets say) a "layman's observation" of production. Just a glimpse into how it works out on the "flexible line"

<comment><USER9> This maybe: [https://www.youtube.com/watch?v=t0E66vaiXeg](https://www.youtube.com/watch?v=t0E66vaiXeg)

<comment><USER10> That's a very nice video of a production plant, but I don't think it's the one in question.

</comment></comment></comment><comment><USER11> "They would tell you that robots are great for repetitive tasks like welding, or painting, but they fail at picking and handling small parts." Are robots really that bad nowadays? I imagine, it would be a problem in VGA-only camera years. Now we have tens of megapixels, lidars, tons of computing power (cheap!).

<comment><USER12> I think the bigger problem is in sensor precision, particularly those sensors responsible for reporting what state the arm in (i.e. so the computers onboard can calculate the position of the arm relative to the object it needs to pick up). Sensors with low tolerances are extremely expensive and those particular robot manufacturers may have opted for less accurate sensors to save on costs. They may work perfectly fine for moving things like doors and frames; but the differences make themselves known as soon as you try to interact with bolts and etc.

When I did robotics we had a perfectly capable Linux box running PID loops several hundred times a second; but our robot still sucked b/c our on board motion sensors were simply not precise enough to give us accurate data to feed the algorithms. As usual, it's all about the data.

<comment><USER13> As someone who works with modern industrial robots, like you'd find in a car factory, the sensors they're using for the arm positioning are unbelievably precise. I can have the robot stopped, with the brakes on, and push on the end of the arm with my fingers and see the joints registering the deviation (and not by 1 or 2 counts, but many more), while I don't perceive any significant movement of the robot.

</comment></comment></comment><comment><USER14> Takaoka Line 2, when Category Theory meets auto manufacturing. Love the carts they move cars on. Curious if Tesla attempts such a line next.

<comment><USER15> > Curious if Tesla attempts such a line next.

and how exactly would they do that? their non-automated lines barely work.

<comment><USER16> That's FUD BS. You don't produce nearly a quarter of a million cars if your production line "barely works". The truth is that they're now routinely exceeding a thousand Model 3 cars per day.

[https://www.bloomberg.com/graphics/2018-tesla- tracker/](https://www.bloomberg.com/graphics/2018-tesla-tracker/)

<comment><USER15> they have a history of manufacturing problems, do they not? they may not be having them at this very moment, but it is well documented and discussed that they have had issues. i think when you're building cars in a makeshift, outdoor assembly line just a year ago, you can't say that you have everything together. maybe they've hit their numbers, but their manufacturing process hasn't been elegant. is this incorrect? and this is a personal opinion, i think founded in some objective reality, but i don't believe teslas to be well-manufactured or designed from a non-owner's perspective. i have sat in them and ridden in one, and it just is not an impressive design and fit and finish to me compared to more entrenched car companies. this is both a design and manufacturing issue, from my understanding.

the point of my comment, which i thought was clear, was a counter to the continual tesla hype in that "can't wait until tesla addresses <this> problem" as if they've conquered all problems that have come before. toyota has it together and has a significant amount of r&d that goes into their advanced assembly lines. it's a bit laughable to think of tesla suddenly taking them on in automation given their historical troubles. or am i wrong?

also, those numbers are fine, but tesla isn't even the largest electric car manufacturer in the world, by sales numbers. and they don't approach traditional car makers. for example, kia sells around 600,000 cars in a single year, and that is including say three times as many models as tesla has available. tesla isn't some manufacturing darling story as far as i can tell. they have no ability to yearly iterate like other car companies. or do they? how do they update their models?

i think the point remains: it is flippant to suggest they can just suddenly innovate on an automated line.

</comment></comment></comment></comment><comment><USER17> > "After pulling a few strings to get into the Takaoka plant, you will see the > Gordian knot become untied."

Um... the Gordian knot was not untied, but rather cut by a sword. It's supposed to be a bold, out-of-the-box solution to a seemingly impossible problem. I don't know if the author of the article chose the wrong word on purpose.</comment>


Bug-free software, possible? - <AUTHOR> http://timesofindia.indiatimes.com/HealthSci/Bug-free_software_possible/articleshow/3254323.cms

<comment><USER2> You can certainly approach it. The problem is, you can't tell for sure when you're there, unless a program has so few execution paths that you can check them by enumeration the way you'd check a proof.

<comment><USER3> Are you familiar with techniques used in avionics software? The methods just ooze with pain for the developer. Examples include no dynamic memory allocation and always running network operations at full tilt so there is no notion of average & peak network usage. For the latter, let's say you want to make a messaging system where module 1 periodically sends a message to module 2.

Rather than just sending a message when it is needed, you determine the maximum frequency you want to send a message, and send the latest message again and again. More deterministic, more verifiable correct, but 100% pain.

</comment></comment><comment><USER4> I don't have high hopes about the possibility of perfectly bug-free software. To be bug-free from an end-user's standpoint, it seems like such a thing would need to be completely free of errors _at every level_ , from specification (understanding precisely what the program is supposed to do, whether in terms of problem domain or customers' requests) to implementation (the bug-free program itself) to the supporting system and hardware (an otherwise perfect program could still fail due to a bug in a floating point processor, for example). Each layer presents its own problems, many of which are far from trivial.

Perhaps a more useful way to frame the question is to ask what approaches could better rule out different kinds of bugs. For example, the strong type system in ML languages (e.g. OCaml) is great for preventing bugs with extensive compile-time analysis (in part due to being derived from a theorem prover), but there is certainly room for improvement. Likewise, well-designed garbage collectors can rule out many kinds of memory leaks.</comment>


Instacart in Talks to Raise $400M at $3B Valuation - <AUTHOR> https://www.bloomberg.com/news/articles/2017-03-01/instacart-in-talks-to-raise-about-400-million <comment><USER2> I feel like I'm taking crazy pills. Every supermarket in Montrela offers phone orders and home delivery for a fee of less than $5 if your order is big enough ($50+). If you want to, you can even do your shopping in the store, and pay about two bucks to have them deliver to your house if it was in a few km.

I just checked, and one of the smaller chains (4 stores) has their own online ordering platform.

<comment><USER3> Their revenue numbers must look absolutely insane, even if they're not actually profitable. I spend more money through instacart right now than any other software, and maybe all other software I use, combined.

<comment><USER4> Not appropriate to count entire cart/shopping spend as Instacart's revenue. Think of that as "GMV". Instacart's revenue is their delivery charges, service charges, and any money the stores pay them / % fees.

See #6 here [http://a16z.com/2015/08/21/16-metrics/](http://a16z.com/2015/08/21/16-metrics/)

<comment><USER3> Great article, thanks.

<comment><USER4> :)

</comment></comment></comment></comment></comment><comment><USER5> Some context, aside from the viability of Instacart itself.

Apoorva Mehta used to work down the hall from us at SomaCentral, a shared office space retrofitted out of an old law firm on Townsend. He and his buddy would be heads down in their laptops every time we walk past their bullpen corner.

They were busy building out a legal tool slash social network slash law something slash annotation something slash whatever. The plan didn't so much change but rather morph every time I said hi.

They were smart though, so I figured they would hit it whenever they get to it. Though they've been at this for awhile now.

While most others tried to keep a social factor going on at the space, they tend to be so busy they would avoid it. Though and now then I do remember getting to grab a drink with them.

A few years later, settling in my new job in Berkeley/Oakland I called up Instacart to deliver me some groceries. But it was only last year I finally looked up the company profile and soon realized it's the same Apoorva Mehta.

From being that dude to 3 billion dollars in a matter of five years is--to quote another commenter-- __insane __.

But he's keeping the silicon valley dream alive.

That's pretty nifty in itself.

The rest of us are just hanging out here writing comments.

</comment><comment><USER6> I was going to post a comment about Instacart here, but instead decided to write a little thing on Medium.

Why Instacart is gonna get crushed by Amazon

[https://medium.com/@a13n/why-instacart-is-gonna-get- crushed-...](https://medium.com/@a13n/why-instacart-is-gonna-get-crushed-by- amazon-48f08bad6e35#.a1jcnmu4g)

<comment><USER7> I think you nailed it. I live in NYC and work too much, meaning I live and die by trading money for time. And every time I use instacart I regret it for the exact same reasons you describe.

The idea of hooking into some kind of warehouse management network makes a lot of sense. Does an API exist? Probably not. But there's an opportunity right there. Partner with a major warehouse management system, preferably one that works with Safeway or Stop n' Shop, and figure out a way to cut out the store visits altogether.

Of course this would mean automating away the shoppers :/.

<comment> jon-wood Ocado in the UK did more or less this. They started out partnering with a large supermarket chain and then started building out their own automated warehouses. The big chains all offer online ordering and delivery, at times with scale that makes it worthwhile for them to run dark stores which are closed to the public and only fulfil online orders. At the other end of the spectrum there are companies like the one I work for offering online ordering and delivery services to smaller independent shops, we provide the website and logistics and they use existing shop staff to pack orders when the shop is quiet.

</comment></comment></comment><comment><USER8> My theory is that the inability of US chains to figure out online ordering is really some anti-competitive agreement to force customers to go into the store because the chains live and die by high-margin impulse buys.

Wholefoods just opened a brand new chain called '365' that has all sorts of sophisticated nonsense like a programmatic tea-bot and yet there is no online ordering. Even the made to order stuff is driven by an interface which could just be made publicly accessible and they'd be done but no, into the store you go.

<comment><USER9> I don't think so.

Supermarkets and e-commerce are entirely different competencies and I have no trouble believing that being good at one doesn't give you a tonne of advantage at the other.

Putting some startup's tea-bot into a store is far easier than integrating that tea-bot into an online ordering system.

<comment><USER10> Most supermarkets in other countries seem to manage deliveries and online ordering pretty well.

For example, all supermarkets in South Africa provide online ordering and delivery. Sometimes without even a delivery charge.

</comment></comment></comment><comment><USER11> Low barrier to entry, crowded space, and low margins...can't imagine this company (or any of the small guys) still being around 5 years from now.

</comment><comment><USER12> I see Uber, Lyft, Amazon Fresh, Instacart, etc and I can't understand why they take the market by storm like that. They are not the first to ever think about that, their service isn't super great (except for Amazon, I'd say) and still, in the US, it seems like the market just sits and waits for them. Then a majority of people jumps in and the thing skyrockets.

Meanwhile, in other countries, small to medium-sized companies have had online groceries ordering for many years, local taxi unions have had websites and recently apps, and still... it's like OMG so much innovation from US companies let's give them crazy valuations.

What gives? Is it market size?

<comment><USER13> I'm not on the Amazon Fresh/Instacart trend yet (and may not be -- I really enjoy grocery shopping and frequent different farmers' markets, so it's not a problem I need to have solved for me).

But as for Uber/Lyft, the reason they're vastly more valuable than whatever local taxi website/app is convenience and lack of friction. When you get an Uber, you don't have to worry about payment or directions; it's all handled for you. When you get to a different city, you don't have to google the local taxi companies and pick one; you just open Lyft and get a ride.

</comment></comment><comment><USER14> Just ended by Instacart subscription for Amazon Fresh. I hated how Instacart would mess with prices. So far, much happier with Amazon Fresh. Costco access gives Instacart a huge advantage, but they abuse it.

</comment><comment><USER15> I guess jumping on the "Uber for X" bandwagon nets you 3Bn.

<comment><USER16> There hasn't been any net yet.

<comment><USER15> True. But I'd assume at the very least it nets you a fat salary befitting a founder of a 3bn company.

<comment><USER17> That's not how salaries work at the startups I've been a part of.

</comment></comment></comment></comment><comment><USER17> As a comparison, the Whole Foods supermarket chain has a market cap of $9.6G. Yow.

<comment><USER18> Yep, and Safeway with over 1500 stores across the US/Mexico has a market cap $8.1B.

<comment><USER16> Instacart is building datasets right now on what people purchase. They can simply bypass their physical market place partners of today and become a retailer without the cost of running physical stores. They would only need to take care of fulfillment, and hey, maybe with machine vision there will be a little Instacart robot that delivers your groceries to you direct from the Instacart warehouse: cutting yet another cost and justifying their high valuation. Who know's if that's what they're up to.

<comment><USER19> > Instacart is building datasets right now on what people purchase.

Why do people say things like this as if Silicon Valley is the only place that's heard of collecting data on customers. You are aware that's what those supermarket savings cards with the barcodes do right?

I guarantee you that Instacart has like the 178th best database of what supermarket customers purchase at best.

<comment><USER16> Of course, I'm only painting a scenario that I think likely is being posed by Instacart to justify their valuation and future prospect, not the superiority or effieciency thereof.

And there is a big difference between having access to a database of customer sales, and actually having those customers...

</comment></comment></comment></comment></comment><comment><USER20> One interesting counter to Instacart are services like grocerkey ([http://www.grocerkey.com/](http://www.grocerkey.com/)) that exist as whitelabel instacart's for stores. That local chain that everyone knows probably doesn't want to send all their customers to instacart if they don't have to. I'm not sold on Instacart being something people use outside of a few major cities, but maybe that's enough for them.

</comment><comment><USER21> Supermarkets can compete by becoming more like Hyvee with restaurants and other amenities. But I agree delivery is amazing.

</comment><comment><USER22> I kind of wonder why this was even leaked. Is anyone going to give Instacart a better deal than that? Or acquire it?

</comment><comment><USER23> That is insane and unreasonable. The bubble has lost it.

<comment><USER24> Why? Grocery shopping is extremely inefficient and has to be done weekly by almost everyone in the world. It's not a small problem that they're solving, so large amounts of cash (and a high valuation) makes sense.

<comment><USER15> One doesn't deserve a high valuation just because the problem they are solving is hard.

The company he came from, Amazon, has more cash than anyone could raise. They have logistical expertise and are perfecting same day shipping at a national scale.

They also offer Amazon "Fresh" which is essentially the same service except they don't skirt labor laws by classifying their employees as contractors.

And non-tech people actually trust Amazon and order quite a bit from them.

For them to succeed outside of a few big cities would require gross negligence and incompetence by Amazon, which isn't impossible. But seems unlikely.

<comment><USER25> > They also offer Amazon "Fresh" which is essentially the same service except > they don't skirt labor laws by classifying their employees as contractors.

At least in DC, the Amazon Fresh (and Prime Now) drivers are definitely just random people in their own car, the same as Instacart.</comment></comment></comment></comment>


Ask HN: Are LineageOS security updates just as secure as OEM's? - <AUTHOR> <comment><USER2> No, because half the vulnerabilities are in device drivers, which are unfortunately device specific binary blobs. Lineage OS has no choice but ship them as is, so they'll not be updated.</comment>


Telescope Time Without Tears – Borda Count Peer Review - <AUTHOR> https://www.youtube.com/watch?v=7c0CoXFApnM

<comment><AUTHOR> Paper describing the proposal for Telescope time-allocation:

[http://arxiv.org/pdf/0906.1943.pdf](http://arxiv.org/pdf/0906.1943.pdf)</comment>


Dave Prosser's C Preprocessing Algorithm (2006) - <AUTHOR> https://www.spinellis.gr/blog/20060626/ <comment><USER2> For anyone interested in examples of software where true preprocessor standard conformance is essential, check out [https://github.com/rofl0r/order- pp](https://github.com/rofl0r/order-pp). It's a functional programming language built on the C preprocessor that can essentially output any sequence of preprocessing tokens, with high-level language features like closures, lexical scoping, first-class functions, reflection, an eval primitive, etc. It also provides data structures like sequences and tuples with many functions for operating on them, and has arbitrary-precision arithmetic support. Pretty neat stuff.

<comment><USER3> Interesting. Would you happen to have a link to some human readable documentation and/or examples? I couldn't make much sense of the grammar in [1] without usage examples.

[1] [https://github.com/rofl0r/order- pp/blob/master/doc/notes.txt](https://github.com/rofl0r/order- pp/blob/master/doc/notes.txt)

<comment><USER4> There are some annotated examples [1].

[1] [https://github.com/rofl0r/order- pp/tree/master/example](https://github.com/rofl0r/order- pp/tree/master/example)

</comment></comment></comment><comment><USER5> Notice that this operates on a lexer stream to transform a source file into the destination without any macros/macro-substitutions remaining. It's not the same lexer as that in the C compiler, which doesn't know anything about macros.

Also, on the C compiler-side, people might be interested in the lexer hack because C isn't a CFG.

[https://en.wikipedia.org/wiki/Lexer_hack](https://en.wikipedia.org/wiki/Lexer_hack)

</comment><comment><USER6> It is left as an exercise to the reader to implement this algorithm efficiently, but I really like the elegant use of recursion.

For those who can't be bothered diffing, the fix in the corrected version is inside the 3rd condition of subst().

</comment><comment><USER7> Looks like it's the correct handling of tokens' hide-sets that is the key. (A hide-set is used for preventing the infinite recursion in the macro-expansion process when a token appears somewhere inside its own definition.)</comment>


The Cliqz search engine uses a typo to slip past your tracker-blocker - <AUTHOR> https://anolysis.privacy.cliqz.com/ <comment><AUTHOR> This is significant because Cliqz claims to be a pro-privacy search engine, as is shown by them getting on the front page four times over the past four days:

[https://news.ycombinator.com/item?id=21716860](https://news.ycombinator.com/item?id=21716860)

[https://news.ycombinator.com/item?id=21694980](https://news.ycombinator.com/item?id=21694980)

[https://news.ycombinator.com/item?id=21684708](https://news.ycombinator.com/item?id=21684708)

[https://news.ycombinator.com/item?id=21676252](https://news.ycombinator.com/item?id=21676252)

</comment><comment><USER2> [Disclaimer: I work at Cliqz]

The name anolysis is not a type, it is derived from "Anonymous Analysis" and which truly means we do all the analysis of the data anonymously.</comment>


Firms that paid for Clinton speeches have US government interests - <AUTHOR> http://bigstory.ap.org/article/99b5f98f78a34d3f9febfd513bb4d206/firms-paid-clinton-speeches-have-us-govt-interests <comment><USER2> I still wish we had a Constitutional amendment that restricted federal employees and appointed / elected officials for a period of 5 years from working from any firm they had contact(1) with in their duties. It should also govern donations to charities run by said people. That and getting rid of every corporate deduction in the tax code would clear quite a bit of corruption out of the system.

I doubt we'll ever get either. A simple tax system means one of the chief ways politicians reward donors goes away, and the speech / job thing is just plain too lucrative.

1) the exact wording is more for lawyers than me

<comment><USER3> So if we pass enough rules the will of the voters should eventually take precedence?

I wonder what incremental changes we could make to get people to go vote their interest and to more carefully examine how elected officials make use of the vast power handed to them.

<comment><USER2> > So if we pass enough rules the will of the voters should eventually take > precedence?

Its a bit more about removing some of the incentives for "bad" behavior. These two are the most problematic to being fair and honest in government.

</comment></comment></comment><comment><USER4> > Clinton has said she can be trusted to spurn her donors on critical issues, > noting that President Barack Obama was tough on Wall Street

Meet the new boss, same as the old boss

</comment><comment><USER5> These are companies that have a responsibility to shareholders to spend funds to benefit the firm.

Obviously they feel the quarter million is well spent, as a bribe to influence government actions.

Anyone who thinks the bribes don't have the possibility to change a politician's actions is quite gullible.

</comment><comment><USER6> Can we all stop pretending they paid for her speeches?

<comment><USER7> Are you implying someone else paid for her speeches instead, or that she gave the speeches for free?

<comment><USER8> I think GP is implying that it isn't really speeches that they were paying for.

</comment></comment></comment><comment><USER9> More on HRC:

[http://youtube.com/watch?v=wK2K5v5bm0Q](http://youtube.com/watch?v=wK2K5v5bm0Q)

</comment><comment><USER10> Speech money is bribe money, plain and simple. These firms don't have any interest in hearing her speech.

<comment><USER8> > These firms don't have any interest in hearing her speech.

Of course they do. Having Hillary Clinton speak helps to cement and perpetuate a firm's status and influence in its industry.

Imagine your employer were to host such a speech. Who would they invite? Business partners, clients, investors...

</comment></comment><comment><USER11> Every firm has government interests. Its naive to think that $225k has any meaningful sway

<comment><USER12> While 225k might be chump change to the corporations; it's naive to think that $225k doesn't have any meaningful sway with the Clintons. Especially given this statement Clinton made regarding the speaking fees:

> "You have no reason to remember, but we came out of the White House not only > dead broke, but in debt," Clinton said. "We had no money when we got there, > and we struggled to piece together the resources for mortgages for houses, > for Chelsea's education.

</comment></comment><comment><USER13> Why $225k is too much? Bill Clinton and Al Gore take similar money. Well known political activists take between $50k and $20k per speech. Actors are charging millions for 20 minute episode. That money seems fair for such prominent speaker.

<comment><USER14> Actors are paid millions to star in movies that are seen by millions of people. More importantly, actors don't run your country or pass laws which affect you personally. If you don't like an actor, it's easy to not watch his movie. If you don't like a politician, you're stuck with her if she gets elected.

Why do people like you always rush to defend obvious bribery anyway?

<comment><USER13> Companies are investing into diversity, for example Intel will invest $300M over a few years. Hiring prominent feminist seems like a good way to do that.

Also such event often makes profit on donations.

<comment><USER7> > "Hiring prominent feminist seems like a good way to do that."

Then you should be looking to vote for Bernie Sanders.

[http://www.theguardian.com/commentisfree/2016/feb/09/i-am-a-...](http://www.theguardian.com/commentisfree/2016/feb/09/i-am- a-woman-voting-for-best-feminist-bernie-sanders-president-hillary-clinton)

</comment></comment></comment></comment><comment><USER15> We have an opportunity to elect the first women as President of the United States I hope we aren't going to pick on small things and ignore that opportunity. Unless she is proven to be any different than any previous President your choices are elect the first women as President or Trump. You decide.

<comment><USER7> The first women president is likely to be remembered for quite a while to come. Do you really want that person to be Hilary Clinton?

Let me put it to you another way... If Bernie had been elected, it's highly likely the vice president would have been Elizabeth Warren. Elizabeth Warren would then have a great platform if she wanted to run for president. Wouldn't you prefer Elizabeth Warren was the first female president over Hilary Clinton?

</comment></comment><comment><USER8> Can't say any of this is surprising...or even interesting. Big shock: companies that have enough money to pay a quarter of a million dollars to have Hillary Clinton speak also have lobbied the government. I think its pretty comical that people seriously think that these companies are buying real favors from the Secretary of State for only $225,000. Hillary doesn't need the money that badly, and if she really were selling favors, you can bet they'd be more than $225,000!

That said, this is a lesson in why public officials should be concerned about the _appearance_ of impropriety, and not just substantive impropriety. The former can still do a lot of damage (both to yourself, and to faith in government) whether you've done anything else wrong or not. It doesn't look good to public officials to be giving paid speeches while they are still in office.

<comment><USER16> Sure, HRC wouldn't cross the street for $200k. However, she and Bill have given speeches like that on a regular basis for years, to the same groups of people. Their campaigns and charitable organizations have been funded by those people too. You don't think that eventually, the cumulative tens of millions of dollars might have an effect on their thinking and decisions?

<comment><USER8> > You don't think that eventually, the cumulative tens of millions of dollars > might have an effect on their thinking and decisions?

And I'm sure all these donors' interests are aligned, right?

And, to be a little less flippant, I do think that the company a person keeps affects a person's thinking. So if a company can buy their way into Hillary Clinton's circle, I accept that they have managed to buy _some_ amount of influence. But I think this influence is negligible compared to Clinton's formal advisers and organic professional network. You also have to remember that most of these companies have far more effective ways of influencing government than paying for some lousy speeches, none of which is Clinton's doing.

<comment><USER17> > And I'm sure all these donors' interests are aligned, right?

Eh? I think they're broadly aligned. They're aligned when it comes to how corporations and executives should be treated by the courts and regulatory agencies. They're aligned on the extent and structure of free trade pacts. They're aligned on corporate taxation. They're aligned with Bill Clinton's claim that the only thing wrong with the American economy is a "skills gap."

Sure there's going to be disagreement (often severe) on particular points here and there but the broad thrust the economic "Washington Consensus" that Bill Clinton championed is a point of alignment among Clinton's donor class.

</comment></comment></comment></comment><comment><USER18> Surprised the Bernie bots are still kicking, this seems like more about nothing than something.

Where's the evidence of any such said corruption?

Just because they received money for speeches doesn't make them corrupt, and if you believe it does well then prove it with evidence you can back up but don't put out false claims. Yes she has gotten a good deal of money from speeches but it isn't the only politician out there (present or past) doing them for large amounts of money.

It's one thing to say you're influenced by lobbyist groups donating to your campaign and whole other to say the speeches are used to influence your policies, they're two separate things.

And even while HRC has gotten a decent amount of donations given to her campaign it's a very small piece of the pie that is shelled out by Corporations and Lobbyist groups, look at the other side for example, do you think they aren't COMPLETELY influenced by lobbyist groups? They practically own the Republican party.

Anyways as I said, more about nothing than something.</comment>


Congresswoman Maloney Wants to Deny Americans Access to Taxpayer Funded Research - <AUTHOR> http://www.michaeleisen.org/blog/?p=807

<comment><USER2> > But they are back at it. A new AAP backed bill -- the > "Research Works Act" > -- was just introduced by Reps Carolyn > Maloney (D-NY) and _Darrell Issa > (R-CA)_.

Again, do not count any Congressman as being "on your side" simply because of single actions on their part. Issa, Franken, et al. -- beware of offering any of them a "free pass" in your mind/thinking/consideration.</comment>


Etcdircd: an ircd backed by etcd - <AUTHOR> https://github.com/heyitsanthony/etcdircd <comment><USER2> There's also RobustIRC that's pretty interesting[0]. It bills itself as "IRC without netsplits" and uses Raft-style concensus underneath (which in this case you're getting through etcd instead).

[0]: [https://robustirc.net/](https://robustirc.net/)

<comment><USER3> Netsplits on IRC are such a visceral experience of distributed consensus failure. I remember as a little teenager on IRC being like "omg what is happening, why are there parallel worlds where weird things happen and everything drifts in and out?!"

<comment><USER4> I'm not convinced this is a solution to the problem though.

As you mentioned with traditional IRC during actual netsplit (not single node disconnecting, but two large networks separating) you will see two networks that will get out of sync with time. In this case (assuming it is implemented correctly) the etcd will just shut down and not allow anyone use it until netsplit ends.

Another issue here is that consensus protocols (raft, paxos etc) expect to have low latency. It doesn't make much sense distribute them geographically, and that will only result in a bad performance.

The biggest issue with Netsplits is getting out of sync, and the syncing back. Many networks solve this by ownership and channel/nick services.

I kind of like how IRCNet solved this problem without involving services:

\- nick conflict - each user gets an unique numeric ID based on server id and connection id, this ID is guaranteed to be unique even during netsplit. On nick collision, servers just reset the nick to that ID

\- channels - they actually had two solutions !channels which are designed in somewhat similar way, where it receives an unique ID and after netsplit one might end up with 2 channels with a different ID. Second solution is +R flag, which only works if there's no netsplit and if no one has an operator status it will grant one person matching one of the masks there.

I like that approach, because it tries to solve the problem without central authority, it's still arguable what is better.

<comment><USER5> Egalitarian Paxos (EPaxos) has much better WAN performance than previous protocols.

      https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf

Most real-world Paxos implementations, like Raft (what etcd uses), simply use Paxos for electing a leader. Afterward, all updates are serialized through the leader. This was a performance optimization, but it works well only on LANs; it sucks for WANs because you've at least doubled the round-trip time for every commit and the leader becomes a bottleneck, so it's especially important to have low latency.

EPaxos was published before Raft and I couldn't find any benchmarks comparing them directly, but in the paper above EPaxos outperforms Mencius (a rotating single leader design) even within a single EC2 cluster. And EPaxos maintains much more consistent performance during latency spikes. So EPaxos may help close the performance gap between LAN and WAN clusters. In the context of IRC where you're geographically distributed _anyhow_, something like EPaxos may impose no appreciable penalty at all, at least if there's sufficient redundancy to ensure a quorum.

<comment><USER6> > Most real-world Paxos implementations, like Raft (what etcd uses), simply > use Paxos for electing a leader.

Ah, the ambiguity of natural language. Just to make this absolutely clear: Raft is not an implementation of Paxos. The "like" here refers to the "electing a leader" bit.

(I assume you already know this, it's just that the sentence as written is ambiguous and I wanted to clarify. I initially read it the wrong way and felt an urge to 'correct' you.)

</comment></comment></comment></comment></comment><comment><USER7> It's nice to see that an easy TLS setup seems to be a priority for this[1], even if it ends up just being a toy project.

[1] [https://github.com/heyitsanthony/etcdircd/blob/master/securi...](https://github.com/heyitsanthony/etcdircd/blob/master/security.md)

<comment><AUTHOR> Privacy is the priority, so TLS is essential :)

</comment></comment><comment><USER8> It's distributed stuff day! Also learned about TiDB today.

</comment><comment><USER9> Kudos for making this AGPL. This is exactly the sort of software that should use such a license.

User's of an IRC server deserve to know whether their operator has started recording messages, added or removed security features, or otherwise be free to review said code.

All chat software should be AGPL.

<comment><USER10> > _User 's of an IRC server deserve to know whether their operator has started > recording messages, added or removed security features, or otherwise be free > to review said code._

I don't understand how this is related - it's probably easier to tap messages outside of the software via packet capture or something than to modify the software to record them.

Also, someone who's interested in recording messages without user consent is probably also totally fine with violating the license.</comment></comment>


To stream everywhere, Netflix encodes each movie 120 times - <AUTHOR> http://gigaom.com/video/netflix-encoding/

<comment><USER2> A friend of mine, Baptiste Coudurier, is the person at Hulu who manages the transcoding of all their videos. He told me they also transcode a video to multiple versions to accommodate different streaming devices and bandwidth characteristics.

I think it only makes sense to lazily-transcode videos on demand (and cache them) when you have millions of videos (eg. Youtube). Netflix and Hulu have much smaller catalogs, so they can afford to transcode ahead of time, and it simplifies their processes.

<comment><USER3> I wonder if there's some lengthy preprocessing step that's common to transcoding regardless of the target resolution/quality. That way you could precompute and store the intermediary format and finish of the encoding to an arbitrary format as required.

<comment><USER4> For two-pass encoding, the video is first scanned to assess complexity, so a second encode pass can work out which frames require a higher bitrate to get the same quality. However this is not used that often, in favour of a quantiser approach, which uses a number to indicate target quality. The encoder then changes the bitrate depending on the frames complexity and the target quality/quantiser. The quantiser approach is much more common because it's easier to specify an output quality that is independent of resolution.

As for the actual encoding, nearly all the computation is very specific to the output resolution/quality. Luckily encoding multiple videos can be spread over many machines, and encoders can take advantage of multiple cores.

</comment></comment></comment><comment><USER5> If this is true, then this should immediately put to rest the explanation (that I'll admit I bought into) that only one video format will win out (H.264 vs Ogg Theora/VP8 is the major discussion point) - as video streaming companies would be unwilling to re-encode their content for more than one format.

Apparently they are quite comfortable encoding as many as 120 instantiations of their videos.

<comment><USER6> In fact, it is not unheard of for us to re-encode the entire catalog when a new, better format or codec comes along, or we add support for a new device.

<comment><USER7> Huh. You guys must have quite the rendering farm for that, eh?

<comment><USER6> EC2. 1000s of instances. :)

<comment><USER8> Spot pricing I hope! =)

</comment></comment></comment></comment></comment><comment><USER6> Using sometimes 1000's of EC2 instances (for a bunch of movies, not just one).

<comment><USER9> this being an ideal application of simd kind of parallelism, i am wondering of a farm of gpu's might be much cheaper (and faster) rather than a general purpose cpu ?

<comment><USER10> AWS has farms of GPUs you can rent by the hour - <http://aws.amazon.com/ec2/instance-types/> \- Look at "Cluster GPU Instances"

</comment></comment></comment><comment><USER11> At the end of the post there's this quote:

> _Simply put, content owners who can deliver quickly and without error are > getting more licensing revenue from Netflix._

Really? Can Netflix afford to be picky with content owners? It seems their main problem is the limited size of their online catalog...? They should take anything they can get!

I just finished watching "Elite Squad 2"; it's such a great movie (really, it is: see <http://www.imdb.com/title/tt1555149/>) that I wanted to watch the first one, but it's "only available on DVD".

Why, why, why?

<comment><USER6> I think you misinterpreted.

We have $X to spend. We have $Y left at the end of the year ($Y < $X). We go though our system and say "who can deliver us the quickest, most error free, content? Ok, give them $Y for whatever they have that our customers might enjoy.

The reason you only see Elite Squad 2 is the same reason that you only see a lot of stuff -- because the price they wanted for the other one was too much.

<comment><USER11> Thanks for the clarification.

(I'm not sure that the technical quality of transmitted files is the best criterium to select content, though, even when it's the end of the year and we're talking about "leftover money").

</comment></comment></comment><comment><USER12> ...everywhere except Linux.

<comment><USER13> This isn't due to transcode issues. They use Silverlight to stream to full- function PCs (be it OSX or Windows). Microsoft hasn't made Silverlight available to the greater GNU community, so it's pretty obvious that they wouldn't distribute to Linux: their contracts with content-holders probably hold them to constricting DRM demands on the videos streamed to devices capable of ripping them off.

Don't blame Netflix for it. Blame the arduous contracts which must be negotiated in order for Netflix to carry the content it does.

<comment><USER14> They have a working Android application which uses native linux code to handle the DRM.

So Silverlight not beign supported on Linux is not the real issue here.

<comment><USER15> Right, it's the contracts, exactly like the parent post said...

<comment><USER16> The issue is it not being supported on regular desktop Linux (excl. Chrome OS).

</comment></comment></comment></comment></comment><comment><USER17> As technology progresses we can probably assume that at some point in the future virtually all devices in use will have fast enough network connections and decoders to play videos that are the highest quality the human eye can see. At this point it would only be worth encoding in this "highest" quality. Of course, by this time we may have new screen technologies such as holographic displays.

<comment><USER16> Yes I can see you mean a time and day when everyone will gigabit connections and what not.

But then you're forgetting: poverty hasn't quite abandoned us yet, has it?

Even with all of the modern technology and what not -- people are still dying of hunger.

We first need to start with these base human needs.

Elevate the standard of living of the people on the other side of digital (and economic) divide, and bring them up to standard of an average first-world- country person.</comment></comment>


Kernel 101 – Let's write a Kernel - <AUTHOR> http://arjunsreedharan.org/post/82710718100/kernel-101-lets-write-a-kernel <comment><USER3> I got this running on qemu by cannibalizing a tiny bit of code from xv6 ([http://pdos.csail.mit.edu/6.828/2012/xv6.html](http://pdos.csail.mit.edu/6.828/2012/xv6.html)) to replace the GRUB dependency. After cloning and building mkernel according to its instructions:

      $ git clone git://pdos.csail.mit.edu/xv6/xv6.git
      $ cd xv6
      $ make

Now you should be able to run xv6 by itself:

      $ path-to-qemu/x86_64-softmmu/qemu-system-x86_64 -serial mon:stdio -hdb fs.img xv6.img -m 512

To run mkernel on qemu, we'll replace xv6's kernel with mkernel's:

      $ dd if=/dev/zero of=mkernel.img count=10000
      $ dd if=bootblock of=mkernel.img conv=notrunc
      $ dd if=../mkernel/kernel of=mkernel.img seek=1 conv=notrunc

Now you can boot up the _mkernel.img_ rather than _xv6.img_ :

      $ path-to-qemu/x86_64-softmmu/qemu-system-x86_64 -serial mon:stdio -hdb fs.img mkernel.img -m 512

(Based on xv6 at hash ff2783442ea2801a4bf6c76f198f36a6e985e7dd and mkernel at hash 42fd4c83fe47933b3e0d1b54f761a323f8350904. Ping me if you have questions; email in profile.)

<comment><USER4> > I got this running on qemu by cannibalizing a tiny bit of code from xv6 to > replace the GRUB dependency.

This thing doesn't depend on GRUB, per se. It requires a multiboot protocol compliant bootloader, and QEMU and Bochs emulator have one built-in.

All you need to do is:

        qemu-system-i386 -kernel kernel

This was mentioned in the last lines of the OP, perhaps they were added after you read it.

<comment><USER3> Ah. Thanks! Yeah, I'd emailed the author with my comment.

</comment></comment></comment><comment><USER5> There's a slight problem in this tutorial in that it assumes ESP (the stack pointer) will be defined by the boot loader to point to an appropriate location for the stack. However, the Multiboot standard states that ESP is undefined [1], and that the OS should set up its own stack as soon as it needs one (here the CALL instruction uses the stack, and the compiled C code may well too).

An easy way to solve this is to reserve some bytes in the .bss section of the executable for the stack by adding a new section in the assembly file:

      [section .bss align=16]
        resb 8192
        stack_end:

Then before you make use of the stack (between `cli` and `call kmain` would be appropriate in this case), you need to set the stack pointer:

      mov esp, stack_end

[1]: [https://www.gnu.org/software/grub/manual/multiboot/multiboot...](https://www.gnu.org/software/grub/manual/multiboot/multiboot.html#Machine- state)

</comment><comment><USER4> This seems like a good start but it's worth noting that it isn't guaranteed to work correctly.

The problem is that the control is passed to C code with no stack space set up. It works out of pure luck because the compiler has decided to keep all variables in registers and changing your C compiler flags might make this fail in interesting ways.

Another thing that is missing is clearing the .BSS section before passing control to the C code. It's not used at the moment, though.

I am pointing these things out because in my own Ring—0 programming projects I spent a lot of time debugging some failures related to stack space and an un- initialized BSS section.

<comment><USER5> > Another thing that is missing is clearing the .BSS section before passing > control to the C code. It's not used at the moment, though.

The Multiboot standard says that the boot loader will clear the .bss section for you - in section 3.1.3: "bss_end_addr Contains the physical address of the end of the bss segment. The boot loader initializes this area to zero"

I don't know what GRUB does if you rely on that fact it can parse ELF files and don't specify the fields like bss_end_addr though. I'm fairly sure it clears it in this case too, but I'm using Multiboot 2 for my OS so the behaviour could be different.

<comment><USER4> > The Multiboot standard says that the boot loader will clear the .bss section > for you - in section 3.1.3: "bss_end_addr Contains the physical address of > the end of the bss segment. The boot loader initializes this area to zero"

Ok, good to know.

It certainly doesn't do that unless you tell it to (using the address tag), and this example (nor my hobby kernel) use that.

So the BSS must be cleared or the bootloader told to do so.

<comment><USER5> I've just checked the GRUB source code and I think it will clear the .bss section even if it's loading an ELF file.

grub-core/loader/multiboot_elfxx.c has a function named grub_multiboot_load_elf32/64 which actually loads the segments of the ELF file. A segment has two fields defining its size: filesz (which is the amount of bytes to copy from the file) and memsz (which is its actual size once loaded). If memsz is greater than filesz, it zeroes the trailing bytes:

      if (phdr(i)->p_filesz < phdr(i)->p_memsz)
        grub_memset ((grub_uint8_t *) source + phdr(i)->p_filesz, 0,
          phdr(i)->p_memsz - phdr(i)->p_filesz);

The .bss section is placed by the linker at the end of a segment and increases memsz by the size of it (but not filesz, to avoid having to place lots of pointless zeroes in the ELF file) - for example this is one of the segments from my kernel's ELF file, which contains the .bss section at the end:

      LOAD off    0x0000000000020000 vaddr 0xffffffff8011f000 paddr 0x000000000011f000 align 2**12
           filesz 0x0000000000004be0 memsz 0x0000000000017678 flags rw-

Here you can see memsz is 0x17678 bytes and filesz is smaller at 0x4be0 bytes. The difference between them is the size of the .bss section.

grub_multiboot_load_elf32/64 is called in the case when the address tag is not present, so the .bss section will be cleared by GRUB in this case as well.

</comment></comment></comment></comment><comment><USER6> This is a great resource for anyone who would like to take this further: [http://wiki.osdev.org/Expanded_Main_Page](http://wiki.osdev.org/Expanded_Main_Page)

In particular, setting up interrupt handlers, paging, and getting a PIC setup is pretty neat.

</comment><comment><USER7> If anybody is doing this, let me share some words of advice based on experience.

Please use a virtual machine instead of doing this on your primary machine. You eliminate the risk of messing up your machine. Also, if you setup the VM properly, you get a debugger.

<comment><USER8> Could you point to how to go about setting a suitable VM up on Debian? I could do with not bricking my machine :)

<comment><USER9> Use QEMU.

[http://qemu.weilnetz.de/qemu-doc.html](http://qemu.weilnetz.de/qemu-doc.html)

</comment></comment></comment><comment><USER10> Looks great, will keep an eye on this!

I also really enjoyed James Molloy's OS kernel development tutorial at [http://www.jamesmolloy.co.uk/tutorial_html/](http://www.jamesmolloy.co.uk/tutorial_html/), which takes you from "Hello World" to some real toy OS kernel implementation.

<comment><USER5> It's worth pointing out there are a few bugs in James Molloy's tutorial [1], and some of the things he does in them aren't exactly best practices - <USER2> example, a few I remember are:

\- Disabling interrupts and paging (which also has the side effect of flushing the TLB) to copy memory around by physical address. This could be done without disabling them by mapping all of physical memory into virtual memory instead (possible in 64-bit mode, but in 32-bit there isn't enough room when your PC has a similar amount of RAM to virtual memory space, in which case you could map smaller parts of it as needed).

\- Moving the stack around to get around the fact that GRUB doesn't set ESP to some well-defined value (instead of defining the stack yourself, which would be much more robust) and then attempting to rewrite the base pointers to fix it. For example, his code can't tell the difference between integers that just happen to have a value in the range of the pointers and a pointer, and will happily rewrite both. Also as ESP isn't defined by the Multiboot standard you could be using any location at all as the stack (such as some memory address that doesn't exist, or your kernel's code itself, or some memory-mapped area for a piece of hardware, etc.) All of which will mean things go wrong. It's better to just set ESP yourself before you enter C - see another of my comments on this submission here [3].

There's actually a newer and much better version of JamesM's tutorials on GitHub, but I believe they aren't quite finished [2].

[1]: [http://wiki.osdev.org/James_Molloy%27s_Known_Bugs](http://wiki.osdev.org/James_Molloy%27s_Known_Bugs) [2]: [https://github.com/jmolloy/JMTK](https://github.com/jmolloy/JMTK) [3]: [https://news.ycombinator.com/item?id=7590753](https://news.ycombinator.com/item?id=7590753)

<comment><USER10> Oh, didn't know there was a follow-up to his original tutorial - I'll have a look at your resources when I get back from work!

</comment></comment></comment><comment><USER11> Very cool. I personally (as a developer without a CS background) find these sorts of posts wonderfully interesting, even if this kernel, as pointed out in this thread, lacks a lot of what a normal kernel does. I'd love to see one of these for a compiler!

<comment><USER12> I highly recommend [http://www.hokstad.com/compiler/](http://www.hokstad.com/compiler/) ; it talks about writing a compiler in a way that makes sense to me - writing it the way you write any other program, rather than throwing you straight in with lexers and parsers and never justifying why we need to do things this way.

<comment><USER13> Glad you like it... I'm hoping to push out the next part later this week, and I've got a few more drafts queued up that "just" needs some proofreading.

Working on getting it to the point where it can fully compile itself now, and hope to get there over the couple of months.

<comment><USER14> I've been reading this series with interest for years. Thank you for writing it.

</comment></comment></comment></comment><comment><USER15> I really like the new HN. Quality of articles is way up.

<comment><USER11> I've found myself spending a lot more time here as well. Was there any code/policy change with the HN site? Or is this new moderators jumping in and helping out a lot? Either way, I agree that it is excellent.

<comment><USER6> About a week or two ago pg added moderation. I cannot find the thread right now, but basically comments are not published until someone with high enough karma (I think 1000 points) approves the comment.

<comment><USER11> Oh, I thought that was only for selected threads.

<comment><USER16> It is, bebop was mistaken. (Unless my comment does not show up, in which case he is correct.)

</comment></comment></comment></comment></comment><comment><USER17> Neat read so far! Not done yet, but I think I've found a small error in kernel.c: the attribute byte of the characters in "my first kernel" should be set to 0x02, not 0x07.

edit: I misread. 0x07 is intentional, 0x02 was mentioned as an alternative. Good post!

<comment><USER18> uaygsfdbzf: your account is marked dead it seems, so most of us can't see what you write.

Here's what he posted:

Not an error. 0x02 means green character on black background. 0x07 means light grey character on black bg. It's explained with one, and coded with another.

<comment><USER19> Not dead for me, and I don't have showdead on.

</comment></comment></comment><comment><USER20> This was fantastic. My only question is, how does one gain knowledge of the required x86 hardware specifics he mentions? I don't know where to begin looking to uncover these sorts of things:

      - The x86 CPU begins execution at the physical address [0xFFFFFFF0]
      - The bootloader loads the kernel at the physical address [0x100000]
      - The BIOS copys the contents of the first sector to physical address [0x7c00]

Is there an x86 instruction manual or is this sort of thing passed down through generations of engineers?

<comment><USER4> > Is there an x86 instruction manual or is this sort of thing passed down > through generations of engineers?

Yes, there is an x86 instruction manuals, colloquially known as "intel manuals".

That's about a dozen volumes of documentation, at a few thousand pages each. It is quite a good example of well written and informative technical documentation.

[http://www.intel.com/content/www/us/en/processors/architectu...](http://www.intel.com/content/www/us/en/processors/architectures- software-developer-manuals.html?iid=tech_vt_tech+64-32_manuals)

</comment></comment><comment><USER8> Can anyone give me an idea how much different this would be for 64bit? Do I just change the nasm directive to `bits 64`?

<comment><USER4> > Can anyone give me an idea how much different this would be for 64bit? Do I > just change the nasm directive to `bits 64`?

A lot different. You can see the boot code of my x86_64 hobby kernel project [1].

The reason is that GRUB/Multiboot protocol is actually doing most of the machine initialization, but it can only set up 32 bit mode. 64 bit mode is missing partly because standardizing the Multiboot protocol is dragging behind, partly because there's no one correct way to do this as you can't have identity mapped memory in 64 bit mode (unlike 32 bit mode).

If you read the OSDev wiki, you can find examples of doing machine initialization "from scratch", ie. after the PC BIOS (or UEFI). This involves arcane details about the x86 machine like setting up something called the A20 line (which was a hack that allows to have 1 megabyte of memory - utilizing a spare pin from the keyboard controller!), etc. Dealing with this shit is not time well spent. (UEFI is a lot easier in some ways, harder in others)

This means that the boot code of your kernel must set up long mode, create an initial page table for virtual memory, etc. Here's my limited 64 bit boot code that sets up one 2 megabyte page [2].

Going to 64 bit mode is not that much more code, but it will make kernel development more painful. In particular, switching CPU modes messes up the GDB debugger which must be patched to work at all. And there's a bit more work involved in all the little things that come with it.

So for educational purposes it would make more sense to stick to 32 bit mode than deal with the nitty gritty details of 64 bit long mode.

[1] [https://github.com/rikusalminen/danjeros](https://github.com/rikusalminen/danjeros) [2] [https://github.com/rikusalminen/danjeros/blob/master/src/arc...](https://github.com/rikusalminen/danjeros/blob/master/src/arch/x86_64/boot/start.s#L57)

<comment><USER21> This makes me really sad. 32bit mode x86 assembly is such a mess compared to amd64 -- I guess it's a good excuse to work with qemu and arm, if nothing else ;-)

<comment><USER22> On the topic of ARM, does anybody know of a similar example like this for ARM? One that just shows you how to pass control to C and do some basic I/O?

<comment><USER22> Too late to edit my post, but I eventually found this: [http://wiki.osdev.org/ARM_Integrator- CP_Bare_Bones](http://wiki.osdev.org/ARM_Integrator-CP_Bare_Bones). I haven't tried it out yet as I'm still getting the cross-compiler toolchain together (doesn't seem to be part of OpenSUSE 12, at least not an obvious part), and it's not as well explained, but still the best I've found.

</comment></comment></comment></comment></comment><comment><USER23> I kinda wish some of the FS utilities were better. Despite FUSE, mounting a block device (like, disk image) as a non-root user is tough, so writing to an image with actual partitions/FS on it is difficult, especially to script, especially if you don't want to sudo in a build script. losetup on a disk image doesn't (to my knowledge) detect partitions… for reasons unknown to me.

Bootsectors are similar. You can't just install grub to a disk image. (You have to losetup it, at the very least, which implies root. Why can't I just install to a file?)

You want a script/build system that allows you, at the very least, to:

1\. Code. 2\. Run build system/script. 3\. Fire up qemu or similar.

You can't be rebooting. Ideally, it'd be great to do this in userspace.

That said, if you're starting out, just do [boot sector] + [kernel] = tada image until you need to do otherwise. (Really, do whatever works and is easy.)

That said, I've found a few somewhat helpful tools.

fuseloop[1] takes a file and offset/size, and exposes a single file. If you have a partitioned disk image, then you can feed it the partition offset, and it gives you back something you can format as an FS. (e.g., you can run ext2fs on.)

Then there's fuse.ext2[2], which mounts an ext2 FS on FUSE, so non-root usable again. Note that I'm linking to my fork of it, since the original didn't build for me, but I didn't write it. (Which I fixed, and sent a pull request, but never heard back.)

Finally — and sorry to peddle my own stuff again — I wrote a Python library for dealing with the MBR.[3] I use it to figure out offsets and sizes in a disk image.

I've had a bit of fun writing a boot loader, and I've managed to get it to load up its stage 2 and switch to 32-bit pmode. Had a fun error where a division instruction was throwing things into a triple fault; see [4] if you want to see how a division instruction can fail _without_ dividing by zero (which was the first thing I checked). The disk layout is currently:

      [boot sector] [stage 2] [kernel] [ partitions, FS, real data, etc. ]

stage2's size is hard coded into the first sector of stage 2 (into itself), and the kernel's location and size will be similarly hard coded into it as well. (When I get there. Disks in pmode are different, as you can't just have the BIOS do all the work for you, sadly!) And by hard-coded, a build script calculates and just re-writes a few bytes.

[1]: [https://github.com/jmattsson/fuseloop](https://github.com/jmattsson/fuseloop)

[2]: [https://github.com/thanatos/fuse- ext2-fakeFS](https://github.com/thanatos/fuse-ext2-fakeFS)

[3]: [https://github.com/thanatos/pymbr](https://github.com/thanatos/pymbr)

[4]: [http://stackoverflow.com/questions/21212174/why-cant-i- step-...](http://stackoverflow.com/questions/21212174/why-cant-i-step-over- this-div-instruction)

<comment><USER5> > losetup on a disk image doesn't (to my knowledge) detect partitions… for > reasons unknown to me.

You can use the kpartx command for this. This site has a good overview: [http://nfolamp.wordpress.com/2010/08/16/mounting-raw- image-f...](http://nfolamp.wordpress.com/2010/08/16/mounting-raw-image-files- and-kpartx/)

</comment></comment><comment><USER24> I have done the same, sort of, only eschewing BIOS for UEFI and thus getting straight to Long Mode :-).

It's pretty basic, currently just boots up and prints the memory map.

It's written in C++11 with the aim to be as clear as possible: [https://github.com/thasenpusch/simplix](https://github.com/thasenpusch/simplix) Have a look :-).

</comment><comment><USER25> I just finished an Operating Systems final today and the entire class was conceptual and learning fundamentals, while I craved to get my hands dirty and actually try to make a simple OS.

I'll definitely be playing around with this. Thanks!

</comment><comment><USER26> Why kernel tutorials always say they need nasm. Gcc already comes with an assembler so simply use it instead of installing other softwares. And if you prefer intel syntax simply add ".intel_syntax;"

</comment><comment><USER27> Looks great, looking forward to dive into this.

</comment><comment><USER28> Why C?

I'm just curious if another language can be used. (c++, go, rust).

<comment><USER29> C is really easy for this sort of thing. I don't know the specifics for using C++, Go, Rust, etc. (Disclamer, I'm a fan of C, and I know some C++ and Rust but little Go). The kernel also doesn't get run-time help, so a lot of features of C++, Rust, and Go would be gone right off the bat (Ex. You can't be spawning threads in Rust or throwing exceptions in C++ without first setting up your environment, which can be a non-trivial task). Using a subset of any of those languages should work, provided the compiler can provide straight ELF binaries that you could boot (I know g++ and rustc can do this, Go is the only one I wonder about.).

If you get to that point, then you have the fact that the C ABI is fairly standard, extremely easy to use, and well-known, so it makes it easy to intermingle assembly and C. If you did it in C++, you'd definitely want to extern "C" any symbol asm is going to call so it isn't mangled, Rust should be the same, no idea on Go (You'd really also want to extern "C" any user-space system calls, because the name mangling might get in the way.). For the last point, you have to be able to use pointers and write to arbitrary memory locations, which is easy for C and C++, possible in Rust, but I don't know about Go.

IMO, the code displayed in this example is really hardly a kernel (It _is_ , but it doesn't really do anything). For a more complex kernel, you may get a benefit from using a language other then C, but for something this simple C lowers the complexity to get a working example with minimal issues going.

</comment></comment><comment><USER30> Awesome read, thanks.

</comment><comment><USER31> I'm not exactly Linus Torvalds but I'm pretty sure a program that prints one line of text is not "a kernel". :)

<comment><USER32> I'm going to give OP the benefit of the doubt on this one. A couple things worth noting, it successfully boots, does not cause a fault of any kind, and is in a position to interact directly with the bare metal. The tools that we use to interact with a *nix system are often just that, bits of code in user- space. This is a kernel-space program. That it does not do any of the memory management or device access yet doesn't mean it's not a kernel.

It might actually be interesting to take these techniques and apply them to something else. Maybe a demo, ala the demoscene, that runs on the bare metal. Maybe implement a game that doesn't have the overhead of an os.

This is less a finished product and more an inspiration. So often I think that people miss that about things. Sometimes things aren't done, sometimes they're just beginning.

<comment><USER33> > Maybe implement a game that doesn't have the overhead of an os.

I'm in CMU's operating systems class right now and that was one of our projects (our second). The first was writing a stack-tracing debug library, the third was a user-space thread library built on top of a particular kernel spec, and then the fourth was to build a kernel basically from scratch, using our thread library as a test program.

Here's a link to the spec for the game we built on the bare metal: [https://www.cs.cmu.edu/~410/p1/proj1.html](https://www.cs.cmu.edu/~410/p1/proj1.html)

That fourth project, or "p3" as it's known around here, is known to be a killer. It's tough to write a whole kernel in 6 weeks (+ 1 week of break), especially when it's expected to have a full virtual memory system, kernel tasks/threads for concurrent programming, program loading, interrupt handling, etc, even with a partner.

<comment><USER32> Wow, that's pretty cool. Thanks for the link.

</comment></comment></comment></comment><comment><USER34> The first question is:

Why?

We already have a whole bunch of operating systems, many of them free.

One of the frequent problems with a lot of free/open source software folks is that they lack direction. This type of thing where we do stuff just to do stuff probably won't fly in one of the leading tech companies.

Why don't you figure out a real problem people have and look for ways to solve that, instead of just doing random "interesting" stuff that wastes people's valuable time?

<comment><USER35> You must be a real joy at parties.

</comment></comment><comment><USER36> Great! except that to be called a kernel it's missing just a process manager, memory manager, filesystem, process separation and hardware abstraction. Yeah I'm _that_ guy, down vote me as you wish, the article is still wrong.

It's a way to load a ring-0 application into grub. Pretty cool, but not a kernel.

<comment><USER37> It's a statically linked program, real memory, diskless system, kernel-mode- only kernel without too much fluff in the video i/o abstraction.

<comment><USER36> Kernel means "core". Kernel-mode-only kernel is a contradiction. If you don't have separate parts, you can't have a core.

<comment><USER37> So if you write core software for an architecture without privsep you must not call it a kernel?

<comment><USER36> I'm not talking about privsep, just different modules. I gonna answer you with another question:

Did MS-DOS had a kernel?

<comment><USER37> Yes. We all lose here today.

<comment><USER36> It depends. Being downvoted by people that believes a memcpy() is a kernel, is victory.</comment></comment></comment></comment></comment></comment></comment>


Ask HN: Good tools to identify processes making laptop hot? - <AUTHOR> Every Macbook Pro I've had becomes very hot when not doing anything.<p>Is there a way to identify what processes are causing this, and kill them if they aren't important? <comment><USER2> There's a "Using Significant Energy" list in the battery menu ([https://www.macrumors.com/2017/01/16/mac-display- brightness-...](https://www.macrumors.com/2017/01/16/mac-display-brightness- significant-energy/))

"Activity Monitor" can tell you which processes use the most CPU time ([https://support.apple.com/en-us/HT201464](https://support.apple.com/en- us/HT201464))

Both correlate decently with "make laptop hot".

</comment><comment><USER3> I'm aware that some Compaq (HP) laptops suffer similarly due to poor design. Their problem IMHO is that DUST being drawn in by the fan cannot escape. Progressively it covers internal components which heat up. Standing laptop on a cake "cooler"* can help keep it cool. This helps improve the flow of air under the laptop AND increases the distance from the desktop.

* Small raised metal tray with rods in horizontal grid.

</comment><comment><USER4> I suspect, not to be mistaken for knowledge, if you slowed your clock down by 10%, which would not be noticeable if you are not doing anything, it will run cool.</comment>


Getting to Zero Exceptions - <AUTHOR> http://yellerapp.com/posts/2015-06-01-getting-to-exception-zero.html <comment><USER2> posted less than a day ago, 28+ comments: [https://news.ycombinator.com/item?id=10841759](https://news.ycombinator.com/item?id=10841759)

repost from last June.</comment>


AMA, HN: Ben Rudgers [Was "AMA, HN [v. 00.00.01a]"] - <AUTHOR>

Ask me anything.<p>[Edit] Why would anyone want to ask me anything?  You probably don't. Hopefully someone more interesting will try it.<p>[Edit] Then why do it? The purpose of this post was to show that the utility of AMA's on HN can be tested, easily. Even if it is more fun to argue about it. <comment><USER2> who are you?</comment>


Backbone.js and Capsule and Thoonk, oh my A scalable realtime architecture - <AUTHOR> http://andyet.net/blog/2011/nov/16/backbonejs-and-capsule-and-thoonk-oh-my-a-scalable/

<comment><USER2> How is this scalable? It seems like this is a single-instance solution that in theory could scale, but this isn't even proven, nor is a clear path given. The author throws out a few vague ideas, and seems to vastly underestimate the complexity of scaling out. Redis cluster also is nowhere near stable, and keeps getting pushed back, so how does this help anyone now?

<comment><USER3> This is not a single instance solution. We can run as many instances as we want across multiple servers. It is currently a single-Redis instance solution, however we have some intelligent sharding planned.

I agree with you about Redis clustering, and would extend it by saying I don't think it'd be helpful for this problem regardless. Intelligent sharding with some gossip for slave promotion is where we'll have to go.

See my follow-up post tomorrow.

<comment><USER2> Yes, single Redis, but multiple instances of the application backed with a single datastore is hardly revolutionary. This is the classic reason why people go with a stateless app layer, so that it's simple to scale at least a single tier of the architecture.

I'm still not sure how this lives up to the "scalable realtime architecture" that dominates the title. People have been building Redis-backed message brokers for a while now. It seems like you have some untested ideas for "intelligent sharding" that could possibly provide scalability, but that's punting on the hard problem. Without this, this is all really pretty pedestrian.

The gossiping sounds like an interesting lead, is this going to be discussed in the post tomorrow?

<comment><USER3> Evolutionary, and probably not even a unique idea. My post does go into the intelligent sharding a bit. Essentially sharding, in the case of &bang, on teams with a simple lookup to find which server a given team is on. Allowing the stateless app layer (as you rightfully call it, other than keeping track of subscription event routing) to connect to multiple Redis servers as needed. You could be on two teams on separate servers and it would work fine.

I also agree that people have been using Redis for messaging, however, what we're doing is a step past that. Again, probably others are doing this. We have atomic verbs for dealing with higher level objects in Redis that use Redis-PubSub for communicating the changes to the objects. This way, whenever an object (feed) is changed (publish/edit/delete/reposition), it can bubble up to the user or processes that care. These feeds are broken down by not only topic but subscribe-able units of interest as well. In this way, processes, users, etc only get updates on data that is relevant to them.

I'm working on converting these verbs to Redis-Lua scripts as my tests have shown it decreases CPU time and reduces atomicity code (especially in Node.js where watch->multi->exec callback stacks can be interrupted by other events). I also expect it to make supporting Thoonk in multiple languages easier as the core code will be shared.

The gossiping is currently being discussed in the Redis mailing list and may end up taking a similar approach to Hadoop's intelligent clients. Antirez would like to provide some tools to make this easier as others have rolled their own Redis gossiping. I'll be researching that more after implementing the team sharding. HA, while related to scaling, is a bit off topic, and I admit to hand-waving here. For now we're focusing more on not losing/corrupting user data.

Perhaps Henrik's title is hyperbole. I doubt the approaches that we're taking are revolutionary, and I imagine similar things been done before. However, we believe that it is a good approach, and we believe in the direction we're taking enough to share it.

Thanks for the intelligent discussion.

<comment><USER2> Cool, thanks for taking the time to go a bit more in depth on this. This is a topic I'm particularly interested in, so I'm looking forward to the follow-up.

</comment></comment></comment></comment></comment><comment><USER4> Looks quite promising. I've played a lot with node.js, Redis and pub/sub wrappers around socket.io and I came to the conclusion that it's this kind of architecture that does the job of scaling well horizontally until Redis becomes the bottleneck.

I'll dig in the source code as I'm very interested in how it's implemented.

<comment><USER3> My post for tomorrow as a quick section on scaling beyond a single instance of Redis.

Essentially intelligent sharding (in &bang's case, by team). Each node.js (or other processes) can look-up which Redis server-set owns a team, and can probably just stay connected to most all of the Redis instances. For HA, slave each shard, AOB, and off-server backups every 15m. Along with a gossip protocol for giving up on masters and promoting slaves.

According to our tests, we don't have to worry about it for awhile, but we've got a plan regardless and will start implementing it soon.

</comment></comment><comment><USER3> I'm working on a post for tomorrow that goes into the details of making a feed-driven single-page app and a bit of the philosophy of our design choices and Thoonk itself.

</comment><comment><USER5> I don't see anything to deal with conflicts and operations crossing each other on the wire. The sort of thing that Operational Transformation deals with.

<comment><USER6> Depends what you're trying to keep concurrent, and your concurrency policy. OT is great for optimistic concurrency on complex structured documents, but probably overkill for simple models (like to-do items) with simple attributes (like title, is-done etc.). For that you could get away with - eg. a revision counter on the model.

Because of the performance implications of different concurrency, you probably don't want it baked too deeply into any framework, since there's some stuff you definitely want strict conflict/error resolution on, and other things you don't.

<comment><USER3> Essentially, that's our take as well. However, we do have a plan for dealing with conflicts and missed updates soon -- probably after Thoonk.js 1.0 slated for late December. Here's how I replied to this question on the blog comments:

The data in the thoonk feeds are never edited locally on the client. Any user- actions that change data go out as a websocket rpc call, and get placed in a job queue. The workers validate the data, check ACL, and then update the corresponding feeds, which then bubbles back up to the user. This happens nearly instantly, and the feed updates are atomic. If two users edit an object, then the last edit wins. For the data we have, this isn't a problem. However, we've got a plan for dealing with concurrency and conflict resolution in the future so that we can can handle being offline for periods of time, and detecting and dealing with conflicts. \--

For &bang in particular, we'll probably just let the user that owns the task resolve the conflicts. "You queued an update to this task while offline, but Bob edited as well. Which version would you like to keep?" In general, you can't edit eachother's data, but you can add to it, so it isn't much of a problem.

Thoonk 1.0 will have update history and incrementing revision numbers for feeds that will give you enough information to resolve your conflicts (whatever the method you choose is) and will help your app recognize and retrieve missed updates.</comment></comment></comment>


Show HN: PatrolServer – Continuous updates of the vurnabilities on your servers - <AUTHOR> http://patrolserver.com

<comment><USER2> 1.) The first thing I noticed was "delivery by mail" in the top text, and it made me concerned; any communication of potential vulnerabilities (i.e. scan results) should be properly encrypted to the recipient and authenticated from you.

2.) The site layout (css) is buggy. On older browsers, some text is positioned partially off screen on the left. Yes, supporting old browsers is a pain, but within reason, it's worthwhile.

3.) The top menu has a white text, but since the top menu stays in place when scrolling on to a white background below, the menu text becomes unreadable (white on white).

Good Luck!

<comment><AUTHOR> Thank you very much for the feedback!

We have had multiple meetings about the security concern of mail communication. And we opted (for now) to send the information by mail, just for the convenience and rapid delivery to the end user. Also in the thought that all information we scan about your server is actually publicly available. So getting the information to the owner as fast as possible is more important than hiding the information (as in security through obscurity). But as we progress, we definitely see the need to hide data and/or encrypt them in the mail. (Especially with the PHP detector)

</comment></comment><comment><USER3> How is PatrolServer's own security? Are results encrypted? Can the results be gathered (i.e. 'site X is vulnerable to exploits A, B and C)? I know that sounds like security by obscurity, but it has to be asked.

I'd much prefer to install a tool on the actual server that reports a local report. But that's hard to monetize.

<comment><AUTHOR> Good question! As you already hinted, we will not disclose all our security measures. But our platform is secured on multiple levels. First of all, all communication between you and our server is encrypted. No eaves-dropping possible. So only you known your own vurnabilities. Second because you have to verify your server, a thirdy party can't get secret vurnability data about your system. So they cannot use our system as easy tool to get attack vectors. Third we separate the scan logic and webserver logic into different servers. And only the scanner has information about passwords and secret information. So while we make our webserver very safe, the scan server is fort knox. One particular nice thing, the scanner server has no open incoming ports. So you cannot access it. Fourth the PHP dectector file will only react to our server and all data exchange is encrypted with keys only known to the scan server. The commands that have to be executed are also signed with assymetric keys, signed on an offline computer. Fifth we keep improving our security while monitoring our systems (with our service but also with other tools). Because new attack vectors are released everyday.

<comment><USER4> I would be interested in a larger writeup on that system, I like it.

<comment><AUTHOR> We will come back with some more details, but at the moment we are having an internal discussion of how many details can be published. So expect it somewhere next week.

</comment></comment></comment></comment><comment><AUTHOR> Thanks everybody! 1 day on Hacker News, 1574 uniques, 103 signups, 15% signup rate and 53% total vulnerability rate. You all have some fixing to do. If you have more feedback, let us know (or spread the love).

</comment><comment><USER5> Eh, it just seems to compare detected versions against the latest version of software. It would be nice if it said _why_ the installed version is vulnerable, links to CVEs, etc.

<comment><AUTHOR> We indeed compare versions against the latest version of the software. Though we also keep track of which versions are supported and also create an inventory with the CVEs and their risk. There are also vulnerability scanners incorporated. That is why we sometimes still show a version is outdated, while we don't know the actual version.

For now we spend more time into giving information on how to keep your system secure and make that as easy as possible. If we can present 1 solution to fix 22 cve exploits, we find that preferable to showing 22 issues that need to get fixed with detailed information about the cve. But should that data should still be findable for experts and is also shown our the data cards. (Except when the data isn't disclosed yet by the software manufacturer) So if you don't see the CVE, please tell us more details about the software that doesn't has vulnerabilities.

<comment><USER5> Okay, that sounds more worthwhile. When I did a simple test against a LAMP server, the only information I got back was "running 5.5.25, latest 5.5.27, this is a vulnerability". I assume you're presenting that info for different software.

<comment><AUTHOR> There is more logic to it. We try to detect if you use ubuntu/debian and suggest those updates. From the moment we know the exploits, we show them. Probably you're talking about PHP? For PHP 5.5.25 there are the following exploits CVE-2015-3414, CVE-2015-3415, CVE-2015-3416, CVE-2015-2325, CVE-2015-2326 and CVE-2015-3152. But none of them have information yet. (see [https://cve.mitre.org/cgi- bin/cvename.cgi?name=CVE-2015-3152](https://cve.mitre.org/cgi- bin/cvename.cgi?name=CVE-2015-3152)) So as soon as they are disclosed, the exploits will automatically be shown (within 5 minutes)

</comment></comment></comment></comment><comment><USER6> Incorrect redirect at [http://patrolserver.com/css/](http://patrolserver.com/css/)

<comment><AUTHOR> Thanks, solved ;)

</comment></comment><comment><USER7> Website is broken on mobile after logging in.

<comment><USER8> Thanks! Problem solved!

<comment><AUTHOR> Indeed the problem is solved. The mobile stylesheet wasn't loaded on production. Sorry for the inconvenience. But glad you noticed and warned us ;)</comment></comment></comment>


Show HN: CertStream – See SSL certs as they're issued in real time - <AUTHOR> https://certstream.calidog.io <comment><USER2> A bit OT, but there is a thing about CT that I keep asking me and I can't seem to find an answer: Say I use a letsencrypt certificate for my nextcloud installation on my home server. Now say that the german police is investigating on me. They call up the BSI and tell them "hey, can you please issue me a signed cert for nextcloud.y0ghur7.xxx so we can mitm this guy? And please, don't log it to the CT, so nobody will ever know"

I will probably never know, because BSI is a trusted CA in all browsers and other http clients so they don't complain (and I am not looking at what certificate the server sent me every time I open my nc page), and nobody else will ever know that that certificate was released. Am I right? So what does CT buy me?

<comment><USER3> At some point browsers will stop allowing certificates that are not logged through CT

<comment><USER2> > At some point browsers will stop allowing certificates that are not logged > through CT

Makes sense. So to be sure nobody issued a cert for one of my properties I would have to check regularly on CT logs to be sure that only certs requested by me are issued. But in that case, if someone requests a cert for one of my properties, and that cert was not requested by me, what do I do?

Do I tell mozilla and google that "someone issued cert id 4d8effdd25 for my nextcloud installation (or my forum where some rebellious users meet up sometimes) to mitm me, but it was not me". Will they belive me? And it will be probably to late anyway, because propagation to a CT log can take up to one day, so they got data on all the traffic for a whole day.

<comment><USER4> It's not part of CT, nor does it fully solve the issue, but you might also like Certificate Authority Authorization. CAA allows you to publish what CAs are acceptable for your domain via DNS. CAs shouldn't issues certificates against that. Of course that doesn't protect against a rogue, compromised or coerced CA, but it does protect against phony requests to the CA.

<comment><USER5> As you said, that only protects against CAs that follow the CA/B Forum Baseline Requirements that require they check CAA at issuance time.

If a government was coercing a CA, they'd just tell them to disable this check. If this can be proven it's grounds to start the distrust process. At the very least, they should fail their next WebTrust audit.

</comment></comment></comment></comment></comment><comment><USER6> One thing I totally dislike with CT is that literally everybody can see all the subdomains that my certificate is valid for (esp. LetsEncrypt), but also for cases where your "normal" wildcard-cert does not work - e.g. _.foo.de is covered, but because wildcards dont go beyond 1 level,_.bar.foo.de is not covered, and so everyone can see that there is one (or more) subdomains at bar.foo.de.

Let's assume an attacker finds a RCE in JIRA, Confluence or Gitlab... now everything the attacker has to do to find a list of candidates is to run a simple grep -i gitlab|jira|confluence|whatever on the CT logs, while he'd have to go the brute-force route before CT.

<comment><USER7> You should not expect a service which has been exposed to the open internet to be hidden.

If your last defense is that nobody knows the domainname, then you've lost. Not knowing the domainname shouldn't be any defense at all.

<comment><USER6> > You should not expect a service which has been exposed to the open internet > to be hidden.

Of course not, but CT dramatically lowers the bar for attackers. That's what I mean.

<comment><USER7> It doesn't really, only when you need to hide subdomains for security reasons. Which you shouldn't do.

CT raises the bar for attackers since they will be logged into the CT if they try to MitM.

Any other attack is not made easier than without CT.

</comment></comment></comment></comment><comment><AUTHOR> Hey folks, developer of CertStream here. You can read more about the motivations and implementation behind this project by visiting the announcement page ([https://medium.com/cali-dog-security/introducing- certstream-...](https://medium.com/cali-dog-security/introducing- certstream-3fc13bb98067)) on my company's blog. I'm also happy to field any questions anyone may have!

<comment><USER8> Interesting project. I was wondering if you think they are any privacy issues around Certificate Transparency, like grouping ownership of domains through the timings.

<comment><AUTHOR> Hmm, that's an interesting thing I haven't given much thought to.

I think that it would be somewhat difficult to pull off a correlation attack/leakage as the CTLs tend to dump in batches vs every poll returning new results, but I think once you remove a lot of the noise (cloudflare SNI certs, testing domains, etc) it'd potentially show some interesting patterns.

[https://github.com/CaliDog/certstream- python/blob/master/exa...](https://github.com/CaliDog/certstream- python/blob/master/examples/stat_windows.py)

This demo would be a decent starting point to that analysis if you'd be interested in toying with it!

</comment></comment></comment><comment><USER9> Interesting timing, considering the talk [0] on this very topic just uploaded to YouTube yesterday morning from DefCon 25. Basically, this is offering his observation (CTL can be used to get a real-time list of new domain names, which can be exploited), as a service.

Seems like Hanno Bõck could at least use a shout out if it was related to his work.

Either way, the talk is worth a watch.

[0] [https://www.youtube.com/watch?v=TMNeSnjZfCI&list=PL9fPq3eQfa...](https://www.youtube.com/watch?v=TMNeSnjZfCI&list=PL9fPq3eQfaaDEQqRyr-0cmF1HTuidm97j&index=11)

<comment><USER10> Shout out isnt needed, i did the same thing as Hanno over a weekend early this year. Been kicking myself since defcon that i didn't submit a talk!

Anyone who reads the certificate transparency log rfc can quickly realize whats possible.

I've also been following calidog since his first medium post, ive got my own similar cert scanner/tracker.

</comment></comment><comment><USER11> There seems to be a big list of domains streaming through in alphabetical order. First there were dozens all starting with J, now there are dozens starting with K. Looks like gemalto is going through the day's domains in alphabetical order and adding them one at a time?

<comment><AUTHOR> Yeah, it's interesting to see how certs are issued from the larger lists, since they tend to come in a deluge. This goes doubly for the cloudflare SNI certificates for their edge nodes!

</comment></comment><comment><USER12> Very cool! I love the simple JSON-over-WebSockets API and would love to see more streaming APIs like this available in the wild.

</comment><comment><USER7> This is quite neat. I'll probably look at the go-library to build something out of that.

In the meanwhile, crt.sh also offers a Atom feed of issued certificates. Definitely not realtime but also works.

</comment><comment><USER13> Sweet landingpage!

<comment><AUTHOR> Thanks! It's a few iterations in, and my designer ([http://www.jweiller.com/](http://www.jweiller.com/)) was definitely responsible for the best looking parts!

<comment><USER14> heads up the <title> isn't getting set, it just shows the URL.

<comment><AUTHOR> Thanks for the heads up! It also isn't displaying the favicon properly it seems :-/.

[https://media1.giphy.com/media/R54jhpzpARmVy/giphy.gif](https://media1.giphy.com/media/R54jhpzpARmVy/giphy.gif) TL;DR - Curse you Webpack! ::shakes fist::</comment></comment></comment></comment>