This is why sites choose to stay vulnerable to Firesheep - <AUTHOR> https://www.google.com/adsense/support/bin/answer.py?hl=en&answer=10528 <comment><USER3> This is a problem we (GitHub) are facing in a big way right now. Google Charts doesn't offer https alternatives, so almost all our users get a big "this site is going to steal all your private information" (mixed content warning). We chose to roll out SSL first, then deal with the hard problem of mixed content warnings (building ridiculous image proxies) later.

I think a lot of developers underestimate how big of an impact this warning is on users, especially on browsers like IE that throw up a dialog on every page that has this warning. Developers understand that it's not that big of a deal — but to a user, it looks like the site is full of viruses, malware and is going to steal all your bank account information.

<comment><USER4> Exactly. Browser makers (including Mozilla/Firefox to a large degree) are responsible for the fact that HTTPS hasn't become the standard protocol as it should have been years ago. It's not only the unproductive mixed content warning but also the insistence of all browsers to only accept expensively bought certificates and throw a very scary and hard to overcome error dialog if a site uses any other kind of cert. While that isn't a problem for big(gish) commercial sites like GitHub, it presents an insurmountable hurdle for private sites and small-time projects for no good reason. For most sites I don't need "secure" origin verification as badly as encryption. The lack of a verifiable server address shouldn't mean that I should be bullied to not use an encrypted connection with it. But even if the verdict is that you absolutely can't have one without the other, browser makers should AT LEAST include trusted root certs of authorities who offer free SSL certificates, too.

<comment><USER5> While your frustration is understandable, I think you're speaking from the perspective of a tech-savvy person and not the average user. If browsers began accepting all free / self-signed certificates, it would be only a matter of time before something like "Firesheep FX" came along and permitted random strangers to MITM anybody's SSL session. Some of us can notice when that happens, but most people won't have a clue unless the browser presented them with a big scary red warning.

However, I agree with you that we need some good free CAs. The difference between free and $10/year is bigger than most of us think it is. Fortunately, there are registrars such as Gandi which will give you free certificates with every domain.

<comment><USER4> > _If browsers began accepting all free / self-signed certificates [...]_

Right now, browsers are accepting any unencrypted old HTTP connection without any warning, while non-verified securely encrypted connections are actively prevented. Tech people can circumvent the block, but normal users cannot. Nor do they have any reason to because the warning they are being shown sounds like the end of the world, while any unsecured connection looks perfectly fine to them. This is something that could be done right now to make everybody more secure, at no cost, but it threatens the business model of companies like Verisign.

Nobody is suggesting that browser makers should display the much-sought-after "lock of absolute protection" icon on any random SSL connection, I'd be fine if they reserve that for paid-for-certs. I'm merely suggesting they show free (or even self-signed) certs the same courtesy as basic HTTP, the most permissive protocol of all time, instead of actively preventing users from using encryption.

I agree with you about the threat of "Firesheep FX" and believe Wifi connections should probably all use WPA2, even at coffee shops where internet access is free. The threat of MITM is real, but the attack can be made more difficult using a number of schemes, and it even includes free certs that offer way more protection than any unencrypted link ever could. Yet, we are currently encouraging unencrypted connections while actively blocking encrypted ones.

If HTTPS could have the same UI mechanisms as, say, an SSH connection I'm convinced the online world would be a much safer place.

<comment><USER6> _Wifi connections should probably all use WPA2, even at coffee shops_

if you just write the password on the wall, it defeats the purpose - <USER2> who logs in is on the same network again, just like a public network

<comment><USER7> Everyone being on the same network isn't too much of a problem. They still can't read each others traffic. See <http://en.wikipedia.org/wiki/IEEE_802.11i-2004> or <http://en.wikipedia.org/wiki/Wi-Fi_Protected_Access#WPA2>

Every device negotiates its own keys with the access point.

<comment><USER8> Very common misconception, but it's still a problem. Any client with the network password can capture the initial key negotiation, and then decrypt the client's subsequent traffic. You can enter the network password in Wireshark: <http://wiki.wireshark.org/HowToDecrypt802.11> .

<comment><USER9> If the shared key is known, it's also trivial to install a rogue access point with the same SSID and a transparent, tampering proxy without a realistic chance of anyone noticing.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> To be accurate, this is not the reason many sites choose not to go with SSL for everything. The real reason is that most sites don't _need_ to be SSL for everything.

I run a travel blogging site, where 99% of all pageviews are from random people off the internet reading people's trip reports and looking at photos. Encrypting all that traffic would do nothing except bog the site for everybody.

Every once in a great while (in terms of total traffic), somebody will log in and post something. That tiny moment could benefit from SSL, since chances are it's happening from a public internet cafe or wifi hotspot. That's the only time a user is actually vulnerable to this sort of attack, so that's when they need to be protected.

But when you look at the internet at a whole, the traffic fraction that needs protecting looks pretty much the same. When you're showing me pictures of cats with funny captions, please don't encrypt them before sending them to me just because you read something about security on HackerNews.

<comment><USER11> The thing that Firesheep brought to people's attention is that the login is not the only thing that needs to be SSL protected. The cookies you get after signing in are often sent in the clear, and that cookie is just as good as your login for gaining access.

<comment><USER12> It's not the same, because with someone's password you can completely lock them out of their account instead of just acting as them.

<comment><USER11> In a lot of systems, you can change the password without knowing the old one as long as you're logged in. Others, you can change the email address, only confirm on the new email address, and then get a password reset.

So if you really cover all of your bases, and require confirmation at every step, then the least they can do is access your data and generally impersonate you until you log out of that session (which no one does) or the session times out (which it won't, because you're still logged in as them).

It's pretty much the same.

</comment></comment></comment></comment><comment><USER13> Honestly, how many sites are aware that they are vulnerable?

It seems like you assume that because the security-oriented 0.5% of the web knows about it, the rest of the web should, too.

For most people, just making sure that their site runs at all is quite enough for them to handle, and keeping current on the latest vulnerabilities is way down on the list.

Additionally, fixing a site takes time. How long has Firesheep been out? A week? Two? You should realize that for many sites, even those staffed by very competent tech people, a month is the minimum amount of time for immediate action.

<comment><USER14> How many sites (that any of us are legitimately worried about) employ webmaster, developers, system admins or other that DON'T know why SSL/HTTPS is important? You can't honestly be giving facebook, twitter, etc a pass on understanding very basic concepts... (sniffing, http (cookies))?

Firesheep has been around for 2+ weeks now, but come on, we've all known this has been possible for forever. I'm 20, and I knew how to do this (and did) /years/ ago. I think Firesheep is just what everyone needed.

There are really good reasons why this is taking a long time and it is NOT lack of knowing that this problem exists.

That having been said, my laptop is now running a LiveCD of x2go's LTSP client and my desktop computer is running the x2go server. Very near-native performance and total security. (I trust my desktops' endpoint).

<comment><USER15> I can understand a possible flaw in a websites attempt to provide secure transfer of data, but just disregarding encrypted communication makes me question their title of web developer.

</comment></comment></comment><comment><USER16> I read the overclocking ssl posting ([http://www.imperialviolet.org/2010/06/25/overclocking- ssl.ht...](http://www.imperialviolet.org/2010/06/25/overclocking-ssl.html)) & I've been seeing plenty of follow up about how SSL is cheap and easy to scale, but I have yet to see one tutorial that describes actually implementing overclocking SSL or implementing it cheaply.

So- to the HN community, is this whole "ssl is cheap" a false meme, or does someone have actual instructions on how to deploy & implement a scalable SSL?

</comment><comment><USER17> There has got to be a sensible way around this. It seems overkill to require every pageview to be over HTTPS, even for otherwise public sites. For example, should these public discussion pages be over HTTPS on hacker news?

On my site I am planning the following: operate the login page over HTTPS and issue two cookies. One is HTTPS only and other other is for all pages. The public (non HTTPS) cookie is only used for identification (e.g. welcome messages and personalisation). However, all requests that change the database in any way are handled over HTTPS and we check to make sure the user has the secret HTTPS cookie as well. Often forms submit to a HTTPS backend and then redirects back to the public page over HTTP. Also, all account information pages (sensitive pages) will be over HTTPS.

This way, the worst that can happen via cookie sniffing is that someone can see pages as though they were someone else. In your case, this is not much of a risk.

<comment> mike-cardwell This is just dangerous. Example. If news.ycombinator.com implemented this dual cookie method. A man in the middle could intercept the page I'm looking at now, where I'm entering this comment in a textarea. They could modify the underlying form to post to the same page as the update form on the profile page, and set a hidden email field. Then when I hit the "reply" button, even though I'm posting to a HTTPS page, I'm not posting to the one I think I am, because the page containing the form it's self wasn't protected by HTTP.

I hope I explained that well enough. Mixed content is _hard_ to do right. Forcing every page over SSL prevents anyone making any modifications to any page, and is just inherently safer.

<comment><USER17> Well, I would say its not 'just as dangerous' as a man in the middle attack is harder to set up.

Good point though. Maybe this could be solved by including a unique access code with the form that is a hashed value of the user'id and the url that you are submitting to (with salting to make this unguessable). Simply check this value upon submission to make sure it matches the URL seen by the controller. That would prevent anyone rewriting a form to submit to a new endpoint.

<comment> mike-cardwell CSRF protection should be implemented even if your entire site is protected by SSL.

Also, I didn't say "just as dangerous", I said "just dangerous"

<comment><USER17> Sorry for misquoting you.

However, with proper CSRF protection your man in the middle argument is not the case is it?

</comment></comment></comment></comment></comment><comment><USER18> Browsers should use two kinds of notifications: "encryption on" (green or red) and "certificate is present" (green or red). Websites that do banking or handle sensitive information should be green/green (SSL-on with verified cert), while ordinary websites could be green/red (SSL-on with self-signed cert).

</comment><comment><USER19> Is the solution to Firesheep to have every logged in page in https? Or is this not necessary?

<comment><USER20> Any HTTP request that includes the session cookie needs to be secured, otherwise the firesheep user will be able to grab the session cookie and use it in their own requests.

<comment> mike-cardwell That is the solution to protect websites from the current iteration of FireSheep. It doesn't fix the underlying problem though. If a version of FireSheep comes out that can do MITM we might have bigger problems.

The solution to the problem is SSL on every page.

<comment><USER21> I'm not sure I'm parsing your post correctly, but as I understand it you're talking about third party websites accepting responsibility to protect you over an insecure network connection. If that's the case, then I think you're mistaken.

Certainly SSL is not required on every page, and MITM tools have been around for some time (including fairly friendly ones like Cain - <http://www.oxid.it/>). At the end of the day companies such as Facebook, Twitter et al have a moral (and in some cases legal) obligation to protect the information assets you uploaded to their systems from compromise. Likewise it is not unreasonable that you take certain steps to protect yourself.

The current version of FireSheep is a real known threat. We don't know what might be in future versions. For protecting against Session ID theft, SSL and the secure flag on cookies are the way to go. Certainly for data that doesn't need to be secure (such as static publicly available graphics), there's no need to use SSL for the majority of use cases.

The use of SSL for delivering dynamic client side code (such as HTML or Javascript) is an interesting issue, but ultimately the user has to bear some responsiblity for their own actions somewhere along the line. Not every network is insecure, not every browser has to support a zillion and one insecure means of using Javascript.

Rather than using SSL on every page and expecting the web sites to do the heavy lifting, consider not using insecure bearer networks, or some sort of means of securing insecure Internet links such as a VPN or SSH tunnel.

<comment> mike-cardwell I do think websites should try to protect their users over an insecure network connection, yes.

<comment><USER21> That's an interesting though. Where do you draw the line? Should websites protect users who don't have AV, firewalls or share user accounts?

<comment> mike-cardwell I don't think websites should protect users that don't have AV, firewalls or share user accounts no. I also don't think websites should protect users who cross the road without looking both ways. None of those things are relevant to protecting the communications between the website and the user.

If the users machine is compromised, that's the users problem. If the users machine isn't compromised, yet the website can't be accessed over an inherently untrustable network like the Internet, then the website has some flaws that it needs to deal with. SSL is a start. DNSSEC is becoming important too and I will be using it on grepular.com when Verisign signs "com" at the beginning of next year.

</comment></comment></comment></comment></comment></comment></comment><comment><USER22> While sites wait for services such as adsense to support SSL, adding a second Secure cookie and requiring on sensitive pages and to perform destructive actions can help reduce risk to users. Depending on the site, it may be OK to skip showing ads on a few authenticated pages. Wordpress implemented this in 2008: [http://ryan.boren.me/2008/07/14/ssl-and-cookies-in- wordpress...](http://ryan.boren.me/2008/07/14/ssl-and-cookies-in-wordpress-26)

This won't protect against active attackers, but is definitely a step forward and will make a full transition easier in the future, when possible.

<comment><USER23> We spent about a week trying this on GitHub. It works pretty well as long as you have no ajax requests. We were basically left with this option:

1) Lose the ajax (and spend a significant time redoing bits of the site) 2) Scary iframe hacks. 3) SSL Everywhere.

I feel like we made the best choice (I certainly don't mind removing any chance we'll have adsense any time soon :). It cleaned up a lot of logic based around determining which pages were important enough to require SSL (admin functions, private repos, etc).

It's brought on some other issues though. Safari and Chrome don't seem to properly cache HTTPS assets to disk, for one. This is an old problem: [http://37signals.com/svn/posts/1431-mixed-content-warning- ho...](http://37signals.com/svn/posts/1431-mixed-content-warning-how-i-loathe- thee) . I'm not too worried about increased bandwidth bills on our end, I'm worried about having a slower site experience. We're also seeing users complain about having to log in every day. Are browsers not keeping secure cookies around either?

</comment></comment><comment><USER24> An alternative? : <http://www.tcpcrypt.org/>

<comment><USER25> > Tcpcrypt is opportunistic encryption. If the other end speaks Tcpcrypt, then > your traffic will be encrypted; otherwise it will be in clear text.

I like that, as opposed to requiring users to have to install some plugin before they can even talk to the server.

<comment><USER26> You may like it but it really reduces the security. A man in the middle can make it look like you don't speak Tcpcrypt by manipulating the first few packets of a connection. It's the same issue as mixing HTTP/HTTPS - the HTTP parts leave you vulnerable. If encryption is not mandatory then it might as well not be there.

<comment><USER24> Which requires active attacks, ie MITM. And _stopping_ MITM is essentially impossible - the best you can do is use our current CA setup, which assumes the original download of the certificate you got wasn't hijacked.

In preventing _passive_ listeners, like Firesheep, this would work 100% effectively anywhere it can work at all. The only way it reduces security is in making people who don't understand MITM attacks feel they're safer than they are - at absolute worst it's like you don't have it installed.

</comment></comment></comment></comment><comment><USER27> Is there an alternative to HTTPS?

</comment><comment><USER28> It's kind of obvious that a major reason is IPV4. Getting another IP so you can run both SSL and plain HTTP is only going to get harder.

</comment><comment><USER29> Is there any reason that the web server can't set up an SSL reverse proxy to fetch the adsense ads ? (Obviously cookies would have to be passed-through...)

It would cost you more bandwidth, but then there's no annoying warning message about mixed content.

</comment><comment><USER30> What's the deal with that page not having any stylesheets? It looks completely broken. Am I the only one seeing this?

<comment><USER31> Turn off adblock and reload the page.

</comment></comment><comment><USER32> it's one of the reasons. probably not the only one.

</comment><comment><USER33> Facebook and Twitter don't run Adsense, it's mostly run on content sites that don't require you log into them.

<comment><AUTHOR> I don't understand your argument here--are you saying we shouldn't mind if the vulnerable sites aren't Facebook or Twitter?

<comment><USER34> hes saying that most sites that use adsense do not require a login, thus do not need https. he is somewhat correct, but not enough for google to just ignore this issue.

<comment><AUTHOR> Most content sites have login systems that people use to customize their experience, post comments or upload content, etc. Millions and millions and millions of people are logged into content sites and are vulnerable to this attack.

Also, I disagree on the premise that adsense is mostly used on content sites. It's used on all kinds of websites.

<comment><USER35> Most notably would be the oodles of forums out there that are ad supported and require logging in.

</comment></comment></comment></comment></comment><comment><USER36> Irrelevant contrarian opinion that adds nothing to the debate, but indicates with certainty I am more interested in being pedantic and scoring points than having a useful discussion.

<comment><USER37> Successfully executed.</comment></comment>


The way Microsoft patched a recent bug raised some eyebrows - <AUTHOR> https://www.bleepingcomputer.com/news/microsoft/microsoft-appears-to-have-lost-the-source-code-of-an-office-component/ <comment><USER3> "A company like Microsoft [...] would never deem manually binary editing as acceptable." is complete bullshit speculation. The guy who has analysed the bug report must be good at assembly language. Maybe he has developed the binary patch very quickly as a proof of his analyse. When this analyse was hand over to development team, they had two options: spending time to setup the development environment with the correct version of sources and tools (error prone), performs again the development using c++, launching all the non regression tests. The other option is to do nothing and bet this binary will never need again to evolve. This is cheaper and the risks seem smaller. IMHO, this beautiful patch shows that Microsoft is open minded and choose smart options instead of bureaucratic ones. I applaud.

<comment><USER4> Came here to write the exact same thing. "The only way this happened is if Microsoft somehow lost the source code of a long forgotten Office component." is completely false speculation/generalization. It is _not_ the _only_ way.

<comment><USER5> More than likely it was a project that only builds in a long-discontinued non- conforming C/C++ compiler that only runs on Windows 95. When I was working at MS very recently I was surprised to learn that all software affected by the Sun lawsuit, way back in 2001, was still strictly verboten, even internally. All builds of Windows 95 through XP SP1 were absent from the internal builds and products shares just as they are from MSDN. I don't believe Microsoft would have somehow lost this source code as the article speculates - <USER2> likely they reasoned that the effort needed to get a build environment set-up and working would take more time than finding someone on the team who really knows x86 assembly well enough to manually fix the bug - and there are plenty of people like that on campus.

</comment></comment></comment><comment><USER6> I did something like this for an old game called Age of Wonders.

[https://github.com/int19h/aow-patch](https://github.com/int19h/aow-patch)

[http://steamcommunity.com/app/61500/discussions/0/2154397748...](http://steamcommunity.com/app/61500/discussions/0/215439774855125498/?tscn=1495150493)

Originally it was a fix for a bug that would show up when running it under Wine. Now the game is re-released on Steam and GoG, and same thing came up on regular Win10.

The funniest part is, I don't even remember what the patch does - I made it 12 years ago! I remember spending a lot of time reading disassembly, trying to figure out what's going on, and eventually figuring just enough to plug the immediate hole. Then I posted the binary diff on a forum, and subsequently forgot all about it, because the bug went away in some Wine update shortly thereafter.

[http://aow.heavengames.com/cgi- bin/forums/display.cgi?action...](http://aow.heavengames.com/cgi- bin/forums/display.cgi?action=ct&f=1,4694,,150)

When I dug that diff out, it made zero sense to me. So now not only there's no source for the game, but I've "lost the source" for the patch, as well - I'd have to reverse engineer it all again to figure out what it actually does. ~

<comment><USER7> > _...because the bug went away in some Wine update shortly thereafter._

So what you're saying is that Wine is more Windows compatible in the case of AoW than Windows itself? :- )

<comment><USER8> Wine makes a point to reproduce all unfixed Windows bugs, because notable software was written assuming they exist. I don't envy Wine maintainers!

</comment></comment></comment><comment><USER9> I'm having a senior moment here, but wasn't it common to include space in instruction-space and initialized data space, designed to permit binary patching by jump table to (currently) un-used pre-assigned regions?

I thought I'd even seen reference to support in compilers to pre-assign such spaces.

If (for instance) this is a product acquired from outside the Redmond campus, run successfully unpatched since B.C. and now needing work, it might actually be more effective to do what they did instead of reverse-compiling code, re- implementing, the whole shebang.

Security nightmare? Sure. Unsafe? depends. NASA pay old-timers the big bucks to keep things working out in space, which probably involves processes which aren't morally far removed from this.

<comment><USER10> > I'm having a senior moment here, but wasn't it common to include space in > instruction-space and initialized data space, designed to permit binary > patching by jump table to (currently) un-used pre-assigned regions?

Are you referring to Microsoft's MOV EDI, EDI at the beginning of every function? That's a hot-patch point.[1]. I think that was more for in-memory patching to get around the problem that if you replace the current code, you might run afoul of some thread actually executing it. That wouldn't necessarily apply to the binary/DLL, which when loaded from scratch wouldn't need any such hot-patching.

1: [https://blogs.msdn.microsoft.com/oldnewthing/20110921-00/?p=...](https://blogs.msdn.microsoft.com/oldnewthing/20110921-00/?p=9583)

</comment></comment><comment><USER11> Contrary to the tone of the article, I wish more of the small and very-likely- security-related patches would come in the form of "actual patches" like this --- and also distributed as actual diffs instead of whole-file-replacements, since this greatly improves the user experience; those of you who have experienced Windows Update taking tens of minutes to download and apply updates may be disappointed and irritated to realise that even if the actual changes are tiny, as in this case, applying the update involves downloading the entire file again. If there are 10 updates which affect a total of 5 KB in a 20MB file, for each one that's installed, you end up redownloading the whole thing, or 200MB of bandwidth used (wasted) to make 5KB of changes. It's not even smart enough to jump to and download just the latest version of all the selected updates.

Another advantage of the small and localised changes, besides rapid and efficient application (and reversion), is the ability to do it to all in- memory copies of the file in addition to the one on disk, eliminating the need to reboot.

 _A company like Microsoft that has solid and complex software development and security practices in place would never deem manually binary editing as acceptable._

Perhaps what we need is better tooling to make this process easier, and I guess in general a more "bidirectional" view of binaries vs. source --- small changes in source should correspond to proportionally small changes in binaries, and also proportionally small updates for end-users.

The current "binaries are sacred and shall only be the outputs of long and complex build processes" attitude is neither efficient nor optimal from the perspective of the end-user experience, or even developers sometimes --- for example, changing a one-letter typo in a string constant and testing the change should not involve hours of compilation, deployment, and installation, but be closer to the minutes it would take to do it in a hex editor.

I posted more on this topic recently: [https://news.ycombinator.com/item?id=15729874](https://news.ycombinator.com/item?id=15729874)

<comment><USER12> Do patches as binary diffs get us most of the way there?

<comment><USER13> They can, but you need the right tooling to make sure small changes don't dramatically change your binaries.

[https://dev.chromium.org/developers/design- documents/softwar...](https://dev.chromium.org/developers/design- documents/software-updates-courgette)

</comment></comment></comment><comment><USER14> Can you imagine the panic they must have felt when they realized that there was security flaw in code that hadn't been compiled in 17 years, that they didn't even write, they couldn't find... and they were responsible for fixing?

I would love to read the post mortem.

<comment><USER15> Probably not much panic, they have people who do that kind of thing all the time. Ever read Raymond Chen's blog The Old New Thing[0]?

[https://blogs.msdn.microsoft.com/oldnewthing/](https://blogs.msdn.microsoft.com/oldnewthing/)

<comment><USER16> Even for people who do not spend much (or any) time on Windows, this makes for _very_ interesting (and entertaining) reading.

</comment></comment></comment><comment><USER17> This was submitted from the original source 3 days ago, with previous discussion: [https://news.ycombinator.com/item?id=15720923](https://news.ycombinator.com/item?id=15720923)

</comment><comment><USER18> Previous discussion: [https://news.ycombinator.com/item?id=15720923](https://news.ycombinator.com/item?id=15720923)

</comment><comment><USER19> I bet getting this to compile was more of a headache than manually patching it. This component - being old itself - probably has tons of dependencies on even older libraries. Code in its teens would need a lot of cleanup to be built (and work) with a modern optimizing compiler and recreating ancient build environment isn't simple either. They did the cost/benefit analysis before fixing a bug.

</comment><comment><USER20> I can think of 10 reasons why other than lost the source code. How about broke the build.

<comment><USER15> Microsoft didn't write it, it is licensed from Design Science (it is just a light weight version of MathType), so it is likely they never had the source code.

<comment><USER21> exactly that.

</comment></comment></comment><comment><USER8> If it's a single component, why don't they just rewrite it ? A company like Microsoft should have the resources. It might be a lot of work, but can they be sure there are no more bugs ? Is keeping a rotting, black box binary responsible ?

<comment><USER22> Compatibility. Office doesn't even use that binary anymore, but still ships it to avoid breaking other apps that depend on it. They wouldn't risk a rewrite.

<comment><USER23> Confirmed. Sometimes the Office team didn't even write the component, but has a dependency on it. When I was on the FoxPro team, Windows decided that it didn't need to ship the Fox ODBC driver anymore because they didn't think anything was using it (use OLE DB and ADO instead; I think they were kind of cleaning house, too), except those things that already shipped with driver. Someone forgot to tell Office, Office stuff started blowing up, and thankfully it was caught in an internal build. Office engineers probably didn't even know who we were, let alone have source code, and Microsoft was a hell of a long way from any kind of standardized build system that another group could understand. Thus it ended up on our plate.

So I whipped up a dummy driver that Windows could ship that just NOP'ed every ODBC call with a text box that said "go download the Fox driver from...". For all I know, it still ships in Windows today ( _VFPODBC.DLL_ , IIRC).

But even that simple task wasn't simple when you're working at Windows scale. First was even getting it checked in, and check-ins at the time were more tribal knowledge than documented. A helpful Windows engineer documented what they knew. But it turns out that there were a lot of assumptions made that were valid if one were on the Windows team, but not if you're an external group. And thus is how the Windows build got broken. Breaking the Windows build is a little different than it might be on your team when the build gets broken. Yeah, maybe you give someone some shit, maybe you make them wear the "I Broke the Build" shirt for a day (don't do that, BTW). But when the Windows build gets broken, emails go out, and war rooms with VPs happen. And that's how I got to meet Chris Jones. He asked why Windows had 1200 testers sitting around doing nothing. Fingers begin to point at the strangers in the room. We said that we followed the directions given to the letter, but there was no formal process for Windows to take check-ins from external groups, so here we are. We were dismissed, and a few weeks later a document started going around on how to check-in to the Windows repo. It was probably a month or more from the time I wrote the dummy (which probably took all of an hour or two) until it showed up in internal builds.

So, yeah, I can kind of see why there might not have been any volunteers to go modify and rebuild a seventeen year old component. Put a two byte patch on the thing and call it a day.

</comment></comment></comment><comment><USER24> It seems sad to me that the same people who pretend to be "hackers" are puzzled by a situation like this one.

Binary patches are a perfectly normal thing, may it be from Microsoft or a random company.

</comment><comment><USER25> I'd imagine that it might be easier to directly patch the existing .exe rather than try to build it from source written in 2000 or earlier.

</comment><comment><USER26> One way or another it's MS failure. Either they don't have a proper environment to build every artifact from its source or they just lost sources. That's what happens when you're trying to cut the costs.

<comment><USER22> It was an outsourced component, they probably never even had access to the source code.</comment></comment>


Continuing frequency deviation in the Continental European Power System - <AUTHOR> https://www.entsoe.eu/news-events/announcements/announcements-archive/Pages/News/2018-03-06-press-release-continuing-frequency-deviation-in-the-continental-european-power-system.aspx <comment><USER2> The short version of the story is that grid electricity in Europe is produced by rotating the turbines 50 times a second and that's where the 50Hz AC electricity comes from.

Apparently to match the demand at given instance electricity plants rotate their turbines a bit faster or a bit slower instead of switching a complete plant on and off[edit: this is not exactly right, see the reply below], which results in slight deviations from the 50Hz standard but it is fine as long as it stays between the limits.

At the end of the day, as the demand increases and decreases, the average frequency would be 50Hz and engineers took advantage of that fact to create clocks that may not be accurate to the second but accurate on average. How do they do that? They count the change in the electricity and assume that 50 changes are exactly 1 second.

Unfortunately, due to political issues in the Balkans, the grid was undersupplied or oversupplied for a prolonged period and this created a deviation from the average of 50Hz and the clocks that depend on this average to be 50Hz also lost accuracy that currently amounts to 6 minutes.

<comment><USER3> > Apparently to match the demand at given instance electricity plants rotate > their turbines a bit faster or a bit slower instead of switching a complete > plant on and off, which results in slight deviations from the 50Hz standard > but it is fine as long as it stays between the limits.

This is slightly imprecise. It actually works the other way round: the demand is "communicated" through the network by slight changes in the frequency.

This can be described best for the example of steam-based generators, as they are typical for the majority of coal/gas/oil/nuclear power plants. When demand for electricity grows, for example because a new factory is powered up, the turbines in the power plants are experiencing larger forces attempting to slow them down, starting with the power plant closest to the new factory.

The speed (and thus the frequency) is constantly measured, and if it drops, the steam valve is opened just a tiny bit more to counteract the drop. If this is not possible because the power plant is already at maximum output, either other power plants have to take the load (and the electricity is then routed through the network to the factory) or additional power plants may have to be added to the network.

The important thing is: the frequency change is a means of communication, it originates from demand changes, and because the entire network is kept in sync, it works to communicate demand changes across the entire network. Power plants providing energy to a sub-part of the entire network always attempt to run their turbines with the exact same speed of 50 rotations per second, which should ideally always conform to the frequency in all other sub-networks that a specific sub-network is attached to.

If just one of these sub-networks does not perform its duty of re-establishing 50 Hertz by powering up their own plant output, the rest of the network must either channel enough power into this sub-network to allow its local generators to re-establish 50 rotations per second by taking some of the load off of them or - if that is not possible or decided against - must also deliberately drop their frequency (and thus the rotational speed of their generators) to match the sub-network that is deviating.

The only alternative to this would be to drop off the deviating sub-network entirely, but that is usually only done in extreme cases of deviation.

<comment><USER4> I have a question about the power grid I've never gotten a satisfactory answer to. I hope someone here can help.

If the USA's power grid is about 4,000,000 meters wide, and the electric signal propagates at roughly the speed of light (3 x 10^8 m/s) ([https://physics.stackexchange.com/questions/358894/speed- of-...](https://physics.stackexchange.com/questions/358894/speed-of-light-vs- speed-of-electricity)), then it will take roughly .013 seconds for the electrical signal to reach from one station to another. This is a lot out of phase! (the period of 60 Hz is .016 sec) How is this managed?

If two generating stations on opposite ends of the country are both contributing to the electrical signal, how is this time lag accounted for? I have two guesses:

1\. The power network is "mapped" such that there is one central generating station and all other generating stations are time lagged based on their graph distance from this generating station

2\. All stations try to generate simultaneously, with their output signals interfering with eachother to a certain degree. This might not matter much because their local loads dampen the strength of their signal.

Either way seems to introduce inefficiencies.

<comment><USER5> Synchronisation is purely local. They are, from the point of view of a sky observer, out of phase by the amount you say, but that doesn't matter. Each generator need only sync up at the time of connection.

Imagine you are on half a tandem bicycle. You are in a room with the chain of the tandem vanishing beyond the wall. You can't see, but clearly someone else is pedalling the thing as the pedals are rotating. Getting on it is tricky, as you have to rotate your feet at the right speed, but once you're on you can sit there and let it carry them round - or you can start applying pressure through your feet to do work and accelerate the chain system.

At this point you discover the thing on the other end of the chain is not a person but a 60Hz synchronous motor-generator. Congratulations, you are in sync with the grid and (when pushing on the pedals) contributing power.

(A corollary of this is that if your grid connection is down or the grid is split into two pieces, you can't start up again until you get a grid input to know you're in phase. See "black start" for details on this.)

<comment><USER6> Okay, but what if you are at point A, and connected to points B and C, which are 1000 km away from you and also 1000 km away from each other, and connected to each other. Now if you would synchronize with B and C, then seen from the sky B and C would be in sync with each other because they're at the same distance from you. But how would B and C then transfer power over the 1000 km link between each other without phase problems?

<comment><USER7> Remember that power stations do not generate the same single phase signal that you tap into in your house. They generate three phase AC current. They also don't link up directly: there are transmission substations in between.

<comment><USER8> This is what I wondered, is network topology carefully arranged to avoid triangular connections, or 'loops' where a generator could interfere (indirectly) with itself?

<comment><USER9> Nope! They're connected in triangles all the time. I was stuck on this phase synchronization for a while, but really it's just the local phase lead or lag that is required.

Think of the phase as a distributed signal that indicates the current state of the local system taking into account all connected generators. It's weird but true!

<comment><USER6> Can you elaborate? It is still unclear to me.

<comment><USER9> I may be explaining it wrong, but here is a reference, particularly 22.8:

[https://www.scribd.com/doc/118169687/Network-Protection- Auto...](https://www.scribd.com/doc/118169687/Network-Protection-Automation- Guide-NPAG-2011-alstom)

This makes it sound more complex than what I learned when I last tried to figure it out =)

They seem to be describing that where two grids meet they go through a synchronizer that disconnects one or the other if they aren't in sync, so implies manually disconnections throughout the network as needed locally.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> As a community of mostly engineering-minded folks, I hope we can take this as a warning to not make our mission-critical systems depend on inputs that are merely convenient side-effects of third-party systems provided free of charge and without obligation. I mean, it doesn't matter how many people post comments here saying things like "50 Hz AC is more accurate than quartz crystal for timekeeping", the fact is that no external agency can take over the action of your quartz crystal and force your clocks to slow down or speed up, whether intentional or not, as an effect of their political actions.

If you want a free source of mostly-reliable oscillations, and it doesn't have any real impact if it falls out of sync, sure go ahead and use something like this. But in the case of time, if you want to sync a clock in Western Europe with something that is free and non-internet based, you're much better off using DCF77 [0], which is specifically designed to synchronize time and comes with specific uptime and quality promises. Still ultimately susceptible to political actions, of course, in extreme cases, but at least you know its _primary_ purpose is to transmit time information, and it is in the control of only one (relatively) stable government as opposed to being subject to the unpredictable changes brought about by interactions between multiple interconnected systems.

DFC77 doesn't solve the same problem as using AC as an oscillator, of course, since you still need an oscillator to keep your clock going during the downtimes that are permitted to the radio signal.

[0] [https://en.wikipedia.org/wiki/DCF77](https://en.wikipedia.org/wiki/DCF77)

<comment><USER11> DCF77 also tells you what time it is, AC can only keep whatever time's been set in sync. The equivalent would be to include a 50-cent crystal oscillator instead.

On a only barely related note, I've been playing with the idea of implementing a NTP synced clock on an ESP8266. It'd wake up, join wifi and sync NTP every few hours, then keep time on its internal crystal in between. It should be a good deal cheaper, work anywhere in the world, and have better indoor coverage (I have a DCF77 clock in an interior bathroom in London, and it never syncs. Kind-of defeats the purpose that I have to move it to a south-facing room for the day to get it to sync).

<comment><USER5> The UI for entering the Wifi password is always the biggest problem with these things. The Dash buttons have an interesting solution involving audio.

<comment><USER12> You could try a USB SATA drive with a FAT16 filesystem containing "/wifipass.txt". USB-1.0 low speed is 1.5Mb/s and while I can't find documentation on ESP8266's GPIO switch speed, it's CPU is 40MHz, so bitbanging should be feasible.

</comment></comment></comment></comment><comment><USER13> My favourite story about grid frequency involves Laurence Hammond, who would go on to create the Hammond organ. He was a designer of AC electric motors, and when he formed his own company, among their first products were luxury AC- powered clocks. The trouble was that they didn't keep very good time, because of fluctuations in the grid frequency.

He solved this problem by making gifts of his clocks to executives of the local power companies, and as if by magic, the timekeeping of all his clocks soon improved.

</comment><comment><USER6> This map of the European power system is linked from The Featured Article: [https://www.entsoe.eu/Documents/Publications/maps/2017/Map_C...](https://www.entsoe.eu/Documents/Publications/maps/2017/Map_Continental- Europe-2.500.000_400dpi.pdf) [complex pdf, doesn't render well in some viewers].

Several fascinating points:

\- Denmark is split between the Nordic (Sjaelland and Lolland) and continental systems. Anyone know why?

\- Even North Africa is synced to Europe.

\- A corner of northeast Poland is fed by Belarus.

\- Cyprus is disconnected, and northern Cyprus apparently has no network to speak of.

\- You can see the DC links to the other systems, and e.g. a back-to-back converter at Alytus, Lithuania, that isolates the European from the ex-Soviet system. But I see no such arrangements at the Belarusian and Ukrainian borders, or at the borders with Syria and Iraq. Is this just missing information, or are these countries also synced in some way?

<comment><USER14> For some reason I'm having trouble opening the PDF, both in Firefox and with evince on Linux.

<comment><USER15> This is really heavy pdf. Try llpp or mupdf.

<comment><USER14> mupdf worked like a charm! Any idea what makes it so performant (or evince so little)?

<comment><USER16> mupdf is, in my experience, much faster than poppler for complex documents, but mupdf-based viewers generally offer less features than those employing poppler's services (like Okular). These two things might be related.

</comment></comment></comment></comment></comment><comment><USER17> Trying to find more details on the specific political issue (ie, not the regional baseline, but the power issue in particular).

Best I've got is an offhand sentence that Kosovo is using more power than it produces and Serbia refuses to balance that consumption.

[https://www.platts.com/latest-news/electric- power/london/kos...](https://www.platts.com/latest-news/electric- power/london/kosovo-serbia-imbalance-impacting-european-system-26903978)

<comment><USER18> I found some more information: [http://www.balkaninsight.com/en/article/government-pays-a- mi...](http://www.balkaninsight.com/en/article/government-pays-a-million-euro- for-electricity-bills-in-northern-kosovo-03-06-2018)

 _Since the end of the Kosovo war in 1999, the four northern Serb-majority municipalities have not paid Pristina for their energy consumption.

To make up for the shortfall, people from other areas of Kosovo had a percentage added to their bills to pay for the north's electricity.

In December, the Energy Regulator's Office announced that electricity bills will be reduced by 3.5 per cent as consumers will no more cover the cost of the four municipalities' power as they have done for the past 19 years._

<comment><USER19> I imagine everyone was Bitcoin mining there and the bill was going thru the roof.

<comment><USER20> Would you believe me if I told you that because of the legal situation (Northern Kosovo does not pay any electricity), there have been tons of crypto mining operations that have sprung up. There are stories of Russians coming all the way to set up warehouses full of mining rigs and generate cryptocurrency using free electricity.

Source: I live here.

</comment></comment></comment></comment><comment><USER21> Wow, I didn't know continental Europe is a single synchronized area. Here in the Nordics (Finland) our grid is separated from the neighbours via DC stations to eliminate the need to synchronize with everyone. Here you can see our grid info: [https://www.fingrid.fi/sahkomarkkinat/sahkojarjestelman- tila...](https://www.fingrid.fi/sahkomarkkinat/sahkojarjestelman-tila/) \- "Sähköverkon aikapoikkeama" is actually the cumulative difference between the 50Hz reference and the actual frequency in the network. When that stays zero, AC clocks keep their time.

edit: I stand corrected, we are in the same frequency domain with Sweden and Norway. I must have confused myself with the DC submarine cables under the Baltic.

<comment><USER22> That's interesting! According to this map [0] the AC connections are only between Sweden, Norway, Finland and a large part of Denmark. At least under the assumtion that the frequencies are separated in the "converter station back-to-back" in Vyborgskaya, Russia.

[0] [https://www.entsoe.eu/Documents/Publications/maps/2017/Map_N...](https://www.entsoe.eu/Documents/Publications/maps/2017/Map_Northern- Europe-3.000.000_400dpi.pdf)

<comment><USER23> "Vyborgskaya" sounds like "station of Vyborg" and not a separate toponym.

</comment></comment></comment><comment><USER24> French grid data from RTE is available from [http://www.gridwatch.templar.co.uk/france/](http://www.gridwatch.templar.co.uk/france/)

I took the frequency data from the last 1200 days, and calculated the cumulative drift. The raw drift graph, assuming 50Hz is nominal, is here:

[http://nrg.cs.ucl.ac.uk/mjh/drift-raw.png](http://nrg.cs.ucl.ac.uk/mjh/drift- raw.png)

This seems bogus; there's a constant drift.

If I compensate for that drift by assuming the nominal grid frequency is 49.9972Hz, the data looks a little more sane.

[http://nrg.cs.ucl.ac.uk/mjh/drift.png](http://nrg.cs.ucl.ac.uk/mjh/drift.png)

You can clearly see the 6 minute deviation this year - it's pretty striking, and certainly nothing this drastic has happened in the last three years. But what's with the positive offset over the previous year? Not entirely sure how much I trust the RTE data.

<comment><USER25> For what it's worth, I have an alarm clock connected to the French grid, and until last month it's been perfectly on time, so assuming it is tuned to 50Hz there is probably something wrong with the source data.

<comment><USER24> Agreed - without knowing more about the data provenance, all I can show here are deviations from the long-term trend. It's enough to see the recent problem clearly, but I wouldn't trust the increase after day 900 to be real. Most likely something changed in how they report the data around that time.

</comment></comment></comment><comment><USER26> I am almost 100% sure that this is, in part, caused by the recent crypto mining surge. Or at the very least was a catalyst for something that's been cooking for 20 years now.

I have no evidence to back this claim, only anecdotal info as I know some people in the area: The Serbian minority in the north does not pay for electricity in Kosovo due to a long standing disagreement with Pristina. The Kosovo government so far managed to find ways to compensate for that, either through loans or by charging more people in other areas.

However, I'm guessing, the bills skyrocketed when the crypto mining craze started so Pristina decided to stop footing the bill.

I've heard of thousands of mining rigs being set up in the area, largely due to the free electricity. So now there's no more free electricity but there is a huge demand and nobody is paying. Hence the deviations.

[http://www.balkaninsight.com/en/article/government-pays-a- mi...](http://www.balkaninsight.com/en/article/government-pays-a-million-euro- for-electricity-bills-in-northern-kosovo-03-06-2018)

edit: wording

</comment><comment><USER27> Two days ago, I noticed that my alarm clock and the oven clock where both about 5 minutes late and I was about to create some conspiracy theory about someone manipulating the radio controlled clocks as 5 minutes deviation in the same direction was too much of a pattern to be a coincidence. But then I noticed that my wrist watch, which I hadn't changed since 3 years, still had the correct time. So I doubt my own pattern recognition abilities and changed the clocks.

When I read about the power shortage today, I was kinda relieved to get the confirmation, that my own pattern recognition was very fine, but that my theories where just not elaborate enough :-D

<comment><USER28> Wow, I had the same! Alarm and oven clocks were off by five minutes from my phone. How strange.

</comment></comment><comment><USER29> I was curious how the "normal" P=IV formula you learn in school worked here, since I wasn't understanding how frequency could possibly change the amount of power outputted. The linked article ([https://www.swissgrid.ch/swissgrid/en/home/experts/topics/fr...](https://www.swissgrid.ch/swissgrid/en/home/experts/topics/frequency.html)) explains the reason: a decrease in frequency means that the generators are actually turning slower, so there's less power generated overall.

<comment><USER30> Also remember that P=IV is the equation for DC, not AC.

<comment><USER29> Yeah, yeah, you now have a couple √2's because you're dealing with RMS voltages and currents, and a cos(θ) for the phase angle, but when it's all combined it ends up differing by some constant factor.

<comment><USER31> Well, no, not if the voltage for example doesn't stay constant nor changes with a constant period, because then P=IV becomes an integral that doesn't simplify like we are used to.

<comment><USER29> I was under the impression that voltage was mostly constant most of the time, except for brief periods where it was switched to keep up with/scale back to match demand. Assuming constant voltage should be a good approximation in this case, right?

<comment><USER32> Imagine a stone being lifted. The weight of the stone is the current, and the height it is lifted to is the voltage.

Both the weight of the stone and the height have an impact on the energy required to do the lifting.

Now imagine you are lifting and lowering it repeatedly. It should be intuitive that doing so 50 times a second requires more power than doing it 25 times a second (One being more lifts than the other, in the same time period).

<comment><USER30> Things are a bit more complicated than this. In your toy system a stone that stays still (DC) would have no power (potential energy), which is false. Also in your toy system m is constant. In general for AC you have both V and I oscillating.

As a matter of fact the average DC power does not depend on frequency.

You could posit a system where g is alternating between positive and negative values and the h reacts to it (phase-shifted). You'd quickly come the conclusion that the potential energy (averaged over a period of oscillation) does NOT depend on the frequency. (The kinetic energy does, but that's where your analogy breaks).

<comment><USER33> > In your toy system a stone that stays still (DC) would have no power > (potential energy).

This is false. In the toy system, weight is current, height is voltage. A stone that stays still has _constant_ voltage, not _zero_ voltage. Thus, it would have a 'power'.

What the system lacks is to define the stone as a capacitive load. Then it would sorta make sense.

It is a hypothetical system, so you can only reason about the aspects the author defined. Tying potential energy in the toy system to real-world potential energy doesn't work.

(Btw, potential energy is not power, it is work. Power is work _over time_.)

> As a matter of fact the average DC power does not depend on frequency.

Uhm. "DC power" stops existing if the frequency ≠ 0, so in that sense it does depend on frequency.

It's true that power itself is not frequency dependent. However, any load is, as reactive losses (parasitic or not) are a function of the frequency. As the power is a function of the load, power ends up being directly tied to the frequency.

(A resistive load cannot exist outside of a perfect DC system, so reactive loads _will_ exist).

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER34> I find stuff like this extremely interesting.

It's incredible how connected and interdependent we have become as a society and how we only notice this if things start to break (or at least run out of spec); even if it only impacts such supposedly minor things as a clock on a baking oven.

<comment><USER35> I'd love to see a gargantuan DOT file mapping out all the world's supply chains, across all industries and economies. I wonder how many 'only one source in the world' (similar to the 1970's chip saws[1]) lynchpin nodes there are...

[1] [https://www.theregister.co.uk/2011/03/22/japan_quake_semi_is...](https://www.theregister.co.uk/2011/03/22/japan_quake_semi_issues/)

<comment><USER36> Here's something similar for >110kV powerlines:

[https://openinframap.org/#4/47.96/21.01/Power](https://openinframap.org/#4/47.96/21.01/Power)

</comment></comment></comment><comment><USER37> > The missing energy amounts currently to 113 GWh. The question of who will > compensate for this loss has to be answered.

This seems to be the primary issue, not the minor drift in cheap clocks.

Also, the press release says nothing about "energy war" so that kind of hyperbole does not add anything of value.

What is going on here is akin to someone tampering with their electrical meter, except done by a state electrical company, so on a somewhat grander scale.

1 GWh = 1,000,000 KWh and in Serbia/Kosovo the price is around €0.07/KWh so the retail amount of this theft is around €8 million. It is significant enough to make an issue of it, but not really a big deal.

[http://ec.europa.eu/eurostat/statistics- explained/index.php/...](http://ec.europa.eu/eurostat/statistics- explained/index.php/Electricity_price_statistics)

</comment><comment><USER5> The UK's [http://gridwatch.templar.co.uk/](http://gridwatch.templar.co.uk/) keeps track of frequency, although not cumulative drift. The UK is DC-coupled to the rest of Europe.

<comment><USER38> [http://gridwatch.co.uk/frequency](http://gridwatch.co.uk/frequency) shows a monthly average of 49.999 for February for the UK National Grid

<comment><USER39> I checked for France : average frequency is 49,99456201 Hz in February with one measurement every 10 seconds (source : [http://clients.rte- france.com/lang/fr/visiteurs/vie/vie_freq...](http://clients.rte- france.com/lang/fr/visiteurs/vie/vie_frequence.jsp) ).

Same info for November 2014 : 49,99994706 Hz.

</comment></comment></comment><comment><USER40> I was just yesterday changing my oven clock. Interesting to find out why here on HN. I wonder what critical systems (besides my oven) will start misbehaving. Traffic lights, maybe?

(edit:Apparently, the emergency broadcast system.)

<comment><USER41> I assume all mission-critical clocks get their tick from a quartz crystal. Clocks running on mains frequency is a very cheap cost cutting measure.

<comment><USER42> If it's mission critical then -- in Europe -- it gets the time from DCF77. It uses an atomic clock to emit a time signal (actually, it uses three and only uses their output when two out of three agrees). It is also compared to further atomic clocks and corrected as necessary.

Even many consumer grade clocks will sync to the radio signal once a day and use a quartz otherwise.

<comment><USER41> I suppose that in the case where the mission itself is critical as well, then indeed it's going to have an atomic clock signal.

</comment></comment></comment></comment><comment><USER43> Ah now I understand why the clock on our microwave oven is running behind.

I am sure I set it correctly after the end of the latest daylight savings period (which they also should forbit, different subject).

It seems getting worse over time. It now runs behind for guess what.. 6 minutes ;)

</comment><comment><USER44> "Clocks in Europe that depend on the grid frequency" which usually means alarm clocks.

I guess a lot of clocks use a crystal now so they're less dependent on the grid frequency.

Also don't buy an alarm clock in the US and bring it to Europe and vice versa, unless it is user adjustable (or you are comfortable with a soldering iron)

</comment><comment><USER45> I preferred the old title. It felt fantastically cyberpunk.

</comment><comment><USER46> ELI5 : [https://www.reddit.com/r/europe/comments/82h83b/disagreement...](https://www.reddit.com/r/europe/comments/82h83b/disagreements_beteeen_kosovo_and_serbia_cause/dvafs2v/)

</comment><comment><USER47> We've updated the title from "Clocks in Europe are slowly deviating because of an energy war in the Balkans" to that of the article.

</comment><comment><AUTHOR> The delay is now up to 6 minutes since January !

<comment><USER48> Correct. This is crazy. Everything from my central heating system, kitchen appliances and bed-side alarm clock is off by 6 minutes.

<comment><USER49> You must have a really old alarm clock. And why wouldn't you correct the time manually if you know it's drifting?

<comment><USER50> What innovations in alarm clock tech am I missing out on that would make me want to upgrade?

<comment><USER51> Being part of an IoT botnet seems to be all the rage nowadays.

</comment></comment></comment></comment></comment><comment><USER52> My microwave adds a minute every week or so because the UK's frequency is slightly different to the the rest of the EU. The clock on the microwave I have is keyed to the continental europe's frequency rather than the UK.

It's actually moderately annoying.

It was a fairly well reviewed (cheap) machine from Wilkinsons, a budget brand. Perils of cheap electronics.

EDIT: Well sounds like it's not the case, TIL, not sure where I picked that up from.

<comment><USER29> > the UK's frequency is slightly different to the the rest of the EU

It is? What does the UK use?

<comment><USER49> It's not. His microwave must be faulty.

</comment></comment></comment><comment><USER53> It's interesting that fixing this delay will involve setting the grid's frequency above 50 to 50.01 Hz which will advance clocks about 15 seconds a day and thus it will take 3 to 4 weeks to bring the clocks back on time.

And they cannot set it higher than 50.01 because at that point mechanisms for bringing the frequency down to 50 would automatically kick in.

<comment><USER24> Where does that 50.01 number come from? For the last 25 minutes, the UK grid figures have been:

50.217999 50.049999 49.990002 50.016998 50.106998 50.091999

They seem to go well above 50.01 for significant periods. Data from [http://www.gridwatch.templar.co.uk/](http://www.gridwatch.templar.co.uk/)

<comment><USER25> The UK is not part of the Continental European Power System.

For the Continental European grid, the article links to [https://www.swissgrid.ch/swissgrid/en/home/experts/topics/fr...](https://www.swissgrid.ch/swissgrid/en/home/experts/topics/frequency.html) which states that in the case of a time drift of more than 20 seconds the frequency will be shifted to 49.990 or 50.010 to correct it.

</comment></comment></comment><comment><USER54> using AC frequency for timekeeping seems ridiculous. Does quartz generator cost anything?

<comment><USER55> I was thinking the exact same thing. I've always assumed that AC frequency wasn't exactly 50Hz and was allowed to fluctuate slightly.

<comment><USER56> That assumption is right. But until now it was always compensated, so that in the mean it would be ~50hz. This correction mechanism apparently isn't enough anymore for the current fluctuations in europe.

<comment><USER57> Not mean, average. That's a very important distinction because if it was mean, then it might never correct, but it does.

</comment></comment></comment></comment><comment><USER58> Related (and 7 years old) article concerning US grid: "Power grid change may disrupt clocks" [0]. It was on HN [1], but with just two comments.

[0] [https://web.archive.org/web/20110701065839/http://old.news.y...](https://web.archive.org/web/20110701065839/http://old.news.yahoo.com/s/ap/20110624/ap_on_hi_te/us_sci_power_clocks)

[1] [https://news.ycombinator.com/item?id=2703264](https://news.ycombinator.com/item?id=2703264)

</comment><comment><USER5> Discussion elsewhere has turned up when this happened in the UK [https://books.google.co.uk/books?id=xnxQfGl8qfkC&pg=PA3&lpg=...](https://books.google.co.uk/books?id=xnxQfGl8qfkC&pg=PA3&lpg=PA3&dq=new+scientist+mains+frequency+31+january+1985&source=bl&ots=id14F5zeEb&sig=dRJUIbKWoORTzPzL1uMQ9FlfKVM&hl=en&sa=X&ved=0ahUKEwjz49Htr9rZAhUqJcAKHVR-C4YQ6AEINjAB#v=onepage&q=new%20scientist%20mains%20frequency%2031%20january%201985&f=false)

</comment><comment><USER59> International disagreements in power supply, loop flows on HVDC lines, incentives for distributed generation that trigger huge distribution build outs that bankrupt utilities, nuclear shutdowns that make it hard to meet carbon goals—its an interesting time on the European grid.

Not that the North American grid doesn't have problems. But the natural gas boom there has provided a nifty way to meet regulatory goals (carbon emmissions) with generator technology that's familiar and very easy to build/control.

</comment><comment> ju-st how can energy go missing because of an energy war? I found this but I don't understand: "Kosovo has been using more power than it produces while Serbia, responsible for balancing the Kosovo grid, has failed to do this, Nies said. As a result, Serbia has been free-riding on the system."

<comment><USER60> Basically, there are people using power on the grid without paying anyone to generate a corresponding quantity.

This means that less power is going into the grid than is being taken out of it, an amount that has accumulated since January to a bit over 100GWh. This missing 100GWh of energy has been collectively sapped from the inertia of the all spinning synchronous machines connected to the grid - mostly generators, but also motors - as a result of which they are spinning slower than they should be. Slowing spinning synchronous generators means a lower grid frequency.

<comment> ju-st Thanks, so its basically stolen at the generation level and thus you can't bill them 100GWh because that energy has never even appeared on the grid and technically nobody has consumed it.

<comment><USER60> No, it has definitely been consumed.

The only impediment to billing them is political.

</comment></comment></comment></comment><comment><USER61> Here's a great movie illustrating how the grid is balanced when a popular tv show ends and millions of people want a cup of hot tea: [https://www.youtube.com/watch?v=slDAvewWfrA](https://www.youtube.com/watch?v=slDAvewWfrA)

</comment><comment><USER62> I'm usually against technical solutions for political problems, but in this case it seems like a no-brainer to switch to cheap and ubiquitous crystal oscillators to decouple timekeeping accuracy from regional political stability.

<comment><USER63> AC grid frequency is much more stable than any crystal oscillator. After any number of years, AC will, on average, drift 0 seconds from UTC. It's being sync'd using atomic clocks to keep it as perfect as possible precisely because people use it as clock.

<comment><USER64> Which is better depends on whether you need short term accuracy or long term. On the long term, grid clock is more accurate. In the short term, crystals are more accurate.

<comment><USER65> This. In Florida US grid zero crossings randomly bounce around +/\- 20nS over a few seconds. Viewing the waveform on a scope can be enlightening (or inflammatory and shocking if one's not properly isolated)

<comment><USER66> I would be willing to bet that this is noise.

</comment></comment></comment></comment></comment><comment><USER67> I sort of wonder how much of a problem that is any more. I have exactly zero clocks that aren't NTP enabled, radio synchronized or just dumb battery watches.

Still an interesting "problem" non the less.

<comment><USER68> Those clocks being off is just a slight annoyance, the actual problem is the threat to grid stability.

</comment></comment><comment><USER69> Good example of how a critical external dependency without an SLA is a problem waiting to happen.

<comment><USER66> Well, the way I read the article, is that one area is violating the SLA, albeit informally specified.

</comment></comment><comment><USER70> As long as it's not 2^32 seconds (~136 years) late/early, NTP will still be able to save the devices it runs on! [0]

[0] [https://rachelbythebay.com/w/2017/09/27/2153/](https://rachelbythebay.com/w/2017/09/27/2153/)

<comment><USER71> These are devices from simpler times: no networking stack, not even an OS, sometimes not even firmware, just the basic function encoded with transistors. There's literally nothing to run _on_. Think water heaters, not RPi.

<comment><USER70> I know! Though strangely, my own kitchen appliances didn't participate in the continental time-shift (my microwave oven is still ticking correctly).

And my remark was more: even NTP can't save ou if you shift the time too much, so it's a good thing it's only ~6 minutes so far.

<comment><USER72> > even NTP can't save ou if you shift the time too much, so it's a good thing > it's only ~6 minutes so far.

Only for the initial step, or if the frequency of the quartz crystal is too fast/slow. The NTP daemon continuously adjusts the clock frequency, so if it was already running and synchronized, the shift will never be more than a few milliseconds.

</comment></comment></comment></comment><comment><USER73> NTP, but for electrical grids

<comment><USER74> connected botnet ready clocks, but for electrical grids?

Or perhaps you just meant setting up a (non-internet) grid time protocol? It would be a risky venture, I'm guessing, as there may be other functions that rely on the 50Hz grid consistency that would also then need their own special protocol on the grid.

<comment><USER71> Well, there's already the peak/off-peak signalling. You could theoretically send another signal, perhaps with time encoded, at midnight or some such. Note that this is a unidirectional broadcast, as opposed to the complexity of an actual bidirectional network.

</comment></comment></comment><comment><USER75> This has been annoying me since it started! The drift is nuts, I have to readjust the clock of my microwave because it's so much delayed and I always think it's earlier than it actually is!

</comment><comment><USER76> Time is relative, you know? :) Maybe time in Bulgaria (where I live) indeed goes slower. p.s. I'll probably get downvotes for my comment..

</comment><comment><USER77> The headline is kind of clickbait, almost no clocks in daily life get their base clock frequency from powerlines.

Edit: Nevermind

<comment><USER78> In Holland on the first day of the month we set off all the war alarms for a minute to test if they still function. This month a lot of them failed, caused by the power frequency

<comment><USER79> Imagine for a second that North Korea or US were involved in the electricity dispute in question... And because of that, as you say, Holland wan't test it's war alarm (which is vital infrastructure I guess) I wonder what would be the position of EU...

<comment><USER80> Do you really think the US is _not_ involved in this dispute?

<comment><USER79> ahahaha conspiration !!!

<comment><USER80> it's not really a conspiracy when energy payment squabbles are smacking you in the face in the news..

[https://www.export.gov/article?id=Kosovo- Energy](https://www.export.gov/article?id=Kosovo-Energy)

[https://www.reuters.com/article/us-ukraine-crisis-russia- gaz...](https://www.reuters.com/article/us-ukraine-crisis-russia- gazprom/ukraines-naftogaz-claims-2-56-billion-victory-in-gazprom-legal-battle- idUSKCN1GC2Z8)

etc etc etc</comment></comment></comment></comment></comment></comment>


YC Rejection – How much should we change to address their concern? - <AUTHOR>

After 2 rounds of intense interviews, got the rejection email last night. Although we agree with YC's reason for rejection (user acquisition), we'd like to confirm that not everyone got the same response before we devote our full energy in pivoting or changing something fundamentally to deal with the issue they raised.
We are super passionate about our idea and already had planned out the milestones we need to hit in the next few months. But if we were to truly address their concern about user acquisition, we need to take a long and hard look at our business model and the overall landscape, but we fear this may jeopardize our schedule.
Anyone else feeling the same? How much are you going to change to address their concerns? <comment><USER2> I'm wondering what they said about user acquisition? for certain startups, which build infrastructures or hardware products, they need some initial capital to show traction. It's hard for them to have a big user base at early stage. What should they do to pass the interview?

<comment><AUTHOR> It not said explicitly, but I believe, for us, long term growth means users change their behavior. We need them to stop using Craigslist. We have the same challenge as early AirBnB.

</comment></comment><comment><USER3> Well is your goal to bring your product to market or to get into YC? Doesn't have to be the end all be all.

<comment><AUTHOR> Thanks for the comment. Our goal is to succeed in the Linkedin/Facebook sense of the word (excuse our enthusiasm, but if not this, we would self fund and be content with organic growth). But I don't want to dismiss their wisdom, as we could be spending our next year building something that will eventually amount to a semi-success/failure, and face a much more difficult pivot.</comment></comment>


Researchers "Translate" Bat Talk (2016) - <AUTHOR> https://www.smithsonianmag.com/smart-news/researchers-translate-bat-talk-and-they-argue-lot-180961564/ <comment><USER2> During my time studying bats we used anabat detectors [https://www.titley- scientific.com/us/products/anabat-systems...](https://www.titley- scientific.com/us/products/anabat-systems/anabat-sd2-bat-detector)

To record bat echolocation and visually analyze the waveforms to determine the species and type of call. Different species echolocate at different frequencies and the calls they make when they're travelling, searching for prey and diving to eat all look different from each other and different between species.

</comment><comment><USER3> A bat translator could have been useful to me when a bat entered my flat at 3 a.m. a few weeks back...

<comment><AUTHOR> Human social behaviour is pretty solid evidence for the fact that merely speaking the same language won't guarantee satisfactory conflict resolution ;-)

<comment><USER3> True. Still wouldn't hurt to try politely before opening the largest window and tooling up a yoga mat and try baseball for the second time in my life. And yes, now I do regret not filming it...

</comment></comment></comment><comment><USER4> Maybe off-topic, but the article says they used a modified machine learning algorithm to nap bat calls to activities and from that developed a translator.

With the improvements to machine translation without parallelism data and voices-to-voice translation, how stupid is the idea that we might someday be able to take enough bat calls and human speech, and create a true translation system? obviously assumes the bats got something vaguely resembling a language we can translate from.

<comment><USER5> i.e., a bat phone.

</comment></comment><comment><USER6> > not just random, as previously thought

Who would think that an animal would just make random noises?

That just ridiculous.

</comment><comment><USER7> They're probably having a good laugh about how much better they dealt with COVID-19 than us humans

</comment><comment><USER8> Ok now how do we start a civil war between them? Let them fight each other to extinction.

<comment><USER9> The bats around where I live, in the Northeastern US, eat a lot of mosquitoes. I'd like them to stay around.

</comment></comment><comment><USER10> Sounds like bats need their own Twitter.

<comment><AUTHOR> This was actually posted on Marginal Revolution today with the exact preface "Let's not give them twitter"

[https://marginalrevolution.com/marginalrevolution/2020/07/he...](https://marginalrevolution.com/marginalrevolution/2020/07/hey- thomas-nagel-bats-argue-a-lot-about-where-the-best-restaurants-are.html)

</comment></comment><comment><USER11> probably accusing each other of not wearing a mask :-P

<comment><USER12> I can just imagine the bat cave now..., "ROOOOBBIN!! You haven't taken out the rubbish!"

"Nah! It was your turn you big bully!"</comment></comment>


Ask HN: How to talk to the boss about promotion - <AUTHOR> Been working in this company for a couple of years - have had growth in terms of responsibility, but that's not been reflected on paper.<p>Recently, I moved internally to a new team and I want to open this topic with new manager. Should I directly ask the question ? Any pointers? <comment><USER2> Yes, I think you should be direct. Some more things to keep in mind:

\- Make sure you gather proof of how you've grown and all your accomplishments. Don't expect your manager or HR to know or keep track of this for you. Even better do this all the time (even when you're not seeing a promotion or raise [1]). Also gather proof that your responsibilities are in line with those of the role you want (ask, look at the job descriptions etc).

\- Think of this as a conversation you're going to have with your boss. It's not a Q&A nor is it a confrontation.

\- Show empathy for any position the company might be in (if for example the company is going through hard times and can't afford to promote you right now).

\- Do NOT threaten to quit if you get a negative response. Instead, make sure to get feedback on how you can improve and make a plan with your boss on how to fulfill any requirements.

\- DO update your resume (should be easy if you're tracking those accomplishments) if you get a negative response. Put feelers out, look at some postings. Quietly.

Good luck.

[1] I built an app called JobRudder for situations just like this. It helps you track your accomplishments, prepare for salary increase meetings, performance reviews, interviews, and more:

[https://jobrudder.com](https://jobrudder.com)</comment>