How to Add Django Database Caching in 5 Minutes - <AUTHOR> http://tech.yipit.com/2011/12/07/django-queryset-caching%E2%80%A6or-how-we-slayed-the-beast-known-as-mysql-joins/

<comment><USER2> Or you can use a database where join performance isn't pathological, and scale out by adding read slaves.

Caching is hard. Caching as an automated 'layer' is much harder. If it were possible to cache in a general way _databases would do it already_. Adding a 'caching layer' is opening a gate into hell. The things is, at first it will be fine. Thousands of engineer hours and hundreds of subtle bugs later, you'll (if you're wise) realize that opening that door let out many demons, you just didn't have the eyes to see them at the time.

"There are only two hard things in Computer Science: cache invalidation and naming things." -- Phil Karlton

<comment><USER3> MySQL does already automatically cache queries internally. However, the scale of that caching is limited, in part because it's difficult to scale MySQL horizontally in the same way as Memcached or other distributed caching systems.

Adding read slaves is a great on many levels. Adding slaves, however, is not without overhead and stale data/programming complexity exists there too.

<comment><USER4> The caching that MySQL does effectively useless unless all of your queries deserve equal priority (times their individual footprint) for memory. That's almost never the case. I don't want relatively low-impact-rare-reads to take up space that I'd prefer to use for high-impact-common-reads, for instance.

It's nice that MySQL caches queries, but it doesn't solve the same problem that an application-level cache does.

</comment></comment></comment><comment><USER5> The timing of this post is funny, as I just got finished reworking our fork of django-cache-machine. As the post points out, the limitation of Cache Machine as it is currently built is that only objects which are already within a queryset can invalidate that queryset. This is fine for selects on primary keys, but beyond that the invalidation logic is incomplete.

My changes ( <https://github.com/theatlantic/django-cache-machine/> ) inspect the ORDER BY clauses (if there's a limit or offset) and WHERE constraints in the query and saves the list of "model-invaliding columns" (i.e., columns which, when changed, should broadly invalidate queries on the model) in a key in the cache. It also associates these queries with a model-flush list. Then, in a pre_save signal, it checks whether any of those columns have changed and marks the instance to invalidate the associated model flush list when it gets passed into the post_save signal. We have these changes up on a few of our sites and, if all goes well, we're looking to move the invalidation at the column level to make the cache-hit ratio even higher.

<comment><USER3> This is an interesting approach. I'm definitely going to check it out. Thanks!

</comment></comment><comment><USER6> Took a look at the Django Cache Machine they mention, at the invalidation scheme (<http://jbalogh.me/projects/cache-machine/#cache-manager>). It stores a "flush list", linking objects to their originating, cached queries. Interesting, though looks like something that could get out of hand quickly - are they storing the "flush list" itself in the cache (else how do all nodes learn of the invalidation) ? That's interesting, though a little creepy (the list gets very large as they appear to be keying on the SQL itself ?) Then they have the invalidation flow along all the relationships - maybe that's OK, but maybe it leads to a lot of excessive invalidation. Also they have a notion of how to avoid a certain kind of race condition there, caching ``None`` instead of just deleting, but it's not clear to me how that helps - you really need a version ID there if you want to prevent that particular condition (else thread 1 puts None, thread 2 sees None, OK we'll put a new one there, thread 3, which started before all of them, doesn't see the None, puts stale data in).

Really if you're caching SQL queries and such, you really, really should be doing little to no modification of cached data - this library makes it seem "easy" which it's not.

<comment><USER3> Your point is well taken; however, as the writer of the article, I'd say this. Our business requirements dictate that slightly inconsistent data is acceptable in certain circumstances. We would not retrieve objects through the caching flow in situations where absolute data integrity is required.

That points makes all the difference, in my mind. To your point about excessive invalidation, that depends on your Read/Write workload.

<comment><USER6> well if I have a system where inconsistent data is OK, I just let the normal cache expiration time logic handle that. Turning down expiration times to a few minutes, even 30 seconds, can still take lots of load off of particular "hot" objects that might be fetched multiple times quickly (such as from a series of AJAX requests) while making it unlikely that large inconsistencies will show up on the screen.

</comment></comment></comment><comment><USER7> full disclosure: johnny-cache author

The top commenter in OP gave a great rundown of these projects and their evaluation of them at YCharts at a Django NYC meetup a month-ish ago; I'm sure his slides are available on the nyc django site somewhere.

All of these projects "automatically" manage cache for querysets, but they do it different ways, and can be susceptible to poor performance under different usage patterns.

From what I can tell, JC adds the lowest amt of overhead to cache misses and hits, and uses the simplest (it's mildly sophisticated, but still straightforward) management algorithm. It's the only one that works fine when using UPDATE queries that do not mention row ids, and (as a result) is the one that most greedily invalidates on writes.

The others are fine projects run by smart people, and depending on your site's situation, I'd recommend some of them over johnny-cache. It's a good idea to evaluate them all, as they certainly did at YCharts (his section on JC was very accurate), and as OP seems to have done.

<comment><USER8> We use johnny-cache on a few projects with great results, thanks for your work!

</comment></comment><comment><USER9> Thanks so much for linking to this post. I was just thinking how I would go about caching QuerySets for a Django project.</comment>


NYC Subway Data REST API - <AUTHOR> http://nycpulse.herokuapp.com/api

<comment> rkuykendall-com App request: Android app, I enter "my" subway stops and the direction I take them ( Ex: The 1 train, uptown, from 116th ), the App simply shows me how many minutes until the next train.

I want something I can quickly open and glance at to know when I should leave, or if I should be sprinting to the station to save myself 20m late at night.

1 minute mock up:

[http://rkuykendall.com/uploads/subway-app-mock- up.jpg](http://rkuykendall.com/uploads/subway-app-mock-up.jpg)

<comment><USER2> The arrival times listed on the website are likely just timetable times. Until the MTA releases real time data, I wouldn't depend too heavily on scheduled train times (especially during rush hour).

<comment><USER3> The MTA already provides real time data for the numbered lines [1] and some buses [2].

[1] [http://datamine.mta.info/](http://datamine.mta.info/) [2] [http://bustime.mta.info/wiki/Developers/Index](http://bustime.mta.info/wiki/Developers/Index)

<comment><AUTHOR> They do, but it's GTFS format...

<comment><USER4> What's so horrifying about the GTFS format?

</comment></comment></comment></comment></comment><comment><USER5> What would really be interesting to play with is the real-time location of each train. The MTA must have it. I wish they would publish it somehow.

I'd love to build a city-wide visualization of the train traffic that shows where it's backing up, where it's flowing smoothly, etc. It probably wouldn't be very useful for me as an NYC commuter, but purely as a visualization it would really be something.

<comment><USER6> They do have real time information for the A-division (numbered trains). The B-division trains are, however, a dogs breakfast with no central monitoring or dispatch. There are plans afoot to spend a couple of billion dollars to perform a dispatch upgrade for BDiv -- but given the generally strapped nature of the MTA, I wouldn't hold my breath for the information to be made available anytime soon.

<comment><USER7> Sort of. The 7 is A division and has no data, and the L is B division and has data.

Plus, who needs realtime train position data when you can just listen for those helpful manual announcements like "An uptown express train is approaching W 4th St."

</comment></comment></comment><comment><USER8> I've tried almost every NYC subway/bus app and I've finally given up. They're all fairly useless IMO. I'd rather just wait 5 minutes for the next train than bother with any of the current apps.

That being said, I wish Google would take on the challenge and include subway info.

\- If somebody made a NYC transit app that I actually felt comfortable relying on... I would love push notifications ASAP of when my frequented trains/buses are down. I always forget to check the night before so please just tell me and make it relevant.

-Also I really want a detailed map of NYC that I can view offline but it needs to be lightning fast.

After all the frustration dealing with those apps, a couple months ago I finally decided to buy a paper map of NYC and start memorizing all that I can. I feel a bit more liberated and a little more like a NYC aficionado (snob).

</comment><comment><AUTHOR> Hi guys, my initial project was to build a visualization like the one @artursapek describes, and ended up making this API because the MTA doesn't have one.

Totally agree that it would be really cool.

I made this nycpulse.herokuapp.com, but it could be so much cooler.

<comment><USER9> Any plans to open source the code?

<comment><AUTHOR> [https://github.com/mimouncadosch/MTA- API](https://github.com/mimouncadosch/MTA-API)

<comment><USER9> Awesome, thanks.

</comment></comment></comment></comment><comment><USER10> Nice work.

I'm not sure how I interpret that arrivals say that a train arrives at my station at 8:42:30 and 8:43:00 (within 30 seconds of each other. Also I've never seen a train arrive at either of those times :). So I wonder how accurate the arrival times are in general.

(P.S. It'd be nice if arrivals was sorted in the response, not sure if that's an artifact of the way this is parsed from the data source or not).

<comment><AUTHOR> Thanks :) Which station are you?

I couldn't find one with a 30 second interval.

<comment><USER10> [http://mtaapi.herokuapp.com/api?id=D38S](http://mtaapi.herokuapp.com/api?id=D38S)

</comment></comment></comment><comment><USER9> For those of you on iOS looking for a polished transit app with support for multiple cities, I highly recommend Transit.[0] It allows you to quickly map trips as well as easily save data for offline access.

[0]: [http://thetransitapp.com](http://thetransitapp.com)

</comment><comment><AUTHOR> "The arrival times listed on the website are likely just timetable times. Until the MTA releases real time data"

@aerialfish you are correct, this data is taken from timetables, as the MTA does not release real-time data. But it is probably very close

<comment><USER3> They do for the numbered lines: [http://datamine.mta.info/](http://datamine.mta.info/)

</comment></comment><comment><USER11> App request too: I would like an application that wake me up 5 minutes before my stop.

<comment><USER12> currently working on this. :)

</comment></comment><comment><AUTHOR> As promised, open source: [https://github.com/mimouncadosch/MTA- API](https://github.com/mimouncadosch/MTA-API)

Feel free to make it better :)

</comment><comment><USER13> How is GTFs "..not particularly useful for developers"? I played around with it a little and found it to be perfectly usable/useful comma-separated values.

<comment><USER14> +! GTFS is super awesome and most apps I know rely on it. Its consistent format that is adopted by transit systems across the globe. couldnt get more developer friendly than that. If you developed with this RESTful API, it would be hard to adapt your app to use the feed for a different city.

</comment></comment><comment><USER15> It has everything but what I was looking for, a notification of when a line is delayed.

</comment><comment><USER16> is this open source?

<comment><AUTHOR> It's not yet, but will be soon :)</comment></comment>


The NSA Is Going to Love These USB-C Charging Cables - <AUTHOR> http://gizmodo.com/he-nsa-is-going-to-love-these-usb-c-charging-cables-1691781672

<comment><USER2> > But that convenience doesn't come without a cost; our computers will be more > vulnerable than ever to malware attacks, from hackers and surveillance > agencies alike.

Cut the crap. USB is just as vulnerable as it always was. The only thing that changed with USB-C was practicality and convenience.

<comment><USER3> The author knows USB is just as vulnerable as it always was.

The thing that changed with USB-C is that you can't avoid plugging in USB devices. If you borrow someone's power cord, you're plugging an unknown USB device into your computer.

Dropping a malicious USB drive in the parking lot was the traditional attack vector for USB devices, which obviously won't work as well with a power cord (a power cord sitting in the parking lot would be a bit suspicious), but it opens up new targeted attack vectors. Someone could easily swap out your power cord with a malicious one when you aren't looking, which might wait for your computer to be inactive, then kick into "fake keyboard mode" to open up a terminal, download whatever executable off of the internet, and run it (with your user privileges). It won't own the whole computer without authentication, but it'll own all of the data in your home folder.

<comment><USER4> > _The thing that changed with USB-C is that you can 't avoid plugging in USB > devices. If you borrow someone's power cord, you're plugging an unknown USB > device into your computer._

You know, they could easily add a mode for it, so it only gets power and no data.

<comment><USER3> Definitely could be, but I'm not holding my breath for any of the major OSes to implement that. You'd need a popup window for every new keyboard to say "This is a keyboard! Is it allowed to be a keyboard?" Which works for a laptop because you always have a keyboard/trackpackpad built in, but could be a problem on a desktop if you can't approve a keyboard/mouse without already having one connected and approved. Do desktops ever include accelerometers? Maybe you could pick it up and shake it to allow a new keyboard.

On the other hand, as I understand it, USB Power Delivery negotiates the voltage over the power wires _only_ , so making an adapter cable that doesn't pass data should still be an option for the paranoid among us. But good luck getting every user at a big company (say, Anthem or Premera) to do that religiously.

<comment><USER4> Actually seems already implemented -- and that the original article was mostlt BS:

"Gizmodo seems to believe the 12-inch MacBook is vulnerable to this direct attack, even going so far as to suggest that the NSA will distribute hacked USB-C power adapters designed to take over your notebook. But unlike Thunderstrike on vulnerable Macs (see "Thunderstrike Proof-of-Concept Attack Serious, but Limited," 9 January 2015), the USB port uses Intel's xHCI (eXtensible Host Controller Interface), which can't be placed into a DFU (device firmware upgrade) mode to overwrite the MacBook's firmware. Thus the MacBook itself can't be infected with BadUSB, so plugging in an unknown power adapter can't give someone control of your MacBook."</comment></comment></comment></comment></comment>


NetCrawlerDetect (a .net std port of JayBizzle's CrawlerDetect) - <AUTHOR> https://github.com/gplumb/NetCrawlerDetect <comment><AUTHOR> For convenience, there is also a nuget package available ([https://www.nuget.org/packages/NetCrawlerDetect](https://www.nuget.org/packages/NetCrawlerDetect))</comment>


Apple is Back in the Enterprise Game - <AUTHOR> http://www.readwriteweb.com/enterprise/2011/01/its-official-apple-is-an-enter.php

<comment><USER2> > _"However, Apple's enterprise strategy has been a mystery for the past > couple years"_

When your strategy is a mystery, you're not in the enterprise game. Playing the enterprise game requires one to have a roadmap.</comment>


SwiftBoot - <AUTHOR> http://www.swiftboot.com/?p=213 <comment><USER2> There are all kinds of topics on boot optimization over on the Embedded Linux wiki:

<http://elinux.org/Boot_Time>

So I'm wondering if they just apply a set of these tricks given your circumstances. Notice that they guarantee a 50% speedup on _your_ board, not that you'll boot as their example. The OMAP 3530 shown is a pretty fast chip. You also don't know what else they're skipping to get this speed.

</comment><comment><USER3> This is awesome! I wonder how they achieved it. Normal boot times for Beagleboard are at least 15-20 seconds.

Perhaps it is using some sort of software suspend/resume functionality? I suspect initializing memory by copying an image directly from flash after U-boot starts up would be about this fast.

<comment><USER4> Would something like uswsusp/tuxonice (software suspend to RAM or disk) be capable of assisting in quick boot times?

Step 1: Build fully functioning embedded system and boot it. Step 2: Suspend it in the state you want it be in on boot Step 3: Hack u-boot to directly load the suspended image

Has anyone done something like this? Is it feasible?

<comment><USER4> I just saw the link posted by joezydeco (above). It has good links to information on this particular avenue of optimization, particularly: <http://elinux.org/Suspend_To_Disk_For_ARM></comment></comment></comment>


Apricity beautiful Linux distro based on Arch - <AUTHOR> http://apricityos.com

<comment><USER2> Why is Chrome, a proprietary software packaged when there is a practically equivalent FLOSS software (Chromium)?

Or, the more general question -- how Pacman packages and Apricity apps relate to each other? The answer seems to be not straightforward, as if Apricity apps were just sugaring for a subset of the Pacman pkg set, then Chromium would be packaged (given Arch ships Chromium, not Chrome).</comment>


New Authentication Scheme Combats Keyloggers, Shoulder-Hacking - <AUTHOR> http://www.darkreading.com/document.asp?doc_id=145104&print=true <comment><USER2> This seems overcomplicated. Here's a simpler interface: all the numbers in your PIN are between 1 and 8. You view the screen by looking into a binocular scope. On each side of scope is a row of four buttons where you can place your fingers. The screen displays random permutations of the numbers 1-8, and you select one digit of your PIN from each permutation.

</comment><comment><USER3> You know what we need? Ubuntu live CD, a solid 10 character password with numbers and upper/lowercase, and fast typing.

The first eliminates the threat of keyloggers. The second prevents shoulder hacking.

Don't need to be a rocket scientist.

<comment><USER4> Does it now?

<http://www.keyghost.com/></comment></comment>


The Islamic Inspiration for the Original World Trade Center Design - <AUTHOR> http://www.slate.com/id/2060207 <comment><USER2> I didn't know this and it was very interesting. But, honestly, not all that surprising.

The beauty of this country is that it's a melting pot of cultures, religions, ideas, colors, and ethnic classes. It's what makes us both different and great.

So, of course, Islam, like every other culture introduced us to some beautiful new ideas. At the end of the day, we're a better more diverse nation because of said influence along with influences from every other group.

Which is why recent events pain me so much. Why can't we just all get along?

<comment><USER3> > Which is why recent events pain me so much. Why can't we just all get along?

I know you're asking rhetorically, but the answer is that there's moral philosophies that are incompatible with each other.

An example: Islamic culture typically believes that women's sexuality can be a destructive and corroding force for herself, her family, and society if not muted and regulated. Progressive multiculturalism believes that any muting or regulating of a person's sexuality is oppressive and wrong.

Thus progressive multiculturalism believes Islamic culture is oppressive and wrong, whereas Islamic culture believes progressive multiculturalism is destructive and corrosive for society and the individuals in society.

The reason we can't all get along is because cultures have violently opposed moral philosophies. Western progressivism wants to liberate and break what they perceive as Islamic oppression. Being that leaders in Muslim society generally have less access to fight back in an information war (since they have less access to media and universities), they fight back in other, more base and primitive ways.

Note that this isn't a moral judgment on either side, just an explanation of why we aren't all getting along.

<comment><USER2> I wasn't asking rhetorically and I am, in fact, a proud Muslim myself.

I appreciate that you used the word 'culture' before your argument but it's still wrong to use the word 'Islam' before 'culture'. It's just culture. Not Islamic culture.

Our moral philosophies are not AT ALL incompatible. At it's roots, Islam is eerily reminiscent of its two previous Abrahamic influences (Judaism and Christianity). If the "West" is incompatible with Islam, then by that same logic it's also incompatible with Christianity and Judaism.

The practices that you mention are heinous and looked down upon by a lot of people, myself included. The places that practice them are either oppressive regimes or full of ignorance. If you read the texts of Islam (the Qur'an, Hadith, etc), women rights are a huge issue and women are very respected.

The problem today is with ignorance, in two ways:

1) Ignorance of people towards Islam. A lot of people have no idea what it is and they're afraid of the mystery. What's worse is that they let a lot (with private agendas) fill in that hazy picture for them. 2) Ignorance of some Muslims themselves that then do many of the problems you're referring to.

I think it's no surprise at all that many of the places that breed these problems (whether it's terrorism or sexism) have low literacy rates, high rates of unemployment, horrible schools, no government, etc. In other words, given these same environments, they would be a breeding ground for bad deeds anywhere. So, again, it's a problem with cultural problems. Not with Islam.

I'm Egyptian American myself. I'd invite you to take a look at some of the more modern Arab or Muslim countries. Jordan, Egypt, Lebanon, some of the Gulf states. If I were to respond to your arguments with an example, it would be my homeland, Egypt, whose population is 85-90% Muslim. We have isolated pockets of such problems, often in some of the poorer rural areas, but for the most part you'll find beaches, tv channels, movies, literature that's as progressive as anywhere else.

<comment><USER4> Terror organizations may primarily recruit from impoverished Muslim societies but they don't always. I have not researched this with any rigour, but there do seem to be quite a few Muslims from wealthy Western nations that have joined terrorist organizations. What draws these Muslims?

This is a genuine question and not rhetorical.

<comment><USER5> Well, the US government started a war for no reason in Iraq that exterminated a million people. Don't you think bringing about the deaths of a million Muslims for no apparent reason might enrage some people?

How many of your friends and family would I have to brutally murder before you said "screw it, I'm going to devote my life to revenge or maybe to just stopping you from killing more people"?

<comment><USER4> From where are you getting your data?

AFAIK sectarian violence has killed many, many more Muslims than the US.

And terrorist organizations existed before the US invaded Iraq.

EDIT: I gather from your comment that you believe terrorism committed against Western civilians is a reasonable response to the way the US has handled the war in Iraq. Correct?

<comment><USER5> _From where are you getting your data?_

Excess mortality figures derived from several published cluster sampling studies and then extrapolated to cover the period of greatest violence.

 _AFAIK sectarian violence has killed many, many more Muslims than the US._

Please read my comment carefully. I said the US government started a war that killed a million Iraqis. The sectarian violence of which you speak was not a significant issue before the war.

 _And terrorist organizations existed before the US invaded Iraq._

Very true. But it seems that various western countries have done other things to kill and maim large numbers of Arabs and Muslims before the war. For example, Mohamed Atta appears to have become radicalized by watching Israeli attacks against Lebanese civilians.

 _I gather from your comment that you believe terrorism committed against Western civilians is a reasonable response to the way the US has handled the war in Iraq. Correct?_

No, that is incorrect. I think attacking random people associated with the government that killed your family is morally wrong. But I also think it is a very understandable response, one that almost anyone here might engage in. Because that's how human beings operate.

I marvel at your inability to distinguish between normative and positive statements.

You never answered my question though: how many of your friends and family would I have to kill before you turned to violent revenge?

<comment><USER4> _Please read my comment carefully. I said the US government started a war that killed a million Iraqis. The sectarian violence of which you speak was not a significant issue before the war._

Your use of the word 'exterminated' makes it sound like you're arguing that the US is directly responsible for Muslims killing Muslims. Sorry if I misunderstood you.

 _You never answered my question though: how many of your friends and family would I have to kill before you turned to violent revenge?_

This is quite obviously a rhetorical question. However, the answer is "not many", but randomly murdering people wouldn't be my response.

I also would like to see a source that says Israel intentionally attacked Lebanese civilians.

<comment><USER5> _Your use of the word 'exterminated' makes it sound like you're arguing that the US is directly responsible for Muslims killing Muslims._

In general, wars tend to cause lots and lots of people to die. That means that countries that starts wars for no good reason are responsible for the resulting deaths.

 _randomly murdering people wouldn't be my response_

I'm glad to hear that your violent revenge would stop just short of murder. Brave talk from a man who hasn't lost much.

 _I also would like to see a source that says Israel intentionally attacked Lebanese civilians._

[http://en.wikipedia.org/wiki/1996_shelling_of_Qana#Mohamed_A...](http://en.wikipedia.org/wiki/1996_shelling_of_Qana#Mohamed_Atta) says "A United Nations military investigation later determined it was unlikely that Israeli shelling of the U.N. compound was the result of technical or procedural errors."

<comment><USER4> _I'm glad to hear that your violent revenge would stop just short of murder. Brave talk from a man who hasn't lost much_

I never said I wouldn't kill _you_ if you killed my friends and family. Murdering random people in response to your actions is not only morally wrong, but also stupid and throughly irrational.

You are apologizing for people who do this because you claim it's "a natural human response". Killing random people is _not_ a natural human response; in my opinion, it's the response of a psychopath.

<comment><USER5> _it's the response of a psychopath_

Perhaps. But I don't think you are very familiar with history. I've read many accounts of soldiers, including American soldiers, killing random people because their friends were killed. In general, such people do not face discipline. In fact, there are many Americans who urged the killing of completely arbitrary Arabs or Muslims in revenge -- many of these people are politicians.

My point is: this sort of psychopathy is quite common and very well accepted in society provided it is directed against out-groups.

<comment><USER4> _Perhaps. But I don't think you are very familiar with history. I've read many accounts of soldiers, including American soldiers, killing random people because their friends were killed. In general, such people do not face discipline. In fact, there are many Americans who urged the killing of completely arbitrary Arabs or Muslims in revenge -- many of these people are politicians._

Soldiers _are_ court-martialed and arrested for killing civilians.

[http://www.commercialappeal.com/news/2008/jul/13/no- prisoner...](http://www.commercialappeal.com/news/2008/jul/13/no-prisoners- america-at-war/) (scroll to bottom)

<http://www.guardian.co.uk/world/2006/nov/16/usa.iraq1>

[http://www.usatoday.com/news/world/iraq/2006-07-09-us- cases_...](http://www.usatoday.com/news/world/iraq/2006-07-09-us-cases_x.htm)

<http://www.foxnews.com/story/0,2933,222220,00.html>

Just because not all are found out and tried does not mean it is condoned by the military or US society. You clearly have your own dogmatic view regarding US foreign policy and attitudes towards Muslims and I can't really argue with someone who has an almost religious belief in a point of view.

<comment><USER5> _Soldiers are court-martialed and arrested for killing civilians._

Sometimes they are. Oftentimes they are not. The fact that some are does not prove that regulations prohibiting the killing civilians are uniformly or frequently enforced. Anecdote is not the plural of data. Surely you see that?

 _Just because not all are found out and tried does not mean it is condoned by the military or US society._

I'd say that when high ranking political figures publicly advocate the killing of innocent people in revenge for the crimes of others without any sanction at all, then...yes, that does imply that American society condones their behavior.

 _You clearly have your own dogmatic view regarding US foreign policy and attitudes towards Muslims and I can't really argue with someone who has an almost religious belief in a point of view._

For someone with "almost religious beliefs", I seem to do a very good job justifying my arguments with evidence. Perhaps that is the main difficulty you face in discussions here.

Speaking as someone who almost joined the military and who who has read and studied a great deal in order to understand how military organizations function in practice, I don't think you're knowledgeable enough on the subject to effectively participate in discussion here.</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment>


Show HN: Web tool to find trading strategies for free - <AUTHOR> http://trading-level.com/ <comment><AUTHOR> This is a free tool i created on my own in the evenings.

It basically tries couple strategies to determine best price levels (prices for limit orders etc). The strategies include a basic distance from the last close price (like: enter the market if the price is prev_close_price + 1%) but also bit more complex strategies (distances from high/low, highest high and lowest low )

To start you just need a csv file from MT4 (so the data of some market)

If you dont have such a file by hand, try mine: [https://filebin.net/90fyxmmrtr6iqa6a](https://filebin.net/90fyxmmrtr6iqa6a)

Once you see something you like you do need still write the code or "trade" it manually

GL HF</comment>


Ask HN: Declarative database migrations? - <AUTHOR> Hi HN, does anyone have suggestions for a declarative database migration tool, ideally compatible with Postgres?
I've used a number of imperative migration systems, most of which are baked into application stacks.  They scratch an itch, but I've always found them a cumbersome and time consuming.
What I really want to do is declare what schema I want, and have a tool (ideally separate from an application stack) which will look at the existing schema and create a diff to update the schema based on what I've declared.
I actually implemented a very basic tool like this for MySQL for a project I was doing a few years ago which created and altered tables and indexes, and there were some pitfalls around destructive changes but on the whole it was very pleasant to use as long as you understood the pitfalls.  I'm having trouble finding any similar tools around though and would love to hear any recommendations. <comment><USER2> The best plan for this today is the same as it was 20 years ago: write your change scripts in SQL.

SQL is easy to learn, easy to read, easy to test, does exactly what you tell it to with no surprises, and drops happily into version control. No tools required because tools can only possibly complicate things.

The only reason the question you ask is in your brain at all is because of the trend 10 years ago to abstract away the database and declare it in configuration files elsewhere. There's no need to do that. The real thing is simpler, easier to comprehend, and guaranteed not to magically map your description of what you want into something different entirely.

<comment><USER3> I think you're right. Keep a text file of your current schema in a vcs. This may be generated from a modelling tool, but you need a text file you can diff and log within the vcs.

SQL isn't difficult to read, modify, test. Ensure you have dev and test servers. You must have someone in your team who is conversant with the db server(s) you target, preferably not an application programmer but someone focused on the database.

There are many schemes to manage changes, but just store a alter and backout script of each change once in production works for me, give them a number and an order to apply.

I'm managing schema for our product set of maybe 1000 db objects as well as doing some reporting, BI and some app dev and scripting. Just ensure that the team use the correct db and can rebuild easily to the trunk or release tag required and you should be fine.

</comment></comment><comment><USER4> I don't know of any declarative tools, although I've written some tips on using Rails migrations here:

[http://illuminatedcomputing.com/posts/2013/03/rules-for- rail...](http://illuminatedcomputing.com/posts/2013/03/rules-for-rails- migrations/)

The problem with a declarative tool is managing change over time. What happens when you want to change a relationship from one-to-many to many-to-many? How does the declarative tool know how to migrate the old data? There are lots of examples where you need to say specifically how to get from point A to point B, especially once you're in production and can't just throw everything away and start over.

</comment><comment><USER5> You could use Sqlalchemy + alembic. This would require you to declare a schema in python. You would not be forced to use sqlalchemy in the application though.

</comment><comment><USER6> [http://www.liquibase.org/](http://www.liquibase.org/)

</comment><comment><USER7> Jetbrains is working on an IDE for what seems to be your use-case.

[https://www.jetbrains.com/dbe/](https://www.jetbrains.com/dbe/)

</comment><comment><USER8> The problem with this is that some transforms you can make (like renaming columns) can't be figured out from comparing two schemas.

</comment><comment><USER9> apgdiff</comment>


Apple rushes to fix major security bug - <AUTHOR> http://www.bbc.co.uk/news/technology-42161823 <comment><AUTHOR> [sic]He found that by entering the username "root", leaving the password field blank, and hitting "enter" a few times, he would be granted unrestricted access to the target machine.

... ...

a member of Apple's support forums had posted details of the flaw more than two weeks ago, though the tone of his message appeared to suggest he saw the vulnerability as a useful feature for troubleshooting rather than a critical security threat.[/sic]</comment>


Google's new voice recognition system works instantly and offline (Pixel only) - <AUTHOR> https://techcrunch.com/2019/03/12/googles-new-voice-recognition-system-works-instantly-and-offline-if-you-have-a-pixel/ <comment><USER2> Google AI blog: [https://ai.googleblog.com/2019/03/an-all-neural-on-device- sp...](https://ai.googleblog.com/2019/03/an-all-neural-on-device-speech.html)

arXiv: [https://arxiv.org/abs/1811.06621](https://arxiv.org/abs/1811.06621)

<comment><USER3> Interesting that they're using RNN transducer. I thought everyone's moved to CNN lately.

<comment><USER4> CNN's are only mentioned because of potential processing considerations as computationally, they are easier to deal with. But given the nature of speech recognition, which is so highly temporally correlated, it shouldn't be a surprise that a recurrent neural network would be used. This is pretty much exactly the purpose the RNN type of model architecture was designed for.

Also if you haven't looked into the properties of how exactly a RNN Transducer functions, I highly recommend doing so. They help resolve a great deal of problems that traditional RNNs and CNNs are unable to deal with.

<comment><USER3> I take it you haven't been following the field lately, have you? It is surprising because convnets just work better for speech recognition ([1] is the latest SOA). I'm guessing gated convolutional LMs are slower than RNN transducers when deployed on mobile. Can someone confirm?

[1] [https://arxiv.org/abs/1812.06864](https://arxiv.org/abs/1812.06864)

<comment><USER5> There is also the transformer approach (eventually with local attention to bound latency), (like I'm doing in my project (Work in Progress) : [https://github.com/GistNoesis/Wisteria/blob/master/SpeechToT...](https://github.com/GistNoesis/Wisteria/blob/master/SpeechToText.md) ), though it's in the same line of thought as the convolutional CTC.

The RNN-T is a nice idea though, if I understand it correctly it's another approach to the alignment problem. In CTC, you are generating sequence like TTTTHHHEE CCCAAATT, which mean that your language model must deal with these repetitions, and you can't train using text without repetitions. In RNN-T you are learning to advance the cursor on either audio sequence or text sequence so as to maintain alignment, kind of like you do when you merge-sort two sorted lists, therefore it outputs THE CAT, and you can use a standard language model.

<comment><USER6> We explored self-attention + CTC in our ICASSP 2019 paper ([https://arxiv.org/abs/1901.10055](https://arxiv.org/abs/1901.10055)). Our implementation uses internal infra, so we're a ways from releasing code :(

Hoping paper details suffice and help with the parameter search, and happy to respond to questions over e-mail. Would love to see an open-source implementation with local or directed attention built out!

<comment><USER3> Very interesting! Perhaps I should revise my original comment to "everyone is moving to transformers lately" :)

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> "But it's sort of funny considering hardly any of Google's other products work offline. Are you going to dictate into a shared document while you're offline? Write an email? Ask for a conversion between liters and cups? You're going to need a connection for that!"

While offline, you might write email drafts, your blog, or even a book:

[https://medium.com/@augustbirch/what-i-learned-writing-an- en...](https://medium.com/@augustbirch/what-i-learned-writing-an-entire-novel- on-my-phone-f1655d09b00b)

What's missing is the ability to make edits using your phone. You can probably speak at over 100 words a minute but then you need to stop to bring up the software keyboard.

<comment><USER8> The offline aspect is hardly the main draw here though. As mentioned earlier in the article, the latency reduction is huge. Another aspect they didn't really cover is privacy implications. Lastly, you may not be offline, but dodgy connections can also be a pain if you need a stable stream of packets going back and forth.

<comment><USER9> I refuse to put an amazon/apple/google surveillance device in my home, so I am very interested in a DIY digital assistant device. I'm aware of a few options but it seems like offline voice recognition is always a little sub-par. I am really looking forward to the day when an offline, open source digital assistant can compare in quality to a proprietary/cloud device.

<comment><USER10> _> I refuse to put an amazon/apple/google surveillance device in my home..._

Do you have a smartphone? Because that's most likely an Apple or Google surveillance device.

</comment></comment></comment></comment><comment><USER11> I just switched my Pixel 1 to airplane mode and tried voice input. Sure enough, it worked offline and it was fast! Very impressive work. (I've tried that before, but in the past it could only understand a few special phrases.) I suppose this new feature came with the security update my phone downloaded a few days ago.

There are lots of ways to spin this, but I see it as a significant improvement for any app that could benefit from voice input. It's immediate and not susceptible to network glitches. The benefit for Google, IMHO, is primarily more sales of updated Android devices.

<comment><USER12> Unless you very recently (meaning today) accepted a download of a new language pack for English, it's likely just the old model, which is perfectly functional, while not being as accurate as the online version.

<comment><USER8> More specifically:

Gboard > Voice Typing > Faster voice typing

It says its an 85MB download for US-English

<comment><USER13> Looks like this is on Android. Gboard iOS app doesn't have this setting.

<comment><USER14> Yes, it's specifically pixel phones.

</comment></comment></comment></comment></comment><comment><USER15> > But it's sort of funny considering hardly any of Google's other products > work offline.

I dunno, Android and a lot of Google's mobile apps that aren't _about_ online communication work fine offline. Actually, a lot of the online communications ones do too, as much as is even conceivable, they just don't transmit and receive offline, because, how would they?

</comment><comment><USER16> Just to be clear: This has nothing to do with "Wake Words" (e.g. OK Google, Alexa, Hey Siri, etc) which have always been handled offline/locally.

This is translating what you said after the wake word from voice to text on the local [Pixel] hardware rather than sending it into Google's Cloud.

The biggest benefits here are speed and reliability. It could also handle some actions offline.

<comment><USER17> Another benefit is privacy, this eliminates an entire set of potentially personal data from being handed off to Google.

<comment><USER18> On the other hand, when you can transcribe locally, uploading whole days worth of eavesropping would not cause a noticeable spike in traffic. I'd consider it more a lateral change than an improvement.

<comment><USER19> install a firewall (there are root/no root) options) and block google keyboard's internet access.

<comment><USER20> What if the data is send by something other than keyboard though? :)

<comment><USER21> This conspiracy goes all the way to the kernel!

</comment></comment></comment></comment></comment></comment><comment><USER22> Does the Pixel have some specific hardware that this uses, or is it simply limited to Pixel to limit the rollout? I am curious if I should get my hopes up to see this on gboard with non-Pixel Android devices.

<comment><USER23> The Pixel 2+ does have a coprocessor for compute workloads (the Visual Core). However users here have reported this working on a Pixel 1, which doesn't have that chip.

The Verge says it may reach other devices later.

It sounds like it's both better than the old dictation model, and significantly smaller.

</comment></comment><comment><USER24> AI systems that are able to work offline are great for privacy.

The thought that every interaction with my phone is being streamed in realtime to a third party server freaks me out.

Kudos to Google for working on this.

<comment><USER25> They can still send the information they collected from your microphone later, when you connect to the internet ...

You want an open source solution, not just an offline solution.

<comment><USER24> Even so, offline AI solutions have been piss poor and Google moving the state of the art in spite of their vested interest in keeping people online is a good thing.

Yes, we want an open source solution, but I'm not going to work on it. So who's going to work on it? Are you?

In absence of resources working towards the ideal, I'll applaud any step in the right direction.

<comment><USER26> To be fair there are others that have been pushing the needle in this space for considerable time. The standout for me is [https://snips.ai](https://snips.ai) Offline, multiple platforms, multiple languages, many parts open source and more oss parts in the pipeline. While certainly not currently aimed at dictation, but instead assistant building and automation. In this space on device speed, privacy & offline are critical. In the case of Snips "piss poor" falls short from the reality of what I have experienced YMMV.

Nonetheless we all benefit from this progress

</comment></comment></comment></comment><comment><USER27> Didn't they advertise something like this a few years ago? I seem to remember trying it and finding that it didn't really work as well as the online recognition at the time.

EDIT: Looks like something was added in Jelly Bean: [https://stackoverflow.com/questions/17616994/offline- speech-...](https://stackoverflow.com/questions/17616994/offline-speech- recognition-in-android-jellybean)

</comment><comment><USER28> This will be great when ported to Lineage!

</comment><comment><USER29> I can't pinpoint when exactly, but on windows XP, there used to be a speech to text engine that worked locally. When you set it up, you had to read some text to train it with your voice. You could constantly train it to improve it.

This was before the cloudamagig, so I wonder it ran on.

Edit: found the link [https://www.techrepublic.com/article/solutionbase-using- spee...](https://www.techrepublic.com/article/solutionbase-using-speech- recognition-in-windows-xp/)

</comment><comment><USER30> At the same time pre-pixel phones get features stripped. "OK Google" now requires phone to be awake and unlocked, or plugged in to work.

</comment><comment><USER31> The other, gigantic shoe that will someday drop will be Google transcribing every incidental conversation. It can already do that, on-device, for every song that's heard, ever. It's a super power, being able to remember every word spoken around you, time and place, but of course it has privacy implications even if all the work is done without their cloud.

<comment><USER32> I've noticed a trend in older folk where they get halfway into a sentence knowing they don't know the name of what they are referencing only to ask those around them for help identifying the reference on the way to making some other point, e.g., "That's like that time in It's a Wonderful Life where _______, gosh, I can't think of his name. What was the guy that was in that movie? Oh yeah, James Stewart. That's like the time in It's a Wonderful Life where yada yada yada"

I'm hopeful that voice recognition assistants will help the burden during Christmas visits :D

</comment></comment><comment><USER33> This is great. I've been working on voice systems for VR and AR applications. On the HoloLens, it's a dream once you have your entire interface speech enabled. Can't wait to start porting to Android. Daydream and ARCore apps are going to see a huge improvement.

</comment><comment><USER34> These end-to-end speech recognition systems are very intriguing. One major limitation is that since they don't model phonetics, they have no great way to deal with highly irregular orthography that doesn't show up in the training data. For example, there is no great way for the system to learn that the pronunciation "black" can be spelled "6LACK" sometimes.

The paper on arXiv goes into how they deal with this. Basically they run a traditional WFST decoder over the output of the RNN-T to take spelling context into account. Still, it's impressive how far the neural system can get with no explicit lexicon or acoustic modeling in general.

<comment><USER35> I'm wondering if we'll see it write emoji at some point.

</comment></comment><comment><USER36> Hrmm, Gboard only? Does it mean they don't/can't use this model for voice commands? I do sometimes dictate messages to my phone but my main use of Android voice recognition is Android Auto commands to navigate or play music.

</comment><comment><USER37> Call me when it can figure out my wife's Italian name, pronounced correctly :-(

<comment><USER37> Downvoter(s): sorry, but it's true. Google can't figure out my wife's name, which is pretty freaking lame, as she's the person I interact with the most in terms of sending messages and emails and whatnot.

<comment><USER38> Perhaps check if this could work for you?

[https://www.google.com/amp/s/www.howtogeek.com/340108/how- to...](https://www.google.com/amp/s/www.howtogeek.com/340108/how-to-add- phonetic-names-to-contacts-in-android/amp/)

</comment></comment></comment><comment><USER39> Call me cynical but I cannot picture Google not tapping into everything you run through their voice recognition software, even if it does work offline. Doesn't mean it won't phone home later.

<comment><USER40> For what? They only really make their money on ads for things you're actively searching for. Everything else they have in ads works rather poorly in the text world. Trying to interpret interests out of task-driven voice commands is way beyond their capabilities.

But, enough of that. I'm holding out until decent voice dictation is standard everywhere and a well understood engineering problem with good open source implementations.

Mostly so I don't have to type address into my car's GPS.

<comment><USER41> I think this is only true under the assumption "voice recognition (and transcription) is solved". I think most would consider this assumption to not be true. If it is not true, then there is value to that voice data, as it can be used to help train.

I guess another assumption could be "they have more than enough voice data to train any future network improvements." While I feel they have a lot of voice data, I am skeptical to say that they wouldn't view more data as useful, or at least potentially useful.

I cannot think of a compelling reason for why they would stop collecting data all of a sudden. Can you? (Serious question, I don't mean to sound snarky)

<comment><USER40> I think you're completely right. I interpreted data collection here as a mining activity for ads preferences. Currently, they fully collect the voice recordings and only seem to provide options on whether the data is connected to your account. I don't see any option to "forget what I said after you've responded."

</comment></comment></comment></comment><comment><USER42> Finally. Over the past year or so I've noticed significant increases in the voice recognition lag across a handful of devices and across multiple wireless carriers.

</comment><comment><USER43> Voice on my pixel 3 is incredible. I normally have problems with voice recognition but this understands me better than some friends I have. It really is magical.

</comment><comment><USER44> What's so special about it? Just tried this on the BlackBerry keyboard and there it works instantly without being connected to the internet as well.

</comment><comment><USER45> Google and its dominance in both AI and reach into everyone's private lives really scares me.

There is a machine that can work totally offline, listen to audio, transcribe it, have a basic understanding and blast me with ads everywhere I go in the digital universe.

It can then psycologically slowly manipulate my behavior via ads making us buy/do things that we don't even realize it.

It's gonna be a scary world for my kids.

</comment><comment><USER46> It would be nice if Siri would at least allow me to turn cellular data back on with a voice command. Turn-to-turn navigation tends to consume a lot of data when I'm abroad using a temporary SIM so I drive without network connection on offline maps but that kills Siri meaning I can't do anything anymore without touching my phone.

<comment><USER47> Disable mobile netork for Maps specificially, while keeping it on for everything else.

</comment></comment><comment><USER48> Now they can save valuable CPU time and your phone will extract advertising keywords from your conversations for them, even without an internet connection. It's way more efficient to cache speech converted to text while offline rather than audio clips. The servers get cleaned up text data, saving bandwidth and storage.

</comment><comment><USER49> For application purposes where you don't want to source the audio from the microphone, is it possible for an Android application to feed audio to Gboard in order to source audio from other sources than the microphone? Maybe the Pixel has a mixer which allows audio from sources other than the microphone?

</comment><comment><USER50> There is an excellent overview over their speech recognition system. [http://iscslp2018.org/images/T4_Towards%20end-to- end%20speec...](http://iscslp2018.org/images/T4_Towards%20end-to- end%20speech%20recognition.pdf)

</comment><comment><USER51> This is an impressive engineering feat. Imagine the applications at edge devices! Microsoft is also trying hard to get their "Intelligent Edge" right.

</comment><comment><USER52> At the risk of being downvoted, any Pixel users enable "Hey Google" recognition on their phones only to regret it?

I'm constantly dealing with the phone interpreting commands intended for a Google Home speaker, which sometimes results in both the speaker and the phone acting on the same command. To my dismay, there's no way to disable Hey Google recognition on the phone after it's been enabled.

Perhaps someone here has run into this issue as well? It's a huge pain point for me.

<comment><USER53> You can register different trigger words, or at least you used to be able to. I had my phone wired to OK Google and the Home wired to Hey Google. Wasn't an issue once I made the distinction. I no longer have an Android so I can't comment on this still working. If you have external parties at your home regularly, that would obviously complicate things.

</comment></comment><comment><USER54> I never understood the need for server-side speech recognition. Did an internship in 2013 for speech recognition on a BeagleBoard with Julius ([https://github.com/julius-speech/julius](https://github.com/julius- speech/julius)), the thing worked with ninety-ish % accuracy (japanese language) and delay comparable to what my tablet gives - but locally.

<comment><USER55> I always figured doing it server-side was just to capture the users' data,either because the company wants a big training set, or for more nefarious purposes like tagged advertising.

</comment></comment><comment><USER56> Hmm, is this how the "what song is playing" feature works? Google claims it works offline (I haven't tested it) but I have a hard time believing that Google is storing information related to every song out there. What about new songs?

<comment><USER57> It was covered on HN when this feature was released. There's a database of >10,000 song fingerprints on-device (IIRC, people found it and it was ~100MB) that's updated based on the most popular songs from GPM/Youtube Music.

[https://venturebeat.com/2017/10/19/how-googles- pixel-2-now-p...](https://venturebeat.com/2017/10/19/how-googles-pixel-2-now- playing-song-identification-works/)

</comment></comment><comment><USER58> been using Google Voice for several years now for most of my communications in text, email, slack, whatever (only on phone, of course).

it is quite good, and very fast. but it's still not there. it has trouble with nuances like "call" vs "called" \-- can't hear that suffix very well in regular speech. for me, it also has a _really_ hard time with pronouns.

many times I'll start off with regular speech, go to look at what was transcribed and notice a couple errors that would make me look like a fool, backspace the whole thing, and then repeating it all gain in a very robot like voice.

it's _almost_ there.

<comment><USER59> > _been using Google Voice for several years now for most of my communications > in text, email, slack, whatever (only on phone, of course)._

Just a heads up, Google Voice is the name of a product that offers telephony service including SMS, and has been around for a decade or so.

[https://www.google.com/voice](https://www.google.com/voice)

</comment></comment><comment><USER60> got it this morning on the way in to work. already used it a bunch and its GREAT.

</comment><comment><USER61> Dictation works offline on iPhone since iOS10

<comment><USER12> Offline transcription has been available on Android as well. This is an announcement of a faster, more accurate model.

</comment></comment><comment><USER62> Finally google has caught up to 1997: [https://en.wikipedia.org/wiki/Dragon_NaturallySpeaking](https://en.wikipedia.org/wiki/Dragon_NaturallySpeaking)

Sure it might work better now, but that's expected when computers are much more powerful than a pentium 100 with 32MB of RAM. Uploading voice to google servers for processing was always just a data grab.

<comment><USER63> Well, try to train your Dragon (no pun intended) and then let another person speak, possibly someone with a different accent.

GBoard works somewhat reliably with 0 training and my Italian accent, that's an apple to oranges comparison.

<comment><USER62> > Well, try to train your Dragon (no pun intended)

The training was in part due to the hardware limitations. Some versions let you skip the training or have only a small training session and let it learn as it went, I'd be surprised if googles systems weren't doing some sort of continuous learning too.

> and then let another person speak, possibly someone with a different accent.

It had profiles to handle that, although it needed training for each profile. With google does it automatically switch profiles or does it mess up what it's learned about you're voice? It's very opaque.

> GBoard works somewhat reliably with 0 training and my Italian accent, that's > an apple to oranges comparison.

I haven't tried the new system but all of google voice recognition has struggled with my Australian accent to the point of being unusable.</comment></comment></comment>


Bus riding, the best-kept secret of NYC transit - <AUTHOR> http://visualizing.nyc/bus-riding-the-best-kept-secret-of-nyc-transit/ <comment><USER3> When I lived in NYC I sometimes had to take the bus downtown from 86th street due to interruptions in subway traffic (which sometimes happened). I remember that during rush hour the buses would advance at a crawling speed there, maybe making three blocks every 5 minutes, which almost made it faster to walk.

So I think the big advantage of the subway system is that it is much faster in general. In less densely populated areas of New York the buses are a good (and necessary) alternative to the subway system since -as the author points out- the density of bus stops is much higher.

</comment><comment><USER4> Does Google Maps have transit coverage of NYC?

I used to know the tube map by heart, but I'm starting to forget - <USER2> Google every time is faster and more reliable (since it takes into account e.g. engineering closures).

<comment><USER5> It does indeed. Probably the most robust and reliable transit coverage of the providers.

</comment></comment><comment><USER6> They're far from a secret imho. Maybe I'm wrong but every one I ride is usually packed unless it's an odd hour. Also, to get to some places you'll pretty much need to ride a bus if you want to get there easily. La Guardia springs to mind. And if you're in Manhattan the cross-town buses can basically be an essential part of life.

But I agree with this article 100%. If you live or travel a lot in NYC learn the bus system, it will only help you.

</comment><comment><USER7> One of the numbers missing between the subway and the bus are the waiting times, the transit times and bus cancellations. What is the use of bus having more routes if it takes much longer to get there.

Nevertheless, interesting use of data.</comment>


Florida political attack ad takes issue with self-driving vehicles - <AUTHOR> http://www.theverge.com/2012/8/13/3240354/florida-driverless-car-attack-ad

<comment><USER2> Time to call Old Glory Insurance; <http://www.hulu.com/watch/2340></comment>


Ask HN: What are your favourite developer blogs? - <AUTHOR> I'm looking for best developer blogs for inspiration <comment><USER2> Hey you'd probably like a project I've built -

[https://discoverdev.io](https://discoverdev.io)

I've created a system that crawls 100s of engineering blogs and picks out 8-12 posts good links every day. I then tag and publish them.

You may find publications you like by going through the archive!

<comment><USER3> This is interesting. Do you have the source code for this somewhere?

<comment><USER2> No clue why you're down voted, but no as of now the source code is not open.

It's a bunch of small / medium sized scripts I've tied together to run the system - I don't think open sourcing it would make sense. Unless of course I spend a bunch of time documenting and structuring the code. Time I unfortunately don't have right now :(

But perhaps in the next couple of months!

</comment></comment></comment><comment><USER4> Joel on Software - joelonsoftware.com

Coding Horror - blog.codinghorror.com

Schneier on Security - [https://www.schneier.com/blog/](https://www.schneier.com/blog/)

Things of Interest - qntm.org

In Pursuit of Laziness - manishearth.github.io

Still Drinking - stilldrinking.org

Shallow Thoughts - [http://shallowsky.com/blog/](http://shallowsky.com/blog/)

Captain Nemo - [https://captnemo.in/archive.html](https://captnemo.in/archive.html)

</comment><comment><USER5> Raymond Chen (MSFT) - The Old New Thing

[https://blogs.msdn.microsoft.com/oldnewthing](https://blogs.msdn.microsoft.com/oldnewthing)

<comment><USER6> This man is a living history of Microsoft and one of the best resources for WINAPI development.

</comment></comment><comment><USER7> Used to be [http://steve-yegge.blogspot.com/](http://steve- yegge.blogspot.com/) but I feel like most "blogs" died at the same time. Internet too big now..

<comment><USER8> Or maybe you outgrew it.

</comment></comment><comment><USER9> Julia Evans, [https://jvns.ca/](https://jvns.ca/)

<comment><USER10> Also has a great list of other blogs, most of which are my favorites now too.

[https://jvns.ca/blog/2016/04/09/some-of-my-favorite- blogs/](https://jvns.ca/blog/2016/04/09/some-of-my-favorite-blogs/)

here's a copy+pastable list: [https://rachelbythebay.com/w/](https://rachelbythebay.com/w/) [http://danluu.com/](http://danluu.com/) [https://aphyr.com/](https://aphyr.com/) [http://www.elidedbranches.com/](http://www.elidedbranches.com/) [http://composition.al/blog/2015/03/31/what-isnt-a-key- value-...](http://composition.al/blog/2015/03/31/what-isnt-a-key-value-store/) [https://www.harihareswara.net/sumana/2016/03/29/0](https://www.harihareswara.net/sumana/2016/03/29/0) [http://blog.plover.com/aliens/dd/intro.html](http://blog.plover.com/aliens/dd/intro.html) [http://blog.jessitron.com/2015/06/ultratestable-coding- style...](http://blog.jessitron.com/2015/06/ultratestable-coding-style.html) [http://akaptur.com/blog/2014/08/03/getting-started-with- pyth...](http://akaptur.com/blog/2014/08/03/getting-started-with-python- internals/) [http://accidentallyquadratic.tumblr.com/post/142387131042/no...](http://accidentallyquadratic.tumblr.com/post/142387131042/nodejs- left-pad) [http://embedded.fm/blog/takingaparttoys](http://embedded.fm/blog/takingaparttoys) [http://gigasquidsoftware.com/blog/2012/09/20/7-john- mccarthy...](http://gigasquidsoftware.com/blog/2012/09/20/7-john-mccarthy- papers-in-7-weeks-1/) [https://mathbabe.org/2016/03/30/apple-vs-fbi-nobody- won/](https://mathbabe.org/2016/03/30/apple-vs-fbi-nobody-won/)

</comment></comment><comment><USER11> Plug: The blog of my file manager [1]. I share my experiences developing a cross-platform app with PyQt. For instance:

* how Google Chrome's auto-update technology works [2]

* which GUI technology to pick if you find Electron too slow [3]

* a video of the release process of my app [4], using the fabulous `desk` tool [5]

[1]: [https://fman.io/blog](https://fman.io/blog)

[2]: [https://fman.io/blog/google-omaha- tutorial/](https://fman.io/blog/google-omaha-tutorial/)

[3]: [https://fman.io/blog/picking-technologies-for-a-desktop- app-...](https://fman.io/blog/picking-technologies-for-a-desktop-app-in-2016/)

[4]: [https://fman.io/blog/live-coding-video-of- fman/](https://fman.io/blog/live-coding-video-of-fman/)

[5]: [https://github.com/jamesob/desk](https://github.com/jamesob/desk)

</comment><comment><USER12> Antirez (the creator of Redis): [http://antirez.com](http://antirez.com)

Nick Craver (lots of StackOverflow / StackExchange stuff): [https://nickcraver.com/blog/](https://nickcraver.com/blog/)

Coding Horror: [https://blog.codinghorror.com/](https://blog.codinghorror.com/)

Troy Hunt: [https://www.troyhunt.com/](https://www.troyhunt.com/)

Steve Yegge: [https://steve-yegge.blogspot.com/](https://steve- yegge.blogspot.com/)

</comment><comment><USER13> If you want to learn Haskell or functional language:

Haskell for all- [http://www.haskellforall.com/](http://www.haskellforall.com/)

Bryan O'Sullivan's blog- [http://www.serpentine.com/blog/](http://www.serpentine.com/blog/)

Monday Morning Haskell- [https://mmhaskell.com/](https://mmhaskell.com/)

Jane Street- [https://blog.janestreet.com/](https://blog.janestreet.com/)

Haskell Weekly- [https://haskellweekly.news/](https://haskellweekly.news/)

Neil Mitchell's Haskell Blog-[http://neilmitchell.blogspot.tw/](http://neilmitchell.blogspot.tw/)

Types & Functional Programming- [http://bitemyapp.com/](http://bitemyapp.com/)

Haoyi's Programming Blog- [http://www.lihaoyi.com/post/WhatsFunctionalProgrammingAllAbo...](http://www.lihaoyi.com/post/WhatsFunctionalProgrammingAllAbout.html)

</comment><comment><USER14> [https://fgiesen.wordpress.com](https://fgiesen.wordpress.com) Most of his posts end up on the first page of hn. In fact there is one right now

[http://nullprogram.com/](http://nullprogram.com/) makes C look fun

</comment><comment><USER15> Mike Ash's blog: [https://www.mikeash.com/pyblog/](https://www.mikeash.com/pyblog/)

Always impresses me with the depth put into each post. Great blog for a Cocoa developer to follow.

</comment><comment><USER16> [http://rachelbythebay.com/w/](http://rachelbythebay.com/w/)

</comment><comment><USER17> I really appreciate the work of Bartosz Milewski: [https://bartoszmilewski.com/](https://bartoszmilewski.com/)

</comment><comment> grey-area If you use Go, you may like this link aggregator I made based on HN [https://golangnews.com](https://golangnews.com) which has a lot of good blog posts on programming in Go.

</comment><comment><USER18> Striked me as odd that this is not mentioned: [http://ayende.com/blog](http://ayende.com/blog) one of the strongest developers in the .net world, hands down.

</comment><comment><USER19> Mechanical Sympathy by Martin Thompson [https://mechanical- sympathy.blogspot.com/?m=1](https://mechanical-sympathy.blogspot.com/?m=1)

<comment><USER20> All of his and fellow Azul Systems guy Cliff Click's talks are worth watching as well, particularly if you care about Java, JVM, low level performance, processor caching.

</comment></comment><comment><USER21> This isn't updated very often but there is a lot of good stuff here.

[https://www.joelonsoftware.com](https://www.joelonsoftware.com)

<comment> grey-area Second this, lots of good posts there, along with Steve yegge's blog. They feel like postcards from the past but there are still a lot of gems. My personal favourites are:

[https://www.joelonsoftware.com/2002/01/06/fire-and- motion/](https://www.joelonsoftware.com/2002/01/06/fire-and-motion/)

[http://steve-yegge.blogspot.co.uk/2006/03/execution-in- kingd...](http://steve-yegge.blogspot.co.uk/2006/03/execution-in-kingdom-of- nouns.html)

</comment></comment><comment><USER22> [https://utcc.utoronto.ca/~cks/space/blog/](https://utcc.utoronto.ca/~cks/space/blog/)

[https://bentrask.com/](https://bentrask.com/)

[http://nullprogram.com/](http://nullprogram.com/)

[https://drewdevault.com/](https://drewdevault.com/)

[https://joeyh.name/blog/](https://joeyh.name/blog/)

[https://bsdly.blogspot.com/](https://bsdly.blogspot.com/)

[http://apenwarr.ca/log/](http://apenwarr.ca/log/)

</comment><comment><USER23> For the most part, I prefer blog/news aggregators to individual blogs. For example, I used to visit F# Weekly every week:

[https://sergeytihon.com/category/f-weekly/](https://sergeytihon.com/category/f-weekly/)

</comment><comment><USER24> [http://tonsky.me/](http://tonsky.me/)

</comment><comment><USER25> Most recently I'm a fan of [http://devcraftweekly.com/](http://devcraftweekly.com/) which some of my employees started as a side project. It's a weekly newsletter on the "Art, Craft and Lifestyle of Software Engineering" but there's also a blog format: [https://devcraftweekly.com/archives/](https://devcraftweekly.com/archives/)

Have a look and check it out. Focus is on development & software engineering as well as remote work and work-life balance.

<comment><USER26> Also we accept suggestion on good blog posts. We read and curate the bests posts we find. Send quality links our way!

</comment></comment><comment><USER27> Devlog of the game "Obra Dinn":

[https://forums.tigsource.com/index.php?topic=40832.0](https://forums.tigsource.com/index.php?topic=40832.0)

</comment><comment><USER28> We curate a big list of interesting blogs here :

[https://github.com/learn-anything/blogs](https://github.com/learn- anything/blogs)

<comment><USER29> These lists are not useful at all honestly, just a mesh of links with no context or info why it would interest someone.

<comment><USER30> You speak for yourself, I hope. Because I (for one) found this useful.

</comment></comment></comment><comment><USER31> You can check out engineering blogs of popular tech companies and start-ups. Here are few curated lists -

1\. [https://www.cybrhome.com/topic/engineering-blogs-of- companie...](https://www.cybrhome.com/topic/engineering-blogs-of-companies)

2\. [https://github.com/kilimchoi/engineering- blogs](https://github.com/kilimchoi/engineering-blogs)

Full disclosure - I'm maker of CybrHome.

</comment><comment><USER32> wingolog.org

Andy Wingo's texts about compilers and optimisation. Very nice, especially if you like scheme.

</comment><comment><USER33> The morning paper by Adrian Colyer: [https://blog.acolyer.org/](https://blog.acolyer.org/)

<comment><USER34> I love this blog!

This is less of a dev blog, though, and more of a research paper blog.

</comment></comment><comment><USER35> I wrote a Reddit comment a few months back listing around 20 of my favorite React-related bloggers: [https://www.reddit.com/r/reactjs/comments/5t8loz/what_are_yo...](https://www.reddit.com/r/reactjs/comments/5t8loz/what_are_your_top_reactreact_native_blogs_that/ddkzooh/) .

</comment><comment><USER36> Off the beaten path, but very interesting still - [https://bvckup2.com/wip](https://bvckup2.com/wip)

It's less of "blog" now, but older entries are a breadcrumb trail of the day- to-day grind of an indie Windows developer.

</comment><comment> github-cat [https://www.pixelstech.net/application/techtrend/](https://www.pixelstech.net/application/techtrend/) This is not a blog though, it shows you the popular technology stuff at the moment you open it.

</comment><comment><USER25> Shameless self-promotion: our Bugfender blog [https://bugfender.com/blog/](https://bugfender.com/blog/)

We share technical posts about mobile development, about scaling our backend and database systems and our experience using Go. On the other end of the spectrum we also write a lot about building a bootstrapped lifestyle business without any external investment.

Some examples:

\- [https://bugfender.com/blog/go-pros-cons-using-go- programming...](https://bugfender.com/blog/go-pros-cons-using-go-programming- language/)

\- [https://bugfender.com/blog/three-years- bugfender-9-5m-users/](https://bugfender.com/blog/three-years- bugfender-9-5m-users/)

\- [https://bugfender.com/blog/why-a-lifestyle-business- beats-a-...](https://bugfender.com/blog/why-a-lifestyle-business-beats-a- startup/)

</comment><comment><USER37> [http://pruthvishetty.com/bookmark#interesting- blogs](http://pruthvishetty.com/bookmark#interesting-blogs) (Not exactly developer blogs, but something the tech community would appreciate).

</comment><comment><USER38> I like factorio's Friday facts. It often has some interesting engineering knowledge.

<comment><USER39> Seconded, [https://www.factorio.com/blog/](https://www.factorio.com/blog/)

</comment></comment><comment><USER40> Not a blog per se, rather a 'vlog'. Really easy to follow video tutorials [https://www.youtube.com/c/JamieBartonCodes](https://www.youtube.com/c/JamieBartonCodes)

</comment><comment><USER41> I built something exactly for this!

[https://engineered.at](https://engineered.at)

I don't have a lot of free time lately but have a few more features queued up, the biggest being automatic topic tagging of articles. Soon!

</comment><comment><USER42> [http://blog.capitalandgrowth.org/blog/](http://blog.capitalandgrowth.org/blog/)

Investor interviews and sales & markting Q&As

</comment><comment><USER43> I say every time I see one of these HN posts: Windytan, phenomenal hardware hacker: [https://www.windytan.com](https://www.windytan.com)

</comment><comment><USER44> There used to be a blog on Ruby called Sneaking Ruby through the System. It was awesome but unfortunately it stopped back in 2010 amidst the Rails craze.

</comment><comment><USER45> [http://thedailywtf.com](http://thedailywtf.com) \- learned some very good lessons from it.

</comment><comment><USER46> [https://rachelbythebay.com/w/](https://rachelbythebay.com/w/)

</comment><comment><USER47> nullprogram.com

</comment><comment><USER29> I like Dadgum's blog. [http://prog21.dadgum.com/](http://prog21.dadgum.com/)

Being an older developer (please don't crucify me he's old compared to people I've worked with), his insight is really interesting to me.

I also use Stacknews to follow tools that interest me. This site aggregates articles and I can read them inline. It's like RSS on steroids.

[https://stackshare.io/news](https://stackshare.io/news)

<comment><USER48> Prog21 is a goldmine, it's a shame he's retired it.</comment></comment>


Ask HN: Scalable way of logging page request data from a PHP application? - <AUTHOR>

A web application I am developing (in PHP) requires the ability to log each page request.
Just like a normal access_log, it will store details like url requested, source ip address, date/time but I also need it to store the User ID of the logged in user (which is stored in a php session variable).
This data will then be queried to create site-wide or per user analytics reports as required at a later date - things such as total number of visits/unique visits, page views in a certain time period, geo-locating the ip addresses and looking at locations, most active times of day, most active members etc.
The obvious thing to do would be to have a mysql insert statement on each page but if the application is receiving thousands of req/sec, this is going to be a hugh bottleneck on the database so I am looking at alternative, scalable ways of doing this without big infrastructure requirements.
A few of the ideas i've had are:
1) Work on a way for Nginx (it's hosted on Nginx with PHP-FPM) to be able to log the user_id from the session/application in the normal web server access_log, which can be parsed and loaded into a database periodically (nightly). This feels like a bit of a hack and will need doing on each web server as the system scales out.
2) Log each page request into Redis which has high write speeds - the problem with this is the lack of ability to query the date at a later date.
3) Log each page request into either Memcache/Redis acting as a cache (or a message queue) and from there it would be regularly extracted, inserted into MySQL and removed.
4) Would something like MongoDB which has more query capability be suitable?
I'm interested in how HN readers would approach this and if anyone has any experience of a similar application (or has come across anything online).
I'm also interested on thoughts on how the data could be suitably structured to be stored in memcache/redis.
Thanks <comment><USER2> My $0.03

The first thing you need to answer is whether you'll do data processing on select, on insert, or as a mix of both.

On select, you only insert raw data (like an access log). You can then pick one of two ways to generate your reports and transform your data. You can do them in real time - when a user requests the data. Or you can do it as a batch process and do the whole OLTP to OLAP offline.

On insert you calculate everything as each hit comes in and fill in your reporting structure directly.

Both the select with real-time report generation and the insert approach give users access to real time data. Select with an offline transformation script will provide cached data until the batch process is run again.

Imma guess that you're users want as-real-time as possible, with some data having to be more real-time than other. This is where you adopt some form of hybrid/mix.

As for actual implementation...First, can you afford to lose data? Now don't jump the gun and immediately say "no". A lot (but certainly not all) of analysis of this type of data works on averages and general trends. If you happen to lose a couple hundred rows in a couple hundred thousands, it doesn't really change the outcome that much. Anyways, it's important to know this because some storage engines support non-durable writes (writes to memory) which are stupid fast, but might result in lost data (or you can possibly write your own buffering logic in the app/web code to spin off another thread and write every 500 hits in a bulk-insert).

Technologies. MongoDB has a couple things going for it. First, writes are fast, and they can either be done to memory (stupid fast) or to disk (or to X replicas). Even writes to disk are pretty damn fast (to X replicas will largely depend on network latency). The other thing MongoDB has going for it is MapReduce...sadly it's single threaded, but whether you are doing real-time reporting on an offline OLTP->OLAP transformation, MapReduce is significantly more powerful than OrderBy.

In chapter 6 of the (free) Little MongoDB Book (<http://openmymind.net/mongodb.pdf>) I actually outlined the initial OLTP to OLAP process we were using for a very modest reporting tool.

Since then, we've switched to Redis and do it all on insert to provide real- time analytics. However, we are only tracking 3 very basic incrementing statistics. You can see the code here: [https://github.com/mogade/mogade- server/blob/master/app/mode...](https://github.com/mogade/mogade- server/blob/master/app/models/stat.rb) (the hit method).

Of course, why don't you just use Google Analytics?

 _EDIT:_

If your data processing needs are truly huge, you'll likely end up with something like Hadoop (which I haven't had the opportunity/pleasure to need, so I can only mention it in passing).

<comment><AUTHOR> Thanks - much to consider!

I can't use Google Analytics because the data needs to be tied to the user id of the logged in user.

</comment></comment><comment><USER3> How much traffic are you really expecting on this project? Dont pre-optimize, most of the time a regular ol' MySQL table will do just fine and people over- worry about performance too early in the development process.

Stick it into MySQL, if it causes problems later, worry about it then (or email me from my email in the profile)

<comment><AUTHOR> I'm probably looking at up to a few hundred req/sec.

Thanks for the advice.

</comment></comment><comment><USER4> Writing to mySQL on every page request is generally a "bad idea" unless you aren't getting much traffic (in which case, it's the easiest solution).

The scalable way to build something like this would be to log your raw data to a service like Scribe Server (open sourced by Facebook) or rsyslogd and then aggregating it in a background process. The aggregated data can be stored in mySQL or whatever datastore you prefer.

<http://scribeserver.svn.sourceforge.net/> <http://www.rsyslog.com/>

(Scribe is a bit more complicated to setup and hook into PHP, but is a much better choice for a highly distributed/scalable system)

<comment><USER5> <http://www.google.com/search?q=premature+optimization+is>

<comment><USER4> I agree completely- most likely OP doesn't need a solution this complex- but it doesn't make any answer any less correct.

</comment></comment></comment><comment><USER5> <http://www.google.com/analytics/> probably does 90% of what you need; add a more user-focused analytics program (there are plenty) and you'll probably get more than you need, executed better than you could yourself.

ps: the inability of many developers to go with a "good enough" solution or to question "requirements" always baffles me. What you think are requirements are probably just rough guesses at what the client might want (I know, I've been writing requirements for a living for 10 years). This sounds like exactly a case of that, although I may be wrong of course.

<comment><AUTHOR> The problem with Google Analytics is that I need the data to be tied to the user_id of the logged in user.

</comment></comment><comment><USER6> If you do go the MySQL route be sure to check out INSERT DELAYED <http://dev.mysql.com/doc/refman/5.5/en/insert-delayed.html>

<comment><AUTHOR> Yes, I use this but the problem is still that if you fire large amounts of traffic at it then you get a lot of these building up and still create a bottleneck.

</comment></comment><comment><USER7> No problem with you asking this here, but you might find this question is better suited to StackOverflow

<comment><AUTHOR> I tried that but didn't get any replies at all....

</comment></comment><comment><USER8> How about putting log lines into memcached, then once a minute write them to disk?

<comment><AUTHOR> This is something I considered but i'm unsure of the best way to store these so they are retrievable without looking up the key.</comment></comment>


Chromium "WontFix" persistence of delete on close cookies - <AUTHOR> http://code.google.com/p/chromium/issues/detail?id=128513

<comment><AUTHOR> Strange that a browser overrides a developers cookie settings.

<comment><USER2> Only if the browser is set to "where I left off" mode. Then it keeps cookies for active tabs, so you are still logged in when you open the browser again. You can change it in settings.</comment></comment>


Ask HN: What value is there in downvoting on HN? - <AUTHOR> I don't see the value in downvoting on HN at all. Users with the ability to downvote will downvote comments they disagree with rather than comments that don't add to the conversation, and since downvoting drives users away, the ones that are left are cold, unfeeling, and happy to live in their bubbles while attempting to isolate everyone else from things they disagree with. <comment><USER2> _" the ones that are left are cold, unfeeling, and happy to live in their bubbles while attempting to isolate everyone else from things they disagree with"_

I think you've just insulted (and mischaracterized) a large number of HN users.</comment>


Troubled £12bn NHS IT system to be scaled back - <AUTHOR> http://news.bbc.co.uk/1/hi/uk_politics/8397854.stm

<comment><USER2> I wonder if this is an example of the "big bang" system. Recall the concept that "large complex systems that work were once small working systems." Or put another way, if it doesn't work when it is small, can it be expected to work when it is large?

Do we know how to build these sort of things organically? I recall the giant FBI system that was scrapped. And it sounds like the FAA ATC system is too large to be replaced.

<comment><USER3> Except the fact that plenty of large private organizations manage to do things like this every day. The NHS is huge but no bigger than Walmart's worldwide organization. But if Walmart spent $25 billion dollars on a system they'd do it with the knowledge that it has to be done right or they'll drive themselves out of business.

While with government agencies you have politicians, who don't know what they're talking about, defining artificial timelines based on political concerns. So the project implodes 7 years later and no one holds the individual politicians accountable.

</comment></comment><comment><USER4> Ahem. This is a _software_ system, right?

Okay, you need some beefy server infrastructure. Heck screw that, let's just build an entire datacenter, we're the government after all. A billion dollars should get you something nice, flower tapestry and all.

So, and then we need this ubercomplex software system. That, like, stores a data-record for a few million user^Wpatients. We probably want to tie that to chipcards and stuff.

Boy such a system can be expensive to build. I'd say a few hundred millions for sure. But hey, it's almost christmas, let's be generous and clock in another billion here - those oracle licenses don't pay for themselves.

Okay, so, excuse me if I'm dense. But what exactly have the other 10 billion dollars been spent on?</comment>


Manage stakeholders and get design into your agile process - <AUTHOR> https://medium.com/@jonatisokon/managing-stakeholders-applying-agile-to-design-6809c2c0613

<comment><USER2> Too many teams get nearly to the end zone, then end up punting.

In contracting, the biggest challenge is selling the solution to the immediate client product owner. If that person isn't completely bought in, it takes a lot of time and effort to overcome their desire to farm ultimate ownership of the deliverable out to a broader stakeholder committee or focus group. In the projects I've worked, at least, that broader stakeholder community is often quite receptive to getting a deliverable vs. asked for design opinions.</comment>


Ask HN: Office politics while working from home - <AUTHOR> Now that a lot of people are working from home: how did office politics change? Is there less of it or do you see a continuation of office politics with other means? Another option is that one is not supposed to talk about this topic... <comment><USER2> There are politics in online games and forums where nobody sees each other. I've been in Slack groups with lots of private gossip channels. I don't think office politics will change much.

One of my favourite quotes: "Civilized men are more discourteous than savages because they know they can be impolite without having their skulls split, as a general thing." \- Robert E. Howard (Conan)

The lack of any physical response may make people even ruder with one another. Though it will cut down on office affairs and the like.

</comment><comment><USER3> I work 2 hours less per day. I take a lot more breaks, and no one seems to notice. As long as they think I am as productive as ever, it's all good, I guess, sort of?

We know how this would normally play out in the office world: "ah! You can work more than before, amazing. You now need to work more. Oh, and we're simply a small startup, the pay raise will be the same, regardless of your performance."

Now I do get my promotions in more free time.

</comment><comment><USER4> It's different.

A lot of the bullshit you'd be aware of due to being able to see people is renewed because everyone hasn't figured out how people operate.

I think we'll see how it works out as companies start purging.</comment>


ISP's top data hog gobbles 2.7TB of data in a month - <AUTHOR> http://arstechnica.com/tech-policy/news/2010/08/isps-top-data-hog-gobbles-27tb-of-data-in-a-month.ars <comment><USER2> That doesn't seem ridiculous - one full hard drive backup over the Internet.

<comment><AUTHOR> I think that Ars agrees with that. It's not that much but it still is if you compare it with the normal average which is about 6-8 GB

</comment></comment><comment><USER3> our hoster charges around 7 euros per TB (and I think they still make a cut). So that would be 20 Eur this costumer is costing them (for a plan that is at least 30 Eur), hardly newsworthy.

<comment><USER4> Server bandwidth pricing != Consumer internet access pricing.

<comment><USER3> I know, but does the difference really justify the news? 2 TB isn't that much.</comment></comment></comment>


Python programming exercises [pdf] - <AUTHOR> http://joaoventura.net/static/files/python_exercises_book.pdf <comment><AUTHOR> Hi HN, this is a book with some programming exercises that I have been using on my classes so that students can get up-to speed with python.

It is quite incomplete and with a lot of errors. Since my available time is short, and I'm not a native english speaker, this is the best result given the time available for it. I plan to include other things and correct the errors eventually.</comment>


If Germany Can't Quit Coal, Can Anyone Else? - <AUTHOR> https://www.wired.com/story/if-germany-cant-quit-coal-can-anyone-else <comment><USER2> Quitting coal is easy, people just don't want to accept paying more for their electricity as well as the NIMB folks that are anti solar, nuclear and wind.

I would happily pay 5x my current energy costs if it meant we could not only halt, but begin to reverse, the damage we've done to our planet.

If we just outlawed the construction of new fossil fuel plants and put timelines in place to take them all offline, people would get over having solar panels on the roof of their neighbors house and be much more inclined to stop listening to lobbyist nonsense about how 'dangerous' nuclear power is and we'd see facilities popping up as fast as they could be built. You can safely bet you'd start finding a lot more energy efficient appliances, processors etc too.

Sure, nuclear still has disposal issues and the sun doesn't always shine and the wind doesn't always blow but it's better than what we're doing now... I'd rather have some site in the desert full of radioactive waste than to have 4/5 of the planet turn into desert from increasing heat.</comment>


Reverse Engineering an Integrated Circuit for Pwn2Win 2017 CTF - <AUTHOR> http://blog.dragonsector.pl/2017/10/pwn2win-2017-shift-register.html <comment><USER2> Reading blog posts like this this is kind of like listening to Usain Bolt talk about how to run fast. Step 1: be better than anyone else. Step 2: work crazy hard.

<comment><USER3> > Step 1: work crazy hard. Step 2: be better than anyone else.

FTFY.

<comment><USER4> If by fix you mean completely change what was being expressed to the opposite, sure.

<comment><USER5> Alright then:

> Step 1: work crazy hard. Step 2: be better than anyone else. Step 3: work > crazy hard.

FTFY.

</comment></comment></comment></comment><comment><USER6> I've actually written the intermediate section of that in production code. The DEF netlist format contains a "wire" as a set of rectangles linking an input A to outputs B,C,D...etc, and we wanted to show only those rectanges involved in e.g. the A-D part of a route. The solution was to make a graph and then use standard graph traversal algorithms to select only the needed rectangles.

This reverse engineering challenge is made a _lot_ easier by all the preparatory work of decapping and photographing the layers not being necessary.

If you like this sort of thing, you may like the Visual 6502: [http://www.visual6502.org/](http://www.visual6502.org/)

</comment><comment><USER7> This is awesome, nice work!</comment>


Music Recommendations in Hyperbolic Space - <AUTHOR> https://arxiv.org/abs/1907.12378 <comment><USER2> Authors learn embeddings in a Poincaré space rather than Euclidean, better for hierarchical relationships. The reference paper is a good read: Nickel & Kiela '17 "Poincaré Embeddings for Learning Hierarchical Representations", [https://arxiv.org/abs/1705.08039](https://arxiv.org/abs/1705.08039)

<comment><USER3> The use of hyperbolic space to encode hierarchies is absolutely genius (in not small part because it should be obvious), given the properties of hyperbolic space to "expand" around fixed points exponentially, exactly like trees do* . It seems like a continuous generalization of trees essentially. Beautiful.

*: [https://en.wikipedia.org/wiki/Hyperbolic_space](https://en.wikipedia.org/wiki/Hyperbolic_space)

"Another distinctive property is the amount of space covered by the n-ball in hyperbolic n-space: it increases exponentially with respect to the radius of the ball for large radii, rather than polynomially. (...)"

<comment><USER4> Genius because it should be obvious? Yes, it should be obvious for people familiar with both areas, and it is actually known for quite a long time. Hyperbolic geometry is used for visualizing hierarchical data since Lamping- Rao 1995 and Munzner 1998, then there were papers about hyperbolic SOMs (2001 IIRC) and lots of papers about the Hyperbolic Random Graph model for scale- free networks. I would say that the paper linked above introducing "Poincare embeddings" (a rather poor name IMO, BTW) does not feel as impressive if you know the details and the earlier work.

</comment></comment></comment><comment><USER4> We did some research a few years ago which was similar in some aspects. We have mapped roguelikes and boardgames into the hyperbolic plane based on the frequency of two games being mentioned together on Reddit. Here are our maps:

Roguelikes description: [http://www.roguetemple.com/z/hyper/reddit.php](http://www.roguetemple.com/z/hyper/reddit.php)

Roguelikes interactive: [http://www.roguetemple.com/z/hyper/rogueviz- online.php?c=-la...](http://www.roguetemple.com/z/hyper/rogueviz- online.php?c=-lab+-dftcolor+383838FF+-canvas+101010+-sag+1+-color+2+-gload+3+-noplayer+--msg0&1=rv%2Froguelikes%2Fedges.csv&2=rv%2Froguelikes%2Fcolor.csv&3=rv%2Froguelikes%2Fcoord-67.txt)

Boardgames description: [http://www.roguetemple.com/z/hyper/redditboard.php](http://www.roguetemple.com/z/hyper/redditboard.php)

Boardgames interactive: [http://www.roguetemple.com/z/hyper/rogueviz- online.php?c=-la...](http://www.roguetemple.com/z/hyper/rogueviz- online.php?c=-lab+-canvas+101010+-sag+1+-color+2+-gload+3+-noplayer+--msg0&1=rv%2Fboardgames%2Fedges.csv&2=rv%2Fboardgames%2Fcolor.csv&3=rv%2Fboardgames%2Fcoord-67.txt)

Since similar games are mapped together, and popular games tend to be close to the center, our maps work great as a recommendation system too. They also look great :)

</comment><comment><USER5> Of all the music recommendation systems out there, this one is my favorite:

[http://www.gnoosic.com](http://www.gnoosic.com)

It has an uber simple interface (you don't even need to log in) but no other system gets me as well as this one.

What surprises me in the given article is the graph on the last page. Why was the treatment group still listening more weeks after they already were getting the old recommendations again?

I wish the graph would reach out further into the past then just one week before the test. So one could get a feel for how much of the variance in the graph is random.

<comment><USER6> This is literally the only recommendation engine I've ever had any success with. IIRC it was written by someone on HN

<comment><USER7> I'm working on a music recommender right now, I'd love your input: [https://news.ycombinator.com/item?id=20584508](https://news.ycombinator.com/item?id=20584508) Mainly my work has focused on re-recommending songs you already know, e.g. in an internet radio kind of setting. I'm thinking of incorporating something like gnoosic for recommending new artists.

</comment></comment></comment><comment><USER8> [https://old.reddit.com/r/MachineLearning/comments/ckm44u/190...](https://old.reddit.com/r/MachineLearning/comments/ckm44u/190712378_music_recommendations_in_hyperbolic/)

</comment><comment><USER9> An interesting idea. What else might this be applicable to? Social networks?

<comment><USER4> Hyperbolic geometry is used in the modelling of social networks and other scale-free networks, see the Hyperbolic Random Graph model.</comment></comment>