Data sets released by Google - <AUTHOR> http://svonava.com/post/62186512058/datasets-released-by-google <comment><USER2> If you want to play around with data, here's another good list of open/free datasets: [http://bitly.com/bundles/hmason/1](http://bitly.com/bundles/hmason/1)

<comment><AUTHOR> Nice, thanks! I wish there was a cleaned up repository of datasets like these, in a unified format, directly accessible to a public MapReduce engine like Elastic at AWS.

</comment></comment><comment><USER3> here's some other data hubs/search engines, endless lists:

[http://datahub.io/](http://datahub.io/)

[http://blog.bigml.com/2013/02/28/data-data-data-thousands- of...](http://blog.bigml.com/2013/02/28/data-data-data-thousands-of-public- data-sources/)

[http://tm.durusau.net/?p=39312](http://tm.durusau.net/?p=39312)

[http://dvn.iq.harvard.edu/dvn/](http://dvn.iq.harvard.edu/dvn/)

_____________

this subreddit seems like a decent place to ask questions

[http://www.reddit.com/r/datasets](http://www.reddit.com/r/datasets)

</comment><comment><USER4> Another one from Google, 1000 scanned books for OCR and other scanned document processing research: [http://commondatastorage.googleapis.com/books/icdar2007/READ...](http://commondatastorage.googleapis.com/books/icdar2007/README.txt)

</comment><comment><USER5> [http://bitly.com/bundles/hmason/1](http://bitly.com/bundles/hmason/1)

[http://commondatastorage.googleapis.com/books/icdar2007/READ...](http://commondatastorage.googleapis.com/books/icdar2007/README.txt)

[http://datahub.io/](http://datahub.io/)

[http://blog.bigml.com/2013/02/28/data-data-data-thousands- of...](http://blog.bigml.com/2013/02/28/data-data-data-thousands-of-public- data-sources/)

[http://iatiregistry.org/](http://iatiregistry.org/)

[http://open.undp.org/](http://open.undp.org/)

[http://data.worldbank.org/](http://data.worldbank.org/)

[https://explore.data.gov/catalog/raw/](https://explore.data.gov/catalog/raw/)

[http://www.data.gov/opendatasites](http://www.data.gov/opendatasites)

[http://data.gov.be/datasets](http://data.gov.be/datasets)

[http://opencorporates.com/](http://opencorporates.com/)

[http://glasspockets.org/work/reportingcommitment/api.html](http://glasspockets.org/work/reportingcommitment/api.html)

[http://thedata.harvard.edu/dvn/](http://thedata.harvard.edu/dvn/)

[http://www.reddit.com/r/datasets](http://www.reddit.com/r/datasets)

[http://archive.ics.uci.edu/ml/](http://archive.ics.uci.edu/ml/)

[http://cleandatahub.org/](http://cleandatahub.org/)

[http://datacatalogs.org/](http://datacatalogs.org/)

[http://archive.org/details/oxford-2005-facebook- matrix](http://archive.org/details/oxford-2005-facebook-matrix)

</comment><comment><USER6> BitTorrent Please! Why does it cost so much? They grabbed our data for free and they have enough free Bandwidth. Let's assume they are greedy, then they could at least offer it through BitTorrent. DVD's for that amount of data is ridiculous. I don't even have a DVD-Reader…

 __ _Can 't afford buying all that + shipping to Europe, but would like to play with the Data for my NLP Project._ __

<comment><USER7> I agree ! I too can't afford it but would really love to play around with that data because i'm just beginning to learn about NLP and stuff. I too feel that shouldn't have been priced and not in a DVD!

</comment></comment><comment><USER8> Here's another good one.

[http://archive.ics.uci.edu/ml/](http://archive.ics.uci.edu/ml/)

</comment><comment><USER9> Here is a good one, [http://cleandatahub.org/](http://cleandatahub.org/) They are trying to aggregate cleaned data sets across the web.

</comment><comment><USER10> no links...

Remember the days when people used to make links on the web because they weren't greedy with their pagerank?

At least Google left us some machine learning data sets after they took all the links. You just can't find them because nobody links to them.

<comment><AUTHOR> I'm sorry for not making it more obvious, but each bullet point in the list ends with a link.

</comment></comment><comment><USER11> Fantastic links throughout this thread.

When playing with new programming languages instead of a 'todo' list I always end up building an XKCD password generator. Interestingly enough, I've never found a frequency/comprehension list worth using to populate it for public consumption.

</comment><comment><USER12> Is there any data set that embodies human relationships with every day objects ?

</comment><comment><USER13> Also:

[https://code.google.com/p/word2vec/#Pre- trained_entity_vecto...](https://code.google.com/p/word2vec/#Pre- trained_entity_vectors_with_Freebase_naming)

</comment><comment><USER14> The ML competition site Kaggle should also get a mention here. [http://www.kaggle.com/competitions](http://www.kaggle.com/competitions)

</comment><comment><USER15> Where is the Web1T dataset? Would you not consider it useful for Machine Learning?

<comment><USER16> I think this list only includes free datasets.</comment></comment>


A laser-powered space elevator? Seattle team leads NASA contest - <AUTHOR> http://www.techflash.com/seattle/2009/11/seattle_team_leads_nasa_power-beaming_space_elevator_contest.html?ana=from_rss

<comment><USER2> As you all likely know, there is currently a 'ghost fleet' (<http://tinyurl.com/lof685>) of huge, empty container ships anchored off of Singapore, due to recession-induced lack of shipping demand.

I was thinking that these would make a great ocean-going launch platform for rockets, or a mobile anchor point for a space elevator.

Having an ocean-going anchor point is good because it solves three problems easily: you ensure that the anchor is always on the equator, you can move it around slightly in the event of storms / upcoming changes in shipping / air traffic / etc, and you aren't chewing up a large chunk of probably valuable real estate.

You might cable the ships together (i.e., turn them into a giant pontoon raft) and then put a platform over the top. Weight is not much of an issue--your flotation would be incredible, and most elevator designs maintain an outward pull anyway. You could convert some of the ships to act as a ferry service bringing cargo to the elevator.

<comment><USER3> But the biggest cost would be the elevator itself, rather than the platform required to hold it. But still, it sounds like something from a Bruce Sterling novel, so it gets my vote.

<comment><USER2> > But the biggest cost would be the elevator itself,

Of course. But the cost of the elevator itself is essentially fixed, regardless of where you build it. The only things that vary are the cost of the location under the anchor point and how optimal that location is.

There isn't a tremendous amount of land on or near the equator (cf <http://www.wall-maps.com/World/PetersProjection-over.gif>); most of Earth's landmass is well up in the northern hemisphere. You've basically got Brazil, part of Africa, and Indonesia. When we want to build our (2nd, 3rd, ..., nth) space elevator, we're going to need to start looking at ocean mounts. Plus, as I mentioned, it has some built-in advantages.

</comment></comment></comment><comment><USER4> <http://news.ycombinator.com/item?id=923793></comment>


Poor Sleep Prevents Brain From Storing Memories - <AUTHOR> http://techzwn.com/2013/01/poor-sleep-prevents-brain-from-storing-memories/

<comment><USER2> I think there's an evolutionary advantage to this. If you don't commit new memories when sleep-deprived, then you don't remember how rough the first few months of parenting a newborn can be (you get very little sleep during that time) and you're more likely to have more children.

</comment><comment><USER3> "The hippocampus stores short-term memories, while the prefrontal cortex is our long-term memory bank." This statement runs contrary to all the current concepts of information storage in the brain. The hippocampus is where long term memories are stored, the prefrontal cortex is involved in working memory, planning and evaluating action outcomes and a bunch of other things we have barely figured out.

</comment><comment><USER4> Can I remind everyone "Correlation does not imply causation"?

<comment><USER5> And in this case there is an obvious omitted factor (which is important to consider when levying this allegation) age. Perhaps being old makes you sleep worse and have trouble retaining memories, but sleep has nothing to do with memories at all (although we have lots of reasons to suspect that it does). They would also have found arthritis, type II diabetes, prostate cancer, and a host of other things reduced memory _if their design was as described by the article_. Given that they have fMRI data it would be fairly simple to run a correlation between quality of sleep and retention within an age cohort and see what you get. Given that they got funding to do the MRIs I'm suspicious that in the actual study they did, I'm not sure you need scans to know the elders often don't sleep well.

But if that works, perhaps it was weight causing poor sleep and memory retention. Or alcohol abuse. I'm not saying this research can't be done but it's a lot more trustworthy if you either 1. randomize subjects and do a treatment or 2. have a much larger dataset and statistically control for these suspicious factors.

</comment></comment><comment><USER6> An interesting family that suffers from a specific sleep deprivation disorder where they simply stop sleeping in middle age and die a few months later.

<http://www.npr.org/templates/story/story.php?storyId=6503414>

</comment><comment><USER7> Great link. I discovered I am highly allergic to linden trees this summer, my first allergy, and was only sleeping 3-4 hours a night, maximum. I was so confused and forgetful of peoples names.

Granted, I also partied non-stop the whole summer.

</comment><comment><USER8> And yet sleep is always the first thing to be sacrificed when we "need to get stuff done".

<comment><USER9> If you use an SRS flashcard program, you'll see strong correlations between days you didn't get enough sleep and lowered retention rates.

Similarly, staying up all night to study for an exam is silly.

<comment><USER10> It's not silly if you never paid attention in any of the classes and you're learning the stuff for the first time. I know that's happened to me before, especially for a lot of the easier courses. Some knowledge that has a high probability of not being retained is better than no knowledge. :)

This is of course a stupid solution to a completely different problem, but those courses didn't inspire me to try my best... if I had another chance, I don't know if I would try harder. I know that sounds bad.

<comment><USER11> No, it sounds like a cost benefit analysis - the only concern is how to judge if a course is worth learning _before_ conducting the analysis.

"I don't need no Calculus" is a fail in the analysis, "I dont need no Media Studies and the effect of the Female Eunach on Porn Pay per TV channels" is not.

<comment><USER10> That assumes that gaining knowledge is the only point of taking a course. Easy credits are also a viable reason. So are required credits.

</comment></comment></comment></comment></comment><comment><USER12> Been there, done that. Inadequate deep sleep also causes your body to skip the muscle/tissue repair process that happens overnight. I wound up with tendonitis this way and eventually burnt out.

</comment><comment><USER13> I think dreaming and the process of storing memories is a key piece of solving Artificial intelligence. IE we must teach the artificial brain to sleep in order for it to store and sort memories. For me it seems similar to a garbage collect function of the brain.

"Do Androids Dream of Electric Sheep?"

<comment><USER14> Your view is very anthropocentric.

</comment></comment><comment><USER15> Your brain is a neural network of neural networks, during sleep, a cost function is applied across the entire grid. Important aspects of your day are done, and redone at high velocity, simultaneously (leading to dreams).

Cost benefit analysis are done against what you might have done, and the results of that, and actions that would have caused more desirable outcomes are projected, as best as it can see, and the habits, and motor neurons are reconfigured accordingly, this explains why when you get good sleep, and you wake up, you find yourself much better able to do tasks than had you not slept. If you don't sleep, you die.

Source of these points:

[http://www.amazon.com/How-Create-Mind-Thought- Revealed/dp/06...](http://www.amazon.com/How-Create-Mind-Thought- Revealed/dp/0670025291)

<https://www.coursera.org/course/ml>

Title is misleading, this function also has to do with encoding short term memories to long term memories. Since the mind only has limited space (limited number of neurons to configure), that only the most useful memories are stored into permanent disk. Disruption of the 7 to 9 hour sleep cycle garbage collects the memories that were about to be stored. The mind queues them up to be dealt with the following day, but sometimes are displaced or missed by more passionate things in the present.

Sleep is one of the most important things you can do to maintain your mind and keep it in top running condition for as long as possible, not too little, not too much, sleep in intervals of 90 minutes. If you consume garbage knowledge on a daily basis, your mind will encode that garbage to permanent disk, and you will become that garbage.

Conspiracy theorists suffer from a mental misconfiguration where the cost function applied to the neural network of neural networks suffers from "over fitting". Finding patterns in randomness leading to conclusions are not valid. A lambda function can be applied against the cost function which will alleviate this. I can do it in software, and when I discover the operating principles of the neo cortex, I will be able to fix all the conspiracy nuts in the local nut house. Take care to not take for granted the fresh slate of your mind while you are young, because when you are old, it'll be mostly full and encoding new skills to disk much more difficult, the cost function is more reluctant to modify the grids since doing so would damage your ability to consume resources, find mates and create more of you. Fill you mind with timeless wisdom and get good sleep before your hard disks become full.

<comment><USER16> "If you don't sleep, you die."

No human has ever died from simply not sleeping (excluding accidents, etc caused by lack of sleep)

[http://www.scientificamerican.com/article.cfm?id=how-long- ca...](http://www.scientificamerican.com/article.cfm?id=how-long-can-humans- stay)

[http://www.abc.net.au/health/talkinghealth/factbuster/storie...](http://www.abc.net.au/health/talkinghealth/factbuster/stories/2012/03/13/3451196.htm#.UQaQ0idEE1M)

<comment><USER17> See also <http://en.wikipedia.org/wiki/Thai_Ngoc> <http://en.wikipedia.org/wiki/Al_Herpin></comment></comment></comment>


Australian state government suspends registration of Uber drivers - <AUTHOR> http://www.abc.net.au/news/2015-09-28/uber-drivers-face-suspension/6808582 <comment><USER2> Each Australian state and territory is handling Uber's entrance into the market differently, but what they all agree on is that Uber doesn't fit the current regulations for private cars.

So far it seems the states are more interested in blocking and fining drivers than reviewing the laws and regulations around private drivers.

I met a driver in Melbourne who was fined twice. He said that Uber paid the fine both times. NSW's policy is far more severe for the drivers; deregistering drivers or vehicles has very real repercussions (could lead to criminal convictions if the driver isn't smart about it). I would say its likely a response Uber's payment of fines.

In Melbourne, taxi drivers recently protested Uber by refusing to pick up passengers. Their strike coincided with a public transport strike (unrelated to Uber) and generally caused a lot of chaos. Because of the costs behind taxi ownership, those protesting drivers took a pretty huge pay hit out of their own pockets to protest.

Seems its an increasingly dangerous time to be an UberX driver.</comment>


Show HN: NetPatch Full-Featured Android Firewall, Full Control Over Your Network - <AUTHOR> https://firewall.netpatch.co/ <comment><AUTHOR> NetPatch Firewall is a very early project, Any questions and suggestions are very useful for it. Please send it here.</comment>


Evolution of the x86 context switch in Linux (2018) - <AUTHOR> http://www.maizure.org/projects/evolution_x86_context_switch_linux/ <comment><USER2> Can someone say why is using the TSS still mandatory with software-based task switching? Is this a requirement imposed by x86?

In looking at the OS dev wiki I see the following:

>"The TSS is primarily suited for hardware multitasking, where each individual process has its own TSS. In Software multitasking, one or two TSS's are also generally used, as they allow for entering Ring 0 code after an interrupt."

Would you not be able to enter Ring0 after an interrupt with a TSS entry? Is this why it is still required?

<comment><USER3> The interrupt stack pointer comes from the TSS. Without that you're still running on the untrusted user stack with no way of bootstrapping a kernel context without corrupting the user state.

<comment><USER2> Oh right, without a mapping its a "chicken and egg" situation. Cheers.

</comment></comment></comment><comment><USER4> I recall reading a paper comparing Linux and Solaris context switch times in ~98 and Linux was 10-100X faster. Solaris did something incredibly slow and safe.

<comment><USER5> Real context switches on Solaris were very slow which is why they had LWP. But Linux process context switches were also faster than Sun's LWP switches.

</comment></comment><comment><USER6> Enjoyed this article, anybody know the significance of adding the do..while(0) loop within the macro starting Linux 1.3?

Was curious if it guards against some C pre-processor issues.

      /** include/asm-i386/system.h */
      #define switch_to(tsk) do {
        [...]
      } while (0)

<comment><USER7> Good answer here: [https://stackoverflow.com/questions/257418/do- while-0-what-i...](https://stackoverflow.com/questions/257418/do-while-0-what- is-it-good-for)

<comment><USER8> Just to add more context, this is a very common cpp (c pre-processor) idiom. You'll find it in most non-trivial C projects somewhere.

<comment><USER9> Abbreviating the C Preprocessor as cpp is very confusing imho.

<comment><USER3> It's a common abbreviation older than C++. You used to even be able to run the c pre processor on arbitrary non-C files by using the cpp command.

<comment><USER8> You still can.

</comment></comment></comment></comment></comment></comment><comment><USER10> An addition to "Linux 2.2 (1999)" is: introduced meltdown vulnerability. That was the then-unknown cost of software context switching.

Later, with Red Hat's 4g4g kernels that Linus rejected, the problem would go away for people who installed Red Hat's version of the OS on systems with many gigabytes of memory.

<comment><USER2> Can you elaborate? How does relying on pure TSS for context switching prevent meltdown?

What were the 4g4g kernels? Might you have any literature and/or on those?

<comment><USER11> Separate address space for kernel and user. Hardware will use TSS to switch address space as needed for syscalls.

<comment><USER2> Do you the reason why Linus rejected this idea?

<comment><USER10> The main reason seemed to be the relatively bad performance of the hardware task switch (loading segment registers and the page table base) that would be required for any system call.

Of course, that turns out to be the fix for meltdown, unless you have the process-context identifiers (PCID) available on Haswell chips and newer. The meltdown fix for older CPUs, such as the Pentium III and Intel Core, is roughly the same as the 4g4g kernel changes.

BTW, the 4g4g kernels were created for a different reason. The kernel needed more virtual address space for itself, and thus couldn't share with user code. This was for a time when people were trying to run 32-bit kernels on systems with 32 gigabytes of RAM.

<comment><USER2> >"Of course, that turns out to be the fix for meltdown, unless you have the process-context identifiers (PCID) available on Haswell chips and newer."

Using TSS based switching is incompatible with PCIDs? Or is it incompatible with separate address spaces for user space and kernel space?

PCIDs are process ID tags on cache lines correct?

<comment><USER10> TSS switching is incompatible with running in 64-bit mode. It is very slow. Doing most of the same actions (reload segment registers and the page table base) in software is also very slow. Prior to the meltdown issue, Linux system calls had avoided most reloads of the segment registers and page table base.

PCIDs are incompatible with older hardware. They are modestly slow. I think the PCID state includes the TLB.

That pretty much means the kernel must support both methods. The PCIDs is used when possible. When the hardware doesn't support PCIDs, Linux must instead reload segment registers and the page table base, either step-by-step in software or via a TSS switch.

BTW, I had to implement x86 hardware task switching for an x86 emulator. The complexity is insane. See my "Who is Hiring?" post if that sounds fun for you.

<comment><USER12> I think you may be a bit confused here. PCID is a feature that lets the kernel avoid a TLB when CR3 is written. With or without PCID, CR3 gets written. The segment registers have nothing whatsoever to do with this on a 64-bit system.

<comment><USER10> The explanation is a bit awkward because I'm trying to describe numerous situations. (pre-2.2 kernels, post-meldown kernels, in-between kernels, 32-bit builds, 64-bit builds, using PCID hardware, not using PCID hardware...) It looks like 7 of the 12 possibilities are valid, more or less.

Some segment registers are reloaded on a 64-bit system. That includes CS, DS, SS, and GS. The 32-bit systems must additionally reload ES. All segment registers are loaded for Linux 2.0 and older, via hardware task switching.

CR3 does not get written for system calls when running on a normal Linux from version 2.2 until the meltdown workaround hit.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER12> Nice article!

If you want to have lots of fun, you could look at switch_mm() on a modern kernel :)

</comment><comment><USER13> This is really well done, bravo.

</comment><comment><USER14> Very thorough. Nice job.

</comment><comment><USER15> anyone knows what Ingo Molnar is doing these days?

<comment><USER16> A ton of stuff related to eBPF, last I saw.

</comment></comment><comment><USER17> I really like how good the article looks when printed. I enjoy reading long, in-depth articles much more when I can read them in print. Unfortunately many blog posts need a lot of tweaking until I can get an acceptable print result. This one looks good enough without any trickery.

Thanks to the author, for caring about the paper people. : )

<comment><USER18> out of curiosity, since it changes how the page prints and I haven't experimented much: have you tried printing while in the browser's "reading" mode? Or does that tend to be worse?

<comment><USER19> Not OP,but if it looks good on "reading" mode it prints nicely. Plus you can adjust font size and print width to conform to your taste.</comment></comment></comment>


Ask HN: What is a startup? - <AUTHOR>

Okay this might belie the idea that there are no dumb questions, but I can't really define what a startup is. I think to myself that I should start one, perhaps a micro-startup, but I don't have a cohesive idea of what that involves. Do you have any Startups for Dummies explanations? <comment><USER2> @robfitz built a really cool startup tool kit: <http://thestartuptoolkit.com/>

I'm pretty sure that if you can complete most of these boxes you'll have found a startup.

</comment><comment><USER3> Colloquially, it's any new business. Average folks will think of a new retail storefront, or new one-man consulting business, as a 'startup' -- and there's no need to disabuse them of that usage, it's reasonable in that context.

In technology circles, it especially connotes a business with extra reward and risk, trying something novel.

In the customer-development/lean-startup formalization of positive startup patterns, Steve Blank defines 'startup' as "an organization formed to search for a repeatable and scalable business model". See:

[http://steveblank.com/2010/01/25/whats-a-startup-first- princ...](http://steveblank.com/2010/01/25/whats-a-startup-first-principles/)

Are you searching for a repeatable and scalable business model? If so, you're a startup! If you form a partnership or corporation to search with more collaborators, then that formation, too, would be a startup.</comment>


The Camera Behind The New Pluto Photos - <AUTHOR> http://www.theatlantic.com/technology/archive/2015/07/the-camera-behind-the-new-horizons-pluto-photos-ralph/398549/?single_page=true <comment><USER2> Article lacks pretty much any technical detail that would make it interesting. Whats the sensor size? What are the typical shutter speed? How do they compress the images? How do they compensate for the noise in now light? Do the cosmic rays impact image quality? Etc Etc.

So, duh.

<comment><USER3> Here you go :)

[http://www.boulder.swri.edu/pkb/ssr/ssr- ralph.pdf](http://www.boulder.swri.edu/pkb/ssr/ssr-ralph.pdf)

<comment><USER4> Although the good photos we are getting are from LORRI, just re-colored with Ralph data.

</comment></comment></comment><comment><USER5> > When you see Pluto looking tan- and sepia-toned in the new, high-resolution > photos, you're looking at data captured by Ralph.

This is somewhat misleading. The high-resolution photos are taken with the LORRI instrument and colorized with lower-resolution data from Ralph.

</comment><comment><USER6> The result is absolutely staggering. I've previously missed the Jupiter picture and had to stop and really look to appreciate just how amazing that photo is, let alone the Pluto one - from the same camera, a mind-boggling distance away from us.

<comment><USER7> Did you notice the aurora on Io?

<comment><USER8> It's actually a plume from a volcanic eruption, which is even crazier.

</comment></comment></comment><comment><USER9> Getting to that image of Jupiter I just had to stop and marvel. Both at God's grandeur in making it, and the ingenuity of the people who made a machine that could get there. And of the two creations, creating people who eventually learn to do that is probably more awesome than creating a planet.

<comment><USER10> I don't get why the down-votes. Is having a belief and expressing it wrong around here/against the forum rules?

<comment><USER11> Content doesn't have to be against rules to be downvoted. I disagree with the opinion in it, so I downvote. It has nothing to do with rules.

<comment><USER12> If someone mentions they like a food or a movie or a song that you don't like, do you downvote?

<comment><USER11> Depends how it's phrased. If someone says "I like liquorice" then I'm not going to downvote. But if someone said "liquorice is the best food in the world" as if it was a fact, then I am going to downvote. If OP said "It is my opinion that Jupiter was created by god" I would not have downvoted. But it's his stating it as a fact which is making me downvote.

</comment></comment></comment></comment></comment><comment><USER13> I'm interested in the data formats and image processing used for all this. I've heard a couple mentions in passing that they're "not in your standard JPG" but I'd like to know more.</comment>


Can a Java programmer be transitioned to COBOL? - <AUTHOR> http://web.archive.org/web/20050214210251/http://objectz.com/columnists/martin/02252002.asp

<comment><USER2> Java and COBOL are just two different languages. I am mainly a java programmer now but I took a COBOL class in college. It was boring. I could go write COBOL if it was requested of me, but why would I want to? COBOL in my opinion is only still alive because of the legacy code that is in place and the money that would be required to replace that code.

"To get business logic to work using these languages is difficult." Really? I don't think so.

<comment><USER3> I have professionally worked both as a COBOL and Java programmer for many years. I understand procedural and OO reasonably well.

While working as a COBOL programmer I was mainly concerned with business logic. All my thinking time was utilised in thinking about the business need, if I needed any clarification I'd talk to the business people who requested the work and they could understand a lot of the terminology I used. I would also do all the testing required.

As a java programmer I would spend a lot of time reading about and understanding various frameworks and have little time left to understand business logic. It is ok because the real testing would be done by other teams. Business people won't even approach me.</comment></comment>


US ends food stamp benefits as Congress debates more cuts - <AUTHOR> http://www.bbc.co.uk/news/world-us-canada-24767932

<comment><USER2> I wish you could down-vote. The title gives you the impression that the U.S. is getting rid of food stamps, but the article clearly states it's lowering the amount a family receives.

Misleading title and just bad reporting.</comment>


Weston on Raspberry Pi Accelerated - <AUTHOR> http://ppaalanen.blogspot.ro/2013/05/weston-on-raspberry-pi-accelerated.html <comment><USER2> The article on the reasons for doing this work is also very interesting: <http://fooishbar.org/tell-me-about/wayland-on-raspberry-pi/>

<comment><USER3> Yes, very interesting. Reminded of <http://prog21.dadgum.com/173.html>

</comment></comment><comment><USER4> Are there any precompiled binaries for this? The Raspberry PI build instructions for Weston look involved - <http://wayland.freedesktop.org/raspberrypi.html>

<comment><USER5> <http://raspberrypi.collabora.com/> though Daniel Stone is intending to update that (hopefully soon) so the .debs include a weston-launch script that does the necessary XDG_RUNTIME_DIR setting and misc setup for a simple easy to run demo.

</comment></comment><comment><USER6> Very cool. There seems to be a lot of very detailed posts from the Wayland/Weston developers recently, is this coincident with the Mir (for want of a better expression) fork?

Still, the synchronization and swapping between GLES and using hardware overlays is a problem that has been solved for sometime on Android. Unfortunately this magic maybe hidden in a proprietary framebuffer Android HAL or more recently the platforms hardware composer HAL.

</comment><comment><USER7> This is pretty actionable stuff. I really like the discussion around the trade-offs. So an interesting question is on my Zynq-7000 Zedboard what sort of Framebuffer would you design to be the "ideal" Wayland or Mir target?

</comment><comment><USER8> This is very good news although now I feel I've just wasted a day speeding up a home-grown software video player for the Pi.

</comment><comment><USER9> Really awesome work, had my hopes up for this and the video does not disappoint!</comment>


99 Prolog Problems - <AUTHOR> https://sites.google.com/site/prologsite/prolog-problems

<comment><USER3> Very nice, Prolog is a vastly under rated language which is very useful for a lot of logic and decision making programming problems.

<comment><USER4> I just started learning Prolog recently and I strongly agree with it's status as being 'underrated'. One thing I've found worth noting is the increasing interest semantic web technologies with no noticeable uptick in the interest in logic programming. This is especially odd when you consider that it's not much of a stretch to look at RDF triples simply as facts and rules in Prolog. I have a feeling that just as Rails rediscovered the mvc work in the late 70s smalltalk community, the semweb world will rediscover prolog along with the larger world of constraint/logic programming.

<comment><USER5> One problem with Prolog is that the "standard" doesn't include a lot of useful functionality that implementations typically provide. Not all Prologs have packaging systems or constraint programming extensions, for example, yet they make Prolog dramatically more useful. SWI Prolog (<http://www.swi- prolog.org/>) and GNU Prolog (<http://www.gprolog.org/>) are pretty good. (Daniel Diaz, GNU Prolog's primary author, also has some excellent research papers on implementing constraint programming.)

Also, for anyone curious about Prolog, I found experimenting with Erlang made many aspects of Prolog make more sense, and vice versa. Erlang was originally a Prolog DSL, and many minor details carried over.

<comment><AUTHOR> I also highly recommend The Reasoned Schemer as another route to understanding the concepts behind Prolog - you essentially build the heart of a Prolog system in a couple hundred lines of Scheme.

The level of density in such a short text as well the ability to work through it without having to sit in front of a computer is invigorating (I've had to reread the first 80 pages or so about 5 times now, of course it might just be that I'm slow ;). It certainly succeeds at showing the beauty of a system that elegantly intertwines FP w/ LP.

Oleg (of Haskell fame) has some excellent further resources, <http://kanren.sourceforge.net/> and William Byrd's thesis is a treasure trove of things to consider: <http://gradworks.umi.com/33/80/3380156.html>

<comment><USER5> William Clocksin's _Clause and Effect_ is a similarly styled book, using Prolog. It also has a couple good case studies.

I also _highly_ recommend Sterling and Shapiro's _The Art of Prolog_.

<comment><AUTHOR> I've also heard only good things about the Bratko book, Prolog Programming for Artificial Intelligence.

I'm curious as to what put you off about TRS? The only thing I found initially frustrating was the lack of hand-holding to be found in the earlier books in the Schemer series. But then I starting working out everything I found confusing w/ pencil and paper and found the going much easier.

<comment><USER5> I'm probably going to give it another try eventually, but I sold it - <USER2> preferred learning logic programming ideas through Prolog itself, rather than Scheme / KANREN.

I've heard good stuff about the Bratko book, too - there's a new edition being published in a few months, I'm planning on checking it out then.

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> Nice collection, would be awesome if there were a way to submit solutions online and have them tested.

</comment><comment><USER7> Great work. So well structured and easy to browse. Thanks

</comment><comment><USER8> same exercise for scala <http://aperiodic.net/phil/scala/s-99/>

</comment><comment><USER9> ...but a bitch ain't one. sorry.

<comment><USER10> Redeem yourself; run with the line and make it Prolog-specific.

<comment><USER11> I got 99 bitches but backward propagation ain't one.</comment></comment></comment>


Finding bugs in SQLite, the easy way - <AUTHOR> http://lcamtuf.blogspot.com/2015/04/finding-bugs-in-sqlite-easy-way.html?m=1 <comment><USER2> I actually (as a big C and C++ fan) quite a damning result for C.

Its easy to say "just programming well" when bugs occur in badly written projects, but SQLite is so well tested, and generally considered well written.

<comment> f- As others have said, parsers for complex formats (be it text-based or binary) are exceptionally hard. There are some classes of C/C++ software where the choice of a language doesn't have such a striking effect. For parsers, the effect is hard to ignore.

But parsers are also the kind of stuff you almost always end up writing in C/C++, and there are semi-compelling reasons for doing so - chiefly, performance and flexibility. You can disagree and make your pitch for Ocaml or JavaScript or whatever, but really, if we had clearly superior choices, we wouldn't be dealing with this problem today (or it would be a much more limited phenomenon). There are some interesting contenders, but the revolution won't happen tomorrow, no matter how much we talk about it on HN.

Perhaps a more fitting conclusion is that if you are parsing untrusted documents, our brains are too puny to get it right, and the parser really needs to live in a low-overhead sandbox. Mechanisms such as seccomp-bpf offer a really convenient and high-performance way to pull it off.

<comment><USER3> In C++, we have Boost.Spirit, and for C there is Bison/Flex. Surely those are better and more safe alternatives than hand-rolling your own parser?

<comment> f- Many of the horribly vulnerable parsers are generated with Bison / Flex, so it's not exactly a robust solution. Plus, especially for binary formats (images, videos, etc), it's hand-written or bust.

</comment></comment></comment></comment><comment><USER4> Who has the best automated fuzzing team... how fast can they grab this stuff (seconds from major open source repository commit time to automatic generation of functional exploit)? There's sure to be teams out there in the seconds/minutes range for a broad range of cases already.

</comment><comment><USER5> Link to afl-fuzz is broken in the post, should be [http://lcamtuf.coredump.cx/afl/](http://lcamtuf.coredump.cx/afl/)

</comment><comment><USER6> > I realized that the developers of SQLite maintained a remarkably well- > structured and comprehensive suite of hand-written test cases in their > repository

Tcl for the win[0].

[0] [https://www.sqlite.org/testing.html](https://www.sqlite.org/testing.html)

</comment><comment><USER7> I guess you can run afl-fuzz along side LLVM's LibFuzzer for even better results. Fuzzing is getting very interesting. =)

[http://llvm.org/docs/LibFuzzer.html](http://llvm.org/docs/LibFuzzer.html)

</comment><comment><USER8> Does anyone have a good writeup on getting started with afl-fuzz? I'm a Python developer and looked into it the other day for fuzzing Python programs, but gave up after a bit because of the documentation (it seems geared towards lower-level languages).

<comment><USER9> I'm sure you have checked out the official documentation ([http://lcamtuf.coredump.cx/afl/README.txt](http://lcamtuf.coredump.cx/afl/README.txt))?

Fuzzing Python programs with AFL will be hard. AFL leverages a version of the GCC compiler to instrument (add additional code to) the resultant binaries. Because Python is not a compiled language this will be difficult. I'm sure there is something you could do to make it work.

<comment><USER10> There's a wrapper for CPython to get it to implement the tracing code, afl- python[1]. Alex Gaynor did a basic intro to it a few days ago which is a better write-up that the project itself has! Of course, this doesn't work so well with mixed C/Python code.

[1]: [https://bitbucket.org/jwilk/python- afl](https://bitbucket.org/jwilk/python-afl) [2]: [https://alexgaynor.net/2015/apr/13/introduction-to- fuzzing-i...](https://alexgaynor.net/2015/apr/13/introduction-to-fuzzing-in- python-with-afl/)</comment></comment></comment>


Please stop using Ruby - <AUTHOR> https://dev.to/jimsy/please-stop-using-ruby-4lf1 <comment><USER2> I couldn't even keep reading through this when I got to the "Ruby is slow" section. Well, my beta testers beg to differ with you — the #1 feedback I've gotten on the app I'm working on is how fast it is. Ruby 2.5, Rails 5+, Turbolinks + Stimulus, and this thing _screams_.

Please folks, there are many articles out there with well-reasoned pitches why you might want to check out another language other than Ruby or another framework other than Rails. This article isn't one of them.</comment>


Show HN: Pinstified - Search Amazon With Pinterest Style Layout - <AUTHOR> http://pinstified.com

<comment><USER2> I made something similar a while back, but for books. Essentially it used the Amazon API to create a Pinterest like interface for Amazon. (But this was before Pinterest, so it was called a Masonry interface at the time.) Here is what I came up with:

<http://bookflavor.com/list/new-york-times-bestsellers-25>

Unfortunately, eventually Amazon decided they didn't like me and cut off my API access. This was because I was doing too much scraping of their service, and because they said the design of my site was causing it to falsely report referrals to Amazon. (Still not sure what was causing that exactly.) At any rate they shut off my API access and I haven't had time to come back and revisit the project since I have been working fulltime.

Best of luck with your site, and watch out for sneaky Amazon restrictions.

<comment><AUTHOR> Hey, thanks for the heads up and sorry to hear about your troubles. Your site is really beautiful. I too was using masonry before "pinterest type layout" became the more apt phrase. Hope you can get back to developing your site. I know those pesky full-time gigs get in the way of our passion, though they're a little more stable. Best wishes and please keep checking back. Any feedback would be helpful. Thanks again:)

</comment></comment><comment><USER3> Very cool :) This past weekend I was just thinking of starting something similar! but I know I would never achieve the great look that your project has.

Congratulations!

<comment><AUTHOR> Yeah it's been quite grueling getting everything working and it took more than a weekend, but it has been fun to work on. Happy hacking, thanks for the compliment, and look forward to seeing your projects:)

</comment></comment><comment><USER4> Fantastic app! Great work.

</comment><comment><USER5> Cool. How do you decide on order of the items in the search results? for instance:

[http://pinstified.com/search?searchCategory=search- alias%3Dj...](http://pinstified.com/search?searchCategory=search- alias%3Djewelry&searchDept=search- alias%253Djewelry&searchButtonInput=Jewelry&searchButtonRefineInput=&fieldKeywords=gold+emerald+ring&searchSubcategory=all&sort=relevance- fs-browse-rank)

vs

[http://www.amazon.com/s/ref=nb_sb_noss_1?url=search- alias%3D...](http://www.amazon.com/s/ref=nb_sb_noss_1?url=search- alias%3Djewelry&field-keywords=gold+emerald+ring)

Also, where do you get data on new and best selling? Is that just the "popular" breakdown on Amazon?

<comment><AUTHOR> You can sort the search results using the refine menu. <https://dl.dropbox.com/u/68927323/search_sort.png>

And

Like you said the "new and best selling" is just the popular breakdown for each category. Though there isn't one for "All Departments". We'll work on that:)</comment></comment>


N2O: Erlang Web Framework - <AUTHOR> http://kukuruku.co/hub/erlang/n2o-erlang-web-framework <comment><USER2> This framework is really out there. It is well ... different and interesting.

Here are some not so conventional things going on:

* Write your page in Erlang.

* Even translates Erlang to JS as a parse transform.

* Websockets (with a fallback mechanism) is the default connection mechanism.

* Don't want to use JSON for some crazy reason? That's alright, user Bert (and ship binary encoded Erlang terms to the browser!).

* Can render stuff on the server and send the whole thing over the websocket connection.

[https://github.com/5HT/n2o](https://github.com/5HT/n2o)

<comment><USER3> >Websockets (with a fallback mechanism)

Not enough frameworks make this as seamless as it should be. So it's refreshing to see it done here.

I've been looking at Phoenix (Elixir not Erlang) but the problem is that I'll have to do logic to detect when WS are available and do the fallback myself which results in more code server side.

I think this is something that should be handled by the framework itself.

<comment><USER4> I'm the author of Phoenix. I'm glad to hear you're giving it a look! I spent a lot of time thinking about the WS layer and fallback support. Ultimately I settled for standard WS with a small multiplexing layer on top. For those that need fallback, they can drop in websocket-js (flash fallback, compatible with native WS api). Have you taken at look at this for your fallback support? [https://github.com/gimite/web-socket-js](https://github.com/gimite/web- socket-js)

Phoenix is still far from 1.0, so any and all feedback is welcome.

<comment><USER5> I'm new to Elixir (and FP in general), but I really like the language. I came across a couple of youtube videos on Phoenix , and it looks really nice (reminds me of Rails). I think something like Sinatra would be even nicer, for 2 reasons: 1\. the standard these days is to have a rest api serving json to web/mobile clients, so a Rails-like framework seems overkill 2\. Pretty much all my friends who are learning web dev on their own, find Sinatra/Flask way easier to get started with than Rails/Django.

Regardless, Elixir (and Phoenix) is a great leap forward compared to the other options for building concurrent, reliable servers. I hope more people try it out before falling for hype/marketing ( _cough_ node.js _cough_ mongodb _cough cough_ )

</comment></comment></comment></comment><comment><USER6> I think there's still room to innovate in the Erlang/Web space. Chicago Boss was a nice improvement because it ties so much different stuff together. It does have a few issues though, like not being very 'Erlangy' in places: it uses lots of parse transform magic. I think it's the right direction though, in that it's fairly general purpose.</comment>


Sony, Sharp and Panasonic lose $21 billion in 2011 - <AUTHOR> http://finance.yahoo.com/news/sony-maps-revival-plan-cut-063224098.html

<comment><AUTHOR> The article is about Sony but thought that tidbit was more interesting, Apple and Samsung are trouncing the "old guard."

 _"The three companies expect a combined loss for the year just ended of $21 billion - more than Sony's entire market value, which has slumped by close to a fifth in the past month. Samsung is 10 times more valuable, while Apple, which Sony executives considered buying in the early 1990s, is worth 30 Sony's."_</comment>


The Ritalin Experiment, Day 4 - <AUTHOR> http://false-epiphany.com/2009/08/the-ritalin-experiment-day-4/

<comment><AUTHOR> Why Day 4 instead of Day 3 or Day 5?

It was a choice between Day 3 and Day 4 -- both of which indicated that the effect had disappeared.

Day 4 seemed more representative. Maybe because of this line: "I mention this because it took me about an hour between deciding to brush my teeth and actually doing it." This got to me. It captured what being distracted is. It's factual and yet subtly affecting.

Another reason for Day 4: Day 3 still sounds like the beginning. It sounds tentative. Like an experiment that might end after three days. Day 4 shows real commitment. It's for real. It says the author is in this for the long haul.

</comment><comment><USER2> I took it once b/c I have mild ADD... it doesn't work if you are sleep deprived...

See how it works if you get lots of exercise, reduce processed carbs and alcohol, and get plenty of sleep...

It helped me focus when I tried it, but then I realized I like being a bit more scattered.</comment>


Apple releases update to fix critical macOS High Sierra security issue - <AUTHOR> https://www.theverge.com/2017/11/29/16715246/apple-releases-high-sierra-root-security-patch <comment><AUTHOR> And here is the announcement from Apple:

Security Update 2017-001 // [https://support.apple.com/en- us/HT208315](https://support.apple.com/en-us/HT208315)</comment>


Stylish suburbs: how ancient Mexican metropolis dodged inequality trap (2019) - <AUTHOR> https://www.reuters.com/article/us-mexico-archeology-teotihuacan-feature/stylish-suburbs-how-ancient-mexican-metropolis-dodged-inequality-trap-idUSKBN1YN0J2 <comment><USER2> The article does not answer the "how" question.

<comment><USER3> The idea that the inequality trap was avoided, is demonstrated in the relics left behind - and in that regard the article does indeed present data to suggest that wealth was distributed because the culture was based around the distribution - free commerce - of skilled artisans producing valuable goods derived from locally available, abundant resources.

<comment><USER4> Essentially, low barrier to entry. Something that free education (not necessarily limited to degree-granting colleges, and perhaps living grants for self-study) and a robust safety net could achieve today.

</comment></comment></comment><comment><USER5> I don't know how much can be really inferred from digging through buried rubble. We know a lot about Rome because they had writing and people who lived then chronicling life there.

Commoners live well in free market societies, and there's no slavery. Perhaps that's how they lived.

For example, the article points out that the height difference between rich & commoners was not so much. But heights improved throughout America in the 1800's (quite dramatically, in fact).

<comment><USER6> > Commoners live well in free market societies, and there's no slavery.

Do you have evidence of this?

<comment> barry-cotter Infant mortality is lower for the bottom decile, almost certainly the bottom percentile in high and middle income countries now than for the very richest and most powerful before antibiotics. Calvin Coolidge's son died of an infected blister he got from a racket playing tennis.

In every continent bar Africa education, healthy lifespan and average wealth as proxied by GDP/capita have improved since WWII. Even in Africa education and health have improved, despite AIDS, and it looks like economic growth might finally take off.

Some of this is due to technological advances but capitalism beats the pants off all alternative models at that, as we can see from the history of the Soviet Union, which managed to almost keep up militarily with the US while spending twice to three times as much as a proportion of GDP.

Slavery is absent in all OECD countries and almost all that are substantially poorer and less economically free.

Yes, we have evidence of this.

<comment><USER7> I don't think looking at GDP/capita proxies quality of life at all.

If you and I pay one another $100 every week we'll both be boosting GDP and doing nothing of material worth aside from paying the taxes on the exchange.

I'm also not convinced that it's absent in all OECD countries given an explicit loophole allowing slavery of prisoners in the US constitution.

<comment> barry-cotter [https://ourworldindata.org/human-development- index](https://ourworldindata.org/human-development-index)

Positive correlation between human development index and GDP per capita; also positive correlation between GDP per capita and education and health.</comment></comment></comment></comment></comment>