The 3 Percent – Why Tech Has a Diversity Problem - <AUTHOR> http://techcrunch.com/2015/07/16/the-3-percent-why-tech-has-a-diversity-problem/

<comment><USER2> Finally, an article that actually talks about the real reasons, instead of blaming it on the hiring process.

There are numerous companies that would LOVE to hire black and latino programmers. In fact, I would go so far as to say that if you are black or latino (and are even halfway competent at programming), you actually have an ADVANTAGE in the hiring process because your race is so under-represented, and the companies want to satisfy their anti-discrimination policies.

The problem in these companies isn't that the candidates are being discriminated against (although there are, of course, plenty of challenges that still need to be overcome in that area). The problem is that the candidates just aren't there!

For every 50-75 white programmers that applied to work at my company, there was probably only 1 black person who applied, and that's not even taking skill level into account.

If we want to solve the racial equality and representation issues in tech, we have to start at the bottom of the funnel. We have to solve the problem at the education, support, and motivation level. We need to get younger black and latino kids and teenagers interested in programming, and we need to make sure that they have the right environment and support to pursue that dream.

</comment><comment><USER3> The problem is good people getting dragged into the pit with/by the bad. Every poverty stricken area has a few good kids just wanting to do the right thing and get out of poverty.

What if we could separate them out, shield them from the negative influence of shittyness and let them flourish?

The only solution is to move them away from the bad because the negative influencers and influences are everywhere, they're in the environment, the school, the home, even the media they relate to.

A boarding school style program would work perfectly. Or a housing project with verified "good" kids and their families. The area could save money on policing and push that straight to education.

As those kids grow, they feed back into the program, becoming tutors, educators, and investors themselves. Giving others in poverty area positive role models to relate to. In other words, removing the downward spiral and replacing it with a positive feedback loop.

<comment><USER4> I think this could be part of the problem. When I was growing up and my dad was getting good work and saving up money, my mom demanded that he move us out of our bad neighborhood. None of my cousins who stayed behind graduated high school, but my siblings and I all went to college.

I'm Hispanic and I never felt like that hindered my job prospects in tech.

</comment></comment><comment><USER5> In my personal experience, charter schools in the nyc area are working wonders. I think that we could learn some things from that program.

[https://en.wikipedia.org/wiki/Charter_schools_in_the_United_...](https://en.wikipedia.org/wiki/Charter_schools_in_the_United_States)

While I admit that there's a self-selection bias with these schools--kids with parents who care about their education will apply for them to attend these schools, and those kids are much more likely to do better in school regardless, which has a self-compounding effect of better classroom environments--the quality of education I see in charter schools tends to be higher than that of standard public schools. At least from what I've seen.

Of course, the author also raises a lot of other points that aren't specific to schools: parents who are incarcerated, safety in the neighborhood, etc...

</comment><comment><USER6> This is the "pipeline excuse" for why various groups are underrepresented in so-called technology jobs.

There is certainly some truth to this. On average some groups receive poorer education than other groups and one would expect them to be underrepresented to this extent given hypothetical fair hiring relative to their proportion of the general population.

But the question is whether some groups such as Blacks and Hispanics are underrepresented relative to their percentage of CS or other engineering degrees for example. USA Today published an article on this:

[http://www.usatoday.com/story/tech/2014/10/12/silicon- valley...](http://www.usatoday.com/story/tech/2014/10/12/silicon-valley- diversity-tech-hiring-computer-science-graduates-african-american- hispanic/14684211/)

It may furthe be added that in many contexts technology employers claim to be desperate for "qualified" technology workers, though they are vague about what "qualified" means. They often seem to imply that a CS degree means qualified.

In fact, technology companies appear extremely picky about who they hire, often informally citing "cultural fit" which is clearly not about technical qualifications despite the high profile statements that the problem is strictly technical qualifications.

In some respects, the technology industry is very diverse with many people from all over the world and many ethnic groups. It is certainly not lily white or even close. It is not NASA mission control in the 1960's with white male engineers with buzz cuts as far as the eye can see. Even the official diversity numbers from the companies, taken at face value, claim about thirty to thirty five percent are "Asian," most of whom would not be classified as white by most people. However, there are various groups, ethnic and otherwise that are rare.

The pipeline is only a partial explanation, one the company's promote because it absolves them of any responsibility.

</comment><comment><USER7> Because too many people believe in "meritocracy", but forget that there are no real objective measures to decide who is better than the other. So it all too often falls to people to decide, and people have biases.

</comment><comment><USER8> Full disclosure: I was at Startup Chile at the same time as the Regalli team. During pitch day they blew us away and I've been happy to watch their continuing success.

Last year I spent nearly the full year (along with others on my team) working to build a team, constantly attempting to hire for a variety of positions. For business positions we had a lot of female and minority applicants. For design positions there were female and minority applicants. For software there were very few (though we did over-represent in our hiring ultimately, because we lucked into some great candidates who happened to be ethnic minorities.)

Why is it that the %3 he refers to apparently more likely to seek business degrees, or art degrees than programming degrees? We certainly did have a diverse flow of candidates for those positions, but we did not for engineering. I don't think that metal detectors in schools are making people less likely to be a programmer.

I think its because programming has become a non-glamourous, non-respected profession. (Look at the wide use of the diminutive "coding" or "coder" and the patronizing words in the job applications "ninja rockstar"\-- these are words you use with people you don't respect who you think are gullible. "You just might be the ninja rockstar coder we're looking for! And just because you're so special we'll be paying you 45 whole thousand dollars! Isn't that neat?"

We're pouring money like crazy into education- I read recently of a New Jersey school system which was spending 20,000 a student, and producing bad results. We've spent trillions on the war on poverty-- in fact, I read recently that something like $87,000 a year is spent per poor person. Hell, if you just gave them that money they wouldn't be poor anymore! Years ago I read that of the money that goes into the welfare system only %25 of it actually goes out in checks to welfare recipients, with %75 going into overhead.

The problem isn't a lack of money- we're spending it. The problem is a lack of incentivizes to spend that money wisely. Can you imagine a private school getting 20k per head and not producing well educated kids? Sure, it could happen, but it wouldn't stay in business for generations like public schools do.

We recognize that government is inefficient at a lot of things- basically everything. So why, if this is the most important thing, do we leave it in the hands of government? Sure if you want government to fund education, that's a different issue... but government is not good at delivering it.

I say this as a graduate of public schools, with the unique advantage of having changed schools more than once a year on average from first grade to graduation from high school. All of them were terrible. I even went to a state wide magnet school for my final two years-- it was not terrible, (my physics teacher taught at stanford, for instance, and would come teach us while working on the results of his experiments) .... but even it was nowhere near what it could have been.

The schools in the middle class neighborhoods are terrible. Why would we expect the poor neigborhoods not to be worse?

<comment><USER9> "I think its because programming has become a non-glamourous, non-respected profession. (Look at the wide use of the diminutive "coding" or "coder" and the patronizing words in the job applications "ninja rockstar"\-- these are words you use with people you don't respect who you think are gullible. "You just might be the ninja rockstar coder we're looking for! And just because you're so special we'll be paying you 45 whole thousand dollars! Isn't that neat?""

While this leans a bit cynical, I do agree that it does accurately describe a considerable about of tech employment, and I really do think that this goes just as far toward explaining the lack of interest in software development positions as a failure of the educational system.

So there are really two things going on - to get more people into software development: first, we need to massively improve access to a good educational system, second, we need to convince the people who can access this educational system to become programmers instead of dental hygienists, mortgage brokers, lawyers, nurses, structural engineers, or physicians. Keep in mind that in San Francisco, dental hygienists earn roughly the same salary as software developers at the median, and nurses earn considerably more [1]. There are still good reasons to want to be one rather than the other, but age discrimination, job stability, and work-life balance are not areas where software development shines.

As it stands, I think it is critical to expand good educational opportunities, but I think that this will simply expand the number of people who decide that other career options are better. In short, we will go from having a lot of people who can't choose to become software developers because they don't have that educational opportunity to having a lot of people who choose not to become software engineers because they now have better opportunities.

[1] check regional salary information at us news best jobs, which is a roundup of BLS data.

</comment></comment><comment><USER10> > If we could ultimately shift the conversation away from why there are so few > Blacks and Latinos in tech and instead talk about why there are only 3% > graduating from some high schools, then I think we stand a real chance of > addressing this issue head on.

I don't think "shifting" the entirety of the conversation to that is a good idea simply because focusing on the first 13 years of education [Kinder-12th] is going to take 15-17 years to show significant impacts. It simply is too long of a period for a political solution to be practical in the near term [which, lets be honest, no politician is willing to commit to a 15+ year program to fix the education system and actually champion it for that long].

Raising the enrollment numbers in college is a 2-4 year issue that neatly fits in a politician's term where they can show results on their next campaign "LOOK WHAT I DID FOR YOUR KIDS?! DO YOU SEE THESE NUMBERS?!".

Anything longer than the 2-4 year window is simply too long for anyone to care in politics.

<comment><USER8> You're presuming-- and you're right-- that politicians don't actually care about fixing things, but about improving their election results. They are self interested. This is a classic example of the Principle Agent problem.

[https://en.wikipedia.org/wiki/Principal–agent_problem](https://en.wikipedia.org/wiki/Principal–agent_problem)

And it exists not just at the level of politicians, I've seen school administrators and teachers-- both of whome I genuinely believe want nothing more than to help the kids under their charge, compromise that principle because it benefits themselves. They're not even aware they are doing it.

So, fundamentally ,the incentives are not aligned with the outcome that benefits society.

Look at fruit at your grocery store. The incentives there are aligned, to a point. Your store gets a very healthy supply of clean, generally safe bananas, apples, oranges and the like. They are shipped across the globe to get you the best, biggest, perfect looking versions of those fruit. The main incentive needed was to deliver food that would not kill you that was safe to eat, and that incentive was met.

When it comes to schools, even the good ones are not delivering a good education. For instance, almost no high schools deliver the tools and skills one needs to be in the workforce. This is totally possible, but the administration of these schools is not focused on education but state requirements.

Now the fruit delivered to your grocer is also more sweet and sugary than these same fruits were centuries ago because they've bred them to please the customer, and that has some society issues that need to be addressed (being "healthy" and eating apples and bananas is too much like eating sugary snacks these days)... but at least we have a choice. If we all only shopped at organic or old style fruit stands the market would shift.

With education we have no choice. You can't chose where to send your kid and you can't even really make an impact on the system because there are so many entrenched interests.

I remember reading about one scandinavian school system (I think?) that had much better outcomes for less money by simply attaching the money to the students. The schools and the money were state run, but the parents decided which school their kids went to. Just that change, caused the schools to compete for students because they wanted the money to grow, and to do that they had to convince their parents that they were delivering a good education.

The incentives were aligned.</comment></comment>


PHP or Ruby? - <AUTHOR>

when it comes to choose programming language and framework which elements I have to consider? what are the criteria to trade off? <comment><USER2> As a language PHP has some problems. That is not to say that it is a bad language. I use PHP with CodeIgniter or Kohana (frameworks) on a regular basis.

That said, I believe that Ruby and Python are both more robust languages on the whole. I suggest doing some generic (read: non-web) scripting in the languages and see what feels right, including community, documentation, and libraries.

Good read: <http://www.codinghorror.com/blog/archives/001119.html>

</comment><comment><USER3> One thing i've always had to consider is what servers actually support Ruby. A few personal projects i've made in Ruby. However, when working on Web applications for other people (Who already have a server), they may not support Ruby (Or PHP for that matter).

I think it really comes down to preference. I learnt PHP first off, then discovered the power of Ruby.

I suggest taking a look at this: <http://www.killersites.com/blog/2006/php-vs- ruby/>

</comment><comment><AUTHOR> the support of the server is not my concern though it can be considered as one criteria + I don't have any preference or feelings about any framework!

So suppose I have an Idea and I'm searching to find out my co-founder who is a developer. shall I go for php developer or ruby? what are the elements which i should consider? and how this choice will effect me as the CEO in the future? for example is it hard to find a ruby developer or PHP? etc....

<comment><USER2> The answer to that: Don't pick them based on the language they are proficient with. If you find an incredibly sage person who writes webapps in COBOL, take them. The language is not going to dictate the success of your venture.

As for numbers, there are generally more PHP developers than Ruby developers, but you want a good developer, not just a developer.

<comment><AUTHOR> According to what I have heardÖ finding a good PHP developer is much more easier than finding a ROR developer coz ruby is new and less developers available and they demand higher salary.

<comment><USER2> So my last line up there had perhaps some unclear meaning. There are more PHP developers out there, but there are good developers to be had in both arenas. As for salaries, you get what you pay for.</comment></comment></comment></comment>


Feeling low? It's okay. Join us in our gaming happy places - <AUTHOR> https://www.rockpapershotgun.com/2019/12/16/feeling-low-its-okay-join-us-in-our-gaming-happy-places/ <comment><USER2> I thought this was going to be an event where people without families or something join the same online space over the holidays. Has that been done?

<comment><USER3> I don't remember being done explicitely for estranged people, but World of Warcraft has a Christmas event every year, with decorations, special items and thematic quests that you would do alongside other players. You probably can find similar things in other popular MMOs. In a similar way, I'm sure online virtual worlds like Second Life or VRChat have players organizing themed parties and such. Then you also have streamers who do a special stream on Christmas Eve, so in a way their communities are also together during the Holidays.</comment></comment>


The Rise of Invisible Work - <AUTHOR> http://www.theatlanticcities.com/jobs-and-economy/2013/10/rise-invisible-work/7412/ <comment><USER2> > When people share and rent things instead of buying them, does that mean we > need to produce less stuff, requiring fewer jobs, ultimately creating less > economic growth?

Hey look! It's our old friend, the Broken Window Fallacy [1]. It has been so long since I've seen it I've almost missed it.

Waste is never good for an economy. If it were, we could have an amazing economy by setting up factories and recycling plants right next to one another and skipping the output. That's massive activity, but zero productivity, and it's productivity that we want.

[1] [http://www.investopedia.com/ask/answers/08/broken-window- fal...](http://www.investopedia.com/ask/answers/08/broken-window-fallacy.asp)

<comment><USER3> Well if that was the case its interesting that products are built to expire. Cause its certainly good for companies.

Waste is just a resource we have not found a way to utililze, we will.

<comment><USER4> > products are build to expire

They aren't - the replacements are higher productivity.

See the PC market for what happens when the replacements are not higher productivity.

<comment><USER3> Eh yes they are. Not all but quite a lot of them.

</comment></comment></comment></comment><comment><USER5> >* the widening gap between productivity and employment in America... is deeply troubling to economists. But it's possible... that the gap could contain all these people we are currently not counting.*

Growing productivity could also just lead to decreasing employment if the same roles are being accomplished by fewer people who do them more efficiently. Having a bit of work on the side essentially amounts to people who are employed doing more, rather than unemployed people having an Etsy or ebay shop which is their main employment. And many of these people probably don't count as "unemployed," as there are advantages to being set up as a full time seller (at least on ebay and amazon) for tax purposes.

</comment><comment><USER6> >So far, the sharing economy's impact has been largely unseen because we (and the Bureau of Labor Statistics) are used to counting employment in whole jobs, or part-time jobs, not something-I-do-on-the-side-while-I-freelance jobs.

I hope it stays that way. I know it's not the best case scenario for the metrics, but I already report enough to the government as is, please don't frame reporting more info as a potentially good thing.

</comment><comment><USER7> Ideally we would measure success in terms of happiness / GDP, not GDP alone.

That said, services like airbnb have externalities in cost which are born by all citizens. Transients staying next door? No thanks. Increased rents? Ouch!

</comment><comment><USER8> This is worthy of discussion because it brings up a point that's been true for a long time: the economy the way it actually exists and the economy the way it exists in many people's minds are two completely different things. So when politicians talk about doing things to get "jobs" \-- I have no idea what that term even means. Most politicians think it all boils down to a bunch of cronyism that ends up with a defense or environmental contractor located in their district. Maybe some kind of ribbon-cutting ceremony that can hit the evening TV News.

The article has some pretty big flaws, however, the biggest of which is that it doesn't even truly understand the degree of dissonance in terms involved. Instead, we get some marketing blurb thrown out, the "sharing economy" and then the entire problem is viewed in terms of that.

Bad analysis always leads to bad advice, then bad solutions. This isn't any kind of new concept that has to do with AirBnB or Uber. _This is the way the economy actually works_ From gin joints to smuggling to gun running to "working under the table" \-- the real economy is messy and fuzzy. It's full of people sharing and trading all kinds of things under the radar. Technology is just multiplying our ability to hook up and help each other out. That's a great thing. I just wish the writer could see the forest instead of just a couple of trees.

But heck, can't have everything.

</comment><comment><USER9> >When people share and rent things instead of buying them, does that mean we need to produce less stuff, requiring fewer jobs, ultimately creating less economic growth?

Well if this was true you could just force people to buy things they don't need then throw them away and create economic growth easily.</comment>


Xcode 5 Finally Makes Interface Builder a Viable Option for Teams - <AUTHOR> http://nilsou.com/blog/2013/08/07/xcode-5-finally-makes-interface-builder-a-viable-option-for-teams/

<comment><USER2> I still don't know why storyboards (all views and segues) are implemented in a single file. I talked to some Apple developers at WWDC last year and this year about the issue. I explained the possibility of actually saving controller/view bindings in separate files (umm, xibs) so you wouldn't get all these conflicts. The storyboard itself would only be a graph of the xibs and their segues to other xibs and whatnot (position on screen in the editor or whatever). A lot less of a chance of a conflict. I'm just giving a high level overview of what we had discussed. I still think it could be implemented better. Break that bad boy up! It should be in a bundle.

</comment><comment><USER3> This is a welcome change. I'm interested to see if storyboards are now team- friendly too.

<comment><USER4> They always have been. The new XIB format is the same as the current Storyboard format.

It doesn't completely prevent merge conflicts though.</comment></comment>


How to improve MessagePack JavaScript decoder speed by 2.6 times - <AUTHOR> https://appspector.com/blog/how-to-improve-messagepack-javascript-parsing-speed-by-2-6-times <comment><AUTHOR> I'd like to share a few interesting performance optimizations that allowed us to significantly increase MessagePack parsing speed at AppSpector. With current msgpack-javascript version we can decode our payloads event faster than native JSON.parse!</comment>


Swift 5 – Swift Programming Language Evolution - <AUTHOR> https://github.com/apple/swift-evolution/blob/master/README.md <comment><USER2> If they're going to break source compatibility again, here's one vote for cleaning up the `if let` and `guard let` syntax. IMO it's rather unreadable and unwieldy.</comment>


Aeron chair history: Herman Miller's office staple and its origins - <AUTHOR> http://www.slate.com/articles/life/design/2012/11/aeron_chair_history_herman_miller_s_office_staple_was_originally_designed.single.html <comment><USER2> I tried the Aeron, it never really fit, I ended up with a Recaro EX and I've never looked back. I've had the same Ex for almost a decade, used it almost every day, and its weathered at least 4-5x better than any similarly aged Aeron that I've seen. It costs a lil more, but you can re-cover it, re-stuff it so its a chair that could last you a lifetime; perfect for those resistant to change :)</comment>


Swift is like Go - <AUTHOR> http://repo.tiye.me/swift-is-like-go/

<comment><AUTHOR> I love Go and I like Swift. This site shows that they are not that different for simple coding. IMHO many Apple/NextStep APIs could use some more simplicity. Right now all these Next Step APIs are pretty verbose and I'm often wondering why simple things need so much API knowledge.

<comment><USER2> My experience is that the learning curve is steep but the payoff is large. The Apple Cocoa/AppKit/CF libs are hefty things to get started with, but man do they make some hard things easy once you get the hang of it. They also tend to push apps to behave more consistently, because people aren't always reinventing the wheel for more complex UX.

</comment></comment><comment><USER3> Except that one is currently open and the other isn't. That carries a lot of weight for some of us.

</comment><comment><USER4> Looks like this hasn't been updated for the changes in the Swift language (since June 2014). That doesn't take away from the content, just be aware that tweaks will be necessary before the Swift will run in more recent versions of Swift.

<comment><USER5> Auther here. I'm mainly working on CoffeeScript and CirruScript. I haven't read Swift code for long.

</comment></comment><comment><USER6> The first one can be shortened for Go:

      package main

      func main() {
        println("Hello, world!")
      }

And it looks even more similar to the Swift version.

<comment><USER7> for println() "These functions are documented for completeness but are not guaranteed to stay in the language."

[http://golang.org/ref/spec#Bootstrapping](http://golang.org/ref/spec#Bootstrapping)</comment></comment>


Pessimal Algorithms and Simplexity Analysis [pdf] - <AUTHOR> http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=0A705B613AD67D4097DE09F4B2155452?doi=10.1.1.37.4489&rep=rep1&type=pdf <comment><AUTHOR> Hilarious parody of a computer science paper. My favourite part is the citation of the Odyssey as "Homer's longest path algorithm".</comment>


NCAA deadline looms over fractious HB2 repeal discussions in North Carolina - <AUTHOR> https://www.washingtonpost.com/news/early-lead/wp/2017/03/28/ncaa-deadline-looms-over-fractious-hb2-repeal-discussions-in-north-carolina/ <comment><AUTHOR> This is a good time to mention that while the bathroom part of HB2 and RFRA are both flawed, I do dislike employment anti-discrimination laws. I suggest a compromise of limiting it to things like manual labor if needed.</comment>


Putty for Mac 9.1.1 has been released - <AUTHOR> https://winonmacs.com/2019/07/01/putty-for-mac-9-1-1-has-been-released/ <comment><USER2> Is there something PuTTY does better than the Mac terminal and its native command line tools?</comment>


Show HN: Persistent async/await state – use JavaScript as a lean workflow tool - <AUTHOR> https://github.com/awto/effectfuljs/tree/master/packages/es-persist <comment><AUTHOR> It is a transpiler and runtime library, it offers writing workflow process descriptions in plain JavaScript, using all modern JS features and any third party library.

When the process is suspended (in `await` or `yield`), its whole state can be saved to DB. It is resumed after some external action, e.g. e-mail confirmation, or some date, or some ticket in issue tracking system changes state etc.

The library doesn't provide DB facilities itself, it gives access to computation state. And it is fairly trivial to save it in any DB. The same for actions implementations.

For example:

``` if (tooLateForToday) await tomorrow ```

The whole EffectfulJS tool is not only for workflow definitions though, it is for embedding abstract effects into JS, there are many more concrete tools can be done on top on it by just implementing its small abstract API.</comment>


Please explain: why Mono? - <AUTHOR>

<comment><AUTHOR> I have been seeing more and more articles and blog posts about Mono lately but I fail to realize the importance of it. I wish I was rude enough to just email to Miguel and ask him why has he dedicated so much time of his life working on it, but I better ask people who's not related to Mono - you.

Seriously, what is their motivation? I mean Mono people's. What they are doing, how I see it, is porting someone else's platform (Microsoft) which by itself is in essense just a copy of another platform (Java)... which _already_ runs fine on Linux! Basicaly it lookts to me that Mono guys are just building a _poor performing copy of copy of something original_ (Java platform) which already is built and works just fine. And have been, for a long time alreay. So long, that people are already getting bored with it.

Why??? Any ideas? \-- tx

P.S. I've been working with .NET for a while. Yes, it's rich and seem to have every imaginable feature, so do Python/Perl/Ruby/Java with their libs (all are 1st class Linux citizens). Why would one want to port .NET to Linux is beyond me...

<comment><USER2> I was curious about C#, started reading about it and I stopped precisely at the point when I saw this piece of code in their tutorial:

if (System.Text.RegularExpressions.Regex.IsMatch(s, sPattern, System.Text.RegularExpressions.RegexOptions.IgnoreCase)) ...

So Miguel de Icaza wants to bring this mess to UNIX? This is ridiculous and so un-UNIX.

(Though GNOME, too, was probably one of the messiest GNU projects so far)

<comment><USER3> Yick, your reaction is understandable. But know that the following is far more typical:

using System.Text.RegularExpressions;

...

if (Regex.IsMatch(s, sPattern, RegexOptions.IgnoreCase))...

<comment><USER2> Oh yes, I know ;) but I wouldn't say it is much less verbose than the first example in terms of the number of words used, and besides, it's Microsoft's tutorial, i.e. it's how they teach novice programmers to program in C#.

</comment></comment></comment></comment><comment><USER4> In my opinion, Mono can make for linux what VB made for Windows: offer a simple-to-learn, simpler-to-use development environment, specially for the corporate world. Many managers prefer to use a "manager-safe choice" like windows/.net to risk on opensource platforms. I think VB was key to the windows commercial success and wide adoption on the mid-90's. Linux didn't had something like that in that time. Many desktop and web apps are being developed on .NET, and the possibility of running them on linux may give it a place it didn't had before. .NET might not be a good choice for a internet startup but it's certainly for corporate and desktop apps. And they can ride over MS marketing pushing the adoption of .NET.

Besides, .NET is a pretty good platform and Miguel and the community are doing an amazing job =)</comment>


Show HN: Doge Seed – Dank mnemonic seed phrases - <AUTHOR> https://dogeseed.com <comment><USER2> Every string seems to start with "very", "much" or "such", and the page does explain this reduces the entropy. I wonder if it could aid a known plaintext attack[0]?

[0] [https://en.wikipedia.org/wiki/Known- plaintext_attack](https://en.wikipedia.org/wiki/Known-plaintext_attack)

<comment><AUTHOR> It can't, as per the BIP39 spec[0], the mnemonic is run through PBKDF2 to generate the seed that is actually used.

So even knowing one or two of the input words doesn't tell you anything about the actual seed.

[0] [https://github.com/bitcoin/bips/blob/master/bip-0039.mediawi...](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki#from- mnemonic-to-seed)</comment></comment>


Complex function plotter with support for split-complex and dual numbers in JS - <AUTHOR> https://wbolden.github.io/complex/#z <comment><USER2> Related:

\- "Visualizing complex-valued functions with Matplotlib and Mayavi" [http://nbviewer.jupyter.org/github/empet/Math/blob/master/Do...](http://nbviewer.jupyter.org/github/empet/Math/blob/master/DomainColoring.ipynb)

\- "Phase Plots of Complex Functions: a Journey in Illustration" [https://arxiv.org/abs/1007.2295](https://arxiv.org/abs/1007.2295)

</comment><comment><AUTHOR> here's the source [https://github.com/wbolden/complex](https://github.com/wbolden/complex)

</comment><comment><USER3> I've wanted something like this for longer than I'll admit. So glad!</comment>


Teenagers allowed to wake up later have fewer absences from school - <AUTHOR> http://news.bbc.co.uk/2/hi/uk_news/education/8579951.stm

<comment><USER2> There's a thing called Delayed Sleep Phase Sndrome. It's a circadian rhythm disorder that affects maybe 1/2 of 1% of people (but seems to be more prevalent among engineers and software people, possibly because DSPS sufferers choose vocations that allow for flexible schedules). Essentially, DSPS people have a natural sleep-wake cycle that's shifted a few hours from the norm. My natural cycle is to sleep at about 2:30am and wake at 11:30. I can force myself into a "normal" schedule, but I can't change my natural circadian rhythm, so I suffer from various effects of sleep disruption that are similar to jetlag. For some reason, a high proportion of teenagers, particularly boys, experience temporary DSPS during their adolescence. It can be as many as 20%. I suspect that the natural creeping of the circadian rhythm that happens during adolescence (as mentioned in this article) kicks into overdrive for some kids, and gives them DSPS-like symptoms during that time. Most of these kids grow out of it and re-establish a normal rhythm into adulthood. People like me, however, keep their DSPS throughout their lives. (And it's hereditary. I got it from my mother) <http://en.wikipedia.org/wiki/Delayed_sleep_phase_syndrome>

<comment><USER3> I'm a little different - I'm fairly certain I have a non-24-hour circadian clock. If I do not force myself, via scheduled melatonin supplements and caffeine/B-vitamin supplements, I naturally land on about a 26 hour circadian clock.

It gets really annoying really quickly.

Tangentially. I wonder at the reason for the increase of DSPS during adolescence among boys. It might very well be evolutionary - the young had to stay up later for watching over camps/habitat at night, perhaps?

<comment><USER4> Adolescent common chimpanzees form bands of males that stay up later than the rest and patrol the group's perimeter. I'm guessing human teenagers are genetically programmed to be the night watch.

(Reposted from below :)

</comment></comment></comment><comment><USER5> Alright, so the research has been conducted, verified, and introduced changes in the lives of teenagers. Now let's move on to adults. Where's my 10am start time at work???

<comment><USER6> Adults aren't teenagers.

As an adult, your [highest daytime alertness is at 10AM]( <http://en.wikipedia.org/wiki/File:Biological_clock_human.PNG>). You don't go from drowsy to alert instantly, so 9AM to 10AM is one of the most alert parts of your day.

<comment><USER7> Adults aren't all identical.

My biological clock is about 3 hours behind, in relation to the light-dark cycle. I become tremendously productive just before noon.

</comment></comment></comment><comment><USER8> In high school, I would fall asleep in class all the time. Teachers would yell, I'd get in trouble, it was awful. It wasn't intentional, I was just so tired that I could not keep myself awake. If I could have stopped it, I would have been much happier, but I couldn't.

Even during my waking periods, I'm sure I wasn't operating at peak mental efficiency.

<comment><USER9> Anecdotal evidence: I can't ever remember being as exhausted as I was having to get up every day for high school. I think my body has adjusted better to less sleep now, but I just remember it being awful.

I felt even worse for the kids who came on the bus. No wonder many of them couldn't stay awake! They probably had to get up at 5:30 every morning just to go to school!

<comment><USER3> Anecdotally, I had an hour and a half bus ride to school back in the day. We were picked up at 5:45 (school started at 7:30). For much of the school year, it was pitch dark out waiting for the bus.

If I wanted to be presentable and eat breakfast, I had to wake up at 4:45.

<comment><USER10> I was similar. Towards the end of high school I moved far away. School started at 8:30, my alarm was set to 6. Bus left just before 7 down the road a little.

It was interesting taking a bus + train + bus to school, but I wasn't really a fan of waking up that early.

</comment></comment></comment></comment><comment><USER11> 10am? That means they were going at 9am? That seems really late compared to when I went to school and it started at 7:30am! I can see where later would cut down on absentees, but it seems to be at a cost of pushing after school activities later and taking away from family time.

I think the four-day school week is a better idea. I think it would help in terms of attendance and would cut costs: [http://news.google.com/news?hl=en&q=four%20day%20school%...](http://news.google.com/news?hl=en&q=four%20day%20school%20week&aql=&gs_rfai=&um=1&ie=UTF-8&sa=N&tab=wn)

<comment><USER12> Another factor is that in the US, there's often a bottleneck in the bus system. It's more efficient to use the same buses for multiple levels of school, but a "bus cycle" may take an hour, and you may have three schools to service separately (high school, middle school, elementary). As a a result, the elementary school may start at 9 AM, but that pushes the high school to 7 AM. And you don't really want to reverse it.

<comment><USER13> > As a a result, the elementary school may start at 9 AM, but that pushes the > high school to 7 AM. And you don't really want to reverse it.

Why not? I'm one of 4 children; every single one of us in elementary school would get up on our own ridiculously early. We would wake up and eagerly watch _Pokemon_ at, IIRC, 6:30 AM. Waking up in the 5s wasn't unheard of. It wasn't until high school and getting up at 6 AM that an older kid would finally start getting up before the younger kids.

<comment><USER14> The problem is having the younger children outside waiting for the bus in the dark before or soon after the sun comes up in late autumn and winter.

<comment><USER13> That doesn't sound so bad. It's going to be cold even if one waits an hour or two.

</comment></comment></comment></comment></comment><comment><USER15> I hate to be that guy, but the headline should say _fewer_ absences.

<comment><USER16> Then don't. Def #6

<http://dictionary.reference.com/browse/less>

<comment><USER17> Cool! But the OED disagrees: <http://www.askoxford.com/concise_oed/few>

The wikipedia article says that using fewer for counting nouns is fairly new: <http://en.wikipedia.org/wiki/Fewer_vs._less>

So I guess you're probably right, and it's not worth correcting people on this matter.

(But I'm probably still going to use fewer instead of less).

<comment><USER18> The distinction can be important. For example, are you one of the less intelligent members of HN, or one of the fewer intelligent members of HN?

<comment><USER19> Cute example but the fewer vs less argument is for when those words affect the same word.

In "less intelligent members," less is an adverb affecting the verb of the sentence, whereas in "fewer intelligent members," fewer is affecting the number of "members." The word switch is changing the entire meaning of the sentence, but the fewer vs less argument is over using the right word for a single, specific meaning.

</comment></comment></comment></comment></comment><comment><USER20> School start time is changing to 30 minutes later than the current time at a school that my fiance teaches at next year. The decision has come with some mixed opinions, but there is certainly some evidence to support this theory. Although, a lot of the kids are pretty pissed, because it means they have to stay at school for 30 mins later into their valuable afternoons.

My opinion though is that once people get use to the new hours (perhaps after 6 months, 1 year, or longer) that it will become the new "normal" and lose its benefits. But my opinion is garbage, because I wake up and go to sleep at terribly insane hours :)

</comment><comment><USER21> This is usually how I did it in high school. I would stay up late either working on actual school work, messing around with video production for tv station, or some web stuff. Instead of waking up tired as hell, I would just go in later. It's better to miss a few hours than take the entire day off OR have that tired feeling ruin an entire day with a domino effect impacting the rest of your productivity. I'm glad people are finally doing something about this.

</comment><comment><USER22> In my high school we had "late" days once or twice a month, where the school day started about an hour later than usual. Also, seniors could schedule their classes so that they could either be done for the day an hour early or show up an hour late. Years later, I still remember the euphoria of waking up at 7 only to realize I had an extra hour to sleep.

</comment><comment><USER23> This would have definitely helped me out in school. One bathroom, 4 people, standing at a bus stop at 7am outside in freezing temperatures, lack of coffee (not allowed in my school even in HS), lack of transportation for after school activities before 5PM, staying up late to use the phone line on my computer after everyone else went to sleep, staying up until ungodly hours to watch MST3k on Comedy Central, etc. Gotta imagine kids today still face some of these reasons for needing a later start. These are the type of people who in the _real world_ would gravitate towards jobs that allowed them the same respect & flexibility.

</comment><comment><USER24> I can personally vouch for this myself in practice.

This is a classic case of quality vs quantity. Depending on my workload, I have sessions of varying time of hyper-activity. If something is highly urgent, then these sessions can endure for as long as I need. The only thing I require is that I'm not that tired.

Luckily, I work in an environment that only requires results. Time is of course tracked, but the ultimate reflection of my value is in my creative thinking and the actual work that gets done.

</comment><comment><USER25> I've been wondering for a while now: why are so many people at HN and similar sites generally the late-riser type? It seems disproportionate.

<comment><USER19> A lot of us have the freedom to be. A disproportionate number of us are owners/founders or otherwise self employed. I go to bed about 8 am :-)

<comment><USER26> I think there is a tenancy for the hacker type to be a night owl, maybe it has something to do with the ability to be able to do work at any hour and the relative peace you get at night.

</comment></comment></comment><comment><USER27> How are parents supposed to get their children prepared or take them to school in office hours?

<comment><USER28> How are parents supposed to pick their children up during office hours?

Combined with the option of riding a bus, I don't think it makes parent's lives any more difficult. Without a bus, I can see it being a big problem for many families, though.

<comment><USER29> Here's the trouble.. School start times for Elementary, Middle and High School need to be staggered. Otherwise they'd need a lot more buses.

But if you put grade school kids at the earlier start time, you'll have little kids that take the bus standing outside in the dark. That is not good.

In some communities with a bus mass transit service you can start HS at the same time as grade school by putting the HS kids on the transit buses. That works--the buses aren't as regulated, but it works. But school districts where this is possible are probably just a fraction.

<comment><USER30> In San Francisco Unified there isn't a high school bus system, and the classes still start at 7:30 AM. It has more to do with hidebound teachers and admins.

<comment><USER29> Of course, not having a high school bus system doesn't defacto mean you'll start class late.

But bussing high school kids means you can't, unless you want to...(see my OP)

</comment></comment></comment></comment></comment><comment><USER4> Adolescent common chimpanzees form bands of males that stay up later than the rest and patrol the group's perimeter. I'm guessing human teenagers are genetically programmed to be the night watch.

</comment><comment><USER31> In a couple Wall Street companies I've experienced, the penalties were harsher for being a few minutes late than just calling in sick, so there were a lot of unnecessary sick days that happened because a person would rather "throw a sickie" than get reamed for showing up at 9:08. It's really stupid and counterproductive.

<comment><USER32> At a previous employer, we were supposed to be in at 8:30am. Our boss even spoke to myself and a friend if we regularly arrived at 8:45, even if we didn't have any specific appointments. (And we stayed _plenty_ late at the end of the day.)

I wouldn't have minded, except that he would regularly be 30-60 minutes late for meetings with clients!

Yes, rules like this can be extremely stupid and counterproductive.

<comment><USER33> Hah. I remember having a job where my boss would complain about how late I was every day. He said I should come in earlier. I didn't. He asked me if I had trouble understanding his instructions. I told him I understood perfectly, but I just wasn't awake at that hour. He then asked, "so you're intentionally disobeying my instructions." I smiled and said yes.

The issue was not raised again.

(Compare this to my current job, where I got weird looks when I asked if I should be in at 9 on my first day. Little did I realize that this was a place where people came in at 7:00 _pm_ and nobody batted an eye :)

</comment></comment></comment><comment><USER34> 1 word: duh.

</comment><comment><USER35> The more we do this stuff to kids, the more the real world is going to kick their ass.

<comment><USER36> You didn't read the article, or the background research behind it, did you?

It's based on research into the natural sleep cycle of adolescent humans, and their cognitive abilities. Turns out that there are changes associated with puberty (who knew?) that make them wake up several hours later and stay awake longer than children; their cognitive abilities also peak later in the day. From age 18 onwards they revert towards the adult baseline.

Assuming you're an adult, do you think you'd work more efficiently if you were rousted out of bed at 5am to start work each morning, and forced to finish at 1pm?

<comment><USER35> No, I absolutely know that I would not work better that way, but my point is, everyone around me is forced to wake up at the crack of dawn and get their ass to work.

I have kids in school, and every year the school changes things around to accommodate some consultant's suggestions regarding grades, start times, recesses, how they learn to read, how they learn math, etc.

Every damn year!! Kids are getting screwed left and right because of studies like this. Last year, my oldest was using one method to learn to read, this year it's something new. And a new way of learning math too.

They had an assembly for us parents explaining to us that they were preparing kids for "jobs that don't exist yet". And that they didn't know if these methods were perfect, but they were based on studies, so.... Kids just end up confused.

This hits a nerve with me. I hate seeing this stuff happen because of studies. I hate when people like you justify stuff like this when you aren't living in it.

Also, how do parents get their kids to school at 10am? How? School needs to start when work starts so we can drop them off on our way.

This stuff screws up kids and will make adjusting to the real world much harder.

<comment><USER37> Are there no buses where you are? During my entire time in public school, I got there in one of three ways: took the bus, I got a ride with my older brother, or I drove myself.

Also keep in mind this would apply to high school students, not elementary school kids. And this isn't a single study, or even a new result. I've known this about teenagers for a while.

<comment><USER35> No. In fact, buses don't go far enough out in some places here, and they don't stop close enough. Plus, I believe they can't afford to run all their buses anymore.

I didn't have my own car until I was a senior in high school. I got dropped off every day (12 years ago now.)

Where we are people live far enough from the school that it's too far too walk, but it's just as far to walk to catch a bus. Getting kids home from school is equally as difficult but most kids are forced to take the long treks by foot or bike. And in the winter, we sometimes get 3+ feet of snow at a time.

It's a big world, I know I shouldn't have started this argument here. I'm frustrated about the education my own kids are getting, and I hate seeing so many people agree with studies that cause sweeping changes in schools.

I'll read the article again. I'm pretty sure I read through it the first time with my mind frustrated with this headline anyway.</comment></comment></comment></comment></comment>


F5Bot: Email Keyword Alerts from Reddit, Hacker News, or Lobsters - <AUTHOR> https://f5bot.com/ <comment><USER2> I wish services like this would move beyond offering everything for free. It's a valuable enough service to charge a nominal fee, be it lifetime or whatever to ensure it sticks around, and give the developer incentive to improve upon it.

I'm guessing the next big web movement will be more paid services that actually have value. Weeding out all the trash that disappears with your data in a year.

Update: Looking at his Patreon. He's making a total of $41. Not even enough to cover his hosting fees apparently. That's why you charge for something worth charging for. [https://www.patreon.com/f5bot](https://www.patreon.com/f5bot)

<comment><USER3> That's not how new services or startups work nowadays. You need critical mass first, and if you put up barriers - like paying money, like ads - people won't use your service. And there'll be a free alternative that takes over.

No, the formula is to invest your own money, hopefully see a lot of growth, get a few rounds of dank investor money to pay for it all and stay afloat, and maybe, MAYBE, come up with a monetization scheme (like idk, reddit gold or premium subscriptions), and then either get bought by one of the big companies or go to the stock market.

<comment><USER2> That's how yesterday worked. Impeding recession is going to dry up that funny money. You're going to have to actually build products people want to pay for to survive. Two individuals being equal, there is no way the one working for completely free is going to outpace the one getting paid to do it. That free service is attractive at first, until you're missing features, support, and longevity that the paid service has.

Example. pinboard.in

<comment><USER4> Pinboard employs a whole one person. It's a good model, but its aims are pretty different from those of your average startup.

<comment><USER2> There's two, and the average startup could learn a lot from how Pinboard is run.

</comment></comment></comment></comment></comment><comment><USER5> The author of F5Bot recently published a guest-post on the Intoli Blog called _How F5Bot Slurps All of Reddit_ [1]. It goes into a lot of technical detail about how the service works and some of the challenges that arise when dealing with such a large number of posts and comments. If you're a fan of F5Bot, or just learning about it now, then you might be interested to check that out.

As others have mentioned, F5Bot is free and completely driven by donations and Lewis Van Winkle's generosity. Please consider giving a donation if you're a user of the service and find it helpful [2].

[1] - [https://intoli.com/blog/f5bot/](https://intoli.com/blog/f5bot/)

[2] - [https://www.patreon.com/f5bot](https://www.patreon.com/f5bot)

<comment><USER6> Thank you for the intoli link. It was a good read, plus I learned about Aho–Corasick algorithm

</comment></comment><comment><USER7> Ah, so _that 's_ how extremists seemingly always find Reddit threads related to certain hot topics and brigade them.

I used to think that it was a simple case of fringe folks being attracted to popular threads. But it happens frequently enough on smaller threads on otherwise moderate, modest communities that I thought something was up.

<comment><USER8> There is also this: [http://wiki.project- pm.org/wiki/Persona_Management](http://wiki.project- pm.org/wiki/Persona_Management)

And this: [https://www.nytimes.com/2018/10/08/us/politics/rick-gates- ps...](https://www.nytimes.com/2018/10/08/us/politics/rick-gates-psy-group- trump.html)

I left reddit because with the API these disinformation campaigns are too easy to create. After watching from afar during the 2016 elections noticing weird new things happening, new patterns, I decided that I no longer knew what was real and left.

</comment></comment><comment><USER9> I've been using this for a few months and it's been really useful. I agree that offering a free version and a "premium" version with a few extra bells and whistles would be a really good idea. It's definitely the kind of thing that I'd pay to have extra features for.

<comment><USER10> Hi, F5Bot author here. I'm working on a premium version, and a couple features are already available through Patreon. It has just been a low priority. If you have ideas for any premium features you would like to see, please feel free to drop me an email sometime (just hit reply on any alert you get).

</comment></comment><comment><USER11> What's the difference to a Google Alert with: "site:reddit.com KEYWORD"?

<comment><USER12> Do Google Alerts even work reliably these days? I set up a bunch some years ago, realized they don't seem to be working, and promptly forgot about this feature. Every month or free some alert I set up then triggers and send me an e-mail, but I kind of expected to get such e-mails _every day_.

<comment><USER13> They are hit and miss. Nearly 100% of the results are from news sites and it seems to lag. Like, I'll get forwards from associates about Topic on Monday then the Google alert on Wednesday

</comment></comment></comment><comment><USER14> Is it worth seeking out an invite to Lobsters am I better off lurking?

<comment><USER15> Lobsters is very low traffic compared to HN. The posts tend to have ~50% cross-over rate with HN as well.

anyhoo, I can hook you up if you want. DM/Mail/Smoke-Signals me.

</comment></comment><comment><USER16> I built a similar service, but for podcasts: [https://www.listennotes.com/alerts/](https://www.listennotes.com/alerts/)

</comment><comment><USER17> I use F5 and it's been genuinely useful seeing where my own blog posts come up! Thanks to the creator for making it

</comment><comment><USER18> Does the author have a timeline for when they'll open source it?

[https://github.com/codeplea/f5bot](https://github.com/codeplea/f5bot)

<comment><USER19> The about page says:

      If I reach $500/month in donations, I will open-source F5Bot.

<comment><USER18> Ah, that's fair. I didn't see that.

</comment></comment></comment><comment><USER20> I always forget about lobste.rs I remember wanting a membership when it launched and getting mad that I couldn't find anyone to send me an invite. Kinda made me feel like it was the no homers club.

</comment><comment><USER21> This is the first time I've heard of it and this is awesome and amazing, and I'm so glad it is free. Thank you for making this available as a great tool!</comment>


Accordion Pagination: A Faster, Smoother Way to Paginate - <AUTHOR> http://uxmovement.com/design-articles/accordion-pagination-a-faster-smoother-way-to-paginate

<comment><USER2> This must be _really_ important if you felt the need to provide a link to your same exact blog post twice in two days (by changing the title of your post so it wouldn't be blocked/detected)!

Actually, now that I look at your submission list ( <http://news.ycombinator.com/submitted?id=UXMovement> ), it seems that you _double-submit your own blog posts by changing the titles_ over and over and over and over (and over) again. I think PG should start collecting ad revenue from you soon!

</comment><comment><USER3> is there really any benefit to users by paginating long articles? it seems like just a ploy to generate more ad views.</comment>


Final words from Arthur C. Clarke - <AUTHOR> http://www.iht.com/articles/2008/03/20/opinion/eddas.php?page=1

<comment><USER2> Ah yes, the old "attribute everything to the guy who just died" routine. A classic. I'm sure nobody else would have thought of geostationary satellites. Oh wait, that's right, someone else did, and Clarke just put them in a book 17 years later.

<http://en.wikipedia.org/wiki/Geostationary_satellite#History>

<comment><USER3> Like a lot of things, somebody else thought of it, but Clarke put popularized it. I'd argue Clarke's job was more important.

Same goes for a lot of songs. Great song writers write a lot of stuff, but until some half-rate performer popularizes it, it's just notes on a page.

For that matter, same goes for good ideas. Lots of folks have them. Takes something special to take an idea and put it into millions of minds.

Clarke was the man. He didn't do everything, but eulogies are made to evoke fond memories, not get translated verbatim into a history book.

<comment><USER2> Eh, I don't know. You have the guy who thought it up (which seems like the easy job, as someone else surely would have thought of it soon after), the guy who wrote about it in a book, and the guy(s) who made it happen.

Only the third truly impresses me.</comment></comment></comment>


Python 3.8.1 Released - <AUTHOR> https://www.python.org/downloads/release/python-381/ <comment><AUTHOR> Here is the full changelog [https://docs.python.org/3.8/whatsnew/changelog.html#python-3...](https://docs.python.org/3.8/whatsnew/changelog.html#python-3-8-1)

It is a signal for many people to move to 3.8

<comment><USER2> What was missing from the original 3.8 release?

<comment><USER3> Nothing. Many people avoid software versions ending in '.0' because they worry they may be buggy.</comment></comment></comment>


The 1950s called and wanted their toolbox back - <AUTHOR> http://rsms.me/2013/01/01/1950s-called-wanted-toolbox-back.html

<comment><USER2> Why are we still using english language? It's such an old tool that I think we should transcend from it to a new level.

It took quite long for personal computers to catch up with super-computers of 50s and 60s. Many tools and concepts might have been conceived 60 years ago but they have become easily accessible much more recently. In late 80s and early 90s most people still had to pay a lot of money for tools like C-compiler and UNIX-like systems. Internet became easily accessible only in 90s.

There are some old techs that are just now becoming easily available like transactional memory. We used to need specialized hardware for that, but now many languages have libraries for software trasactional memory. Some are even implementing it as a language feature. This development is likely due to rise of consumer multi-core devices.

</comment><comment><USER3> The spaceship example the author uses is a good example of why I think he is wrong. My toolbox at home is full of tools passed down from my late grandfather's toolbox. I'm pretty sure if I was building a spaceship, a lot of the tools in it would be exactly what I need to use, as well as new takes on those tools like my pneumatic wrenches.

Sure I'd use crazy new tools in addition, but its not like the fundamentals of a field have to be constantly reinvented; they're just honed, refined, expanded and specialized. The space-flight specific bits would have their own new ways of working on them specific to them, but at the end of the day I'd still just be bolting a bunch of shit together. If the new tech introduces some improvements to the fundamentals to support itself, those improvements will slowly get rolled back into the base systems.

Calling for a revolutionary breakthrough is silly, his analogy of cars is a good example. The automobile's history has just been a slow evolution of better ways to perform the same old tricks. Revolutions within software are going to be smaller because the field already exists. At a high level nothing has changed, while on smaller scale the last few years have completely redefined source control.

</comment><comment><USER4> LISP is a fundamental programming abstraction. To say languages are based on it is disingenious. Lets put the clock back further - programming is based on math, which is 10,000 years old.

</comment><comment><USER5> It's not right to call text-based programming one-dimensional. It would be if we wrote the code on a single line. It's actually 2D. We have indentation to communicate nesting and scope, taking advantage of the visual/spatial sense.

</comment><comment><USER6> quite a few errors in facts and concepts here.

modern programming programming languages are quite different from those of the 50's

personally, I think the industry missed a beat in the non standardization of CASE tools.</comment>


Ask HN: What's the business and finance model behind new retail stores? - <AUTHOR> Say I have an idea for a great new bricks and mortar retail store.
What does that cost?  It can't be cheap to do properly - to fit out a beautifully designed space - minimum $50K to $100K?
Say you had an idea for retail that was going to cost $250,000 to create the retail environment..... what are the options to finance that - are there angels who specialise in retail or do you just have to have a house and get a second mortgage?
Where do people get the money to do that?  There's so much retail out there but not alot of that side of business gets discussed. <comment><USER2> So, my wife has a (very small) retail clothing store, started about 15 years ago. Since then, she has compared notes with many other owners of retail stores. The number one feature that is shared by those who are still in business five years later, is not taking on a bunch of debt.

There is a cycle: spend, see results, learn from that. If you take on a bunch of debt, it's like you're doing all the spending at first, then getting the results, then doing the learning (for your "our startup went bust, here's what we learned" blog post). In a way, a business loan is sort of like a waterfall software project. Don't do that. Do agile.

Rent a small space, probably subleasing from somebody else, and see what works. Then do more of what works. It is harder at first, but a lot of people who take out big loans, don't realize how much they don't know yet, and spend the money on all the wrong things. What are the right things? It depends on your market, your product type, your customers, and therefore the best way to find it out is to do it on a small scale and watch carefully what works.</comment>


Obama After Dark: The Precious Hours Alone - <AUTHOR> http://www.nytimes.com/2016/07/03/us/politics/obama-after-dark-the-precious-hours-alone.html <comment><USER2> Makes me feel a bit better about my late nights. I'm also a night person, usually work after the rest of the family has gone to bed. Sometimes I feel guilty - I know my wife would like me to go to bed with her, I know I'd like it better if I was able to function better at 6:30 am.

I guess there are a bunch of people like me out there. Wish I could do it in the White House, though. Sweet study :)

</comment><comment><USER3> What transpires is that he works a lot. Overall I think he will remain as a good president. I only wish he had taken more courageous decisions on whistlebowers and drone policies for instance.

</comment><comment><USER4> The first image of him reading in his "private" study, accompanied by a photographer.

<comment><USER5> Well, this is purely a PR piece, what did you expect. ;)</comment></comment>


Smart People Use Firefox, Even Smarter People Use Chrome - <AUTHOR> http://www.kinlane.com/2010/07/smart-people-use-firefox-even-smarter-people-use-chrome/ <comment><USER2> And the smartest people refuse to click on linkbait headlines.

There is nothing in that "article" but unsubstantiated claims. No data, no results, nothing.</comment>


Ask HN: HN user ghosted 5 years, why? - <AUTHOR> Just randomly ran across a user that appears to have not realized their account was ghosted by mods 5 years ago; as in they have appear to have been commenting for 5 years not realizing that their account was ghosted.
As far as I am able to tell, they were out of line when they got ghosted, but wow, umm they may go the rest of their life using HN has a ghost.
Assuming they could fix there behavior, how does this make any sense?  How many users are there like this? <comment><USER2> Why are you posting this here, where there's a high chance we wouldn't see it, instead of sending it to firstname.lastname@example.com as the site guidelines ask? It seems to me that if you cared primarily about the user, as opposed to stirring up meta drama, you would do the thing that actually could help them.

</comment><comment><USER3> I thought people with "showdead" enabled could see those and if they upvote enough of their comments the comments become visible and the user becomes unghosted. Is that not the case?

<comment><AUTHOR> Yes, that's correct, if you have enough rep you're able to "unghost the dead" and "see the dead" at any rep simply by creating an account, clicking "show dead" — though always assumed a user would notice or they'd just stop posting. Though there's to my knowledge no easy way to only see the shadowban users aside from writing script that watches the new comments.

Pro tip, if normally people vote or comment on your comments, you're not getting either, access HN via a private session without cookies and not signed in, then click to where your comment would be, if it not visible, you have been ghosted; to be extra sure, then create a throwaway account, click profile, turn on "show dead", load the profile that might be shadowban, click comments, and if the comments say [DEAD] with no [FLAGGED] you have been shadowban; email mods if you have questions via the "contact" mailto link in the footer.

Now, looking at all the comments, some which intent appears to be helpful, makes me wonder if shadowlands for users are not as toxic as the behavior they seek to remove; that is they isolating a user, making they think no one cares to reply to them, etc.

</comment></comment><comment> tech-historian Is "ghosted" the same thing as shadowbanned?

<comment><AUTHOR> Yes, that's correct; meaning if you go to your profile, click "show dead" then click "comments" on the main navigation, then keep scrolling down, clicking more until you find a comment marked [DEAD] but not also [FLAGGED] — then click the users profile, then comments, then click through there comments; some will not be [DEAD] because a user had "show dead" on and unghosted the comment if they had enough rep.</comment></comment>


Mark Zuckerberg's Patent - <AUTHOR> http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PG01&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.html&r=1&f=G&l=50&s1=%2220080046976%22.PGNR.&OS=DN/20080046976&RS=DN/20080046976

<comment><USER2> Wow! A summary of privacy settings. Amazing! It's so original and clever. Boy, I wish other software could have a summary of privacy settings. We could even generalize this to summaries of other sorts of settings.

My head spins with the genius of it. I think I need a lie down.

<comment><USER3> I'm not a shilling for Zuckerberg but you are being too hard on him. First, the patent isn't for "summary of privacy settings", it is for a systems/methods to generate the said summary. Second, a patent isn't supposed to be original or clever. The criteria are 1) novel, which can be a non- original, i.e. a derivative work. 2) non-obvious (to one skilled in the arts) or in other words non-obvious to a patent examiner who doesn't know much except for some prior patents in the field.

<comment><USER4> Any software patent is 'non-obvious' to a non-programmer. That's the problem with them.

<comment><USER5> them = patents, or non-programmers?

<comment><USER6> Patents, of course.

<comment><USER7> Well the real problem is the patent examiners probably aren't programmers, thus won't have a good idea of what should be patentable.

<comment><USER8> For most CS students/programmers, I'd wager that being a patent examiner isn't very alluring. This is even more evident at career fairs where representatives from the USPTO have to get in your face to grab your attention. You just don't see too many students lining up to talk to them, rather than say, Google or Microsoft.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> Sounds like a good candidate for the EFF patent busting project: <http://w2.eff.org/patent/>

Anyone know of any prior art?

(I don't know if they pick patents to bust on their own or take requests)

EDIT: If the above poster osipov is correct and this patent is legit/novel, please disregard my comment.

<comment><USER10> It isn't novel. See my reply above.

</comment></comment><comment><AUTHOR> I request everyone to donate to EFF.

<comment><USER11> amen

</comment></comment><comment><USER12> This is a joke right? I mean, it sounds like the patent for a web form. Get info, use info, deliver it back to user. Replace privacy summary with almost anything else and you have a basic function of most websites.

</comment><comment><USER13> Lets all say it the way it should be: this is a joke!! A patent for a high school level process programming? Come on!

Sorry I had to post another comment after actually reading the whole stuff.

</comment><comment><USER13> If this patent weren't from the founder of Facebook it will almost sound funny. I have never seen anything like this.

<comment><USER14> IMHO, MZ as inventor of said patent makes this funny*scary^2.

</comment></comment><comment><USER15> Look at the patent app again. It's not just privacy settings; it's the blueprint for the mechanism that the Data Portability group has been talking about. That's why Zuckerberg doesn't want Facebook to really play their reindeer games. They want to build it themselves and have Facebook as the hub of the wheel with spokes off to everything else.

</comment><comment><USER16> After reading the discussion I have to point out the obvious: it's an application not a patent.

</comment><comment><USER17> I thought it was going to be about a system of being a dick to everyone you've ever done business with or who helped you early on, but this is almost as obvious.

<comment><USER13> Say it loud! more details please.

</comment></comment><comment><USER18> I wonder if they submitted this patent application in crayon. The abstract reads like it was written by a third grader.</comment>


Unsolved Problems in Blockchain Sharding - <AUTHOR> https://medium.com/nearprotocol/unsolved-problems-in-blockchain-sharding-2327d6517f43 <comment><USER2> I am (perhaps naively) wondering whether a blockchain can be sharded hierarchically instead.

Each shard works like a channel like in the Lightning Network and the channel can have sub-channels, this means the channels all have their own blockchain, i. e. they are blockchain shards. To verify a transaction from A to B it's like a tree walk from a tree node A to the other tree node B. First verify the channel A is in, then go to the parent channel till the common parent channel is reached then descend again to the partner B's channel, verifying on each step ascending and descending.

<comment><USER3> That is somewhat similar to what Vlad Zamfir's sharding is doing ([https://medium.com/nearprotocol/so-what-exactly-is-vlads- sha...](https://medium.com/nearprotocol/so-what-exactly-is-vlads-sharding-poc- doing-37e538177ed9))

For a given transaction you'd only need to verify log(n) chains, but the problems from the top level post remain. The question is how you verify the transaction in a given chain. If you verify the entire chain, than you are expected to end up verifying all the chains (since ultimately cross shard transactions will be coming from all shards), removing any benefit from sharding. If you somehow trust that the shards were doing their job properly, then you need to somehow deal with shards being corrupted.

<comment><USER2> I am probably naive again, but it seems for me that you don't need to verify other transactions, only the one between A and B. You could ignore the shards/channels not affected by the transaction.

Another optimisation would be that for example an exchange tries to pull all its local transactions into the same channel. Or people join the retail's channel before paying. The idea is to avoid ascending as much as possible. In other words, a variant of Lightning Network!

</comment></comment></comment><comment><USER3> Alex from Near is here, the author of the post. Would be happy to answer any questions.

</comment><comment><USER4> It currently takes several weeks to spin up a Stellar validator. Last time I checked, you needed several days to a week to do the same for Ethereum. This has probably not improved. Both are growing rapidly and it's becoming a problem for adoption.

Both have on disk data well over a TB. And it's early days. This stuff is getting progressively less feasible to run for normal users as more users start using this and start adding their own transactions. Long term, sharding is not going to be optional.

<comment><USER5> Blockchain just suffers from the classic dilemma of being ahead of its time.

Once we have infinite power from practical fusion we can mine coins for cheap.

And when quantum computers are common you can update the chain or do sharding in no time at all.

<comment><USER6> Mining a bitcoin block always takes 10 minutes and each block can only handle around 5000 transactions because that's what bitcoin was designed to do. It doesn't matter if you're powering all miners on earth with a dyson sphere or a single hamster wheel, it will still take 10 minutes.

</comment></comment></comment><comment><USER7> Thanks for expressing these problems clearly and succinctly.

Does the Fisherman approach involve validators randomly checking other validators for phony transactions?

Will validators be incentivized to do this?

Is that why they are called fisherman? Because honest validators are "fishing" around for opportunities to catch invalid transactions and increase their stakes?

How long is the challenge period currently being discussed?

The other attack vector that the fisherman opens, "grieving" attacks, whereby malicious nodes launch a series of false reports, knowingly sacrificing their stakes presumably to overwhelm the system and push through a double-spend or phony token mint or something.

Am I interpreting how this attack works correctly?

Has any thought been given to making the challenge period as long as it needs to be to process all challenges, and incentivizing goodwill from those affected by the slowdowns with a fractional return of the slashed stakes... Maybe like a reverse gas cost? Is that insane?

<comment><USER8> The main issue with fishermen is that they can't help much with data availability issues, because an attacker can just make the data available later during the arbitration process, which leads to a vector for DOS attacks against the arbitration system.

<comment><USER7> So the data availability problem renders the fisherman solution useless in a pragmatic sense?

<comment><USER7> Has sharding just recreated the double-spend problem by shattering the consensus mechanism that solved it into 100 pieces?

Are we now free-floating, hoping to find a new solution to the double-spend problem on a fragmented system?

</comment></comment></comment></comment><comment><USER9> If complexity is increasing you are going in the wrong direction _unless_ that complexity maps directly to a real problem or constraint domain.

Cryptocurrencies are just simple accounting systems and they run on general purpose computers, so no and no.

I don't think this whole direction works unless you can find an approach that doesn't add much complexity or even simplifies things in some way. I feel the same way about byzantine proof of stake schemes.

This is definitely an engineering intuition that I have, but I think I can ground it to some extent in thermodynamics and learning theory. If the universe isn't mandating complexity, it's superfluous.

</comment><comment><USER10> This is interesting, but where the article falls down for me is tying it back to actual real-world use cases. I find it very difficult to think about technical trade-offs without being able to tie it back to the actual interests of the people the system serves. It becomes too easy to solve technical problems in ways that do not solve the human problems that the system is meant to serve.

</comment><comment> Ar-Curunir Small correction: SNARKs _can_ be used for general computations. Efficiency takes a hit, though.</comment>


When to Stockpile Computer Parts? - <AUTHOR> http://www.interact-sw.co.uk/iangblog/2014/01/17/pc-stockpiling

<comment><USER2> I hope that Mac SE stored at mom's will be worth something since the Dell inspiron running xubuntu and Win 98 is not cutting it any longer</comment>


Upton: A Web Scraping Framework - <AUTHOR> http://www.propublica.org/nerds/item/upton-a-web-scraping-framework <comment><USER2> Looks like a very nice integrated solution for data scientists and researchers. It's interesting to see the different shapes tools like this take, depending on their target users. I've been happy with node.js + cheerio[1] and it's simplistic jquery-like API:

        request 'http://website.com/list_of_stories.html', (err, body) ->
            $ = cheerio.load(body)
            callback $('#comments li a.commenter-name').map(cheerio::text)

Plus if you need to handle javascript/ajax, just replace that with jsdom/chimera with minor changes.

[1] [http://npmjs.org/package/cheerio](http://npmjs.org/package/cheerio)

<comment><USER3> I found using cheerio with request and phantom.js makes anything possible.

[https://github.com/mikeal/request](https://github.com/mikeal/request) [https://github.com/sgentle/phantomjs- node](https://github.com/sgentle/phantomjs-node)

</comment></comment><comment><USER4> Cool library, Jeremy...another Ruby scraping framework you might want to check out and improve upon is Artsy's Spidey:

[https://github.com/joeyAghion/spidey](https://github.com/joeyAghion/spidey)

Has a similar approach but also leaves storage (and caching) up to the end- user.

<comment><USER5> Thanks, Dan. I'll check that out, especially for good ideas on how to solve things I haven't solved yet. Spidering and scraping seem to be very related, but not quite the same -- and I admittedly know nothing about spidering.

</comment></comment><comment><USER6> Instead of (or in addition to) manually creating scrapers, you can use Diffbot to automatically extract this type of information from news articles using computer vision: [http://diffbot.com/products/automatic/article](http://diffbot.com/products/automatic/article). It also allows you to create rules with a WYSIWYG editor: [http://diffbot.com/products/custom/](http://diffbot.com/products/custom/)

<comment><USER7> I was starting a project by using Scrapy: I was essentially going to query everyday a page (a search page accessed by a POST call), getting the results (a list of links) and then get every single result page (download an XML file).

The project was paused but I'm thinking about restarting it, and I was thinking if something like diffbot or import.io could be useful for me.. any experience doing these kind of stuff?

</comment></comment><comment><USER8> Does Upton have any way of dealing with JavaScript or ajax calls? For a lot of the scraping I do in Python, this is crucial for me. I use Selenium's Webdriver (along with beautiful soup or lxml) now for that - definitely open to other options.

<comment><USER9> There's phantomjs for that. Though I've actually had the most success with perl's WWW::Mechanize::Firefox and a headless X server.

<comment><USER10> You don't even need phantomjs, really. You can use python + webkitgtk+ through the gobject bindings. The problem with phantomjs is that it's using an old version of webkit from an old version of qtwebkit from qt 4.8, whenever that was released. By comparison, webkitgtk+ can be compiled from upstream webkit whenever you please.

If you insist on controlling webkit through javascript, you can use gnome- seedjs. But this is problematic/annoying because there's no commonjs implementation yet.. in phantomjs you can require() node modules in the outside context. Not so much in gnome-seedjs..

Also, X means it's not actually headless, even if you're using xserver-xorg- video-dummy or xvfb. For this reason, phantomjs got rid of the X requirement a number of versions ago.

<comment><USER11> Do you have any other resources about using webkitgtk with python for this sort of purpose?

<comment><USER10> No, I haven't found a definitive tutorial or reference (even for webkit). In general, look around for "import gi.repository.webkit" and you will find relevant things. I am not very sure how the other phantomjs developers are learning webkit things.. probably just reading code.

</comment></comment></comment></comment></comment><comment><USER12> For the past couple of years, I've always done any web scraping with trusty python + beautiful soup or elementtree. I've recently started doing it with Clojure + Enlive (mostly as an excuse to use clojure for less academic excercises) but I really like it.

From that perspective, Upton looks pretty cool, especially the debug mode.

<comment><USER13> Have you taken a look at Scrapy ([http://scrapy.org/](http://scrapy.org/))? My evolution has been from Perl to Python and I recently did a project with Scrapy that left me pretty happy.

<comment><USER14> Scrapy is really awesome. It's a soup to nuts queuing, fetching and extraction workflow tool and if I had to start a larger than trivial project (like an rss reader or shopping aggregation site) I would base my spider toolchain on it.

<comment><USER13> Yea - I was definitely impressed by it and I just got started. It felt as if I was able to get rid of all the boilerplate and just focus on getting the next page that needed to be crawled and the information to extract.

</comment></comment></comment></comment><comment><USER15> I've been using [http://diffbot.com/](http://diffbot.com/) for this sort of stuff, together with [http://oembed.com/](http://oembed.com/)

<comment><USER16> Been loving Diffbot for my startup. Only downside is you very rapidly approach the freemium tier for any production loads. It works fantastically well though.

</comment></comment><comment><USER17> I use YQL to do web scraping. It lets me do something like this:

`select * from data.html.cssselect where url="www.yahoo.com" and css="#news a"`

Could you elaborate on the benefits of using Upton instead of this?

<comment><USER5> AFAICT, YQL can only handle scraping individual pages that way.

Upton can scrape a whole set of pages. If you have a page that lists the pages you're interested in; suppose you're interested in HN commenters on front page posts, you could specify the front page URL and a selector for links to comment pages, and Upton would _automatically_ scrape those pages and return them to you.

Upton could even write the commenter names to a CSV for you with just a filename and a CSS selector/XPath expression.

It's not stuff you couldn't do with YQL or Python/BeautifulSoup. But it's stuff that I didn't want to have to write over and over each time I wrote a new scraper.

<comment><USER17> Makes sense! Thanks for clarifying that.

</comment></comment></comment><comment><USER18> This looks really good. We're mainly Python focused and have been working on a tool to try to 'train' a crawler to extract specific elements of a page. As this is a crawling thread I hope you don't mind me asking some advice on where to take it from here :)

Here's how it currently works:

1) It has a queue of domains that I have pre-processed. For the initial purposes I've restricted it to pages that I think are ecommerce based on $ signs, add to cart/basket type links etc

2) There is a visual tool that I then use to select certain parts of the page - eg price, product, image etc. I save these out as xpaths

3) Once I have done one URL I send a crawler to that domain and extract other pages that fit the profile of an ecommerce page and try to use the same mapping as number 2 above to extract the data

I have done a small video to show it in action:

[http://www.screencast.com/t/riB3iiVMiSk](http://www.screencast.com/t/riB3iiVMiSk)

I'm not sure if I'm doing this the right way. If a site/page changes structure then I may have to re-map the data. I was hoping that someone would have some pointers for me in terms of any other ways to do this. Also with Javascript- heavy sites I've had some problems

If anyone has any knowledge of screen scraping, where it can be done more automatically, I'd really appreciate a steer!

<comment><USER19> I've done almost exactly this in the past. There's a hell of a lot of fiddling in keeping the xpaths both stable and general enough to be useful.

One approach I found absolutely vital was to have a rewriting, caching proxy between the crawler and the upstream site. This proxy allowed me to rewrite the page content into something much simpler for the crawler to get to grips with (RSS or Atom, say). I used Celerity ([http://celerity.rubyforge.org/](http://celerity.rubyforge.org/)) with a hacked-on Mechanize API to do the rewriting, which let me handle JS-heavy pages _almost_ as easily as static HTML ones. My original inspiration for this was _why's Mousehole (the source for which is here: [https://github.com/evaryont/mousehole](https://github.com/evaryont/mousehole), I've got no idea if it runs on recent Rubies).

The proxy also gives you somewhere to raise an alert if, all of a sudden, your scraping fails because of an upstream change.

One tool I always intended to make some use of, but never got round to, was Ariel: [http://ariel.rubyforge.org/](http://ariel.rubyforge.org/). It looks like it ought to be able to totally remove the need to manually extract xpaths.

<comment><USER18> Thanks for this. I'll check these out

</comment></comment></comment><comment><USER20> Recently I've been using CasperJS for my scraping needs. [http://casperjs.org/](http://casperjs.org/)

</comment><comment><USER21> Web::Scraper is a really good one for perl. ([http://search.cpan.org/~miyagawa/Web- Scraper-0.37/lib/Web/Sc...](http://search.cpan.org/~miyagawa/Web- Scraper-0.37/lib/Web/Scraper.pm)).

<comment><USER22> Typically with Perl there is more than one module for this :)

\- pQuery | [https://metacpan.org/module/pQuery](https://metacpan.org/module/pQuery)

\- Mojo::UserAgent | [https://metacpan.org/module/Mojo%3a%3aUserAgent](https://metacpan.org/module/Mojo%3a%3aUserAgent)

\- Scrappy | [https://metacpan.org/release/Scrappy](https://metacpan.org/release/Scrappy)

\- Web::Query | [https://metacpan.org/module/Web%3a%3aQuery](https://metacpan.org/module/Web%3a%3aQuery)

\- Web::Magic | [https://metacpan.org/module/Web%3a%3aMagic](https://metacpan.org/module/Web%3a%3aMagic)

Above are specifically for scraping but one shouldn't forget WWW::Mechanize & LWP.

My preference over last few years is with pQuery. However Web::Query is Tokuhiro's pQuery _improvement_ and Mojo::UserAgent looks very nifty.

</comment></comment><comment><USER23> What separates this from Nokogiri? (Don't take that as critical, more working code out in the world is better. Just wondering, as I use Nokogiri heavily for our company chat-bot, and couldn't tell the answer at a quick glance.)

<comment><USER24> Reposting a comment by the author from the article:

> Upton depends on Nokogiri, which is basically the BeautifulSoup port for > Ruby.

> If you just used vanilla Nokogiri, you'd be responsible for writing code to > fetch, save (maybe), debug and sew together all the pieces of your web > scraper. Upton does a lot of that work for you, so you can skip the > boilerplate.

</comment></comment><comment><USER25> Recently I've been using Fake Browser (fakeapp.com) for web scraping. While it's inefficient for large jobs, it's awesome for hacking together quick scripts. With Fake you can write your scraper in JavaScript and it run in an actual browser so it's very visual process. It's great for instances where you want to get past complicated authentication systems without writing code. Just sign in manually and start your script.

</comment><comment><USER26> Sounds pretty cool. Also, for those operating in "java land" there are Commons HttpClient[1] and Apache Tika[2] which, together, are a pretty potent combination for scraping web data.

[1]: [http://hc.apache.org/httpcomponents-client- ga/](http://hc.apache.org/httpcomponents-client-ga/)

[2]: [http://tika.apache.org/](http://tika.apache.org/)

</comment><comment><USER27> What about import.io? Anyone has experience using it?

Link: [http://import.io/](http://import.io/)

<comment><USER28> Is this somewhat familiar to embedly.com? That's what I currently use, and it seems to work fine, but am curious if that's the best thing to be using (I'm typically just grabbing the image thumbnail, but it'd be nice to grab some text if it exists too.

</comment></comment><comment><USER29> Always used anemone with great results [http://anemone.rubyforge.org/](http://anemone.rubyforge.org/)

Seems like it's not being actively developed, but again, never had a problem.

Edit: realize this is focused on single page scraping with data extraction. Could use them nicely together in fact.

<comment><USER5> I had never seen this before. I will check it out, at least for inspiration.

I'd say "scraping" is a little more focused on extracting data from specific pages as opposed to ALL pages as in "spidering", but the two are certainly cousins if not siblings. Anemone would probably be good at the same sorts of tasks Upton is designed for (i.e. scraping data contained on multiple pages).

</comment></comment><comment><USER30> Nice lib, you may take some ideas from pismo: [https://github.com/peterc/pismo](https://github.com/peterc/pismo) , more metadata oriented but returns a nokogiri doc as well.

<comment><USER5> Thanks, will check it out.

</comment></comment><comment><USER31> PHP SimpleDOM Class makes scraping just like working with jQuery on the backend. [http://simplehtmldom.sourceforge.net/](http://simplehtmldom.sourceforge.net/)

<comment><USER32> I used this for many years, but the memory footprint is terrible.

I would recommend querypath. Very small footprint and takes a fraction of the cpu time.

</comment></comment><comment><USER33> For anyone interested in a Node.js crawler, my company recently released roach: [https://github.com/PetroFeed/roach](https://github.com/PetroFeed/roach)

</comment><comment><USER34> What I need is something that can scrape .NET sites with lots of weird and signed AJAX stuff just to populate a select control.

I'm using node.js with several libraries and nothing has worked so far.

<comment><USER35> You might want to look at one of the headless web browsers, like phantom or casper,

[http://phantomjs.org/](http://phantomjs.org/) [http://casperjs.org/](http://casperjs.org/)

<comment><USER34> Already tried that, and already failed.

Some JS in the page makes webkit die.

I think a headless Firefox is my only hope.

</comment></comment></comment><comment><USER36> Is web scraping legal?

<comment><USER37> If anything, it might be against TOS but ASAIK, breaking TOS is not illegal.

<comment><USER38> Who was the guy who got jail time for breaking a TOS? Think it was iPhone related.

<comment><USER37> Are you thinking of weev? [http://en.wikipedia.org/wiki/Weev#AT.26T_data_breach](http://en.wikipedia.org/wiki/Weev#AT.26T_data_breach)</comment></comment></comment></comment>