Let's Crowd-Write a GitHub Novel - why-el https://github.com/why-el/crowd-written-novel <comment><AUTHOR> It does seem like an interesting experiment, but after browsing a few random commits - and personally, any group project needs an initial structure for it to work. I can't remember right now but I have a faint memory of the subject somewhere, probably in college.

I would go over to [http://creativecommons.org/choose/](http://creativecommons.org/choose/) and pick the least restrictive license or which ever works for the group / project.

You could integrate it with gitbook as fendent mentioned (I have no experience with gitbook past the homepage), or leanpub. I personally use leanpub which integrates nicely with github and the royalty fee is sweeter when compared - despite whether it will be the goal of the group or not. Or the funds can be donated, whatever gets the group motivated or is in line with the group. However, you do need to provide some sort of direction to the group. But as you mentioned, if it's just an experiment (the replies here are in itself data) go for it.

If you do decide to give it structure, you can take a look at a very basic book I set up a while back.

[https://github.com/teichopsia-/valse_of_silence](https://github.com/teichopsia-/valse_of_silence) There are others out there that are better but I don't know of any.

</comment><comment><USER2> Novels written by groups are generally unreadable. This is because they have no focus, and because the immediate cheap laugh always wins. It's a shame, because with a little more forethought, such shortcomings could be fixed. Ie: start by communally developing an outline, main characters, etc. It would also be interesting to use Mechanical Turk in a systemic way to have "the crowd" determine which of several proposed plot-points is most consistent with the whole, etc.

<comment><USER3> Experiments like this either fundamentally don't understand what it is about a novel that makes it wonderful or do it for the (excuse me) novelty of it.

A novel that works on any sort of level does so by means of compelling, empathetic characters, relatable tones, great storytelling, and rich language. Crowdsourcing fails to be more effective than a single author at all of these elements.

<comment> why-el You do realize this is in nowhere pretending to be near as effective as a novel written even by the most okayish of writers right?

</comment></comment></comment><comment><USER4> There's a platform for pretty much just this thing [https://www.gitbook.com/](https://www.gitbook.com/)

<comment><USER5> Seconded. They also allow you to connect the gitbook account to GitHub, and version the book on your own repositories.

</comment></comment><comment><USER6> I wonder if it's possible to create a streamlined platform for this kind of work. Like others mentioned being able to start at an outline stage and then move on to fleshing the parts out would be interesting. I could see the outline being its own repo and each "unit"(probably chapters) get their own repo and maintainer. Nightly builds could be conducted to merge the work in the repos into a coherent whole, reconciling missing parts with their outline bullet points. An interface over the "build" to make quick commits and PRs would be nice too.

Additionally a community style guide and "story bible" would probably be in order. I believe a lot of writing projects which involve a large number of writers like video games and TV series do things like this. On an unrelated note, are there any tools to enforce natural language writing style compliance? Like code linters for human text?

<comment><USER7> It kinda depends on what you mean, but GNU Style and Diction [0] come to mind.

[0]: [http://www.gnu.org/software/diction/](http://www.gnu.org/software/diction/)

</comment></comment><comment><USER8> Is the merge step really necessary? Let people create their own branches from whatever starting point they like best, and let all branches/versions co- exist. It's not a piece of software you're developing after all; there may be many different viable paths, and darwinism can take care of selecting the winners via attrition of paths. I realize things can get fairly chaotic. The number of forks or sub-branches of a given version can act as a good crowd- sourced indication of popular snapshots.

Disclaimer: we've been toying with this idea for a while, and here's a paper (and tool) that came out of it: [http://onlinelibrary.wiley.com/doi/10.1002/cpe.3420/abstract](http://onlinelibrary.wiley.com/doi/10.1002/cpe.3420/abstract)

<comment><USER9> I actually built exactly this into a web app, and tested it with a few dozen authors from a local writing club. Writers could add/branch the book at the paragraph level. Readers could open up the book at the beginning or at any arbitrary node, and there was some magic that would automatically pick the longest and/or most interesting branch to "fill" the book as you read. Readers could of course also browse any other branch. This was surprisingly easy to build with Neo4j as the data store.

I never did anything more with it because it was obviously at its best with a small number of users, who all happened to be meeting in person regularly; I had no ideas about how to scale up that experience to the wider net. Still, it was a ton of fun while it lasted.

<comment><USER8> That's really cool! I think one way to make the experiment scalable is to add a social network overlay, where people can "follow" those other authors they're watching, so they can build off their contributions.

We've been using these ideas in the classroom as well: post an assignment where multiple solutions are possible, and let students build off each other's contributions (implemented as a Moodle plugin). I want to expand this into note-taking and wikis in MOOCs.

</comment></comment></comment><comment><USER10> Subreddit for the discussion of this project: [http://www.reddit.com/r/GitHubNovel/](http://www.reddit.com/r/GitHubNovel/)

</comment><comment><USER11> Going to be difficult to manage everyone appending text to the same place! Every PR will be a merge conflict.

<comment><USER12> Merge conflicts only happen when the commits happen on the same line, presumably a novel is larger than a line. So there shouldn't be a problem after a while.

Maybe you were making a joke... actually you were probably making a joke.

You were making a joke. Sorry.

<comment><USER11> Not a joke. People will be appending text to the end of the file constantly. Git usually conflicts when two people modify the end of a file. Merging one PR will invalidate every other PR.

Edit: even the very first two pull requests conflicted. [https://github.com/why-el/crowd-written-novel/pull/2](https://github.com/why- el/crowd-written-novel/pull/2) "This pull request contains merge conflicts that must be resolved."

Probably a critical flaw in this project.

<comment><USER12> Might be a bit early to tell how that's going to play out. Personally I wouldn't expect concatenations to be the majority of work, but more fill in the blanks and fleshing out of details.

That's just my guess though.

edit response: It's just starting out! There's only one sentence! This is not confirmation of your statement.

<comment><USER11> The first five pull requests have been text additions and 4/5 have conflicted looks like.

I don't know why you assume more people would edit a novel than write to it.

<comment><USER13> Editing's going to be a nightmare...

</comment></comment></comment></comment></comment></comment><comment><USER14> In the wrong hands, crowd-sourced writing can be used as a tool for great evil: [https://web.archive.org/web/20141006094052/http://www.totali...](https://web.archive.org/web/20141006094052/http://www.totalitarian.info/public/assets/totalitarian.pdf)

</comment><comment><USER12> Haha, What an approachable idea. Might as well fire off some pull requests and have them get rejected.

</comment><comment><USER15> I built a blog on wordpress which contained a story written by 6 writers. It turned out very well actually (Check the story section [http://viehebdomadaires.com/blog/](http://viehebdomadaires.com/blog/)). I had to do this by adding a new writer as editor on the blog every week. It was more challenging but having something like this would be great.

There were some rules for the writers:

Each writer who follows the first writer, will have the opportunity to take the story forward during their week. They can work with the characters already introduced in the story and have the option of creating up to 2 new characters. These extensions to the story would have to be under a maximum of 1000 words.

</comment><comment><USER16> [http://en.wikipedia.org/wiki/Exquisite_corpse](http://en.wikipedia.org/wiki/Exquisite_corpse)

</comment><comment> why-el Hello HN, this is the author. The repo needs structure and some organization, which I specifically said I forwent during my initial launch as an experiment to see what structure will emerge by simple PR merging, and if too many git conflicts, what structure could get us out of that. If git is all too cumbersome for it, then we'll learn something.

I am also in Europe and it's so late, so I will snooze for a few hours.

<comment><USER11> All seven open PRs have merge conflicts. This is just a very public lesson about knowing your tools before starting projects :) A wiki might be a better medium for this type of project.

<comment> why-el I was completely aware of the issue and I chose specifically to let it happen (As was very clear from the README). This is an exercise in how the community can bend the nature of Git to serve a nice purpose. Absolutely no issues if this fails, in fact I'd learn more if it fails then I would if it succeeds.

</comment></comment></comment><comment><USER10> I have been thinking about developing a platform for such a thing. What are your thoughts on how it should look like? What would be useful as an MVP?

<comment><USER17> Perhaps you should have a "Trie of stories" ([https://en.wikipedia.org/wiki/Trie](https://en.wikipedia.org/wiki/Trie)). Every possible sequence of letters, and thus every story, is simply a path down this trie, where each node has children containing the letters of the alphabet and punctuation. Users could mark down notable places on the trie where there is a story.

<comment><USER18> That reminds me of Borges' story, _The Library of Babel_. It concerns a library which contains every possible book. Many great books would be contained within, but it would be impossible to differentiate them from books that devolve into pure noise halfway through.

[http://hyperdiscordia.crywalt.com/library_of_babel.html](http://hyperdiscordia.crywalt.com/library_of_babel.html)

</comment></comment></comment><comment><USER19> What is the ideal timeframe till completion? Who decides what gets merged? What is the target length? What is the preferred contribution size?

<comment> why-el Timeframe, I have no idea. For merging, I'd go with consensus unless it starts to bikeshed, then I'll BDFL it.

</comment></comment><comment><USER20> Just submitted a small pull request as an encouragement. It would be interesting to see how it turns out to be. Good luck!

</comment><comment><USER21> Just a suggestion: Poetica might be a better platform for this. [https://poetica.com/](https://poetica.com/)

</comment><comment><USER22> Christ, this is why all those stereotypes of Silicon Valley exist.

<comment> dav- Stereotypes such as?

<comment><USER23> Such as trying to apply popular technologies to everything (in the best case, because of shortsightedness and in the worst case because you have a financial interest in the popular technology and its adoption for everything) instead of realizing that, in many cases, while you are holding a hammer, not everything is a nail.

Not making any comment about the submitted link, but I understand where that feeling comes from.

</comment></comment></comment><comment><USER12> I think we broke him.</comment>


Apple's Joswiak: Wundrbar is Very Special - <AUTHOR> http://blog.wundrbar.com/2009/03/apples-joswiak-wundrbar-is-very-special.html

<comment><USER2> Title is misleading: it made me think Joswiak had specifically mentioned Wundrbar. A somewhat more accurate title would be: Wundrbar's experience with the iPhone app approval process.

</comment><comment><USER3> I think Apple's statistics are probably highly misleading.

I doubt that 95% of all submissions are approved. That's probably 95% of all _apps_ are _eventually_ approved, perhaps after several rejections.

The 98% statistic probably means of the _submissions_ that are approved, 98% take < 1 week.

</comment><comment><USER4> While Apple's app approval process has been well-established as opaque and cruelly arbitrary, no one can be surprised that this app, of all apps, should have taken a long time to review. Almost by definition it steps on an awful lot of toes; I'm sure that if Apple has, as I think they've been documented, an interest in preserving the use space of their own apps AS WELL AS certain high-powered partners, then surely no one at Wundrbar can be surprised that their app, which in many ways is nothing more than a series of front-ends to other people's data, might have raised more than one eyebrow.

</comment><comment><USER5> Has anyone else noted that his name bears an uncanny resemblance to a blend of "Jobs" and "Wosniak"?

<comment><USER3> Yes. And they seem to call him "Joz" = "Jobs" + "Woz"

</comment></comment><comment><USER6> Wundrbar is like 70 apps in 1. Maybe they were testing each app separately?

</comment><comment><USER7> When I submitted my app, Apple actually called _ME_. Kind of ridiculous, but it was pretty cool. They were super friendly and wanted me to make some quick changes before approval, which I did, and it was rather seamless experience.

The code signing ritual...now that was a different story.

</comment><comment><USER8> "Meanwhile, for individual developers, the process disproportionately favors development of so-called 'crap apps.' Why should developers invest time in developing complex apps that might take months to review, when they could develop simple apps of limited utility that are more likely to get into the App Store in a day or two?"

This point seems spot on. Has anyone else been afraid to devote months to an app for the same reason? Seems like a scary investment given that you will just be subject to Apple's whims indefinitely.

<comment><USER9> My "crap app" got rejected for being of "minimal functionality" even though it duplicates the functionality of a physical device found in many restrooms in Japan, so it seems Apple is indiscriminately applying their policies regardless of application quality, leading to some making it through and others not. Case in point:

[http://createdigitalmusic.com/2009/03/16/apple-rejects- free-...](http://createdigitalmusic.com/2009/03/16/apple-rejects-free-iphone- tool-for-artists-because-of-minimal-user-functionality/)

I would hardly call this as having "minimal functionality". Yet Apple thinks so. But neither I nor Apple should have that call beyond apps that are illegal, overly resource intensive or malicious. The market should be able to decide. This behavior hinders development across the board, especially when you have no insight into why an app is rejected or what it will take to get it approved. Will Shipley has a great post on his blog about this very issue:

[http://wilshipley.com/blog/2008/09/iphone-app-store-let- mark...](http://wilshipley.com/blog/2008/09/iphone-app-store-let-market- decide.html)

</comment></comment><comment><USER10> Wait, I thought only 96% of apps got approved at all.

<comment><USER3> 96% get approved, of that 96%, 98% supposed get approved within a week.

</comment></comment><comment><USER11> Is it unreasonable for me to think that 98% of them should be approved in 1 business day?

<comment><USER12> Probably.

Even at some steady state, an app has to come into their queue and be assigned to someone. Each of those folks is probably already busy. And the approval process requires going through all the major features of the app, checking for UI gaffes, profanity, etc.

So even if they had a perfect process, you'd imagine it'd take two or three days.

</comment></comment><comment><USER13> I think this is mostly because wundrbar has similar functions as the new SpotLight in iPhone OS 3.0.

<comment><USER3> Huh? Spotlight lets you search your music, mail, notes, calendars, and a few other things stored on your phone. Wundrbar lets you search various services on the internet. There's zero overlap, AFAIK.</comment></comment>


A closer look at the Java 11 HTTP Client - <AUTHOR> https://golb.hplar.ch/2019/01/java-11-http-client.html <comment><AUTHOR> [https://www.reddit.com/r/java/comments/fgty65/a_closer_look_...](https://www.reddit.com/r/java/comments/fgty65/a_closer_look_at_the_java_11_http_client/)</comment>


FCC wants to auction off more public broadband spectrum - <AUTHOR> http://www.fcc.gov/blog/hot-times-spectrum-policy#comment-257959357

<comment><USER2> I think the auction process is flawed. It fosters an oligarchy of corporations with deep pockets and consumers end up with absurdly expensive services like texting. Sure the government gets revenues, but it just gives the taxing authority to private entities. Land lines are a natural monopoly but cellular service is not.

<comment><AUTHOR> I couldn't agree more.

Government shouldn't be auctioning off anything public, whether it is parking meters (Chicago), airports, or airwaves. We've literally had generations of future public revenues sold off for the political gains realized in single elections.

$50 Billion sounds like a lot money, but in reality it is about .4% of our GDP. It is also only a one time deposit for something a corporation gets to lease for probably 50-100 years (I'm unsure of the details). The previous auction saw AT&T and Verizon making off with prime broadband real-estate for less than $30 Billion between them.

Land-lines and many other privately held "public utilities" are often subsidized by direct government cash grants, or through tax breaks.</comment></comment>


Ask HN: How to avoid being a digital pack rat? - <AUTHOR>

I am finally admitting to myself that I am a digital hoarder.  I just upgraded my laptop and realized how much stuff I drag along each time I upgrade.  I have VMs from the last three laptops and two workstations that I keep lugging around.  At this rate my next laptop will have to have a 1tb drive just for the old VMs.  I tell myself that I will cleanup later but I am at a point where it will take a significant effort to consolidate and minimize.<p>How can I avoid this in the future?  How do you streamline your file storage and application management? <comment><USER2> As storage grows, won't your old stash become very easy to store?

<comment><AUTHOR> That was my thought but a 500GB hard drive in a notebook fills up fast when you have 40g - 100g virtual machines. I was counting on technology to "fix" an organization issue. Now I just have a lot more of the same issue.</comment></comment>


How I Hacked Facebook's Secure Files Transfer Service for Employees - <AUTHOR> http://www.nirgoldshlager.com/2013/01/how-i-hacked-facebook-employees-secure.html

<comment><USER2> This is an example of how a competent cracker (as opposed to a script kiddie) works. He's:

1) Identified a badly protected side entrance to use rather than the front door

2) Painstakingly researched the third party product (similarly one could investigate a third party library used in a bespoke codebase)

3) Figured out the adaptations the target organisation had made to it and guessed some mistakes they'd made

4) Eventually hit on a cookie modification attack made possible by limitations found in that publicly-available codebase.

Smart.

<comment><USER3> Let's also add the fact that he reported it so that it can be fixed immediately. :)

<comment><USER4> While certainly commendable, it does not make the difference between a script kiddie and a competent cracker :)

</comment></comment></comment><comment><USER5> "This Version of ionCube was not vulnerable to a possible decryption"

Does he just mean "this version doesn't have a readily available dissassembler yet"? Even if they chose the path of compiling to native code, if you own the box, it can't be that hard to get the code.

<comment><USER6> I wish he'd gone down this route further. It looks like there's code at the top of the file that "decrypts" (decodes?) the contents of the PHP file. I'd be curious why he felt this wasn't worth reversing.

I'm also curious if there's a way to dump 'disassembled' PHP after it's been loaded into the PHP processor. If it's going through eval() at the end, then shouldn't the plain-text source be available in a string somewhere?

<comment><USER7> I believe these obfuscators work by forcing you to install a binary-only .so zend extension in php.ini, which intercepts these encoded files. The files themselves should decode to already-halfway-parsed PHP opcodes or so, which is probably injected into the PHP/Zend virtual machine.

Figuring out the encoding scheme is probably a lot of boring disassembly work, which in the end just lets you decode a bunch of PHP opcodes which themselves would take a lot of work to make sense of.

But perhaps the Vulcan Logic Dumper could help out a little bit. <http://derickrethans.nl/projects.html#vld>

</comment></comment></comment><comment><USER8> I like this but what the hell is the problem with base64?

'Still alive these day?'?

It's useful for what it does, always has been, never been a security feature.

<comment><USER9> The problem is that they're using it for encryption.

<comment><USER10> They are using it to make sure the value doesn't interfere with the syntax of HTTP headers, i.e. for escaping, not for encryption. That's exactly what base64 is for. They simply have no encryption at all.

<comment><USER11> Can email interfere with HTTP headers though? There's no \r\n, ";" or "=" in valid email.

Anyway, I have seen sites where it was used as a security measure. Or so the authors thought I guess. Storing login password in url parameter? Seems safe if it is encoded.. But it was years ago.

<comment><USER12> It sounds like you confused encoded with encrypted, encoded is no different than plain-text. The encoding is just to get a valid string for the URI (so a password can contain special URI characters) and is instantly reversible.

<comment><USER11> No, I haven't. Should have put the "security measure" in the quotes.

</comment></comment></comment></comment></comment></comment><comment><USER13> Why does an enterprise level SaaS have such weak security? It is a rookie mistake for a company who emphasizes security.

<comment><USER14> I don't think you've worked with many 'enterprise' level software products. Enterprise basically means has a salesforce willing to slog through an RFP.

</comment></comment><comment><USER15> I understand that facebook has other important things to do ,than create their own file sharing service( for internal purpose ), but I can't stop asking this question, why not create their own thing ?

<comment><USER7> Well you pretty much answered your own question, didn't you?

Surely it is cheaper to buy an off-the-shelf solution, instead of spending a lot of money on wasting engineering time building and supporting a product that is not core to their mission.

</comment></comment><comment> cool-RR I'm confused. This looks like an interesting hack, but I don't understand what's going on here, despite reading the post 4 times and watching the video twice.

What is this "Password Recovery" page? Is this for emailing a person a reset link to a password? Is it for changing your password? What is the cookie used for? What is the flawed logic in the system?

Can anyone explain this more clearly please?

<comment><USER16> I think the process was that a user was sent a 'forgot password' link. When they clicked the link, they were redirected to `wmPassupdate.html` with the cookie `referer={base64(email)}` set. Then, when users submit the password update form, the `referer` cookie is used to authenticate them, but it can be trivially generated.

<comment><USER17> tptacek will be shortly along to explain how password reset links are typical and one of the most common mistakes.

How you should just "take the hit" (the cost is trivial) and store a randomly generated nounce in your database rather then do shenanigans like encoding user information in the url with secrets and bad cryptography and what not.

See? We listen! :)

<comment><USER18> Do you have any examples of someone further explaining what you are suggesting? Or if you have time, can you further elaborate on this point? I glanced at the nounce article on wikipedia and if I understand correctly you are suggesting:

1) user creates account (which generates nounce)

2) when password resetting via email auth via nounce.

3) when password is reset regen nounce

Is that right? Just trying to better understand what appears to be a good approach to password resets.

<comment><USER17> Here's a recent post of somebody else channeling tptacek: <http://news.ycombinator.com/item?id=4940291>

I got it from a quick 'password reset' search here, I recall a lot of these discussions, if you're curios click around on a few more results. :)

</comment></comment></comment></comment></comment><comment><AUTHOR> Here is another Facebook vulnerability, a more serious one -- a password reset bug: [http://chingshiong.blogspot.in/2013/01/facebook- bug-4-passwo...](http://chingshiong.blogspot.in/2013/01/facebook- bug-4-password-reset.html)

</comment><comment><USER19> For a non-facebook user/employee, can someone please explain which service that is?

<comment><USER20> Internal company file sharing.

<comment><USER21> Not just internal, but external as well. Instead of emailing a large file to a vendor, you would send them a link.

<comment><USER20> Oh true!

</comment></comment></comment></comment><comment><USER22> The date in the last screenshot (19/3/2012) doesn't match the date in the rest of the images (7/1/2013).

<comment><USER23> The last screenshot is a capture of Windows Media Player playing back a desktop video capture session; basically screenshot inception. The author provides no information on timelines, so it's likely that he found this exploit back in March of 2012, but is just releasing the write-up now.

</comment></comment><comment><USER24> Depending on user supplied data for password changes? Not a good idea...

</comment><comment><USER25> _Soon i will publish OAuth bypass in Facebook.com, Cya Next time!_

hrm?

</comment><comment><USER26> Windows Media Player?

O_o

<comment><USER27> What's wrong with WMP? :)</comment></comment>


Kaizen, the oldest productivity improvement tool - <AUTHOR> http://www.rediff.com/money/2007/jan/23kai.htm

<comment><USER2> Kaizen is the concept of continuous and neverending improvement. In Japan it stems from W. Edwards Deming's principles of statistical process and quality control imported from _America_ in the 1950s as part of Japan's post-WWII reconstruction efforts in the manufacturing sector.

Its citation here is virtually tautological: the tool to improve productivity is Kaizen, which means improving continuously! How stupendously helpful.

 _"Think, rethink and attempt Kaizen, with 100 per cent clarity and understanding;"_

Apparently all you need to get to 100 percent clarity and understanding on a fuzzy concept is a bulleted list instructing you to do so. Enlightenment, here I come.

</comment><comment><USER3> "To be effective, Kaizen needs 100 per cent participation from everyone involved." "Think, rethink and attempt Kaizen, with 100 per cent clarity and understanding;"

If you happen to not see improvement, you can't blame him!</comment>


5 Reasons Startup CEOs Should Answer Support Emails - <AUTHOR> http://blog.snapengage.com/2011/08/5-reasons-why-startup-ceo-should-answer-support-emails/

<comment><USER2> Maybe my startup is different than others, but I have to do everything at least some of the time. I can't imagine a startup ceo saying "I refuse to do X".

<comment><USER3> I can't imagine anyone at a small startup refusing to do something unless it would be seriously detrimental due to their lack of experience on the task at hand. I know the article is geared towards CEOs, but being able to touch so many different areas is one of the best things for anyone to be a part of at a startup, including the customer support aspect. There's so much you can learn.

</comment></comment><comment><USER4> Couldn't agree more. The bonus reason sums it up best: "It's the right thing to do". This kind of stuff pays off in the long-rug I bet. A few times I have replied to emails from startups and the reply-to is their CEO. I get excited, send off an email, but am disappointed when I see an employee replying to my email which was forward to them by the CEO.

</comment><comment><USER5> They are at least reasons to monitor the email and to respond to select ones. If your support has a general framework to work upon, they can handle most emails very well. Handling all/most or handling none of the support are both bad options.</comment>


HTML5 Security Cheatsheet - <AUTHOR> http://html5sec.org/

<comment><USER2> I find the title very misleading. Although this website seems to be a nice summary about HTML5 security, I don't see anything that qualifies as "cheat sheet". That title seems to be pure link bait.</comment>


Show HN: Meteorite – Smarter GitHub notifications - <AUTHOR> https://meteorite.surge.sh/ <comment><AUTHOR> A little about this project - I always wished GitHub notifications were treated like first class citizens. I wanted to know why I'm getting a notification and how important it is to me. Each day at work, I get a lot of notifications and sometimes its hard to figure out which ones I should focus on first.

For example, imagine I had two notifications: one was for somebody commenting on an issue I opened, and the other was from somebody @mention'ing me on a PR that my review is requested on which has 50 comments and is also a week old. It'd be great to know the difference so I can address the more important things first.

So basically I made a simple web-app that does all this — shows you all the notifications that are relevant to you, tells you why you're getting them, and scores them based on importance so the more urgent notifications can float to the top. It's all open sourced so anyone can pick it apart, make suggestions, or help improve it — I hope other people find this as useful as I do.</comment>


Hip tech startup Uber ditches hip tech startup Slack - <AUTHOR> http://www.theguardian.com/technology/2016/apr/04/slack-uber-apps-parting <comment><USER2> What does the word startup mean anymore?

</comment><comment><USER3> Yet both get some free publicity from this.</comment>


How much will you pay for Google Reader per month? - <AUTHOR> http://blog.raywu.co/post/45318528374/pain-point-clearly-exists-shutting-down-google-reader <comment><USER2> It's an irrelevant question -- when you work on the scale that Google does and have the data that they can analyze, it's easy to make the business case that it is either a good business product or not worth the time.

It's easy to make a survey and petition and whatnot, but power users are always the minority (the long tail) and do not drive a business alone. Especially one where the engineers supporting any product are paid six-figures a piece and the product is not directly monetized (i.e. you're not paying for it, you're not hosting the data, etc.)

<comment><AUTHOR> I agree that power users don't drive the business (you put it very nicely).

But what is the bottom line (how many six-figures engineers), and it troubles me to not see Google try and find a market for this product.</comment></comment>


Web Scraping with Beautiful Soup (2014) - <AUTHOR> http://web.stanford.edu/~zlotnick/TextAsData/Web_Scraping_with_Beautiful_Soup.html <comment><USER2> Cute. That's what BeautifulSoup is good for.

I'm a longtime user of BeautifulSoup.

BeautifulSoup does not use or create "the DOM". It does convert HTML into a tree, but that tree is somewhat different from a browser's Document Object Model. For most screen-scraping purposes, this doesn't matter. But if the page uses Javascript to manipulate the DOM on page load, it does.

I have a tool for looking at a web page through BeautifulSoup. This reads the page from a server, parses it into a tree with BeautifulSoup using the HTML5 parser, discards all Javascript, makes all links absolute, and turns the tree back into HTML in UTF-8, properly indented. If you run a page through this and it still makes sense, scraping will probably work. If not, simple scraping won't work and you'll probably have to use a program-controlled browser that will execute JavaScript.

Some examples:

HN looks good: [http://www.sitetruth.com/fcgi/viewer.fcgi?url=https://news.y...](http://www.sitetruth.com/fcgi/viewer.fcgi?url=https://news.ycombinator.com)

AFL-CIO, the site used in the article, looks great: [http://www.sitetruth.com/fcgi/viewer.fcgi?url=aflcio.org](http://www.sitetruth.com/fcgi/viewer.fcgi?url=aflcio.org)

Twitter's images disappear: [http://www.sitetruth.com/fcgi/viewer.fcgi?url=https://www.tw...](http://www.sitetruth.com/fcgi/viewer.fcgi?url=https://www.twitter.com)

Adobe's formatting disappears: [http://www.sitetruth.com/fcgi/viewer.fcgi?url=https://www.ad...](http://www.sitetruth.com/fcgi/viewer.fcgi?url=https://www.adobe.com)

Intel complains about the browser but looks OK: [http://www.sitetruth.com/fcgi/viewer.fcgi?url=intel.com](http://www.sitetruth.com/fcgi/viewer.fcgi?url=intel.com)

Grubhub gives us nothing as plain HTML: [http://www.sitetruth.com/fcgi/viewer.fcgi?url=grubhub.com](http://www.sitetruth.com/fcgi/viewer.fcgi?url=grubhub.com)

Same for Doordash: [http://www.sitetruth.com/fcgi/viewer.fcgi?url=doordash.com](http://www.sitetruth.com/fcgi/viewer.fcgi?url=doordash.com)

(No scraping restaurant menus with BeautifulSoup.)

Cool stuff in pure CSS works fine: [http://www.sitetruth.com/fcgi/viewer.fcgi?url=css3.bradshawe...](http://www.sitetruth.com/fcgi/viewer.fcgi?url=css3.bradshawenterprises.com/cfimg/)

(You don't really need Javascript any more just to get the page up.)

<comment><USER3> Ooo this is really neat. Thanks for sharing!

</comment></comment><comment><USER4> My web scraping toolkit (Python):

-Beautiful Soup

-Requests Lib

-JSON Lib

-Selenium

-Urllib2

-Cookielib

Handles 99% of things I encounter. With dynamic sites, you're often better off simulating the request than controlling the browser.

Then you get to parse your way through someone's annoyingly formatted JSON.

</comment><comment><USER5> I enjoy using scrapy because it allows for a bit more functionality. Check it out if beautiful soup is too simplemente for your needs.

<comment><USER6> Which would you say is easier for a beginner(to scraping and to python in general) to use?

<comment><USER7> BS4 is easier but scrapy is more powerful. E.G., with scrapy you can program to follow all 'next' links and download 100s of pages in parallel. BS4 is more for grabbing content from a single page.

</comment></comment></comment><comment><USER8> Absolutely love BeautifulSoup, I use it almost everyday in my job. There isn't really a scrapy vs. BS4 divide as you can still use the library with Scrapy, as opposed to it's standard parsing functionality. It also works well with lxml which is considerably faster.

It's also possible to build very performant and large scale crawlers with just BS4 and requests . Though managing the architecture is a bit of a pain, but it definitely can be done.

There are also a number of cases where it's better to use Bs4 than scrapy.

Also using PhantomJS, Selenium and BS4 can provide you with a very powerful data scraping solution.

</comment><comment><USER9> I recently used BeautifulSoup in a Wikipedia scraping project. It's definitely a great tool but it had a few annoying functionality issues.

I had some preprocessing using decompose(), which can take lists of attributes. However it cannot decompose multiple attributes at once such as anchors with a certain title and table elements in one call. It felt cumbersome to call it multiple times.

It seems like Scrapy does exactly what I needed though (click on the first link in the article), which would have saved me a ton of headache. So it goes.

</comment><comment><USER10> NEWBIES BEWARE!!! USE THE LXML PARSER ... screw the inbuilt html parser</comment>


An Undocumented Princetonian - <AUTHOR> http://www.nytimes.com/2010/01/03/education/edlife/03alien-t.html?ref=edlife&pagewanted=all <comment><USER2> Great story/</comment>