Is Apple experiencing a problematic decline in software quality? - <AUTHOR> http://www.tuaw.com/2014/10/15/is-apple-experiencing-a-problematic-decline-in-software-quality/ <comment><USER2> I'd just like to point out that this is likely not to get better with this (arbitrary) decision to release a new OS every year. It used to be that we'd have a new release roughly every 2 years (from 2003 - 2011). This meant: 1. it was in development for 2 years and thus probably stabler to begin with, and 2) there were then 2 years to iron out the bugs. You'd get the new OS, there were some understandable kinks, but then by 6-12 months later it was _pretty solid_ and you had another year of a solid OS ahead of you. It was perfectly reasonable to wait on the OS and let the early birds kick the tires first.

This is no longer possible. It feels that as soon as version N is out, they are scrambling to make version N+1. There is no downtime. There is no stabilization phase. You are eternally in brand-new isn't fully working mode. _Of course_ the software is going to be worse.

Couple this with the stark reality that Apple has simply _run out of ideas_ in terms of software. Every new version of OS X boils down to: 1. arbitrary UI tweak (forcing developers to refresh), 2. Gimmick features in Mail.app/Safari (RSS in Mail/Safari, Postcards in Mail, yet another 3d effect to re-arrange your tabs in Safari, annotations in Mail, etc etc etc), and 3. regressions of features that worked for _years_. Occasionally .Mac/MobileMe/iCloud will be renamed in hopes everyone forgets about the last round of data loss bugs/hopefully people get excited about this vague thing they don't really know the scope of.

<comment><USER3> >This meant: 1. it was in development for 2 years and thus probably stabler to begin with

No. Shorter release cycles don't imply loss of quality. I claim the opposite in fact. Rolling releases are much better for quality. You're releasing smaller more focused features and battle-testing them in the field.

>There is no stabilization phase.

There are always stabilization phases. Not every major release will be an overhaul. Most will be incremental updates.

<comment><USER2> _> No. Shorter release cycles don't imply loss of quality. I claim the opposite in fact. Rolling releases are much better for quality. You're releasing smaller more focused features and battle-testing them in the field._

The jury is out on this. We have seen that longer release cycles with teams that take things seriously can be very successful (mission critical software). We've also seen that short release cycles can be very effective. The context of your software matters a great deal here, as well as the basic mechanics of how your software is delivered. You can't just take the philosophies of web sites and apply them across the board and declare it objective fact:

1\. Take for example the fact that with a web service you can deploy quickly to a small subset of users, and grow it as it proves itself. This is quite ideal, but regardless of whether its even practically _possible_ with OS software, its certainly not what Apple does. The combination of a short release cycle with a worldwide release means that "catching your bug" may mean catastrophic data loss for a large amount of customers.

2\. With a web service you have the option of rolling back a bad version, or, simply deploying yet another release. Again, whether its possible with OS's or not, Apple certainly doesn't (or can't?) employ this strategy. The best they can do oftentimes is simply remove the update, but everyone that has it installed keeps having the bug until a) your fix is out and b) they actually install said fix.

 _> There are always stabilization phases. Not every major release will be an overhaul. Most will be incremental updates._

There certainly _could_ be stabilization phases, but the proof is in the pudding and thats not how it feels. All the last releases have seen major additions, and most the regressions I run into either take years to fix (dual monitor support) or still remain increasingly broken (Messages to name just one example). There is a simple reality to having a year to stabilize without the pressure of a PR push of new features vs. balancing your stabilization with coming up with compelling things to slap on a website and show in a keynote.

</comment></comment></comment><comment><USER4> Apple's OSX design work seems stagnant as well. I find Gnome since version 3.12 (yes Gnome) looks better than Yosemite. I use Yosemite at work every day and find it's a step down from Mavericks. Most of the improvements seem to be to apps I never use (Maps, Safari). The new dark top bar is embarrassingly ugly compared to Gnome dark theme. The icons look childish and half-baked.

Gnome's design in minimal, clean, and feels out of the way instead of being flashy. Most importantly Gnome is improving rapidly. The new 3.14 looks amazing, where they revamped all the small details (icons, resolutions). A great demo of it here: [https://www.youtube.com/watch?v=VhK_2M0B8Qo](https://www.youtube.com/watch?v=VhK_2M0B8Qo)

I used to love getting new OSX releases. I'm really surprised to see it get sidelined in recent years. Maybe all of the good designers are working on iOS?

<comment><USER5> I second your enthusiasm for Gnome. It really is in a great place right now and is my shell of choice on Linux. But I disagree with the Yosemite criticism. Unlike Linux, OS X actually has a large degree of uniformity between third-party apps thanks to Cocoa and (historically) the human interface guidelines. Apple just can't make sweeping changes to the OS X design without severely damaging the coherence of the OS X ecosystem; design changes have to be incremental. And they're doing a pretty good job. Yosemite brings design changes specifically optimised for high-DPI (font, line weights) and a general toning down of brashness (e.g. the new dock). I imagine the new 'dark mode' (which, let's face it, will be unseen by most consumers) is just a signal of things to come; I wouldn't be surprised to see a dark set of Cocoa controls in a future release too. But it'll take time.

Honestly, I'm glad that Apple isn't being brash on the Mac. They can get away with it on iOS (the 6->7 transition) because it's such an active platform, with so much demand, that developers have put in the time and money to re- design, but a similar design transition on OS X would take much much longer and I imagine would be a lot more painful for the end user.

<comment><USER4> Fair enough, OSX is still a fine operating system. My taste for change is that of early adopters so I'm at least happy Gnome is filling that void for me.

</comment></comment></comment><comment><USER6> Apple has had a decline in software quality since around the Intel transition. Prior to that OS X had become really very respectable, but during it lots of typical things started happening, like suspend not quite working sometimes, and external display support going to hell. It doesn't seem like they ever recovered.

It's hard to admit now, but at one point I even liked XCode, but again, that was before they iTunesd it.

Maybe it's age, but in my mind they definitely did get worse, and as a result I've moved to Windows (7) as my main machine after over a decade in Mac land.

<comment><USER7> I literally went the opposite. I switched to OSX soon after the Intel transition and loved it, after a decade in Windows land. I honestly just much prefer the look and feel of Apple products and software. I find them more enjoyable to use, consistent, more intelligently thought out and much less hassle.

I also used to hate Xcode 3 but as of Xcode 5 I've really grown to like it.

You're my bizarro opposite. I bet you hate the colour blue ;)

<comment><USER8> I've been a MacBook user for 5 years, but Windows File Explorer is still light-years ahead of the atrocity that is Finder.

<comment><USER9> I've tried to "go mac" about 3 times now, at quite a bit of expense, and the horrible and endless well of failure that is Finder continues to rebuff me. Even as Explorer continues to change and ends up with some weird corners, it's such a much better and more efficient file manager.

I'll find myself going weeks, enjoying my rMBP, then I'll need to do some heavy duty file management and end up in Finder-hell and immediately wish I was using _something_ better. DirectoryOpus, Midnight Commander, _anything_.

It says something when the CLI is not only faster to use, but more user- friendly and discoverable than the GUI.

In fact, I'm _about_ to start work on a bunch of file management and I've been putting it off for a week _just_ because I don't want to interact with Finder.

Gahhh, it's a terrible terrible piece of software.

(maybe I should just break down and install this [http://www.ragesw.com/products/explorer.html](http://www.ragesw.com/products/explorer.html))

<comment><USER10> I always read about people hating Finder, and I never understand why.

There are some things that I can't live without: being able to drag a file or folder into a file open/save dialog (try doing this in Windows! HOHO), and the column view. It really makes file management a breeze, and I rarely drop into the Terminal, unless I'm doing some heavy-duty renaming, or stuff like that.

Could you elaborate on the Finder problems you're encountering?

<comment><USER9> I could probably provide you with pages of issues but they all bundle into a couple of major categories

1 - unbelievably poor context presentation, e.g. if I paste? where in the file system will something end up? Who knows? Where am I in the filesystem? I can guess! Is it correct? not usually. How about let's play the game of which kind of file is this? Because filenames usually end up with a '...' somewhere in them and I have to fiddle with Finder _every single time_ to get it wide enough to stop truncating filenames so I can figure out which file is "P1250416.JPG" vs "P1250417.JPG". Folders mix in with filenames when sorted so navigating up and down the tree takes forever (and is harder to do keyboarding). And on and on and on.

2 - completely nonstandard keyboarding and navigation, e.g. ENTER to rename, cmd+o or cmd+down to open a file? really? In which way are either of those possible intuitive? It appears Finder is trying to follow some impossibly ancient keyboard shortcut system that was probably put in place 30 years ago and doesn't make any sense at all.

Here's a typical use-case for me. I just shot about 3,000 photos and I want to do a quick pass on the photos and delete bad photos and move photos of a certain kind (photos with a certain subject) into another folder. In explorer it's a matter of hitting "enter" then "right" until I see one I don't like then hitting "delete" to remove it then continuing with "right" until I find more to delete or finish the bunch. Moving ones with a specific subject involves me ctrl+mouswheel until the thumbnails are as big as possible so I can see the subjects, then ctrl+lmb on all the ones I want to move then ctrl+x, move into the folder, ctrl+v (and now they're all sorted and not scattered all over the place like in Finder and I moved them using the completely system consistent cut-paste keyboard hotkeys) and alt+left to go back (just like a browser). Other niceties like being able to maximize and then restore back to the default window size with a couple mouseclicks are also smoother.

In Finder all of this becomes work instead of a few minutes of repetitive keypushing. I can almost do the entire workflow without a mouse in Explorer, and where I have to mouse it kind of makes sense (picking specific items from a group) over a keyboard. But in Finder, just to get started, I have to buy and install a couple pieces of software.

Sure most of these things can be "fixed". If I install this or that extension, and customize finder in this or that way and remap such and such keyboard hotkey I can kind of end up with a sane workflow and filenames I can actually see. But I shouldn't have to fix shipping software. Everytime I open Finder I'm asking myself if anybody at Apple actually uses it.

Take a look on the internet for people complaining about Finder and most of the complaints are more or less along the same lines. The complaints are consistent and have been going on for years. Solutions have been hacks since forever as well and usually involve installing $100 of replacements, addons or fixes.

What's amazing to me is that these problems don't exist in any other GUI file manager I've ever used, from the Amiga to my TI calculators. Finder is just rubbish.

</comment></comment></comment></comment></comment></comment><comment><USER11> My iPhone 6+ has shown more bugs in 2 weeks than the three iPhones I had prior to that (the original, 3G, and 4). Has hard locked at least half a dozen times, particularly when receiving calls.

<comment><USER12> Similarly, Safari frequently locks up or crashes on my iPad -- behavior that started with 8.0.0 and has continued through 8.0.2. I rarely remember this happening in the past.

<comment><USER13> I can crash my entire iPad every time I try to start a slideshow while connected via AirPlay in one of my apps (Portfolio). From the technical side of things all it does is load a full resolution image into a CALayer instance and fade it in, which has worked on every previous iOS version but for some reason crashes the entire device on iOS 8.

</comment></comment></comment><comment><USER14> > _Is Apple experiencing a problematic decline in software quality?_

No, blogs and articles are experiencing a problematic decline in long and medium term memory.

Apple has always (under Jobs or not) had ups and downs, in both software and hardware quality control.

Remember how OS X 10.1 was unusable, the problems with Lion, when it first came out? The file-loss bug in the FS? And tons of other things besides.

As for hardware, well, Jobs first love child was the cube, with the overheating problem (and the not-selling-well problem). Then we had the iBook G3 logic board issues (for tons of models). Battery issues. The G5 Pro cooling goo leak issue. Etc etc. And of course, as any long time Mac buyer knows, a classic advice is "never buy the first revision of a product".

Part of it, for hardware, is that a bug in a production run e.g. for Dell doesn't affect that many people (because Dell puts out 50+ different models, whereas Apple puts out a few, so each of Apple's has tens of millions of buyers). And of course the press doesn't care much for a fault in Dell or HP or whatever production run, whereas the slightest BS in an Apple production run is a "*gate". And of course Apple does more daring stuff with machining, weight, thinness, internal design etc than most companies, so there's always a chance to screw some things that's bigger than in just assembling some brick- sized plasticy laptop.

</comment><comment><USER15> No, I don't think so. People are perceiving it that way because a few unrelated issues have cropped up lately. Coincidence, not correlation.

<comment><USER16> But coincidence is a type of correlation, is it not?

<comment><USER17> Strong correlation requires a large number of 'coincidences'

</comment></comment></comment><comment><USER18> You took the word out of my mouth! It is by far and it is also losing some of its legendary attention to detail.

I've noticed the amount of patches going up as well as the stability of software go a little bit down. But that's not all, most major unix tools in Mac OS X are 2 to 3 years outdated and Safari is a bug nest.

To bring some balance to my criticism I fell that innovation at Apple has not decreased but the quality of products has.

<comment><USER19> As referenced in this article[0] a major blocker for shipping newer versions of the command line toolchain is the fact that newer versions of these tools moved to an incompatible GPLv3 license.

[0]: [http://robservatory.com/behind-os-xs-modern-face-lies-an- agi...](http://robservatory.com/behind-os-xs-modern-face-lies-an-aging- collection-of-unix-tools/)

<comment><USER20> Oh come on, that's far from serious. There are 6 pieces of software there that are old because newer versions are GPLv3. Grep could be replaced with bsdgrep, and nano could be replaced with pico and nobody would notice. The other 4 aren't that important for day to day use except bash and OSX should have switched users to zsh a long time ago, or simplified the shell by using tcsh as default.

The older version of openssh is rather perplexing, though, but they haven't updated any of the other bsd utilities really and the pf firewall is ancient too.

</comment></comment></comment><comment><USER21> I've noticed this over the past few years. What I've also noticed is a move by Apple towards marketing not what their products can do, but by counting how many new features they they've added to their software. They're still far ahead of their competitors in terms of quality and usability of their products, but not what they once were.

</comment><comment><USER22> Apple said: "iOS 8 includes over 4,000 new APIs that let you add amazing new features and capabilities to your apps"

Maybe not a good thing.

<comment><USER23> it's like an author advertising word count... weeee...

More APIs == more bugs, more nastiness under the hood and a rats nest of keeping track of the OS.

</comment></comment><comment><USER24> Wait - this supposes they had quality to start with. Not in my experience, and I have been using their offerings since almost the beginning.

I think the difference is that people are finally feeling the confidence to state the obvious - why is the software so sucky?

<comment><USER25> >I think the difference is that people are finally feeling the confidence to state the obvious - why is the software so sucky?

I agree; the Jobs distortion field is wearing off. There's a lot of cognitive dissonance to get past (this must have been better before because I liked it before), and this dawning recognition is getting projected onto Apple.

<comment><USER26> But the distortion field that before exist a magical distortion field is still strong...

Similar to: Jobs make everything at apple, apple say it invented first (about everything), people that buy apple is a lunatic fanboy, bla, bla...

You have a more stable software or not. You can show numbers, or at worst, anecdotal evidence. But blaming in a distortion field?

</comment></comment></comment><comment><USER27> It is difficult to maintain focus on software as a hardware company (and vice versa). While Steve Jobs was around he was able to emphasize importance of both hw & sw. Over time the part of the company which brings the money receives increasing investment at the expense of the other. For Apple, maintaining a balance will be a challenge going forward. QA seems to be the first compromise.

</comment><comment><USER28> It's all the little things. iOS 7 introduced a ton of glitches in the Music app that still aren't fixed in iOS 8.

</comment><comment><USER29> The latest reports on personal data leaking onto removable devices like usb drives are quite worrying too:

[http://www.reddit.com/r/netsec/comments/2jbfgp/osx_appears_t...](http://www.reddit.com/r/netsec/comments/2jbfgp/osx_appears_to_leak_data_metadata_when_indexing/) [https://news.ycombinator.com/item?id=8450848](https://news.ycombinator.com/item?id=8450848)

(PS: Had to paste these links into Notes.app after copying the urls from mobile safari's share sheet, and then copy from there to paste here because mobile safari won't allow paste in text fields from its own share sheet in ios8... How's that for QA)

</comment><comment><USER30> Yosemite has been great but I still run into a few bugs here and there with iOS 8.0.2. I think they should consider decoupling hardware and software releases.

</comment><comment><USER31> This comments section has lots of people saying " _x_ was when things got really bad" followed by people saying "actually _x_ was the pinnacle." (And some then following up with "it was actually _y_ when it got bad.")

This has been the pattern in The Comments since I started using Apple stuff 15 years ago. Likewise, "Apple used to innovate and now they're out of ideas."

Which isn't to say that things haven't perhaps taken a downturn lately: there have unquestionably been an unusually large number of publicized missteps in the last couple of months. Some technical, some social, some real, and some media-hyped. But an apparent clustering at one moment doesn't assure or even imply a long-term trend. Five and ten years from now I feel confident the same comments will be being made.

People get annoyed at the un-Steve-like-ness of Cook's apparent focus on quantifiable things like "CustSat". But, when you're serving literally hundreds of millions of customers in a blog-driven media world, it's hard to know what would be a better way to measure these things. If customers are becoming less satisfied, you can be certain the executive team knows it.

Still, it seems apparent that there's an immediate PR issue. (And simultaneously, it should be noted, iPhones are flying off the shelves in unprecedented numbers and at likely higher ASPs than ever.) If I were Cook, I'd do what I could to make really sure we got our act together in the short term, look for some opportunities to buy back some goodwill, and then I'd probably keep doing roughly what we'd been doing for the last decade or so while weighing the feasibility of some of the commmon suggestions, such as decoupling OS releases from hardware releases.

</comment><comment><USER32> I suspect this is more reflective of a scale issue than a direct decline attributed to the capabilities of the company.

I suspect that when a company reaches a certain amount of market penetration where the number of active users reaches into the 100s of millions, that number of eyeballs over that many devices begins to show the cracks which maybe got glanced over previously.

My comparison for this is, of course, Microsoft. As they were coming up, we were forgiving of their set-backs, but into maturity, people started looking for alternatives, their had to be something better because their stuff was SO buggy. I point to the Vista 'fiasco' as an example. Was it really a 'fiasco'? Or was it just that, even when what was considered a small number of people upgraded, and recognized issues, that small number of users was so large that it brought major attention to the issues.

When 10% of your users have issues and you have 10 million users, that's 1 million voices. When 10% of your users have issues and that's 100 million users, it's considerably larger.

Diversity of hardware platforms further exacerbates this issue. When it was just the iPhone1-3, the hardware wasn't considerably different. Bring on the iPhone 3, and increased pixel count, and you start to notice a few more minor issues, then iPhone 5 with different screen layout, handled well by apple, but not seamless. Now start adding some devices having fingerprint scanners and some without, some with health data gathering and fingerprint scanners some with one of these things, some with none. Sure, you 'should' be able to test for these small differences, but it gets considerably more complicated with each iteration. Apple has done a good job of getting people to retire old hardware, or not cause a fuss about not being able to upgrade, but they're still getting into a realm of device numbers they hadn't experienced before.

</comment><comment><USER33> I completely agree with the article an will recommend reading Apple Core Rot: [http://macperformanceguide.com/AppleCoreRot- intro.html](http://macperformanceguide.com/AppleCoreRot-intro.html)

</comment><comment><USER34> Yosemite is pretty fantastic.

</comment><comment><USER35> Yes, but in my experience this is an industry-wide problem not specific to Apple.

I know I'm getting a bit old so there's surely some amount of 'get off my lawn, kids' going on here, but I kind of expect new software to mostly suck these days for many reasons but with the 'webification of everything' (in terms of pervasive use of JavaScript, throwing out all the old UI frameworks and replacing them with web DOMs or web DOM like systems, replacing graphics systems with WebGL/Canvas, etc) being a primary cause.

(awaiting the downvotes...)

</comment><comment><USER36> I own 2 separate apple PCs. And seriously considering replacing one of them (Mac Mini) with Mac Pro ($3000 starting price !!) for video work. And probably getting another mac laptop for wife.

I got on the apple fan train kinda late, like right around when they transitioned to Intel.

With that said, I sometimes feel Apple hitting it big with iPhones was similar to someone winning a lottery. Often when someone wins jackpot, he grows distant from friends/family because of his money. He loses his direction in life due to the sudden infusion of wealth.

I feel that way with Apple.

</comment><comment><USER37> If this is real, I don't think it's about release timing at all, but the technical complexity of implementing and integrating the specific new features they've chosen.

I don't have a list of specific examples to back up that claim, but I encourage other apple devs here to think about recent releases from that perspective.

</comment><comment><USER38> I don't see how you can deny it. They're doing a lot of things and some of them have very hard deadlines.

<comment><USER39> They can fix this problem. Decouple the hardware and software releases. There is nothing wrong with a new device that runs the older OS for a couple of months. Look at it this way:

\- New device with a new OS that requires a few rapid updates in the weeks following release because the OS was rushed and is very buggy. > If one of these updates go wrong, which has happened, the user is stuck with a broken device or they could downgrade to an already buggy and less stable OS as before.

\- New device with the "old" but stable OS that continues to function well until the new, stable OS comes out. > Even if the new OS's initial release is troublesome, a downgrade will take the user to the older but stable OS, so the device is still functional.

Which provides the better user experience? I vote for the second option, but Apple have opted for the first option without any clear long-term benefits.

</comment></comment><comment><USER40> I'm wondering why no iOS/OS X devs aren't mentioning the "Quality" of Swift "1.0". The amount of issues is really stunning and IMHO, nowhere ready yet.

Xcode doesn't seem to play along with Swift either, the whole integration is quite buggy yet.

</comment><comment><USER41> It's the secrecy. Apple can't test things extensively because the whole world is trying to get a pic of their latest devices. That coupled with huge scale is bound to create hiccups, the real test is how fast and how well they address the issues.

<comment><USER36> But we are talking about software here, not hardware...

</comment></comment><comment><USER42> I personally believe iOS 8 has been the exception, not the rule.

Yosemite, including the Public Betas, has been rock solid.

<comment><USER43> I definitely would not say that Yosemite Betas have been rock solid.

<comment><USER44> For betas they have been good. I don't set the bar high for those. The last public beta is working really well. Yosemite reminds me of KDE in some of the stuff Apple has done.

<comment><USER43> The later versions have been decent but I installed the first public beta and really regretted it for like a month or two until it got a bit more stable.

<comment><USER45> Surely you did not disregard all the warnings on the download website and the installer itself and installed it on the main partition of your primary work computer?

<comment><USER43> Whether I did or not is irrelevant. Grandgrandparent was saying that the betas have been solid, I disagreed.

</comment></comment></comment></comment></comment></comment><comment><USER46> How bad would OSX have to get to actually have the year of Linux?

</comment><comment><USER47> Betteridge's law of headlines strikes again. [http://en.wikipedia.org/wiki/Betteridge's_law_of_headlines](http://en.wikipedia.org/wiki/Betteridge's_law_of_headlines)

Edit: Why the downvotes? It's true. The answer is No.

<comment><USER25> I prefer 'Blot's Headline.'

[https://news.ycombinator.com/item?id=7954867](https://news.ycombinator.com/item?id=7954867)

</comment></comment><comment><USER48> The first rule of Apple software quality is that you don't talk about Apple software quality. It's like complaining about Rolex watches not being accurate.

Typical error on a Rolex is a minute or two a month, sometimes worse. Rolex doesn't even provide chronometer certification for their watches. As their CEO says, "We are not in the watch business. We are in the luxury business".

That's the market Apple is aiming at.

<comment><USER49> If a Rolex stops keeping time, it's conceivable it would still be worn. But if a BMW no longer starts up, the driver's not going to sit behind the wheel.

Whether it's the (mostly) info-appliance market of iOS, or the productivity market of the Mac, Apple's products are much more like BMWs than Rolexes. The luxury is intertwined with the functionality.</comment></comment>


Python Libraries you should know about - <AUTHOR> http://doda.co/7-python-libraries-you-should-know-about <comment><USER2> I'm only surprised to not see any of Kenneth Reitz' work on this list, e.g. Requests. In fact in that vein is rauth, an OAuth client lib built on top of Requests (<https://github.com/litl/rauth>). Full disclosure: I'm the author of rauth. :)

<comment><USER3> Hey. Lots of people suggested this so I added a notice to the blogpost to further explain why I chose these libraries:

> I specifically excluded awesome libs like requests, SQLAlchemy, Flask, > fabric etc. because I think they're already pretty "main-stream". If you > know what you're trying to do, it's almost guaranteed that you'll stumble > over the aforementioned. This is a list of libraries that in my opinion > should be better known, but aren't.

</comment></comment><comment><USER4> Since everyone is asking (and no one is doing), I put together a simple benchmark for pyquery, bs4, and lxml (cssselect/xpath).

<https://gist.github.com/4061368>

All it does is grab paragraphs from python.org's html a couple thousand times.

        ==== Total trials: 100000 =====
        bs4 total time: 31.6
        pq total time: 9.3
        lxml (cssselect) total time: 5.4
        lxml (xpath) total time: 4.3
        regex total time: 8.9 (doesn't find all p)

What does it mean? Unless you're running thousands of queries for parsing, it doesn't matter which library you choose. My computer old and slow. Pick which one is the easiest, that you'll fight the least with. Don't put energy into unnecessary optimization. Using a good library is like choking someone, they'll fight for a little while until they pass out. (you'll remember this analogy next time you want to switch libraries, do you really need to choke someone to get your job done?) After they pass out, it's smooth sailing and you don't have to worry. Don't rock the boat unless you have to.

<comment><USER3> Haha, thank you mercuryrising. That's quite the interesting comparison. A few thoughts:

\- Python.org is quite a simple web app, it would be interesting to run this against something a little more complex like a long wikipedia article or <http://www.nytimes.com/> or even the Alexa Top 500

\- It would also be interesting to split the times between parsing and selecting as I feel that's where the difference between pq and lxml comes in.

All in all it seems BS4 is quite a bit faster than I gave it credit for (especially factoring in parsing+selecting instead of just the former)

<comment><USER4> Python.org was pretty simple. bs4 really hits the brakes on the NYT page.

        ==== Total trials: 100000 =====
        bs4 parsing time: 0.6
        bs4 selecting time: 146.3
        pq parsing time: 0.0
        pq selecting: 15.7
        lxml parsing time: 0.0
        lxml (cssselect) selecting time: 12.4
        lxml parsing time: 0.0
        lxml (xpath) selecting: 11.5

<comment><USER3> That seems more in line with my experience :) This is the stuff lxml shines at. Thank you for testing this out.

</comment></comment></comment></comment><comment><USER5> You should probably add requests <http://docs.python-requests.org>. Anyway great list! I'm already using most of them and they are awesome.

<comment><USER3> Hey. Author of the blog post here.

I specifically excluded awesome libs like requests, SQLAlchemy, Flask, fabric etc. because I thought them too "main-stream". If you know what you're trying to do, it's almost guaranteed that you'll stumble over the aforementioned. I tried to compile a little bit of a list of libraries that SHOULD be better known, but aren't.

<comment><USER5> I see your point but in my opinion, requests is quite different from SQLAlchemy, Flask and fabric. I wouldn't put them at the same level.

</comment></comment></comment><comment><USER6> When compiling lists of the best Python libs, one definitely has to check out Pocoo: <http://www.pocoo.org/> They are a bunch of dudes who are incredibly skilled at putting together great API:s. All their libraries, from pygments, jinja2 to sphinx are well-documented and extremely simple to use.

</comment><comment><USER7> One I'd like to add is Docopt: <http://docopt.org/> and <https://github.com/docopt/docopt>

It makes it very simple and intuitive to build command line apps.

<comment><USER8> And if you want to have both command line args _and_ config files, check out my project: <https://github.com/ipartola/groper>. It even supports creating a sample config file out of the options you have defined.

<comment><USER9> This might just be your examples, but it seems a bit verbose.

For example for:

define_opt('server', 'daemon', type=bool, cmd_name='daemon', cmd_short_name='d')

Can I just write define_opt('daemon') ?

<comment><USER8> It is a bit verbose, you are right. Let me explain why all of these are necessary. (BTW, argparse is almost as verbose [1]).

server - this is the section/module. I could grab this from the name of the current module, but that means you must use unique module names, and not change them. Otherwise, your config files would stop working.

daemon - required, obviously, since this is the name of the option.

type - I can't assume a type for you. By default it is a unicode instance, so you can omit this parameter if that's your use case.

cmd_name/cmd_short_name - I can try to assume that cmd_name is the same as the second positional argument, but once again, there could be a conflict. For example, if you want to have options.db.filename and options.log.filename, you can't use cmd_name = 'filename' for both. cmd_short_name is even worse, since here you may want a specific letter to be used (such as an upper case D instead of d). Note that these parameters are optional, since most of your values will likely go in the config file, not on the command line.

An alternative API might look like this:

      define_opt_int('server', 'shutdown_timeout')

      define_opt_bool('server', 'deameon')

      define_opt_cmd_unicode('server', 'pid_file', cmd_name='pid', cmd_short_name='p')

With a fallback to:

      define_opt('log', 'level', type=lambda x: x if x in LOG_LEVELS else 'DEBUG', cmd_name='log-level', cmd_short_name='L')

Any feedback is greatly appreciated.

[1] <http://docs.python.org/dev/library/argparse.html>

<comment><USER10> So maybe just define_opt('server.daemon'), with the rest given sane defaults and customizable as needed?

Interesting module nonetheless!

</comment></comment></comment></comment></comment><comment><USER11> The very first suggestion complains that BeautifulSoup is too slow, but as of version 4, it's actually just a navigation layer on top of your preferred parser. So it's as fast as lxml, and as easy to use as, well, BeautifulSoup.

<comment><USER12> I used BeautifulSoup for a project once because of all the accolades it gets here but found it to be less than robust. It might be sufficient in a scenario where you have a single site you need to scrap but I found it was totally unreliable when used across a wide range of sites and esp. sites with foreign language content.

<comment><USER13> I started using PyQuery yesterday, after using BeautifulSoup for a long time. It seems much easier to use.

      pq_page = pquery(url=PAGE_URL)

Note that PyQuery has some encoding issues too (or rather the sites I were scraping were too bad, showing two different encodings in meta tag!), here are two different things I have done to workaround:

      page = requests.get(PAGE_URL)
      pq_page = pquery(page.text)

If that doesn't cut it (because requests detects it wrong too), try forcing the encoding in requests:

      page = requests.get(PAGE_URL)
      page.encoding = 'utf-8'
      pq_page = pquery(page.text)

</comment></comment></comment><comment><USER14> Is PyQuery fast? The whole premise of the first point was that BeautifulSoup was too slow, but then he didn't provide a comparison between them.

<comment><USER3> Hey Permit. This is a really good point. Looking at PyQuery's source code [0], it really does nothing in terms of parsing other than calling lxml's fromstring function and then works with the result of that when evaluating queries. So it would probably be a tad slower than pure lxml since it also does other bits (like checking for URLs and then fetching them for you) but from looking at the source code, I'd think that the overhead is minimal.

[0] [https://github.com/dsc/pyquery/blob/master/pyquery/pyquery.p...](https://github.com/dsc/pyquery/blob/master/pyquery/pyquery.py#L179)

<comment><USER15> Do you have any more up-to-date data than a 2008 comparison?

<comment><USER3> Hey, I commented on this a bit further down in the topic: <http://www.crummy.com/2012/1/22/0>

</comment></comment></comment></comment><comment> think-large I <3 you OP. So much right now. I didn't even know i needed these until I read this post and now I know and I'm so happy.

I know matplotlib comes with Python(x,y) but that's a pretty awesome one too.

</comment><comment><USER16> The coolest part of dateutil isn't the parser, it's the recurrence rules and recurrence rule sets. Doing that on your own is extremely error-prone if you have a non-trivial recurrence.

<comment><USER3> Cool, thank you for making me aware of that. I haven't had to use something like that but it's good to know. I'll put it in the blogpost.

EDIT: I added it below the `parse` example. Thanks again!

</comment></comment><comment><USER17> Great list. In particular, fuzzywuzzy and pattern caught my eye in a "how- have-I-not-heard-of-these" kind of way.

</comment><comment><USER18> If you do any non-trivial work with decorators, the `decorator` module is a must: [https://micheles.googlecode.com/hg/decorator/documentation.h...](https://micheles.googlecode.com/hg/decorator/documentation.html). Think of it as @functools.wraps on steroids (though this probably doesn't do it justice). FWIW I think it should be in the standard library.

</comment><comment><USER19> No mention of pandas or nltk?

I'd never heard of pattern before, and while it looks like it's a nice bundle of features, I'm concerned by the fact it references pyWordNet by name even though it hasn't been an independent project since 2006 (<http://osteele.com/projects/pywordnet/>). Has anyone actually used it?

</comment><comment><USER20> Here are some Ruby libraries I've used that are similar to those Python libraries:

pyquery equivalent: Nokogiri (<http://nokogiri.org/>). Lets you select elements with jQuery-like selectors. Uses libxml2 as its parser.

watchdog equivalent: watchr (<https://github.com/mynyml/watchr>). Run code when the filesystem changes.

path.py equivalent: rush (<http://rush.heroku.com/>). Provides a far better API to the filesystem than the standard library.

I also found this equivalent to fuzzywuzzy, but I've never used it: amatch (<http://flori.github.com/amatch/>)

</comment><comment><USER21> Excellent collection! I will add one more.

Python Imaging Library - Today's web is full of images and PIL makes it easy for image manipulation. Although, it's not extremely performance efficient at very large scale.

<comment><USER22> In a similar vein, I recently discovered Wand (<http://dahlia.kr/wand/>) which is a gorgeous API built on top of ImageMagick. It's a lot more limited in functionality than what PIL offers, but for common operations like scaling, cropping, or extracting EXIF data it's much nicer to work with.

</comment></comment><comment><USER23> I have a few scripts that could benefit from PyQuery, didn't know about that one. Also path.py looks like it will save me some time to. Thanks!

<comment><USER24> path.py is a must have, I am amazed how often it's being rediscovered, despite the fact that I try to advertise it whenever it's relevant. But it's quite old at this point, maybe we should be more excited about efforts like this one: <http://www.python.org/dev/peps/pep-0428/>

</comment></comment><comment><USER25> for fuzzy date work i tend to relay on the parsedatetime module:

<http://code.google.com/p/parsedatetime/>

it seems to accept syntax similar to the 'at' command does (and obviates the need for my python C module to do that parsing based on the scheduler parser for 'at'). examples include "1 day ago", "ten hours from now" and the like. very useful.

</comment><comment><USER26> Speaking as someone with approximately 4h of Python experience -- great list. I'll be using the sh lib right away.

<comment><USER27> Unless you're using Windows :(

The docs got my hopes high, but they fail to mention sh isn't supported on Windows -- I saw that by looking at its source on GitHub. I'll take a look at pbs, but this kinda bummed me out.

<comment><USER28> Do yourself a huge favor and develop on Linux. VirtualBox + Ubuntu 12.04 = Free.

<comment><USER27> Sure, I have an Ubuntu box. Thing is, while I would be doing _myself_ a huge favor, the same wouldn't apply to the users of what I'm trying to create. Perhaps the world doesn't need yet another Git GUI, but if I leave out Windows support, the world will need it even less ;)

</comment></comment></comment></comment><comment><USER29> i have reservations about dateutil. it's certainly true that the built-in time and date routines in python are ugly, but dateutil's parsing doesn't try to give a single, consistent interpretation of an input date/time - instead it parses it in chunks, where one chunk can effectively overwrite another. so you can have input that is illogical or inconsistent and dateutil will happily give a single "right" answer instead of flagging an error.

for example, see this bug on so - [http://stackoverflow.com/questions/10575919/strange-date- par...](http://stackoverflow.com/questions/10575919/strange-date-parsing- results-in-python)

in short: it's ok for non-critical cases where you just need "some date" from input. but don't use it if you would rather have an error than an incorrect interpretation.

</comment><comment><USER30> Thanks for sharing. These are a nice set of libraries I never used. I am bookmarking this page.

</comment><comment><USER31> The latest BeautifulSoup uses lxml (version 4+). How does it compare to PyQuery?

</comment><comment><USER32> You have a typo. Search for 'PyQyery', twice.

<comment><USER3> Thank you, fixed.

</comment></comment><comment><USER33> This is quite a cool idea:

>>> path('a') / 'b' / 'c' path('a/b/c')

It would be fun to have that in Ruby!

<comment><USER34> Alright, let me say that I do not like this part of path. Operator overloading is fine, I mean the + is overloaded all to hell. But seeing the division symbol in weird places in a project I took over was distracting first, then irritating, and by the time I figured out what was going on I did not want it despite seeing some benefit.

<comment><USER35> I agree. It reminds me of FancyRoutes which I also found off putting - <http://news.ycombinator.com/item?id=1100248>

_Path.py_ may already do it (docs are limited so I'm not sure) but something like below would be better:

      my $foo = dir('a', 'b', 'c');

This is how Path::Class works in Perl (<https://metacpan.org/module/Path::Class>). This way the path is a filesystem agnostic directory object.

NB. Twisting operator overloading isn't all bad though. For eg. IO:All nearly gets it all right and being a little mad I will sometimes use it :) <https://metacpan.org/module/IO::All>

</comment></comment></comment><comment><USER36> watchdog is pretty buggy on linux at least.

<comment><USER3> Hey, can you talk more on that? I found it to work quite well on my machine and servers, it's also an integral part of another open source library that I'm building. [0]

[0] <http://github.com/doda/imagy></comment></comment>


Space Launch Now – Spaceflight News and Events - happy-go-lucky https://spacelaunchnow.me <comment><AUTHOR> That name is really similar to [https://spaceflightnow.com/](https://spaceflightnow.com/) You might try picking a different name to avoid confusion.

</comment><comment><USER2> Thanks so much for building ;)

Do you have any interest in creating a full space content channel? Heading down to Cape Canaveral or Texas to livestream launches?

<comment><USER3> I do already have a YouTube - however maintaining the application is a full- enough part time gig. I plan on trying to grow the YouTube to the point I can hand it off and revenue share with a creator.

</comment></comment><comment><USER4> May I ask how many man-hours it takes to maintain the data each month? Are you able to automate much of it?

<comment><USER3> There is a small staff that voluntarily help source data, essentially as easy as watching Twitter. Not really possible to automate due to it being journalistic in general.

</comment></comment><comment><USER5> Similar, ad-free app: [https://play.google.com/store/apps/details?id=com.nextspacef...](https://play.google.com/store/apps/details?id=com.nextspaceflight.android.nextspaceflight)

</comment><comment><USER6> I enjoy this app because I get to hear about launches happening all around the world in real time. My only complaint is that it gives live notifications about launches I'm already watching, popping up over the YouTube stream.

</comment><comment><USER7> One thing I've been looking for is a reliable but simple iOS app that will send notifications when a launch is going to happen. Usually with an hour and t-5 minute warning I think would be best.

</comment><comment><USER8> Similar to this: [https://www.howmanypeopleareinspacerightnow.com/](https://www.howmanypeopleareinspacerightnow.com/)

</comment><comment><USER9> It looks to me December 2019 was a very eventful month, launches happening almost every other day.

</comment><comment><USER3> Woah - didn't expect to see this here - appreciate the OP sharing here.

</comment><comment><USER10> Is it possible to get notifications for _just_ spacex launches with this?

<comment><USER11> You should check out the SpaceX subreddit for that. Incredible level of detail. A bit much on the Elon fanboyism, but you can spot that.

</comment></comment><comment><USER12> Why does it say that all the scheduled upcoming launches are "NO-GO"?

<comment><USER3> That essentially means the launch date is subject to move, based on un- confirmed/un-reliable sources, etc - GO for Launch means there is publicly posted information from the launch service provider or some other first party source that confirms the date.

Most launches end up being No-GO until the week or so before.

</comment></comment><comment><USER13> The pagination links have lots of spaces before the page number.

</comment><comment><USER14> It would look better if it used fewer fonts.</comment>


Ask HN: Need some input on a new startup project - <AUTHOR>

Hi there, just launched a new startup project so you can easily share datasets with the world.
DasData in pre-release phase and would be very grateful if anyone is interested to play with it. Register free at http://dasdata.co<p>Thanks! <comment><USER2> [https://news.ycombinator.com/item?id=6455799](https://news.ycombinator.com/item?id=6455799) for ref.

<comment><AUTHOR> thank you man!

</comment></comment><comment><USER3> Nice service :)

<comment><AUTHOR> I appreciate!</comment></comment>


Ask HN: Do you use and love C#? - <AUTHOR> I'm particularly frustrated with it right now, and I would love to know how others who do not absolutely hate the .NET/C# ecosystem feel so I can have a balanced view. <comment><USER2> My experience with .NET was quite frankly awful (and very dated so it might've gotten better). C# on the other hand I always found to be a joy to use, and always equated it to "better Java". Though most of my college courses used C# so I might just be partial to it.

<comment><AUTHOR> This is what I expected ("better java"), and was disappointed when it was basically just more of the same. My college courses were in Java (and of professional work after college), and while I strongly disliked it as well, if I had to pick I would pick Java(or some other JVM language)+JVM over C#+CLR -- C# just didn't deliver on being better than Java in any appreciable way to me.

Could you think of something specific that you find a joy to do in C# that isn't a property of the IDE? "true" async/yield (completely stopped coroutine) is one of the things I was excited about, but quickly annoyed when the interface to `Task`s was so bloated and unintuitive... Why would somone call `.CountinueWith` on a task and NOT expect the result? Maybe I'm just expecting a specific interface out of `ContinueWith` that isn't there

</comment></comment><comment><USER3> I love c#, although that is waning as the language accrues more and more features, it's starting to feel like c++ and the stewards seem to want to move it in every direction possible, from low level high performance to high level application code.

The .net ecosystem has always been a mixed bag. MS did there best to kill off OSS usage with some very "me too" offerings early on. It's always been way too tied to windows. It can often be as enterprisey as the worst java has to offer. Nuget is used for far to many things like loading Visual Studio plugins.

.net core is a clusterfuck and seems to have embraced the move fast and break things motto, this is absolutely not something I or the companies using .net want. Everything is behind the dotnet build tool, even the compiler as an independent component is hidden away from you. The whole stack is moving way to fast for enterprise customers, which was there bread and butter.

<comment><USER4> As an avid fan and (previous) user of C#, the thing I struggled with was everything moving to Nuget...everything.

It felt very much like Nuget was an attempted solution to DLL hell but it was just another version of that same hell itself but at least now on your VS machine you didn't have to find wherever your company stored their libs

</comment></comment><comment><USER5> I don't use C# any more (outside of personal projects), but I absolutely love it as a language.

ASP.NET has always been a mixed bag for me, and I've always felt tied down by my decision to restrict myself to the Windows platform. More often than not, I've found that peoples grievances towards C# or .NET has been more to do with Windows issues, outside of actual issues with the language.

Outside of Windows, the sheer amount of control over the ecosystem has always turned a lot of devs away, despite it being something that .NET devs love. Give me Visual Studio and a decent set of standard Windows tools and I'll be in heaven, whereas someone that is used to choice might not feel the same.

</comment><comment><USER6> Yes but with a few caveats, the language is getting quite big now, I think it would be daunting to start from scratch. The core part of the framework is OK, Forms hasn't kept up with the times and WPF is far too complex. Neither are cross platform so I'm not starting anything new with it.

</comment><comment><USER7> What is the frustration ? Is is related to language or your understanding of the ecosystem and platform ? I absolutely love c#, use it everyday.

</comment><comment><USER8> One part of me thinks that they are not going to treat Linux (my favorite platform) as first class, so this prevents me from investing in C#.

It might not be true but I am not stating a fact, merely expressing my bias against the dotnet platform. I hope they succeed anyway :)

We now use python, ruby, rust, go religiously.

</comment><comment><USER9> Sure, but it's less a case of love and more that the language stays out of my way.

I care about the application and the end user's experience. Language is just a means to that end. C# makes it fairly easy to encode what I'm thinking in a safer way than, e.g., C or C++.

</comment><comment><USER10> C# cannot express generic algorithms on unsafe pointers unlike F#. C# has dounting inconsitencies in generics inference, delegate and event handling, operators and in expression-statements, unlike F#. I use C# but love F#.

</comment><comment><USER11> I love it for the efficiency it provide to the developpers. But, as a Linux user, I won't use it for my personnal projects.

</comment><comment><USER12> I would be curious what issues you have with C#/.net

<comment><AUTHOR> While it's tempting to air my perceived grievances I'm going to decide against it -- I've seen/done enough with C# that I know I will never choose it for a personal project (or work on it for a professional project) ever again, not when options like Go, Haskell, NodeJS + Typescript, Rust, Lisps (clojure/clisp/racket/scheme) and even Java exist. I'm sure if I typed out some long list of things I think are sufficient about the language, someone will take it as an opportunity to "dispell myths" or show me how if I write ~20 lines of boilerplate, it'll be just like the language was ergonomic from the start.

The C# community doesn't need another naysayer, but I did want to find out (maybe this is the wrong place to ask) from people who enjoy using it why they do.

<comment><USER12> My use case is I write a GUI tool in C#/.net, publish it and then hand it over to field service and production and they never bug me again unless they need a feature added.

> and even Java exist.

When you mention this, I think you are trolling. I can see starting a greenfield project in C#/.net. There isn't any reason to start a green field Java project. Just like there is little reason to start a greenfield C++ project.

<comment><AUTHOR> Ahh, that's an interesting use case, I haven't written many GUI tools and the last time I wrote one was in Java (Swing) and it wasn't terrible but wasn't great either, so C# is pretty good/ergonomic for it?

When I say java, I mostly mean a JVM language... yeah I wouldn't really start a new project in Java, though I've heard nice things about JavaFX on the GUI side of things.

<comment><USER12> Implementing GUI apps in C#/.net with visual studio basically just works. C# as a language is 'fine'. And generics seem to be well implemented. Microsoft documentation for simple stuff is reasonably clear.

I think .net timers not so much.</comment></comment></comment></comment></comment>


Why is YouTube auto playing Trump's attack ads on the homepage? - <AUTHOR> https://www.reddit.com/r/youtube/comments/icf295/why_is_youtube_auto_playing_trumps_attack_ads_on/ <comment><USER2> Dark money finishes it's journey in the form of ad buys. That seems to be the point where news orgs lose interest in covering it.</comment>


A New Way of Note Taking That Helps You Remember More and Procrastinate Less - <AUTHOR> https://www.inc.com/minda-zetlin/brain-expert-jim-kwik-note-taking-memory-retention-productivity-overcoming-procrastination.html <comment><USER2> Benjamin Franklin ivented the t notes way before this. I learned it in college.</comment>


Mercurial (hg) with Dropbox - <AUTHOR> http://www.h4ck3r.net/2010/05/11/mercurial-hg-with-dropbox/ <comment><USER2> _You're never going to work directly in the Dropbox directory,_

That is never really backed up, and doesn't make a lot of sense.

I've been doing this for a while - but using Dropbox as my working directory [1]. All he is doing here is using Dropbox as a backup - but such services exist (e.g. BitBucket) and are easier to sync to (one push and you don't have to wait for Dropbox to then sync).

Dropbox is useful for maintaining a consistent working directory between computers. I can hack around at work, go home to finish up and then commit. Brilliant :)

1\. [http://www.errant.me.uk/blog/2010/04/a-nifty-way-to- manage-c...](http://www.errant.me.uk/blog/2010/04/a-nifty-way-to-manage-code/)

<comment><USER3> Can't believe I've never thought of this.

So just to be clear - your working directory is in dropbox and you edit, compile etc in place and no bandwidth, sync , whatever issues?

<comment><USER2> I've never noticed bandwidth/sync issues. Sometimes it takes a couple of minutes to get it's head round things; but not often (Dropbox really is great at "just working").

It also syncs over the local network which speeds things up no end when I work at home.

I don't compile any of these because they are python, php or ruby projects (I only write C at work :P). I do have a Ruby/Redis project that I run from the Dropbox folder for testing - that has never generated any problems to my knowledge.

<comment><USER4> I do the same thing, except with C# instead. I've never had any issues with compilation.

</comment></comment></comment></comment><comment><USER5> This isn't bad, but why wouldn't you also host the live code on Dropbox as well. One of the biggest bonuses I see about Dropbox is the fact that I am, essentially, backed up _all the time_. If his computer blew up before he hit "push", he wouldn't have a copy. If my computer blows up, as long as I hit save, and Dropbox is synced, I'm good to go. If anything, I'd host my repo somewhere offsite that doesn't use Amazon S3, so that I'm extra backed up.

<comment><USER6> Be nice because it hurts Dropbox.

I usually save my working source file every small amount of time (quite often) and every single change would create a history version on Dropbox storage server which I don't need that much. I'm not sure how many versions are kept but it is totally a waste of bandwidth as well as storage. So, pick what & when a folder/file to be synced is one of the most key features that I'm waiting for. Yup, cause it's very useful for live projects in Dropbox folder.

<comment><USER7> Wouldn't the better solution be for Dropbox to cache those trivial updates to limit bandwidth? It's not as if code is the only thing that might have multiple small updates. Seems an optimization they'd want to solve for themselves, rather than relying on user behavior.

And if there's a limit to the version history, isn't the storage concern moot?

</comment></comment></comment><comment><USER8> And if I work offline on different computers now and then, pushing to the Dropbox which then merges the repos in a Dropboxey way? Are .hg directories safe for concurrent modification?

<comment> brown9-2 This is what I wanted to ask too. What if two users both push updates to their individual .hg folders in the Dropbox location?

I think the author might be sacrificing some reliability for the fact that he/she will be the only one working on the "shared" repo.

</comment></comment><comment><USER9> Interesting idea, but why not just use some disk space on a remote server that you pay for anyway? I use a VPS for this and use cron jobs for rolling S3 backups. (And, I need the VPS anyway.) If someone is not a developer, however, using Dropbox sounds like a good idea.

</comment><comment><USER10> Interesting. I do something with SVN (and haven't had a problem so far). I have my Visual Studio project directory located in my Dropbox so the current file saves end up backed up.

I also have a local SVN server binaries and configuration stored in my dropbox, aswell as the svnroot. It means whatever computer I go to, I instantly have a local SVN server running too.

Naturally I don't have 2 computers running svnserv at the same time. It was the lower barrier to entry than SVK and the VS Git Extensions aren't quite there yet.

</comment><comment><USER11> For those who can't afford a premium private repos, it's a good idea. You'd better off not symlinking the repo in Dropbox dir, or you'll keep dropbox very busy when you are editing files.

</comment><comment><USER12> I hate how everything has shadowing, can't read properly.

<comment><USER13> The Readability bookmarklet is pretty handy in this case. Also works great if you need to cleanup a page from Google cache. <http://lab.arc90.com/experiments/readability/>

</comment></comment><comment><USER14> One thing that prevents me switching from SVN at the moment is that I like the fact that every time I commit I'm also performing a backup.

Is there anyway to set up hg to mirror all commits to another remote repository? Would that not be better than hoping your repository never goes over the Dropbox 2GB free limit?

</comment><comment><USER15> Ah thats one of the gems I read today. Awesome, the same can be done for git I guess. I'm going to try it today.

P.S: I have a wicked-cool thought. A nice web interface to browse (shared-)git repos on Dropbox would be cool. I'm going to try writing one this weekend.

<comment><AUTHOR> Yeah, git can work similarly too, though I can't remember the clone flag (--bare I think?). My dumb brain hurts whenever I try to do something moderately complex with git, so I tend to stick to hg.

<comment><USER16> I've been doing this for a while with git as a result of reading this post: [http://www.cimgf.com/2008/06/03/version-control-makes- you-a-...](http://www.cimgf.com/2008/06/03/version-control-makes-you-a-better- programmer/)

works good..

</comment></comment></comment><comment><USER17> If you're one or two devs, save yourself a bunch of hassle and use Kiln (kilnhg.com) for free. Yes, I'm a Kiln dev...

</comment><comment><USER18> why not just use github ?

<comment><USER19> Private repos?

<comment><USER20> Unless I'm misunderstanding, we offer private repos at GitHub. <http://github.com/plans>

<comment><USER19> Not for free though -- and bitbucket only has one private repo for free.

</comment></comment></comment></comment><comment><USER21> I was a happy Dropbox user. Until I read that they only accept engineering resumes from grads of "top tier" universities. Sorry, but I can't entrust my data to a company with such fallacious and just plain stupid hiring practices.</comment>


Islamic AppStore: a Store of Ramadan and Islamic Mobile Apps for All Devices - <AUTHOR> http://arabcrunch.com/2011/07/islamic-app-store-massive-collection-of-ramadan-and-islamic-mobile-apps-for-all-devices.html <comment><USER2> _Wonders why there's no comments yet_

I think app stores like this serve a good goal, first they bring together apps that some companies might have tried removing from the normal app store in fear that someone would get offended, and also it makes a safe haven for the religious (or otherwise belonging to a group.) people that normally have no where to go for those unique apps that remind you when the best time to pray would be etc etc.

I'd like to take a second to point out that I almost didn't post anything in fear of someone being offended, and then realized, well this wouldn't have been put here if someone didn't want to talk about it.

</comment><comment><USER3> Very interesting, however it doesn't seem to have many iPhone apps.</comment>


Ask HN: How to get alpha feedback of a product? - <AUTHOR> I just released the alpha verison of my product called Taskord.
Taskord is free open source social media for tasks! where users can post public and private todos, ask questions, and launch products. Taskord helps you to get things done together with the members of the community.
https://taskord.com
How can I gain alpha users/testers? <comment><AUTHOR> Here is the URL: [https://taskord.com](https://taskord.com)

<comment><USER2> _Taskord helps you stay social with your todolist, so you can get things done together._

I don't know what this means, exactly. Part of what is confusing is some of it seems to be about idividuals ("you") and part of it is about other people ("social" and "together"). If "you" is the collective "you" it's probably better to use "team" or something like that for the first one.

Overall it's not something I am likely to use because it does not solve a problem I am currently having. Good luck.</comment></comment>


A glimpse inside CryptoWall 3.0 - <AUTHOR> http://blog.brillantit.com/?p=15 <comment><USER2> > Deactivating: Shadow Copies Startup repair Windows error recovery

> And stopping: Windows Security Center Service Windows Defender Windows > Update Service Windows Error Reporting Service and BITS

Won't these actions require administrator privileges? How do the program escalate its privilege if it starts by user clicking on a js file?

<comment><USER3> UAC gave Windows Vista a bad reputation for annoying prompts all the time, so in the interest of user experience and convenience, starting with Windows 7 Microsoft intentionally made it much less bulletproof: any Microsoft signed executable can bypass UAC. This includes common system executables which can be caused to load arbitrary dlls, this includes the control panel to turn off UAC entirely. It hasn't and won't be "fixed", it's just how it all works.

[http://www.pretentiousname.com/misc/win7_uac_whitelist2.html](http://www.pretentiousname.com/misc/win7_uac_whitelist2.html)

<comment><USER4> > any Microsoft signed executable can bypass UAC.

If UAC is set to Medium (the default).

If you're serious about security you should set it to High.

<comment><USER5> Unfortunately this results in constant annoying prompts.

I don't know what OSX does, but it seems like I'm prompted to escalate far less often than I would be on a high UAC system. How do the requirements compare?

<comment><USER6> Your usage of OSX must be different than mine, because I get prompted far more than I'd like.

</comment></comment></comment></comment></comment><comment><USER7> I wonder whether various browser vendors are investigating ways to disable loading Wordpress sites given their propensity to being taken over by exploits. The browser vendors collectively disabled/stunted Flash, maybe it's time for Wordpress to go down?

<comment><USER5> Is Wordpress really that insecure, or is it the fact that it's the single most popular blogging platform out there, requires very minimal technical competence to set up, and therefore its users are doing dumb things with the configuration?

<comment><USER6> I'd call it a combination of it being extremely popular (attackers want it), minimal technical competence, PHP, and a history of really bad bugs.

<comment><USER8> WordPress in itself isn't that unsafe; they are pretty diligent in fixing bugs and security issues. _However_ that is not true about plugins, themes, various theme/plugin frameworks developed for WordPress[1], or custom-tailored things[2]. Also, we must factor in the end-users lack of technical competence[3].

Source: I work for a hosting company with lots of WordPress sites.

[1]: In my experience, there's lot of purposely back-doored, or easily exploitable themes and plugins. Also, let's not forget that users and/or developers often get these illegally (without even knowing they did it; it happened a few times).

[2]: The entry barrier for PHP/WordPress is _extremely low_ and many of these developers are beginners and lack even basic understanding of security or _even_ how things work; they base stuff off from an overly-simplified tutorial written by someone only slightly more experienced than they are. There are also inherent language and CMS issues here, but I won't go into those.

[3]: We actually have users who don't to update WordPress or any of the plugins for ages. _AGES_! The other day, I had to argue with a client why having a WordPress version from 2006 (something from the 2.0.x release series) is a bad idea. This is either because the developer stopped supporting and abandoned development of some component their site depends on, or because of legacy custom-tailored code that was a once-off purchase.

<comment><USER9> As a fellow shared hosting engineer I pretty much agree with [1] and [2], and for [3] I also feel your pain, but we don't mess about any more arguing the toss.

We now scan for egregiously out of date WordPress installs and warn customers that their site will be at risk of being disabled if they don't upgrade to the latest version. If after a couple of days we see no action then we pull the site.

If we detect sites serving dodgy links then they're instantly shut down until the customer can prove they've secured the site.

In 99.9% of cases our customers are happy we do this because they're mostly businesses and serving malware damages their brand and reputation. We do get the occasional user who refuses to co-operate, and if they do we serve them notice to take their business elsewhere.

<comment><USER8> > If we detect sites serving dodgy links then they're instantly shut down > until the customer can prove they've secured the site.

We have a similar approach to this actually. The exception being that we clean the malware ourselves, sadly. I tried to say is a bad idea multiple times, but no luck. What makes things worse, we have a few "spoiled" clients that keep getting their websites hacked (there's 3 such WordPress and Joomla development resellers) and they started expecting us to clean their websites. _Sigh_

Also, I tried to argue a few times that we do the scan-and-warn thing, but I got turned down with the counter argument that it would generate more backscatter on our support department than it would be worth it.</comment></comment></comment></comment></comment></comment>


Nokia Map Project Sheds Light on Belarus's Roads - simba-hiiipower http://www.nytimes.com/2013/08/12/technology/nokia-map-project-sheds-light-on-belaruss-roads.html

<comment><AUTHOR> This such a disappointing article it borders on journalistic dishonesty. They make out Belarus to be some Terra Incognita being discovered by some brave Nokia volunteer(?) explorers. It is simply not the case, Nokia's competitors in mapping space, Google and OpenStreetMap already have excellent maps. Just compare maps of Zhodzina, a provincial Belarusian town I chose at random between Bing (Nokia)[1] , Google [2], and OpenStreetMap [3]. Nokia's map is hole, whereas Google and OSM have streets and names associates with them. This pattern follows from Minsk to the provinces.

The details offered by the article may be of some interest, but their veracity simply can not be trusted due to glaring omissions, half truths, and down right falsehoods in the article on the broad level. Shame, shame, shame on NY Times.

[1] [http://binged.it/14p2lTY](http://binged.it/14p2lTY) [2] [https://www.google.com/maps?q=Zhodzina,+Minsk+Province,+Bela...](https://www.google.com/maps?q=Zhodzina,+Minsk+Province,+Belarus&hl=en&ie=UTF8&ll=54.102565,28.331981&spn=0.020885,0.052099&sll=54.103043,28.326359&sspn=0.020885,0.052099&oq=Zho&hnear=%C5%BDodzina,+Minsk+Province,+Belarus&t=m&z=15) [3] [http://www.openstreetmap.org/#map=15/54.1038/28.3231](http://www.openstreetmap.org/#map=15/54.1038/28.3231)

</comment><comment><USER2> Students collecting geo-data to improve the mapping of their local area, and donating it to a single corporate entity in a way that neither competing corporate entities nor free culture projects can use. Hmm.

Also, this puff piece is about Nokia's Here maps, yet when I go to their website (or Bing) Belarus is an empty wasteland. But in Google maps and OSM, there's an incredible amount of detail. So what problem where they attempting to solve? They seem to be claiming the government is to blame for the problem, but that doesn't seem to have held Google back.

edit: and on the mention of OSM, I have noticed that the amount of OSM contributers who are Russian-speaking has exploded recently. Looking at the comparison between OSM and Nokia/Navteq maps in the area I can now understand why.

<comment><USER3> > _... that doesn 't seem to have held Google back._

To the contrary. Google's map data in Belarus also comes from volunteers. For all we know, most of the volunteers who supplied the data for Google Maps and OpenStreetMap may also be working on Nokia Maps now.

The problem they are attempting to solve is that Nokia maps does not have Belarus map data. The fact that Google and OSM have that data is useless to Nokia, and to all GPS device companies that license from Nokia.

The incentive for these volunteers is that in a few years, they'll be able to tell a visitor, "Just enter the address into your GPS," knowing that it'll work. If they supplied the map data only to OSM and withheld it from commercial organizations like Google and Nokia, then they won't be able to accomplish that.

</comment></comment><comment><USER4> Open Street Maps seems to have pretty detailed data of Belarus.

</comment><comment><USER5> >>In countries where geographic borders are in dispute — like India and Pakistan, which are at odds over demarcation of the Kashmir region — Nokia has produced separate digital maps for each country, with each version displaying that country's preferred border.

Does google maps or openstreet map follow similar strategies? I always wondered how issues like these were handled as they are so complicated for a single organization to deal with.

<comment><USER6> Google maps and OSM show both the borders. For an India focused website I am currently working on, I went with jvectormap for this reason ( it is just a map to show cities and towns as points ) - [http://www.ruralindiaonline.org/map/](http://www.ruralindiaonline.org/map/)

</comment></comment><comment><USER7> OpenStreetMap have pretty good data, actually, not just for Belarus, but also all sorts of areas in the ex-USSR where google/apple/nokia can only manage a blank space, or a 1 point per 10km vector road. Saved my bacon repeatedly while rallying across that neck of the woods, and you can very easily load OSM data into a garmin or whatever.

Edit: looks like google have been making lots of headway in this area too - [https://www.google.com/maps/preview#!q=rostov+on+don&data=!4...](https://www.google.com/maps/preview#!q=rostov+on+don&data=!4m10!1m9!4m8!1m3!1d96311!2d-2.5907094!3d51.468489!3m2!1i1680!2i930!4f13.1) \- this time last year, there was only one road in Rostov, according to them, and it was dead straight. Made navigating fun, as I failed to get the OSM tile for the area, as it was an unplanned side-trip.</comment>


Show HN: To Do app (Angular, .NET, MongoDB) - <AUTHOR> http://todo-17.apphb.com

<comment><USER2> How is this To-Do app different from others? Is the source code available somewhere?

Tha landing page is non-descriptive and you could atleast introduce your app here in comments.</comment>


Maze Generation: Wilson's algorithm - <AUTHOR> http://weblog.jamisbuck.org/2011/1/20/maze-generation-wilson-s-algorithm <comment><USER2> Ah, the pleasure to see an interactive widget like the ones demonstrating the algorithm, right clicking and finding out it's not flash.

<comment><USER3> In fact, the mazes are implemented in CoffeeScript:

<https://github.com/jamis/csmazes>

</comment></comment><comment><USER4> How would you express the algorithmic complexity of an algorithm like this, that incorporates a random walk which must hit a particular point before the algorithm can proceed?

In theory, this algorithm could run for any arbitrarily large amount of time. Although the probability of that is low, doesn't it make the worst-case time for this algorithm unacceptable?

<comment><USER5> You're absolutely right. Both this one and Aldous-Broder both have a worst- case where the algorithm never terminates. As for whether the algorithm is "unacceptable", that depends on the application. For games? Yeah, this is probably far from your best option for generating mazes. But for cases where you absolutely must have a uniform spanning tree, your options are limited. Wilson's is much better than Aldous-Broder, but still not perfect. Robin Houston has described a variant that combines both Aldous-Broder and Wilson (doing AB until about 30% of the field is filled, and then switching to Wilson's) which empirically improves the odds quite a bit, but as long as you're doing a blind random walk, you're pretty much never guaranteed to finish.

<comment><USER4> Ok... that's what I thought. Thanks.

By the way, thanks for this series of articles... I've found it most diverting. I guess I'm the sort of person who enjoys "recreational maze generation" :p

</comment></comment></comment><comment><USER6> Here is my version in Clojure: <http://clojure.pastebin.com/dd5ccDkP>

Runs pretty fast, even for large mazes.

</comment><comment><USER7> Empirically, I think I "like" the mazes this generates better than the Prim's algorithm mazes, which from what I've seen tend to look extremely uniform (haven't made one / thousands, just seen a couple). _Far_ more computationally expensive, but unless you're making a _huge_ maze I doubt it matters much.

</comment><comment><USER8> So far I like the recursive subdivision algorithm best. It also has the possibility to stop the recursion for different subdivisions at different times, to generate larger dungeons in your mazes. That's not so easy to do with the other algorithms.</comment>


Source for Predator Object Tracking Algorithm - <AUTHOR> https://github.com/abelsson/TLD

<comment><USER2> One thing that would be interesting for this app would be to setup so that it would send you an email/sms whenever it saw a face it did not recognize, along with the pic.

I'd love to see this as part of a home security system in which it will send you images when it starts seeing people it doesnt recognize.</comment>


Mitch McConnell Moves to Expand Bill Barr's Surveillance Powers - <AUTHOR> https://www.thedailybeast.com/mitch-mcconnell-moves-to-expand-bill-barrs-surveillance-powers <comment><USER2> If I might offer a more neutral and IMO significantly more informative source on Senate FISA amendments;

[https://www.lawfareblog.com/senate-proposes-five- amendments-...](https://www.lawfareblog.com/senate-proposes-five-amendments- fisa-reform)

It's quite extensive and talks about each of five amendments including the various side-by-sides that McConnell has put forward.

Here's the part just about web browsing content vs metadata;

> _Courts have struggled with the line between what constitutes the "content" > of communications and what does not. For example, most courts have ruled > that a first-level web domain, like www.amazon.com, is considered to be > dialing-routing-addressing-signaling, not "content," and as such could be > collected without a warrant. But a domain name that results from a search of > "anarchist cookbook"—a 1971 book that contains instructions for > manufacturing explosives—would be considered "content." The Wyden-Daines > amendment is designed to clarify that all Americans' internet website > browsing and internet search history information would be off-limits without > a warrant._

> _McConnell has authored a side-by-side to this amendment that would seem, on > its face, to similarly limit an application under Section 215 to exclude an > order authorizing or requiring the production of internet website browsing > information or internet search history information. But the side-by-side > would limit such an application only "to the extent such information > constitutes contents of a communication, as defined in section 2510 of title > 18, United States Code." This qualification is key to understanding the > McConnell alternative amendment: Rather than requiring a warrant for > collection of Americans' internet website browsing and internet search > history information, it essentially restates and reifies the existing, > somewhat confusing, state of affairs. In essence, the McConnell amendment > maintains the current status quo regarding treatment of internet website > browsing and internet search history information._

<comment><USER3> Lawfare is hardly neutral - they have been cheerleading the expansion of surveillance powers for the intelligence bureaucracy for years.

<comment><USER2> I doubt you could find any site that could be described as absolutely neutral, but if you read both articles, TFA is nakedly political while Lawfare seems decided _less_ so, wouldn't you agree?

<comment><USER3> No, I view them both as nakedly political.</comment></comment></comment></comment>


Ask HN: How does Twitter do geolocation? - <AUTHOR>

I've recently been traveling and along the way have been tweeting.  I always check out my location as Twitter determines it.  I'm really amazed with how specific their geolocation system gets.  How are they doing geolocation on such a specific level?<p>Also, where are they getting their spatial overlays?  Sometimes I'm tracked to districts in my own town I never even knew the name of.  I'd love to have access to the source they're getting this data from. <comment><USER2> Probably Google Location Services. That's what Mozilla uses: <http://www.mozilla.com/en-US/firefox/geolocation/>

</comment><comment><USER3> Probably from <http://geoapi.com/> which Twitter acquired in 2009.</comment>


Comparing JSON modules for Python - <AUTHOR> http://deron.meranda.us/python/comparing_json_modules/

<comment><USER2> I'm surprised speed benchmarks are left out. There are a number of posts that do some simple tests. I like cjson because it's so fast -- it performed about 50x faster than simplejson for the strings I was decoding.

</comment><comment><USER3> It should be noted that simplejson is now included in the standard library. Reference: <http://bugs.python.org/issue2750></comment>