Secure your browsing using a home VPN - <AUTHOR> http://sriramk.com/blog/2011/08/ddwrt-pptp-vpn.html

<comment><USER2> Shameless plug: my new app and service, Cloak (<https://www.getcloak.com/>), is a zero-hassle VPN. It's currently in beta for OS X.

I'd love feedback from the HN community!

Grab the app, enter your Cloak credentials, and you're done. If Cloak sees you're on on a password-less wireless network, it automatically activates. Cloak's servers are cloud-hosted; the client selects the back-end server that will give the lowest latency. Under the hood, Cloak for OSX is built on industry-standard OpenVPN.

I expect Cloak will exit beta when the iPhone/iPad client is finished, probably in early September. But if you'd like to try it sooner, drop me a line [davepeck at getcloak dot com] and I'll send you a special HN invite code. Cheers!

<comment><USER3> Another shameless plug here... We provide a cheap (39.95$/year) unmetered VPN service at <https://www.privateinternetaccess.com>

We support all devices (OpenVPN, PPTP, IPSEC/L2TP) and have servers in US west/midwest/east, UK, and Switzerland.

<comment><USER4> What is your logging policy?

<comment><USER3> We don't keep any logs.

<comment> St-Clock Hum, I see this on your privacy policy page:

INFO WE COLLECT

From Clients of our Service

Date and time you connect/disconnect to/from the service. Source IP you use to connect to the service. Bandwidth usage E-mail provided by Paypal. Paypal payment data.

And then this:

DISCLOSURE

If under subpoena, PrivateInternetAccess.com may release data in order to comply with legal obligations or in order to enforce the PrivateInternetAccess.com Terms of Service and/or other agreements.

PrivateInternetAccess.com may release data in order to protect the rights, property and/or safety of PrivateInternetAccess.com, its constituents, and/or other visitors and clients.

<comment><USER3> That's correct, we log connections TO our VPN but log absolutely nothing in terms of traffic sent THROUGH the VPN. So whatever you do while connected is not logged and therefore un-subpoena-able.

</comment></comment></comment></comment></comment></comment><comment><USER5> From TFA:

> Here's why I picked PPTP and I believe using it with very long > passwords/passphrases is acceptable.

Bear in mind that the author is looking to use this in open hotspots such as coffee shops etc. I would not advise that people implement this.

As the author points out, there are a number of vulnerabilities in PPTP, the most serious of which is that the initiation protocol is susceptible to an offline brute force attack using tools like asleap[1].

To be clear, the attacker does not need a rogue access point, nor association with an access point for this to work. They can just passively sniff away, then at some point later go through the pcaps, crack it offline and do what they want. There's an episode of Hak5[2] covering this as well as this useful straight to the point video of asleep and THC pptp-bruter[3]

[1] <http://www.willhackforsushi.com/Asleap.html> [2] <http://revision3.com/hak5/asleap> [3] [http://blip.tv/g0tmi1k/cracking-vpns- asleap-and-thc-pptp-bru...](http://blip.tv/g0tmi1k/cracking-vpns-asleap-and- thc-pptp-bruter-3375795)

The solution is to use L2TP and IPSec if you can and aren't jailbreaking, or to use a TLS VPN if you have jailbroken or don't have iDevices.

<comment><USER6> Author here. Like I said in the post, the third constraint I had was something that would work with dd wrt - and that doesn't support l2tp. Thanks for the bit on the offline cracking though.

<comment><USER5> No worries, I understand your reasons for choosing PPTP, but I thought it important to highlight the consequences.

It is clear you have put a lot of effort into it though and it's more well written than a lot of guides I've seen, hence my interest in the first place!

</comment></comment></comment><comment><USER7> I've got 3 low quality VPSs, priced from $0.99 to $2.50 per month, that I picked up on a whim over the years from deals posted to <http://www.lowendbox.com> (no affiliation). On one server, I have squid running and bound to localhost. On my machine, I have autossh set up to maintain a constant port-forwarding connection established (via port 443 for maximum firewall/filter accessibility) with the squid server. I have my local web browsers proxying over that connection.

So I'm pretty safe from snooping by my employer, home ISP, or whatever 133t hackers are sniffing traffic at McDonald's when I'm browsing and sipping a coffee.

It's not the most elegant solution, but it does the trick. I supposed some day I'll mess around with OpenVPN (which I have deployed before, and do really like), but only when I'm bored or otherwise have nothing else better to do with my time.

<comment><USER3> Just fyi, all your non-HTTP Flash traffic (Flash sockets) is probably not being proxied at all. Even if you have a SOCKS proxy set in the browser.

The easiest way around this is VPN, but you can also do crazier stuff like use ProxyCap (for windows or OSX) or iptables redirect/transocks setup in Linux: [http://coderrr.wordpress.com/2009/07/29/how-to-force- flash-o...](http://coderrr.wordpress.com/2009/07/29/how-to-force-flash-or-any- program-to-use-a-socks-proxy-using-transocks-and-iptables-in-linux/)

<comment><USER8> Linux has tsocks and proxychains which override the connect() method with a LD_PRELOAD library and force traffic to go through a socks proxy. Easier than iptables since at least proxychains can be installed with apt-get on Ubuntu(don't know about other OSes).

<comment><USER3> Only problem with the LD_PRELOAD method is that it doesn't work for all (many) apps. I believe one case it breaks for is non blocking connect()s, not sure if there are others. Although the iptables setup is a lot more complicated it pretty much works for everything.

<comment><USER9> An example I have found is rtorrent. There is also <https://github.com/apenwarr/sshuttle>, which looks promising, but I haven't used it. My vps provider doesn't provide tap for openvpn to work.

</comment></comment></comment></comment></comment><comment><USER10> For the other 50% of smartphone users that are on Android (as per this <http://news.ycombinator.com/item?id=2855717> ): you can use OpenVPN and certificate authentication, Cyanogenmod supports it _natively_. Tunnelblick is the OSX client of choice for OpenVPN.

</comment><comment><USER11> For the same use-case, a very quick howto on a disposable EC2 proxy for coffee shop browsing...

[http://flatterline.com/index.php/2009/04/23/disposable- proxy...](http://flatterline.com/index.php/2009/04/23/disposable-proxy-for- secure-coffee-shop-browsing/)

<comment><USER12> I use this, but it can be a little annoying now that some sites like StackExchange block EC2 netblocks (I guess because they have lots of juicy content ripe for scraping). Yelp is another one that comes to mind that does this as well.

<comment><USER13> What??? For real? Fuck. I was going to set up an EC2 vpn endpoint this very day.

<comment><USER12> Yep. Some more discussion here:

<http://news.ycombinator.com/item?id=2441535>

It sucks, but I can see the reason why they just blanket block EC2 rather than trying to find any more nuanced way to identify scrapers.

If I were running scraping bots on EC2, rate-limiting me isn't a real threat since the instant I get throttled for misuse I can just destroy the instance, start up a brand new one on a different IP address, and continue as I was before.

</comment></comment></comment></comment><comment><USER14> If you'd rather not have to mess around with stuff, and have $5 to spare, we can help: <https://wonderproxy.com/signup/vpn>

It was a huge pain in the neck to get native support on Mac & various windows flavours. But our normal clients needed a good solution for testing flash & silverlight apps.

<comment><USER15> I was contemplating setting up a VPS for this, but a turn key solution at $5\month would be much better.

Can you comment on how you can offer 100GB\month VPN for $5 but the basic proxy plan is 2GB\month for $20? What use case does the proxy plan meet over the VPN other than more server locations?

<comment><USER14> The VPN locations are places with easy access to cheap bandwidth, the Proxy plan includes locations like South Africa, Columbia, and New Zealand where the cost of maintaining a presence is significantly higher. We're considering including the VPN locations (London, San Antonio) in the Proxy plan free of charge.

</comment></comment></comment><comment><USER6> And Linode has an outage in their datacenter which has brought down my site. Nice timing, Linode.

</comment><comment><USER16> If you want a fast, free, easy-to-use VPN, just get AnchorFree's HotSpot Shield. Most downloaded free VPN in the world. AnchorFree's CEO was also selected as one of Inc's 30 Under 30 this year.

DISCLAIMER: AnchorFree's CEO is one of my dearest of friends for nearly 12 years, but that's not why I'm promoting HotSpot Shield here. It just really is that good, and I use it all the time on public hotspots.

</comment><comment><USER17> I've been traveling abroad and set up a VPN on a Linode box for some basic security and to bounce through the US for things like Netflix and Hulu. If nothing else this post's comments have unveiled some good turnkey VPN solutions, so thanks!

</comment><comment><USER18> So even after reading the article, I'm unsure if its an appropriate set up for me. I just want a way to access my NAS at home from my iphone (mainly just web traffic). I'm not worried about securing my traffic from a public hotspot because I almost never use them. I just want to be able to connect over 3G or from my work wifi. Is PPTP still a high risk with a long passphrase? Or is the risk related to connecting to the VPN from an unsecured network? Also I should mention that I have a DD-WRT capable router, and I don't want to have another machine running just for VPN purposes. Thanks

</comment><comment><USER19> Also worth nothing: the typical 4MB WRT54G router doesn't have enough room for OpenVPN if you use the stock OpenWRT images.

I've got a stripped down .config and image that does include OpenVPN, if anyone's interested.

</comment><comment><USER20> I have noticed that I'm sometimes unable to connect to my VPN from, for example, the wireless network of a hotel I'm staying at. Has anyone else run into this issue and is there a straightforward workaround?

</comment><comment><USER21> the simplest and pretty easy to do is ssh tunnel to the trusty home machine running vncserver, and then vncviewer into it via the tunnel. vpn is too heavy for home. and the home machine does not even need to run a full blown xserver, i.e. no video card needed.

</comment><comment><USER22> So VPN by someone who knows nothing about security. Fail.

<comment><USER23> Why don't you update your comment with a detailed breakdown of why it's so fail?

<comment><USER22> Anybody still implementing PPTP deserves the insecurities that are inherent. A much more elegant, and secure, solution would just be tunneling via SSH dynamically. It's easy to do, requires no setup and if done right (i.e. certs for auth) it is a more maintainable solution. I generally use IPsec - because, it is still by and far, the most secure of the VPN solutions with platform interop today. Setting up and maintaining an IPsec tunnel for the masses is not trivial unfortunately - which is why I'd recommend SSH or potentially OpenVPN (which has nice IPv6 support).

The author of the article on how to implement references the below...

<http://www.schneier.com/pptp-faq.html>

...I think there's plenty there to disuade you. Would you build your webapp on a framework that stores user credentials in plaintext as a feature? No. Enough said.

Why go through the trouble of implementing something that's known broken? _sigh_

</comment></comment></comment><comment><USER24> Contrary to the article's assertion, OpenVPN is _not_ natively supported by iOS.

<comment><USER25> "If you're ok with not being able to use this from non-jailbroken iOS devices, you should use OpenVPN instead of PPTP as I do so below"

I had to read it three times, but you are both in agreement.

<comment><USER24> The post was edited after I posted. Guess I have to start Pinboard-ing everything I respond to now.

<comment><USER6> I couldn't let that awful phrasing stand :)</comment></comment></comment></comment>


New 'hog-nosed rat' discovered in Indonesia - <AUTHOR> http://www.bbc.co.uk/news/world-asia-34451265 <comment><USER2> "The rats have since been preserved and are lodged in a museum in Indonesia."

Yes, this seems like the right thing to do upon discovering a new species.</comment>


Hyper-growth in SaaS - <AUTHOR> http://christophjanz.blogspot.com/2015/03/hyper-growth-in-saas.html

<comment><USER2> This was an awesome post - really enjoyed it!</comment>


Ask HN: A search engine for learning resources? - <AUTHOR> Hey all, I'm thinking about scrapping together a human curated cross between product hunt and a search engine to streamline learning and understand what resources are optimal to follow. What do you all think? <comment><USER2> Hi, you can create own search system on bubblehunt.com. Add any resources about learning in your profile and create own best search engine for this topic. You can become independent information provider like miniGoogle.</comment>


From Dec 1, 2019 Chinese govt will have full access to every server in China - <AUTHOR> https://www.chinalawblog.com/2019/10/chinas-new-cybersecurity-system-there-is-no-place-to-hide.html <comment><USER2> >In cases where the Ministry of Public Security determines there is an Internet security issue, it has the right to perform a remote access inspection. the scope of which is set out in Article 10. Prior notice of remote access is required.

>...it is not clear whether notice goes only to the cloud provided or to both the cloud provider and the customer of the cloud provider

>...but my guess is that the cloud customer will never know unless its cloud provider tells them.

So much for setting a datacenter in China (Apple and others) and expect that your customer/business/proprietary data is not being shared with other Chinese State Owned Entities (SOEs) (i.e. your competitors)

<comment><USER3> Wonder the feasibility of having ghost servers which just act as fake servers to your external origin servers.

<comment><USER2> Well with the tightening of the Great Firewall (GFW) the chance of getting data out without their knowledge is becoming slimmer and slimmer

</comment></comment></comment><comment><USER4> How will any foreign owned entity realistically do business?</comment>


Belgium ends 19th-Century telegram service - <AUTHOR> http://www.bbc.co.uk/news/world-europe-42359914 <comment><USER3> The German post offers fake telegrams: They can be sent via telephone or internet (incl. CSV upload for mass spendings) and will be delivered by regular mail. If entered till 3am it will be delivered on the same day. 160 characters for 12.90€. [https://www.deutschepost.de/de/t/telegramm.html](https://www.deutschepost.de/de/t/telegramm.html)

<comment><USER4> It's the same in Italy [1]. Until just a few years ago it was still somewhat used in special circumstances (wedding, death of friend/relative) to send a formal message when you could not be present in person.

[1] [https://www.poste.it/gamma/telegrammi.html](https://www.poste.it/gamma/telegrammi.html)

<comment><USER5> We used to have that in Sweden as well, but it seems like it was discontinued a few years ago.

</comment></comment></comment><comment><USER6> The service was offered on things like trains and cruise-ships and of all the miraculous things, you could send actual money through it as well. Sounds like a pretty magical service for the 1930s. There were even photos and drawings over it: [https://en.wikipedia.org/wiki/Wirephoto](https://en.wikipedia.org/wiki/Wirephoto)

<comment><USER7> Hence the idiom "to wire [money]".

<comment><USER8> In general terms, transferring funds via ACH is over a wire (or perhaps more specifically, a fiber optic filament) as well.

</comment></comment></comment><comment><USER9> When I was young, I sometimes got birthday telegrams from my uncle. I loved them - only a single sentence in all caps.

So I am feeling a little bit nostalgic right now. But, of course, I would not want that world back - where you had to pay a small fortune to "instantly" deliver a short message.

And "instantly" meant within a few hours, not milliseconds like today.

<comment><USER10> I remember getting one -1- telegram. (Received for my Confirmation from a distant relative in the US. Must have been May 1993.)

This was such an event in the cottage I lived in at the time that the local postmaster dropped by on his way home from work to deliver it, rather than just toss it in the mail for delivery next day (as was SoP, if I remember correctly - I seem to remember telegrams were telexed from whatever entity served as the national gateway to the nearmost post office, where it was printed and put in local mail; this procedure was written out in the phone books, and provided for a glimpse into an exotic world for a kid stuck in West Podunk, Norway in the eighties...)

Still get a chuckle out of it, as my relative (who had emigrated from Norway as a kid, and thus spoke Norwegian) had apparently gamed the system a bit - <USER2> order to pay for fewer words, he'd strung multiple words together like 'CONGRATULATIONSON THEBIGDAYMAY GOD BLESSALL'

(Wonder how that worked out - presumably foreign language text was billed at a higher rate?)

<comment><USER11> > in order to pay for fewer words, he'd strung multiple words together

I remember reading [1] that nineteenth century telegraph users could buy popular code books that allowed them to save money by substituting single words for common phrases. Obviously both parties needed to use the same code book, and unsurprisingly the telegraph companies tried to ban the practice.

[1] I think this was in _The Victorian Internet_ by Tom Standage. A very interesting read btw.

<comment><USER12> The same technique is still used by stenotypists.

Some years ago, I read a newspaper story about a blind stenotypist who records the proceedings at the Saxonian parliament. He said that he uses a set of a few thousand abbreviations specifically designed for parliament speeches. Then someone takes his papertrail and expands the abbreviations into the actual proceedings.

<comment><USER13> The practice was also used in shorthand. In addition to being a phonetic transcription, the secretary would also utilize standard abbreviations and contractions to keep up with the spoken word. The documents would then be sent to a secretarial pool of typists who would transcribe it to long-hand. Schools would train secretaries to write their shorthand identically, so that it was legible to other secretaries and the abbreviations were universally understood.

</comment></comment></comment></comment></comment><comment><USER14> I remember the anecdote about ending the use of a Morse code only distress channel by the Coast Guard. On the last day, the land based radio operators spent the time sending each other Shakespeare quotes and similar until they received an actual emergency distress call! Seems a small cargo ship had lost a lot of systems - including rudder control - and were forced to use the Morse channel to ask for help.

</comment><comment><USER15> I am always sad to hear that another part of history dies out. I'm not sure why, since I've never in my life used a telegram, but it's somehow representative of an age that is no more. Including anything from steam engines, trains, intercontinental ship travel and a whole range of things that have long since been improved on, but which I'm still sad to see go.

<comment><USER16> Intercontinental ship travel has not entirely died out. Well, for pure transportation it largely has, but for tourism or entertainment you can frequently find intercontinental legs on cruise ship voyages. These are not routinely offered and usually occur when the ship itself needs to sail between continents.

Because these voyages have multiple consecutive days at sea with no land activities, they are usually considered less desirable than cruises that dock every day in port. As much as there is to do (relatively) on big ships, most passengers get bored. Those of us who enjoy solace and contemplation can enjoy the time however.

On modern cruise ships, all of these journeys are fast and uneventful, what with weather radar and forecast, fast travel speeds, and stabilizers that nearly eliminate rolling motion in calm waters. Plus satellite Internet and often cellular phone connections. So they rarely ever have the drama that we imagine from a century ago.

<comment><USER17> You can also book rooms on some container ships (yes, container ship cruises are a thing!).

The downsides of such a cruise are readily apparent: Lack of amenities during the cruise, few to no people to talk to/with (unless you know the common language of the crew - which could be almost anything - but even then, they have work to do!), etc.

You are also limited on where you can travel from and to, and of course when you arrive at your destination, you probably have to get creative to gain transport from there to wherever.

There's also the fact that you won't usually have a set and known time for when to board the ship for leaving (it might be in port and scheduled to leave at a certain time, but might be held up for one reason or another outside of the port, or delayed leaving for some reason or another), or when it will arrive at its destination (like any ship, I suppose - but container ships are working vessels, and they may be held out of port for certain admin purposes, or some other law-related reason, or they may have to dock elsewhere for repairs - or who knows what else).

That all said, from what I have investigated, if you want a little (or a potentially a lot) of adventure, and don't mind the whole uncertainness of the experience, container ship cruising can be a very inexpensive (compared to a regular cruise ship). I don't think it compares to flying (but I never delved that deep into the pricing), but it certainly could be worth it from an "about my vacation" story perspective.

<comment><USER15> The last time I had a look at this the cost was $130 per day. This seemed a tad on the expensive side for a trip that could take a few weeks.

</comment></comment></comment></comment><comment><USER18> This image shows the worldwide wiring map in 1891 for those interested:

[https://upload.wikimedia.org/wikipedia/commons/9/9a/1891_Tel...](https://upload.wikimedia.org/wikipedia/commons/9/9a/1891_Telegraph_Lines.jpg)

<comment><USER19> And many of the same paths are today used for fiber optics.

<comment><USER18> That's so true! I hadn't thought of that, but you're right.

</comment></comment></comment><comment><USER20> I rememeber us getting a telegram from an aunt who lived abroad in the 70s. The local post office phoned us with the message and the written telegram itself was delivered by the postman, next day. I kept it for awhile.

</comment><comment><USER21> From the article:

 _Just 10 businesses and a handful of individual customers have kept the Belgian system going until now. It has been chiefly used by bailiffs, who had need of a system which provided legal guarantees of dispatch and receipt._

Would email programs supporting PGP fill this role?

</comment><comment><USER22> We all use www.telegram.org nowadays.</comment>


Ask HN: Any cloud computing instances cheaper than AWS EC2 Micro? - <AUTHOR>

Hello guys, I am looking for a cheaper solution than AWS EC2 Micro instances. Basically my site will not get much traffic and won't need much ram (~256 mb will be fine), I also need very small disk space. What do you recommend? <comment><USER2> If you don't have server sided scripts you wouldn't need a virtual root server in the first place, but rather host your pages at github, wordpress or the likes. If you do have active components you could go to Google App Engine, Heroku or PythonAnywhere. If you do need a virtual root server, it's hard to beat a t1.micro, especially if you don't need it 24/7.

</comment><comment><USER3> You can try appfog(paas).. and if you are coding on nodejs, you can try nodejitsu(paas).

</comment><comment><USER4> google app engine - free.

<comment><AUTHOR> App Engine is definitely not free. One of my very small apps used to exceed front-end quota and caused me a bill.</comment></comment>


This Is What Python Beginners Have to Deal With - <AUTHOR> http://pythonforengineers.com/this-is-what-python-beginners-have-to-deal-with/ <comment><USER2> > Now, compare that to what happens when I search for the same in Python. I > get this:

> > [https://docs.python.org/2/library/re.html](https://docs.python.org/2/library/re.html)

I clicked and I saw the kind of technical documentation I love to read: precise, with examples. There is an introduction to Python's flavor of regular expressions, which highlights the potential problems (backslashes). This is not targetted at novice _programmers_ , though.

> That's the generic re library page. I have to scroll down to see which > function I need.

Did you notice the table of contents on the left? You might want to try Ctrl+F. But you might have to scroll down too, I don't deny it. You have to read, too.

> And even then, all I get is a hundred words of generic description. No > examples.

Ctrl+F "example" highlights the table of contents. There is an example for "findall".

> I then have to Google Python regex findall examples, and go through a dozen > blogs written in 2008.

[http://lmgtfy.com/?q=regex+findall+example#](http://lmgtfy.com/?q=regex+findall+example#)

Not really, there are plenty of recent posts (stackoverflow, etc). And having articles from 2008 would not even imply that they are incorrect. In order to see how old your post is, I have to look at the source, but even though it is quite recent, it contains factual errors.

<comment><USER3> I think this is the exact type of comments that the author comes out against.

You aren't putting yourself in the shoes of a beginner.

Let's say I'm a beginner, and I want to just experiment with the language a bit (I've already read some tutorials or am not interested) - since I'm just trying to get the feel of the language and ecosystem, I'm not trying to get a deep understanding of the language and tools - I want to match a regex.

In the C++ and PHP examples he provided, the basic example is easily found - it's basically two actions - search in google and scroll to the example.

What you are suggesting takes several more steps - ctrl-f for a text that you might get wrong (you've searched for sample instead of example for instance), iterate until you get the right place, etc...

And that's assuming you even realize that you should run ctrl-f, why should you?

The main issue though is that if I'm a beginner I'm doing this kind of searches several times a minute - to have to stop and realize what to look for every time is extremely frustrating.

I've worked with Ruby,Python, Java and Javascript - python's (and django's) documentation is extremely verbose and rarely straightforward compared to the other ones.

<comment><USER2> > You aren't putting yourself in the shoes of a beginner.

How do you know if I am putting myself in the shoes of a beginner or not? And what kind of beginners are we talking about? Are they comfortable with a browser? Do they already know how to program something? You can't expect a single document to be relevant for all kind of people. The reference documentation has a clear target audience, which apparently conflicts with what you expect from it.

> And that's assuming you even realize that you should run ctrl-f, why should > you?

> (you've searched for sample instead of example for instance)

Okay, so you are talking about absolute beginners, with no experience whatsoever with programming. And the C++ and PHP pages are somewhat useful for them?

> I've already read some tutorials or am not interested

> Let's say I'm a beginner [...] I want to match a regex.

Now, you are saying that you already have experience with programming? Google "python tutorial regex" gives plenty of results, the first of them being currently "Regular Expressions HOWTO" ([https://docs.python.org/2/howto/regex.html](https://docs.python.org/2/howto/regex.html)).

Absolute beginners need a teacher, a mentor, or a step-by-step book, or a very gentle tutorial, or an online REPL. That's fine. You can't please everybody in the reference documentation.

<comment><USER4> _And what kind of beginners are we talking about? Are they comfortable with a browser? Do they already know how to program something?_

I'll add to that, "Are they even _computer-literate_ in any way?" I'd worked at a college a few years ago, and it is sometimes shocking how little the newcomers know about just using and troubleshooting basic problems with computers, nevermind learning how to write programs for one... and IMHO it's only gotten worse. Knowledge of things like basic filesystem concepts ("What's a directory? How do paths work?") in particular seem to be disappearing.

</comment></comment></comment></comment><comment><USER5> You know what puts off beginners?

Articles plagued with stuff like "this is horrible", "this is shit" and so on and son.

I see too often articles that just try to denigrate other technologies, saying that they suck, they are horrible, they are shit, they should be considered harmful, etc, just to try to make the author sound smart-assed.

The truth is that most stuff is the way it is for a reason. Maybe the situation has evolved and the reason is no longer valid, or maybe the people that built the thing had different goals than you. But understanding why it was valid for the other person and why it is maybe not valid for your use-case in an empathetic way will only make you a better engineer. We are all smart people trying our best to achieve something using technology. We are on the same boat. Let's stop insulting each-other.

If I was a novice and saw and article like this, I don't think I would want to hang out with such an overreacting crowd. Python is a libre project: instead of insulting other people's work to sell your book, try improving it. I can understand why proprietary software sometimes sucks -- it is disempowering -- but with libre software you are just kindly given a gift. If you don't like it, improve it and give back to the community, don't denigrate.

<comment><USER6> Many people have been and are contributing. But when there isn't critical mass or appreciation for changes they move on to other things. So the python community is not only left with the flaws but can't attract people to fix them.

</comment></comment><comment><USER7> I pick Python up maybe 3 or 4 times a year - usually to automate something or to explore some algorithm in a notebook - so I'm pretty much an eternal beginner (in respect of the language, rather than the development process). This means I'm hitting up the docs pretty much constantly and I do find them mildly frustrating. It's an organisation thing. As someone stated up thread, the docs are really comprehensive, which means they resist skimming. There's no distinction between - say - a language _guide_ and a language _reference_. As someone coming at Python from a comprehensive background in other PLs I just want to see a damn example, I don't want or need to read a treatise on operator precedence or regex syntax. Fortunately this problem is easily solved by adding the word 'example' as a search term - and then going back to the super comprehensive docs to find out what I just did wrong with my regex syntax!

For a _complete_ beginner, the documentation for most any language is just line noise at first, until you gain the appropriate context (for which there are a pile of easily googleable resources) and thereafter the more comprehensive the better because you don't know _what you don 't know_.

Maybe if there weren't eleventy bajillion easily accessible 'how to X in Python' articles or SO wasn't a thing there would be some pressing need for differently organised docs, but there are and it is, so y'know, meh.

Edit: over opinionated autocorrect.

</comment><comment><USER8> I think the Python docs are fine. I use them all the time. (The re docs in particular.)

It's not beginner-focused, but it'd suck if the current docs were replaced with docs for beginners.

The re page has over 7,000 words. I need those 7k words. They help me get things done.

Perhaps a solution might be something like a [https://simple.docs.python.org](https://simple.docs.python.org) type site: A simplified version, parallel to the existing documentation, consisting mostly of examples. I'd use it.

<comment><USER9> The issue with the Python docs is that you have to ACTUALLY READ IT. Some language have documentation that you can just sort of skim and pick out the bits you think you need.

The Python documentation, and the Django documentation have additional information in the text. Describing how a module or function works, what you can expect and how you should use it. Important information is often "hidden" in the text and if you're the sort of person that doesn't actually read the documentation you're going to get it wrong.

The Python docs are absolutely fine, because actually tries to teach you how stuff works and why it is the way it is.

<comment><USER10> I think the docs are often too verbose. Many pages start with a long winded intro that misses the main point. Some pages never get to the point. Docstrings are sometimes several pages long.

Example of not getting to the point: Extending/Embedding.

There is no fundamental difference. In both cases you're just calling a bunch of functions from libpython.a.

</comment></comment></comment><comment><USER11> I've never encountered the "X requires unstated requirement Y, which fails to build because of obscure compiler error" issue, I would have liked if the author named X and Y.

However the documentation is totally a fair point. The "re" module is an excellent example - it starts off with a handful of relatively verbose introductory paragraphs and then dives straight into an extremely comprehensive list of special characters. The good news is that all the information you need is there _somewhere_ but sadly it's hard to get there. Another commenter pointed out the Table of Contents, which for some reason my eyes just completely ignore by default. Maybe the font colour doesn't contrast with the background (aquamarine-ish on blue) or the line-spacing makes it tough. Certainly the numbered prefixes do not help, or the fact that the headings aren't actually particularly useful (sorry if you're responsible for the documentation, I like it overall I'm just searching for reasons why it's hard to read the side-menu).

I wish I had the time to help out, and the UX know-how to contribute to making it better. I'd feel awful if I pitched in and made things _worse_ :(

<comment><USER12> I tried installing OpenCV2. I needed to get Pip, then I tried to install OpenCV2, but it required a whole bunch of other stuff, which I had to compile, which failed.

Was a very frustrating afternoon.

<comment><USER13> If on Windows, google Christoph Gohlke's (spelling?) big list of pre-compiled Python modules.

<comment><USER12> How do I know to do that? Most projects just say "Install OpenCV" or, if you're lucky, give you the step-by-step for doing a pip install. Assuming you already have pip.

<comment><USER14> I remember the first time I sat down to really do something with python, step 1 was setup a virtual environment...

I never got to step 2 that week

</comment></comment></comment></comment></comment><comment><USER15> The typical narrative of the mediocre duct tape engineer.

"I want to put something together fast by trial and error, without having to go through actual thinking or understanding. I don't care about details, just give me an example that I can copy, paste and duct-tape with another piece of code, so I can ship my product and charge some idiot a fortune for some spaghetti with meatballs code."

Documentation does not necessarily have to follow the format of a tutorial and can vary in levels of detail. Tutorials on this topic do exist. There, LMGTFY: [https://docs.python.org/3/howto/regex.html](https://docs.python.org/3/howto/regex.html)

How did I find it? Google: regular expression site:python.org

<comment><USER16> Is it really wrong to learn something new with examples and without having to read a full book before typing the first few lines of code? I prefer to learn by trial and error.

<comment><USER15> It is a valid option to explore by trial and error. But once you have obtained something that seems to work, if you can't explain why it is working, it is a bad moment to stop.

If you construct an entire system by trial and error without an understanding of how and why it's working, you are exposing yourself and your customer to serious risks.

Even if things do not crash, if your code gets audited (e.g: when another person joins your team), you will have a bad time explaining your choices. As a professional engineer, it is expected of you to know what you are doing, and fudging code is the easiest way of getting sacked to the sound of a trumpet.

<comment><USER16> Right but the article was about a beginner learning a new language, I don't think a beginner is expected to have audit-proof code for quite some time.

</comment></comment></comment></comment><comment><USER17> I've been learning Python as it seems to be the preferred language in education at the moment in UK and I'm expecting my kids to be using it. I work in arts & craft but have a strong hobby background in computers.

I started with a Coursera course [https://www.coursera.org/learn/python](https://www.coursera.org/learn/python) which was pretty easy (you can run the videos at 1.5 or 2x).

That course is based on a free gratis book by the lecturer (there a chapter on regex that covers re; with examples too).

I took a good few hours trying different IDEs and settled on PyCharm community edition, it has very easy doc lookup. ([http://alicious.com/editors-ides- python-code/](http://alicious.com/editors-ides-python-code/) is a brief summary of most of my IDE-for-python search.)

Currently I'm doing the Web Data course (using Beautiful Soup) and the only problem but is that _by choice_ I'm using Python 3 when the course is written for 2. It covers regex pretty well.

</comment><comment><USER18> I'm kinda surprised about the community argument. I've generally felt that people are nice enough, if a bit conservative. Though I guess this is the community that produced Python 3k, so something must have gone wrong

Especially to hear that someone went from Python to Ruby... I still feel raw for getting yelled at in IRC for trying to understand how blocks differ from anon. functions and about not doing things the "Ruby way"

<comment><USER19> I also found that part odd.

At least in stackoverflow, compared to R community, I found python community to be more helpful and more polite.

<comment><USER20> Agreed. The community is the part I like most about Python. I've largely stopped attending talks and such years ago, but still attend events every now and again (mostly the annual local conference) purely because I find the community friendly and interesting.

</comment></comment></comment><comment><USER21> I feel a big problem with Python these days is that it's not one Python community. It really is becoming more and more fractured and efforts go into weird places.

For instance we have the core developers which do Python 3, then we have some separate volunteers that work on the packaging infrastructure which is largely based on cobbling systems on top of the horribly broken distutils/setuptools setup. For documentation people don't want to touch it that much because which version should it go in? Python 2 which everybody is using or Python 3?

This schism in the community is causing more and more issues as time goes on.

</comment><comment><USER22> An odd article; the site is adware for a book, if the documentation was better he or she wouldn't have any sales!

</comment><comment><USER23> 2 out of the 3 negative points talks about how horrible the documentation is and yet I almost every day see blog posts, tutorials, books etc being posted online. If it really is so horrible why are these 'authors' apparently, not contributing to those horrible docs to make them better?

The gripe about packaging however is justified in a small manner, although we are way better than we used to be.

<comment><USER24> > If it really is so horrible why are these 'authors' apparently, not > contributing to those horrible docs to make them better?

Incentives (building their personal brand) is much more tilted towards blog posts, tutorials, books etc. than improving the standard documentation.

<comment><USER25> But if you're complaining primarily to further your personal brand (not because you genuinely want to see a problem addressed), why should your complaints even be taken with merit?

</comment></comment></comment><comment><USER26> Once there was a discussion on HN about a specific command line tool, how easy command lines were and how even an idiot should be able to install and use it. I decided to test that theory by attempting to install it on a fresh copy of Windows, and documenting every step. I did get it installed, but did not succeed in using the command line tool for the desired application. There was also a number of difficult steps that nontechnical users would not know anything about and get frustrated by or turned away from. There were serious mistakes in the instructions. The biggest issue was dependencies which had further dependencies, all of which had their own difficult installation instructions. At least it didn't require compiling anything!

[https://news.ycombinator.com/item?id=11453086](https://news.ycombinator.com/item?id=11453086)

</comment><comment><USER27> The trouble I had installing numpy, scipy and theano on python was over 3 days and only numpy & theano works.

3 days to go over the internet find out what the error... then trial and error.

Yes its difficult. And as the author says,"people get down voted"

</comment><comment><USER28> Remember that Python is several decades old, and version 2 is about 16 years old. Like UNIX man pages, the docs (as well as best practices, idioms, etc.) were influenced by programming methods of the day. There was much more time to read the docs, think, and experiment. There weren't many answers to be found.

The methods of the modern developer are much different: 1) run into problem, 2) look for library to solve problem, 3) install library from package manager, 4) look for examples of library that fit problem, 5) copy-paste-modify example, 6) commit.

I can see why the docs are insufficient for today's beginners.

<comment><USER29> PHP is just as old and when I learnt PHP 14 years ago the docs were really friendly back then too, I loved being able to go to php.net/func_name and get a good description of how it worked. Of course this is for the standard library, which is what the article talks about, but PHP's standard library (at least then) had much more of the kitchen sink thrown in it.

</comment></comment><comment><USER30> If python documentation is bad then, god forbid, these poor users should try to use anything else such as node.

As far as technical documentation does, the python documentation is usually ranging from pretty good to very good.

</comment><comment><USER31> I can't believe an advertisement has gotten so many up votes.

The only problem I had when learning python is that it took me a while to figure out that you have to put __init__.py in every folder. Easiest language I've ever learned and made projects with, and I had previously learned Java/C/C++/Erlang

</comment><comment><USER32> 1st problem, do pip install pip --upgrade in your virtualenv as the 1st thing. Another possible solution, install the corresponding .rpm/.deb package

2nd problem: go to this page first [https://docs.python.org/3/py- modindex.html](https://docs.python.org/3/py-modindex.html)

3rd problem: yeah, kind of. Except for the logging module. Eff the logging module, it's a "java made in python" crap

<comment><USER33> Structlog FTW!

I only use the logging module these days when I've no alternative. The API for configuring logging is terrible.

</comment></comment><comment><USER34> >Installing libraries

I think pip and easy_install solves this problem. Or did I read the point incorrectly?

>docs are horrible

Also I never had seen the JS documentation of Mozilla, it is just beautiful. I have been using Python without any documentation related problems for that time, I actually am fond of Python documentation, for some reason it struck a chord with me.

The thing is, I learned Python by using the docs as the starting point, studied the tutorial and later the standard library, never really had a problem sorting through the docs ever. For everything there is a global module index, go to the a-z, click on a, go to the respective class/function and it has a good example, at least for the stdlib

I love Go language's docs more though.

<comment><USER35> Try "pip install pycurl". It'll fail. The solution is to install a C compiler and (on Ubuntu) run apt-get install libcurl4-openssl-dev. That's very different to other languages - they don't expect you to have a bunch of different development environments.

<comment><USER3> That's not really true, lots of languages requires certain modules to be built from source with a C compiler.

Ruby and Node.js comes to mind.

Though in general the community behind them do a much better job of hiding this complexity from the user - to the point that most times you don't realize you've compiled it from source.

</comment></comment></comment><comment><USER36> Is there a good place to go for critiques on your project's documentation?

I'm currently working on updating the API docs for the company I work for, so we're paying people. However, what should open source projects do?

One thing I've seen that works poorly is encouraging new contributors should work on the documentation. The problem with this is that a new user only has knowledge of what they are confused by--not the broad understanding of the project that allows them to compose a document that teaches a coherent mental model. It seems like that can only be done by someone who is familiar with the project, but then they need a way to test that their writing actually makes sense.

</comment><comment><USER37> I really like python, I use it all the time now and I don't start any new stuff in Ruby. But it was a struggle to get over the hump. I pretty much never use the Python docs, for the author's example, I would just google 'python array length stackoverflow'. I would only look at the official docs if something piqued my interest and I wanted to know more. Whereas with Ruby I go straight to chapter 4 of the pickaxe book and it answers the question 99 times out of 100. Also because its a PDF it works reliably on the train! I would dearly love to find a similarly written reference for python.

<comment><USER33> I'm not sure if this is what you're looking for, but Sphinx, which is used for generating the Python docs, is capable of producing docs in a number of formats, including ePub and PDF, and offline versions of the docs have been available for download in those formats ever since the changeover to Sphinx: [https://docs.python.org/3/download.html](https://docs.python.org/3/download.html)

</comment></comment><comment><USER38> 1) is really something that the maintainers of pip should deal with but unfortunately they don't see it as their problem.

<comment><USER33> I really don't know of anything like pip that can handle that case in any language, and I'm not entirely convinced that somehow resolving the plethora of naming conventions for packages between different distros (and even different _vesions_ of those distros) is really the job of a language-specific package manager.

Nobody's done it because any potential solutions are much too brittle. On Debian derivatives, it's not too big a problem, but go outside of that, and even high-level tooling is a mess (the tooling on top of RPM varies from distro to distro, and even version to version). On macOS, if you're using a ports-like system, you might be dealing with fink, MacPorts, brew, pkgsrc, &c.

Any competently-managed project, OTOH, will list the dependencies for a number of major distros and OSs so that the person installing it will have an idea what to look for.

I'm all ears for a reasonable solution to this, if you have one.

<comment><USER38> >I really don't know of anything like pip that can handle that case in any language, and I'm not entirely convinced that somehow resolving the plethora of naming conventions for packages between different distros (and even different vesions of those distros) is really the job of a language-specific package manager.

I actually started probably the only project to actually make a stab at solving this: [https://github.com/unixpackage/unixpackage](https://github.com/unixpackage/unixpackage)

However, I'm loath to continue working on it because there are few tasks more tedious, filled with annoying edge cases and thankless than this. Plus I know it would never make it into pip because "it's not pip's job".

>I'm all ears for a reasonable solution to this, if you have one.

The only one I can think of is to packages like libpq-dev and libjpeg-dev and even the C compiler installable via pip and get the maintainers of pillow/psycopg2/whatever else to remove all their horrible hacky code to look for these dependencies on the system itself.

This will balloon the size of virtualenvs and increase installation times but it'll be worth it. Not only will it eliminate those horrible installation errors making everybody happy (not just beginners), it will eliminate a nasty source of brittleness (subtly different behavior caused by different versions of these packages on different platforms).

<comment><USER33> > However, I'm loath to continue working on it because there are few tasks > more tedious, filled with annoying edge cases and thankless than this.

That, in a nutshell, is the problem.

> Plus I know it would never make it into pip because "it's not pip's job".

There are good reasons it wouldn't make it into pip: something like that is more the realm of setuptools, rather than pip.

> The only one I can think of is to packages like libpq-dev and libjpeg-dev > and even the C compiler installable via pip and get the maintainers of > pillow/psycopg2/whatever else to remove all their horrible hacky code to > look for these dependencies on the system itself.

The 'horrible hacky code' is autotools 90% of the time, and while I'd like to see if die in a fire and for pkgconf to replace it as far as library detection goes, it's not going away any time soon.

Also, there is next to no chance of what you propose happening: you're essentially proposing a parallel system package management system, and at that point you'd might as well just propose that the Python packaging toolchain integrate, I don't know, maybe pkgsrc into itself.

<comment><USER38> >That, in a nutshell, is the problem.

That problem can be solved with money. I've seen Jessica McKellar offer to shower PSF money over people who want to help make installation of python easier for beginners on _Windows_. I've not seen her offer to shower money over anybody wanting to solve this problem.

And, as I mentioned before I don't believe a solution like that unixpackage would actually make it into pip. They'd refuse it.

 _That_ , in the nutshell, is the problem.

>There are good reasons it wouldn't make it into pip: something like that is more the realm of setuptools, rather than pip.

Maybe. I'm never really sure where one ends and the other begins, to be honest. Perhaps that's part of the problem.

>you're essentially proposing a parallel system package management system,

I am, yes. And as ridiculous as that may sound it's utterly crucial.

>and at that point you'd might as well just propose that the Python packaging toolchain integrate, I don't know, maybe pkgsrc into itself.

That would be my other proposal, yes. Either pkgsrc, guix or nix.

>Also, there is next to no chance of what you propose happening

Want to know what's really shit about python?

<comment><USER33> No, this is a significantly bigger problem than simply getting things to work well on Windows.

Have you proposed this to anyone in the PSF?

> Maybe. I'm never really sure where one ends and the other begins, to be > honest. Perhaps that's part of the problem.

In a nutshell, leaving out various details: pip downloads packages, asks them for their dependencies, downloads the dependencies, asks them for their dependencies, and so on, and so on. Then it does a topological sort and asks them to install _themselves_.

The process of installation is done by the setup.py file within the package[1]. That's setuptools.

The line between the two is pretty clear; this is similar to the split between dpkg/apt or rpm/yum, except setuptools itself has some built-in dependency resolution support.

[1] Assuming it's not a wheel.

> I am, yes. And as ridiculous as that may sound it's utterly crucial.

And a security nightmare.

> Want to know what's really shit about python?

Do you know what's shit about dealing with just about every language out there? Pick a language with its own package management, and the moment it steps outside of its own runtime environment, this problem crops up.

Want to know why I think there's little chance of this working? It's nothing to do with Python and everything to do this being a nightmare for sysadmins.

<comment><USER38> >Have you proposed this to anyone in the PSF?

I'm not interested in getting involved in their politics.

>And a security nightmare.

Absurd.

>Do you know what's shit about dealing with just about every language out there?

Not this. You might have noticed a variety of java programmers saying "nope, not an issue here".

>Pick a language with its own package management, and the moment it steps outside of its own runtime environment

The issue is not at all to do with stepping outside of the runtime environment and everything to do with running code in an uncontrolled environment. The solution is to control the environment as much as you can and fail cleanly if you can't.

>Want to know why I think there's little chance of this working?

No, I have no interest in what you think any more.

<comment><USER33> > I'm not interested in getting involved in their politics.

Then why did you even bring them up?

> Absurd.

Not if you're a sysadmin.

> Not this. You might have noticed a variety of java programmers saying "nope, > not an issue here".

Because JNI is barely used. This isn't the case with Python, or Ruby, or a host of other lanuages.

> No, I have no interest in what you think any more.

Don't be an ass.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER39> I see many people like python docs, and I agree, but the article talks about the difficulty for __beginners__.

That being said, maybe the python docs need something equivalent of wikipedia's 'Simple English' definitions.

Maybe by adding a 'simple' prefix (ie: simple.docs.python.org/3.5/library/re.html) that takes us to the very simplified version of the current documentation. Links to the in-depth docs are obviously necessary as well.

<comment><USER40> The main problem with the python docs is not the verboseness. It's that they are entirely unstructured. If there just was a standard list of bold parameter - explanation and finally return value it would help immensely.

My own favorite is urllib.request.urlopen, the description mixes arguments, return values and exceptions in a blob of text two pages long. It's very confusing to read if you are a beginner just trying to fetch a file. The examples are way, way down on the page where you will never find them, the table of contents is so long you can't even see there is an example section unless you scroll down the menu.

On the bright side though, everything is explained in the text. It just less accessible than some other projects' documentation.

</comment></comment><comment><USER41> Here's a cached version (thanks, Google!) since the site seems to be down: [http://webcache.googleusercontent.com/search?q=cache:3GQ0z6z...](http://webcache.googleusercontent.com/search?q=cache:3GQ0z6zB72cJ:pythonforengineers.com/this- is-what-python-beginners-have-to-deal-with/+&cd=1&hl=en&ct=clnk&gl=us)

</comment><comment><USER42> >> I tried to install X. But when I ran it, it threw up a generic error, which meant (after Googling) that it needed Y. Okay. So I installed Y. But Y needed Z. And Z needs a C/C++ compiler. Okay, I'm on Linux, I have those.

If package maintainer does not specify dependencies, what can be done, really?

<comment><USER38> setup.py doesn't give a consistent mechanism to specify those kinds of dependencies.

<comment><USER43> How so? Any package in setup.py can declare a install_requires listing, including version matchers, of its (direct) dependencies. Those dependencies themselves should take care of listing theirs and so on completing the chain so that tools like pip can do their job.

That doesn't mean you can't run into a situation where a dependency cannot be resolved, but I don't see any problems around specifying them.

<comment><USER44> I think the author meant packages output of pip, like this issue:

I want to install the pip package cryptography. I need to make sure Python.h is available. I need to make sure (on Debian) that lib-ffidev and lib-ssldev are available. This isn't said anywhere by pip (except Python.h, probably), and it's not the same on every system, let alone every Debian based system.

<comment><USER43> Ah, right, that makes sense. I don't think it's a fair criticism though, as you're crossing the boundaries between packages provided by your language ecosystem vs. those provided by your OS (distribution). There's no easy way to specify this in a way that would work on all platforms or you'd have to start adding logic to pip that knows how to apt/yum/pacman/port install those things.

It's the same if you look at Ruby, Node, even Go and many others. The second you need native libraries/binaries all you have to go on is the hope that the author has documented them along with the rest of the installation instructions. The current only way around it is hoping that your distribution has packaged up the library you need (and is up to date enough for your requirements) in their native format and declaring the required dependencies.

<comment><USER38> There's two ways around this:

* Provide a way to detect the package or lack thereof and fail in a consistent, easy to understand way. This part actually wouldn't be too hard.

* Provide a way to use the package via nix, guix or pkgsrc instead and enforce its usage so the OS package manager stops mattering. This has the additional benefit that you can specify versions (fairly important since different versions of those dependencies can have far reaching effects).

</comment></comment></comment></comment></comment></comment><comment><USER45> oh please, don't compare python docs to php docs, php docs show one function per page, because their api is WRONG, python shows them often per module level, how to compose the api of a module and use them with actual examples, the doc is useful for experienced developers.

I feel this article is advertising their book, maybe the only point to consider is the somewhat broken status of pip that have to deals with system libraries dependencies, but whatever, I still use pip every day and with multiple production deploys.

Is there some good comment about python docs, packaging management, that isn't so much a rant?

<comment><USER46> This being said, I find the PHP docs' styling to be easier to read. The python docs are all over the place.

This being said, the PHP doc linked to in the article is 15% document, and 85% user comments (why do docs even have user comments?). Yuck.

<comment><USER16> While I understand the idea of not having user comments, back when I was doing php - and remember this was a time without StackOverflow - checking the docs user comments would get me going very quickly as some particular use cases would always show up there (and with improvements from other users too).

</comment></comment></comment><comment><USER47> I'd suggest using ZealDocs or Dash if you want fast searchable documentation.

</comment><comment><USER48> Regarding libraries, I often check unit tests for examples. But yeah, regex and similar docs suck.

</comment><comment><USER49> I'd like details on the "installing libraries" comment. for python, most distros come with a package, otherwise compilation is rarer.

If compiling python, or even something like opencv that requires compilation on any random linux distro - yes, you need to be able to figure it out. linux distros aren't all standardised.

<comment><USER21> > I'd like details on the "installing libraries" comment. for python, most > distros come with a package, otherwise compilation is rarer.

Installing python modules from a system's package manager is not something people should do. It causes more issues than it solves.

<comment><USER50> Can you go into more detail? So far, for my basic scripting needs, system packages have always been fine.

<comment><USER51> First of all, distro packages essentially always lag behind the released upstream versions, so you have to begin with figuring out which version the distro packaged, and find documentation for that, include that version in all your questions to the mailing list for the library, and hope that you're not on a version so old that it's no longer supported upstream.

Then there's the way distros like to make changes for "consistency" with the rest of their system. Command-line entry points get renamed or have options and behavior changed to reflect the distro's standards, but now the thing you're running isn't the thing upstream wrote or documented, so you're even more on your own if you hit trouble.

Also, it's not unheard-of for distros to take something that's a single packaged entity upstream, and split it into multiple distro packages, possibly without making it entirely clear that to get full functionality you'll have to install multiple distro packages. And, again, this is specific to the distro and their choices, so of course upstream won't document it and may be just as baffled as you are when you ask why something that's important to you doesn't seem to be in the package you got.

Finally, the Python ecosystem is built around the _Python_ packaging and distribution toolchains. You're going to find plenty of expertise in things like using pip and virtual environments to develop and deploy Python, but all that goes out the window when you switch to using the distro package manager and packages. So you're giving up a lot of tooling built by and for Python engineers, for something which had to be a one-size-fits-all solution for the operating system it's bound to.

So while you can do all right with system packages and tooling for quite a few use cases, the dangers aren't that far away and are ready to hurt you, badly, as soon as you stray just a little bit outside of what the distro optimized for, or need help with a package beyond what the distro and its packagers are able to provide.

<comment><USER50> Hm okay, at least the first two things don't apply for me since Arch Linux updates usually within a few days, and likes to stay as close to upstream as possible.

</comment></comment></comment></comment></comment><comment><USER52> Maybe instead of trying to teach others with a 'book project' he should try teaching himself and learn the actual language then he wouldn't have these kind of imaginary 'issues'!!

</comment><comment><USER53> Has anyone made tldr pages for python packages?</comment>


Don't Recycle - <AUTHOR> http://web-poet.com/2009/04/22/web-ideas/

<comment><USER2> Call me poetically-challenged, but I'd really appreciate an explanation. I don't get it...at ALL. How does earth day have anything to do with not recycling? What does URL shortening have anything to do with that either?

<comment><USER3> Internet content should not be recycled: Videos and pictures are forwarded forever through emails consuming bandwidth and storage and therefore energy and resources. This may also apply to copycat internet ideas whose number is overkill that most sites don't survive and can be considered just waste of resources. URL shortening is considered by the author as one of these.</comment></comment>


Firefox Will Block Navigational Data URIs as Part of an Anti-Phishing Feature - <AUTHOR> https://www.bleepingcomputer.com/news/security/firefox-will-block-navigational-data-uris-as-part-of-an-anti-phishing-feature/ <comment><USER2> missing from title "...only for top-level navigation".

Nothing to see here. They are late to the party even. This won't break any legit site out there. At all.</comment>


Unable to email anyone from primary domain name; thanks Google Mail and G Suite - <AUTHOR> https://mailman.nanog.org/pipermail/nanog/2019-October/103817.html <comment><USER2> Great writeup. I'm running an app that monitors social news sites like HN and sends the matching posts to the user. Most of my emails are marked as malicious, and my G Suite account was even blocked once.</comment>


Chomsky: The U.S. behaves nothing like a democracy - <AUTHOR> http://www.salon.com/2013/08/17/chomsky_the_u_s_behaves_nothing_like_a_democracy/?source=newsletter <comment><USER2> > The MIT professor lays out how the majority of U.S. policies are opposed to > what wide swaths of the public want

This is a feature of the design of the U.S. federal government. The framers of the Constitution were very familiar with the history of governments and knew that direct democracy was a disaster. It devolves into mob rule, and let's face it, most people don't know much about most things.

Chomsky says that the lower 70% of the population has no effect on policy, but there is plenty of evidence to the contrary. The U.S. system of income taxation is progressive and refundable, meaning that there is substantial transfer of wealth from the richest people to the poorest. The upper income brackets pay a disproportionately large percentage of all income taxes.

U.S. states fund universal public education through high school; a policy that benefits the poor and middle class far more than the rich, who can afford to send their children to private schools. Health care funding like Medicare and Medicaid

I'm not saying that these are bad! Just that they demonstrate the extent to which U.S. policy does benefit a broader swath of the population than you might think from reading this essay.

Ironically, some of the things that Noam Chomsky likes the least are in fact popular with a majority of the U.S. population. Pew reports that 61% of American citizens approve of the drone assassination program, for instance [1].

[1] [http://www.pewresearch.org/fact-tank/2013/07/25/big- gender-g...](http://www.pewresearch.org/fact-tank/2013/07/25/big-gender-gap- in-global-public-opinion-about-use-of-drones/)

<comment><USER3> > Chomsky says that the lower 70% of the population has no effect on policy, > but there is plenty of evidence to the contrary. The U.S. system of income > taxation is progressive and refundable, meaning that there is substantial > transfer of wealth from the richest people to the poorest.

No, it doesn't mean that, because income tax isn't the only tax (or even the only tax _on incomes_ , as the payroll tax, which is regressive, also exists, and, on top of that, forms of income that tend to be distributed more to the richest -- long term cap gains -- have preferential tax treatment over other income), and because while the nominal rates are progressive the distribution of deductions and credits (including refundable credits) are not.

<comment><USER4> Even with preferential treatment of LTCG, high earners still pay a disproportionate share of income taxes (which include capital gains taxes).

The real bias in the tax code is, perhaps unsurprisingly, around the median income, where many expenses that the poor do not incur to the same degree (home ownership, college tuition, IRAs) are highly subsidized, with most credits and deductions dropping off as you hit a higher income level.

</comment></comment></comment><comment><USER5> >The Federalist Papers were basically a propaganda effort to try to get the public to go along with the system. But the debates in the Constitutional Convention are much more revealing. And in fact the constitutional system was created on that basis.

Does anyone have any insight into this? Is there truth to this claim?

</comment><comment><USER6> Chomsky's dogmatism and oversimplification of complicated factors weakens his points ( _some selective polls I 'm not linking to show overwhelming support for things the government doesn't do_ \- but compared to how many things with overwhelming support that _is_ done by the government?). There are certainly _plenty_ of examples of ways the US is not a perfect democracy, or even a perfect democratic republic, but it is trivial to disprove that it behaves _nothing_ like one.

<comment><USER7> If it is trivial, can you disprove it? Quacks like a plutocracy...

We need better democracies.

<comment><USER8> > We need better democracies.

Yes. But for that we need better people

<comment><USER9> Or, perhaps less corrupt and more broad minded actions from government/business. Positive change can begin with enforcing transparency and social and environmental concerns that presently all but escape the first world's de-facto economic rationalist system, which is rapidly infecting all corners of the planet, crowding out any space for existing or proposed, often functional and noteworthy alternatives. Recommended read: [http://en.wikipedia.org/wiki/Debt:_The_First_5000_Years](http://en.wikipedia.org/wiki/Debt:_The_First_5000_Years)

</comment></comment></comment></comment><comment><USER5> >Yet economic policies have changed little in response to one electoral defeat after another. The left has replaced the right; the right has ousted the left. Even the center-right trounced Communists (in Cyprus) – but the economic policies have essentially remained the same: governments will continue to cut spending and raise taxes.

I like his ability to cut to the truth of the matter. What the people vote for and desire is quite secondary at the end of the day.

</comment><comment><USER10> > The reality is that there is massive state intervention in the productive > economy and the free-trade agreements are anything but free-trade > agreements. That should be obvious. Just to take one example: The > information technology (IT) revolution, which is driving the economy, that > was based on decades of work in effectively the state sector – hard, costly, > creative work substantially in the state sector, no consumer choice at all, > there was entrepreneurial initiative but it was largely limited to getting > government grants or bailouts or procurement. Except by some economists, > that's underestimated but a very significant factor in corporate profit. If > you can't sell something, hand it over the government. They'll buy it.

> After a long period – decades in fact – of hard, creative work, the primary > research and development, the results are handed over to private enterprise > for commercialization and profit. That's Steve Jobs and Bill Gates and so > on. It's not quite that simple of course. But that's a core part of the > picture. The system goes way back to the origins of industrial economies, > but it's dramatically true since WWII that this ought to be the core of the > study of the productive economy.

Can someone comment on this? I am of course aware of ARPA and DARPA but is it really as one sided as he says?

<comment><USER11> The beginnings of computers being used for artillery projections and code breaking and then transferred over to commercial enterprise almost wholesale is documented fairly well in a book called Turing's Cathedral, but there were all sorts of pissed off people when Von Neumann went ahead and commercialized stuff that invented by others and was top secret. The british couldnt even comment on the fact that they had computers for a good 30 years.

</comment></comment><comment><USER12> This is a link to the transcript of the speech. If you'd prefer to listen (or watch) the talk the recording is available on youtube: [https://www.youtube.com/watch?v=btlgQs0UDxY](https://www.youtube.com/watch?v=btlgQs0UDxY)

</comment><comment><USER13> Hey, original poster - this is a blatantly and thoroughly political article.

While it might very well be interesting and worthy of discussion _somewhere_ , that somewhere should be somewhere _else_.

<comment><USER14> The above comment is correct, according to the guidelines.

It's probably not worth complaining anymore though, this site really turned a corner with the NSA stuff.

<comment><USER4> Seriously. Now Chomsky is working his way onto the front page with some regularity?

Not to mention when I woke up this morning the top post was somebody writing for an online magazine of "Art-activism, eco-anarchism, subversion and sedition" positing a conspiracy by the plutocrats to put people in make-work jobs.

Alas, I lost my flagging privileges around the time we had a front page with half a dozen postings about a plane to Cuba that no one of interest was on.

</comment></comment></comment><comment><USER15> OK, Mr Chomsky. Why don't you go talk about democracy to your friend, Fidel "Butcher" Castro?

<comment><USER16> This has got to be the most stupid, ad hominem, ignorant comment of the whole HN.

Congratulations!</comment></comment>


How Balanced Automates Testing and Continuously Deploys - <AUTHOR> http://blog.balancedpayments.com/balanced-payments-operations-automated-testing-continuous-deployment-jenkins/ <comment><AUTHOR> If these kinds of problems interest you and you're looking for a real challenge, contact us! We're always looking for sharp and talented individuals that can make an impact.

NmQ2ZjYzMmU3Mzc0NmU2NTZkNzk2MTcwNjQ2NTYzNmU2MTZjNjE2MjQwNjU2MzZlNjU3MjY1NjY2NjY5NjQ2MTY1NmI2MTZkNmY3NDc0NmU2MTc3Njk=

<comment><USER2> No offence. (and also using your excellent service) But.

My statistical background have always questioned me, are these guys, who are posting rebuses on job pages are actually searching for rebus solvers or need their job done?

Or did they ever conduct a practical coding experiment with applicants who love rebuses and who dont..

<comment><USER3> It took me exactly 20 seconds to get the email address behind this ... and I'm not a professional rebus solver. I just happen to know how a base64 and hex encoded string looks like.

<comment><USER4> dammit - I thought I had decoded gibberish for 5 minutes there ! COuld not work out what I had done wrong.

when you say 20 seconds did you reach for a bash command ? interested knowing which.

<comment><USER5> Here is some bash kinda:

curl -s [http://blog.balancedpayments.com/balanced-payments- operation...](http://blog.balancedpayments.com/balanced-payments-operations- automated-testing-continuous-deployment-jenkins/) | html2text | grep -P '[^.*]=$' | head -n 1 | base64 -d | sed 's/../0x& /g' | xxd -r -c 100 | rev

<comment><AUTHOR> You should apply. I'm looking for people like you :)

<comment><USER6> email@example.com

I really wouldn't hire someone who wants to use grep to parse out html[1] :P

Then again, I would not like to work for a place where engineers don't understand this ;)

[1] <http://stackoverflow.com/a/1732454/366152>

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> Any light on how to pull back once a change has been deployed? If it's version controlled you can check out a previous version but do you automate tht?

<comment><USER8> Hi there. I'm the author of this post.

We use git, so it's easy to revert commits, or push an update with '-f' that resets to an earlier version. Once we push this new commit (that simply puts us back to an earlier state) to our release branch, it's picked up by the testing system just like any other commit and pushed out.

<comment><USER7> That seems like a slow process if you're "oh shit, rollback, rollback, rollback!"

<comment><USER8> If we're already at the "Oh shit" stage, then we might be happy sacrificing test coverage to go back to known-good code, in which case we would run our fabric `deploy` task and have our rollback completed in ~30 seconds. The idea behind this process is that we never have "oh shit" moments like that :)

(Please see my other response to a similar question, as well)

</comment></comment></comment></comment><comment><USER9> We used a similar continuous deployment process for Canvas, minus the staging server (now that we're doing iPad software, things have changed). A nice next step is to deploy to a single server at first and only roll out to the rest of the servers once that one has been audited - if that server starts failing, divert traffic from it until it gets back to a healthy state.

<comment><USER10> This is a great idea. Can you share any insight into how to monitor this via an automated process as part of a deploy and how to finish scheduling the remaining deploy once it has been verified?

I'm guessing we could use Jenkin's join plugin for this and have a job that just waits for x minutes but time does not necessarily correlate to a feature being used.

<comment><USER9> I'm still doing that step manually as I figure out how I want to automate it. An easy way would be to emulate your load balancer's health check (or interface with your load balancer if you can), in our case this is AWS's ELB which just hits a /ping endpoint on the instance that gets handled reasonably deeply in our stack and puts that instance out of service after enough consecutive failures.

It helps to look at what your past failures have been when going live in production to know what to audit during incremental rollout. In our case, /ping has stopped responding before, but some other common cases were increased 500 response rates or severely decreased performance across average or peak response times. These can be used as metrics when we automate. It doesn't help much for infrequently-used features, but I think the main idea is to prevent all of your instances from going down/going haywire at once together, which is unlikely to be a problem caused by such features even if you rolled out to all at once.

Would be interested to hear if you get that working with Jenkins to handle incremental rollout, I haven't tried yet. I'm not even sure how this would be done in Fabric but that might be another option (and have Jenkins call Fabric and block on its completion).

<comment><AUTHOR> wahnfrieden,

Can you reach out via support @ balancedpayments.com? Come hang out on our IRC - irc.freenode.net #balanced.

Would love to talk to you more!

</comment></comment></comment></comment><comment><USER11> The article mentions it takes 10 minutes to do a release so if you have revert to an earlier version you could be looking at significant (for a payments processor) downtime. I presume you have a strategy for rolling back directly on the servers?

<comment><USER8> The whole idea behind this testing infrastructure is so that we're fairly certain that any deploy isn't going to leave us scrambling to revert to an earlier version. Since we've implemented it, we haven't had to do that yet -- it's caught all manner of issues large and small that we're glad never made it into production.

The 10 minutes is to run the full suite of tests. In the case of an emergency, if we needed to get code out and wanted to bypass tests (e.g. reverting to an earlier version), we could run our deploy task manually, in parallel on all machines, and be done in about 30 seconds. This is obviously a nuclear option that we'd hope to never have to use, but if we were reverting to known-good code after a botched deploy, it might be our best bet.

This process is a safety net. If we want to work without a net, of course we have more options -- this is the tradeoff that we've made.

EDIT: small clarification

</comment></comment><comment><USER12> 950 unit tests? The number of unit tests sounds very small. I was expecting 5000+. Why so small? What makes your team decide to write a unit test?

<comment><USER8> Hi there. Why would you expect 5000+ tests, or anything more than 100? It seems like the number of tests that is the "right" number would be very dependent on the size of the codebase, which wasn't mentioned in the article.

We write tests when new code is added, or a bug is fixed and we want to make sure it doesn't reappear. Hopefully we add new tests without prompting, because that's the right thing to do, but our coverage enforcement will goad us into writing tests if we forget. Ideally, new code should be testable by only a few tests -- if it requires more, then it's probably too complicated and should be refactored.

<comment><USER13> > Hi there. Why would you expect 5000+ tests, or anything more than 100?

It might be a factor of testing methodology and style, many people consider that each assert is a test/each test should only assert a single things, so the number of tests grows large. I was also surprised at a "mere" 950 unit tests (our web frontend has maybe 10% test coverage and reports almost 700 tests, but that's because the test runner counts each assertion as a test, not each test case — of which we have 200)

<comment><USER8> Ahh, I see. Right, 950+ is the number of unit tests we have, each of which contains a number of assertions. If we count those, it's well into the thousands, so I guess the original question wasn't that far off the mark.</comment></comment></comment></comment>


Delay-Tolerant Networks Might Be the Killer App Ham Radio Needs - <AUTHOR> https://faradayrf.com/delay-tolerant-networks-killer-app/ <comment><USER2> I work at JPL and have begun using the ION implementation of DTN as part of a distributed computing research project. The development of this stack is very active, and momentum is pretty solid internally. It wouldn't surprise me to see more use of DTN in the near future. The lead devs are active and responsive despite being promoted away from front line work most of the time. Also, this is TRL 8/9, so ION / DTN is pretty stable.

</comment><comment><USER3> I looked into this in 2007 - 2008, when I was in college. I was doing a NASA internship, working on an engineering study of IP communication over deep space links. However my directive concerned IP protocol specifically. DTN was an interesting case study to me, but we weren't trying to use the DTN protocol so I never tried to actually set it up. I thought it was a neat idea, but I hadn't kept up with developments in that field since then.

<comment><USER4> I think of this as well, especially when looking towards the future with Mars colonies etc. However, I see it as a mix of data dumps of the internet as we know it and some sort of email system as well.

<comment><USER2> I actually think IoT is a great application, as devices become producers, but also relays for messaging. The use case for this functionality is orbiter relay, but there's nothing to prevent any moving agent to be a "router" in DTN, no matter how long it's offline.

</comment></comment></comment><comment><USER5> Mirror link please? (Oddly Google on Firefox Mobile doesn't appear to offer cache links)

And IMO, the shot of adrenaline that ham radio needs is for the FCC to take the gloves off of data modes on HF. 300 baud, are you kidding me? The same RF bandwidth that analog voice gets to use could deliver magnitudes more as a digital signal.

<comment><USER6> Its a weird case of legislating good engineering into political law. Its not needed; should be able to just quote FCC part 97.101(a) "In all respects not specifically covered by FCC Rules each amateur station must be operated in accordance with good engineering and good amateur practice"

Google up "digital transmission on fading multipath channels" or similar and for example there's a nice whitepaper from National

[http://www.ni.com/white-paper/14931/en/](http://www.ni.com/white- paper/14931/en/)

The ionosphere makes massive incredible multipath distortion causing intersymbol interference making anything over 100 or so symbols per second basically broadband noise aka jamming. Even if by some miracle a (short NVIS?) hop were decodable between hams X and Y, the ionosphere being what it is means everyone else on the band would be hearing distorted broadband noise.

Most of the work recently (like last 30 years) has been in very low baud rate systems. From memory, doesn't FT8 run at 5.86 baud? It uses multi channel fsk and some FEC weirdness to push a remarkable number of bits per symbol.

Like a decade ago I used to talk to people using Olivia mode which runs at 31.25 baud which frankly was too fast for conditions. Olivia also encoded quite a few bits per symbol. Generally symbol rates (baud rates) have been dropping while bits/sec has been increasing for a very long time. In the really old days like RTTY the baud rate was the bits/sec rate so the ratio of bits/baud was 1, it hasn't been that low in a long time. It was very frustrating and weird (an annoying!) to watch a waterfall display as multipath fading drifted thru a band and destroyed just the mark or just the space carrier of a RTTY signal. 45 baud rtty is probably the highest baud rate you'll see on HF digital ham radio.

<comment><USER7> "Most of the work recently (like last 30 years) has been in very low baud rate systems. From memory, doesn't FT8 run at 5.86 baud? It uses multi channel fsk and some FEC weirdness to push a remarkable number of bits per symbol."

It also makes communications possible where normal modes just don't work well. For example, six meter (50MHz) band contacts where the signal signal level is extremely weak and the opens very brief.

</comment></comment></comment><comment><USER8> He's re-inventing UUCP.[1]

[1] [https://en.wikipedia.org/wiki/UUCP](https://en.wikipedia.org/wiki/UUCP)

<comment><USER9> Yeah, yeah everything old is new again.

The one thing I like about faradayrf is they're at least trying. Right now the state of packet radio is either 1200bps bell tones with no FEC or modern modulation and on the other end something like FT8 which is incredible but has almost zero bandwidth and good for only communicating "I heard you".

I'd love to see a more modern packet infrastructure that isn't stuck in the 80s.

<comment><USER10> Spot on ;)

In fact this post was meant to start the discussion you are alluding to. Let's not reinvent the wheel as so much as take what we have available to us now and use it combined with some new technology to propel ham radio into modern communications.

<comment><USER9> Yeah, I would love to see some modern N-QAM, FEC spec make it's way into the APRS space. The number of times I see 144.390 key up at S8+ and no packet has to be at least 50% of the time.

Some of the stuff that happens in DMR space is pretty cool, even simple TDMA opens up so many possibilities. You could do Voice+APRS at the same time over the modern repeater network. For instance we've got BawFaw that covers most of SW Washington, would be awesome to be able to run something like that for packet radio.

</comment></comment></comment></comment><comment><USER11> Curiously, the "relay" mentioned in ARRL, is a form of a DTN. They were doing this a hundred years ago, just manually!

</comment><comment><USER12> Tried to comment on the page before it died. There's at least one vaguely plausible chain of events that could lead to DTN being ubiquitous: if/as vehicles start communicating with each other as we approach full autonomy, incorporating DTN into those communications would seem a possibility.

<comment><USER10> Woops, try again. Network issues this morning with the website.

(I am one of the co-founders of FaradayRF ;)

</comment></comment><comment><USER13> I wonder if there's opportunity for collaboration with Secure Scuttlebutt, which is essentially a DTN: [https://www.scuttlebutt.nz/](https://www.scuttlebutt.nz/)</comment>