Enough with the Salts: Updates on Secure Password Schemes - <AUTHOR> https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2015/march/enough-with-the-salts-updates-on-secure-password-schemes/ <comment><USER2> Not mentioned in the article, is why salting is important even if rainbow tables aren't part of your threat model:

There is more than one reason, but they all boil down to one fact; two accounts with the same password will hash to the same value if you do not have a salt.

As an example, if an attacker gets a dump of all the hashed passwords, the weak passwords will be immediately apparent just from an analysis of duplicates.

<comment><USER3> I almost feel as though this is at least half of the point of using salts. We're trying to make things slow, so forcing an attacker to crack the same password twice (or more) in cases where it's reused seems beneficial and very cheap. Weak passwords will still be cracked first but it will still require at least SOME work.

<comment><USER4> That was the reason salts were used in /etc/passwd. You couldn't tell if John and James had the same password, and you couldn't tell if John used the same password on multiple machines. Or more critically, if the root password was the same on the entire cluster.

As you two have already discussed, it mostly provides herd safety. You can't target the 'weak ones' because all the passwords look roughly the same.

</comment></comment></comment><comment><USER5> People need to read this article very carefully before forming any sort of take-aways.

The title and the start of the article suggest that salts are irrelevant. Then only a few paragraphs later, the author states that salts are indeed crucial.

Later in the article, we learn that this article isn't really about salts at all, but the importance of a high work factor / rounds.

The problem with the way this article is presented is that a casual reader will be left with the idea that salts are no longer important, which is false.

A better title would be something like, "Salts are important, but so is a high work factor"

<comment><USER6> Careful. Salt _quality_ is almost totally unimportant. And there are no well- known, well-reviewed password hashes that aren't randomized. The lesson I'd take away from an article like this is: "don't DIY your password hash".

Historically, every discussion I've seen that tried to acknowledge the "importance" of salts devolved into people defending their DIY "secret-salt" schemes. "Important salts" seem like an invitation to reject bcrypt, scrypt, and PBKDF2.

I think part of the problem is the term. I think we should stop talking about "salts", and instead just say whether a given password scheme is _randomized_ or not.

<comment><USER7> Google AppEngine is part of the Google Cloud Platform. It's a pretty big deal.

If you want to use it, (at least with Python) you need to roll your own. You can't upload scrypt, bcrypt or PBKDF2 with your app, since code with C isn't kosher on there.

<comment><USER8> Isn't PBKDF2 in the stdlib as hashlib.pbkdf2_hmac?

<comment><USER9> Google has had support for PBKDF2 in PyCrypto since around August 2012[1].

1: [https://code.google.com/p/googleappengine/issues/detail?id=5...](https://code.google.com/p/googleappengine/issues/detail?id=5303)

</comment></comment></comment></comment></comment><comment><USER10> Salt is a requirement but not a panacea. AshleyMadison's salted MD5 proves that well enough. The point of a salt is so that same password does not result in the same hash for two different users. The key point is that attack time scales linearly with user count, which makes a difference in the actual crack rate _when the breach is large_.

When cracking passwords, to crack the most hashes, your inner loop is iterating the _salts_ from the database while the outer loop moves down the dictionary. Crucially, that means the salt should always be the first bits fed to your hashing function, or else the cracker can optimize away some hashing cycles across users. HMAC will do the right thing here if salt is passed as the 'key' (although the way it expands the key to the block size is somewhat flawed). Feeding the salt to the hashing function first approximately halves the crack rate versus appending salt at the end, by preventing any reuse of hash function state in the inner loop. [1]

[1] - [http://article.gmane.org/gmane.comp.security.phc/2325/](http://article.gmane.org/gmane.comp.security.phc/2325/)

</comment><comment><USER11> Serious question: Why isn't there a "best practice" for this? A library everybody uses to store passwords in the most secure way possible, and when more secure ways are found, the library is updated and not your code?

Everything is so low level, manual and error prone for such a basic part of app security.

<comment><USER12> The library can't be updated automatically, because suddenly all your stored passwords wouldn't work.

If it's a big monolithic library like Spring, they just deprecate APIs and add new ones (which is in fact what they do); if you're in a microframework language like Node, you're on your own to rip out your md5 module and switch it to bcrypt.

<comment><USER13> > The library can't be updated automatically

Why not?

> because suddenly all your stored passwords wouldn't work.

Passlib handles that with a concept of "deprecated algorithms"[0]: a cryptcontext has a list of algorithms it can accept as inputs to validate against, and a subset of these can be marked as deprecated[1]. The passwords hashed with deprecated algorithms will validate but they'll be flagged as needing an upgrade. The `verify_and_upgrade` method will return both the validity of the password and a new hash if it should be upgraded, so the normal pattern is:

        valid, new_hash = pass_ctx.verify_and_update(password, old_hash)
        if valid:
            if new_hash:
                # store new hash for user
            # password was valid
        else:
            # password wasn't valid

[0] [http://pythonhosted.org/passlib/lib/passlib.context- tutorial...](http://pythonhosted.org/passlib/lib/passlib.context- tutorial.html#deprecation-hash-migration)

[1] in recent versions you can even set the _deprecated_ list to "auto", this will consider all algorithms but the default (the one used if you encrypt with the cryptcontext without requesting anything specific) to be deprecated. It's a thing of beauty.

</comment></comment></comment><comment><USER14> Interesting side note: one of the most successful altcoins (a few years ago, back when people cared about this stuff) was called litecoin. The major difference is that it used scrypt instead of SHA because it was hard to parallelize. The idea was this would keep rich groups from taking over the network by throwing money at hashing clusters.

I built a machine in a milk crate that had 5xR9s to mine it. I broke even and sold it after I got bored :) It was fun for a month or two.

<comment><USER15> This was probably the most notorious misuse of scrypt.

Memory usage of scrypt was tuned to take 128KB, which made Litecoin mining ~10 times faster on GPUs than on CPUs.

</comment></comment><comment><USER16> Is anyone here using Scrypt in production?

I had do decide on a hashing scheme recently and ended up going with Bcrypt just because of how new Scrypt is.

<comment><USER17> I know Keybase ([http://keybase.io](http://keybase.io)) uses scrypt, but that's the only one I know of off the top of my head.

<comment><USER18> Details, in case it's helpful/interesting to anyone: [https://keybase.io/docs/api/1.0/call/login](https://keybase.io/docs/api/1.0/call/login) . Also of note, the hashing is done client side, not server-side.

<comment><USER4> Doesn't that mean the hash is the password, not the password? If I steal the password file what keeps me from just logging in as everybody?

<comment><USER19> The client's hash is the password but the password file contains salted hashes of the client's hash. An attacker that has the client's password hash can replay it, but they don't know the user's password. An attacker with the password file doesn't know any clients' password hashes or whether any of the password hashes are reused by different users.

<comment><USER20> Yo dawg, I heard you liked hashes...

Meme reference aside, this is actually a sane way to do things.

</comment></comment></comment></comment></comment></comment><comment><USER21> I never understand why PBKDF2 gets so little love. It's easy to use, it's easy to implement correctly, it's standardised. scrypt is cooler, but PBKDF2 is just fine.

<comment><USER22> PBKDF2 is actually a crappy standard:

\- The work factor depends on the length of hash. For example, if you compute and store 64 bytes of PBKDF2-HMAC-SHA256, an attacker needs to only compute the first 32 bytes to verify the guess, and with PBKDF2 it means doing half the work.

\- In most common use (with HMAC), it inherits HMAC "vulnerability": a password longer than the hash function block is equal to the its hash (see this Twitter thread: [https://twitter.com/dchest/status/421595430539894784](https://twitter.com/dchest/status/421595430539894784)). For example, `plnlrtfpijpuhqylxbgqiiyipieyxvfsavzgxbbcfusqkozwpngsyejqlmjsytrmd` and `eBkXQTfuBqp'cTcar&g*` have the same PBKDF2-HMAC-SHA1 hash. (Scrypt is also vulnerable, BTW, as it uses PBKDF2-HMAC-SHA256.)

\- PBKDF2 is commonly used with hashes which are fast on GPU or custom hardware and require tiny memory, thus making defense/attack cost ratio very bad. See scrypt paper for cost estimates ([https://www.tarsnap.com/scrypt/scrypt.pdf](https://www.tarsnap.com/scrypt/scrypt.pdf)).

BTW, we had Password Hashing Competition ([https://password- hashing.net](https://password-hashing.net)), the (tweaked) winner of which will hopefully be widely used in the future.

<comment><USER23> It's difficult to implement efficiently:

* If you use the underlying hash as a black-box, each HMAC invocation compresses 4 blocks. An optimized implementation will only require 2. Many implementations (e.g. the .NET one) don't include that optimization and thus suffer from a 2x slowdown compared to the attacker's implementation.

* If you use HMAC naively, it'll hash the whole password on each iteration, opening a DoS attack. Django made this mistake (see [http://arstechnica.com/security/2013/09/long-passwords-are-g...](http://arstechnica.com/security/2013/09/long-passwords-are-good-but-too-much-length-can-be-bad-for-security/) )

It abuses the underlying hash:

It xors together the outputs of different iteration numbers. This can interfere with the feed-forward of certain hash constructions. As far as I can tell this does not cause security issues, but it's still very dirty.

\---

Personally I'd go so far as to say that it's worse than PBKDF1 with a sufficiently wide underlying hash.

</comment></comment></comment><comment><USER24> Reading this title I was wondering what news they have over PBKDF2/Scrypt/Bcrypt and whether it would really be better. Read through it quickly and was disappointed. Unlike they claim, this advice was popular even in 2010: [https://security.stackexchange.com/questions/211/how-to- secu...](https://security.stackexchange.com/questions/211/how-to-securely- hash-passwords)

</comment><comment><USER25> "The multiple salts don't really do much since they're all likely known to an attacker or can be quickly calculated given knowledge of the other two component"

I've never heard before that you can calculate the sitewide salt (i.e. in memory abd unknown to attacker) from having both the user password and the per-user salt. How is this done?

<comment><USER10> The second half of that sentence is false, assuming the site salt has enough entropy, i.e. 32-bytes from a CS-PRNG.

However, if your server is compromised, pulling the site-wide salt (aka 'pepper') from memory is presumably trivial, so you typically assume it is known to the attacker.

<comment><USER25> I'm not really familiar with the field, is it really trivial to compromise both the database and the application server? And is it really that trivial to gain data out of memory?

Is there a good resource that reviews attacks that actually occurred and analyzes what and how it could have been prevented?

</comment></comment></comment><comment><USER26> These days, what is wrong with rolling your own encryption of passwords using key strengthening and salting?

Like this:

hash = sha1 applied recursively 4071 times ( password . salt ) . salt

Serious question. Don't just say religiously, "just use bcrypt". Tell me what is really wrong. What attacks will succeed today? Any crypto enthusiasts in the audience?

<comment><USER12> The non-expert answer is: Because crypto math is difficult for randos to analyze, and it's not obviously the case that running an algorithm 4,000 times is 4,000 times harder to break than running it once.

And from a purely pragmatic perspective, why even consider it? You have a bcrypt/scrypt library, writing your own code is harder and almost certainly worse, so there is literally no reason to spend even a second thinking about it.

<comment><USER26> Yes, there are reasons to think about it. For example if I want to understand the principles that make the usual algorithm strong.

There is a reason why Apple makes every app implement its own receipt validation scheme. So a dedicated attacker can't crack all apps at once. Same here.

</comment></comment></comment><comment><USER27> Aren't some new password hashing schemes coming out now?

[https://password-hashing.net/](https://password-hashing.net/)

<comment><USER20> There's still some tweaks to be made to Argon2 before it's finalized. Mostly GPU resistance.

Then there's the standardization for the crypt(3) format.

</comment></comment><comment><USER28> Of course all this might go out the window once quantum computers become mainstream.

<comment><USER22> How so? Currently known symmetric crypto algorithms are secure against attacks by quantum computers. More than that, hash functions, which are building primitives of password hashing functions, are used in "post-quantum crypto", e.g. for signatures ([https://en.wikipedia.org/wiki/Post- quantum_cryptography#Hash...](https://en.wikipedia.org/wiki/Post- quantum_cryptography#Hash-based_Cryptography), [http://sphincs.cr.yp.to/](http://sphincs.cr.yp.to/)). (Although more research should be done on memory-hard functions.)</comment></comment>


Would people be as disappointed if Apple released iPhone 4S as iPhone 5? - <AUTHOR> http://www.marco.org/2011/10/04/iphone-4s-by-any-other-name

<comment><USER2> Probably.

The Big New Feature is the voice thing, and it really needs to prove itself in the field -- it risks being written off as a gimmick, and one that'll make you self-conscious whenever you try to use it.

The potential excitement is clouded by skepticism.

Without that, what's left? International roaming is something a small fraction of us will use. Sprint as a carrier is hardly a fundamental transformation. I mean, don't get me wrong, I'll take the incremental improvements to CPU and camera and software (I'm about to replace an Android on its last legs) but they fail to fascinate.</comment>


How to avoid working for a douchebag - <AUTHOR> http://www.codeanthem.com/blog/2010/07/how-to-avoid-working-for-a-douchebag/

<comment><USER2> " _It just takes a single question to be quite effective at filtering out the douchebags from the rest:_ "

      "As a manager, what do you do to help
      your programmers do their job better?

Indeed. If anyone I was interviewing had ever asked me that question I could have talked for at least a couple of uninterrupted minutes. And it's pretty clear plenty of the ... subpar managers I've worked for would have failed it for one reason or another.</comment>


Hearing your touch: A new acoustic side channel on smartphones - <AUTHOR> https://arxiv.org/abs/1903.11137 <comment><USER2> Pretty cool paper: they train LDA model to predict password from sound of taps with 61% accuracy! This required background app running on device with access to microphone. The obvious extension is to predict password from video clip of someone entering it!

<comment><USER3> The front camera has very little chance to register any finger motion on screen, though.

Adding accelerometer (tilt / shift) data could help a little bit, because it's correlated to the motion of typing hands, too.

<comment><USER4> It helps quite a bit. The relevant web APIs were retroactively rate-limitied to mitigate these risks.

[https://bugs.chromium.org/p/chromium/issues/detail?id=421691](https://bugs.chromium.org/p/chromium/issues/detail?id=421691)

[https://arxiv.org/abs/1410.7746](https://arxiv.org/abs/1410.7746)

</comment></comment></comment><comment><USER5> For PIN codes, would randomizing the placement of the keys on the virtual 10-key keyboard mitigate this?

<comment><USER6> Yes, and it'd help with simple finger oil on screen analysis, but it'd make for a frustrating ux.

<comment><USER7> LineageOS has this as an option.

It's actually not so bad UX-wise.

</comment></comment></comment><comment><USER8> Could be cool to combine with an accelerometer and have an accurate keyboard with no moving parts and without multitouch. Could be quite rugged.

</comment><comment><USER9> Yet another reason why the mobile OS should (freeze / swap out / disable) an app upon entering the background ... at a minimum the user needs to have more control over degree to which above happens ... of course powers to be love a smartphones current total lack of such privacy

</comment><comment><USER10> I wonder how much accuracy you could get doing the same thing with the IMU.

<comment><USER11> > In controlled settings, our prediction model can on average classify the PIN > entered 43% of the time and pattern 73% of the time within 5 attempts when > selecting from a test set of 50 PINs and 50 patterns. In uncontrolled > settings, while users are walking, our model can still classify 20% of the > PINs and 40% of the patterns within 5 attempts.

[https://www.cs.swarthmore.edu/~aviv/papers/aviv- acsac12-acce...](https://www.cs.swarthmore.edu/~aviv/papers/aviv- acsac12-accel.pdf)

</comment></comment><comment><USER12> Brings a new meaning to touch typing.</comment>


Why I left Medium and moved back to my own domain - <AUTHOR> https://arslan.io/2017/07/30/why-i-left-medium-and-moved-back-to-my-own-domain/ <comment><USER2> _" Writing comments to Medium posts feels awkward because each comment is treated as a blog post."_

This is also awkward from the reader point of view. Trying to follow a comment chain on medium is frustrating as it only shows the first level, diving in switches pages, and often, a need to press a second "load all comments" button. Then the back button to wind your way back up.

Is there some sound reasoning for why it was set up this way? Some benefit I'm not seeing?

<comment><USER3> I suspect it's because they didn't/don't want people to comment in the usual way - you can highlight text on an article and leave short comments in the relevant place if you want to do that.

I think the comments section at the end was intended more as a place for lengthy reaction posts. It's why comments have "recommend" rather than "like" as a button. I might "like" someone saying "great post!" but I'm not going to "recommend" it.

It often feels like Medium has made a product for what they want online discourse to be, rather than what it actually is. I don't hate them for trying.

<comment><USER4> Commenting on articles is a terrible method of interaction. Anti-spam is hard, and spammers always win. Trolls always win. Comments rarely add much to the discussion from the article, and they're often hard to follow. There's no reason to leave a comment saying "great post!". It's spam. It's unnecessary.

Blogs are a wholly inappropriate place for back-and-forth discourse to happen simply because if I have a blog, it's my blog. It's not a discussion board. I inherently have the advantage, I control the conversation, and it's all about me. I have the ability to delete comments I disagree with. That's not discourse, that's an echo chamber.

I completely agree with Medium. Just because blog posts have historically had comments doesn't mean it's the right choice. The minute people accepted "pingbacks" as a valid form of comment was the moment comments died. V!AGR@ and P3N15 type spam sealed the deal.

<comment><USER5> > Commenting on articles is a terrible method of interaction. Anti-spam is > hard, and spammers always win. Trolls always win.

But that's practically what HN is, yet I like to believe it's a good platform without too much spam or trolls.

<comment><USER6> That makes me curious as to the mechanisms hn employs to combat spam. I've never seen a comment I've been able to tell was spam, which is impressive.

<comment><USER7> Visit your HN profile then set "showdead" to "yes" to see some of such comments.

<comment><USER8> There's some stuff that shows up there, but surprisingly little.

I've been browsing "showdead" for quite some time. My main complaint is that there's no obvious way to tell if an account has been shadowbanned or not. I'll report questionable instances to hn (via email), and on something like 3/4 of cases, the account has been, though it's not clealry evident.

I'll vouch for some flagged items. Flag others. Generally I think the comment ranking is pretty good. There's a tendency for somewhat fluffy comments to be excessively highly rated, but as compared to other systems (say, Reddit), HN compares favourably.

There's also a slight tendency for the hivemind to punish critical, but valid, comments. But again, compared to most other systems I've seen, not particularly, and the situation often evens out.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> I don't understand why people write programming blogs on Medium. I think that managing a live website with its own server (rather than only working on other people's projects) is a very important skill.

If you don't want to manage WordPress, try a static site generator like Metalsmith: [https://github.com/segmentio/metalsmith](https://github.com/segmentio/metalsmith)

Deploy on Digital Ocean or Linode for $5/month. Free hosting options include Github Pages and Netlify.

<comment><USER10> I can't speak for others, but one reason why I'd use something like Medium is _precisely_ to ensure that I don't manage anything by myself except for the content production part.

I have a long list of attempts at setting up my own site, but because it's mine I wanted to get it just right. And so down the rabbit hole I go: making sure my server is super-secure (because it's important hey I'll learn in the process!), finally trying out Caddy, using Ghost (node.js), building my own static site generator using React for the backend, which of course means updating and perfecting my Ultimate Front-End Setup.

And once that's done, I want to do right by the design, and of course make sure my html/css/js is impeccable and that it all runs on every browser under the sun.

Of course I love doing this and I learn a lot from it, but if my goal is to write stuff and put it online, this approach never really works. I'm happy it doesn't, because it means I still love what I do more than writing about what I do, but it's one very legitimate reason to use something like Medium.

<comment><USER11> I've had largely the same experiences. Recently I decided to challenge myself to instead keep it simple, rather than treat the whole thing as a "learn some new thing along the way" type exercise. The end result is I actually have a blog I'm actually writing on that is absurdly easy to maintain, rather than the usual (for me anyway!) 75 percent complete side project in some new technology I'll never get around to finishing.

Simple VPS, webserver, letsencrypt, static site generator, cloud flare for CDN duties and a little CSS. All I do now is write a new article in Markdown every now and then, then run a simple shell script to regenerate the site and rsync the newly minted static pages to the server.

Personal sites or new blogs are all too often victims of premature optimisation - your amazing google PageSpeed score is nice, but who cares if no one is reading.

<comment><USER12> > Simple VPS, webserver, letsencrypt, static site generator, cloud flare for > CDN duties and a little CSS.

Why didn't you host on Github Pages and you could have saved all that but having the same flexibilty regarding a SSG use case?

<comment><USER13> > Why didn't you host on Github Pages

Tried that once, fell down a 3 day rabbit hole trying all the static blog generators.. ended up picking WordPress.

<comment><USER14> Same. For all the times this is recommended, I haven't found a static blog generator that really works well. WordPress is still the easiest and most flexible to get running, but it's a PITA to keep it maintained.

After trying Lektor (fine for general use, but hard to theme; no pre-baked templating/theming system at all) and Hugo (tbh I don't remember the exact thing here, but I got really frustrated trying to do something and ended up deciding that themeless Lektor was better), I'm now holding out hope that Ghost, while not a SSG, will be good and simple enough that I can self-host a "blog" without falling back to raw HTML editing or WordPress. Started out by having to edit the package dependencies to versions that work on recent versions of Node... :\

<comment><USER15> Write your own generator, it's pretty easy! Blogs don't have to be fancy, most programming blogs need just paragraphs, images, and code blocks. Then you can add fancier features (interactive diagrams, fancy animations, a nice deployment pipeline, etc.) as you go along.

</comment></comment></comment></comment></comment></comment></comment><comment><USER16> About the dickbars:

1) I uBlock them on my phone and tablet. Unfortunately there are plenty of random sites on Medium with their own domain. I don't follow any of them and get there from HN or similar sites. Blocking all of them doesn't make sense because I'll probably never get back to that domain. I agree with the author, the button in the middle of the page really sucks.

2) On desktop, the button and the dickbar never show because I'm running with NoScript and they are a JavaScript thing :-)

The author misses another inconvenient feature of Medium, the login. It's either by oAuth (I don't remember which providers) or by email. That means they send me an email with a link to click to login. In theory it's ok, because obviously email is safer and more convenient than a password stored in a password manager (/sarcasm). The first time I used it the mail didn't arrive until the day after, so I've been primed against it. Probably almost everybody just login with Facebook or something so, quoting the author, "I'm not a part of his [Ev Williams] vision".

<comment><USER17> In case you're not aware, NoScript also exists for Android: [https://noscript.net/nsa/](https://noscript.net/nsa/)

Not quite as functional as its bigger brother, but an option nonetheless.

<comment><USER18> NoScript is also integrated into [https://play.google.com/store/apps/details?id=com.brave.brow...](https://play.google.com/store/apps/details?id=com.brave.browser) This uses Chrome so there's likely higher compatibility with mobile sites

<comment><USER19> script blocking may be built in but I don't think noscript is.

<comment><USER18> Other than name/logo what's the difference? Brave allows you to block all scripts by default and only whitelist specific sites.

<comment><USER19> Noscript does more than just turn javascript on and off. I recall the creator of noscript writing a post a few years ago saying that the plugin model of chrome didn't allow for all the things noscript does. That may have changed since and umatrix may be able to do things that the noscript guy said couldn't be done but I don't know

<comment><USER18> Brave's features are not an extension. It's is a fork of Chromium. Any mechanism they include in their build has none of the limitations extensions have and they're not extensions at all.

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> To the point criticism of Medium, very detailed and the OP is picking the right issues of Medium.

While Medium looks so beautiful and clean at first glance it really disappoints when you use it on a daily base both as a creator and as a user. Everytime I use Medium, I am surprised that Medium is successful. Its appearance feels definitely premium and significantly of higher value than any other blog system but the usability is a nightmare.

So we are back to square one. Which blog system should we use? SSG on Github Pages?

<comment><USER20> > Which blog system should we use?

Hosting your own WordPress site? It's simple and easy. Why did anyone ever want to be on Medium in the first place? Most likely for the name association and the visibility that comes with it.

I think we have a responsibility to promote self-reliance as part of technological culture. If we want people to move away from centralization (which I do), we have to teach by example (which I do).

<comment><USER21> _" Hosting your own WordPress site? It's simple and easy."_

I would think that, for at least 90% of those publishing on Medium, that's neither simple nor easy.

Even ignoring that, hosting your own Wordpress site means taking responsibility for keeping it up to date, security-wise. That hurdle alone probably makes most bloggers use some third-party blogging provider.

<comment><USER20> It's just an example, there are many choices, some of which are presumably even simpler.

But yes there will be some responsibility involved, I think that falls into healthy technical culture as well. It's the difference between taking care of something and getting someone else to take care of it. At the end of the day, someone has to do it, and we shouldn't always simply trust others with it just because it's easier.

</comment></comment></comment></comment><comment><USER22> The best part of reading technical content is the discussion in the comments – Medium makes that impossible.

<comment><USER23> I've seen many technical articles link to an HN post for discussion instead of using Disqus or hosting on something like medium.

<comment><USER24> I get it, too. HN is a threaded comment system, which we've been iterating on since the USENET days.

Not that threaded comments are the solution for everything, but it excels at visually separating each of the distinct discussions that may arise around a given topic.

</comment></comment></comment><comment><USER25> Medium is a blog platform and a news aggregator. I use it mostly for consuming news by using tags of interest. Good luck with that writing your own blog on your own server. Oh, and the daily newsletter is a delightful joy.

Discoverability is key to the success of Medium.

[https://medium.com/tag/swift/latest](https://medium.com/tag/swift/latest)

[https://medium.com/tag/kotlin/latest](https://medium.com/tag/kotlin/latest)

[https://medium.com/tag/python/latest](https://medium.com/tag/python/latest)

[https://medium.com/tag/java/latest](https://medium.com/tag/java/latest)

Or by topic

[https://medium.com/tag/design/latest](https://medium.com/tag/design/latest)

[https://medium.com/tag/UX/latest](https://medium.com/tag/UX/latest)

[https://medium.com/tag/AI/latest](https://medium.com/tag/AI/latest)

<comment><USER12> Why should I discover content on Medium?

This job is done by HN and Reddit and unfortunately Facebook if I am on FB. I don't have time to get sucked in another feed.

<comment><USER13> Specific content. For example I follow the Laravel tag on medium, I've not seen anything regarding Laravel on HN in a fair while.

</comment></comment></comment><comment> francis-io Using something like Jekyll to create a static blog lets you put it on AWS S3. For all but the highest traffic sites the bandwidth cost is trivial and you totally remove all the issues with uptime and securing a server.

<comment><USER26> Same applies to GitHub pages.

<comment><USER27> GHP means you're subject to github's downtime, which tends towards more frequent than s3 (outside us-east-1). Worse, you can't run HTTPS with GHP and a custom domain.

You could use CloudFlare's flexible TLS, but it's unsecured from their system to your site, which has resulted in carrier level content being injected[0].

[0]: [https://news.ycombinator.com/item?id=12091900](https://news.ycombinator.com/item?id=12091900)

<comment><USER28> That's what got me to host my Jekyll sites on my own too (lack of proper end- to-end SSL on a custom domain).

Also, if you're not using S3 / CloudFront you can use Let's Encrypt to issue free SSL certificates on any cloud hosting provider.

It's an awesome set up once you know how to get everything up and running.

I even recently released a course[0] that explains how to do it step by step (setting up a custom domain, hosting multiple sites on 1 DigitalOcean server, A+ SSL ratings for both nginx and Apache, etc.).

It takes less than 1 minute to deploy and protect any custom site (or web app), since the course comes with ready to go configs and scripts too.

[0]: [https://httpswithletsencrypt.com/](https://httpswithletsencrypt.com/)

</comment></comment></comment></comment><comment><USER29> Switched long time ago to Blogger--free to use your domain, change template design as you like as much as you like, connect external widgets like Disqus (my blog to check how Blogger may look: [http://blog.babich.me/](http://blog.babich.me/)).

Also, a huge thing is integration with Google Photos and Google Drive. You may insert in a post photos from your Google Photo right by selecting them inside an image addition menu--no need in hustling with links, just look through Google Photos inside Blogger. BTW, it's possible to see photos uploaded via Blogger in Google Photos folder inside Google Drive (Google Photos app itself does not show photos uploaded via Blogger).

You also may connect FeedBurner (Google company) in Blogger settings and collect subscription emails and also automatically send emails to subscribers via FeedBurner when a new post is published. BTW, every tag used for tagging posts in Blogger becomes dedicated RSS feed. So, if you write about different topics (e.g. coding and travel), it's easy to set up for people to follow RSS or subscribe via email only to a topic they are interested in and not everything you write.

And it's all free!

Medium feels like long-form Twitter with all these replies treated as posts. This is a terrible experience. Especially when you follow RSS feed of somebody on Medium and comments & replies are delivered and treated in the feed the same way as posts. I guess it's due to the fact that Ev was also Twitter founder and has only one construct in his head as to posts and replies.

<comment><USER30> One of the things I hate the most about Blogger is their stupid JS template where on mobile swiping left/right (to move the page if you're zoomed in) switches to the next/previous post instead.

<comment><USER29> I just rewrote the template completely. It's not that hard. So I do not have to deal with standard templates issues.

</comment></comment></comment><comment><USER31> A commentator on Arslan's post mentions "Medium's foolproof wysiwyg editor." This surprised me. I find their WYSIWYG editor to be incredibly frustrating, counter-intuitive, limiting, and obnoxious.

Allow me to belabor only the last point.

Imagine typing the following sentence (I'm using pipe | to show the cursor):

      Clifford is a big dog.

But it's missing something, so you move your insertion point:

      Clifford is a big| dog.

and you type "<space>red." In any sane editor, the result will be exactly what you typed:

      Clifford is a big red| dog.

But on Medium, the result is:

      Clifford is a big red|dog.

Clifford is a big reddog? Ffffff. This happened because when you typed the first space after "big", the result was not the insertion of a space but in fact the same as if you had hit the "right arrow" key:

      Clifford is a big |dog.

Not cool. I have a strong habit of inserting words with the surrounding spaces already inserted. How dare Medium forbid me to type the way I want?

I know that Medium does this because they are on a crusade to kill the usage of two spaces after sentence. That doesn't excuse insane frustration of the editing process. No wonder most Medium posts don't look like they have ever experienced the most cursory proofreading!

</comment><comment><USER32> I don't think this is an either/or situation.

Medium's value is in exposing new, (arguably) high-quality content to its readers.

Every so often, when you write an extra high-quality article, post it on medium as well, with a way for the reader to subscribe. It'll expose new readers to your writing and you still keep your content on your own platform.

</comment><comment><USER33> The OP forgot to mention the worst (IMHO) problem: Medium does not support multi-language content (more here: [https://medium.com/@oleksiy/multilingual- content-management-...](https://medium.com/@oleksiy/multilingual-content- management-a-challenge-that-faces-bloggers-and-a-punch-for-medium- ea1ecb3e357a)). If I write posts in English or Hungarian it is okay. But when I start to write in both my readers will see unreadable gibberish when they come to my page and half of the content will be noise for them and it can't be helped.

That's why I do it the other way around. I have my own domain and page which runs on a custom Jekyll / GitHub Pages setup and I __import __stories to Medium from my page.

This way I can keep using medium and get more readers for free but Medium will display the "Originally posted at ..." line at the bottom. Win-win! You should try this out and use Medium for what it is useful for.

</comment><comment><USER34> Moral of the story: Try to never rely on another company's platform for your business. It's hard enough creating a successful business, but to add a layer of risk on top of that is not smart.

<comment><USER35> Github, an ISP, mobile phone company, rented offices, an OS, hardware, and the list is almost endless. I actually think it's important to outsource as much as you can and focus on your core business.

<comment><USER34> you're obviously not understanding my comment

</comment></comment></comment><comment><USER36> I never understood how writers of any kind would let Medium (or Tumblr, or ...) rip all control off their hands.

<comment><USER37> Thirst for (fast/instant) fame.

</comment></comment><comment><USER38> This is good. Medium is, in a sense, the anti-internet. It centralizes where no centralization is needed.

</comment><comment><USER39> I feel one of the main reasons Medium is so successful is that they cracked discoverability and rank high on Google for any topic (sort of like Wikipedia). Hence, I am hesitant to move away from it although I really want to build a sustainable, long-term business on my own domain.

Any ideas on this?

<comment><USER40> My advice: Publish on your own domain, syndicate to Medium (and LinkedIn, and whatever) shortly after.

Always import the story[1] so that Medium adds a rel="canonical" link, and link back to the canonical version (e.g. "Originally published at XXX" in the text of the story itself.

[1] [https://medium.com/p/import](https://medium.com/p/import)

<comment><USER41> Any idea how to get code formatted properly?

[https://firstname.lastname@example.com/applied-formal-logic- brute-...](https://firstname.lastname@example.com/applied-formal-logic-brute-force- string-search-71e417b7f157)

<comment><USER40> If you type ``` on a new line, Medium will switch to code input mode.

If you forget, Control/Command-? brings up keyboard shortcuts help, and the 2nd page has the shortcuts for code formatting.

</comment></comment></comment></comment><comment><USER42> I recently wanted to move my Tumblr personal blog to Medium however after a few days of nightmare and still unable to setup a custom domain despite having paid $75 fee i requested a refund and closed my account. Their customer service and documentation kind of sucks!

<comment><USER43> It's sad that a process like this is still somehow manual. I can't imagine what technical debt they must have where they can't automate custom domains.

<comment><USER44> I believe it's by design. They want their brand "medium.com" visible. They don't want to become a commodity blog hosting.

</comment></comment></comment><comment><USER45> Personally, I never write comments on blog posts, but I participate in communities like Hacker News and a couple random subreddits.

I prefer including links to Hacker News or Reddit in the blog post. I'm not interested in handling authentication, moderating users, or dealing with spam. News aggregators usually do a great job with all those points.

The Webmention [0] spec solves this problem, but sadly, it hasn't been widely adopted. :(

[0] [https://en.wikipedia.org/wiki/Webmention](https://en.wikipedia.org/wiki/Webmention)

</comment><comment><USER46> At [https://dev.to](https://dev.to) we have a similar product to Medium in a lot of ways. One thing we try to do is offer the benefits of Medium in terms of distribution, but also encourage users to easily make their own site the canonical source, because we don't want to force vendor lock in when you get to this point.

I'd like to think we also do a good job dealing with a lot of the UX issues that the author outlines :)

</comment><comment><USER47> Why lose flexibility and switch to medium in the first place?

<comment><USER48> Because when I switched to Medium my blog posts went from ~50 impressions to 1000+ impressions

Exposure.

<comment><USER49> Are you using a custom domain with Medium?

<comment><USER48> No I opted for no-domain because my articles can be sequestered to any publication where they will receive better exposure than if I had reserved it for my own.

</comment></comment></comment></comment><comment><USER50> From a branding and long-term point of view, it's really important to "own" your content by hosting it on your own domain and even server space. The advantage of a platform like Medium is that it can increase the reach of your content, but it's too risky to go all in with them since who knows how long they'll be around, or what limitations they might add down the line.

But there's a way to have your cake and eat it too: you can publish first on your self-hosted WordPress site, and then republish automatically on Medium with a canonical tag pointing back to your WordPress site. This means you get the SEO and control benefits of WP, and the reach benefits of Medium.

I wrote a guide on how to make this magic happen: [https://illuminea.com/ultimate-guide-to-wordpress- medium/](https://illuminea.com/ultimate-guide-to-wordpress-medium/). Of course this post is also reposted on Medium with the canonical tag :) [https://medium.com/@miriamschwab/6425c2d5e5c4](https://medium.com/@miriamschwab/6425c2d5e5c4)

</comment><comment><USER51> It's not easy to run a team (engineering) blog with static site generators. Authoring in markdown, reviewing, external contributions etc takes up a lot of time.

With medium we've encouraged the team to run personal blogs and add interested articles to the team publication. It's a bottom up approach.

In return, we get viewer-ship when medium recommends our articles on other (medium) blogs or via tags. Medium has a sense of community. Speaking strictly for myself, running your own site is tad selfish, it lacks as sense of community. Right now medium does that, We'd only move out of medium if there's something that does it better.

</comment><comment><USER52> I find the first part particularly interesting. I always contemplated of whether or not I should start blogging. And for me it always seemed an obvious choice between whether or not I want to host my own blog, which would mean a lot of energy and money investment, or if I should start a medium page, possibly losing control over my texts and page views.

When you now say that it's a hassle to setup and even costs money, why should anybody use Medium? Do people not know how much you already gift a page if you let them host your content? They should pay you and not the other way around.

</comment><comment><USER31> Not to be a horrible self promoter (it's just the ideas I care about), but my 2012 discussion of the problem with new blogging platforms still applies, from a general perspective, to Medium: [https://alanhogan.com/the-problem-with-new- blog-platforms](https://alanhogan.com/the-problem-with-new-blog-platforms). (tl;dr is that eventually you will grow out of virtually any non-Wordpress platform, especially one you don't control yourself.)

</comment><comment><USER53> See also: [http://practicaltypography.com/billionaires- typewriter.html](http://practicaltypography.com/billionaires-typewriter.html)

<comment><USER54> HN really need to add Matthew Butterick's sites to spamlist.

<comment><USER55> Why so? I just skimmed read the linked page (after pasting the URL manually...) and it doesn't seem any worse than other articles posted here.

<comment><USER54> Because if you follow the link you get [http://practicaltypography.com/graylist.html](http://practicaltypography.com/graylist.html). That page has about the same amount of content as a typical spam page.

<comment><USER55> He's just blocking high traffic referrers. Which I agree is odd. It's not like his site is media heavy.

<comment><USER56> He attempts to monetize without resorting to ad networks. A laudable goal, even if the methodology is idiosyncratic.

</comment></comment></comment></comment></comment></comment><comment><USER57> I had no interest in using medium, but this is one extra little reason for not fixing my unbroken pelican+GitLab pages setup.

</comment><comment><USER58> Why remove all your posts from medium.com? Why not post to both wordpress and medium.com?

<comment><USER59> I haven't followed the progression of Google search indexing, but in the old days, the 'rule of thumb' was that having the same content on two different domains would hurt your rankings.

</comment></comment><comment><USER60> Does medium allow you to monetize your content and put ads? If it's not the case then why do people write for Medium. What's there to gain?

</comment><comment><USER61> Medium is blocked in China. And there are as many people learning English in China as the entire US population.

That's the biggest deal breaker for me.

</comment><comment><USER62> One thing that really annoys me is embeded gists don't render in medium's mobile app.

</comment><comment><USER63> Interesting read. Personally, I prefer Medium for the convenience.

</comment><comment><USER64> My blog is hosted by Medium on my own domain and I think it kicks ass. All of the issues the author brought up in his post seemed pretty minor to me. Just my opinion, but Medium has the best blogging software out there, by far.

</comment><comment><USER65> Irony: I am getting a "This site can be reached".

</comment><comment><USER66> Privacy Badger blocks most medium sites for me :)

</comment><comment><USER67> Setting up a domain was difficult. Really?

</comment><comment><USER68> i generally avoid medium.com blogs because the ui is bad. i hate the comments system.

</comment><comment><USER69> may I ask which theme you use? it looks very nice..

<comment><USER70> [https://wordpress.com/theme/revelar](https://wordpress.com/theme/revelar)

</comment></comment><comment><USER69> i did some google search for

"gatsby themes" "gatsby disqus"

almost nothing useful turns out.. any ideas?</comment>


Good Tax Law Book? - <AUTHOR>

Can anyone recommend a good tax law book? I need one that is a good introductory text for someone without a strong background in law. <comment><USER2> Not offhand. Are you looking to understand how tax litigation plays out, or the tax code? If the latter, bear in mind that whatever is current is likely to be obsolete by the end of the year and perhaps beyond, given the (theoretical) mutual interest of both main political parties in enacting some sort of fiscal 'grand bargain' that would involve substantial changes to the tax code.

On the other hand, if you just want to get the a super-basic overview, the IRS itself has a guide aimed at students that is surprisingly comprehensive and well-written: <http://apps.irs.gov/app/understandingTaxes/index.jsp>

duke university has a good collection of research materials, but they are really aimed at legal scholars: <http://law.duke.edu/lib/researchguides/fedtax/> Tax law is really messy and complex (like our politics), so it doesn't lend itself well to simply analyses :-/</comment>


Python startup time: milliseconds matter - <AUTHOR> https://mail.python.org/pipermail/python-dev/2018-May/153296.html <comment><USER3> I've always been disappointed by how large software projects, both FOSS and commercial, lose their "can do" spirit with age. Long-time contributors become very quick with a "no". They dismiss longstanding problems as illegitimate use cases and reject patches with vague and impervious arguments about "maintainability" or "complexity". Maybe in some specific cases these concerns might be justified, but when everything garners this reaction, the overall effect is that progress stalls, crystallized at the moment the last bit of technical boldness flowed away.

You can see this attitude of "no" on this very HN thread. Read the comments! Instead of talking about ways we can make Python startup faster, we're seeing arguments that Python shouldn't be fast, we shouldn't try to make it faster, and that programs (and, by implication, programmers) who want Python startup to be fast are somehow illegitimate. It's a dismal perspective. We should be exercising our creativity as a way to solve problems, not finding creative ways to convince ourselves to accept mediocrity.

<comment><USER4> This isn't an attitude of "no" \- it's an attitude of "yes" to other things. The arguments are that making Python startup fast makes other things worse, and we care about those other things.

Here are some other things we can say "yes" to:

\- Rewrite as much of Mercurial in Rust as possible, which will provide performance improvements well beyond what Python can possibly offer. [https://www.mercurial-scm.org/wiki/OxidationPlan](https://www.mercurial- scm.org/wiki/OxidationPlan)

\- Spend resources on developing PyPy, which (being a JIT) has relatively slow startup but much faster performance in general, for people who want fast performance.

\- Write compilers from well-typed Python to native code.

\- Keep CPython easy to hack on, so that more people with a "can do" spirit can successfully contribute to CPython instead of it being a mess of special cases in Guido's head.

Will you join me in saying "yes" to these things and not convincing ourselves to accept mediocrity?

<comment><USER5> I'm so longing for a Python(like) compiler.

MicroPython put together a Python in 250kb. Why the hell can't we make an LLVM frontend for Python that can use type hints for optimization? Sure, you lose some dynamic features as you optimize for speed, but that's the dream. Quickly write a prototype, not caring about types, optimize later with adding types and removing dynamicism.

I'm currently learning Racket and LLVM and I have about 70 more years to live. I'm gonna try make Python fast on slow weekends 'til I die.

<comment><USER3> Unless your Python compiler can use cpython modules without a massive performance penalty, it's going to see very limited adoption. The ecosystem matters.

<comment><USER6> We had a chance with ctypes and now CFFI to move away from the platform calcifying cpython module interface that is overly coupled to the cpython runtime. I am very disappointed in the lack of affordances that cpython gives to alternative pythons to support their work. The stdlib is a crufty mess that is overly coupled to cpython as well. The batteries are corroded and need to be swapped out for a modular pack.

</comment></comment></comment></comment></comment><comment><USER7> I have to say that my first reaction was: "maybe you shouldn't use python for this, then". If you are using a language in a way that it gets worse in subsequent versions, that's a good sign that they're optimizing for something other than what you care about.

The programming language R does not, as I understand it, optimize for speed, because they are optimizing for ease of exploratory data analysis. R is growing quite rapidly. So is python, actually. It doesn't mean that either one is good at everything, and it's probably the case that both are growing because they don't try to be good at everything. A good toolbox is better than a multi-tool.

<comment><USER8> (I authored the linked post)

While the "maybe you shouldn't use Python" comment could be construed as trolling to some, there is definite truth to your line of reasoning and I agree with comment.

I absolutely love Python as a programming language for the space it is in. But as someone who needs to think long term about maintaining large projects with lifetimes measured in potentially decades, Python has a few key weaknesses that make it really difficult for me to continue justify using it for such projects. Startup time is one. The GIL is the other large one (not being able to achieve linear speedups on CPU-bound code in 2018 with Moore's Law dead is unacceptable). General performance disadvantages can be adequately addressed with PyPy, JITs, Cython, etc. Problems scaling large code bases using a dynamic language can be mitigated with typing and better tools.

Python can be _very_ competitive against typed systems languages. But if it fails to address its shortcomings, I think more and more people will choose Rust, Go, Java, C/C++, etc for large scale, long time horizon projects. This will [further] relegate Python to be viewed as a "toy" language by more serious developers, which is obviously not good for the Python ecosystem. So I think "maybe you shouldn't use Python for this, then" is a very accurate statement/critique.

<comment><USER9> If one needs Rust, C/C++ level of performance I doubt there is much Python can do and one can wonder if Python was ever the right tool for such a project.

<comment><USER10> It's a great tool for prototyping.

<comment><USER11> If you expect to need the performance of a statically typed, compiled language I don't see why you'd prototype in a dynamically typed, interpreted language.

<comment><USER12> That's why build systems still look like black magic infused with even darker sh, and a bit of perl sprinkled all over, presumably because the previous maintainers were all out of goat blood.

</comment></comment></comment></comment></comment></comment><comment><USER13> I totally understand that milliseconds matter in the use case described in the article.

For me, personally, I use python to automate tasks - or to quickly parse through loads and loads of data. To me, startup speed is _somewhat_ irrelevant.

I built a micro-framework that is completely unorthodox in nature, but very effective for what I needed - that being a suite of tools available from an 'internet' server, available to me (and my coworkers) over port 80 or 443.

My internet server, which runs python on the backend (and uses apache to actually serve the GET / POST) literally spits out pages in 0.012 seconds. Some of the 'tools' run processes on the system, reach out to other resources, and spit the results out in under 0.03 seconds (much of that being network / internet RTT). To me, that's _good enough_ \- adding 30 or even 300 milliseconds to any of that just wouldn't matter.

I totally get that if Python wants to be a big (read bigger?) player then startup time matters more...but for my personal use cases, I'm not concerned with the current startup time one bit.

<comment><USER14> As expected, language start up time only matters to _some_ people. Often in my case, Python is used to build command line tools (similar to the case of Mercurial).

In such an event, the start-up time of the program might _dominate_ the total run time of the application. And on my laptop or desktop with a fast SSD with good caching and a reasonably fast CPU... that still ends up being 'okay'.

But once I put that on an ARM chip with a mediocre hard drive - some python scripts spend so long initializing that they are practically unusable. Whereas the comparable Perl/BASH script runs almost instantaneously.

Often to make Python even practically usable for such systems I have to implement my own lazily loaded module system. Having _some_ language which allowed me to say...

        import(eager) some_module
        import(lazy) another_module

Which could trigger the import process only when that module becomes necessary (if ever).

<comment><USER15> Have you tried moving import statements into the functions where they are invoked? My understanding is this is effectively the same as lazy loading the module[1].

[1] [https://stackoverflow.com/questions/3095071/in-python- what-h...](https://stackoverflow.com/questions/3095071/in-python-what-happens- when-you-import-inside-of-a-function)

<comment><USER14> I have, and it actually works well (performance wise). The maintenance burden is a little higher.

<comment><USER16> A little Python preprocessor that lets you annotate your lazy modules sounds like a fun little toy project, actually. Not something I'd use for real, but it would be fun to build.

<comment><USER17> 3.7 makes it easier to use dynamic imports. [https://snarky.ca/lazy-importing- in-python-3-7/](https://snarky.ca/lazy-importing-in-python-3-7/)

</comment></comment></comment></comment></comment></comment><comment><USER18> Sort of related story: we needed a scripting language able to run on an x86 RTOS type of architecture compiled with msvc and looked into CPython because, well, Python is after all quite a nice language. After spending a considerable amount of time to get it compiled (sorry, don't recall all the issues there, but main one was that the source code assumed msvc == windows which I know is true for 99% of cases but didn't expect a huge project like CPython to trip over) it would segfault at startup. During step-by-step debugging it was astonishing how much code got executed before even doing some actual interpreting/REPL. Now I get there might not be a way around some initialization, but still it simply looked too much to me and perhaps not overly clean either. Moreover it included a bunch of registry access (again, because it saw msvc baing used) which the RTOS didn't have in full hence the segfault. Anyway we looked further and thankfully found MicroPython which took less time to port than the time spend to get CPython even compiling. While not a complete Python implementation, it does the job fur us, and it gets away with startup/init code of just something like 100 LOC (including argument parsing etc). Yes I know it's not a fair comparision, but still, the difference is big enough to, at least for me, indicate CPython might just be doing too much at startup and/or possibly spend time on features which aren't used by many users and/or possibly drags along some old cruft. Not sure, just guessing.

<comment><USER19> [http://boo-lang.org/](http://boo-lang.org/)

<comment><USER20> Context?

</comment></comment></comment><comment><USER21> Mercurial's startup time is the reason why, for fish, I've implemented code to figure out if something might be a hg repo myself.

Just calling `hg root` takes 200ms with hot cache. The equivalent code in fish-script takes about 3. Which enables us to turn on hg integration in the prompt by default.

The equivalent `git rev-parse` call takes about 8ms.

<comment><USER22> Wow, that's quite a difference.

But 8ms is still too slow for me. :) I implemented the Git recognition code myself in my own prompt using the minimal amount of FS operations [1], and it renders in 5 ms from start to finish, including a "git:branch-name/47d72fe825" display.

[1] [https://github.com/majewsky/gofu/blob/master/pkg/prompt/git....](https://github.com/majewsky/gofu/blob/master/pkg/prompt/git.go)

<comment><USER23> (I work on Git in my copious free time)

One of the reasons git-rev-parse takes slightly longer than your implementation is that you just unconditionally truncate the SHA-1 to 10 bytes. E.g. run this on linux.git:

        git log --oneline --abbrev=10 --pretty=format:%h |
        grep -E -v '^.{10}$' |
        perl -pe 's/^(.{10}).*/$1/'

You'll get 4 SHA-1s that are ambiguous at 10 characters, this problem will get a lot worse on bigger repositories.

Which is not to say that there isn't a lot of room for improvement. The scope creep of initialization time is one of the things that tends to get worse over time without being noticed, but Git unlike (apparently) Python makes huge use of re-invoking itself as part of its own test suite (tens of thousands of times), so it's naturally kept in check somewhat.

If you have this use-case I'd encourage you to start a thread on the Git mailing list about it.

</comment></comment></comment><comment><USER24> This is truly a problem. Even more so if you host your application on a network directory. Loading all the small files takes ages. I really wish there would be a good way to compile the whole application with all the modules into one package once you're ready to release. I really wish the creators of Python would have given such use-cases more consideration.

Edit: I'm aware that there are solutions that put everything a program touches into a kind of executable archive. A single file several hundred Megabytes in size. I've tested it. It doesn't really pre-compile the modules. The startup time was exactly the same.

<comment><USER25> Nuikta ([http://nuitka.net/](http://nuitka.net/)) already does that and much more:

\- it compiles your program and make it stand alone so you can distribute just the exe

\- it makes it start faster

\- it makes it run faster

\- it's fully independant of the system python. Actually your system doesn't even need a python at all

I don't get why it's not used, it's very robust, compatible with 3.6 and on some of my script I get about x4 speed up just on start up alone.

<comment><USER26> First time I hear about this, and I've looked for alternatives to cxfreeze and its cousins in the past.

Any time I see something like this, I feel like I'm hearing about some homeopathic cancer cure. If Nuitka actually does what it says it does, it's solving a big recurrent problem for the Python community, so why is nobody talking about it?

<comment><USER25> That's the question I'm asking.

Not only it's a beautiful tool, but the author has been quietly and steadily working on it for 8 years. Compatibility is the number one goal.

The guy has a lot of rigor and humility, so maybe communication suffered ?

<comment><USER27> hg allows loading modules at runtime. maybe thats a problem.

<comment><USER25> I doubt it, since you can pass pass manually a list of all modules you want nuikta to embed with --recurse-plugins=MODULE/PACKAGE

<comment><USER28> But the problem is that you can't know the list you need at embed time for hg, because extensions are arbitrary python files discovered at runtime.

</comment></comment></comment></comment></comment></comment></comment><comment><USER29> Here's what has worked for me:

1\. Don't do that. Either write the driving app in Python or write the subprocesses in an ahead-of-time compiled language. Python's a great language but it's not the right tool for everything.

2\. Be parsimonious with the modules you import. During development, measure the performance after adding new imports. E.g., one graph libraries I tried had all its many graph algorithm implementations separated into modules and it loaded every single one of them even if all you wanted to do was to create a data structure and do some simple operations on it. We just wrote our own minimal class.

<comment><USER30> > Don't do that. Either write the driving app in Python

Even if you write the driver in Python, you don't necessarily want to call the program you're testing in the same process. You might want independent launches of a command-line tool, so that you test the same behavior people get when they run the tool. Otherwise, your test suite might trip over some internal state that gets preserved from run to run in ways that command-line invocation wouldn't.

<comment><USER29> Good point, but I didn't mean to sound specific to testing apps. I just meant, in general, write big apps using Python top-down and something precompiled if you must spawn lots of external processes.

</comment></comment></comment><comment><USER31> The slow startup combined with the general lack of interest of the Python ecosystem to try to find a solution for distributing self contained applications was the biggest reason we ended up writing out CLI tool in something else even though we are a Python shop.

I'm really curious why there hasn't been much of a desire to change this and it even got worse as time progressed which is odd.

<comment><USER32> This is disappointing to me too, but I think there are some problems baked in to the language that make it hard.

\- Imports can't be parsed statically.

\- Startup time has two major components: crawling the file system for imports, and running all the init() functions of every module, which happens before you get to main(). The first is only fixable through breaking changes, and the second is hard to fix without drastically changing the language.

The import code in CPython was a mess, which was apparently cleaned up by importlib in Python 3, through tremendous effort. But unfortunately I think importlib made things slower?

I recall a PyCon talk where as of 3.6, essentially everything about Python 3 is now faster than Python 2, EXCEPT startup time!

This is a shame, because I would have switched to Python 3 for startup time ALONE. (As of now, most of my code and that of my former employer is Python 2.) That would have been the perfect time to address startup time, because getting a 2x-10x improvement (which is what's needed) requires breaking changes.

I don't think there's a lack of interest in the broader Python community, but there might be a lack of interest/manpower in the core team, which leads to the situation wonderfully summarized in the recent xkcd:

[https://xkcd.com/1987/](https://xkcd.com/1987/)

FWIW I was the one who sent a patch to let Python run a .zip file back in 2007 or so, for Python 2.6 I think. This was roughly based on what we did at Google for self-contained applications. A core team member did a cleaner version of my patch, although this meant it was undocumented until Python 3.5 or so:

[https://docs.python.org/3/library/zipapp.html](https://docs.python.org/3/library/zipapp.html)

The .zip support at runtime was a start, but it's really the tooling that's a problem. And it's really the language that inhibits tooling.

Also, even if you distributed self-contained applications, the startup time is not great. It's improved a bit because you're "statting" a zip file rather than making syscalls, but it's still not great.

In other words, I have wondered about this "failure" for over a decade myself, and even tried to do something about it. I think the problem is that there are multiple parts to the solution, the responsibility for these parts is distributed. I hate to throw everything on the core team, but module systems and packaging are definitely a case where "distributed innovation" doesn't work. There has to be a central team setting standards that everyone else follows.

Also, it's not a trivial problem. Go is a static language and is doing better in this regard, but still people complain about packaging. (vgo is coming out after nearly a decade, etc.)

I should also add that while I think Python packaging is in the category of "barely works", I would say the same is true of Debian. And Debian is arguably the most popular Linux package manager. They're cases of "failure by success".

<comment><USER33> > The import code in CPython was a mess, which was apparently cleaned up by > importlib in Python 3, through tremendous effort. But unfortunately I think > importlib made things slower?

AFAIK importlib is entirely written in Python and kinda portable across Python implementations, while previously most was C code. It's not surprising something gets slower when written in Python.

> Also, even if you distributed self-contained applications, the startup time > is not great. It's improved a bit because you're "statting" a zip file > rather than making syscalls, but it's still not great.

PyQt applications on Windows typically take two or more seconds before they can do _anything_ , including Enterprise's favourite start-up pastime, splashscreens. Except maybe if you rolled your own .exe wrapper that displayed the splash before invoking any of the Python loading.

That's really, really poor in the age of 4 GHz CPUs from the factory, RAM big enough to fit multiple copies of all binaries on a PC and SSDs with at the very least tens of thousands of IOPS.

<comment><USER32> Yeah the time it takes is really mind-boggling if you think about it. I recently had occasion to run Windows XP in a VirtualBox on fairly underpowered Macbook Air.

It not only installed really fast, but at runtime it was fast and responsive! And so were the apps! Virtualbox recommends 192 MB of RAM for Windows XP, and it works fine. Amazing. Remember when everyone said Windows was slow and bloated?

On the other hand, I tried compiling Python 2.7 on a Raspberry Pi Zero, which is probably around as fast as the machines at the time of XP (maybe a little slower). This was not a fun experience!

Actually I just looked it up, and the Pi Zero has 512 MB of RAM. So in that respect it has more power. Not sure about the CPU though... I think I ran Windows XP on 300 Mhz computers, but I don't remember. Pi Zero is 700 Mhz, but you can't compare clock rates across architectures. I think they're probably similar though.

\---

FWIW I think importing is heavily bottlenecked by I/O, in particular stat() of tons of "useless" files. In theory the C to Python change shouldn't have affected it much. But I haven't looked into it more deeply than that.

<comment><USER33> IIRC the foundation originally compared the RPi's CPU to a Pentium II running at 266 MHz, which seems about right to me.

IME/IMB startup is almost always CPU bound (to a single CPU thread, of course). Note that the Linux kernel also caches negative dent lookups, so these "is there something here?" stat()s will stay in the dentry cache.

</comment></comment></comment></comment></comment><comment><USER23> Best out of 5 times on my Debian testing laptop for a "hello world", in order of worst to best:

        ruby2.5:     83ms (-e 'puts "hi"')
        python3.6:   35ms (-c 'print("hi")')
        python2.7:   24ms (-c 'print("hi")')
        perl5.26.2:  8ms  (-e 'print "hi"')
        C (GCC 7.3): 2ms  (int main(void) { puts("hi"); })

<comment><USER34>
      $ time ruby --disable-gems -e 'puts "hi"'
      hi

      real    0m0.009s
      user    0m0.008s
      sys     0m0.000s

<comment><USER35> Sure, two can play that game. Let's add `-S`, which disables the site module, to the Python invocations.

        perl ........... 0m0.012s
        siteless py27 .. 0m0.018s
        gemless ruby ... 0m0.021s
        siteless py36 .. 0m0.025s
        siteful py27 ... 0m0.034s
        siteful py36 ... 0m0.049s
        gemful ruby .... 0m0.089s

</comment></comment></comment><comment><USER36>
      in the temple of tmux
      for the cult of vi
      we sit and wait
      for venv to activate

</comment><comment><USER37> Given it is known how slow Python at starting up, I am puzzled why Mozilla continue to use it in build scripts. Perl is just as portable but starts up like 10 times faster.

<comment><USER8> I wrote the linked post and maintain the Firefox build system. The reason is that in 2018 (and for the past 10 years honestly) and it is far easier to find people who know Python than Perl. Python is essentially the lingua franca in Firefox land for systems-level tasks that don't warrant a compiled language. As I said in the post, Rust will likely infringe on Python over time due to performance, lower defect rate, long-term maintenance advantages, etc.

<comment><USER38> >As I said in the post, Rust will likely infringe on Python over time due to performance, lower defect rate, long-term maintenance advantages, etc.

Indeed. As hg is moving to use more and more Rust:

[https://www.mercurial-scm.org/wiki/OxidationPlan](https://www.mercurial- scm.org/wiki/OxidationPlan)

<comment><USER39> indygreg knows because he is the author of that wiki page. :)

[https://www.mercurial- scm.org/wiki/OxidationPlan?action=info](https://www.mercurial- scm.org/wiki/OxidationPlan?action=info)

<comment><USER38> I didn't realize that so thanks for pointing it out. But sometimes I also comment for third parties reading along to pick up some info as well.

</comment></comment></comment></comment></comment><comment><USER40> Naive question: If the startup time matters because you're imposing that startup time hundreds or thousands of times - why not remove the startup time?

I'm saying, use the emacs model. Start hg with a flag so it simply keeps running in the background while listening on a port. Run a bare-bones nc script to pipe commands to hg over a port and have it execute your commands.

This isn't a new problem, nor is it even a new solution. No complete re-write of the interpreter or the tool required.

Anyways, that's my 2¢

<comment><USER41> There's a paragraph in the OP about how they've actually done this:

> Mercurial provides a `chg` program that essentially spins up a daemon `hg` > process running a "command server" so the `chg` program [written in C - <USER2> > startup overhead] can dispatch commands to an already-running Python/`hg` > process and avoid paying the startup overhead cost. When you run Mercurial's > test suite using `chg`, it completes _minutes_ faster. `chg` exists mainly > as a workaround for slow startup overhead.

Just like this isn't what the usual `emacs` command does (it's `emacsclient`), it isn't what the usual `hg` command does either. There are some disadvantages to this solution and some assumptions it makes, which have apparently led the Mercurial maintainers to conclude, like the Emacs maintainers, that it won't work as the default. Hence the desire for solutions that will.

</comment></comment><comment><USER42> I hate to admit it but it's partly why I don't use clojure (pardon the side- topic) more. I can't bear the boot process and the overall cost.

Python is free to tinker, and all similar interpreters are joyful to use. Anything else is probably better for heavy duty jobs environments.

<comment><USER43> I feel the same way about Clojure. For a LISP, where interactive development via the REPL is supposed to be one of the value-add of the language, it falls completely short in that aspect. They even have entire libraries and design patterns (Component, etc.) to work around the issue, but I find it ridiculous that your entire program structure is dictated by the fact that the REPL boot up time is too damn slow.

</comment></comment><comment><USER44> Knock, Knock, who's there? ---- Long Pause --- Java!

</comment><comment><USER45> Python is great for prototyping or even real apps if performance isn't so critical. However, more than once I've found myself in the situation where I wrote a bunch of Python code and then end up starting that code up from another app, just like the thread discusses and I immediately feel like this is an anti-pattern.

What's even more annoying is that my Python code usually calls a whole lot of C libraries (OpenCV, numpy, etc.) So it's like this: app->OS process->python interpreter->my python code->C libraries. That just really feels wrong so I'd like two things:

1) better/easier path to embed python scripts into my app e.g. resident interpreter

2) some way of passing scripts to python without restarting a new process, this may exist and I'm unaware

</comment><comment><USER46> Startup time has also been the biggest gripe I have with Julia so far. Otherwise it's a truly fantastic language to work in. I wasn't able to put the `__precompile__()` function to good use it seems - the time it takes to execute my program didn't change at all for some reason. Or maybe it's not actually the startup time that caused the problem, but the time it took to perform file IO. Anyways my program now takes even much longer time to startup than the Python equivalent (though it runs much faster once started), which is a real disappointment.

<comment><USER47> precompile doesn't store native compiled code. Though I know from talking to the compiler developers that this is high on the 1.x list. It's an annoyance but at least it has a clear solution in sight.

</comment></comment><comment><USER48> Truly solving this problem is difficult, but you can hack around it with a zygote process to remove a substantial amount of overhead, in exchange for RAM. While this is generally more of win for server processes, you can see it applied to a CLI proof of concept:

[https://github.com/msolo/pyzy](https://github.com/msolo/pyzy)

</comment><comment><USER49> I agree Python's startup time is too slow. But one trick you can use to improve it some is the "-S" flag, which skips site-specific customizations. On my Ubuntu system it brings Python 3.6 startup time down from 36ms to 18ms for me; still not great, but it helps.

The drawback is this may screw up your Python environment, not sure how easy it is to work around it if it does.

</comment><comment><USER50> Proposed solution: steal undump from emacs. [https://news.ycombinator.com/item?id=13073566](https://news.ycombinator.com/item?id=13073566)

Perhaps it would be possible to read in the source files, compile them, and preserve an image of the state immediately before reading input or command line.

<comment><USER51> I'm pretty sure Python 3 already does this, and that's what the __pycache__ directories it creates when running a command are for.

<comment><USER52> Those are only bytecode. It helps a little but you still have to load the file from the filesystem and run it on import.

</comment></comment></comment><comment><USER53> I was kind of amazed how penalized a script could be by collecting all its "import" statements at the top. Once somebody's command couldn't even print "--help" output in under 2 seconds, and after measuring the script I told them to move all their imports later and the docs appeared instantly.

</comment><comment><USER54> I'm a long time python user, but never really peeked under the hood. However, I have a few ideas.

Optimized modules loading: maybe loading a larger 'super' module would be faster than several smaller ones? For example a python program could be analyzed to find it's dependent modules, and then pack all these into a 'super' module.

Once the python program executes, it would load the single 'super' module and hopefully bypass all the dynamic code which each module runs when imported to load up.

As mentioned previously, this is just off the top of my head and would certainly warrant more investigation/profiling to confirm my hypothesis.

</comment><comment><USER55> I'm pretty sure it's too late by now for Python, but I've had some success with compiling C-based interpreters [0] to C; that is, generating the actual C code that the interpreter would execute to run the program. That way you can reuse much of the interpreter, keep the dynamic behavior and still get nimble native executables.

[0] [https://github.com/basic-gongfu/cixl#compiling](https://github.com/basic- gongfu/cixl#compiling)

</comment><comment><USER56> Should be able to hot boot the VM with the right tooling. You can reuse HPC "checkpoint" code from supercomputing environments as a generic hammer for Python/Ruby/JVM. Some Russians figured out how to do it in userspace without a kernel mod: [https://criu.org/Main_Page](https://criu.org/Main_Page)

</comment><comment><USER57> People here comment about how python is slow, but even fast/slow is I'll defined in my opinion. You don't see people hacking tensor flow (generally) in native languages to speed it up, they just enable CUDA. I'm imagining fast definition is limited to massively parallel server workloads with io.

</comment><comment><USER58> Reminds me of buildout. It's awful piece of software. We used in previous Flask project, and a simple flask shell takes 3 minutes to start. If you type `import` in CPython shell it will literally freeze for a few seconds. Because it injects one sys.path for each packages specified!!!

</comment><comment><USER59> I'm just curious why more people don't make use of chg to avoid the mercurial startup time. It seemed to solve it for me - are there drawbacks?

<comment><USER60> At a guess: They didn't hear about it (keeping your ears open is a cost not everyone wants to pay). They don't want to bother with setting it up. They don't want to bother with maintaining it (even if it's as simple as reinstall every time you get a new computer).

<comment><USER59> That's fair. My experience with it so far has literally just been aliasing hg to chg. It performed all the magic in the background for me.

</comment></comment></comment><comment><USER61> A recent article in ACM Queue included an off-hand remark that Go's compile time is often faster than Python's startup time. Just sayin'

</comment><comment><USER62> Would it be feasible to keep a set of Python interpreters around at all times and use a round robin approach to feed each already-on interpreter commands then perform an interpreter environment cleanup out-of-band after a task is complete?

<comment><USER63> Or just use the operating system's `fork` system call?

There's also nailgun for Java which sounds like it works a little differently: [http://martiansoftware.com/nailgun/](http://martiansoftware.com/nailgun/)

<comment><USER64> I guess a fork()'ed process triggers copy-on-write behavior in the kernel once the process starts running. So that's latency (the copying) you could still optimize away.

<comment><USER65> You might want to measure it before you optimize it! Oftentimes I find that forks where I don't write much are quite inexpensive, with little COW action.

<comment><USER64> You could be right.

Also, could it be that Intel's cache hierarchy plays some really smart tricks behind the scenes, to make this fast?

</comment></comment></comment></comment></comment><comment><USER66> Slowness is the elephant in the room in Python land. It's like everybody has decided to cover their eyes in front of this massive pachyderm. A massive delusion

<comment><USER67> Delusion? I don't think many cover their eyes. More likely they've come to accept that for their use cases the performance is good enough and the convenience gain well worth it.

</comment></comment><comment><USER68> It's weird to see someone make this pitch when C systems software development regularly requires us to try and shave off microseconds. Millisecond delays mean you've already fucked up.

</comment><comment><USER69> For use cases where performance is important, using an interpreted (implementation of a) language is a bad idea.

There are many great reasons to use Python, but execution speed is not one of them.

<comment><USER24> But what else do you switch to?

Imagine you import tons of modules which often are only available in Python. This gets you going really quickly with your project and it runs very smoothly. Transferring this to C++ would probably take so long you won't even finish to find out before you run out of funding.

I have hopes that Rust or some descendant of Rust will get us there in maybe 10 years but in the meantime it would be better to get Python up to speed as good as possible.

<comment><USER70> For some use cases, Go might be a good alternative to Python. It's performant, yet simple and readable and it has a great ecosystem.

<comment><USER71> I find laying out a go project with dependencies to be miserable though. I write a lot of go and python code and why on God's green earth did google decide to handle dependencies by having you effectively git clone a library and then have these huge tree deps subdirectories and so on I don't know. It's mess. I vastly prefer the approach of either having one canonically designated folder to install dependencies to where each dependency is a top level directory that can be scanned or having them all stored in a project folder relative to root of the workspace similar to node_modules then the current mess.

Drives me nuts. Look at this layout if you don't know what I'm referencing:

[https://golang.org/doc/code.html#remote](https://golang.org/doc/code.html#remote)

Yes I have used dep

And yes I have all kinds of shortcut commands for navigating my Go workspace

But look at this canonical example From the docs and it's easy to see this is a giant mess that is utterly unnecessary. No other language I've used has had such a gross problem with dependency layout. It also leads to gross import strings.

It's one of my biggest criticisms of Go to be honest.

Thst and it's reliance on environmental variables that have to be set perfectly in order to actually do anything (thank god for direnv [https://direnv.net/](https://direnv.net/))

<comment><USER70> Isn't what a dependency folder like node_modules pretty much what the vendor folder is to go. Have you tried using a dep mangement tool like glide before?

<comment><USER71> Thst is one I haven't tried I tried a few others whose names I forgot because most alt managers seemingly stopped development but this looks active. Thank you!

I now would shift my argument to the fact that Google should really just adopt this as their standard if it works as advertised.

Link for those who haven't seen it:

[https://glide.sh](https://glide.sh)

</comment></comment></comment></comment></comment></comment><comment><USER72> IIRC CPython devs reject performance-related patches if they cause the code to become "less readable".

>> I believe Mercurial is, finally, slowly porting to Python 3.

I just gave up on Mercurial since it didn't let me push to BitBucket nor to an Ubuntu VPS via SSH.

For better or worse, Git just works.

<comment><USER73> I'm confused, since my daily workflow is pushing to Bitbucket via hg and ssh.

<comment><USER74> My work is considering switching to Git mostly because we think adopting Bitbucket will force us to. Is that not true? I'd love some reasons to stay with hg...

<comment><USER75> Bitbucket was originally for hg though. Why would switching to git be better?</comment></comment></comment></comment>


The Great Paper Caper - <AUTHOR> http://www.gq.com/long-form/the-great-paper-caper

<comment><USER2> Interesting read. What he did was immoral, but I think everyone can agree he is brilliant.

<comment><USER3> How more immoral is it to illegally counterfeit money than it is to legally counterfeit money? Law and morality often have no bearing to one another these days, even if they should.

The Fed steals 2-4% of your savings every year, even if you don't live in the US by exporting inflation worldwide. How is this guy any worse? I might argue he's less bad because $200mm is a drop in the bucket compared to 2% of the world's savings, which is on the order of trillions.</comment></comment>