Ask HN: Push Payments to Debit Cards? - <AUTHOR>

Hey guys I was looking into a solution that would allow me to pay my app users with a push to debit card type of service.
I know stripe does thishttps://stripe.com/blog/send-money-to-debit-cards as well as balanced payments https://www.balancedpayments.com/push-to-card
I was wondering if anyone knew anyone else who would do this? I reached out to balanced payments but their solution won't be ready till the second quarter to start. Stripe seems like the best choice but I just wanted to reach out and see if anyone had any suggestions or know anything?
Thanks. <comment><USER2> [http://www.quora.com/Which-credit-card-processors-will- enabl...](http://www.quora.com/Which-credit-card-processors-will-enable-you- to-push-money-to-a-customers-credit-debit-card-as-the-primary-transaction)

<comment><AUTHOR> Thanks WMF. I was looking for a service that would have this as an API or infrastructure behind it. I would rather not have to build out something entirely from scratch.

</comment></comment><comment><USER3> I'm interested in this as well. Are there any legal issues with sending people money on a large scale?</comment>


What's next for SemVer - <AUTHOR> https://words.steveklabnik.com/what-s-next-for-semver <comment><USER3> The absolute most important thing they could do is add a fourth number, at the front, that is meaningless - the vanity number. vanity.major.minor.patch. There has been so much wasted time, and incorrect versioning, just because people have not "felt" that breaking changes constituted "major" semver updates - just to then break many production systems due to semver ranges only working iff major, minor, and patch actually attempt to mean anything.

This is a losing battle -- we are never going to convince people of this. So let's just acquiesce. Let's allow the first number to advance at whatever rate you like. It's the "human" portion of the version number. That way, everything to the right of it, no one will care about and can serve its actual purpose. Ultimately this isn't even a new idea, its just "the PR version" vs. "the build number" just like native apps have had for ages.

<comment><USER4> Not sure if it's a good idea.

A better idea in my opinion would be to have 2 versions:

* an "API" version that specifies technical changes.

* a public version more representative of big milestones in the project.

It's roughly what is done in the C/C++ ecosystem with sonames.

But I agree, that's indeed a flaw of SemVer IMHO. According SemVer, even if you are adding tons of new cool and useful features, while maintaining compatibility of existing one, you are not supposed to increase the major number.

From a "social" point of view, it gives the feeling a project is not really active.

And even from a technical perspective, bumping the major in case of significant changes can actually be a good idea. When I see an X.0.0 release, I kind of expect major changes inside the project, and even if the project remains compatible, significant changes generally bring bugs and instability, in some context it might be a good idea to wait for a version like X.1.4 to remain stable.

With semver, 1.42.0 -> 1.43.0 can be a small add on of one method or tons of new methods. And even 1.2.2 -> 1.2.3 can be an entire rework of the internals, while the API stays exactly the same.

Don't get me wrong, SemVer is great, it has enabled far less painful dependency management and better tooling, but some human and qualitative aspects are lacking in this versioning scheme.

<comment><USER5> I can't find any reference to not being able to increment the major version if you maintain compatibility. Only the reverse is specified, you must increment it if it is broken. [https://semver.org/#spec- item-8](https://semver.org/#spec-item-8)

<comment><USER6> I'd think the problem with bumping the version number if you're remaining compatible is that it signals to the end-user that you're not compatible. i.e. when I start using 2.x the constraints I'll put in my list of libraries is < 3.0 (because otherwise my project could randomly break upon 3.0's release, depending on whether it's compatible or not).

</comment></comment></comment></comment><comment><USER7> I'm excited to see this. I would hope to see someone from the Go community get involved in the maintainer group, though. Russ Cox's philosophy on versioning is now baked into the Go language pretty deeply, and it's (at least partly) motivated by problems with semver, real or perceived:

> _Where I think semver falls short is the idea that "having to bump major > versions" is a step that will make you "think through the impact of your > changes and evaluate the cost /benefit ratio involved." Quite the opposite: > it's far too easy to read semver as implying that as long as you increment > the major version when you make an incompatible change, everything else will > work out. The example shows that this is not the case._

(from [https://research.swtch.com/vgo-import](https://research.swtch.com/vgo- import), one of the treatises on go versioning)

Despite this criticism, Go is _sort of_ still using semver, at least as a way to compare string version identifiers. I think it's fair to say that Go users form a significant community using semantic versioning. It would be a mistake to leave them out of RFC discussions on the topic.

<comment><AUTHOR> I didn't previously reach out to anyone from Go specifically because they do not really use SemVer. As a group, we're open to including others than this initial roster, but didn't think it would be particularly relevant to their interest. If that's incorrect, someone please get in touch!

<comment><USER7> The language implements semver directly in the Go language source: [https://github.com/golang/go/blob/47d5a4433513c3870a9994b50e...](https://github.com/golang/go/blob/47d5a4433513c3870a9994b50ed4fa2af628c742/src/cmd/go/internal/semver/semver.go)

This is used to resolve module loading, according to this logic: [https://github.com/golang/go/blob/aab0b704d88b7dad10c767ad94...](https://github.com/golang/go/blob/aab0b704d88b7dad10c767ad941619d59887203c/src/cmd/go/internal/modload/query.go#L17-L35)

<comment><AUTHOR> Thanks for the citations! Reading this, it's semver-ish for sure; that said, the "ish" is exactly why we're doing this. I'll try to figure out how to get in touch with someone.

<comment><USER8> Hi Steve, I've looked at this relatively carefully, and I believe that Go does 100% follow the semver spec (modulo any bugs in the implementation).

However, while that happens to be my personal belief, I am not any type of world-class semver expert, nor am a semver maintainer, and hence always happy to learn more (especially from, say, an actual semver maintainer such as yourself ;-)

I think you are saying "-ish" in your comment above at least partly based the two chunks of linked code above, including this comment there:

      // This package follows Semantic Versioning 2.0.0 (see semver.org)
      // with two exceptions. First, it requires the "v" prefix. Second, it recognizes
      // vMAJOR and vMAJOR.MINOR (with no prerelease or build suffixes)
      // as shorthands for vMAJOR.0.0 and vMAJOR.MINOR.0.

It is true that Go treats the leading "v" as a requirement for VCS tags to be recognized as encoding a semantic version. That said, as far as I am aware, requiring the "v" prefix for VCS tags is allowed by the semver spec, and it might be possible to draw a distinction between a semver version (which does not include a "v") vs. how a given semver version is encoded into a VCS tag (where an encoding is allowed to include a "v", as far as I am aware). For example, there is an FAQ that was added a few years ago to 'master' at github.com/semver/semver that seemed to address this:

 _prefixing a semantic version with a "v" is a common way (in English) to indicate it is a version number. Abbreviating "version" as "v" is often seen with version control. Example: git tag v1.2.3 -m "Release version 1.2.3", in which case "v1.2.3" is a tag name and the semantic version is "1.2.3"._[1]

I am also aware that a leading "v" vs. no leading "v" for VCS tags can trigger some impassioned debate, so I might regret posting this comment. ;-)

Regarding the second piece from that comment above -- that particular Go package does have the ability to parse something like "v1" or "v1.2" (without the three integers required by semver). However, the result is not interpreted as a valid semver version by the overall 'go' tool. For example, a VCS tag such as "v1.2" that the 'go' tool finds on a git repository will _not_ be interpreted as a semantic version (because it does not have the required three integers). However, being able to parse something like "v1" or "v1.2" is used for example as part of a version query mechanism. For example, you can do something like "go get firstname.lastname@example.com" as a way of asking "please give me the highest available release version in >= v1.2.0 and < v2.0.0". In other words, it is a short-hand for a particular type of version query, which I would guess would not be in violation of the current semver 2.0 spec? If interested, there is some more information about that query mechanism (which is called a "module query") in the Go doc[2].

Finally, here is a snippet from the Go doc[3] stating semver is used (and there is a link to [https://semver.org](https://semver.org) in that section as well):

 _The go command requires that modules use semantic versions and expects that the versions accurately describe compatibility_

I wouldn't be shocked if you have a different take on some or all of what I said above, but wanted to at least share my personal understanding...

[1] [https://github.com/semver/semver/blob/master/semver.md](https://github.com/semver/semver/blob/master/semver.md)

[2] [https://golang.org/cmd/go/#hdr- Module_queries](https://golang.org/cmd/go/#hdr-Module_queries)

[3] [https://golang.org/cmd/go/#hdr- Module_compatibility_and_sema...](https://golang.org/cmd/go/#hdr- Module_compatibility_and_semantic_versioning)

<comment><AUTHOR> Hey hey!

So, primarily I am saying "ish" for two reasons: one, as I said below, I mis- understood how versions were actually used within Go. There's a lot of stuff out there, and keeping the three or four implementations I _do_ know well is tough enough. Second, I don't want to say, definitively, that any particular implementation "does not implement Semver", because I think the spec is deficient enough that it's really hard to say in general.

On to your specific points:

> That said, as far as I am aware, requiring the "v" prefix for VCS tags is > allowed by the semver spec

This is true, SemVer says nothing about VCSes.

> However, being able to parse parse something like "v1" or "v1.2" is used for > example as part of a version query mechanism.

Right, so that's what I thought this was getting at, and the general "range" concept isn't in the spec, so all of that is fine, spec speaking.

However, that doc comment describes it a bit differently than you are. It describes them as version numbers. So it's possible that the doc comment is a bit mis-leading, maybe. That's very reasonable! This is why we need to clean up the spec text.

<comment><USER8> Makes sense. I guess the way I would summarize it is I believe the end-to-end Go system is 100% compliant with semver 2.0 spec. (And that is true to my knowledge _even though_ there happens to be an internal-only 'semver' package that _also_ contains Go-specific functionality related to semver, and that internal 'semver' package both must be used properly and is used properly to keep the overall end-to-end system semver spec compliant, including proper use of functions like isSemverPrefix(version) and similar to differentiate between what is allowed in a semver VCS tag vs. what is allowed for what is effectively a range-based query, etc.).

Again, happy to learn otherwise...

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> For an interesting take on versioning, see the Spec-ulation keynote by Rich Hickey [1]. It specifically touches upon SemVer being itself versioned.

[1]: [https://www.youtube.com/watch?v=oyLBGkS5ICk](https://www.youtube.com/watch?v=oyLBGkS5ICk)

<comment><USER10> This is an excellent talk, i found myself fully agreeing and kind of relieved that i'm not the only one thinking that you shouldn't break backwards compatibility and if you do, then rename the API. I'd only add that for libraries renaming "FOO 1" (with "1" often implied) to "FOO 2" isn't a good new name since someone else might want to pick up the slack and continue developing a "FOO 2" that is backwards compatible.

One of my favorite parts of the talk is the X.Y.Z changes in semver essentially translating (from a user perspective) to imscrewed.dontcare.dontcare :-P.

Sadly it seems that this talk has been largely ignored. I'd like everyone who provides any library, framework or anything else that other people are supposed to build on - especially on the perpetually broken Linux desktop - <USER2> watch this talk.

<comment><USER11> SemVer bit starts at [https://youtu.be/oyLBGkS5ICk?t=825](https://youtu.be/oyLBGkS5ICk?t=825)

Have to agree, excellent talk.

</comment></comment></comment><comment><USER12> This is great!

Natalie Weizenbaum and I used SemVer when we created pub, the package manager for Dart. It was a good choice, but we definitely ran into some subtle problems over the years. Briefly:

* Like Steve notes, Semver doesn't specify any syntax and semantics ranges, constraints, or other "groups of versions". Every package manager just rolls its own thing, hopefully in a mostly compatible way.

* SemVer used to specify an ordering for build versions ("1.2.3+1" versus "1.2.3+2") and then later removed that. Pub still uses the older semantics because a package manager has to do _something_. If the user requests "foo 1.2.3" and both "1.2.3+1" and "1.2.3+2" exist, what's the package manager supposed to do? Pick randomly? So we still technically use the older SemVer spec where this ordering is defined.

* It's hard for a package to transition into a period of experimental development. Let's say you maintain foo which has lots of users. Foo 1.2.3 is out and lots of people are using it. You want to start working on a major, breaking revision to foo. You need to do that in a way that doesn't spontaneously break existing users, but you also want to ship something so that you can get feedback from early adopters.

Right now, the way you do this is by shipping a series of pre-release versions like 2.0.0-dev.1, etc. That works, but users are often confused that a range like ">=2.0.0" does _not_ include those dev versions, nor does "<2.0.0". It doesn't feel to me that SemVer gracefully handles the idea that previously stable packages may go through periods of instability. Pre-releases are sort of the imaginary numbers of SemVer where they don't really fit the natural linearization of version numbers.

Many projects now have "channels" of development where users can indicate whether they want to be on "stable", "dev", "bleeding edge", etc. When it comes to pre-releases and experimental versions, I think users generally want to indicate whether they want to be on the bleeding edge or not. Today, at least in pub, they have to indicate that using some carefully crafted version constraint that only includes pre-releases. It would be nice if that was a more direct way to express that.

(The fact that SemVer doesn't define ranges compounds this. I think our interpretation of ranges mostly follows npm.)

<comment><USER13> One could argue that if you are about to majorly break back compat, that you should just rename your project. Even calling it `foo2` is better than pretending it's `foo 2.0`. It may slow adoption of your new work, but at least it's staying honest.

<comment><USER14> SemVer doesn't seem to distinguish between "major" and "minor" breaking changes, instead saying all breaking changes are major version changes. So making new major versions cheap seems like a good way to document any breaking changes, even if they are "minor" breaking changes.

</comment></comment></comment><comment><USER15> 1) In the Windows ecosystem almost every version has 4 parts. (e.g. file headers in exe, dll, assembly etc.) So it's impossible to use semver which only allows 3 parts. This seems like a major unnecessary roadblock.

2) IMO packaging tools should have fields to indicate the versioning scheme they follow, for example:

{ "version": "1.0.0", "version_scheme": "[https://semver.org/spec/v2.0.0.html"](https://semver.org/spec/v2.0.0.html") }

{ "version"="19.02", "version_scheme": "[https://calver.org/"](https://calver.org/"))

{ "version"="126.96.36.199", "version_scheme": "[https://altver.example.org/winver/22.214.171.124.html"](https://altver.example.org/winver/18.104.22.168.html"))

{ "version"="18.104.22.168", "version_scheme": "custom")

Currently the tools pretend to know that "all" versions follow the same scheme. But in reality they just don't and never will.

Maybe also allow a prefix instead, but make it explicit and unmistakable (i.e. not "v1.0.0", but "semver2-1.0.0").

</comment><comment><USER16> The map is not the territory. Similarly, the semantic version number is not the API.

Why are we relying on fallible humans to examine an API and encode an assertion about its backwards compatibility into a number? Further, why are we then handing this (often dubious) assertion of backwards compatibility to machines to depend on?

Can't we create a set of rules/tests that will _actually execute_ the API to determine whether it's backwards compatible? Why aren't we doing that more?

<comment><USER17> Figuring out how to make a computer decide that one API is compatible with another would be bloody hard. I hesitate to say 'impossible' because today, I can subscribe to much of what I deemed impossible two decades ago.

If you can figure this out, it would be an absolutely amazing feat. Best of luck!

<comment><USER18> Fully typed functional languages (Haskell for example) have this ability.

<comment><USER19> I'm not sure that is correct. A function that takes no arguments and returns 2 and a function that takes no arguments and returns 3 both have the same type signature. I'm not super strong on the theoretical side of computer science, but, my understanding is that the only general solution to determining if our function returns 2 or 3 (or some other integer) is equivalent to the halting problem.

<comment><USER18> Yes, obviously. The API is the same though. The interface.

<comment><USER19> The signature of the interface is the same, yes. But, if the behavior changes it can still break users. I'm fully in support of strong typing and SemVer and all that good stuff - I was just responding to your comment which I interpreted to mean that a language like Haskell has a full solution to the problem of breakage, which I don't believe it does. It's certainly in a better position that many other language in the types of breakage it can detect just from the function signatures, however.

</comment></comment></comment></comment></comment></comment><comment><USER20> I want to see some support for alpha, beta, pre-release, and snapshot versions.

The problem is that everyone does it differently, and build systems have a hard time distinguishing these non-production-ready versions. For example, I type "mvn versions:display-dependency-updates" and I get a suggestion that I use "junit 4.13-beta-2".

<comment><AUTHOR> The SemVer spec does have pre-release versions [https://semver.org/#spec- item-9](https://semver.org/#spec-item-9)

But, I guess you're saying that certain kinds of names in the pre-release space should have defined semantics?

<comment><USER21> "-SNAPSHOT" suffix already has defined semantics in Maven world, which would make sense to replicate elsewhere, since it's very convenient for CI. Alfa, Beta etc. could be used to improve multi-environment CD (eg UAT envs could accept RCs, production - only stable versions and so on).

</comment></comment></comment><comment><USER22> I'm not sure if it's already in the spec, but it would be nice to get a very hard definition of when to call a project 1.0.0.

It would also be nice to see it evolve to handle projects like Docker and Ubuntu, where Docker said goodbye to SemVer because it couldn't fit their use case.

Also, SemVer seems too optimistic about major updates. After being in the industry for several decades, breaking changes are depressingly rare, and SemVer says basically nothing about this, or the fact that major versions include bugs-as-api-contracts.

<comment><AUTHOR> [https://semver.org/#how-do-i-know-when-to- release-100](https://semver.org/#how-do-i-know-when-to-release-100)

</comment></comment><comment><USER23> Is there something like "test-driven versioning"?

It would be rather simple: "Here is the testsuite for our (public) api. We support the use cases tested, but no more.

If you want your use case covered, please submit a testcase.

If we change an existing test, we increase the major version.

If we add tests, we increase the minor level.

If the test suite remains the same and we only change the code, it's a patch- level increase."

This could be verified automatically, increasing trust.

<comment><AUTHOR> SemVer talks a lot about compatibility, but doesn't define it. Each project is supposed to on its own. You can just say this for any given project, and it would be compatible.

<comment><USER11> I prefer Rich Hickey's definition of compatibility [1]:

A function is compatible if it requires no more than it did, and provides no less than it did in the previous version. A library is compatible if it provides no fewer functions, and requires no additional context (transitive dependencies) to use it, and all of its functions are compatible as defined.

When you wish to require more, or provide less, you make a function with a new name.

[1] As linked elsewhere: [https://youtu.be/oyLBGkS5ICk?t=1189](https://youtu.be/oyLBGkS5ICk?t=1189)

</comment></comment></comment><comment><USER24> I would like to work similarly to how WHATWG makes HTML a "living standard", with rolling releases tracked by [http://caniuse.com/](http://caniuse.com/)

instead of working so hard to name changes with oversimplified numbers, our change management systems could do all this for us with metadata

see also [the package manager we need]([https://gist.github.com/dominictarr/7c3319a0b0bad22bde2c](https://gist.github.com/dominictarr/7c3319a0b0bad22bde2c)), [Towards "annex", a Fact Based Dependency System]([https://speakerdeck.com/markhibberd/towards-annex-a-fact- bas...](https://speakerdeck.com/markhibberd/towards-annex-a-fact-based- dependency-system)), ["Towards "annex", a Fact Based Dependency System" by Mark Hibberd]([https://www.youtube.com/watch?v=JjYAnBhF2JU](https://www.youtube.com/watch?v=JjYAnBhF2JU))

<comment><USER7> Frankly, I think browser compatibility is one of the nastiest, trickiest bits of web development. I'd want to find any way to avoid that sort of thing. The "living standard" model merely describes underlying complexity without resolving any of it, which puts the burden on individual developers. I think we can do better.

<comment><USER24> > The "living standard" model merely describes underlying complexity _without > resolving any of it, which puts the burden on individual developers_.

Don't blame people when it's their tools that are at fault.

"If you want to teach people a new way of thinking, don't bother trying to teach them. Instead, give them a tool, the use of which will lead to new ways of thinking."

\- Buckminster Fuller

<comment><USER7> I'm confused. Can you explain which tool is at fault, and for which problem, here?

<comment><USER24> coding needs to be more automatic

api changes aren't such a big deal

they only seem like it now, because _the editor_ is the primary coding tool

code should be generated from content. roughly speaking:

\- [https://en.wikipedia.org/wiki/Code_as_data_(disambiguation)](https://en.wikipedia.org/wiki/Code_as_data_\(disambiguation\))

\- [https://en.wikipedia.org/wiki/Code_generation_(compiler)](https://en.wikipedia.org/wiki/Code_generation_\(compiler\))

\- [https://en.wikipedia.org/wiki/Round- trip_engineering](https://en.wikipedia.org/wiki/Round-trip_engineering)

\- [https://en.wikipedia.org/wiki/Model- driven_engineering](https://en.wikipedia.org/wiki/Model-driven_engineering)

further, I believe:

> code is unrefactored config

> config is immature content

> content is unrefined code

</comment></comment></comment></comment></comment><comment><USER25> Interesting to see this topic coming up. I just recently complained to Microsoft about how Azure Artifacts' "Universal Packages"[1] only supports strict Semantic Versioning with no extras (no additional version number components past patch and no commit hash). This is problematic because there are still several versioning schemes that use four version components, most notably Windows binaries. I was really interested in using Universal Packages to store application binaries as part of a set of new CI pipelines I'm architecting; unfortunately, I have to resort to NuGet, which still supports <major>.<minor>.<patch>.<buildid>+commithash syntax.

I realized the issue was with Semantic Versioning itself. The spec should be changed to mention that any number of additional dotted version components are allowed after <major>.<minor>.<patch> but that they have no semantic meaning. This would help address legacy versioning concerns like mine while not being too prescriptive.

Only having three version components is additionally problematic because there is no place for CI build IDs in the version. Imagine I'm working in a bug-fix branch called 1.5.4. I might have several commits that all trigger CI pipelines. I don't want to increment the patch component until I actually merge the bug fix into master, but I still want my CI system to generate unique version numbers per build. There currently doesn't seem to be a way to do that without being forced to use pre-release identifiers (e.g., 1.5.4-beta.1, -beta.2, etc.), which unnecessarily complicates the CI pipeline.

[1]: [https://docs.microsoft.com/en- us/azure/devops/artifacts/quic...](https://docs.microsoft.com/en- us/azure/devops/artifacts/quickstarts/universal-packages?view=azure- devops&tabs=azuredevops)

<comment><USER26> Timely for me as well. Another reason for adding an additional version number component is this: I use an open source library that I need to fork. Since the library uses the `patch` number component, it's hard to release forked versions to NPM without having conflicting patch numbers.

For example, if the library is at `0.15.4` I don't want to release my updates as `0.15.5`, `0.15.6` etc. because when the main library updates to `0.15.5` we have conflict/confusion. Instead, I'd like to `0.15.4.1` to indicate my `.1` update to `0.15.4`. When they update to `0.15.5` I can merge and release a version at `0.15.5.1`

<comment><USER25> Exactly.

In my case, I'm using Azure Pipelines' "build ID" concept as a globally-unique integer. Given any fourth version component (e.g., the _125_ in 22.214.171.124) I can identify the exact unique build that generated that artifact. No two builds across all artifacts will ever share the same build ID. It's a great way of finding the build of any artifact even if all I have is a version number.

</comment></comment></comment><comment><USER27> I'm surprised there's no involvement from PHP yet (although, who would be involved?)

PHP's renaissance has a lot of thanks to give to Composer - our own package manager that has been fantastic.

Composer should definitely be represented in the maintainer's group.

<comment><AUTHOR> [https://github.com/semver/semver/issues/495](https://github.com/semver/semver/issues/495)

</comment></comment><comment><USER28> What's not new: Semantic Versioning is an excuse to break existing code...or more cynically, SemVer is a promise to break your code. The author cites Ruby and Javascript as examples of semantic versioning's success. But trying really hard to not break existing code is a core principle of both languages development. MINASWAN [0] and principal of least surprise in Ruby . A consensus formal specification process for ECMAScript [1] (aka "JavaScript").

[0] [https://en.wiktionary.org/wiki/MINASWAN](https://en.wiktionary.org/wiki/MINASWAN)

[1] A breaking change looks like this, [https://itnext.io/a-tiny-disastrous- ecmascript-change-fadc05...](https://itnext.io/a-tiny-disastrous-ecmascript- change-fadc05c83e69)

<comment><USER29> It's neither an excuse nor a promise to break code. It's merely a version numbering scheme that makes it explicit which versions will break code. If you're using semver and you don't want your next version to break code, go ahead and release it with an increased minor version. Major version increases aren't for you then.

<comment><USER28> Semantic versioning reduces the social cost of breaking changes. It makes them more likely.

<comment><USER30> Suppose we replaced Semantic Versioning with any other approach (of your choice) that still disclosed the same amount of information about whether API was identical, had an additive change, or had a breaking change. Would you object to that model too ?

If so, it seems like you'd be effectively arguing that authors should simply not disclose important changes to the API. It is hard to imagine that position being defensible.

If not, why do you consider Semantic Versioning to be more hazardous?

<comment><USER28> Additive changes aren't a problem. If I use the method 'spam' from library '{foo : most_recent}' adding 'eggs' to 'foo' doesn't require me to edit my code to {foo : 16.8.37}. The most recent version still works.

If the library author decides to do something that breaks calls to 'spam' (or 'eggs', etc.) then a protocol that calls the new library 'foobar' (not 'foo') means '{foo : most_recent}' still works. I can use continue using 'foo.spam' if it meets my needs.

\+ And there's no problem if I want to add '{foobar : most_recent}' to the code if I want to use 'foobar.eggs'.

\+ And I can even use 'foobar.spam' alongside 'foo.spam'.

\+ And my build system can be simplified to '{foo}, {foobar}' as the ordinary case. With language like '{foo : 23.8.13}' only needed for rare cases. For example when my code depends on the effects of an obvious bug that was patched.

The premise of semantic versioning is that breaking changes for whatever reason are ok. And look, that's fine. "It's your library. It's free. You don't owe me anything." I just prefer not to rely on people who will change "print x" to "print(x)" based on divine revelation rather than the effect it has on everyone who relied on the documentation saying _use 'print x'_. And no automated migration tools because semantic versioning.

</comment></comment></comment></comment></comment><comment><USER31> In my usage of SemiVer I felt it lacked what I call a "gestalt number" which would be the first number, representing broad changes to the design which (almost always) necessarily include API changes. But API changes can happen within the same "gestalt".

<comment><USER32> Why does a SemVer spec have to deal with that? Why not have a "gestalt" number/indicator for purely informal use, but keep the SemVer part as is?

<comment><USER33> That's essentially a 4-part version number. For most projects it's over the top and almost no-one used or uses 126.96.36.199 version numbers. I've seen libraries use "library3 1.6.2" [1] but it feels very strange and confusing.

[1] [https://github.com/etorreborre/specs2/releases](https://github.com/etorreborre/specs2/releases)

</comment></comment></comment><comment><USER34> SemVer is great, I think people try to read too much into it. It's there so that your automatic dependency system wont pull a breaking change without you first checking if it actually did break anything. I think waiting for semver major releases is an anti-pattern. I rather have a very small breaking change, rather then a bunch of breaking changes at once. So if you use SemVer, don't be afraid to do a Semver Major, avoid it if possible, but don't make it such a big deal. Ohh and don't use it for marketing, use a cool name instead, like NameOfProduct "ninja tiger" (instead of v9.2.2)

</comment><comment><USER35> Nice to see this happen. I'm glad that it will get more attention and some possible improvements.

One thing I'd like to see is a specified practice for versioning projects that "wrap" another project where the "inner" project's version is significant to the user.

For example, a long time ago I wrote a Ruby gem to expose jQuery's templating library to the Rails asset pipeline. My gem had a version, but each version also had the jQuery Templates source code packaged within it, which of course was also versioned. The most useful thing I could do for the gem's version was to have it match the version of JQT that it contained, but if I needed to make a new gem release to change some of the Ruby wrapper without changing the packaged version of JQT, there was no "official" or universally obvious way to denote this. I could add an extra number on the end, but it wasn't obvious that this extra number wasn't part of JQT's version just from looking at the four-part number. I believe some package managers use the name of the system in the "wrapper version" to denote this, e.g. "1.0.0.ubuntu4" but even that is lacking because you really need two represent two different fully semantic versions. I'd be great to see a solution for this specified.

<comment><USER36> That may be more interesting than it looks like at its face, as that give a tie-in to declare dependencies on system libraries.

</comment></comment><comment><USER37> What are SemVers current flaws? Maybe I missed a link in the article.

<comment><AUTHOR> Here's the largest, IMHO: none of the matchers are in the spec. It purely talks about versions themselves, not how you specify ranges. It's only really got half of the important stuff in it!

<comment><USER38> Could you elaborate on this?

<comment><AUTHOR> "Matchers" or "ranges" (even the terminology is split up thanks to the lack of a spec!) are not defined in semver. It only defines how to version a particular software artifact, it cannot answer the question "does version 1.2.3 fit the constraint > 1.0.0?"

Does that make sense? The "> 1.0.0" part is undefined. That includes things like "is that space required there?", and so is an interoperabilty issue.

<comment><USER39> I don't see what's ambiguous in that.

> does version 1.2.3 fit the constraint > 1.0.0?

I would answer, yes, 1.2.3 is greater than 1.0.0. I can't see how anyone would interpret it differently.

<comment><USER12> That's an easy one. Try:

Does "<=2.3.4" include "2.3.3-some.prerelease"?

Justify your answer. :)

<comment><USER40> At least with Gitflow and Githubflow workflows, it is my understanding that we change the version number in the release/feature branch.

With that understanding in mind (regardless of it matching your understanding, I mean), then clearly, "2.3.3-some.prerelease" is in fact less than "2.3.4", or it would have been named "2.3.4-some.prerelease".

Therefore, I believe it is clear that workflow must necessarily be considered to understand SemVer.

<comment><USER12> So if a user has the version constraint ">=2.3.3 <2.3.4", should they get that pre-release version?

<comment><USER40> This is covered explicitly in the SemVer specification section 11[0].

The answer depends on if there exists a released "2.3.3" version, in which case, the pre-release is not selected, because pre-release versions have less priority than release versions.

The spec provides the following example:

1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-alpha.beta < 1.0.0-beta < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0.

[0] [https://semver.org/#spec-item-11](https://semver.org/#spec-item-11)

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER41> What I don't like about SemVer is that all breaking changes look exactly the same.

That's why I like to use SemVer with a small twist - a fourth number prepended to the regular three: A.B.C.D.

`B`, `C`, and `D` are the regular SemVer Major, Minor, and Patch. But `A` is "Generation" \- something you increment only for _fundamental_ changes to the codebase .i.e. 22.214.171.124 is a different generation of the product than 184.108.40.206, while 22.214.171.124 is the same generation, only with some incompatible APIs.

<comment><USER42> Rust sort of already has this, though implemented in a worse way. They use 0.[major/minor].[minor/patch]. When the library author wants to indicate that their library is stable, they go to 1.0.0 and promise that they'll never make a breaking change.

The problem with that approach is that you never know if a change is major or minor, since many projects use the 0.A.B differently. Even if projects were standardized, you're missing some information compated to A.B.C or A.B.C.D.

<comment><AUTHOR> This is an additional thing I'd really like to gain clarity on in the spec itself. Given that ranges have no definition currently....

Oh, also

> they go to 1.0.0 and promise that they'll never make a breaking change.

That's not what 1.0 means; 2.0 can be produced with breaking changes.

<comment><USER43> I think GP means that as a cultural thing, the Rust/Cargo ecosystem has a strong aversion to breaking changes, and as a result tends not to do anything that would require a bump to 2.0.

</comment></comment></comment></comment><comment><USER44> What's the punchline? This post doesn't detail at all what's next for SemVer. What changes are coming? What's wrong with SemVer that needs fixing?

<comment><USER45> They clearly have some deficiencies in mind; an example or two would go a long way toward helping this post.

<comment><AUTHOR> I think that getting all of the implementations on the same page is possibly even _more_ valuable than any changes. For example, since ranges aren't in the spec, npm and Cargo treat "1.2.3" _very_ differently. Since we can't break either ecosystem, that means that that specific aspect will have to be either implementation-defined or "choose one of these two ways," but we can at least discover where these kinds of cases are, and possibly fix them, or properly spec them, as needed. This matters a lot for interoperability. For example, Notion is a Rust project that manages JavaScript stuff; for this to work, Rust's implementation will need a "do it like npm" flag. This is really tough when it's not even spelled out anywhere what the possibilities are.

The lack of this causes more issues than real actual problems with the text as it is today, IMHO. For example, the Go code above requires a "v", which is not in the current spec, but some implementations handle it, some don't. Driving these things out is important!

</comment></comment></comment><comment><USER46> Here's a hypothetical feature of all package managers:

* for all stable packages requested by the user, _always_ upgrade to the most recent stable version that has the same major number as the one currently installed

Can npm, apt, cargo, guix, etc. safely do this presently for the set of packages that use semver?

If the answer is no, then what is the value to using semver?

<comment><USER47> If I consume library X at version 1.2.3, develop my application, and then library X is upgraded to 1.2.4, should I pull in this new version? What about if it is upgraded to 1.3.1?

The answer is maybe, but there are a few questions that need answers first:

1\. The author of the library has stated that there are no backwards incompatible API changes. What is my confidence in this assertion?

2\. In asserting that there are no backwards-incompatible changes, they have asserted that there is no new behavior as the result of bugs. Am I sure of that?

3\. By stating my application is using v1.2.3, I am stating that I am _correctly_ using the published API. Am I? For example what if an API returned a sorted list, but that was not a guarantee of that API?

4\. Are there benefits to upgrading that make the risks above worthwhile?

<comment><USER46> > 1\. The author of the library has stated that there are no backwards > incompatible API changes. What is my confidence in this assertion?

Prefix all of my responses with, " _If_ semver is to be meaningful to the developer."

1\. Your confidence should be so high that an ongoing pattern of mismatch between statement and reality spells the end of that software project.

2\. With the same level of confidence as above, yes.

3\. I need clarification. Are you saying that a major version >= 1 of a dependency you rely upon in production returns a sorted list, but then a new bugfix version now outputs an unsorted list?

4\. Performance, bugfixes, the ability to use new features with impunity.

And _of course_ my replies are idealistic and cannot work in a large number of extant development contexts.

But let me throw the question back-- given your righteous skepticism, what in the world is the point of having _three_ different version numbers if you cannot trust any of them without further in-depth research? If a bugfix can break one of your dev's assumptions about a poorly documented interface, what meaningful hierarchy could there be among the dots of semver? Why not have a single incrementing number?

edit: clarification

<comment><USER6> > 3\. I need clarification. Are you saying that a major version >= 1 of a > dependency you rely upon in production returns a sorted list, but then a new > bugfix version now outputs an unsorted list?

Not OP, but I believe they are saying that the API happens to return a sorted list (either for their particular inputs, or in general), but this was never a guarantee of the API (and was presumably not covered by tests), and so the package maintainers changed the implementation to no longer return a sorted list (maybe they found a faster algorithm).

I have to say that I think this is a good case for having tests that are tied to particular versions, like in this comment: [https://news.ycombinator.com/item?id=19138853](https://news.ycombinator.com/item?id=19138853)

Or perhaps in this specific case the user of the library will add the test (to their own codebase) that checks the output is always sorted, and it will fail to upgrade to a version where this isn't true. In general I'd be pretty happy to see people write tests for the subset of behavior they require from a library, although I appreciate that's potentially a whole lot of work, particularly in ecosystems without property-based testing.

<comment><USER46> Attempting to extend the "semver skeptic" OP's original questions: _where_ would such information be represented in semver?

Whatever probabilities you give to it appearing in the major/minor/bugfix slot, the point is that the skeptic dev cannot safely assume that it will appear in the major slot. If it doesn't appear in the major slot, that means there is a mismatch between what the package author/maintainer considers to be backwards-compatibility and what the author assumes it is. Given the sad state of FLOSS documentation, this is a common type of mismatch where bugs hide.

That means the skeptic dev has to pin to some version that they've tested, only upgrading for a critical bug or some feature valuable enough to warrant the increased time building tests for the new version and hunting for "mismatch" bugs.

I think that skepticism characterizes a critical mass of npm usage. If that is indeed the case then those devs do not trust the minor and bugfix slots to convey meaningful backwards-compatibility information. There might as well be a single incrementing version integer with a changelog at that point.

Edit: clarification

</comment></comment></comment></comment></comment><comment><USER48> The clause allowing 0.x releases to contain breaking changes needs to go. Also, the first release should be v1; so many projects never leave 0.x despite being depended on by millions of projects and everytime they push a new minor all hell breaks loose as people have to manually look at the changelog and decide to upgrade.

<comment><USER49> _the first release should be v1_

If this were done, how would v1 be any different than v0 is now? Isn't it nice that people who want to avoid all that hell breaking loose can simply not use a package that hasn't gotten to v1? Why would you want to do away with that?

</comment></comment><comment><USER50> The link to [https://github.com/orgs/semver/teams/maintainers/members](https://github.com/orgs/semver/teams/maintainers/members) 404s for me -- I'm guessing due to permissions?

<comment><AUTHOR> I left a comment below about the problems; the link apparently 404s because of GitHub, see the footnote I added next to the link. Sorry about that! Not under my control :/

</comment></comment><comment><USER51> Since for SemVer "to work, you first need to declare a public API" maybe tools should require a file VERSIONING (or SEMVER?) containing something like "<PRODUCT> follows SemVer 18.104.22.168 <[https://semver.org/spec/v2.0.0.html>](https://semver.org/spec/v2.0.0.html>). The public API of <PRODUCT> consists of ... We pledge to change the major version when we make incompatible changes to this public API. ... We consider the following as incompatible changes to the public API: ... We do not consider the following as incompatible changes to the public API: ..."

</comment><comment><USER52> I was curious about the empirical evidence supporting the notion that SemVer is "better" than what came before it. Does anyone have any pointers.

</comment><comment><USER53> ComVer is what's next.

[https://github.com/staltz/comver](https://github.com/staltz/comver)

<comment><USER54> I quite like that scheme. It saves all the debate about whether to increment the 2nd or 3rd number.

The downside to such a scheme in terms of production releases (rather than libraries), is I have where 2.3 might need a fix for production, while 2.4 is being worked on. So the natural release number is 2.3.1. With ComVer you are forced to make your production fix 2.4 and rename the WIP version 2.5 which can be a pain depending on your JIRA/CI setup etc.

</comment></comment><comment><USER55> I love how elm enforces semver versioning.

When you write a library, it will detect API changes and upgrade the version accordingly.

</comment><comment><USER56> Very exciting to finally see semver get some traction in one way or another! A very welcome decision and announcement, also supported by Haacked's post[1].

[1]: [https://haacked.com/archive/2019/02/11/semver- collective/](https://haacked.com/archive/2019/02/11/semver-collective/)

I have been quite active around the semver repo for a few years after 2.0.0 was released initially and participated in a lot of significant discussions that, to this day, still remain unresolved without either affinity to them or a stated rejection. After many years of seeing the same questions and suggestions pop up with no progress in sight, I just slowly lost hope and once issue updates started piling in to close issues at the initiative of @jwdonahue (thanks, by the way) without advancing any discussion or subject, I just unsubscribed. I am really exited about the things that are to come and will gladly try to participate in discussions to the best of my abilities.

Currently, there are quite a few things that SemVer needs to decide on (not necessarily implement), but imo the important points are:

1\. How are versions selected? You mentioned this in the thread already, but this point is hugely lackluster about the current situation. See issue #205 for the biggest discussion to date and many cross-references.

1.1 Build metadata, especially in the context of version selectors, kind of "just exist". They serve mostly as a "catch all" for various things you might want to annotate a version, but there are no rules or use cases for them.

2\. (Automatic) development/post-release builds come up frequently when trying to version snapshot builds, however Semver does not have a solution for this. See issue #200.

3\. Clarify the "limbo" that are 0.x.y releases. To me, the spec is clear in its semantics for this, but the wording can be misleading to some. Basically, in the current situation semver just does not apply to 0.x releases _at all_ (for the purpose of providing a "stable API") and this should be clearly communicated, especially for version selection.

4\. Put more emphasis on what a "Public API" is, either in the spec itself or the FAQ. Many people have problems with this abstract concept and ask about it on the issue tracker.

5\. Another topic that comes up frequently is demand for a "generation" or otherwise comparable fourth number to declare a kind of reboot for versioning. Some package managers have this (pacman, for example) and it is sparingly used, but there is no stance on whether or not SemVer would be interested in such an optional component or whether this is to be handled elsewhere. See issue #213.

</comment><comment><USER57> My biggest problem with SemVer is that determining what is and isn't a breaking change can be very hard in modern languages. Worse yet, if the library is invoked from a different language, some things that weren't breaking can become so (and vice versa). And sometimes, said different language is actually a new version of the same language.

For example, in C#, is adding a new overload to a sealed class a breaking change? The general consensus is "no", but doing that can easily break existing code by making it ambiguous (if there was only one implicit conversion or upcast to consider before, but now there are two).

But what about adding a _new_ method, with a completely different name - surely that's not breaking (to remind, our class is sealed, so there's no descendants; and in C#, methods don't override base class unless explicitly requested)? It wasn't - until C# 3.5 added extension methods. Now, every time you add a new method to your class, you are potentially breaking someone else's extension method with the same signature - and worse yet, it will compile silently, and just call a different thing.

Okay, what about adding a new property? Can't be overloaded, and there's no extension properties (yet; they might still be added). There's still a problem: when calling an overloaded method with a lambda - also added in C# 3.5 - as an argument, one of the criteria for choosing an overload is that the body of the lambda is valid after substituting the types for that particular overload - this includes checking members. For example, suppose we have:

       class Foo {}

       class Bar {
          public void Frob() {}
       }

       class Program {
          static void Qux(Action<Foo> a) {}
          static void Qux(Action<Bar> a) {}

          static void Main() {
              Qux(x => x.Frob());
          }
       }

This compiles, because only Bar has Frob, and thus Qux(Action<Bar>) is the correct overload. But if Foo also gets some Frob at some future point, this code breaks - the overload is ambiguous now. In more complicated cases with other types involved, this can also result in a silent choice of a different overload.

But, on the other hand, if you were to use the same class from F#, it wouldn't be a problem, because it simply doesn't try to do this sort of type inference on top of overloading. So for F#, this isn't breaking at all... today. It might be tomorrow, though, as it was with the extension method example above (F# didn't add support for those for a while after C# did, so it wasn't broken by that initially).

I guess the TL;DR is that "breaking API change" is an oversimplification of the real thing - in reality, what's breaking depends on the entire toolchain _and_ the way the library is used by the client code, not just the API itself. So our criteria for a major semver increase simply aren't well-defined. And that needs to be done - separately for various languages due to different semantics - before any given version number can be more than just one person's guess.

<comment><AUTHOR> SemVer itself as a spec expects projects to define exactly what compatibility means, it's true. We went through all of these kinds of cases with Rust, for example, and defined what kinds of changes increase what kinds of version numbers. Given that it's based on language features, it gives a good basis for the ecosystem, but when you don't have that basis, yeah it's much much tougher.

<comment><USER57> Is there any kind of centralized place where "what does semver breaking change means for language X" can be aggregated? I think it can be useful, at least until this is something that language designers routinely include in specs.

But there's remains the question of what happens if a library is written in one language, but used from many others with different features that can make non-breaking changes breaking, and vice versa. One could say that the owner of the library is generally not responsible for breaking other languages, but in many cases the library is cross-language by design.

<comment><AUTHOR> Not currently. Rust has done this, but I'm not aware of other languages having done so.

</comment></comment></comment></comment><comment><USER58> I have a soft spot for Semitic Versioning [0] myself.

I don't know. Great artists know when to stop. I can't help but feel that this discussion crossed that line some time ago.

[0] [https://github.com/ajalt/fuckitpy#versioning](https://github.com/ajalt/fuckitpy#versioning)</comment>


A scheme to encrypt the entire web is working - <AUTHOR> http://www.wired.com/2016/04/scheme-encrypt-entire-web-actually-working/?utm_content=buffercffd2&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer <comment><USER2> And yet Wired is still being served over http.

<comment><AUTHOR> Very interesting observation.</comment></comment>


Show HN: Cloud Sharing of your Servers - <AUTHOR> http://CloudSlice.io <comment><AUTHOR> Hi, this is a variation of a previous product (still existing) that we built. It's a low cost, smart way to share server images that you store on our cloud. You can share servers with anyone, pretty much by emailing them a hyperlink. It would be great to hear thoughts and feedback from the HN crew!</comment>


Show HN: Moment – A Fun Way to Share Stories - <AUTHOR> https://moment.rocks

<comment><USER2> I guess the obvious question is how is this different from Snapchat (minus the fact Snapchat disappears after x amount of time)? Looks nice!

<comment><AUTHOR> Thanks, feroz1! In addition to the ephemeral messaging aspect that makes lightweight chatting fun, Moment also doesn't require any personal information to sign up and use, instead it uses a set of unique URLs for chat room access. And, the GIF-like messages embody a richer chatting experience that I think add a lot to the interactions you can have with friends through the app.</comment></comment>


Ask HN: Sales/BD folks, do you use any location analysis tools? - <AUTHOR> I've been working on developing tools for location analysis for the past few months and am looking to gauge interest to see if it's useful for sales/business development.<p>I'd love to hear your thoughts. <comment><USER2> You're creating figures for lifetime value? What are you basing on?

<comment><AUTHOR> Not really. More like we're just looking to see if any sales/BD folks actually use any location/mapping tools for their work.

I'd assume that lifetime values and sales forecasting would probably come from one's own data.</comment></comment>


The Little Printf - <AUTHOR> https://ferd.ca/the-little-printf.html <comment><USER2> " _It is; and because you 're the most familiar person with these fires, you get to only work on them more and more, until your employer hires someone else to cover your old job, the one you loved. If you care hard enough about your work to be the one doing the stuff everyone else hates, you're thanked by doing more and more of that work you don't like, until that's all you do. And then there's nothing left for you to enjoy._"

Sigh.

<comment><USER3> This stood out as the most realistic advice in the story to me. I've seen a lot of people stuck working on stuff they don't like just because they're the one who knows the most about it. The farther you go down that path, the harder it is to get away.

<comment><USER4> This one stood out to me as well. Like you, I agree that it's accurate. I've both seen and lived it.

The others bother me because they're essentially malicious caricatures. This one feels like a dose of sympathy.

<comment><USER5> Yeah. They're malicious caricatures, but as mentioned in the text, they're all different sub-personalities I've at least had at some point in my career.

Also do note that the text is heavily copying The Little Prince, to the extent that all characters are also malicious caricatures of adults, with only one of them actually being perceived positively, so I tried to play to the original.

</comment></comment></comment></comment><comment><USER6> _" The games people play, the roles and reputations they chase and entertain, the fleeting pleasure they derive from solving intricate problems, is all fun for a while. Ultimately though, if you do not solve anything worthwhile, if you forget about the people involved, it's never gonna be truly fulfilling."

"It is the time you have spent on your system that makes it so important", the man added, "and when you lost sight of why it made sense to spend time on it, when it became a game of pride, it caused more grief than relief.

"Developers have often forgotten this truth; If you lose sight of things, working on your system becomes its own problem, and the most effective solution is to get rid of the system, given it's the problem."_

This story is easily one of the best depictions of the software engineering profession I've ever come across. I originally got hooked on programming because of the thrill of solving "intricate problems"... but the fleeting nature of that thrill only becomes more and more apparent as time goes on. Fundamentally I think it's necessary to have a deeper motivation, a bigger "why" behind what you do, in order to sustain your satisfaction from day to day.

</comment><comment><USER7> A real gem. If you have time check out his other work:

[https://ferd.ca/](https://ferd.ca/)

Some of it is Erlang specific but other posts are quite general about distributed systems, overload handling, etc.

He is also the author of [https://learnyousomeerlang.com/](https://learnyousomeerlang.com/)

</comment><comment><USER8> Priceless if you read "the little prince"[1] - if not, stop now and invest the hour to do that first. I make all my (adult) guests listen to the audiobook when driving through Tarfaya, where the author was working as a pilot for French aerial post.

[1] [https://en.m.wikipedia.org/wiki/The_Little_Prince](https://en.m.wikipedia.org/wiki/The_Little_Prince)

<comment><USER9> There's also a musical! [https://www.youtube.com/watch?v=ViK7qT8TS5w#t=3m58s](https://www.youtube.com/watch?v=ViK7qT8TS5w#t=3m58s)

</comment></comment><comment><USER10> I know that this article is fiction, but there's a common misconception about the Dunning-Kruger effect presented in the story that is worth understanding. People think that Dunning and Kruger showed that less competent people think that they're more competent than more competent people think they are, but this is not what Dunning and Kruger showed, if you'll excuse the complicated sentence construction (the article linked below has a graph and explains this a bit better).

Actually, Dunning and Kruger simply showed that a person's estimate of their own skill tends to be _closer to the mean, plus a little_ than their actual skill level.

The Dunning-Kruger effect has probably been a bit distorted because we like the idea that thinking you are good at something is evidence that you are not, in fact, good at that thing.

See also "what the Dunning-Kruger effect is and isn't" \- [https://www.talyarkoni.org/blog/2010/07/07/what-the- dunning-...](https://www.talyarkoni.org/blog/2010/07/07/what-the-dunning- kruger-effect-is-and-isnt/)

<comment><USER11> Hm, that relationship is also what you'd get if people were perfect at estimating their own skill while the proxy being used by the researchers was imperfect, isn't it?

<comment><USER10> Well, you would get _broadly speaking_ a similar relationship. General error in the measurement would appear as a regression to the mean. However, Dunning and Kruger addressed this in their paper, directly, in section 4.1.3. The actual metric that the researchers measured was actual class rank and estimated class rank. Class rank is well-defined, and estimated class rank is also well-defined. That's not to say that class rank is a perfect choice of metric, but you _are_ asking students to estimate something which can be exactly measured.

The paper showed a difference in the size of the error between top and bottom quartile students that was larger than you would expect if it were just uniform error. When you think about it, this seems pretty damn obvious... if you are bad at something, you are also bad at knowing that you are bad at it. But if you are good at something, you have a more accurate understanding of your own skills. In short, if you're good at something, you probably know that you are good at it.

This is what goes against the popular notion of the Dunning Kruger effect. Many people seem to use Dunning Kruger as some kind of twisted justification for impostor syndrome, where you are not permitted to think you are good at something because it would paradoxically imply that you are not good at it. I say this, just because I know too many people suffering from impostor syndrome, and this incorrect impression of Dunning Kruger seems to be a contributing factor.

</comment></comment></comment><comment><USER12> I find it a little sad that he did not credit Antoine de Saint-Exupéry who wrote this story and created the drawings in 1943 after he crashed his aeroplane in the sahara desert and and met the real little prince there. Also, that the comments so far are very complimentary but it is not clear that the writers know the original story and I am most concerned that the warning about the Baobabs is not included!

      https://en.wikipedia.org/wiki/The_Little_Prince

<comment><USER5> because Le Petit Prince is one of the most translated and sold books ever, and that the title was a very direct reference, I just didn't feel it would be necessary to include a note saying it's based on the little prince story since it felt obvious to me. My bad.

<comment><USER13> You're right, it is obvious

</comment></comment></comment><comment><USER14> it took me until the 5th chapter to realize this is a parody of "The Little Prince"

</comment><comment><USER15> :,)

</comment><comment><USER16> I call my little Printf Snigl [0], but (s)he's been telling me the same story.

Can't do it any more, I have no more fucks to give about saving the world and awesome profits.

[https://gitlab.com/sifoo/snigl](https://gitlab.com/sifoo/snigl)</comment>


Saudi Arabia's Reckless Extremism - <AUTHOR> http://www.nytimes.com/2016/01/11/opinion/mohammad-javad-zarif-saudi-arabias-reckless-extremism.html <comment><USER2> TL;DR - written by Mohammad Javad Zarif, the foreign minister of the Islamic Republic of Iran.</comment>


Show HN: Scoopinion, The Weirdly Accurate Newsletter - <AUTHOR> https://www.scoopinion.com

<comment><USER2> so...this thing tracks my browsing habits?</comment>


Turkey bans YouTube - <AUTHOR>

<comment><USER2> This is going to complicate my last-minute application to Y Combinator...</comment>


GitLab Annex: Large Binaries with Git - <AUTHOR> https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/ <comment><USER2> What is the difference between GitLab Annex and git-annex?

[https://git-annex.branchable.com/](https://git-annex.branchable.com/)

Edit: Just found the relevant quote from the article

> _In GitLab 7.8 Enterprise Edition this problem is solved by integrating the > awesome git-annex._

I guess the repeated branding of "GitLab Annex" just seems a bit strange to me.

<comment><USER3> GitLab Annex is git-annex integrated in GitLab. Meaning you have the authentication and protection that GitLab gives you with git-annex.

</comment></comment><comment><USER4> I am trying to understand why putting large binaries under source control is a good idea in the first place. Unless you have a way to make sense of the diffs this will not help right? and to to make sense of the diffs, you need to understand the structure/format of the binary at least after you have the diff, or maybe even to make a useful diff, right?

<comment><USER5> Diffing is just one small part of version control. Keeping binary files in version control is the number one step away from messy file-name-based versioning.

Many design studios are forced to use project1.ver1.psd project1.ver2.psd project1.ver3.psd, etc and so on in order to version their files. Single psd files can be on the order of high hundreds of megabytes to low gigabytes for high resolution ready-for-press files.

Not being able to diff the files is not a problem from an organisational point of view. Of course, in an ideal world there would be diffing of large binaries in a way that makes sense, but thinking there's no use in versioning binaries is very short sighted.

<comment><AUTHOR> This exactly. We keep our game's asset in SVN at the moment only for the convenience of versioning. Artists are still locking files to ensure no conflicts are happening (and therefore no merges are needed).

It's much easier to naively do a repository checkout/update than manually detecting changes (or rolling your own solution using rsync or similar).

Especially when considering the game has over 100Gb of raw assets. SVN, Git or Perforce might not be the best tools for such a task but it works great.

</comment></comment></comment><comment><USER6> Duplicate of [https://news.ycombinator.com/item?id=9063705](https://news.ycombinator.com/item?id=9063705)

</comment><comment><USER3> GitLab engineer here, let me know if you have any questions.

We're quite excited about GitLab Annex and curious to hear what people think about it.

</comment><comment><USER7> Joey Hess mentioned this in the git-annex devlog: [http://git- annex.branchable.com/devblog/day_256__sqlite_conc...](http://git- annex.branchable.com/devblog/day_256__sqlite_concurrency_argh/)

</comment><comment><USER8> Most grownup software shops use Perforce precisely because it can handle large repositories with large amounts of non-source non-textual content. Maybe this will bring Git in the same league with the big boys.

<comment><USER3> We believe source control should be everywhere and git isn't great yet to work with large binaries. Git(Lab) Annex makes things much better for people working with these kinds of repositories.

We'd love to get feedback on it.

<comment><USER9> As I mentioned in another thread, efficient storage of these files is necessary, but not sufficient. Exclusive locking is a requirement for working with editable (but not mergeable) binary files in a team environment. It's the reason most games studios use Perforce or SVN instead of git. Tracking locks using a separate system does not scale, it needs to be integrated and enforced by the VCS.

PlasticSCM has a hybrid approach that is worth studying.

<comment><USER10> Woukd you mind explaining a bit more why locking is a requirement? It sounds more like an organisational issue rather than something _required_ in SCM.

Taking the game studio example, why would two developers/artists/whatever need to work on the same asset at the same time? Locking stops them from stepping on each other's toes, but it also means one has to wait for the other to finish their task before they can do anything.

<comment><USER9> Stepping in each other's toes is part of the game artist workflow. :)

Seriously, with the tens of thousands of assets in a medium to large game, the way they can be reused across the game, and the peculiarities of art and in- house tools, it's not uncommon for two artists to try to modify the same asset. Sometimes it's accidental, sometimes it's necessary, sometimes it is an oversight, sometimes it is an organisational issue as you say... but locking detects the conflict and prevents it from turning into wasted hours or days.

Relying on a separate tool to manage this issue is a notable increase in friction. Since the SCM handles changes and conflicts for source code (and thankfully allows merging), why wouldn't it do so for art?</comment></comment></comment></comment></comment>


Arizona Senate votes to accept tax payments in Bitcoin - <AUTHOR> http://fortune.com/2018/02/10/arizona-bitcoin-taxes/ <comment><USER2> I can't wait for the obsession with "Blockchain Technology" to final fizzle out for the few realistic non-groundbreaking applications to emerge.

Every "out of touch with tech except when some crazy new dangled "stuff" seems to be printing money" is coming out of the woodwork to get a slice... except they're the exactly people being fleeced to print this new money.

They don't understand the the reason adding blockchain to your name gives you a 100% bump in stock isn't because blockchain technology is revolutionary, it's because of people just as clueless as they are and just as scared of missing out.

<comment><USER3> It's just the dotcom bubble all over again

<comment><USER4> Tulips anyone?

[https://en.m.wikipedia.org/wiki/Tulip_mania](https://en.m.wikipedia.org/wiki/Tulip_mania)

</comment></comment></comment><comment><USER5> "the primary goal is to foster technological innovation in the state"

"sending a signal to everyone in the United States and possibly throughout the world that Arizona is going to be the place to be for blockchain and digital currency technology in the future."

So this isn't the solution to a problem, it's an advertisement.

<comment><USER6> An advertisement is a solution to a problem (a problem of people's behavior not being what you'd prefer it to be.)

</comment></comment><comment><USER7> > Includes a provision mandating that cryptocurrency payments be converted to > U.S. dollars within 24 hours of their payment

So there's a massive, planned liquidation period (Aka tax time) - this can't be exploited at all.

<comment><USER8> Tax season is almost 4 months long in the US.

<comment><USER7> My intuition was that taxes would be paid mostly near the deadline. I ran the numbers [0] based on IRS numbers to check this intuition. While it is a matter of degree, and it is definitely less pronounced than I initially thought, the last week sees roughly double the number of filers than the minimum week, with another peak in early February.

I'm not aware of a comparable comodity market where there would be a seasonal, forced sell-off to actually speak to the effects. I am, however, convinced that this at least presents an opportunity for adverse behaviors which could make this an untenable policy.

It also doesn't speak to that there are 8 months for the markets to prepare for exploit this period.

[0] [https://www.irs.gov/newsroom/2018-and-prior-year-filing- seas...](https://www.irs.gov/newsroom/2018-and-prior-year-filing-season- statistics)

Results (2017):

Week : Filers (Millions)

January 27 : 8.3

February 3 : 11.1

February 10 : 11.4

February 17 : 9.6

February 24 : 8.9

March 3 : 8.1

March 10 : 7.6

March 17 : 7.2

March 24 : 6.9

March 31 : 7.5

April 7 : 8.9

April 14 : 13.1

April 21 : 13.9

<comment><USER6> Tax _filing_ and tax _payments_ are not necessarily simultaneous.

</comment></comment></comment></comment><comment><USER9> Cool. Call me when I don't have to calculate capital gains taxes every time I spend Bitcoin.</comment>


Is downloading really stealing? The ethics of digital piracy - <AUTHOR> http://nzherald.co.nz/business/news/article.cfm?c_id=3&objectid=11432353

<comment><USER2> What are we supposed to be "stealing"? Information is a mental construct that humans use to make an easier sense of the world. It's like a component of PCA. Or some tree in a random forest classifier. It's not an independent object existing "out there". (Unless you believe in platonic objects: in which case you need to bring evidence to court to support your — apparently — religious beliefs.)

Information is just brains. And brains are already owned by minds/souls occupying/generated by them. The same goes for hard drives, DVDs, USB sticks, individual DNA molecules, etc. You can only "steal" information by stealing a brain or an actual USB stick. The same goes for "owning" it. (I will grant you that unauthorized access is a crime of trespassing, but in most cases copyright violations are not that.)

tldr: You can't just invent an imaginary category that you pretend is ownable, in pursuit of some social goal, and then threaten to put other people into a rape cage if they don't follow along with your fantasy. Well, you can, but that would make you an immoral bastard™.

[Analogy:

Look, imagine someone comes to court and says that he's suing his neighbor for blocking his driveway for an hour. He is asking for $1M damages. The judge asks what the damages are for, and the plaintiff answers: "For the death of my pet and resulting emotional suffering."

The judge asks: "Which pet?" The plaintiff says: "My imaginary invisible unicorn." How well do you think that would go?

Now imagine if the plaintiff admits that he actually doesn't even believe himself in the existence of the unicorn but just wants to invent his existence as a legal fiction to deter blocking driveways.

In case you didn't get it, this was all analogy for intellectual property. Downloaded from my imaginary platonic farm.]

</comment><comment><USER3> "That they can profit in this way provides an important incentive - aside from the intrinsic value of the productive activity itself - for them to engage in socially useful productive activity."

That's just it. This is a business model built on threat of violence and coersion that provides additional incentives over and beyond what content creators would get without it. Just because a business model is able to collect money - extortion, say, or confidence games - does not mean the model itself is ethical. The argument that we need to keep the existing incentives in place is much like arguments made by any other entrenched interests who are getting the benefit of some societal system or government program and want to continue it because - to hear them tell it - the repeal would make the sky fall.

Even if it leads to ever increasing absurdity like the USA endlessly extending the age of copyright to prevent works from falling into public domain, or the software patent race and sunk costs in buying porfolios, leading to speculative legal fishing for money.

The history of SCIENCE, open source software, and freely available wiki sites show that people will contribute content and create great and useful collaborative works even in the absence of monetary compensation and business models that coerce people into paying. In fact, even the question of ego / attribution seems to be answered - people don't have to get "top billing" in a movie or be listed as a book's primary author in order to contribute. And the result is often better than a commercial enterprise, for a variety of reasons. It evolves with time, is accessible from more environments, and leads to more wealth creation in the long run around it. The internet protocol suite, linux, webkit, wikipedia, publicly accessible science, all of it.

The human motivations are very well explained here:

[http://m.youtube.com/watch?v=u6XAPnuFjJc](http://m.youtube.com/watch?v=u6XAPnuFjJc)

It is somewhat ironic that anarcho capitalists decry government for using force and then ask for more of it to protect capitalism and property, in areas where a naturally occuring gift economy without coersion and which confers basic freedoms on all users has been shown to do better. This irony can be best seen in the way every anarcho capitalist switches from deontological arguments (what is moral in and of itself) that they use to criticize systems of government, to consequentialist (what the outcomes would be) as soon as the time comes to justify the use of force to maintain property and incentives of capitalism for something. "But think of all the great movies that wouldn't have been made because no one would have any incentive." Yeah, and think of all the crimes that would have been committed if no one would have government.</comment>


Interviewing as an Outsider: Becoming Visible in Tech - <AUTHOR> https://medium.com/@polymorphism/interviewing-as-an-outsider-how-i-finally-got-seen-in-tech-f50803166462 <comment><USER2> "People in the Valley get admired when they drop out of elite schools, but if you're not dropping out of Stanford, you don't get congratulated. It's all dependent on your demographic situation."</comment>


Falsificationism and Falsifiability - <AUTHOR> http://en.wikipedia.org/wiki/Falsifiability

<comment><USER2> Every time I stumble into a religion "vs" science debate, I tell the participants to look up this page (or at least this term). Falsifiability underlies the scientific method, and it's the element of 'Science' that's normally not appreciated or understood (at least by the kind of folk who have _that_ argument.</comment>


I'm a startup founder and I had sex with an investor - <AUTHOR> http://www.businessinsider.com/sexism-women-silicon-valley-tech-why-startup-founder-sex-investor-2017-7?utm_content=buffer7bf31&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer-ti <comment><USER2> So... I'll chime in if you guys want to talk to me.

First, I left unroll.me in 2012 so blame whatever ethics you want but I had nothing to do with that.

Second, this post is about female responsibility in the ecosystem. Clearly you guys, except one of you, didn't read the whole post.

I posted it on Medium. I know no journalists. The editor in chief of BI reached out to me and said it was one of the best pieces she has seen on this topic, can they run it.

So ask away. I'm all ears

<comment><USER3> thank you for clarifying that you were not connected to the abuses of the unroll.me platform.

My anger is directed at the responsible parties, and I apologize if I have offended you.

If what you worked hard to build was twisted and abused by others then that is truly a painful thing.

may you find peace

<comment><USER4> You may want to edit your original comment.

</comment></comment></comment><comment><USER5> Found this very interesting and brutally honest in the details around the circumstances of the consensual night out (she mentioned several times that the meeting being purely business was all made up in her head) and her equally honest opinion on what happened to Cheryl Yeoh (she doesn't agree that was assault).

Pity how a personal account with no clear villain or wrong doing is not getting much attention as it deserves, she is being very honest here and leaving herself vulnerable to being called both a SJW and a traitor to other women. I appreciate her honesty and complicated analysis of her own motivations and subsequent reactions.

<comment><USER6> Being moderate on these topics seems to invite scorn from all "sides". Can't win, really.

<comment><USER7> Nope. And there's a large and growing group who self-silence (or even.. censor) because being moderate attracts as much vitriol as taking a side - because BOTH sides attack.

I don't get why saying "Many issues are complex" is some kind of war crime.

</comment></comment></comment><comment><USER8> So some how the investor is still painted as at fault here? It's not like this dude was her boss and by her in admission the guy was lording "I'm an investor" over her head. Got to be honest, this piece feels like extended click bait.

<comment><AUTHOR> i totally agree on this.

</comment></comment><comment><USER9> Wait, wait, wait. She was married and a mother when she had consensual sex with this investor, and explicitly one who would not be investing in her company? And her husband didn't leave her? Maybe I'm old fashioned, but I was too distracted by that to pay attention to the rest of the article.

Bottom line is if you don't want to cheat on your spouse, don't hang out with the opposite sex one on one--especially if there's alcohol in the mix.

<comment><USER10> I had the same thought, but then I reminded myself that there have been several years between 2011 and today. She's married now (for less than a year), not when the events described take place.

</comment></comment><comment><USER11> This seems like a good and thoughtful article to me. I don't know why some people are hating

</comment><comment><USER12> In Hollywood it is the casting couch.

In Washington and on Wall St it is business as usual.

So why should a startup be any different?

</comment><comment><USER3> Unroll.me: You stole peoples mail spools and sold them to the opposition. Selling peoples lyft receipts to Uber -- you should be ashamed of yourself and the legacy of violation that you left in your wake. To me -- this is one of the worst abuses of peoples trust that they granted to you. You are scum.

I have no sympathy for any of your antics or tales.

<comment><USER13> Thanks for bringing this up. Too often, Valley gossip, masquerading as news, is merely which people are friendly with reporters/bloggers and the resulting pieces are usually lazy journalism and one-sided.

<comment><USER2> I commented on this below...</comment></comment></comment>


Why Are Drought Balls Black Instead of White? - <AUTHOR> http://space.io9.com/why-are-drought-balls-black-instead-of-white-1724040253

<comment><USER2> A somewhat disturbing quote from the abstract of one of the journal articles linked to in the article:

 _High light intensity promoted the production rate of bromate, but decreased its total generation amount due to acceleration of chlorine decomposition._</comment>


Ask HN: Has Intel not made any architectural improvements in 3 years? - <AUTHOR> So I recently decided to upgrade my home media server essentially because remote desktop and the whole thing was laggy and because I thought I could get something more energy efficient.
So the old one is an Intel® NUC Kit DN2820FYKH powered by a actively cooled Intel® Celeron® Processor N2830 (1M Cache, up to 2.41 GHz). 4GB DDR3 RAM, SSD.
The new one is a Chinese NUC powered by a passively cooled Intel® Core™ i7-7500U Processor (4M Cache, up to 3.50 GHz). 8GB DDR3 RAM, SSD.
I thought one of the best test was to use Handbrake to encode a chapter of a 1080p x264 movie to x265.
Here are the results with the power used by the whole server mesured directly at the power outlet:
Core™ i7-7500U (Q3 2016) | Idle : 8 watts  | Max cpu : 35 watts  | encoded 4660 frames in 586.04s | 7.95 fps / 35 watts / 0.22 fpsw
Celeron® Processor N2830 (Q1 2014) | Idle : 10 watts | Max cpu : 12 watts | encoded 4660 frames in 2400.00s | 1.94 fps / 12 watts | 0.16 fpsw
So, somehow I got what I wanted. At idle state, which the server is 98 % of the time, I got 20 % lower power use. I also now have 4 times more power at my disposal should I need it.
What seems also interesting is the ratio of frame per second encoded per watt = energy efficiency. From 0.16 to 0.22 is an energy efficency increase of +37.5 %. At the same time CPU Lithography went from 22 nm to 14 nm : -36.36 %. About the same percentage !
I may be jumping to the wrong conclusions here, but it seems to me that Intel spent the 3 years seperating 14nm and 22 nm doing nothing else than reduce lithography and made 0 architectural improvements. Am I wrong? <comment><USER2> I'm really not sure what changes they've made, but I think it's just a coincidence that -36% and +37% are similar magnitudes. E.g. imagine instead the lithography shrank by 99.999%. Is there reason to expect 0.32 fpsw?

<comment><AUTHOR> All righty then, my bad.</comment></comment>


Do Not Use Task Manager for Memory Info - <AUTHOR> https://mahdytech.com/2019/01/05/task-manager-memory-info/ <comment><USER2> This article correctly states that committed memory is that in use + memory that's being paged out. Now why would you want to know the committed memory over the actual physical RAM in use?

I can trivially create an app that memory maps a massive file and will show several GB of committed memory. This won't be in use of course, memory mapping files so that the OS will page in/out as required is intentional. Those GB of committed memory aren't something you should care about. I'd be scared if someone looked at the committed memory use of a program that correctly uses mmap and caused someone to exclaim "OMG this is uses TB of RAM!".

Task Manager is doing the right thing here. It's showing you want's actually paged in and in use right now.

<comment><USER3> OOOOOOO, something I actually know!

> Now why would you want to know the committed memory over the actual physical > RAM in use?

Because in Windows, committed size is relative to physical size. You can commit a lot more than RAM, but watch your page-file grow.

malloc() can fail on Windows for this reason. This is not the same on Linux or any of the BSD's I've tried. :)

I experienced/discovered this August last year.. sometimes understanding a lot about Linux can make you blind to the architectural differences Windows has.

I wrote some cross-platform CPP to show it[0]

[0]: [https://gist.github.com/dijit/cb2caa1a40d48e03613f5af0e518d6...](https://gist.github.com/dijit/cb2caa1a40d48e03613f5af0e518d626)

<comment><USER4> > This is not the same on Linux

Linux lets you choose an overcommit policy. [https://www.kernel.org/doc/Documentation/vm/overcommit- accou...](https://www.kernel.org/doc/Documentation/vm/overcommit-accounting)

<comment><USER5> You can choose an overcommit policy on Linux, but most library developers on Linux have chosen the default and regularly allocate wide swaths they don't intend to use.

This is a real pain when moving an application from FreeBSD to Linux, as effective limits on memory are lost (ulimit set at ~90% of ram results in a malloc failure and a clean crashdump rather than death by thrashing, or an untrapable oom kill).

There could maybe be a middle ground where malloc would allocate large chunks of address space for ease of administration, and then ask the OS to commit those pages in smaller chunks as needed. Often, there's not much a lot you can do when allocation fails, but it's way more actionable if the failure is returned from a syscall vs failing when you write to an unbacked page, which could happen basically anywhere in your program.

</comment></comment></comment></comment><comment><USER6> For versions of Windows before 10, I'd agree with this 100%. Windows 10 (or maybe 8-8.1?) pulled in a lot of functionality from Process Explorer and is much improved.

For developers Process Explorer (and ProcMon and a few other utils) is likely an improvement, but frankly if you're doing Windows development you should already have learned about them and probably some of Nir Sofer's tools as well (nirsoft.net). For 90% of people (even developers) you probably don't need what Process Explorer provides.

Side note, in Process Explorer if you turn on the lower pane (View menu or Ctrl-L) you can view all handles that a process has open, including file handles. That can be useful for identifying unrecognized processes.

<comment><USER7> Here's the link to Process Explorer: [https://docs.microsoft.com/en- us/sysinternals/downloads/proc...](https://docs.microsoft.com/en- us/sysinternals/downloads/process-explorer)

It also has an option to replace Task Manager so that it comes up when you do Ctrl+Shift+Esc, etc.

</comment></comment><comment><USER8> This article seems to have a high-level overview of how virtual memory works, without mentioning it at all…I find the terminology use rather strange too. Saying that the virtual address space as is "reserved by the OS for each process" was a bit confusing to to me.

<comment><USER9> The article confuses address space and memory reservations as you note. It again makes the confusion when it suggests the virtual size column in Process Explorer is a reflection of the address space[1]. It also suggests that working set, private bytes, and committed memory are the same thing[2]. The article does not even live up to its click-baity title: Task Manager's default memory column of "private working set" is a decent measure of how much memory a process is uniquely using and task manager has the ability to add all the other measures of memory usage that the article mentions.

For a better explanation of virtual memory in Windows, I recommend Mark Russinovich's article[3]. His tool VMMap[4] is useful for visualizing the memory usage of an individual process.

[1]: The reserved memory is really large for 64-bit processes that use Control Flow Gaurd: [http://www.alex-ionescu.com/?p=246](http://www.alex- ionescu.com/?p=246)

[2]: Task Manager and Process Explorer add to the confusion by calling the same memory different things (Process Explorer's "private bytes" number is the same as Task Manager's "commit size" number on Windows 10 1809).

[3]: [https://blogs.technet.microsoft.com/markrussinovich/2008/11/...](https://blogs.technet.microsoft.com/markrussinovich/2008/11/17/pushing- the-limits-of-windows-virtual-memory/)

[4]: [https://docs.microsoft.com/en- us/sysinternals/downloads/vmma...](https://docs.microsoft.com/en- us/sysinternals/downloads/vmmap)

</comment></comment><comment><USER10> We took the site down.

Archive.org has a copy/mirror: [https://web.archive.org/web/20190106190255/https://mahdytech...](https://web.archive.org/web/20190106190255/https://mahdytech.com/2019/01/05/task- manager-memory-info/)

<comment><AUTHOR> Thanks for the mirror! It should be back up now.

</comment></comment><comment><USER11> The bigger problem is that there's no good number that captures the memory impact of a process on modern unified-memory page-cache-ful architectures. Practically nobody gets this right, and the author of the article is himself glosses over some important details. Every byte of address space is either allocated and backed by some memory object ("reserved" memory) or it's unallocated. Commit charge is a measure of the amount of memory that the system has guaranteed will be available in the worst case, should all faultable address ranges be faulted, but that's not the same thing as memory actually being used. For example, if you MapViewOfFile a 1GB file PAGE_READONLY, you burn very little commit --- just enough for the page metadata --- but if you change page protections to PAGE_WRITECOPY, then you incur an extra 1GB of commit charge, even though you still haven't faulted anything into memory or reserved more address space --- and that's because you _could_ legally COW-fault every page in that file, and the kernel has to commit (thus the word) to providing that memory in the worst case.

I work on Linux these days, which is even more confusing, because thanks to overcommit, most people don't distinguish these different kinds of memory allocation, even though the distinction between commit and reserved memory exists on Linux too. (The kernel just lies about satisfying commit charges unless you tell it not to lie to you. Most people are happy with overcommit's optimism.)

Anyway, the key thing to realize about modern virtual memory subsystems is that "memory consumption" is an incoherent concept. You can derive lots of different numbers from memory management statistics, but each of these numbers is useful for a _specific purpose_. There is no one number that will give you an accurate measure of the impact of a particular process in all scenarios. People constantly say, "Look: just give me a number that I can plot on a dashboard and drive down over time". No such thing exists.

Task manager has to pick one of these numbers to show users by default, and its choice, roughly equivalent to Linux Private_Dirty, probably isn't terrible, since it's a decent proxy for how much RAM you get back if you kill the process. I don't think total commit charge is as good a choice, since with a large pagefile (which everyone should have) total commit can be much larger than total resident memory. Linux PSS is another popular choice, since (unlike Private_Dirty) it reflects the impact of a program's use of shared memory, but PSS behaves in perverse ways --- e.g., starting an instance of memory-hungry process can make PSS _decrease_ because some pages in this program are distributed across more processes, increasing the denominator in the PSS calculation.

Are you worried about running out of page file space? Yes, you want to look at commit. Are you wondering why you're seeing a large number of page faults starting a game? Commit won't help you, but RSS might. It really depends on the situation.

I wouldn't take the advice in the article at face value. If you want to understand the impact a particular program has on the system's memory behavior, you need to understand how the virtual memory system actually behaves, and that's non-trivial.

<comment><USER12> Lovely post, but I now realize that I am rather ignorant of the interplay between processes and the kernel with regards to memory allocation.

Any ideas on cool stuff to read to remedy this?

<comment><USER13> Drepper's _What every programmer should know about memory_ is a little old and goes into perhaps unnecessary detail at times, but is a great place to start.

[https://lwn.net/Articles/250967/](https://lwn.net/Articles/250967/)

</comment></comment></comment><comment><USER14> Does anyone understand the story behind Process Explorer? For 10 years it's been an invaluable tool. But despite being developed by Microsoft you have to install it like third party software. Doesn't even come with an installer! How come it never got integrated into the main OS?

<comment><USER15> It originally started out as something from NTInternals. They were building small utils with capabilities similar to what they had become used to in their Unix background. Then they focused on all the low level tools which led to MS buying them.

Not having an installer is actually a bonus. Far too many "simple" Windows programs seem to need Gigabytes of DLLs installing to Windows folder and spray themselves all over the system. /Windows bloats massively after a couple of years of active use. An exe I can keep in a folder, or drop into the path and simply delete if no longer useful.

<comment><USER14> Well an Installer doesn't mean it has to install something bloated. Make a folder for it in C:\Program Files\, copy over the .exe and .chm, done. I can do this manually easily enough just think it's notable no one ever did it for this tool.

</comment></comment></comment><comment><USER16> David Solomon (author of inside windows nt) used to call it "task mangler" for this very reason :)

<comment><AUTHOR> Totally :) If I had a dime for every one who mixed working set and private bytes because of using Task Manager

</comment></comment><comment><USER17> Nah, if something hangs I use the key combination that starts up the Task Manager. Not Perf Mon. There I instantly see what's the problem. I kill it (or use the temporary Pause) and go back to work.

This is the most occurring case where I look into memory. This will probably be the case for most windows users who know about the Task Manager out there and there is no reason for them to "not use Task Manager" anymore.

I hate those generalizing click bait headlines...they should at least come up with some equal justification for that.

</comment><comment><USER18> I am surprised by how little thought MS always put into the Task Manager compared to every other process manager for Windows, including MSs own Process Explorer.

Even on Windows Server 2016, it doesn't show the per-process memory correctly for processes using over 128 GB.

Had to learn that the hard way when the overall RAM usage was pretty high but I couldn't find any individual process using that much. Then opened Process Explorer and boom, SQL Server was using over 130 GB (cubes..).

</comment><comment><USER19> One small tip to whoever designs these apps: count memory in mibs - this is a way more intuitive, apps that consume less than a mib are extremely rare.</comment>


Working Together In "War Rooms" Doubles Teams' Productivity, Researchers Find - <AUTHOR> http://www.sciencedaily.com/releases/2000/12/001206144705.htm

<comment><USER2> Interesting. I wonder what team size they were using for this comparison. My personal experience has been quite different, so I wonder if our 'war room' scenario had too many people in it.

<comment><AUTHOR> Interesting point. There are probably a lot of variables that would make something like this work or fail. Team size, company culture, the kind of work, and things like that.

</comment></comment><comment><USER3> Working in a war room scenario also basically forces you to do actual work. You can't exactly be browsing Hacker News since everyone else will see your monitor.</comment>


Tell YC: SoCal YC Facebook Group - <AUTHOR>

I'm sorry if this borders on spam, but I thought posting here would be the best place to let you all know about the Facebook group I created for SoCal YC'ers.<p>For some time now I've tried to connect with SoCal YC readers and hackers, but haven't made much progress. I know you guys are out there! You can find the group here: http://www.facebook.com/group.php?gid=83407422326. Anyone from north of LA down to SD, please join. I'd like to get a little get together going for the near future, and would love to have a great turn out. <comment><USER2> You should make the group about "Hacker News", not YC.

</comment><comment><USER3> Can you push to a twitter account? For personal reasons I do not use facebook, but consider me included.

I'm in Pasadena, CA and am on-board with your thoughts. Thanks.

</comment><comment><USER4> Yeah, create a twitter and maybe even a meetup.com thingee. It's always great to meet with other hackers.</comment>


Fun with HTTP Headers (2005) - <AUTHOR> http://www.nextthing.org/archives/2005/08/07/fun-with-http-headers?Cneonction=close

<comment><USER2> Not headers, but console.log ascii art never hurt anyone

see: [http://www.nytimes.com/interactive/2014/10/01/us/retracing-t...](http://www.nytimes.com/interactive/2014/10/01/us/retracing- the-steps-of-the-dallas-ebola-patient.html)

p.s. article needs more d3 Larry, Baden and Karen

</comment><comment><USER3> It gets me every time I see the SCG wiki there.

I actually did put Set-Cookie2 with Comment and CommentURL for the session cookie into a web framework. But when Opera gave up I deleted the code :-(</comment>