Messenger.js - dead simple API for multi-server communication in Node.js - <AUTHOR> https://github.com/weixiyen/messenger.js

<comment><USER2> One of the things that RabbitMQ does which this doesn't seems to be much more support for variable topography. This seems to need to know of all receivers when you create a "speaker" which in dynamic networks doesn't seem like such a good idea. Of course you can have each receiver register with some sort of registry and have speakers interrogate that, but then you're back with a single point of (logical) failure.

Discovery and dynamic usage doesn't seem to be covered in many of these projects (cool as they are) which seems to me to be a more challenging and useful extension.

<comment><AUTHOR> It's a very good point. I plan to add automatic discovery where every speaker and listener maintains a current list of the communication topography.

I figure I'll use this space to explain "why" messenger.js even exists.

One reason I wrote messenger.js is because I find it easy to install, work with, and deploy over using RabbitMQ in node. Personally I was using zeromq with node-zeromq.

This means, I need to install zmq, install the node module for it, and anything else that it may need to work properly. In addition I have to do this on dev and production.

On top of that, I also have to write an abstraction over it similar to how the messenger.js listener works right now. To me, there seems to be too many moving parts to the puzzle just to get some simple communication going.

I fell in love with the DNode API at one point, but I have to start my servers first and clients second. It doesn't appear to be a way to capture error events and try to auto-reconnect. There is a ticket for it but it seems neglected. For me, I don't want to worry about that. I want that abstracted away. I also wanted it in TCP, not HTTP.

Of course, the benefit of RabbitMQ is in a separate language, someone probably already wrote the adapter for you to communicate with services of different languages.

I'm not trying to compete with RabbitMQ here though. Just trying to get node.js communication to work in a scalable fashion with a 1 line install that just works.

<comment><USER2> Yeah that's fair enough - I'm working with RabbitMQ in Node at the moment, and I've got a simple domain specific wrapper around it, which I find fairly easy. Of course, not everything on it is in Node, which is one reason this wouldn't work for me - but if it works for you, that's great!

I'll be keeping an eye on it anyway - you never know when something like that might solve a problem you encounter.

</comment></comment></comment><comment><USER3> This looks promising. Trying to figure out how to benchmark this to test the "insanely fast" claim though. Any ideas? Maybe even add a tiny infographic on there?

<comment><USER4> Absolutely. Such a claim needs a serious backup in the domain of messaging, especially when the competition includes fast and reliable solutions such as Apollo or RabbitMQ.

</comment></comment><comment><USER5> How does this compare to AMPQ solutions? This definitely looks like a great use of Node though.

</comment><comment><USER6> What makes you think this is faster than existing solutions? I see no benchmarks.

<comment><AUTHOR> I will put them up. In theory it should be faster than something like DNode or anything that uses DNode (hook.io) because it doesn't have the TCP tear down from using HTTP to carry what is probably small payloads most of the time.

Right now it's as fast as Node.js will allow TCP to be, if that makes any sense.

Benchmarks will come as soon as I get some free time to sit down and do them all. I'll plan to compare with zmq+nodezmq, rabbitmq+nodeamqp, dnode, hook.io. Any others?

<comment><USER7> You do know what HTTP keepalive is, right?

<comment><AUTHOR> Eh, correct, but you still pass unnecessary headers each time as part of the payload. It would even up the speed theoretically, but it's still a request reply model that demands a response, and you really don't need that for pub/sub for example.

TCP just feels cleaner as a protocol for this kind of stuff but I would certainly concede that speed of HTTP isn't as bad as people would make it out to be.

Also, I'm considering adding createFastSpeaker and createFastListener to the API for people like game makers who don't need to guarantee delivery of certain data over UDP.

</comment></comment></comment></comment><comment><USER8> While it's nice to see projects such as this, it's somewhat of a pity that most of such solutions are specific to a single language, runtime or environment. I recently started working on something similar, but which is supposed to glue code from any language or environment together (somewhat like AMPQ etc, but on a slightly lower level, I guess...): <http://dev.yorhel.nl/doc/commvis>

<comment><USER9> Kind of like 0MQ, but with JSON?

<comment><USER8> Ah, I hadn't heard about 0MQ yet. Just glanced at it and it looks quite similar, but with very different communication primitives.

I certainly need to play with that to find out more. Thanks for the mention!

<comment><USER9> np!

</comment></comment></comment></comment><comment><USER10> So, not to be dense, but could you give an example of an application that would make good use of this library?

It feels very websocketish, but with some more infrastructure around it. Would people use this instead of something like socket.io, or are the 2 solving different problems?

Looks pretty cool, either way.

<comment><USER11> I think it's more like Hook.io than Socket.io. One use could be server monitoring or deployment. You can also check out the Hook.io hook libraries to see how that has been used: <https://github.com/hookio/hook.io/wiki/Hook.io- Libraries>

This seems like a simpler API to me.

<comment><USER10> Thanks for the reply, I'll check out Hook as well.

I'm fairly new to working with Node, so any kind of context for use-cases around some of these libraries is always a help

<comment><USER12> i assume messenger works the same way, but the nature of hook is the processes are completely decoupled. essentially, you are just creating an event emitter that works cross-process. one benefit of this decoupling is it enables you to spin up and take down these processes without affecting the rest of the program.

</comment></comment></comment></comment><comment><USER13> The examples and output don't seem to be synchronized (for example in the pub/sub example). But more interesting, how can the client reach servers on a different network/ip? since that is the more common case for distributed servers...

<comment><AUTHOR> messenger.createSpeaker('220.127.116.11:2222', '33.333.333.3:2000')

on other server

messenger.createListener(2222, 2000)

You'll of course have to update your IP Tables.

</comment></comment><comment><USER14> Oh no, I don't think the internet can take another 'so fast it destroys everything you know about interneting' node.js project :(

</comment><comment><USER12> similarly, check out hook-io by nodejitsu. i actually don't see the benefit of messenger.js at all in comparison. perhaps its simplicity, if that happens to be beneficial in your scenario.</comment>


Ask HN: Why are non-technical managers paid more than engineers? - <AUTHOR> This has been the standard at every software company I've worked at so far.
There are 2 open positions: Software Engineer, and Project Manager.
The Project Manager position is filled in 2-4 weeks. This person is offered more compensation than the engineers they're managing. Often their skillset appears (to engineers at least) to consist only of the most basic skills required to function in an office environment. The emerging pattern is that the winning candidate is chosen based mostly on their physical presentation, and their ability to "win conversations."
The Software Engineer position is filled in 3 months if we're lucky, and then stays open in perpetuity, because the need for engineers outpaces the company's ability to hire them. We screen thousands of resumes trying to find somebody who's even worth interviewing. We hire a recruiting agency, throw money at Greenhouse, conduct on-site rounds with a 90% fail rate, and pay mid-5-figure finder fees to Triplebyte and Hired.
The Software Engineer has a skillset that is beyond the comprehension of most people, and is so large and varied that some part of it is even beyond the comprehension of their peers and vice-versa. There's often a lifetime of work behind the development of that skillset, since they were in their teens or even earlier, most of it above and beyond any standard educational curriculum.
Software companies would seem to be one of the most (if not the most) extreme examples of supervisors being more replaceable than their direct reports in hiring.
So why are software companies so willing to offer more pay for a managerial role that is easy to hire for, and so unwilling to offer more pay for a technical role that is one of the hardest to hire for? Why does it seem like market forces just don't apply here? <comment><USER3> Senior engineer speaking. 20 years of experience in the software industry.

Management is knowing of to make people work together toward a common goal.

1\. A managerial role is not _easy_ to hire for if you're looking for someone who has (for a start) correct skills & experience to manage a team (of engineers only, or of various backgrounds).

One of their roles is to help his team focus on what they want to/can do for everyone to succeed. That implies a lot of ingesting, structuring information and communicate it in all directions, all of the time. You need practice, and skills, to handle that without becoming completely numb.

2\. Technical skills aren't everything. Far from that. When hiring, it's among the last items on my checklist. You want people to be motivated for the job, to be capable and willing to work in a team, to be willing to learn and spread what they learnt. What they know and can bring in total derives from that.

3\. You get 1 manager for several members of a team. And a manager also coordinates with other managers & team members. Budget-wise, you can take and compare the manager compensation with any one team member's compensation but that does not make much sense.

4\. Companies success being the result of (mostly) the technical excellence of their teams is the worst toxic professional myth I've encountered; it took me 5 years to see it and 10 more to accept it.

Purpose/strategy (know what you want) + design (know how to get there) + logistics (know who/what to ask for it) trumps most of the rest.

The bonus is if you also have the technical skills to do it but that's not a requirement.

Being an engineer is great, I love it. But it's far from being THE ultimate valuable role in a company.

<comment><USER4> Number 2 is so hard for a lot of engineers to grasp. Technical knowledge does not translate to strong management skills. In fact, I would say at a certain point deep technical expertise becomes a hinderance on your ability to manage people.

<comment><USER5> technical skills are necessary to make educated decisions and judgments. Managers on high tech projects without such skills blindly shuffle people between roles usually base on which person is nicer to talk.

<comment><USER3> Right. For every single person, yes.

But there are many people to make an organisation work.

There, experts provide their expertise, managers make calls (hopefully for everyone - that's also part of their job, informed calls).

If the call was wrong, you can question & identify if it is either the manager's judgement that was debatable, or the piece of expertise provided, or both. And then decide what to do: change the methods, change the role, change the teams, move the people.

If you have a single person being the expert and making the calls, all you have is a bus number of 1.

<comment><USER5> All those iterations take resources from somewhere else. Manager with strong tech skills can reduce number of mistakes and frictions significantly and improve projects chances.

> you can question & identify if it is either the manager's judgement that was > debatable

You are probably referring on some VP level person, so I think they usually have even lower expertise, and will just fire manager who doesn't deliver, which is correct, VP shouldn't dig to such low level of execution.

</comment></comment></comment></comment></comment><comment><USER6> Have you heard about the law of supply and demand? Because it is bullshit. Salaries are not set based upon some fictional law of economics. Instead, look to sociology and biology to explain managerial salaries. Developers may be harder to recruit than managers but occupy a lower rung on the totem pole than them. Hence cannot be paid more than managers. Because pay is status and leaders have more status than followers. The more people you lead the more status and money.

<comment><USER7> I agree that Pay is status; but I believe with a bigger pay comes bigger responsability.

Landing on a role with bigger responsability must be properly compensated or you won't be able to fill the position.

<comment><USER8> Correct, but I'd like to point out that responsibility does not imply consequences. If the higher-ups mess up they can blame it on those below them. Usually it is done in an indirect way by _setting expectations_. For a lot of people, if they are the ones setting expectations they mentally take themselves out of the equation, assuming they are good. They might be wrong.

</comment></comment></comment><comment><USER9> Programmer jobs are harder to fill because programmers have a long list of personal preferences, trick questions, random algorithms to quiz your depth of understanding on, arbitrary answers and stack-specific questions in their arsenal for rejecting candidates. Everyone else focuses on what the job actually entails and how much they will be making.

<comment><USER10> I also feel like many programmers enjoy making interview questions as tricky as they can and finding every possible reason to reject other programmers they're interviewing in order to appear superior in abilities. How can it be that every company complains they cannot hire engineers fast enough, pay 10s of thousands to outsourced recruiters and recruitment services, and yet there are many engineers, not short of decent ones, having to go through at least 5-7 interviews in order to land one offer.

</comment></comment><comment><USER11> Project manager can mean different things in different companies. I've seen PM's negotiate complex contracts with customers and as a result bring in a lot more revenue for the company without any additional development work. For the regular PM who's just managing internally, if they are good the should help reduce the amount of unnecessary work from the business and have developers working on the most crucial work. Being able to negotiate with different 'kinds' of people is a real skill, and it's not just basic office skills.

That said I think the "cushy" thing about being a PM though is there isn't a new framework to learn next year - you can just keep building on your existing people skills. How to win friends and influence people is more relevant and enduring than any tech skills from the 1930's (or even the 1990's), for example.

But overall IF they are good they are worth it. (If they are bad they cost just as much and are much more harmful than a bad dev)

</comment><comment><USER12> Accountability.

At least in theory, whether a project goes well or badly, it is the leader that is accountable for that result. They get paid well to bring success to a project. They get fired fast for failing.

As a dev, you typically are not held to that level of accountability. If you do good work on a crappy project, you just get moved to a different project when it all blows up, instead of being unemployed. So with less personal risk, you also get less compensation.

<comment><USER13> This depends on where you work, including it's org and/or team culture. I've been on teams where management wanted details - what is our delivery plan (PMs and technical leaders communicate this to more senior management and involve engineers to ask questions, understand what's achievable, set dates and expectations with stakeholders, clearly define CX, write or review customer stories, etc).

I've seen teams where a director level sat down with engineers to understand the launch plan, what system metrics are/will be available, what are technical dependencies? Does team A own them or team B? What's the escalation path in case of failures? Have we stress tested? Failure tested? If dependencies all died, how resilient are our systems? Do they fail gracefully? What's the forecasted peak request rate? What's our infrastructure spend? The list goes on...

I've been on other teams where management is high in the clouds - <USER2> only and even the immediate or other technical management refuses to get involved in any kind of planning, estimates, or generally having any accountability of deliverables. The couple overachievers in the engineering team are basically the ones responsible for the success if the project succeeds (and sometimes they're also owning the definition of success and success metrics). If this scares you it should.

My point being, there is certainly a broad spectrum.

These experiences are from 3 of the FAANGs.

<comment><USER14> Is 1st, F or G? 2nd the commerce A? 3rd N?

</comment></comment></comment><comment><USER15> In my experience, a good project manager makes a good team fantastic. A bad project manager makes a good team dysfunctional. While I can't speak for all the project managers you work with, one who can turn a good team of more than three or so engineers fantastic is worth significantly more than one single good engineer.

</comment><comment><USER16> > There's often a lifetime of work behind the development of that skill.

This is literally true of every senior job role.

It seems like it's mostly software engineers that don't get this.

</comment><comment><USER17> > So why are software companies so willing to offer more pay for a managerial > role that is easy to hire for, and so unwilling to offer more pay for a > technical role that is one of the hardest to hire for?

It really depends on which part of the "software world" the company is situated in and the value they put in engineering.

A lot of companies have this exaggerated structure to value the work done by the project manager and business analyst (usually spec-ing out work and abstracting customer interaction from the rest of the team). The effects of abstracting customer interaction allows them to be valued highly, which will make them reside on the top of the chain and get compensated accordingly. The rest of the team doesn't matter that much as long as they've got the right qualifications to convert requirements into working code.

This stackoverflow answer will give you an idea:

[https://softwareengineering.stackexchange.com/questions/4577...](https://softwareengineering.stackexchange.com/questions/45776/why- do-business-analysts-and-project-managers-get-higher-salaries-than-programme)

<comment><USER18> Unfortunately I've seen most project managers, BAs, and product managers that have no understanding of the VOC, requirements, or the engineering.

Very common:

Product managers typically come from sales/business development and can't articulate VOC or requirements in an intelligible way, even though that may be a listed duty of theirs. They are great at taking their customer base to the baseball game and knocking back a few beers, but engineering ultimately has to collect those insights on their own.

Project managers that have only a PMP or a some kind of tangential management qualification treat every project the same way but do not have the technical understanding to be useful. In their eyes rushing schedules, gate clearing, and release trumps any other concerns and often the result is shipping a turd product with bare bones checkbox requirements (that they never actually appreciated, understood, or cared about) that flops. But they get to claim they navigated X number of product releases on their resume.

Business analysts possess half the knowledge they need to be competent. You need expertise in the business, customers, products, requirements, research, and analytics (basically a step just below the base knowledge you need to do systems engineering without having the technicals to develop a systems solution). You usually get someone with just enough knowledge in one area of computer science, analytics, or business (but almost never a complete suite of knowledge to provide useful input into the process).

Engineering is hard. It's a mixture of multiple disciplines of knowledge and wearing many hats. What holds back engineering salaries is that management, sales, and business types compete with engineering to improve the bottom line, seeking to minimize/outsource the importance of technical solutions and maximize other methods/budgets like marketing gimmicks, cutting costs, and inserting processes in a kludge or bureaucratic manner with the illusion of accountability (I have never witnessed actual accountability) via their hierarchies being closer to executives and board members who tend to have a disproportionate influence on matters like budget for competent engineers.

I have seen enough turd products released that do insurmountable damage to a company that I run if I detect the scent that engineering is treated as just another resource that translates requirements into shipped products and is not on equal footing and input as sales, business process, and management.

</comment></comment><comment><USER19> Just wanted to point out that typically in the US, the role you seem to be describing is that of an engineering manager, not a project manager. If your company is finding engineering managers in 2-4 weeks, it probably is not a top-tier tech company.

At the highest paying tech companies (FANG) there are usually 3 "manager" roles:

\- Product Manager: Interfaces between marketing, engineering and design to manage the development of new features and products. Not necessarily a people manager and does not directly manager engineers.

\- Project Manager: Manages scheduling, task assignment, communication between teams, assigns bugs. Does not directly manage engineers.

\- Engineering Manager: Primarily a people manager, usually was a SWE previously. Does manage engineers.

Of the 3, the project manager path is probably the least lucrative at the same level. Product and engineering tend to have similar pay.

Good engineering managers have a unique blend of managerial skill AND technical skill. On the other hand, there are lots of anti-social weirdos that are great ICs in engineering. As a result, I'm inclined to say that supply and demand is working pretty well.

<comment><USER20> This is the most correct answer. In my company software engineers easily make more than project managers. People manager make a bit more than engineers but not that much. 10-20k more.

Soure: glassdoor.com

</comment></comment><comment><USER21> In general, in the business venue, the big share of income goes to those that can offer some form of command and control to a business venture.

</comment><comment><USER22> Maybe the non-technical managers are better at salary negotiation than the software engineers?

</comment><comment><USER23> Be thankful you're not a musician.

<comment><USER24> The best comment so far!

</comment></comment><comment><USER25> Assumption of facts not in evidence. At many companies including the biggest, engineers are paid more than peer product/project/program managers, and most managers are engineers also.

</comment><comment><USER26> Project managers do not always make more than software engineers. In fact, entry level PMs often make significantly less. It depends on the org and seniority of the position really.

</comment><comment><USER27> Because they set the pay scales, and engineers aren't organized enough to force them to set them differently.

</comment><comment><USER28> The managers are more valuable to the company than the engineers...

</comment><comment><USER29> I actually think a lot of Project Managers do a good job, and are valuable, but surprised nobody has mentioned David Graeber's Bullshit Jobs theory.

That is documented at [https://en.wikipedia.org/wiki/Bullshit_Jobs](https://en.wikipedia.org/wiki/Bullshit_Jobs) and [https://www.vox.com/2018/5/8/17308744/bullshit-jobs-book- dav...](https://www.vox.com/2018/5/8/17308744/bullshit-jobs-book-david- graeber-occupy-wall-street-karl-marx).

It specifically mentions "taskmasters" who manage/create work for others, and project managers might be put into that category by some (but again, not by me as a general rule).

</comment><comment><USER30> Scale</comment>


Are 80 Characters per Line Still Reasonable in 2020? - <AUTHOR> https://hackaday.com/2020/06/18/ask-hackaday-are-80-characters-per-line-still-reasonable-in-2020/ <comment><USER2> I think the answer is "yes" although exceptions to the rule should be permitted. For example, if you split a long logger calling line into multiple shorter lines, it can hurt readability of the actual program logic a lot.

Having more or less everywhere 80 chars limit also makes it easy to display multiple buffers in my Emacs next to each other, which is not a must have, but nice to have.

</comment><comment><USER3> Readability: the Optimal Line Length

[https://baymard.com/blog/line-length- readability?utm_source=...](https://baymard.com/blog/line-length- readability?utm_source=designernews)</comment>


How Generalised Problems Kill Startups - <AUTHOR> https://medium.dave-bailey.com/how-to-reconnect-with-your-purpose-as-a-founder-6d18e80ccc9c#.43r94gfqq <comment><USER2> The real issue is that you can't solve the original problem upfront. You have to work your way acquiring new tools, capabilities, ideas... And it is easy to get lost in the woods.

Which reminds me of [http://imgur.com/gallery/t0XHtgJ](http://imgur.com/gallery/t0XHtgJ)

As a side note I think most of the problems don't have a unique solution, but have a set of solutions that lessen that problem to some degree. Some people tend to get lost finding the perfect solution for which they need perfect tools that don't exist.

<comment><USER3> is that Walter White?

<comment><USER4> No, it's Hal: [https://en.wikipedia.org/wiki/Malcolm_in_the_Middle](https://en.wikipedia.org/wiki/Malcolm_in_the_Middle)

(yes, it's the same actor)

</comment></comment></comment><comment><USER5> >We all start businesses because we're frustrated with something.

I disagree. Sometimes it's because we see other people frustrated with something and think we can help, or more generally because we had an idea to provide value to some group of people without necessarily being in that group of people ourselves.

<comment><USER6> I don't know. I have to say, most startup origin stories I've heard seem to begin with personal pain. Kalanick and Camp started Uber because they had problems getting a cab in Paris. Zuckerberg created Facebook/mash to have a central Harvard student directory. Stripe because Patrick C had so much trouble setting up payments for other projects.

At the very least, another person's pain causes you pain via empathy or some sort of chain reaction. I had a successful website business that was born solely out of a friend's constant requests for updates. I hacked a CMS together and a business was born. His pain was getting his website updated–my pain was him bugging me.

<comment><USER7> Those stories are usually made up after the fact. People naturally have the tendency to create interesting narratives to explain phenomenon in their lives. It's also just better PR. I know this sounds cynical, but it's more often than not true.

Kalanick may have had a hard time hailing a cab...but that probably wasn't inconvenient enough to want him to devote his entire life to building a company solving that one specific problem that he randomly had in his life.

<comment><USER8> Is that really so hard to believe? Building a social network or a ride sharing app is something you do in a couple of months maybe a year. When it takes off though, that's when you make the decision to dedicate your life to it for the years to come.

</comment></comment></comment></comment><comment><USER9> Why would anyone start a company to "solve a problem X"? You should start trying to sell something people want to buy. In the meantime you may end up solving a problem.

<comment><USER10> People buy things to solve a problem, not so you can sell them something. It goes back to being focused on the customer and the whole disruption theme. If your in the tiny nail business and make the best damn tiny nails in the world that's great but when 3M comes along with that putty crap and your sales fall through the floor then it hits you, you were actually in the help customers hang stuff business.

</comment></comment><comment><USER11> The author doesn't mention who the founders are pitching to; I assume he is talking about pitching to investors or other people who might lead to investors. IMO this is the actual issue rather than the "generalised problem."

"Generalised problem" is a by-product of not talking to the right people. Talking with the target customers/users will lead to better understanding of the actual problem.

</comment><comment><USER12> Reminds me of the quote by V. N. Vapnik: "One should solve the problem directly and never solve a more general problem as an intermediate step."

<comment><USER13> That quote would make sense if the quoted had say, isolated the medicinal properties of penicillin, but Vapnik invented Support Vector Machines which have solved generalized cases of classification but directly solved 0 problems (if you say something like running an OCR algo on documents whose purpose remain unamed that were: translated from human speech or thought, printed on paper, then scanned; is solving a problem directly I question your ability to speak of general and direct problem solving any further).

<comment><USER14> Perhaps Vapnik was speaking from experience when he said that "generalized problems suck."

</comment></comment></comment><comment><USER15> What are some examples of this phenomenon in action?

<comment><USER16> A classic example is Webvan.

The basic idea was that ecommerce was going to kill store-based commerce. The founder was talked out of selling absolutely everything and focused in on groceries. They spent a billion dollars building out warehouses, buying trucks, etc, etc. They solved the general problem of delivering groceries.

However, nobody buys a general solution. Each person picks a specific solution to their specific problems. And most people a) already had ways to get groceries they were happy enough with, and b) WebVan never was particularly great for anything, just adequate for a bunch of things. WebVan would have known this if they had started trying to solve the problem for, say, people living on a single block. But because they scaled up while solving a generalized problem, they were screwed.

This is in sharp contrast to Amazon, which started out with book sales, nailed that, and has spent 20 years gradually expanding to other products, markets, and approaches.

<comment><USER17> There was another older service called HomeGrocer that was doing quite well... until WebVan bought them out and converted them to WebVan tech.

From Wikipedia:

> In September 2000, stockholders approved a $1.2B all stock buyout by the > cash-rich competitor Webvan of the much larger HomeGrocer.com. HomeGrocer > was rebranded to Webvan, the management team fired and the technology and > processes converted to Webvan. The Webvan technology did not work very well > and their execution was poor. Most telling, the eight different HomeGrocer > facilities converted to Webvan saw overnight sales declines of one-third, > then over 50% within two weeks and never recovered. Formerly profitable > HomeGrocer facilities quickly ended up with significant losses. Studies of > what happened after the merger were not kind to the original Webvan > management who spent over $500M before going bankrupt less than a year > later.

<comment><USER18> How come WebVan was the cash-rich one then?

<comment><USER17> Huge amount of VC funding during the dot-com bubble, not from any actual profits.

</comment></comment></comment></comment></comment><comment><USER19>
        When we spend so much time resuming the details, it's easy to forget them altogether.

"Resuming the details"? Should that be revising?

</comment><comment><USER20> Being general or specific is just a thought process about an idea. You can go from one end to another and vice-versa while ideating and even when you build stuff. You can assume the whole world is going to use your product. But when you try to define that first set of faces who may use, everything starts to become specific and it questions the overtly general assumptions you made.

</comment><comment><USER21> > This is paradoxical; it's easier to pitch a general problem, while consumers > purchase to solve a specific problem.

I'm not understanding what the paradox is here. This seems intuitive to me.

<comment><AUTHOR> The paradox is that founders generalise a problem to make it sound more valuable, and in doing so lose sight of the specific problem which is actually more valuable. At least, that's how I understand it.

<comment><USER21> I mean that's how so many startups work though.

You start with a big grand vision, find fit in a specific niche / market / vertical to solve a specific problem for a specific customer, then generalize it back to more customers and more verticals.

<comment><AUTHOR> Well, that's the story winners are incentivised to tell - they had some grand plan and every step was designed perfectly to lead to the next.

</comment></comment></comment></comment><comment><USER22> TL;DR:

You have a very specific problem P you want to solve. It's too large when specified to pitch successfully, so you compress it with a destructive, one- way generalization function G, getting you a general problem G(P). You spend so much time pitching G(P) that you forget what P was.

The problem: customers don't want you solving G(P), they want you to solve P. But G is a one-way function, so you can't get P from G(P).

Solution: note the problem P down somewhere before you start pitching G(P). Reread the notes when you get your funding.

<comment><USER23> Thanks for that, the MVP craze makes G really small as well IMO - it's so damn difficult to build something worthwhile in your spare time but I'm nearly there! It's going to be so satisfying launching something I believe in even if I have another 6-8 months of features to add to it!

</comment></comment><comment><USER24> The real problem Medium is trying to solve is not to get you to read interesting things, but to grow their brand.

No, I will not "tweet this".

<comment><USER25> I don't think Medium put that there, I think the author did... You can embed tweets directly into Medium posts.

<comment><USER26> I've heard it all now. You can pre-select the things YOU YOURSELF SAID that are especially profound/awesome/tweet-worthy? This is like, NOT EVEN an echo chamber, because the sound never even makes it out of your own head. You're your own editor, reviewer and audience. Tweet THIS! _grabs crotch_

<comment><USER27> I hear you, but there's a insightful saying that explains that:

"Quoting yourself is cool, as long as you're cool" \-- nandemo</comment></comment></comment></comment>


How to Interface with Big (Engineering) Companies? - <AUTHOR> In brief: How do you get started interfacing with huge (engineering) companies?
The long story:
I've long been interested in marine logistics and shipping, I've built boats and even developed a small-scale autonomous boat (read: RC sailboat) and with rising incidence of crewmember ransom, increased trade volume and changes in the quality and availability of autonomous system algorithms -- I think it's time to retrofit an existing vessels with an autonomous stern system, with backup human-based satellite navigation from land.
I reached out to MARINTEK and Marorka, asking for some relevant contacts within their respective organizations, and it may not surprise you that the response I got back was somewhere between puzzled and threatening.
I don't think it would be feasible to mount an operation to build boats from scratch, but if I could convince one of these organizations to help identify legal barriers to unmanned vessels and license some autonomous shipping technology I build, it could have a shot.
Do I just keep bothering them until they crack or is there a defined procedure for reaching out like this? Is hearing something like this simply too insane for big corporate entities? <comment><USER2> I don't think there is a playbook for this one. It is possible to make a sale like this, but it is also possible you could bang your head for years, or talk to the wrong person and anger the company, or...

What I can say is you should (i) file for micro-entity status at the U.S. patent office, and (ii) file a provisional patent application (costs just $65 if you are a micro-entity) and do this as fast as possible (tomorrow!)

The worst thing is you are going to bang your head against them and then they will rip you off, and going down the patent road is a protection against that. Once you file a PPA you can say you have something "patent pending" and you have priority if you file a real patent in a year. That is a more involved process, but if you are tiny and dealing with big co's, you need all the legal ammunition you can get.

Some people will think you are a jerk if you are following the patent route, and let them think that. If you have a patent it will be a valuable asset for anyone you license it to, so it is not just a stick, it is a carrot.</comment>


Netflix customers up in arms over the new Netflix Silverlight player - <AUTHOR> http://blog.netflix.com/2008/10/opt-in-for-new-netflix-movie-player.html?commentPage=3

<comment><USER2> The title is misleading because none of those problems cited (except lack of support for the .0001% of the population using a PPC Mac) have anything to do with Silverlight. I've seen perfectly good video in that format. The Olympics for example.

If they had the same problems with a Flash player, it wouldn't be news, nor would you have given it that linkbait title. I expect better from you nickb.

<comment><USER3> It's subjective, I suppose. I haven't done any rigorous testing, but the Silverlight player running through Firefox on my MacBook Pro seems lower quality than Firefox running the old player through a Win XP VMWare instance.

</comment></comment><comment><USER4> What's wrong with the Silverlight player?

Runs fine on my Mac, except for a strange interaction between it and the NVidia chipsets that causes it to tear video (apparently it doesn't happen on ATI-based Macs.) There's a workaround you can use with Firefox that fixes that, at any rate.

Certainly seems to suck less CPU than Flash, which can peg both cores of my MacBook Pro just playing a YouTube video full-screen.

<comment><AUTHOR> Read some of the comments.

* Silverlight doesn't run on PPC Macs

* lots of video quality issues: poor picture, blockiness

* lots of audio syncing issues: audio is several seconds behind on many videos

* can't revert back to the old player once you "upgrade" and most customers are saying that they were conned into "upgrading" by Netflix.

Finally, this all started in November of 2008 and comments are still pouring in and it's March of 2009.

<comment><USER5> I watch it daily on Mac (mini), never had any issue. Love it. I guess I should pour in my comments-- though they get them indirectly when I beam to people about how much I love Netflix streaming.

</comment></comment></comment><comment><USER6> This seems a bit crazy. To my surprise, Silverlight has worked awesome on my mac. This link is just a trial for a new beta player. Is the "up in arms" part that they still don't support PowerPC?

PowerPC is sorta on its last legs anyways. Even a profitable startup like Netflix has to invest wisely, and pouring development resources into building an entirely new player on the mac (since Microsoft wont support silverlight on PPC) would be a pretty poor business decision.

Sorry guys, you're going to have to upgrade at some point. You just aren't really an install base that has any weight any longer.

</comment><comment><USER7> I always find it amusing to read the comments for stuff like this. It's just so obvious that you will always get a people complaining about something.

I mean, you have a large number of complaints that Netflix is forcing people to use software made by "evil M$". You have people complaining that PPC isn't supported. You have people complaining that they wasted time supported Mac at all. You have people complaining that their decade old CPUs aren't supported, and others complaining they need to get beefier encoding to allow for better quality.

</comment><comment><USER8> The post linked is from October 2008. It's now March 2009. Is there any more current information? I dismissed the article as soon as I saw it was outdated.

</comment><comment><USER9> Does anyone know why they used Silverlight?

<comment><USER10> There are several reasons but one of the biggest is that Microsoft's DRM solution is/was the most acceptable to the studios.

<comment><USER11> Some quotes from disguntled Netflix customers:

"You can bet I won't be renewing if they don't fix this garbage"

"lower connection speed and lower quality than before Silverlight"

"uggh, Individualization failed. Unable to playback protected (DRM) content. Error 8152. Still does not work on the Mac."

"horribly pixelated, sporadic and painful to watch at times"

"Never really had a problem with the WMP player. Now, I was forced to "opt" in to the new player. Video is jerky/choppy on all Starz titles and many others. And by jerky, I mean unwatchable. Thanks Netflix!!!!!!!!!!!!! What worked fine yesterday is now worthless. Amazing that they can actually make Blockbuster an option again."

If the studios want to push people into downloading over BitTorrent, they're going the right way about it.

Having said all that, kudos to Netflix for not pulling all the adverse comments from their website.

<comment><USER12> _If the studios want to push people into downloading over BitTorrent, they're going the right way about it._

Yeah, I only use Netflix out of utter desperation. If I really want to watch something, and just can't find it on Usenet or TPB, then I grudgingly accept the fact that I will have to use Netflix. The UI and video quality are both terrible. Why is it that I can stream full-HD video for free, but I am only able to _pay_ for low-quality DRM'd crap?

</comment></comment></comment></comment><comment><USER13> Not sure what everyone is talking about. I use netflix on osx and it's by far the best online streaming experience I've had.... equivalent to iTunes in quality, but with better error handling for flaky connections.

The DRM also works seamlessly.... so the end user wouldn't even know the content was protected.

I just wish it worked on linux... I don't know if moonlight plans to support DRM...

</comment><comment><USER14> The quality on their player is worse than your average Scene DVD rip--and given how bad such rips usually look, this is embarrassing.

How long is it going to take, how many hundreds of millions of wasted dollars, how many failed companies, before people realize that if you're going to charge people to watch video, you should show it in reasonable quality, not godawful low-bitrate VC-1? And it isn't as if good encoders are expensive or something like that.

</comment><comment><USER15> I don't like it because the video skips about every 5-10 seconds. The audio is fine. It's really annoying, especially if you're watching an action movie.

Hrmph.

<comment><USER16> I have this problem when I use Safari, even the new version of Safari, but it all works fine when I use Firefox.

</comment></comment><comment><USER17> I've been using the Silverlight version on Boxee for a few months and I have had few issues. I do think an opt-in opt-out program would have been a better move.

</comment><comment><USER18> Does anyone know why they decided to make it so people couldn't switch back to Flash? It seems to me that if they lifted this ridiculous restriction, all these people pissed at Netflix would be happy.

<comment><USER10> > back to flash?

Flash never was an option.

<comment><USER18> Isn't that how streaming to PCs works if you don't opt in to Silverlight?

If not, substitute "flash" with whatever the current option is.

<comment><USER6> Nah, the "Opt In" is just for the new version of the Silverlight player.

Netflix on demand has always been silverlight based.

</comment></comment></comment></comment><comment><USER19> Has anyone hacked the streaming format? It seems like it should be possible.

Netflix on Tivo works well.

<comment><USER3> The whole point is that its got baked in DRM.

Anyway, what's the point? Everything's already on bittorrent, and everything on streaming is also available on easily rippable DVD.

</comment></comment><comment><USER20> I don't know, but this is the type of shit that really makes me mad. If you decide to do business with a large corporation that has enough internal resources to deploy a half-assed product that is based solely on the decision to add competition to an existing market, then this is what you get.

What kind of crap is netflix(and microsoft) trying pull here? -There is no reverting back to the old-player (a subtle TOS) -No Mac support -Silverlight has to be installed on all PC's that use the netflix service ...this list goes on

Microsoft's primary focus has always been to use tacky business strategies that will increase shareholder profits rather than building great, dependable products....and it really needs to stop.

<comment><USER21> >No Mac support There is Mac support, but only for Intel based systems. PowerPC systems are out of luck.</comment></comment>


A Trailblazing Plan to Fight California Wildfires - <AUTHOR> https://www.newyorker.com/magazine/2019/08/26/a-trailblazing-plan-to-fight-california-wildfires <comment><USER2> This part was especially depressing:

 _> The California Air Resources Board restricts prescribed burns to days when pollution is at acceptable levels and the weather likely to disperse emissions from fire. In practice, this means that burning can occur only during a few weeks in the spring. In summer and autumn—the seasons when forests would burn naturally—the state's air usually falls foul of the Clean Air Act. These are also the months that are most prone to uncontrollable wildfires, whose smoke is far more damaging to human health than that from prescribed fire. But, perversely, because wildfires are classified as natural catastrophes, their emissions are not counted against legal quotas._

<comment><USER3> That they are still taking about the same prevention techniques for decades is depressing. You would think a state home to some of the worlds best and brightest engineers could come up with a better solution to this than the same old tired argument of forest fire prevention and prescribed burns.

<comment><USER4> I volunteer in the Danish national emergency service. A year ago we and Multiple other countries went to Sweden to help out with massive forest fires. I'm talking speeding for an hour in varying levels of smoke and not be half way through our area. I've never experienced fires like that, mainly since we dont have forests like that, but it became pretty clear to me how much the low vegetation meant for the life of the fire. The fire could be burning slowly through the ground vegetation and everybody would pretty much ignore it. But if it hit bigger bushes and spread to the treetops all hell broke loose. Others trucks were written off as a loss and personnel airlifted away on that account. As for a newer approach they stragedly cut down parts of the forest, both for wood production and to vary the age of the forest. This seemed to have great effect when trying to contain the fire.

</comment></comment></comment><comment><USER5> It's surprising and a little disappointing when articles like this don't include photos. It was a really interesting read, but I had a hard time visualizing the "healthy" forest that the writer described at the end.

Meanwhile it is interesting how many of the problems we face are principally cultural in nature. Just as the "dense, dark European forest" isn't actually healthy in California, there are a great many other aspects of contemporary life in the US, especially in the Southwest, that are not healthy, but are cling to by most people as part of their cultural narrative. Monoculture grass lawns and parking lots at every destination come to mind, but there are many, many examples.

I wonder if and when we'll ever learn effective ways to bring about culture change when called for. It seems like being able to and interested in moving in small increments would be key, but often times the scale of change that the scientists / visionaries / leaders are calling for don't translate into incremental change in obvious ways. Further, the people leading the charge generally want to see sweeping paradigm shift, and often oppose incremental change as "half-measures."

These are tough problems to solve, but I'm optimistic that some of the lessons that have become widely known in software world (agile, etc) will be able to productively translate into the broader society.

</comment><comment><USER6> I'm sitting in Paradise, CA right now with only one neighbor's home left visible among dozens. There are about 1/3 as many tall trees (mostly pines 30m/100 ft tall) as before since the loggers have monetized them, P&GE didn't want to take any risks when rebuilding the power grid and the city/county/state motivation for future fire remediation. (IMO, the haphazard removal of trees is quite ugly and unnatural.) The policy of putting out all fires quickly set the stage for tragedy, leading to what was feared most. Living at the urban-wildlife interface needs resilient, prepared dwellers who safeguard their structures and allow fires to burn around without threatening lives or property.

</comment><comment><USER7> Pairs well with today's discussion of multi-stakeholder coordination on a US military airplane: [https://news.ycombinator.com/item?id=20792427](https://news.ycombinator.com/item?id=20792427)

</comment><comment><USER3> What is trailblazing about this? A state home to best and brightest engineers on the planet and they can't figure out how to put out a fire. But they can make self driving cars, rockets to mars, uber scooters, and delivery drones.

The answer to this isn't the forest service or the state government. The answer to this is in the private sector.

Instead of making the next facebook how about saving the state from fires that could wipe out its offices?

Is this really the best California can come up with? Does anyone know of any ventures in this space that are up to the challenge?

There is nothing trailblazing about the plan in this article. It's more of an iteration on the same failed policies of the past that got us here in the first place. The forest service is ill suited for this challenge.

<comment><USER8> What in the world are you talking about?

You can't throw 10 Elon Musks at a megafire and expect it to put the fire out. The thermodynamics of putting out a huge fire don't make sense. The logistics of preventing any such fire from starting are impossible.

 _There is nothing trailblazing about the plan in this article. It 's more of an iteration on the same failed policies of the past that got us here in the first place. The forest service is ill suited for this challenge._

The article is very clear on how we got to this point, and on how this plan is different.

<comment><USER3> My comment was clear. I am talking about throwing 10 Elon Musks at the problem. Heck let's throw 50 Elons at it.

The article spends more time describing the Californian landscape and history of fire fighting than what it proposes as a solution. Good luck keeping up with the super blooms. California is a large state with lots of unreachable areas. Where and when these things start is clearly unpredictable.

Mediocrity at its finest.

<comment><USER9> _I am talking about throwing 10 Elon Musks at the problem. Heck let 's throw 50 Elons at it._

We're trying to prevent the huge wildfires, not give some narcissistic idiots platforms for personal aggrandizement while they burn the entire state to the ground.

Prescribed burns work. The problem is that the season is too narrow and the area required too big for existing resources to handle. Private enterprise won't solve that. It will just make everything more expensive and result in less fire prevention activity.

<comment><USER3> Prescribed burns work. Agreed. It's a partial solution we already know about.

Would love to hear some new ideas in 2019 however, but maybe that's asking too much.</comment></comment></comment></comment></comment>


Ask HN: Would You Pay for a Recommendation? - <AUTHOR> Would you pay for expert advice before making a purchase?<p>For example say you want to start playing guitar, you pay to be connected to a guitarist that gives you personalized advice on what best guitar to buy. Or for example, you are a parent, what is the best video game console + monitor you can buy for your son?
Would appreciate to hear some thoughts. <comment><USER2> Yes, and not just for the recco, but for the "why you suggest this / what's the competition" discussion that compresses months or years of relevant knowledge down to say, 30min of advice hopefully tailored to your need.

The problem is, where do you find that being sold, and how can you trust the vendor, for all but a few recognized fields where such consultations are normal practice.

<comment><AUTHOR> Connect buyers with experts who have no bias towards any particular brand or product.

</comment></comment><comment><USER3> Personally no. Half the joy of shopping in a category I'm unfamiliar with is learning the difference and figuring out what I prefer. And recommendations, no matter who they're from, can be wildly biased and subjective.

In your example above, best video game console will vary wildly based on who the intended audience is for.</comment>


Curated list of free books - <AUTHOR> https://github.com/learn-anything/books <comment><USER2> Actual title: "Free and non free books all can learn from"</comment>


A Litmus Test for Entrepreneurs - <AUTHOR> http://hbswk.hbs.edu/item/2978.html

<comment><USER2> What's so odd to me is that reading through the working knowledge section and others on the site is that the Harvard Business School just doesn't seem to get a lot of things about web business anymore.

This article comes off as being written by someone who, while incredibly credentialed, just seems like an outsider. It seems like an article my parents would write and they dont necessarily have a great grasp on the way business is evolving.

I'd much rather read certain blogs than the stuff coming out of hbs these days. Its odd but they're missing developments almost as badly as traditional media.

 __Note this semi-rant is directed on the whole at hbs' working knowledge site and only a little bit at this specific article.

<comment><USER3> Well, in their defense the article was written in 2002. ;)

</comment></comment><comment><USER4> The section on "Do you have the patience to start small?" reminds me of pg's recent comment:

<http://news.ycombinator.com/item?id=116280>

</comment><comment><USER5> All the talk of "are you a closer?" reminded me of Al Pacino's speech in Glengarry Glen Ross: <http://www.youtube.com/watch?v=TROhlThs9qY>

<comment><USER6> Yeah, Alec Baldwin was brilliant in that role.</comment></comment>


Can Dropbox Live Up to $10B Valuation? - <AUTHOR> http://www.bloomberg.com/news/videos/2016-04-08/can-dropbox-live-up-to-10-billion-valuation <comment><AUTHOR> I found the talk interesting. Some points:

Sam Altman has talked to the fund managers who marked dropbox down. They said it was mostly for accounting and Sam offered to buy shares at the lower price but they didn't want to sell

Dropbox's plan to move from syncing your files to keeping your teams in sync

Bloomberg's video player was awful for me. Crashed about 5 times.

</comment><comment><USER2> It probably will. I can see dropbox branching out into cloud hosting for websites. Web 2.0 valuations have been remarkably stable. 90% of the time, valuations either stay flat or keep rising.</comment>


The Price of Tomorrow: Why Deflation Is the Key to an Abundant Future - <AUTHOR> http://thepriceoftomorrow.com/ <comment><USER2> Book ad. Don't bother with the link unless you actually care about the book.</comment>


Tesla restarting production defying local order - <AUTHOR> https://www.wsj.com/articles/tesla-to-restart-production-elon-musk-says-11589230278 <comment><USER2> Comments moved to [https://news.ycombinator.com/item?id=23146897](https://news.ycombinator.com/item?id=23146897).

</comment><comment><USER3> Wondering how much it costs to cancel my Tesla order</comment>


Drinking pop daily can shorten your life: Study - <AUTHOR> http://www.torontosun.com/2014/10/17/drinking-pop-daily-can-shorten-your-life-study <comment><USER2> That seems interesting, but what exactly is "pop"? And why is there a photo of a SODA in the article?

:P</comment>


Advanced Encoding and Decoding Techniques in Go - <AUTHOR> https://blog.gopheracademy.com/advent-2016/advanced-encoding-decoding/ <comment><USER2> The tutorial makes it seem more complicated than it actually is. JSONDog and the constructor example has no place there.

<comment><AUTHOR> Which specific example do you mean, and do you have any suggestions on how to improve it?

I felt this was a good approach given how aliasing builds off of it, but I may have missed a better approach. These are just ones that have worked well for me.

<comment><USER3> I believe grandparent may be referring to the first example. It's a fair bit of reading to get to the good stuff, and if you're skimming it can look complicated. Personally I think the approach of building up the solution is instructive.

</comment></comment></comment><comment><USER4> Is this advanced? If you have ever or will ever attempt to implement an idiomatic encoder/decoder in Go, you will learn everything this post talks about.

<comment><AUTHOR> It is a relative term.

I use it here because most introductory posts about encoding/decoding in go (like [https://blog.golang.org/json-and-go](https://blog.golang.org/json-and- go)) don't touch on these at all. Instead many people figure each of these out separately over time.

<comment><USER5> I'm inclined to agree with the grandparent here. It's good that you're willing to contribute to the already impressive catalogue of Go reference material but your guide is really more of an introduction into the JSON marshaller than advanced techniques about encoding and decoding in the more general sense. Sadly I do feel a little mislead by the title since I was hoping for something a little more challenging than JSON - though had I known it was just JSON I certainly wouldn't have opened the page so maybe the title works? ;)

On a positive note, it does look like it's well written. Detailed with clear examples and laid out in a clean design that's easy to follow.</comment></comment></comment>


JQuery Interactive Tutorial - <AUTHOR> http://net-informations.com/jq/default.htm <comment><AUTHOR> Scripting languages are programming languages mostly (but not necessarily exclusively) used for scripting that don't require an explicit compilation step. Scripting languages usually sits behind some programming language. These are designed for integrating and communicating with other programming languages. These languages usually have less access to the systems native abilities since they run on a subset of the original programming language. For ex. Javascript will not be able to access your file system.</comment>


Why to use 'python -m pip' - <AUTHOR> https://snarky.ca/why-you-should-use-python-m-pip/ <comment><USER2> To prevent accidentally installing in the global interpreter I have this in my .bash_profile:

        # pip should only run if there is a
        # virtualenv currently activated
        export PIP_REQUIRE_VIRTUALENV=true

<comment><USER3> Do you type "sudo" by accident a lot?

<comment><USER4> If you do frequent admin work on a computer it's trivial to "type sudo by accident a lot", so I don't understand the spirit of the question...

</comment></comment></comment><comment><USER5> >Now you may be saying, "I always install the latest versions, so that would mean Python 3.8.0 was installed last since it's newer than 3.7.5". OK, but what happens when Python 3.7.6 comes out? Your pip command would have gone from using Python 3.8 to Python 3.7.

This seems less like a reason to use `python -m pip` and more a reason to endorse languages with saner versioning and less need to sandbox twenty different versions from one another.

<comment><USER6> Nah - what you simply need to do is to access the pip binary in the directory of the virtualenv (using PATH) you're working with and simply stop using the system interpreter.

That being said I would like to inquire which languages _you_ could give as an example that work perfectly well with the system-provided compilers when you are using the latest features.

<comment><USER7> virtualenv is a nasty kludge and needs to die

<comment><USER8> Do the problems it purports to solve just not interest you or did you have an alternate suggestion?

<comment><USER7> It creates more and worse problems than it solves. People find it alright to write code that only works in a specific combination of language and package versions, and that is a shameful state of affairs.

<comment><USER9> That has nothing to do with virtualenvs though.

</comment></comment></comment></comment></comment></comment><comment><USER10> Just use [mini]conda. Everytime someone posts about a python package issue or virtual env issue, it's something that conda already had solved before 2014. And yes, pip is fully supported within a conda environment.

I really can't figure out why conda is not ore popular.

<comment><USER11> Have you tried using it with larger projects? You can easily lose a day waiting for dependency resolution to finish — it takes _forever_.

<comment><USER12> It's not fast on large environment.ymls, but recreating your environment isn't something you do daily so it's not that big of a deal for us.

<comment><USER9> Any project with CI should recreate the environment a lot.

The lack of a Conda lockfile makes it impractical to use outside of toy research proof of concepts.

<comment><USER12> > Any project with CI should recreate the environment a lot.

That's a very inefficient way to run your CI, with conda and pip alike.

Instead, you could build your environment once in a Docker image and use that as your build image.

Saves a lot of time on your builds, guarantees reproducibility, and will work even when package servers are unavailable.

<comment><USER9> We do, but docker caching is not terribly easy to implement in a project with lots of concurrent builds. At least not for me.

We build images based on their commit hash, caching off the last commit hash. That works but has issues with merges and the first commit to a branch.

We also do it based on the branch name, but Docker has issues around specifying multiple cache from arguments in the CLI. That causes unnecessary invalidations on branches.

That all leads to more rebuilds than I would like.

</comment></comment></comment></comment></comment></comment><comment><USER13> Pip is a powerful but low-level tool, which probably shouldn't be user-facing. It doesn't have a full resolver, locking mechanism, or environment management. Likewise setuptools.

For installing global command-line scripts, I like pipx instead of pip. [https://github.com/pipxproject/pipx](https://github.com/pipxproject/pipx)

For development, I like Poetry instead of pip. [https://poetry.eustace.io](https://poetry.eustace.io)

<comment><USER14> Haven't used Python in a long time, but last I remember, pipenv was the tool of choice. Frankly, I'm overwhelmed by all these choices. Does Poetry, pipsi, pipx, pipenv, and pip + venv all accomplish similar goals? Is it just a matter of preference?

<comment><USER13> Pipenv is a replacement for venv + pip + pip-compile.

Poetry is a replacement for pipenv + setuptools.

Pipsi is unmaintained and replaced by pipx.

<comment><USER9> Poetry doesn't manage virtual environments for you, does it?

<comment><USER13> It does.

</comment></comment></comment></comment></comment><comment><USER15> > "Let's say I have two versions of Python installed, like Python 3.7 and 3.8 > (and this is very common for people thanks to Python coming installed on > macOS and Linux, let alone you may have installed Python 3.8 to play with it > while having previously installed Python 3.7). Now, if you were to type pip > in your terminal, which Python interpreter would it install for?"

Yeah but you just kick the can to which python happens to be the system python or activated python referenced by "python". It's exactly the same problem.

If I as a user have to be careful of which underlying Python installation is being referenced, then I want to be responsible for activating the conda environment I need and using "regular" commands like plain "pip" after that. The "python -m" idiom is not important for this use case, as it doesn't actually solve the problem (ensuring the right referenced Python).

<comment><USER16> > Yeah but you just kick the can to which python happens to be the system > python or activated python referenced by "python". It's exactly the same > problem.

Exactly right. And on any sane Linux system the pip command should get you a version compatible with the system Python, which means this is a pointless extra step. Unless these instructions are supposed to be for Windows, or MacOS, in which case it should probably say so. (And doesn't everyone on Windows use Conda anyway?)

</comment></comment><comment><USER17> if you use a per project venv I see no reason not to use `pip install`. if you don't use a per project venv, well, do that!

<comment><USER18> Yeah, venv's are a must. And to the article's "forgot to activate the venv" complaint, I highly recommend direnv.

On top of activating the venv when (and only when) you're inside your project directory tree, you can use dotenv for loading all relevant env vars (AWS creds, API keys, etc.,) and add your project's root dir to PYTHONPATH with path_add.

<comment><USER19> The PIP_REQUIRE_VIRTUALENV environment variable also helps with this.

<comment><USER20> Thank you! Now if only this was documented!

Come to think of it, only there was a PIP_DEFAULT_VIRTUALENV_PROJECTDIR variable which would causes pip to look upwards until it found a setup.py file, and then use a .venv directory in that directory by default...

</comment></comment></comment></comment><comment><USER21> Maybe it's just on my installation, but pip comes with versioned executables. That is, I can call "pip3.7" or "pip3.4". What's the benefits of using the -m approach in this case?

<comment><USER22> You should be using a virtualenv

</comment></comment><comment><USER23> Pip is so bad vs rivals, conda is so much better I'm amazed it's not the recommended package manager.

<comment><USER24> What about installing from source ? conda does not support it at all. For some esoteric python packages which use C++ or Cython backends, it is sometimes impossible to install those packages and installing from source becomes the only viable option. pip is quite awesome for this.

<comment><USER25> This is where conda excels because it uses its own toolchain and therefore doesn't need to make any assumptions about the system. Someone needs to have packaged it, ideally using conda-forge, but it should then just work.

Pip installing all from source is tricky to get working correctly as soon as you require a newer compiler or dependencies. Providing wheels works well but is a pain to set up correctly with good platform coverage.

That said, it doesn't need to be one or the other. Conda integrates well with pip so you can install a base layer with conda then pip install your more esoteric dependencies.

<comment><USER26> It doesn't integrate well. Conda HQ has a bunch of warnings: [https://www.anaconda.com/using-pip-in-a-conda- environment/](https://www.anaconda.com/using-pip-in-a-conda-environment/)

</comment></comment></comment></comment><comment><USER27>
        alias pip python -m pip

then stop worrying about it

<comment><USER28> just don't include that in a machine where the user doesn't know it's there until it's the last thing they check to explain wtf is going on ;)

</comment></comment><comment><USER29> As someone who is stuck on a very old version of python (2.7) on win32 due to Reasons (vendors, basically), how I wish I could even try to install pip; unfortunately, it will only recommend upgrading to a newer version of python when I attempt. All roads seem to lead to this place, a one-size-fits-all dismissive "lol upgrade."

Packaging is one of my least favorite things about the language I love the most.

<comment><USER30> Can you at least upgrade to latest version of python 2.7? If you're on at least python 2.7.9, pip should be already installed because it's included by default.

</comment></comment><comment><USER31> This kind of stuff is my least favorite attribute of Python. Surely there must be a better way?

<comment><USER32> Once you figure out how venv works it's really simple to set up a python project. It's just another quirk with just another programming language, they all have them. People don't complain about how hard it is to learn to program, but learning how to use some pip commands is too difficult for them?

<comment><USER33> Sure it is a quirk, however it could really be better.

Anyway, I think the fact that people has more patience with learning a language than tools is that learning a language is fun, while tooling is just boring.

</comment></comment></comment><comment><USER34> "that's why you should use virtualenv"

</comment><comment><USER35> TL;DR because Python is a mess, and installations are often messed up.

<comment><USER33> Python itself is not a mess, the packaging part of Python is.

However, Python is also one of the oldest programming languages still in use and one of the first to have the concept of installable dependencies (easy_install).

Now it is kinda difficult to fix without breaking something, however I think we eventually will have a solution, like __pypackage__ [https://www.python.org/dev/peps/pep-0582/](https://www.python.org/dev/peps/pep-0582/).

</comment></comment><comment><USER36> Okay, or just use pipenv.

[https://github.com/pypa/pipenv](https://github.com/pypa/pipenv)

<comment><USER37> Not sure why this is downvoted. Only the uninformed are using (usually multiple, lol) requirements.txt files + raw `pip` and managing venvs themselves nowadays.

<comment><USER36> There's a huge rift between hobbyist Python users and professional developers. That's okay, but the former group doesn't always like it when the latter group chimes in...

</comment></comment></comment><comment><USER38> I usually use pip3 install --user. Then just add ~/.local/bin to PATH. It hasn't caused any issues for me thus far. I also use venv, but only in CI to make sure that other people's environments don't get messed up. I personally prefer using pip3 install --user.

<comment><USER39> I'm guessing you just never use Python 2? Otherwise it's hard to see how you don't run into issues.

<comment><USER40> It's EOL in under two months.

<comment><USER39> It's also

      Required By: apitrace apm asciidoc bluefish bzr cloudprint-cups dblatex epydoc fio gconf gemrb getmail gmock gnome-doc-utils gtk-recordmydesktop inkscape ipcheck ipython2 java-openjfx java11-openjfx java8-openjfx jcl jmc kig kodi-bin kodi-gbm kodi-wayland lhapdf lib32-apitrace libgda libkate libvirt-python2 libvolk lilypond mailman mcomix mediaproxy mercurial mftrace mod_wsgi2 moosefs mopidy mysql-python mysql-workbench nss-pam-ldapd ntop pluma pychecker pydb pylibacl pynac pypanel pyrex pyrit pysol-sound-server rdiff-backup repo rtaudio scribus sgmltools-lite shedskin singularity spambayes tellico texmacs tuxpaint txt2tags vim-ultisnips wesnoth wicd wifite xpra
      Optional For: android-tools arduino armagetronad boost boost1.69 brial bugzilla cairo-dock-plug-ins cantor cryptominisat5 csound dia ecasound ecryptfs-utils efl efl-docs faust freedroidrpg freeradius gammu geda-gaf gif2png git gogglesmm graphviz gvim hivex inn john kcachegrind-common kodi-eventclients kross-interpreters libdnet libevent libglade libieee1284 libnewt libproxy libuhd magma marsyas mate-menus mathomatic mediawiki munin-node ncmpc net-snmp non-timeline notmuch openconnect opensips plan9port postgresql pycharm-community-edition python-pywal recoll roundcubemail rrdtool singular skktools spectmorph spring subversion texlive-core texlive-music vim vim-latexsuite xf86-video-qxl

Regardless, let's not turn this into yet another argument about why anyone should/shouldn't upgrade to Python 2. I was just trying to confirm that the parent is not doing anything that requires Python 2.

<comment><USER34> All packages that matter were already ported, some new packages work only with version 3, many popular packages already dropped support for python 2, a lot will drop in December.

Most (all? actually I don't know any from your list that do) packages that you listed are no longer maintained

<comment><USER39> It seems you missed the second half of my comment.</comment></comment></comment></comment></comment></comment>