Rietveld - code review Google style - <AUTHOR> http://code.google.com/appengine/articles/rietveld.html

<comment><USER2> Back in November 2009 I was on vacation in San Francisco and decided to drop by the Google campus in Mountain View to visit some friends.

Of all the things they showed me there, Mondrian (of which Rietveld is a fork of) impressed me the most. More so than dinosaur skeletons, space ships, 3D Google Earth terminals, or free amazing meals.

I was working in a startup at the time and we did not do code review at all. After seeing the Google workflow and having their build processes explained to me in detail, I was trying to set up code review before commits at the startup too. I hoped to improve code quality with this. Unfortunately the tools we tried were terrible and so it never took off.

To this day, not getting the necessary processes in place remains one of the biggest regrets of working there. Now I work at another place where we don't do that either. Too bad.

</comment><comment><USER3> My last team switched from Subversion/Rietveld to Mercurial/Review Board about a year ago, and I have to say that I much prefer the latter.

Even leaving aside the advantages of a DVCS such as Mercurial, I think that Review Board has a cleaner, more responsive UI. The "interdiff" feature (i.e., show diffs between changesets in this review request, not against the trunk) is pretty great, too.

Also, as a fairly mainstream Django app, it's not dependent on the App Engine runtime, which lets you run it on any standard Python/WSGI-capable web server on any recent version of Python. (App Engine only supports 2.5.)

If you're using Mercurial or Git, I'd highly recommend you check out Review Board; the initial setup is easy if you've deployed any WSGI apps before, and the web API makes it pretty easy to tie into your ticketing/CI/IRC bot/etc.

<comment><USER4> I think the Go project uses Mercurial but does code reviews using Rietveld. They have some python code which helps in sending patches for codereview to Rietveld.

I haven't used Review Board, but I have however used Rietveld and I don't have any complaints about the UI. It also provides the "interdiff" feature, just choose the patchsets you want compared for a specific file, when you go to the diff page.

</comment></comment><comment><USER5> have a look at gerrit[1] if you are a git user. it's a complete rewrite of rietveld (in java) and tied closely to git which is because the main contributor is Shawn Pearce (who is also main contributor to git[2]).

[1]: <http://code.google.com/p/gerrit/> [2]: <http://git-scm.com/about>

<comment><USER6> I think Gerrit is a really impressive piece of software. Please correct me if I'm wrong but if you're looking for a major user of Gerrit I believe the Android OSP uses it.

As much as I like Gerrit I don't think I can use it for our team because Gerrit wants you to review commits, and our team works in terms of whole branches. Reviewing your individual commits made early in the branch history probably isn't useful as bugs may be fixed--or the code entirely changed--by later commits. Anyone else have this problem with Gerrit, reviewing commits vs. reviewing branches?

<comment><AUTHOR> I'm sorry, but I think that your team has this backwards.

If you wait until a branch is done, then reviewers are being asked to look at big chunks of code. That is a lot more effort, and it is harder to get reviewers to do it. And when they do do it, it is harder to get them to do a thorough job of it. Furthermore when they notice design issues, it is harder to get developers to go back and redo all of their working code to make it better.

Therefore if you want effective code review, you need to review early and review often. The smaller a change is, the easier it is to review, the easier it is to be thorough, and the easier it is to accept suggested changes.

My conclusion is therefore that choosing to do code review after the fact means that you are guaranteeing no proper code review. But when you review early - literally on every commit - it is then much easier to maintain a truly effective code review process. It initially "feels" much heavier. But after you get in the rhythm, your code will be much, much improved.

<comment><USER7> We use gerrit at my company, and what you say is true. Developers are told to do one feature or bug fix per commit/gerrit change. It really does help with the review process and things move much faster.

</comment></comment></comment></comment><comment><USER8> Why not just use Crucible if you are running a small startup? There license cost is $10 right now, and it has a lot of niceties. <http://www.atlassian.com/software/crucible/>

<comment><USER9> Seconding this. We are using it, alongside Jira and Fisheye and they are really great, well-done tools. Atlassian will gain a customer when we grow large enough.

</comment></comment><comment><USER10> Doesn't bugzilla (although ugly) handle the same workflow that he is describing? We have svndiff linked in for the side by side diffs, but that is the only thing missing.

<comment><AUTHOR> I don't see how bugzilla handles the same problem. The key feature of the workflow is that a commit cannot succeed until code review is complete. This is very important for reducing how often you break the build. With bugzilla integration your bugzilla tickets contain the history of commits that are tied to that bug, but that is a different problem.

</comment></comment><comment><USER11> Anyone know why google is so committed to perforce?

<comment><USER12> We take for granted today that there are several fast, stable DVCS generally understood by an average engineer (in other words, the bar for using a DVCS has been lowered in the last few years and it's not a purely academic endeavor.) When Google was ramping up, Perforce was dramatically faster and more reliable than the almost all of its competitors. It was one of the most common source control systems to find at Valley companies.

Why are they _still_ using it? Inertia, I assume.

<comment><USER13> I was way happier at Google using Perforce than I am today using Mercurial.

The essential difference that annoys me to no end: \- In Mercurial / Git, you must be synced to tip in order to push. (to avoid creating new heads) \- In Perforce (and other old style VCS), you can commit as long as the files that you are committing are based on the newest version.

It is impossible to have any sort of sizeable team commit to the same DVCS repo. With ~12 software engineers, I already have to sync 25 times a day, even if I'm the only one touching my area of the codebase.

Most Google engineers commit to the same repository, and everyone gets the benefits of building from head (instant fix propagation from other systems, rather than waiting for releases). There is no way for everyone to work on the same repo with a DVCS afaik. It would require some sort of complicated hierarchy of repos.

<comment><USER14> If you do not sync and test your changes against the latest, I'd argue that you're being careless - how do you know things still interact properly?

Feature branches are the way to work (for me). You're pretty much on your own in there, and once the thing's done you push to main.

<comment><USER15> If you don't sync and _code_ against the latest, how do you know things still interact properly?

This is the old optimistic vs. pessimistic locking debate. I remember that at my first job, we used SourceSafe, and it would lock the files when you check them out so that nobody else could edit it. When we switched to CVS, I asked "But what happens if people make incompatible changes to different regions of a file and conflict detection doesn't catch it?" The answer was "Then the next person who checks out the code gets a compile error and we deal with it. It just doesn't happen that often in practice."

I've found that the problem you've had just doesn't happen that often. In the two years and hundreds of changes I've made at Google, I can think of a handful (< 5) of times that a CL has broken the build or caused a bug because of a bad sync and yet not caused any conflicts. And even then, the continuous build catches it and it either gets rolled back and tested properly before resubmit, or somebody patches it and we move on.

I've found that any feature branch that lives longer than a week becomes essentially impossible to integrate - in the time necessary to bring it up to head, head has changed enough that you then need to integrate it again, and so on. This obviously depends on team size though - if you've got 10 developers working on a piece of code, it's going to have a lower change rate than if you have 500 developers working on it. Of course, if you have 10 developers working on the code, the chances are miniscule that one will submit something that conflicts with your change in the window after you sync & test, and you can just yell out across the room "Hey, anybody submitting anything that conflicts with my change?"

<comment><USER16> "I've found that any feature branch that lives longer than a week becomes essentially impossible to integrate - in the time necessary to bring it up to head, head has changed enough that you then need to integrate it again, and so on."

DCVSs don't stop you from doing your commit, doing a naive merge, and then optimistically pushing. That option is still there, and is partly why so many people rebase before pushing their changes. Of course, pessimistic integration is equally available as an option.

If you find the overhead so high, it suggests that it's process holding you back and forcing you to pessimistically integrate, rather than the DCVS doing so.

</comment></comment></comment></comment></comment></comment><comment><USER17> If this would have come around 6 months ago, then I would have used it. Now, I'm satisfied with GitHub and its code-review facilities. We already ditched SVN.

<comment><USER18> Rietveld has been open source for years.

<comment><USER17> Now that you say it, I think I remember my co-founder evaluating it at the time and deciding not to bother. I forget why.

</comment></comment></comment><comment><USER19> the date on that link is May 2008. havent started seeing the video yet, but curious to know why this is news now

<comment><AUTHOR> It is something that came to my attention recently which I thought would be a best practice of interest to people on HN. I hadn't seen it discussed here. Therefore I posted the link, and apparently some other people agree with me about its value.</comment></comment>


Intel Xeon processor with FPGA now shipping - <AUTHOR> https://fpgaer.wordpress.com/2018/05/24/intel-xeon-processor-with-fpga-now-shipping/ <comment><USER2> From The Next Platform ([https://www.nextplatform.com/2018/05/24/a-peek- inside-that-i...](https://www.nextplatform.com/2018/05/24/a-peek-inside-that- intel-xeon-fpga-hybrid-chip/)):

 _The initial workload that Intel is targeting is putting Open Virtual Switch, the open source virtual switch, on the FPGA, offloading some switching functions in a network from the CPU where such virtual switch software might reside either inside a server virtualization hypervisor or outside of it but alongside virtual machines and containers. This obviates the need for certain Ethernet switches in the kind of Clos networks used by hyperscalers, cloud builders, telcos, and other service providers and also frees up compute capacity on the CPUs that might otherwise be managing virtual switching. Intel says that by implementing Open Virtual Switch on the FPGA, it can cut the latency on the virtual switch in half, boost the throughput by 3.2X, and crank up the number of VMs hosted on a machine by 2X compared to just running Open Virtual Switch on the Xeon portion of this hybrid chip. That makes a pretty good case for having the FPGA right close to the CPU – provided this chip doesn't cost too much._

<comment><USER3> That seems like a weak benefit. I don't think the fpga part will be cheap. In fact it will probably be significantly more expensive than the CPU. So then why not buy a chip with twice the cores if running twice the VMs is what you want to do with it? Also you can buy an AMD EPYC chip with twice the cores for about the same price you'd pay only for the CPU part of this chip/half the intel cores.

<comment><USER4> >Also you can buy an AMD EPYC chip with twice the cores for about the same price you'd pay only for the CPU part of this chip/half the intel cores.

Indeed. What benefits the most from being turned into FPGA circuit are stateless or minimally stateful circuits like media decoders/encoders.

Complex stuff like fancy routers, classifiers, internet protocol inspection/handling will not gain a hundredfold speedup unlike the stuff above. This is why cheap x86 based routers are still a thing.

At the moment, I am involved with one cloud provider in China that bids big on cheap FPGAs and RDMA. AWS and Azures can be defeated in detail.

The plan is following: provide hardware accelerated "building blocks" of any modern dotcom business.

Need memcached? We have it running on RDMA, from a bare metal ASIC, 10 times faster than any x86.

Need transcoding? We have it available on RDMA, from a bare metal ASIC, 10 times cheaper than any AWS instance on buck/megabyte.

Need API proxy for TLS/Gzip with gigabytes per second throughput? We have it running on RDMA, from a box with 4 PCIe accelerators, and AWS has nothing to offer for this use case other than "buy a hundred of top tier high performance instances and put them behind load balancers"

<comment><USER5> Pretty sure AWS has had f1 instances for some time already.

<comment><USER4> Yeah, but for those one you still have to do all the HDL programming stuff yourself. Alibaba on other hand aims to provide everything ready to use over a cute RDMA api

</comment></comment></comment></comment></comment><comment><USER6> On a whole different level, if you are merely interested in FPGAs in your computer then the PicoEVB neatly goes in the same M.2 slot as wifi cards do and communicates over PCIe, even if just PCIe Gen 2 x1. As far as I am aware this is, by far, the cheapest way to get an FPGA on the PCIe bus.

<comment><USER7> Don't new laptops only have one of those, for their hard drives?

<comment><USER8> It's confusing me. wifi cards don't need m2 slot, I thought it was a standard made for SSDs. Usually wifi cards use basic mini pcie sockets, even on old (2006>) 12" laptops you can find 2, albeit sometimes only one socket is installed (the other is empty space with pins there, some people even soldered one manually)

<comment> loa-in-backup Directly from Wikipedia:

Buses exposed through the M.2 connector are PCI Express 3.0, Serial ATA (SATA) 3.0 and USB 3.0, which is backward compatible with USB 2.0. As a result, M.2 modules can integrate multiple functions, including the following device classes: Wi-Fi, Bluetooth, satellite navigation, near field communication (NFC), digital radio, Wireless Gigabit Alliance (WiGig), wireless WAN (WWAN), and solid-state drives (SSDs).

</comment></comment></comment></comment><comment><USER9> Note that this is a ~$2500 processor with a ~$5000 FPGA stuck to it.

</comment><comment><USER10> Depending on the price, this will be interesting for I/do-something/O workloads like video encoding/decoding. Depending on the price, looking forward to it. Of course, depending on the price. :)

<comment><USER11> You won't have enough resources for video encoding, most probably. Arria 10 FPGA has 67M _bits_ of memory total (8.5M _bytes_ ), including one simulated with registers. As far as I remember, just storing 1920x1080 (HD) frame in 4:2:0 mode would require 4M bytes, half of memory. 4K would require 4 times as much resources.

You may do some on-line processing like running neural net on the video content, but that's about it. Don't expect anything super exciting from that chip.

Yet, it will bring joy to high frequency traders. The systems there do most of the work in FPGA, including UDP/TCP/IP packets processing and offload some work to CPU (broadcasts about network topology are handled on CPU, for example). They also would like to receive CPU computation results as fast as they can and this chip is exactly that.

<comment><USER9> You have more than enough resources for video encoding, because all modern codecs have a macroblock structure. You don't need to keep the whole current frame in at once, you can slide a window around it. (Conversely, you _do_ need to keep the matching area of the previous frame and maybe even the next frame in order to do proper inter-prediction). That said I would assume GPUs are more suited to this task.

By FPGA standards this thing is _enormous_.

<comment><USER11> You are also limited by number of access paths into the memory which holds, say, macroblock. I forgot about this issue, sorry.

For block RAM in previous generation FPGAs from Altera there were one read and one write paths. To have two read paths you would need to copy block RAM as many times as you need read paths. This means that if you search for block content inside a macroblock with N parallel accesses you would need N copies of macroblock stored.

Tabula's time shifting tech allowed for up to, I believe, twelve paths into the block RAM, six for read and six for write (they were time-scheduled to 1-read-1-write block RAM operating at six times the frequency). I thought this thing would be a road for superscalar FPGA CPUs, but Tabula was closed.

You can imagine not using RAM at all, but then you will spend other resources in FPGA.

These are problems with video compression I see here. I think they are substantial but not unsurmountable and require a balance to solve. It is just me that I saw the balance is not in favour of FPGA.

<comment><USER12> Such a shame that Tabula closed down. They had some of the most interesting tech -- hopefully someone is able to pick it up someday. As someone who works with FPGAs, I think a solution like theirs is the only way FPGAs will become mainstream.

<comment><USER13> Altera took many of the employees (although quite a few are now at AWS) and it's rumored they also bought the IP. A lot of the core ideas will live on in Stratix, hopefully.

</comment></comment></comment></comment></comment></comment><comment><USER14> Could this be a hint at Intel's supercomputing and AI strategy? They cannot compete with GPUs on flops with Xeon, but an embedded FPGA might get them closer.

It is a risky strategy however. Even if they can attain similar performance, which I doubt, programmability remains the big problem for FPGAs. I know Intel is pushing openCL but it simply does not have an ecosystem for software right now, and it remains to be seen if they can even enable much of the feature set of openCL on an FPGA.

<comment><USER15> >programmability remains the big problem for FPGAs

I agree because there's added levels of complexity with HDLs that SW doesn't have to deal with. What would be nice is if there was a tool that you could declare your problem (functions) and your constraints (throughput, LUTs available) and it would figure out the memory, ALUs and pipe-lining needed to solve the problem.

<comment><USER16> That is called high-level synthesis and there are some tools for this available (and I know at least one company who produces a proprietary toolchain - including hardware - for this).

In my experience, high-level synthesis makes things a lot easier for the programmer, but you still have to be aware of very low-level details (and you still can get bitten by abstractions that you don't fully understand).

<comment><USER17> That company might fruitfully make their system available within a datacenter / "VPS"-type context, eg the way Cloud9 does it. ([https://c9.io/](https://c9.io/), you make a free account, you instantly get a Web based editor connected to a Debian VM)

If their system accepts "mostly ordinary" code in eg C, a fair amount of stuff would probably work with it, so it could scale beyond educational exploration/tinkering, too.

<comment><USER18> Here is a service I know of: [https://reconfigure.io/](https://reconfigure.io/)

</comment></comment></comment></comment></comment><comment><USER19> This is great! I think that some time in the future, FPGA will be a vital part for data processing.

In theory, an application with a computation-heavy task could program the FPGA to provide part of that task in hardware (think the hot innermost loop body).

What I am worried about is the infrastructure that is needed to make this happen: Is there even support for this in our compilers? What would support look like?

<comment><USER9> > In theory, an application with a computation-heavy task could program the > FPGA to provide part of that task in hardware (think the hot innermost loop > body).

This is absurdly far away; it reminds me of the decades of assumption that 4GLs are going to obsolete programmers or the decades of trying to cross- compile C to FPGAs, badly.

It doesn't help that there's a huge infrastructure barrier caused by closed tools. Imagine if Intel brought out a processor with a proprietary instruction set where you were only allowed to use their FORTRAN compiler (no C, let alone anything more modern or JIT) with a per-seat license. That's where FPGA tools are.

We won't see a Cambrian explosion in FPGA tooling until they are made properly open. Building using open-source tools needs to be actively supported by the manufacturer.

There are also conceptual obstacles; FPGAs are sufficiently different that programmers have to re-learn and re-write idioms in order to get usable results. It's as big a jump as going from Javascript to CUDA.

<comment><USER20> With the end of Moore's law, won't this kind of thing become increasingly necessary, though?

I agree with you about the problems of closed-source - but I feel like the end of Moore's law will encourage creative solutions to performance problems - which FPGAs would at least make technically possible. And, even if it's like the old days when people bought compilers and access to source, people will still do it - since it'll be the best way to get an edge on the competition.

</comment></comment></comment><comment><USER21> I thought FPGAs were limited to either cases where you need some custom hardware _now_ or where demand is low because, if demand is high enough and you can wait, custom ASICs are cheaper, lower power, and faster.

What has changed here? Are FPGAs faster now or are many more people experimenting, creating sufficient demand for custom hardware at short notice?

<comment><USER22> CPU performance increases are a lot slower now so it now makes sense to spend time on custom RTL. In the past you would just wait 8 months and buy the new twice as fast CPU.

FPGAs are a waste of Silicon compared to ASICs, but for fixed applications they are less wasteful than CPUs.

</comment></comment><comment><USER23> This is cool! I have so many questions, like how does it work with context switching? How do they protect access to the cache?

<comment><USER24> They don't, it's an Intel CPU.

</comment></comment><comment><USER25> does it mean that you can load your designs on that fpga ? where can we find what features are exposed to sw ?

</comment><comment><USER26> Looks like a future AWS instance type.</comment>


Equal Pay Is Not Just a Women's Issue, It's a Family and Economic Issue Too - <AUTHOR> http://womenwanttobeontop.com/index.php/2016/04/12/equal-pay-is-not-just-a-womens-issue-its-a-family-and-economic-issue-too/ <comment><USER2> The gender based wage gap is a myth: [http://www.huffingtonpost.com/christina- hoff-sommers/wage-ga...](http://www.huffingtonpost.com/christina-hoff- sommers/wage-gap_b_2073804.html)

<comment><USER3> Another article by the huffington post saying the opposite

[http://www.huffingtonpost.com/entry/no-the-gender-pay-gap- is...](http://www.huffingtonpost.com/entry/no-the-gender-pay-gap-isnt-a-myth- and-heres-why_us_5703cb8de4b0a06d5806e03f)

<comment><USER4> FTA:

> But the argument that "life choices" made by women are the real reason > behind the gap is, in itself, an absurd oversimplification. Sure, many women > choose to stay home or cut back their hours after having children. But many > others don't opt out. They're forced out because they cannot afford child > care, or find a full-time job that affords them any kind of flexibility.

There is no reason to believe that lack of flexibility or not being able to afford day care wouldn't hit men in the same way, too.

The people who call it a myth show that the gap disappears as soon as you control for these variables. Actually, the gap disappears to a large degree by looking at actual hours worked instead of just the yearly salary of jobs with the same description.

<comment><USER3> There is strong reason to believe the lack of flexibility would hit women much stronger than men.

Census that shows 82.6% of custodial single parents are female in the United States. [http://www.census.gov/prod/2009pubs/p60-237.pdf](http://www.census.gov/prod/2009pubs/p60-237.pdf)

Article summarizing studies that show that even in marriages that attempt to be egalitarian both spouses agree the female ends up taking on more of the household responsibilities. [http://www.nytimes.com/2015/07/31/upshot/millennial-men- find...](http://www.nytimes.com/2015/07/31/upshot/millennial-men-find-work- and-family-hard-to-balance.html)

<comment><USER4> Well, but then that simply is a _choice_ , isn't it?

I would not expect to be paid in full if I work part-time - why should that be true if I would be a woman?

</comment></comment></comment></comment></comment><comment><USER5> Besides the gap myth (or misconception, I would call it) that others have already pointed out, this article contains some other questionable statements.

> Numerous studies have shown that equal pay for women would significantly > boost the American economy—in one year, the U.S. would have produced $447.6 > billion more income

Surely this income not going to women is going to someone else -- male workers, presumably, or to shareholders. $450B wouldn't just come out of thin air, would it? The linked page doesn't explain how they came to this number. I'm assuming they just took the difference women earn on average times the number of working women.

</comment><comment><USER3> I know there have been a couple of companies that are making the move to openly discuss employee pay. It's something that surely needs to happen. Twice in my short career I've run into very obvious pay discrimination revealed after an employee on the way out told me their salary. Once to me, once to a fellow female developer. Excited to see both voluntary moves toward this as well as legislation.

<comment><USER6> What was the nature of the discrimination? How bad was it?

</comment></comment><comment><USER7> Framing the problem as "benefits gap" as opposed to "pay gap" sounds like a good way forward I think. The article rightly points to the difference in terms of parental leave and other benefits, which are usually not brought up when talking about pay gap.

</comment><comment><USER8> These studies have been debunked countless times. When all is equal (hours worked, degree, and experience), women get paid pretty much the same amount as men.

[http://www.wsj.com/articles/the-wage-gap-myth-that-wont- die-...](http://www.wsj.com/articles/the-wage-gap-myth-that-wont- die-1443654408)

<comment><USER3> Another article by the wsj saying the opposite. [http://www.wsj.com/articles/gender-pay-gap- widens-1445389259](http://www.wsj.com/articles/gender-pay-gap- widens-1445389259)

<comment><USER8> I posted the article as an example, but there are many other studies on it coming to the same conclusion.

</comment></comment></comment><comment><USER9> Not that I expected greatness from a website called womenwanttobeontop.com but this article is just bad. Not in the sense that the issue isn't important, but that the authors points are flimsy which does more to harm her message more than help it.

> women still make 79 cents to a man's dollar

If you're opening your article with this line you're already pandering to your base. Nobody who would read this article critically would take that line seriously. Especially since the cited link isn't a study or any kind of evidence, it's another uncited article which simply parrots her statement.

Taking aggregate values is useful in the sense that it signifies that _something_ is probably wrong, but then she assumes that the aggregate data is meaningful on the individual level.

If men and women were paid exactly the same for the same work it's still possible to wind up with a wage gap in aggregate. That doesn't mean that there isn't a problem, but it would mean that the problem isn't wage discrimination.

> For women of color, this gap is even higher.

She linked the exact same useless 'article' for this statement that she did for the previous one. Worse, the linked article doesn't even _mention_ the gap between different races.

The question that's actually interesting is if there's a wage gap between whites and blacks (yes) and if the gap for black women is larger than the combined gap between men and women and whites and blacks. One outcome is more significant than the other.

> Numerous studies have shown that equal pay for women would significantly > boost the American economy—in one year, the U.S. would have produced $447.6 > billion more income

The "study" (not a study, singular) cited in the paragraph is from the Institute for Women's Policy Research which provides no evidence for her claims and seems to be founded on the completely fallacious economic reasoning that the money required to fill the gap doesn't come from anywhere and will have no other economic effects.

Well no shit the economy will improve if you just magically create 21% more wealth from nowhere.

> The income women bring in is also increasingly critical to the financial > security of family units, and when women make less that means less money for > groceries, gas, rent, college tuition and other things families need.

This has nothing to do with women. This has to do with the current state of the middle class, job security, and having a functioning social safety net.

> Women in senior leadership positions need to mentor younger women and > advocate for their advancement in companies.

The hypocrisy is staggering. Women should be using their positions of power and influence to advantage other women simply because of their sex.</comment>


Plasma shot could prevent coronavirus. But feds and makers won't act - <AUTHOR> https://www.latimes.com/science/story/2020-07-10/injection-prevent-coronavirus-feds-manufacturers-fail-to-act <comment><USER2> Clickbait title:

> But the idea exists only on paper...[Dr. Fauci said] scientists should first > demonstrate that the coronavirus antibodies that are currently delivered to > patients intravenously in hospital wards across the country actually work. > "Once you show the efficacy, then the obvious next step is to convert it > into an intramuscular" shot.

<comment><USER3> So they don't want to do it because it hasn't been proven to work. And at the same time it can't be proven to work if it isn't allowed to be done.

How do we get out of this conundrum?

<comment><USER2> You do a proper study. It sounds like the resistance is coming because they're trying to push it through and skip a step.

</comment></comment></comment><comment><USER4> This is a post you would see on Reddit..</comment>


Uber vs. Piketty - <AUTHOR> http://cafehayek.com/2015/08/uber-vs-piketty.html

<comment><USER2> > While working together earlier this week on a business trip to California, > my Mercatus Center colleague Ashley Schiller and I were chatting about Uber > and the assaults that governments are now launching on this amazing > innovation

Calling Uber amazing innovation means you have no idea what you are talking. It is neither amazing nor innovation.

And if the government was really assaulting Uber they would have used RICO.

<comment><AUTHOR> Well, in the context of the thesis that companies like Uber are allowing people to partly convert consumption assets into production ones, to an economist it strikes me as not unreasonable he'd describe it as both "amazing" and an innovation.

That's certainly a very interesting concept, and new in the modern era, I think. Closest I can think of right now is selling surplus solar power your home is generating back to the utility, a somewhat different thing.

<comment><USER2> You know about the many self thought programmers that learned on their own PC-s ... since the late 70s.

<comment><AUTHOR> I don't think education is this sort of consumption or production, in fact, when did people start buying PCs more for consumption? E.g. games and telecom/the Internet?

Comparable might be buying more PC that you need, and selling the extra cycles. There have been for some time schemes that do that for non-profit uses, e.g. math ([https://en.wikipedia.org/wiki/Great_Internet_Mersenne_Prime_...](https://en.wikipedia.org/wiki/Great_Internet_Mersenne_Prime_Search)) and science ([https://en.wikipedia.org/wiki/Folding@home](https://en.wikipedia.org/wiki/Folding@home)), I now notice a huge list of them: [https://en.wikipedia.org/wiki/List_of_distributed_computing_...](https://en.wikipedia.org/wiki/List_of_distributed_computing_projects) But I think I would have heard of any "make money at home!" schemes....</comment></comment></comment></comment>


Fighting Node callback hell with PureScript - <AUTHOR> http://andreypopp.com/posts/2014-07-21-fighting-node-callbacks-with-purescript.html

<comment><USER2> While PureScript looks very cool, if you're interested in solving callback hell without changing the whole language, StratifiedJS is a superscript of JS that adds straightforward sequential-style code which is async under the hood, with a bunch of concurrency constructs built into the language:

[http://onilabs.com/stratifiedjs](http://onilabs.com/stratifiedjs)

(Also, not terribly relevant to the discussion, but the fact that almost every line is abruptly broken mid-word without any hyphenation makes the article rather jarring to read)</comment>


It's not what programming languages do, it's what they shepherd you to - <AUTHOR> https://nibblestew.blogspot.com/2020/03/its-not-what-programming-languages-do.html <comment><USER2> This is what people mean when they say Haskell is "opinionated."

Haskell shepherds you into separating out IO code from library code to such an extent that literally any function that has an IO action taints the value returned from that function, causing it to be an IO value, and trying to pass that IO value into another function makes the return type of that function IO, too. Parametric polymorphism is the default, too, so it also shepherds you into writing general purpose code. Haskell is full of these little decisions where it just won't let you do something because it's not "correct" code, and they kind of don't care if that makes coding in it a fight against the compiler.

Rust took that philosophy and applied it pointers. Every value has a lifetime and an ownership which makes it quite hard to do things that aren't memory safe.

Both Rust and Haskell wrap values that can fail in little boxes, and to get them out you have to check which type of value it is, and in C# there's nothing stopping you from returning null and not telling anyone that you can return null, and just assuming people will check for null all the time. Haskell has a philosophy of "make invalid code unrepresentable." The concept of a value being in a box, rather than null being a possible value makes it impossible to use that value without getting it out.

People who write Go love that concurrency is easy and Go fmt has enforced a single canonical style. Building these sorts of things into the language goes a long way in getting them adopted and becoming the norm.

I think we saw a rise of the easy, anything-goes, screw-performance scripting languages. I think the next fashion seems to be in enforcing "correct" coding style. They all have their place.

<comment><USER3> The haskell situation sounds like generally a good thing but I am not sure I would like it very much if this also applies to logging.... It does not sound like great fun to have to change the signature of a function when it needs to log something and then change it again if it no longer needs to.

<comment><USER4> For logging, you can use unsafePerformIO. Of course, you would call it inside a special function that can do logging. In fact, there are functions in Debug.Trace that do exactly that (to standard output).

Similarly, I used unsafePerformIO (again put into a convenient function) to save a checkpoint data in a large computation. The computation is defined pure, but it calls the function to checkpoint, which does in fact IO under the covers.

Remember, type safety is there to help you. As long as the function performing the I/O doesn't affect the outcome of the computation, everything is safe.

<comment><USER5> > As long as the function performing the I/O doesn't affect the outcome of the > computation, everything is safe.

except it's not! Your IO action may not affect the outcome of the computation but it may launch the missiles in background, which changes everything. The less contrived example would be - "computation is fine and is not affected, yet we have our [production cluster deleted / disk space run out / money sent to wrong recepients] by the IO action".

<comment><USER6> Despite the name, unsafePerformIO isn't automatically akin to undefined behavior in C. It _can_ cause undefined behavior if misused, the most obvious example being the creation of polymorphic I/O reference objects which act like unsafeCoerce—but that would be affecting the outcome of the computation. If the value returned from unsafePerformIO is a pure function of the inputs then the only remaining risk is that any side effects may occur more than once or not at all depending on how the pure code is evaluated. As long as you're okay with that there isn't really any issue with using something like Debug.Trace for its intended purpose, debugging.

There are better ways to handle _logging_ , of course—you generally want your log entries to be deterministic, and the ability to produce log entries (as opposed to arbitrary I/O actions) should be reflected in the types.

</comment></comment></comment></comment></comment><comment><USER7> This is my main issue with C++. For a while my job was to get game engine codebases running, integrate tools and move on. So I saw a lot of big C++ codebases. Nearly every one had the same bad behaviors. Tons of globals. Configuring build options from code. Header mazes that made it clear people didn't actually know what code their classes needed.

I then worked for awhile developing a fairly fresh C++ code base. The programers I worked with were very willing to write maintainable code and follow a standard and it was still really damn hard to keep things like header hygene.

When I go back to the language I can't believe how much time I spend dealing with minor issues that stem from the bad habits it builds. For years I would refuse to say any language was good or bad. Always I insisted you use the right tool for the job. And there are some features of C++ that when you need them you have to use that language or maybe C in its place. But the shortcomings are unrelated to language's issues which largely seem to come from a focus on backward comparability. And so even used in its right application it seems incredibly flawed. And I pretty much believe it's a bad language now.

Disclaimer: I learned to program with C++, I understand its power and for years I loved the language. I also understand there are situations where despite its shortcomings it is the right choice.

<comment><USER8> Why are globals considered bad? I'm seriously asking. I, too, have been told hundreds of times over the course of my career, and I never questioned it. I want to question it now, because I've never understood why people work SO HARD to remove and avoid globals. I seriously doubt that the time and effort I've seen spent on removing and avoiding globals has been time well spent. And I'm quite sure that the effort spent on that is not comparable to the amount of problems prevented by not having globals. There's just no way globals can be dangerous enough to justify the size of globals-cleansing efforts I've seen.

Game development often has a very large global state, and game problems are often inherently global state manipulation problems; you need globals in order to even __have the game __in many cases.

<comment><USER9> Global state is nearly impossible to test in any decent automated fashion. When writing unit tests, globals are the bane of your existence.

If you're relying on globals for passing data, they are also difficult to reason about in multithreaded code.

There are means by which you can share data, that data if instantiated at the code entry point, can be shared in such a way as to never need globals, and rely on decent patterns for sharing between code points.

Yes, there is a trade off in adding parameters to functions, references in classes, but these can be avoided by adopting patterns like inversion of control, etc.

Basically globals are a bad pattern because they make it hard to test and hard to reason about data access patterns.

<comment><USER8> I don't understand. If you have a good understanding of the code you're writing, you won't put yourself into a position where globals cause problems unless you're being very stupid, and if you do, normal use of the program will detect those problems, right? Certainly bug reproduction steps and a debugger will figure out what's going on.

You mentioned unit tests, and these are another thing I don't fully understand. Obviously testing your code is important, and automated tests are good. My beef with unit testing comes with the requirement that all methods and functions have multiple tests each for success and failure conditions, and that results in test code which outweighs tested code by several times. When you discover that the architecture you've been putting together isn't going to work, which is something that happens approximately 100% of the time if you're doing anything real, you now have (say) 5,000 lines of code that needs rework, and 50,000 lines of test code that need to be thrown away and rewritten.

That is A LOT of effort to shove onto yourself to avoid a few global variables, to me. That's so much effort that many projects will just not make the change and ship software that they know is insufficient, and then they'll graft on whatever functionality can't be attained natively with the given architectural decisions rather than redesign.

The ability to paralyze yourself with the weight of unit test code seems like an extremely high price to pay to avoid some global variables.

<comment><USER10> > When you discover that the architecture you've been putting together isn't > going to work

One of the underappreciated benefits of unit tests is it quickly teaches you how to write good code. It turns out testable code is also code that tends to be well architected and doesn't need to be rewritten. Basically writing tests leads you to being a better programmer

<comment><USER8> I have never witnessed that in 15 years of working at places which write unit tests. I've witnessed a LOT of unit tests which test nothing and manually return the pass/fail result desired so the indicators stay green.

<comment><USER10> That's very unfortunate.

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> That's a great metaphor for language smells! Some more anecdotes:

\- Python shepherds you into using list comprehensions, even when it's almost always premature optimization and much harder to read than a loop. As a language smell that's not bad, it's just the worst I could think of in a post-v2 world. Luckily there's `black`, `flake8`, `isort` and `mypy`.

\- Bash shepherds you into using `ls` and backticks, useless `cat`s, treating all text as ASCII, and premature portability in the form of "POSIX-ish". Luckily `shellcheck` takes care of several common issues.

\- Lisp shepherds you into building your own language.

There's also tool shepherding:

\- IDEA shepherds you into refactoring all the time, since it's the only IDE which does this anywhere near reliably enough. (At least in Java. In other languages renaming something with a common name is almost guaranteed to also rename unrelated stuff.)

\- Firefox shepherds you into using privacy extensions.

\- Chrome shepherds you into using Google extensions.

\- Android shepherds you into installing shedloads of apps you hardly ever use.

\- *nixes other than Mac OS shepherd you into using the shell and distrusting software by default.

\- Windows and Mac OS shepherd you into using GUIs for everything and trusting software by default.

<comment><USER12> > Windows and Mac OS shepherd you into using GUIs for everything and trusting > software by default.

Not sure how familiar you are with modern Windows, but you can try powershell. You can manage everything in Windows with it, and it's a very cool shell language with a lot of feature.

(It's also open source and multi platform)

<comment><USER13> Do you have any good sources for places to start learining powershell? I'm pretty comfortable in bash, but powershell scares me.

<comment><USER11> As someone who used PowerShell for a year and has used Bash for 10+, PowerShell is much less scary than Bash. The scariest thing about PowerShell is .NET, which (at least 8 years ago or so, so cue someone correcting that below) had extensive but often low quality documentation, with undocumented features you basically _had_ to use to write useful code, uselessly trivial code examples (think "`0 + 0 == 0`" as an example of arithmetic) and some bad names. That's not to say Bash is _better,_ just that they still had some way to go.

</comment></comment></comment></comment><comment><USER14> The term for this in the field of human machine interaction is (perceived) affordance, popularized by Norman.

This sounds like less active guidance than nudging or shepherding. Creating affordances is still an active design choice though.

[https://en.wikipedia.org/wiki/Affordance](https://en.wikipedia.org/wiki/Affordance)

[http://johnnyholland.org/2010/04/perceived-affordances- and-d...](http://johnnyholland.org/2010/04/perceived-affordances-and- designing-for-task-flow/)

<comment><USER15> Not sure the concept applies cleanly here. Affordance is about perceiving possibilities from an _interface_ or environment.

For the subject of programming language idioms, the main factors are restrictions or patterns the language offers, and how naturally they fit within their context, not just perception by the user.

<comment><USER14> I'm curious, how do you make the distinction that restrictions/patterns/context are not just as much perveived properties of the interface to the device you are programming?

Are you perhaps perceiving perception as just visual perception? (See what I did there? :)

</comment></comment></comment><comment><USER16> Paul Dourish reviews some work on this topic in his book Stuff of Bits, see pages 8-9: [https://mitpress.mit.edu/books/stuff- bits](https://mitpress.mit.edu/books/stuff-bits)

It's related to, but definitely not the same as, linguistic relativism. Programming "language" might be a bit of a misnomer, because it creates a false equivalency to natural language. Just as different subfields of mathematics were created to solve different problems, so too were different programming "languages" inspired by different subfields and their notations. With that view, it's unsurprising that some ways of doing things highlight certain methods of solving problems and obscure or impede others.

</comment><comment><USER17> Working in a Java/Kotlin environment, everyone always handles all null cases when working in Kotlin, but they are frequently overlooked in the Java applications. Many of the Java apps compensate with more levels of catch-all exception handlers targetting unexpected NPEs. The only time we get NPEs in Kotlin is when Kotlin allows them because of the Kotlin/Java interop problem.

Working with Javascript/Typescript, we need to rely on linters to enforce safe practices in Javascript.

</comment><comment><USER18> Something i like about rust is it shepards you to fast running programs and away from null pointer errors.

something i like about go is it shepards you to write code any other go programmer can follow easily

something i dislike about c# is it has the tools to let you write very very fast code but shepards you to use non devirtualized interfaces over heap allocated classes tied together with linq overhead.

<comment><USER19> > something i like about go is it shepards you to write code any other go > programmer can follow easily

Sure, the syntax and indentation levels are all the same, but that's not really the difficult parts of programming. The difficulty comes from abstractions, indirections and other abstract things that Go, just as any language, let's you do however you want.

There are of course codebases made in Go where the indirections makes no sense and are hard to follow, just as in any language.

What Go shepards you into is to make really verbose code (well, compared to most languages except Java I guess), where everything is explicit, unless hidden by indirection. This is both a blessing and a curse.

<comment><USER20> Go limits the number of available abstractions--yes at the cost of verbosity-- but compared to a language like C++, it's minuscule in its size. The end result is that you can keep the entire language in your head, and you don't have to go digging into the bowels of the internet to figure out how "turing complete template metaprogramming" works since the last developer decide to use that cool feature they just discovered.

<comment><USER19> > Go limits the number of available abstractions

Again, it doesn't, as abstractions are not built from syntax of the language but from the indirections developers create with the syntax provided. I agree that the abstractions the standard library provides are smaller than in other languages, but outside of that, anyone can create their own abstractions (as it should be).

<comment><USER21> I would say Go doesn't limit the number of abstractions, it limits the number of ways in which abstractions can be created. Or maybe better put, it doesn't provide a very rich set of abstraction facilities.

</comment></comment></comment></comment></comment><comment><USER22> It is the language plus the community. And not just the language.

As an example, there is nothing about Ruby that makes it more or less prone to monkey-patching than many other dynamic languages. But once a certain number of popular frameworks did that, there was no getting away from that. (Rails even has a convention around where you put your monkey patches.)

<comment><USER23> > there is nothing about Ruby that makes it more or less prone to monkey- > patching than many other dynamic languages.

Python disallows making changes to fundamental types like `int` and `list`. It's not possible for a Python framework to support something like Rails' `2.days.ago`.

Interestingly, I don't think this was an explicit decision made when designing Python - it's just a side effect of the built-in types being written in C rather than in Python itself.

<comment><USER24> > Python disallows making changes to fundamental types like `int` and `list`.

Not directly via Python but you can achieve if it you really want [https://github.com/clarete/forbiddenfruit](https://github.com/clarete/forbiddenfruit) :D

</comment></comment></comment><comment><USER25> I like the concept and I particularly like the way I feel nim shepherds me:

* I very rarely need to come up with a name for a function or other identifier. The correct name can be reused for multiple use cases thanks to the type system and prox overloading. * to spend a little time designing the interface before jumping in the code * but also to think what I really need to accomplish and get to it instead of building a grandiose architecture * to have consistent apis * to steer away from OOP * to rely on minimal dependencies and to be kind of minimal in general * to use the correct tool for the problem (macro are not easy to write and that's good otherwise you will abuse them. Instead they are great to use) * to build main table code * ...

I would be interested in what other nimmers think are the good shepherding.

One might also think of what is bad shepherding of nim, although nothing comes to mind at the moment.

<comment><USER26> > * to use the correct tool for the problem (macro are not easy to write and > that's good otherwise you will abuse them. Instead they are great to use)

Just wanted to add: Nim has Macros (which are comparable to lisp forms) and Templates, which are closer to C and Lisp macros and are much harder to abuse; It also has built-in inlines and generics, which are essentially the prime use for templates and macros in languages that lack those.

It also has stuff like operational transforms, which lets you state things like "whenever you see x*4 and x is an integer, use x shl 2 itself", so that you CAN apply your knowledge in the places where you know better than the compiler, while still writing the code you meant to write (multiply by 4) and not the assembly you wanted to generate.

Right tool for the right job is a very good description, which I don't think many languages can claim - definitely not minimalist ones like K, nor kitchen sink ones like C++ (no, an accidentally turing complete template expansion system is NOT the right tool for compile time computation).

</comment></comment><comment><USER27> The author is fairly insightful on various human behaviors in computing systems.

You might enjoy his Mesonbuild build system[0], which has a full manual [1]. It's already used in bunch of high visibility projects [2].

[0] [https://mesonbuild.com/](https://mesonbuild.com/)

[1] [https://meson-manual.com/](https://meson-manual.com/)

[2] [https://mesonbuild.com/Users.html](https://mesonbuild.com/Users.html)

</comment><comment><USER28> When I program in C++, there are lots of things one must consider. Should this be const, public/private, virtual, should I create a class, should I first create an abstract base class, should I create a factory, should I implement the PIMPL idiom, should this be a template function. The list of concerns is nearly endless. When I write in Python I tend to mainly think about solving my problem. In C++ I will naturally think more about performance and in Python that concern comes only if something seems slow. I make no claims about which is better, just that the language definitely affects me and the approach I take.

</comment><comment><USER29> This also can change over time. For example, 15 years ago PHP shepherded you to include every file you were using explicitly, making it hard to reason about a given project if you weren't the creator.

A big effort ensued to change that — class autoloading became the standard, and a large community arose around that standard.

Similarly, JavaScript shepherded you towards some bad practices that the community has now found ample remedies for.

<comment><USER30> > This also can change over time. For example, 15 years ago PHP shepherded you > to include every file you were using explicitly, making it hard to reason > about a given project if you weren't the creator.

Huh, that seems backwards to me? Wouldn't the explicit approach make it more obvious what scripts were relevant?

<comment><USER31> As someone who has written a lot of PHP and is still maintaining a few PHP codebases... I don't miss the old ways at all. But I can see how the parent comment can be misleading for someone who's not too familiar with PHP (apologies if my assumption is wrong).

TL;DR in many ways modern PHP is more explicit than old PHP

The old way wasn't as explicit as the comment makes it. It was just painful. Imports in PHP are global, so every file can use any function/class already imported. You could explicitly define all dependencies at the top, but that's not what the language shepherds you to. In reality most imports were implicit (explicit in ANOTHER file), and you'd just import what you needed that hadn't been already imported (I don't think I've ever worked in a codebase where every file has all dependencies declared at the top). The old ways had so many downsides:

\- moving things around could produce fatal errors just because the implicit imports changed

\- any framework/cms using require instead of requice_once would limit your ability to import files (including the same class twice creates a fatal error)

\- very poor support for IDEs

The modern way is quite pleasant to work with:

\- everything should be in a namespace

\- every used class should be declared at the top of the file (like Java)

\- great IDE support (auto imports, auto complete, click to go to definition)

Yes there's some extra complexity in the autoloader, but in my experience it's negligible. If you use an IDE you might never even see it.

So yeah I don't think PHP got any less explicit. The opposite actually, modern libraries/frameworks tend to be much easier to navigate.

<comment><USER29> Yeah, my explanation wasn't good. Yours is much better – thanks!

If anyone's curious what the old way looks like, have a look at WordPress's codebase – lots of imports scattered around everywhere with no one single class-loading system.

</comment></comment></comment></comment><comment><USER32> Yes! Another example.of this is C# and F#. Both build on the same .net base and both are Turing complete languages with elements of OOP and FP, but wow, the code that those communities write are completely different. They each steer you in different directions.

</comment><comment><USER33> Usually people say language X "encourages" you to do Y. or "biases", but "shepherds" is a fine word too.

C, C++, and Rust shepherd the programmer to use array of structure memory layout. (though it's usually not great for vectorization)

<comment><USER34> Do you have examples of languages which don't encourage you to use array of structures?

</comment></comment><comment><USER35> An imo really nice but lengthy explanation of this is given by Robert Martin here: [https://cleancoders.com/video-details/clean-code- episode-0](https://cleancoders.com/video-details/clean-code-episode-0)

He calls this restriction by paradigms.

<comment><USER36> Thank you for sharing this. This is a _much_ more thoughtful overview of the topic.

</comment></comment><comment><USER37> Is shepherding present in every aspect of life?

My guess would be that it is and that shepherding is what we talk about when we say that we can learn from every aspect of life.

If what I wrote is correct, than shepherding is the teacher of reality. But I guess it's on us to decide when we've learned enough and move on.

I was at first wanting to ask if shepherding is present in video games as well, but then I realized what shepherding could actually be.

SHEPHERDING; The part of an aspect that _can_ teach you something.

_can_, because it's up to you to decide if you'll learn anything.

Is shepherding always negative or can it be positive?

Also, if we would always strive to fix what shepherding teaches us, would that mean that in infinite amount of time we would reach perfection?

And, last question I swear, is shepherding subjective or objective. Or is it both?

<comment><USER38> Shepherding is when some behavior is encouraged by being made easy/rewarded, or by other behavior being made harder/punished. So shepherding absolutely happens in every aspect of life - systems move toward low energy states, water flows downhill and living creatures tend to follow the path of least resistance. When parents do it, we call it parenting. When governments or companies do it (via taxes/subsidies or pricing), we call it nudging. When groups do it, we call it socialization.

It's a really useful tool to analyze systems and organizations with - not by looking at what they make possible, but also by what they encourage and discourage - most of the time, the latter are what really matter (the average case is usually what determines the long-term impact of something, not the best or the worst case). When Netflix has auto-play at the end of a stream, they shepherd you toward binging. When Animal Crossing has things you need to wait wall-clock time for, they encourage you not to binge. When free-to-play games come with loot boxes, they don't force you to do anything, but they might still be shepherding gambling-like behavior.

</comment></comment><comment><USER39> This is also why CMake, even Modern CMake, is so bad. You have to fight tooth and nail for something even remotely maintainable.

</comment><comment><USER40> Title reminds me of "guide you to the pit of success" (ie, a slippery slope w a positive ending), which IIRC I first encountered in a post by Zeit cofounder G Rauch, writing about NextJS.

<comment><USER41> That's exactly what I thought about. I think this is the blog post that made this phrase popular: [https://blog.codinghorror.com/falling-into-the-pit-of- succes...](https://blog.codinghorror.com/falling-into-the-pit-of-success/)

<comment><USER42> This is exactly what I thought of while reading the OP article. I love this metaphor and use it often

</comment></comment></comment><comment><USER43> " Yet, in practice, many Perl scripts do XML (and HTML) manipulation with regexes, which is brittle and "wrong" for lack of a better term. This is a clear case of shepherding. Text manipulation in Perl is easy. Importing, calling and using an XML parser is not."

Importing, calling, and using an XML parser is straightforward in Perl. The same for HTML: think about what you want to process, what you want to ignore, and write your callbacks accordingly.

<comment><USER44> This struck me as clueless part of the article.

Perl coding was a significant portion of my career, and much of that involved web services, involving XML. I've never come across code that attempted to parse XML with regexes in Perl. It was always done with easy-to-import-and- call XML DOM or SAX parsers.

</comment></comment><comment><USER45> You read the code. The computer program you.

<comment><USER46> There is probably more truth in this than people realise at first glance.

</comment></comment><comment><USER47> This is a very good take. I've had a whiff of this idea for a while now, but never fully-formed it. As the author says, it's something that's been missing in lots and lots of language and framework debates and has caused people to talk past each other over and over. It's also a very helpful tool when examining why the cultures around certain technologies are the way they are.

</comment><comment><USER48> I believe these things, and is why I think different programming languages are good for different purposes and are why I use different programming languages rather than only one kind. (For example, Haskell isn't a programming language I use much although I do use it sometimes because sometimes what I make is what it seems like good for to me.)

</comment><comment><USER49> Haha, the idea that a "proper" scripting language is more portable than a shell pipeline is amusing. Sure it is, if you can guarantee everyone has the same version of Python, and all the right libraries at the right versions, etc. But if you can do that, you can probably have them all install the GNU userland on their Macs, too.

</comment><comment><USER50> Be the shepherd, use Lisp :D

Seriously though, what other languages other than Lisp (all the mainstream ones at least) give you the freedom to change the language and/or create DSL's with the same ease? And you can still do your 'bare metal' in C if you really really need to and bring it in.

<comment><USER47> I actually think the OP perfectly explains the core _problem_ with Lisp: Part of the value of shepherding is getting everyone on the same page. When everyone's their own shepherd, nobody is on the same page.

<comment><USER51> Javascript is this on steroids: It wants to be inspired by Lisp, by Self, by Java, and the resulting melange smells like every paradigm while fitting none of them. Trying to write applications in Javascript is trying to corral the minds of hundreds of people who each had a different idea of what Javascript is and what it does and what it wants _you_ to do. The resulting system is doomed to total incoherence.

<comment><USER52> What's mostly resulted in Javascript land is people using opinionated (and often severely constrained) boilerplate generators like create-react-app, with opinionated linters like Prettier (alongside transpiling Javascript from other languages like TypeScript), that force a coding still despite the myriad ways to achieve things - I've actually found that to mitigate a lot of these issues well, but it can't deal with the boatload of "bad" advice online to wade through.

<comment><USER51> Even with opinionated generators, what happens when you pull in a library that comes from a very functional mindset, a second that treats JS like Smalltalk, and a third that was thrown together by a novice and has become the de facto standard for its purpose?

The issue doesn't just lie within the baseline of the code you write, but in how many disjoint dialects that code must interact with and partially conform or contort to in order to engage with.

</comment></comment></comment></comment></comment><comment><USER53> Even broader look on this is which run-time properties the resulting _typical_ code has.

Most languages _tend_ to produce a distinct behavior patterns and properties, such as (in no particular order):

\- multi-threaded / single-threaded / multi-process

\- frequency of memory safety bugs

\- broken locale / unicode handling

\- broken or non-standard network settings handling

\- only synchronous I/O calls (or in contrast only async I/O)

\- releasing memory back to OS (or not)

\- fragmenting heap over time (or not)

\- fast / slow text processing

\- fast / slow binary codecs

</comment><comment><USER54> "shepherding: An invisible property of a programming language and its ecosystem that drives people into solving problems in ways that are natural for the programming language itself rather than ways that are considered "better" in some sense"

Seems like most could lend a little more weight to: 'does this lang align with how do you want to think about/represent the problems you're solving'

<comment><USER55> The corollary for language and API designers: whatever you make easiest is what people will do.

If there is a 'right' way to do something, make that the default or the simplest calling pattern. If there is a new 'right' way, don't route the new way directly through the old way. People will think they're cutting out the middleman by keeping the old calling convention as long as possible.

<comment><USER56> > If there is a new 'right' way, don't route the new way directly through the > old way. People will think they're cutting out the middleman by keeping the > old calling convention as long as possible.

It's best if the old way becomes a shim onto the new way --- that shows that the new way works, and encourages moving to the new way -- nobody wants to keep a shim they don't need, and if the new way can do everything, the compatability shim isn't needed.

</comment></comment></comment><comment><USER57> JavaScript shepards you into monkey patching

<comment><USER58> Shimming and polyfilling (i.e. ensuring standard functions are present and what not) sure. But I think most codebases avoid monkey patching (i.e. adding new functionality to built ins).

Maybe it's up for debate if the language itself guides people in that direction, but no one seems to do it anymore, so I personally don't think it does.

<comment><USER47> The new culture around modules actively shepherds people _away_ from monkey- patching. Whether or not Webpack is considered "part of the language" for these purposes is fairly academic, but yes, the reality is that people don't really do that any more.

</comment></comment></comment><comment><USER59> There's a similar blog post by Gabriel Gonzalez which focuses on the incentives set in Haskell:

[http://www.haskellforall.com/2016/04/worst-practices- should-...](http://www.haskellforall.com/2016/04/worst-practices-should-be- hard.html)

</comment><comment><USER60> I would have liked to have seen more examples of what different languages shepherd you to do.

Python, in my experience, shepherds you to import a lot of modules rather than writing your own. There's so many helpful ones available and that's kind of a reality shock if you switch to some other languages.

</comment><comment><USER61> I like this. The insight is true of tools generally, not only languages. It's true at the feature/affordance level. Also at the tool level. "To a man with a hammer everything looks like a nail."

</comment><comment><USER62> My take was always language quality is not about expressive power, because it's fairly difficult to come up with a non Turing complete system anyways, it is about what is easy to understand and modify.

<comment><USER46> Isn't it a balance?

VB6 code was easy to understand on one level, but its lack of expressiveness lead to a lot more code. On the other hand when I look at Ruby code there is too much syntactic sugar for a non-ruby programmers to understand it without looking some thing sup.

</comment></comment><comment><USER63> ZX Spectrum Basic shepherded you into using goto keyword which I've heard it's the single most evil thing to do in software development.

<comment><USER64> In Speccy Basic a GOTO jumped to a line number, so if you added code in that changed your numbering the GOTO broke. That was managed by using numbers that left space (10, 20, and so on), but it was still horrible in a big program. Another issue was that you couldn't go back (that's what gosub was for). They also made code hard to follow because the execution jumped around all over the place, but that was less of a problem.

These days languages that have GOTO usually jump to a label so it's not quite as bad. They're still likely to end up as spaghetti though.

And yes, I am old.

<comment><USER23> Modern languages usually only support goto within a single function. This significantly reduces the potential for spaghetti, compared to 8-bit BASICs that allowed you to jump anywhere within a program.

</comment></comment></comment><comment><USER36> While this topic is fun, it smells like complete BS to me.

Why? Because the problem of people choosing the wrong tool ( _not_ the tool "shepherding" the person into using it) is older than programming [1].

We can do better than to (once again) make up an elaborate way of blaming our tools.

[1] [https://en.wikipedia.org/wiki/Law_of_the_instrument](https://en.wikipedia.org/wiki/Law_of_the_instrument)

<comment><USER65> What is the right tool in each of these cases? If you have a PERL codebase, do you take XML with a totally different language?

<comment><USER66> 2004-2008 I worked at The Press Association - we were processing gigabytes of XML a day, many of it in real time feeds pushed out to the likes of the BBC, large newspaper publishers, and betting shops.

All of this was done with Perl. None of it was done with regular expressions - it was done through XML::LibXML, and thus libxml. Whenever I hear other perl devs complain about XML processing my answer is always the same - just use XML::LibXML.

Choosing a different language might gain some advantages, but if you still choose the wrong libraries you're still going to have pain.

<comment><USER65> But if "choose the right tool" includes "use the right libraries", how does it differ from "write good code"? At that point arguing that you should use the right tool is arguing about being disciplined?

</comment></comment></comment></comment><comment><USER67> Typo

> shepherding An invisible property of a progamming language

</comment><comment><USER68> If all you have is a hammer, everything looks like a nail.

<comment><USER69> My favourite when working in teams is: "use the right tool for the right job", which presupposes you know many tools and have experience working with them ... and that the shared set of tool knowledge in the team is more than just a hammer and a nail.

</comment></comment><comment><USER70> Facts.

</comment><comment><USER71> I agree with all the points brought up in this article except for this curveball:

> Turing complete configuration languages shepherd you into writing complex > logic with them

Everything discussed in this post is a consequence of language design or best practices. I don't think shoving complex logic into yaml files would be considered either. This is more a possibility as a result of Turing completeness, I don't think the language design has anything to do with it. All the other "shepherding" examples are clearly intentional choices by the language designers.

<comment><USER72> I think that the author of the article specifically speaking of configuration DSLs that were intended to be Turing-incomplete but somehow grew full Turing- completeness and therefore became bastard languages. YAML as a config DSL is sub-par (but I'll take it over JSON any day), and truly declarative, Turing- incomplete config languages are great, as is just using a fully-featured scripting language such as a Lisp or Python, but the middle-ground with a custom config language that is also Turing-complete is usually just bad.

Not sure I have any examples of what that might be. Perhaps nginx? or apache? Lots of complex software have very complex config languages that might accidentally be Turing-complete, but only fools would actually use them like that.

<comment><USER73> Often the turing completeness comes not from the config _format_ but from the semantics. A good example of this is Logstash config, where your config is an array of steps to apply, each of which can filter things in certain ways based on certain conditions, and before you know it you've used 100 lines to badly and buggily implement a 5-line script in a real language.</comment></comment></comment>


Nothing about pedestrian safety that more technology won't fix - <AUTHOR> http://cityobservatory.org/pedestrian_lack_of_technology/ <comment><USER2> The simplest thing to do is to just put the cars underground. Dig trenches, build reinforced concrete boxes in those trenches, leaving surface streets for pedestrians only.

True, this isn't a cheap fix. No, it's not intended for anything but downtown areas, where the density is high enough to make it worthwhile.

This would open up a lot of space in major cities - new buildings could be built in the former streets, leaving narrow European-style streets on either side. This is space that could be filled with cafes, shops, restaurants, etc.

What makes this possible? Uber and the electric cars. With Uber, you don't need to park the car. And electric cars don't require nearly the kind of ventilation that gas cars do! (Although some degree of ventilation is still required, because no one should die if and when an electric car catches on fire!)</comment>


ESA experimental spaceplane completes research flight - <AUTHOR> http://www.esa.int/For_Media/Press_Releases/ESA_experimental_spaceplane_completes_research_flight

<comment><USER2> The IXV is more similar to the SNC Dream Chaser than any of what SpaceX is doing. Though you cannot compare them either, apart from that they are both spaceships, utilize the lifting body concept, and have (at different levels) autonomous control.</comment>


JDK 7 Features - <AUTHOR> http://openjdk.java.net/projects/jdk7/features/ <comment><USER2> I wonder what's going on in that Java group at Oracle. How can it possibly take three more years or so to implement something like list and map literals or a data picker UI component?

The one and only change to the VM that I would like to see is not even planned: structured value types. This single issue is responsible for Java's massive memory usage. Half of that memory is used for completely unnecessary pointers and it will lead to C or C++ grabbing the title of 100 year language.

<comment><USER3> Memory usage is fairly moot, and will just become more and more moot.

When you're watching a 3d projected HD video with 5.1 audio on your smartphone, what's using the memory... A few pointers? Or the actual media? And in any event, we'll likely have a few TB of memory even on phones to play with in a few years.

<comment><USER4> You chose an interesting example. The movie can be played and processed quickly exactly because it has an explicit and probably hardware mapped in some places memory layout. It's processed with as few indirect operations as possible.

And this was exactly the complaint in the parent post - this is not possible to do with Java. If you have a list of Integers, you have a list of Integers. (please correct me if I'm mistaken here: ) so every integer needs an explicit pointer in the list itself + the overhead of boxing. It's not the pointer to the list that matters - it's the fact, that your list takes ~4 times as much memory (pointer, lock, tag, value) - or maybe more - I don't remember the object implementation details for JVM.

For scenarios where you operate on big tables of data, it matters, because it kills cache performance and uses up many times the amount of memory that you'd normally use.

<comment><USER2> That's exactly what I meant. In the case of int you can work around it using int[] but if you need more structure, like with a complex number type, or tuples, pairs, etc, the workarounds get really tedious. Basically what you can do is stuff it all into a large byte[] and deserialise on access. That's slow and kills productivity.

<comment><USER3> It's hardly a 'work around' :/

      int[][]

or

      reals int[]
      imaginaries int[]

<comment><USER5> Java's multidimensional arrays aren't stored contiguously in memory because rows can be different lengths. If you have an int[][] a, then a[1] is simply a pointer to an int[]. You can't escape the excess of pointers.

If you try something like having an array for each component as a substitute for an array of structs, then that places fairly strict upper bounds on the length of the arrays before which you are guaranteed to have killed cache locality.

</comment></comment></comment></comment></comment></comment><comment><USER6> This means that Oracle chose Mark Reinhold's plan B (<http://blogs.sun.com/mr/entry/rethinking_jdk7>).

From the point of view of a web application developer, JDK 7 is not very interesting. JDK 7 will contain a few language updates like switch-cases based on strings. No major changes or features that would radically change a web app developer's coding routines. What do you think?

<comment><USER7> As a web application developer, on Java, I think there is a lot to look forward too.

Mostly in JSR 292.

I see a point where I'm running the best of ruby/python/groovy as core parts of a Java deployment environment.

Edit: Not that you can't now, but now this will bring very good reasons to run a hybrid environment combining the best of both systems.

<comment><USER8> Unfortunately JSR 292 will not deliver as you'd thing.

Some people don't have such a good impression of it, e.g. Rich Hickey (Clojure): <http://clojure-log.n01se.net/date/2010-06-01.html>

         method handles are just going to be an entirely new thing hotspot et al
         are going to have to be taught to optimize. They are already awesome at
         optimizing ordinary class method dispatch
         bobo_: invokedynamic is not needed for removing reflection
         ....
         it is just a different way of doing call site caches, which you can already
         do today with classes and methods

What JSR 292 does is to simplify the work a compiler architect needs to do, since implementing call-site caches is hard work, not to mention memory management since interpreters like JRuby are generating lots of classes that are entering PermGen ... so stuff like java.dyn.AnonymousClassLoader may provide some relief.

But on the other hand, compilers will still need to support Java pre-7, since adoption in the enterprise is really slow (many companies are still on version 1.4)

There is a backport for invokedynamic though, but it remains to be seen if it's any good: <http://code.google.com/p/jvm-language-runtime/>

Notably missing features with a lot more bang than InvokeDynamic:

          tail-call optimization
          fixnums
          coroutines

All in all, Java 7 is a monumental failure and Oracle is waisting resources on merging JRockit with Sun's JVM, instead of saving it.

Say what you want about .NET, but their releases have been coherent with each one adding value. If only Microsoft's management would see this as an opportunity and open up .NET a bit, probably not.

<comment><USER7> Everyone could implement their own solution to call-site caching (and they do), but to push it down into the JVM for a single clear defined way of doing it, grants much more to optimization efforts. What benefits one system, should benefit them all.

There will be a schism between libraries that move to Java 7 only, and support their own call-cache mechanism, but I doubt any of the established projects would ditch their existing solutions, it would most likely only effect new libraries.

If an enterprise is busy being tied up in Java 1.4, and missing out on the drastic speed and memory improvements in hotspot 1.6 line, then I doubt that adding dynamic languages is a priority. JRuby for example requires 1.6 (from the literature I've seen and personal experience, but they don't explicitly say so on their site)

I also disagree generally about how Oracle is dealing with Java 7, but I am willing to see what the end result is.

Microsoft has been heavily involved in the IronPython/IronRuby and Mono efforts. They do see the power in .Net cross system.

<comment><USER8> > _Microsoft has been heavily involved in the IronPython/IronRuby and Mono > efforts. They do see the power in .Net cross system._

Mono is kind of behind ... I have high hopes for version 2.8/3.0 with the new garbage-collector, but the prelease crashed with a segfault which kind of turned me off for now, I'll retry it when the final version is released.

What bothers me about it is that currently memory-management sucks, some bugs remain unfixed (tail calls don't work properly, hence F# is unusable, and AFAIK fixing that bug requires some major changes), and there are not many Linux-specific APIs for server-side stuff ... like the async I/O apis are there only for compatibility, but don't work properly.

I also kind of expected Mono to be more than a .NET clone, and yet there is no alternative to the ASP.NET, which is heavy and is made with .NET's constraints in mind. On Mono it leaks memory for instance.

They could do more for Mono than they already did ... like have 1 or 2 experienced engineers help them with the garbage-collector / or grant them the IP to learn from / use the garbage collector from .NET.

Also, the C# ECMA standard is 2 versions behind, what's up with that?

</comment></comment></comment></comment></comment><comment><USER9> The main motivation for adding closures was not having to include a big pile of boilerplate interfaces coming from the jsr 166 fork/join framework. Now fork/join is in JDK7, but closures wont make it till JDK8, which means we'll be stuck with the crappier API forever.

In my opinion, JDK7 wont have enough incentive for corporations to upgrade Remember how long it took for JDK6 to be adopted? So why release a JDK version if the majority of people wont use it? Why not spend a year or more on building new cohesively designed APIs with closures in mind?

<comment><AUTHOR> Actually, that's not accurate. The motivation for adding closures was to avoid adding those additional interfaces for the _ParallelArray_ abstraction that goes over fork/join. Fork/join has always been planned to go in for JDK 7. AFAIK, ParallelArray will _not_ go into the new JDK 7 plan so the API stuff you mention is not an issue.

<comment><USER9> Oh, thanks for clarifying that, I thought ParallelArray was part of it.

</comment></comment></comment><comment><USER10> I like Scala, but I'm still surprised a simpler "better Java" alternative hasn't taken off like:

<http://code.google.com/p/stab-language/> (C# syntax compiled directly to the JVM)

<http://boo.codehaus.org/> (Python syntax, normally for .NET, but there is a pure JVM fork in there somewhere)

The barrier to adoption these days is a robust Eclipse IDE--if either of these projects had one, I think they'd see a lot of adoption.

<comment><USER11> I think Fan might be the simpler better Java - but like you say I dont think it taken off. <http://fantom.org/>

</comment></comment><comment><USER12> As a java developer who's moved over to ruby the JDK 7 feature list was starting to make me think it wouldn't be such a painful language to work with. Just 3 features would have made me think about working with java again. Type inference, closures and collection literals. Anything else they're working on for the java language is wasted time. They've never been great at writing APIs, they should stick to improving the language and letting the open source community drive the APIs forward.

</comment><comment><USER13> No closures :-(

<comment><USER14> They're still coming, but have been put off until JDK 8, which is scheduled for late 2012 (about 6 months later than the original JDK 7 scheduled date). By the fast-moving standards of the software world this does make them a long way off, though...

<comment><USER15> Especially when you are writing Java code to run on client machines. There it's not even safe to rely on 1.4 features yet.

So I guess I'll be able to use closures the moment I can stop trying to make my web pages work in IE 6: in the yeat 2040

<comment><USER16> That's maybe the one most important problem arising with Java if you are using it in an environment you do not know or not know well (as it is, on client machines - especially using Java on the web). As of it, using Java 7 features in Web pages - no way, maybe no way ever if you do not want to exclude a big mass of people out there on the market.

<comment><USER17> People still write client side Java? _shudder_ I love Java, but good god would I never write an Applet again.

<comment><USER18> You do realise that 'client side' includes people's own machines, right? You don't need to do applets for those.

Besides, applets are way way better now than they used to be.

</comment></comment></comment></comment></comment></comment><comment><USER19> Keep using Scala ;-)

<comment><USER20> Your write Java the Langauge should just die! The should work on the VM/GC and JIT. Programmers should work with Clojure or Scala.

</comment></comment><comment><USER17> It might be just sybtactical sugar, but I would really like to see some sort of annotation for auto generating setters and getters at compile. Eclipse makes it pretty easy to make these, but it sure would clean up class clutter.

<comment><USER18> Why bother? All Java IDEs do that with a keystroke.

</comment></comment><comment><USER7> There are a lot of great steps forward for the Java language in 7. My only question is when is Oracle going to release it so it can start being adopted?

</comment><comment><USER21> TLS 1.2 = Transport Layer Security: <http://tools.ietf.org/html/rfc5246>

</comment><comment><USER22> Java is everywhere. Berkeley CS162 course uses Java language to teach OS Design and Internals - they wrote a pseudo-code examples in Java, a-la SaveRegisters(). It is a madness, Isn't it? ^_^

<comment><USER3> Why madness? It has a clear, logical, consistent and easy to understand syntax.

<comment><USER22> Really? Easier than C syntax, which is a more appropriate in this context, and on which Java's syntax is based upon?

<comment><USER3> Way easier than C syntax. Even though I have long mastered various assembly languages and higher level languages, C pointers and their arithmetic still confuses me.

There's nothing worse than something like bar = *[&foo + 88] or whatever it is.</comment></comment></comment></comment>


What is data science? - <AUTHOR> http://radar.oreilly.com/2010/06/what-is-data-science.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+oreilly%2Fradar%2Fatom+%28O%27Reilly+Radar%29&utm_content=Google+Reader <comment><USER2> Although submitted later, for this submission:

<http://news.ycombinator.com/item?id=1398402>

the person submitting it has taken the time to trim the URL of all the crap that this one has:

      utm_source=feedburner &
      utm_medium=feed &
      utm_campaign=
        Feed%3A +
        oreilly%2Fradar%2Fatom +
        %28O%27Reilly +
        Radar%29
        &
      utm_content=Google+Reader

Please, people, trim the URLs and give the duplication detection mechanism a chance. It would make HN a marginally better place.

Alternatively, please, PG, implement the mechanism I suggested ages ago:

<http://news.ycombinator.com/item?id=1012215>

It's not perfect, but it would help keep discussions in one place instead of getting spread over multiple submissions.</comment>


HHS Coronavirus Data Hub – Hospital Capacity Data - <AUTHOR> https://protect-public.hhs.gov/pages/hospital-capacity <comment><AUTHOR> The new HHS website replacing the previous CDC data is displaying per-state hospital capacity data based on a statistical method outlined near the bottom of the page. Can anyone here speak to the effectiveness of the statistical method they chose?

Looking at AZ, their numbers are a good 8 - 10% lower than the AZ Department of Health Services numbers. And a quick glance at other hotspots like Florida, it seems their capacity numbers may be under-represented in the HHS data.

I don't know enough about statistical analysis to know if their method improves what may be imperfect data, or if it may be politically motivated to purposefully show lower numbers.

EDIT: Here is the method as outlined on their website:

Statistical Method: For hospitals that report both the total number of beds and the total number of occupied beds, calculate the percentage % of beds that are occupied. Next, weights are calculated for reporting facilities by taking the % of total beds associated to hospitals that report their numbers. Then, a total estimated number of inpatient beds is derived by dividing the total number of beds by the weight. Next, the number of unreported beds is derived by subtracting the number of reported beds from the total estimated beds to get the estimated number of unreported beds. Lastly, the total estimated number of occupied beds is calculated by adding the number of reported occupied beds to the expected number of unreported occupied beds (based on the binomial distribution). The above calculation is done by state and nationwide for all three of the metrics of interest.</comment>