The Miraculous 1880s - <AUTHOR> http://spectrum.ieee.org/geek-life/history/the-miraculous-1880s <comment><USER2> In my opinion, the 1880s are no more impressive than the 1990s.

\- Hubble

\- International Space Station

\- World Wide Web

\- Human Genome Project

\- GPS becomes fully operational

\- Drones / UAVs

\- Large Hadron Collider project gets underway

\- Wi-fi (every bit as amazing as the coin operated vending machine)

\- Web / Internet search engines

\- Vast breakthroughs in understanding stem-cells

\- Streaming media, explosion of digital media

\- E-commerce

\- Consumer e-mail (far more important than the electric iron)

\- Instant messaging

\- Mass adoption of the graphical user interface

\- Digital cameras commercially available

\- Commercial lithium ion batteries

\- Protease Inhibitor, countless drug breakthroughs, advancements, vaccines

\- Cloning, eg Dolly

\- Vast numbers of breakthroughs around mobile phones, smartphones, PDAs, screens, touch, etc.

\- Linux

\- First robotic vacuum cleaner (again, every bit as impressive as the electric iron)

\- GMOs commercialized

\- Voice over IP becomes a commercial product

Some of these things were obviously continuations, or built upon previous technology. That is true of many of the 1880s breakthroughs as well.

<comment><USER3> It's a bit odd to see "Large Hadron Collider project gets underway" in the same category as "Hubble [Space Telescope]", given that HST got underway in the 1970s. Also, Willadsen cloned mammals in the 1980s using embryonic cells. (Dolly was the first mammal cloned from adult cells.) And "consumer e-mail" was available already in the 1980s with Minitel, CompuServe and Prodigy, among other organizations, as well as from BBSes.

That's not to say there have been no wondrous things in the 1990s, only that the details are hard to pin down.

There's a belief that the rate of technological growth it itself advancing, so that the rate of growth now is much more than the rate of growth 100+ years ago. See for example [https://en.wikipedia.org/wiki/Accelerating_change](https://en.wikipedia.org/wiki/Accelerating_change) for people who believe this is the case.

This essay is trying to counter-balance, or ground, that belief.

The problem, described at [http://www.innovationfiles.org/is-technological- change-speed...](http://www.innovationfiles.org/is-technological-change- speeding-up-how-can-you-tell/) , is that it's hard to measure the rate of technological change. Among other things, we suffer from bias error because we forgot that some apparently simple things - my favorite example is the vertical file system, dating from the late 1800 - didn't exist until recently.

</comment></comment><comment><USER4> Interestingly, Jules Verne (the then equivalent for technology- and exploration-worshipping hard sci-fi authors of today) was reaching the height of his output around that same decade (and the next), with some of his most significant works having been already published in the 15 years preceding it. It was definitely an age of great enthusiasm about technology and progress, and I think Verne's books capture that energy pretty well.

If the 1880s were the miraculous decade in terms of technological breakthroughs, the first 3 or 4 decades of the 20th century were a radical fracture in terms of fundamental thinking across the whole culture: Einstein, Godel, quantum physics, post-modernism, dadaism, Picasso, Joyce's Ulysses. I think we have yet to fully comprehend the magnitude of that tectonic shift in how we think of and understand... well... everything. It was _massive_ and it touched _everything_.

And there's a common thread to that whole massive change. If I had to summarize it in one word, it would be 'relativity'. We've stopped thinking in absolute terms, and started to understand the value and validity and truth of the many different points of view. A century later, that kind of understanding is still percolating into politics, social mores, and daily life.

</comment><comment><USER5> Don't get me wrong, I like the article, but it dismisses a whole bunch of silicon valley stuff as "variations on a theme" of microprocessors and parsing radio spectrum. Fair enough but including the invention of Coca Cola and Edison opening a power plant? That's like counting "Apple starts selling iPhones in China" as an innovation.

The "safety" bicycle? (the chain drive was invented in 1879) First practical street railway? (It's like a railway, on a street.) The Wall Street Journal?

It's always worth putting technological progress in context. It reminds me of a 1999 WIRED article where they asked people to pick which 20th century technology would most shape the 21st century, to which one expert replied (I paraphrase), "if I weren't restricted to 20th century technologies, I'd say the telephone" (pointing out that the majority of the world's population lacked access at the time). Someone (maybe the same person) nominated the bicycle as the most important technology of the 20th century.

<comment><USER6> To address one point, the safety bicycle was the point in the development of the bicycle that made it possible for women to ride while wearing long skirts. This was a huge step for women's autonomy as it allowed women a way to travel independently. So yes, the safety bicycle was a big deal, though I agree that it's nonsense to try to rank decades.

[http://www.annielondonderry.com/womenWheels.html](http://www.annielondonderry.com/womenWheels.html)

<comment><USER7> I don't think it's really nonsense to try to rank decades, as long as you have some specified criteria. For inventions by lasting impact, some decades are clearly superior to others. The 1750s, for instance, basically had nothing. I can't find a single thing invented in the 1750s that we still use today or inspired further inventions directly. I'm sure there's a couple, but I would argue that since it's hard to find them they're relatively unimporant.

The first decade of the 1800s? The first modernish battery, the arc lamp, morphine, the steam locomotive, the first automobile, the steamboat.

Perhaps it's _silly_ to rank decades, but I don't think it's nonsense.

<comment><USER8> Ben Franklin invented the lightning rod in the 1750s. Harrison's marine chromometer. Achromatic lenses: interestingly, they were really from the 1730s and then sat on: [https://en.wikipedia.org/wiki/Achromatic_lens](https://en.wikipedia.org/wiki/Achromatic_lens). The navigational sextant: also found in Newton's unpublished writings. (But I'm disinclined to feel smug about these delays, compared to some bits of computing history.) "First high-quality cement using hydraulic lime since Roman times." The [https://en.wikipedia.org/wiki/Lever_escapement](https://en.wikipedia.org/wiki/Lever_escapement). Lind's treatment of scurvy, and an early attempt at vaccination. Some chemistry of practical import.

So, I agree: the 1800s beat the 1750s in the invention cage-fight.

Source: [https://en.wikipedia.org/wiki/1750_in_science](https://en.wikipedia.org/wiki/1750_in_science) and following, from which I left out science and math discoveries.

<comment><USER7> I don't know how the lightning rod didn't come up in my googling!

Total aside, re: the chronometer, the 1750s definitely was not when it came into existence. He may have refined the design during that time, but the H1 chronometer was completed during the 1730s. I'm not sure if there's an exact date around for its completion.

<comment><USER9> H4 was the one that was good enough, no?

<comment><USER7> The H4 was completed in 1761.

</comment></comment></comment></comment></comment></comment></comment><comment> bra-ket also

\- Lilienthal gliders developed in the 1880s that inspired Wright brothers to create the first airplane: [https://en.wikipedia.org/wiki/Otto_Lilienthal](https://en.wikipedia.org/wiki/Otto_Lilienthal)

\- Heaviside's work on Maxwell equations

\- Boltzmann's work on thermodynamics

\- Hertz discovers the photoelectric effect and radio waves

\- Hollerith receives a patent for electric tabulating machine, one of the first computers

\- Design of rocket propulsion by means of explosives (a precursor for Project Orion): [https://en.wikipedia.org/wiki/Nikolai_Kibalchich#Final_lette...](https://en.wikipedia.org/wiki/Nikolai_Kibalchich#Final_letter)

<comment><USER10> The 1880s also saw Chicago's population grow by 118% and New York's by 23%.

Compare that to today, when we treat urban population booms of 5% a decade as 'breakneck growth' (e.g. San Francisco, from 777k to 815k from 2000 to 2011).

<comment><USER11> Chicago in 1880 was starting with a pretty small base, though...

<comment><USER12> 1880: 503,185

1890: 1,099,850

<comment><USER11> OK, bigger than I thought. Big enough that the comparison to SF is not totally invalid...

Except that SF has the problem that almost nobody can afford to _live_ there, so everybody commutes. So how big is the SF metro area? 7,000,000 in 2000 - though that includes San Jose and Oakland. Those are places that send commuters to SF, true, but a bunch of them don't go there too, and so it's getting really hard trying to figure out how to come up with validly comparable numbers...

<comment><USER10> Why does SF have that problem? To a significant degree, because more people can live in buildings like this: [http://cache- graphicslib.viator.com/graphicslib/media/c8/ver...](http://cache- graphicslib.viator.com/graphicslib/media/c8/very-busy-paris-streets- photo_1489864-770tall.jpg) than buildings like this: [http://c718756.r56.cf2.rackcdn.com/26000/93334/2010-08-25/35...](http://c718756.r56.cf2.rackcdn.com/26000/93334/2010-08-25/35b9b123d2a5a4838c5cab39f69780ce.jpg) yet it's illegal to replace the latter with the former in SF. Unlike, say, Chicago in the 1880s.

<comment><USER13> The bulk of Chicago _isn 't_ highly multistory buildings, and most of the early skyscrapers were office towers, not residential.

What Chicago has that San Francisco hasn't is _land_. The city sprawled both north and south along Lake Michigan, and westward.

San Francisco has less space in the city proper, yes, but it also wasn't filled out fully until the 1950s. Now, of course, it _is_ fully built up, and the only way to add more people is to grow up rather than out.

Chicago, though, _also_ has highly developed suburbs, which were principally served until the 1960s by commuter rail, radiating out in all directions. Contrast with SF which had only one line south (CalTrain) through what remains a comparatively sparsely developed area (the Peninsula), and limited East Bay service (though the Key System _did_ exist until the 1950s).

It's simply easier to build out more metropolitan infrastructure in the midwest.

Chicago is also within reasonable rail distance of much of the East Coast. By the late 1850s, it was less than two day's travel from New York, and well under one by the 1930s (about the same, by rail, as now).

[http://railroads.unl.edu/documents/view_document.php?id=rail...](http://railroads.unl.edu/documents/view_document.php?id=rail.str.0241)

San Francisco was four _weeks '_ travel in the 1850s, and still 3-4 days by the 1930s.

Today, California is increasingly practically limited by water access.

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> [https://en.wikipedia.org/wiki/Michelson%E2%80%93Morley_exper...](https://en.wikipedia.org/wiki/Michelson%E2%80%93Morley_experiment)

Happened 1887. A load of other things mentioned by other people here, too.

I wonder if there was some sort of geographic centre to it. A lot of it sounds like it happened in the US.

</comment><comment><USER15> This is pointless: If you let me define the terms, I can 'prove' that the Ancient Romans invented radio, computers, and whatever else, and nothing of note has been created since 500 CE. Picking out the 'first' of something is _always_ an exercise in defining terms, meaning the person who does it gets to pick the winners by defining some key words correctly.

(First computer: The rock. It enables partially-mechanized computation. The abacus is merely a variation on the theme of using pebbles to count, and nothing later counts at all.)

(First radio: Using torches to send messages. After all, all light is in the electromagnetic spectrum, just like the modern FM band.)

</comment><comment><USER16> This author doesn't seem to be familiar with Belle Époque [https://en.wikipedia.org/wiki/Belle_%C3%89poque](https://en.wikipedia.org/wiki/Belle_%C3%89poque)

The Gilded Age from 1871 until we screwed it all up with WW I is well known in history as an age of major advancement.

</comment><comment><USER17> Coinciding with the Long Depression of 1873-1896? [https://en.wikipedia.org/wiki/Long_Depression](https://en.wikipedia.org/wiki/Long_Depression)

Those good old technological explanations for secular stagnation don't look great over longer timescales.

</comment><comment><USER18> Question: If you can choose any 100-year span in human history to live, which one would you chose?

My answer was 1869 to 1969. Imagine the world without cars, electricity, airplanes, telephones, skyscrapers... You start from there. And then within 100 years man lands on moon. You would have witnessed two biggest wars in human history, multiple cultural revolutions, art revolutions, medical revolutions... It would be the most interesting years in the human history to live. Now compare that to our time span :).

<comment><USER19> I am quite happy to be missing out on my share of world wars thought I read plenty about them.

Otherwise I often think I was born too late, that I missed so many opportunities if I had just been a few years older. If I had only bought Google when I had the chance, if I hadn't been such a young fool, if I was in the work place during the boom I could have gotten lucky. Oh man the ifs go on forever. So I throw up my hands and look around to see if I can avoid an if today.

</comment></comment><comment><USER20> And how many stem cell treatments were available in 2005?

There are lots of innovations in tech happening outside of microprocessors.

<comment><USER13> How much are stem cells improving quality of life?

How much did clean water, flush toilets, and municipal waste removal improve quality of life and life expectancy?

[https://ello.co/dredmorbius/post/MsQfdPAn_0XUdZUoReBfbg](https://ello.co/dredmorbius/post/MsQfdPAn_0XUdZUoReBfbg)

[https://d324imu86q1bqn.cloudfront.net/uploads/asset/attachme...](https://d324imu86q1bqn.cloudfront.net/uploads/asset/attachment/1468664/ello- xhdpi-281770e7.jpg)

</comment></comment><comment><USER21> Biotech? Nanotech? Transportation? Software? Information accessibility? Social technologies? Genetic engineering? Space travel? Artificial intelligence?

We do a lot more than processors these days.</comment>


Launch HN: Humanloop (YC S20) – A platform to annotate, train and deploy NLP - <AUTHOR> Hey HN.
We're Peter, Raza and Jordan of Humanloop (https://humanloop.com) and we're building a low code platform to annotate data, rapidly train and then deploy Natural Language Processing (NLP) models. We use active learning research to make this possible with 5-10x less labelled data.
We've worked on large machine learning products in industry (Alexa, text-to-speech systems at Google and in insurance modelling) and seen first-hand the huge efforts required to get these systems trained, deployed and working well in production. Despite huge progress in pretrained models (BERT, GPT-3), one of the biggest bottlenecks remains getting enough _good quality_ labelled data.
Unlike annotations for driverless cars, the data that's being annotated for NLP often requires domain expertise that's hard to outsource. We've spoken to teams using NLP for medical chat bots, legal contract analysis, cyber security monitoring and customer service, and it's not uncommon to find teams of lawyers or doctors doing text labelling tasks. This is an expensive barrier to building and deploying NLP.
We aim to solve this problem by providing a text annotation platform that trains a model as your team annotates. Coupling data annotation and model training has a number of benefits:
1) we can use the model to select the most valuable data to annotate next – this "active learning" loop can often reduce data requirements by 10x
2) a tight iteration cycle between annotation and training lets you pick up on errors much sooner and correct annotation guidelines
3) as soon as you've finished the annotation cycle you have a trained model ready to be deployed.
Active learning is far from a new idea, but getting it to work well in practice is surprisingly challenging, especially for deep learning. Simple approaches use the ML models' predictive uncertainty (the entropy of the softmax) to select what data to label... but in practice this often selects genuinely ambiguous or "noisy" data that both annotators and models have a hard time handling. From a usability perspective, the process needs to be cognizant of the annotation effort, and the models need to quickly update with new labelled data, otherwise it's too frustrating to have a human-in-the-loop training session.
Our approach uses Bayesian deep learning to tackle these issues. Raza and Peter have worked on this in their PhDs at University College London alongside fellow cofounders David and Emine [1, 2]. With Bayesian deep learning, we're incorporating uncertainty in the parameters of the models themselves, rather than just finding the best model. This can be used to find the data where the model is uncertain, not just where the data is noisy. And we use a rapid approximate Bayesian update to give quick feedback from small amounts of data [3]. An upside of this is that the models have well-calibrated uncertainty estimates -- to know when they don't know -- and we're exploring how this could be used in production settings for a human-in-the-loop fallback.
Since starting we've been working with data science teams at two large law firms to help build out an internal platform for cyber threat monitoring and data extraction. We're now opening up the platform to train text classifiers and span-tagging models quickly and deploy them to the cloud. A common use case is for classifying support tickets or chatbot intents.
We came together to work on this because we kept seeing data as the bottleneck for the deployment of ML and were inspired by ideas like Andrej Karpathy's software 2.0 [4]. We anticipate a future in which the barriers to ML deployment become sufficiently lowered that domain experts are able to automate tasks for themselves through machine teaching and we view data annotation tools as a first step along this path.
Thanks for reading. We love HN and we're looking forward to any feedback, ideas or questions you may have.
[1] https://openreview.net/forum?id=Skdvd2xAZ – a scalable approach to estimates uncertainty in deep learning models
[2] https://dl.acm.org/doi/10.1145/2766462.2767753 work to combine uncertainty together with representativeness when selecting examples for active learning.
[3] https://arxiv.org/abs/1707.05562 – a simple Bayesian approach to learn from few data
[4] https://medium.com/@karpathy/software-2-0-a64152b37c35 <comment><USER2> This looks pretty great, though the SaaS model is an absolute non-starter for my own usage unfortunately. We've been pretty prolific users of Explosion AI's (Makers of SpaCy) Prodigy [1] and actually the interfaces look very similar. What would you say the core differences are between Humanloop and Prodigy?

1: [https://prodi.gy/](https://prodi.gy/)

<comment><USER3> Thanks! Prodigy is a good tool and we definitely were inspired by some of their UX decisions. Reducing each decision to a small atomic unit and avoiding context switching makes a lot of sense.

Our starting place is similar to Prodigy in that we also see active learning as a key piece of the puzzle but we think to make active learning work reliably really does need taking into account parameter uncertainty. As far as I know Prodigy doesn't do this. We are also working to make our active learning work at the level of batches and be cost-aware. Often the most valuable examples to label for the model are the most time consuming for humans and we work to trade this off.

A few other differences are that we do offer a cloud hosted solution so getting set up is much faster and it's more natural for us to be able to accomodate team annotation and quality assurance. By providing a hosted model we also give you the option of deploying features very quickly and continuing to improve them post deployment.

I'd be curious to know the barriers that Saas introduces for you?

<comment><USER2> Good to know! I work with a lot of highly classified information. Much to my teams frustration, the notion that any of which will be cloud hosted on GCP or AWS was laughable until somewhat recently. It will probably be years until we, and I imagine many institutions like ours, can take advantage of something like this. I appreciate the effort with the VPS / private cloud hosting option, but without an on-prem deployment option, this wouldn't even make it passed initial discussions.

</comment></comment></comment><comment><USER4> [https://jacobbuckman.com/2020-01-17-a-sober-look-at- bayesian...](https://jacobbuckman.com/2020-01-17-a-sober-look-at-bayesian- neural-networks/)

"But in practice, BNNs do generalize to test points, and do seem to output reasonable uncertainty estimates. (Although it's worth noting that simpler approaches, like ensembles, consistently outperform BNNs.)"

</comment><comment><USER5> It looks awesome!

I see the snorkel logo on the website and they recently also launched snorkel flow for data annotation and model training. There isn't much detail on that, but I wonder is there any advantage humanloop has over that?

On the same track, prodigy also has a prodigy team version that is being ready for launch forever. So glad you guys are few steps ahead.

I am also building a labeling interface myself because I couldn't find the right product for my needs(I have tried tools like label studio, doccano, prodigy, dataturks and ml annotate). They just miss one thing or the other. I really wish there is one place that features like HTML support, hierarchical labels, active learning, batch labeling, project tracking, multi user management and most important the UI/UX are all well put together.

<comment><USER3> Hi, sorry I missed this is earlier! We're in many ways complementary to weak labelling techniques (like Snorkel) and are actually working to include them in the tool. We think weak labelling is a great way to overcome cold starts and active learning then helps you improve rapidly.

The big difference between us and Snorkel is our emphasis on active learning and HITL deployment. We think the existing paradigm of ML deployment is very waterfall and slow.

Would love to hear about the annotation interface your building. Agree there should be one place with all those features! (We're hoping it will be Humanloop ;) ).

</comment></comment><comment><USER6> Is this something we will be able to buy and run on our servers? I don't think we're the only ones wary of working hard to develop IP for a different company.

Also predictions/month pricing is just really challenging and incompatible with many downstream business models. The value has to be really huge to justify that.

<comment><USER3> The model you train and data you upload are yours to own, unique to you and don't get shared across users or reused for any other tasks so hopefully you shouln't feel too much like your building our IP ;)

In terms of deployment options, we're trying to lead with cloud hosting by default but know that for a lot of people the whole reason they're annotating in house is privacy so we've been exploring deploying in your VPC and for larger enterprises on-prem.

Interested to hear more of your thoughts on the pricing model, this is something we're still iterating on so I'd be interested what you think would be most compatible with your use cases?

</comment></comment><comment><USER7> Neat how do you compare yourself on the annotation capabilities with Datasaur.ai which launched in the last YC batch?

In terms of training the models for deployment -- do we own the artifact? Can I move that into my own model repository?

Also how do you feel this compares to using fine tuning on a publicly available BERT family model which is already fairly fast and easy not requiring a huge corpus, speaking from experience of recently having done so?

Are the benefits more from the tight feedback loop and already standing infrastructure?

<comment><AUTHOR> All great questions!

Datasaur are great. I hope Ivan would think it's fair that I'd describe their current product as as a modern, cloud-hosted Brat ([https://brat.nlplab.org/](https://brat.nlplab.org/) – this remains very popular!) with the features to make that work with teams. As you point out we're focusing on the tight integration of annotation and training enabling you to move faster and iterate on NLP ideas... essentially trying for move a waterfall ML lifecycle to a an agile one.

Fine tuning on BERT is the way to go. It's what we do, and that already reduces the data annotation requirements by an order of magnitude. Doing that offline in a notebook is still wanted by some (you can use our tool just as the annotation platform, and download the data and you'll still get the efficiency benefit through active learning) but integrating or deploying that model is still a time-suck. Having the model deployed in the cloud immediately has a load of supplementary benefits (easy to update, can always use the latest models etc) too, we hope.

(edit: typos)

<comment><USER8> Firstly, congrats on the launch! Active learning is a super interesting space.

You say it's possible to download the data and use Humanloop for annotation only while still benefitting from active learning. I'm curious about your experience with how much active learning depends on the model. Are the examples that the online model selects for labelling generally also the most useful ones for a different model trained offline?

<comment><AUTHOR> Cheers. It's a good thing to be wary of. Poor use of active learning will end up biasing the data according to the model it's trained on – so that data won't be the best X samples to train on a different model. Most of this issue comes from bad active learning selection methods. If you have well calibrated uncertainty estimates and sample for diversity and representiveness too, it's far less of a concern.

</comment></comment></comment></comment><comment><USER9> #1 and #2, if they work as advertised, are great features but a lot of other companies claim to do this but have failed.

One of the biggest problems I have is image annotation using CVAT - the tool works when the task is simple annotation but outputting the annotation data and integrating it has been a pain-point. Also CVAT has a tool is great but has a lot of missing features :/

<comment><USER10> Related: Diffgram is working in the Vision (Image & Video) space. Not NLP yet.

Integration paint points are mentioned often. We are working on solutions here, eg: [https://www.youtube.com/watch?v=w7yiW5wpnMg&t=128s](https://www.youtube.com/watch?v=w7yiW5wpnMg&t=128s) Imagine adding bucket event triggers as next step here

Some really exciting features coming soon that make this even better. [https://diffgram.readme.io/docs/what-is- diffgram](https://diffgram.readme.io/docs/what-is-diffgram)

Can try shared platform and do private install for actual [https://diffgram.com/user/new](https://diffgram.com/user/new)

We would love your feedback on missing features please feel free to email me directly email@example.com

</comment></comment><comment><USER11> This workflow is still so complex to get right. Really excited to see more tools for it and try it out ourselves!

At visitorX we're building a fairly large bank of comments and a tagging system and Humanloop looks really great for that.

</comment><comment><USER12> Cool! I'm curious, how do you compare your service to Scale (scale.com)?

<comment><USER3> Hi,

Raza here (one of the other co-founders). Good question! I think our visions are quite different even if our starting points look similar.

Scale has always positioned themselves as an API to human labour and their goal is to abstract the labelling task away from the end user as much as possible. So scale works really well when you can easily outsource your annotation task.

Our ultimate goal is to try and give domain experts the ability to teach ML models themselves. We're much more focussed on NLP and on tasks that require domain expertise and are hard to outsource. For people where deep domain expertise matters or their are privacy concerns, Scale isn't really an option and we're building tools for them.

On another point, Scale makes its money by charging per annotation so we think they aren't as incentivised to reduce how much you need to label.

thanks!

</comment></comment><comment><USER13> Is it English-only or true NLP that would work with multiple languages? Congrats for the launch!

<comment><USER3> We wrap a lot of popular frameworks and have implementations of most SOTA models. By default we use a multilingual BERT model so it should work out of the box on different languages.

</comment></comment><comment><USER14> Humanloop is such a great name for an AI platform :) Congrats on the launch!

<comment><AUTHOR> haha so great to hear! For a while google search kept trying to auto correct it to 'human poop'

</comment></comment><comment><USER15> This is really cool! Wish you all the best of luck!!

</comment><comment><USER16> Do you allow for on-premise inference?

<comment><USER17> Our default deployment option is cloud first for both training and inference at the moment, but we have thought about the ability for users to export a trained model. Either exporting the model parameters in some standardised format, or a compiled predict function, or a docker image that encapsulates a full inference service, etc. So if you could use this kind of export within your application, this would allow on-premise inference. This is something we could probably make available pretty quickly if necessary for your use case.

</comment></comment><comment><USER18> What type of annotations do you offer?

<comment><AUTHOR> Right now document level classification and span tagging within text documents. These can also be combined (as in the landing page screenshot) so that for a given input, you're learning multiple tasks at once as you annotate.

The core of this platform should generally be independent of the data input type and the output labels, so we're building out other annotation options for our business customers. If there's a use case you would like it to support, it would be great to chat jordan[at]humanloop.com :)

<comment><USER19> >> text documents

Congrats on the launch! Would Humanloop be able to support HTML files or URLs? A client of ours has a need to annotate verbose web pages.

<comment><USER3> At the moment we dont support the ability to render the HTML but it is something that has come up before. One of the teams we're speaking to wants to classify blog posts and would like to be able to preserve their formatting. If this is something that's important to you we would consider adding it so maybe drop me an email at raza[at]humanloop.com and we can discuss?

<comment><USER19> Thank you for your reply. Yes, preserving the formatting is important for us too.

</comment></comment></comment></comment></comment><comment><USER20> Great stuff!</comment>


USA dismantles env. Pro. Laws - <AUTHOR> I wonder why this just goes unchallenged again and again.<p>https://www.scientificamerican.com/article/trump-administration-completes-climate-dismantling-with-methane-rollback/ <comment><USER2> Please use the original headline of the article. Add own commentary in a comment after submission. [https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

</comment><comment><USER3> Because the law gives authority to agencies to create rules, and the people who direct those agencies make the rules they want. The problem is that the law gives a lot of leeway for creating policies. This leads to rules changing when new administrations take power. Just look at all the executive orders that get revoked and instated when a president is elected.</comment>


[2016] Why are EU regulators going after Android?–It's all about the Play Store - <AUTHOR> https://www.theverge.com/2016/4/21/11477032/android-antitrust-eu-google-play-store <comment><AUTHOR> In light of the recently announced fine [1] and Google's response [2], I thought, some people might be interested in more information on the circumstances leading up to this point.

Another good read is this text from 2014: [https://mobiforge.com/news- comment/android-forks-why-google-...](https://mobiforge.com/news- comment/android-forks-why-google-can-rest-easy-for-now)

And the recent change preventing logging into Google accounts on uncertified devices: [2018] [https://arstechnica.com/gadgets/2018/03/google-starts- blocki...](https://arstechnica.com/gadgets/2018/03/google-starts-blocking- uncertified-android-devices-from-logging-in/)

[1]: [https://news.ycombinator.com/item?id=17556497](https://news.ycombinator.com/item?id=17556497)

[2]: [https://www.blog.google/around-the-globe/google- europe/andro...](https://www.blog.google/around-the-globe/google- europe/android-has-created-more-choice-not-less/)</comment>


Tell HN: Recruitment ads for TheMuse every other week gives negative impression - <AUTHOR> <comment><AUTHOR> An ad is posted every week by the looks of it: [https://news.ycombinator.com/from?site=themuse.com](https://news.ycombinator.com/from?site=themuse.com)

Is it such a bad workplace that they cannot fill the positions or why is this spam necessary?</comment>


Computers Are Taking Design Cues from Human Brains - <AUTHOR> https://mobile.nytimes.com/2017/09/16/technology/chips-off-the-old-block-computers-are-taking-design-cues-from-human-brains.html <comment><USER2> That article took a very long time to say that computer chips are becoming more specialized (like brains! because brains aren't homogenous).

Just a bunch of fluff.

<comment><USER3> Thank you for the heads up, this is exactly why I read comments before the article.</comment></comment>


Interpreting Analog Sticks - <AUTHOR> http://blog.hypersect.com/interpreting-analog-sticks/ <comment><USER2> Great post. Over of my favorite game hacks is players who carve out their controllers in games that don't clamp magnitude. Let's them move faster in diagonals. Sneaky sneaky!

<comment><USER3> Another trick, which can be used in some games such as Doom 64, is to reset the neutral position such that down is neutral, so up is 2x speed.

<comment><USER4> You can do this with a lot of gamecube games because the system would set to default what ever values the system was receiving at bootup; turned out to be a great way to avoid costs by having different manufactures make the controllers

</comment></comment></comment><comment><USER5> I remember using a crappy joystick in the DOS racing game Screamer. The acceleration/steering was mapped to the unit square, but my joystick had a circular hole in the base, limiting movement. It made it impossible to turn around, since you couldn't accelerate and give max steering at the same time.

I fixed the joystick with a metal file.

[https://youtu.be/Ka56PLmXCqI?t=28](https://youtu.be/Ka56PLmXCqI?t=28)

<comment><USER6> What was the name of the joystick? Do you have a photo of it so I can understand what you mean?

<comment><USER5> It has been long lost, I think. Maybe my brother still has it.

I can't find a pic of the exact model, but theese illustrates my point.

Compare the holes in the bases where the stick is attached:

Square: [https://www.google.se/imgres?imgurl=http%3A%2F%2Fimages10.ne...](https://www.google.se/imgres?imgurl=http%3A%2F%2Fimages10.newegg.com%2FProductImage%2F26-280-020-06.jpg&imgrefurl=http%3A%2F%2Fwww.newegg.com%2FPC- Game- Controllers%2FSubCategory%2FID-123&docid=NebpvZDkpxDqrM&tbnid=VWTpcwk0Up3mvM%3A&w=640&h=480&hl=en- se&client=safari&bih=559&biw=375&ved=0ahUKEwiN3dyNyf_MAhXFXSwKHfExAbMQMwgvKA0wDQ&iact=mrc&uact=8)

Circle: [https://www.google.se/search?q=joystick&client=safari&hl=en-...](https://www.google.se/search?q=joystick&client=safari&hl=en- se&prmd=ivn&source=lnms&tbm=isch&sa=X&ved=0ahUKEwiS0-WLyf_MAhVBjSwKHXdqDqAQ_AUIBygB&biw=375&bih=559#imgrc=-ngP3P3xN6Y-8M%3A)

<comment><USER6> Thanks :)

</comment></comment></comment></comment><comment><USER7> Good thing tuning's a thing. I always knew friends' controllers were not equally matched and it was not a level playing field. For this reason alone wanted my own system. Well described concepts and illustrations and I enjoyed the humor in the writeup.

 _" With controller age and abuse, this area will grow, shift and distort. Every value within the inner dead zone is mapped to zero, giving us a reliable origin point for the game logic."_

</comment><comment><USER8> I wonder how difficult would it be to infer the actual sizes of deadzones of this particular controller from inputs during gameplay.</comment>


Are you living in a Computer Simulation? - <AUTHOR> http://fermatslibrary.com/s/are-you-living-in-a-computer-simulation <comment><USER2> What about the chance of failure of a simulation? When you believe that simulations are nested like virtual machines inside virtual machines even a small chance of failure would make it highly unlikely to live in a very deeply nested simulation and thus increasing the chance that you are near to the physical reality.</comment>


JL on Female Founders - <AUTHOR> http://www.foundersatwork.com/1/post/2008/07/female-founders.html

<comment><USER2> I think that natural selection might explain it.

What characrerises reproductively successful men? Or, to put it another way, what qualities do women seek in a mate? Women tend to prefer dominant men of high status with lots of money, and there is little reason to believe that their ancestors differed. For much of human history male status came from being a skilled hunter and warrior and from the ability to influence others through muscle or wit. This history has left men more disposed than women to strive for status and engage in the risky, competitive and sometimes aggressive behaviour often required to do a successful startup. Women with a taste for winners would likely be in a position to pass their preference on to more children than would women with a penchant for failures.

Women would not ordinarily enhance their reproductive success through direct competition and risk-taking. Indeed, it would promise little reproductive payoff and could be a very dicey strategy because a bad outcome would imperil a woman's future reproduction and the well-being of existing children. Instead, women, to a far greater extent than men, have enhanced reproductive success by looking after their children as they grow up, resulting in stronger bonds between mother and child than between father and child.

This explanation also hints at the fact that women would make good CEO's of large corporations where security is more important than risk taking.

Note: this isn't in in any way meant as a degradation of women - merely a possible explanation of the cause.

<comment><USER3> Unfortunately, your explanation does not go the whole way.

Consider the evolution of biological genes:

a) Men and women share mostly the same genes (the y chromosome is a desolate wasteland), and there are few (if any) known mechanisms by which gene merging is different for male and female humans.

b) Dominant, decisive, intelligent, women are likely to share similar evolutionary advantages to men. Note that I leave high status out of this equation.

Compare the mathematics to the evolution of culture:

a) There are numerous known cultural differences between what men and women grow up with. There are also many known mechanisms for which cultural _meme_ merging is different for men and for women. Our culture directs different memes towards men and women, to a far greater extent than gene selection does (if it does at all).

b) High status women are in fact less likely to have many kids, as is shown by an abundance of demographic data. Therefore, the mechanism by which they can spread what's responsible for 'high-statusness' is limited mostly to memes. But culture strongly moderates this influence of dominant, high status, intelligent women. Consider the treatment Hilary Clinton is getting now. Even in a more ordinary circumstance, male bosses can wield power in a way that would label their female cohorts 'ice queens'. So the influence of powerful women is labeled a bad thing, and the adoption of those values by other women is severely inhibited.

\---

So, for this case, at first glance, it seems like the evolution of cultural memes completely dominates the influence of the evolution of biological genes. Does anyone see a hole in this argument?

<comment><USER4> Yes. The reference to cultural memes is a chicken an egg argument; by claiming "complete domination" you essentially overlook the origin of said memes.

I'm not armed with sources, but I daresay that nature has given everything specialization, so much so that applying the same ideas to the sexes is not the slightest bit strange. More strongly said, there is nothing surprising when one sex exhibits traits much more ostensibly than another. While I am not referring to the superiority of a specific sex, specie, or trait, it actually seems more unnatural to claim equality on things like attitudes towards risk and attribute difference to culture.

Back to your arguments specifically. These come from a fuzzy memory, but they provide a complementary view.

a. Even a single genetic difference can lead to drastically different results. We are close enough to chimps. Men and women have different brains; different areas activate up when shown the same picture. It's not a stretch to say that risk evaluation is done differently as well.

b. I recall at least two matriarchial societies in which women called the shots. Nevertheless, the men were still the fighters, which is perhaps the most competitive, aggressive, and risky behavior possible. Women can be dominant, intelligent, and decisive, and still partake in less adrenaline- laden activities. There may be something else in the motivation.

a. I think this is a good and important point, and am not trying to underplay it, but this argument is usually the convenient one, and the hardest to make any systematic observation. Thought experiment: try comparing startup founder numbers of Japanese men to those of American women.

b. Again, true enough, but also: the sperm of a high-status males, whom are shown to produce more offspring, still hold good chance to yield daughters.

Does this poke a hole in your argument?

<comment><USER3> _you essentially overlook the origin of said memes._

I don't overlook the origin. Your thrust seems to be that culture only magnifies innate difference. Perhaps this is true in simple cases, but I think that there are particular examples where culture takes on a mind of its own.

I point out that existing cultural bias might be due to biological bias interacting with environmental factors long extinct. This bias is encoded and preserved, unlike almost everything else in the animal kingdom, in human culture, which can be perpetuated and passed on. Existing cultural bias might just be a carry over from prehistoric times, entirely irrelevant to whether or not there are now meaningful differences in aptitude, in this, the present.

 _it actually seems more unnatural to claim equality on things like attitudes towards risk and attribute difference to culture._

I never claimed equality, I simply pointed out that the differences you'd expect due to biological selection are smaller in magnitude than those due to cultural selection, and so therefore born-attitudes towards things like risk should be largely outweighed by cultural ones.

My mind wouldn't scream inconsistency if I heard that women, on average, were intrinsically, say, 30% less likely to take big risks. That kind of difference seems reasonable. But the data on entrepreneurship implies a difference many dozens of times that.

I don't think this magnitude is consistent with other risky behaviors -- many women do risky things as well, like dating strange men in remote places, running away from home, abusing hard drugs, or keeping a pregnancy in ill- health. This difference in magnitude may be explained by cultural factors, because men have greater cultural support when projecting high-status creating cultural memes (especially those encouraging risky behavior).

 _this argument is usually the convenient one_

Perhaps I was unclear. This is a very different argument than the one typically given, because it uses the very same mathematics that the evolutionary biologists use, only applied to culture. And what you find is that the factors in culture -- the influence of the same kind of mathematics, is much stronger.

 _the sperm of a high-status males, whom are shown to produce more offspring, still hold good chance to yield daughters._

Can you explain the relevance of this?

<comment><USER5> The easiest way to prove something is genetic rather than cultural is to examine different cultures throughout the world, especially isolated ones like pygmies or aborigines.

The paradigm of man as provider and woman as nurturer, and the mating ramifications of that (i.e. women preferring men who are able to provide, which most closely translates in our society to wealth, and men preferring women more fit to raise children, which in our society most closely translates to healthy appearance) are virtually universal.

That would strongly suggest that it is genetic rather than cultural.

<comment><USER3> _The easiest way to prove something is genetic rather than cultural is to examine different cultures throughout the world, especially isolated ones like pygmies or aborigines._

I agree that this is easy, but I am not sure that it is in fact accurate. To do so you must show that the particular cultural values you're exploring do not share anthropological linage. This is a very difficult problem. How do you propose to handle that case?

 _The paradigm of man as provider and woman as nurturer, and the mating ramifications of that (i.e. women preferring men who are able to provide, which most closely translates in our society to wealth, and men preferring women more fit to raise children, which in our society most closely translates to healthy appearance) are virtually universal._

So would you draw the line from this evolutionary argument like so: men have a greater instinct to produce wealth or high status, which causes them to produce more startups.

The problem I have with this is that's its so vague. If we don't actually try to nail down or postulate an actual theory, people can be lazy and undisciplined in their thought. They might say, for example _"here are some good reasons that the sexes might be intrinsically different; any existing difference is just the way things are, due intrinsically, and due to genetics"_. My point is that the disciplined way to approach this question is that you actually need to make specific predictions, or you risk degenerating into religious war.

So, _in which ways_ is this instinct for men to provide manifested?

Compared to startups, law or medicine or finance seems a surer method to acquire status and prestige, and you probably come out ahead on average. More women are in these fields than in startups.

And physical science and engineering aren't particularly high prestige careers, nor do they make much wealth, nor do they seem to improve procreative chances. And these fields are particularly male centric.

So then usually someone points out that men like to take risks. They aim for the massive payoffs. If that's so, one should examine the _structure_ of risks that men take versus women.

So, we have a set of hypotheses, backed up in varying amounts by data. Men are much more likely to do one off things to impress people. I guess this is called machismo. They're much more likely to take physical risks, or health risks. They may be more likely to take risks in their social stature, so long as they have little to lose. They're more likely to risk the state of a relationship for some impulse, someone or something that they want.

By contrast, I postulate, women are more likely to risk themselves emotionally. They are more likely to invest themselves in one particular friendship or relationship, even if it risks not panning out. Men, by contrast, _fear commitment_. They're more likely to keep their eggs out of one basket (see especially, for example, studies on the messaging patterns and viewing patterns and requests and satisfaction on dating sites). Women are _more_ likely to invest themselves emotionally in some community, cause, person, or idea.

Doesn't this sound like a startup?

It's striking how many female entrepreneurs describe starting a startup like having a baby. Mena Trott (in founders at work) describes it like having a chemical in your brain that blocks out the painful moments, leaving only the other ones in memory (This effect is described also of pregnancy, and recently they've actually discovered that _such a chemical exists_ ). Are women more ready to devote themselves like this? Are women more emotionally prepared for a startup? Perhaps.

So, we have _conflicting_ attitudes towards risk. On one hand, we postulate, men are more likely to be driven by promises of extreme status and wealth. On the other, women are perhaps more emotionally prepared, in some areas, when they get there. It seems like this should be a fairly balanced game.

But it's not. Female founders make up 3% of the YC pool. The risk and wealth hypothesis can't handle this alone.

Something has to take up the slack. And I think that it has much to do with the fact that different memes that define success in this society propagate to men and to women.

<comment><USER5> You can't simply say that since men are more hesitant to commit to relationships, and women less, that the same holds true in startups or careers, or that it translates at all. Both genders exhibit behaviors where mating is concerned that are very different from how they behave in other situations.

You should really read up on evolutionary psychology, I think you might enjoy it.

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> _"... By nature, startups are very non-discriminatory ..."_

I don't know about that, maybe in a homogeneous ideal Startup.

Startups by nature tend to be a blended team soup of testosterone, technology and immaturity. Of the three I don't know which is worse. Testosterone is vital because it gives drive and competition. But has the downside of cruelty and bullying. Technology circles also tend to attract a certain cloistered _male_ culture (online or in meat-space). Hostile to difference and eager to argue. The nastiness occurs in arguments over technology is in part due to the makeup of Startup founders, Homo logicus. There is a Coding Horror article discussing Coopers observations if you are interested ~ <http://www.codinghorror.com/blog/archives/000091.html> The last of the Startup ingredients is "immaturity". There's a reason the term "grown-up" is bandied jokingly around Startup-up culture. It has less to do with _"Age"_ and more to do with understanding how the _"real world"_ works and how to deal with it. Put the founder team under enough stresses, let them get tired (because they are working hard) and put before them a decision that has to be made, now! Now place any male founder in this environment and tell me if _"discrimination"_ of some sort will not happen. Let alone a female.

It's no co-incidence that a question in the application asks how long the founders have know and presumably worked together. Because this is a sort of heuristic of future success. It's interesting that Mitch Kapor discussed the issue of poor behaviour in Startups (but not specifically with founders). Behaviour that would never be tolerated in big business at StartupSchool '07 ~ [http://feeds.feedburner.com/~r/Ycombinator- StartupSchool/~3/...](http://feeds.feedburner.com/~r/Ycombinator- StartupSchool/~3/106570486/Startup_School_2007-Mitch_Kapor.mp3) [ approx 20Mb] Kapor was most proud of the culture created where good behaviour resulted in a non-discriminatory workplace. It must be easier to do this when you build a 1000 employee company and have a Business background and wide experience.

</comment><comment><USER7> The numbers alone don't work out. 48% of the class or 06 at MIT were female (up from like 2% back in the day) and most of them were serious hackers. Yet today almost all of the girls who didn't go to grad school are working for either one of the big tech companies (Oracle, M$ etc) or an investment bank.

With the whole goddamn responsibility of keeping humanity alive on top of them undoubtedly there's a genetic disposition for women to be more risk averse. That though probably does not matter as much because women have clearly been extremely competent in pretty much every other field now for decades.

So the problem has to lie in social conditioning. Cultural expectation and etiquette has always been the most powerful destroyer of human potential in both men and women, from the day they are born (and thus being most effective at it)

Heard this joke? Why do baby boys wear blue and baby girls wear pink?

Cos they don't have a fucking choice!

</comment><comment><USER8> No one's going to even suggest sexism?

Or the fact that many, many people (men and women) look at any woman between 25 and 40 and expect them to leave to have a baby and raise a family?

<comment><USER9> Women have stormed the academy in most fields, and around half of new medical doctors are female. I think it's time to retire knee-jerk sexism as the main explanation for the lack of women in tech.

While I'm confident that women face more barriers than men, it's dubious that sexism is sufficient to explain such extreme numbers. The same goes for math professors, Wall Street traders, and the NBA for that matter.

More smart women choose to become doctors or lawyers, it seems, than to study physics or to roll the dice on a startup. And a lot of those 25-to-40 year old women _want_ to have a baby and raise a family, even if it means a less resume-impressive career. (My sister, an M.D. who specifically chose a family- friendly ophthalmology specialty, is one of them.) And what's wrong with that?

</comment></comment><comment><USER10> This is sad, but from a biological perspective, it's oddly tautological. Men are organisms that run huge risks in hopes of huge payoffs, while women run smaller risks for more certain rewards. One could consider a male lion to be an entrepreneur, and a lioness to have a steady job. However, in nature both strategies are equally successful, while in the modern world being an entrepreneur is more effective.

On the flip side, men are incarcerated much more frequently than women, and for similar reasons. This is somehow not as surprising.

</comment><comment><USER11> I'd have to think that making a start up succeed is so hard that any gender barrier, whether perceived or real, would seem minor in comparison.

<comment><USER3> One of the principles that YC is founded on is that many of the very best founders were sitting on the fence. So even if there's some minor force pushing one group of people backwards instead of forwards, you'll discover a big difference in the number of people on each side.

</comment></comment><comment><USER12> So, geeks are mostly male right now. It makes sense that geek startups will be mostly male, as well.

It's very difficult for women to break into any field that is male dominated. The fact is, just being different causes a woman to be treated differently in a male-dominated environment. That makes it hard and un-fun. This is exacerbated by the fact that some guys really are jerks, and enjoy exerting the power that comes with picking on someone in the minority -- women, in this case.

If something is more difficult, there are fewer people who are willing to do it, regardless of gender, race or creed. So when women break into a male dominated profession, it first it starts out at a trickle; just those who are very brave.

I really hope that this changes pretty soon. Women have a lot of things to bring to the table that men don't -- namely, they're (on the average) familiar with a lot of markets that men aren't (on the average) as familiar with. There's a lot of money to be made there, and a lot of cool new stuff to be invented.

</comment><comment><USER13> So I wanted to make sure no-one posted this view point before.

1stly in a modern 1st world country like the US or Japan birth-rates have been falling for along time. I think this a strong indication that gender roles are becoming less important in culture. Additionally you can see this in the current political strife on similar topics.

So to put all of those things aside, lets just look at the numbers. Marquette the school I went to had an engineering degree program with 40-60% female. It was the highest in the country while I was attending, and the topic of choice .....

Biology.

So, I had the chance to ask many of them why they chose that field and the response was ALWAYS, that they wanted to help people.

I think ability is not the root here, but it is the drive, and I think the women out there that have the personality and the drive to be entrepreneurs do not have an economic model that matches their interests.

I am willing to bet that as greentech becomes a viable industry to start a company you will see more women entering the entrepreneurship game. So, if it is still some sort of preference to help people and the world, why is that so? I don't know but thank God someone wants to do something more.

I think that if you are going to support a climate for female entrepreneur's you are going to have to find a way to create a community (YC seems to be a good start) and a way for the process to feel personable rather than abstract.

Jessica, maybe you can organize a few geek dinners to get female entrepreneurs together talking about what they would of like to have had when they were starting a company? I am sure you will find it much different than what the guys needed.

</comment><comment><USER14> From what I gather it's not so much that female founders are turned away from doing startups as they just never turn towards it to begin with... anybody out there with thoughts on why?

I have often wondered this. If one assumes the prototypical founder is a hacker, and the stereotypical hacker was a nerd in school... then we come back to: why aren't more nerds female? Are little girls discouraged from tinkering and making things in favor of other activities? I'm asking seriously, as I am a guy without sisters, or really any close female relatives, so I have no idea how girls are raised.

<comment><USER15> I don't think there's a single culprit. Now, I'm not a sociologist, and I'm simplifying here, but it seems to me that it starts in childhood with the selection of toys and judgments on character that parents and family members apply. Boys get Legos, girls get dolls. This gets reinforced in grade school where perhaps girls are rewarded more for excelling in "creative" activities while boys are subtly steered in a different direction. In high school, science and history books are almost devoid of women. Meanwhile television and movies continue to portray stereotypical views of women. By the time college and career roll around, you've got both subtle and explicit forms of discrimination to contend with -- not to mention a complete lack of role models at the CEO and founder level in Silicon Valley. So the cycle continues.

</comment></comment><comment><USER16> actually if you think about it, women would actually have a leg up when it comes to getting funding etc. Since there are so few of them, they have a much higher chance of actually getting to talk about their idea in front of VCs who want to see what they have to offer.

<comment><USER12> Why does there being fewer of them increase their likelihood of getting an opportunity to pitch? Because they are a curiosity?

<comment><USER16> Well I wouldn't use that exact word, but it works.

<comment><USER3> I really don't think that being a curiosity helps much.

<comment><USER16> hey if it gets you in the door to give a presentation to a VC, more power to you

<comment><USER3> Presentations are a cost. If they won't lead to funding, they give information to the VCs, they waste your time, and they get your hopes up and then slowly disappoint you.

If being a curiosity gives a negative first impression, this likely outweighs any additional time to speak.

</comment></comment></comment></comment></comment></comment><comment><USER17> If you want to make a startup, you do it. If you want to raise money, then sure, you do run the risk of not being able to. I think that could be for many reasons.

These days though bootstrapping is well within most peoples reach, male or female. There are no excuses.</comment>


Sweden's recycling is so revolutionary, the country has run out of rubbish - <AUTHOR> https://www.independent.co.uk/environment/sweden-s-recycling-is-so-revolutionary-the-country-has-run-out-of-rubbish-a7462976.html <comment><USER2> This article is non-sensical. Recycling is a huge technology problem. It requires ability identify several different kind of material precisely from a massive hodge-podge, classify them and treat them individually. I don't think anyone has solved this problem, let alone solving in cost effective way so you come out revenue positive. The most "recylers" take easy route of dumping, burning etc. This article says Sweden burns "recyclables" to generate heat for winter. I would assume they only do this for high quality papers. There is no technology advances described in this article.

<comment><USER3> Or you could get people to sort them into various recycling bins. This is how it is in Japan, amongst other nations. A good chunk of this recycling problem can go away with the right culture.

</comment></comment><comment><USER4> tldr: they "recycle" by burning everything

<comment><USER5> [https://www.treehugger.com/energy-policy/no-sweden-does- not-...](https://www.treehugger.com/energy-policy/no-sweden-does-not- recycle-99-percent-its-waste.html) says "they incinerate about 50 percent of their waste to make heat and energy."

Followed by "And even in their own website, they admit that is not the best approach, that it is not really recycling, and that it takes less energy to actually recycle and reuse than it does to burn and manufacture a replacement from scratch." ... "In summary: Incineration is not recycling, and therefore Sweden does not recycle 99% of its waste."</comment></comment>


Low Hanging Fruit of Programming Language Design - <AUTHOR> http://250bpm.com/blog:112 <comment><USER2> You know, sometimes duplicating code is just better. I've been around long enough to see many attempts at refactoring to _share code!_ crash and burn. The hoops and complexity that some people go through to DRY (don't repeat yourself) are worse sometimes than the cost of maintaining two copies.

<comment><USER3> > The hoops and complexity that some people go through to DRY (don't repeat > yourself) are worse sometimes than the cost of maintaining two copies.

The number of bugs are proportional to the lines of code, this is undeniable from empirical data. Ergo, fewer lines of code will tend to yield fewer bugs. So if your code is literally the same, there's no reason not to extract it into a function.

That said, the moment you have to start adding parameters in order to successfully factor out common code, ie. to select the correct code path taken depending on caller context, that's when you should seriously question whether the code should actually be shared between these two callers. More than likely in this case, only the common code path between two callers should be shared.

<comment><USER4> > So if your code is literally the same, there's no reason not to extract it > into a function.

If the two pieces of code are likely to change in different ways, for different reasons, that is a strong reason not to extract it into a function even if they happen to be character-for-character identical for the moment.

Less code means fewer bugs, but that doesn't mean I should be working on the gzipped representation.

<comment><USER3> > If the two pieces of code are likely to change in different ways, for > different reasons, that is a strong reason not to extract it into a function > even if they happen to be character-for-character identical for the moment.

The future is much more malleable than your immediate needs.

But even if your future turns out to be true, the code you need to refactor is then already extracted to a function, so you can easily duplicate that function, make your localized changes, and change the callers to the new function. So this is still the best route.

> Less code means fewer bugs, but that doesn't mean I should be working on the > gzipped representation.

Don't be absurd. gzipping doesn't preserve your program in human-readable form. Extracting code into a reusable function makes your program more human readable, not less.

<comment><USER5> > Extracting code into a reusable function makes your program more human > readable

When done properly, yes. When done to the point where a five line function is created with ten inputs (yes, this is real), no. But DRY tells us that the five lines of duplication is unconditionally _worse_.

Hell, I've even seen things like logging/write tuples (i.e. log the error, write the to a socket) encapsulated, even though the only non-parameter code ends up being the two function calls.

Anything, taken to extremes is bad. The problem with DRY is it encourages that extremism.

<comment><USER3> > When done properly, yes. When done to the point where a five line function > is created with ten inputs (yes, this is real), no. But DRY tells us that > the five lines of duplication is unconditionally worse.

I'm not convinced by your example. There are plenty of mathematical calculations taking numerous parameters that I think should be in a distinct function.

Even for non-mathematical calculations, 5 lines of code that are used repeatedly as some sort of standard pattern in your program should also get factored out. Like your logging example, ie. you consistently log everything in your program the same way, then sure, refactor that into a shared function. Then if you suddenly find you need to log more or less information, you can update it in one place.

Of course, I understand your meaning that sometimes factoring out doesn't make sense, but if you find repetition more than twice as per DRY, refactoring seems appropriate.

<comment><USER6> You DRY up code by writing indirection. That's the expense of all abstractions. You can't believe that all indirection is worth it at all costs, so I'm not sure what point you're belaboring.

<comment><USER3> > You can't believe that all indirection is worth it at all costs

I think I've been pretty clear about the costs and when this is worth it, particularly in my first post in this subthread, which I'll quote here:

> That said, the moment you have to start adding parameters in order to > successfully factor out common code, ie. to select the correct code path > taken depending on caller context, that's when you should seriously question > whether the code should actually be shared between these two callers. More > than likely in this case, only the common code path between two callers > should be shared.

Or if you want a more concise soundbite: refactor if your indirection is actually a clear and coherent abstraction.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> As "michael" points out in the comments on the article, this has already been invented, it's called a macro and has been available for decades in various LISPs. Wikipedia lists a number of other languages [1] that support macros.

[1] [https://en.wikipedia.org/wiki/Macro_(computer_science)#Synta...](https://en.wikipedia.org/wiki/Macro_\(computer_science\)#Syntactic_macros)

<comment><USER8> The problem with having powerful macros is always the same: as a project grows and grows you end up inventing your own little dialect of the language which is opaque to any 3rd party reading your code unless they take the time to unravel your macros.

I think Rust did two things right in that regard: firstly the macro invocations are always suffixed with '!' so you know it's not a regular function call right away. Secondly Rust macros are so quirky, ugly and painful to implement that you only ever use them as a last resort, so people tend not to abuse them too much.

<comment><USER9> > _as a project grows and grows you end up inventing your own little dialect > of the language which is opaque to any 3rd party reading your code unless > they take the time to unravel your macros._

This is bad use of macros, or an ugly macro system.

Macros, at least in Lisp, made code even _clearer_ to understand; because they let you create the constructs that make the problem domain map more directly, straightforwardly, easily, to the programming language.

So they reduce line count, they reduce need for kludges or workarounds. They allow more straight code.

But this is within the Land of Lisp, where writing a macro isn't something "advanced" nor "complex" nor "esoteric". In the Lisp world, writing a macro is 95% similar to writing a run-of-the-mill function.

<comment><USER10> No true Scotsman would ever write macros in such a confusing manner!

But really, this is a recognized problem of Lisp, and has been called the Lisp Curse. [0] One is _never_ programming in "just Lisp", but rather in Lisp plus some half-baked DSL haphazardly created by whoever wrote the program in the first place.

Also, don't confuse readability with understanding. Yes, DSLs are typically easier to read, but only _after_ you come to understand the primitives of the language. When every program has its own DSL with its own primitives, even programs that do similar things... That becomes quite a burden.

[0] [http://www.winestockwebdesign.com/Essays/Lisp_Curse.html](http://www.winestockwebdesign.com/Essays/Lisp_Curse.html)

<comment><USER11> You're replying to an, arguably, "no true Scotsman" argument with a straw man of your own:

> but rather in Lisp plus some half-baked DSL

Why does it need to be "half-baked"? Why do you assume that writing a good DSL is impossible for most Lisp users? Are you sure it's actually the case?

<comment><USER12> If I understand "the Lisp curse" correctly, the claim is that Lisp often winds up with a "half-baked DSL" because making DSLs in Lisp is so _easy_. You can do it without putting very much thought into it, so it's easy for the original author to just slap something together.

Note well: This is my understanding of the claim. I take no position on whether it is true.

</comment></comment></comment></comment></comment></comment><comment> aaron-lebo The author is absolutely right that theres's a ton of data from languages/open source projects that if analyzed could prove massively useful.

But I'm not sure that the metrics in the referenced article have much to do with language design. Most of what they are measuring (their conclusion says as much) are people copy pasting files and entire projects. JS by their stats is the worst about this but JS is a language that reduces duplication more than Java (also measured with less copying).

Seems to be a social or dev skill issue rather than language design. Honestly most languages have an excellent tool for reuse - the function/method, which isn't used enough to remove all duplication as is.

 _We presented an exhaustive investigation of code cloning in GitHub for four of the most popular object-oriented languages: Java, C++, Python and JavaScript. The amount of file-level duplication is staggering in the four language ecosystems, with the extreme case of JavaScript, where only 6% of the files are original, and the rest are copies of those. The Java ecosystem has the least amount of duplication. These results stand even when ignoring very small files. When delving deeper into the data we observed the presence of files from popular libraries that were copy-included in a large number projects. We also detected cases of reappropriation of entire projects, where developers take over a project without changes. There seemed to be several reasons for this, from abandoned projects, to slightly abusive uses of GitHub in educational contexts. Finally, we studied the JavaScript ecosystem, which turns out to be dominated by Node libraries that are committed to the applications' repositories._

Reading the conclusion again, they're capturing the existence of node_modules/, which has nothing to do with language design.

<comment><USER13> Further info about node_modules from the paper

 _Because the JavaScript sample was so heavily (78 out of 80) dominated by Node packages, wehave performed the same analysis again, this time excluding the Node files. This uncovered jQueryin its various versions and parts accounting for more than half of the sample (43), followed froma distance by other popular frameworks such as Twitter Bootstrap (12), Angular (7), reveal (4).Language tools such as modernizr, prettify, HTML5Shiv and others were present. We attribute thisgreater diversity to the fact that to keep connections small, many libraries are distributed as asingle file. It is also a testament to the popularity of jQuery which still managed to occupy half ofthe list._

</comment></comment><comment><USER14> In visual studio in the late 2000s, code snippets were all the rage, a new feature to allow you to write boiler plate quickly.

Since .Net 3.5 I barely use them, I really only use the built in ones for foreach and property accessors. The addition of lambdas and short-hand property accessors vastly reduced the boiler plate code in C#. I imagine the addition of async/await has also helped a lot with other certain types of code.

My round-about point is that existing languages can improve to reduce code duplication.

</comment><comment><USER15> I find (given enough time for it that is), more and more, that if code is duplicated or bad or both, that I can rewrite it into some (unexisting) DSL that makes it elegant. Then when trying to rewrite it to the language at hand, it becomes (a lot more) inelegant again. So is low hanging fruit there an Alan Kay strategy of writing DSLs for specific domains (which would put languages like Lisp, Ocaml or Haskell in a good place) or should language designers find ways to make the language itself better for these kind of problems (and is that possible, for instance; advanced types OR elegant macros?).

The other comment about Django (which would also work for Rails); a lot of times, big enough libraries/frameworks written in a language, are actually DSLs which would suggest we lean toward the 'Alan Kay' 'make everything a DSL' solution. The mathematician/CS person in me wants the 'improve the language' solution. But low hanging fruit for language design I don't see here.

<comment><USER16> The thing about DSL's (or really any programming language abstraction) is that they make it easier to understand the intent of the code at the expense of making it harder to understand the effect of the code. I find that the cognitive load of maintaining a project with lots of DSL's is high. The advantage of e.g. java over scala is that while java code is often more verbose and ugly, it holds fewer surprises. Like anything there is a trade-off here, adding more DSL's can be better or worse.

<comment><USER15> > find that the cognitive load of maintaining a project with lots of DSL's is > high

Yes, that is currently why I design that kind of scenario in a (possibly blue sky) DSL 'on paper' (if there isn't one handy) and then translate that back to libraries and structures in the language i'm working in. I just notice that this becomes annoying sometimes if the existing solutions really do not match the DSL I envisioned.

</comment></comment></comment><comment><USER17> "In Java, 70% of duplicates were due to code generation. For C++ is was 18%, for Python 85%, for JavaScript 70%."

In the case of Java and JavaScript, there's already a language that is built on top of them and has macros. It's Clojure/ClojureScript.

<comment><USER18> Two actually, you forgot Scala/Scala.js ;-)

disclaimer: I won't argue about the superiority of one macro system over another, the point is that macros exist in both languages.

</comment></comment><comment><USER19> A nice complement to this is Andrej Karpathy's Software 2.0 essay. Namely that it is getting much easier to collect the data. Than it ever will be to write the complex programs which can anticipate every case.

Software 2.0

[https://medium.com/@karpathy/software-2-0-a64152b37c35](https://medium.com/@karpathy/software-2-0-a64152b37c35)

One Model To Learn Them All

[https://arxiv.org/abs/1706.05137](https://arxiv.org/abs/1706.05137)

</comment><comment><USER20> > In Java, 70% of duplicates were due to code generation

I don't think, in Java's case, that fixing this is low hanging fruit. Java's more reliant on code generation than comparable languages for a reason. Most the codegen I've encountered is stuff that would be hard to implement in a more elegant way due to limitations of the platform's runtime environment (such as type erasure) that make it hard to handle a lot of the dumb verbose mapping work and stuff like that at run time.

Someone else mentioned Clojure. I haven't tried Clojure yet, but I'm willing to believe it - dynamic typing seems like it could easily be a secret weapon on the JVM, since the platform is so halfhearted at static typing. But I think that teams that are committed to using a static, infix, non-S-expression language may be painted into a bit of a corner here.

<comment><USER21> A lot of our duplication in our Java-projects is due to Java classes generated from wsdls or xsds or similar. I quite like it, actually. Can generate it from a known xsd and version it as it's own module.

How could a language solve this? I know F# can compile some stuff using things online, but this means the build is suddenly not reproducible. And we could always only generate that code on the fly when compiling, but that always wreak havoc on some tooling or IDEs when the code you're referencing isn't there until compile-time.

<comment><USER20> Something like F#'s type compilers would work on the JVM. You can make the build reproducible by just making sure that the inputs the type compiler uses are consistent. Practically speaking, the biggest difference between that approach and more traditional code gen is that you don't have to add a whole bunch of build steps and you don't have to have team arguments over whether generated code gets checked into source control or not. It's actually great for the IDE, which does know all that type information at development time because it can invoke the type compiler as a service. F# is the only language I've seen where you can write inline SQL queries and get a red squiggly if you mistype a field's name. (IntelliJ comes close, but, as far as I've been able to find, it only works if you stick all your SQL queries in resource files.)

The other approach is to just handle these sorts of things at run time, using the richer run-time information that your code has access to. So, e.g., if you're binding an XML file to a List<Record<Employee>>, in C# it's possible to actually express something like List<Record<Employee>>.class. The mapping function can just look at that and figure out how to map data dynamically. You, the developer, then get to own your own domain objects instead of having to rely on ones that are auto-generated by some code generation process. You don't have to accumulate a bunch of XML files and XML schema files and associated obnoxious-to-maintain clutter. And you can do it without resorting to a bunch of custom weirdness like Jackson's TypeReference class.

</comment></comment></comment><comment><USER22> Joe Armstrong talked about immutable globally available code. Only then would we not need duplication. #lazyweb please find a link to it.

Why do we need modules at all? [http://lambda-the- ultimate.org/node/5079](http://lambda-the-ultimate.org/node/5079)

[http://erlang.org/pipermail/erlang- questions/2011-May/058768...](http://erlang.org/pipermail/erlang- questions/2011-May/058768.html)

</comment><comment><USER12> Duplicating my comment from the other discussion of this article:

> In Java, 70% of duplicates were due to code generation. For C++ is was 18%, > for Python 85%, for JavaScript 70%.

C++ is the real outlier there. That could be because C++ code is much harder to generate, but I don't buy that it's that much harder than Java. Or it could be because C++ templates aren't considered "code generation". Or it could be because C++ doesn't get used for projects with that much boilerplate code. Or...

<comment><USER23> A C++ project that features generated code will probably run the generator during the build process - and possibly build it too, if it's written in C++ itself - and so the generated code might well not end up in the repo.

</comment></comment><comment><USER24> Duplicating code is acceptable, IMHO, if it is either clear the two files are going to diverge quickly, or if it looks like they will not be changed at all in the foreseeable future.

Trouble starts if there are two source files that do almost the same thing but slightly different, and then you need to change them; now you probably need to change these duplicate files in lock step, and that is a rather error-prone process. IIRC, this kind of problem was the reason for adding templates to C++. And say what you will about C++, but templates are a very powerful tool.

I fail to see the connection to language design, though. Is the author saying that one should add some feature to make duplication as unnecessary as possible (like templates in C++)? Or that the tooling around the language should be better suited to automatic code generation?

FWIW, I think code generation is a very powerful tool; in a way, code generation is meta-programming. (Is there a distinction at all?) And I think that there is a lot of potential in this area. Go has supported the "go generate" command for a while now, and I have seen a few very interesting use cases (e.g. ffjson, which generates code to serialize/parse Go data types to and from JSON more efficiently than the builtin reflection-bases mechanism).

Someone once said (I forgot who), "I'd rather write programs that write programs than write programs." That sums it up pretty well, I think. ;-) Okay, okay, so now I _do_ see the connection to language design. Sorry, my fingers were faster than my mind this time.

</comment><comment><USER25> C# is really bad about this. They seem to be heavy into the Magical Boilerplate coding style in which your "empty" project consists of thousands of lines of duplicated/generated code which you then make slight edits to. My opinion is that if some piece of code is so common that it needs to be inserted into every project - then it should be a library call. Apparently that's a minority opinion though..

<comment><USER26> This isn't my experience with C# (an empty project starts with a `Class1.cs` file if it's a library or `Program.cs` if it's an application), so you might be using something on top of C# that needs all of that boilerplate.

<comment><USER27> Sounds like project templates in Visual Studio - which can populate your project with vast amounts of stuff. This can good or bad depending on the context.

<comment><USER25> An empty ASP.NET core application contains ~35,000 lines of code.

<comment><USER28> No. Just looked, and the only large file generated is the dependency cache (~7000 lines of json) which you never touch and can always be regenerated with dotnet restore.

The rest is ~100-150 lines code+configuration and a readme file (~180 lines).

<comment><USER25> Look- I get that a lot of this stuff (jquery..) is libraries, and some of it is the visual studio solutions file, but if it's libraries, then _why was it copied into my project directory?_

       1031 ./.vs/config/applicationhost.config
         19 ./.vs/WebApplication1/v15/.suo
          3 ./WebApplication1/.bowerrc
         10 ./WebApplication1/appsettings.Development.json
          8 ./WebApplication1/appsettings.json
         10 ./WebApplication1/bower.json
         24 ./WebApplication1/bundleconfig.json
         35 ./WebApplication1/Controllers/HomeController.cs
          1 ./WebApplication1/obj/Debug/netcoreapp1.1/CoreCompileInputs.cache
          0 ./WebApplication1/obj/Debug/netcoreapp1.1/TemporaryGeneratedFile_036C0B5B-1481-4323-8D20-8F5ADCB23D92.cs
          0 ./WebApplication1/obj/Debug/netcoreapp1.1/TemporaryGeneratedFile_5937a670-0e60-4077-877b-f7221da3dda1.cs
          0 ./WebApplication1/obj/Debug/netcoreapp1.1/TemporaryGeneratedFile_E7A71F73-0F8D-4B9B-B56E-8E70B10BC5D3.cs
         24 ./WebApplication1/obj/Debug/netcoreapp1.1/WebApplication1.AssemblyInfo.cs
       9830 ./WebApplication1/obj/project.assets.json
         17 ./WebApplication1/obj/WebApplication1.csproj.nuget.g.props
          5 ./WebApplication1/obj/WebApplication1.csproj.nuget.g.targets
         25 ./WebApplication1/Program.cs
         27 ./WebApplication1/Properties/launchSettings.json
         60 ./WebApplication1/Startup.cs
          7 ./WebApplication1/Views/Home/About.cshtml
         17 ./WebApplication1/Views/Home/Contact.cshtml
        108 ./WebApplication1/Views/Home/Index.cshtml
         14 ./WebApplication1/Views/Shared/Error.cshtml
         73 ./WebApplication1/Views/Shared/_Layout.cshtml
         18 ./WebApplication1/Views/Shared/_ValidationScriptsPartial.cshtml
          2 ./WebApplication1/Views/_ViewImports.cshtml
          3 ./WebApplication1/Views/_ViewStart.cshtml
         22 ./WebApplication1/WebApplication1.csproj
         37 ./WebApplication1/wwwroot/css/site.css
          0 ./WebApplication1/wwwroot/css/site.min.css
          0 ./WebApplication1/wwwroot/favicon.ico
          1 ./WebApplication1/wwwroot/js/site.js
          0 ./WebApplication1/wwwroot/js/site.min.js
         44 ./WebApplication1/wwwroot/lib/bootstrap/.bower.json
        587 ./WebApplication1/wwwroot/lib/bootstrap/dist/css/bootstrap-theme.css
          0 ./WebApplication1/wwwroot/lib/bootstrap/dist/css/bootstrap-theme.css.map
          5 ./WebApplication1/wwwroot/lib/bootstrap/dist/css/bootstrap-theme.min.css
          0 ./WebApplication1/wwwroot/lib/bootstrap/dist/css/bootstrap-theme.min.css.map
       6757 ./WebApplication1/wwwroot/lib/bootstrap/dist/css/bootstrap.css
          0 ./WebApplication1/wwwroot/lib/bootstrap/dist/css/bootstrap.css.map
          5 ./WebApplication1/wwwroot/lib/bootstrap/dist/css/bootstrap.min.css
          0 ./WebApplication1/wwwroot/lib/bootstrap/dist/css/bootstrap.min.css.map
        105 ./WebApplication1/wwwroot/lib/bootstrap/dist/fonts/glyphicons-halflings-regular.eot
        287 ./WebApplication1/wwwroot/lib/bootstrap/dist/fonts/glyphicons-halflings-regular.svg
        771 ./WebApplication1/wwwroot/lib/bootstrap/dist/fonts/glyphicons-halflings-regular.ttf
         93 ./WebApplication1/wwwroot/lib/bootstrap/dist/fonts/glyphicons-halflings-regular.woff
         72 ./WebApplication1/wwwroot/lib/bootstrap/dist/fonts/glyphicons-halflings-regular.woff2
       2377 ./WebApplication1/wwwroot/lib/bootstrap/dist/js/bootstrap.js
          6 ./WebApplication1/wwwroot/lib/bootstrap/dist/js/bootstrap.min.js
         12 ./WebApplication1/wwwroot/lib/bootstrap/dist/js/npm.js
         21 ./WebApplication1/wwwroot/lib/bootstrap/LICENSE
         24 ./WebApplication1/wwwroot/lib/jquery/.bower.json
       9831 ./WebApplication1/wwwroot/lib/jquery/dist/jquery.js
          4 ./WebApplication1/wwwroot/lib/jquery/dist/jquery.min.js
          0 ./WebApplication1/wwwroot/lib/jquery/dist/jquery.min.map
         36 ./WebApplication1/wwwroot/lib/jquery/LICENSE.txt
         39 ./WebApplication1/wwwroot/lib/jquery-validation/.bower.json
        997 ./WebApplication1/wwwroot/lib/jquery-validation/dist/additional-methods.js
          3 ./WebApplication1/wwwroot/lib/jquery-validation/dist/additional-methods.min.js
       1397 ./WebApplication1/wwwroot/lib/jquery-validation/dist/jquery.validate.js
          3 ./WebApplication1/wwwroot/lib/jquery-validation/dist/jquery.validate.min.js
         22 ./WebApplication1/wwwroot/lib/jquery-validation/LICENSE.md
         43 ./WebApplication1/wwwroot/lib/jquery-validation-unobtrusive/.bower.json
        415 ./WebApplication1/wwwroot/lib/jquery-validation-unobtrusive/jquery.validate.unobtrusive.js
          4 ./WebApplication1/wwwroot/lib/jquery-validation-unobtrusive/jquery.validate.unobtrusive.min.js
         22 ./WebApplication1.sln
      35413 total

<comment><USER29> That's not an empty .net core project; that's a sample project that you can run and see a whole website which itself is documentation on creating .net core site.

However, I'm not sure Visual Studio gives you any way to avoid this when you use the GUI to start a new core project. It is however possible to create a non-Core ASP.NET project with literally nothing in it.

<comment><USER28> VS sure does offer you a way to avoid this. Simply choose the "Empty" Template when creating a new ASP.NET Core project. (Duh. :) )

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER30> There's a lot more to a programming language than the ability to avoid duplicated code, and making the language more expressive is not always better. The flexibility of the C++ preprocessor makes tooling (editor support, static analysis, incremental builds, etc) much more difficult to write, and as a result, state-of-the-art C++ tooling has a lot of disadvantages compared with (say) Java tooling.

</comment><comment><USER31> I would argue that many of the large extensions, plug-ins, and packages are already their own programming languages. For example you don't just "learn Python", you can "learn Django".

</comment><comment><USER32> There is an interesting related paper " _Copy and Paste Redeemed_ " [1] that is based on the assumption that a lot of copy/paste happens (and discusses why this is not necessarily a bad thing). [1] investigates how copy/paste can be auto-detected, how similar pieces of code can automatically be merged together, and how abstractions can be automatically created from copy/paste code.

I have not used the tool myself, so I cannot comment on how well it works in practise. But I found the idea intriguing.

[1] K. Narasimhan, C. Reichenbach, Copy and Paste Redeemed [http://creichen.net/papers/cpr.pdf](http://creichen.net/papers/cpr.pdf)

<comment><USER33> good approach IMO

</comment></comment><comment><USER34> Normally you shouldn't be checking generated code into version control. This only measures GitHub users who did that anyway.

<comment><USER12> Normally, yes, but...

If you don't check in the generated code, that means that you have to run the generator every time you build (or at least every time you get a clean copy of the tree and build). For code that changes very rarely, that may not be a net win.

Also, if you don't check in the generated code, and then you upgrade to the latest version of the generator, surprising things can happen (or even if different people have different versions of the generator installed).

So there can be cases where checking in generated code can be at least a reasonable thing to do.

</comment></comment><comment><USER35> The pain of implementing an ORM and the generated code between OO and Relational Databases might be low hanging in it's explanation, but writing a language that does this is probably a Hard Problem. I wouldn't call that low hanging, but I might be really happy to use such a language. ([https://blog.codinghorror.com/object-relational-mapping- is-t...](https://blog.codinghorror.com/object-relational-mapping-is-the- vietnam-of-computer-science/))

</comment><comment><USER36> The fact that this is labeled as "low hanging fruit" tells us that the author has never designed or implemented a language.

</comment><comment><USER37> I was trying to express an inchoate thought the other day, something to the effect that, "total volume of code in the world should be shrinking about now". I think this article could be interpreted as pointing to that.

I imagine a Grand Refactoring... The "great compression" of '23, or something...

</comment><comment><USER38> > In other words, 70% of the code on GitHub consists of clones of previously > created files

I interpret this as code _reuse_ which is almost the opposite of duplication.

<comment><USER4> Duplication will always be reuse - why are you duplicating if not to use?

Reuse is not always duplication, however, and reuse without duplication is quite often better.

<comment><USER38> Repetition (like in DRY) is duplication without reuse. When I think of duplication in the negative sense, I think of repetition, not reusing a module or library. Another negative thing with duplication is taking up more disk space, but a good file system will take care of that! Then there is "DLL hell", or "library hell" where two programs use the same library, but different versions ... How do you reuse without duplication ?

</comment></comment></comment><comment><USER39> I'm a lisper at night so I tend to have a heavy use of macros as well as rather abstract patterns you can use in any language. This works for me because

a) I'm the only maintainer of my own side projects (most of the time I never finish nor publish them ... including a cool wysiwyg mouse-driven Clojure POC debugger)

b) I tend to perform multiple full rewrites of my projects (up to 4 times).

c) My code tends to get obscure very quickly since I enjoy giving it an "ontological" twist, and eventually this leads to less code but the code gets less understandable from the static perspective of the source file it sits in, so to get a good grasp of what it does, one needs to run the code and observe what it does when it is evaluated (at macro-expansion time or run-time): this is why I semi-successfully attempted to write the debugger mentioned above.

Meanwhile at work, I have not performed a full rewrite yet (understandably so) and the goal is to keep the code as flat and linear as possible so that anyone in the team can grasp what it does in one glance. Obviously this leads to a lot of repetition, but this is for the greater good.

Currently I'm working on improving my dev experience with Clojure from two angles:

1) Saner macros : I've been tweaking Clojure's reader so that code generated with the backquote reader macro gets printed in its original form when using pprint. For instance `(a b c) expands to (my-ns/a my-ns/b my-ns/c) and becomes `(a b c) again when printed with pprint. I'm also thinking about expanding macros in temporary files in order to get sane stack traces for code generated by macros, but this is a surgically more complex thing to do.

2) "Macros" that expand and persist in the very file they are written in. This allows for in-file debugging and should address point c) from above. Example: At first the content of your file looks like:

      (debug
        (+ 1 2))

When you evaluate the file, it turns into (i.e. the file gets rewritten as):

      (debug
        (+ 1 2)
        ;;= 3
        )

Since the debugging/inspecting of how the code behaves at runtime gets persisted in the file along with the code itself, it should allow for a broader and more direct understanding of what the code does. Since this can also be used as a language/library level snippet system, I've also been considering using these in-file persisting macros as a templating engine for code. In particular, if it is augmented with a conflict management system à la git, one should be able to have flat/linear yet automatically generated code and still be able to overwrite what's in a code template expansion without losing the benefits of automatic code generation.

</comment><comment><USER40> I highly recommend you read about composability, various levels of abstraction and how it all plays together. A good programming language should strive for clean interfaces and expose relatively lower level methods that can be composed in different ways to build more complex logic. E.g. Ruby/Python exposing the methods they are exposing now, gems/packages use it to provide a higher level construct, django/rails combines these gems/packages to expose even higher level constructs.

So there is no point in including the most common auto-generated code in the programming language itself. Because the tool that generated it can keep evolving independent of the language and actually uses the lower level constructs exposed by language. By including the logic within programming language we'll just bloat the language and loose out on the beauty of composability.</comment>