A "software engineer" should be required to be licensed - <AUTHOR> https://twitter.com/stephendpalley/status/1249685093653843974 <comment><USER2> In many engineering disciplines one needs to be a "Professional Engineer" to well actually do engineering stuff (I'm looking at you Structural Engineering). In some places (Canada for example) the title is reserved exclusively (and by law) for those who got an accredited engineering degree and go on to apprentice under what would be a thought of as a PE in the USA.

I personally don't have a problem with the title "engineer" being reserved for what would amount to a US "professional engineer".

I'll note that PE don't require either an advanced degree nor a million dollar liability policy, so I'm not sure why such would be required from a "software engineer".

Really in the end, I don't think programmers should be calling themselves architects or engineers. I feel like we do so because we covet the prestige associated with these professions, but work in one that is not old enough for our own secret societies or iron rings or titles.

For what it is worth, internally in my company I have co-opted and use the title "Jedi Master". It is just as valid/invalid in my opinion as software engineer or architect and is way cooler.

<comment><USER3> > " _I 'll note that PE don't require either an advanced degree nor a million > dollar liability policy, so I'm not sure why such would be required from a > 'software engineer'._"

Leaking customer data for a medium sized business would probably easily exceed $1 million in emergency software fixes, PR, and goodwill (and that's leaving aside the cost to the customer), yet that happens constantly because of bad developers continuing to implement known bad practices.

</comment></comment><comment><USER4> While I think having the equivalent of the Professional Engineer (in the US) for software or hardware/electrical is probably a good idea for very critical products (self-driving cars, medical devices, planes, etc) distributed to the public, a blanket requirement would significantly impair a lot of the economy.

<comment><AUTHOR> The thread is a mess, but he clarifies that he's referring to those who develop software for "critical infrastructure."

</comment></comment><comment><USER5> "React Engineer". Cringe!</comment>


Ulysses: a structured/semantic document editor for OS X - <AUTHOR> http://www.the-soulmen.com/ulysses/index.html

<comment><USER2> Scrivener is terrific: <http://www.literatureandlatte.com/scrivener.html>. I've written book-length stuff in Emacs/LaTeX before, and I might go back to that for typesetting, but I'd write in Scrivener.

<comment><USER3> Using MultiMarkdown in Scrivener and then generating LaTeX works pretty well for me, though I haven't written anything book-length with it.

[http://fletcherpenney.net/multimarkdown/using_multimarkdown_...](http://fletcherpenney.net/multimarkdown/using_multimarkdown_with_scriv/)

</comment></comment><comment><USER4> This is really cool. There are a lot of areas where the state of the art really is Word and Excel. For some use cases, a horizontal application is perfectly adequate. However, all the rest are opportunities for enterprising hackers.

<comment><USER5> I actually made a poetry book in LaTeX. It wasn't THAT hard.

</comment></comment><comment><USER6> Anyone here ready to earn some karma by vouching for it and giving a few paragraphs on why I absolutely must try this out? :-)

<comment><AUTHOR> I can tell you why I like it. There's an app that's been repped for here called Scrivener, and like Scrivener (it predates Scr., actually), Ulysses is an application that is supposed to make writing large amounts of text more sane than standard word processors. It essentially combines elements of outliners and information managers with a composition interface.

In general the UI for Ulysses is _very_ thoroughly thought out, and somewhat idiosyncratic for that fact. Many people find it lacking or uncomfortable in one way or another, but I can tell you that every element of the UI has been considered and applied to fulfill the designers' very specific goal.

One of those UI values is that it's very semantically correct. There's no formatting at all, except auto-formatting that you define yourself, which will recognize plaintext tags. The way this shakes out is that I will write a sentence like this: The &&quick&& brown fox jumps over the lazy dog.

And then define a tag called 'emphasis', which is formed by a '&&' wrap, and displays itself in a certain way depending on whether it's being viewed in a window, fullscreen, printed, et cetera.

If you are a fan of working in plaintext and you are interested in more structured ways to write long-form text, I think it's a great program.

EDIT: changed example markup to something more exotic for HN's own parser

<comment><USER7> What is a "composition editor?" or for that matter, an "information manager"?

<comment><AUTHOR> I don't know what a composition editor is. But an information manager is, generally, one of the wide range of applications that take unstructured, file- like data and provide an application interface to them. What that entails varies, but there's often indexing (and powerful search), tagging, and sometimes NLP–a snippets manager, or a PIM like Eaglefiler, Together, Yojimbo, DEVONthink and many many more.

EDIT: You might mean, "what's a composition interface?" In which case it's a text-editor component, with some added UI sugar like current line highlighting, word count, splitting, typewriting scrolling, full-screen mode, et cetera. You know, an interface for composing text.

</comment></comment></comment></comment><comment><USER8> Pretty cool. I do have to say that I prefer LaTeX though. If you're not into writing LaTeX you may want to try LyX (<http://www.lyx.org/>). It's a WYSIWYM LaTeX editor for Windows and UNIX-like (including OS X).

Heh, although when I say "Ulysses" out loud it makes me think "useless."

<comment><USER9> Apples and oranges.

LaTeX is a formatter. (Actually a full-blown interpreter, but it's generally used for producing typeset text output from command files.)

Ulysses and Scrivener are an utterly different class of application; they're used for composition and editing of complex compound documents, with an emphasis on managing the structure (e.g. they both make it utterly trivial to change the order of chapters in a book, swap subsections within a chapter, write the text with no distractions, or mess around with the outline).

LyX is, effectively, a WYSIWYG editor for LaTeX (that is handicapped by not being flexible enough to generate arbitrary LaTeX documents, and so it tends to be restricted to using a set range of LaTeX templates).

Might as well be comparing motorcycles and ro-ro ferries.

Finally: parading your lack of knowledge of classical mythology or modernist literature really doesn't do you any favours.

<comment><USER8> First, thanks for the information. I didn't understand from the site that Ulysses is actually more of an organizational tool. It wasn't until I read this[1] review article that I understood that.

However, you yourself made a few mistakes. Technically, LaTeX is not a formatter, it is a markup language for the TeX typesetting system. In addition, it is actually _not_ an interpreter (in the same way that HTML is not an interpreter).

LyX is, as I said, a _WYSIWYM_ editor for LaTeX. It is also allows LaTeX insertion and imports, so it should be arbitrarily flexible, not handicapped.

 _Finally: parading your lack of knowledge of classical mythology or modernist literature really doesn't do you any favours._

Hmm. Actually I have read Ulysses, A Portrait of the Artist as a Young Man, Finnegans Wake, as well as the Odyssey. James Joyce is probably one of my picks for greatest authors who ever lived. My little remark was one of whimsy, commenting on a little phonetic wordplay, not an expression of ignorance.

[1] [http://www.macobserver.com/tmo/review/ulysses_is_a_heroic_wr...](http://www.macobserver.com/tmo/review/ulysses_is_a_heroic_writers_application/)

</comment></comment></comment><comment><USER10> Ok so after I decided that it was interesting and cool, my first thought was, "What's the emacs-mode for Ulysses?". I couldn't find one so it looks like I may have to actually download and install it. Anyone know of similar emacs or vim modes for writing?

<comment><AUTHOR> I don't know of any such vim mode but I'd be very interested to hear about it. The closest I have is a script that turns on things like wrap-by-word and scroll-by-visual-line (given that in prose, what vim thinks of as lines, we think of as paragraphs)

</comment></comment><comment><USER11> Writeroom is pretty great too.

<http://www.hogbaysoftware.com/products/writeroom>

<comment><USER3> It's even better when combined with QuickCursor (free download), which allows you to, with a keystroke, edit the contents of almost any textfield in WriteRoom. It also works with other editors such as TextMate.

<http://www.hogbaysoftware.com/products/quickcursor/>

</comment></comment><comment><USER12> Do I understand well : is this something similar to what would be a good GUI for reStructuredText ? (I don't have OS X anymore to try it, unfortunately)</comment>


Attracting high-paying clients to design and development agencies - <AUTHOR> https://www.reddit.com/r/Entrepreneur/comments/6ztybx/how_i_attract_high_paying_clients_to_my_design/?15 <comment><USER2> From 3 months ago: [https://news.ycombinator.com/item?id=15237398](https://news.ycombinator.com/item?id=15237398)

</comment><comment><USER3> If there's one consistent issue that plagues startups, consultants, and even large established business it's marketing.

And that whole thread touches on a subject that I wish would get more traction here on HN: content engineering.

Content Engineering (building tools and useful online software projects) is an incredibly effective way to build an audience and awareness by helping people instead of writing yet another 500 word blog post or dumping six figures of VC money into FB or Google ads.

It honestly feels like cheating. I think tools just occupy a different part of people's brains than articles and certainly a different spot than ads.

This is a hot topic for me as I was looking through the analytics for my site yesterday and found that our Subject Line tester [1] was driving about 10x the traffic of our dozen 2000+ word carefully crafted and educational blog posts [2]. F'ing ridiculous

1 - [https://sendcheckit.com/email-subject-line- tester](https://sendcheckit.com/email-subject-line-tester) 2 - [https://sendcheckit.com/blog](https://sendcheckit.com/blog)

Here's some other good examples:

Clearbit's Logo API [https://clearbit.com/logo](https://clearbit.com/logo)

ForAGoodStrftime [https://www.foragoodstrftime.com](https://www.foragoodstrftime.com)

Atom [https://atom.io/](https://atom.io/)

Golden Ratio Typography Calculator [https://pearsonified.com/typography/](https://pearsonified.com/typography/)

<comment><USER4> This is also a great way to understand your target market (rather than sending out surveys and creating landing pages).

I built [https://page.rest](https://page.rest), [https://screen.rip](https://screen.rip) & [https://pdf.cool](https://pdf.cool), in the last 3 months which gave us a great understanding of our target market and problem space.

It's also a way to win trust. If you can solve a small problem well, people will trust you with their bigger problems.

</comment></comment><comment><USER5> Marketing is not: "Spending money on ads, Spending years building a social following, Going to networking events, Cold calling"

but is: "posting it on relevant forums, sharing it with bloggers, sending it to journalists to potentially write about"

Isn't that sort of a contradiction, if the thing I want to market is a side project in itself?

I guess I'll try his strategy of talking about one of my side project's outputs instead of the side project itself: Learning Chinese with movies and music.

[https://www.youtube.com/channel/UCIUS5dz58-4i1ZQF0qouSaQ/vid...](https://www.youtube.com/channel/UCIUS5dz58-4i1ZQF0qouSaQ/videos?shelf_id=0&view=0&sort=dd)

</comment><comment><USER6> I can't seem to find it but I think unsplash was a similar side project that was used as marketing for a different company.

<comment><USER7> You're exactly right!

Unsplash [1] was a side-project for our company freelance developer and designer platform, Crew [2] (Crew was later sold to Dribbble; Unsplash now operates independently).

We knew as creators that finding beautiful, useable imagery was one of the hardest challenges in any creative project, so we decided after shooting some photos for our homepage header image, to give the rest of the photos from the shoot away for free for anyone to use.

We setup a $19 tumblr theme, a domain, and posted the site to here.

What happened next was amazing: the site was flooded with traffic thanks to trending on HN, thousands of photos were downloaded on the first day, and most shockingly to us, other creatives began submitting their own photography to Unsplash.

Fast forward a year later and Unsplash was the main source of traffic and revenue for the parent company Crew. I think it peaked at around 60% of our monthly revenue due to customers finding Crew from a simple 'Made by Crew' link on Unsplash. At the same time, Unsplash was doing around 1M downloads a month.

We eventually sold Crew to Dribbble to focus full time on Unsplash, so it's no longer a side project. It's grown a lot since those early days, with more than 20M photo downloads every month, half a million free-to-use photos, and is integrated into thousands of applications, including Trello, Adobe, and more.

If you're interested in more about Unsplash, we've got all the main milestones here: [https://unsplash.com/history](https://unsplash.com/history)

1 - [https://unsplash.com](https://unsplash.com) 2 - [https://crew.co](https://crew.co)

<comment><USER8> That history page is very clever - never seen anything like that before, and it's such a great way to build a brand.</comment></comment></comment>


Life on HIV treatment – a personal perspective - <AUTHOR> https://www.aidsmap.com/Life-on-HIV-treatment-a-personal-perspective/page/1255106/

<comment><USER2> Heavy stuff, viruses are nasty and HIV is one of the nastiest. Fortunately there has been fairly steady progress in halting the worst aspects of HIV but the stigma remains and for some reason HIV sufferers are viewed in a totally different light than say cancer or ebola victims.

When the epidemic hit in the 80's I worked with a bunch of people producing a newspaper about Amsterdam and I vividly remember the first descriptions of a new and mysterious disease that was attacking various subgroups in the clubbing scene (mostly gay people and drug users at the time, since then it has spread well beyond that and but even today the association with those groups is strong, stronger than probably warranted, HIV is a disease that certain groups are still more susceptible to but it's gone mainstream and you'd be a fool to think that because you're not in an elevated risk-group that you're somehow safe or immune).

<comment><USER3> i would say people are far more scared of someone recently suffering from ebola than HIV!

</comment></comment><comment><USER4> Another way to look at this article: the latest (and arguably most promising) HIV prevention techniques involve treatment-as-prevention[0]; in other words, taking HIV antiretrovirals as prophylactics. Ten or fifteen years ago, this wouldn't have been possible[1], because the treatments were unpleasant enough that nobody would want to take them daily if they didn't have to.

This is an incredible paradigm shift for HIV prevention, because it completely overturns the way we've been conceptualizing it. We've always thought of HIV like polio, which we are eradicating with a vaccine. Instead, the end of HIV/AIDS may very well come in the form of the same medications we've already developed. More importantly, we already have a great distribution mechanism for HIV medications in (e.g.) sub-Sarharan Africa, which means that we don't need to re-train people and create new programs; we would just need to expand the ones we already have.

None of this would have been possible with the drugs we had in the 90s, even if we had wanted to get FDA approval for PrEP (which of course, would have been political suicide).

[0] [https://en.wikipedia.org/wiki/Tenofovir/emtricitabine](https://en.wikipedia.org/wiki/Tenofovir/emtricitabine)

[1] Technically the drug that's used for this today came out 14 years ago, but it wasn't as commonly-used, and most of the drugs back then still sucked.

<comment><USER5> I guess my question would be, if people fail to practice safe sex, they could easily fail to take their prophylactic medication as well.

<comment><USER4> That's the same concern that was raised with birth control pills back when they first came out, but they've proven to be incredibly effective when measuring actual use (rather than "ideal" use, which is ridiculously inaccurate).

Given how many people are already on at least one daily medication, adding another pill is a pretty low barrier. It's much easier to remember to take a pill every morning than it is to remember to (a) stock up on condoms, (b) bring them with you on a date, (c) put one on in the heat of the moment, etc - you get the idea.

This isn't just speculation; it's supported by the clinical trials and post- clinical data as well, which is why PrEP is not only approved by the FDA, but actively recommended by the WHO and other organizations.

<comment><USER5> I'm not questioning that it works for some, I'm just wondering how many it actually works for.</comment></comment></comment></comment>


Ask HN: Was the number of comments removed from ranking system? - <AUTHOR> I noticed that there a few old (10h+) submissions with moderate amount (90-100) of upvotes and comments on the front page.<p>I remembered that it used to be that the ranking will be inversely proportional to both time and number of comments (I can't find it anywhere in FAQ now), which would render these submissions lowly ranked due to high number of comments. Was the ranking system changed? <comment><USER2> My impression is that there is some degree of correlation between threads with high numbers of comments and the comments exhibiting unproductive behaviors. Correspondingly, this makes it more likely that the story will be flagged. Likewise I think there is some correlation between stories that are "more mainstream news" and high numbers of comments exhibiting unproductive behaviors: these stories are even more likely to be flagged.

I've never seen anything suggesting the raw number of comments affects score negatively.

</comment><comment><USER3> I doubt that number of comments ever mattered, except that a comment also acts as an upvote.</comment>


To create a new category, name a new game - <AUTHOR> https://www.linkedin.com/pulse/create-new-category-name-game-andy-raskin <comment><USER2> I wish this was easier to do as an employee: you see the writing on the wall, they pay you to be an expert in a field, but no one listens to your "we should be moving in this direction". You're met with one of my most annoying phrases: "this is how we've always done it."

The opportunity passes by, now your company is playing catch-up and managers above you are asking how they missed it. If you say "I told you so," you get a reprimand instead of acknowledgement.

<comment><USER3> It really depends on the direction.

A lot of times, devs want to move in a new direction of whatever the hip tech stack is this year. This would help their career by adding the tech to their resume but the business benefit is almost always net negative.

Rather than arguing with a developer, they might just give a response that is outside of the developers' domain. The real reason is that it was a bad idea for the business.

<comment><USER2> That's a fair point. The perspective of developers is different from that of a manager with perhaps a longer term view or other knowledge than the people in the trenches.

</comment></comment></comment><comment><USER4> He writes "goodbye, ownership, hello usership" like that's a good thing. That's the business experience of outsourcing a service. You don't own that part of the business any more. You're just a user of some service that doesn't want to be bothered with your problems.

<comment><USER5> For what it's worth, a large portion of the HN community is involved with SaaS business. SaaS epitomizes that concept

</comment></comment><comment><USER6> >Because once someone buys into your new game, it becomes their orthodoxy. They become fiercely loyal to it as an organizing principle for how they act in the world.

>Until someone shows up with another story about an even newer game. That's inevitable, of course, but hopefully it doesn't happen until you've gotten very successful helping customers win at what will suddenly become an old game.

All seems a bit postmodern (post-truth I think it's the current rendition of that) ... I'm more interested in what's objectively better.

This sounds more like "we got well marketed to, drank the coolaid and now don't want to change (because we'd lose face, demonstrating our susceptibility to being conned ... which seems to me like the big problem in UK [and USA] politics at present).

<comment><USER7> Once you've created your objectively better thing you still have to go out and sell it. This is about how to do that.

<comment><USER8> Plus objectively better usually doesn't mean across the board. It's only objectively better in some particular aspect.

</comment></comment></comment><comment><USER9> I'm sorry, but I find this post insufferable. It just reeks of corporate buzzwording with a dash of disingenuous manipulation.

Also, unrelated: I hate those "conversational" customer service bubbles. I don't know what I expect to happen when I click them, but I know that "Jennifer H", whose picture is shown next to it, is not sitting at a keyboard eagerly awaiting my questions. Maybe I expect a chatbot, maybe I expect outsourced customer-service-farms, but I don't click buttons when I don't know what they really do, especially when I know they're avoiding being up-front about it.

<comment><USER10> I hate buzzwordy articles (and chatbox popups) too, but this article is better than usual and I think the point is a good one. There are situations in which you can't win at the old game, but if you reframe what you're doing as a new game, you can turn yourself into an early defining player instead of a hapless latecomer.

We see this in technical domains too. For example, I think Clojure succeeded this way. (We can argue about how much Clojure has succeeded, but it's a Lisp—we have to grade on a curve.) I don't think any attempt at improving on Common Lisp, no matter how technically solid, could have achieved that without "naming a new game", to use the article's vocabulary. Hot newness dominates improved oldness: every element of the set of hot newness beats every element of the set of improved oldness. Elixir/Erlang is another example. I say this article is one of the rarest things, a piece of marketing literature (like Crossing the Chasm and maybe the Innovator's Dilemma) that is useful to engineers.

<comment><USER9> It came across to me as "Just use this one weird trick and you can convince people of _anything_!" The runner example in particular seemed to emphasize the convincing over the content. I suppose it's possible to use the same technique benevolently, but the article didn't seem concerned with that question.

<comment><USER11> Engineers' emphasis on content over convincing is why we often fail to persuade. I think dang is right that this is a piece of marketing literature that's useful to engineers.

</comment></comment></comment></comment><comment><USER12> > _"If AI can diagnose cancer better than doctors, why can't AI understand > [sales] reality better than a salesperson?"_

[citation needed]

</comment><comment><USER13> Classic con-man tactic. "This is the NEW thing, it's a game-changer, you can get in on the ground floor"

</comment><comment><USER14> "Goodbye, ownership. Hello, usership" is a great tagline for the insane, rent- seeking business models that are en vogue in Silicon Valley.

</comment><comment><USER15> This is probably better well known as a "paradigm shift".

[https://en.wikipedia.org/wiki/Paradigm_shift](https://en.wikipedia.org/wiki/Paradigm_shift)

<comment><USER16> I think paradigm shift, at least when accurately used, runs much deeper than this.

<comment><USER15> Maybe you need to completely reevaluate what you think a paradigm shift really is? Could be you've been thinking about it wrong the whole time.

<comment><USER16> Perhaps, but evidence doesn't seem to point that way :) I agree it's not an easy bright line distinction, but it's the sort of term that dilution makes far less useful.

</comment></comment></comment></comment><comment><USER17> It makes for a good setup for the piece, but the whole "do or don't carboload before endurance racing" is still very much a subject of debate.

<comment><USER18> His point is "train your body to burn fat instead of carboloading". I don't get it, why not do both?

<comment><USER19> Yeap basically you should do your long runs slowly and more or less fasted. It sucks and sometimes you crash but it helps train you to burn fat more efficiently. You still want to carbo load before your long (20+) race...</comment></comment></comment>


Poor-Quality Antidepressant Studies - <AUTHOR> http://asserttrue.blogspot.com/2013/03/77-poor-quality-antidepressant-studies.html

<comment><USER2> It's too bad the author of this piece didn't include a reference to a remarkable study that combined published antidepressant studies, most of which found them to be effective, and studies that were funded, carried out, but then not published, which largely found them to be ineffective. The study's conclusion: "Meta-analyses of antidepressant medications have reported only modest benefits over placebo treatment, and when unpublished trial data are included, _the benefit falls below accepted criteria for clinical significance_." [Translation: antidepressants do not work.]

The study: [http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fj...](http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0050045)

After this study was published, it ought to have been replicated, or its conclusions called into question or falsified, or newer studies of greater scope be carried out, or antidepressant manufacturers be forced to include a warning on their antidepressant products: "This drug is actually a placebo".

But ... none of these things happened. The study had precisely no effect on the pharmaceutical industry or the public.</comment>


FlickIM: Bored Startup Makes Awesome IM for the iPhone - <AUTHOR> http://www.techcrunch.com/2007/07/16/flickim-bored-startup-makes-awesome-im-for-the-iphone/

<comment><USER2> I don't buy it. Why do they have funding, but just up and decided this week what to do?</comment>


Show HN: Convert blog articles to high-quality audio - <AUTHOR> https://websitevoice.com <comment><USER2> If this is not a wake up call for me, I don't know what it should be. I had this same exact idea for 9 months now but never moved to execute. Reading through site it even has the same wording as in my head.

As a developer I wanted to build something like this and it looks easy(without AI part). But following the startup gurus and podcasts I figured that idea should be validated first and to find actual customers is more important than building technology. Here started troubles. I have no skills in marketing and when diving deeper it looked cumbersome for me. I spend free credits on AdSense but it only brought 8 emails for empty newsletter.

I'm interested did you do the research on user base? What would be a market for this?

Where are you planning to grow the site? Do you have monetization plan?

p.s. found a link in comments for read2me.online, and just from demo they sound really good for ears

<comment><USER3> Likewise, I spent a day building a proof of concept but lost interest after.

</comment></comment><comment><USER4> Hello, Very interesting. I've been using Edge's built-in voice reading a lot more recently for the exact case you mention on your homepage: listening while working. I have two questions for you after reading your TOS and Privacy Policy. I was expecting to see "you" scraping data from each site or otherwise monetizing the service, but nothing stood out. Either your TOS/PP is incorrect, or you don't have an obvious income stream for the service you are rendering.

So the questions: Who are you? and How are you doing this for free?

Thank you,

<comment><AUTHOR> why it really matters on who I am? I'm a developer with passion to help people listen to podcasts and the idea was simple. How can I offer it for free? I do a lot of work to make it happen, there will be a premium version with more features which are being cooked with the help of many developers who loved the idea too.

<comment><USER5> It matters because one of your potential customers thinks it matters. And presumably because services without a business model don't have a long future.

<comment><AUTHOR> timClicks that is not true, there are many products out there that do not rely on capitals, It does not matter who I am personally. The service is free of charge until we cannot make it free of charge anymore for whatever reason. We are focusing now on user's feedbacks who already added the widget. you can find more answers on our business model there. [https://www.indiehackers.com/forum/finished-mvp-to-turn- blog...](https://www.indiehackers.com/forum/finished-mvp-to-turn-blogs-to- podcasts-with-ai-what-do-you-think-06553ca565)

</comment></comment></comment></comment><comment><USER6> There are some UI issues from what I noticed:

At small breakpoints, the UI (especially the "try it now" button) can overlap some text. Sizing also seems off.

Another issue I noticed is the play button itself, the entire circle area is clickable, however only the play/pause icon can actually be clicked.

There are also numerous accessibility issues. If you're going to advertise this as something to improve accessibility, it might be good to actually concentrate on that for your landing page.

I do hope the UI can be improved, as the product looks solid.

</comment><comment><AUTHOR> I'm not here to pitch the idea to make people like it, if you do not like it, be constructive on a given problem. To get a feeling more on how it feels on a blog here is an example: [https://websitevoice.com/blog/text-to-speech- dyslexia-and-re...](https://websitevoice.com/blog/text-to-speech-dyslexia-and- reading-disabilities/)

if the solution does not appeal to you, it surely appeals to others.

<comment><USER7> Couldn't listen to it for more than 20 seconds, got quite robotic and monotone for me. This would be like flash and auto play videos, cool at first then most annoying thing ever created.

<comment><USER8> monotone? hahaha you should try the default text to speech from the operating system to see what is monotone

</comment></comment></comment><comment><USER9> [https://read2me.online](https://read2me.online) Another similar service

<comment><USER2> Pricing looks expensive, but quality is top notch. Do you know which TTS engine it is?

</comment></comment><comment><USER10> This looks -- and sounds! -- awesome. I'm not sure if this can compete with Schema's 'Speakable' [1] property (for smart speakers), but for mobile/web browsing this looks amazing.

Looking forward to seeing new developments and also the pricing plan eventually.

1: [https://developers.google.com/search/docs/data- types/speakab...](https://developers.google.com/search/docs/data- types/speakable)

</comment><comment><AUTHOR> We created this free web app to convert articles to high-quality audio. Our mission is to help people with learning disabilities, those who prefer listening over reading, and who are always on the go to enjoy consuming your content online. Please give it a try and let us now what you think. Your feedback is highly appreciated :)

<comment><USER11> I'm working on a project that overlaps slightly (only with this feature). Would you be open to offering programmatic access in the future via an API?.

<comment><USER2> @jamil7 what are you working on? I wrote here that I've had pretty similar idea, wonder what's your take on that?

<comment><USER11> @mindwork DM me if you feel like it be cool to hear what you're working on.

</comment></comment></comment></comment><comment><USER12> I added it to an old unused part of my website.

[https://www.folj.com/lateral-solutions/](https://www.folj.com/lateral- solutions/)

Apparently my javascript doesn't play nicely with programmatically injected elements. The Show All Solutions button still works.

<comment><AUTHOR> good, at least you powered your blog with voice. happy to see it useful for you.

</comment></comment><comment><USER13> Just some hopefully constructive criticism, I found the voice used on the "Smart Natural Voice" section to be the most robotic sounding of them all. Specifically, the "Say goodbye to robotic voice, this is nothing like you ever heard before." line sounded very robotic and a bit awkward.

I'm not sure if maybe the content of the copy is a little clunky or if it's the voice itself, but that section sounded worse than the other ones, IMO. If possible, I would think about toying with that. The girl's voice in the "Voice Pitch and Speed Control" section sounded far more natural and impressive.

</comment><comment><USER14> Can I use this to convert other articles? Could I pay to get 32kbit Opus?

</comment><comment><USER15> Do you have plans to publish a versioned release with support for Subresource Integrity?

Also, is there somewhere that the following is documented?

\- Required origins for each type of content so that this can be compatible with a Content-Security-Policy

\- Required use of browser features so that this can be made compatible with a Feature-Policy

\- Commitments from yourself that the lists of required origins/features will not be added to without prior notice, in order to give time for blog operators to update their headers accordingly

Thanks

</comment><comment><USER16> I've wondered for a while with NPR doesn't have both audio and text of all of their programs.

Sometimes I can listen, but not read (driving). Sometimes, I can read but not listen (work, or just when I'm without headphones). I realize they want high quality voice, but if at the moment all they have is text that's ok with me. Not having both of these often means I just skip that article and never come back to it

<comment><USER17> radio ambulante (NPR's spanish podcast) has full transcripts in english and spanish of all their content. The quality of the journalism and topics are outstanding as well.

Super useful for learning Spanish.

</comment></comment><comment><USER18> Amazon Polly [1] and Google Text to Speech [2] are unfortunately far superior.

[1] [https://aws.amazon.com/polly/](https://aws.amazon.com/polly/)

[2] [https://cloud.google.com/text-to-speech/](https://cloud.google.com/text- to-speech/)

<comment><USER19> I have to agree, plus they're free for what most users would need.

Also noteworthy is the official Amazon Polly for WordPress plugin - [https://wordpress.org/plugins/amazon- polly/](https://wordpress.org/plugins/amazon-polly/)

</comment></comment><comment><USER20> Don't have time to test it now. How does it sound? Is it better than the robotic text to speech voice we all know? Wonder if you can run this offline so you can save the files when you build your static site and no scripts are required on the frontend.

<comment><USER21> The actual text to voice synthesis isn't acutally bad, but the audio quality is terrible. The audio artifacts are so bad I could barely listen to the demo snippets. It's too bad they ruined a quite good product by skimping on the most important quality, audio quality.

I wonder how OP missed this? Audio quality has to be flawless when using headphones for extended times. It's not like you need a lot of bits for audio, so it's not something you save on.

Please fix!

<comment><USER22> Sounds good on my speakers

<comment><USER21> Headphones.

</comment></comment></comment></comment><comment><USER21> You should showcase your product better.

Add a text input box so that prospective users can try it themselves.

</comment><comment><USER23> Typo: "We use Artificial Intelligence and Machine Learning to constantly improve our voice algorithms to make your website text-to-speech is as realistic as possible."

<comment><AUTHOR> Thanks PapaSlug, i like your username. The Typo is fixed now.

</comment></comment><comment><USER24> Constructive criticism , I could not find any samples of the audio on your site.

Recommendation, add the plugin to your own homepage for potential users to test it out before having to signup.

<comment><AUTHOR> thanks for the input:

there is a demo button here on this page where you enable your website. You can even test the feature without signup.

[https://websitevoice.com/convert-text-to-audio- free](https://websitevoice.com/convert-text-to-audio-free)

<comment><USER24> Apologies, looks like your widget domain is being blocked by my work and was preventing the demo button from doing anything.

</comment></comment></comment><comment><USER25> What's the catch? How are you providing this for free.

<comment><USER26> according to what he said on another thread it will be free and it will have some paid features on top

</comment></comment><comment><USER27> Is there a demo of the UK voice anywhere? I could only find the American one.

Very impressed with the audio quality. What's the revenue model?

<comment><USER28> >Very impressed with the audio quality

To me it sounds not much better than Google's standard wavenet TTS (seen here: [https://cloud.google.com/text-to-speech/](https://cloud.google.com/text-to- speech/)) which is admittedly pretty state of the art but nowhere as good as actual human dictation.

</comment></comment><comment><USER29> I use Read Aloud Firefox extension to listen to blog posts.</comment>


RedditStorage - <AUTHOR> https://github.com/Rossem/RedditStorage <comment><USER2> If you had a language model (say, trained on existing comments from Reddit), you could encode the data in the comments in English, and make the abuse a little more subtle.

<comment><USER3> .. Much like how gfycat encodes their links

eg.:

\- [https://gfycat.com/JaggedIdealFrillneckedlizard](https://gfycat.com/JaggedIdealFrillneckedlizard)

\- [https://gfycat.com/ThirstyAmbitiousBuzzard](https://gfycat.com/ThirstyAmbitiousBuzzard)

\- [https://gfycat.com/AlertSpicyBlueandgoldmackaw](https://gfycat.com/AlertSpicyBlueandgoldmackaw)

<comment><USER4> This is also how hipku stores ip addresses as haiku

demo - [http://hipku.gabrielmartin.net](http://hipku.gabrielmartin.net)

explanation - [http://gabrielmartin.net/projects/hipku/](http://gabrielmartin.net/projects/hipku/)

<comment><USER5> No wonder why this is everywhere:

      The hungry white ape

      aches in the ancient canyon.

      Autumn colors crunch.

[https://www.google.com/search?q=%22%22The+hungry+white+ape+a...](https://www.google.com/search?q=%22%22The+hungry+white+ape+aches+in+the+ancient+canyon.+Autumn+colors+crunch.%22&oq=%22%22The+hungry+white+ape+aches+in+the+ancient+canyon.+Autumn+colors+crunch.%22&aqs=chrome..69i64j69i57.3103j0j9&sourceid=chrome&es_sm=122&ie=UTF-8#q=%22The+hungry+white+ape+aches+in+the+ancient+canyon.+Autumn+colors+crunch.%22)

<comment><USER6> According to the overview on the of hipku website, it uses only monosyllabic words, unlike this haiku.

<comment><USER4> That's not entirely true - it only uses monosyllabic words for IPv6 addresses because there's no other way to fit enough bits into the right number of syllables.

For IPv4 addresses, there's loads of space, so I can afford to use some longer words.

</comment></comment></comment></comment></comment></comment><comment><USER7> There should be a contest: Who can find the most implausible data storage medium? (Rated according to various criteria such as ingenuity, reliability, max. data read/write rates, latency, storage size, costs…)

<comment><USER8> To get reliable and free storage, photo hosting is usually the easiest way. Flickr offers 1TB, picasa/g+ offers unlimited storage with some hidden quoats. Everything that allows lossless photos lets you store arbitrary data. Depending on how careful you wanna be you can store hundreds of GBs per account.

Email attachments used to be a great way a while ago but nowadays using multiple gdrive/dropbox/onedrive accounts is much easier.

They are easy to create in large numbers (especially if your ISP has dynamic IPS) and as long as you're even a little bit careful, nearly impossible to ban. Add some redundancy across different services to that and a $2 VPS that gives you tons of upload bandwidth and you've got yourself as many TBs of free,fast and reliable online storage as you want.

I spent so much time as a teenager with no money and some python skills coding storage solutions like that. I'd say it was to store movies and tv shows for myself but in retrospect I mostly did it because it was so much fun to develop.

<comment><USER9> Yeah, I wrote something that stores data to Flickr last summer: [https://github.com/namwen/hoardr](https://github.com/namwen/hoardr) . I kind of had a reason but it was more for the enjoyment of getting it to work.

<comment><USER10> There is also hyperglobalmegastore [https://github.com/adrian- bl/hyperglobalmegastore](https://github.com/adrian-bl/hyperglobalmegastore) All data is encrypted and you can even mount your flickr 'drive' using fuse.

<comment><USER11> This project needs more stars!

</comment></comment></comment></comment></comment><comment><USER12> Well there's some pretty amusing abuse. I recall there was a botnet a while back that got it's commands from a subreddit as well. Quite brilliant actually- who would suspect reddit as a command server?

<comment><USER13> > who would suspect reddit as a command server

Everyone who used IRC as a command server from years past. It turns out that things useful for human communication tend to be useful for computer communication.

Usenet, Email... hell... I'm sure BBS would have been used if modems were popular enough back in the day.

<comment><USER14> Every time I see an api for sending and receiving any type of file or text, I think botnet/building a secret chat system on top of it.

<comment><USER15> Every time I see a service offering some resource as "unlimited", I think of using it as a free backend.

<comment><USER14> me too, I have about 3tb of stuff from the Napster days and I've always wanted to upload it all somewhere so I can stream it on my phone.

</comment></comment></comment></comment></comment><comment><USER16> If this idea appeals to you, you may also be interested in the 2009 paper _Graffiti Networks: A Subversive, Internet-Scale File Sharing Model_ [0] by Andrew Pavlo.

tl;dr: the researchers discovered that MediaWiki instances were good soft targets.

[0] [https://www.cs.cmu.edu/~pavlo/static/slides/graffiti- dc401-o...](https://www.cs.cmu.edu/~pavlo/static/slides/graffiti- dc401-oct12.pdf)

<comment><USER17> The mini-saga embedded in the presentation was pretty funny

> Concluding Remarks >  Off probation at the end of this semester!

<comment><USER18> I got put on probation for redirecting my ~/.bash_history to /dev/null and removing my `finger` information with `chfn`. Universities can be pretty ridiculous with their disciplinary actions.

<comment><USER19> Why... why would redirecting your .bash_history to /dev/null be a punishable offense? I assume it's so they could check for evildoing on your part, but that seems like a ridiculously idiotic way of doing it.

</comment></comment></comment></comment><comment><USER20> An expected reaction from the reddit admins...

[http://www.reddit.com/r/programming/comments/38kn2g/redditst...](http://www.reddit.com/r/programming/comments/38kn2g/redditstorage_a_cloud_storage_that_uses_reddit_as/crvx3tp)

</comment><comment><USER21> One of the co-creators here; as a disclaimer, we didn't mean to threaten to break reddit at all. We're amazed that someone even found this repo since we abandoned it back in January, and that it's even gotten any amount of attention. Honestly we just built this in a week over winter break cause we were bored

</comment><comment><USER22> This is pretty much exactly how binary newsgroups got started. Not to be all "I thought of it first" but I had thought it would be funny to do something similar on Twitter.

<comment><USER23> Woah, that'd be a lot of tweets. Even reddit with its 10000 char limit per comment has loads of comment trains if you want to store a sizable amount of info that way. With twitters 140 char limit, that would be a huge amount of tweets. You'd probably run the risk of being identified as a spammer if you send that many tweets at once...

<comment><USER24> 140 unicode characters. Which actually gives you quite a lot of space to work with.

</comment></comment></comment><comment><USER25> I like the proof of concept, but I hate that anyone would abuse Reddit this way.

<comment><USER26> More insightful than most of the comments on there.

</comment></comment><comment><USER27> Can anyone do a rough cryptoanalysis of the code? It uses AES block cipher in CBC mode with a random iv. Which attacks is this open to?

First, I suspect it's lacking a secure integrity check (MAC), so is weak against chosen ciphertext attacks.

        def encrypt(self, plaintext):
            plaintext = self.pad(plaintext)
            iv = Random.new().read(AES.block_size)
            cipher = AES.new(self.key, AES.MODE_CBC, iv)
            return iv + cipher.encrypt(plaintext)

I'm also not sure about his padding of zeros to attain the AES block size - was there a more secure padding?

        def pad(self, s):
            return s + b"\0" * (AES.block_size - len(s) % AES.block_size)

</comment><comment><USER28> Wouldn't it be funny if reddit just randomly edited the comments to break the encoding...

<comment><USER29> This sounds like a /u/jedberg type of thing to do.

<comment><USER28> I'd only do it to people I know after backing up the original. I wouldn't want someone to _actually_ lose their files.

<comment><USER29> One could argue that that's their fault for giving /u/rram "root" access to their "database."

</comment></comment></comment></comment><comment><USER30> Shame an encryption key is REQUIRED, could be a useful way to transfer files between Reddit users. Of course the file has to be encoded, but the encryption should be an optional extra.

<comment><USER31> You could always share the password, or even redistribute a modified version of the program with a hardcoded password.

<comment><USER21> That was the idea; one of our original goals was to make a system to quickly share small files over reddit. The issue is you have to store the password for each file somewhere

</comment></comment></comment><comment><USER32> Don't use this for anything important, and certainly not with a non-unique password.

Key is derived from a single SHA256 (can be brute-forced very rapidly), cyphertext isn't authenticated (can be tampered with or corrupted without anything noticing), and the padding function is broken (strips trailing NULLs, so no good for binary files).

</comment><comment><USER33> Interesting idea... Since image formats already store a huge BLOB, how much more would it take to make ImgurStorage?

(Ideally, it would be slightly more elegant than just renaming a zip file.)

<comment><USER34> This isn't an area that I'm particularly strong in, but I think that the way that imgur compresses images[0] might have a noticeable effect on this.

[0] [https://help.imgur.com/hc/en-us/articles/201424706-How- does-...](https://help.imgur.com/hc/en-us/articles/201424706-How-does-Imgur- compress-my-images)

<comment><USER35> I've run a few photo sites and one of the things I do on all wild incoming JPGs is do a minor compression on them and if that saves more than about 30% on the file size I just use the compressed version. Then anything that's been camouflaged in there gets dropped.

</comment></comment></comment><comment><USER36> Pretty clever. If it was stored in reddit's wiki system instead of comments, it could have a revision history!

</comment><comment><USER37> I think this will break your ToS with reddit and result in a ban on the account. That said, I don't know. It's kinda cool though.

<comment><USER38> Only in the "hacking the system part". Otherwise it's an abuse of a service. There's plenty of cheap data hosting elsewhere on the net.

</comment></comment><comment><USER39> Presumably something like this is what's happening in /r/A858DE45F56D9BC9/

<comment><USER34> Possibly. Searching up some of the content on there revealed this[0], so at least some of it is (most likely) data.

[0] [http://a858.soulsphere.org/](http://a858.soulsphere.org/)

Edit:[1] shows that this is most likely a false-positive.

[1] [https://www.reddit.com/r/Solving_A858/comments/24vml1/mime_t...](https://www.reddit.com/r/Solving_A858/comments/24vml1/mime_type/chb5k2e?context=3)

</comment></comment><comment><USER40> Welp. This won't last very long. :-)

</comment><comment><USER41> I've been preaching the similarities of reddit to newsgroups and IRC forever so this seems like a natural evolution to me. Probably fairly easy for reddit to shut down though unfortunately.

Now if ISP's would start offering their own cached usable versions of reddit we would be getting somewhere :)

<comment><USER42> And that somewhere would be Usenet 2.0.

</comment></comment><comment><USER43> I had the same idea few months ago, I've even coded simple POC in Java which mapped submissions in subreddit to files. You could use FUSE to create virtual device and map files in mounted folder to comments. For Java I was using fuse- jna - there should be binding for Python.

<comment><USER36> Someone made a reddit FUSE filesystem (I don't know if it still works though)

[https://github.com/ianpreston/redditfs](https://github.com/ianpreston/redditfs)

</comment></comment><comment><USER44> Lol, I've thought of doing this so many times on Facebook, Google+, Twitter and reddit. Seeing the amount of points this gets, I guess I should have done it. I didn't because it seemed so pointless: they'll just block accounts using this.

</comment><comment><USER45> Somewhat related project i had going... [https://github.com/meesterdude/reddit-rust- servers](https://github.com/meesterdude/reddit-rust-servers) ([http://ruru.name/reddit-rust-servers/](http://ruru.name/reddit-rust- servers/) show/hide columns to see more options)

I used to run the rust servers sub. I would have people post JSON posts, which i would then spider and generate a JSON DB from, and created a UI (see the gh- pages branch) to grab the JSON and present a searchable/filterable way of finding servers that are relevant to you.

</comment><comment><USER46> I thought about creating an anonymous peer-to-peer network like BitMessage but over Twitter instead of over TCP/IP. The main benefit is that for the watching government hardware your traffic will flow to twitter, not to some suspicious computers. Of course if government can talk to Twitter, it might find out that activity, but not all governments can talk to Twitter.

Another improvement might be not to send base64 abracadabra, but instead send some readable texts (autogenerated or fragments from wikipedia) and encode message as a slight deviations (typos, etc) using steganography. But it would require a lot of messages to transmit enough data.

<comment><USER21> yeah that'd probably speed things up significantly; we already ran into speed issues with PRAW in terms of how fast it can upload comments

</comment></comment><comment><USER47> Perhaps something like this could explain [http://www.reddit.com/r/A858DE45F56D9BC9](http://www.reddit.com/r/A858DE45F56D9BC9)

</comment><comment><USER48> You could randomly spread this over various subs, that and add erasure coding. This way if a chunk or two goes missing you can reconstruct the original blob.

</comment><comment><USER49> A nice new example of what's called "parasitic storage." This kind should be easy enough to detect on Reddit's end: encrypted and binary data look very different from text. Further, if a site allows binary, it's different from crypto. The only type that's hard to filter is custom stego whose patterns look similar to normally accepted traffic. Extra true if it's a high volume site.

</comment><comment><USER50> From 2010: [https://nealpoole.com/blog/2010/12/bit-ly-file-storage- cleve...](https://nealpoole.com/blog/2010/12/bit-ly-file-storage-cleverness- and-chutzpah/)

</comment><comment><USER51> Reminds me of: [https://twitter.com/manzoor_e/status/604072602114605056/phot...](https://twitter.com/manzoor_e/status/604072602114605056/photo/1)

</comment><comment><USER52> how do you get a Mac OS X GUI around this if it is written in python? Can you do the same with perl, php, and other languages? Interface Builder has always been a stumbling block for me to even begin to learn Obj-C or Swift.

<comment><USER53> It says in the readme he uses wxPython (wxWidgets). You could also use Qt as I believe both use native Cocoa underneath. You can't do everything that is possible if you were coding on Obj-C or Swift, but the stuff you can do looks native. Both have bindings in many languages.

</comment></comment><comment><USER54> Please don't do this . . .

</comment><comment><USER17> I remember once briefly thinking how fun it'd be to do something like this, before realizing with the spam filters the way they are it'd probably be the last thing I ever did on the site.

Neat proof-of-concept though

</comment><comment><USER55> RedditStorage reminds me of a couple business models we tried out that tanked..

The first was a new business where we would go to trade shows, conventions, hell even fast food places, and just collect as many free beverages, condiments, napkins et cetera as possible. Then we'd sell them online.

The other one didn't do much better. We'd go to a Lowes Tool Rental, and just rent a bunch of tools and then re-rent them out of our truck in the parking lot. They had to have them back an hour before Lowes closed for the night.

Our current business model is, we go to bars and hit on people, and if we get their phone numbers, we add it to a subscription service where other people can have access to it.

Honestly, I feel we're no more in the wrong than RedditStorage is.. /s

</comment><comment><USER56> > RedditStorage uses an AES encryption algorithm which requires you to choose > a password (e.g. "bunny")

Some people still don't know what a password is? =D

</comment><comment><USER57> What happens when someone uses this to pollute popular subreddits? People will get pissed off...

<comment><USER58> I guess they will simply block the username and delete the comments.

</comment></comment><comment><USER59> that is awesome in its complete disregard of reddit :) and a death sentence to itself if it gains popularity as reddit-admins will have to ban the accounts/discard the content :) so it's not that secure a storage idea...

nice little engineering work though. kudos.

</comment><comment><USER60> Yeah, it can be done, but who the fuck needs that ?

</comment><comment><USER61> does reddit not have some sort of posting throughput limit?

<comment><USER62> If they don't, they will soon.</comment></comment>


Metaflow, Netflix's Python framework for data science, is now open source - <AUTHOR> https://metaflow.org <comment><USER3> After going through a lot of marketing fluff, I landed on this useful page which explains Metaflow basics: [https://docs.metaflow.org/metaflow/basics](https://docs.metaflow.org/metaflow/basics)

Here's my understanding:

\- It's a python library for creating & executing DAGs

\- Each node is a processing step & the results are stored after each step so you can restart failed workflows from where it failed

\- Tight integration with AWS ECS to run the whole DAG on cloud

I don't know why their .org site oddly feels like a paid SaaS tool. Anyway, thank you Netflix for open sourcing Metaflow.

<comment><USER4> looks like airflow with ML tools?

<comment><USER5> It provides a Python DAG building library like Airflow, but doesn't do Airflow's 'Operator ecosystem' thing.

It also is very opinionated about dependency management (Conda-only) and is Python-only, where Airflow I think has operators to run arbitrary containers. So Metaflow is a non-starter I think if you don't want to exclusively use Python.

Airflow also ships with built-in scheduler support (Celery?) or can run on K8s. Metaflow doesn't have this. Seems to rely on AWS Batch for production DAG execution.

Airflow ships with a pretty rich UI. Metaflow seems to be anti-UI, and provides a novel Notebook-oriented workflow interaction model.

Metaflow has pretty nice code artifact + params snapshotting functionality which is a core selling point. Airflow is not as supportive of this so it's harder to do reproducibility (I think). This is encapsulated by their "Datastore" model which can locally or in S3 persist flow code, config and data.

<comment> savin-goyal Metaflow does come bundled with a scheduler that can place jobs on a variety of compute platforms (current release supports local on-instance and AWS batch). In terms of dependencies, we went with conda because of its traction in the data science community as well as excellent support for system packages. Our execution model also supports arbitrary docker containers (on AWS batch) where you can theoretically bake in your own dependencies. In terms of language support, we have bindings for R internally, that we plan to open source as well.

I wouldn't qualify metaflow as anti-UI. For model monitoring, we haven't found a good enough UI that can handle the diversity of models and use cases we see internally, and believe that notebooks are an excellent visualisation medium that gives the power to the end user (data scientists) to craft dashboards as they see fit. For tracking the execution of production runs, we have historically relied on the UI of the scheduler itself (meson). We are exploring what a metaflow-specific UI might look like.

As for comparisons with Airflow, it is an excellent production grade scheduler. Metaflow intends to solve a different problem of providing an excellent development and deployment experience for ML pipelines.

<comment><USER6> Thanks for open sourcing this! This seems like precisely the kind of tool I've been looking for. One thing that would be really great is support for HPC schedulers like SLURM. It seems relatively straightforward to add, so I might give it a shot myself.

<comment><USER7> Happy to help either through our gitter chat or firstname.lastname@example.com.

</comment></comment></comment></comment></comment></comment><comment><USER8> How is this different / better to existing tools or workflows? I don't like to criticise new frameworks / tools without first understanding them, but I like to know what some key differences are without the marketing/PR fluff before giving one a go.

For instance, this tutorial example here ([https://github.com/Netflix/metaflow/blob/master/metaflow/tut...](https://github.com/Netflix/metaflow/blob/master/metaflow/tutorials/01-playlist/playlist.py)) does not look substantially different to what I could achieve just as easily in R, or other Python data wrangling frameworks.

Is the main feature the fact I can quickly put my workflows into the cloud?

<comment><AUTHOR> Here are some key features:

\- Metaflow snapshots your code, data, and dependencies automatically in a content-addressed datastore, which is typically backed by S3, although local filesystem is supported too. This allows you to resume workflows, reproduce past results, and inspect anything about the workflow e.g. in a notebook. This is a core feature of Metaflow.

\- Metaflow is designed to work well with a cloud backend. We support AWS today but technically other clouds could be supported too. There's quite a bit of engineering that has gone into building this integration. For instance, using the Metaflow's built-in S3 client, you can pull over 10Gbps, which is more than you can get with e.g. aws CLI today easily.

\- We have spent time and effort in keeping the API surface area clean and highly usable. YMMV but it has been an appealing feature to many users this far.

Hope this makes sense!

<comment><USER9> > using the Metaflow's built-in S3 client, you can pull over 10Gbps, which is > more than you can get with e.g. aws CLI today easily

Can you please explain how you were able to better the performance of aws cli.

<comment><AUTHOR> you need more connections than what a single AWS CLI process open to saturate network on a big box. You can achieve the same by doing some "xargs | aws cli" trickery but then error handling becomes harder.

Our S3 client just handles multiple worker processes correctly with error handling.

</comment></comment></comment></comment><comment><AUTHOR> hey, I'm one of the authors of Metaflow. Happy to answer any questions! Netflix has been using Metaflow internally for about two years, so we have many war stories :)

<comment><USER10> The centralized DAG scheduler seems like a pretty important part. How much will not having Meson hamper the usability?

<comment><USER11> Hi bhtucker, We do have plans to integrate with production-grade schedulers in the very near future. [https://github.com/Netflix/metaflow/issues/2](https://github.com/Netflix/metaflow/issues/2)

</comment></comment></comment><comment><USER12> This looks exciting! I'll play around with the tutorial and try to set up the AWS environment this weekend. I have several questions.

1\. At what sort of scale does Metaflow become useful? Would you expect Metaflow to augment the productivity of a lone data scientist working by himself? Or is it more likely that you would need 3, 10, 25, or more data scientists before Metaflow is likely to become useful?

2\. When you move to a new text editor, there are some initial frictions while you're trying to wrap your head around how things work. So, it can take some time before you become productive. Analogously, I imagine there are initial frictions when moving to Metaflow. In your experience, after Metaflow's environment has already been established, how long does it take for data scientists to get back to their initial productivity? It would be useful to have a sense of this for the data scientist who would want to sell their organization on adopting Metaflow.

3\. Many data scientists work in organizations which have far less mature data infrastructure than Netflix, and/or data science needs of a much smaller scale than Netflix. In particular, I may not even have batch processing needs (e.g. a social scientist working on datasets which can be held entirely in memory). In that case, is Metaflow useful?

4\. What's the closest open-source alternative to Metaflow on the market? Off the top of my head, I can't think of anything which quite matches.

<comment><USER7> 1\. Metaflow should best help when there is an element of collaboration - <USER2> small to medium team of data scientists. Collaborating with your self is also another scenario when Metaflow can be useful since it takes care of versioning and archiving various artifacts.

2\. Keeping the language pythonic, without any additional need to learn a DSL has definitely been key to Metaflow's adoption internally. That said, this is something we are open to hearing back, esp. with this OSS launch.

3\. Yes - definitely think so. Personally my favorite is the local prototyping experience part; when everything can fit in memory and is blazing fast. There is an also an open issue for fast-data access, which you can upvote if interested in seeing it open-sourced.

4\. We don't think there is an exact equivalent as well. :)

<comment><USER5> Re 4, aren't Kubeflow and Lyft's recently open-sourced "Flyte" pretty similar?

If you don't consider them basically equivalent, what would you say are the key differences?

<comment><USER7> Thanks for pinging on this.

re: Kubeflow - imho it is quite coupled to Kubernetes. We don't intend to be tied to a specific compute substrate even though the first launch is with AWS. We do follow a plugin architecture - so I'm hoping Kube happens sometime.

re: Flyte - I'm less informed on this but happy to educate myself and get back.

<comment><USER5> That's true of Kubeflow. I'm not sure that project will be as keen on being as "compute substrate" agnostic as Metaflow too, given its connection with Google.

If you feel inclined jump in the Flyte Slack and share your thoughts :). At my company we're on Kubeflow/Argo now, but things are developing quite a lot in this space so keen to not be myopic.

<comment><USER7> Thanks for sharing the context. Hopefully we can have a (fast) follow up with Kube integration depending on demand.

</comment></comment></comment></comment></comment></comment><comment><USER13> Is there a reason to use this over DVC[1] which is language and framework agnostic and supports a large number of storage backends? It works with any git repo and even polyglot implementations and can run the DAG on any system.

Currently using DVC, MLflow just for metadata visualization and notes on experiments, and Anaconda for (python) dependency management. We are an embedded shop so we don't deploy to the "cloud."

[1]: [https://dvc.org/](https://dvc.org/)

<comment><AUTHOR> We are good friends with the DVC folks! If the DVC + MLFlow + Anaconda stack works for you, that's great. Metaflow provides similar features. The cloud integration is really important at Netflix's scale.

</comment></comment><comment><USER14> My team has a similar library called Loman, which we open-sourced. Instead of nodes representing tasks, they represent data, and the library keeps track of which nodes are up-to-date or stale as you provide new inputs or change how nodes are computed. Each node is either an input node with a provided value, or a computed node with a function to calculate its value. Think of it as a grown-up Excel calculation tree. We've found it quite useful for quant research, and in production it works nicely because you can serialize entire computation graph which gives an easy way to diagnose what failed and why in hundreds of interdependent computations. It's also useful for real-time displays, where you can bind market and UI inputs to nodes and calculated nodes back to the UI - some things you want to recalculate frequently, whereas some are slow and need to happen infrequently in the background.

[1] Github: [https://github.com/janushendersonassetallocation/loman](https://github.com/janushendersonassetallocation/loman)

[2] Docs: [https://loman.readthedocs.io/en/latest/](https://loman.readthedocs.io/en/latest/)

[3] Examples: [https://github.com/janushendersonassetallocation/loman/tree/...](https://github.com/janushendersonassetallocation/loman/tree/master/examples)

</comment><comment><USER15> I am disappointed that when I click on documentation, "why metaflow," I get a bunch of cartoony BS instead of a simple text explanation. Glad these folks don't write RFC'S.

Edit: just went to the Amazon CodeGuru homepage. Fantastic! Wish they were all like that.

</comment><comment> purple-again We are on Azure using Spark via Databricks. We had to abandon sci kit learn because of this choice. Does your service require AWS and can it be used in conjunction with Spark? Thank you for your time and consideration.

<comment><USER16> What about databricks made you abandon sklearn?

<comment><USER17> MLLib I would think - [https://spark.apache.org/docs/latest/ml- guide.html](https://spark.apache.org/docs/latest/ml-guide.html)

</comment></comment></comment><comment><AUTHOR> btw, if you happen to be at AWS Reinvent right now, you can get a stylish, collector's edition Metaflow t-shirt if you drop by at the Netflix booth at the expo hall and/or ping us otherwise!

</comment><comment><USER18> How does this compare to snakemake[1] and nextflow[2]?

[1] [https://snakemake.readthedocs.io/en/stable/](https://snakemake.readthedocs.io/en/stable/) [2] [https://www.nextflow.io/](https://www.nextflow.io/)

<comment><USER19> The fact that metaflow works directly in Python piques my interest. I can lint it, I can test it, I can format it, I can easily extend it.

I've been hesitant to commit myself and my collaborators to yet another DSL -- and that's part of why I haven't seen much to offer in snakemake and nextflow.

<comment><USER7> Yes - that's our thinking too. Compilers finding your typos for variable names seems helpful for user productivity.

</comment></comment></comment><comment><USER20> Can anybody provide a good comparison e.g. with Meltano?

I am not affiliated with the Meltano people, but I like the idea of keeping the system modular, what seems to make it easier to replace components.

I have no doubt that we will see better replacements for every component of a data pipeline in the coming years. If there is only one thing to do right, then it´s to not bet on one tool but keep the whole stack flexible.

I am still missing well established standards for data formats, workflow definitions and project descriptions - hopefully open source ninjas will deliver on this front before proprietary pirats will destroy the field with progress-inhibiting closed things. It seems to be too late to create an "Autocad" or "Word" file format for datascience, but I see no clear winner atm, but hopefully my sight is bad - please enlighten me!

<comment> savin-goyal I am not familiar with Meltano, sorry.

</comment></comment><comment><USER21> Seems like a cool addition to the DAG ML tooling family. Thanks for sharing! Do you support, or plan to support, features commonly found in data science platform tools like Domino ([https://www.dominodatalab.com/](https://www.dominodatalab.com/))? I'm thinking of container management, automatic publishing of web apps and API endpoints, providing a search for artifacts like code or projects, etc.

<comment><AUTHOR> Good question. We have many common features covered:

\- Container management: See [https://docs.metaflow.org/metaflow/dependencies](https://docs.metaflow.org/metaflow/dependencies)

\- Search for artifacts: see [https://docs.metaflow.org/metaflow/client](https://docs.metaflow.org/metaflow/client)

\- Automatic publishing of web apps: we have this internally but it is not open-source yet. If it interests you, react to this issue ([https://github.com/Netflix/metaflow/issues/3](https://github.com/Netflix/metaflow/issues/3))

Let us know if you notice any other interesting features missing! Feel free to open a GitHub issue or reach out to us on [http://chat.metaflow.org](http://chat.metaflow.org)

</comment></comment><comment><USER22> This looks like a fantastically clean API for Python data and ML pipelines. Congratulations!

It would be great to have a scheduler and monitoring UI that are equally lightweight.

<comment><AUTHOR> Metaflow comes with a built in scheduler. If your company has an existing scheduler, e.g. for ETL, you can translate Metaflow DAGs to the production scheduler automatically. This is what we do at Netflix.

We could provide similar support e.g. for Airflow, if there's interest.

For monitoring, we have relied on notebooks this far. Their flexibility and customizability is unbeatable. We might build some lightweight discovery / debugging UI later, if there's demand. We are a bit on the fence about it internally.

<comment><USER5> > For monitoring, we have relied on notebooks this far.

This is pretty interesting approach. Notebooks are a natural environment of Data Scientist and for monitoring they'd provide really good discoverability, flexibility, and interactivity.

Do you use more traditional monitoring that will alert you somehow if a workflow fails, or a workflow hasn't run at all for X hrs/days?

<comment><USER11> At Netflix, we rely on the workflow scheduler for such alerting and bundle in a layer of triggering mechanism (custom notifications and such)

</comment></comment></comment></comment><comment><USER23> The link in the docs to the CloudFormation template source is broken: [https://docs.metaflow.org/metaflow-on-aws/deploy-to- aws#clou...](https://docs.metaflow.org/metaflow-on-aws/deploy-to- aws#cloudformation-template) Instead of /Netflix/metaflow-tools/aws it should probably be /Netflix/metaflow-tools/tree/master/aws

<comment><USER7> Thanks for reporting it. We ll fix it. Sorry for the inconvenience.

<comment><USER23> No worries, on the whole the documentation is top notch.

</comment></comment></comment><comment><USER24> Very interesting project. I love that this allows you to transparently switch "runtime" from local to cloud, like spark does, but integrated with common python tools like sklearn/tf etc. Looking forward to test metaflow out myself.

<comment><USER7> Thanks. Let us know how you like the prototyping -> scaling out & up journey.

</comment></comment><comment><USER25> i looked over the tutorials and curious to know, whether the tutorials are representative of how Netflix does ML ?

is data really being read in .csv format and processed in memory with pandas ?

because I see "petabytes of data" being thrown everywhere, and i am just trying to understand how one can read gigabytes in .csv process do simple stats like grouping by in pandas - shouldn't simple SQL DWH do the same thing more efficiently with partitioned tables, clustered indexes and the power of SQL language ?

i would love to take a look at one representative ML pipeline (even with masked names of datasets, features) just to see how "terabytes" of data get processed into a model

<comment><AUTHOR> Good question! A typical Metaflow workflow at Netflix starts by reading data from our data warehouse, either by executing a (Spark)SQL query or by fetching Parquet files directly from S3 using the built-in S3 client. We have some additional Python tooling to make this easy (see [https://github.com/Netflix/metaflow/issues/4](https://github.com/Netflix/metaflow/issues/4))

After the data is loaded, there are bunch of steps related to data transformations. Training happens with an off-the-shelf ML library like Scikit Learn or Tensorflow for training. Many workflows train a suite of models using the foreach construct.

The results can be pushed to various other systems. Typically they are either pushed to another table or as a microservice (see [https://github.com/Netflix/metaflow/issues/3](https://github.com/Netflix/metaflow/issues/3))

</comment></comment><comment><USER26> It's so simple and intuitive to run two steps in parallel. Thank you, Netflix!

<comment> savin-goyal You're welcome! :)

</comment></comment><comment><USER27> this seems like it's very similar to [http://metaflow.fr](http://metaflow.fr) is there any relation, or is this a name collision?

<comment><AUTHOR> no relation

</comment></comment><comment><USER17> How does it compare to dragster.io?

[https://github.com/dagster-io/dagster](https://github.com/dagster-io/dagster)

<comment><AUTHOR> Orchestrating a workflow, which is what Dagster does, is just one part of Metaflow. Other important parts are dependency management, cloud integration, state transfer, inspecting and organizing results - features that are central to data science workflows.

Metaflow helps data scientists build and manage data science workflows, not just execute a DAG.

</comment></comment><comment><USER28> At first glance, I see BASIC's GOTO statement.

</comment><comment><USER29> How does this compare to Kedro?

</comment><comment><USER30> Awesome!!!</comment>


Pearsons, Who Pledged $100M to UChicago, Want Their Money Back - <AUTHOR> https://www.chicagomaroon.com/article/2018/3/5/pearsons-want-100-million-back-from-univeristy-of-chicago/ <comment><USER2> One of the more ridiculous things from the article:

> _Apart from ensuring that the Institute had a world-class in-house faculty, > the contract also mandated that the University host an annual "Pearson > Global Forum" (PGF) that would convene leading conflict scholars and > policymakers from around the world.

> The University, according to the suit, was obligated to hold the first PGF > by October 31, 2018, but said that it could not host the event by then as > "it had not planned or done the necessary preparatory work." The Pearsons > alleged that the University informed them that it would attempt to meet this > obligation by that date by instead involving the Institute in the 2018 Irish > Catholic Bishops' World Meeting of Families Congress, an event which, > according to its website, "brings together families from across the world to > celebrate, pray and reflect upon the central importance of marriage and the > family as the cornerstone of our lives, of society and of the Church."_

<comment><USER3> It seems like there was a mismatch in expectations that goes to the heart of how universities and philanthropy work.

The university thought it was getting a donation, along with some vanity conditions, comparable to naming a building or a chair, which it could fudge while spending the money on whatever it wanted.

The foundation thought it was paying the university to carry out something very specific on its behalf.

<comment><USER4> > The university thought it was getting a donation, along with some vanity > conditions, comparable to naming a building or a chair, which it could fudge > while spending the money on whatever it wanted.

I think it's clear that the conditions were more than vanity, and were actually qualitative.

</comment></comment></comment><comment><USER5> From reading the article, doesn't seem like UChicago was very interested in actually running the institute as much as sticking it with operating expenses and doing the bare minimum.

I'm sure more will come out - this will be an interesting lawsuit.

<comment><USER6> Yep. In grad school, the university took like 50% of the funds for our project, probably to plug budget gaps. I agree with the first comment:

"Relying on a university administration is generally not prudent. It is clear that the Family envisioned an effective "new business," in the spirit of a venture or a "start-up" and those kinds of projects take serious business execution. In my experience, the Pearson's complaint merely exposes the tip of the iceberg as far as what the donor "sausage machine" looks like in a major university. Lesson? Donors need to act more like active investors, despite the general hard resistance universities put up."

They were expecting university administrators to provide "serious business execution".

</comment></comment><comment><USER7> Can someone clarify this for me?

 _> A few days before Robinson's appointment as institute director, the University also announced that two scholars, an associate professor at Columbia University and an assistant professor at New York University, would assume named full professorships at the Pearson Institute._

 _> "For the other two chaired positions, the U of C hired two junior, non- tenured professors from academic institutions that are ranked below the U of C in national academic standings," the complaint reads._

Putting aside the difficulty of rating universities by any objective metric, when did UChicago become more highly ranked than NYU and Columbia? I'm surprised about both, but I'm rather shocked about Columbia. According to US News and World Report[1], UChicago is in a two-way tie for third place among all universities in the US. I'm aware it's a good school, but I didn't realize it's "officially" beating out Stanford, MIT, Berkeley, CalTech and every Ivy except for Harvard and Princeton.

_____________________________

1\. [https://www.usnews.com/best-colleges/rankings/national- unive...](https://www.usnews.com/best-colleges/rankings/national-universities)

<comment><USER8> Forgetting about "official" rankings for now, Iirc uchicago has had more nobel prize winners passing through in some capacity than anywhere else, and its graduate programs have been among, or the best for a long, long time. It may not have had "kitchen table" renown but it was well known in academics.

For example, several years ago a couple friends I knew withdrew their undergrad applications to Harvard when they were granted acceptance to uchicago because of the mathematics curriculum (it wasn't an objective choice necessarily, but it was a well thought out one that was academically motivated).

It was a little odd that its graduate programs were almost universally held at the pinnacle of academics, while most people didn't think the undergrad was that good. That changed when uchicago accepted the common app (looks more selective) even though the rest didn't change as much. Suddenly it skyrocketed in the rankings. It was always that good, though.

<comment><USER9> [https://www.bestmastersprograms.org/50-universities-with- the...](https://www.bestmastersprograms.org/50-universities-with-the-most- nobel-prize-winners/)

Most Nobel Prizes (per "unnoficial" counts that have some weird inflation for some universities but not Chicago!):

1\. Harvard

2\. Columbia

3\. Cambridge (UK)

4\. Chicago

5\. MIT

6\. Berkely

7\. Oxford (UK)

<comment><USER10> In 1976 I applied to the 4 of the 5 top grad schools in math (I let my mother decree that Berkeley was too far away from where she lived -- if that sounds weird, please realize I was 16 years old at the time). Those were Harvard, MIT, Chicago and Princeton. I was accepted at all except Princeton, and visited the other 3. There was no apparent academic difference between the Harvard and MIT departments, because both schools said one could study interchangeably at either of them. Harvard easily beat MIT on other criteria.

As for Chicago -- well, when I asked Irving Kaplansky to volunteer something I might not yet know, he told me the history of the department. That didn't impress me. I eventually picked Harvard over Chicago for several reasons:

\-- Chicago had a fixed course curriculum. Harvard had zero actual course requirements. \-- Harvard's faculty seemed more approachable by and engaged with the students. \-- I was more attracted to Harvard's resources in other departments. (And in fact, I wound up post-doccing at Harvard in public policy.) \-- Cambridge was a lot nicer area to live in than the area around University of Chicago.

I made the right choice. But I surely would have been happy with Chicago as well.

</comment></comment></comment></comment><comment><USER11> On a much smaller scale but country singer Garth Brooks donated $500k to his hometown hospital and then got his money back after they didn't follow through.

No written agreement but he still won a lawsuit and was awarded damages.

[https://www.nytimes.com/2014/11/07/giving/when-a-donation- st...](https://www.nytimes.com/2014/11/07/giving/when-a-donation-steers-off- course.html)

</comment><comment><USER12> This reads to me a little like an investor lawsuit transposed into the world of academia. In this case the Pearsons have the role of a philanthropic VC fund, and the Pearson Institute was a startup whose product would be groundbreaking research. And now the Pearsons have filed suit alleging that UChicago misled investors.

<comment><USER13> _> This reads to me a little like an investor lawsuit transposed [...] the Pearsons have the role of a philanthropic VC fund_

In case you weren't aware, "Venture Philanthropy" is a real, concrete term used since the 1960's and apropos![0] "Philanthropic VC [Fund]" is an oxymoron, though, I'm sure it's been used jokingly.

[0] [https://www.investopedia.com/terms/v/venture- philanthropy.as...](https://www.investopedia.com/terms/v/venture- philanthropy.asp)

</comment></comment><comment><USER14> I was briefly a PhD student in government at one of the schools mentioned herein, and it made a big splash in the field when Blattman & Dube moved. A few notes.

1) Robinson/the institute using its resources to lure two rising stars rather than established names is par for the course (though giving them named professorships so early in their careers is unusual, I think). I doubt the Pearsons are up on how such things typically operate.

2) The number of people who are both "senior scholars studying conflict resolution" and "researchers doing data-driven, rigorous work" might be in the single digits, or an empty set. The people who come to mind are mostly in their 40s, which is mid-career.

3) I personally hope the Pearsons win their lawsuit. First, giving to a wealthy university is really bad from an effective altruism POV, and if this kind of acrimony discourages such donations, _and_ if subsequent substitute donations create higher utility, then that's great. Second, if universities only accept gifts whose conditions they are capable of fulfilling, I would hope for less administrative bloat. Or, just as possibly, the Pearsons' winning won't have the desired effects at all, who can say.

</comment><comment><USER15> I feel like part of the story is missing because I have trouble imagining that UC couldn't easily find someone to head the institute. I would think that getting to head a well-endowed institute for a top-3 university would be one of the few steps up a Harvard professor could make--and then to have trouble even filling other tenured professorships?

Are the Pearsons notoriously difficult people? Was there a perception or whisper campaign that the institute wouldn't be as academically free or rigorous as advertised? Was Diermeier maneuvering behind the scenes to torpedo the institute? I get the sense that a lot of the story is still hidden.

<comment><USER16> According to the article, this isn't the first time they've done this:

>The Pearsons are no strangers to lawsuits regarding their philanthropic contributions. In 2011, Thomas Pearson sued Garrett-Evangelical Theological Seminary, his father's alma mater, alleging that a $1.2 million gift made in their name to the Methodist seminary had been misused. The United States District Court for the Northern District of Illinois ultimately ruled that the Seminary had respected the terms of the contract and dismissed all of the Pearsons' claims.

My hunch is that they grossly overestimated what a political science department could do to "reduce conflict around the globe".

</comment></comment><comment><USER17> As a UChicago alum I'm somewhat bewildered by their actions, and saddened for the bad press and more importantly what seems to be a missed opportunity to take advantage of an unprecedented gift. I'm curious how this will be seen to reflect on Robert Zimmer. Although he may not have been directly involved in the project, I'm sure he had a lot to do with the arrangement and negotiations, and the buck ultimately stops with him -- assuming, of course, that UChicago screwed up as they are accused of having done.

</comment><comment><USER18> I'm not suggesting this is what's happening, but: does anyone know the financial implications of making a (presumably tax-deductible) charitable donation that you later revoke?

<comment><USER19> I'm sure it is structured in a way that mitigates that issue. E.g. donate to an intermediary trust for the tax benefit, then that trust makes the gift to the university. In fact the article seems to name this entity as the Pearson Family Foundation.

</comment></comment><comment><USER20> > failed to fulfill several key contractual obligations

I don't understand how you can attach strings (i.e. conditions ) on gifts, telling the receipt how they can or cannot use the gift.

> In the suit, the Pearsons declared that they had lost all confidence that > the University would be "an appropriate or capable steward of the Pearson > Family legacy" as it had allegedly failed to fulfill several key contractual > obligations that were agreed upon when the gift was made.

This would be akin to telling my dad that I will gift him a car for his birthday, but on the condition that he can never sell it or rent it out or re- gift it to his friend. If he doesn't do as I say, I'll sue him to get the gift (car) back.

Or is it that corporate gifts are not the same as personal gifts?

[EDIT: What's with the downvotes? I'm not trolling, this is a serious question.]

<comment><USER21> From the article it seems like they wanted to make sure the money was going to good use and it wasn't.

<comment><USER22> You're not addressing the question, though. This was not a contract for services rendered. It was a gift.

<comment><USER23> According to the article it was a contract. The University of Chicago entered a contract with the Pearson family to host an institute. That the university _reported_ it as a gift in the press was perhaps unfortunate, but it is understandable as it has better optics.

If the idea for the Institute had been the universities, and they were soliciting donors to fund it, and Pearson was one of multiple donors, sure _that_ would qualify as a gift, but as the article is written, this was a very different beast.

<comment><USER24> It can still be a gift, even if strings are attached.</comment></comment></comment></comment></comment>


Ask HN: Why are successful open source projects usually developed by small teams - <AUTHOR> Many people fear using new open source project if it is developed by a small team or even a single person. This is one of the top concerns from our users [1].
But the history of Open Source Software proves that many successful projects are initially created by a single person:

  - Linux was created by Linus Torvalds.
  - Nginx was created by Igor Sysoev.
  - Memcached was created by Brad Fitzpatrick.
  - Redis was created by Salvatore Sanfilippo.
  - Python was created by Guido van Rossum.
 The list may be continued (mention missing projects in comments).
It would be great to have answers on the following questions:
1) Is there positive correlation between the initial size of the team and the success of open source project?
2) Why people prefer using new open source projects developed by big teams instead of projects created by small teams?
3) What about architecture quality and code quality? Do they differ between projects created by big teams and projects developed by small teams?
4) What about bus factor? Is this valid concern for OSS developed by small teams? Are there prominent examples?
Let's discuss these questions.
[1] https://github.com/VictoriaMetrics/VictoriaMetrics/ <comment><USER2> Hi, I'm maintainer of Wekan [https://wekan.github.io](https://wekan.github.io) . After original creator of Wekan, I have been maintaining Wekan since 2016-12 [https://github.com/wekan/wekan/wiki/FAQ#what-was-wekan- fork-...](https://github.com/wekan/wekan/wiki/FAQ#what-was-wekan-fork--wefork) . Some stats at [https://www.openhub.net/p/wekan](https://www.openhub.net/p/wekan) .

1) Success is, does somebody continue to maintain project, and continue to code. There has been many contributors during these years, but it depends when each of them have time to contribute, or do they get busy doing something else.

2) I think it depends what software someone likes to use.

3) Architecture depends on decisions of original software creator. If someone is fixing bugs, adding features and improving architecture, it's OK.

4) Bus factor is mainly, is someone still using the software and interested in improving it. There are very many projects without maintainers.

</comment><comment><USER3> I think this observation applies to software development generally, not just open source projects. In "The Mythical Man Month", Fred Brooks talks about the concept of "the surgical team". One thing he stresses is the idea of having a single, overarching vision of what the architecture should look like. This becomes harder to achieve as more people are added to a project. More opinions are tossed around and debated. Lots of people have lots of good ideas, but not all good ideas are compatible with each other.

</comment><comment><USER4> In my experience, good software is developed by _tight_ teams. Small teams usually have a clearer direction and can respond to change faster, (startups are a good example).

Conversely, large distributed teams usually produce more generalised software with less direction (see [1] conway's law).

But _Team size_ is just one (maybe the least important) of several factors that contribute to an Open-Source project's success. It's hard to distil why some project's are successful, but it generally comes down to:

\- Pain points that the solution fixes, (is it way better than alternatives)

\- Complexity of solution, (how hard is it to set up)

\- Motives of the project, (is it for-profit? consultant-ware? etc..)

\- Size of the community, (argueably the most important factor)

[1] [https://en.wikipedia.org/wiki/Conway%27s_law](https://en.wikipedia.org/wiki/Conway%27s_law)

</comment><comment><USER5> I had an important project while working for a big company. I was the only person writing code and I had to coordinate with and wait for approval from 4 different managers only one of which had technical background (and was the busiest). Oh and each of them was paid at least twice as much as I was.

</comment><comment><USER6> Coherence in thought can be a multiplier in productivity.

<comment><AUTHOR> Does this mean that small teams usually produce better code than bigger teams?

<comment><USER6> Only if they have alignment.

</comment></comment></comment><comment><USER7> >Many people fear using new open source project if it is developed by a small team or even a single person.

I fear because that makes for a smaller/single point of failure. Some guy or a few people make something really nice and useful which you like? Well, he's gotten bored now and has decided to stop developing/maintaining it. Now you're screwed for some time until other people pick it back up depending on demand. Feels like it happens way too often, so I'm not too interested in picking up new things too much compared to established projects where people are actually paid to do it or at least have some responsibility.

<comment><USER8> > Well, he's gotten bored now and has decided to stop developing/maintaining > it.

You can still pay the person to maintain the software or even outright hire him/her if the software is useful to your company.

<comment><USER9> Yes, but that is significantly more expensive than paying of a license from a team which already has lots of other companies paying for licenses.

Though it is less expensive than trying to do the same thing when a proprietary dev shop loses interest.

</comment></comment></comment><comment><USER10> Bus factor.

You may start using a project, everything is fine, but then you hit the bug. And due to author unavailability(vacations, full-time job, just an abandoned project, etc) to solve the problem you need to fork it and maintain on your own or just find another thing.

<comment><AUTHOR> It is interesting to look at known examples of open source projects suffered from bus factor.

Open source projects allow fixing the bug on your own. Then the bugfix may be pushed to the original project. Popular abandoned projects may be forked by the community in order to continue the evolution.</comment></comment>