We Need DevOps for ML Data - <AUTHOR> https://tecton.ai/blog/devops-ml-data/ <comment><USER2> In my experience, the problem has a lot to do with how teams organize around ML.

When you have engineering team separate than a data science team, you'll inevitably have unproductive conflict & politics. One team might be incentivized for stability and speed (engineering or ops) and the other model accuracy (data science). The end result can be disastrous... An engineering team that wants to bend nothing to help data scientists get their work in production. Or a data science team that only cares about maximizing accuracy, even if it might destroy prod, or be impractical to implement in a performant way.

To hit the sweet spot on accuracy, speed, and stability, you need to have one team that focuses on the end feature. It needs to be cross-functional and accountable for doing a great job at that feature. And the data scientists need to be possibly more focused on measuring and analyzing the feature's success, rather than just building models for models sake.

I'd recommend the book Agile IT Organization Design if you're interested in good team design patterns

<comment><USER3> This. In my experiences across larger enterprises, data science teams rarely hold the key to production environments and therefore, relies heavily on IT to productionizing ML. And I completely agree that data scientists need to be focused on measuring and analyzing success as opposed to churning more and more models.

<comment><USER4> In many cases, this is not just because production teams don't want data science teams to be able to deploy to production --due to lack of trust or confidence- but data science teams often don't want this responsibility.

<comment><USER2> There's also perhaps a syndrome of not wanting to do the organizational work to do ML well. Instead of changing the whole org by integrating ML into every team, a data science team is hired doing god knows what. There's status assigned to being the "data scientist" and they work away, siloed, on fun sounding deep learning models. In this mode, if they produce anything it's impractical, divorced from the product realities, and rather hard for the main engineering/product org to maintain or implement.

The reality is there's more work to embracing ML than hiring data scientists. Everyone needs to understand ML a little, and it needs to be OK to critically question data science work from product and engineering angles.

<comment><USER4> Another aspect of this I've observed -- Personal sense of value (and industry pay goes into this) contributes to partitioning of work. If we're charitable, it's from a belief of comparative advantage, and if we are brutally honest about some people, it's because people often feel that "_____ isn't a good use of their time." This is also fed by the "sexiest job of the 21-st century" saying that's been created.

We see this in data science and machine learning where people complain about spending their time cleaning data, etc... when their time should be spent "generating insights/etc." We also see that those insights are interesting but not very useful if they aren't actionable, too costly or too impractical to implement.

Ultimate value is related to being able to contribute to and achieve the holistic outcome, but the lens of success is often focused on models or insights instead. This is a cultural and organizational problem, rather than a technological one. It also takes a dose of humility to appreciate the true value of the so-called dirty work.

<comment><USER3> Another point is that while we technologists love to marvel at data science and machine learning, it still begs the question of what value does it bring to the business. Does the added responsibility of creating all the infrastructure and processes worth it to justify a 5% increase in conversion rates? As you say, even the dirty work has a cost and whether that cost is worth paying to find out that there's nothing you can do to improve the business. That's why all the massive multi-year central data warehouse cleansing type projects keep failing without yielding much value. There's just a lack of focus on delivering incremental value with these data projects.

<comment><USER5> I think currently it creates new possibilities to do business. Computer vision is that point that nowadays it is more engineering than data science so adding something like somewhat good object detection is not that hard. NLP is probably same point where cv was 5 years ago so we start seeing very good NLP models.

<comment><USER3> Definitely. Even as an engineer working on CV 10 years ago, the hard part wasn't object detection but rather network bandwidth to stream incredible amounts of data and processing it in real-time.

Spoke to an experienced engineer who used to lead NLP at MSFT and same comment. NLP models are already fantastic and it isn't very hard to build a smart chatbot. The implementations these days are just very poor because they are not well thought out from a user perspective.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> I see this as more of an organizational challenge than a technology challenge.

Getting ML models into production isn't particularly hard... if you put an engineering team on it that know how to write automated release procedures, design architecture that can scale and build robust APIs to surface the data.

But in many companies the engineers with those operations-level skills and the researchers who work on machine learning live completely separate lives. And then the researchers are expected to deploy and scale their models to production!

That's not to say that this organizational problem cannot be solved with technology/entrepreneurship. If a company can afford it it's likely much cheaper to pay an external company to solve your "ML in production" problems than to re-design your organization such that you equip your internal ML teams with the skills they need to go to prod.

<comment><USER7> I disagree. It's not about getting the data where it needs to be. It's about data version control at a very fine level with very large datasets (in a way that is efficient). It's about detecting changes in model results base don changes in data. It's about tracking provenance of data in the datasets. It's about potentially controlled access to the data (eg allowing models to use health care data without actually knowing the underlying data). It's about detecting bias in datasets over time.

It's actually quite complex, which is why generally speaking very few people do anything like this. I am unaware of any general solution to this problem, either in industry or academia.

<comment><USER3> You've brought up a lot of very interesting points that we're actually looking to solve regarding data distribution changes, data version control and reproducibility, privacy guards and bias detections with our startup Orchestra ([https://orchestrahq.com](https://orchestrahq.com)).

Would love to chat if you have further thoughts around the subject - there's a ton of problems we're looking to tackle in the space and would be good to get input.

</comment></comment></comment><comment><USER4> Fig 4 looks like it's derived from Hidden Technical Debt in Machine Learning (2015).

[https://papers.nips.cc/paper/5656-hidden-technical-debt- in-m...](https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine- learning-systems.pdf)

As someone else says in this comment thread, this is very much an organizational problem, and cannot be viewed as just a technology problem.

The common behavior of individuals and teams is the pursuit of solutions that solve problems for them. Problems here with ML, and as we've seen with "Data Science," along with other magic technologies is that having an appreciation for the domain or context goes a long way. Being familiar with entire process, or "pipeline," is valuable, and role/functional silos often lead the problems people experience.

For some classes of machine learning problems and associated data, sourcing solutions from vendors can work, but as with any tools you can procure, you need the right people to use them appropriately. This also applies to "DevOps" which is used for comparison in the blog post.

\--> DevOps example -- the philosophy seems to be about having software developers also share build/release and infrastructure responsibility. But some organizations have made "DevOps" teams to silo build/release and infrastructure work... they ended up renaming what used to be called their Build/Release or SysAdmin teams. Siloing things to be "someone else's" problem doesn't result in the major transformations that are needed.

Now imagine what happens if we substitute MLDevOps for DevOps above.

I'll continue to say "The Role of a Data Engineer on a Team is Complementary and Defined By The Tasks That Others Don't (Want To) Do (Well)"

<comment><USER8> > The Role of a Data Engineer on a Team is Complementary and Defined By The > Tasks That Others Don't (Want To) Do (Well)

Those types of tasks are also often not recognized or rewarded by management, despite being a hugely critical part of the system. I believe the incorrect hiring of scientists who are often strong in terms of core theory or number of papers published but have no clue about building real production ML systems is a huge organizational problem, often causing ML teams to fail to deliver any real value.

</comment></comment><comment><USER9> Wow, I probably have seen 10 of these kind of companies over the past few months. Personally I believe (and hope) the winners in this space are going to be modular open-source companies/products as opposed to the "all-in-one enterprise solutions"

<comment><USER10> Could you please mention what are the other solutions you've got to see in this space?

<comment><USER11> [https://dolthub.com](https://dolthub.com) is the cool kid right now. There is pacaderm, git lfs, IPFS.

Really what we need is version control for data, it's not just an ML data problem. It's a little different though, because you would like to move computation to data, rather than the other way around

<comment><USER12> The utility of version controling production-sized (not sample training data) data (as opposed to code) is something I've having trouble grasping unless I'm missing something here -- and I may be, so please enlighten me.

It seems to me to be able to time-travel in data you almost need to store the Write-Ahead Log of database transactions and be able to replay that. Debezium captures the CDC information, but it's a infrastructure level tool rather than a version control tool.

In data science, most time-travel issues are worked around using bitemporal data modeling: which is a fancy way of saying "add a separate timestamp column to the table to record when the data was written". Then you can roll things back to any ETL point in a performant fashion. This is particularly useful for debugging recursive algorithms that get retrained every day.

But these are infrastructure level approaches. I'm not sure that it's a problem for a version control tool.

<comment><USER13> I worry about retraining every day. Isn't that a flag that says "It hasn't learned a thing and actually I'm just improving my backfitting score"?

<comment><USER14> as long as it works well on out of sample data at deployment time, it is okay.

Until some major data drift happens, but you would notoce it anyways

<comment><USER13> Honestly, I've heard people in Vegas tell me the same about their strategies vs. slots. Genuinely, if you have made money from this - well done, take it out now, congratulate yourself. If you haven't...

</comment></comment></comment></comment></comment></comment></comment><comment><USER15> I was the main data science engineer at one of my previous companies. We used tools like airflow for running python scripts to import data, clean/transform it, train models, and even test various models against datasets. We also used Azure for similar things.

It's easy to do "dev ops" for machine learning. Basically, just automate everything and implement gatekeeping mechanisms along with active monitoring.

It's true, though. I had to cobble together a lot of custom things at the time, but it wasn't that hard to do.

<comment><USER16> I'm the CTO at a data science company, and this has been my experience too. I've been lucky enough to have quite a few engineers go from zero practical experience to being able to train and deploy complex ml solutions, and the most successful solutions have always involved a combination of just a couple of tools: \- airflow and/or celery for running data extraction and transformation jobs \- pandas and numpy for data wrangling \- sklearn, xgboost, lightgbm, pytorch or tensorflow for training/inference \- flask or Django to serve results

It's a handful of technologies, but they're (generally) mature, battle tested, and well documented.

<comment><USER2> Generally true. Though I will say that in larger orgs, you will occasionally get someone doing some ML they read a paper on that's not well supported by major tooling. I mean it's the same trend chasing you see in engineering...

</comment></comment></comment><comment><USER17> Good god it's hard to do this at a non-tech company. MLOps would be great, but we don't really have "Ops," just IT, since our main business is not software. And we don't have Dev either, so we don't have anyone to really emulate on the inside. Our data scientists are foremost analysts who can write some Python, they don't know OO or memory optimizations or anything. They've never used a bash prompt or know what one is. Management thought we could orchestrate this huge waterfall schedule for a project and now it's falling apart as we open each new box of surprises...

<comment><USER18> If you don't have a dev, how are you collecting any data to begin with?

</comment></comment><comment><USER19> I 'll disagree with most comments that it's mainly an organizational problem. Creating tooling for things like:

\- managing different data sources

\- versioning data

\- monitoring how new data affects the model

\- testing that certain SLAs are met before new features are deployed

\- ability to rollback

\- data & model quality monitoring

is technically challenging.

Obviously there are engineers that will quickly hack something together and will falsely think that they have a good enough MLOps solution. I have been part of such teams.

Most companies are not Google, Facebook, or Uber. The large ones very often don't have the know-how to create a robust technical solution around this process, and even if they do it can take them years and the smaller ones lack both the resources and technical expertise.

I'm always looking for new ideas that can become successful business and when I saw the Uber Michelangelo here on HN a few years ago, I was thinking that selling similar tooling to other companies, had great potential. Seems that the right team to create that company was the one the built Michelangelo itself :)

</comment><comment><USER20> I really find it difficult to put into words just how little I care to pay for a web ui so I can "manage" my data.

Data pipelines are a real problem though, and I'm very interested in what startups do with this space.

<comment><USER21> > Data pipelines are a real problem though

Can you please elaborate more, thanks.

<comment><USER22> Its not trivial to create and manage Data Pipelines if you care about scale, serving a wide range of inputs and outputs, or making this data easy to surface and spread throughout your org (i.e. making it actually useful to regular people).

"Static ETL" like running the same database load every day at 1:00am isn't a super challenging problem. Doing it across many tables with complex transformations and multiple steps easily can be. You really have to consider reliability, processing speed, failure methods and other problems that dont really arise until you hit a certain scale.

There's also the issue of what people want out of a Pipeline that's changing. If you want people to be ""data driven"", then that means they need easy access to potentially all of your company's data on an ad hoc basis. So now your boring ETL 1 am pipeline isnt really serving any of these new usecases.

How do you create flexible pipelines that can be created from any dataset on an ad hoc basis? This is where tools like Airflow or Prefect come in. Creating a platform that can create these types of Pipelines is a real problem.

And before you even ask yourself _how_ to process this data, you need to also ask _where_? If you want to do what I outlined above - making your data more accessible and easy to use - then you probably need to rework how you're storing your data. But Data Lakes (and others) are a whole topic in and of itself.

</comment></comment></comment><comment><USER23> "We need fewer data scientists and more data janitors" \- anon

</comment><comment><USER24> Honest question (though I suppose the clickthroughs to the comments are likely to be a biased sample): is "getting to prod" really the gatekeeper/bottleneck for most ML? I would have thought "a model that works" is much harder, especially given how hyped the field is and how many people are trying to tackle problems that are I'll suited to the current batch of ML techniques.

Unless the issue here is data collection in prod to start training your model.

<comment><USER18> >is "getting to prod" really the gatekeeper/bottleneck for most ML?

The most common bottleneck is collecting the right data. It can take years, or even a task force just to get the right data before the data scientist can begin.

>I would have thought "a model that works" is much harder

It depends how experienced the data scientist is. Early on into a project a data scientist can do a feasibility assessment. They should identify what is possible, and how possible. Sometimes some data science projects are heavy on the research side where where it can take 2 weeks to 3 months to figure out if something is possible. Sometimes the feasibility assessment ends up being incorrect and a goal is shown to be impossible.

Once research is done it usually takes 4 weeks to 6 months for a data scientist to build a model. The upper bound is rare and happens because of recursive refinement to increase accuracy, trying to get every last drop out of what is possible.

In contrast it can take months to years for the company to begin to collect the right data for a data scientist to be able to begin to do what benefits the company. Sometimes crowd source projects need to be created just to collect the required data. It then takes an average of 3 to 6 months for productionization if there is clever feature engineering in the model. Note: When I say productionization, I mean all the way to the end customer, so setting up and maintaining pipelines, frontend devs updating websites to add the service, and whatever else is necessary. There is more work involved on the production side, but it can be split up to multiple engineers.

<comment><USER24> That's exactly what I suspected (I work at a dl/ml hardware co), thanks!

</comment></comment></comment><comment><USER25> Managing data is not an IT job. Data is just unformatted information, and should be managed and governed by those who are trained in Information Management: Modern day Librarians.

IT own the platform, and the software. They should never own the data as well.

<comment><USER26> I agree with this, and I think data librarian is a role that any "data-driven" company needs. IMO it makes a lot of sense for data scientists to fill that role, but I think that's an issue for many. Data scientists may think being a librarian and organizing the knowledge base is beneath them, or maybe management thinks it's beneath them. Execs tend to not care about the state of knowledge infrastructure as long as their reports get to them when they expect.

</comment></comment><comment><USER27> This is close to home. Our approach to DevOps for ML Data is to use S3 as the git core and build immutable datasets and models on top of S3 object versioning. I wrote the piece below on "Versioning data and models for faster iteration in ML" earlier this year. The key idea is for every model iteration to be a pure function F(code, environment, data). Ideas welcome: [https://medium.com/pytorch/how-to-iterate-faster-in- machine-...](https://medium.com/pytorch/how-to-iterate-faster-in-machine- learning-by-versioning-data-and-models-featuring-detectron2-4fd2f9338df5)

</comment><comment><USER28> I'm reminded of: [https://blog.acolyer.org/2019/06/03/ease-ml- ci/](https://blog.acolyer.org/2019/06/03/ease-ml-ci/)

</comment><comment><USER29> This startup is trying to build the next GitHub for ML Data: [https://dagshub.com/](https://dagshub.com/)

<comment><USER30> This seems pretty cool.

</comment></comment><comment><USER31> Data is hard to automate and standardized pipelines and processes are really helpful. This is interesting.

</comment><comment><USER32> Isn't this DataOps?

<comment><USER33> I was thinking the same, it could be that the OP isn't familiar with the term yet.

</comment></comment><comment><USER34> Is this not just a really long advertisement?

</comment><comment><USER35> Rule number one: define your terms as you introduce them. On and on about ML. But what is it?

I had to search to see it was Machine Learning.

How hard is it to define it the first time you use it?

I can bet lots of people were scratching their heads but didn't bother to look it up or continue reading...

<comment><USER36> Genuinely curious, did you think "ML" stood for anything else? My day to day work is not machine learning but if I ever see ML, "machine learning" is the first thing I think of.

<comment><USER37> There is also the ML family of programming languages: [https://en.wikipedia.org/wiki/ML_(programming_language)](https://en.wikipedia.org/wiki/ML_\(programming_language\))</comment></comment></comment>


Microsoft OSS image and video annotation for DL now bundled in Azure DSVM image - <AUTHOR> https://github.com/CatalystCode/VoTT/ <comment><USER2> Cool!</comment>


The infinitely profitable program - <AUTHOR> http://peetm.com/blog/?p=55

<comment><USER2> The headline is pure link-bait, but coaxing functionality out of a zero-byte program _is_ a clever hack, so kudos to the author for that.</comment>


Why Germans Love Their Health Insurance - <AUTHOR> https://global.handelsblatt.com/politics/why-germans-love-their-health-insurance-763779/amp <comment><USER2> If you want to link AMP page (blasphemy :-) ) please link the google CDN page. It scrolls better and fewer bits need to be transmitted. Thanks.

[https://global-handelsblatt- com.cdn.ampproject.org/c/s/globa...](https://global-handelsblatt- com.cdn.ampproject.org/c/s/global.handelsblatt.com/politics/why-germans-love- their-health-insurance-763779/amp)

</comment><comment><USER3> I think it is a good rule that if our country is having a problem, we should look at how other countries handle the issue and see if they are doing it better. It seems pretty clear just about every other industrialized nation handles health care better than the US.</comment>


Stories and Tips: 50+ Interviews with Facebook, Twitter, Amazon and Others - <AUTHOR> http://blog.robertelder.org/50-interviews-with-facebook-twitter-amazon-others/ <comment><AUTHOR> Hi, I'm the author. If any of you have experience interviewing candidates for software positions, and you disagree with any of the advice I've provided in this post, I would be happy to hear it.</comment>


Project Wren - <AUTHOR> https://projectwren.com <comment><USER2> This is great! Individuals taking responsibility for their impact is a great way to convince other entities from doing the same.</comment>


FOUT is back in Chrome 50 - <AUTHOR> https://productforums.google.com/forum/#!topic/chrome/kBzYJZV-qMM;context-place=topicsearchin/chrome/category$3Amac%7Csort:relevance%7Cspell:false <comment><USER3> For those of us who aren't in the know, apparently FOUT stands for "Flash of Unstyled Text": [https://css-tricks.com/fout-foit-foft/](https://css- tricks.com/fout-foit-foft/)

<comment><AUTHOR> Yeah... I probably should have commented that earlier.

If anyone wants to see an example of FOUT, you might be able to on my website with Chrome 50. I am currently experiencing this and then I found this bug report. Reload a few times if you don't see it at first.

[http://jakedeichert.com](http://jakedeichert.com)

edit: FOUT usually means a fallback font is briefly shown before flickering and rendering a loaded web font.

edit: As people have pointed out below, my situation likely has nothing to do with this possible bug. My website layout is very lightweight and Chrome is rendering it probably faster than the other browsers I tried which explains why I'm seeing FOUT more often in it. And yes, I'm loading my CSS asynchronously which is the main reason behind my FOUT.

<comment><USER4> You are using loadCSS to load the font from google fonts. loadCSS is designed to asynchronously load a stylesheet. Given an asynchronous load of the font, I'm not surprised you're seeing occasional FOUT. You should adopt the recommended <link> tag technique for your webfonts, unless you're willing to follow the more advanced font performance / FOUT guidance from Zach Leatherman.

<comment><AUTHOR> I just switched from using a blocking link tag to loadCSS today which indeed loads it asynchronously. I'd say 1/5 times I am not seeing FOUT.

However I'm not seeing FOUT on Safari for Mac and iOS and Firefox for Mac at the moment.

*edit: I definitely understand that you're right Paul. If the website loads slow, FOUT is expected. However, I'm seeing it more than what I expected to with loadCSS combined with this bug report I found. So I'm kind of leaning toward the idea that it is a bug in Chrome 50. If not, I'm wrong and it's only my loadCSS causing it.

<comment><USER5> When you load web fonts asynchronously, you're allowing the browser to render the page without them. Since your site is very light, Chrome is able to perform the first render practically immediately (like within 100ms), so the FOUT is very noticeable.

Btw, once the CSS `font-display` property becomes available, you'll be able to control web font rendering to some extent, but as of now, this type of FOUT is the best approach, since, if you went with FOIT, you'd risk up to 3 seconds of whitespace in case your fonts experienced network congestion.

<comment><AUTHOR> Okay this is a logical explanation. I am probably wrong then.

</comment></comment></comment></comment></comment></comment><comment><USER5> Better FOUT than FOIT. When I have to look at large areas of whitespace while the page loads, I get irritated. Just let me read the text, you stupid website! :P

<comment><USER6> It seems there is no good answer (short of not using custom fonts) - <USER2> FOIT irritates me as well, the inevitable repositioning of everything on the page when the custom font loads is also deeply infuriating.

</comment></comment><comment><USER4> Here's the browser view of this brief interaction: [http://i.imgur.com/IWFEKGt.png](http://i.imgur.com/IWFEKGt.png) [0]

Basically, the stylesheet finished transferring and the browser followed up with laying out the page. Some data:uri requests were in flight and when they finished (200ms later) the page was laid out again and now applied the fonts. Typekit has historically detect in JavaScript when a font loads because they like to ensure the text stays invisible (across browsers) while the page hasn't finished loading the fonts. But in this case, it looks like there's a problem with either Typekit not finding the load "events"[1] of these fonts, or Chrome mishandling data uri fonts in some way. We've filed [https://crbug.com/607788](https://crbug.com/607788) to investigate further.

Loading fonts well is hard. Here's some advice on doing it well: [https://gist.github.com/paulirish/d511793a1c3b74b31460](https://gist.github.com/paulirish/d511793a1c3b74b31460)

[0] You can explore the same recording here: [https://frontend.chrome- dev.tools/serve_rev/@db7757b63f29696...](https://frontend.chrome- dev.tools/serve_rev/@db7757b63f29696b9b465949f9704a0d9d149f2b/inspector.html?loadTimelineFromURL=https://dl.dropboxusercontent.com/u/39519/fout- trace.json&experiments=true) (Chrome only. Check the screenshots checkbox)

[1] [https://drafts.csswg.org/css-font-loading/](https://drafts.csswg.org/css- font-loading/) was standardized for these purposes. It's available in Chrome, Opera and Firefox to reliably detect when a font has finished loading.

[‽] As mentioned later by hatsix, the font-display spec introduces a CSS property to provide control over FOUT/FOIT to the site: [https://tabatkins.github.io/specs/css-font- display/](https://tabatkins.github.io/specs/css-font-display/) It will definitely bring a lot of sanity to a something that's been a problem since at least 2009 :) [http://www.paulirish.com/2009/fighting-the-font-face- fout/](http://www.paulirish.com/2009/fighting-the-font-face-fout/)

<comment><USER7> I've been hearing reports of increased FOUT lately and noticed it myself. I just so happened to use someone else's computer this evening tonight and did not see the issue. Turns out that I am seeing FOUT on every page load with Typekit fonts in Chrome 50, but not Chrome 49 or Firefox.

Are you getting the results from your tests in 49, 50, or both? It seems like Chrome 50 was the catalyst for the issue.

<comment><USER4> I'm digging in a little more and seeing some odd behavior, but it's unclear if this stuff is new to 50. Star this ticket to follow along: [https://crbug.com/607788](https://crbug.com/607788)

</comment></comment></comment><comment><USER8> I believe this is intentional, and part of the CSS-Font-Display spec.

[https://tabatkins.github.io/specs/css-font- display/](https://tabatkins.github.io/specs/css-font-display/)

The previous behavior (Flash of Invisible Text) was considered an anti- pattern, as it prevented text from being read on slower connections.

The new spec allows CSS to specify the behavior.

<comment><AUTHOR> At first glance, that document states that Chrome (and the other browsers) have a 3 second timeout before showing the fallback font.

For me, I'm seeing a fallback in less than 100ms before flickering to my web font. So the timeout doesn't seem to be working right now maybe?

</comment></comment><comment><USER9> Chrome 50 also introduced some overly eager GC bugs [1] and throws weird new WebGL warnings [2].

Basically all our WebGL projects are affected and clients already started to complain that textures aren't showing or that preloaders are stuck. Hopefully the fixed release gets shipped soon.

[1] [https://bugs.chromium.org/p/chromium/issues/detail?id=604844](https://bugs.chromium.org/p/chromium/issues/detail?id=604844) [2] [http://stackoverflow.com/questions/36876232/three-js- there-i...](http://stackoverflow.com/questions/36876232/three-js-there-is-no- texture-bound-to-the-unit-0-error)

<comment><USER10> Interesting, I encountered problems with image decoding. Image.src = dataURL stopped being reliable. I suspect that the code was refactored to support createImageBitmap ([https://developers.google.com/web/updates/2016/03/createimag...](https://developers.google.com/web/updates/2016/03/createimagebitmap- in-chrome-50)), and some kind of race condition or something causes some decoded images to be correctly sized but blank.

We fixed this by making users that support it use createImageBitmap, but I've also seen some Windows Chrome crashes that I suspect are from that call.

EDIT1: Also, thanks for the tip about the WebGL error. I've been seeing that too.

EDIT2: D'oh it looks like you also spotted the reason for the src attribute as well. I should have clicked the link before responding. ;D

</comment></comment><comment><USER11> I know it's "bad" but my solution to this is just to stuff everything in the head tag.

Instead of @import I just base64 encode things as data URIs. One big fat CSS file with everything the browser needs.

Get the content loaded once and the cache generally takes care of making it fast enough.

<comment><USER12> The problem here is that you have to use Typekit because you don't actually own the fonts.

<comment><USER11> I'm not familiar with Typekit, but from the link it applies more broadly.

> I've discovered a bug when using the Webfont Loader for Google Fonts or > Typekit.

I've found most of the Google Web Fonts permit transformation and embedding. You just need to read the licenses closely.

</comment></comment></comment><comment><USER13> If they could review webfont rendering in Chrome on Windows overall I would be so happy - it really is much worse than FF, IE and Edge.

<comment><USER14> I have difficulty viewing very thin text, and unfortunately the way Chrome renders common fonts like 10-12pt Arial and Verdana makes me strain my eyes. Firefox and Edge work okay for me.

</comment></comment><comment><USER15> A good example of FOUT: [http://recordit.co/vsAZQc5ihD](http://recordit.co/vsAZQc5ihD)

</comment><comment><USER16> It'd be nice if unstyled text wouldn't be as distracting, or rather, that the transition from fallback font to desired font wouldn't look bad.

What if the CSS could specify basic font metrics which the fallback would be shaped to fit until the actual font loaded?

</comment><comment><USER17> Got really excited when I thought this was FAUT (Find As U Type) like Firefox has</comment>


Building a $5k ML Workstation with Tiitan RTX and Ryzen ThreadRipper [video] - <AUTHOR> https://www.youtube.com/watch?v=iahQOQZVdog <comment><USER2> If you go with air cooling on a threadripper, I suggest going with a Noctua cooler instead of Dark Rock. Dark Rock extended the size of the heat plate to match the TR cpu size, but they didn't cover it with heat pipes, Noctua did. Cooling performance really suffers on the 3990X because there are chiplets on the edge of the cpu. the 32 and 24 core models it may not matter so much.

See: [https://www.kitguru.net/components/cooling/luke- hill/threadr...](https://www.kitguru.net/components/cooling/luke- hill/threadripper-3990x-cpu-cooling-comparison-how-to-tame-the-beast/11/)

On non threadripper cpus I actually like Dark Rock better. Cooling is the same as Noctua but it looks cooler and was quieter for me.

<comment><USER3> Also if you go air cooling and are confident with your abilities, consider delidding to drop temps. more (5 to 20C). If you go air cooling I would assume it is on the basis of long term stability, so don't use liquid metal either. Also invest in a nice PSU (gold minimum) with your peak load pulling only 75% of the rated max wattage

Edit: Like most things look at components real-world testing figures, in this case, wattage, as opposed to TDP when planning

<comment><USER4> Ryzen CPUs are soldered, delidding has minimal impact.

<comment><USER5> To be honest I can't recall any AMD CPUs that weren't soldered... apart from those that didn't have an IHS in the first place ;)

Intel was just like "we can save 4 $ BOM cost on a 500 $ part there" with their 4th-9th generation of CPUs. No thermal headroom? No problem.

<comment><USER6> 0.8%? Could be worth billions.

</comment></comment></comment></comment></comment><comment><USER7> Good choice on the 24 GB Titan RTX (so you can do at least batch size = 1 for Bert-Large). Not sure if that's the reason it was chosen though to be honest. If you want to do convnets only then you would do better with NVLinke'd 2080 Tis.

Secondarily, I would suggest that you guys not use windows but instead Ubuntu 18.04 or 20.04 LTS and just install Lambda Stack ([https://lambdalabs.com/lambda-stack-deep-learning- software](https://lambdalabs.com/lambda-stack-deep-learning-software)). It's a debian PPA that we maintain at Lambda to keep all of your deep learning drivers, CUDA, CuDNN, TensorFlow, and PyTorch up to date with just apt. It's free!

<comment><USER8> Interesting! Is Lambda Stack going to work on 20.04? The link mentions only 16.04 and 18.04

<comment><USER9> I've asked Lambda Labs 2-3 times if they are going to do an Ubuntu 20.04 stack, but I have not had a reply yet.

<comment><USER7> It works for 20.04 LTS and we've updated the website to reflect this, thanks for bringing it to my attention!

</comment></comment></comment></comment><comment><USER10> If you only have $1K or less to spend, and you don't already have a sufficient PC that you can upgrade with a big GPU...

A non-Threadripper Ryzen, a big GPU, and a big PSU in a big case will go most of the way for most people, and leave you with an easy incremental upgrade path for bigger GPUs (or maybe add a second GPU).

Slightly dated info for my current ML server, which is nicely quiet in my living room, thanks to Noctua: [https://www.neilvandyke.org/machine- learning/](https://www.neilvandyke.org/machine-learning/)

(Side note that's not in that page: I like to use older ThinkPads with transplanted vintage keyboards for my workstations, so I needed to make a separate box for the GPU. But life would be easiler, with a lot less juggling complexity, if I simply had the big GPU in my laptop rather.)

<comment><USER11> I recently bought a P73 thinkpad specced out like this, and it's great.

However, putting a GPU and lots of RAM into a laptop makes it very, very heavy so it's worth thinking about if that's acceptable for you.

<comment><USER12> I just bought a specced out 1950x on a x399 with 64gb ram (plus the box, power, etc) for about $800, which I think is the fair price for 3 year old hardware. It needs a GPU, but for my usecase it's perfect.

I'm also in Europe, so prices are higher.

<comment><USER11> Yeah, me too. I spent a lot of money on this machine, but amortised over about five years (which is how long my last one lasted), it's acceptable (that's what I keep telling myself anyway).

</comment></comment></comment></comment><comment><USER13> Here's my recommendation (I've built several such machines for my own use):

1\. Go with a 1600W PSU from EVGA or Corsair. Other brands are hit or miss if you ever need very high current on the rails. This will manifest in your machine suddenly powering off when all 4 GPUs are hit with data at once (as is typical at the start of an epoch)

2\. Use a mobo with evenly spaced GPU slots, such as ASRock TRX40 Creator. That way you can install 4 GPUs eventually and use that 1600W PSU. You also get 10GbE for distributed training, which is nice.

3\. Don't waste money on Titan RTX, get 2x2080ti's instead. Then after a while get two more. Buy blower cards which blow hot air _out_ of the case.

4\. Use an extension cable to install SSD and do not install it under a GPU - it'll die eventually due to overheating.

5\. Air cooling is fine

6\. If you have more than 2 GPUs learn how to adjust fan speeds on GPUs. Crank them to 85-100% while training to prevent throttling.

</comment><comment><USER14> Here is their list:

PCPartPicker Part List: [https://pcpartpicker.com/list/Jhyzcq](https://pcpartpicker.com/list/Jhyzcq)

CPU: AMD Threadripper 3960X 3.8 GHz 24-Core Processor ($1348.00 @ Amazon)

CPU Cooler: be quiet! Dark Rock Pro TR4 59.5 CFM CPU Cooler ($89.90 @ Amazon)

Motherboard: MSI TRX40 PRO WIFI ATX sTRX4 Motherboard ($389.99 @ B&H)

Memory: Corsair Vengeance RGB Pro 64 GB (4 x 16 GB) DDR4-3200 CL16 Memory ($329.99 @ Amazon)

Storage: Sabrent Rocket 4.0 2 TB M.2-2280 NVME Solid State Drive ($399.98 @ Amazon)

Video Card: NVIDIA TITAN RTX 24 GB Video Card ($2499.99 @ Newegg)

Case: Corsair Crystal 570X RGB ATX Mid Tower Case ($179.99 @ B&H)

Power Supply: Corsair RMx 1000 W 80+ Gold Certified Fully Modular ATX Power Supply ($204.99 @ Best Buy)

Case Fan: Corsair LL120RGB LED 43.25 CFM 120 mm Fans 3-Pack ($120.99 @ Best Buy)

Total: $5563.82

Prices include shipping, taxes, and discounts when available

Generated by PCPartPicker 2020-07-15 11:13 EDT-0400

<comment><USER15> You can spend half of the specified costs on every single one of the listed components with zero impact on your ML work productivity. $330 for 64gb of ram, really?

<comment><USER16> That is high end RAM binned at 3200Mhz. Consumer RAM is mostly 2133/2400 and with servers often using 2666. RAM at 3200 (PC4-25600) gives you about 25% more peak bandwidth than RAM at 2400 (PC4-19200) and about 16% more than 2666 (PC4-21333)

<comment><USER17> Not true, 3200 CL16 can be nearly as cheap as slower RAM nowadays, with multiple brands at sub-$60 per 16GB stick[1]. OP is paying extra for RGB, as another commenter points out.

[1] [https://pcpartpicker.com/products/memory/#sort=price&U=4&Z=1...](https://pcpartpicker.com/products/memory/#sort=price&U=4&Z=16384001&S=3200,5000&L=30,160)

</comment></comment></comment></comment><comment><USER18> It's worth noting that if your ML work is entirely CUDA based (as often happens), you likely won't benefit from a Threadripper CPU. Downgrading to a Ryzen 9 or even 7 will reduce costs by a good bit. The savings can be pocketed or put toward a second Titan RTX + NVLink (48Gb usable VRAM).

<comment><USER19> Should note (from someone who has a few of these systems at my lab) unfortunately the consumer RTX cards don't do memory pooling. This means that although NVLINK is good for inter-GPU comms it doesn't actually allow you to run giant models that need the entire 48GB of memory for a backwards pass (treat the combined cards as "one card"). Not typically a problem for most people but worth mentioning

<comment><USER18> From [https://www.nvidia.com/en-us/deep-learning- ai/products/titan...](https://www.nvidia.com/en-us/deep-learning- ai/products/titan-rtx/):

"NVIDIA TITAN RTX NVLink Bridge

The TITAN RTX NVLink™ bridge connects two TITAN RTX cards together over a 100 GB/s interface. The result is an effective doubling of memory capacity to 48 GB, so that you can train neural networks faster, process even larger datasets, and work with some of the biggest rendering models."

<comment><USER19> Yeah you're not wrong, but it's a bit misleading. This allows you to run faster, but it does it by allowing you to use a larger batch size (arguably not best practice but your mileage will vary). Memory pooling is a bit different in that you can treat the combined cards as a single card from TF/pytorch.

<comment><USER20> But batch size is prob least problem since you can do data parallelism (send half batch to each gpu, combine on cpu).

I think only model bigger than gpu mem is where you really wish for nvlink on v100s.

</comment></comment></comment></comment></comment><comment><USER21> I'm a bit concerned the build uses a Gold Certified power supply unit?

Even for cheaper builds for non-ML workstations I would still only use Platinum and nothing less. I've been told Titanium is excessive but I mean I leave these things on for a while and power is expensive.

For the DIY enthusiast or the WFH researcher, also the amount of heat involved can be a considerable cost in cooling or utility cost which varies substantially by floor of a building. It's probably not good, but not that bad to aircool this many GPUs as I've done in the past but it definitely means I'm paying a lot for A/C in the summer but almost nothing in the winter.

I think Smerity even said he heated his small bedroom through the San Francisco winter off of one GPU while researching YOLO.

Point: These things get hot. They require a lot of electricity. You should be concerned about a good PSU even for smaller builds. My energy cost for a 6GPU rig ran me about 1/3 of my total rent for a small apartment. That's electricity BEFORE I calculated my A/C bill which was separate and also substantial. My landlord hates me because I initially talked him into including it with my rent.

All in all, it still makes sense to keep investing in local workstations, on- premises builds. No security concerns about a cloud, no futzing around with integrated notebooks, you own it you control it, and the price point up front is extremely attractive compared to base rates for cloud computing even on specialized hardware like a TPU.

The numbers I come up with for batches still have a wide gap of several thousand USD most of the time, and then there's how much time it takes and how likely their service breaks.

So kudos for the person who put in the effort to put this together and share. Any and all efforts towards making ML/DS affordable and DIY rises the tide for all boats.

Question to the audience: Does anyone build GPU rigs like this for cryptocurrency anymore? I was only able to build a workstation once the price for GPU cards crashed.

<comment><USER2> The efficiency delta between Gold and Titanium is really small. Optimizing that for heat reasons would be optimizing less than 1% of total heat output. And most cases keep the power supply thermally separate from the rest of the stuff anyway.

This guy has a very oversized Gold power supply, the efficiency would be ~92% with gold vs 94% with platinum. Maybe a smaller titanium one would be a better overall choice I guess.

<comment><USER21> Generally I think going jumbo is good for cooling, because honestly the weight for this build probably doesn't matter as I imagine it's not getting transported often.

Conversely, this is not a build for overclocking per se. However I think it's a safe assumption we are running at capacity of over 90% for multiple days or weeks even. If batches run into over a month, probably time to get a server rack instead?

It is worthwhile to note you're not going to save any money in efficiency for computation you don't use.

<comment><USER22> There's no way this build draws 900W.

</comment></comment></comment></comment><comment><USER23> What do people use ML for these days? I do computer programming, and I have done some work with video encoding, but this just seems like a huge investment money wise. So I am curious what use it is.

For my needs the most intensive thing I do is compile some large programs or encode some large video, which you can get a computer for that for like $800.

<comment><USER24> ML is typically used to find correlations in data. If something happens over and over again, there is a high chance it will happen again. Having such an algorithm that has identified this correlation allows it to identify when it will happen again. This allows for what is called predictive analytics.

This can be as simple as identifying when a customer will end their service with a business, as there might be a pattern before previous customers have left, predicting when new customers are going to leave, and giving them a coupon or similar right before they would otherwise leave. This problem is called customer churn.

It can be as complex as identifying when hardware will fail ahead of time, or even bio-ware. For example, I did a project that predicted when people were falling into depression before they could tell they were with a high accuracy rate. I also predicted other future medical issues ahead of time, like the probability an elderly person is going to fall over within the next handful of days.

On the business side there are a lot of use cases for ML, but it falls more into analytics than engineering, as it's about predictive insight.

</comment></comment><comment><USER25> Now is a particularly bad time to build a rig, since new NVIDIA cards are launching in a couple months. The value of a used 2080Ti (Turing) will tank, because Ampere cards will be available with similar performance for half the price.

<comment><USER26> Agreed. I need to update my gaming rig. Waiting for \- Ryzen 3 \- next round of GPUs from both vendors (although ML folks likely stay nVidia of course) \- with luck, a better pcie4 SSD will be out by then too.

I really wouldn't build one now unless I had to.

</comment></comment><comment><USER27> Interesting video, thanks for sharing. Just curious if you have one with tests or benchmarks for the completed build and/or temps at high loads? Would be cool to see :)

<comment><AUTHOR> Those will be coming!

</comment></comment><comment><USER28> I think that tape-like piece that you removed from the SSD compartment is actually the thermal pad that makes contact between the SSD and the MSI heat sink cover so you may actually want that!

</comment><comment><USER29> Thanks for the video! Could you comment on the differences between the Titan RTX and the V100? I am a bit confused because the V100 is significantly more expensive ($7k on Amazon even for the 16GB version) and has a slower clock speed, yet it is the standard in ML research papers. I see that it has ~10% more CUDA cores, but it doesn't seem like this would warrant a 3x price increase.

<comment><USER30> The other commenter mentioned the "pro-level" tradeoffs, but there's something else too: Nvidia's licensing won't let you use GeForce cards in the cloud. If you're building a datacenter, you have to use the Teslas.

</comment></comment><comment><USER15> 2x 2080ti would be faster than titan rtx, provide the same amount of memory, and would be cheaper.

<comment><USER19> Unfortunately multi-GPU training doesn't scale linearly yet [1] so it's often a better call to get a larger card then two smaller ones, at least for the single-model case.

[1] [https://github.com/keras- team/keras/issues/9204](https://github.com/keras-team/keras/issues/9204)

<comment><USER31> (non-TF) Keras has notoriously bad multi-GPU support though (and was generally not well optimized. Case in point, the latest version just re-exports/forwards to tf.keras).

Looking at something like [https://lambdalabs.com/deep-learning/gpu- benchmarks](https://lambdalabs.com/deep-learning/gpu-benchmarks) or [https://github.com/tensorpack/benchmarks/tree/master/other-w...](https://github.com/tensorpack/benchmarks/tree/master/other- wrappers), multi-gpu scaling on 2080tis seems pretty darn close to linear. Plus, there are benefits to having more than one accelerator handy on a local workstation. For one, it's much easier to have multiple experiments running simultaneously or to run parallel training (e.g. hyperparameter search or RL episodes). Given that only the uber-expensive enterprise cards have proper virtualization/time sharing, trying this workflow on a Titan RTX will most likely be suboptimal unless you _always_ run models that can make use of most of the memory and compute (no RNNs, no Neural ODEs, etc.)

</comment></comment></comment><comment><USER32> I am curious about the opposite end of the spectrum. What is the smallest and cheapest self contained setup that can be a serviceble development box for someone doing ML / AI type work? Does not need to run the production load, but has to be capable enough to allow local development activity that is still representative enough.

So far the best I have identified is Intel NUC8 + nVidia GPU via Thunderbird. But it is still $1000 at least by the time you have it all together.

NB: I know lots of people will say, just do it with cloud, but I work in a setting where much of my data cannot be put in the cloud, and also where the cost structure of funding well allows for fixed capital expenditure but not variable cloud costs.

<comment><USER33> Just buy a case, motherboard, cpu, GPU, ram, psu, and build it. At the extreme low end you can buy a refurb Dell tower and drop in a new GPU.

</comment></comment><comment><USER34> With the size of air-coolers these days, and how they all have integrated heat pipes, I'm beginning to wonder if we've crossed the distinction barrier with liquid-coolers.

Holy moly is that a big heatsink.

<comment><USER35> Air cooling is wonderful. I didn't watch this video but I use a giant Noctua heatsink on my system (Intel 6950X, not quite as demanding as Threadripper but hardly a cold-running CPU) and I have zero regrets. I have used all-in-one watercoolers, and ... they just die after a few years. I have owned three and all three of them died in less than 2 years each. Each time, I very sadly woke up in the morning with the realization that I'm not going to be using my computer today. AIOs have not, in my experience, been quieter or yielded better temps than air cooling. So I am not sure it's worth it.

Obviously a homebuilt watercooling rig is going to be way better than AIOs, but it requires extensive maintenance. It's water. Stuff will grow in there. It evaporates. Joints loosen over time and could end up shooting water all over your $6000 workstation.

To me, it's not worth it. Threadripper requires a big heatsink. So be it.

<comment><USER36> Linus Tech Tips did a video in which they were unable to beat a high-end air cooled nuctua set-up with any sort of water cooling solution. And by unable to beat, I mean both temp-wise and sound-wise. Maybe, just maybe, you might be able to beat air cooling on one dimension with a very expensive custom loop, but it's totally not worth it.

Water cooled PCs do pretty look cool though.

<comment><USER37> Those air cooled units with the heat pipes are effectively heat pumps, and heat pumps are crazy efficient at cooling. They rely on evaporative cooling, which soaks up substantially more energy, so much so that it's possible to to get temperatures noticeably below ambient using the method.

Converting water to steam takes about (from the top of my head) 5x more energy than it takes to increase it by one degree. I imagine the coolant used in these heat sinks is at least this efficient, if not more, and is something that evaporates well below 100°C.

If heat sinks were like using cars and roads to transport people. A water cooled setup would be like expanding the size of the roads to increase the number of cars that can transport people. While those heat pipe coolers are like replacing cars with trains, so you can do a lot more the same amount of space.

<comment><USER38> > Converting water to steam takes about (from the top of my head) 5x more > energy than it takes to increase it by one degree.

I think this greatly underestimates the potential of evaporative cooling. The "specific heat of water" is 1 calorie/gram °C --- that is, one calorie can heat one gram of water by one degree Celsius if no phase change is involved. The "heat of vaporization of water" is more than 500 calorie/gram at 100°C. That is, the energy necessary to convert a given quantity of liquid water from just below boiling to steam is not 5x the energy necessary to raise that amount of water 1°C, but 500x! You are probably remembering that this is 5x the energy necessary to take liquid water all the way from 0°C (almost frozen) to 100°C (almost boiling).

</comment></comment></comment></comment></comment><comment><USER39> When he said training on Google colab took one day and on his computer took 20 mins, did he compare with google colab CPU? The difference seems too large.

</comment><comment><USER40> Being unfamiliar with ML work, when does it make sense to build one of these vs spinning up some instances on AWS or gcloud?

<comment><USER41> I think it really depends on how much you care about ML and how performant you actually need it to be. If you are a hobbyist or prototyping something speculatively for work, perhaps a cloud instance is prudent. If ML is your life's work, I'd probably consider throwing down for a proper rig so you don't get killed on cloud hosting fees.

</comment></comment><comment><USER42> Does anyone do a measure of how long it would take such a workstation to pay for itself (including some nominal amount of operational cost for electricity) compared to simply doing ML on AWS/Azure/GCP? Seems like such a metric could be a useful measure for comparing such machines.

<comment><USER43> A comparable workstation costs about a month of on-demand EC2 time or 3 months of spot instance time.

AWS GPU instances are really expensive.

The most cost effective imo is to build a workstation for development and then deploy to AWS spot if you need a cluster.

If you can't use a workstation for whatever reason, then use the new AWS feature to "stop" spot instances and use the spot instance as your workstation while being conscious of the high hourly cost and shutting it down when you're not working.

<comment><USER44> Azure ML/GPU instanced are also really expensive.

I did the maths recently and figured out I could put together a machine with a couple of 2080 Ti's and have it pay for itself in a couple of months.

I'm very seriously considering doing it, especially as I'm the only data scientist, if I had a team I'd be more in favour of going to the effort of setting up cloud-based training jobs etc

<comment><USER43> That's what my partners and I did. But we bought refurbished 1080 cards for about $300 each and Ryzen 9 hardware.

We're waiting for the 3000 series to come out which should be a large performance/dollar improvement over the current gen cards due to the smaller transistor size.

</comment></comment></comment></comment><comment><USER45> i have a threadripper 1920x with 2x2080ti.

When running cpuburn, i get around 65 tdie temp and with gpuburn, the upper card gets to around 86 and the lower one to 81.

i have right now a water cooler for the cpu, 3 inlet fans (bottom back) and 2 outlet fans through the water cooler on top. i was wondering what temperatures I should aim for and what an optimal fan configuration is. i have a couple of fans laying around.

the case is lian li O11 Air and the mobo is a taichi x399.

anyone any tips?

also I would want to use SLI. but then I would have to remove the fans on the gpu. Do I need then to water cool the gpu or what is the solution?

if anyone of the moddibg pros here can help that would be awesome :-)

</comment><comment><USER46> Please fix title Tiitan typo.

</comment><comment><USER47> A bigger box helps reduce cooling and power expenses. I built a ThreadRipper tower with 2080TI last year and used BeQuiet 900 for it with very nice results.

</comment><comment><USER48> Did not expect a video that wholesome.

</comment><comment><USER49> Did you consider NOT using NVidia? Both the capital cost and operational costs are huge compared to fairly high end Radeon cards (NO, I"m not an AMD employee - I merely choose based upon my personal budget)</comment>


The Elite Club That Rules the Diamond World Is Starting to Crack - <AUTHOR> https://www.bloomberg.com/news/articles/2019-07-29/the-elite-club-that-rules-the-diamond-world-is-starting-to-crack <comment><USER2> >It's a perfect storm: you have too much supply, not enough demand

My wife lost the diamond in her wedding ring and we were talking about getting it replaced. I have two daughters in their early 20s and they were aghast at the idea of replacing it with another diamond. They were adamant that nobody should buy diamonds for jewelry and instead should use a synthetic stone. They claim that "everyone" (which apparently means everyone they are in contact with on social media) knows this. If this kind of thinking really is widespread among the younger set then I'm not surprised that the demand is dropping.

<comment><USER3> I was blown away when someone told me they wanted a natural diamond. It seemed crazy to me, I don't get why someone would want to pay more for an identical (except for the fact that it was likely mined unethically) stone. I don't think any of my friends have a natural diamond but that's probably partially the people that I hang out with.

<comment><USER4> I've been poking my head in periodically and have yet to see the massive savings of lab-created diamonds, or even moissanite. I keep expecting that the market will have shifted—last time I expected it was when moissanite for jewelry's patent ran out—and one or the other will be down to only a little higher than low-quality replacements like CZ, but it's not happened. They still seem really high to me. Cheaper, yes, but still expensive. Especially lab-created diamond.

I mean I'm on board with not buying natural diamonds and all and it's not like I wanna buy any of these for myself anyway, but I don't think there'll be a full collapse in the natural diamond market until the two best alternatives put some _serious_ price pressure on them.

[EDIT] just checked again, 0.75c and up lab diamonds of middling (by gemstone standards) quality or better sure seem to still be like $2,000+. And 0.75c is merely on the large side of "modest".

<comment><USER5> Large moissanite stones are vastly cheaper than equivalent-quality natural diamonds at trade prices; if you can afford a decent .75ct diamond, you can afford a 15ct moissanite. When cut correctly, moissanite is visibly superior to diamond in fire and brilliance. That's a large part of the reason why small synthetic stones aren't as cheap as you might hope - it takes about the same amount of skilled labour to properly cut a natural diamond, a lab-grown diamond or a moissanite. CZ stones are cheap and look cheap, in large part because they're crudely cut.

<comment><USER4> Yeah, I checked Moissanite again after the post and it actually has gotten pretty cheap. Maybe $400 for a pretty good 0.75ct. That's more like it, and yeah, probably is cheap enough that it's mostly the cut you're paying for.

</comment></comment></comment></comment></comment><comment><USER6> The breakup has been on going for about 7 years now. As the article says, the old way of buying diamonds was a take it or leave it proposition. About 7 years ago though, some of the smaller players in the field (Dominion, Rio Tinto, Okavango) decided to start auctioning their rough stone every couple of months. They invited dozens, up to 100 bidders for each auction. This was the first time buyers could get a real market price for the rough stone, and there were no repercussionsn if they were outbid. It's become the industry practice now to sell a portion of the rough using auction software and using that market price to set the prices for these sight holders, who still buy a large proportion of the rough. Even DeBeers and Alrosa use auctions to set prices. The sales of diamonds has changed more in the last decade than it did the previous 100. (Disclosure, my company supplies software for these auctions).

<comment><USER6> To add to my comment a little, and this was touched up in the article. The vast majority of rough diamonds are sent to India for cutting/polishing, before being sold to the retail channels (Tiffany, Zales, Wal-Mart, etc). The profit margin for the miners is rather small, the profit margin for the cutters/polishers is razor thin, and the profit margin for the retailers is HUGE.

<comment><USER7> Is it at all possible to buy direct from the cutters and polishers?

<comment><USER6> Just speculating here, maybe if you fly to India and visit their cutting shops in Surat.

The fairest prices as I understand it are on BlueNile, which charges about 10-15% above cut price, which is much better than the 100+% you'll find in retail.

<comment><USER8> +1 for BlueNile, you can do better than them but they are a huge improvement over the mall stores.

</comment></comment></comment></comment></comment><comment><USER9> I recently purchased a moissanite[1] ring and was very happy that it was a choice. My disillusionment with diamonds started when I read this article[2] on how the industry manufactured demand. The argument against diamonds and for moissanite is succinctly made by [http://diamondssuck.com/](http://diamondssuck.com/) (I have no affiliation. I am just grateful that the author made it, and I try to share it as much as possible.)

[1]: [https://en.wikipedia.org/wiki/Moissanite](https://en.wikipedia.org/wiki/Moissanite)

[2]: [https://www.theatlantic.com/international/archive/2015/02/ho...](https://www.theatlantic.com/international/archive/2015/02/how- an-ad-campaign-invented-the-diamond-engagement-ring/385376/)

<comment><USER10> Moissanite makes a beautiful stone.

<comment><USER11> They're pretty, but they're not diamonds. They refract the light differently. They're more colorful. I can spot one from several feet away. We went with a man made diamond and love it. Less than half the price of an equivalent natural stone.

<comment><USER12> What's a man-made diamond?

<comment><USER13> A diamond made in a laboratory

</comment></comment></comment></comment></comment><comment><USER10> I'm an amateur gem faceter -- if you want something that's actually _rare_ , choose a non-diamond colored stone. A ruby or sapphire is far more rare.

Personally, I advocate for lab created stones. It is flawless, you can get the exact color you want (I have something like 20 different colors of sapphire), and won't break the bank. You can spend your money where it counts -- paying for the skills of the cutter -- rather than on a chunk of rough that has been cut sub-optimally to maximize preserving the weight of the stone rather than the brilliance of the cut.

<comment><USER14> How do you find a good jeweler/faceter that works with synthetic stones? All the downtown big name jewelers seem to work with natural.

</comment></comment><comment><USER15> Good.

We purchased a lab grown/man-made diamond on purpose. Even "non-conflict" diamonds are mined in immoral ways. And while in theory it is great to bring outside money into some of the poorest areas of the world, the work conditions and atrocities cannot be ignored[0][1][2].

Ideally people should move beyond diamonds entirely. There's plenty of precious stones. But in the meantime lab grown diamonds are the most moral way to buy a diamond.

[0] [http://www.nbcnews.com/id/15842524/ns/world_news/t/diamonds-...](http://www.nbcnews.com/id/15842524/ns/world_news/t/diamonds- journey-grim-reality-tarnishes-glitter/#.XT86ruhKhPY)

[1] [https://en.wikipedia.org/wiki/Child_labour_in_the_diamond_in...](https://en.wikipedia.org/wiki/Child_labour_in_the_diamond_industry)

[2] [https://www.huffpost.com/entry/all-diamonds-out-of- afric_b_2...](https://www.huffpost.com/entry/all-diamonds-out-of- afric_b_2628013)

<comment><USER16> Where did you purchase the lab grown diamond?

<comment><USER15> There's a number of stores on the internet that offer them. You just buy the diamond then have them set into a band of your choice.

I don't think large jewelry chains commonly offer them.

<comment><USER17> I wouldn't be surprised if their supply or natural diamonds was contingent on them not selling artificial ones.

<comment><USER18> Or maybe they're owned by the companies selling natural stones?

Wouldn't be the first time that someone tried to own the whole market from end to end.

</comment></comment></comment></comment></comment><comment><USER19> It seems some posters here are missing an important point about engagement rings: they are a gesture of financial commitment. The expensive price tag is a feature, not a bug that can be solved by creating chemically-identical but inexpensive alternatives.

If lab-grown stones become the norm for ethical reasons, but they are priced too low, I can see culture shifting to new gestures of commitment.

<comment><USER20> > It seems some posters here are missing an important point about engagement > rings: they are a gesture of financial commitment.

And all this time I thought it was to indicate a commitment to marriage.

> The expensive price tag is a feature, not a bug...

Feature? For who? The diamond industry?

<comment><USER21> Yea, not sure how this is a feature. I want to prove my loyalty so I'll waste a bunch of what will soon be your money buying you a temporary ring?

I made an engagement ring from a coconut husk in about ten seconds and it did the trick just fine. No need to spend any money really, let alone the "three month's pay" rule of thumb that gets tossed around.

<comment><USER22> In many cultures the purpose of jewelry is to give the woman wealth independent of the husband. When she is a housewife with much less earning potential she will be completely dependent, and so the gifts of gold mitigate that.

Diamond though are a terrible way to store wealth. As soon as you walk out of the store they lose significant value, and they will only depreciate.

</comment></comment></comment></comment><comment><USER23> Diamonds have had their supply artificially restricted for decades, inflating their prices. That, combined with the ethical dilemmas (blood diamonds, etc.) associated with the stones, it doesn't make sense to me to get them. They don't even look prettier than an emerald or sapphire either, there's just too much hype.

<comment><USER2> My wife doesn't like colored stones, so emeralds and sapphires are right out. We are looking at Moissanite stones. Not synthetic, but significantly cheaper than diamonds.

<comment><USER24> Moissanite is also synthetic (typically grown via a chemical vapor deposition process). It's not carbon (like diamonds), but made of SiC, a material long used for abrasives and more recently for high-power electronics and LEDs. The 4H-SiC polymorph is actually extraordinarily rare in nature, even moreso than diamonds!

We chose moissanites as the primary stones in my wife's engagement rings and everyone talks about how pretty it is. It's a bit more "firey" or "sparkly" than a regular diamond (because of more birefringence), but I guarantee no one outside a very highly trained jeweler would know the difference (if that's important to you). It's worth it to spend the money on the higher quality ones though, since just like diamonds, the inexpensive ones will have yellowish or greenish tints. Even still, they're at least an order of magnitude cheaper than similar diamonds ($600 vs. $6,500 at my quick search for ~1 carat).

</comment></comment></comment><comment><USER25> Some years ago I went to Tate modern in London and saw a massive queue in the engine room,where was a medium size black sort of a kiosk. Having no idea what's inside the booth,yet too curious to ignore it, I join the queue and after 10-15 min. I get to go inside.I walk in and there's 'For the Love of God',which essentially a skull covered in diamonds. The room was black and had some special lightning to make the skull shine.The whole thing looked so ridiculous and was on the same level as something made in China for $10,the ultimate bling bling.I didn't even stop to observe it and left the booth. Diamonds nowadays are like Gucci or louis vuitton products: even if what you are wearing is original,you still look ridiculous..

<comment><USER26> This I think [https://en.wikipedia.org/wiki/For_the_Love_of_God](https://en.wikipedia.org/wiki/For_the_Love_of_God)

I agree with you.

<comment><USER27> I do have to wonder if it wasn't made by Damien Hirst[1] whether we'd even be discussing it. Even setting aside my perceived vulgarity (a criticism I would apply to much of Hirst's work), it doesn't seem particularly new or interesting.

[https://en.wikipedia.org/wiki/Damien_Hirst](https://en.wikipedia.org/wiki/Damien_Hirst)

<comment><USER26> Where _art_ is concerned, vapid peacocks flock.

</comment></comment></comment></comment><comment><USER28> Natural diamonds have been the most shamelessly wealth-signaling purchase a person can make, ever since the artificial process was perfected. As far as I know there is literally no difference between an artificial and a natural diamond except the fact that the latter is more scarce and harder to get, and therefore more expensive. Unlike other traditional wealth-signal purchases, there is no pretense of utility or even pleasure or fashion. It's purely to say, "I can afford this."

<comment><USER6> Natural diamonds are far more plentiful than lab grown diamonds.

<comment><USER28> I mean, technically, since lab grown ones are made on-demand and haven't been an option for nearly as long. Let's not get pedantic. The point is they're much cheaper to "produce".

</comment></comment></comment><comment><USER29> Simple test about diamonds: Putting so-called "investment diamonds" and heritage pieces to one side, has anyone who bought an item of diamond jewellery here ever experienced selling it for remotely close to what they bought it for?

Investment diamonds are a semi self-regulated shell-game of hedging which tries to be like gold, but basically isn't. Gold is equally irrational in some ways, but it is used by the state banks as part of their hedge. I believe only diamond producing nations use diamonds as their financial strategic reserve. Nobody else seems to regard them as a viable mechanism for national financial resilience.

Consumer grade diamond is a propped up market where the cost and price disfunction is about as big as it can be.

Pink diamonds, now thats a scarce resource. As long as small girls are obsessed with the colour pink, and the Argyll mine in Australia has reserves (the pipe is exhausted but they banked huge numbers of stones) there is value in scarcity.

Buy shares in bort: industrial construction needs diamond grit

<comment><USER30> > Consumer grade diamond is a propped up market where the cost and price > disfunction is about as big as it can be.

A few years ago I purchased a pair of diamond earrings for myself at one of the big chains. Kay I think it was. When I paid I was prompted to get a paper receipt or have a receipt emailed to me. I hit paper and then decided I probably should have gone with email in case I needed to get them serviced or anything so I asked the woman helping me if I could change my selection. The two women working spent quite a bit of time at the computer trying to figure it out before one of them said "I'm just going to do this and be done with it."

I should have known that wasn't a good sign. They must have added my email address as some sort of corporate/store account because I was emailed a copy of every receipt for every. single. purchase. made at that location from then on.

I called the location, I called their headquarters, I talked to someone in IT at HQ. The receipts kept coming and no one at Kay knew why nor did they seem to care. I finally just gave up and abandoned the gmail address but I've often wondered if I'm sitting on one of the most comprehensive consumer diamond data sets of "retail price" vs "purchase price" in the world.

<comment><USER31> You should take a look at the data, or share it! Either way I'm super curious.

<comment><USER29> if the invoices count caret you can plot $/caret over time.

if the invoices don't count caret, you can count $/purchase over time, and then do text analysis to group the purchases.

</comment></comment></comment></comment><comment><USER32> As a nerd, once I found out that lab grown diamonds and other gems are chemically and physically the same as their natural counterparts for a fraction of the price, it became VERY difficult for me to buy natural gems. It's hard for me to justify being stupid. Thankfully my fiancee was on the same page.

<comment><USER33> Why do you want a diamond (natural or otherwise) on your finger at all?

<comment><USER34> Unfortunately even if you might feel a certain way, there might be someone else important in your life expecting a Real Diamond Ring whose opinion of you is more important than a few thousands of dollars

<comment><USER35> If their opinion of you is dependent upon that, that fact just might be worth its weight in gold.

</comment></comment></comment></comment><comment><USER36> >He argues that demand remains robust, and millennials will covet diamonds the same way their parents and grandparents did.

RIP diamond industry.

</comment><comment><USER37> Well if this is based on old traditions indicating a gesture of financial commitments then we should expect more to be buying engagement rings now:

Apparently some women are earning more than men, why should I spend my hard earned cash to impress your friends family over a rock? Why not invest the money into something productive.

An idiotic tradition.

[https://www.google.com/amp/s/techcrunch.com/2019/03/04/googl...](https://www.google.com/amp/s/techcrunch.com/2019/03/04/google- found-it-paid-men-less-than-women-for-the-same-job/amp/)

<comment><USER38> The whole diamond ring thing is basically the result of a really effective advertising campaign. It's definitely dumb to get a diamond ring when you can get a much more beautiful, intricate ring for significantly cheaper.

</comment></comment><comment><USER39> The article does a good job of pointing out issues with the diamond industry including margins getting smaller due to consolidation in the space and lack of financing.

However, a couple of points are just old and mute at this point: 1\. De Beers' monopoly in the diamond world has long been over. It now supplies only about 20 percent of the world's rough diamonds. 2\. Demand of diamonds is creeping up whereas supply is limited: [https://www.bain.com/contentassets/7b9f04068ca64542a17b71713...](https://www.bain.com/contentassets/7b9f04068ca64542a17b7171365277cb/bain- diamond-report-2018-fig31_1440x810.gif?width=1440&height=810&mode=crop)

Disclaimer: We, at Liquid Diamonds, are building an online exchange for naturally mined diamonds; think Nasdaq for diamonds. [https://liquid.diamonds/](https://liquid.diamonds/)

</comment><comment><USER8> Something I've noticed over the years: on websites that have a female majority demographic, diamonds are generally talked about in revered terms, but the polar opposite sentiment is seen on male dominated websites.

Pinterest: effusive towards diamonds. Popular articles include "What diamond shape fits your personality type?"

Reddit: hates diamonds. Popular articles include "Diamonds are a myth created by corporations."

</comment><comment><USER40> Beyond the horrific conflict-side of things surrounding this industry, it is this artificial market inflation and artificial scarcity that really bugs me. If there's anything that i think should go the way of the dodo bird, its this industry. Bleh.

</comment><comment><USER41> Diamonds are industrially useful but calling them "precious stones" or "permanent" has always seemed silly to me. They literally burn up if you heat them in an oxygen atmosphere. Most rocks don't do that.

<comment><USER42> They do burn, but not easily. They only seem to char slightly with an oxy acetylene torch (I assume that's what is being used here) [https://www.youtube.com/watch?v=hF_yqriai7E](https://www.youtube.com/watch?v=hF_yqriai7E)

It might be a heatsinking thing, as diamond does transfer heat well, and the metal underneath would continue to carry it away.

</comment></comment><comment><USER8> This thread is kind of funny on a meta-level.

People are saying things roughly equivalent to: "I purchased cheese for my wife because she wanted it" and others are countering with "cheese isn't rare, lab-grown cheese is superior, my wife doesn't even like cheese, my wife likes nuts more than cheese, acktshually nuts are superior to cheese, cows are mistreated, I'm not purchasing cheese for my future wife, your wife should have bought you cheese, check out these cheese-alternatives that are even better than cheese, there are secret vaults full of uneaten cheese, eating is bad for your health, etc. etc."

There's still a lot of emotion tied to diamonds. Diamonds will truly be on their way out when a thread about diamonds is mostly ignored instead of becoming a bizarre flame war.

</comment><comment><USER43> To see the actual market in action, try selling off your spouse's engagement ring and see the value it brings on the secondary market.

<comment><USER44> Is the answer: almost nothing?

</comment></comment><comment><USER45> When I become lucky to propose to someone, I am not going to go down the route of a diamond that everyone else has. Something more precious like Morganite or Alexandrite (unless they can be lab grown too?)

<comment><USER46> Why is that? Just curious.

When I proposed to my wife I went with a Sapphire as the main stone, mostly because I personally find clear rocks boring to look at (and my wife generally agrees).

<comment><USER45> Same reason as you just stated. Why give a loved one something that a lot of other people will have, try and be more unique.

<comment><USER47> Careful with that. I had the same thought, but was proven 'wrong' because my SO did not want anything but diamond, just because of the 'tradition.'

No amount of reading online would get her to change her mind.

She was worth it though.

</comment></comment></comment></comment><comment><USER48> Links from bloomberg.com are always behind a soft-paywall. or is this just happening to me?

<comment><USER49> I'm able to see it, but this banner is on the bottom of the page "This is your last free article." I wonder if it the number of articles resets monthly, or if this is my last one until I clear my cookies.

<comment> Digit-Al Open in a private window or use outline.com</comment></comment></comment>


Human embryonic stem cells restore gerbil hearing - <AUTHOR> http://www.nature.com/news/human-embryonic-stem-cells-restore-gerbil-hearing-1.11402

<comment><USER2> Interesting. But the link to the primary article is broken.</comment>


5 Red Flags of Startups - <AUTHOR> https://rayyanislam.squarespace.com/blog/5redflagsofstartups

<comment><USER2> I only counted 4 red flags</comment>


Coding Practice - <AUTHOR>

I am a Computer Science Undergrad and I love to code.. The obvious way that people say to get better at coding is to code more but I want to know of some good projects that other people work on in their free time.. <comment><USER2> Many people suggest working at reading through other peoples code, I've heard good things about ID's famous code for example.</comment>


Scammers Are Stealing Bitcoin on Twitter With a Classic Scheme - <AUTHOR> https://www.wired.com/story/classic-scam-steals-bitcoin-on-twitter/ <comment><USER2> 'a fool and his money are soon parted' will always hold true no matter what you use for currency.</comment>