Ageing is not an inevitable fact of life - <AUTHOR> http://www.bbc.com/future/story/20150915-the-secrets-of-living-to-200-years-old <comment><USER2> Funny they don't mention the hydra, an amazing little creature, almost(?) immortal: [http://www.wired.com/2015/02/absurd-creature-of-the-week- hyd...](http://www.wired.com/2015/02/absurd-creature-of-the-week-hydra/)

<comment><USER3> Lobsters and some jellyfish are functionally immortal as well. [https://en.wikipedia.org/wiki/Turritopsis_dohrnii](https://en.wikipedia.org/wiki/Turritopsis_dohrnii)

<comment><USER4> Smithsonian magazine disagrees on the immortality of lobsters:

[http://www.smithsonianmag.com/science-nature/dont-listen- to-...](http://www.smithsonianmag.com/science-nature/dont-listen-to-the-buzz- lobsters-arent-actually-immortal-88450872/?no-ist)

Although, since they will eventually die of energy depletion from growing too large, if they actually stopped growing they might go on ... a lot longer?

<comment><USER5> > Although, since they will eventually die of energy depletion from growing > too large, if they actually stopped growing they might go on ... a lot > longer?

If humans stopped ageing they'd go on a lot longer, too. But as long as they can't it's a hypothetical scenario in both cases.

<comment><USER6> Maybe it's because I'm not a lobster, but halting growth to keep energy requirements low seems like an easier problem.

</comment></comment></comment></comment></comment><comment><USER7> The BBC has a few interesting articles on aging. Here's another:

[http://www.bbc.com/future/story/20140520-the-girls-who- never...](http://www.bbc.com/future/story/20140520-the-girls-who-never-age)

If you read these articles, you're left wondering if there's some existing genetic solution waiting to be found that'll add a healthy 50 years to our lives.

<comment><USER8> If I could only live to be 80, but as healthy as I am now, I'd sign for it

<comment><USER9> Exercise. It's not "as healthy as I am now", but it makes a big difference. Anecdotal evidence, but my 73 year old father still rides his bike over mountains (and shot his best ever golf score last month). He _does_ complain that he wishes he had the body of a 60 year old ;-) My wife's uncle has to be pushing 90 and still runs 2 km every day. If you just keep going, it's amazing what you can accomplish even in the later stages of life.

Obviously this is survivor bias, but if you look after yourself and have a bit of luck, there's no reason you can't be healthy enough to enjoy life well past 80.

<comment><USER8> I do a minimum of 5-10 hours a week of aerobic exercise (mostly cycling), to make time for more, I'd either have to quit my job or board the kids somewhere.

I also have some anecdotal evidence. My father was always an avid cyclist as well, until his hip joint wore out. He had to give it up, and in the space of a year turned into an old man. I'm sorry, unless something else kills you in your prime, it will happen to all of us sooner or later.

</comment></comment></comment></comment><comment><USER10> Just three days ago The New York Times published a piece about immortality through cryonics: [http://www.nytimes.com/2015/09/13/us/cancer-immortality- cryo...](http://www.nytimes.com/2015/09/13/us/cancer-immortality- cryogenics.html)

Now BBC publishes an anti-aging piece. They feel like "submarines", paid/encouraged pieces to promote a certain point of view.

Yudkowsky, is that you? I'm all for it and I'm glad someone is doing this, but curiosity is killing me.

<comment><USER11> Try Google/Calico Labs, Genentech, Aubrey de Grey, Peter Thiel/SENS.org, Craig Venter/Human longevity, GenoPharmix, The Buck Institute, Berkeley Labs, Cynthia Kenyon, Brin, Page, Jack Ma, AOL 2.0.zuckerberg and the list goes on... this is real and it's happening.

"What controls aging? Biochemist Cynthia Kenyon has found a simple genetic mutation that can double the lifespan of a simple worm, C. elegans. The lessons from that discovery, and others, are pointing to how we might one day significantly extend youthful human life." — [http://www.ted.com/talks/cynthia_kenyon_experiments_that_hin...](http://www.ted.com/talks/cynthia_kenyon_experiments_that_hint_of_longer_lives)

"Google Launches New Company, Calico, to Extend the Human Life Span" - [http://goo.gl/2LgUKd](http://goo.gl/2LgUKd) "The Race To Extend The Human Lifespan Is Heating Up: Google & Craig Venter Leading the Pack" \- [http://goo.gl/xrHWgv](http://goo.gl/xrHWgv)

"For too many of our friends and family, life has been cut short or the quality of their life is too often lacking. Arthur Levinson is one of the crazy ones who thinks it doesn't have to be this way. There is no one better suited to lead this mission and I am excited to see the results." - Apple CEO Tim Cook

"Ten years ago, we thought aging was probably the result of a slow decay, a sort of rusting. But Professor Kenyon has shown that it's ... controlled by genes. That opens the possibility of slowing it down with drugs." — Jeff Holly, Bristol University

Also see, "The Breakthrough Prize in Life Sciences honors transformative advances toward understanding living systems and extending human life. The prize was founded in 2013 by Sergey Brin and Anne Wojcicki, Mark Zuckerberg and Priscilla Chan, Yuri and Julia Milner, and Jack Ma and Cathy Zhang." [https://breakthroughprize.org/Prize/2](https://breakthroughprize.org/Prize/2)

Real space travel is not solved until aging and extending human life for a bit is solved. This is why I admire companies and individuals in this area over any other area. It's simply the most innovative accomplishment that can be done in the history of humankind and with the largest rewards.

</comment></comment><comment><USER12> This is wrong. Whales live longer because they are bigger. Larger animals have slower metabolisms and consequently longer lifespans. See this paper: [http://www.ncbi.nlm.nih.gov/pubmed/15855403](http://www.ncbi.nlm.nih.gov/pubmed/15855403)

<comment><USER13> But the article also mentions the naked mole rat, which is about the size of a mouse but lives ten times longer, and Brandt's bat, which weighs as much as a sugar cube but lives 40 years.

<comment><USER12> Not sure about the mole rat, but bats cheat by shutting down their metabolism (basically hibernating) most of the time.

<comment><USER14> Why do people try to boost their metabolism (or demise) these days then? I strongly believe that being physically lazy (i.e. not expending much energy and keeping metabolism slow), but mentally active, and walking 30-45 minutes a day is the healthiest lifestyle one could have (if they can afford it).

<comment><USER12> Because heart disease is the main killer, and obesity and poor vascular fitness resulting from inactivity are major risk factors. Plus people want a tight ass.

<comment><USER14> I don't exercise and I'm fine although genetically I have 30% higher risk of obesity, so, just don't eat more than you have to and find pleasures in creation and discovery, in socializing, not in eating!

</comment></comment></comment></comment></comment></comment><comment><USER15> "They live a lot longer than human beings, yet they are living in the wild, without going to the doctor or any of the perks of human society.."

Followed by:

"By studying these whales and other extraordinarily long-lived creatures, de Magalhaes and colleagues hope we can find new medicines that will similarly slow down the human body's decay and delay death."

Has to be one of the most ironic things I've seen lately and that's saying something. We are studying organisms that live long lives without medical intervention in hopes of creating medical interventions to keep humans alive longer. Take more pills, go to the doctor more, pay more money for "health". At some point, surely we will begin to question what it means to be alive. I'd love to live longer but not at the expense of quality of life and taking pills for the rest of my life is lower quality in my view.

<comment><USER16> Maybe human understanding will evolve and we will start studying these things in order to better understand the lifestyle that gets them this result rather than how to make a pill to replicate some piece of the chemistry.

Great comment. Thank you for making it.

</comment></comment><comment><USER17> Not only it is an inevitable fact of life, it (and death) is the single most important attribute of life. Especially in the case of humans aging and death limit the amount of damage one single person can do.

I'm not saying it's absolutely impossible to change. Just that it shouldn't.

<comment><USER18> I'm not saying you're wrong, but I completely disagree with you.

I would argue that the primary reasons people fail to better fulfill their potential are lack of time, poor health and grief or hopelessness in the face of death. I also strongly believe that the epidemic of disastrous short-term thinking among us (particularly our 'leaders') is directly linked to the fact that each of us knows we're going to die all too soon.

I could meet you halfway and say that in our current form, we may not be ideally suited for significantly longer lifespans-- but we can change ourselves with drugs, technology and genetic engineering, so it would be a temporary problem.

The only paradise I believe is possible is one we create, and we cannot do it with this pathetically paltry handful of decades we're allotted.

<comment><USER19> Sorry but no...

We have 7 billion people on this planet already and the last thing we need is for them to be living longer.

You can't fulfill your potential in 75 years? Why would that change if you had 150 years? Hopelessness in the face of death? I am 46 and that is not even in my thought process yet...

We don't need drugs, technology and genetic engineering to live longer. We need less poverty (better/fairer distribution of wealth), less disease, and less war for a better quality of life.

Who wants to be here in 100 years anyway? I don't want to live in a world of 20 billion people with little resources on a barley habitable world.

<comment> saiya-jin these movements are not about everybody living much longer, only selected wealthy elite. otherwise, this cash would be dedicated to matters you mention (and I fully agree with what you write).

Just look at people beyond 80 - most of them are a mere shadow of their former self in all aspects. Living till 120 just for the sake of living sounds more like a nightmare - that tells me they are terrified of death, meaning all are godless in their heart no matter how often they practice their religion. I welcome it (hopefully at proper age) - I will make room for another generation, just like the previous one did for me. I have my time here, and that's enough.

<comment><USER20> _these movements are not about everybody living much longer, only selected wealthy elite_

Just like cell phones cost thousands of dollars and will only ever be toys for the rich.

 _Just look at people beyond 80 - most of them are a mere shadow of their former self in all aspects._

That's exactly what we're saying: aging is awful, and we should fix it.

 _I have my time here, and that 's enough._

And that's why you'll refuse all medical treatment after you turn 60, right? (The expected lifespan for 20 year old people in 1850, before our godless and unnatural medicine started working: [http://www.infoplease.com/ipa/A0005140.html](http://www.infoplease.com/ipa/A0005140.html))

</comment></comment></comment></comment></comment><comment><USER21> Re: the title. _What_ are inevitable facts of life?

<comment><USER22> Well, there's the laws of physics, and the Second Law of Thermodynamics pushing the arrow of time forwards. Beyond that, not much is _actually_ inevitable: people just prefer to rationalize anything awful as Inevitable because this gives them a comforting sense of metaphysics.

<comment><USER23> 2nd law of thermodynamics only applies on macroscopic scale. So not really inevitable.

Plus, the laws of physics are based on a set of axioms, which don't necessarily have to be true.

<comment><USER22> >Plus, the laws of physics are based on a set of axioms,

Well no. They're based on empirical observations.

</comment></comment></comment></comment><comment><USER24> death and taxes next on list of problems to be tackled by scientists.</comment>


Ask HN: What does your BI stack look like? - <AUTHOR> How does your company visualize and report on what's happening in the business? <comment><USER2> Postgres -> Metabase

I believe this is the best combination of cheap/powerful for early-stage startups. My very non-technical cofounder is able to use metabase's simple GUI interface to create graphs/insights (even joining and aggregating across tables!), and for anything complex I can step in a give a helper SQL query. We have around 10M records we aggregate around for daily insights.

Metabase can also be run as a pseudo-desktop/web application to save additional cash (we don't do this though).

<comment><USER3> UX guy from Metabase here. Glad to hear it's working well for you. Happy to answer any questions folks might have.

<comment><AUTHOR> Wow great to see you in the thread. No questions from me but great job on the latest update.

<comment><USER4> Thanks!

-Another Metabase team member

</comment></comment></comment></comment><comment><USER5> If you have a data team: Stitch / Segment -> BigQuery -> Dataform -> BigQuery -> Looker

I work with many companies helping them set up their data stack, and from what I've seen this is pretty much the optimal set up. These tools all require very little maintenance, are relatively cheap (compared to the man power required to set these things up from scratch internally), and scale well as companies grow and complexity increases.

If you don't have a data team: Segment -> Amplitude/Mixpanel

If you don't have a dedicated data team, you can do a lot with tools like Amplitude and Mixpanel. Get your engineering team set up with Segment and tracking all of the important interactions within your product, and set Segment to pass those events to Amp/Mix. Self serve analytics in these tools is then pretty easy and will serve you well until you start hiring a data team.

Full disclosure: I work for Dataform (used to be a customer, loved it so much I decided to join them)

<comment><USER6> Thanks for sharing, I don't think there's an "optimal setup" in the objective sense but good to know this works well.

<comment><USER5> Good point. I guess I mean, if you put a gun to my head and forced me to pick one, this is what I'd pick (and I think it works for a pretty meaningful percentage of companies that are looking for an answer to the original question)

</comment></comment></comment><comment><USER7> As a data scientist (startup / Yahoo) turned product manager (4 startups), I've used a variety of stacks in the past - from plain SQL, to Mode, to Mixpanel, Looker, Interana, and Hive.

Recently we started using PopSQL ([https://popsql.com](https://popsql.com)) and love it.

When I don't have a dedicated data team, my philosophy is:

1) Make it difficult to get wrong answers

\- Don't use Google Analytics. It's too easy to generate incorrect charts, and too difficult to verify them.

\- Have a limited sandbox of reports for non-SQL writers

\- Keep the SQL close to the report, so it's easy to verify the underlying query.

\- Push people to learn even basic SQL

2) Make it quick and easy to ask iterative questions \- PopSQL is way faster than Mode. Like 20x faster.

3) For metrics that matter (e.g. KPIs), instrument them directly and even build a custom analytics dashboard if it's important. (beware dashboard clutter! [https://twitter.com/andrewchen/status/1193619877489192961](https://twitter.com/andrewchen/status/1193619877489192961) )

<comment><USER8> $20/user/month seems incredibly expensive for an editor that we would have traditionally paid a one time cost for. I assume this has the saving and sharing of queries built in, but that is a lot of money for some storage. Are there other major benefits I am not seeing? (It is not my intention to attack you or make you defend this product, I am just curious if I am missing something - there are a lot of similar products with similarly questionable value propositions)

<comment><USER7> Saving, shared queries, easy link sharing, and basic charts with some intelligence. It also has excellent query UX (auto-complete, etc.).

Most of all, it's really fast. It fundamentally depends on your DB of course, but PopSQL doesn't add any extra bloat the way Mode does.

</comment></comment></comment><comment><USER9> At Grove, we are:

Airflow -> S3 -> DBT with Spark/EMR or Redshift/Spectrum -> Redshift data marts -> Looker

At least, that's the way we like our pipelines to work. In practice we have a couple of extractions that land directly in Redshift (we extract Zendesk data, for instance, with Stitch Data). We use Snowplow for click stream analytics. And we'll likely move from Redshift to Snowflake (or mayybbbeee Bigtable) in Q2 of 2020.

We used to do all of our transforms via DBT within Redshift but have been offloading the heavier-duty pieces (like Snowplow event processing) to Spark jobs because they were too taxing on Redshift.

We've gone through 3 "generations" of Looker reporting — gen 1 was just wrapping LookML around our schema and forcing Looker to do the joins and generate SQL for everything we wanted to know. Second generation involved a bunch of common table expressions within Looker itself that worked, but were developed without much thought as to data mart design. Gen 3 is where we are now with super deliberate scoping and implementation of warehouses in DBT. Before any of that we rolled our own tool [0].

Happy to answer any questions. We've gone from ~30 to ~1000 employees in the 3 years since we started using Looker and have learned a thing or two along the way.

[0] [https://github.com/groveco/django-sql- explorer](https://github.com/groveco/django-sql-explorer)

<comment><USER10> Very similar to our progression, do you have a good way you manage your data dictionary and catalog?

<comment><USER11> I don't want to derail the thread, but I recently started a company ([https://syndetic.co](https://syndetic.co)) that's working on this problem. We've been focusing on the external data dictionary use case (how does a data- as-a-service company explain a dataset to their prospective customers) but we've been encountering a number of companies that are evaluating data catalogs and other internal tools for their data science teams.

I would really appreciate getting your perspective - I'm steve (at) syndetic.co

<comment><USER10> Was just thinking this would make a great business opportunity. Thanks, we'll check it out!

<comment><USER12> This space desperately needs some new options.

The legacy players are a mess, the new cloud native offerings are overly Engineering focused and immature, and the few startups in this space are recreating the legacy solutions in a SAAS formats

Go forth and claim the significant checks that I and others plan to write for these mediocre offerings.

</comment></comment></comment></comment></comment><comment><USER13> Luigi, AWS S3, DBT, Snowflake and Re:dash (currently analyzing Metabase or Looker to allow queries without SQL)

Luigi runs our scrapers and other workflow management tasks (e.g. DB backups).

All raw data lives in S3. We make an effort to be able to recreate the whole data warehouse from the raw data, so if any cleaning/normalization process fails, we have this safety net. I'm curious to hear if others use a similar pattern, or if there are better options.

DBT handles both loading the data from S3 into Snowflake (by creating the Snowflake Stages), and transforming the data in Snowflake. This isn't how DBT is used usually, but it felt wasteful to add Stitch or another tool to load the data into Snowflake, as snowflake supports it out of the box. I also created a `setup_snowflake` macro operation that creates our users, warehouses, databases, etc., in Snowflake (a kind of "poor man's Terraform")

I don't think Snowflake requires introduction. It's an amazing tool. We used Postgres before, but Snowflake is much much better, even though our DB is pretty small (~200 GB).

Finally, we use Re:dash as a BI, but I'm checking other options that allow usage without SQL (currently Metabase and Looker).

<comment><USER14> You might want to check out Holistics.io, allowing your non-technical to explore data without writing SQL.

It's an alternative to Looker but more affordable, and more powerful than Metabase.

<comment><USER15> If you setup your data well and have denormalized data tables, metabase is really good.

</comment></comment></comment><comment><USER10> Looker on top of Redshift. Events streamed in from Segment and ELT transforms managed by Airflow. Looker gives you nice visualizations and charting with RBAC and some some lightweight ETL functionality. The real advantage of Looker is their modeling layer which sits on top of your physical data and is done via a DSL called LookML. Source control is managed via a wrapper around git. The end result is that analysts can move lightning fast building their own models in SQL or do analysis via drag and drop explorations. Looker's customer support is the best I've experienced and hasn't changed since Google acquired Looker. We're likely moving off Redshift to Snowflake in the next 6 months because it is slow to pick up changes and we want to decouple storage and compute for scaling reasons. Airflow is an incredible platform but needs customization and templated workflows to make it great. Data build tool (DBT) is how we plan on managing ETL code via Airflow in the near future. We're also adding Spark, but more for data science.

<comment><USER5> If you're interested in an alternative to dbt that's a little more analyst friendly, check out Dataform. A lot of teams are using Dataform between the warehouse and looker, to handle transformation logic using pure SQL, but in a more user friendly format than Airflow. Get in touch if you'd like to chat! dan at dataform.co

</comment></comment><comment><USER16> We're a small, bootstrapped company with 2 people. Some data is millions of rows others are billions.

Goal for us is KISS. Keeping everything as simple as possible -- both in terms of infrastructure, ease of use and cost.

Primary we're using Metabase in front of multiple MariaDB instances. Metabase is absolutely fantastic!

We also have a couple of additional small projects:

\- A small Node.js app that receives events live via fluentbit, saves them off in MariaDB and sends text message notifications via Twilio when certain thresholds are exceeded

\- A small "ETL" app that polls the Stripe and QuickBooks API to mirror data in a MariaDB database so we can easily access it from Metabase

</comment><comment><USER17> Stitch/Fivetran/Custom -> Snowflake -> dbt -> Periscope

Lots more documentation here [https://about.gitlab.com/handbook/business- ops/data-team/](https://about.gitlab.com/handbook/business-ops/data-team/)

We have many of our KPIs embedded in the handbook (look for KPI index mapping link).

This is also our primary project where you can see all of our dbt code [https://gitlab.com/gitlab-data/analytics/](https://gitlab.com/gitlab- data/analytics/)

<comment><USER18> The gitlab wiki has some amazing content on it. Thanks for making it public.

</comment></comment><comment><USER19> Sorry to be that one avoids-the-question comment, but:

Without design sessions to figure out your data store design (look up Kimball, Immon), and then monitoring/testing to make sure everything is running smoothly, any data stack will be hell.

Badly designed data brings fatigue and rancor and unnecessary work and is a huge hard-to-quantify money suck.

<comment><USER5> IMO the right solution to this is not spending a bunch of time upfront trying to design your warehouse. You just want to adopt a tool that makes it easy to continuously refactor into more optimal formats as your warehouse scales. Happy to discuss more if you're interested! dan at dataform.co

</comment></comment><comment><USER20> We use Snowplow directly into Snowflake and report on this using Mode. We are a data native SaaS firm, and we set things up like this about 5 years ago and it has served us well. Streamlined and high performance. For all other sources, we use Stitch -> Snowflake, with one semi-custom Singer.io tap also running on Stitch. All this was simple to set up and means we don't have to worry about getting the data into one place. Of course, we then have lots of work to do in Snowflake and Mode to get the data the business needs. We share the reports from Mode to various teas via Slack. Hope this helps.

<comment><USER5> Are you using Dataform Graham? I don't think we've spoken if you are, I'd love to chat! dan at dataform.co

</comment></comment><comment><USER21> At Amplify, Matillion/Stitch/Fivetran/Custom -> S3 + Snowflake -> Matillion for transforms -> Looker + Tableau.

It's important to say out loud that a lot of analysis also happens within product-local reporting systems, or in "offline" Google sheets.

We are currently working on building out the same stack, terminating in a Powered By Looker instance, for customer-facing aggregate reporting.

The engineering and data science teams do great with Looker, but Tableau still covers use cases for non-engineer business people (think: the finance, customer operations, training, etc) who want to start from source data plus a few sidebar Google sheets, tinker around, and then develop a rough report without waiting for iterative cycles with the data engineering team. We're thinking hard about how to get those use cases into the warehouse + Looker faster.

</comment><comment><USER22> Data infrastructure is scattered, siloed, excel sheets and google sheets stored in various places (personal g drives, company g drives, some network share somewhere, an ms sharepoint site, sometimes in development git repos or various wikis)

Reporting infrastructure is manual massaging and lots of powerpoint.

My company isn't that large, but bigger than you'd think for such a "system".

<comment><USER23> > but bigger than you'd think for such a "system".

I suspect you underestimate how large some of the organizations are that use a variant of this system.

</comment></comment><comment><USER24> After reading the comments here, I decided to give Metabase[0] a shot. Awesome analysis tool. Very impressed. Works even with large SQLite databases.

[0]:[https://www.metabase.com/](https://www.metabase.com/)

<comment><USER25> Being a Clojure [0] team, we naturally use Metabase as well. Analyzing a Postgres data warehouse input from Segment, as well as our production db.

[0] - [https://github.com/metabase/metabase](https://github.com/metabase/metabase)

</comment></comment><comment><USER26> Data warehouse:

\- Amazon Redshift (data sourced from many different backend DBs; e.g. PostgreSQL, MySQL, other Redshift instances etc.)

\- BigQuery (Some teams store the data for their projects here. For reporting, they're selectively transferred to Redshift.)

Reports:

\- Tableau (extracts of pre-aggregated data from Redshift)

\- Looker (connects live to and executes its own queries on Redshift)

Anything that is based on pre-aggregated (rather small, e.g. n_rows < 5mil) data is visualized on Tableau. If users want to work on a self-service tool they use Looker which can connect to (cleaned) raw data optimized for this use case.

ETLs for raw data run daily on Redshift. Reports are also scheduled on Redshift (with Airflow) but the refresh interval is report-specific.

</comment><comment><USER27> We keep it very simple as well.

Postgres read-replica for cheap realtime stuff, Redshift for the larger sets, Airflow to manage transfers, and Metabase to visualize and share. We also collect data from a bunch of APIs, but those each run via their own job and store in Postgres.

We also try to define wide short tables in Redshift that enable users to use Metabase to ask questions.

I was very happy with Metabase. Being that we can't afford Looker right now (but we would all love to) it is pretty solid.

</comment><comment><USER28> Stitch/Airflow/Other -> Snowflake -> dbt -> Snowflake

Everyhting goes through S3 because Snowflake storage is on it.

dbt is amazing, we began using it a month ago and it already transformed the way our data team work. It really is a value multiplier for everyone. Data engineers are happier because they don't need to write and maintain data transformations, analysts are happier because they can maintain their own SQL pipelines & the whole company is happier because we now have a great documentation tool to explore our data.

We also are big fans of Snowflake, make operating a data warehouse a breeze.

Then, we use a mix of Redash & Tableau for reporting.

Redash for static reporting (open to the whole company) & Tableau to create more complex data tools we expose to some internal teams; Marketing, Risk, Finance ...

</comment><comment><USER29> Postgres (read replica) -> Redash

Segment -> Amplitude (but using it less and less)

I am surprised no one mentioned [https://redash.io/](https://redash.io/) till now (a lot cheaper than looker/mode/Periscope with all the basic functionality that you might need).

<comment><USER30> Redash is far too expensive for what it does imo. Haven't used the other solutions though.

</comment></comment><comment><USER31> We use Segment for event tracking, Postgresql for transactional data and a number of spreadsheets and third-party integrations with Stitchdata. Since our data is relatively small, we use PG as a data-warehouse and heavily use DBT for ETL. The people who are familiar with SQL just use DataGrip, for the UI we use our tool [https://rakam.io](https://rakam.io).

Shameless plug: It's basically similar to Looker but it plays well with the product data and integration with Segment as well.

</comment><comment><USER32> Informatica -> ADLS -> SQL Server -> PowerBI also, in the same department: NiFi -> HDFS -> Spark -> Hive -> NiFi -> ADLS -> PowerBI and: NiFi -> Azure Event Hubs -> Azure container instances -> Event Hubs -> Streaming Analytics -> PowerBI

I'm pretty fond of the last stack for streaming dashboards in the sensor data realm.

</comment><comment><USER33>
        -> Kafka-connect -> Snowflake -> SQL/sf-tasks -> Snowflake -> Looker
        -> Alooma        ->
        -> custom        ->

Using Kafka-connect, we're able to serve up near real-time (2-5 mins) insights on device generated events.

We probably need to use some kind of ETL tool to replace custom SQL and sf- tasks. Unfortunately, we haven't been able to find a tool that handles this in a non-batch (even if it's micro-batching) form. Snowflake change-streams and tasks allows us to ETL in a streaming-like fashion.

We're ingesting everything from raw/transformed/aggregated events, micro- service DBs (as fast as they sprout up), netsuite/salesforce, mixpanel, MySQL, MongoDB... Billions of rows of data across multiple data-source accessible to internal and external customer in a matter of seconds. It's been an incredible challenge, especially with only a team of 2-5 people.

</comment><comment><USER34> Python, Redshift, Tableau. But if you are starting from scratch I'd suggest to focus on the etl piece with python and send pdf reports generated with latex. Too many people get distracted by the fancy reporting stuff and don't do the 2 things that matter: good etl with good sql db; analytics that is tied to results the company cares about

</comment><comment><USER18> It's interesting seeing the various stacks being used...Here at One Medical:

[ onemedical.com, mixpanel, Google Sheets, Salesforce, etc ] -> S3 (Amazon EMR) -> [ Tableau, Aurora MySQL ]

It's a nice & clean stack for data engineering.

Airflow is used for orchestration and is heavily customized with plugins. Zeppelin notebooks are used by most analysts.

We'll probably be replacing Aurora MySQL w/an MPP database like Snowflake or Redshift. MySQL is a crutch, but easy to deploy and manage to a point.

Several python frameworks also do various ETL & data science functions, but everything generally revolves around S3 & Airflow.

Amazon EMR is a great swiss army knife for ETL. Moving between Pig, Spark & Hive is trivial. Tableau is a bit of a challenge. Tableau seems to give users too much rope to hang themselves with.

Also, we're hiring: [https://www.onemedical.com/careers/](https://www.onemedical.com/careers/)

</comment><comment><USER35> In my previous company, we did TreasureData->Tableau.

TreasureData is a platform like Segment and lets you bring your customer event data as well as data from SaaS tools (like Salesforce, Zendesk) into a data warehouse hosted by TreasureData. It worked great but had the downside that all the data was in TreasureData and we were kind of locked into it. Segment kind of solves that problem because it has connectors to Redshift/Snowflake etc so you can keep ownership of your data warehouse but the warehouse sync delay (in our version) was a problem.

Also, BI was just one of the use cases. We wanted to send the data to 3rd party tools (like Facebook ads) based on some logic (some were simple rules but we had complex ML driven lead scoring models too). TreasureData was more flexible on being able to run those rules and activate the results but ideally we wanted to run them on top of our own warehouse in AWS.

</comment><comment><USER36> Sharing an awesome business intelligence tools list [0] I created ca. 2 years ago to get a better understanding of what's out there.

[0]: [https://github.com/thenaturalist/awesome-business- intelligen...](https://github.com/thenaturalist/awesome-business-intelligence)

</comment><comment><USER37> It covers many more bases than business intelligence, but Forest Admin. And most of its functionality is free.

[https://www.forestadmin.com/](https://www.forestadmin.com/)

It generates a very beautiful CRUD admin dashboard automatically via reflection.

Allows building drag-and-drop data viz dashboards, saving commonly-used custom queries as "scopes", and even building your own HTML views if you need to get really fancy (think tracking live deliveries on a map, etc).

Also has Stripe and Intercom integrations.

I really can't hype this enough. Have been using this on nearly every app I've built the past three years.

The core team also answered my emails as a never-paying customer within 2-3 days the few times I have mailed them over the years I've used it.

<comment><USER38> Wow, that looks incredible. I don't know why, but I've never really thought to look for admin site solutions out there, instead opting for janky in-house built ones. I'll definitely be playing around with this.

</comment></comment><comment><USER39> Stitch -> Redshift (with DBT for data modeling) -> Looker

For a smaller company, it makes a lot of sense for us to use off-the-shelf tools like these rather than rolling our own pipelines. Every once in a while we run into issues where Stitch isn't quite as flexible as we'd like (can't set dist/sortkeys etc), but that's the tradeoff you make for its simplicity.

DBT is amazing and I can't recommend it highly enough!

Looker works for analytics, but we're starting to do more data-sciency work, and it doesn't have built-in Jupyter notebooks or anything like that. Does anyone have a solution they use and like for hosting and sharing notebooks alongside a stack like this?

</comment><comment><USER40> A lot of these answers seem to focus on app analytics (e.g. collecting clicks, page views, etc. from Segment). How are people collecting / integrating financial data (e.g. sales, subscriptions, expenses)?

<comment><USER33> So how are you collecting/integrating financial such as what is contained in salesforce/netsuite/etc?

My recommendation would be

\- an ETL as a service such as fivetran, stitch, etc

\- into a data warehouse such as snowflake, big-query, etc

\- transform/aggregate/normalize/magic data using some ETL tool such as dbt, matillion, etc

\- analytics tool such as Looker, Tableau, etc...

</comment></comment><comment><USER41> Ive done 2 data pipelines, one alot like what most people are are talking.

The other I had to build for a startup with millions of monthly uniques but only seed funding (cant do a 30+k a month data eng bill).

Went with custom event emission->kenisis->(druid & S3) and used imply ([https://imply.io/](https://imply.io/)). Easy real time data analytics, auto "enrichment with druid lookups from a RDBMS, and a simple ui for slice/dice investigation metrics. All in all costed lest then the cheapest looker license.

</comment><comment><USER42> We made our own reporting. Seems crazy to pay multiple 3rd parties to look at our own data. We're using Apache Druid on the back end. (Giant fortune 50 non tech company)

<comment><USER43> SQL server -> R server running on CentOS

Pipe stuff to Domo for the investors.

</comment></comment><comment><USER44> Daily ETL in form of sql scripts from prod dbs to internal data warehouse for use via Saiku, case-by-base Shiny apps and some bash email-reporting.

</comment><comment><USER45> Postgresql > pgadmin

For less technical people > metabase

For automated reporting and storing historic trends > Klipfolio

For near real time automated operational reporting > kloud.io

</comment><comment><USER46> Formerly: Oracle Hyperion -> ETL to Azure -> PowerBI

Currently: Internal Data Warehouse -> RDS -> Internal web app (Django, React)

<comment><USER47> Was that Essbase or HFM? I had lots of fun once getting bulk data out of HFM and into spreadsheets via a web app.

<comment><USER46> Essbase, which made it a pain to get everyone off of Excel.

</comment></comment></comment><comment><USER48> Custom ETL -> BiqQuery -> Datastudio & Metabase

We initially considered Stitch and other -as-a-service ETL but ~500 lines of Python later we had our own thing. I also experimented with FDW: [https://unwindprotect.com/postgres-fdw](https://unwindprotect.com/postgres- fdw)

</comment><comment><USER49> Does Excel qualify here?

</comment><comment><USER25> On a meta note, if you're interested in viewing & sharing stacks, that's the primary feature of the startup I'm working on: Vetd (app.vetd.com). The communities we host (often VC portfolio companies) share their stacks and leverage for discounts.

\- CTO (chris at vetd.com)

<comment><USER33> What does this have to do with BI stacks?

<comment><USER25> Companies using our platform add all sorts of items to their stack, including BI tools/services.

<comment><USER33> Nice. So you do hypothetically provide a stack similar to

        Fivetran -> Snowflake -> Looker

<comment><USER25> We don't currently have any way to define relationships between products, but I like that idea...

It looks something like this [0] when you view the aggregated data of the companies in your community, and you can also view each company's individual stack.

[0] - [https://imgur.com/a/UXLN5KJ](https://imgur.com/a/UXLN5KJ)

</comment></comment></comment></comment></comment><comment><USER50> We use BigQuery as the data warehouse - there are airflow jobs to periodically load data into BQ - most of these are simple python scripts.

Metabase for most of our simple BI metrics - Tableau for some advanced users doing more complicated stuff.

</comment><comment><USER51> App, Web, IoT device send realtime events to a fluentd system with many events going to Segment. Web front end also loads Segment for various ad trackers. IoT device also uploads telemetry and other logs in both realtime and asynchronously and those end up in some Postgres databases.

Segment syncs our event data periodically to our data warehouse (Redshift).

We have a readonly replica of the eCommerce DB for live stats (Postgres).

And there is a time series db for system/IoT telemetry (InfluxDB).

Most of our BI queries are done in Mode. Some are done in Grafana (data in our InfluxDB and some live data). Spot check against Google Analytics or FB ad tracker...

</comment><comment><USER15> MariaDB > Metabase Stitch/Segment > BigQuery > Metabase Stitch/Segment > BigQuery > Google Data Studio (Curmudgeonly stakeholders refuse to log in to BI tools... but require reporting anyways...)

</comment><comment><USER52> MySQL -> replication -> MySQL -> Metabase & Tableau

We want to switch to postgres because of features, but "The devil you know is better than the devil you don't", so we just kinda sticking with MySQL.

</comment><comment><USER53> A bit different from the crowd here. After much searching, our BI stack has been fully converted to data streaming using: StreamSets DataCollector >> Kafka >> AWS S3 >> AWS Redshift >> Dremio >> Jupyter Notebooks. Great to have Jupyter take on data prep and data analysis tasks while external tables in Redshift are very fast with minimal ETL. Dremio has been great as a virtual data mart with great performance on Parquet data in S3 and desktop BI tool integration.

</comment><comment><USER54> I work at Snowflake where we're dogfooding our own stuff.

</comment><comment><USER55> We've recently changed from : Segment -> BigQuery -> Looker to Segment -> BigQuery -> Dataform -> BigQuery -> Looker.

The addition of Dataform into our stack has completely changed the game, allowing us to maintain one source of objective truth and maximise looker (minimising the joining etc. in LookML, instead pointing it straight at tables in BigQuery).

This in turn means our downstream data users are happier to use a fast and reliable looker instance!

<comment><USER36> Your first comment ever?

Any affiliation with DataForm?

</comment></comment><comment><USER14> Postgres, Pipedrive, Zendesk -> BigQuery, then BI using Holistics.io

Holistics handles both ETL, transformation and self-service visualiation (Looker alternative), all in 1 tool.

</comment><comment><USER56> Custom pipes mostly coordinated by Luigi (heavily customized) -> Redshift -> DBT -> Looker.

Some spark mixed in at the ingestion and transformation phases.

Like someone else said in this thread, we're currently battling Redshift scaling problems and are looking to offload more of the raw data to S3 using Spark to create read views.

No data catalog right now but the Looker setup is extremely well-maintained. Hoping to deploy Lyft's Amundsen to fix this more generally.

<comment><USER31> Since your Looker setup is well-maintained, did you consider adopting the data catalog in Looker, what would be the challenge doing it?

<comment><USER56> The main reason is that only a subset of all our datasets are actually exposed in Looker, and Amundsen would have applicability to other data sources - certain production semi-OLAP DBs, the future hive-on-S3 deployment... that we have no plans to expose in Looker at all.

That said I haven't looked closely at how far we could go with Looker, I'll check it out.

<comment><USER31> I see. We actually develop a BI tool similar to Looker but also want to implement a data catalog feature embedded into it. I'm familiar with Amundsen but I believe that this kind of tool needs to have tight integration with the BI tool and Amundsen still has a way in terms of integrations. (searching among saved reports, dashboards and collaboration with teams, etc.). Otherwise, using spreadsheets is still easier and maintainable.

I would love to hear the features that you would need in a data catalog system.

</comment></comment></comment></comment><comment><USER57> Dave from Chartio here, wanting to share our new book describing the 4 stages of setting up your ideal data stack here - [https://dataschool.com/data- governance/](https://dataschool.com/data-governance/).

It covers BI a bit, but mostly the stack that BI sits on top of. It's an open book so we're always looking for suggestions and experiences such as those shared here.

</comment><comment><USER58> We built a tool to help non-technical folk transform their data into useful formats for BI ([https://www.data-monkey.com](https://www.data-monkey.com)). The tool currently supports JSON, CSV, Excel and text files, and comes with features to merge/filter/transform data. It's free to try and use if you're interested - we'd love to hear your feedback!

</comment><comment><USER59> We built our own tool at Datastreams ([https://datastreams.io](https://datastreams.io)) to collect data. We currently collect several thousands of events per second, mainly web data.

Events are mainly streamed to one of the following: Cloud buckets(S3, etc), HDFS, SQL-db or Cassandra.

Most clients use one of the following visualization tools: PowerBI, Qlik or Tableau.

Our clients are mid to enterprise size.

Disclaimer: I work at Datastreams

</comment><comment><USER60> Is there any way that we can provide BI to our customers using an OSS tool?

Our databases store all our users data. I'm thinking of using something like pgsync to sync all database postgres to a new postgres and then having redash or metabase set up to connect.

Alternatively using locked filters on metabase and embedded questions.

All our data for our users are in postgres and they very much want BI insights. Not sure how easiest way for this...

<comment><USER61> For his purpose you can check 'Team Sharing' feature of our BI tool (SeekTable): [https://www.seektable.com/help/share-report-to- team](https://www.seektable.com/help/share-report-to-team)

</comment></comment><comment><USER62> Ideally: Podio , excel sheets, other program backend-> Pentaho -> MySQL/MongoDB on Google Cloud, Google Analytics-> R scripting(DBI connections, BigQuery and Hadoop managed through R) -> shiny-proxy, APIs, Hadoop for Big Data computation-> Rmarkdown Reports, shiny-dashboards, a little Tableau and a little Power BI.

MySQL will be moved to Postgres for better Performance soon.

</comment><comment><USER63> Custom Collector(analytics) -> Clickhouse -> Custom ETL Scripts -> Clickhouse -> Re:dash We tried metabase which is awesome , but Redash is also great and easy to setup as well if your team knows sql then Redash is better We also looked at druid and after some benchmarking we settled on Clickhouse, realtime queries even without etl runs within seconds in clickhouse

</comment><comment><USER64> We are a small data consultancy, so we use other/more diverse things for customers, but our internal stack is fairly simple:

Hubspot/Jira/G Suite → (Python) → PostgreSQL → (SQL) → Tableau

Since we are Tableau partners, we have a bunch of internal licences either way. We host Tableau Server, ETL, and PostgreSQL ourselves, all on Centos cloud servers.

</comment><comment><USER65> {SQL Server, Oracle} -> ETL to SQL Server (Hybrid graph db)-> SQL Server Analysis Services -> {Power Bi, Tableau}

<comment><USER66> SSAS is my favorite cube / OLAP data store. I know I'm in the minority, but I find it really user friendly for devs and users.

</comment></comment><comment><USER67> Product Manager at Replica here.

I've set up the following stack:

Segment -> (postgres DB, Intercom, Heap Analytics, Full Story)

postgress = Data dump for future usage. Intercom = CMS + Communication. Heap Analytics = Product analytics. Full Story = Session analytics.

[https://replicastudios.com/](https://replicastudios.com/)

</comment><comment><USER68> Various -> Airflow -> BigQuery -> Looker

We have a variety of data sources, from Mixpanel to PostgreSQL to black-box systems that can only dump out CSV files. We use Airflow to manage pulling from those, doing some transforms, and writing everything into BigQuery. Looker then handles the reporting and browsing.

</comment><comment><USER69> ETL: Python & SQL

Warehouse: PostgreSQL

Reporting platform: Looker

Easy, agile, and cheap.

</comment><comment><USER70> Oracle -> IBM Cognos

A nightly rebuild using ETL scripts written in sql. Not cheap or glamorous, but solid for our needs.

</comment><comment><USER71> Data Sources (Analytics, DBs etc) -> Stitch (stitchdata.com) -> Redshift -> Periscope (periscopedata.com)

This setup has worked pretty well for us so far. I've learned of a few tools from this thread that might help us to better manage data sets and views - specifically DBT and Dataform.

</comment><comment><USER72> @sixo @numlocked I'd be happy to share more about Snowflake's architecture. As others mentioned in this thread, Snowflake completely separates storage from compute -- eliminating Redshift scaling issues.

Drop me a message, would love to chat.

email@example.com

</comment><comment><USER73> We use Retool.com with our Postgres & other data sources.

The benefit we can also build tooling and workflows, in addition to the metrics, tables and charts.

Early on you don't necessary don't exactly what end up needing, so malleable tool is useful.

</comment><comment><USER74> Are any of you offering embedded analytics in your products? Mostly applies to mid-market/enterprise SaaS platforms but interested in learning more about how you offer analytics/BI to your customers in your products.

<comment><USER75> Hi! You may like to check out what our team has built at www.holistics.io. The pricing metrics of our embedded dashboards cater for you to scale to unlimited viewers, and you don't need to recreate a duplicate copy of your user accounts into our system for it to work. Check out the video below to find out more!

[https://www.holistics.io/guides/embed- analytics/](https://www.holistics.io/guides/embed-analytics/)

</comment></comment><comment><USER76> We use the standard Microsoft stuff. SQL Server, SSIS, SSRS, PowerBI. There are some pain points, but for the most part, it works, and it's pretty inexpensive if you're already a Microsoft shop.

</comment><comment><USER77> No mention for Athena? Right now we are heavily using it at our org.

</comment><comment><USER78> At Qubole, I have this setup for internal reporting

MySQL -> Data export using Sqoop through Airflow -> S3 -> Spark -> Jupyter Notebook

PS: Qubole is a data platform which makes ETL pipeline setup easy.

</comment><comment><USER79> how does one end up doing BI | Data Engineering ? currently a frontend | full- stack JS dev. tired of that world and want to switch to something stable ?

<comment><USER13> I'd highly recommend the Kimball books, e.g. "The Data Warehouse Toolkit".

Unless you're working with a lot of data, the technical parts of a data warehouse are relatively easy to grasp (the dbt team wrote some great posts on how they structure their projects [1]). After that, what really makes a difference is how you structure our data warehouse, how you model your data in a way that allow you to query it efficiently and ask many questions. That's where the Kimball books shine.

[1] [https://community.snowflake.com/s/article/Use-Case-How-We- Co...](https://community.snowflake.com/s/article/Use-Case-How-We-Configure- Snowflake-at-Fishtown-Analytics)

</comment></comment><comment><USER80> Datacoral, which is our ETL and manages all materialized views within our warehouse, replacing our need for data engineers Redshift Mode & Looker

</comment><comment><USER81> Kinesis Firehose -> S3 -> Snowflake -> Sigma

</comment><comment><USER82> MySQL and Pebble Reports [1] for reporting.

[1] [http://pebblereports.com/](http://pebblereports.com/)

</comment><comment><USER83> Infra: Postgres -> S3 -> Looker

Business: Segment -> customer.io/Zapier/Heap Analytics + Looker

Support/Success: Intercom+Slack -> frame.ai

<comment><USER84> +1 for Heap – pretty simple to segment, query, and organize data in my experience.

</comment></comment><comment><USER85> Kafka streaming to validate and attach model data, fed into Druid by Flink, queried via a custom front end.

<comment><USER33> > and attach model data

Can you elaborate on this? Do you mean enriching events and/or windowing/aggregating/summarizing?

Are you using KSQL?

<comment><USER85> Yeah, enriching the events as they pass through - so we have a Kafka Connect JDBC source polling the source of truth DB every X seconds for model changes (we looked into Debezium for streaming model changes, but the additional complexity didn't bring us much benefit based on how often our model changes), and then we pull that into a GlobalKTable and join the KStream as appropriate.

We also use Kafka streaming with a persisted event window store to deduplicate across a window of 2 hours.

<comment><USER33> Mind if I ask you what you used to capture source changes if Debezium didn't work out for you? Is there a specific Kafka Connector for JDBC that you're using?

<comment><USER85> Nope, just the JDBC Connector configured for Postgres. It wasn't that Debezium didn't work out of us, more that it brought in more complexity (e.g., [https://debezium.io/documentation/reference/1.0/postgres- plu...](https://debezium.io/documentation/reference/1.0/postgres- plugins.html)) than the payoff justified.

If we had a need for near instantaenous model updates, then I'd definitely go Debezium. We didn't use it because we didn't need it - a generic KC JDBC source with a poll period of 5 seconds met our needs.s

</comment></comment></comment></comment></comment><comment><USER86> We basicallly don't have a BI stack. Which is silly, but I guess we've had other priorities.

<comment><USER19> Better to have "nothing but prod" than something bad, imo/tbh.

</comment></comment><comment><USER87> Client I'm working for:

* SAP -> SAP services -> tableau. * Some depts use Excel -> Python -> tableau.

</comment><comment><USER88> Embulk + Airflow => Postgres => Apache Superset(as UI/Tool)

ELT process so more DBT in the future

</comment><comment><USER89> HubSpot -> AzureSQL -> PowerBI

HubSpot -> AzureSQL -> Tableau

Fully automated syncing with user friendly signup

</comment><comment><USER90> Data warehouse: Postgres Data visualizations and reporting: Tableau, DataGrip

</comment><comment><USER91> We use a simple `Postgres -> CSV -> Pandas` pipeline.

</comment><comment><USER92> ETL: Domo data connectors, Python, SQL

Warehouse: PostgreSQL

Reporting platform: Domo

</comment><comment><USER93> does anyone uses elasticsearch + kibana?

</comment><comment><USER94> apps on pivotal to extract, transform, combine data -> to ms sql db -> powerbi to visualize

</comment><comment><USER95> BI?

<comment><USER96> Business Intelligence. Making reports. Usually over a large set of data. A normal query would be too slow so the results are pre calculated. Maybe you will calculate the average and total sales per city then dump the results into a table. From there you can quickly generate more granular reports like sales per state.

</comment></comment><comment><USER97> Postgres -> QueryClips

</comment><comment><USER93> does anyone use ElasticSearch + Kibana?

<comment><USER98> I do. I work in data consulting. I prefer tableau on top of snowflake for 99% of jobs, but for massively scaling insert performance elasticsearch has a niche. Kibana seems like a mishmash of different products duct taped together, but for the most part plays nice.</comment></comment>


Google vs. Oracle America - <AUTHOR> https://en.wikipedia.org/wiki/Google_v._Oracle_America <comment><AUTHOR> I keep thinking of the potential fallout of copyrightable APIs. I don't know who's right here, but I can't help but think that copyrightable APIs would stifle the proliferation of good software.</comment>


Python + Django + Rpy + R = Web App (by Simon Knapp) - <AUTHOR> https://github.com/Sleepingwell/DjangoRpyDemo

<comment><USER2> This looks like a great project; however, I really wish there were a more descriptive README.

</comment><comment><USER3> There is always a good time for some shameless auto-promotion, so I'll mention Triggr, a package which allows one to convert R process into a fast, event- based XML-, JSON- or even more freestyle RPC server: <https://github.com/mbq/Triggr>

</comment><comment><USER4> Is this just a project with all the glue worked out between these projects?

</comment><comment><AUTHOR> The original post is here: <http://www.linkedin.com/groups/R-web- apps-77616.S.91223876>

</comment><comment><USER5> I know many folks like ggplot2 much more than anything available for Python. What are the advantages of using R over Python?

<comment><USER2> R has exceptional libraries for doing statistical analysis. This kind of setup could be a great platform for building a statistical software suite as a service. Sort of like SAS or STATA or SPSS in the cloud.</comment></comment>


The Art of the Selfless: Better Habits for Better Developed Applications - <AUTHOR> http://uxmag.com/technology/the-art-of-the-selfless

<comment><USER2> Being the salesman personality I already know why we need your skills. Even better to know one personally that says it perfectly how we mesh to compliment one another and why we need each other.</comment>


London Man May Be Cured of HIV After Stem-Cell Transplant, Researchers Say - <AUTHOR> https://www.wsj.com/articles/london-man-may-be-cured-of-hiv-after-stem-cell-transplant-researchers-say-11551746867 <comment><USER2> Dupe of [https://news.ycombinator.com/item?id=19307160](https://news.ycombinator.com/item?id=19307160) which already has a few comments.</comment>


The Secret to Raising Smart Kids - <AUTHOR> http://www.scientificamerican.com/article.cfm?id=the-secret-to-raising-smart-kids

<comment><USER2> A better title would be "The Secret to Keeping Kids from Complacency"</comment>


On-demand pick up of your unwanted items. $5 a bag - <AUTHOR> http://goodandtidy.com

<comment><USER2> I live in Boston and literally just spend the better part of my day trying to make a donation. I packed everything up and dragged my stuff downstairs to the concierge desk in my building, walked 15 minutes to my parking garage to get my car, drove back to my building on a busy street and double parked to quickly load up the trunk. Then I drove what should have been a 10 minute drive to the Kenmore Square Goodwill but turned into a solid 30 min because of all the traffic from BU's graduation ceremony. When I got there, all the parking spots were blocked off and police were forcing traffic to keep flowing. I went around the block several times looking for parking before deciding it was pointless. I quickly Googled the next closest Goodwill store, which is in Jamica Plain, and decided to go there instead. There's no graduation in Jamaica Plain today but the Goodwill store there is also on the main road with very little parking...

<comment><USER3> You can always donate to Boomerangs with a little more ease. They're a few buildings away from the Goodwill on Centre and you can pull into their back parking lot while you are dropping off. Cool store, too.

</comment></comment><comment><USER4> As a Midwestern suburbanite, I don't think I understand this service. When it comes to trash removal, the city already comes around once per week and collects whatever I put out there. The cost of this is baked into my city taxes and can't be opted out of so far as I know. There may be some technical limit as to how much they'll take at once, but I've never hit it. They even come around and take big stuff, like couches and mattresses, a few times per year. Is this type of service not common throughout the US?

This is an interesting idea for donatable goods like clothes, but I lose the write off and I'm not sure waiting around for a pick-up is really more convenient than just driving the stuff to a drop off location myself.

The only real advantage I see here is getting rid of stuff that's hard to dispose of: car batteries (you know, cause I'm always throwing those away), poisonous or flammable liquids, medical waste (which I don't think I've ever had), etc... but I can't believe that is profitable cargo for them to haul away at $5/bag.

What am I missing here? Am I just the wrong demographic for this?

<comment><USER5> I feel similar in some ways. I live in an urban apartment building, we have a "recycle room" where I can leave anything I can produce and it magically goes away. Having said that, when I rented in a different part of the city we had a very small bin and shared it with the apartment downstairs. The bin would often fill up and I'd find myself holding onto trash because I had no room left for it and no time to drive to the dump. In that case $5 would have felt like a good deal!

</comment></comment><comment><USER6> I just returned home from hauling most of my clothing to the local Goodwill. On a related note, I look forward to seeing more people wearing startup/conference t-shirts around NYC, since they comprised the lion's share of my wardrobe...

I've been researching these services for the past few weeks since I've committed myself to liquidating most of my belongings (I live in a small apartment with no closets -- thanks New York! -- and was inspired by this blog post: [http://www.raptitude.com/2011/01/i-dont-want-stuff-any- more-...](http://www.raptitude.com/2011/01/i-dont-want-stuff-any-more-only- things/)), and would prefer to see them in the hands of friends and those in need rather than a landfill.

Does anyone have recommendations for where/how to donate more valuable items? For example, I want to donate an expensive drone I never use, but would prefer to see it go to a classroom/after-school program. Thinking I'll search DonorsChoose.org and offer things to teachers, but is there another service?

</comment><comment><USER7> "Anything that fits into a bag."

I've got a propane torch, a fire extinguisher, various household cleaning supplies, and about 100 lb of ceramic tile to get rid of. And I have some 3 mil "contractor bags" than can handle more weight than I can carry.

My current plan is to drive the hazardous waste to recology. If someone could do it for me it'd be nice, but I'm not sure that's even legal. Still working out the tile thing. Not sure Habitat would want it. I've also got a bunch of paint, but I know where to take that.

<comment><USER8> Latex paint can usually be disposed of in normal garbage. Just open the cans and let the paint dry out first.

<comment><USER9> Some cities have a household hazardous waste disposal location where one can drop off used chemicals, paint, etc. They will separate stuff that can be reused and properly dispose of the rest. The one in our city, we just backup our car and they take out the stuff while you sit in the car and complete the paperwork and show an id.

</comment></comment></comment><comment><USER10> This really could be Uber for trash.

I pay something like $50 a month for trash collection services, or around $12 a week. That includes yard waste and recycling.

If I can fit my trash and recycling into two bags per week and find something else to do with my yard waste, I'll save money by using GoodAndTidy for weekly trash pickup.

I like their website and the idea, but it seems like a service that may have trouble becoming profitable based on their $5/bag fees.

<comment><USER11> I posted my uber for trash comment first, ironically.

I don't think you're kidding though, are you.

</comment></comment><comment><USER12> In Paris it's free, you just have to go on this website and tell what you are throwing away:

[https://teleservices.paris.fr/sira/jsp/site/Portal.jsp?page=...](https://teleservices.paris.fr/sira/jsp/site/Portal.jsp?page=formengine&form=ramen)

BTW in the URL: "&form=ramen" I suppose that the developer was hungry.

<comment><USER13> Probably a concatenation of RAMassement d'objets ENcombrants.

</comment></comment><comment><USER5> I'm struggling to find a privacy policy, is there one? This could be the cynic in me, but I feel there is more to learn from my unwanted items than almost any other data source (aside from my email?). Gaining information about people from what they discard is possibly a great business model, but I think the situation should be made clear in a privacy policy.

Edit: Changed trash to "unwanted items"

</comment><comment><USER14> My first thought from "properly dispose of, we don't want it in landfills" was, this is for hazardous waste -- you know, that box of old paint cans and dead SLA batteries that you know your city has a plan for, if only you could be bothered to figure it out. If not, maybe the website should make that more clear.

</comment><comment><USER15> "Currently in the San Francisco Bay Area." Does not pick up in Palo Alto, Mountain View or Foster City. That's the just few addresses I've tried. A service area map would be helpful.

<comment><USER16> Also, the whole thing is on a *.com domain and yet from your comment only services a portion of a single city? I am disappointed.

</comment></comment><comment><USER17> Where I last lived, there was at least one charity that had free pickup by truck. Not on demand exactly, but they could take anything useful if it was boxed- or furniture. You could leave your stuff on the porch and they would shift it for you.

This seems strictly worse and very expensive if you want to get rid of any significant amount of stuff.

<comment><USER18> This one takes none-useful items too. The reason I want to get rid of the stuff is that it is not useful (unless you really want an old CRT tv or a case of broken computers, I guess).

</comment></comment><comment><USER19> They just updated their text and dropped San Francisco from the list of supported areas. Currently they are only in Berkeley.

I would use this service. I easily have couple bag of not-quite-trash that wants to be dealt with. But really, what are they going to do with an old Nokia cellphone charger? Or a 24" LCD monitor with DVI and VGA input but the HDMI input is broken. Or slightly damaged clothes with rivets and zippers I keep meaning to repair but have never gotten around it, so might as well throw out.

For $5, I can't imagine I'll get any sort of post-processing report about what they got out of my crap or what they managed to do with it.

Hell, I've got a bunch of crap that I keep meaning to put up for sale on ebay/craigslist that I never get to. I doubt I'm alone in that regard.

</comment><comment><USER20> In Mountain View, and likely elsewhere in the Bay Area, charitable organizations make the rounds once a month to basically do the same. They provide the bags, collect them, and presumably sell what they can for cash. As for batteries and other hazardous materials, there's a protocol to have the city pick them up for no charge. I've never had a problem getting rid of stuff and feeling like I'm doing it responsibly. It's hard to understand what value this adds beyond immediacy.

At least with the services I already take advantage of, it's all done in batches and my contribution is low compared to the overall haul. This is on- demand, so it's likely not as efficient.

<comment><USER21> Any links you can share to these services? Will have a bunch of stuff to give away soon and I'm not familiar with the various organizations in MV that help with this.

<comment><USER20> [http://donatedav.org/donation-pickup.html](http://donatedav.org/donation- pickup.html)

<comment><USER21> That's awesome, thanks for the link! Are there other non-profits that have similar services or just DAV?

</comment></comment></comment></comment><comment><USER22> I believe CRTs are normally ~$20 each to dispose. I wonder how many I can fit in a bag.

<comment><USER23> Just drop them off at Best Buy, they claim to recycle them...

</comment></comment><comment><USER24> Would be nice if they gave receipts for tax purposes, like the Shyp/Goodwill partnership did: [http://blog.shyp.com/goodwill/](http://blog.shyp.com/goodwill/)

<comment><USER25> Considering they are charging $5 a bag,I doubt they are a nonprofit.

</comment></comment><comment><USER26> I was looking for extra information so I entered my email into the field. Figured they'd sign me up for a mailing list like what all the sites out there do.

I was surprised to see it say "Thanks, expect an email from us first thing in the morning!"

I can't find a way to cancel the request. Hopefully it'll be in the email. I understand the need for rapid on-boarding but this was a bit of a surprise.

<comment><USER27> I'm sure they get a million I don't have anything after all replies to their morning emails if that's how they operate.

</comment></comment><comment><USER28> I would use this. I have a bunch of old batteries, light bulbs, and random machine parts thats I been meaning to recycle but never get around to it. If these guys would properly recycle those things for me, I would pay more than $5 because I am trying to be conscious about what I put in the landfill.

</comment><comment><USER29> The District of Columbia gives each household a couple of large-item pickups every year. (Large item example--old storm windows, say 4 feet by 2.5.) Latex paint, as someone remarks, can be allowed to dry out and dumped with regular garbage. The Fort Totten transfer station accepts nastier stuff--solvents, oil-based paint, electronics--one Saturday every month. And with Goodwill on South Dakota Ave. NE and Community Forklift in Hyattsville one can get get rid of pretty much anything still usable.

</comment><comment><USER30> So they pick up someone's trash and give it to some non-profit as a donation?

Poor non-profits. It will cost them more to sort through and dispose unwanted items than the value received.

<comment><AUTHOR> Sounds like they're sorting through the contents: "We'll give away or properly dispose of your items."

<comment><USER30> Disposing items properly costs money. Donating items doesn't. So the incentive for them is to "donate" as much trash as possible. Then non-profit org will have to deal with it.

Sorry, I just don't like non-profits being treated as a free dumping site.

<comment><USER31> I hear your concern and I hope they have no intention of treating non-profits as a free dumping site. I would like to give them the benefit of the doubt and not just assume the worst of them.

</comment></comment></comment></comment><comment><USER32> As my wife and I are currently going through Mari Kondo's [http://tidyingup.com/](http://tidyingup.com/) process, this would be great to have in London.

Recycling is generally great here, but there are certain things that are actually quite valuable, but are just way too much hassle to get rid of one- by-one.

</comment><comment><USER33> I'm not fully sure why someone would use this. Is it for donating stuff, or garbage disposal?

For my unwanted items, I donate them via [https://givebackbox.com/](https://givebackbox.com/) \- which gives you a pre- paid UPS shipping label and lets you ship to Goodwill for free.

</comment><comment><USER34> I'm moving and have recently sold and donated a three bedroom house full of stuff rather than haul it.

Maybe this a specifically urban need? Up here in my small mountain town (< 4000 pop.), Goodwill, Hospice, Hospitality House (homeless shelter), and Habitat for Humanity all pick-up free as do the thrift stores.

<comment><USER35> Same options exist in Columbus, Ohio. You just have to work around their schedule, they only come to certain parts of town on certain days of the week/month.

</comment></comment><comment><USER36> This seems good. I like that this has the potential to expand the total pool of donated items and help reallocate resources to people who need them, while also helping people clear out their homes too.

</comment><comment><USER37> Free in most of the Europe. I am surprised this one will be paid tbh.

</comment><comment><USER11> It's uber for the waste management business.

Instacartage.

Asbestos. Redefined.

Or something.

For those of us that lived through the first dotcom bubble this is all getting more and more familiar.

</comment><comment><USER38> No no no! Good-and-Tidy just doesn't get the business model here. They need to do it for free, and make it up on volume.</comment>


Ask HN:  How much rent can I afford in San Francisco? - <AUTHOR>

I'm moving to San Francisco soon and I'm just trying to get a rough idea of what I can afford.  The general rule of thumb is rent should be 33% of income, but that seems a bit tough in SF with rent prices what they are.  What do most landlords go by? <comment><USER2> If you are willing to be flexible in terms on location, you can get cheaper rents but still be close to the city proper via public transit. Places down in the peninsula will be even cheaper, but you will have to factor in the price of the commute (caltrain from mountain view to SF everyday can get expensive). There are also the google shuttles, but I have not used them.

Living with room-mates will also save you a bunch of money obviously. I am down in Palo Alto right now but thinking about moving up to SF, so long as I can convince the people I want to live with to find a place cheap enough for me. Good luck!

</comment><comment><USER3> Landlords go by whatever the market will bear. SF is a great place to live and everybody living there is willing to pay more to do so.

My advice is decide what minimum qualities you must have in a place and find something that meets or exceeds those for the least money.

But try to make a short commitment initially. You will doubtless learn a lot more about the market and your own needs once you're here.

</comment><comment><USER4> The standard formula used by rental companies for tenant approval is the combined incomes of the renters should be 40x the monthly rent of the apartment. Of course, preference is given to those making more money. If it's being rented by the owner there is usually more room for negotiation.</comment>


Browser Fingerprinting: An Introduction and the Challenges Ahead - <AUTHOR> https://blog.torproject.org/browser-fingerprinting-introduction-and-challenges-ahead <comment><USER2> Fun fact about the example fingerprint in the article:

On almost any other distro you're just going to see "Linux" for the platform. The reason "Fedora" is in this particular one is that there are packages that are installed by default on both Firefox and Chrome that add it to the useragent string for no reason.

I know this because, for a few glorious days, there was a bug in the Chrome plugin that meant that there was another "Fedora; " added for every redirect in a redirect chain (i.e. "Fedora; " to "Fedora; Fedora; " etc.), which broke a whole bunch of whitelists and fingerprinting bot detection scripts.

There's absolutely no reason for it, there's no benefit to having there, but everytime someone files a bug about it the maintainers don't see what the problem is and WontFix the bug. [1]

[1]: [https://bugzilla.redhat.com/show_bug.cgi?id=1266569](https://bugzilla.redhat.com/show_bug.cgi?id=1266569)

<comment><USER3> User agent strings discourage feature detection and proliferate as browsers converge and imitate one another. They should be dropped entirely.

Sadly maintaining exceptions for broken sites that demand them is too much for one person.

</comment></comment><comment><USER4> I did a little bit of research on browser fingerprinting years ago and even tried to write my own library to do it and ever since, I've always questioned how useful it actually is.

Using the list of measurement points on [https://amiunique.org](https://amiunique.org) as guide most of the things that are constant about my computer like platform, browser, or requested language are not really unique to me and are shared by a large percentage of the other users who have come to the site.

On the other hand, most of the data points that are unique to my machine change semi-frequently. User agent and version change on browser updates, timezone changes when I travel, screen size and resolution change when I plug into my external monitor, new fonts will slowly be installed over time, and even things like how the canvases are rendered can change slightly depending on how much strain my GPU is under at the time I get fingerprinted.

Just plugging in to my external monitor was enough to get amiunique to treat me as a different user. (if you want to try, be sure to clear your local storage and cookies in between visits as the site saves a uuid there and will serve you your previous results if it finds it).

I'm sure there's some magic formula that gives different weights to different data points that can give a decent guess at who you are, but I doubt it can say with 100% accuracy that you are who it thinks you are.

It seems to me all it would take to defeat fingerprinting is a browser extension that modifies the browser apis to randomly slightly alter the requested data (add a random font to the list, add some nonsense to the user agent, etc). Sure, the fingerprint would still be unique, but it would be unique on every visit which would defeat the ability to track a user across visits.

*I'm not an expert on this subject at all, so if I got something wrong, please correct me

<comment><USER5> Cookie + IP Address + Fingerprint + URL Referral + Social Media

It's possible to be anonymous if you only log into websites with proper privacy policies, delete cookies & cache every day, and hide behind carrier NAT.

Adding a random font shouldn't stop fingerprinting, people install new applications all the time. You make a lot of points, but ad tech can get around it.

Ideally ad corporations would be forced to delete records on people after 30 days.

<comment><USER6> > delete cookies & cache every day

I'm browsing everywhere in Safari's Private mode by default, 3 years no problems.

<comment><USER7> It is becoming increasingly difficult though. Google captchas everywhere. Authentication requesting 2 factor because it thinks it is a new browser. The web is progressively becoming unusable unless you are loaded with cookies and tracked. And the culprits are the likes of Amazon, Cloudflare, Google, etc. Not easy to move them.

<comment><USER6> Cloudflare somehow got better lately, I'm curious to know how :) Google search works fine and I'm not using other Google services. I see surprisingly few captchas. Yeah, I'm on VPN too, maybe they treat it differently.

<comment><USER8> Better at what?

<comment><USER6> Less captchas.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> One case of possible fingerprinting I encountered in an ad recently surprised me by how many different obscure tests it seemed to perform. The entire script is 80kB minified, and it looks like it's just hundreds of tests that check for all kinds of properties of the environment.

Many of the test seem to look for the presence of specific APIs, or test various features of the JS environment. Which I'd expect to not give more information than the user agent, but of course is not as easy to fake as the user agent it.

I'm not sure this script is entirely about fingerprinting, but I really can't see any other reason for it. Though the minification of course makes it a bit harder to see exactly what it does.

The script in question is the following:

[https://static.adsafeprotected.com/sca.17.4.95.js](https://static.adsafeprotected.com/sca.17.4.95.js)

<comment><USER10> While it's indeed used for "ad tracking", this snippet is actually closer to a white-hat tool. This is from Integral Ad Science [https://integralads.com/](https://integralads.com/) which attempts to detect ad fraud. If you hate ads, it won't matter, but in the fight to make ads less invasive, potentially more useful, and still viable as an economic model for those who want to participate with it, Integral and WhiteOps ([https://www.whiteops.com/](https://www.whiteops.com/)) both do some great data work to try to stop disruptive bots.

If one hates all ads and tracking, of course, then none of this will matter, but just thought I'd point out the "why" for the fingerprinting in this case.

</comment></comment><comment><USER11> Tor's anti-fingerprinting measures are available in Firefox. They're off by default, but you can enable them by going to the about:config page, searching "privacy.resistFingerprinting" and setting to "true"

Interesting discussion from six months back: [https://news.ycombinator.com/item?id=19323032](https://news.ycombinator.com/item?id=19323032)

<comment><USER12> Even with `privacy.resistFingerprinting` enabled, my fingerprint is unique all-time on [https://amiunique.org](https://amiunique.org).

<comment><USER13> I wouldn't put much stock in amiunique right now. It might be great, but needs a bigger sample size.

I am unique purely based on my content language: "en-AU,en,en-US"

While we like to think we're special in Australia, I'm not THAT special.

</comment></comment></comment><comment><USER14> > all Tor users should have the exact same fingerprint

Wouldn't it be better to impersonate at random a different combination of fingerprints for each page? Otherwise any browser with this fingerprint would be risking being blocked for being part of the Tor network.

</comment><comment> seph-reed The link to amiunique.com was pretty interesting. In particular, I enjoyed the webgl render. Hardware/driver dependent things such as that are.. a conundrum of anonymity.

<comment><USER15> If someone didn't read the OP and tried to go to the amiunique, the website is dot org

[https://amiunique.org](https://amiunique.org)

<comment> seph-reed ah. bollocks. thank you for the correction.

<comment><USER16> Tis editable, albeit difficult to find, in case you weren't aware (click timestamp of the comment you'd like to edit).

</comment></comment></comment></comment><comment><USER17> As long as browsers are heading into the direction of more and more functionality, APIs and complexity, I get the feeling that being un- fingerprintable is going to be a lost battle. The best you can do is to be a unique device every single time and thereby defeat server-side profile building.

</comment><comment><USER18> I have "resist fingerprinting" set on Firefox, but it doesn't seem to do much good. Any suggestions other than to entirely turn off JS, which to me seems akin to not using the majority of the web?

<comment><USER19> I accept that this reply is basically saying "Turn it off", but look at umatrix. You can allow/block JS by default but then add certain hosts or domains to the allow list.

I browse with JS disabled, but when I hit a site which doesn't work properly (and which I want to use still), I can allow only the domains which it needs to work.

It's quite surprising how quickly many sites now load!

</comment></comment><comment><USER20> I was interested in hearing what they did to prevent fingerprinting with regards to things like window size, but it seems like they just discourage the user from changing it :( It looks like this isn't going to be improved anytime soon.

<comment><USER21> I would love a button or menu action to RESET my window to the standard size in regular Firefox but so far I can't find any easy way to do it. You'd think an extension would do it but when I checked, all the extensions related to window size did different things.

<comment> toper-centage I guess some window managers like i3wm let you do just that.

</comment></comment></comment><comment><USER22> As long as browsers are inherently insecure in regards to fingerprinting protection, the only viable protection is to create an extensive list of known JS fingerprinters.

Additionally one needs to disallow all third party javascript sources by default.

Both strategies are possible with a good content blocker.

These strategies protect against all known and almost all unknown fingerprinting scripts. The only scenario where it doesn't work is unknown first-party scripts, but cross-site tracking is impossible.

</comment><comment><USER23> > TBB masks the underlying OS by claiming it is running on a Windows machine.

Note that for some time now TBB on Linux identifies as Linux in both the platform string and the useragent.

Edit: the platform string is mentioned later in the article.

</comment><comment><USER24> I've been using this[1] for a while, not so much for the privacy, but to segregate internet things from my desktop user.

      1: https://gitlab.com/edneville/newuserbrowser

</comment><comment><USER25> The advantage is that if you block all the cookies you can get a new identity by just resizing your window.</comment>


Google/Motorola and SpeakerBlast - <AUTHOR> http://ryanspahn.com/my-google-NDA-experience.html <comment><AUTHOR> For fun, read the comment pages at [https://news.ycombinator.com/threads?id=LargeCompanies](https://news.ycombinator.com/threads?id=LargeCompanies) for more history. (hint, there is a reason why the account is named like that)</comment>


How to start a live video streaming website and mobile app? - <AUTHOR>

<comment><USER2> Ignore the technology and do some research to find out if there's a market for what you want to build.</comment>


Share your desktop as a webcam in OBS, Hangouts, Zoom etc. - <AUTHOR> https://lukespear.co.uk/blog/share-desktop-as-webcam-in-obs-loopback/ <comment><USER2> On iOS you can use the NDI HX capture app (currently free) to stream via low latency NDI protocol to your PC. There's a video guide at Newtek.com.

App link: [https://apps.apple.com/de/app/ndi-hx- capture/id1501247823](https://apps.apple.com/de/app/ndi-hx- capture/id1501247823)

Their NDI HX camera app is also currently free. It turns your iPhone / iPad into a low latency wireless webcam.

You can use these with the OBS NDI plugin or the software provided by Newtek which turns an NDI source into a virtual camera.

<comment><USER3> Just what I was looking for. It is quite literally cheaper to buy a used or cheap smartphone than a webcam right now.

Is there anything this good an Android? There's a lot of apps that promise to do it but they look a bit suspicious and I'm wary about putting them on my phone.

<comment><USER4> It says iOS 13.1 or later required. Brand new or refurbished at least, iOS devices that can run iOS 13 aren't that cheap yet. So yeah for sure I'd be interested in an android equivalent. Little knowledge of the ecosystem but I assume getting an okay Android device mostly just for this would be pretty cheap.

</comment></comment></comment><comment><USER5> The title of this is confusing. It should be something like "Share OBS output as a webcam in Hangouts, Zoom etc."

OBS can do vision mixing between Webcams, pre-recorded video files, Screen shares, NDI streams etc. with picture in picture, chroma key and many other features.

I actually wanted to do this about 10 years ago. I posted a question on superuser and it's still getting upvotes now: [https://superuser.com/q/78515/742](https://superuser.com/q/78515/742)

</comment><comment><USER6> I'm on ubuntu 18.04 and haven't been able to get v4l2loopback related setup to work with OBS using the obs-v4l2sink

* With the packages OBS Studio or the newest compiled

* With the DMKS v4l2loopback kernel module or the newest compiled

EDIT: Thank you for helpful comments. I managed to get this to work by:

1\. Purging my existing v4l2loopback-dkms

2\. Compiling latest v4l2loopback kernel module

3\. Compiling latest OBS Studio

4\. Compiling latest obs-v4l2sink plugin

5\. sudo rmmod v4l2loopback

6\. sudo modprobe v4l2loopback devices=1 video_nr=10 card_label="OBS Cam" exclusive_caps=1

7\. unplugging my webcam and plugging it back in

If you're currently having problems, before trying 1-4, try 5-7. It seems like v4l2loopback is getting loaded on boot, and some combo of that and my existing webcam was messing this up. Once I rmmod the kernel module and put it back, capture works but my webcam is gone. When I unplug my webcam and put it back, it all works. It may very well work fine with the standard v4l2loopback-dkms and obs packages without having to compile latest.

<comment><USER7> Weird, worked pretty out of the box here besides some of the v4l2 parameters for application compatibility. My system is a bit outdated though, v4l2loopback-dkms 0.9.1-4ubuntu0.1 and OBS 24.0.3 (64 bit, installed through whatever the quickstart guide suggested for Ubuntu).

I'm using this to initialize the loopback device:

        #!/bin/bash
        set -e
        if lsmod | grep "v4l2loopback" &> /dev/null ; then
            echo "v4l2loopback is loaded"
            lsmod | grep "v4l2loopback"
            sudo rmmod v4l2loopback
        fi
        sudo modprobe v4l2loopback devices=1 video_nr=2 exclusive_caps=1
        lsmod | grep "v4l2loopback"

<comment><USER6> I think you led me to the problem. I updated the original comment, but I think the issue was that v4l2loopback was being loaded without any options and screwing me up, so rmmod ing it and reloading it seems to help

<comment><USER8> You can set default module options through a file in /etc/modprobe.d/. If you're using systemd, it can also be configured to load the module automatically on startup. This is what I'm using:

/etc/modprobe.d/v4l2loopback.conf:

        options v4l2loopback devices=1 video_nr=10 card_label="OBS Cam" exclusive_caps=1

/etc/modules-load.d/v4l2loopback.conf:

        v4l2loopback

Working flawlessly so far in Hangouts and Zoom using OBS Studio and the obs-v4lsink plugin.

</comment></comment></comment></comment><comment><USER9> I'd been looking into OBS/obs-virtualcam until I discovered that macOS support wasn't there.

Not free, but I have successfully tested ManyCam, which I believe will work well for my streamlining the process of switching between screenshare and secondary camera sources for Zoom.

<comment><USER10> I made a plugin that sorta works for this last week: [https://github.com/johnboiles/obs-mac- virtualcam](https://github.com/johnboiles/obs-mac-virtualcam)

<comment><USER4> Awesome! Even if it is buggy or won't work for me when I try it. Great that you already have this shipped. Not saying it is buggy. Just if it is, no problem having this out already!

Been waiting for something for OBS for a while. since 2019 when Mojave made Syphon injecting much harder due to security changes. And demand for it and any other similar injecting almost entirely waned.

</comment></comment></comment><comment><USER11> Is there something I can use besides Adobe Character Animator to stream a cartoon version of myself to Hangouts and Zoom? Something that talks when I do, uses my webcam to copy my movements, press keys to move arms, etc?

<comment><USER12> Snap Camera comes close.

[https://snapcamera.snapchat.com/](https://snapcamera.snapchat.com/)

</comment></comment><comment><USER13> I use this solution in a similar way, but I usually share a single window. I use this command, change /dev/video2 with your virtual output and WINXID with the XID of the window you want to view (use xwininfo to get it).

      gst-launch-1.0 ximagesrc use-damage=0 xid=WINXID ! videoconvert ! video/x-raw,format=YUY2 ! videoconvert ! queue ! v4l2sink device=/dev/video2

Also, I usually combine it with scrcpy, so I can share my Android phone/tablet's screen on my computer's videoconference session.

</comment><comment><USER14> It's also a pretty good solution if you just want to display a static image or loop a 5 minute clip giving the appearence of having video enabled and making your screen more identifiable.

<comment><USER15> Just a five minute loop of thoughtful nodding and the occasional brow furrow.

</comment></comment><comment><USER16> Just use webcamoid and get it done and over with. It is less hack.

<comment><USER17> You say that like it's a quick install. You have to build it on your machine apparently:

[https://github.com/webcamoid/webcamoid/wiki/Raw-build-and- in...](https://github.com/webcamoid/webcamoid/wiki/Raw-build-and-install)

<comment><USER18> The release page doesn't have what you want?

[https://github.com/webcamoid/webcamoid/releases](https://github.com/webcamoid/webcamoid/releases)

<comment><USER17> At first glance, those might work, but when I go to the README, there's no mention of those binaries, and it links to the wiki under "Build and Install".

However going to the project website at [https://webcamoid.github.io/](https://webcamoid.github.io/) it does link to the executables, which is a little clearer.

I guess I've gotten used to going to the Github repo for projects more than most would.

</comment></comment></comment></comment><comment><USER19> I use SCFH DSF[0], updated for almost a decade and English support is a bit limited but works mostly fine

[0] [http://mosax.sakura.ne.jp/yp4g/fswiki.cgi?page=SCFH+DSF](http://mosax.sakura.ne.jp/yp4g/fswiki.cgi?page=SCFH+DSF)

</comment><comment><USER20> Good timing on the article - I was just setting this up yesterday with some luck (Snap Camera doesn't seem to respect the virtual cam instance) but still very cool.

</comment><comment><USER21> What's the use of this? I'm genuinely confused. Can't you already share your Desktop in Hangouts/Zoom/etc..?

<comment><USER22> I don't get it for OBS, but for the teleconf apps it can let multiple people share screens at the same time (rather than taking over presenter role), and conceivable work better in browser versions that don't support desktop sharing (I think chromium requires an extra plugin).

Edit: also, the page conspicuously mentions several times that it can be used for "educational purposes" to steam video running on the desktop, so it seems the intention is pirating video.

<comment><AUTHOR> A little uncharitable to say the entire intention is pirating, especially given the global pandemic, quarantining and social issues this can help with.

But yes, showing legal or illegal copies of movies you own or don't own is possible if one were so inclined, among the myriad legit uses. I can see why it looked like that was implied.

<comment><USER22> If the page hadn't been so "wink wink nudge knudge" about it, you might have a point. But they went out of their way to mention "for educational purposes" in some very awkward ways, several times.

</comment></comment></comment></comment><comment><USER23> This is a godsend. Zoom screen share doesn't seem to work at all on Wayland.

</comment><comment><USER24> you can already share your desktop in zoom

<comment><USER15> Right, but this is about streaming something else instead of the webcam video.</comment></comment>