Generate random fake data and populate your application for easier development - <AUTHOR> https://random-data-api.com/ <comment><USER2> I've tried this and other libraries like Faker and in my experience it's not worth it for anything besides maybe testing design mockups. When a project is small, I learned I might as well just launch early and test with real data. When a project is not small, a lot of time is spent trying to replace all the fake calls and data intertwined with what's real and getting confused in the process.

</comment><comment><USER3> I also recommend [https://robohash.org/](https://robohash.org/) to generate avatars/photos for your various fake users.

I noticed their API has a low rate limit (which is fair for a free service), so I make sure to add some small delay (2-3 secs) to my seeding / fake data process so that I don't hit it too frequently.

</comment><comment><USER4> There is a typo in the FAQ. It reads, "A sa single developer,..."</comment>


Partially Derivative: The One About Sarah Connor - <AUTHOR> http://www.partiallyderivative.com/news/2015/1/16/episode-10-the-one-about-sarah-connor

<comment><USER2> The podcast is as entertaining as it is informative. It's really a goldmine of good resources/ideas and it's my favorite podcast to date.

<comment><USER3> Thank you so much!

</comment></comment><comment><USER4> Derives me crazy!great podcast! Thanks!</comment>


Show HN: An open-source and decentralized COVID-19 contact-tracing app - <AUTHOR> https://github.com/RaphaelJ/covid-tracer/blob/master/README.md <comment><AUTHOR> Hi, OP here,

I started working on this app a couple of weeks ago, trying to iterate on Singapore's contact-tracking app by adding data anonymization and decentralization features.

I ended up with a solution that is similar to the D3P-T, with some implementation differences.

The app is available as an APK for Android [1]. As for now, the iOS app is not easy to distribute, as Apple does not allow coronavirus-related apps to be published on the app store.

\--

[1] [https://github.com/RaphaelJ/covid- tracer/blob/master/release...](https://github.com/RaphaelJ/covid- tracer/blob/master/release..).</comment>


Launch HN: Hubble (YC S20) – Monitor data quality inside data warehouses - <AUTHOR> Hey everyone! We're Oliver and Hamzah from Hubble (https://gethubble.io/hn). Hubble runs tests on your data warehouse so you can identify issues with data quality. You can test for things like missing values, uniqueness of data or how frequently data is added/updated.
We worked together for the last 4 years at a startup where we built and managed data products for insurers and banks. A common pattern we saw was teams taking data from their internal tools (CRM, HR system, etc.), application databases, and 3rd party data and storing it in a warehouse for analysis. However, when analysts/data scientists used the data for reports they would spot something suspicious and the engineering team would have to manually go through the data pipelines to find the source of the problem. More often than not it was simple things like a spike in missing values because an ETL job failed or stale data because a 3rd party data source hadn't updated correctly. We realised that reliability/ trustworthiness of the raw data was essential before you could start abstracting away more interesting tasks like analysis, insight or predictions.
We wanted to do this without having to write and maintain lots of individual tests in our code. So we built Hubble, which connects to a data warehouse and creates tests based on the type of data being stored (i.e. freshness of timestamps, the cardinality of strings, max value of numbers, missing values, etc.). We've also added the ability to write any custom tests using a built-in SQL editor. All the tests run on a schedule and you'll get an email or slack alert when they fail. We're also building webhooks and an Airflow operator so you can run tests immediately after running an ETL job or trigger a process to fix a failing test.
Instead of asking users to send their data to us, the tests are run in the data warehouse and we track the test results over time. Today we support BigQuery, Snowflake and Rockset (which lets us work with MongoDB and DynamoDB) and are adding more on request.
We're planning on charging $200 a month for a few seats, and $30-50 for extra users after that.
We're still at an early access stage but want the HN community's feedback so we've opened up access to the app for a few days, you can try it out here https://gethubble.io/hn. We've added a demo data warehouse you can start with that has data on COVID-19 cases in Italy and bike-share trips in San Francisco. Thanks and looking forward to hearing your ideas, experiences and feedback! <comment><USER2> Customer here (comment not solicited!). We've been trying out Hubble for a month or so and it's looking really promising.

I love the idea of being able to outsource the creativity/problem solving of predicting things that could go wrong with our data to a service that specialises in just that, and I can totally see how they can automate this in a big way as they grow.

</comment><comment><USER3> How does hubble compare to Great Expectations or DBT for pipeline testing? It looks like more emphasis on automated profiling than "having to write and maintain lots of individual tests" and obviously hubble being a saas offering is the big difference?

Also any plans to profile and test file-based stores as well? There's a lot that can go wrong in a pipeline before data even reaches BigQuery or Snowflake, and you may help your customers save money if you could profile data in S3 before it goes through a potentially expensive transform process.

Best of luck, though! Data testing is a very real need in most data organizations I've been in, and I'm glad more and more tools seem to be popping up recently to help with it.

<comment><AUTHOR> Thanks! We love DBT and take a lot of inspiration from their work. We're putting a lot of effort into suggesting the right tests based on the data types, sources, and field names. A lot of these tests are pretty repetitive to write so we want to make it easy to spin them up.

We've also found that keeping a history of the state of the warehouse over time is really useful context for determining whether a test has failed (example: this table tends to update every 30-40 minutes so we'll set a threshold at an hour).

We also handle the scheduling, which is surprisingly annoying to manage (we built a couple of internal tools for this in the past). That's something we really missed with great expectations (you get this with DBT cloud). Testing files is an interesting use case, to an extent we support this using Athena or Bigquery external tables for json/csv/parquet. We're intentionally limiting it to SQL for now.

<comment><USER4> Very interesting tool, I am trying to do this with Dataform/Looker, and feel like some kind of inference like below would be great.

> this table tends to update every 30-40 minutes so we'll set a threshold at > an hour

Can you achieve these tests with metadata or do you need 100% read access to the database?

I also wonder if this would work as part of a Analytics Engineering CICD process? Something like how dbt cloud will block pull requests that fail certain criteria.

<comment><AUTHOR> Metadata is a valuable place for finding information like load times, rows inserted / updated. Currently we just rely on read-access and raw SQL. A common way users are doing this now (and we are internally for our analytics data) is using, for example, the Fivetran logs table to monitor ingestion times and inserted rows, rather than querying the raw tables.

For CICD, absolutely we want to support this as well as stopping/conditional execution in DAGs (e.g. airflow). We're launching webhooks very soon

</comment></comment></comment></comment><comment><USER5> this is interesting! running tests on data is certainly a pain point for me, and there doesn't seem to be nearly the kind of infrastructure available as for, say, tests for code functionality.

Is this open source? Sending my data to a third party is a no-go, as is having a third-party connect to the database. Something part of a managed hosting service, though, or an add-on to an existing trusted hosted service that has gone through compliance (e.g. Heroku, AWS), would be more palatable.

<comment><USER6> This was the same pain point we had when we saw how good the tools were for testing our software vs our data.

It's not open source but we can deploy on-prem (or cloud-prem more accurately) pretty easily. We're also going to setup as an add-on available through AWS marketplace. Feel free to shoot me an email if you want to see if this can work for you hamzah[at]gethubble.io

</comment></comment><comment><USER7> Running a full table scan on BigQuery every hour can get quite expensive. Do you support some sort of deltas?

I signed up. Unlike the video, I do not see Redshift as an option. Any idea when Redshift will be supported?

How does billing per user make sense here? What prevents me monitoring thousands of tables under single user? Your workload costs will be higher than $200 here, no?

Do you have a set of fixed IPs you're connecting from to allow me to whitelist you?

<comment><AUTHOR> Full table scans can get expensive. We're adding support for incremental tests so for append-only tables you'll only test the recent rows. This is especially useful if you use partitioned tables in bigquery.

Actually in the first version of the product we automatically tested every column in every table. The tests are more selective now, which is partially due to cost and partially because nobody wants to navigate through 10,000 tests.

Redshift will be supported this week! We have a list of new sources to get through and it's right at the top. We've been emailing over the IP for whitelisting but we'll add it to the connection page too.

As for pricing, we're experimenting. Our costs do scale with number of tests (more scheduled tasks, more historical results stored). At the moment we retain the last month or so of test results, which is manageable for pretty large workloads.

<comment><USER7> Looking forward to Redshift!

BTW, you don't need to navigate 10K tests... you only need to navigate the failing ones.

</comment></comment></comment><comment><USER8> Co-founder of intermix.io here (which we sold in March). We came more from the performance monitoring angle (specifically for Redshift), but then shifted to a product that works horizontally across all warehouses, to track usage, workflows and user engagement. "Shift to Data Products" was the narrative we started using in Q4 2019. If you read the copy on the current intermix.io website, I think you'll find yourself nodding. (FYI - we got bought by a small PE Fund that is rolling the product into Xplenty, an ETL product).

My experience is that monitoring data quality is a still an under-appreciated discipline. I've found that most teams still have an "not invented here" mentality, or don't even know they have the problem! That can lead to a "oh, we can just fix it when it happens" type of mentality. But your timing may be better than ours - we started back in 2016.

I haven't played with your product (yet), only took a look at this thread and your website. Some observations:

\- SQL Editor - big plus! I think giving your users a space where they can take action is a super value-add, we didn't have that.

\- nice work running the tests inside the customer's warehouse. That has two benefits for you. 1) you're not incurring the cost to crunch the metadata, it can get quite expensive, depending on the number of tables in the warehouse. 2) you're avoiding data access issues, getting access to the warehouse was always a hurdle, even though we only needed access to the system tables.

\- pricing model. I think the per-seat model is the way to go. We tried charging by number of rows, and size of the warehouse (number of nodes), but then you run into weird situations with customers who are dealing with huge historic datasets, but really only look at the last 30 of data.

My unsolicited $0.02 is that you think hard about distribution. I think you want to think about hitching your wagon to the cloud marketplaces, and Snowflake's marketplace. For example, attaching themselves to Snowflake is what made all the difference for Fivetran.

I have a bunch of more scars that I can share if you care to know them :-)

<comment><USER9> Fantastic blog post, thanks for sharing.

So I guess if you had to pick arbitrary revenue/data/fte cutoffs, do you see the org chart of these adopters as you've described looking a certain way? Let me try to rephrase that.

Do you think there's a step function of "here you need one DBA who is a holy librarian" and "here we need a gitlab styled data team with SLAs and the data equivalent of HR business partners who get assigned to the BU"?

Tangential to your comment but curious if you believe the human side scales akin to the infrastructure side.

<comment><USER4> Where is the blog post?

</comment></comment></comment><comment><USER10> Have you considered picking a different name? Searching for "Hubble" for whatever reason is going to return millions of irrelevant results for your customers.

<comment><USER11> I can't think of a worse name for SEO purposes. You'd have to fight through a well loved and well known space telescope, the astronomer it was named after, and Hubble contact lenses, which has raised ~74MM.

<comment><USER12> If a customer is looking for you specifically, they will find you (e.g. "hubble data" as stated above). If they are looking for a "data quality monitor" then the SEO will need to reflect that. The name is largely irrelevant at that point, it's merely a moniker.

In the grand scheme of problems a new company has, this is so trivially minor that I can't fathom this having any tangible effect on the success of a company. It's one thing if there's another data warehousing company called "hubble", but that's not the case you're making.

<comment><USER13> Hubble data brings up, as I would expect, data from the Hubble Space Telescope. Not one of the first page of results points to anything else but HSTS information.

<comment><USER12> The product literally just launched -- give it a few weeks, it'll show up.

<comment><USER13> I don't know who's advising you on SEO, but you will not ever outrank STSCI, NASA, ESA, AWS Open Data's HSTS archive, The Planetary Society, the National Academy of Sciences, or the ESO on "hubble data" as long as Hubble is still what people think of when they hear Hubble. The telescope and related sites/agencies/organizations have a 22 year head start building a relevant link profile in Google. And if you did, Google would get suspicious.

Hubble is fine as a name if you pick the right keywords to target in your marketing, but "Hubble data" is never going to show a link to something that isn't at least tangentially related to the telescope.

<comment><USER14> You're getting carried away with the example "hubble data." The point is that people will modify their search terms until they find the company they're looking for. If they don't know the company they're looking for then they will search by use case (eg, "detect data drift"), in which case the search results for the company name don't matter.

</comment></comment></comment></comment></comment></comment></comment><comment><USER15> I signed up and I think the concept is promising. It was very easy to add a couple of tests. SQL interface is handy and convenient, but sometimes still limited. It would be good to add a support for some custom scripts (i.e. Python, R). Another important thing for my team would also be seamless integration with other tools (i.e. email, SMS, Slack) to notify the team about the failed test(s).

</comment><comment><USER16> +1 for alleviating data scientists/engineers of boring, repetitive manual tasks and empowering them to focus on the more challenging stuff

</comment><comment><USER17> What does the tech stack look like?

Is there any caching for those situations where you may read the same historical data over & over?

<comment><AUTHOR> Yes, we store the historical value of each test so you can always scroll back through time and see the state of the data warehouse at any given point.

For example, if you have a test that counts the number of rows "COUNT(*)" \- that value will be recorded. So you can look back an hour/day/week and see how many rows the table had without executing any SQL. These values are stored in a time series db, so querying history is fast.

Our tech stack: monolith backend in python + postgres + react. The test themselves are all SQL queries and run in the data warehouse.

</comment></comment><comment><USER18> Do you have/think you need an on-prem version?

<comment><AUTHOR> Yes we can run the whole stack on-prem. We realised very early that on-prem would be needed for many users. So we've made it easy to spin up Hubble in a k8s cluster in your cloud or on bare metal.</comment></comment>


2012 Is Bullshit; 2020 Is When We'll Be in Trouble (2012) - <AUTHOR> https://www.vice.com/en_us/article/8gvbg5/2012-is-bullshit-2020-is-when-well-really-be-in-trouble-fob-0003000-v19n10 <comment><USER2> Very interesting, but unless I'm literally just missing it on the mobile view, this is an extremely short interview?</comment>


Ask HN: Appcelerator's pros, cons and alternatives? - <AUTHOR> If code reuse between platforms and native user experience are desired, is Appcelerator a good option in 2016?
Most opinions I'm finding are years old, so I'm not sure if they're still relevant.
The official site sells it really well, and both the IDE and the analytics platform seem like good products.
Any thoughts? What are the alternatives? <comment><USER2> I've built several cross-platforms apps, but never used Appcelerator, the fact that it used C# (Does it still ?)was the main reason behind the fact that I didn't try.

I've used Cordova/Phonegap (where you build webViews, not native apps), which is an OK solution. It's pretty good for rapid prototyping, but the user experience isn't as good as a native app.

>What are the alternatives?

For code reuse & native apps, React native would be a good and modern alternative too.

React native : [https://facebook.github.io/react- native/](https://facebook.github.io/react-native/) List of examples apps : [https://github.com/maxenceC/open-source-react-native- apps](https://github.com/maxenceC/open-source-react-native-apps)

It's doesn't feel mature yet, but it's a way better solution than webviews.

I've tried some IDE specific for React native, and Deco ([https://www.decosoftware.com/](https://www.decosoftware.com/)) was great !

<comment><USER3> Appcelerator is JS rather than C# (Xamarin does use C#)

</comment></comment><comment><USER3> I've used it for several years and done a _lot_ with it. Most of the niggles have been ironed out over time, though Android does tend to lag behind with things a little.

Most recently they introduced Hyperloop (not that one!) which allows direct access to native API's, use of cocoapods etc directly from JS. That would be fantastic if it didn't require a team licence of $1200/seat upfront to even use. With React Native taking off and Nativescript also offering native API access, I can only see it withering behind the massive paywall now which is a shame.

Personally, most of my new projects will now be React Native as I'm confident it'll be around and well supported for longer.

</comment><comment><USER4> If you plan on doing any work with BLE, I highly advise against Appcelerator. You will have an uphill battle. Our company has some legacy Appcelerator apps and it has been difficult to maintain them over time. Learn Swift, use Xcode, nothing comes close to 1st party.

</comment><comment><USER5> ionic.</comment>


Court has confirmed coordinated raids of Zwiebelfreunde in Germany were illegal - <AUTHOR> https://blog.torservers.net/20180704/coordinated-raids-of-zwiebelfreunde-at-various-locations-in-germany.html#update <comment><USER2> A small remainder: German police also confiscated donor lists for anonymity projects like Tails [0]

In that context, I consider it highly unlikely that this data wasn't already shared with all kinds of institutions. Sure, CCC and Zwiebelfreunde can do a bit of legal dance and make German police pretend nothing like that ever happened, but one would be naive to take their word for it.

[0] [https://news.ycombinator.com/item?id=17483393](https://news.ycombinator.com/item?id=17483393)

</comment><comment><USER3> I mean, I am really, really glad that at least it's been declared illegal, as the whole thing is just insane (and reeks of the typical authoritarian overreach as is custom in Bavaria), but the fact it happened in the first place is still deeply disconcerting.

<comment><USER4> They'll probably do it again.

<comment><USER5> I'm not a German, but Dutch. Doesn't this generate a lot of media attention and precedence? I thought these kind of actions were especially sensitive in Germany?

</comment></comment></comment><comment><USER6> Are there any consequences to prevent this from happening again? I am unfamiliar w/ German approaches to illegal police activities (even if barely illegal and retroactively determined), but I know in other countries the absence of any consequences allows these things to repeatedly happen.

<comment><USER7> Effectively there will be no consequences. Technically Zwiebelfreunde could file charges against the judge who signed the orders or the police who executed them, but effectively they will not be prosecuted, let alone convicted. Nearly all illegal police activity in Germany had no real consequences, because there is a "high trust" in them by ruling parties.

Some examples from the last year:

[http://www.spiegel.de/panorama/justiz/g20-polizei-soll- recht...](http://www.spiegel.de/panorama/justiz/g20-polizei-soll-rechtswidrig- granatpistole-eingesetzt-haben-a-1176695.html)

[https://www.hessenschau.de/panorama/durchsuchung-des-asta- st...](https://www.hessenschau.de/panorama/durchsuchung-des-asta- studierendenhauses-war-rechtswidrig,rechtswidrige-durchsuchung-asta-100.html)

</comment></comment><comment><USER8> They're requesting in writing that no analysis was done, because then when they get the equipment back and see that yup, analysis was done, they can then I assume sue for... Something? How's that work in Germany?

<comment><USER9> > _They 're requesting in writing that no analysis was done, because then when > they get the equipment back and see that yup, analysis was done, they can > then I assume sue for... Something?_

I'd assume if you're involved in a lawsuit against the police, and they make a factual claim like "we did not analyze items not listed in the warrant", that there's some sort of perjury type charge there.

While the police are allowed to lie to you, that's usually restricted to interrogations ("your partner has already flipped") not lying to the lawyer for the person suing you.

<comment><USER10> This isn't how it works in the United States at least. The police are allowed to lie to you, your lawyer, the media, anyone they want to, as long as they are not under oath. They may be held civilly liable for lies that cause damages but only after overcoming their qualified immunity. Lieing to the court even when not under oath could lead to a contempt change.

<comment><USER9> >This isn't how it works in the United States at least. The police are allowed to lie to you, your lawyer, the media, anyone they want to, as long as they are not under oath.

That seems like you're playing semantics.

A __contempt charge __is a type of criminal charge. My contention is that if a lawyer, during discovery, asks the police a question like "did you analyze the data" and they lie they'd face issues.

So it looks like we're in agreement :)

<comment><USER5> You are in spirit in agreement, but I don't think he played on words. Just stating (how awful, personal opinion) how it is in the US.

</comment></comment></comment></comment></comment><comment><USER11> Google Translate of the German court text quoted in the blog post:

The challenged search and seizure orders and confiscation orders of the district court of Munich were unlawful. As far as they were still valid, they were therefore to be lifted. Moreover, their illegality was ascertained. The applications for surrender of the confiscated objects were also to be granted in this respect. The measures can no longer be considered covered by the Code of Criminal Procedure. The investigating authorities are right to bring proceedings against the persons responsible for the calls made on the websites cited. However, the assumption made by the investigating authorities that evidence can be found during the search of the data subjects or on the confiscated data carriers is not justified in the known circumstances. There is no sufficient probability of finding relevant data. There is no evidence that those affected, their association onion friends e.V. or the grouping "Riseup Networks" belong only to the environment of unknown offenders. It is also not immediately apparent that they can find information on the perpetrator environment or the perpetrators. The only connection point is the fact that the unknown perpetrators had set up an e-mail account at "Riseup Networks". However, "Riseup Networks" offers (as intended) anonymized Internet services. The account can be set up anonymously (on recommendation). E-mail traffic via the configured account is encrypted. Therefore, there is very little chance of getting information about the perpetrators and acts via "Riseup Networks" and its database. In addition, there is the additional factor that according to the information available so far, those concerned with their association "Zwiebelfreunde e.V." can not be equated with "Riseup Networks". The link exists, as far as can be seen, only in the support of the network by collecting donations

<comment><USER12> I think the translation from deepl.com is a bit better:

"The challenged search and seizure orders and seizure confirmation orders of the Munich District Court were illegal. If they still existed, they were therefore to be repealed. Moreover, their unlawfulness could be established. The requests for surrender of the confiscated items were also to be granted in this respect. The measures can no longer be regarded as covered by the Code of Criminal Procedure.

The investigating authorities are right to take legal action against those responsible for the calls on the websites cited. However, in the known circumstances, it is not justified for the investigating authorities to assume that evidence for these proceedings can be found during the search of the persons concerned or on the confiscated data carriers. There is insufficient probability of finding relevant data. There is no evidence that those affected, their association Zwiebelfreunde e.V. or the grouping "Riseup Networks" belong even to the environment of the unknown perpetrators. It is also not immediately apparent that they contain information on the perpetrators' environment or perpetrators. The only connection point is the fact that the unknown perpetrators had set up an e-mail account at "Riseup Networks". "However, "Riseup Networks" offers (as intended) anonymized Internet services. The account can be set up anonymously (on recommendation). E-mail traffic via the configured account is encrypted. Therefore, there is very little probability of obtaining information about the perpetrators and crimes via "Riseup Networks" and its database. In addition, according to the information available to date, those affected with their association "Zwiebelfreunde e.V." cannot be equated with "Riseup Networks". The only connection, as far as is evident, is to support the network by collecting donations."

Translated with www.DeepL.com/Translator

</comment></comment><comment><USER13> It's a good thing people are willing to risk this much to make an important point.</comment>


Building Flickr's new Hybrid Signed-Out Homepage - <AUTHOR> http://code.flickr.net/2014/04/23/building-flickrs-new-hybrid-signed-out-homepage/

<comment><USER2> Just wondering, is there a reason flickr doesn't bother alerting users on IE7 and below that their browser is out of date? This page degrades down to IE8 (where it just seems to drop all scrolling effects and animations), but is entirely broken in IE7.

I know Twitter does this and goes as far as showing the mobile version of their site instead.</comment>


So long, Sonos: Meet the open-source audio system that will never die - <AUTHOR> https://www.zdnet.com/article/so-long-sonos-meet-the-open-source-audio-system-that-will-never-die/#ftag=CAD-03-10abf5f <comment><AUTHOR> I think there is a lot of merit to the idea of an open-source hardware and firmware platform for cloud-based AI assistant/audio systems. It seems like it would be wins all the way around. Google and Amazon can fight over features, stability, security, and UI. Consumers don't have a brick if they take a chance with a small startup that goes belly-up. The hardware wouldn't end up in a landfill. Network configuration and debugging could be standardized, making security and support much easier. And enthusiasts or people with special needs could swap out the OS.

</comment><comment><USER2> This is a great idea but I believe Sonos has some patents on syncing audio across separate speakers that is patented. Someone will need to figure out an open way around that.</comment>


Show HN: iLend – Keep track of the things you lend and borrow - <AUTHOR> https://ilendapp.com/ <comment><USER2> Great idea and execution! However, I fear this may lead me to be less likely to lend things by making me more contemplative on the odds of me getting the things back in good condition or at all, and I wonder if this may actually be worse for me and my friend group. I'm sure I've loaned several books and never had them returned, but is this really that bad, especially given that I can't remember any specifics items or lendees?

<comment><AUTHOR> It just helps you keep track of all the things. I personally lent so many things that it helps me track who has what. ;)</comment></comment>