Resilience: An Ad Blocker Without Compromises - <AUTHOR> https://resilienceblocker.info/ <comment><USER2> _> Resilience doesn't sell out your privacy with "acceptable ads". Resilience won't be blocked by your web browser's developers._

Excellent. I'm glad to see uBlock Origin getting some much needed competition, and a good take on Chrome's adblocking downgrade.

 _> Transparent HTTP/HTTPS proxy for Windows, Linux and macOS. _

Not so great. Getting HTTP/1 right is _hard_ , getting SSL right is _hard_ , integration is quite hard just as well. And then there's HTTP/2, and WebSockets and other protocols. Possible results of failure: connection reset, encryption downgrade attack, malicious script injection, credentials stealing, browsing history stealing, XSS, SSL spoofing, IDN homograph attack. Sadly Chrome seems to leave no better option at this point.

OTOH, in the longer run it makes sense for ad blockers & similar tools to be ran as a wholly separate process, offering one more layer of separation.

<comment><USER3> > I'm glad to see uBlock Origin getting some much needed competition,

Why does uBO need competition? It appears to me that it offers the best combination of features and performance out there.

Moreover, the choice to implement as a proxy has some serious drawbacks, features like uBO's cosmetic filters won't be practical to implement in a proxy.

<comment><USER4> Competition in any space is extremely valuable. If for whatever reason uBlock Origin went away tomorrow there would be a void to fill. Furthermore, competition drives innovation. A separate project may figure out a better or more efficient way to accomplish the same goals and they could potentially share their findings with the uBlock team or vice versa and everyone benefits.

<comment><USER5> > If for whatever reason uBlock Origin went away tomorrow there would be a > void to fill.

If for whatever reason uBlock Origin went away tomorrow there would be a fork. uBlock Origin is the second (or third, I forget) fork of the original project.

<comment><USER6> uBlock Origin is still almost exclusively maintained by the original author: Raymond Hill. He was already behind uBlock and is now working on uBlock Origin (the original uBlock being basically stale).

</comment></comment></comment></comment></comment><comment><USER7> I welcome more competition in this space, especially ones that operate at a different abstraction level than others, but this one doesnt look active anymore.

I know brave isnt the most popular product / business model, but their rust reimplementation of the Adblock Plus syntax, a rewrite of uBlock Origins and Cliqz, might hopefully be a valuable contribution to the open source world.

Regardless of how people feel about the ethics of ad blocking, being able to dynamically block parts of websites has become as or more important than anti- virus, for safety.

[https://brave.com/improved-ad-blocker- performance/](https://brave.com/improved-ad-blocker-performance/)

[https://github.com/brave/adblock-rust](https://github.com/brave/adblock-rust)

Although it will make troubleshooting a nightmare, necessary protection at multiple levels of the stack is becoming a reality. NextDNS (or quad9, pihole, adguard) combined with Brave (or uBlock Origin, Cliqz) in combination are going to keep people safer, despite the shortcomings of both dns based and active page filters and parsing.

It would be nice to be able to manage and maintain filter list configurations across all browsers, devices, and dns. Some sort of central management that updates and propagates NextDNS, Brave, uBlock Origin, uBlock Matrix, and Dark Reader. uBlock Matrix and Dark Reader are especially cumbersome to use between Chrome and Firefox on different devices.

<comment><USER8> > I know brave isnt the most popular product / business model

Curious what you mean by this. Are BATs unpopular or is there some other issue with the browser itself?

<comment><USER9> Do you know of any website that accepts them? I've seen them on some exchanges, but the current procedure is to sell them directly. Who's buying them (besides speculators)?

<comment><USER10> _raises hand_

I am working on a matrix/xmpp/mastodon paid hosting, which I am racing to get the Ethereum token payment gateway finished by the end of the month and planning to accept BAT/DAI/ETH.

</comment></comment></comment></comment><comment><USER11> Interesting but is the project even alive anymore? Last commit was 9 months ago.

Combined with it saying it's early early alpha, no instructions and no builds means it doesn't look to be in a good state to use.

</comment><comment><USER12> Does anyone remember Privoxy [https://www.privoxy.org/](https://www.privoxy.org/) ? I've been filtering and removing junk from the web since the early 2000s.

With https now prevalent, filtering with a proxy server remains annoying. I vastly prefer the model of uBlock Origin for desktops, and a combination of Firefox + uBlock origin and a dns blocker (Blokada) on Android.

<comment><USER13> What about [https://www.admuncher.com](https://www.admuncher.com) ? :) Since 1999.

<comment><USER14> Ad Muncher is broken/abandonware (hasn't been updated since 2014).

</comment></comment></comment><comment><USER15> Or you could use a browser that doesn't actively try to prevent you from using adblocking extensions.

<comment><AUTHOR> Tor is by far the best browser. However as a part of compartmentalisation I use un-googled chromium and right now it is struggling to install anything (others have this issue.)

</comment></comment><comment><USER16> > Resilience is an ad blocker for your computer that works with any browser on > any operating system.

This might be a bit difficult on some of the mobile platforms, at least with the code that's currently in the repository :P

</comment><comment><USER17> HTTP is filtered? No problem, let's transfer ads through websockets or webrtc!

</comment><comment><USER18> Once we've all blocked ads from every possible angle, what is the model we'll be using to pay for social media, photo sharing, email, video sharing,. instant messaging and so on?

I dislike ads as much as the next person but I don't comprehend what monetary model is being lined up to replace their revenue generating abilities, for service providers

Aside from lots of small subscriptions.

<comment><USER19> Cash is king.

(full disclosure: Protonmail and VPN subscriber, donate to Marcel Borkhorst M66B for his excellent Netguard app, never had FB/IG accounts, and only used Twitter once to solve an issue with TMobile's TForce, which was shockingly efficient, never had the app).

</comment></comment><comment><USER20> This is great; a proxy is definitely the way to go given how many processes use http.

I used to use glimmerblocker but it gradually stopped really working. One great feature was that it was a rewrite proxy so I could, for example, suppress auto play videos, a feature I had missed since Omniweb almost 20 years ago

</comment><comment><USER21> Can anyone explain the following text taken from the site?

"Chrome executives: I know you think this won't get traction because it'll never be as easy to install as uBlock. Watch me as I make it every bit as easy by the time you finish switching to declarativeNetRequest."

<comment><USER22> Chrome is switch some rules around that will render current ad blockers useless.

</comment></comment><comment><USER23> If you don't have an iOS app then I don't see how you can say "all operating systems" with a straight face.

</comment><comment><USER24> Looks dead, seems the maintainer gave up.

</comment><comment><USER25> Strong words from a dormant project.

</comment><comment><USER26> Is this any different than a PiHole?

<comment><USER24> PiHole is nice because you can protect all the devices in your house at the same time, but it's annoying bec if you want to unblock something bec a page is messed up you need to go to PiHole service to unblock it and it's not easy to set up temp unblocks

</comment></comment><comment><USER27> Any advantages over a hosts file?

<comment><USER17> You are not restricted to a per-host filter, but can filter by URL

</comment></comment><comment><USER28> Nadim apears to have ambitions of being appointed Censor of Rome.</comment>


Show HN: My recreation of cyberpunk/futuristic UI in rust - <AUTHOR> https://ivanceras.github.io/futuristic-ui/ <comment><USER2> Wow, crazy timing! You'll love what I just posted haha

I really like the sounds too - are they original and/or free to use?

<comment><AUTHOR> This is based from arwes.dev including the sounds. The license is MIT.</comment></comment>


Interviewing candidates before you check references? You're doing it wrong - <AUTHOR> https://also.roybahat.com/hiring-someone-check-references-before-you-interview-ba05c5b46e37 <comment><USER2> Too many dishonest people wasting the time of job applicants. Anyone looking for a job is a fool to submit references before the final stage of receiving a job offer. Disagree with your premise 100%.

<comment><USER3> Also, nobody will want to be anyone's reference anymore if it requires talking to dozens of people every time the candidate applies for a job. (If employers check references in the early stages of a job application, a reference will have to interact with everyone who doesn't want to hire the candidate in addition to everyone who wants to hire the candidate.)

The article says: _" Note the response time — whether they get back to you quickly or not might be useful information in and of itself."_

Not replying quickly can mean many things. It might mean that the reference is on vacation, for example.</comment></comment>


Ask HN: Review my site: BHeardUSA.com - Ye-Ha

The three static pages (Welcome, Realtime & Trends) are meant to be self-explanatory.  We want our mothers to be able to use the site.  Please comment on content and proposed service. <comment><AUTHOR> \- Your design is a great start -- The guy holding the megaphone gets the point across very well.

\- Trends are a great idea, and showing them by area makes it a lot more powerful (for example, i'd love to see what people are saying about the election right now, here in south jersey).

\- My first question when visiting the site is -- How will the public people see my postcard?

\- My next question is -- why should I use this instead of twitter, where I can do things like @reply the public person.

\- The top message "send positive/negative postcards"... conflicts with the bullet list of the things I can use the service for...

\- I'm not sure navigating between realtime and trends to set the geographic area of search is very clear to a user. Not quite obvious enough

\- I think you can lean things out a bit -- do you really need a log in box taking up that much space on every screen?

\- Real time: the map is a neat attention grabber, and gets me interested a bit the first time visiting, but I'm not sure it would be as enticing on later views. As mentioned above, I'd work on a more direct way to see trends in an area.

Great start, hope this was helpful!

<comment> Ye-Ha Yes it was!

TY TY - we really appreciate it. Already made one changed based on your feedback.

</comment></comment><comment> Ye-Ha The post card gets delivered to the target person via USPS.

You can also send fake dog poop. Currenly looking for some that can be recycled.......

</comment><comment><USER2> <http://bheardusa.com/></comment>


Neuroscientists decode the brain activity of the worm - <AUTHOR> http://phys.org/news/2015-10-neuroscientists-decode-brain-worm.html <comment><USER2> Ok, that is pretty cool. Next up, inducing neural activity in worms to drive them around. If you can go both ways (observe and predict, and induce and demonstrate) you really do have the worm's brain all figured out.

Then you build a mining robot with mineral sensors that can detect the ore your interested in. Now you strap in your worm brain and you have your local computer translate motion signals into robot motion, and mineral detection into food detection. And voila you have a robotic miner which forages around for ore to process.

<comment><USER3> Driving mice is already possible. [https://med.stanford.edu/news/all- news/2010/09/light-workout...](https://med.stanford.edu/news/all- news/2010/09/light-workout-scientists-use-optogenetics-to-effectively- stimulate-muscle-movement-in-mice.html)

I believe this was also done in insects, although I can't find the article.

With C. Elegans, the difference is that we will be able to understand it in far more detail.

As for mining robots, biology already has you beat. The entire earth in enervated with slow-growing mining bacteria. They are probably a better place to start for industrial purposes.

</comment></comment><comment><USER4> _" They used worms genetically engineered with a fluorescent protein that caused the worm's neurons to flash when they were active."_

Now that's impressive.

So can OpenWorm, the C. Elegans simulator, get the same patterns?

<comment><USER5> I'm currently working on extending this technique to higher order primates. It's unbelievably cool that these protein sensors work. If you're curious about the details, check out GCaMP and calcium imaging, or these papers (paywall)

[https://en.wikipedia.org/wiki/Calcium_imaging](https://en.wikipedia.org/wiki/Calcium_imaging)

[http://www.nature.com/nature/journal/v499/n7458/abs/nature12...](http://www.nature.com/nature/journal/v499/n7458/abs/nature12354.html)

[http://www.jneurosci.org/content/32/40/13819.full](http://www.jneurosci.org/content/32/40/13819.full)

</comment></comment><comment><USER6> It sounded as if they have mapped all of the connections between neurons in previous work. Which suggests these connections are genetic rather than learned, and the same in every individual worm. While that makes studying them much easier, I find it surprising. I suppose that's how innate abilities are conferred on the next generation even in humans where many connections will indeed be individually learned/constructed. Very interesting.

<comment><USER7> Yes. The worm c.elegans has 302 neurons, connected in a specific way.

From the article:

"For almost 30 years, the list of connections between individual neurons has been known. Despite the low number of neurons, its neuronal networks possesse a high degree of complexity and sophisticated behavioral output; the worm thus represents an animal of choice to study brain function."

<comment><USER8> Apparently they just found two more in males:

[http://www.nature.com/news/surprise-mystery-neurons-found- in...](http://www.nature.com/news/surprise-mystery-neurons-found-in-male- worms-1.18558)

</comment></comment></comment><comment><USER9> Full paper: [http://www.cell.com/cell/pdfExtended/S0092-8674(15)01196-4](http://www.cell.com/cell/pdfExtended/S0092-8674\(15\)01196-4)

</comment><comment><USER3> Interestingly, C. Elegans also autofluoresces- without the need for any special genes to be cloned in. When it dies, a bright wave of light passes through its body.

<comment><USER5> Lots of proteins have natural fluorescence. You'll notice this if you shine blacklight in ocean water - it'll glow brightly from all of the little microorganisms that aren't intending to fluoresce but do so incidentally. There are also bioluminescent organisms but that's different.

<comment><USER3> Sure, IIRC from my college chemistry classes it's mostly Trp or Try fluorescence.

In this case, it's still considered fairly puzzling why the flash is so intense or what its "actual function" or whether it's protein fluorescence at all: [http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3735983/](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3735983/)

</comment></comment></comment><comment><USER10> Is it possible to access the data of how the 302 cells are connected somewhere? Would be awesome to emulate in software.

<comment><USER11> This might be what you are looking for [http://www.wormatlas.org/neuronalwiring.html](http://www.wormatlas.org/neuronalwiring.html) specifically this file [http://www.wormatlas.org/images/NeuronConnect.xls](http://www.wormatlas.org/images/NeuronConnect.xls) that you can read in pandas</comment></comment>


Who Is Winning the Food Delivery War? - <AUTHOR> https://priceonomics.com/who-is-winning-the-food-delivery-war/ <comment><USER2> A tangent - but I live in Jakarta, Indonesia. Here around 2 years ago a local motorbike ride-sharing startup called Gojek so ago switched on a service called go-food. There legion of drivers would now pick up any food from any restaurant you wanted and drop it to you.

It killed most food delivery startups within a few weeks, they have since expanded the service to allow pickup of pretty much anything. Supermarkets, convenience stores, toyshops, pharmacies, pet-shops, hardware. Theres even a generic option to buy "something" from a shop at "address".

Its pretty easy to look at this and think this is the end-game in developed countries too. For at least those purchases where convenience and timing is a factor and cost not as much. All these disparate products are actually a very similar UX and solving the same problem.

Regular grocery shopping behaviour (or the startups running deliveries) hasn't been much effected, presumably as its a more bespoke use case.

<comment><USER3> In the dotcom there was a famous example of arbitrary shopping-deliver-as-a- service called Kozmo.com. The problem in western countries is that the wages for delivery cannot possibly work for the way people use such services. It's not even clear this will work well for restaurants outside of specific affluent areas.

</comment></comment><comment><USER4> > On average, customers go to physical grocery stores 3.2 times per month

This seems crazy to me -- not only must be purchases be _enormous_ , but fresh meats and vegetables don't keep for 9 days between each visit. Am I missing something?

<comment><USER5> That doesn't sound that weird to me. Two things:

I expect most people aren't shy about buying several weeks worth of meat and freezing it. Quality doesn't drop off _that_ much. Quality differences are more noticeable with frozen vegetables, but I suspect a lot of people are fine with those too. Then there are frozen microwave/oven dinners and things like that, as well as non-perishable/shelf-stable items.

A household that exclusively cooks all their meals might make 7-10 grocery trips per month, but I expect that most people don't do that, and even those who cook all their meals probably don't rely on fresh meat & veggies for every single meal. And note that they don't cover people who go out to eat at restaurants at all in that list, so they're not accounted for anywhere.

On a side note, regarding your surprise at the purchases being large: most Americans, especially those who live in rural or suburban areas, don't do the kind of daily or near-daily stops at small convenience stores to pick up a few quick things that you see in dense cities. They drive to the grocery store, load up as much as they can fit in their cart, fill up their car, and go back home.

<comment><USER6> On the frozen meat point, I'd bet large sums of money people can't distinguish between fresh and frozen-within-3-or-so-months meat. In fact, frozen meat can be reliably made even tastier than fresh if you cook it straight fron the freezer with no thaw. It's a cool little hack really. By cooking from frozen you get more time to sear, char, and crisp the outside of the meat while the inside doesn't overcook. It's next to impossible to get the right mix of seared/charred outside but tender lightly cooked inside with the normal thickness stock cuts (~1inch) you get at grocery stores. With frozen, can do it no problem everytime. On that note, another trick, if you must cook without freezing, is to just get a custom thickness cut. For example, I need a 2-3inch NY strip if I want to grill straight from the refrigerator and get the right outside to inside done-ness.

</comment></comment></comment><comment><USER7> Pizza and chinese restaurants are likely dominating the food delivery war in the US. They don't seem to be considered in this study.

</comment><comment><USER8> In China there are a heap of last-mile delivery providers, but only two clear winners with national coverage: [http://ele.me/](http://ele.me/) and [http://waimai.meituan.com/](http://waimai.meituan.com/) ... statistics of recent but nebulous origin claim Ele.me (Alibaba-backed) has 55% (42% + 13% after acquiring Baidu Waimai last month) and Meituan (Tencent backed) has 41% of the national market. (I would regard these figures suspiciously.)

Overall the finished meal delivery concept has proven very popular with Chinese consumers, probably because of growing working hours, smaller families/living situations and worsening traffic.

The government recently announced rules for the delivery sector's employees including no right to enter private premises, requirement to be punctual, requirement to be quiet and respectful, etc.

In terms of Asian regional providers, Food Panda has pretty good traction in a few countries and there are a number of smaller startups in most Southeast Asian markets, though none we are aware of with fundamentally unique models. Uber Eats and Deliveroo are trying to get in to some markets like Hong Kong.

</comment><comment><USER9> I feel like Food Delivery is a place where competition may actually have driven up prices for the consumers. I was gone from SF for two years and when I came back to the same neighborhood, I felt that food delivery became more expensive because of DoorDash, etc.

Places that used to not charge delivery fees now do, maybe because now they realized they could? Or because they have to compete for delivery people. Also, some apps now charge a 20% default tip. It is not rare to get three types of fees on top on one delivery order, even for something simple as a burrito.

<comment><USER10> Don't forget the price markups! If you compare prices on Doordash or Yelp vs restaurants' websites you'll often find that the prices themself are 10-20% higher.

</comment></comment><comment><USER11> In the US can you not order food for delivery directly from supermarket, same for restaurants?

I would have thought in the UK Tesco, Sainsburys etc do as well as any delivery only services (Ocado, Amazon Fresh etc).

I would have guessed direct restaurant orders are as large as justeat/uber eats style services. They would certainly deserve a place on the pie chart...

I also dont believe the average US consumer spends $105/month on meal kits...

<comment><USER12> > In the US can you not order food for delivery directly from supermarket, > same for restaurants?

I think it's available in larger, older cities (e.g. on the East Coast), but in the rest of the country it's a lot rarer. I think maybe it's much the same reason we no longer have bellboys, ushers or elevator attendants: minimum wage & payroll taxes rose to the point that it wasn't cost-effective for grocers or customers.

<comment><USER13> In the UK we have higher minimum wage, and higher petrol duties, and yet we manage to have extensive supermarket delivery.

And it's been happening for many years - the first "online" shopping from home happened in the UK, from Tesco, in 1984.

[http://www.bbc.co.uk/news/magazine-24091393](http://www.bbc.co.uk/news/magazine-24091393)

</comment></comment></comment><comment><USER14> Just remember - if you don't want your data packed up and shipped off for marketing purposes, don't apply for a loan from Earnest :)

<comment><USER15> Seriously. It's one thing to conduct the analysis in-house and sell the results, it's another to just sell the data outright.

<comment><USER6> I don't think they sold it though. This is Priceonomics thing now -- companies pay them to write stories using the company's data as a more highbrow form of content marketing. I assume Earnest commissioned this vs them selling or giving away the data to Priceonomics.

</comment></comment></comment><comment><USER16> The thing that leaps out at me is that People spend $134 when ordering online and $155 in-store. That simple stat shouts a huge amount about the changes coming. That's 15% revenue, and I expect it is almost entirely impulse purchases, much of which will be high fat high sugar content.

But losing 15% revenue is huge.

I also would ask where the joined up services are - I mean imagine we all shared some domestic services for our houses in a street.

Maid service, (vaguely) communal eating (certainly communal food prep), the same plumber and boiler services, the same x y z

I think that handing over that part of my life for someone else to think about is attractive. And it is not really on radar.

I just wonder if it is possible

</comment><comment><USER17> Seeing various fast foods in America and what generally people eat , it looks like easy to cook especially the prep work is less. Still I am surprised to see why many ppl don't cook . It's so much fun, more than you get to eat healthy and save lots of money .

<comment><USER18> > Still I am surprised to see why many ppl don't cook

Many people don't know how to cook and I assume that a lot lack basic knowledge about what food is healthy too. I wonder why this isn't taught at school.

</comment></comment><comment><USER19> The data here seems highly suspect, every other analysis I've seen puts Uber eats at a much larger market share than 0.1% of the market.

</comment><comment><USER20> I'm very skeptical of their data considering it doesn't include Peapod, the oldest and largest internet grocery delivery company.

<comment><USER21> If you take a look at the original blog post upon which this article is based, they list exactly which companies they include in each category:

[https://www.earnest.com/blog/food-economy-spending- data/](https://www.earnest.com/blog/food-economy-spending-data/)

"Brick-and-mortar includes Publix, Whole Foods, Kroger, Safeway, Trader Joe's, Costco, and Albertsons. Grocery delivery includes Instacart, Fresh Direct, Foodkick, Amazon Fresh, and Thrive. Meal Kit services include Blue Apron, Hello Fresh, and Purple Carrot. Prepped meals include Freshly, Thistle, and Munchery. Restaurant delivery services include Ubereats, Doordash, Seamless, GrubHub, and Caviar."

</comment></comment><comment><USER22> Why isn't Postmates on the graph with DoorDash etc?

</comment><comment><USER23> tl;dr = consumers</comment>


OpenSolaris vs Linux - <AUTHOR> http://www.tuxradar.com/content/opensolaris-vs-linux

<comment><USER3> Does it have access to 17+ kPackages, like Debian does?

What about the tools duality? GNU and Solaris concepts are sufficiently different to require two set of binaries, and $PATH split-personality.

Also, Sun (or Oracle) is firmly in control of kernel development. There's no community yet, nor signs of an emerging one, so there's some danger of standing in abandonware land.

Apart from that I like OpenSolaris fine. However, its chiefest advantage is zfs, and btrfs should be there in 3-4 years. So the clock is ticking.

<comment><USER4> Packages: You're right, that is a weakness. However, the important stuff is there, and compiling other things is rarely a problem. OK for experienced admins, bad for newcomers.

Tools: Solaris has a strong history of standards compliance, so you've got the standards based tools, and the GNU tools. Different audiences prefer different tools. GNU tools are now the defaults to cater for people coming from Linux.

Sun/Oracle: Yes, this is an issue, and the uncertainty over the future worries me somewhat -- you'll note Oracle only promised to spend more developing Solaris, and OpenSolaris was not mentioned. Linux is also dependent on big corporates for development, but there is more than one, reducing risk. However, the Solaris model does result in a very clear vision of where things should go, and I personally really like the consistent and stable result - <USER2> wondering whether the new scheduler causes performance woes.

I'd also add Zones as well as ZFS as it's chief advantage, as the combination is absolutely wonderful for development and testing. Also see here [http://www.c0t0d0s0.org/archives/5917-The-impact-of- virtuali...](http://www.c0t0d0s0.org/archives/5917-The-impact-of- virtualisation.html) about the performance benefits of Zones for virtualisation.

<comment><USER3> > OK for experienced admins, bad for newcomers.

How difficult is to create Solaris packages from source? I must admit I never had the problem.

> I'd also add Zones

Well, Linux has VServer.

> as well as ZFS

I agree zfs is nice. Particularly, given L2ARC and ZIL on SSDs, with large SATA raidz2 or raidz3 behind them.

However, btrfs is making the right noises. Whether they deliver on them or not is not yet known, but if they do Solaris needs quite a few more tricks up its sleeve by then. I hope Oracle doesn't drop the ball.

<comment><USER4> Solaris packages are easy to create, but getting things to compile can be tricky as lots of open source code assumes Linux-isms, so you have to port to standards. (BSDs have the same problem.)

Zones are integrated into the very core of the OS, not bolted on.

btrfs does sound like an improvement on ZFS in a few ways, especially in it's COW b-trees, taking advantage of a few more years of development. Competition will be good for ZFS, but right now, ZFS is probably the best shipping and stable FS.

</comment></comment></comment></comment><comment><USER5> Is it just me, or is having to download a file, run a terminal and execute 5 commands just to get flash not a sign of a 'proper desktop OS'?

<comment> old-gregg I find it sad and depressing that a buggy browser plug-in has managed to rise to a status of "OS breaker". You can't be a successful smartphone upstart without Adobe blessing these days (unless you're Apple)

Even github.com brings up "plugins are missing" bar in Firefox. Github? WTF? By the way how do I disable plug-ins in FF like I had them turned off in Safari? It keeps bugging me with that annoying bar.

<comment><USER6> I believe GitHub only uses flash for their repo location copy button which lets you copy a repository location onto your clipboard without having to select it and then copy it manually.

<comment><USER7> Yeah, I didn't even know until I installed ClickToFlash.

I don't know why they do though, maybe flash has better clipboard access than javascript..?</comment></comment></comment></comment>


Why Adobe AIR Is Not Google Gears, Prism, JavaFX or Silverlight - <AUTHOR> http://ruby.sys-con.com/read/514448.htm <comment><USER2> The article was fine to skim but when I went to find out more about the other, Tim Negris, I found that he was recently the VP/Marketing at SCO. Weird. <http://ir.sco.com/ReleaseDetail.cfm?releaseid=170511></comment>


Why do companies treat their office address like a company secret? - <AUTHOR> What are they afraid of? Don't they realize that for every angry customer that might show up at their office to cause trouble, there are 1000 prospective employees that just want to get an idea of how long their new commute will take? Sometimes, it's hard to even figure out what city they're in. <comment><USER2> It depends. But one of the reasons I can think of is if it is an early stage startup or small company. They might be using co-working space like WeWork or only have a virtual office and couple of employees working remotely/from home.

In such case they want to appear bigger than they are and if you could google their address and find out it's a virtual office with 200 companies registered there you might decide to look for a bigger / more established company instead of going with them.

But even if your company's address is just a virtual office, you should list it visibly on your website in my opinion. It doesn't look good when you appear to be hiding it. There's nothing wrong with small business or new startup not having a proper office from beginning.

</comment><comment><USER3> A comical thing about many "cutting-edge" SaaS startups: no one wants to do the website; no one owns it. I have seen it many times: marketing people imploring the programmers to push a little change to the website, only to find that no one is available for the task.

Logically, it makes sense as websites usually get updated only once in a while and it would foolish to dedicate a resource to it. But, it also adds a great difficulty in building a tiny thing, even if something like a contact page.

I guess big companies have more organised processes and teams to handle this. But, in startups, everyone is already working on the core product and everything else seems like a distraction.

</comment><comment><USER4> It's possible that if a company is very small, it works out of a co-working space or has all of its employees working remotely in different locations. They may be hiding this to make it appear to customers that they're bigger than they really are.

But I agree that not telling prospective employees where they're located will deter people from applying. For example, two places within a few miles of each other in NYC can have drastically different commutes (time, comfort and/or safety), so there's no way I'd consider applying for a job if I didn't know the company's location.

</comment><comment><USER5> A business Im involved in does online surveys. As with anything there can be a mistake or misunderstanding over an account credit or voucher type thing. Some people can get very angry over this rather than lodge a report with our member services team. We have seen people call into offices and give death threats over $2. And while you can find our office address online, we keep it a little hidden and don't encourage the location or phone number to be public. So basically for us its to keep potential lunatics away.

</comment><comment><USER6> My current company's CEO is a bit of an icon in the startup world, and his fans can be...intense. If we leave the office door unlocked during work hours, people can (and have) just waltzed in hoping to meet him. People also send all sorts of weird gifts.

Since we were basically working out of a house for the longest time, whoever was on hand would have to deal with the unwanted guest. If you're alone in the office and some big dude comes in saying "Well I won't leave until I meet your founder!", that can be scary.

As Gustomaximus puts it, it is to keep potential lunatics away.

</comment><comment><USER7> Great question ! I have often wondered that too.

I can understand small companies, trying to cover up the fact that they are still operating out of their garage or spare room. But when large corporations do it, I am just flummoxed. For example if you look up Apple in the Sydney (AU) whitepages, there is no address given just a free call number. I just checked, Google now show their address, but in the past they didn't either.

</comment><comment><USER8> I don't think it's sinister, I just think it's looked upon as extra work, with no obvious upside.

For would-be competitors who want to send in spies, it's easier to move on to other companies that publish addresses.

In the big 5 companies, the more hush-hush the R&D activities, the harder the particular site is to find.

</comment><comment><USER9> Can you give examples? Afaik most publish their address?

</comment><comment><USER10> To prevent crazy people from doing something crazy. These buildings aren't open to the public. Outside of your employment use case (in that case just ask the recruiter for the building location; I've never had a problem with that), the people trying to show up to an office building are probably trying to complain or make a scene.</comment>


Tiny-dnn – A C++11 implementation of deep learning - <AUTHOR> https://github.com/tiny-dnn/tiny-dnn <comment><USER2> If you're interested in C++ Frameworks that take a similar approach (e.g. a comprehensive C++ API, statically defined networks etc.) and you aren't afraid of meta-programming, then I'd also take a look at the dlib implementation: [http://blog.dlib.net/2016/06/a-clean-c11-deep-learning- api.h...](http://blog.dlib.net/2016/06/a-clean-c11-deep-learning-api.html)

<comment><USER3> I was shocked to read in that article that MS Visual Studio still does not support the full C++11 spec, which is now over 5 years old. What? I don't use MS products at all, but this really surprises me. Why are they so far behind?

<comment><USER4> What would that be? Visual Studio 2015 seems like fully covered except C99 preprocessor, _if_ this table fully covers all C++11 features.

[https://msdn.microsoft.com/en- us/library/hh567368.aspx](https://msdn.microsoft.com/en- us/library/hh567368.aspx)

Why they have been behind? Wild guess, Microsoft thought that C#/.NET would be the future for everything and scaled back on C++, but with Windows 10 they realized that the world still consist much of C++, so now it is a equal member again together with .NET and HTML5/JavaScript, i.e Universal Windows Platform.

<comment><USER5> > Wild guess, Microsoft thought that C#/.NET would be the future for > everything and scaled back on C++, but with Windows 10 they realized that > the world still consist much of C++, so now it is a equal member again > together with .NET and HTML5/JavaScript, i.e Universal Windows Platform.

Actually it is a bit more complex than that.

.NET used to belong to DevTools and C++ to WindowsDev.

WindowsDev lost the political wars to DevTools when MS went full .NET, but given the technical and political battles that caused Longhorn failure, WindowsDev gained ground.

Hence why starting with Vista the "Going Native" motto was born, and COM gained ground as the way to get more OS APIs to user space.

Alongside this, "Going Native" wave, Singularity and Midori meant bringing those efforts back to .NET thus creating .NET Native.

UWP original design, WinRT, can actually be traced back to .NET before the CLR was created. When they were designing COM 2.0 Runtime, as the future way to use VB, C++ on Windows.

Also before you rejoice too much about C++'s role, check how many C++ related talks are there at Build, Ignite and Connect().

C++ is seen as the official systems programming language for kernel programming, drivers, games and GPGPU. For everything else, the documentation or conference sessions tend to focus on .NET languages.

<comment><USER6> > C++ is seen as the official systems programming language for kernel > programming, drivers

Kernel and driver development is still very much C.

<comment><USER7> I thought Windows did away with C completely and only supports C++ now?

<comment><USER5> The official statement from Herb's blog was into that direction.

[https://herbsutter.com/2012/05/03/reader-qa-what-about-vc- an...](https://herbsutter.com/2012/05/03/reader-qa-what-about-vc-and-c99/)

The modern C support has only been done to the extent required by ANSI C++, and by integrating clang frontend with VC++ backend (C2).

Microsoft is doing with C2 something similar to LLVM for all their languages (.NET Native, VC++, clang frontend).

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> When I looked for a small ANN library with little external dependencies and which could be statically linked I settled on FANN[0].

Worked reasonably, solved my problem as well as I hoped it would. It is rather limited in features though- no training on GPU, single-threaded by design, etc.

tiny-dnn appears to have a lot more choices regarding network architecture, parallelization options. Would definitely have tried tiny-dnn first if I had known about it.

[0]: [http://leenissen.dk/fann/wp/](http://leenissen.dk/fann/wp/)

<comment><USER9> Fann doesn't do DCNN, arguably the most important right now.

<comment><USER3> > arguably the most important right now.

But for very specific uses, right? DCNN excel at image and video, but are DCNN also common for other tasks?

<comment><USER10> Surprisingly (to me at least) convolutional neural networks (CNN) are also useful for natural language processing. (I am using CNN for NLP at work.)

<comment><USER11> CNNs in general excel at infering/decoding "context" from the vectors, so it makes sense to use them for language. But yes, there are cases where they don't make sense/just add complexity.

</comment></comment></comment></comment></comment><comment><USER12> "98.8% accuracy on MNIST in 13 minutes training" \- that's really slow isn't it? I can probably write vanilla JS or Python code that does that in 10 lines or so and would finish in around 5 seconds.

Why is this taking so long?

<comment><USER13> > I can probably write vanilla JS or Python code that does that in 10 lines or > so and would finish in around 5 seconds.

Really? There are certainly frameworks that you can use to achieve this performance in that amount of code, but I would be really interested to see that in vanilla JS/Python.

<comment><USER12> Hey, sure - here's 9 lines of code that implement a perceptron. I get ~95% of precision after a second with 400 samples.

```js

         const dotSign = (v1, v2) => v1.reduce((prev, x, i) => prev + x * v2[i], 1) > 0 ? 1 : -1

        module.exports = (data, weights = Array(data[0].content.length).fill(0)) => {
          for(const {label, content} of data) {
            const delta = (label - dotSign(content, weights)) / 2;
            weights = weights.map((x, i) => x + delta * content[i]);
          }
          return { perceive: vector => dotSign(vector, weights), weights };
        }

```

I can upload an electron app that does this with mnist if interested.

<comment><USER14> how long does does it take to get from 95% -> 98.8% though?

<comment><USER15> This. You can also get near 98% accuracy with vw and one quadratic interaction over the pixel space. It takes seconds. But that last .8%, forget about ever getting there with infinite training time.

</comment></comment></comment></comment></comment><comment><USER16> Sorry if the question is not relevant but is that difficult to use that project to implement something like DeepDream ([https://github.com/google/deepdream](https://github.com/google/deepdream)) ?

<comment><USER17> This should be possible and fairly easy to do, I guess - it's just forward and backward pass with your favorite model through the net and collect the loss for all octaves. You will have to port a few image utility functions (like roll, zoom, etc.) if not available. I ported the basic Deep Dream example to JavaScript [1] and it was not that difficult.

When looking through the code I saw that BP for LRN is not implemented yet, so you cannot pick a model using LRN (or you have to implement it).

[1] [https://github.com/chaosmail/caffejs/blob/master/docs/assets...](https://github.com/chaosmail/caffejs/blob/master/docs/assets/scripts/deepdream_worker.js)

<comment><USER16> Thank you!

</comment></comment></comment><comment><USER18> Contributor here. Welcome to ask question. We are working on GPU support now.

</comment><comment><USER10> This looks interesting. I cloned the repository and am checking it out. The examples all use supplied with the repo binary image training data. It would be good to document data preparation better (sorry in advance if I just missed seeing that).

20+ years ago, I made my living on C++, but dropped it. Looking at the example programs using C++11 features, I am motivated to get up to speed on C++11. I put writing a NLP example for tiny-dnn and sending a pull request on my todo list.

</comment><comment><USER19> Can anyone recommend a book, long article or tutorial on deep learning? I'm not in the field but interested in what particular advances have been made that deep learning is suddenly so much more attractive than regular neural networks etc were back in the 00's.

<comment><USER17> Check out this blog post about deep learning for computer vision applications [1]; it describes very well the historical development and the current advances. In short, not too much changed from LeCuns ConvNets in the 80s. Disclaimer, I wrote this article.

[1] [http://chaosmail.github.io/deeplearning/2016/10/22/intro- to-...](http://chaosmail.github.io/deeplearning/2016/10/22/intro-to-deep- learning-for-computer-vision/)

</comment></comment><comment><USER20> When I saw "tiny-dnn" I though. Huh? Tiny Dot Net Nuke?? No such thing exists. DNN is too much bloat!

<comment><USER20> Wow HN got no sense of humour today.

<comment><USER3> HN has lots of humour and an appreciation for clever wit and sarcasm. I'm afraid that your effort, while noble, failed to reach this bar.</comment></comment></comment>


Don't use Hadoop when your data isn't that big (2013) - <AUTHOR> https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html <comment><USER3> I have seen so so so many projects get bogged down by the need to use a "big data" stack.

I think my favorite example, was a team that spent six months trying to build a system to take in files, parse them, and store them. Files came through a little less than one per second, which translated to about 100kb. This translated to about 2.5Gb a day of data. The data only needed to be stored for a year, and could easily be compressed.

They felt the need to setup a cluster, with 1Tb of RAM to handle processing the documents, they had a 25 Kafka instances, etc. It was just insane.

I just pulled out a python script, and combine that with Postgres and within an afternoon I had completed the project (albeit not production ready). This is so typical within companies it makes me gag. They were easily spending $100k a month just on infrastructure, my solution cost ~$400 ($1200 with replication).

The sad part, is that convincing management to use my solution was the hardest part. Basically, I had to explain how my system was more robust, faster, cheaper, etc. Even side-by-side comparisons didn't seem to convince them, they just felt the other solution was better some how... Eventually, I convinced them, after about a month of debates, and an endless stream of proof.

<comment><USER4> The problem you had was that you were trying to convince management to act in the best interest of the company rather than their own best interest. They would much rather be in charge of a $100k a month project instead of a $1k a month project. Also the big solution required a bunch of engineers working full time which puts them higher up the ladder compared to your 1 engineer working part time solution.

You were saving the company money but hurting their resumes.

<comment><USER5> Reminds me of almost every technology project I've seen in finance! It's all about building complex empires, rather than simple, functional solutions. Sometimes, it's also just about using up the assigned technology budget so that it's not scaled down the following year!

<comment><USER6> Funny, I had a phone screen today and in the midst of asking about how I saw my future and what I wanted to work in, the recruiter was tiptoeing around a situation where someone at the company had suggested React or something and the devs had pushed back (successfully) on the basis that the site _didn 't need it_. I got the feeling he was trying not to step on my dreams of being a cutting-edge JS frontend trailblazer, but it is really a point in the company's favor (to me) that they were able to resist the urge.

Basically, they are looking for web developers but it seems like they have to filter out all the frontend ninja rockstars.

<comment><USER7> Why didn't they "need" it? How do they decide what they "need"?

This sounds like typical anti-change pushback, which I have learned can actually be a good thing. However, this anecdote is severely lacking in insight; much like most people's support of, or opposition to, change. Further, like the widespread belief that sentences shouldn't start with conjunctions; much less conjunctive adverbs.

<comment><USER8> Um, maybe they didn't need client side rendering? Plain 'ol server-side HTML templating was faster and simpler?

Not every web project needs react, or even JS.

<comment><USER9> At this point, I'm not sure I agree. I am...not a fan of JavaScript, to put it mildly (though ES6 does a lot to suck less). But for my money, nothing out there is better at templating even static HTML than React and JSX. The compositional nature of components is _way_ better than any other templating stack I've ever used and it's pretty easy to bake it into static HTML (Gatsby and `react-html-email` being awesome projects for it).

I'm sure there are declarative, object-oriented (rather than text-oriented) templating engines out there that use an approach like React's. But I would consider using an imperative, text-oriented templating language a yellow, if not red, flag in 2017.

<comment><USER8> I use Twirl, included in the Play Framework (Scala/Java).

It is functional (well, as functional as React), and templates compile to plain 'ol functions, so compatibility and static typing is the same as the rest of the your program.

Obviously, if I needed a SPA or something, it's not what I would use, but again, not everything should be an SPA.

<comment><USER9> You don't need to use React as a SPA, though, is what I'm saying. (When using React, those component trees, too, compile to 100% plain-ol'-functions.)

Twirl is fine, insofar as it's attached to the Play (that's not to impugn you for picking it, my history with Play is colorful and frustrating). I wouldn't raise a flag for that. But _not_ using something in this vein definitely is, and React is probably the most accessible way to do it for the 90-95% case of developers.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> The other point that's commonly missed is that Hadoop is really only useful when both inputs _and_ outputs are too large to fit on one machine. If you have a big-data input but a small-data output (which is very common in a lot of exploratory problems), you can get away with a simple work-queue setup that sends results into a shared RDBMS or filesystem.

At the beginning of my current project, I had a job that involved 35T of input, but the vast majority of records would be ignored, and then for each successful one, only a few hundred bytes of output would be generated. Rather than Hadoop, I setup a simple system where a number of worker processes would query Postgres for the next available shard, mark it as in-progress, and then stream it from S3 and process it. When they finished, they'd write a CSV file back to S3. The reduce phase was just 'cat'.

The resulting system took a few hours to build (few days, including the actual algorithms run), and it was _much_ more debuggable than Hadoop would be. You could inspect exactly where the job was, what shards had errored out, and which were currently running on machines, and download & view intermediate results before the whole computation finished. You could run the workers locally on a MBP if you needed to debug a shard, with no setup needed.

When I was at Google, we had a saying that "The only interesting part of MapReduce is the phase that's not in the name: the Shuffle". [That's the phase where the outputs of the Map are sorted, written to the filesystem and eventually network, and delivered to the appropriate Reduce shard.] If you don't need a shuffle phase - either because you have no reducer, your reduce input is small enough to fit on one machine, or your reduce input comes infrequently enough that a single microservice can keep up with all the map tasks - then you don't need a MapReduce-like framework.

<comment><USER11> > it was much more debuggable than Hadoop would be. You could inspect exactly > where the job was, what shards had errored out, and which were currently > running on machines, and download & view intermediate results before the > whole computation finished. You could run the workers locally on a MBP if > you needed to debug a shard, with no setup needed.

To the extent that's true it's an indictment of Hadoop's implementation. Doing all those things in Hadoop ought to be trivial; maybe there are a few tools you'd have to make a one-time effort to learn, but reusing them ought to save you effort over making a custom system every time.

<comment><USER12> That was certainly the case in the past. We backed away from it ~5 years ago after wasting too much time investigating cases where a daemon had leaked memory, hit the heap limit, and failed in a way which caused work to halt but produced no visible error state other than a stack trace in one of the many log files. Local testing wasn't viable since you needed something like 16-20GB to run stably, and back then we didn't have individual dev systems with enough RAM to run that kind of overhead and still have enough left for the actual work.

</comment></comment></comment><comment><USER13> One problem I've noticed is that there are no good "medium data" tools.

Column stores are crazy fast, but there isn't much simple tooling built around things like parquet or ORC files. It's all gigantic java projects. Having some tools like grep,cut,sort,uniq,jq etc that worked against parquet files would go a long way to bridge the gap.

Something like pyspark may be the answer, I think it may be possible to wrap it and build the tools that I want.. like

      find logs/ | xargs -P 16 json2parquet --out parquet_logs/
      parquet-sql-query parquet_logs/ 'select src,count(*) from conn group by src...'

I've been testing [https://clickhouse.yandex/](https://clickhouse.yandex/). I threw it on a single VM with 4G of ram and imported billions of flow records into it. queries rip through data at tens of millions of records a second.

Edit: another example... I have a few months of ssh honeypot logs in a compressed json log file. Reporting on top user/password combos by unique source address took tens of minutes with a jq pipeline. The same thing imported into clickhouse took a few seconds to run something like

      select user,password,uniq(src) as sources from ssh group by user,password order by sources desc limit 100

<comment><USER14> For "medium data", my company has found a lot of success using dask [0], which mimics the pandas API, but can scale across multiple cores or machine.

The community around dask is quite active and there's solid documentation to help learn the library. I cannot recommend dask enough for medium data projects for people who want to use python.

They have a great run down of dask vs pyspark [1] to help you understand why'd you use it.

[0] [http://dask.pydata.org/en/latest/](http://dask.pydata.org/en/latest/)

[1] [http://dask.pydata.org/en/latest/spark.html](http://dask.pydata.org/en/latest/spark.html)

<comment><USER15> I've been trying to change all of my Luigi pipeline tasks from using Pandas to Dask, so that I can push a lot more data through. Seems like an easy process so far, and I like the easy implementation of parallel computing.

</comment></comment></comment><comment><USER16> There's a meta-pattern here. I've seen countless (especially younger) dev teams focus on new and hot tech to the detriment of solving the business problem. I've seen this happen in two, sometimes overlapping, cases:

1\. When the devs' agenda is about learning new tech rather than solving business problems. The ways to solve this are to incentivize devs at the business problem level (hard) or find devs who care more about solving business problems instead of learning hot new tech (easier).

2\. When the product management function is weak within an org. Product defines the requirements, and makes trade-offs around the solution. A strong PM will recognize when a bazooka is being used to kill a fly, and will push dev to make smarter trade-offs that result in a cheaper, faster, more maintainable solution. This is especially challenging when the dev team cares more about shiny tech than solving business problems.

<comment><USER17> It's not necessarily devs pushing for hyped technologies that don't fit the business problem.

Half- or nontechnical managers often follow tech hypes as well and may push for the project to use "Big Data technology" simply because it makes them feel more important to lead a project that is part of the hyped topic.

<comment><USER16> Agreed! This falls into the "weak product management" category imho.

FWIW I think most devs could learn how to push back on poor decisions like this made by non-technical managers as it really shouldn't be the manager's responsibility to dictate technology choices for a solution. It _should_ be their responsibility to push the dev team to make tradeoffs in order to achieve a business result.

Devs who are good at understanding the business objectives and pushing the non-technical team to make better decisions are both wonderful to work with and command higher salaries / fees.

</comment></comment></comment><comment><USER18> In 2017, with Spark's Catalyst engine and DataFrames data structure (allowing SQLesque operations instead of requiring writing code in map-reduce paradigms), you can have the best of both worlds in terms of big data performance and high usability. Running Spark in a non-distributed manner may sound counterintuitive, but it works well and makes good utilization of all CPU, RAM, and HDD.

Spark is orders of magnitude faster than Hadoop, too.

<comment><USER19> Spark is great, but honestly who writes pure MR jobs anymore anyway? At least use hive so that your engineers can stick with sql syntax.

There's also Facebook's presto, which is so much faster than hive it will make your head spin!

<comment><USER20> Spark is not pure MR. Spark has Spark Dataframes and Datasets with SQL-like syntax. You can even write pure SQL and not mess with the Dataframe API at all.

Presto is nice, but you can't use it for an ETL job. It is great for analysis.

<comment><USER21> Presto works great for ETL. It supports CREATE TABLE, INSERT and DELETE for Hive data. Many companies use it for ETL. See examples here:

[https://prestodb.io/docs/current/connector/hive.html#example...](https://prestodb.io/docs/current/connector/hive.html#examples)

</comment></comment></comment></comment><comment><USER22> I really like the article and it agrees with what we've been accomplishing at my company. Datasets top out at 2 GB and clients ask for "big data" solutions that don't make sense. My only complaint is that I don't feel comfortable sharing it with my co-workers because of the vulgar image on there... :/

<comment><USER23> For datasets maxing out at 2Gb you can try www.CSVExplorer.com . It might be a good way to give your clients access to their data.

Disclaimer: I built CSV Explorer.

</comment></comment><comment><USER24> Honestly this is an outdated article. In 2017 Hadoop is a more like OS/Platform/Ecosystem with FS (hdfs)/ Scheduler / Applications etc. Spark, Presto, Hive now ensure that you no longer have to write Map-Reduce jobs. While I understand the message about 600 MB being enough to fit in memory and speed offered by command line tools etc. Yet its better to just use Spark (which has local mode with convenient in-memory caching) so that when you have to create a company wide data handling/processing system you can just "plug- in" your code into a Spark/Presto/Hadoop cluster.

Finally if you are truly looking for speed while maintaining portability, these days I would recommend using Docker containers with external volumes created on tmpfs thus providing both speed of an in-memory implementation while being agnostic to both OS and FS.

<comment><USER25> Honestly, you also must factor out the price of distributed computing, and take into account the fact that processing power and disk size have increased a lot in that timeframe also.

Then you don't need a new OS/Platform/Ecosystem with FS (hdfs)/ Scheduler / Applications to maintain, host and secure to compute thing that could be done with local scripts, or even java / cython programs. or distribute on a few servers what can (often) be distributed easily.

that still covers most of the use cases.

<comment><USER26> Not sure that you understand what you're saying.

You don't need a new OS/Platform/Ecosystem to run Hadoop/Spark. It is literally just half a dozen Java apps that you run on typical hardware on your regular Linux OS. That is specifically what it was designed for. So rather than write your own Java apps and distribute it yourself why not use a platform that is proven, performant and with Spark will allow you to do things that you could never write yourself.

<comment><USER25> The OS/Platform/Ecosystem was a reference to the post I was replying to.

To be clearer, I don't think you need such things when you just need one or maybe a few servers talking together when you remove the cost of parallelism (see article about COST) and all the hadoop machinery. I have an OS, platform and ecosystem, it's called unix and works for 95% of my needs.

Frankly I've seen my current project be implemented partly on single node (to be replicated for later scalability) and partly multiprocess python. The python is maybe 10x speedier and has 10% the code and run complexity. We tried to run it on a shared cluster of hundred nodes for giggles, the python finished running before the other one started launching. Not taking into account the setup / maintenance.

The COST model is a good one indeed.

So I'm roughly paraphrasing the article (which I point anyone talking about needing big data to, just before asking how big their data exactly is.)

</comment></comment></comment></comment><comment><USER27> Rule of thumb: if the data fits on an ordinary PC, it's not Big Data. The right tool is probably an Ordinary Database.

<comment><USER26> Rule of thumb: Don't make stupid rules of thumb.

Your point is simply not true. Many users of big data platforms e.g. Spark aren't doing so because of the volumes of data. It's because they want to do machine/deep learning on a proven and popular platform. With technologies like Caffe, Sparkling Water, Tensorflow all available on the one platform.

<comment><USER28> Is Tensorflow available on Spark? That sounds strange.

</comment></comment></comment><comment><USER29> "Don't worry too much if you only have small data. The others' isn't as big as they say, either".

</comment><comment><USER30> There's a really good paper that somewhat formalizes this: [https://www.usenix.org/system/files/conference/hotos15/hotos...](https://www.usenix.org/system/files/conference/hotos15/hotos15-paper- mcsherry.pdf)

<comment><USER31> Excellent! Thanks.

</comment></comment><comment><USER32> Imagine my glee when reading this thread after a long day of trying to get a simple answer as to where to deploy a JAR / where data should be landed.

7 hours later, 4 email chains and around 12 people involved, and still no answer, and they've all gone home...

I've been working in a project for an insurance conglomerate for the past 6 months (total project has been running close to a year) that has very recently hit the rocks.

The client has gotten fed up of our inability to deliver, so canned the long term plans of the project in favour of a stop gap solution in order to fulfil some short-term business goal.

We've acquired a relatively ludicrous amount of hardware (i'm talking $100,000+) for what is essentially the equivalent of SELECT COUNT(*) FROM ALL TABLES, which is then filtered into some slow, unusable BI tool that costs $999+ per user / per year.

The real tragedy is they already have geo-diverse SQLServer instances running that would have allowed them to do this in 1 evening. 1 year and $1m+ in billable's and we somehow are going to fail to deliver even that.

<comment><USER33> Wonder which of the big consulting firms is running this one....

</comment></comment><comment><AUTHOR> Related: [https://aadrake.com/command-line-tools-can-be-235x-faster- th...](https://aadrake.com/command-line-tools-can-be-235x-faster-than-your- hadoop-cluster.html)

</comment><comment><USER34> But..but..all the cool kids are using it!

Seriously though, this also applies to a good chunk of ML "applications". Most of times, a 'simple' SVM would work perfectly.

<comment><USER35> Most of the time, a simple decision tree can handle "advanced AI" requirements.

<comment><USER36> Or linear regression

</comment></comment></comment><comment><USER37> I may be mistaken here, but I thought the point of Hadoop wasn't merely to hold more data, but to do larger, distributed, computations on that data. You might have only 10GB of data but need to perform heavy computations on them, requiring a large cluster, with each worker needing to exchange data with other workers periodically.

<comment><USER38> Yes, this and also parallelizing disk I/O. For example you could fit a 5TB table on a single machine, but if you have an operation that requires doing a full scan (e.g. uniqueness count over arbitrary dates), that will take a very long time on one disk. Yes you could partition into multiple disks, but Hadoop offers a nice generalized solution.

</comment></comment><comment><USER39> Unfortunately still relevant... I can't count the number of times when investors are pushing on a CEO which in turn starts to worry about scalability & ask the team if Cassandra isn't a better fit etc (when the whole data can indeed be kept in memory or some midsize USB key).

<comment><USER40> Cassandra is not limited to use cases that need to scale-out, but it also excells at stuff that needs to be always-on, regardless of data size. Sure, you can do many nice things with data in memory or on a midsize USB key, but when your centralized hardware fails, you have a problem and in the best case you restore from a backup - and during downtime your customers go to your competition.

</comment></comment><comment><USER41> Anyone have suggestions for tools to help with data analysis on moderate numbers of very large records? We're working with close a million records per year, which isn't "Big Data", but the records themselves usually contain over a thousand fields. (Is that "Wide Data"?)

Right now the records are stored in JSON documents, and we're working on ways to process those documents for reporting and analysis. There were some previous attempts to store the records "normalized" in SQL, but it quickly became difficult to modify and maintain.

<comment><USER42> Depends on not only the input, but also the output. What kind of analysis do you to perform?

<comment><USER41> We want to be able to generate things like pivot tables (charts, graphs, etc) using location and/or time data from the records compared with many other fields in the record. We're feeding a little of the data into Microsoft SSRS, but setting it up is time-consuming and the results are slow and ugly.

I've done this in the past with OLAP cubes (SSAS) with much larger numbers of small records. The difference here is there are so many more fields and we have 20-30 new or modified fields each year. It doesn't seem like this should be a unique problem, but I'm having a hard time finding tools that might make it easier and less time-consuming. Maybe I'm just not asking the right questions.

</comment></comment></comment><comment><USER43> In this order, I recommend:

1\. Bash. Can you get the answer you need with awk, sed, sort, etc.? Linux commands behave nicely with CPU and RAM, and "Command-line Tools can be 235x Faster than your Hadoop Cluster" ([https://aadrake.com/command-line-tools-can- be-235x-faster-th...](https://aadrake.com/command-line-tools-can- be-235x-faster-than-your-hadoop-cluster.html)).

2\. SQLite. The maximum database size is 140 terabytes, and SQLite can join different database files together. First I would recommend using bash to cull your data down before import, if possible ([https://www.sqlite.org/limits.html](https://www.sqlite.org/limits.html)).

3\. PostgreSQL. Again, I would recommend using bash to cull your data down before import, if possible ([https://www.postgresql.org/about/](https://www.postgresql.org/about/)).

      Maximum Database Size		Unlimited
      Maximum Table Size		32 TB
      Maximum Row Size		1.6 TB
      Maximum Field Size		1 GB
      Maximum Rows per Table	Unlimited
      Maximum Columns per Table	250 - 1600 depending on column types
      Maximum Indexes per Table	Unlimited

More: "Bane's Law (Going Deep)" ([https://news.ycombinator.com/item?id=8902739](https://news.ycombinator.com/item?id=8902739))

</comment><comment><USER44> It always cracks me up what people consider "big data" to be. I once wrote a few Julia scripts to process terabytes of data from the results of molecular dynamics simulations. Just spin up a few high-memory cloud instances, process for a couple hours, and then shut down. Total cost: about $12. It's amazing what a couple billion CPU cycles per second can do. All without Hadoop, Spark, etc.

</comment><comment><USER45> Something no one has mentioned directly: For startups selling software to fortune 500 companies, they generally have to be very open about the architecture of their software, how it all works, etc. Sales are always a "build vs buy" decision with these companies.

If the architecture diagram looks complex, it makes for an easier sale, i.e. "well clearly that would take us forever to build/integrate/whatever, so buying seems easier" whereas if it's just a database and a app server, it's can be a much harder sale.

obviously all the usual caveats apply, but in general, demonstrating "technical complexity" to the customer is something a lot of sales and CEOs will push for.

</comment><comment><USER46> I'm not sure how valid the last point is (my Data is 5TB or bigger) when you can create a 2TB (memory) x1 instance [https://aws.amazon.com/ec2/instance- types/#memory-optimized](https://aws.amazon.com/ec2/instance-types/#memory- optimized) . Also someone else mentioned using Spark's Catalyst engine and DataFrames data structure. My previous experience with Spark it would automatically swap the data so it would be feasible to work with 5tb of data.

<comment><USER47> The article is 4 years old. Scale numbers accordingly. 5TB of data today can fit in RAM on a single server.

</comment></comment><comment><USER48> Shameless pitch:

Google BigQuery has a 10GB free storage tier and a 1TB per month query tier.

If your data is small enough, you get a fully managed "serverless" analytics database for very little, potentially free.

(work at G)

</comment><comment><USER49> Previously on HN: [https://news.ycombinator.com/item?id=6398650](https://news.ycombinator.com/item?id=6398650)

</comment><comment><USER50> How come the author, HN user yummyfajitas, has not posted for over a month? Was he banned?

<comment><USER51> Just conjecture, but I imagine that he got tired of the mods accusing him of trolling (in his uniformly high-quality and civil commentary) due to his non- standard political views.

</comment></comment><comment><USER52> This is still not an uncommon problem.

Interestingly, we (Cloud Dataproc team) have been trying to work in the opposite direction. A few months ago we launched single-node clusters so people can use Spark on one VM instead of creating crazy-huge clusters just for lightweight data processing. No sense in using a a fleet of cores for simple stuff. :)

Disclaimer - work for Google Cloud (and think we need better tools for simple data processing)

</comment><comment><USER53> If one has tons of workloads (like in hundreds to thousands) to run concurrently on a lot of the data, Hadoop is pretty solid.

YARN is also allowing interesting stuff.

</comment><comment><USER54> AWS is launching instances from 4 to 16TB of memory in the coming year: [https://aws.amazon.com/blogs/aws/ec2-in-memory-processing- up...](https://aws.amazon.com/blogs/aws/ec2-in-memory-processing-update- instances-with-4-to-16-tb-of-memory-scale-out-sap-hana-to-34-tb/)

Even terabytes of data isn't "big" anymore.

<comment><USER55> In the era of "big data", TB was never considered big.

<comment><USER56> it's big enough that your conventional RDBMS and app server stack starts to struggle with it

</comment></comment></comment><comment> dis-sys A 2016 work project I did requires to crawl 100 million web pages from around 100k different web sites each night. There was no existing infrastructure what so ever. I was told that it is a big data project and internet scale processing power is required (and actually budgeted for it).

Built a Python crawler to pull data from the Internet using EC2 spot instances, as they are dirty cheap. After just a few days happy coding, I eventually got to a place where I can reliably (spot instances are not that reliable) pull the data, compress them and download them to our own data centre for processing (policy rather than design decision) for under $50 with the majority spent on transferring the final compressed output.

Big data? 100 million small files a run is a typical small data project, big data means trillions of files a day, or something like exabytes scale data volume.

<comment><USER57> You're defining the term big data as "an amount of data that seems impressive to me". That's a pretty loose definition, and two people could have wildly different ideas.

I prefer to define big data as "an amount of data that cannot fit on any one computer." Once you start using distributed systems, either with sharding, NoSQL, or similar tech, you're in big data territory. Going from one computer to two you greatly increase the complexity of the system.

</comment></comment><comment><USER58> More and more I reach for pyspark for medium sized tasks because I like the API and if I need to scale it I can. I don't think this is as black and white as it used to be now that big data framework interfaces are getting as good or better than small data interfaces.

</comment><comment><USER59> We fell in this "trap" as well. Whilst working on a marketing automation system, we were integrating a Google Analytics/Piwik clone. Our guestimates indicated we were going to be storing around 100GB of events per month. We geared up and got working. The team built complex Hadoop jobs, Pig & Sqoop scripts, lots of high-level abstractions to make writing jobs easier, lots of infrastructure, etc. etc. After about 2 months we scrapped the "big data" idea and redid everything in two weeks using PostgreSQL. As most of the queries were by date, partitioning made a huge difference.

I recall one of the classes was named SimpleJobDescriptor. At the near end it was 500+ lines long. Not so simple after all.

</comment><comment><USER60> I didn't even read the article, but I agree with the title already. The amount of companies asking for knowledge in Hadoop for data only bigger then excel limit is not trivial... At least in the area where I work, this is the case

<comment><USER61> I hate these sorts of titles, because some of us DO work with petabyte scale datasets. I know the article does mention the types of use cases that Hadoop is good for, but the title just comes across as arrogantly dismissive.

<comment><USER62> And we cheer for you guys. We really do. You push the edge, it's the right thing to do.

Some of us though, from time to time, have these side company projects. Then ,after a round of consultants and meetings (and nobody asking us for input) we had this 500MB database, and an architect who was pushing for all the big data keyword to be on his resume, even though all math and business numbers said we'd double that in _twenty years_.

We are talking zookeepers and 20 servers per geo separate datacenter, spark, ml, Casandra, hadoop, MR, java everywhere, logstash, etc etc etc. Presentations in hundreds or PowerPoint slides and endless meetings about direction and future.

So the article, and MySQL, have it's places. Especially when one can finish the job in two weeks.

<comment><USER61> Of course, I am not saying that there aren't people who misuse 'big data' technology for 'small data' problems. You could even keep a provocative title like, "Are you sure you need big data?" or, "Your data must be this big to ride" or something else both cheeky and not dismissive.

</comment></comment></comment></comment><comment><USER63> I've been around this loop over the past few months. Something to bear in mind: Hadoop of 2017 is quite different to the Hadoop of 2013.

Our data isn't enormous, by any means - 160G in one particular instance that's being used for proof of concept, it'll add up to 5-20T should it reach production. The catch is that it's 160G in MySQL; it's only 5G or so once it's been boiled down to Parquet files in HDFS. Columnar stores can be a really big win, depending on the shape of your data.

We use Impala for our queries. It's quite good tech; it's much faster at table scans than everything that doesn't describe itself as an in-memory database. That means writing SQL much like you would with Hive, only it runs faster.

I tried out both Citus and Greenplum to give PostgreSQL a fair shot. First problem: PostgreSQL is limited to 1600 columns in a table, and the column limit for a select clause isn't much bigger. We have several times this number of columns in our largest analytic tables. Not the end of the world, you can cobble things together from joins and more special-purpose tables.

Second problem: CitusDB doesn't come OOTB with a column store, and it's far too slow when using a row store. I didn't bother trying to compile the column store extension to use with Citus; the pain ruled itself out. I continued ahead with Greenplum, focusing on columnar storage - row storage is consistently poor.

Third problem: Greenplum is cobbled together from a pile of duct tape, an assembly of scripts and ssh keys to keep the cluster in sync. It does not inspire the same kind of confidence for operational management as HDFS, whether rebalancing the cluster, expanding the cluster, or decommissioning nodes (not supported with Greenplum, AFAICT).

Fourth problem: Impala simply runs faster than the Postgres derivatives, and its lead increases the more data you have. Impala seems to do table scans over twice as fast on identical deployment environments.

Indexes only help when the operation being performed can use the index. As it happens, most analytic queries do full scans, or have predicates that are either not very selective (randomly skipping rows here and there) or are really selective (date bucketing, which maps well to typical Hadoop partitioning strategies). I had some hope that indexes would help for joins; but Greenplum didn't elect to use my indexes, and when I forced their use, it ran slower. The ancient version of Postgres that Greenplum is forked from doesn't help much either, since it can't e.g. use covering indexes to avoid looking back to the table.

If it was my startup, I'd take a risk on something like MonetDB, or look harder at MemSQL, given what I've seen about how data has shrunk with column stores. But from what I've seen and measured, Postgres doesn't really cut it for analytic queries.

<comment><USER64> > PostgreSQL is limited to 1600 columns in a table, and the column limit for a > select clause isn't much bigger. We have several times this number of > columns in our largest analytic tables.

Is that typical for this kind of work?

<comment><USER63> What's "this kind of work"? :) The company I work for does reconciliation as a service, something that's pervasive in finance. We support customer-defined schemas; in fact that's one of our selling-points. I could start talking about the kinds of things we've done with MySQL to make this perform well - it's not difficult, just a bit unorthodox.

Anyway, the diversity in customer schema leaks out into the Hadoop schema, where we'd much prefer to give customers data using column names they're familiar with, and we also want to give them rows from all their different schemas in a single table (because many schemas have overlap by design). The superset of all schema columns is large, however. The problem can be overcome with more tooling - defining friendly views with explicit column choice - <USER2> having the option to implement that (and go to market sooner), vs a requirement to implement that, adds up to a distinct advantage for tech that can support the extra columns.

<comment><USER65> Edgar F. Codd just started spinning

<comment><USER63> Something you need to bear in mind is that distributed joins are very expensive; you have a better time designing your schema such that related data can be placed logically close together, whether it's arrays / maps inside rows (for one to many), or very wide rows (denormalizing what might be a star schema).

(I know, in a column store having related data in another column isn't actually close together; but it can be stepped through at the same time, it doesn't need a join to be correlated, it's correlated naturally.)

</comment></comment></comment></comment></comment><comment><USER66> This reminds me of [https://news.ycombinator.com/item?id=8908462](https://news.ycombinator.com/item?id=8908462). Use command line tools when applicable

</comment><comment><USER67> I agree with the sentiment of the article, but why does this site ask for notification permissions? It's a blog - if I want notifications about new posts I'll add your RSS feed to my reader.

</comment><comment><USER68> I'm biased in that I've been doing nothing but Hadoop consultancy for almost 10 years now.

I have done maybe a hundred to two consulting hundred projects based on Hadoop (as you can imagine it's mostly short-term and troubleshooting stuff) and am a ASF committer myself. So take this with a grain of salt.

I've seen the kinds of projects the article and lots of people in here refer to and I agree with lots of what's been said.

I also agree that Hadoop (the whole ecosystem) probably not has a single "best of category" tool that ticks all the checkboxes but IMO that's totally missing the point.

What it gives you is: \- (Mostly) Open source stack (compare that to your Informatica/Talend/IBM stack that big companies have and not to your Python jobs they don't), lots of the code is crap (see my Apache JIRA history) but it's there and you can attach a debugger to your running cluster, that's fantastic and very hard to do with e.g. Python (at least I'm not able to find good ways of doing it)

\- Proven technology, it works most of the times and if not there's tons of SO answers, mailing lists, books.

\- All the boring enterprise stuff: Encryption at rest and on disk, strong authentication, strong authorization, auditing, high availability, failover etc.

\- Integration into Monitoring & alerting tools, distributions figure out the important stats and thresholds for you

\- Mostly easy to operate these days even at large scale thanks to Cloudera Manager et al. - no need to manually run/install stuff

\- "Coolness factor"

\- Costs way less than established tools with e.g. external storage filers or similar specialized hardware

\- It's not trivial but also not super hard to find people with Hadoop/Spark experience

\- Existing purchasing agreements with Hardware vendors or Cloud providers

\- Business/C-level gets and supports it

Edit: Three more I thought of:

\- You can buy licenses and support for the tools. Both of which have been discussed multiple times here on HN, a donation doesn't work that well for companies

\- You can buy indemnification from license problems from at least Cloudera but I think Hortonworks as well

\- If the same stack has been used elsewhere in the company already chances are there are processes in place, someone has already vetted it, done the open source checks etc.

See what I don't mention are things like performance or absolute money amounts or even the amount of data (as the whole ecosystem is much more than "big data" now) etc. as we see surprisingly little projects that care about specific SLAs or features. It's all relative.

That said: At least 50% of the projects would still be better served by a simpler solution but that still leaves a whole big chunk of projects where Hadoop makes sense for other than technical or monetary reasons.

But I very much disagree with this part of the post: > The only benefit to using Hadoop is scaling.

</comment><comment><USER69> I agreed with the article at the time it was posted. Now I use Spark for various sized data sets. Specifically Zeppelin + Spark is a great combination.

Then again, Spark doesn't really need Hadoop, I see more and more people using it with Kafka and Elasticsearch for all sorts of fun stuff.

And as other commenters pointed out, you get read-only SQL (very powerful SQL) for free. The other day I joined an elasticsearch result with a CSV file in SQL.

</comment><comment><USER70> In a world you can buy a 24 TB RAM 8-socket 192-core 384-thread box able to hold 20 2.5" devices and a mix of 16 devices picked from NVMe storage, GPUs or other coprocessors (giving you up to 16 coprocessor hosts, each with 72 x86 cores, giving you 2304 threads and 256 GB of RAM), the odds of you actually having big data (as in "intractable from a single box") are remarkably small.

</comment><comment><USER71> I still enjoy remembering a friend who works at one of the week known big data companies tell me "our team works on tools for medium data"

</comment><comment><USER72> I feel like there should be a general law or rule of sorts -- don't use technology for the sake of using technology.

</comment><comment><USER73> I'd add that the benefits of avoiding bigdata(tm) can extended further into the 5TB+ space if you're happy to run at a bit of a delay.

I.e. If it's ok for the crunching to take 6 hours to produce intermediary aggregates which can then be crunched in less time , you can avoid spark and Hadoop for much longer.

</comment><comment><USER74> Nice article and it still holds despite being a few years old. There's a general lack of understanding or appreciation for this "medium" data world which is actually where most of "big" data lives (i.e. too big for Excel but not PB or many TB).

</comment><comment><USER75> There's a very similar argument in this article that's trended on HN a couple of times before: [https://news.ycombinator.com/item?id=12472905](https://news.ycombinator.com/item?id=12472905)

</comment><comment><USER76> Looking at the comments, one valid reason a company might be using big data without actually having big data is the potential need to scale and grow. Don't forget that many of them are expecting a 10x increase in users.

<comment><USER77> Don't forget that most of them never get there. Often because they are not able to find & make the product which the market wants/needs. Sometimes partly because engineering is too slow and unadaptive, possibly because they are overcomplicating things unnecessarily.

</comment></comment><comment><USER78> Yup, 600mb is pretty tiny. I have a netbook from 2011 that I used for crunching numbers for my dissertation. It can handle 600mb csv files in memory using numpy. The slowest part is reading from disk.

<comment><USER79> The craziest example I saw was a blog post titled something like "How I used big data to analyse my problem" and she was talking about 1200 rows. Big data that could fit in a floppy disk!

Buzzwords are hell of a drug

</comment></comment><comment><USER80> Don't use Hadoop even if your data is big. Spark replaced it.

<comment><USER26> What are you talking about ?

Hadoop is basically HDFS + YARN. Spark is a replacement for MapReduce only.

<comment><USER80> Except cloud computing. HDFS is only useful if you own hardware. HDFS is still useful for these workloads, but calling it "Hadoop" when you are using "Spark" as the execution engine doesn't make sense. You are using Spark, which depends on HDFS for local installs only. And if you can be in the cloud, you should.

It is also worth noting that many people use Spark against a database like Cassandra and not HDFS. So it isn't even universal for local installs.

I was a Hadoop evangelist, but its time has passed. It is a foundation, not the tool you use to get work done.

</comment></comment></comment><comment><USER81> > Don't use Hadoop when your data isn't that big

Did we really have to weaken the title's statement? Bending to the common denominator will always lead to milquetoast titles.

</comment><comment><USER82> In my experience, there are not many problems in the course of everyday business that cannot be solved with SQLite's CSV import feature and/or its virtual tables.

</comment><comment><USER83> > Too big for Excel is not "Big Data"

This. Should be sold as posters with pre-printed "Happy birthday mr manager".

</comment><comment><USER84> "There is no computation you can write in Hadoop which you cannot write more easily in either SQL, or with a simple Python script that scans your files."

How about base64decode? it is possible in SQL but not pretty. There are other examples as well. I do agree that Hadoop is probably an overkill for 90%+ of uses but there are certain set of problems which are helped by having a general execution environment.

<comment><USER47> You conveniently ignored the "or with a simple Python script that scans your files" part.

</comment></comment><comment><USER85> For 95% of applications Hadoop is overkill

</comment><comment><USER86> The thing that's so frustrating to me--and many others, based on the comments --is how much of this gets pushed to management and product people.

Our team just acquired a product manager for the first time. And for the most part I absolutely adore her. She's really great at pacing things out to a timeline that works for us and pulling us out of our idea that a hacked up solution that you do in a couple of days isn't really a product. And she's very much pushing against putting those into production, all of which I agree with.

But she doesn't understand technology or needs. She's constantly having meetings where she pitches our manager on product ideas that don't even make sense compared to the client needs. Many of these are buzzword-based. But it makes me seem like an asshole where every team meeting, I have to be like, "No. That's not what the client asked for. The client asked for a solution to this problem x. The client (doesn't matter if it's internal or external) often doesn't know what the right solution is. Good product people and managers know that to get to a good place, the key is to get the client to describe the problem instead of the solution.

I'm not a product person. I don't know jack dick about products except for how to make them. But the level of miscommunication I see even in our small organization is astounding.

You get one manager, one product manager, and two stakeholders, and one engineer in a meeting together. It's absolutely unbelievable to me what management and product take away from those meetings.

It's so far from the reality of the problem to be solved that it sometimes makes me nuts. Like it has today.

I think that our whole way of doing things and our hierarchy is completely broken and backwards. You want to get something done? Put a software engineer in a room with an operations person who is on the ground. Talk about the problem. Propose a solution.

Take it to the management and let them sort out priorities among themselves. This bullshit about management being the first point of contact virtually guarantees that buzzword-based development is going to happen.

I know that we are supposed to be the sacred cows, and that we need to be protected. Fuck that, I say. Let me interact with the people on the ground whose problems I'm trying to solve. Don't put a non-technical product person who doesn't get any of the details right between me and the end-user. I'll take the time out of my day two days per week to go to the office and have meetings, and I do the rest at home where I can be productive and write code.

Sorry for the rant, but fuck all of this. We have a very broken system, in most cases.

</comment><comment><USER87> Hadoop's carbon footprint should make it one of the top threats to civilization.

</comment><comment><USER88> I don't know, 21PBs seems like a lot to me....

<comment><USER64> From the article:

> But my data is more than 5TB! Your life now sucks - you are stuck with > Hadoop. You don't have many other choices (big servers with many hard drives > might still be in play), and most of your other choices are considerably > more expensive.

</comment></comment><comment><USER89> Data are plural, eg: "Data arent't that big." I believe datum is singular.

<comment><USER90> Not true in the vast majority of usages. Maybe in scientific writing, but this is a blog post.</comment></comment>


Jitsi Meet Security and Privacy - <AUTHOR> https://jitsi.org/news/security/ <comment><USER2> Short answer: like Zoom, multiparty Jitsi meetings are encrypted point-to- point, not end-to-end, and Jitsi can monitor and record your multiparty meetings.

<comment><USER3> ... unless you self-host the server. What you can do with jitsi, but not with zoom.

<comment><USER4> Then its still not end-to-end encrypted. The difference is that the only point where its no encrypted is under your control as well. So you're right that your meetings cannot be recorded if you host your own instance.

Unlike Zoom, Jitsi never claimed that their system was e2e encrypted which is a huge difference if you ask me. Apparently its a technical limitation of WebRTC, so I'd assume no webbrowser based solution can be e2e encrypted as of now. Which is ashame since the browser compatability makes meetings with people outside your company (or with less tech savy family members) so much easier.

edit: misunderstoot your point

<comment><USER3> Thanks for the clarification!

As your edit already states, not the point I was trying to make, but still, good to clear this up.

There was interesting discussion yesterday by the devs of mediasoup, why it is not in webRTC yet: [https://news.ycombinator.com/item?id=22761816](https://news.ycombinator.com/item?id=22761816)

</comment></comment></comment></comment><comment><USER5> Are there any cryptographic and/or network designs which allow end-to-end encryption of a group video chat without full meshing?

<comment><USER6> The point is not the crypto, but what WebRTC supports. As of now it does not. See [https://webrtchacks.com/you-dont-have-end-to-end- encryption-...](https://webrtchacks.com/you-dont-have-end-to-end- encryption-e2ee/) for reference

<comment><USER7> Of course WebRTC is only a requirement if you want to be browser-based. And even then, some solutions use WebRTC data channels for video instead (e.g. Zoom's web version). They could do it, although you'd of course still be trusting the server to serve you the right thing.

</comment></comment></comment><comment><USER8> Wouldn't IP multicast (if it were routed by ISPs) be a perfect fit for a scalable p2p video conference solution? Crypto wise, I guess, a room key could be negotiated.

</comment><comment><USER9> If you are using webrtc in the browser e2e is not possible. Which is why it's critical you have the ability to self host such solutions.

<comment><USER10> What are you talking about? Webrtc has mandatory encryption. The only issue is using a simulcast middleware server. If you're doing mesh p2p then it's e2e.

<comment><USER9> Obviously - I was referring to when you have a SFU, such as Jitsi and comparable to zoom.

The concept of e2e encryption doesn't even make sense in p2p, the fact that direct communication between two peers is "end to end encrypted" is self evident. It's only when there is a central server that talking about e2e as a concept is relevant.

<comment><USER10> No? TURN servers relay webrtc traffic and it's still e2e encrypted

</comment></comment></comment></comment><comment><USER11> Basically same situation technically as Zoom, without the PR fiasco

<comment><USER12> Except with Jitsi, the server that can potentially see the traffic is under your own control.</comment></comment>


The Day You Became a Better Writer by Scott Adams - <AUTHOR> http://dilbertblog.typepad.com/the_dilbert_blog/2007/06/the_day_you_bec.html <comment><AUTHOR> One para in the article says: Humor writing is a lot like business writing. It needs to be simple. The main difference is in the choice of words. For humor, don't say "drink" when you can say "swill." "Drink" should be simpler than "swill" to me. What do you say?

</comment><comment><USER2> (June 2007)

<comment><AUTHOR> I got the hint that I was supposed to mention the date of the article in the title. It's my first post on HN:) I'll take of this point in future posts. Thanks:)

<comment><USER2> It warrants attention only if it's not recent in origin (say, the past few months or so).</comment></comment></comment>


Best Remote Jobs Websites in 2019 - <AUTHOR> https://remotehub.io/blog/remote-jobs-websites <comment><USER2> Don't discount the power of posting on LinkedIn for all recruiters who have remote positions to please contact you. I know some recruiters are forbidden by their clients to name the company that's hiring remote workers either because the company's PR dept is insanely jealous of controlling how their name appears in all forms of media or because they don't want their on-prem FTEs to realize they could probably be working from home... or both.

</comment><comment><USER3> Remotehub would be more useful if you could filter/search by role/position and programming language/technology/skill i.e. software engineer, or developer, and java or python or ruby etc

<comment><AUTHOR> Thanks! Yep, makes sense. I implemented this today: [https://remotehub.io/founder/remote-jobs- skills](https://remotehub.io/founder/remote-jobs-skills)</comment></comment>


Like Monopoly in the Depression, Settlers of Catan is the board game of our time - <AUTHOR> http://www.washingtonpost.com/wp-dyn/content/article/2010/11/24/AR2010112404140.html

<comment><USER2> Catan "serves as a model for solving contemporary problems such as trade imbalances, nuclear proliferation, and climate change." Huh? Catan is a GAME! You play with your friends! It is fun! Its creator was trying to improve the gameplay of Monopoly, not solve the world's problems. Why oh why must journalists strain to tie everything in with their conception of the zeitgeist.

</comment><comment><USER3> .../what version/ of Settlers? Original Flavor gets dull after a short while, I find.</comment>


Chrome Bug Lets Attackers Steal Web Secrets via Audio or Video HTML Tags - <AUTHOR> https://www.bleepingcomputer.com/news/security/chrome-bug-lets-attackers-steal-web-secrets-via-audio-or-video-html-tags/ <comment><USER2> TLDR: only size of reply in crude ~50KB increments. Nothing as bad as Edge full CORS bypass (+FF size leak) from a couple of months ago [https://jakearchibald.com/2018/i-discovered-a-browser- bug/](https://jakearchibald.com/2018/i-discovered-a-browser-bug/)</comment>