How Australia's $49B internet network came to be ridiculed - <AUTHOR> http://www.bbc.com/news/world-australia-41577003 <comment><USER2> First common mistake: Yes, Australia is geographically huge. No, it is not sparce. it is a highly urbanized economy in terms of geographic concentration of the population. Therefore, the oft-quoted extreme expense "oh, its so much more expensive to do it in OZ compared to ..." is bullshit: its not. There are of course some very expensive locations to service, but thats a USO problem. Overall, the cost of deployment of the CAN is within scale of any other urban economy. Higher than Singapore? Sure. But, comparable per head of population to NZ.

Second common mistake: The argument which led to the MTM replacing FTTP was not about technology. It was about the cost of capital works, and the economic model of recovery. Labor was stupid enough to float a mixed-economy pluralist argument hinting at a future independent non-utility model with zero cost (long term) to the government. This hung them because it allowed the Liberals to go hard on cost recovery and privatization. If the intent had been to reserve a national utility function in layers 1 and 2, and promote competition above this, we'd be in a different conversation.

That conversation was about structural separation of the former monopoly Telco. It was quashed by Labor, in a stupid reactive move. It was floated, and the banking sector indicated a preference. Reactively the union movement and Labor party moved to reject it: if the banks wanted it, it must be bad.

Talk about a huge missed opportunity.

The NZ model, by comparison is gold: structural separation. Crown holdings for the fiber. A pot of money, re-circulating to deploy to the cities, each region re-fills the pot to move to the next town. Far simpler. And.. fibre to (most) homes.

<comment><USER3> Yeah this is spot on.

Other comments are blaming the Liberals, but the reality is if you are an effective leader trying to create a huge infrastructure project that you care deeply about, you need to come up with a solution that will last through a change in government, with aligned economic and political contingencies. If you are just governing for fun then you do what Rudd did. It always lets you blame the next government though, as he is doing now.

<comment><USER4> To the Aussie's credit the infrastructure and the regulations instilled on businesses there have always been Federally imposed.

Since the mid-90s the government has imposed many regulations for control that has impeded much of the free market progress for network infrastructure. Those regulations coupled with needed external transit installations have not helped. For a long period of time the government would pick and choose companies based on their ideals rather than their talent. This has hindered progress from both sides of the fence by liberal and conservative administrations.

I ran a semi successful ISP and consulting company for a decade in the US and I have always enjoyed the tiering system we setup vs other country Infrastructures. It wasn't until recently when the FCC gained control that states started to use the new regulation power to hinder progress. I am still on the fence about the whole thing because the exploitation has not been bad in my area but the horror stores I have heard in other states make me sad. Hopefully it will get back to normal.

<comment><USER2> I'm not personally sold on the baby bell model of regional monopoly. I think the whole FIOS debacle, and what happened when google started offering citywide fibre says a lot about how US telco industry operates. Lobby your way to the people who make rules and then get rules passed to shut down anything which freaks your own monopoly of indifference.

<comment><USER5> I am curious and not sure what to search for. What are you referring to? Link?

<comment><USER5> What was the "whole FIOS debacle, and what happened when google started offering citywide Fibre"? Why the down votes? This sounds interesting, but I am not sure what to search for to learn more.

<comment><USER2> I do not live in the USA. What I am told by my colleagues who do, is that the post bell regional breakup created monopolies by location in order to prevent national dominance but give each region a reliable income in a time of voice call logistics. Fast forward to the emergence of fiber as a viable technology: either you live in a location serviced by a telco who is willing to re- engineer to fiber or you don't. Verizon exemplifies the quandary, it's a huge cost, they have to carve out special data models for cable companies to stop them refusing to drop coax, it's impossible to charge sanely, their shares tank. I am sure there are happy customers but there is no clear equality of service. It's a patchwork minefield.

</comment></comment></comment></comment></comment></comment></comment><comment><USER3> By 'fit for service' they mean 12Mbps down, or average copper wire speeds. What's the point?

Although the original goal was great, in hindsight it appears that it could not be delivered anywhere near the original budget. For context, after a prolonged period of centre-right government, a centre-left government was elected, led by Kevin Rudd. He had a progressive agenda, but due to a combination of his unmitigated egotism and internal party conflicts, he instituted a variety of programs that were poorly conceived and executed, and was ultimately ousted by his own party in a fairly disasterous manner. The National Broadband network was a product of this highly disappointing and dysfunctional period of government. Since then the problems have continued across a change back to the centre-right government, who with a very slim majority have essentially stuffed around for the last few years with a lame duck leader who seems hamstrung by the ultra conservatives in his own party.

For my part, I'm dropping all wires and switching to 4G/LTE, where I can stream 4K content, get decent uploads and still pay less than I would on the National Broadband network.

<comment><USER6> Where do you get your cheap 4G/LTE?

Best I can find is prohibitively expensive, especially if I wan't to stream 4K content...

<comment><AUTHOR> Vivid Wireless offer unlimited 4G data capped at 12 Mbps for $90 a month. Its on the Optus network so your experience will depend on what the coverage is like where you live.

<comment><USER7> > Its on the Optus network

Actually it's a completely separate network to Optus' 4G network. Optus bought them several years ago.

It's also highly throttled and utterly shite for most users. I was one.

<comment><AUTHOR> Useful to know. I was considering moving over to them to save money on my ADSL2+ connection. My area has fantastic 4G speeds without any congestion issues as its semi rural and I was expecting them to be quite good.

<comment><USER8> Have been with vivid for about 12 months, was supposed to be a temporary arrangement while I waited for my building's strata to fix the internal wiring between my apartment and the MDF in the basement. All of the NBN horror stories and the strata's malignant incompetence mean I'm still using it and it's pretty solid in my experience.

I'm in central Sydney and normally average around 10mbps down

</comment></comment></comment></comment></comment></comment><comment><USER9> This article didn't mention what Australians are choosing to do when they have a range of speeds. According to the figures NBN released 80% of customers aren't even paying for the top speed they could. Similarly, this piece from Vox notes that fast internet just is not nearly as valuable as proponents claimed.

[http://www.theaustralian.com.au/business/technology/only-17p...](http://www.theaustralian.com.au/business/technology/only-17pc- of-nbn-users-choosing-highspeed-packages/news- story/20ab3be87d9cc9ca6760c3889e29fb41)

[https://www.vox.com/new-money/2016/10/17/13230500/gigabit- ne...](https://www.vox.com/new-money/2016/10/17/13230500/gigabit-networks- chattanooga-google)

<comment><USER10> The craziest part is that I was getting 100/60 that actually ran FASTER than it's advertised speed, unlimited (truly) for $69 a month, with TPGs private FTTB network. But I had to move, and now I'm on an Opticom network; $99 a month for slower upload, but I do get a static v4 IP, so that's a plus for what I use it for. And this is all in Fortitude Valley, basically the centre of Brisbane...

<comment><USER11> 'tis true. Can vouch because that's the plan/infrastructure I'm on in Melbourne.

I can only extrapolate my own experience, but perhaps if there was some truth in advertising enforcement. I've had NBN plans elsewhere, and I never chose the top plan, because I'd still be limited by other parts of the internet I was communicating with in practice, and everyone knew the top rate plans quoted speeds are literal nonsense. Why pay extra when you get the same performance on the bottom/mid plan?

</comment></comment></comment><comment><USER12> Tasmania got very lucky with the rollout, we are mostly covered with FTTP, Fixed Wireless and only some FTTN.

We are also lucky enough to have a local business (Launtel) pushing the constraints of the NBN by offering Gigabit at prices that aren't unreasonable ($14.50/day). They are treating it as a new pricing model, why bill the same as ADSL when there are no costs associated with setup and plan changes? They are billing daily which allows more flexibility meaning you could spin up a connection for a day (At an Airbnb for example) or change speed depending on your usage.

This is all in beta for them at the moment and only available in Tasmania. Here is a speedtest I ran yesterday from my home FTTP connection [https://www.speedtest.net/result/6738001151.png](https://www.speedtest.net/result/6738001151.png)

<comment><USER13> Wow. Just wow. My advice is buy cheap real estate and start an internatianal incubator. The weather alone should keep people indoors and focused... only half joking!

<comment><USER12> It was like I had jumped into the future when this was connected.

The Tasmanian Government is actually offering grants to incubators at the moment [http://www.startuptasmania.com/incubator- support](http://www.startuptasmania.com/incubator-support)

</comment></comment></comment><comment><USER14> The tl;dr is that Australia started a fibre-to-the-premises project for the whole country (93% of premises, plus rural 7% on wireless), then a conservative government got in power and decided to deliberately destroy it. To do this, they changed it to fibre-to-the-node (DSL) instead. Instead of gigabit to the home, they're spending more money providing people with connections anywhere between 4mbps and 100 at the absolute most, with no possibility of upgrades. The cost to do this, is, of course, far more money in the long term than installing the fibre. Some people in metropolitan areas will get connected to the existing coaxial network, which is going to have huge contention problems as it's stretched far beyond what it was ever intended for.

It's the biggest trainwreck of an infrastructure project in Australia's history and will set every person and business back for decades, and the blame lies squarely with the Liberal Party.

The current head of the project has gone on record saying that Australians don't want fast internet and wouldn't use it if it was free.

<comment><USER15> Wait, if the Conservative party decided to destroy it how is that the fault of the Liberal Party?

<comment><USER16> The Liberal party is the conservative party. No Conservative party in Australia AFAIK.

The Liberal party is broadly social-conservative, pro-business.

The Labor party is broadly speaking socially liberal and pro-labour.

<comment><USER17> You're on the money, but just a bit out of date. We have a (very) new far- right party called the Australian Conservatives now. The creation of a former Liberal party senator.

<comment><USER18> Ahh, make australia great again.

</comment></comment></comment></comment></comment><comment> dis-sys I lived in Australia for 12 years, they fed me fake news about the progress of the NBN on weekly basis. In the end, they didn't deliver it at all - I was living in Sydney's inner city suburb but there is no way to get it 10 years after the promise was first made.

Moved back to Shanghai in 2017, was really happy about the fact that CCP didn't play the same game, they deliver concrete results. 500Mbps fiber costs me $450/year, with an extra $20/year, I can screw the GFW - streaming 4k from youtube is easy and I am watching it now.

I really like the fact that there was no general election topic or endless + meaningless debate on whether such a fiber network is good or not, CCP just deliver what they believe is good and take responsibility from there. Life is short, how many 10 years you have to waste on craps like that?

It is also worth pointing out that in Chinese definition, wasteful spending on tax money is serious corruption, you'd be seeing trucks load of CCP officials sent into jail should they dare to spend $49b without delivering the network on time.

</comment><comment><USER19> In Australia, the Liberal party leadership is clever enough to design national computer communication networks. Wait.... no they're not even close to that clever. But it doesn't stop them anyway.

What we can be sure of it now that Tony Abbott and Malcolm Turnbull have completely ruined our chance at having a world class NBN, they'll not do what is needed to actually make it world class.

It's really strange that politicians simply are not driven by making things awesome. They're driven by something else and it's certainly not the goal of getting to the best possible outcome for the people and the country.

So I have HFC coming into my home right now.... where is my symmetric 10Gbps connection?

[https://www.nbnco.com.au/blog/industry/hfc-what-is-full- dupl...](https://www.nbnco.com.au/blog/industry/hfc-what-is-full-duplex- docsis-3-1.html)

I loathe politicians with the burning heat of a thousand supernovas.

<comment><USER20> Don't waste that anger. Vote left (Or at least be very careful with your preference flows) at the next election. And get everyone you can to start talking/pushing for a Royal Commission into this.

<comment><USER19> I loathe all politicians including the left wing. I just loathe the right wing more.

None of them represent the citizens. They represent their own grasping self interest.

And I certainly don't agree with the far left who seem to have an anti corporate/anti capitalist whiff to them ... I'm a committed capitalist AND green and they is no political party aligned to that.

<comment><USER21> Sick of politicians?

Sustainable Australia party is who you're looking for. William Bourke is eminently sensible. Dick Smith joined. The anti-corruption fighter who wrote "Game of Mates" Dr Cameron Murray is running for them too.

[http://www.sustainableaustralia.org.au](http://www.sustainableaustralia.org.au)

[http://www.sustainableaustralia.org.au/qld](http://www.sustainableaustralia.org.au/qld)

<comment><USER22> Unfortunately, Dick Smith's reputation has been somewhat tainted by getting involved with a racist national disgrace:

[http://www.sbs.com.au/news/article/2017/08/17/hanson-dick- sm...](http://www.sbs.com.au/news/article/2017/08/17/hanson-dick-smith-meet- over-capping-immigration)

<comment><USER21> Somewhat, although he backed away from One Nation after Pauline wore a Burka in parliament.

It's a shame the media has conflated any discussion of an immigration policy with racism (ie because of One Nation)

Dick's not running though, so checkout the candidates and party, they are academics and scientists for the common good.

</comment></comment></comment></comment></comment></comment><comment><USER23> _> After the conservative government attained power in 2013, the NBN struck more upheaval, with copper wire ordered for the final stretches of connection, surrounding neighbourhood broadband nodes._

How would pulling copper make it any better? It surely couldn't make it cheaper and at the same time would only use an inferior technology. Why would anyone pulling a new network today use anything but fiber optics?

<comment><USER24> They're trying to use existing copper for most of it, but as predicted by all the experts it's not working (the Government promised 50Mbps to all and up to 100Mbps but many people only get ~35Mbps - of course, the parts done by the old Government are capable of gigabit), and the cost has blown out by more than $15bn anyway... So yeah, should have just stayed with the fibre.

</comment></comment><comment><USER25> Reminds me of Dave of EEVblog (in AU, obviously) discussing about how stupid the pricing, bureaucracy and outdated are on various, overlapping internet service providers, especially NBN.

[https://youtu.be/haImi4cWau0](https://youtu.be/haImi4cWau0) (2017)

[https://youtu.be/nJxQZr4dz3g](https://youtu.be/nJxQZr4dz3g) (2014)

</comment><comment><USER26> The BBC blamed, in part, "Australia's preference for underground wiring" as if it was a fashion statement.

In reality Britain simply lacks serious termites (and is in general underequipped in terms of dangerous animals, if you use Australia as a baseline).

I think this article buries the sins of the Liberal party (though the whole problem cannot be laid at their feet). I still have hope she'll be right in the end.

</comment><comment><USER27> I truly fear the day the NBN comes to my area. I am currently with Telstra and fortunately have HFC, I get pretty reliable speeds between 112 and 115mbps, with ping of 9ms. I am a developer and I rely on my currently decent connection to make money, video call clients in other countries, transfer large files (to and fro) if speeds were to suffer resulting in the inability to communicate properly, it would be crushing.

The fact that retailers are not purchasing enough bandwidth means you're buying a Ferrari with a broken engine that is being pulled by a single horse. My area is slated for NBN 2019, let's hope they do something about the bandwidth before then (apparently at an estimates hearing a solution was promised by Christmas this year). What a joke.

<comment><USER28> Really? What's your up rate? I am with Telstra HFC and get 2Mbps on the upload on home plan. NBN gives 40Mbps.

<comment><USER27> Telstra with added speed pack gives me 115mbps down and 2mbps up. I actually get a little over 2mb though. If I go onto the NBN and the bandwidth issues aren't sorted out, you can have the fastest speeds all you want but still experience ADSL speeds during peak congestion.

</comment></comment></comment><comment><USER29> > In 2013, Australia ranked 30th for average internet speeds. The NBN was > meant to improve that, but the most recent standings ranked Australia 50th.

Well, I'm pretty sure 50th place today is a lot better than 30th back then still. They might not have the best internet around, but it probably got better for the users in that timespan. It smells to me like typical misused statistics in search of click-earning controversy.

<comment><USER30> How is this misusage of statistics? Given that internet speeds are a moving target, it really only makes sense to compare with other nations. If Australia doubles their internet speed that's great and all, but if the rest of the world makes a 10x improvement, sure doesn't seem like much of an accomplishment.

</comment></comment><comment><USER31> Running FTTH to every home was always a pipe dream (excuse the pun). Some kind of technology mix was a necessary compromise _however_ the balance is wrong - percentage of FTTH is way too low. My home town is literally (and inexplicably!), divided down the middle - one half with FTTH, the other FTTN.

<comment><USER32> >Running FTTH to every home was always a pipe dream (excuse the pun).

Why do you say this?

<comment><USER31> Because of the astronomical cost involved in running fibre to _some_ premises. And not even talking about city vs bush - two city dwellings in close proximity can have vastly different cost of installing fibre.

I would qualify my original statement with 'as part of a NBN-like project'. I didn't intend to suggest that not every home should have fibre, or in deed will have fibre at some point in the future, just not at the tax payer's expense.

</comment></comment></comment><comment><USER33> I will be moving to Melbourne next year. What kind of internet speed can I expect, what areas should I be looking at for a good connection at home? I currently have 200/20 in Berlin, presumably will need to downgrade my expectations...

<comment><AUTHOR> Depends on the area but expect 10 Mbps or less. NBN has not really moved into capital cities yet. I am looking forward to them rolling out FTTN connections on copper laid 50 years ago and insulated using newspaper. Even if you get an area with NBN keep in mind they are not obligated to provide more than 12 Mbps during co existence and 25 Mbps after. With 5 dropouts a day considered acceptable service.

If you can find a unit with TPG FTTN go with that. It will be cheaper and uncongested.

</comment></comment><comment><USER34> It's interesting to compare Australia's grand experiment with Stockholm's. Australia's proposal, before retrenching, was 93% coverage at 100 mbps (later, one gigabit), with a government investment around $30 billion AUD. Completion was promised on an eight-year timeline.

Stockholm, in comparison, followed a very different model: [https://www.stokab.se/Documents/Stockholms%20Stokab%20-%20A%...](https://www.stokab.se/Documents/Stockholms%20Stokab%20-%20A%20Blueprint%20for%20Ubiquitous%20Fiber%20Connectivity%20FINAL%20VERSION.pdf). A city owned company was created. No public money was contributed; the network was financed by the utility taking loans. The government imposed no coverage requirement. Service was expanded based on demonstrated demand, and it took 17 years to cover 90% of the city. The utility started turning an overall profit in 2008, after 13 years of operation. The Stockholm system is expensive-- customers are charged a set-up fee to recoup construction costs, and although prices must be transparent, they are not rate-regulated. NBN Co., in contrast, is subject to rate regulation and does not recover construction costs as part of a setup fee.

Broadly speaking, there are three lessons to take from this:

1) Urban areas and rural areas should not be part of the same program. "Universal service" sounds nice in theory, but in practice the challenges of deploying in rural areas can torpedo the project. In Sweden, the government _did_ help deploy fiber in rural areas: [http://www.ftthcouncil.eu/documents/Opinions/2013/Rural_FTTH...](http://www.ftthcouncil.eu/documents/Opinions/2013/Rural_FTTH_Nordics_Final.pdf). Sweden required dark fiber to be deployed to within a few kilometers of rural areas, but rural residents had to pay for their own hookups. The government subsidized 50% of the cost of those hookups, up to about $600 USD per property.

2) Coverage requirements are a bad idea. When you force the operator to cover areas where the business case doesn't yet exist, you're forcing them to charge their existing customers more to subsidize new customers. That can make the product too expensive for existing customers.

3) It's okay for fiber to be expensive, at least at first. Charging more to early adopters to recoup large fixed costs is a basic business model in the tech industry. Telia in Sweden charges about $120/month for gigabit, and in many cases customers are liable for construction setup costs. That's okay--it allows the operator to quickly recoup construction costs and use ongoing revenues to expand the network. Sweden does not regulate rates to make fiber affordable for poor people, nor does it force fiber operators to subsidize service themselves.

</comment><comment><USER35> So here's my tale of what happened:

Party A was in power, and having infighting problems. Party B was headed by an absolute arse of a man, who used the tactic "Say No! to EVERYTHING". Didn't matter how benign or banal, the opposition party would say no. A flagship policy like the NBN was particularly focused on - and if you look at interviews with the opposition communications minister at the time (now our prime minister), it was clear that he was toeing the party line and didn't think changing the policy was a good idea.

Fast-forward to the next exection, and Absolute Arse gets into power. Turns out he can't govern at all (despite 20 years on the front benches) and he definitely isn't going to backtrack on something he's attacked so routinely. He's kicked out by his own party. At this stage, the NBN has been gutted, and its doom is sealed.

In steps the comms minister as the new PM, but he is now beset by factionalism. He knows what a turkey the NBN is, but he doesn't have the political power to do an about-face on his party's policies.

In short, we're fucked. By the time any other party can get in, the NBN will be 'mostly complete'. The most annoying thing is that the folks who really need good internet is business, not residences (think graphic design, medical imagery, etc), and the party that fucked the NBN is the 'party of business'. Enterprises can already dictate their own network backbones, but small-to- medium businesses cannot.

<comment><USER36> "Enterprises can already dictate their own network backbones, but small-to- medium businesses cannot."

Before that election, I can remember having an argument about the need for the NBN with a friend who works for a very large company. He argued that the internet in Australia was fine because their teleconferencing video in the boardroom worked well. I told him that not only would they have had a very specialised connection, that in the very near future, households would see multiple, concurrent, hi-res video streams, and that business demands would be high too.

We're a home of four. It's not uncommon for the kids to be streaming Netflix while I work (with a basketball stream running), and my wife uploads large photos and streams a TV show on her phone while she works. We're still a year away from getting NBN and our internet absolutely crawls.</comment></comment>


Massive increase in SMTP attacks - <AUTHOR>

In the past 24 hours I'm seeing a massive up tick more like a tidal wave of SMTP port attacks. The majority are from Comcast, TWC and Cox cable IP addresses.
My theory is that hackers have breached some kind of significant Windows vulnerability.
Does anyone else see this? At all? Any increase in attacks on your systems, any port any surface? <comment><USER2> Yes, seeing some phony SMTP requests from Comcast and Cox (easily blocked by Spamhaus), but not in a larger amount than the usual. By how much was it a _" massive up tick"_ on your systems?

<comment><AUTHOR> From none several days earlier to, it was happening every couple of minutes when I made the submission to HN.

edit: Oh and yes definitely I use several blocking sites in my Postfix configuration but I checked the reputation on those IPs at that time they were not in the registries yet. Plus these, I should have explained, were failed authentication attempts, not just plan spammers. You know using made up names like harry, sally but sometimes valid ones like info.

I'm not math wiz but I doubt they could be lucky enough to guess the password on account with tries at minute or more intervals, I also run fail2ban which usually stops them too.

Still it just sort of jarred me, the number increased so fast. I was worried there might be something else going on I could be missing.

<comment><USER3> I host SMTP for about 20 domains, and I think if your starting point is zero then you've just been lucky.

Even for non-popular domains I get relay attempts, and spam deliveries, every few seconds. Round. The. Clock.

It could just be that your server hadn't been detected, either via port-scans, or via DNS lookups, and now it has been.

</comment></comment></comment><comment><USER4> [http://chilli.nosignal.org/mailman/listinfo/mailop](http://chilli.nosignal.org/mailman/listinfo/mailop)</comment>


Dogs snub people who are mean to their owners: study - <AUTHOR> http://www.france24.com/en/20150612-dogs-snub-people-who-are-mean-owners-study <comment><USER2> this is why when Bonita and I first started our long-distance relationship, while she was excited that I offered her a plane ticket to visit me in California, her concerned friends convinced her that I should visit her first in Nova Scotia, in the dead of winter.

this so her dog could check me out.</comment>


KODAKCoin cryptocurrency - <AUTHOR> https://www.kodak.com/kodakone/default.htm <comment><USER2> This is really exciting to me.

I know a lot of folks will probably see this as an old company trying to catch a ride on the cryptocurrency hype train, but to me this has always been the dream of cryptocurrency; a new class of technology to help solve types of problems previously very difficult to address, and it's awesome to see a >125 year old company taking a shot at adopting it for a novel use case (licensing of and payment for photography).

I'd very much rather see more stories like this than more early stage ICOs from alt coins made by a handful of people with unknown/dubious intentions.

</comment><comment><USER3> A note to the crypto fans: _this_ is what will actually be useful about blockchain technology -- tying proof of transaction to something _that is actually itself of value_.

<comment><USER4> What do you mean? I didn't see any details on how this will work on Kodak's site.

<comment><USER5> "The KODAKOne image rights management platform will create an encrypted, digital ledger of rights ownership for photographers to register both new and archive work that they can then license within the platform."

<comment><USER6> Why do you need a block chain for this? The functionality can also be created without this.

<comment><USER7> With a blockchain, they could set things up such that the ownership data is not owned or controlled by any one company. I'm not sure what Kodak would get out of setting something like that up. But there doesn't seem to be another reason they'd be using a blockchain, and they probably aren't just being stupid.

<comment><USER6> This is just another attempt to get money from eager "investors" through an ICO (according to Kodak it will be SEC regulated, I would be very surprised, it would probably be the first)

</comment></comment></comment></comment></comment></comment><comment><USER8> How are they going to prevent nefarious people from doing a mad rush to upload others' work, and then collecting royalties for IP that isn't theirs?</comment>


Show HN: RTFA, a Chrome extension to hide HN comments until you read the article - <AUTHOR> https://chrome.google.com/webstore/detail/read-the-effing-article/angdimfeecdnooacbnnppmlbobcnpbln

<comment><USER2> I prefer the inverse. Reading the comments from an educated audience first, then diving into the article with a more firm understanding of the landscape.

<comment><USER3> I agree. The order I use: read the comments, read the article, and THEN post a comment if I have something to add.

<comment><USER4> Perhaps it should just prevent you from _writing_ a comment before reading the article.

<comment><USER5> I like to read the article first, get a feel a of what it's about and then read comments to see further analysis. Reading article first let's me think about it before I get influenced (negative or positive) by reading the comments.

I agree, commenting without reading the article and only reading the comments is a sure-shot way of making a mistake.

</comment></comment></comment></comment><comment><USER6> Hiding the Reply links might make sense if you can't stop yourself from commenting before reading, but hiding the comments themselves doesn't make sense. I find that HN (along with LWN) is one of the few sites with high-value comments, and I often want to read the first few to get context for the article.

<comment><USER7> Yes, I often read the comments, decide that the article is uninteresting on that basis, and leave it at that.

</comment></comment><comment><USER8> I prefer using the comments/discussion themselves as an indicator to whether I'll find the article a worthwhile read in the first place - how often do users actually comment/discuss an article without reading it beforehand? What's the point in that?

<comment><USER9> Yup. Too many times have I slogged through six pages of text only to visit the comments and find, "This article was complete bullshit, and here's a well- reasoned argument citing excellent sources explaining why you just wasted your time reading it".

</comment></comment><comment><USER10> Usually the title of an article is a poor indication of the article's content, structure and tone. By skimming the comments I can usually get some idea of these attributes and then decide if it's an article that interests me.

Also, the HN comments often load much faster than the page itself (full of crazy JavaScript, CSS, ads, useless graphics, etc).

</comment><comment><USER11> The people who need it the most won't use it. But you are onto something.

The next step: Get in touch with Google so that it's included into official Chrome in some way. It doesn't have to be installed by default - just a one time pop-up when a user first visits Hacker News. An offer to install it.

The next step +1: Encourage people to expand the extension to other sites. Write it in a modular way, write documentation.

</comment><comment><USER12> One thing though, if the site's unreachable, often in the comments there's a mirror or a summary. If you can't turn the a into a:visited, you'd never get the link to show up if the site gets overloaded by the HN effect.

</comment><comment><USER13> I don't have a preference about which one to read first, for ex. with articles like "I stopped using X and why you should stop too", I find more value in reading the comments first, because usually the top comment will be a counter- argument more worthy than the original article.

But with articles that teaches something new to me, for ex. "Scaling X to a million users", I prefer to read the article first.

</comment><comment><USER14> This is almost ironic, since the people who install this are the ones who are conscientious enough to need it the least.

</comment><comment><USER15> I very much prefer to first read the comments because it gives me quality data to determine whether the article is worth a read or not.

</comment><comment><USER16> I'm looking for a tool to sort and show comments according to time. Specially for Ask HN and Show HN.

</comment><comment><USER17> I would prefer an extension that stopped me reading the article until I had read the comments.</comment>


Political Moderates Are Lying - <AUTHOR> https://quillette.com/2018/07/02/political-moderates-are-lying/ <comment><USER2> No political party has ever been worthy of being anyone's tribe at least in the USA. But mankind naturally gravitates toward tribes and hierarchical communities. I want to be in a tribe. I consider family friends or extended family to be tribe like but it lacks any leaders. Tribes are not really all about ideologies. It is more like family relationships. Growing up my extended family was my clan or my tribe. We grew apart and we lack our original matriarchs and patriarchs. Politics assuming tribal roles because tribal life is in demand and people like to be part of something. Even if they are hoodwinked into thinking they are part of something that values them when it is clear to many people it is simply a lie.

</comment><comment><USER3> How far must the two parties diverge before enough space opens up for one or more new groups to reside in the middle? That might be a nice outcome to have, and the possibility is almost enough to make me hope the vicious cycle continues...</comment>


Man searches landfill for HDD containing $7.5M worth of bitcoins - <AUTHOR> http://www.bbc.co.uk/news/uk-wales-south-east-wales-25134289

<comment><USER2> In fact, more accurately, man _doesn 't_ search landfill:

        "Why aren't I out there with a shovel now?
         I think I'm just resigned to never being
         able to find it."

Same story, different source, significant discussion:

[https://news.ycombinator.com/item?id=6809065](https://news.ycombinator.com/item?id=6809065)</comment>


Opening the code of our X-Pack features - <AUTHOR> https://www.elastic.co/blog/doubling-down-on-open <comment><USER3> This is incredibly user-hostile, and if you read this expecting this is an open-source release, consider yourself clickbaited.

Want to contribute to the open-source version of Elasticsearch? Want to ensure you're running an open-source database without running a custom EULA past your legal department?

Well, make sure that after 6.3 is released, you don't accidentally download the default distribution, or browse the official Github repository, or clone from Github, as all of those will have components subject to an as-yet- unreleased, non-OSI-approved EULA (see the final paragraph in [https://www.elastic.co/products/x-pack/open](https://www.elastic.co/products/x-pack/open)), which may have clauses that trigger liability to Elastic if you accidentally flip a switch or look at the wrong thing. (I am not a lawyer, this is not legal advice, but I shouldn't have to be to know if I can use software freely, and I can only guess worst-case scenarios because the terms aren't released.)

If Elastic were truly committed to transparency, they would release the terms of their new EULA immediately, provide a way to access the head of the open- source portion of the repository without accepting the EULA, and make it clear what the pricing ramifications are if one opts into certain flags or reads certain files in a commercial capacity. (Pricing requires a custom quote, but a quick Google search suggests that the lowest tier in which REST and node-to- node communications are encrypted begins in the tens of thousands of dollars - not the kind of thing you want to take lightly.)

Now more than ever, it's important that projects like [https://github.com/floragunncom/search- guard](https://github.com/floragunncom/search-guard) , a free and open-source (Apache 2.0) alternative to some of these X-Pack components, are supported, and that the word is spread around them. (EDIT: Yes, they have commercial components, but those are in a separate repository, so you avoid any entanglements. Why isn't Elastic doing the same?)

And hopefully, if Elastic doesn't maintain a fully-OSS repo, someone will create and maintain a friendly fully-OSS fork that can be combined with third- party software to carry on the legacy of this otherwise excellent database in the right way.

<comment><USER4> One of the awesome things about our products is that people care deeply about them. There is one thing, in particular, I want to address.

'If Elastic were truly committed to transparency, they would release the terms of their new Eula immediately...'

We are working on it. This change is not one that was taken lightly and we have to ensure we have the right language for when we do release the license, etc. We will put it online as soon as we are able.

We are committed to transparency and, also, we are committed to our users.

Yes, we are asking for a fair bit of trust...and I hope we continue to prove ourselves worthy of that trust.

<disclaimer: I work at Elastic in Developer Relations>

<comment><USER5> Sorry if the following sounds like a personal rant: it is.

You're saying that people care deeply about your product. In reverse, I don't get a feeling you care much about the community around your products. Since you're working at developer relations, I'd like to point out that you're still funneling people to your IRC channels via [https://www.elastic.co/community](https://www.elastic.co/community), but once people arrive there, they barely get any help from elastic people. Despite multiple requests, the channels are still not logged or searchable, so questions get asked over and over again. I'm a long-time lurker there and especially questions about how to contribute to the projects end up in silence. People asking how to work on tickets (in the tickets) as part of a university course or as part of a Gsoc assignment: silence.

There's barely anyone offering guidance on how to contribute to the open source project beyond "you'll need to sign the CLA." The CLA is contributor- hostile. Anything people touch as part of their work cannot be contributed unless they get legal involved - a show stopper for many. The CLA, at least in some versions required full copyright assignment and indemnification - I, personally, can say that it stopped me from providing any kind of fixes or improvements.

I used to run the Berlin elasticsearch usergroup which started out before elastic, the company, even existed. At some point it used to be one of the largest ES UGs world-wide, still, any kind of support from elastic beyond personal support from some developers: nonexistant. Heads up and infos for feature announcements so that we could prep a talk fitting the announcement: Not there. Sending a speaker for an announcement? Impossible. At some point, elastic even tried to charge us for the privilege of running a UG. The best offer we received was an offer to pay for pizza. We did get a honorable mention at the first elasticon, IIRC.

> Yes, we are asking for a fair bit of trust...and I hope we continue to prove > ourselves worthy of that trust.

Good luck. I've heard these words before. Any kind of interaction with elastic, the company, that I had as a community member was borderline hostile. I wish things would improve, but I've pretty much lost faith.

<comment><USER6> On the other hand,

Their discourse community is very active. ElasticSearch people help out there a lot. They are people in the end, and they can't be everywhere. They are spread thin and they do their best. Open source doesn't mean super human. Lack of resources doesn't mean "hostile". If you think not being able to respond is hostile, then wait till you really see the hostile part of the world.

For everything ElasticSearch does, this is a very critical and negative outlook.

<comment><USER5> > They are people in the end, and they can't be everywhere.

I agree, but they mention IRC as an official contact and support channel. Either do that and then support it, or don't. But you can't say "you can get help on IRC" and then not be there, that just does not work. People do actually come to the channel and expect answers and they do treat volunteers there like they're paid support staff - an expectation that is understandable given that it's listed on the official page.

> Lack of resources doesn't mean "hostile".

No, but I didn't say so. I said the interactions were borderline hostile. I won't recount the episodes here, though, but generally they were about asking us to go out of our way to organize things and then drop the ball. If you ask me to do stuff for you for free, then you'd better follow through. Lack of resources is no longer an excuse then.

</comment></comment></comment></comment></comment><comment><USER7> Yeah so has anyone actually tried to get ElasticSearch up and running lately? I just tried and had a terrible time, despite the fact that I was using ElasticSearch + Kibana, and it was dockerized, and it was on Kubernetes (there's more complexity, yes, but all those tools make deployment simpler once you understand them, not harder -- writing a pod resource config to get a thing running means I don't have to run around my system changing settings, I just put all of it in one place). XPack was just another stumbling block while trying to get everything running.

The combination of lack of documentation, inconsistent/changed configuration (ENV vs YAML vs values that just don't exist anymore), breaking changes between versions that rendered Kibana completely useless, and the recent (?) removal of plugins that expose web APIs (so I couldn't use something like elastic-head. This is all in Kubernetes btw -- maybe it's just that I wasn't smart enough to get it done, but it's so easy to write functional (if not well-configured) configurations for other databases, I was at a loss for words when nothing I tried worked right.

I got so angry trying to set up ElasticSearch that making a F/OSS competitor is now #2 on my list of projects-to-do-next. I'm sure the thought is naive but I need to find out for myself that there's no easier way.

Imagine if the team behind Prometheus had focused on search instead of metrics? That's the kind of tool I want to use. A tool as focused, easy to start, clearly documented, and straightforward as prometheus.

<comment><USER8> Elasticsearch is a mess. It's so full of historical warts.

One major problem is that none of their documentation is actually reference documentation -- if you look for the formal schema (for things like mappings and the query DSL), the list of endpoints and their allowed parameters, the full list of settings etc., you won't find them listed anywhere. For example, does "keyword" mappings support the "enabled" property? What does the "index_options" setting actually do when combined with the "index" setting? Hard to tell any of this without trying them out. Turns out "dynamic_templates" mappings support _any_ combination of the above, and will never complain about invalid combinations, whereas property mappings do. The whole environment variable vs Java property mess that you mention also exists.

They do deserve credit for trying to clean it up. The last few releases have been pretty brutal in how they've been deprecating (and later removing) legacy features and tightening the semantics, the newest and most dramatic of which is the deprecation of multiple type mappings per index. And they've been pretty good at explaining what's going to happen. So the warts are getting fewer. On the flip side, you have to follow the release notes religiously if you want to keep up to speed, since each release now tends to remove a bunch of features or add strict validation where there previously was none, and it becomes harder to upgrade. (If you want an important bug fix that hasn't been backported, things could get expensive.)

It's interesting how the Elasticsearch team let their focus be derailed by this new industry obsession with analytics and logs. It's not something ES was originally built for, and it turned out to be good at it mostly by accident. It's not terrible at it, but Elasticsearch shines the most for its original purpose, as a content index with rich full-text search capabilities. (Areas where it works less well include scaling edge cases such as high-cardinality aggregation buckets and high numbers of unique field names.) I wish they'd rather worked on things like joins and fixing the need for the "nested" object type, which is a ridiculous hack, but since those things aren't needed for analytics/logs, they haven't happened.

(Pet peeve time: One problem that rarely gets mentioned is that Elasticsearch's "eventually consistent" model has two parts. There's the part where replicas may be out of sync with primaries, but there's also the problem that on each individual node, index operations don't become visible to queries right away, not until the next segment "refresh", which by default happens every second. There's no API to ask about the refresh state, so right not the only way for a read followed by a write to be consistent is to ask the write to wait for refresh (or force a refresh), which is the opposite of what you want; the wait should be on read, not write. Given that ES now has a sequence number associated with shards, I'm surprised they haven't tied those numbers together with refreshes so you can ask about which sequence number the index is currently "at".)

So I think Elasticsearch is definitely ripe for disruption. I don't know of anything else that is able to compete at the moment, at least not in a single package; Solr isn't really in the same league.

<comment><USER9> It's surprising when someone expects consistent ops from elastic when it's built on something that has none of it (lucene).

At least solr doesn't pretent to be something it isn't (database).

<comment><USER10> >At least solr doesn't pretent to be something it isn't (database).

I worked on implementing ElasticSearch at my company and one of the things they mention clearly is ElasticSearch should NEVER be used as source of truth (primary database).

<comment><USER11> I think I understand where the OP was coming from.

ElasticSearch + Kibana often gets positioned and used as an open source alternative to Splunk. In that context it is in all respects operating as a primary database since often the source logs are transient.

</comment></comment></comment></comment></comment><comment><USER12> Background: They introduced X-Pack by bundling it with the default distribution as a time-limited trial without explicitly stating that it is just a demo. People who did the update were bitten weeks later when it just stopped working because the demo license had expired. [Documentation of this was ridiculously bad and I only learned through this post that there apparently was some way to get a free license (maintaining an instance for a 501(c)(3), I just assume we'd have qualified).]

This looks like an attempt at fixing their karma balance, but until I've reviewed the EULA I am pessimistic about the value of "allowing for some derivative works". And I don't really get the "allowing for some [..] contribution". <zyn>Is a patch that improves performance by 10% welcome but I'll have to pay them to make them accept a patch that improves performance by 100%?</zyn>

<comment><USER13> _Disclosure: I work at Elastic, primarily on the X-Pack features in Elasticsearch_

> They introduced X-Pack by bundling it with the default distribution as a > time-limited trial without explicitly stating that it is just a demo.

I assume you're referring to the Elastic docker images, as I believe that's the only place where we've ever bundled X-Pack without any explicit opt-in (Our windows installer also includes X-Pack, but it's clearly marked as an optional & commercial component).

We totally underestimated the confusion and difficulties that would cause, and it was fixed with the 6.0 release.

The difficult was balancing the different needs of different users, with the constraints of how docker containers are typically managed. X-Pack requires a file-system level install, which is not something that docker users expect or want - an image should be built once and then be essentially immutable. No one likes to have to enter the container in order to install a new plugin.

Since it was possible to disable X-Pack functionality, or install a free license for a subset of the features, it seemed like shipping the container with X-Pack pre-installed and letting users dial it back as needed, was the better option compare with shipping without X-Pack and forcing customers to reconfigure their container so that they could get the features that they needed.

We didn't expect that there would be so many users for whom Docker was the primary/initial point of contact with our stack. We believed that we would be mostly working with users who already understood what X-Pack was and what they were getting. When it became clear that that wasn't true, we had to come up with a solution, which is what we shipped in the 6.0 release. We didn't want to change the behaviour in a minor release and cause more confusion for users who were relying on X-Pack being installed, so it wasn't simply a case of changing the way the images.

The X-Pack licensing code was built on the premise that X-Pack was a plugin the was explicitly installed so that those who installed it would know what was going on. And when it was written, that was true. One of the consequences of that assumption was that it would automatically generate a trial license on start-up, and then you could install your own purchased license after the fact. In order to offer Docker containers that worked the way users would expect, we had to make changes to that so that X-Pack could be installed, but default to only enabling the free features, so since 6.0 we are now able to provide 3 different docker "flavours" \- pure open source, basic (free license), platinum (trial license for paid features).

We do want people to use X-Pack. We believe that the free (basic license) features offer something useful that users should know about, and the success of the company relies on users knowing that our commercial features exist, and being able to evaluate them and decide if that's something they want to purchase. But the docker situation was never intended to be a "bait-and- switch", it was just a problem that caught us by surprise and took some time to rectify.

 _For historical accuracy, the inclusion of X-Pack in our docker images was not the introduction of X-Pack, it had been available for quite a long time before that and the underlying commercial IP was several years old before we started publishing Docker images._

<comment><USER14> How would you do a time limited trial for an immutable widely distributed Docker image?

There either has to be some fingerprinting and a phone home, or a licence you "manually" set to the env.

Since I am unaware of any sufficiently reliable host/cluster fingerprinting opportunity in say Kube, I assume it needs a licence to be configured in the pod yaml file.

What I don't understand is how this confusion can arise given the explicit licence installation.

Or where did I go wrong?

<comment><USER13> It depends on what "immutable" means to you.

Elasticsearch is stateful, it's essentially useless if you don't have some form of persistent storage for it, because its purpose is to act as a datastore. So we make no attempts to treat the Elasticsearch containers as if they're stateless in the "no permanent storage" sense, but you can point your Elasticsearch data directory to a mounted volume and keep that state away from the core image.

But having an immutable image is a bigger deal. The expectation that users have of docker images is that you don't need to edit anything on the image files themselves, but that the configuration is provided from an external source, typically through environment variables.

Installing _X-Pack_ is an image change. It stores new files in the plugins, bin, and config directories of Elasticsearch, and will typically require changes to your main cofiguration file. That's something docker users generally don't want.

Installing a _license_ is a state change. The license is loaded with an HTTP API, and we store that data in the data directory alongside the stored documents. So for most users it's not a violation of their "immutable" expectations.

The other expectation is that you can simply start the container, and run it to completion and then terminate it - you don't want a process that expects to be started, and then reconfigured, and then restarted, etc. That affects other aspects of how you can configure Elasticsearch, but doesn't impact the licensing issue quite so much.

<comment><USER14> Thank you for the explanation. Data directory shared across all pods is the answer then.

Given that the process was automagic, I can see the potential for confusion.

Seems you are handling the fallout nicely though.

</comment></comment></comment></comment></comment><comment><USER15> The important part is near the bottom, free components of X-Pack will no longer require the cumbersome registration process. Also of some concern is that extra care will be required to find a version of the software that is Apache 2.0 licensed as the proprietary licensed X-Pack components will be included by default in the code base instead of as a separate install.

\---

"Also, X-Pack features will now be bundled into the default distribution. All free features are included and enabled by default and will never 'expire', and commercial features are opt-in via a trial license. The license for free features never expires,you no longer need to register to use these capabilities. In addition to this, an Apache 2.0-only distribution will be created for download."

</comment><comment><USER16> Here we have a company that is simply opening up visibility of the code of their commercial offering, which will enhance customer relationships vs. keeping the code black box —— and it's observed as user-hostile?

Open core isn't evil. Companies need to make money, and shouldn't have to choose between fully-open-source vs. closed-source.

Fully-open-source companies have a really hard time of building a good company, because despite the great work they put out into the world, other enterprising cloud companies can build closed-source cloud services around it. For example, I've paid Compose.io and other companies real money around MongoDB, and never a cent to MongoDB the company. Same goes w/ Docker, who has struggled to build a great business because they open-sourced so much of their value.

So yeah, Elastic is trying to make it more likely that you'll choose to buy a license, because the health of their business depends on getting paid customers.

</comment><comment><USER17> Eleven uses of the word free, all used in the gratis, not libre, sense of the word.

'Open' is in the same category for me now as 'cloud' \-- too nebulous, too little consensus on what it actually means, often used in a 'don't worry about the details' hand-wavy way.

</comment><comment><USER18> This thing of having the source "out there" (but not OSS) worries me and is very confusing for users because it creates dangerous grey areas.

If you think like me, check out the __actually __FOSS(GPLv3) alternative to X-Pack security I created:

[https://github.com/sscarduzio/elasticsearch-readonlyrest- plu...](https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin)

</comment><comment><USER19> Their licence is still proprietary, though. At the least people with a technical curiosity in X-Pack will be able to install it. Technically it's illegal, but Elastic probably doesn't care until the moment they start gaining momentum.

<comment><USER20> I wouldn't put it past the Elastic team to include some sort of phone-home mechanism here.

Nor would I put it past them to consider legal demand letters as part of their sales funnel.

<comment><USER4> In 6.3, telemetry ('phone home') is an opt-in only feature for both the free and paid components of X-Pack. We won't be tracking any identification, so even if users opt-in to telemetry, there is no way to turn that into a sales strategy.

Suing users is a pretty terrible business practice and not, at all, a funnel creation opportunity.

Re. License, there are a few FAQs on [https://www.elastic.co/products/x-pack/open#faq](https://www.elastic.co/products/x-pack/open#faq) that may help clear up any misunderstandings. Free features are free and governed by the Elastic EULA but are enabled by default. Paid features are available through an opt-in trial and, if they help solve a problem, can be purchased.

<disclaimer: I work at Elastic in Developer Relations>

<comment><USER21> and this trial can't be disabled (to use them for an unlimited amount of time) by someone by modifying your source because your license prohibits that? Am I correct in that assumption?

</comment></comment></comment></comment><comment><USER22> I'm a cynical es admin, but I'm pretty sure this is just a polite way of saying "you're still gonna need support" more than it is "it's good to go, run it yourself!"

Between ES and RMQ, I find myself running software written before the cloud was big, but trying to be the drivers of the cloud -- those two pieces consume more resources, both physical and operational then the rest of our stack three times replicated, combined.

</comment><comment><USER23> Dear Elastic people reading this thread: We hope you wont do evil with your licence.

If you don't do evil (e.g. write anything that could mess other people IP when working with Elasticsearch, prevent compatibility with other products, scare people from even looking at these codes or at the whole repository ) it will be.. a setep in the right direction.

If you do evil it will cost you dearly in the end, with FUD about companies should be enormously weary of having anyone even look at those repositories.

Hope we'll continue to cheer and celebrate your achievements.

</comment><comment><USER24> Saw this in my inbox yesterday and was excited... until I read it. Nothing changed. Ultimately Elastic is still looking for a revenue model _beyond consulting_ that makes sense.

Our company needed to build a bunch of this ourselves, simply because of the bundling cost. At the end of the day many, many organizations would love the ability to PURCHASE A SINGLE PLUGIN not all of x-pack. Ignoring the cost - <USER2> reality is that x-pack is useful in large enterprise, and a reasonably tiered model to develop against would help startups get going, and tie us in for the long-term. Instead, we built our own.

The sheer number of times I've had direct conversations with people at Elastic (from CEO down) about this makes me cringe. We rely on Elastic. We currently have our own ecosystem around it. We're also going to (over time), probably have to open source bits and pieces, so that others don't have the same pain.

</comment><comment><USER25> Being honest X-PACK is just a pain for the small developer, it should be opt- in by default and only installed by the user not the other way around, I would probably be using some parts of it like that.

I used to love Elastic even have a couple of small pull requests merged. I've felt bad for using such an awesome software for free and not giving back. But X-PACK changed that, I am now a bit afraid of putting my trust in the software and the first thing I do is uninstall X-PACK when I get my hands in an elastic cluster or github repo that carries it.</comment>


Makers unite - the revolution will be home-made - <AUTHOR> http://www.bbc.co.uk/news/technology-19347120

<comment><USER2> Here's another one: The Storm Umbrella (can withstand 65 miles p/hour winds). <http://www.stormparaplu.nl/> Created by a student in Holland that got sick and tired of today's umbrella's.

</comment><comment><USER2> Like the Indian entrepreneur who took on Proctor & Gamble and build his own DIY machine for making low-cost sanitary napkins. [http://www.mamamia.com.au/health-wellbeing/the-man-who- wore-...](http://www.mamamia.com.au/health-wellbeing/the-man-who-wore-a- sanitary-pad-to-help-his-wife/)

<comment><USER3> I love that on so many levels! Especially as they're microfinancing the machines to provide local employment. As for the revolution, it's like they say - the future belongs to the creators, the innovators, and the entrepreneurs.

<comment><USER2> Startling to know how it is possible to invent, create and make a nice profit selling sanitary napkins for _a fraction_ of the price of the existing ones. Makes you think about what else we are paying way to much for.

</comment></comment></comment><comment><USER4> I think the article misses one important point. Most of the components these makers use are still being done in factories. And, almost surely, if you want to mass produce those makers' stuff you'd still need to approach a factory.

Unless what they mean is that there won't be a need for mass produced items anymore as soon as a lot of makers around the globe can freely reproduce other makers' designs.

<comment><USER5> The history of the Printrbot kickstarter project[1] and business[2] provides an interesting anecdote. The original idea was to make available affordable 3d printers, compatible with repraps (that is, mostly reproducible by other 3d printers, except for electronics, motors, power supplies and a handful of standard metal parts that can be bought cheaply anywhere) -- but as the popularity exploded, they realized it would be easier and cheaper to provide a version that was mostly laser cut -- they had exploded past the level where large scale manufacturing became cheaper.

It is also interesting to note that the laser cut version is capable of producing the parts needed for the other, self-reproducible design.

There are two parts to this revolution so far: Designs prototyped on a reprap can be printed/cut in any material, at any scale -- from wood to aluminium.

Designs that work in abs plastic: cups, glasses, lamps, sockets, casings for certain electronics, dolls, toys, action figures, clips, replacement parts for various items around the house -- they can all be printed at home, right now.

See for example the updates for the Hex Bright[3] kickstarter project -- the first prototype parts are 3d printed in plastic, the final design is in aluminium.

This is indeed a revolution, and it will change industry. Maybe not _all_ of industry, but a large part of it.

Think about it: if you can prototype it, you can order 500.000 copies of it. That by itself is significant.

Then consider the fact that the price of milling and printing equipment is moving into the range where individuals can bootstrap themselves, also for producing industrial strength parts -- and significantly building the parts needed for production equipment -- and we are closer to a democratization of the means of production than we have ever been.

[1] [http://www.kickstarter.com/projects/printrbot/printrbot- your...](http://www.kickstarter.com/projects/printrbot/printrbot-your- first-3d-printer)

[2] <http://printrbot.com/>

[3] [http://www.kickstarter.com/projects/527051507/hexbright- an-o...](http://www.kickstarter.com/projects/527051507/hexbright-an-open- source-light)

[edit: formatting]

</comment></comment><comment><USER6> A video of Chris Anderson - Makers: The New Industrial Revolution talk from a few weeks back

[http://www.watershed.co.uk/dshed/chris-anderson-makers- new-i...](http://www.watershed.co.uk/dshed/chris-anderson-makers-new- industrial-revolution)</comment>


Death of the sampling theorem? - <AUTHOR> https://markusmeister.com/2018/03/20/death-of-the-sampling-theorem/amp/ <comment><USER2> I don't recall ever reading a rejecting peer reviewer for one journal calling out the same work accepted in another ("peer reviewed") journal. Before the web, I guess there wasn't much for a reviewer to do other than complain to their peers. Now the low quality journal is called out in public. And a rejecting peer reviewer calling out the author for ignoring direct contradictory evidence against their work. The threat of breaking peer review anonymity is now a force to be reckoned with.

<comment><USER3> I do not see how this article poses a threat of breaking peer review anonymity. The only anonymity broken here is the author's own, only in this particular case, and after the fact. What is the purpose of reviewer anonymity? It protects the reviewer from being pressured into accepting a paper (and to protect science from that happening), which is not a risk here, and it prevents reviews from turning into long-running disputes about the validity of a rejection, but in this case, the issue is one of the authors willfully ignoring objective, valid and significant objections to their work.

</comment></comment><comment><USER4> Non-AMP version, which works with JS disabled:

[https://markusmeister.com/2018/03/20/death-of-the- sampling-t...](https://markusmeister.com/2018/03/20/death-of-the-sampling- theorem/)

And here's JS-free rendering of the notebook:

[https://nbviewer.jupyter.org/github/markusmeister/Aliasing/b...](https://nbviewer.jupyter.org/github/markusmeister/Aliasing/blob/master/Tsai_2017_Critique.ipynb)

<comment><USER5> Thx. You are not the hero we need, but the hero we deserve.

</comment></comment><comment><USER6> I find it funny how Nyquist theorem is often seen as a highly theoretical rule with complex technical background. But if you think about it, it's super straight-forward. You just need to sample each up- and down-deflection of a wave at least once.

Trying to "beat" this rule is like trying to beat the Pythagorean theorem. If you want to derive higher frequencies, you'll quite obviously have to make additional assumptions about the measured wave.

<comment><USER7> It's not straightfoward. Your intuition only works for periodic signals. Nyquist applies to all L2 functions.

<comment><USER8> Non-periodic signals can be thought of as a Fourier series of periodic sine waves though.

</comment></comment></comment><comment><USER9> It's theoretically plausible that the noise has some structure, and with some bayesian priors, one can classify some of the variation as "probably noise" vs "probably signal". Of course it works best if you know what the signal is before you start, but if that's the case why are you trying to experimentally construct it? :-)

This is a common subtlely problem with statistical denoising (aka pattern recognition, aka lossy compression). Commonly the source data and noise models are synthetic formulas (simple sine wave or aussian), and knowledge of the data-generation procedure can infect the supposed algorithm. (a subtle form of training your model on your test dataset)

<comment><USER10> > but if that's the case why are you trying to experimentally construct it?

People are able to interpret speech under very high levels of noise due to having strong priors / being able to guess well / having shared state with the speaker.

Of course you should take advantage of whatever priors you have about the signal; we know rather trivially from information science that information gain / transmission rate is _maximized_ when the listener knows as much as possible about the source. This is also key for compressed sensing.

You see analogues of this pattern everywhere in the world: e.g. people learn most quickly when they have _some_ context about the problem space already.

The real question is why the sender would produce a signal with any level of predictability in it, rather than a signal that purely maximizes the effective bit rate of transmission (e.g. minimize redundancy modulo noise). But it's easy to see that physical constraints (e.g. how the human voice works) are at least partly responsible for the enforcement of regularity, at least for many signals that we're interested in in practice.

<comment><USER11> _> People are able to interpret speech under very high levels of noise due to having strong priors / being able to guess well / having shared state with the speaker._

In the information theoretic sense, this just means there is enough redundancy in the signal that you can error correct the noise. If you're in a noisy room talking to a coworker, your shared context and priors are really redundant bits of information. If she says something about "nachoes" but you couldn't tell if she said "nachoes", "tacos" or "broncos", the nachoes next to you add redundancy to the signal.

So in that context, what you're saying is that we should take advantage of any redundancy we can find in an incoming signal.

<comment><USER12> The problem is there is no real way to estimate redundancy of a real signal of specific form, or more importantly discerning between likely alias vs likely true signal given a signal with unknown statistics. The statistical approach might work for simple clean signals but makes critical mistakes in case of complex ones. Specifically you get to estimate true phase and true magnitude in a given subband...

<comment><USER11> Yes, agreed. My point was to basically say that priors (as a form of redundancy) do not undercut the sampling theorem.

</comment></comment></comment></comment></comment><comment><USER13> While it might seem arrogant of the author to take this stance, the Sampling Theorem generalizes extremely well and is considered an extremely stable result. My favorite generalized Sampling Theorem is [https://arxiv.org/abs/1405.0324](https://arxiv.org/abs/1405.0324)

<comment><USER9> It's hardly arrogant to vote in favor of a 100-year-old mathematical proof over a new supposed refutation.

<comment><USER14> It is an argument from authority. That is asking a lot from the reader, i.e. to suspend believe or go research, the later of which is not too much to ask if direct at an acceptable audience.

"Arrogance" literally is akin to "to ask", e.g. interrogate ... unless the "a-" is akin to "anti" instead of "ad-" or whatever, so that arrogance would be like walzing in and taking without asking or making an assumption without due inquiry.

</comment></comment></comment><comment><USER15> Usually claims like this fall into two buckets: (1) crazy, (2) existing idea spun as something new. (Like airless tires, which have been around for a hundred years.)

This is quite clearly (2).

> _Effectively they can replace the anti-aliasing hardware with software that > operates on the digital side after the sampling step_

Sampling with filtering in the digital domain after sampling is absolutely old hat, found in everyday tech.

Let's look at audio. We can sample audio at 48 kHz, and use a very complicated analog "brick wall" filter at 20 kHz.

There is another way: use a less aggressive, simple filter, which lets through plenty of ultrasonic material past 20 kHz. Sample at a much higher frequency, say 384 kHz or whatever. Then reduce the resolution of the resulting data _digitally_ to 48 kHz. There we go: digital side after sampling step.

This is cheaper and better than building an analog filter which requires multiple stages/poles that have to be carefully tuned, requiring precision components.

<comment><USER16> First sentence from the original article: "A team from Columbia University led by Ken Shepard and Rafa Yuste claims to beat the 100 year old Sampling Theorem"

> (2) existing idea spun as something new

The existing idea is oversampling and is often spun as beating the sampling theorem. But this is unlike the airless tires example, because, while oversampling works, it isn't beating the sampling theorem but rather based on it.

You _can_ filter after sampling, but that filter is _not_ a replacement for the analog ant-aliasing filter. To avoid aliasing you have to somehow limit the bandwidth of your input.

<comment><USER15> Right; it's not a replacement. But you can then pretend (due to naivete or dishonesty) that the analog one doesn't exist and claim you've beaten the sampling theory. Why: because the input stage before the sampler naturally has a limited frequency response and you happen sample well above that. Thus a filter is _de facto_ there but wasn't formally designed in.

</comment></comment></comment><comment><USER17> Just a consideration (I am mathematician, but have nothing to do with signal processing or its mathematics; so don't overestimate my knowledge in this area):

Let us consider a periodic signal (though, I believe, this assumption can be weakened). If we have to assume that all frequencies from the inverse of the period length up to the Nyquist frequency might occur in the signal, the sampling theorem gives us the information that we need.

Now some "practical" consideration: Assume that the signal that we want to sample is very "well-behaved", e.g. we know that lots of its low frequencies are 0 or near 0 (or in general have some other equation that tells us "how the signal has to look like"). So if we reconstruct the frequencies of the signal, but in these Fourier coeficients some other value than 0 appears, we know that it has to come from "noise" that originates in some higher frequency above the Nyquist frequency.

My mathematical intuition tells me that it is plausible that such a trick might be used to reconstruct signal exactly even if we sample with less than than 2 times the highest occuring frequency. Why? Because we know more about the signal than what sampling theorem assumes. So a "wrong" reconstruction for which the sampling theorem can show its existence if there occurs a frequency higher than the Nyquist frequency is not important for us, since we can plausibly show that this signal cannot have 0s in the "excluded low frequencies" Fourier coefficients.

This would explain to me why such an impossible looking algorithm works so well in practice.

It it very plausible to me that such a trick is already used somewhere in engineering.

<comment><USER18> This is called compressed sensing.

<comment><USER17> Thanks.

<comment><USER19> Compressed Sensing originated in math, you might be interested in the original paper: [https://statweb.stanford.edu/~donoho/Reports/2004/Compressed...](https://statweb.stanford.edu/~donoho/Reports/2004/CompressedSensing091604.pdf)

</comment></comment></comment></comment><comment><USER20> With the rise of "big data", and "machine learning", we should painfully expect shit-science to become the norm (if it's not already). If you ever read a research paper somewhere in the bio-something space, you'll share the sentiment ("double-blinded this... double-blinded that... our unpublished data analysis pipeline... _96% accuracy_ ")

<comment><USER21> I particularly enjoy the re-discovery of numerical integration in 1994 by a medical researcher: [https://fliptomato.wordpress.com/2007/03/19/medical- research...](https://fliptomato.wordpress.com/2007/03/19/medical-researcher- discovers-integration-gets-75-citations/)

</comment></comment><comment><USER22> This article reminds me that I am on the low end of intelligence for the average Hacker News reader.

<comment><USER23> I'd take with context. Everything in this article should be understood by a 3rd year electrical/computer engineer as standard knowledge. Because I'm a computer engineering graduate, I can understand it. However, give me an article with '3rd year standard knowledge' from any other engineer discipline and I'll have no idea what's happening

<comment><USER24> One of the lessons college taught me is that I can learn anything, but I can't learn everything.

<comment><USER25> One of the lessons many engineers who pass the PE exam learn is, "not with that attitude you can't".

</comment></comment></comment></comment><comment><USER7> This isn't about the sampling theorem, it's a misapplication of compressed sensing.

</comment><comment><USER26> Similar to the conservation of energy in physics you can ask yourself "is there enough space for the information?". If somebody tries pulls two signals out of one signal, they better don't have too much entropy each. For a given bandwidth (number of independent samples), a given signal energy, and a given noise energy, there is a hard limit to the amount of information you can pull out.

If your signals are not "full entropy" or "white noise", you can do all sorts of funny tricks to increase the SNR to some extent. These tricks sure are nice but they veil the true issue at hand which is about information and capacity.

</comment><comment><USER27> I love this post for two reasons. First, it's a great reminder to question assumptions about high abstraction mathematical models that tries to say something about reality. And to not take what a community of even scientists say as stable truth. Ask a room full of signal processing engineers if it's possible to do better than the sampling theorem, and they'll confidently – with bravado – tell you that such an idea is ridiculous. Second, the end is a great example of a "proof by incentive" argument, which is one of my favourite ways to produce trust in a theorem.

<comment><USER28> I read this comment before reading the article and got excited to read an article showing how the "establishment" was wrong. Much to my surprise, the article was saying the exact opposite! I'm still not really sure how to read this comment in a way that makes sense.

<comment><USER11> I think that's because the comment is ambiguously distinguishing between academics and engineers _or_ lumping them together.

<comment><USER29> The comment is digitized brain noise in ASCII form. It just shows that if there's too much unfiltered noise before keyboard sampling, the signal can't be reconstructed.

</comment></comment></comment></comment><comment><USER30> so, hold on, he's wanting to charge people $10 (to get a file) to attempt to solve the problem, then give a prize of $1000 to anyone who can do it?

Hmmm.

Interesting.

<comment><USER31> He does provide a Jupyter notebook to try out your preliminary ideas.

From what I've seen people in the DSP community are very anal about the inviolability of the sampling theorem. So charging $10 may be his way to punish novice "fools" who think they know better.

I'm not sure I disagree.

<comment><USER32> If you understand the sampling theorem, its meaning is self-evident, you just count the degrees of freedom. Ask a mathematician how many samples do they need to recover the coefficients of a polynomial of degree 10. They will say: 11 samples are necessary and sufficient, if you have less than 11 samples then you cannot recover any coefficient. If you have 11 or more samples you can recover all the coefficients.

The sampling theorem is exactly the same thing but for trigonometric polynomials.

<comment><USER25> I don't understand it from this angle, but to me, the general description of what they employ sounds very analogous to how noise cancelling headphones work for pilots. The noise generated by the engines, which normally would drown out any signal of humans conversing through air, is of known frequencies, and a synthesized version of that noise can be played back 180 degrees out of phase along with the original audio, cancelling the noise, revealing the signal. No one is implying that the headphones do this without sampling the engine noise, they might just not have to rely on the actual measurements in question to glean the necessary information to synthesize a signal capable of such cancellation. It reads to me like this broadband thermal noise is predictable, which is why they are able to synthesize it in the first place.

<comment><USER32> The statement of sampling theorem has nothing to do with noise. It only states that there is conservation of information.

<comment><USER25> You misunderstood. I was relaying that I, too, have a decent grasp of sampling theory, and that if I understand the article's main premise, then saying they have disproven Nyquist's Theorem with this example is akin to saying that being able to draw both 30 degree and 90 degree angles with a compass disproves the idea that angle trisection is generally impossible.

They have simply shifted in time the sampling of the noise. Even if they argue they synthesize the noise in realtime, they had to have sampled it at some point to know what to synthesize in the first place. It was at this point that they were required to sample at at least 2MHz to accurately quantify their noise at 1MHz.

The mechanism used in pilot's noise cancelling headphones is much the same. they don't sample the engine noise as its being made, they synthesize it from known frequencies.

</comment></comment></comment></comment></comment></comment><comment> a-dub It seems a little tongue in cheek to me. Sampling with no low-pass antialiasing filter in general seems like a different problem from sampling with no low-pass antialiasing filter where you _know_ that they only spectral content above the nyquist cutoff is gaussian noise.

That said, his takedown of their approach makes sense to me. It's hard to really judge it though as my signal processing is getting _really_ rusty these days. I guess that's why we have professors. :)

</comment><comment><USER33> But ... they only want signals up to 5Khz, which is bugger all. So you could oversample (even a stock audio chip will do 96Khz) then apply a low pass filter with an FIR. Why wouldn't this work?

</comment><comment><USER34> Tldr: authors of paper claim their digital signal processing method is superior to those methods imposed by sampling theory; in practice, their method works only on examples where there is high level of a priori knowledge of the signal, i.e., not in any practical environment.

<comment><USER35> Close, but not quite: compressed sensing really works in a lot of domains. You "just" have to know of a basis set in which your signal is sparse. This basis set can be (and often is) overcomplete; i.e. you don't need a minimal orthogonal set of basis functions. So if you know that "most pixels of an MRI are black" or "music only has a few non-zero Fourier frequencies", you can apply compressed sensing techniques to recover/reconstruct the underlying signal. This is a fairly mild and merely structural form of a priori side information, as opposed to having to know detailed precise prior distributions or etc.

See [https://en.wikipedia.org/wiki/Sparse_dictionary_learning](https://en.wikipedia.org/wiki/Sparse_dictionary_learning) and [https://arxiv.org/abs/1012.0621](https://arxiv.org/abs/1012.0621) for the gory details.

<comment><USER36> Most of the research in compressed sensing type applications now is focused on more sophisticated prior distributions than sparse/Laplacian. These more general bayesian approaches provide significant performance improvements over LASSO etc.

<comment><USER35> Sweet, great to hear. Any particular papers you'd recommend checking out?

</comment></comment></comment></comment><comment><USER37> there is so much useless discussion here. what the blogger takes issue with (and refutes convincingly) is that ALIASED THERMAL NOISE can be RECONSTRUCTED. it clearly cannot (if you think you can, this is your chance to make $1000). it doesn't matter what rate you sample at, nor is there any need to invoke compressed sensing - what the authors of the original papers do has nothing to do with it.

<comment><USER38> couldn't have said it better myself (I'm the blogger)

</comment></comment><comment><USER39> I suppose you could say the antialias filter doesn't reduce the noise, but rather coherently averages the signal to increase it's amplitude. If the desired signal is noise-like itself, then I don't see how the scheme would work.

<comment><USER40> A simple (low pass) anti-aliasing filter has a linear drop-off, meaning the greater the difference between the frequency of the signal and the frequency of the noise, the better the signal to noise ratio. On the other hand, sampling random gaussian distributed noise will be random gaussian distributed noise no matter what frequency you sample it at.

[https://en.wikipedia.org/wiki/Low- pass_filter#/media/File:Bu...](https://en.wikipedia.org/wiki/Low- pass_filter#/media/File:Butterworth_response.svg)

<comment><USER12> Remember that the lowpass will change the phase or time response of the original signal. If there is anything in the frequencies and phases affected by the lowpass it will get distorted.

The trick works only because the signal is spike like. I reckon it will miss some of the spikes in real life on real signals or produce spurious spikes. The statistics are not accurate at any given point in time just in the limit.

<comment><USER41> I was going to point this out (phase response distortion) but you beat me to it.

During my university period, I had to digitally sample an ECG (electrocardiogram) signal. Since the shape of the signal is important in diagnosis and must be preserved, I used a Bessel filter [1].

[1] [https://en.wikipedia.org/wiki/Bessel_filter](https://en.wikipedia.org/wiki/Bessel_filter)

</comment></comment></comment></comment><comment><USER42> Death of the clickbait?

</comment><comment> Ono-Sendai There's no problem with doing anti-aliasing digitally later based on samples you have collected. You don't need some kind of physical analog device to do it.

<comment><USER43> Only if you oversample to cover the entire noise frequency range.

<comment><USER44> Indeed, and digital audio systems typically combine some sort of oversampling with digital filtering. They still need an anti-aliasing filter, but the filter works at a higher cutoff frequency and can be much simpler.</comment></comment></comment>