Coronavirus could force ISPs to abandon data caps forever - <AUTHOR> https://techcrunch.com/2020/03/12/coronavirus-could-force-isps-to-abandon-data-caps-forever/ <comment><USER2> I think that data caps aren't a good solution simply because they don't have a good connection with network capacity. Many wireless carriers have started doing "deprioritization" after a certain amount of usage which is a lot more consumer friendly. If the network has capacity, heavy users see no difference in their connection. If the network is capacity constrained, it favors lighter users so that they get their money's worth.

My school did this when I was an undergrad. The top 100 users on campus often used more than half the bandwidth. That was fine and the school was more than happy for them to use the internet heavily - the connection had already been paid for, might as well have it used. However, the school did put them at a lower priority when there was congestion. Most of the heavy users didn't even notice.

I'm really hopeful that wireless technologies will improve competition over the next decade. If three wireless companies can provide home broadband competition in the future, we won't be beholden to the single provider that most of us deal with now. AT&T, Verizon, and T-Mobile all bought a bunch of millimeter wave spectrum in the latest auction (in addition to their current holdings). T-Mobile is gearing up for a big home broadband push. They're hoping to have 9.5M home broadband customers in the next three years which would probably make them the third largest after Comcast and Charter who have around 26-27M I believe (and around double Fios' customer base). Having even a little competition would mean that they'd have to compete for my business rather than being able to hike prices every year.

<comment><USER3> A lot of UK broadband is advertised as "unlimited" but subject to a fair usage policy. I guess they must use a similar system to the one you described. It's pretty decent, and I'm never really thinking about my usage, which is what I like (I don't want to feel like I'm going to run out of bandwidth).

Interestingly, my ISP gave me a breakdown of my data usage, and I was under a meager 30gb/mo ever since my contact started. That surprised me as I watch a fair bit of YouTube etc, however, I guess it just doesn't add up to all that much.

<comment><USER4> > Interestingly, my ISP gave me a breakdown of my data usage, and I was under > a meager 30gb/mo ever since my contact started. That surprised me as I watch > a fair bit of YouTube etc, however, I guess it just doesn't add up to all > that much.

My hypothesis (could be wrong, very anecdotal, no evidence):

1\. YouTube does some pretty heavy compression to begin with (as several popular YouTubers have complained over the years) 2\. YouTube in its default settings switches to the lower resolution when it detects you are not actively watching the video

I think this one two punch means that you sip on data even when you think you use YouTube a lot.

<comment> frosted-flakes > when it detects you are not actively watching the video

How? By its very nature, watching video is passive. Unless they watch you through the camera.

It actually infuriates me when YouTube auto-lowers the resolution (usually because I have slow Internet). I almost always set it to 720p or 1080p, but I have to do it every time because it doesn't remember my setting.

</comment></comment></comment></comment><comment><USER5> No it wont. Sure they might waive fees for a 60 days or so but to think that these corporations would give up their free money piggy bank is untethered from reality. If anything they will use it to try and get people to upgrade to a higher tier.

<comment><USER6> A shiny nickel says that right now, there is an ISP sales VP asking if the terms of service for customer lines exclude using a VPN to get in to work, and if this can be detected and used as a forced upsell.

<comment><USER7> I'm actually hoping that ISPs start to squeeze in this direction just because it's something that might actually garner enough anger/support to force legislators to say what exactly buying "internet service" means. Businesses aren't gonna be happy when people start expensing their internet service.

So _please_ force customers to upgrade to a $2-3x "business class" package if they want to use their internet to do any work/commercial activity.

<comment><USER8> Until the ISPs decide that all freelancers and any VPN are "business use" and then the legislators don't do anything about it.

</comment></comment></comment></comment><comment><USER9> "Dats caps" have many variations.

Americans are used to the version where tricks and ISPs advertise "unlimited" but have arbitrarily defined and enforced caps with no transparency where tut may get throttled, random packets dropped, that sort of thing.

Australia OTOH (IMHO) handled this much better.

The ACCC (like the FTC/FCC but with teeth and without regulatory capture) said it was false advertising under the Trade Practices Act to say something was "unlimited" if it has soft or hard data caps.

What this did in the nascent days of Australia's broadband was force ISPs to offer an explicit quota that was advertised. It allowed Javier users to pay more and lighter users to pay less.

Some hardliners here will rejects even that but non-American ISPs have one big coat and limitation that American ISPs don't: they have to pay to connect to America. And transpacific bandwidth certainly wasn't free.

Nowadays unlimited is more the norm and it's actual unlimited too. This is just market maturity as external bandwidth is a lot more plentiful than 10-20 years ago.

Of course, the connection speeds still suck but that's a whole other thing.

My point is that companies need to be stopped when they make false claims about "unlimited" or anything else.

</comment><comment><USER10> This will probably only change through regulation. The only way you'll get regulation is by making politicians view this as very good PR for themselves.

(This is what happened in Israel, though it's rare to see such effectiveness here)

</comment><comment><USER11> What are the data caps? How many GB of up and down traffic?

Here in Norway only mobile data is capped. ADSL and fibre are effectively unlimited. As far as I know this is generally the case in Europe.

<comment><USER12> Here in the US, my Comcast internet connection comes with a 1TB (up and down combined, i think) per month cap. If you go over they give you a warning the first few times, then charge some outrageous amount per gigabyte that you go over. If you ask them how much for an uncapped connection, they will lie to you and claim that option does not exist. Only after blowing the cap a few times will they admit that they will remove the cap for an _additional_ $50/mo.

This is the only ISP that services my house in a residential area of Seattle.

<comment><USER13> Even working from home I think would have trouble blowing through 1TB in a month. How many hours of video conferencing would that cover? Initial Google search says about 4hrs/GB, so 4000hrs/TB, or over 5 months of nonstop video conferencing?

<comment><USER14> Video conferencing is not the only thing people do with their residential internet.

<comment><USER13> It's the most data intensive thing I can imagine that fits with working from home due to Covid-19.

Obviously if you're hosting a website from your home server you can go above that, but why would you do that instead of using a real host and ssh-ing in, which wouldn't even register against 1TB?

<comment><USER15> You're assuming that nobody has a job that requires transferring large files on a regular basis.

<comment><USER13> Why do those files have to be transferred to your home? Can't you use ssh/remote desktop to connect to your office desktop and work with them there?

<comment><USER16> AutoCAD over local NFS chokes, I can't imagine trying to run it over VDI.

Some things are just IO and processor intensive, and for those, having the data be local is a much more efficient process overall.

<comment><USER17> > AutoCAD over local NFS chokes, I can't imagine trying to run it over VDI.

VDI lets the "desktop" actually running the software potentially be more local to the storage than local NFS, even though both are more distant from the end- user, so I don't see the problem.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER18> I've often thought there should be like an optional slowlane and fastlane for downloads. Last night I downloaded a 20GB game and it nearly pinned my internet connection and downloaded in about 20 mins. But I knew i wasnt going to play if for several hours and there would be no reason to "brownout" my neighbors. I want fast internet when i want it. But also see a time and place for "download this, eventually"

<comment><USER19> This _should_ be possible client-side... You just negotiate the TCP ACKs to have a slower response rate, essentially self-throttling the quality of the connection.

<comment> frosted-flakes OneDrive and SyncThing allow you to set a max transfer speed, so it's definitely possible. OneDrive in particular will absolutely kill my network if it's not throttled (I have slow 10/1mbps DSL). It took a while for us to figure that out; for months we were randomly experiencing terrible Internet speeds.

I don't even bother with OneDrive anymore because our Internet is so slow that stuff was _always_ syncing. I use SyncThing between my phone, laptop, and desktop instead, which can sync over the local network.

</comment></comment></comment><comment><USER20> They had the chance to price data with some relation to their costs and instead chose to squeeze a few customers while providing everyone inferior service.

Wireless still needs to have a price associated with usage; usage monopolizes a shared resource, EM spectrum, and there should be a cost for that.

For wired or fibre as the CEO of Sonic, Dane Jasper, has always said data is in reality mostly too cheap to bother metering.

</comment><comment><USER21> Doubtful. I wfh and have hit Comcast's cap a few times in the past. Radioactive? I have no fucking choice even though less than a mile away in all directions there's century link.

The solution is to get a lower speed. I went from 300 to 250 to 200. It's not noticeable. I'd like a gig, but it's just not worth it. This way I pay less too. Fuck Comcast and the monopolies we have here.

</comment><comment> Funes- We should use open-source mesh wireless networks, and do away with ISPs once and for all.

<comment><USER22> Wireless mesh networks are all fun and games until you try to scale up.

First you have to deal with the huge coordination issue of getting everybody in line and on line to actually form a mesh.

Then if you actually succeed with that you have to deal with physics, maintenance and upgrades.

Most, if not all networks that start as meshes, devolve into centralized networks.

Then all you have is another ISP.

The root cause to why wireless mesh networks don't exist in any meaningful way is that traditional ISP networks are more cost effective.

</comment></comment><comment><USER23> What? LOL. No it won't. A few providers will lift them temporarily and then go right back to how things were in a couple of months.

> the very fact that the limits can be lifted at will or certain high-traffic > categories (such as a broadband company's own streaming TV channels) can be > exempted [... proves that] there is definitively enough capacity for the > network to be used without those caps

Someone doesn't understand how the Internet works.

<comment><USER24> Please connect the dots here; what you've written is nonsensical without further explanation. The technical argument in TFA is pretty simple: if this traffic level is feasible now, it will also be feasible in a couple of months. If instead you're making a point about how horrible FCC is, then be more explicit about it.

<comment><USER25> The cable company's own streaming channels are presumably served from servers on the company's own infrastructure, and the traffic from them to your home does not leave their own infrastructure. So, presumably, that does not count against your internet cap because from the cable company's point of view it is not using the internet. To them, it's part of their intranet.

<comment><USER24> I don't think that's really what we're talking about? Anyway, it's incorrect, because backhaul is cheap. The whole reason we have to deal with Comcast etc. is the many barriers to competition over the "last mile".

</comment></comment></comment></comment><comment><USER26> The best way to do data is (connection_fee + usage/MB). That's the most customer-friendly way to do things. It allows customers to alter their usage according to their billing ability. Fixed line usage is probably in the decicents per GB. Wireless is probably in the $8/GB range. All acceptable. I may have the numbers wrong but I think I'm probably in the ballpark.

<comment><USER22> You are pretty close on the numbers.

Fixed network bandwidth is centicents per GB, and that's for wholesale IP transit. Peering bandwidth can be cheaper at scale. For all intents and purposes fixed bandwidth is too cheap to meter on consumer connections. As such a single fixed monthly rate for unlimited usage at the chosen line rate is the appropriate solution. Also the most consumer friendly way.

In regulated markets, wholesale costs for a GB of wireless traffic is less than $3. Here you can argue for billing by usage, but there are markets with true unlimited wireless services, and they do just fine.

<comment><USER26> Ah neat. That's cool. I like that. It could also be like fixed below usage and per GB above to mirror the costs. Essentially, small enough that a normal consumer won't see a problem but your torrent 24x7 guy is paying his dues.

I have gigabit fibre here in SF and if I ran at full tilt I'm doing 100 MB/s × 3600 s/hr × 0.01 cents / GB × 0.01 GB / MB = 36 cents / hr. And that's $260 on traffic per month. I think it's fine to pay that in overages if you're using at full tilt.

I totally get the advantage of not worrying about that. I prefer my unlimited which just has a soft limit but I'd prefer to pay for what I use without subsidizing the 0.01 percentile customer.

I'm also a big fan of traffic shaping for wireless because honestly my WhatsApp message is probably more important than my YouTube video. Pause the latter to send the former. Every time. The real tragedy is that we don't have the attention (physically) to be able to auction instantaneous spectrum bandwidth.

<comment><USER22> You dropped a zero. There are 1000 MB in a GB, so your cost of running a Gbps line at full tilt is more like 3.6 cents per hour.

In other words, bill everybody a dollar per month for bandwidth and worry about it only if average usage per subscriber goes over 10 TB per month.

<comment><USER26> Listen, I have a graduate degree in Mathematics, so a mistake like that was inevitable, _especially_ because I was trying to be careful about it.

That changes everything! The full-bore user is only $30 over the non-user. I have completely changed my mind. Why even bother measuring this nonsense? This explains why my provider doesn't charge per usage for my symmetric gigabit plan.

</comment></comment></comment></comment></comment><comment><USER27> I _am_ sympathetic to the ISPs. It's good for the planet and emergency situations when we each use wi-fi instead of using cellular data. However, I think the ISPs simply haven't shown enough creativity in incentivizing users to use available wi-fi networks over cell data. I imagine very few people even know that we should want to prioritize wi-fi.

<comment><USER28> >It's good for the planet and emergency situations when we each use wi-fi instead of using cellular data.

How do you figure?

<comment><USER27> Everyone using cell signals takes up that much bandwidth that could otherwise be used for emergency personnel - I'm not saying not to use cell towers, I'm just saying that if you could choose between the two with no penalty on either, wi-fi would be preferable.

Also I would think it's far cheaper energy-wise to transmit the signal from my living room to my study rather than beaming the signal from a cell tower miles away. I could be wrong.

</comment></comment></comment><comment><USER29> I called Comcast and had 2 options. Pay $50/month for unlimited or pay $25/mo for unlimited, but I would have to forgo using my own modem/router and use theirs. Decided to just pay the $50, but we'll see what happens. I could see them increasing the cap or forgiving overages for a few months.

<comment><USER30> You must maybe use their modem (how they can check it I don't know, maybe they don't give you the configuration parameters for the network?) but you always have the possibility to add your own router after it and basically use their modem/router as a simple modem.

It's what I did in my country before they made illegal for ISPs to force you to use their modem/router, simply used the modem/router of the provider as a modem, forwarding all the ports to my own router that I used to do actual routing, WiFi access point and all other stuff that a router does.

<comment><USER31> They also charge you a monthly fee for using their modem.

<comment><USER32> They waive that in a lot of cases, as they're trying to build out their network of xfinitywifi hotspots you see everywhere. I still ran my own modem, but it's a point to keep in mind.

<comment><USER33> If it's providing a WiFi hotspot, then it's their _router_ that you're using, not just their _modem_.

<comment><USER32> They give you a combo router and modem by default when you run their modems (for the reason given above). The WAN port is just a downstream port for the internal router.

I don't think there's a way to disable the router itself in most cases these days, just some of the services it provides.

</comment></comment></comment></comment></comment></comment><comment><USER34> For those in the eCommerce area, are you seeing more or less consumer spending right now (outside of supplies ie toilet paper, masks, hand sanitizers..)

</comment><comment><USER35> Are data caps from ISPs still a thing? Where i live, only mobile operators have data caps, regular ISPs ditched that about ten years ago.

</comment><comment><USER36> They seem to have plenty of capacity, no issues in Bay Area despite many more people WFH or working less and streaming more.

<comment><USER18> personally my work uses nearly no bandwidth. a little Stackoverflow here and there. probably peaks with video calls.

Certainly doesnt use even close to 4k streaming netflix.

</comment></comment><comment><USER37> One can only hope. I absolutely loved Three UK for their unlimited 4G, it was my only Internet provider.

</comment><comment><USER38> Verizon hasn't even waived data caps for wireless, so I doubt it.

</comment><comment><USER39> We'd get free electricity and gas too then. Not gonna happen.

<comment><USER40> The difference is that with gas and electric, there is an increased cost to the utility to deliver more electricity and gas.

With bandwidth there is almost no additional cost to deliver more bandwidth. Yes in aggregate they need to upgrade their equipment, but if they just got rid of data caps tomorrow, they would incur almost no extra cost.

If electricity and gas suddenly became fixed cost, the cost to the utilities would go up dramatically.

<comment><USER41> > With bandwidth there is almost no additional cost to deliver more bandwidth

Depends how awful your infrastructure is. The big ISP in Ireland was still using microwave links as backhaul for ADSL until at least 2012 or so (in one particularly ridiculous case, a whole village only had 8Mbit/sec backhaul), and I believe there are still some developed countries where that kind of setup is common.

Interestingly, back in the mid noughties, said ISP either gave ADSL customers a 20GB cap or no cap, depending on if their local exchange had a fibre backhaul. To confuse the issue, they referred to the latter as "fibre powered broadband".

<comment><USER22> The grandparent's comment still stands.

Upgrading backhaul is a fixed cost. There is no incremental cost for usage.

Very much different than power generation.

</comment></comment></comment></comment><comment><USER42> It's a great time to increase prices if you ask me.

<comment><USER43> Good thing nobody's asking you then.

<comment><USER44> Do you think it's a good time to increase prices?

<comment><USER43> I enjoy how you edited this from "Did anyone ask you?"

No. Why the hell would the time of a disaster be a good time to increase prices for people who rely on these services.

Go away.

<comment><USER44> Lol, no, I asked you this question because the other guy said "Did anyone ask you?"

That way you could answer yes to him. It looks like he deleted his comment.

</comment></comment></comment></comment></comment><comment><USER45> The concept that data caps are somehow uniquely unjust is dumb. Bandwidth is a scarce resource, like seats on a train. Nobody gets mad that the train company charges per ride instead of only selling monthly passes. Be mad that your ISP has a monopoly and charges too much. Don't be mad about a pricing scheme that is completely reasonable if the prices involved are fair.

<comment><USER46> I would agree with you if there were evidence to support your claim, but there is not.

Actually there is evidence to the contrary. [https://www.nextbigfuture.com/2013/02/cable-companies- make-9...](https://www.nextbigfuture.com/2013/02/cable-companies- make-97-margin-on.html?amp)

The truth is fiber bandwidth is cheap.

<comment><USER42> There's lots of evidence that internet is cheap. Too bad we don't have those ISPs to provide it. Have you ever thought how hard it is to dig a 50m hole in the ground to put fiber in it?

<comment><USER47> It can be a lot easier than digging trenches. Where I lived I've watched various google & TW contractors running fiber on existing utility poles. It surprised me how fast those guys can move, I'm talking a couple guys and a truck can probably do a few miles a day. Last time I saw it, they had a couple trucks doing some prep work ahead of a guy literally walking along under the cable pulling that spinner thing. They had wired from the office building I was in, down the road a couple miles, by lunch.

(random youtube video) [https://www.youtube.com/watch?v=aUMjCvMxp-M](https://www.youtube.com/watch?v=aUMjCvMxp-M)

<comment><USER42> Most people move fast if they are paid accordingly.</comment></comment></comment></comment></comment>


Nerdiness not linked to glasses - <AUTHOR> http://www.cbc.ca/technology/story/2008/03/28/glasses-nerds.html?ref=rss <comment><USER2> Reminds me of this research study:

Cichlid = fish

"Ethanol and three beverage alcohols were examined for their effects on the aggressive responses of the convict cichlid. At an ethanol concentration that gave negligible effects on the basal level of these responses, rum was observed to depress aggression."

<http://www.springerlink.com/content/t2p3167173282h4m/></comment>


Building an AngularJS Directive for Cat Videos - <AUTHOR> http://davidandsuzi.com/building-an-angularjs-directive-for-cat-videos/

<comment><USER2> Shouldn't the final template have a {{code}} after embed/ in the src attribute?

<comment><AUTHOR> oops - good catch. Actually, it should have been {{video}} - it's been updated. thanks.</comment></comment>


Brain has ability to adapt and change through life - <AUTHOR> http://articles.sun-sentinel.com/2010-03-22/health/sfl-human-brain-030910_1_brain-science-connections-change <comment><USER2> In other news, the sun rose today.

This is an anecdotal article telling no one anything new, nor is it conveying an opinion or any sort of knowledge you wouldn't expect most HN-ers to know.

For some actually interesting reads: <http://cpirf.org/stories/352> [from 1997]

<http://iospress.metapress.com/content/6c5glqvnjgmc5194/> [only if you have a subscription or your school does... but interesting nonetheless]

<http://en.wikipedia.org/wiki/Neuroplasticity>

The list goes on.</comment>


Sony Ericsson Brings the "Funny" - <AUTHOR> http://www.androidguys.com/2010/09/21/sony-ericsson-brings-funny-series-xperia-x10-clips-video/

<comment><AUTHOR> After watching a couple of clips, here's my rant: This is why Apple wins and companies like Sony Ericsson are left in the dust and will continue to be. It's not Steve's reality distortion field, etc. etc., but as pointed out innumerable times, it's the attention to detail. These supposedly funny videos are not only totally humorless (and in some cases offensive, e.g. the Models one) but what is worse brainless. And even worse: They smell clueless wannabe. As wannabe as Microsoft's Seinfeld ad spots. They are obviously the product of a PR group working with a cool ad agency trying to imitate some misguided notion of coolness or something.

And this is exactly what separates the truly great from the also rans.</comment>


Ask HN: How do you work in a larger Scrum Team? - <AUTHOR> We are a relatively large Scrum Team of 8 (soon to be 9) developers. Apart from the developers we also have other roles in the team, for example a designer, a tester,... All in all we are 14 people.
We frequently run into problems because of this size. Most of the time there are multiple projects going on, with different people creating smaller groups focusing on one of the various projects. It's really hard to be on the same page about everything that's going on. Our standups tend to drag on, probably because the stuff people are talking about only ever is interesting for a small part of the team. We are not pulling in the same direction because we are pulling multiple things.
To me it feels obvious that the team is too large, but it's not an easy thing to change. So in the meantime I would like to focus on improving what we have.
Does anyone have experience working in a team of this size? Any ideas about how we could improve the way we work? <comment><USER2> We ran whole company (startup) standup morning meetings until we reached 18. Regular employee feedback was that the meeting is too long. Also regular employee feedback was they'd like to learn more about what other people are working on.

We continued the standup meetings and forced people to be prepared and keep it short. Anything over three sentences got cut off. If somebody was late no comments were allowed and nothing will get repeated for that person. Looking back it was best for keeping the team (company) together.</comment>


Ask HN: Covid – Cause and Effect - <AUTHOR> I'm ultimately looking for a reason why this hypothesis is wrong. It is partially about the role of tech during the pandemic.
Assuming it is generally accepted that COVID originated in bats, then it's reasonable to look at characteristics of bats that are unique. Perhaps even find a correlation between the behavior of bats and humans.
Bats are nocturnal. Most species of bats never get sunlight. One study I found as an armchair researcher points to evidence of a vitamin D deficiency in wild bat populations: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3897589/
Here is a study done back in may about Vitamin D levels and the severity of COVID: https://www.sciencedaily.com/releases/2020/05/200512134426.htm
Now consider the changes in human behavior as we have become glued to technology.
We stay inside more, we avoid the sun at all costs.
This means we all likely have lowered vitamin levels (Vitamin D and Calcifediol specifically).
To top all of this off, our response has been to stay inside even more than we used to, allow tech to dominate all areas of our lives, thus worsening the problem.
Also consider the populations hardest hit, there would be vitamin D absorption problems as well.
Consider senior citizens who likely stay inside even more than the rest of us.
This would explain some of the death rates in nursing homes when compared to the larger senior population.
Is it possible we have basically trolled ourselves with our technology addiction and the solution to this current problem is to just go outside? It can't be that simple right?
EDIT: Here is a link I just found in another thread on Hacker News showing a correlation between the two:
https://www.medrxiv.org/content/10.1101/2020.06.01.20112334v1 <comment><USER2> Staying inside was never the actual response to Covid-19. The CDC and local guidelines specifically say that going outside is a safe activity. Where I live I see a lot of people outside, distant from each other, taking walks and hanging out in the park like any other summer.

In fact, one of the only safe ways to spend time with friends is outside. I know of many people who had socially distant gatherings outside - in the past they may have spent this time indoors in close quarters.

I don't assume that everyone is literally staying inside any more than they were before. It's not like the office was outside, either. Heck, the 20% of America that is now unemployed might be spending _more_ time outside.

Reducing the risk of contracting Covid-19 is about people, place, time, and space. Sitting in a restaurant for an hour with trapped air mingling amongst talking and eating people is a lot worse than passing someone on a sidewalk. Nothing about the guidelines have anything to do with literally remaining indoors.

Also, the response isn't about maximizing outcomes once you get the virus. It is about preventing the hospital systems from being overrun. In other words, whether or not staying home helps you survive Covid-19 once you get the virus wasn't ever a part of this mitigation.

The government can't make people suddenly not be obese or have heart conditions. They can't make you eat healthy and get the vitamins you need. All they can do is stop the hospital from running out of beds and staff.

As far as your theory, I dunno. I don't see anything beyond a very weak link. Humans are pattern builders, and we all want to shortcut ourselves to see a pattern out of very different pieces of information. I'm not going to guess either way on your theory because I don't want to jump to conclusions.

<comment><AUTHOR> I actually just found this link in another thread about a correlation between the two: [https://www.medrxiv.org/content/10.1101/2020.06.01.20112334v...](https://www.medrxiv.org/content/10.1101/2020.06.01.20112334v1)

Fair points about the response to the virus. However, if there is anything to the sunlight idea then it's going to look questionable in hindsight.

</comment></comment><comment><USER3> FWIW, I've been spending more time outside during the pandemic. Sunlight and fresh air are are almost always good for you, and this pandemic seems to be no exception.

The authorities responding with lockdown or "shelter in place" are their one- size-fits-all plan to threats. They only have a few sets of procedures, none of them tailored to a droplet-borne respiratory pandemic.

Leading with similarities to bats is unnecessary, and probably undermines your claim. There's evidence that vitamin D deficiency has a big effect on severity, and staying inside reduces sunlight-generated vitamin D, so you don't need to invoke bats. Just take vitamin D supplements get get outside.

<comment><AUTHOR> I guess the point with bats is that staying out of sunlight as they do created the conditions that led to a virus like covid being able to take root. As a result of a technology addiction, humans started living in their own caves, and also became susceptible to the same kinds of disease as a result. Maybe I'm projecting my own experiences... It seems that people in general have stayed indoors more as technology got cheaper and widespread.

</comment></comment><comment><USER4> It is important even males depilate their arms to get any level of UV penetration to produce the D vitamin</comment>


Are you thinking, or are you farting? - <AUTHOR> http://slash7.com/articles/2009/6/10/are-you-thinking-or-are-you-farting

<comment><USER2> _It's a cycle of awesomeness._

I'm not convinced... It's an interesting article, but it seems to me this cycle can just as easily devolve into mental masturbation. "I'm thinking X. Am I thinking this because Y or Z? What's the meaning of thought anyway?" Keep doing this for too long and you might turn into some kind of philosopher.

Now, I agree that some self-examination is essential (as Socrates put it, the unexamined life is not worth living), but I'd argue that what's needed is a balance between thinking and doing. And doing requires putting aside all those fabulous thoughts and focusing on the task at hand for a while.

The two are not exclusive, nor is one superior to the other. They are a bit like Reason and Passion in Khalil Gibran's writings (<http://www.katsandogz.com/onreason.html>):

 _Your reason and your passion are the rudder and the sails of your seafaring soul. If either your sails or your rudder be broken, you can but toss and drift, or else be held at a standstill in mid-seas. For reason, ruling alone, is a force confining; and passion, unattended, is a flame that burns to its own destruction. Therefore let your soul exalt your reason to the height of passion, that it may sing; And let it direct your passion with reason, that your passion may live through its own daily resurrection, and like the phoenix rise above its own ashes._

 _I would have you consider your judgment and your appetite even as you would two loved guests in your house. Surely you would not honour one guest above the other; for he who is more mindful of one loses the love and the faith of both._

 _Among the hills, when you sit in the cool shade of the white poplars, sharing the peace and serenity of distant fields and meadows -- then let your heart say in silence, "God rests in reason." And when the storm comes, and the mighty wind shakes the forest, and thunder and lightning proclaim the majesty of the sky -- then let your heart say in awe, "God moves in passion." And since you are a breath in God's sphere, and a leaf in God's forest, you too should rest in reason and move in passion._

<comment><USER3> _"I'm thinking X. Am I thinking this because Y or Z? What's the meaning of thought anyway?"_

In the thought example about the "Milk" in the article I found myself asking "Who does that?" and now I'm wondering if that helps further explain to me how women tend to think into things.

I'm curious to know here who would have gone onto thought #3 in the milk example? If I'm enjoying something I'm simply enjoying it... I'm not trying to determine if I'm _really_ NOT enjoying it and thus trying to convince myself that I AM enjoying it.

<comment><AUTHOR> Haven't you ever come out of a situation and, at the end, when it was all said and done, realized that you had talked/fooled yourself into it?

Self-congratulatory tones can be indicators of something that is _very bad_.

It's important to not get high on your own supply. A lot of what we tell ourselves is wrong, or at least misguided. If you question your own intentions and observations, you can avoid making big mistakes.

For example, I know lots of people who tell themselves "I didn't get what I want because x." where x is something that is obviously completely beyond their control, and therefore not their problem/responsibility.

But if you want to protect your ego, telling yourself that it's because of x-ism is totally effective. It also cripples you.

Which is why you have to ask yourself, "What's my motive here?" and "What do I have to lose if I think about it from a different perspective?"

Also, considering most of the people who have promoted self-examination and self-understanding over the years have been men, your bit about women is somewhat disingenuous.

<comment><USER3> _\-- Haven't you ever come out of a situation and, at the end, when it was all said and done, realized that you had talked/fooled yourself into it?_

No. Before I ever enter a situation I already know whether or not I need to convince myself. I'm pretty aware of my own intuition, and the more I need to "talk myself" into doing something the more I'll question myself before I skip over my gut instinct.

It's been extremely rare when I've come out of a situation where I trusted my intuition and been unhappy with the result. On the other hand, I've had many times where I overrode my intuition and thought "damn, should have trusted myself"

 _\-- It's important to not get high on your own supply._

I don't. It's called being proactive and is functionally similar to the techniques of mindfulness. Steven Covey writes about it in his first chapter of "Seven Habits of Highly Effective People". And please, entrepreneurs ignore reality more than most people I've ever known. Start up a business against all odds... convincing yourself to keep going with an idea even though you haven't succeeded after years? How is that _NOT_ being high on your own supply?

 _\-- If you question your own intentions and observations, you can avoid making big mistakes._

This is the big fallacy of questioning your every thought and intention. If _only_ I could just monitor every thought, every motive, question my every step, then I could do it perfectly -- I would avoid the big mistakes. Wrong. You might have perfectly reasoned every thought, justified every intention, and still fail simply because you didn't have all the information.

So with that in mind, that's why it's much better to simply DO. Ensure one's intuition is in agreement and then DO, don't think into it. Mistakes are going to happen anyway, and that's when you'll do the reflecting.

<comment><USER4> _\-- If you question your own intentions and observations, you can avoid making big mistakes._

 _This is the big fallacy of questioning your every thought and intention. If only I could just monitor every thought, every motive, question my every step, then I could do it perfectly -- I would avoid the big mistakes. Wrong. You might have perfectly reasoned every thought, justified every intention, and still fail simply because you didn't have all the information._

I like to compare this to Gödel's first incompleteness theorem. No matter how much we examine our thoughts, we're still applying the same brain to them. You might hope to catch some of the problems by applying second, third and fourth thoughts to your first thoughts, but there's bound to be a flaw you can't work around, "an arithmetical statement that is true, but not provable in theory" ;)

<comment><AUTHOR> The "You" that is doing the analyzing can obviously learn and grow... otherwise, what's the point of doing anything?

And, if you read the original article, you'll see it's not about "catching problems" at all.

<comment><USER4> See, that kind of hostile attitude is what usually turns people away from any advice you might offer. Your assumption that I haven't read the article -- because my wording of my interpretation of it differs from what you expect -- is unwarranted.

The article itself was rather interesting and I certainly didn't mean to imply that metacognition is useless. From my point of view, not engaging in any metacognition ever will eventually lead to what I call problems, among which are unwarranted assumptions and failure to consider alternatives. Those are "problems" that I like to "catch" and correct in my own thinking. However, I do believe that one has to draw a line somewhere; otherwise, one risks indulging in what another commenter named "mental masturbation".

As for personal growth, I believe there will always be limits to our thinking that cannot be (directly) surpassed by more thinking ;)

<comment><AUTHOR> I didn't intend to sound hostile. I think we're talking past each other here :)

Also, I agree with you 100% on mental masturbation. (I also blog occasionally @ <http://www.justfuckingship.com/>). But I _am_ tired of every freakin web app looking, acting, and being essentially the same.

We also define problems differently. I think it's a bigger issue that all web app are basically the same, than if a given individual sits and thinks _too hard_ and never does anything. But that's because I have barely any problems with executing any more... because I've worked on it.

Have you heard of the categorization of failures?

Failure Level 1 - Doing the right thing, but doing it badly.

Failure Level 2 - Doing the wrong thing, whether you do it well or badly.

I mostly like to poke people about Level 2. No blog/article/motivational speaker/whatever can do _everything_ so I pick my battles :)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER5> _Thought 1: I want to make a forum. Here's how I will make the front page that lists all the forums, making it clean and easy to see the most popular stuff, even if it's older...

Thought 2: But wait. Do I need a front page that lists all the forums? Am I optimizing instead of thinking?

Thought 3: Are forums really the right tool for the job?

Thought 4: How would I recognize the right tool for the job? What does communication mean? How does this change on the internet?_

I do this, but unlike the author I've come to regard it as generally a bad habit. Too much thinking gets in the way of _doing_. These days I try to avoid over-thinking everything, and concentrate on just making something - anything, and seeing what happens when I launch it.

<comment><USER6> this is sometimes a good idea, sometimes not (think Space flight, for example).

The current thinking, probably stemming from Google's design-by-stats methodology is that this scales down to startups, you build something and morph it based on feedback. We used to call that design by committee, but clearly when you give it a new name its a different animal :)

I think what's more likely to have happened is that the problems have changed from things like best way to compress these bytes, faster state machine for this network stream, better LCS algorithm to find my n-gram etc, to problems that involve many social factors for their success. When this happens the only way to find those factors is to involve the social community that is creating them. Often those doing the iteration/refine cycle with the developers are not the same as those ultimately using a product, but when it works, it works well.

Software is seeing a shift towards large scale problems, like making the accumulated history of mankind available to anyone, anywhere; or supporting complex life decisions; or connecting and staying connected with multiple disparate sets of 'friends, etc. These are radically different problems being solved in different ways.

But not every problem is solved best this way.

</comment></comment><comment><USER7> Link broken (goes to edit login). Try here:

<http://slash7.com>

<comment><USER8> Or [http://slash7.com/articles/2009/6/10/are-you-thinking-or- are...](http://slash7.com/articles/2009/6/10/are-you-thinking-or-are-you- farting) for the direct link :)

</comment></comment><comment><USER9> Tangentially related. Asking "Why 5 times"

"Taiichi Ohno is known to have said that "having no problems is the biggest problem of all." He viewed problems not as a negative but as a "Kaizen opportunity in disguise." Whenever problems arose, he encouraged his staff to investigate the problem at the source and to as "ask 'why' five times about every matter"

[http://www.shmula.com/382/ask-why-five-times-about-every- mat...](http://www.shmula.com/382/ask-why-five-times-about-every-matter)

<http://en.wikipedia.org/wiki/5_Whys>

<comment><USER6> 5, of course, being utterly arbitrary, and even when it does work, its largely luck. Root cause analysis is a science, why5 is guess work, done 5 times.

</comment></comment><comment><USER10> Are we human, or are we dancer?

<comment><AUTHOR> Arggggh, why did you have to bring that up? I'm going to have to drink it out of my head later today.

</comment></comment><comment><USER11> observation without judgment seems very similar to the idea of mindfulness explored in vipassana meditation.

<comment><AUTHOR> Yes, vipassana meditation changed my life.

But I want to follow Terry Pratchett's lead and teach unobtrusively. He's really good at it. So, as a step 1, I didn't ever use the words Buddhism or meditation.

<comment><USER11> enlightenment on the down-low! Sneaky.

</comment></comment></comment><comment><AUTHOR> Oh my god, I'm an idiot. That'll teach me to try to use a utility for multiple clipboards. :(

Yes, the direct link is

[http://slash7.com/articles/2009/6/10/are-you-thinking-or- are...](http://slash7.com/articles/2009/6/10/are-you-thinking-or-are-you- farting)

D'oh.

<comment><USER12> I assume one of the editors will fix it at some point... or you could resubmit.

<comment><USER2> It's already accumulated a fair few comments, I don't think resubmitting would be appropriate.</comment></comment></comment>


Say What You Mean (in JavaScript) - <AUTHOR> http://iterative.ly/2010/05/18/say-what-you-mean-in-javascript/

<comment><USER2> Replacing "if" constructions with boolean operators is powerful. The simplistic example in the article doesn't really show the purpose of using || and && to control flow.

It's scalable. What if you have to test a third expression? Surely this:

foo() && bar() && buzz()

is more readable than

if(foo()) if(bar()) buzz()

It's flexible. Since an "if" statement is not an expression in most languages, you have to be careful where you place it. For example, what if you wanted to pass the result to a function?

qux(foo() && bar() && buzz());

Javascript (and a few other languages) have the neat feature with boolean operators that they return the result of the last tested expression. So the above example would pass the result of buzz() to qux if all statements are true, otherwise it would pass false. Here is the same code with "if" statements:

var res = false;

if(foo()) if(bar()) res = buzz();

qux(res);

There are many more examples of why you should be used to constructing expressions with boolean operators. It doesn't matter much for the simplistic example in the article, but the same technique allows you do write more complex code in succinct ways.

<comment><USER3> This does not respond to the issue raised in the article.

In all of jlongster's examples, the result of the '&&' operator is being used. So, someone reading the code can mentally consider the '&&'s to be logical operators.

However, in the article's examples, the result was not being used at all. So, the '&&' was being used not as a 'logical operator' but as a 'conditional operator', which is a non-intuitive use.

Also, '&&' and '||' are not Boolean operators -- they are logical operators, where order matters for short-circuiting.

<comment><USER4> For what it's worth, I've often seen code like jlongster's example, but never seen code like the OP's example.

</comment></comment></comment><comment><USER5> While I endorse the argument for coding for readability (all things being equal), I don't agree with his example. I find the && syntax in javascript completely readable.

<comment><USER6> I find it readable if you're trying to test both sides of the && to be true -- if(foo() && bar()){ // do this }

However, the way it's written in the article, the intent is to merely test foo() and then execute bar() if foo() returns true. Even though I'm familiar with &&, it took me a minute to understand this as well. I'd be eqaully annoyed seeing this in production code.

<comment><USER7> I'm with the GP on this one... Readable code is very important. Using the short-circuiting property of && doesn't decrease readability in most cases (unless someone isn't familiar with the language, or the concept of using short-circuiting for flow control).

</comment></comment></comment><comment><USER8> I guess what you find readable also depends on where you're coming from. When I first saw that foo() && bar() line in the OP, I knew what it meant immediately. It's because I use it myself, and I've seen raganwald's andand operator and heard of Haskell's Maybe monad.

I've used it for so long that it's come to mean "if the first exists, then do the second", and not just "this expression is true if both are true", and I don't really confuse the two.

<comment><USER9> You'd exaggerate the problem at hand too if you were pushing some strange variant of the language criticized. Any programmer experienced with C-style languages recognizes logical constructs with && and || immediately.

If

      function (x) {
        x = x || someDefaultValue;

for setting non-falsy default argument values isn't a JavaScript idiom, I don't know what is.

<comment><AUTHOR> But that's not the code I was criticizing. In fact, it's precisely because that's the idiom I associate with boolean operators in JavaScript that the code snippet I posted struck me as unintuitive.

As to "pushing some strange variant": Guilty as charged. Have you tried CoffeeScript? It's _fantastic_. The code you gave above would be written as

        (x) ->
          x ||= someDefaultValue

<comment><USER9> > Have you tried CoffeeScript?

Nope. I'm apprehensive about adding another layer to the development process when I know JS very well and don't find it at all limiting. So I'm probably not part of your target audience.

<comment><AUTHOR> I was skeptical at first too, and I wouldn't recommend it to anyone who doesn't already have a pretty good understanding of JavaScript. But I'd urge you to try it for a week or so. It's not a layer of abstraction like GWT; it's just a nice, succinct syntax that compiles statement-by-statement to JavaScript.

</comment></comment></comment></comment></comment><comment><USER10> I prefer this:

        if(foo()) bar();

No {} necessary if it's just a single command.

<comment><USER11> I prefer to always use {} for readability and consistency's sake, regardless of line count. Also reduces human error.

<comment><USER10> Me too. But if the goal was to shorten "if(foo()) { bar(); }, then I could just leave the {} braces out instead of making it "foo() && bar()".

</comment></comment></comment><comment><USER12> I think sometimes what's "readable" to one person isn't to another. The more you hit a coding convention, the more readable it becomes (as with anything). So noted. Move along. There are bigger fish to fry.

<comment><AUTHOR> Point taken: I'm being really, really picky in this post. It's is far from the most confusing code I've seen today.

But I'm making a larger point: JavaScript tends to be a lot less readable than code written in other languages not just because of the language itself but because of the byte-squeezing habits web developers have picked up over the years. I think JavaScript tools have advanced to the point that we can finally drop those habits.

</comment></comment><comment><USER13> && and || are commonly used as "short circuit" operators in JavaScript. I have no problem with this.

</comment><comment><USER14> The old rule of thumb was, "Online, go for readability. Batch, go for speed." This almost always translates into, "Online, use 'if' or 'case'. Batch, use Boolean."

In the first case, maintenance programmers will thank you. In the second, users processing millions of records will thank you.</comment>


How to replicate Google "Hangouts On Air" stream combining functionality? - rob-olmos http://stackoverflow.com/questions/19483397/how-to-replicate-google-hangouts-on-air-stream-combining-functionality

<comment> rob-olmos The original library I was experimenting with was from Main Concept. Basically we were just decoding the frame to it's raw format, reading out the bytes/pixels of two frames side by side with some black pixels in between, and then trying to encode this new frame. I hope there's an easier, more widely applicable way.</comment>


Tutanota aknowledge being under attack - <AUTHOR> https://www.facebook.com/tutanota/posts/4666783140013399?__tn__=-R <comment><USER2> I'm actually a customer of Tutanota, and their product has been pretty good in the last ~2 years I've used it.

Short of hosting your own email, I can't think of other email providers as secure as tutanota. They tick off a lot of boxes: open source, hosted in Germany, encrypted mailboxes, DKIM/DMARC support...

The only feature that might give people pause is that their search functionality isn't great. I have pretty low volume of emails to my personal inbox so it doesn't matter, but it might be frustrating to use as a work inbox.

<comment><USER3> How does it compare to ProtonMail?

<comment><USER4> I like Tutanota much more (except for the last week) for a few reasons. The ones that come to mind include email search, more affordable pricing, and more flexible domain allowances. Also, they tend to roll out new features on a faster timeline, such as the recent addition of a new domain verification method that few providers currently support (that I forgot the name of?) and an in-app calendar.

</comment></comment></comment><comment><USER5> Their blog post contains this interesting note: [https://tutanota.com/blog/posts/ddos-attack- tutanota/](https://tutanota.com/blog/posts/ddos-attack-tutanota/)

> We have long wanted to publish a status page. However, as a privacy-first > email service, we cannot use Google services to host a status page (like > most services do).

I guess Facebook doesn't count though.

<comment><USER6> I can't get to their status page, which would seem to defeat the purpose of a status page.

I wonder if a static page on Netlify would comply with their privacy stance.

<comment><USER7> The fact that a status page is not reachable would itself imply the status of the service whose status it reports, no? Especially since visiting the status page is not normally done unless someone is having problems with the service.

<comment><USER8> Not really; part of the point of a status page is that it lets you distinguish between "their service is down" and "the route between me and their service is down".

</comment></comment></comment></comment><comment><USER6> If you prefer something that's not Facebook:

[https://twitter.com/TutanotaTeam/status/1306344728745644033](https://twitter.com/TutanotaTeam/status/1306344728745644033)

I don't know if we will get a report as to what happened, but Tutanota is frequently used for…perhaps not entirely legal transactions. _(Not that I would ever do such a thing, perish the thought.)_ I wonder if they made the wrong person/drug syndicate angry…

<comment><USER9> They supply a steady stream of spammers and phishing gangs with email addresses to use when signing up for email sending services. We blocked their domains entirely because it is all garbage.

<comment><USER6> Huh, the email service that everyone uses to buy "research chemicals" via Bitcoin attracts a shady audience? Shocking.

<comment><USER10> FWIW, I find those types tend to recommend ProtonMail more than Tutonota

</comment></comment></comment></comment><comment><USER11> >Tutanota

>3:09 PM (Sep 20 2020)

>We are very sorry about this ongoing outage. We are working on fixing this. While someone wants to stop you from using encrypted emails, we are committed to not let that happen. Thank you very much for your patience and for weathering this storm with us.

</comment><comment><USER12> Weird fb quirk that I never noticed, if you click on the link it goes to the post. Then if I click 'go back' it doesn't go back here, it opens the fb front page. Then I click 'go back' again and I'm here. They somehow Load the front page and then direct me to the post so quickly I don't notice it when I'm going to the post. I'm on iOS Safari...

<comment><USER13> Might be a mobile quirk? I tried it on Mac Safari and it went back like normal.

<comment><USER12> Given how spotty it is, it might be some weird A/B testing. I'm also logged in to Facebook which might make a difference.

</comment></comment></comment><comment><USER14> If like me you never heard of them before:

Tutanota is an end-to-end encrypted email software and freemium hosted secure email service.

</comment><comment><USER15> They've been down most of the day in North America. I've been using the service for about a year or so, without any problems but they have had at least some down time almost every day for the last few weeks. How common is it for botnet attacks like this to last so long?

</comment><comment><USER16> "You must log in to continue."

One would hope encrypted email providers would know better than to force logging in to facebook to read their posts.

<comment><USER17> I don't think that's up to the page owner. And you still have the option of selecting [Not Now] and reading the post

<comment><USER18> I have often found that organizations (government or private) have no idea how to configure privacy for their FB announcements so there is no way to view public content without logging in, even if it is ordinarily available to anyone with a FB account.

</comment></comment></comment><comment><USER19> They don't use DNSSEC, which means they can't support DKIM.

They've given control of their DNS to NSA linked Amazon.

They'll probably move their webmail to FBI linked Cloudflare next.

<comment><USER20> Virtually no DKIM zones are DNSSEC signed.

DNSSEC itself is linked to the NSA.

</comment></comment><comment><USER21> I tried their service 3 years ago, i couldn't receive emails and had to contact them through reddit... at least i got a refund.</comment>


Today is IBD day, why should you care? - <AUTHOR> http://metro.co.uk/2015/05/19/what-world-ibd-day-means-to-me-and-why-you-should-care-too-5203567

<comment><AUTHOR> Hi guys, I am posting this because person close to me has Chrohn disease and its pretty nasty disease.

There is nice overview here [https://www.youtube.com/watch?v=Keqzt83KMVA](https://www.youtube.com/watch?v=Keqzt83KMVA).</comment>


1 in 4 Statisticians Say They Were Asked to Commit Scientific Fraud - <AUTHOR> https://www.acsh.org/news/2018/10/30/1-4-statisticians-say-they-were-asked-commit-scientific-fraud-13554 <comment><USER2> I was going to give a lecture across several departments about my PhD research in bioinformatics. The night before the talk I was generating some new figures and saw something weird, which led to me digging through source code and discovering a bug which invalidated the last 18 months of research and all my conclusions. I went to my advisor with this problem and he told me to present it anyways. I refused. I suffered consequences for that, including getting stonewalled by my advisor whenever I tried to publish a paper. I wish I had actually been in a position to refuse.

<comment><USER3> This is par for the course with computational research. I discovered a bug in my code the _last week_ before submitting my PhD dissertation. Luckily, all of my code was organized in a pipeline that could automatically regenerate everything on the fly, but I needed a supercomputer. The queue was too long for Titan, so I set up an impromptu cluster on Azure (it was the only cloud provider with Infiniband at the time) and paid $200 out of pocket to regenerate the correct figures.

I wouldn't be surprised if a significant portion of published computational research has bugs that totally invalidate the conclusions. I think we need to push hard to require all taxpayer-funded research to make any code that results in a journal article publicly available.

<comment><USER4> Open source doesn't ensure quality code.

Ideally the code would be part of the peer review process, but code review is really expensive, so who knows how that would play out.

<comment><USER5> It's simple:

We stop publishing in papers, and instead adopt smaller chunks of our work as the core publishing units.

Each figure should be an individually published entity which contains the entire computational pipeline.

Figures are our observations on which we apply logic/philosophy/whatyouwannacallit. Publishing them alongside their relevant code makes the process transparent, reproducible and individually reviewable, as it should be.

We can then "publish" comments, observations, conclusions etc on those Figures as a separate thing. Now the logic of the conclusions can be reviewed separately from the statistics and code of the figure.

<comment><USER6> > Each figure should be an individually published entity which contains the > entire computational pipeline.

I agree in principle. But, for the experimental sciences, we need better publication infrastructure to make this practically possible.

For example, consider a figure that summarizes compares, between several groups, the mechanical strain of tensile test specimens for a given load. Strain is measured from digital image correlation of video of the test. Some pain points:

1\. There is a few hundred GB of test video underlying the figure. Where should the author put this where it will remain publicly accessible for the useful lifetime of the paper? How long should it remain accessible, anyway? The scientific record is ostensibly permanent, but relying on authors to personally maintain cloud hosting accounts for data distribution will seldom provide more than a couple years' of data availability.

2\. Open data hosts that aim for permanent archival of scientific data do exist (e.g., the Open Science Framework), but their infrastructure is a poor match with reproducible practices. I haven't found an open data host that both accepts uploads via git + git annex or git + git LFS and has permissive repository size limits. Often the provided file upload tool can't even handle folders, requiring all files to be uploaded individually. Publishing open data usually requires reorganizing it to according to the data host's worldview or publishing a subset of the data, which breaks the existing computational analysis pipeline.

3\. Proprietary software was used in the analysis pipeline. The particular version of the software that was used is no longer sold. It's unclear how someone without the software license would reproduce the analysis.

Finally, there's the issue of computational literacy of scientists. In most cases, the "computational pipeline" is a grad student clicking through a GUI a couple hundred times, and occasionally copying the results into an MS Office document for publication. No version control. Generally, an interactive analysis session cannot be stored and reproduced later. How do we change this? Can we make version control (including of large binary files) user-friendly enough that non-programmers will use it? And make it easy to update Word / PowerPoint documents from the data analysis pipeline instead of relying on copy & paste?

If any of these pain points are in fact solved and my information is out of date, I would be thrilled to hear it.

<comment><USER5> 1 ans 2: I like IPFS for this, check it out

3: analysis that uses propriatory is marked appropriately as second class

> computational literacy of scientists

Welp...

</comment></comment></comment></comment></comment></comment><comment><USER7> The pressure to get positive results is just too high to be compatible with good science. When you have people's whole life and career on the line all the time (publish or perish) then people are going to do what they have to stay employed.

Like Gresham's law bad science drives out good, because it is much faster to do bad science than good. Those that try to maintain quality can't pump out as many papers as the bad and so lose on the grant treadmill.

Any solution that does not address the incentives is doomed to fail. Not fixing this problem will kill science.

<comment><USER8> Science is a field where failure should be celebrated; see Thomas Edison and the lightbulb.

<comment><USER9> ... Except that Edison succeeded.

What kind of failures should we celebrate? There are two kinds that I can see: a failure to produce a result because of a lack of skill or knowledge and a failure to produce a result because of a demonstrable or provable reason. The last one is actually a success because that kind of failure produces new knowledge. But we have a culture in far too many fields where this is seen as not publishable.

<comment><USER10> I have always wanted a 'Journal of Cautionary Tales', for studies that were well thought out, approached correctly, and just didn't work.

</comment></comment></comment></comment><comment><USER11> Beyond overt fraud, I feel like there's so much incentive for imprudent analysis.

Several times when I've worked on a project at work that involves data analysis, I get really impatient responses when I don't come to a firm conclusion. We're not even talking about highly charged subjects. I'm not aware if my stakeholders are biased toward one conclusion or another -- I think they're just very upset that my analysis contains uncertainty. They expect me to be able to massage any kind of data to derive clear and obvious facts.

Thankfully, I can refuse this without much consequence, but it's really opened my eyes to the potential pressures to corrupt the integrity of fact-finding in even mundane circumstances.

<comment><USER12> > They expect me to be able to massage any kind of data to derive clear and > obvious facts.

Not to be rude, but if that's in your job description, then I can understand where they are coming from. As someone in IT, it's always my fault that the computer is not working even if the ten year old device simply broke. That doesn't mean they are trying to get me to do an imprudent analysis.

Sure, there is a difference between operations and trying to find correct answers to questions (i.e. science), but they do have money to make, and if they're not (even slightly) asking you to bias your work, I can understand if they put a bit of pressure on getting the results clear.

Edit: As u/jeremyjh points out, I completely misread this post. If it's about uncertainty due to not enough data, then please completely disregard what I said. (I can't delete the post anymore because there exists a reply to it.)

<comment><USER13> That isn't how statistics works. You can't remove uncertainty from a given set of data.

<comment><USER12> Oh _that_ kind of uncertainty. I completely misread that post.

</comment></comment></comment></comment><comment><USER14> Reminds me of a project (not in medicine) I had a couple years ago where my group was in the data science role for another group. We helped out with experimental design, data acquisition, and analysis. After all of our work, when we crunched the numbers, we found that the hypothesis being tested didn't hold. The people funding the work were not at all happy, and went so far as to claim that "the plots must be wrong". We showed them everything we had done, and nobody could find any issues with the analysis. But, they were irritated because we didn't want to play the dishonest game of farting around with plots and numbers to make it look like the experiment worked. So, they angrily broke up with us. On the plus side, when I looked a year or two after, it looks like the grumpy people haven't made any progress and are either out of business or something.

<comment><USER15> At least you had the data... I've been a number of times in a situation where I was supposed to conduct extensive, thorough analyses of (expensively acquired) data that was evidently useless. I still had to go through the motions, knowing full well that if somehow the analysis yielded the "desired" results, any caveats would be ignored.

</comment></comment><comment><USER16> "Prove our new train signaling system is as secure as the old one. If you can, you'll get your degree, if you can't, you find someone else to sponsor your research/degree and we find someone else to do the proof." \-- Deutsche Bahn (government held national train company in Germany) in around 2001 to a friend of mine (they and him found "someone else")

<comment><USER17> In government this is so common as to have a name.

[https://en.m.wikipedia.org/wiki/Policy- based_evidence_making](https://en.m.wikipedia.org/wiki/Policy- based_evidence_making)

</comment></comment><comment><USER18> Even within the statistics community, there's a spectrum of quick-and-dirty vs fully rigorous. People with the ability and inclination to be fully rigorous often get treated as pedants and perfectionists (in the bad sense).

I often get that with business partners. "The data says <likely X, but with caveats/nuance/uncertainty/under certain assumptions we can't justify>" to which they respond "Can we just say X?" Or "can we get numbers on Y to support a presentation on Z?" when Y _seems_ to support Z, but actually you can't draw that connection, so it's misleading.

Stuff like this happens because people treat extra rigor as pedantry and are comfortable making supporting assumptions that aren't supported by data. The people making fraudulent requests aren't aware that they're fraudulent (usually). In my experience, they just think they're being practical.

<comment><USER19> I agree, and see this regularly.

"Picking battles" is one way to describe my counterpoint: caveat exactly as much as needed so that a proper decision can be made with the risks involved.

If you want to query your whole team for a joint lunch location but coworker X is out, it is still appropriate to say (assuming more than you and X on team) that you asked the team and you decided on lunch place Y. It's not rigorous (X is left out) but it's still accurate.

This is very different from, say, regulatory or securities reporting where ambiguity is not appropriate.

</comment></comment><comment><USER20> It's "nearly 1 in 4" in the text, but I guess they altered the statistics slightly to make their case look better.

</comment><comment><USER21> That happens, a lot. Not only statisticians.

My friend was half way over her 5 years vesting with a startup as when the CFO asked her to help then improve their numbers due a foreseen investment round. The idea was basically bump revenue basing it solely on the GMV masking returns and not calculating discounts and shipping. They also wanted to hide running costs by forcing vendors to agree on a 90+ days billing cycle, they also pushed the CTO on turning off parts of the system during weekends and holidays and forbid PTO until the deal was closed.

She refused doing the number masking and was asked to leave.

<comment><USER22> The story I heard related to an actuarial consultant who, after analyzing data and suggesting a defensible year-over-year rate hike to their insurance company client, was asked to change their analysis to support the rate hike the insurance company wanted.

And, of course, if the consultant didn't play ball, the insurance company could always consult with a different actuarial firm.

<comment><USER23> That's the name of the game for certain types of consulting--support the already-made decision. Mostly, but not exclusively, seen in management consulting to support staff reduction, outsourcing, and the like.

</comment></comment></comment><comment><USER24> The troubling part of this article (and _especially_ the comments) is how easy it is to be labeled "anti science" today when questioning certain studies. I'd hate to think our culture is causing us to lose the required academic rigor to make data-based decisions.

<comment><USER25> It's a tricky thing. There is a lot of high quality science. There is some lower quality science -- outright fraud even, unfortunately. And then there is science reporting, which is all over the map, and can easily be misinterpreted.

Humanity has built up an enormous amount of legitimate scientific knowledge and understanding. There has been much difficulty, confusion, and dead-ends along the way -- and it seems this continues to be the case. But it really is an incredible thing, how much we know at this point. It took me a long time and a lot of self-learning to develop a rich appreciation for this.

Of course, there is still much for us to figure out. And we should keep doing so, difficulties be damned (and hopefully mitigated over time).

</comment></comment><comment><USER26> Very interesting study. I'd love to see a follow up on 2 things:

1) Did the statisticians refuse or comply with the request?

2) When they refused, how did the requester react? Were the requesters actually malicious, or just bad at statistics? If everything was fine once the statisticians explained that removing "just an outlier" wasn't a valid option, then this report isn't quite as concerning, and is maybe just an indication that more researchers need to hire statisticians to help them out.

</comment><comment><USER27> I was working on the Cingular and ATT merger, my upper management asked me to fudge the numbers on roaming. I would gave the numbers in blue and orange, and blue (att) had better data handoffs on 3g. It made orange look so bad, (shitty network), they made me combine them, to which tanked ATT's network stats to the c-level.

One of many times, stats made someone look bad, and they made me change them.

Most the time, support metrics are altered to make it look like support contracts are hitting everything contractual.

This is daily business everywhere in tech. Mostly fudging to downright lying, just depends on the importance of the data if some mone is tied to it.

I don't approve of this shit, but lucky, I never been asked to commit fraud, like reporting sales that don't exist...

<comment><USER28> We've had this issue before with people scraping our websites, as our software gets better at detecting scraping, our number of page impressions went down.

Luckily our board was pretty understanding the reduction in infrastructure strain took a nice percentage out of operating costs.

</comment></comment><comment><USER29> Sensitivity / ablation studies plus datasets opensourcing should be made mandatory for every computational proposition and research asap, so that results can be checked by peers and by the community at large. With publish- or-perish pressure, democratisation of tools, rogue parties and hyperspecialisation, is it getting to the point almost nothing would sustain any stringent scrutiny? Science cannot afford opacity any more, more than that in these blurred times.

</comment><comment><USER30> _Of the less serious offenses, 55% of biostatisticians said that they received requests to underreport non-significant results._

Sigh.

</comment><comment><USER31> That's why one should big consulting companies like McKinsey, Bain, BCG, etc, who don't need to use hardcore statistics to prove whatever the board/CEO wanted in order to execute some policy.

<comment><USER32> You mean like good ole Arthur Andersen? Even the big boys will play dirty for the right price.

</comment></comment><comment><USER33> ...in the USA, in medical research...

<comment><USER34> That title is too long for HN. Sometimes I wish there was a limit longer than 80 characters so that no one has to resort to slightly more sensational titles.

<comment><USER35> I don't know, it's actually more upsetting if it's just medical research? Otherwise I could've told myself it happens less when it really matters.

</comment></comment></comment><comment><USER36> This didn't even cover the most frequent type of fraud which is to redo the experiment, or reanalyze the data until a sample of noise looks like a signal and then pretend you found something (as illustrated here: [https://www.xkcd.com/882/](https://www.xkcd.com/882/)).

I got out of academia and stopped trusting most university research because I observed too much of this culture of fraud.

</comment><comment><USER37> I wonder how prevalent this sort of thing is in other fields. I also wonder how prevalent fraud is generally, and the extent to which it is driving our decreasing faith in the academy. Also, how much (if any) of this fraud is motivated by political (instead of strictly personal) outcomes.

</comment><comment><USER38> What is the "American Council on Science and Health"? See here: [https://www.sourcewatch.org/index.php/American_Council_on_Sc...](https://www.sourcewatch.org/index.php/American_Council_on_Science_and_Health)

1 in 4 Statisticians weren't asked to commit scientific fraud. Within the article:

"Researchers often make "inappropriate requests" to statisticians. And by "inappropriate," the authors aren't referring to accidental requests for incorrect statistical analyses; instead, they're referring to requests for unscrupulous data manipulation or even fraud."

This isn't even remotely close to what the title of the article claims.

</comment><comment><USER30> I have serious health issues. I've gradually gotten healthier, in part because I am skeptical of a lot of studies. I know how borked so many of them are, not a thing most people want to hear me assert. I get a metric fuck ton of flack over my skepticism.

I kind of have a mental box where I squirrel away little tidbits that meet two criteria: 1. They seem to come from rigorous studies and 2. They also fit with my general understanding of how life works.

Over time, I mentally group things -- a la A and B seem related -- without assuming that I know how they relate. I seem to have an inordinately high tolerance for ambiguity.

Most people seem to need An Answer even if it's wrong. They have two categories -- black and white -- and when presented with purple or pink or blue, they force fit it to one of their existing categories and don't confuse them with the facts.

I wasn't trying to prove anything to anyone. I was just trying to deal with my life. But having gotten substantially healthier, I now wonder how to talk about such. It seems a wasted opportunity for the world for me to not share, but the world has treated me pretty horribly and done all in its power to tell me to STFU, I'm just crazy and spouting nonsense.

So I sometimes think I should write what I understand to be true and then carefully back it up with citations to try to support it. Then I read articles like this, throw my hands in the air and go "Why bother?"

The way I have been treated seems particularly unfair when you learn how much "real scientists" cook the books. Like why? It feels like pure prejudice when I read things like this.

</comment><comment><USER39> 1 in 4? How do we know that number is not made up?

In fact there is a 25% chance that the person involved in working that out has been asked to commit fraud at some point.

<comment><USER40> >In fact there is a 25% chance that the person involved in working that out has been asked to commit fraud at some point.

But there's only a 10% chance of that.

</comment></comment><comment><USER41> Another statistic just in: 3/4 statisticians lie about not committing scientific fraud

</comment><comment><USER42> To be honest I'm surprised it's not higher. Relevant discussion: [https://news.ycombinator.com/item?id=17789308](https://news.ycombinator.com/item?id=17789308)

The bottom line is that there's not much incentive for doing the boring statistical validations (I can tell you that _no one_ likes doing statistics apart from statisticians, and not even all of them) and verifying that everything is reproductible, and a _huge_ incentive for, let's say, 'arrange' a thing or two so that the paper looks better. So many people just kinda do it, and it slips through the cracks because:

-In many fields, the reviewers are not statisticians themselves

-No one really bothers to download the data, setup the environment and libraries, and reproduce the exact steps taken to obtain the very same figures in a paper. Which is understandable given how doing all of that can be such a chore. Anyway, most of the time the exact steps aren't even described. No, jupyter notebooks aren't a solution either (it would take too long to explain why and the post is already long).

-In many cases, the results turn out to be true anyway so people don't notice they were initially put forward with fraudulent validations

-When they turn out to be false, people just shrug and move on with what's actually true. Bogus results often fail to stand the test of time and get forgotten quickly despite initial hype. No one bothers to say: "Hey, that paper from 8 years ago is bullshit and their authors are hacks!" because nobody cares.

-There's a huge psychological barrier to actually call out one of your peers and affirm that they're an impostor and their work is bogus. Especially when said impostor happens to be a big name in the field, that part of you still doesn't believe they would commit fraud, not to mention the social repercussions and backlash of doing such an accusation. We scientists aren't a very confrontational bunch.

So most of the time it kinda works and we're all bumbling along hoping to find some modicum of truth at the end of the road with minimal harm done. But of course sometimes you get these guys who kick off their entire career on a high profile, much hyped fabricated result (Woo-Suk) or even a _series_ of bogus papers (Sato), and _that_ may lead to long-term harm. The good news is that hyped papers or very prominent figures quickly attract scrutiny from their peers, and sooner or later reality catches up with them as labs around the world fail to replicate their 'breakthroughs'.

All in all, I'd say we're doing fine. We're just not the ethereal source of truth that some people hold us to be, the very same people who, after claming that 'God is dead', are very quick to replace Him with His sillicon-based equivalent around which we would act like priests, except with lab coats in lieu of clerical garments.

<comment><USER43> Ok, so why _aren 't_ jupyter notebooks a solution?! I was gonna suggest that also in response to another comment above suggesting figures should be published 'alongside their relevant code'.

<comment><USER42> They are so _unwieldy_. JSON formatting means the ipynb format is a huge mess to handle by simple means, so you _have_ to open a browser window to do anything. Loading, editing, formatting, saving is just... ugh. You can't do proper diffs (though some tools are trying to alleviate that). Loading, stopping and restarting kernels is excruciatingly slow. Because cell execution isn't necessarily done in order it can become very, very easy to lose yourself and feel like you're back in the 70's experiencing GOTOs and the joys of spaghetti code. And that's from _my_ point of view with a technical background. Imagine explaining notebooks to a biologist to whom writing any line of code is still something new and daunting. Imagine their reaction when they happen to mess up one cell in their own execution order and alter all the subsequent workflow, try to grok some of the magic %commands, or get some cryptic error message due to some config file not having the proper rights because they installed a library with 'pip --user' and not 'sudo pip'. It's not realistic to think that an entire community of people who aren't technically minded, many of which actually loathe or fear anything looking like code, is going to adopt a tool that even technically minded people struggle to use they way it's intended.

On top of that, many steps necessary to reproduce a pipeline typically need to load enormous datasets. Terabytes of simulated protein structures, hundreds of gigabytes of sequencing reads, phylogenetic trees, alignment files, what have you. Once you somehow acquire that dataset, you need the appropriate tools, many of which need to be specifically compiled for your platform, then run them onto a powerful machine if you don't want the pipeline to take months to complete, etc. (And that's if you didn't use any proprietary software or any GUI based application with no command line interface.) You can't exactly load all of that into Github+Mybinder and call it a day. You can't ask a community of people that doesn't like coding to learn about Docker containers either.

Nevertheless, we (at our lab) _do_ use notebooks when we can because we know they're fashionable. We can only present parts of our results, though, due to the aforementioned constraints, but it's still pretty looking and people like them, so we write short demos using them.

</comment></comment></comment><comment><USER44> "Can you make the error bars a bit smaller please?".... "um no, I'm affraid they are as I've calculated "

</comment><comment><USER45> Is this result statistically significant?

</comment><comment><USER46> What's odd to me is that some people seem to, even in the face of this, think it's fair to criticize liberal arts journals for sometimes publishing bad papers. People are just people, in every field.

</comment><comment><USER47> "1 in 4 Statisticians Say They Were Asked to Commit Scientific Fraud"

It would be hilarious if this report were debunked on the grounds that they inflated the number of statisticians surveyed

</comment><comment><USER48> ... and 1 in 4 not prepared to answer truthfully, 1 in 4 not having worked in the field long enough, and 1 in 4 not competent enough to know the difference.

/haha, only serious.

</comment><comment><USER49> How do I know this study isn't part of the 1/4?

</comment><comment><USER50> Most statisticians work for people who are trying to sell something. Statistics is not always the best way of selling something.

</comment><comment><USER51> I think that this should prompt scrutiny in fields that are highly dependent upon statistical analysis, such as climate change. However, this very suggestion is intolerable to many and in a way equivalent to blasphemy against the church science. Scrutiny will not be tolerated by academe with regard to ideas held as sacrosanct. I find this culture in academia pretty disgusting.

</comment><comment><USER52> I wonder what the stats are in China.

<comment><AUTHOR> 80% of China's clinical trial data are fraudulent, investigation finds

[https://www.bmj.com/content/355/bmj.i5396](https://www.bmj.com/content/355/bmj.i5396)

</comment></comment><comment><USER53> '25%' would have been a more appealing headline ... :D

</comment><comment><USER54> Is this fraudulent stats?

</comment><comment><USER55> The other 3 in 4 just lie about it.

<comment> munk-a Alternatively just one statistician lied, the one conducting this study.

</comment></comment><comment><USER56> I can't help but wonder what the rate is that didn't admit to being asked to commit scientific fraud because they did.

</comment><comment><USER57> And the other 3 lied</comment>


Ask HN: Is it common to give up rights to inspection in stock options? - <AUTHOR>

As a stockholder of a private company, you have certain rights to see the financial health of the company. Is it common in startups to make sure the employees waive their rights once they exercise their options? <comment><USER2> I would use caution when valuing any contract that contained significant provisions contrary to transparency.</comment>


GCC Explorer - <AUTHOR> http://xania.org/201205/gcc-explorer Interactive tool to see what GCC is up to. <comment><USER2> I remember one time during compilers class I tried to figure out the largest prime number in the denominator that GCC would replace with multiplication and bit shifts. I got up to a number in five digits and stopped counting. Just goes to show the amount of work that goes into compilers and the importance of speed for some program writers.

<comment><USER3> It likely has code that handles the generic case. See [http://ridiculousfish.com/blog/posts/labor-of-division- episo...](http://ridiculousfish.com/blog/posts/labor-of-division- episode-i.html)

</comment></comment><comment><USER4> Handy tool. It would be nice if it could show side-by-side diffs of the output coming from different GCC versions.

<comment><USER5> Or GCC vs LLVM, for that matter. If such link is permanent it could be linked from GCC or LLVM missed optimization bugs.

</comment></comment><comment><USER6> Yet more bonus points if you implement a way to correlate (roughly) the assembly code with the corresponding C++ code. Like, when you select lines in the C++ code, it highlights the corresponding assembly instructions. Not trivial, especially at high optimization levels.

<comment><USER7> Actually, it's not that difficult. Get GCC to generate DWARF [debugging] information, and then use that to provide the back-references. (Reference: DWARF 3.0 spec, available at <http://dwarfstd.org/doc/Dwarf3.pdf>. See page 92.) This is how GDB does the same.

The output is a little weird (I've had it execute line 56 before line 55, for example) and you do have to be careful, as not all lines translate directly. But it would be at least be a nice starting point. There are even some nice libraries out there that parse this information!

EDIT: Hmm, I just noticed that the source is on Github. If I get some time this weekend I'll try adding support for this.

</comment></comment><comment><USER8> As a cross-platform developer, I would love to see clang and MSVC as compiler options so that I'm able to compare the various outputs when trying to track down undefined/implementation defined behaviors.

As it stands though, this is a fantastic tool! It's right up there with ideone for me.

</comment><comment><USER9> What's happening in the signed case? Signed division and modulo have been redefined in newer C/C++ standards. He talks about writing C but the syntax looks like C++ 11? at least i don't think he means the C "auto" here.

</comment><comment><USER10> This is great. I often compile with gcc -S but then get swamped by the massive amount of included stdlib stuff and name manging, so it's hard to find my code.

</comment><comment><USER11> Great, but I was hoping for access to libc so I could see if the new string/popcnt/rand instructions are being used with -march=corei7

</comment><comment><USER12> Please use vanilla GCC as ubuntu (and other distros) stupidly edit system wide spec files to add things like fstack-protector.

</comment><comment><USER3> <http://llvm.org/demo/index.cgi> is a similar tool for clang.

</comment><comment><USER13> For bonus points, get a compiler building with Emscripten and compile the code locally in the browser.

</comment><comment><USER14> Very cool tool, I can easily see this becoming a standard tool for learning assembly!

</comment><comment><USER15> Nice. Please add more GCC versions (3.x and 4.0 through 4.3?) and the most recent clang/clang++ releases! Direct pastebin-style links would also be great.

</comment><comment><USER6> Very cool. Thanks!</comment>