Ask HN: Who is your go-to web hosting company? - <AUTHOR> I'm looking for Web hosting (either shared or VPS) for a personal owncloud setup. All the reviews I've seen online say different things or are obviously sponsored by a particular company so don't give a proper representation of the quality of the services.<p>Any thoughts or recommendations? What d'yall think of godaddy, hostgator and bluehost compared to AWS for value for money? <comment><USER2> For shared, I've never used a service that could measure itself with NearlyFreeSpeech. Of course, I don't host anything huge or very important, but I suspect that even if I did I would still go with NFS. Their pay-for-what- you-use pricing, realm options, and support (for an extra but fair fee) are simply phenomenal. I have nothing at all to complain about with NFS, whereas I hold plenty of grudges against godaddy and hostgator.

</comment><comment><USER3> The best shared/pseudo-vps hosting that EXISTS in the world is: [https://www.webfaction.com/?aid=9281](https://www.webfaction.com/?aid=9281) (referral link)

Source: Have hosted from php/mysql stuff to 350+ requests/second uwsgi-python- flask-postgresql webapp on a 10$/month shared-space.

I challenge every commenter on this thread to find a better host than webfaction!!

<comment><USER4> does it come with cpanel license?

<comment><USER3> they have their own custom panel (better)

</comment></comment></comment><comment><USER5> I've been using Linode for about 18 months, never had any problems. There's been two hardware upgrades during that time. I'm on the $20/month plan.. I use it to host several domains. I've also used AWS EC2 but found it was probably overkill for my own personal needs, as I don't have the need to spin up and down instances or scale or anything like that.

<comment><USER6> i would have to say dont use linode - two companies were hacked in the last 2 years and the intruders got in by rooting all of [one section? of] linodes datacenter.... all in all i would say if they didnt learn their lesson the first time its not advisable to use them. probably better off with hetzner but idk. there should definitely be a breach tracker site.

</comment></comment><comment><USER7> I just use DigitalOcean and don't think about it much anymore. They're reliable, flexible and cheap.

Also, at the risk of triggering your "sponsored by a company" radar, here's my promo code, which will get you $10 off:

[https://www.digitalocean.com/?refcode=9fad2cc71fbc](https://www.digitalocean.com/?refcode=9fad2cc71fbc)

<comment><USER8> And if you are a student you can get 100$ credit from the Github student pack.

[https://education.github.com/pack](https://education.github.com/pack)

</comment></comment><comment><USER9> I've used DigitalOcean for about a year and been happy with it. Lately I've been looking at Atlantic.Net and Vultr.com as well because they offer either cheaper or have slightly different priorities in their pricing levels.

For example, Vultr offers more memory and less hard drive space in their $5/month level.

These days I'm trying to get to a point where I have deployment software in place that would allow me to use Ubuntu LTS anywhere in a fairly short period of time. It's not that I'm actively looking to leave DigitalOcean, it's that I would like to be able to recommend something like Atlantic.Net's GO plan to somebody to put up a quick static site for $12 a year and know that it actually works because I've tried it myself.

</comment><comment><USER10> I don't run owncloud, but if I did, I would want it to be, you know, my own cloud!

I have an intel NUC [1] in my house, in the DMZ of my router, and running a script to change it's ip address at my DNS provider when my house ip address changes

I've got a lot of stuff running on that, which can be accessed by me from anywhere, but it lives in my house, and so enjoys more legal protections and physical security than a VPS.

[1] [http://www.amazon.com/Intel-Next-Computing-Black- BOXDCCP847D...](http://www.amazon.com/Intel-Next-Computing-Black- BOXDCCP847DYE/dp/B00B7I8HZ4)

</comment><comment><USER11> I've had a "reseller" account with httpme.com for at least 10 years. Signed up with them back when the original owner was very active on WebHostingTalk (if I remember correctly, user "Aussie Bob"). Prior to them I bounced around at least 5 different shared providers and had my own servers colocated for a while. I've had minor problems that were all quickly resolved (one of my friend's sites I was hosting had a script that began to eat up too many resources, and they killed it).

The interface is standard WHM/cPanel. No SSH.

<comment><USER12> He runs AussieHost.com now.

</comment></comment><comment><USER13> Well if you don't trust the big review sites you at least recognize bullshit when you see it. That said, my startup is trying to give an actual unbiased look at what people think about web hosting companies. It works by aggregating and analyzing twitter sentiment about all those brands you mentioned (it only works on bigger guys who get enough volume of discussion). Take a look [http://www.reviewsignal.com/webhosting/compare](http://www.reviewsignal.com/webhosting/compare)

HostGator/BlueHost is a near identical product from the same company (EIG) and the quality is quite low. GoDaddy doesn't have that much better of a reputation but it's really hard to tell sometimes between quality of their service and people who don't like them because of what they've done in the past (sopa, elephant killing, advertising, etc). Doesn't help that they are huge either.

AWS is expensive if you're comparing it to shared hosting. They are nowhere near the same product. AWS is great if you need AWS features and are a technical person who can handle all that (or afford paying a lot for support/someone who can). The other companies are providing management/support. If you goal is installing owncloud my first question would be are you going to install/manage it or not? That generally cuts down which companies might make sense.

<comment><AUTHOR> I plan on installing owncloud myself

</comment></comment><comment><USER14> I was sick of mainstream companies after endless problems and issues and I found vultr on a web hosting forum: [http://www.vultr.com](http://www.vultr.com)

And it's awesome. A good hosting provider is one you forget about. When you do things right, people won't be sure you've done anything at all. So far, I didn't have any problems, didn't need any support and nothing went wrong. And the reviews, when you finally find them, are usually great.

I remember comparing performance when I made the decision. And basically, all the major players suck big time. GoDaddy, HostGator and BlueHost are shit. AWS works for large scale stuff, for everything else it's shit. If you want quality, go with a small provider. The mom and pop shops of the tech world.

Bigger companies use cost reduction to increase profits and squeeze out every last penny. Small companies and startups are hungry for clients so they focus on great services and products.

<comment><USER15> Even though it's obvious, it's considered good etiquette to disclose when you're using a referral link.

Also, you're on a new account. There's also another reply vouching for Vultr on a new account in this thread, albeit without a referral link.

Something smells.

If I were OP I wouldn't even consider vultr.

> > [https://news.ycombinator.com/item?id=9118593](https://news.ycombinator.com/item?id=9118593)

> > [https://news.ycombinator.com/item?id=9118652](https://news.ycombinator.com/item?id=9118652)

<comment><USER14> Yes, you got me. I created my account 10 days ago and commented on various threads just to post a referral link to a web hosting provider.

</comment></comment></comment><comment><USER16> DigitalOcean is awesome for both side projects and complex ones, let us not forget OpenShift (redhat), heroku (it is free :D) and linode that has support for Tor. But DO is awesome for starters. For registrar I suggest NameCheap, I have used Godaddy and I am sad I did. AWS is beautifully awesome for high scalable projects.

</comment><comment><USER17> I've had good experiences with HostGator since moving there from a smaller company that basically stopped responding to support requests. They seem to update software regularly, and it's cost effective.

I've heard bad things about GoDaddy from various sources, and that's why I opted not to go with them. Not that I'm putting up any content that is infringing or even remotely in a grey area, but I've heard their policy as a company is to take down your site at the smallest complaint and then they only look into the complaint after you ask why your site is down.

All that said, I'm still on the lookout for a hosting stack (hosting provider, OS, and application) that allows me to rapidly prototype new ideas (Drupal looks awful without a lot of customization and I'm not clued up enough for Docker).

</comment><comment><USER18> There's no reason to deal with shared hosting providers anymore. VPSes are so incredibly cheap that it doesn't make any sense.

In order of price (low to high): DigitalOcean, Linode, AWS.

Take your pick. They're all good.

<comment><USER19> But then I'd have to manage and maintain a server. What if I don't want to be a server administrator?

<comment><USER18> It's not that time consuming. Digital Ocean will set up LAMP for you (or even install Drupal or Wordpress if that's your thing). Secure SSH, set up your app, set up monitoring and leave it alone.

<comment><USER20> I want to move from shared hosting. Looking at DO. Have scores of sites to migrate. I use NameCheap for domain reg and point to host. Can I host multiple domains at DO? How does it work for multiple sites?

Would setup be something like:

1\. Get DO account 2\. Install LAMP 3\. Setup multiple sub domains in DO? 4\. Create site directories in DO and copy site files into them? 5\. Point domains to sub domains?

</comment></comment></comment></comment><comment><USER21> I get my domains through DreamHost. Their "unlimited" hosting is not, and it's spotty, but if you register domains through them they give free domain- registry privacy proxying. I have bit of a hangup about my personal info being freely available through lookup.

I'm up for re-evaluating my hosting soon, but right now I'm through Lithium Hosting. Haven't done much app-related stuff, but they've been fine for regular low-volume hosting stuff, and I pay $10/month (legacy plan). Web- faction sounds nice.

</comment><comment><USER22> If all you are concerned with is cost go with something like digitalocean. If you intend to run your business on it I recommend aws.

</comment><comment><USER23> Quite a few of my sites are hosted with EUKHost, in terms of support they've always been near perfect and I've never had to wait for help. Servers are reliable too, can't think of any major downtime in the last 5 or so years.

Tried AWS and preferred Digital Ocean instead, the $5 servers are quick and it's easy to spin one up to use for various needs.

</comment><comment><USER24> I use Linode, Rackspace and AWS pretty heavily. Overall I recommend Linode the highest. Good prices, decent server-monitoring built in, and I love how much bandwidth they give you for each server.

AWS kills me on the bandwidth charges. The cost-calculators are practically worthless unless you're really good at estimating your potential bandwidth usage.

</comment><comment><USER25> In my experience with OwnCloud it is a rather expensive thing to run. Running it on a $10 VPS regardless of the host, you're likely going to have a bad time.

With DigitalOcean, the SSD will definitely help, but I'd look at the $20 plan just for the RAM. However expect to have to tweak OwnCloud's environment a little to get the best out of it.

</comment><comment><USER26> Disclaimer: I work for Rackspace.

We have a once click Orchestration install of OwnCloud that will install and fine tune your server according to their best practices and specifications,. We also have a developer+ program where you can get $50/mo for 12 months in credit to try us out (developer.rackspace.com/signup)

</comment><comment><USER27> I use DigitalOcean. I know people have had issues with them but so far its been nothing but a good experience. I also prefer being close the metal as far as a server goes so it fit my needs perfectly.

</comment><comment><USER28> I like DigitalOcean for my hosting, and NameCheap as a registrar. DigitalOcean has a ton of tutorials and other good resources to learn how to use its platform, and NameCheap is, well, cheap.

</comment><comment><USER29> I've used RamNode and had a great experience using them.

<comment><USER30> Same here.

</comment></comment><comment><USER31> Check www.vultr.com it's cheep and you can spin your own ISO for OS install. Best I found so far.

</comment><comment><AUTHOR> ITT: Op asks for honest opinions about webhosts because other sites just give me a bunch of referral links.

Gets a bunch of referral links.

But thanks for those who've offered good advice, I think I might go for digitalocean.

<comment><USER3> I think I explained my choice ?

Just try it, you have 60days money back guarantee. But it's not a complete vps.

</comment></comment><comment><USER32> DigitalOcean

</comment><comment><USER33> Hetzner, since a long time.

</comment><comment><USER34> linode (a VPS web hosting provider).

N.B: I do not trust any shared hosting company.</comment>


Ask YC: Entrepreneurial/Startup Reading List? - <AUTHOR>

My friend and I are looking to start our own small web business in the coming months, and I am looking to expand my entrepreneurial and business knowledge base.  I am looking for books that deal with starting a business as well as managing software teams.<p>So far on my list, I have Peopleware by Tom DeMarco, Groundswell by Charlene Li and Josh Bernoff, and Raving Fans: A Revolutionary Approach to Customer Service by Ken Blanchard and Sheldon Bowles.  Does the YC community have any other suggestions to add to my list? Thank you! <comment><USER2> "Starting Something" - The story of Neoforma, from startup to IPO, written by one of the two founders. Has especially great insights on the people aspect of the journey (VC's, employees, family, partners, angel investors...etc).

'The Search' - talks about Google and search companies in general, how they started, how they developed their business model, and the evolution of the industry in general. Especially fascinating is Bill Gross's story, and how he started ToGo.com (later became Overture) and invented a lot of the things that people commonly only associate with Google. It was also the first time I read the in-depth story about the founding of Google. inspiring...

'Founders at Work' is terrific also. Can't wait for the second part.

I'll throw in an extra recommendation. Not exactly a book, but a podcast actually. It's called "Venture Voice". It a series of interviews with entrepreneurs (Ev Williams of Twitter/Blogger/Odeo, Jay Adelson of Revision 3 and CEO of Digg, Calicanis of Weblogs Inc, Kawasaki, Mena Trott of Six Apart, to name a few). You can think of it as an audio version of 'Founders at Work'. Those two are a real must especially for web entrepreneurs.

</comment><comment><USER3> Check this list out: [http://www.bookhuddle.com/list/491/Recommended- Technical-and...](http://www.bookhuddle.com/list/491/Recommended-Technical- and-Entrepreneurship-Reading)

This list is made up from books I've read, books friends and coworkers have read, books on Joel Spolsky's Management Training Program, the Personal MBA reading list, and books recommmened by people on Hacker News.

</comment><comment><USER4> I've been reading 'Founders at Work' by YCs own Jessica Livingston. Very good book for insight into the beginnings of startups and the startup process in general.

Definitely a must read!

<comment><AUTHOR> Sounds great, I'll check it out, thank you!

</comment></comment><comment><USER5> The Wisdom of Crowds by James Surowiecki Takes a whole new approach to management, and solutions.

How to Win Friends & Influence People by Dale Carnegie

</comment><comment><USER6> "Crossing the Chasm" Geoffrey Moore "Entrepreneurship and Innovation" Peter Drucker "Cities and the Wealth of Nations" Jane Jacobs

Also, check out "The Programmer's Stone" blog for great ideas on software team building and management.

</comment><comment><USER7> Mythical Man-Month by Fred Brooks teaches great lessons about managing programmers and programming as a team.

On a less technical note, Growing a Business by Paul Hawken is great book about letting a company grow naturally.

</comment><comment><USER8> <http://ycombinator.com/lib.html>

</comment><comment><USER9> "The Monk and the Riddle" by Randy Komisar</comment>


Frustrating Client - <AUTHOR> https://joelx.com/frustrating-client/13111/ <comment><USER2> Call a lawyer.

If you can adequately show the work you have done and if you tracked hours (even better), then you should have no problem getting the company to pay you. If they're making as much money as you say they are, they'll want to avoid any bad publicity at all costs.

Compile a list of the following:

\- All forms of communication with the client; emails, Slack messages and phone call logs

\- Compile a list of commits made to your source control repository, including whom did what

\- A list of any created design assets or other materials

\- Time tracking data (if any)

\- If they want to play hardball, see if this company has a history of not paying people for their work. If they did it to you, there is a chance they've done it to others or jibbed people in other facets of their business

Then hand the list over to a lawyer. I've done this before, usually a scary letter from your lawyer which will cost you a couple hundred dollars is all that is usually need to scare a company into paying.

</comment><comment><USER3> Why would you brag here with so contentless piece of blog post? Apart from getting the most obvious advice, as hoodoof gave you.

The post says literally nothing of any use to anybody. Some guy (team for hire) got screwed over by some unnamed company, that's the whole article. Wow.

</comment><comment><USER4> If you have solid written documentation that confirms without a doubt what you are owed, then just give it to a lawyer and forget it.</comment>


Korean team designs graphene-based transparent flexible battery - <AUTHOR> https://www.graphene-info.com/korean-team-designs-graphene-based-transparent-flexible-battery <comment><USER2> Graphene-Info.com is a shill page offering networking and promotion services to the nascent "graphene industry". The Korean team's work, while interesting on the basis of pure science, has no indication of ever being practical or marketable. They haven't even bothered applying for a patent.

</comment><comment><USER3> Some perspective: over the past 5-6 years, I have seen a raft of companies pop up promising better batteries. Their business plans are all fairly similar, but none really succeed (that I have seen); most are treading water, and few have produced returns. It seems breaking into the battery market is very difficult. Glass batteries, AC batteries, cheap graphene from biomass, silicon anodes, none have really succeeded, _yet_. Maybe a move to intermittent energy sources (e.g. wind, solar, hydro) which require batteries and transmission will drive adoption of new technology. If I were to guess, it will be a shortage of lithium, nickel, cobalt, etc.

<comment><AUTHOR> Thanks for sharing. Seems you are knowledgeable about the industry. Any views on companies, or technologies, that you consider promising?

</comment></comment><comment><USER4> Is there any source for a better quality larger photo? Would love to actually see how this looks like...

<comment><USER5> Try this [http://www.hankookilbo.com/News/Read/201904231788312607](http://www.hankookilbo.com/News/Read/201904231788312607)

<comment><USER6> thank you! this is pretty amazing to see. i suspect lithium ionization will prevail a while longer but it's good to see some radical pragmatic application of any potential alternatives!

</comment></comment></comment><comment><USER7> So the implication is transparent phones?

<comment><USER8> phones don't really have a design problem that can be solved with transparency other than a gimmick.

However being able to integrate the battery and the display into the same structural element opens design options for other applications e.g. smart glasses.

It also can be used for other applications such as say UV/IR spectrum solar panels with the battery integrated into them which could be made into a drop in window or structural glass for architecture or for automotive applications for example.

<comment><USER9> Almost none of the recent changes in phones have been real problems. Gimmicks sell phones.</comment></comment></comment>


Tess: the mental health chatbot that thinks like a therapist - <AUTHOR> https://www.theguardian.com/society/2018/apr/25/meet-tess-the-mental-health-chatbot-that-thinks-like-a-therapist <comment><USER2> It seems like this is intended to be rolled out by healthcare providers. It's not a free service. They recommend using Signal or Tor for privacy, rather than Facebook Messenger, although their demo runs on Facebook Messenger.

[http://x2ai.com/#howitworks](http://x2ai.com/#howitworks)

[http://x2ai.com/privacy-policy/](http://x2ai.com/privacy-policy/)

</comment><comment><USER3> See [http://swordartonline.wikia.com/wiki/Yui](http://swordartonline.wikia.com/wiki/Yui)</comment>


Interested in launching DeliveryRunners.com in your area? - <AUTHOR>

I have the domain deliveryrunners.com. any one interested in running a delivery business in your area? <comment><USER2> Are you going to be running a delivery service in your area?

<comment><AUTHOR> A friend of mine that owns a pizza shop is going to work on part of the city. I'm probably going to focus on the development and the app.

</comment></comment><comment><AUTHOR> I'm going create the website and a phone app for it.</comment>


Psd.js layer visualization demo - <AUTHOR> http://meltingice.github.com/psd.js/layers.html

<comment><USER2> This is really cool but I can't get the PNG node module to install on my server or on my laptop</comment>


Burning it down on the way out – how can non-tech founders protect themselves? - <AUTHOR> Hi all,
As the title suggest, I'm writing in hopes of assistance with solving a problem that many of us "non-technical" founders face -- a technical person has all the leverage to make a lot of crummy things happen to a business overnight.
People can put trust in their team members, but let's face it -- shit happens, and when it does, it would be nice to know that the owners are going to stay the owners and the IP/code stays with the company.
What should we do that doesn't seem accusatory and scream "the sky is falling"?
SO... Non-Technical Founders: What are some techniques that you have used / seen to help with this?
Technical Founders/Tech people:
What are some things that you've thought to yourself "These idiots don't realize I can just X,Y,Z" or "If they were smart, they would/wouldn't have done X"
What about legal things? Different tools/software? Procedures?
I couldn't find any information on this topic, so anything would be useful. <comment><USER2> > shit happens, and when it does, it would be nice to know that the owners are > going to stay the owners and the IP/code stays with the company

This is why we have contracts and lawyers. You shouldn't start up a company without first having paperwork that clearly defines who owns the company, its intellectual property and other assets. For example, who owns what percentage of the shares, what is the corporate governance structure (who are the CEO, officers and board), etc.

This applies even to a business partnership of two people.

</comment><comment><USER3> As other comments have pointed out, there's not a lot you can do if the technical person wants to bring the house down.

But there are other cases you should cover against, like the techie getting run over by a bus. The first thing that comes to mind is to make sure you have an admin account and/or a recovery procedure (a master password printed on a piece of paper in a safe, access to the account's email address to reset the password, etc.) for _everything_.

Everything in this case is at the very least: AWS account (including the root account to manage the bills), e-mail provider, Gsuite admin rights, whoever you brought your domain names from, Dropbox, bank/PayPal/Stripe, GitHub/BitBucket, your telco, the admin accounts on your corporate PCs, and just about all the services you use on a daily basis. The email accounts and DNS are IMO the most critical; you can retrieve a lot of stuff if you control the e-mail or the domain name. BUT this assumes the accounts were set up with a corporate email, which may not be the case if you're a small company - probably a personal e-mail was used to register the first services, which are also the most crucial. And if the techie knows what she's doing, she will have enabled MFA, so e-mail alone probably won't cut it.

Just ask yourself: if the tech person dropped off the face of the earth, what would you be locked out of?

</comment><comment><USER4> A model you might look at if you are a majority or 100% shareholder is to have a separate AWS account that you control 100%. You can have write-only S3 buckets that another AWS account can post to. You should be able to do this with other cloud systems as well. You can then use replication rules to copy data from your production AWS account to this backup one.

Although I am the technical co-founder in this situation, I am protecting myself against mistakes made by senior technical staff. It would also protect to a measure against malicious action.

I certainly hope I never have a dispute with my business partner. If I did, I would try to resolve it amicably and if necessary resort to the courts. At no point would I even dream of cutting off my nose to spite my face by damaging production systems for example or deleting code. That just gives the other side ammunition at best and at worst the consequences can be very serious [0]

As others have noted, there is little you can do to stop a trusted director- level partner or employee if a switch flips and they suddenly become evil. Setting up a write-only backup system is about the best you can do... but if you're non-technical who are you going to trust to do this?

[0] [https://www.theregister.co.uk/2008/12/29/terry_childs_trial/](https://www.theregister.co.uk/2008/12/29/terry_childs_trial/)

</comment><comment><USER5> There's not a lot one can do. It's sort of like preventing your bodyguards from assassinating you. They know all your procedures, it's a matter of trust. Even as a tech co-founder, it's difficult to control as it's easier to hide back doors and time bombs as the code gets more complex.

There's a few measures you can take: 1\. Daily backups of code repositories, on a computer you trust. 2\. Periodic backups of the production server. 3\. Check your contract agreements on IP with a proper lawyer. IP is very sensitive as a lot of techies also have side project.

But in general, trust your techies. People are reciprocal. If you start putting in more safeguards, they start putting in their own. Don't isolate them from meetings/clients, don't put too many levels of security, don't hire people to "check up" on that guy, trust the intern.

Most engineers just want to build stuff. If you let them build stuff they will be very happy and loyal.

</comment><comment><USER6> How about just don't screw over your technical cofounder? Amazing how non technical founders can't resist trying to screw over the tech guys.

</comment><comment><USER7> You have your principles backwards. Non-technical founders should not stay the owners of a technology company if shit happens. The ownership should be usurped by technically proficient people. It is called 'nerd's revenge'.</comment>


Perkeep: Personal Storage System - Permanently Keep Your Stuff, For Life - <AUTHOR> https://perkeep.org <comment><USER2> I run Perkeep and I am amused by the infrequent emails I get from Google going "This app has access to your google accounts! We don't think it's safe!!! PANIC STATIONS!!"</comment>


Should I pursue someone who stole my idea? - <AUTHOR> http://answers.onstartups.com/q/24541/5722

<comment><USER2> It's not right, but it happens all the time. IMHO, if you can get past the rage, sell him the bank of hours. If he really wants to talk with you that badly, ask for an agreement with a piece of the action (thought I doubt he'll sign it).

Listen, ideas truly are a dime a dozen. Implementation is hard. Facebook, myspace, friendster...all the same idea. If this guy can do it, good for him.

Take the money and run. If you can't do that, bring the same idea to a competitor and see if they bite and will make you a deal.

</comment><comment><USER3> The OP gave away too much up front, and it's hard to fault the other guy (from a business perspective) for just implementing what he was given for free. He was effectively consulting with no contract or compensation.

That said, it's a fairly bad-faith move to take the idea and implement it without paying for it, and if the OP can afford it, should just cut ties with the company and move on.

<comment><AUTHOR> What do you think of the OP's plan for developing his own implementation of the idea?

<comment><USER4> It'll forever be tainted because the other guy launched the product first. The OP will look like a copycat at best or just like he feels his former business partner is in the worst case.

We saw how it worked out with the Winklevi, the other dude actually implemented the idea so he deserves the spoils.</comment></comment></comment>


Canada Is Allowing Dying People to Do Psychedelic Mushrooms - <AUTHOR> https://www.vice.com/en_in/article/y3zxqm/canada-is-allowing-dying-people-to-do-psychedelic-psilocybin-mushrooms <comment><USER2> Not a naive question: why would Canada (state) need to authorize patients to consume anything?

I get that substances are regulated but we already can get legal access to substances that we shouldn't consume too much or at all (ie. spirits, tobacco, sugar, salt, bleach...) so, why would dying patients (or anyone, really) not be allowed to consume regulated mushrooms?

I get that the real problem noted in this article is not on the right of the patient to consume, but the access to it - anyone distributing mushrooms to those patients need legal coverage to do so.

Still, I find it really hard to understand why (most, yet, not all) societies are so obsessed in regulating the access and consumptions of certains things while vast majority of unregulated things can and are as deadly as those regulated.

<comment><USER3> > Still, I find it really hard to understand why (most, yet, not all) > societies are so obsessed in regulating the access and consumptions of > certains things

Some things are deemed net negatives for society, we already can't trust people not to drive while high or after drinking alcohol, if anyone could buy cheap meth at Walmart I can't imagine it would be a good thing.

I'm all for legalising, or at least decriminalising, psychedelics but some people really can't be trusted to ensure their own safety and the safety of others. On top of that it's a bit like seatbelts or motorcycle helmets, some people think it's their own choice but when they crash they cost $$$ to the public health system and they're fucking over every single tax payer in the country.

I already know the rebuttal to that point "simply do it safely", well, just as some people can't drink alcohol safely, some people can't handle drugs safely. The only sensible way to legalise it would be to allow it under full medical supervision, which brings a whole other set of problems

> while vast majority of unregulated things can and are as deadly as those > regulated.

If you're talking about alcohol and tobacco they're legal because they're historical and heavily lobbied. If alcohol didn't exist and someone came up with it in 2020 it wouldn't be legalised. Tbh in my opinion half the food sold in supermarkets should be as illegal as coke but that's another topic.

(By the way I always found the "if alcohol and tobacco are bad but legal then drugs should be legal" argument kind of moot, the logical conclusion would be to criminalise tobacco and alcohol rather than decriminalising everything else)

<comment><USER4> A big reason alcohol can't be made illegal is that is ridiculously easy to make yourself. Given time sugary liquids will spontaneously start fermenting into alcohol.

Getting your hands on drugs requires connections, and those connections can be more or less viably disrupted.

<comment><USER3> If you know how to access tor you can get lsd delivered via USPS in the us. It's not anywhere as hard to get as it used to be. I'm in Berlin right now and could get lsd (and all kind of other drugs) delivered to my door in less than 2 hours, it's the same in any major EU cities.

edit: and you don't even need tor really, everything is happening through signal these days.

<comment><USER4> The current political climate is favorable for drugs. Don't mistake that for it being impossible to stop.

A determined government could attack what you just described in so many ways. By shutting down tor. By owning tor. By tracing the envelopes. By setting up fake drug markets to catch buyers. By controlling access to precursors. By looking for labs. By bribing members of these organizations (I doubt they are 1-man operations).

<comment><USER2> I question the ability of government to stop everything deemed ilegal, immoral, dangerous. A (seemingly) arbitrary line is drawn somewhere for reasons that I things goes beyond what is rational.

</comment></comment></comment></comment></comment></comment><comment><USER5> From day 0 we all have begun the process of dying.

</comment><comment><USER6> "Allowing"</comment>


The most advanced directional signs on Earth - <AUTHOR> http://breakfastny.com/points/

<comment><USER2> that's cool! i hope they can take a beating though, because they look like a good swing could take them down, and the types of people that will be near the places their marketing have a tendencty to smash things.</comment>


Ask HN: Most regrettable design decision in computer industry? - <AUTHOR>

I think that there is a type of design decision that cause a lot of trouble and trivial benefit. If designers had been aware of problems, the world would have saved tons of time and money.
For example:
Big-Endian and Little-Endian:
The difference of endian forces developers to be careful of target architecture. If early processor vendors had communicated and established which is the standard, there would have been less trouble and no need to invent byte order mark(BOM).
32bit IP address:
Of course it is difficult to judge how many devices will be in the world, but sometimes I have an uneasy feeling about shifting to IPv6. I think that the industry will experience confusion and need to devote much effort than we think.
What do you think is most regrettable design decision in computer industry? <comment><USER2> There are 2 ways to handle strings: 1\. All strings are prepended with a length variable 2\. Use a NULL character to represent the end of the string

The Unix guys originally chose option 2. To them it made sense because it used fewer extra bits and it allowed you to do strings of any arbitrary length. Unfortunately, every single buffer overflow bug is because of that decision.

</comment><comment><USER3> \- plain text authentication (FTP, SMTP, etc).

\- 2 digits for years of 4 digits

\- 32 bits ints for counting dates since epoch

</comment><comment><USER4> Having a computer industry.

All the other design decisions are not regretable within the context of a computer industry, since they just promote more invoicing. On the contrary, the best thing you can do, is to create more such design decision, so you can charge customers more: to implement those design decisions, to reimplement them in a different language for a different platform, to reimplement them when the new extended version of the design decision becomes necessary, and so on. If you make a good design decision, you're set for life.

</comment><comment><USER5> If people would have just agreed about exactly what a 'newline' was, that would have been nice.

</comment><comment><USER6> While not directly related to h/w, one of my favorite reads has been "Unix Haters Handbook":

[http://web.mit.edu/~simsong/www/ugh.pdf](http://web.mit.edu/~simsong/www/ugh.pdf)

Note I am a Unix//Linux fan myself, but it was really eye-opening to read that doc. (it also confirmed some of my early suspicions on what I perceived to be various design flaws).

Also the book was released in the early 90s', and since then, some of the issues discussed in it has been fixed in later releases of *BSDs' and Linux.

It is still an enjoyable read :)

</comment><comment><USER7> Non replaceable power adapter-to-laptop cables, they save maybe a buck at most but cost us soo much more in replacement PSUs and fried laptops.

Daylight Savings Time

</comment><comment><USER8> 640k

GOTO

Buffer Overflows

DRM

</comment><comment><USER9> CAPTCHA

<comment><USER10> Because spam allowance is a much better design decision?

<comment><USER9> Since when has CAPTCHA stopped spam?

</comment></comment></comment><comment><USER11> null

<comment><USER12> Tony Hoare: "I call it my billion-dollar mistake. It was the invention of the null reference in 1965. ..."

[http://qconlondon.com/london-2009/presentation/Null+Referenc...](http://qconlondon.com/london-2009/presentation/Null+References:+The+Billion+Dollar+Mistake)</comment></comment>


Internet user arrested for linking to other websites  - <AUTHOR> http://www.fsf.org/blogs/community/mccarthy

<comment><USER2> _Federal officials charged 32-year-old Bryan McCarthy of Texas with one count of criminal infringement of a copyright. He allegedly posted illegal streams of the UFC, NFL, NBA, WWE and other sports programming._

 _According to today's release, McCarthy made more than $90,000 in profits from online merchants who paid him to advertise on the site. Since it was seized on Feb. 1, the site has received 1.3 million hits._

He wasn't indicted simply for linking to other websites. You, upon passing around a link to a copyrighted Youtube video, are not a potential felon. Here's the issue:

      § 506. Criminal offenses
        (a) Criminal Infringement. —
          (1) In general. — Any person who willfully infringes a copyright
          shall be punished as provided under section 2319 of title 18,
          if the infringement was committed —
            (A) for purposes of commercial advantage or
            private financial gain;

Note last sentence.

It is still perfectly reasonable to disagree with the prosecution (I don't, but I respect your right to). But it's still incumbent on the FSF not to be deceptive.

<comment><USER3> You leave out (B) and (C) of that paragraph:

        (B) by the reproduction or distribution, including by
            electronic means, during any 180-day period, of 1
            or more copies or phonorecords of 1 or more
            copyrighted works, which have a total retail value
            of more than $1,000; or

        (C) by the distribution of a work being prepared for
            commercial distribution, by making it available on
            a computer network accessible to members of the
            public, if such person knew or should have known
            that the work was intended for commercial
            distribution.

Not the "or".

<comment><USER4> How does it matter? The retail value of the UFC/NFL live broadcasts would be several orders of magnitude more than $1,000 to the networks/stations licensing them, and even if you let that slide based on what the end viewer is paying instead of the licensees the guy still hits C face-first.

So he's matching either ABC, or AC, either of which put him in the running for a free vacation at club fed.

<comment><USER2> It matters to the extent that one might want to build a case against the law by suggesting it could ensnare innocuous (or even accidental) infringement; the argument being, "look how easy it is to run afoul of (B) or (C)".

And that's an argument I have a lot of sympathy for, even if it isn't germane to this guy's situation.

<comment><USER5> Well I dunno. Part C seems to only apply to works being "being prepared for production", which most folks wouldn't even have access to. Part B would be the easiest for your casual pirate to run afoul of, but if you're pirating $1000 worth of stuff within a year then... yeah, you're moving out of "occasional vice" territory and into the "just watching lots of stuff without paying for it" territory.</comment></comment></comment></comment></comment>


Berlin – St. Petersburg Software Companies - <AUTHOR> Hello! I'm searching for software company which has offices in Berlin (Germany) and St. Petersburg (Russia). Under my current circumstances I need to find a way to be employed in Germany but be able to partly be in St. Petersburg. I heard at list one IT company with offices in both cities exists. If anybody knows something or can suggest anything, please, give a feedback! Thanks in advance! <comment><USER2> Yandex has its offices in both of the cities you've mentioned

Most of the global companies who dare not to cease their operations in Russia since the infamous events should have an office in St.Pete as it's the second Russian city by population with great universities and relatively low cost of living.</comment>


Ask HN: How To Build a Community? - <AUTHOR>

I do not suppose there is anything else harder to do on the internet. So how do you think a community like hn or reddit or digg or facebook or any other can be built? <comment><USER2> Suggestions that have worked for me on a smaller scale:

1\. Assign users meaningful names on signup. I pull them from the email address (whatever precedes @ if the username isn't already used in the system). This works out much better than assigning names like "user14132" because it makes casual participants seem more involved than they otherwise would.

2\. Also, there's a huge difference in site engagement between burying comments after a splash page and putting them on the front page sidebar. There is a trade-off between getting people into your funnel and encouraging casual participation. If you're building community it will come at a cost.

3\. Send out email notifications on direct replies in community areas (but be careful not to open your systems to spamming). People love getting automatically notified whenever someone replies to them.</comment>


Shop less, live more – save the planet. It doesn't sound that bad to me - <AUTHOR> https://www.theguardian.com/commentisfree/2019/aug/31/shop-less-green-future-sustainable-environment-retail <comment><USER2> > Only 29% of women actually say they enjoy shopping

Strange, all women I know enjoys(some addicted) to shopping. Makes me wonder how many other things I am related to biased samples.

A mini gist of what I takeaway from this:

\- women are conditioned to enjoy shopping but only minority of the whole enjoys it

\- buying things are bad but those things create jobs(marketing, managing, sales, trasport etc.)

\- shopping is sort of hollow happiness and wears off quickly

\- someone launched a challenge involving buying no new cloths in October

\- author advices to shop locally & physically & think about every expense

\- or instead of shopping spend that money in experience

[ edit: formatting ]

<comment><USER3> Women are attacked in Western cultures for their shopping habits by their families/spouses for the overall level of spend they make. It might be enough of a taboo subject in their minds that in a poll they may be more likely to rate their enjoyment lower than it actually is.

<comment><USER4> You're a weirdo if you don't care about your clothes and accessories beyond clean, free from visible wear and fitting properly, at least in high school and your 20s, and that's why a lot of us force ourselves to care, at least for a decade or so.

I am now a proud jeans and t-shirts lady who throws on a scarf and not- sneakers when I need to look a little fancier.

Think about why lots of young men who don't actually like beer or booze that much go drinking with the guys anyway. Some of them even over consume to the point of angering their wives/girlfriends.

<comment><USER5> Each person has their own path to running out of f*cks to give. Kudos to getting their sooner than later.

<comment><USER6> Agreed. Be it a spouse, family, work or friends. At some point we all find our limit.

</comment></comment></comment></comment></comment><comment><USER7> Anyone in the rich first world is welcome to try a morally and ecologically superior way of life by moving to a poor third world country and making a living there. I'm not sure small corrections in our lush lifestyles will cut it or deserve bragging rights...</comment>


The Surprising, All-Natural Anti-Nutrients and Toxins in Plant Foods - <AUTHOR> http://kindredmedia.org/2011/11/the-surprising-anti-nutrients-and-toxins-in-plant-food/ <comment><USER2> You're missing a "T" in the front. It's "The ..." not "He ..."</comment>


Other Operating Systems - <AUTHOR> http://www.azillionmonkeys.com/qed/os.html <comment><USER2> Interesting, but ...

> Why is it that so many people are interested it writing Operating Systems? > You wouldn't think it was because they are somehow dissatisfied with Windows > would you?

Many of these OS' predated Windows by good bit.

</comment><comment><USER3> A wonderful collection of dead links, sadly enough.

<comment><USER4> BeOS/Haiku (<http://haiku-os.org/> \- not a dead link, surprisingly)</comment></comment>


Exploring San Francisco and Silicon Valley - <AUTHOR> http://paulstamatiou.com/exploring-san-francisco-silicon-valley

<comment><USER2> If you can put up with it, I'd recommend doing a short term lease while you get a feel for the city. There are advantages and disadvantages to different parts, and I don't think you'll have a good idea of what's right for you until you've been immersed/found employment, etc. Bring a bicycle and plan on spending some time getting acquainted with the city.

Something like Lower Haight, SOMA, or Duboce Triangle area is centrally located and would be a good starting point to explore.

<comment><USER3> I will second Duboce Triangle, I live in this area and find that it has excellent bus and train routes to just about everywhere in the city.

</comment></comment><comment><USER4> Congrats on the move, Paul, and good luck in San Francisco!

</comment><comment><USER5> Be prepared that SF != Silicon Valley. Different culture, different companies, 1 hour+ commute or so each way.

<comment><USER6> To be honest I used both terms in the title for SEO purposes.</comment></comment>


Ask HN: Good examples of registration ui - <AUTHOR>

I did some searching and found this link: http://ui-patterns.com/explore/collections/account-registration<p>It has some good examples but I wanted to hear your opinions on what works and what doesn't. I personally like HN and Instapaper's simple registration forms and was wondering why there aren't more services using similar sign up forms. Wouldn't having one field for an email address and collecting all of the other cruft after validation be easier for end users? <comment><USER2> Here are my thoughts on registration forms:

You should always have absolute minimum number of fields required to start using the service.

Don't have any optional fields (there are plenty of better places to ask for that information).

If there is information required to use certain parts of the service then ask for it when they want to use that part.

<http://tumblr.com> and <http://posterous.com> examples of this. I'm actually a fan of the single password input too... I implemented a similar form recently: <http://hellohype.com/accounts/register>, only with a username field instead of a url field.

I'm not a big fan of going to the extreme of Instapaper though. They don't require a password at all, so a malicious user can access and modify another user's account.</comment>


The top five lessons Game Of Thrones can teach us about online security - <AUTHOR> https://www.getcloak.com/blog/2013/05/06/top-five-lessons-game-thrones-can-teach-us-about-o/

<comment><USER2> Also, the fact that Ned Stark has bastard children is an allegory for "Don't have multiple inferior browsers installed on the same machine." Come on, this whole thing was a real stretch.</comment>


Dweb: Building a Resilient Web with WebTorrent - <AUTHOR> https://hacks.mozilla.org/2018/08/dweb-building-a-resilient-web-with-webtorrent/ <comment><USER2> See also [https://github.com/mozilla/libdweb](https://github.com/mozilla/libdweb) for stuff that browsers should/could have that would make much better to write true decentralized applications.

</comment><comment><USER3> Thanks Feross for creating webtorrent. The biggest problem I see with this and also ipfs is that the latency is up to 30seconds even for files that are distributes over many clients. Is there any work in progress to reduce the latency?

<comment><USER4> From the theoretical point of view we can get huge improvements both in terms of torrent availability and latency, by making use of "fountain codes".

Basically it is a family of error correcting code which splits the data in N+M distinct pieces, and allow you to reconstruct the data once you have any distinct N of these pieces. This allow you to seed any one of these small pieces yet allow the reconstruction of the whole data provided there are enough other people seeding other pieces (Typically number of "seeds" will increase between x100 and x1000 (Note that it obviously don't increase the outgoing bandwidth of your network, but it makes it a lot easier to conserve and distribute rare files). Which also means any of these pieces can help getting the first packets needed to begin streaming which reduces latency. The negative aspects, are : \- bigger "torrent" files (because you need to store the hashes of more pieces). \- Some time you'll download an extra piece. \- More compute power needed on the clients to decode the stream. \- More complexity in the code.

Why is it not implemented ? Mainly because it is not a standard yet. There is quite a lot of freedom in the ways to implement it, and it will probably fracture the pools with plenty of non compatible protocols. And if you have a new version of the protocol to improve some things you need the seeds to re- encode the data to be protocol compatible.

That is also probably a subject which is encumbered by software patents.

Economically targeting the fat tail of rare torrents is probably not worth it.

It's fun algebra you should check if you don't know it already :)

<comment><USER5> Decentralized storage is already barely economical compared to centralized storage, if it makes sense at all.

While this might reduce latency, it sounds like in practice it'd multiple the redundancy of the system and make it even less cost effective?

<comment><USER4> Quite the opposite (it's like a RAID storage system), because of the error correcting scheme, you need a lot less storage to provide a good level of redundancy. You only need to store pieces of size 1/N * file size, and as long as you can get N of those pieces you can reconstruct the file. To guarantee a high level of availability, you only need that there are more than N hosts up containing distinct pieces. So typically you will use (~3 * size of file) of storage on your network, instead of (number of seeds * size of file) bits, and still provide with the same level of service as if number of seeds where seeding the whole file. So each node is able to store a lot more pieces of other files.

<comment><USER5> I might need to look up how they work because that's basically what my next question is: how do the chunks overlap and how does that affect the probability of a file being unrecoverable?

If I have a stupid simple scheme where I split a file into 100 pieces and replicate each piece 3 times, I have a 100C1 * p^3 chance of irrecoverability where p is the independent probability of pieces dropping out.

I guess I just want to know, given arbitrary p, what the probability of the file being gone is.

<comment><USER4> Error correction codes are quite magical. Instead of splitting in 100 pieces each piece 3 times, you split it in 300 distinct pieces of size 1/100 * filesize, which any 100 can reconstruct the file. So basically you can drop 2/3 of the pieces and yet be able to reconstruct the file.

So as long as your nodes uptime is greater than 33% + eps, the file is not gone. The eps is quite small because of the law of great number.

For example if your node uptime is 50%, the pieces will be in average available on 300 * 50% = 150 +- C*uptimevariance /sqrt(N) nodes which is greater than 100 with probability ~ 1, so the file will be recoverable almost surely.

<comment><USER5> Wow, Gokay codes were as advanced as I got but Fountain codes sound downright magical in comparison.

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> The web should be distributed, big thanks Feross!

This should also be in the spirit of Tim Berners-Lee.

</comment><comment><USER2> Honest question: when should I prefer this over IPFS?

<comment><USER7> So for keeping something in ipfs you need to have a server running ipfs seeding the files. You can use ipfs.js but it still needs to connect to an IPFS server. Webtorrent don't require any server running. All you need is the webtorrent.js file. That said you can still use a third party service like Infura for putting files in IPFS without running your own server. But Infura discards the file after seeding it for like 2 weeks. So you are not in total control of the file. In webtorrent you don't need any third party service. All you need is a browser with the webtorrent.js file. At the same time it is reccomended to run a STUN server to increase the reach of the files seeded by your webtorrent application. If you are creating a webtorrent application without a STUN server chances are that a lot of people would not be able to connect to it. I think IPFS on the other hand don't have this issue. Once a file is uploaded and is seeded by atleast one server it should be accessible to everyone else. (I am not 100% sure about this though). So choose wisely.

<comment><USER8> But the problem with Web Torrent is that the seeds go away when you close the tab, whereas with IPFS the page keeps getting seeded.

<comment><USER7> It is also the advantage of webtorrent. All a person need to seed a file in webtorrent is open a browser window. You can't rely on users of your application to start an IPFS server and seed the files.

If you want to seed a file for long term you can use the webtorrent-cli or webtorrent app as well.

</comment></comment></comment></comment><comment><USER9> Financial cost of self expression isn't much of a barrier. Having static website hosting in public cloud is affordable to quite many, few dollars in month and more if you have traffic. How does WebTorrent affect load times for users of your site?

</comment><comment><USER10> So.... this is a re-invention of FreeNet?</comment>


Internet Pastime by Age Group - <AUTHOR> http://reflectionof.me/internet-pastime-by-age-group <comment><USER2> What th..? They left off "trolling"!!!</comment>


Ask HN: I have 10 months to spend. What should I learn? - <AUTHOR> Due to weird circumstances I have about 10 months with secured income without having to work. When the 10 months are over I need to get a new job.
Up until now I have been working mainly with UX/IA/Design, but would like to transition more towards web and/or app development.
I should mention that on the side of my day job I have launched a few Rails apps and also iOS apps, so I am not starting completely from scratch.
Things I am considering:
* ASP.NET MVC (Where I live this is requested from a lot of companies)
* Backbone.js / AngularJS / Ember.js
* Android development
* Swift (I will most likely look into this anyway)
* Or just deepen my knowledge in Ruby/Rails.
Any feedback and ideas on what I should focus are welcome! <comment><USER2> No matter which one of these you choose, I'd recommend working on real (preferably open source) projects and putting them on Github so that you have something to show when you apply for jobs after 10 months.

<comment><USER3> Totally agreed! A recruiter I spoke to yesterday said, "I see that you have a presence on GitHub..." referring to one of my more popular OS projects, and then rambled on for about 15 minutes about his open source friendly client. I'm not saying you want to attract lots of rambling recruiters (because that's boring) but it means that people are recognising hard work and OS contributions as a good thing. It helps.

<comment><AUTHOR> I will definitely try to work on my GitHub presence. Thanks!

</comment></comment></comment><comment><USER3> Lucky you!

If I were in your position I'd use a couple of months to do some Swift development, and a couple to do some Android development using Android Studio. Might be worth trying out something for Google Glass and/or Oculus Rift if you have the necessary hardware. Or maybe grab a programmable quadcopter and give that a go?

Don't forget to consider learning about some other disciplines too, never know where things will take you :) or are you totally set on spending your time on programming?

<comment><AUTHOR> I will surely look into other things too.

I have found myself getting more and more into health, nutrition, yoga and meditation the last years. So the dream would be to somehow combine it.

<comment><USER3> Well, if you keep at all of it then hopefully you'll stumble onto an idea where you can combine it all. Good luck!

</comment></comment></comment><comment><USER4> The rest of the stack. Spend some time with layers 3&4 and you won't regret it. It was the most boring class I has in school and I use it every day. Read this book: [http://goo.gl/ZGifKa](http://goo.gl/ZGifKa) (Internetworking with TCP/IP Vol.1: Principles, Protocols, and Architecture (4th Edition)).

Knowing how TCP/IP works will give you an edge over other devs for understanding data movement and endear you to the IT/network/security folks (who do you favors).

<comment><USER5> There is 6th edition too: [http://www.amazon.com/Internetworking-TCP-Volume- One-Edition...](http://www.amazon.com/Internetworking-TCP-Volume-One- Edition/dp/013608530X)

</comment></comment><comment><USER6> It looks like the thing that's going to really blow up soon is IoT (or, at the very least, it looks like it's going to be the most lucrative, un-sexy- sounding thing on the planet in 5-10 years). If you have a good grounding in C, familiarize yourself with contiki (which, although it's not being used now, is likely to become the backbone of any third-party IoT node that isn't running linux) and some basic digital electronics.

</comment><comment><USER7> From a Professional Development & Leadership perspective, tech guys often overlook basic presentation and sales skills, useful if you need to rally project buy-in or gain support for an idea. Here's a place to start> [http://pitchanything.com/](http://pitchanything.com/)

</comment><comment><USER8> I learned Clojure/Clojurescript the last half a year which was my sabbatical. Coming from design into frontend, functional programming blew my head off and made me a way better programmer :)

</comment><comment><USER9> Pick a job that you want, gather the requirements for that job, and learn them. Aim for something hard.

</comment><comment><USER10> Travel. And play with girls :)

</comment><comment><USER11> Start a business.</comment>


JSON will be a core type in PostgreSQL 9.2 - <AUTHOR> http://people.planetpostgresql.org/andrew/index.php?/archives/244-Under-the-wire.html

<comment><USER2> This is nice, but it's a shame that after going to the trouble of writing a JSON parser in C and incorporating it into the main Postgres codebase, they didn't go one step further and add a json_project_key function that pulls values out of a JSON blob. That, combined with functional indices, allows you to store JSON blobs in your database _and_ index on values inside them, or put another way, get the data model benefits of a NoSQL database [1] without sacrificing the mature Postgres internals.

The Heroku Postgres guys have been playing with this idea [2] using the PL/V8 plugin, which embeds Javascript as a supported language inside Postgres (and thus makes it trivial to implement the json_project_key function), but if Postgres is going to natively support JSON parsing then it shouldn't take an addon module to achieve this.

[1] Attempting to forestall the thread-jacking: I know NoSQL databases have other benefits besides their data model, but for some applications that's certainly _one of_ the benefits. [2] <https://gist.github.com/1150804>

<comment><USER3> I can't help but think this is just step 1 in bringing native support for JSON indexing and querying.

<comment><USER4> On the other hand, as far as I can see XML support still does not allow for indexing or querying of the document itself, at least in core, even though it's been a postgres datatype for a long time.

</comment></comment></comment><comment><USER5> Interesting. Being able to output the results of a query as JSON will be useful when running something like Node.js. I can't shake my feeling that something like this doesn't belong in the SQL statement, though- choice of output format seems like it's something connecting libraries ought to do.

That said, having seen 'core type' I instantly imagined being able to query based on JSON properties, which doesn't appear to be the case. Not surprising, because it would be a huge amount of work.. but it's nice to imagine.

(before anyone says anything- yes, I know NoSQL exists. But a hybrid solution using Postgres would be very interesting)

<comment><USER6> You might be interested in postgresql's hstore: <http://www.postgresql.org/docs/9.0/static/hstore.html>

It's a key-value store that allows querying, which is what I think you are lamenting in your comment. Here's a bit more about how to query and index with hstore: <http://lwn.net/Articles/406385/>. It's pretty simple and doesn't have anywhere near the number of querying possibilities that MongoDB has, but it can be used for on-the-fly column names (similar to JSON). You can only query on the root node's children, unlike the open-ended possibilities in NoSQL.

<comment><USER5> Oh, that is very intersting, thanks!

I'm (quite happily) tied to Postgres because I'm using PostGIS, but the ability to add freeform data to a location would be ideal. Looks like I may already have a solution here.

<comment><USER7> Cool, PostGIS! What are you building? I worked as GIS engineer in 2010. PostGIS is pretty cool technology.

<comment><USER5> I'm building a 'taxi tracker' app, for want of a better term- gives you estimated trip times, allows you to share the journey with other people so they know where you are.

That part doesn't require PostGIS as such, but it's for an NYC city government app competition, so we have all sort of city datasets to use. I've used PostGIS to help make custom map tiles (preview at <https://twitter.com/#!/taxonomyapp/status/149565007384940545>), highlight the outline of the building you're heading to... all sorts. It's been a fantastic learning exercise.

<comment><USER8> I don't know exactly what your GIS needs are, but it might be worth seeing if MongoDB's geospatial indexing meets them if you are interested in querying free-form data.

<http://www.mongodb.org/display/DOCS/Geospatial+Indexing> [http://www.mongodb.org/display/DOCS/Geospatial+Haystack+Inde...](http://www.mongodb.org/display/DOCS/Geospatial+Haystack+Indexing)

</comment></comment></comment></comment></comment></comment><comment><USER9> For the record, the title is misleading and Andrew Dunstan (the guy whose blog this entry links to, a major PostgreSQL contributor and one of the people involved in having a JSON type in Postgres) clarifies:

[http://people.planetpostgresql.org/andrew/index.php?/archive...](http://people.planetpostgresql.org/andrew/index.php?/archives/245-I-didnt- say-that.html)

</comment><comment><USER10> My first reaction to this is - "Ugh. Haven't we been down this road with XML?" I don't want the format du jour baked into my datastore. Now, if it had a field that stored some sort of abstract lists-and-hashes structure (sort of redis-in-a-field), that sounds more interesting to me. I realize JSON is isomorphic to just that, but it still seems like flavor of the week.

<comment><USER11> Look at PostgreSQL's hstore data type - that's pretty much what you're asking for.

<comment><USER12> hstore is all kinds of awesome, but one limitation is that the values are pure text. You can nest an hstore inside another, but only if you're smart enough to know that it SHOULD be treated as an hstore; as far as Postgres is concerned, it's text that happens to contain '=>'.

If JSON support means first-class type support in a nested object, that's a huge leap forward.

</comment></comment></comment><comment><USER13> Pretty neat. I think this is the important bit though: "Basically, his patch just parses the text to make sure it was valid JSON, and stores it as text."

So, if you already have an environment where you are certain you are inserting valid JSON, it's not much different than just using the text type today.

<comment><USER14> > So, if you already have an environment where you are certain you are > inserting valid JSON, it's not much different than just using the text type > today.

Today, yes.

Tomorrow, when you launch an API and suddenly hundreds of different apps are talking to your systems, no.

Putting rich descriptions of data right next to the data is a good thing in the long run.

</comment></comment><comment><USER15> I think it's worth mentioning that MySQL has a nice collection of UDF's to handle JSON object output, arrays, and so on: <http://www.mysqludf.org/lib_mysqludf_json/index.php>

</comment><comment><USER16> it would be super awesome if JSONH <https://github.com/WebReflection/JSONH> (json for homogenous collections) was standardized and adopted. it would be a great csv replacement and a perfect fit for recordsets, providing minimal overhead.

</comment><comment><USER17> Erm... JSON is pure text - so not much of an innovation in truth - I can store it in SQLight right now.

<comment><USER18> I think you're missing the point. With this feature you'll be able to write a query that returns a JSON string of the results, possible as a field in a larger result set. It also validates that the JSON parses on the way in.

<comment><USER17> Thanks for the help - so it is a "result type" (not so much core) - OK that will save a few lines of code - unless I have to write as many lines to handle result sets that do not parse...

I suspect I am not a big fan of abstractions.

<comment><USER19> I'm with you, this smells of feature creep. I don't see a big benefit over serializing the result set on the client. This is literally a one-liner if you're using an ORM or something like PEP 249.

The new constraint allowing you to ensure the string stored is valid JSON sounds useful though.

<comment><USER20> > I'm with you, this smells of feature creep.

There are good reasons to have JSON support in the DBMS, such as querying and indexing on the fields within a JSON document.

My only criticism is that postgres has a great extensions mechanism, and it would be better if effort were spent improving the ecosystem around that. I can't think of any reason for it to be in core if installing an extension were a trivial and widely accepted practice.

But, it takes time to really develop that ecosystem. And there are users that want JSON support yesterday.

EDIT: from a technological standpoint, there is no reason why it can't exist as an extension. PostGIS is much more sophisticated on all counts, and it's an extension.

<comment><USER21> I agree with you insomuch as it's ideal for an extension, and argued for its inclusion as a 9.3 feature (looks like Christmas might come early, though). However, I concluded with this:

      Right now the perception of Postgres...actually, databases in general,
      including virtually all of the newcomers -- is that they are
      monolithic systems, and for most people either "9.3" will "have"
      javascript and indexing of JSON documents, or it won't.  In most cases
      I would say "meh, let them eat cake until extensions become so
      apparently dominant that we can wave someone aside to extension-land",
      but in this case I think that would be a strategic mistake.

[http://archives.postgresql.org/pgsql- hackers/2011-12/msg0078...](http://archives.postgresql.org/pgsql- hackers/2011-12/msg00782.php)

So this is definitely a social problem. Why not move UUID out of the core? (not uuid-ossp, which implements generation, but uuid parsing and storage) Because, for now, one cannot assume extensions for really common useful functionality.

Luckily, I think things are on the right path to at a future juncture that even a commonly desired data type can live as an extension.</comment></comment></comment></comment></comment></comment>


Ask HN: Largest Non-Recoverable Dataloss - <AUTHOR> Whats the largest non-recoverable storage loss which has been on either of SSD or HDD. (by Pb?)<p>I would imagine few enterprises which suffered one want to broadcast it. <comment><USER2> The collapse of the Western part of the Roman Empire would qualify. No matter how much life went on as before in the rural parts (vide Auson and his _Mosella_ , which tells of peace), the loss of scientific and cultural knowledge is just staggering.

Wikipedia has an article: [https://de.wikipedia.org/wiki/Bücherverluste_in_der_Spätanti...](https://de.wikipedia.org/wiki/Bücherverluste_in_der_Spätantike)

<comment><AUTHOR> I was really thinking about modern times. Post computer age.

But your answer is interesting, and goes to the library of Alexandria I guess.</comment></comment>


Venture Fund Economics: Gross and Net Returns - <AUTHOR> http://avc.blogs.com/a_vc/2008/08/venture-fund--1.html

<comment><USER2> 40 points and no comments, that's special.

I thought it was interesting :)

<comment><USER3> It certainly provides a better understanding of what the VC's are thinking about when they're investing.

Of particular note was how soon they have to identify the "losers" and cut them off from further funding.</comment></comment>


Running Awk in parallel to process 256M records - <AUTHOR> https://ketancmaheshwari.github.io/posts/2020/05/24/SMC18-Data-Challenge-4.html <comment><USER2> Hm. I'm fully aware that I'm currently turning into a bearded DBA. And I may be just misreading the article and I probably don't understand the article.

But, I started being somewhat confused by something:

> Fortunately, I had access to a large-memory (24 T) SGI system with 512-core > Intel Xeon (2.5GHz) CPUs. All the IO is memory (/dev/shm) bound ie. the data > is read from and written to /dev/shm.

> The total data size is 329GB.

At first glance, that's an awful lot of hardware for a ... decently sized but not awfully large dataset. We're dealing with datasets that size at 32G or 64G of RAM, just a wee bit less.

The article presents a lot more AWK knowledge than I have. I'm impressed by that. I acknowledge that.

But I'd probably put all of that into a postgres instance, compute indexes and rely on automated query optimization and parallelization from there. Maybe tinker with pgstorm to offload huge index operations to a GPU. A lot of the shown scripting would be done by postgres, the parallelization is done automatically based on indexes, while eliminating the string serializations.

I do agree with the underlying sentiment of "We don't need hadoop". I'm impressed that AWK goes so far. I'd still recommend postgres in this case as a first solution. Maybe I just work with too many silly people at the moment.

<comment><USER3> If you're dealing with such a static single-machine dataset, why not go for SQLite instead of PostGres?

There'd be much less setup overhead

<comment><USER2> Sqlite is single-writer for a single database. I mean. Of course, VHRanger, only your team will need to write to the database, and your team will make sure that only one person on your team will write to it. Eh. I've been there too many times. Oh, but yes, your team will also figure out the fallout if things to wrong. Ah..

Ok. Maybe those are enterprise concerns: Sqlite doesn't scale regarding multiple users reading and writing. Of course it's a read only dataset, but do you know the bouquet of views and derived tables data scientists create around a read-only dataset? Hah. Oh and of course these are not critical, but if they get lost, shit hits the fan because it takes multiple weeks to rebuild them.

I've been in that swamp enough times to just install postgres and stop caring. Takes me 2 more hours now, but avoids weeks of discussions in the future.

<comment><USER4> Remember that we're comparing this to awk…

</comment></comment></comment></comment><comment><USER5> I'm sure that with tools like MPI-Bash [1] and more generally libcircle [2] many embarrassingly parallelizable problems can easily be tackled with standard *nix tools.

[1] [https://github.com/lanl/MPI-Bash](https://github.com/lanl/MPI-Bash) [2] [http://hpc.github.io/libcircle/](http://hpc.github.io/libcircle/)

<comment><USER6> I have lately used a shell `for` loop that only emits part of the iterated values (not all are eligible for further processing) and then fed the loop directly to the `parallel` tool via the pipe operator.

The results were indeed embarrassingly parallel.

I am a fan of some languages that seem better equipped to utilise our modern many-core machines and I'd still write a longer-living system with better guarantees in those languages -- but many people ignore shell goodness at their own peril.

<comment><USER7> Big agree. Gnu parallel, xargs, and make -j are all very useful basic tools for embarrassingly parallel workloads.

I've been developing a simulation software that does Monte Carlo over realizations of a simulated universe, and xargs and (later) parallel were super useful for parts of the workload. All the parallel job instances run the same simulation code, but for a different simulated universe, all controlled by a random number seed, so you can generate an ensemble of simulations with basically:

       head -NumberOfSimsWanted seeds.txt | xargs simulation.py

<comment><USER8> I have lately been working on a genetic algorithm system, written in single- threaded Perl. When I started off wanting to do, say, 20 simulations, I just used parallel, and it worked great on the 8 threads of my desktop machine. Now I've done a bit more and want to do long-running simulations that might take a few hours, I've taken to just firing it all at AWS Batch and having a script that uploads the results to S3. Now I can do 1-200 instances at the same time, which my own hardware isn't up to in a reasonable timeframe.

<comment><USER6> I have a pretty strong workstation (iMac Pro) that I currently don't use all the time. And a gaming PC that I barely touch in the last 9-10 months. Hit me up if you want me to lend you some CPU power sometime.

<comment><USER8> Cheers, but the configuring of it would be more hassle than it's worth I suspect. As it is, using AWS is very cheap as I can have it use spot instances. So hundreds of CPU-hours used for in the order of US$10-20.

<comment><USER6> I suspect setting that up would indeed be a hassle. :)

Thanks for mentioning AWS Batch, I didn't know about it and will look it up.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> It's odd that TFA has this focus on performance but doesn't mention _which_ awk implementation has been used; at least I haven't found any mention of it. There are 3-4 implementations in mainstream use: nawk (the one true awk, an ancient version of which is installed on Mac OS by default), mawk (installed on eg. Ubuntu by default), gawk (on RH by default last I checked), or busybox- awk. Tip: mawk is much faster than the others and to get performance out of gawk you should use LANG=C (and also because of crashes with complex regexpes in Unicode locales in some versions of gawk 3 and 4).

<comment><AUTHOR> Thank you for your comment. I should have clarified that I use gawk version 4.0.2. I mean to update the post but somehow am unable to push to repo or login to my github.

Edit: Done.

<comment><USER10> They seem to be currently having a few problems with log-in: [https://www.githubstatus.com/incidents/q3cfsrp1qb6l](https://www.githubstatus.com/incidents/q3cfsrp1qb6l)

</comment></comment></comment><comment><USER11> this somewhat reminds me of taco bell programming [http://widgetsandshit.com/teddziuba/2010/10/taco-bell- progra...](http://widgetsandshit.com/teddziuba/2010/10/taco-bell- programming.html)

<comment><AUTHOR> Excellent! Didn't know there is a name for this. Agreed the work is a case of Taco bell programming!

<comment><USER12> There is also [https://adamdrake.com/command-line-tools-can- be-235x-faster-...](https://adamdrake.com/command-line-tools-can- be-235x-faster-than-your-hadoop-cluster.html)

</comment></comment></comment><comment><USER13> I've always wanted to build a parallel awk. And call it pawk. And have an O'Reilly book about it. With a chicken on the cover. pawk, pawk, pawk! This is a true story, sadly.

<comment><AUTHOR> I too wanted to do almost exactly the same. I want to port awk to GPUs by hacking its code and adding OpenCL tags at right places. Someday!

</comment></comment><comment><USER14>
        !($1 in a) && FILENAME ~ /aminer/ { print }

This uses a regular expression. As regex is not actually needed in this case, you might be able to get better performance with something like this:

        !($1 in a) && index(FILENAME, "aminer") != 0 { print }

</comment><comment><USER15> I like the idea of using AWK for this. But you can give kdb/q a try. 250M rows is nothing for kdb, and it seemed that you can afford the license.

<comment><USER16> kdb/q is amazing. I have seen some experienced folks make jaw dropping data computations at the drop of a hat on gigabyte sized data. Makes everything else look laughably _puny_.

I suspect if it was open source, it would probably be the most popular big- data storage and computing platform.

<comment><USER17> Hmm... 256M records is still not that much for postgres or mysql so it shouldn't be too much for kdb/q. Several gb of data also shouldn't be too much.

</comment></comment></comment><comment><USER18> gawk has been my go-to text processing program for many years. I have written a number of multi-thousand line programs in it. I always use the lint option. Catches many of my errors. One of these had to read a 300,000,000 byte file into a single string so it could be searched. The file was in 32-byte lines. At first, I read each line in and appended it to the result string, but that took way to long since the result string was reallocated each time it was appended to. So I read in about 1000 of the 32-byte lines, appending them to a local string. This 32000-byte string was then appended to the result, so this only was done 10000 times. Worked fine.

</comment><comment><USER19> Spending _minutes_ on these tasks on hardware like this is pretty silly. awk is fine if these are just one-off scripts where development time is the priority, otherwise you're wasting tons of compute time.

Querying things like these on such a small dataset should take seconds, not minutes.

<comment><AUTHOR> Thank you for your comment. Most of the solutions indeed take less than a minute. Solution to problem 1 & 3 took 25 sec and that to 5 took 26.

The solution that took 9 minutes involved processing the abstract in each record. The abstracts are quite sizeable on some of these publications. Processing millions of them took time.

The solution that took 48 minutes involved a nested loop effectively reaching the iteration count for 216 years times 256M records which comes to about 55B.

Hope this clarifies things a bit but I am not claiming this to be the most optimized solution. I am sure there is scope for refinements -- this was my take on it.

</comment></comment><comment><USER20> First, I think it is great that you found a tool that suits your needs. A few weeks ago I was mangling some data too (just about 17 million records) and would like to contribute my experience.

My tools of choice were awk, R, and Go (in that order). Sometimes I could calculate something within a few seconds with awk. But for various calculations, R proved to be a lot faster. At some point, I reached a problem where the simple R implementation I borrowed from Stack Overflow (which was supposed to be much faster than the other posted solutions) did not satisfy my expectations and I spend 4 hours writing an implementation in Go which was a magnitude faster (I think it was about 20 minutes vs. 20 seconds).

So my advice is to broaden your toolset. When you reach the point where a single execution of your awk program takes 48 minutes, it might be worth considering using another tool. However, that doesn't mean awk isn't a good tool, I still use it for simple things, as writing 2 lines in awk is much faster than writing 30 in Go for the same task.

</comment><comment><USER21> Definitely an under-appreciated tool. Very useful for one-off tasks that would take a fair bit longer to code in something like python.

</comment><comment><USER22> I am often impressed by the things that can be done with these old-school UNIX tools. I'm trying to learn a few of them, and the most difficult part are these very implicit syntax constructions. How is the naive observer to know that in bash `$(something)` is command substitution, but in a Makefile `$(something)` is just a normal variable? With `awk`, `sed` and friends it gets even worse of course.

Is the proper answer 'just learn it'? Are these tools one of these things (like musical instruments or painting) where the initial learning phase is tedious and frustrating, but the potential is basically limitless?

<comment><USER23> Some of it you pickup or remember from the context of the file you're looking at, but you really should take the time to read the manuals for these tools from cover to cover at some point if you're making extensive use of them. In the case of GNU bash & make: [https://www.gnu.org/savannah- checkouts/gnu/bash/manual/bash....](https://www.gnu.org/savannah- checkouts/gnu/bash/manual/bash.html) & [https://www.gnu.org/software/make/manual/make.html](https://www.gnu.org/software/make/manual/make.html)

</comment></comment><comment><USER24> [https://mobile.twitter.com/awkdb](https://mobile.twitter.com/awkdb) was a joke account made in frustration by a coworker trying to operate a Hadoop cluster almost a decade ago. Maybe it's time to hand over the account...

</comment><comment><USER25> Your system had 512-core Xeons? Did you mean that you had 5 12-core xeons? Or 512 cores total?

<comment><AUTHOR> 512 Intel Xeon CPUs configured over 16 nodes. This one: [http://www.comnetco.com/sgi-uv300-the-most-powerful-in- memor...](http://www.comnetco.com/sgi-uv300-the-most-powerful-in-memory- supercomputer)

<comment><USER25> Wow. I didn't realize you could run so many CPUs in one address space. This thing is basically one huge computer. Would love to open up Gnome System Monitor and see 512 cores!

<comment><USER26> Core counts has gone up quite considerably in the last few years, certainly in a compute farm context. 10 years ago, you'd be looking at 2U servers having 8 cores (dual quad core) as high density. These days, 2U servers can pack 4 sockets, and processors are in the high 20 something cores, so if you've got deep pockets you can get over 100 cores in 2U.

<comment><USER27> AMD Epyc can give you 128/256 cores in a 2 socket system. Not that expensive.

</comment></comment></comment></comment></comment><comment><USER28> and here I am working on a big distributed system that has to handle 200k records a day (and hardly does successfully). sigh.

</comment><comment><USER29> Turning json data into tabular data using jq was pretty neat. So many json apis in use today yet still a need for csv and excel docs.

<comment><USER30> Excel can natively connect to and parse json nowadays.

</comment></comment><comment><USER31> You couldn't have used FS="\036" or "\r"?

</comment><comment><USER32> Amazing work. Keep it up.

</comment><comment><USER30> Why not just use data.table?

The solution would be much less error prone and most likely much quicker as well.

<comment><AUTHOR> Thank you for the comment. One reason to use awk was that I wanted to see how far I could go with Awk. I will checkout data.table. Does it offer any kind of parallelism?

<comment><USER30> Many common functions in data.table are parallelized under the hood by OpenMP.

</comment></comment></comment><comment><USER33> Slightly off topic, but as a swift developer ([https://swift.org/](https://swift.org/)) the usage of swift/T in this project really confused me. Is swift/T in any way related to Apple's swift language?

The naming conflict makes googling the differences fairly challenging.

<comment><AUTHOR> Both Swift's are different. I was aware of potential confusion which is why I did not use Swift in the title and mention explicitly in the blog that this is not Apple Swift. Fun story: Apple contacted Swift/T team before launching Apple Swift and made a mention of it in their page.

<comment><USER33> Cool! Thanks for the explanation. That answers my biggest question of who came first.

</comment></comment></comment><comment><USER34> Try mawk if you can. I find that it does even faster.

<comment><AUTHOR> Indeed I did try mawk and found it to be faster. However, when I try setting locale "LC_ALL=C"; the performance of both awk and mawk were almost the same. I also let it at awk in favor of portability -- mawk is not available on most systems and needs to be installed.

</comment></comment><comment><USER35> GNU Parallel + AWK = even less code to write.</comment>


How to make a product video for under $150 - <AUTHOR> http://polaske.tumblr.com/post/91289197615/how-i-created-a-professional-product-video-for-146-55?one=one

<comment><USER2> still something is missing to be professional. Anyway is a lot better than other home made (:

<comment><AUTHOR> Most definitely not quite pro. And thanks!</comment></comment>