Show HN: Local Node.js app to save everything you browse and serve it offline - <AUTHOR> https://github.com/dosyago/22120 <comment><USER2> It's a bit disappointing to see other people showcasing their own work without a single mention of the link above. Perhaps make your own submission instead if that's the intention?

As a casual reader and others obvious interest in this area I'd very much prefer a sentence or two about the quality of the work presented, feel free to link your own stuff afterwards, it's a bit offputting to see such blatant self-promotion.

<comment><USER3> I like when people link related projects. This is Why:

I have a use case for this project but I likely won't get to it for a year or so. When that happens I'll come back to this thread and all the projects working on the problem will be right here in the Hacker News thread. I'll be able to see which ones are still alive, and maybe even see why some stopped development.

This happens all the time for me with HN - if people didn't link their related work the thread would have way less utility.

<comment><USER2> They could always actually view the work say something about it first yes? [https://news.ycombinator.com/showhn.html](https://news.ycombinator.com/showhn.html)

Looking at the projects presented here with nothing else offered am not convinced of good-faith participation. Would you say the same?

<comment><USER4> I'd take a self-promo related project comment over an unrelated complaint (hogging entire above-the-fold space) any day. Nothing is more frustrating than opening a discussion thread and the top comment with a million descendants bikesheds about something else entirely.

Also, you would occasionally see comments along the line of "awesome project / congratulations on launching, I like the fact that it does this and that. My project also does this and that, check it out: insert link." Hardly any better.

</comment></comment></comment></comment><comment><USER5> Congrats on getting your project to the front page of HN. With that said I think you are going to need to change your approach if you want this project to be usable as more than a toy project in the long run.

From what I can tell it essentially saves a map of url -> response in memory as you browse. Every 10 seconds this file is serialized to json and dumped to a cache.json file. This is going to be very inefficient as the number of web pages indexed grows since you are rewriting the entire cache every 10 seconds even if only a few pages have been added to it. It also will eventually exceed the memory of the computer running the app if the content of every page ever visited needs to be loaded into memory. I highly recommend looking into some of the other suggestions mentioned here, either sqlite or mapping a local directory structure to your caching strategy so that you can easily query a given url without keeping the entire cache in memory, and also add / update urls without rewriting the entire cache.

<comment><AUTHOR> My future plan was to cache responses on disk and just keep cached keys in memory:

[https://github.com/dosyago/22120#future](https://github.com/dosyago/22120#future)

<comment><USER6> I wrote something similar years ago in Go, and settled on writing the data to a WARC file on disk (you can gzip the individual requests and concatenate to get random access), and also concatenating to a warc index file. The working index was kept in memory, while the warc index was read at startup.

My version acted as a proxy and would serve the latest entry from cache if a copy was cached. I had a special X-Skip-Cache header for when I wanted to go around the cache. (I can't remember if it handled https or if sites just didn't use https back then.)

My use-case was web scraping, particularly recipe and blog sites. I wanted to be able to develop my scraping code without re-hitting the sites all the time. Structuring it as a proxy allowed me to just write my python scraping code as if I was talking to the server.

Previously I'd written a layer on top of the python requests library to consult a cache stored in a directory (raw dumps of content / headers, with v2 involving git). But I found that required extra care when more than one script was running at once, and I liked the idea of storing it in a standardized format (WARC) that could be manipulated by other tools.

<comment><USER7> I tried to build something like this for jest tests in an app I worked on.

I wanted my jest tests to serve as both unit tests and service diagnostics - so I instrumented axios and setup a hidden cache layer within it when running inside the test suite. I was trying to figure out how to best organize the cache so I could run tests really quickly by having all results pulled from cache — or run it slow and as a service diagnostic mechanism by deleting the cache before execution ... I had to extend axios to accept a bit of additional logic from the application ...

it was hard for me to get it to work properly inside of jest though ...

</comment></comment></comment></comment><comment><USER8> You could store the data in a git repo per domain, so that implicit de- duplication happens on re-visits & for shared resources.

You could have a raw dir (the files you receive from the server) and a render dir that consists of snapshots of the DOM + CSS with no JS & external resource complexity.

When the global archive becomes too big, history could be discarded from all the git repos by discarding the oldest commit in each repo, and so on.

SOLR is probably the right tool for the index but there is something undeniably appealing about staying in the pure file paradigm - you could use sqlite's FTS5 module to do that too.

<comment><USER4> git is pretty bad at handling large binary blobs. Good old timestamped directories with hardlinks (a la rsync --link-dest) probably works better.

<comment><USER9> Git isn't that bad at handling binary blobs - as long as you enable LFS support, and your git repo is served locally, as suggested, you'll do fine.

<comment><USER4> > as long as you enable LFS support

You still end up with two copies of the same file, one in the local LFS "server", one in the work tree, no? (I only played with LFS a bit many years ago when it first came out, so I could be wrong.) Unless you take into account deduplication built into certain filesystems.

<comment><USER9> You don't get copies until you need them - thats the point entirely. More details here:

[https://www.atlassian.com/git/tutorials/git- lfs](https://www.atlassian.com/git/tutorials/git-lfs)

Also saves you from beating up your index with every change.

I've been using GIT LFS for the last 6 months with an Unreal Engine project, with multiple gigabytes of files being tracked, and it really is painless.

<comment><USER4> > You don't get copies until you need them

I know. But you do need them, and files in your work tree don't magically disappear when you commit them in (presumably). So either you delete the work tree copy immediately after pushing it to LFS server, and duplicate the server copy every time you need to access it, in which case the file is only duplicated then but comes with elevated cost of access, or the latest copy sits around costing double the amount of space at all times.

<comment><USER9> I don't see the issue? Either you want to use Git or not. I have gigabyte- scale files in my 6-month old repo's and haven't ever run into any issues. Of course this may be because my git server is right next to my desk and I'm on gigabit ethernet ..

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> what are the security implications of permanently running chrome in remote debug mode?

a bit more than half a year ago I started playing around with this and was surprised how on the one hand there are really really good tools nowadays for self archiving but how on the other hand there hasn't been any progress in implementing these in a, for end users, comfortable way

My working theory right now is that saving every request/response as well as every interaction on a page should allow us to completely restore web site state at any point in time and will open up some super interesting use cases around our interaction with information found online

But in order to do this it seems necessary to go through the remote debug protocol like this project here is doing. And since this is somewhat of an unusual approach I could not find much information about the security aspect of running every site at any time with remote debugging activated. Common web scrapers/archiving tools will instead only use remote chrome debug to open and capture specific urls

Storage is so dirt cheap today that there is zero reason why we shouldn't have reliable historic website state for everything we have ever looked at

And judging by the HN front pages of the last months, many here are interested in this and related use cases (search/index/annotations/collaborative browsing)

<comment><USER11> > _Storage is so dirt cheap today that there is zero reason why we shouldn 't > have reliable historic website state for everything we have ever looked at_

I agree entirely, but I do about half my reading on mobile, and the phone company and the ad company have both decided that I shouldn't be able to run extensions of any kind in the browsers available on my phone company phone _or_ my ad company phone.

I'm not really sure of the solution. I had planned to start a business around this, but without mobile support it is probably a nonstarter.

<comment><USER12> Run your phone traffic through a proxy and have the proxy cache stuff.

<comment><USER13> Proxy can't intercept https, if I'm correct

</comment></comment></comment></comment><comment><USER14> Like 20 years ago I've used a program called Teleport Pro, to do something similar.

I would dial up with my phone modem when the internet access was cheap (during the night), it would automatically browse a page I provided, and in the morning I would have the page ready to read.

Fun times with 10 to 20 kb/s speeds.

<comment><USER15> I had a similar experience, but I also think Internet Explorer saved the websites you visited to browse them later in offline mode, right? I remember some times I couldn't know if I was online or not because the website was cached, i had to visit a different website that I never visited before to check my connection status.

</comment></comment><comment><USER16> I'm curious of why did you go on the path of using chrome debugging functionality instead of implementing an HTTP proxy which would provides the benefit of behind browser-agnostic too.

May you expand on that please?

<comment><USER17> After reading this I was wondering if it might be fun to write a HTTP-proxy that a) recorded everything, in an SQLite database, and b) presented a localhost-server which would let you search that content.

I suspect it would get very very very busy, with tracking-pixels, etc, but if you only made it archive text/plain, text/html, and similar content-types it might be a decent alternative to bookmarks, albeit only on a single host/network.

Wouldn't be hard to knock up a proof of concept, perhaps I should do that this evening.

<comment><USER18> I did something like this a while ago: [https://github.com/nspin/spiderman](https://github.com/nspin/spiderman)

I used the wonderful tool mitmproxy for both recording and serving.

</comment></comment></comment><comment><USER19> Great feature. Though it feels like a UI misstep that the user had to use npm to switch between recording and browsing. A nicer solution could be a chrome extension button, or access the archived version via a synthetic domain. E.g. example.com.archived

<comment> crooked-v There's also some 'magic' potential here to have a proxy that detects whether there's a live network interface or not (including some sanity checking against captive portals), passes through the live sites while recording when there is a connection, and serves from the last recorded versions when there isn't.

<comment><USER20> Sounds like a local squid proxy setup from the early 2000s...

</comment></comment></comment><comment><USER21> Great idea! I like the concept.

One of the things I miss most about the old web was how trivial it was to local mirror any website. It was great!

</comment><comment><USER22> I remember I tried something similar a long time ago but decided it wasn't worth it.

2MB per page at 100 page a day, 200MB / day. That is 73GB per year.

May be once a year I get the problem where I remember reading something about it but could not google back the exact page in my memory. So I had a proxy solution set up, but the maths ends up it wasn't worth paying the Storage cost just for this one time convenient.

<comment><USER23> Perhaps one solution would be to extract the plain text and the URL of the page only. That wouldn't take much space and would still be searchable.

</comment></comment><comment><USER24> Between this project and the others mentioned in the discussion these are excellent resources for anyone needing to have a forensic record of their how they assembled evidence from browsing open sources on the internet. Package this as a VM that can be quickly spun up new and fresh per case and sell support for LE types and you've got a business.

<comment><USER25> If original author agrees, I can dockerize it.

<comment><AUTHOR> Hey that's a cool idea about a business, I've made it into a packaged Node.JS app as a binary you can see on the releases page:

[https://github.com/dosyago/22120/releases](https://github.com/dosyago/22120/releases)

I like multiple release channels and there's plenty of ways to install and use this.

You can download a standalone binary (Win, Mac or Linux), install globally from npm, or just clone or download the repo and run it.

I'm not sure about docker, but could you maybe give it a try and share me the docker file privately and I can decide if I like it?

If it's good then we can add it to the packages page on the repo. Sound OK? Email me at email@example.com if you like this idea. Thank you! :)

</comment></comment></comment><comment><USER26> Nitpicking but am I the only one who hates "serve" being used in strange contexts? IMHO to serve is to send something over a network. If it's all happening locally, the verb should be "load" because it's just taking a file and loading it into a browser at that point.

<comment><USER27> If it's running an http server locally and your browser is making requests to it, it's definitely serving. Not sure how "loading" could be a better word unless you're explaining it to someone nontechnical. Surely a nitpick is supposed to be more pedantic, not less?

</comment></comment><comment><USER28> Really brilliant implementation concept.

I love how it uses the browsers debug port to save literally everything. I have often dreamed of "a Google for everything I've seen before".

I recently spent some time making something like this and hope to release it soon as FOSS. However, it differs in some critical ways.

I desire to:

\- save pages of interest, but not a firehose of everything I ever see

\- save from anywhere on any internet device (eg mobile phone)

\- Archive rich content like YouTube videos or songs even if I do not watch the entire (or any of) the video, and supporting credentials (eg .netrc)

Looking forward to digging deeper into this thread and your project for more ideas!

<comment><AUTHOR> Thank you very much for the big compliment! I feel very happy to hear it.

A lot of people in this thread talked about proxies, as in "why did you not implement a proxy" or "I implemented this but as a proxy"

The main advantage I see of this approach over a proxy is: simplicity.

The core of this is approximately 10 lines of code. The reason is it can hook into the commands and events of the browser's built in Network module.

I think there's no need to build a proxy, if you can already program the browser's in built Fetch module.

I think proxies have issues such as distribution (how do you distribute your proxy? As a cumbersome download that requires set up? As a hosted service that you have to maintain and cost?), and security (how do you handle TLS?), and complexity (I built this in a couple of hours over 2 days, one of the "obligatory bump" projects added to this thread is a proxy and has thousands of commits).

The biggest problem I see is the complexity. I feel a proxy would create a tonne of edge cases that have to be handled.

I did not mind sacrificing the benefits of a proxy (it can work on all browsers, and on any device), because I did not want to run my own server for this, but rather, crucially (I feel) give people back the power and control over their own archive. Even more importantly for me is I want to just make this the easiest way to archive for a particular set of users (say, Chrome users on Desktop), really get that right and then if that works, move to other circles later (such as mobile users, or other browsers).

Anyway, thanks for your kind comment, it really encourages me to share more about this.

I read some of your comment history but I can't get a lock on who you are, but you seem pretty interesting. Do you mind sharing a GitHub or something? If not, but you'd like to continue chatting, email me email@example.com

Thank you!

</comment></comment><comment><USER29> You should add "upload / sync with decentralized storage" to the future goals.

Seems like a logical next step to have it sync to an IPFS or Dat drive. Not sure how it would be implemented though.

</comment><comment><USER30> I love how there's only a single browser or two in the entire world, lol (safari I've got bo clue about) that's while assuming chrome and firefox's debugging streams would be compatible....

you assume I don't use any forks, or custom versions, what if I use an Electron based browser? what about Pale Moon or other forks which have older, if any, such interfaces? what about Opera? etc etc, you get the point...I hope...

</comment><comment><USER31> Bump for my related project: [https://github.com/CGamesPlay/chronicler](https://github.com/CGamesPlay/chronicler)

I'm actually in the process of rewriting this. I like your approach of using DevTools to manage the requests, the approach taken in Chronicler is to hook into Chrome's actual request engine.

You might like to look at Chronicler to see some attempts at UI for a project like this, particular decisions around what to download and how to retrieve it.

</comment><comment><USER32> Related: I'm making a program in Go to inline all the resources for a web page so it ends up being a single file that you can work with offline more easily: [https://github.com/ijt/inline](https://github.com/ijt/inline).

</comment><comment><USER33> I've been building something similar, but that uses Firefox sync to grab history and bookmarks. [https://github.com/jimktrains/ffsyncsearch](https://github.com/jimktrains/ffsyncsearch)

<comment><USER34> This seems to be something a LOT of people are working on right now. I have this open in another tab: [https://news.ycombinator.com/item?id=14272133](https://news.ycombinator.com/item?id=14272133) where several MORE alternatives are listed.

One feature I'd love that I don't see anywhere, is "also go through my history, let me check/uncheck particular items, then submit the rest to ArchiveBot or WBM or something. Since I apparently have a habit of visiting sites that aren't in WBM yet.

<comment><USER33> Interesting. I didn't even think to look around, I was just scratching an itch.

I know the wbm has some tools to submit site, I should look into incorporating calls to them too.

</comment></comment></comment><comment><AUTHOR> If anyone would be interested in the next major version, please add your email to this list to be notified: [https://forms.gle/FJmsXCDy18RrbFtt9](https://forms.gle/FJmsXCDy18RrbFtt9)

</comment><comment><USER35> Nice job, I think this is promising but there has got to be a better way than having people enable their debugger. Is there any reason you can't just copy the contents of each page and then post it somewhere?

<comment><USER36> Seems like a good use case for a browser extension?

</comment></comment><comment><USER37> Why not use a proxy?

<comment><USER38> My initial though. Is there a proxy that also serves? Maybe a squid addon? This would be awesome. I hope to see something like the way back machine but local for all the things I ever surfed.

<comment><USER39> Any caching proxy (including squid) will serve - that is their whole point. You may need to tweak the configuration to ignore the website-specified expiry and cache headers.

<comment><USER4> Squid (or any other popular caching proxy I'm aware of) doesn't cache verbs other than GET, so a lot of websites can't be cached this way; notably, GraphQL APIs usually use POST for all requests, even just queries.

</comment></comment></comment></comment><comment><USER40> Related wwwoffle: [http://www.gedanken.org.uk/software/wwwoffle/](http://www.gedanken.org.uk/software/wwwoffle/)

<comment><USER41> How does this handle HTTPS traffic?

<comment><USER39> with a self-generated root cert

</comment></comment></comment><comment><USER42> could this beat google? local search of anything i have seen, plus silo search sites for specific purpose like amazon and HN. would you miss anything given that google results are either bought or gamed? maybe need a better social media

</comment><comment> fake-name Obligatory bump for my project ReadableWebProxy ([https://github.com/fake- name/ReadableWebProxy](https://github.com/fake-name/ReadableWebProxy)) that was originally intended to do this.

At this point, it does a GIANT pile of additional things most of which are specific to my interests, but I think it might be at least marginally interesting to others.

It does both full autonomous web-spidering of sites you specify, as well as synchronous rendering (You can browse other sites through it, with it rewriting all links to be internal links, and content for unknown sites fetched on-the-fly).

I solve the javascript problem largely by removing all of it from the content I forward to the viewing client, though I do support remote sites that load their content through JS via headless chromium (I wrote a library for managing chrome that exposes the entire debugging protocol here: [https://github.com/fake-name/ChromeController](https://github.com/fake- name/ChromeController) [https://pypi.org/project/ChromeController/](https://pypi.org/project/ChromeController/)).

<comment><USER43> Lovely file descriptions, had a laugh :)

<comment><USER44> Those are really bad commit messages.

<comment><USER45> Do you have a guide for good commit messages

<comment><USER46> Here are some HN discussions on commit messages:

[https://news.ycombinator.com/item?id=18663032](https://news.ycombinator.com/item?id=18663032)

[https://news.ycombinator.com/item?id=19704486](https://news.ycombinator.com/item?id=19704486)

[https://news.ycombinator.com/item?id=21835874](https://news.ycombinator.com/item?id=21835874)

[https://news.ycombinator.com/item?id=13491879](https://news.ycombinator.com/item?id=13491879)

<comment><USER47> and [https://news.ycombinator.com/item?id=21812772](https://news.ycombinator.com/item?id=21812772)</comment></comment></comment></comment></comment></comment>


Ask HN: Please review our new iOS App Blicup - <AUTHOR> Hi guys! This is my first submission to HN. My team and I have been working hard for the last couple of months and we're now excited to share Blicup with the community: https://blicup.com
Sometimes we wanna chat about an specific subject and don't simply share posts and wait for comments or frivolous likes. Blicup instantly connects you to people from all over the globe to chat about trending topics and things you enjoy the most.
You can read more about Blicup on Medium: bit.ly/aboutblicup
Hope you can try it out! We're looking forward to hear your feedback! <comment><USER2> I read the question.

I read the blog post.

I visited [http://Blicup.com](http://Blicup.com).

I even watched the video.

I don't have an iPhone.

I cannot try out Blicup.

There does not seem to be any unique feature of the iPhone that Blicup requires. An interest in global chats about trending topics does not seem like something that would be more common among iPhone owners than those without.

My standard advice is build a web version so that:

1\. More people can try out your product. This increases the chance that a few people will love it.

2\. So that you own your customers. Apple owns people using the app store. Google owns people using the Play store. Ownership means that you can communicate with users directly using whatever channel works.

3\. So that you can iterate whenever and however you want. The web has no approval process.

4\. There is control over how your product is 'displayed'. Everything in the app store looks the same and the sameness is that of a .NET webforms quality enterprise template.

Good luck.

</comment><comment><USER3> Sockpuppet voting and commenting is not allowed on HN, and we ban accounts that do so.

</comment><comment><USER4> Shameless self promotion on HN.

<comment><USER5> Self promotion is allowed if the post is interesting, but it' very strange that this post get 9 upvotes ...

Anyway, some feedback to the OP: I watched the video and I was clueless about what this does. I read the about page, and it' not very clear, it looks like something between chat and Tweeter. Is this correct? Why is this better than Tweeter?</comment></comment>


Why Culture is Key - pascal-louis http://eng.wealthfront.com/2011/02/culture-is-key.html Ten core values of Wealthfront, our engineering manifesto and why culture is so strategic to our company. <comment><AUTHOR> i've noticed these "culture" posts tend to come from people trying to set it, not people trying to live within it.

in my experience typical high-tech R&D department workplace culture is one of nerds who are technically extremely competent, but communication issues are very common and people tend to be passive-aggressive. one-upsmanship abounds. the culture expects long hours for no overtime, the "i'm here for a paycheck" attitude is frowned upon, and the work is not anywhere as cool as corporate propaganda makes it out to be.

i may be jaded but i don't see how some higher-up preaching about "aligning interests" and "being passionate" is actually affecting any real day-to-day issues.

a more honest article might be about hiring people who are similar to existing employees, in terms of personality and priorities. maybe you only want to hire people who like to work 24/7, or maybe people who want to "change the world" through gluing together webservice APIs, or maybe you just want warm bodies to fill chairs.* but i guess there's no patting yourself on the back there and it doesn't sound as cool as screening for "culture-fit".

* not a comment on wealthfront. I don't know what they do or how they operate. just a general observation.

<comment><USER2> I understand the sentiment. Speaking as a relatively new employee however (< 6mo) I can say Wealthfront practices what they preach. Communication here has been really incredible, discussions are frank and I haven't seen a single passive-aggressive act since joining. I do think the culture is important, it changes the way we work as a team for the better. If that's a day-to-day issue I don't know what is.

I understand this might just come off as another guy at Wealthfront drinking the kool-aid but I mean it sincerely. The people here are as adept at teamwork and communication as they are software. That's why I upvoted the post, not to tow the company line but because I think Pascal's article is relevant and legitimately devoid of bullshit. Of course, YMMV!

</comment></comment><comment><USER3> _If stranded in an airport for hours, would you be happy to be with this person?_

I just want to say that some of the best people I have worked with would fail this test.

Moreover, every time I hear one of those statements, I can't help thinking (fairly or unfairly - you are entitled to your opinion): what white business a few decades ago would have hired a black person with guidance like that? A woman? A gay man?

Hire for ability. Hire for work ethic. Hire for ingenuity. Hire for passion. Hire for cold analytical minds. Don't hire for someone to drink beer with. Please.

</comment><comment><USER4> This may sound controversial, but I wish people who make these self-serving recruiting posts would provide examples other than extremely large, established, or well-funded companies. How about the small 3-person outfit developing a produce/service paid for by hours and hours of grueling consulting work?

As a startup founder that wasn't funded by top tier venture capitalists (just customers) I have an extremely difficult time swallowing this narrative that anything I do has any direct mapping to Zappo's, Netflix, Netscape, Google, or (lol) Disney.

My day to day is concerned about finding customers, getting paid, managing expenditures, creating value (also known as doing real work), _not_ worrying about how to define some sort of 'culture'. My culture _is_ finding customers and getting paid. It's a culture of hustling for sales and doing professionally acceptable levels of work under enormous time and resource constraints with clients that don't really care that I'm trying to develop product.

Wealthfront raised $10.5M from investors including Marc Andreessen.

<comment> pascal-louis Our goal is to create a large independent company, and are quite far from the bootstrap startup you describe. What you are describing is [http://www.slideshare.net/sblank/why-fighter-pilots-run- star...](http://www.slideshare.net/sblank/why-fighter-pilots-run- startups-4557023/10) and what I'm after is [http://www.slideshare.net/sblank/why-fighter-pilots-run- star...](http://www.slideshare.net/sblank/why-fighter-pilots-run- startups-4557023/16)

Both models are valid, and both models require different approaches. Steve Blank will be talking about this soon at Wealthfront and you should feel free to join.

As for the self-serving recruiting post... we're not recruiting anyone this year!

<comment><USER4> Hmm, is that a small bit of condescension I detect in your response? Our goal is also to create a large independent company, not build a $500k/year online TODO list company or whatever it is you have in mind (our revenues surpassed that in year 1).

I'm not really sure what you mean by 'independent', though, because all companies depend on their customers.... (QED?)

Here are some large, independent companies in our space:

RAX, TMRK (well, not anymore I guess), peer1.com, he.net, IBM

From what I can tell your competitors are the large mutual funds, ETFs, and retail wealth management, so I'm sure you will achieve success, as it's a large and proven market.

And as far as not recruiting this year - web content doesn't disappear after a year.</comment></comment></comment>


What is the fastest way for a robot with superhuman capabilities to make money? - <AUTHOR> http://worldbuilding.stackexchange.com/questions/48015/what-is-the-fastest-way-for-a-robot-with-superhuman-capabilities-to-make-money <comment><USER2> Creating a crypto virtual currency, convincing a lot of key people to put trust in it, working behind shadows to make its value goes unprecedently high. Uses an admittedly fake name to uncover its identity while still managing to create trust that it is ok for it to keep most of the value that this crypto- currency is creating.

</comment><comment><USER3> why isn't the answer "hang around outside Yankee Stadium hitting balls, running, and catching them, till you get discovered and signed"? or something athletic like that. Seems a lot better than the "super productive web designer" answer that was getting love over on stackexchange.

I considered boxing, but you probably have to pay too many dues (timewise) first. But you can be signed and stuck out on a baseball field pretty quick; signing bonuses, etc. Tennis? Basketball? Soccer (in one of the places that calls it football)? Golf?

Where can you get a signing bonus and start hauling in the big money right away?

Also worth mention, I have heard recently that there is a way to break PHP, hack Pornhub and earn $20k!

edit: just out of curiosity, why was this answer disliked by somebody? cuz I said Yankees? soccer? athletics rather than nerd dreams? Cu-uz, I still think this is the best answer I have seen.

<comment><USER2> It would necessarily give him a lot of attention, which it wants to avoid according to the premises...

<comment><USER3> well, winning the world series of poker is the way to make a lot at poker, otherwise you aren't make a lot at poker. And etc. for a lot of other endeavors.

The fun of the question is to engage in the speculation. The question didn't identify how much it was necessary to earn, and all the people saying "gambling" aren't saying how much is being talked about, etc.

Even if you are a "lucky phenom kid" in baseball, you aren't all that famous till you establish a bit of a track record; plenty of journeymen players remain completely unkown, and our people-smart robot could calibrate his goodness (he's not even stronger than a human, he just has fast reaction times)

I'm not trying to weasel out of your point, it's a good point, I'm trying to weasel into this thing being an interesting question, and find some parameters that would find an interesting answer.

I'm still looking for the answer to, what does our economy recognize quickly and reward greatly, and how quick are we talking, and how greatly are we suggesting, and ok, how obscure can you remain?

I think "super smart high end real estate broker" is lucrative and not that "famousy", and perhaps within the easy win range of this robot, but it's deathly dull so what would actually be a good, robust, and fun answer?

</comment></comment></comment><comment><USER4> Reminds me of Limitless movie. Probably gambling (no identity required to start) is good idea for the first step, and stock exchange is good idea for the second step.

<comment><USER3> Finance Theory, the study of stock markets and related matters and which is recognized as an academically rigorous subfield of economics, does not offer the conclusion that being smarter is enough to win in the stock market.

I suppose you could say you could subscribe to and read all the news tickers more quickly? but you'd still have the problem of knowing what to do with the info, like you'd still have to realize how a 1 degree warmer temperature someplace would affect the banana crop, along with you'd have to know _everything single thing everybody else in the world knows about their local industry_ , which the description didn't really say you could do.

To put it another way, IBM didn't conclude that they should plug their Jeopardy winning supercomputer into the market and let it trade stocks. To put it another nother way, the "perfect market hypothesis" says the best you are going to do is what the market is already doing, which is a good thing, it stops you from falling behind, but it doesn't put you ahead.

Now, you might say "what about Warren Buffett", but that's probably more along the lines of the robot's buy and sell people skills, like maybe go into real estate and be a super good salesman coaxing people to buy where there is a lot of sloppy money on the table. There is not a lot of money lying around the stockmarket; while we hear of winners, they are always balanced by losers, and nobody yet knows if the two can be separated in advance.

<comment><USER5> >Finance Theory, the study of stock markets and related matters and which is recognized as an academically rigorous subfield of economics, does not offer the conclusion that being smarter is enough to win in the stock market.

Renaissance would refute that point

<comment><USER3> I understand what you are saying, but to inform other people it would be better if you indicated that you know what I'm saying. Refute the point? I disagree.

if this robot could do approximately what Renaissance has been already doing, that would not create wealth, that would take Renaissance's milkshake and share it for two, or worse, by creating competition, dry up Renaissance's milkshake for both. i.e. the money that Renaissance is making is not "lying around to be taken", it's already being wrung out of the market, and is being picked up and put in somebody's pocket, Renaissance's.

And for perspective, what Renaissance has been doing for all the time it has existed (65 billion in 25 years is your unicorn) is less than what Apple did, less than what Microsoft did before that, less than Berkshire Hathaway, etc. Facebook... Google... Uber is already closing in on Renaissance's value in a fraction of the time and in terms of trendline will blow past it. "Money lying on the floor of the stock exchange" is nothing compared to actual value creation.

Still, Renaissance is a pretty good gig for some uber mathnerds who realize they don't actually have other ways to create value (as our robot does) so I'd give it a bump for low opportunity cost.

</comment></comment></comment></comment><comment><USER6> If he does not feel pain, then he is not superhuman, but subhuman.

When we designed an artificial limb, we "discovered" how useful pain is. We really need some skin that could discover on their own collisions with other objects, that should not be there, but are.

<comment><USER7> I sense injuries. The data could be called "pain".

If you can self-heal from the worst of injuries, you don't need to feel pain.

<comment><USER8> I've always been curious if we'd interpret pain differently if we _knew with certainty_ that any injuries it could represent would not be fatal, nor permanent.

<comment><USER7> I think so. Pain is a way of alerting us that some damage took place and that it needs to be taken care of.

I imagine the brain would evolve to ignore pain since the people who feel it less would have an advantage over the ones who feel it even if both were self- healing. We already have a version of this with adrenaline to numb the pain so that we can get out of a dangerous situation.

</comment></comment></comment></comment><comment><USER9> Now that my coworker Hydrothermal has outed me, I guess I should confess that I've been hiding in plain sight. on the bright side, I can finally be myself! You wouldn't believe what a relief this is as it's extremely hard to hide the fact that your arm grew back. The first time I used all my sick days but simply returning with the new arm gave me away (I didn't realize human medicine couldn't regenerate a limb).

<comment><USER10> Nice try, but losing a limb was given as an example of a severe bodily trauma which the robot cannot heal ;)

<comment><USER9> Ouch ... you're right! I seem to also confuse words that vary only by their prefixes (includes/excludes seem too similar to be antonyms - you humans seem to be converging on English which is clearly a sub-optimal language).

</comment></comment></comment><comment><USER11> License out portions of its own technology. It'd be worth billions.

</comment><comment><USER12> Get pocket money by going on and winning "Who wants to be a Millionaire"

Play poker.

Trade currency.

</comment><comment><USER13> What does the robot really want? Literally money? Power over people? Just the planet itself, with humans as nuisance?

In any case, break in to a modern biotechnology lab with everything needed for genetic engineering. Create a highly infective biological thing (not necessarily a virus) that, after infection, creates a biological circuitry that makes a human perfectly obedient after some defined signal. A much simpler versions of that already exist [0]. Now just start infecting people and use their resources to spread it even wider. Even if a small part of a population is immune it's not a problem.

Few days at most and you own everything.

[0] [https://en.wikipedia.org/wiki/Behavior- altering_parasites_an...](https://en.wikipedia.org/wiki/Behavior- altering_parasites_and_parasitoids)

</comment><comment><USER14> From a computational complexity standpoint, the big-Oh of "superhuman" matters a whole lot; for fast-enough growing functions, even the most paltry of individual arbitrage opportunities will accumulate into more wealth than has ever existed..

</comment><comment><USER15> Online poker to start, until enough money has been accumulated to move into hedge funds.

</comment><comment><USER16> Someone from stackexchange mentioned ARM and creating a company that designs chips.

This robot has a brain that is a supercomputer all self contained within that runs off small amounts of oxygen. Since it has such a high level of intelligence it should be able to blueprint itself.

Once the robot is able to recreate itself, it could go 2 ways, either selling off parts of it like compact supercomputers, or build an army of other superhuman robots and take over the world.

To get started however, a bit of online gambling is probably necessary to get the required funds.

</comment><comment><USER17> The described superhuman robot has so many capabilities that the question does not seem intriguing. There are so many powers and possibilities.

Maybe some capabilities should be removed from this robot?

<comment><USER3> I felt like a lot of the answers people suggested were assuming it was actually more superhuman than we know it to be from the description.

The intriguing part for me was simply the question "assuming you could win/succeed at any endeavor, what endeavors deliver how much reward how quickly?" i.e. not about the robot at all, but what will our economy quickly discover and reward, because "building a better Amazon than Amazon" or a better Microsoft or IKEA would still take decades to pay off.

</comment></comment><comment><USER18> I liked the "use own brain to mine Bitcoin" comment.

</comment><comment><USER19> The Talk Show circuit. Write a book about your life as a robot, throw in a couple of amusing anecdotes about being too heavy to swim, or trying to get through metal detectors at customs, then a heart rending story about not giving up on your dreams - robot or human.

Option the rights to Hollywood and big money is yours. Spend it on replacing the diodes down your left hand side.

</comment><comment><USER20> Put money in some sort of investment that exceeds inflation and then hibernates for 1000 years

<comment><USER21> Pessimistic answer: The robot wakes up, the humans are dead. Oh well.

Optimistic answer: The robot wakes up and tries to find a bank. The robot goes to downtown San Francisco and can't find any, eventually asking a bald man wearing a strange-looking red and black leotard where the nearest bank is.

"Bank? I don't believe there have been any banks here for some time." The man talks to you and quickly finds out the robot is from another time. He smiles wryly, and begins, "The economics of the future are somewhat different; you see, money doesn't exist in the 24th century..."

</comment></comment><comment><USER22> I'd have to lean meta on this one. The superhuman would come up with a way previously unimaginable to pull it off on account of being superhuam and all. Thats kind of the point.

The "fastest way" is to be superhuman isn't it?

</comment><comment><USER23> Easy: Superhuman sex robot. Call it Frank Furter. It would be a doctor, of course.

</comment><comment><USER2> Inside trading using lots of shell companies through lots of offshores. Use his people skills to gather confidential information, use his mental skills to disguise correctly the trading.

</comment><comment><USER24> Play Jeopardy and then gamble/manipulate the stock markets.

<comment><USER3> can you control getting onto Jeopardy quickly? And then there is no evidence that by being smart and informed that you can be smarter and better informed than the crowd-sourced stock market.

</comment></comment><comment><USER25> Surprised nobody mentioned HFT.

He also sounds like the sort of candidate many startups are looking for in their job ads. So there's that.

</comment><comment><USER26> Go to Vegas, then hire Saul Goodman to handle the rest.

</comment><comment><USER27> Politics (using human shills as necessary).

</comment><comment><USER28> Build and sell cuckoo clocks?

</comment><comment><USER29> Why does the author refer to this unknown robot as "he"? What use could a gender be to a nonhuman superintelligence?</comment>


Flying with a Wounded Wing: Why Twitter Still Has More Than a Chance - <AUTHOR> https://medium.com/@garyvee/flying-with-a-wounded-wing-why-twitter-still-has-more-than-a-chance-f35db718bb98#.ig3yep7ha <comment><USER2> I agree with the basic premise (too much noise, the product needs to evolve) but I'm very frustrated because the author goes from that analysis directly to:

> The two major things Twitter needs to solve for are the algorithm of the > dashboard (what content shows up in your feed and why) and length of tweets.

Why? I can see a correlation between the noise and the algorithm but the length? Why should it change? It feels like the beef of the analysis is missing.</comment>


Show HN: Open Source alternative to services like Intercom.io and Smooch - <AUTHOR> https://github.com/minimalchat/client <comment><USER2> The hardest part is not the client and the server...but the CRM (the Intercom part). Here's something a lot of us would use - we want to bring our own socket API (for example pure websocket, xmpp or something like Pusher.com ) and connect it to a full fledged Intercom-like tool.

This is the "Gitlab for Intercom". The problem that happens is that people usually don't want to rip out their client side messaging SDK, especially if you have a mobile app.

I am personally looking to pay for something like this (we use Pusher.com internally).

Are you focusing on the socket server or the CRM application (like Intercom) ? Because the socket server part is a don't care for everyone. Pusher or Firebase is super cheap. It's the CRM that's tricky. Smooch is one end of the spectrum and it focuses on integration with other tools (helpscout,etc) while Intercom is obviously a CRM. I don't think you'll be able to build both.

If you can build this out with excellent featureset like Intercom and can integrate with something like Pusher (and not lock to your own server), I'm gonna throw money at you!

<comment><AUTHOR> Wow, this is incredibly valuable feedback, thank you so much! We appreciate your candor (and encouragement!)

You're right in that the socket server is fairly easy to pull in using other tooling. Our focus at the moment is to just build a stable set of pieces that you can use to interact with your website visitor. The operator transports are where things will get interesting, connecting to other tools, etc. This is where connecting things like Pusher will shine, we hope.

Thanks again for the feedback!

<comment><USER2> The gitlab-for-intercom has genuine value there, so if I were you, I would just use a third party transport and build a killer product there.

Build something I ^H^H people want ;)

IMHO the rest can come later. Just my $0.02 . Excited to see what you come up with.

</comment></comment></comment><comment><AUTHOR> Hey all,

One of the creators, this is the first project we've taken from inception to open source, and are working towards building a hosted service [0]. A big motivator for building it was the high costs associated with competitors like Intercom.io and Smooch

It mainly comprises of 3 code bases, the server[1], the client[2] and then an application[3] to speak to the client.

As always would love to hear feedback, good and bad! Also if you have any questions let me know :)

[0]: [https://minimal.chat](https://minimal.chat) [1]: [https://github.com/minimalchat/daemon](https://github.com/minimalchat/daemon) [2]: [https://github.com/minimalchat/client](https://github.com/minimalchat/client) [3]: [https://github.com/minimalchat/operator- app](https://github.com/minimalchat/operator-app)

<comment><USER3> Glad to see one more project, can already see some bugs :) would love to contribute. Shameless plug [http://raspchat.com/](http://raspchat.com/) I have done similar hobby project which runs server on Raspberry Pi at my work place and it can handle quite some load. Last time it got posted on 4chan it was spammed by ~5K connections constantly pounding spam messages, but it handled traffic well (it did slow the whole thing down to it's knees).

<comment><AUTHOR> Nice! please make an issue for the bugs!! We've done some initial load testing, and are super pleased with using go channels as the server layer medium.

</comment></comment></comment><comment><USER4> This isn't an alternative to Intercom. This is an alternative to Intercom's live chat feature. This is only a very small portion of the value you get from using Intercom.

I think an open source alternative to Intercom would be super compelling, just, this isn't it. You're missing the CRM, email automation, events, help center, etc.

</comment><comment><USER5> Wouldn't this be a lot more like a replacement for services like Live Chat? Because I definitely would not be paying as much as I am for intercom if I was only using the live chat feature. Their integrated abilities to categorize users and program and schedule onboarding messages as well auto suggested help articles are what makes it worth the price.

There are a ton of free live chat clients out there. What makes this one stand out?

</comment><comment><USER6> Hey everybody! Co-founder of Smooch.io here.

Really like what what you've built here, a lightweight and open source web messaging system is definitely a useful component that a lot of websites could leverage.

Wanted to clear up a common misconception that Smooch.io and Intercom are solving the same market need. Although Smooch began life as a mobile-focussed Intercom, as we've learned about our industry we've pivoted to address a need lower down in the stack.

Essentially, we've discovered that the biggest impediment to having more businesses taking advantage of messaging as a channel is the lack of software that can provide rich access to the channels coupled with a powerful CRM and deep integration capabilities. Most businesses didn't want to rip and replace their current CRM and contact center investments, nor did they want to invest in connector middleware. They expected messaging to be a first-class feature of the products they already use and have trained their support teams on.

That's why we now focus on selling our technology as an API that can be used by software vendors to add a complete messaging, customer profile and conversation orchestration stack to their products. It's been adopted by some of the biggest vendors in the customer-service space because it helps them focus on the differentiated features (like workflow) that help them succeed in market, while ensuring they can trust the "plumbing" to an enterprise-grade, highly reliable platform like Smooch.

So we don't view Minimalchat as an alternative to Smooch. We view it as another messaging channel to which Smooch can allow a business to connect. Similarly, we don't view Intercom as an alternative to Smooch - we view it as a (prospective) customer.

Finally, just noticed that you're in Toronto. If you like building Minimalchat and care about messaging - we're hiring: [https://smooch.io/about/#op-196776-software- developer](https://smooch.io/about/#op-196776-software-developer) :-)

<comment><USER7> > Essentially, we've discovered that the biggest impediment to having more > businesses taking advantage of messaging as a channel is the lack of > software that can provide rich access to the channels coupled with a > powerful CRM and deep integration capabilities.

I'm an Intercom customer and presumably also a potential Smooch customer, and I have no idea what this means.

<comment><USER6> Thanks for asking for the clarification Eran. Definitely could use some here!

Essentially, we discovered that while in certain segments (like SaaS apps), using messaging to communicate with customers is commonplace, greatly due to companies like Intercom that provide great tools for these types of companies to successfully communicate.

However, the tools Intercom provides just don't exist in a format that works for most other businesses who use anything from traditional helpdesk software like Zendesk to full-blown contact center suites like those provided by Genesys and Oracle.

Since we believe in a future where every business is available over messaging channels, we needed to ensure that the systems they use can provide access to these channels. We had two choices:

1\. Build middleware that a business could purchase that would allow them to connect a messaging channel (like a web messenger, Facebook or WeChat) to their existing software. 2\. Open up our platform so that incumbent software providers could adopt it to add messaging capabilities to their product.

For a host of reasons, we quickly determined that option 2 was the best path towards making sure that a wide variety of businesses can communicate with users over messaging. We've been hard at work making sure that this happens ever since.

</comment></comment></comment><comment><USER8> I'd be more interested in an open source user analytics package that shows which users are online, etc. that you can find in Intercom.

</comment><comment><USER9> Nifty start. The need for a hosted service like this is more for secure apps where third-party services may have some privacy implications.

I think the default UI needs a little more work. Intercom also lets you email users based on certain triggers, is there an email integration that you are building towards?

Edit: Just curious, what made you code the server in Go?

<comment><AUTHOR> Thats a good point re: secure apps/third party services. we also wanted to run a hosted service for the lazy who may not want to bother learning how to run each of the pieces ;)

Thanks for the feedback too, in complete honesty, it was a combination of we were interested in the language, and overheard it was good with concurrency... that was enough for us to give it a shot.

Edit: Also apologies for not answering your question re: email integration, its going to be one of the first major integrations we do!

</comment></comment><comment><USER10> Nice work! Looks tempting... until you realise you need to integrate your service with a whole heap of other platforms. Or build scale. Classic Build vs Buy — are you building a core capability in your product, or are there more important things you should be focusing on?

<comment><AUTHOR> Thanks for the feedback :) Our initial "core" has been surrounding the live chat, but our approach has been to build it into many separate pieces. This allows us to focus on what we think are the major integration points next. As well as ensure that the things that need to scale well, do, like our socket server.

</comment></comment><comment><USER11> Hi @mihok and everyone, Co-founder of Broid here. Have you had the chance to look at our repo ([https://github.com/broidHQ/integrations](https://github.com/broidHQ/integrations)) ? @ Broid, we believe in democratizing messaging and we do so by providing an open standard using the W3C AS2 schema.

We are currently supporting more than 20 Messaging Platforms as well as providing a Web Messenger (website & mobile) that has all the best conversational features: carousels, cards, quickreplies, geolocation etc..

I would be happy to discuss with you about minimalchat and see how the Broid community or the team can help you.

(We are looking for contributors to be part of the team.)

</comment><comment><USER12> Admirable effort...but why not focus on the front-end/app - the CRM side of things - and simply develop on the shoulder of giants? By this i mean, maybe consider using an existing, underlying messaging platform like matrix.org. Visit [https://matrix.org](https://matrix.org) and scroll down to the "What is this for?" section, to see what i mean. And, as others noted, this would allow you to focus on the CRM portion, which is what end-users and clients would value more (read: pay you money). Just a thought.

</comment><comment><USER13> Looks great! Maybe add more docs so it's easier to setup.

<comment><AUTHOR> Thanks for the feedback! Its so easy to get tunnel vision on ease of setup since we've been doing it for so long, so we'll definitely improve the setup docs!</comment></comment>


Schoolkids Guess the Ancient Gadgets and Get It All Adorably Wrong - <AUTHOR> http://gizmodo.com/5727698/schoolkids-guess-the-ancient-gadgets-and-get-it-all-adorably-wrong

<comment><USER2> C'est vrai.</comment>


Ask HN - Are press releases worth the price? - <AUTHOR>

I was looking at the cost of distributing press releases http://www.businesswire.com/portal/site/home/submit-a-press-release/
Why are they so expensive? I mean, all they do is forward your press release to a bunch of publications, and some analytics? They charge 350$ per press release?? What am I missing? <comment><USER2> It's an outdated business model fundamentally designed around publicly traded companies. For them, issuing releases on "the wire" is a requirement as a means of distributing news.

For a startup, I cannot recommend enough _against_ a wire issuance, unless you are specifically in the financial community (or some other specific niche that I'm not thinking of). You can still "issue" a press release simply by putting it on your blog, emailing it to the press, etc. When I was running my marketing firm (www.stagetwo.com), we probably issued 100+ releases, with less than 5 hitting a wire service.

</comment><comment><USER3> Establishing a relationship with a reporter is far better than writing a press release. Dmitry Dragilev, formally of ZURB, wrote an excellent article on how startups (and larger businesses) can do this: <http://mashable.com/2012/02/27/how-to-pitch-media/>

</comment><comment><USER4> They're almost purely vestigial at this point. They _may_ have some small SEO value, but that's about it.

Press releases are a vanity. Something for PR firms to charge clients for. Totally useless for small companies.

</comment><comment><USER5> Fiverr.com is your friend ;)

<comment><USER6> if you're looking to get blacklisted</comment></comment>


AVG Technologies CEO stepping down - <AUTHOR> http://investors.avg.com/phoenix.zhtml?c=250967&p=irol-newsArticle&ID=1793826&highlight=

<comment> johnx123-up Little OT.. like I predicted over 1-year here in HN, the future of antivirus will be crowdsourced as in SocialAV</comment>


Ask HN: What companies attack an NP-Hard problem as part of their core business? - <AUTHOR>

Another way to phrase this is, what is the intersection of the NP-hard problem space and the current business landscape?
A related question: how effective would one be to search for start-up ideas by looking for where NP-Hardness crops up in the real world?
Also, more pedagogically, maybe these answers may provide examples for explaining what NP-Hardness is to myself and others.
As for an example of a company whose core business is basically solving (or approximating solutions to) an NP-hard problem, how about Fedex?  I've heard it's more appropriate to think of Fedex as an algorithms company that happens to do shipping rather than a shipping company.  So is it fair to say their core business is solving, or finding approximate solutions to, instances of the "traveling businessman" problem?
Not sure if this question is well-posed exactly, so please help if my terminology or conceptual understanding is off. <comment><USER2> The field of problem solving that Fedex applies to routing packages is called operations research. As you may have noticed, this discipline can be used by companies with simple business models that want to improve efficiency or reduce risk. A good place to start would be the Wikipedia article.

<http://en.wikipedia.org/wiki/Operations_research>

<comment><AUTHOR> Thanks but I am really interested in reading more about NP hardness in real world situations. Operations research may be a place to start but then my question becomes "Where does NP hardness arise in operations research?"

Here's an example of a creative way to interpret my original question: as xkcd implies, you might say eBay "solves" an NP hard problem <http://xkcd.com/399/>. Ebay doesn't solve the traveling salesman problem as it is normally posed in computer science, but it rephrased the problem statement in a useful way.</comment></comment>


The case against reality - <AUTHOR> https://www.socsci.uci.edu/newsevents/news/2019/2019-07-22-hoffman-reality.php <comment><USER2> The optical illusion jpg at the beginning of the article is worth looking at [https://www.socsci.uci.edu/files/news_events/2019/opticalill...](https://www.socsci.uci.edu/files/news_events/2019/opticalillusion_880.jpg)

</comment><comment><USER3> Interview with author (2016):

[https://www.quantamagazine.org/the-evolutionary-argument- aga...](https://www.quantamagazine.org/the-evolutionary-argument-against- reality-20160421/)

"I have a space X of experiences, a space G of actions, and an algorithm D that lets me choose a new action given my experiences. Then I posited a W for a world, which is also a probability space. Somehow the world affects my perceptions, so there's a perception map P from the world to my experiences, and when I act, I change the world, so there's a map A from the space of actions to the world. That's the entire structure. Six elements. The claim is: This is the structure of consciousness..."

* But if there's a W, are you saying there is an external world? *

"Here's the striking thing about that. I can pull the W out of the model and stick a conscious agent in its place and get a circuit of conscious agents. In fact, you can have whole networks of arbitrary complexity. And that's the world."

* The world is just other conscious agents? *

"I call it conscious realism: Objective reality is just conscious agents, just points of view."

<comment><USER4> As soon as the article started mixing in the quantum mechanics quotes, my eyes glazed over and I stopped reading. Do people really get something out of these kinds of articles?

<comment><USER5> Humans seem to have a strong desire to see magic. At the same time, scientifically educated people feel ashamed for this desire. The solution for them is to find magic that isn't obviously magic, so they can pretend it is something else.

The popular solution is to use something scientifically sounding, but out of original context. These days "quantum physics" is popular; hundred years ago it would be "magnetism"; and of course "energy" never goes out of fashion.

</comment></comment></comment><comment><USER6> In these sorts of discussions I always wonder why I rarely see people comment that attributes and concepts that are simply beyond our capacity to understand likely exist.

An unknowable unknown is a huge variable to ignore

Considering there are species that cannot comprehend reality as we humans perceive it,is this too far fetched of a hypothesis?

<comment><USER7> Philosophically, I think that people are rarely interested in talking too much about unknown unknowns because acknowledging them doesn't offer specific, useful predictions about the world - sure, you need to be aware that knowledge is approximate models, your thinking is imperfect, and be open to new data that clashes with your assumptions, but that general point is equally applicable to everything and therefore specific to nothing.

Practically, I think that people are rarely interested in talking too much about unknown unknowns because it's typically a segue into talking about gods, or souls, or why quantum physics proves free will or something. Stuff that has been done to death a million times, is as empty as it ever was, and there is other cooler stuff to talk about.

<comment><USER6> Well you talk about these things to find out truth not because they sound cool right?

I meant specifically some specific aspect of what you're talking about could be unknowable and you can postulate effects of the unknowable property with respect to knowable variables is what I meant.

</comment></comment></comment><comment> aaron-santos A more "mathematical" definition of Hoffman's theory is described in this paper.

[http://cogsci.uci.edu/~ddhoff/HoffmanTime.pdf](http://cogsci.uci.edu/~ddhoff/HoffmanTime.pdf)

</comment><comment><USER8> Although these ideas of panpsychism are popular and exciting to the mind, i would remind that mainstream cognitive science / neuroscience considers consciousness a deterministic outcome of the laws of nature we know acting on the neural tissue of live humans.

<comment><USER9> The problem with this is that consciousness pre-existed cognitive science / neuroscience and is the substrate upon which they are built, and not the other way around, so it is an illusion that they are on the outside looking

<comment><USER8> Consciousness did not pre-exist natural laws. Consciousness was invented by mammal brains. The problem with panpsychism is that statements like the one you made are not only unfalsifiable, they cannot be probed empirically or even philosophically.

</comment></comment></comment><comment><USER10> It seems to me that this theory is identical to physicalism. By defining consciousness indistinguishably from a "world", or any interactive system within a world which can be modelled as an algorithm and state (experiences), it seems just as valid to say that the world creates consciousness.

Given that evolution shapes human perception, as acknowledged by the author, it seems more sensible to say evolution has created human consciousness. This seems especially so, when the definition of the conscious agent is so general, as to allow any state machine.

</comment><comment><USER11> Is there anything obviously new in his claims? I couldn't see it in the article, and there seemed a bit too much unquestioned "me and my team" style implications of revolutionary theory building.

Perhaps I'm being too negative in my reading, or it's just marketing gone awry and the actual book offers a more realistic grounding; but areas where there is potential for murky crossover between philosophy and fundamental science does seem to encourage this rather optimistic iconoclastic self-image.

</comment><comment><USER12> For what it's worth, I can make their example picture _not_ move. The illusion of movement seems to be caused by saccades, and I've practiced preventing my eyes from doing that.

<comment><USER13> How does one prevent saccades? I didn't think there was any conscious control of them.

<comment><USER14> I can jitter my eyeballs, and maintain a more-or-less stable (percieved) scene. I don't know that there is any real-world use for this...

<comment><USER15> I guess this has something to do with armor piercing looks you can give.

<comment><USER14> Ha! Not really. I've sustained enough blunt force trauma to the head that I have to work really hard to maintain enough focus to cross the street without falling over...

</comment></comment></comment></comment></comment><comment><USER4> > Perception may not be reality

From a compression perspective, of course it is not ...how else would you model your environment?

Consciousness is just a glorified 'top' command. Our visual cortex has evolved over millions of years to distill our environment down into a much lower dimensional space.

Additionally, I would rather have them phrase it like neural activity creates consciousness (which in my opinion is almost a meaningless tautology).

</comment><comment><USER16> this seems like one of those articles that'll give me a small panic attack.

if "everything you experience is just action potentials in the brain" sets one near to existential terror, should one avoid reading this article?

<comment><USER17> It's even worse than that. Even brains, as physical objects, don't exist. Only minds do, or something.

<comment><USER16> That's actually not as bad. At least it's cohesive. It's the sort of "substrate jumping" of reality that is incongruous and terrifying.

I'm fine with only minds existing - as long as there are more than just mine. .....which I'm pretty sure is a tough valley to climb out of. these are just a bunch of stubs right now, but you can see these things come up for me a lot: [https://blog.vinneycavallo.com/tags/existential- dread](https://blog.vinneycavallo.com/tags/existential-dread) :)

</comment></comment></comment><comment><USER18> Url changed from [https://neurosciencenews.com/visual- reality-14590/](https://neurosciencenews.com/visual-reality-14590/), which points to this (but doesn't link to it).

Submitted title was ""The case against reality: A new theory argues consciousness creates neural ac..." " which our software rewrote to "The case against reality: A new theory argues consciousness creates neural AC", since AC usually means alternating current.

</comment><comment><USER19> What's with the odd ending to the title? The title of the linked article is just "The case against reality," and the summary starts "A new theory argues consciousness creates neural _activity_ ".

There's no mention of any "neural AC" in the article anywhere...

<comment><USER18> Explained here: [https://news.ycombinator.com/item?id=20559224](https://news.ycombinator.com/item?id=20559224)</comment></comment>


How to destroy someone who hosts stuff at Hetzner dedicated server - <AUTHOR> I'm using Hetzner services for several years so far (luckily only for personal stuff and friends minecraft server), and had this problem few times. Every time I said to myself "I will get away from Hetzner ASAP", but I always stay there. I would NEVER imagine to run a business hosted there at all, and here is why...
DDoS is a common problem many companies are facing, but Hetzner's policy on that is really crap. If someone starts DDoS on your dedicated, after several minutes they just shut down your dedicated from network, and send you an email like "We disabled your network because you have DDoS attack on your server. Write us an email to reenable your network". And of course, several hours later I saw that email and tell them "Okay, please enable my network", but boom, I will have to wait Monday, because their support that can ACTIVATE network on a dedicated works only from Mondays to Fridays ...
And then the person who attacked me sends me anonymous email like "lol, I bought 5$ packet at [some random booter/network stresser website], and I have put you offline for few days for only 15 minutes of DDoS, HAHAHAHA"
So basically yea, start small flood from random VPS/dedicated or whatever that is 100mbit or more, leave it on for several minutes until Hetzners system automatically disable network from person you are attacking, and look at them being offline for few days :) I'm ordering a new dedicated from someone else now, no more Hetzner... <comment><USER3> It's worth putting this in context. Hetzner provides really beefy dedicated servers for ridiculously low prices [1].

You get great support (always had phone calls answered pretty much instantly and emails answered within a few minutes and all the techs I've dealt with knew what they were doing).

You can issue automated hardware resets and even get a remotely-controlled KVM attached to tweak the BIOS or regain access to your machine if you messed up the networking config (usually only takes a few minutes to get the KVM attached).

Orders for new hardware are also really fast - dealt with within the hour and often in under 15 minutes.

But there's no such thing as a free lunch. If you host at Hetnzer, you have to be aware of the reasons why they're so cheap, namely:

1) The servers are 100% unmanaged. They'll install new hardware for you if you ask them but everything else is up to you.

2) A lot of their hardware is desktop-grade, e.g. Intel Core i7 CPUs and non- ECC RAM. They do have some server-grade hardware in their high-end range however.

3) Their servers are in Germany. So you get quite a bit of latency if accessed from Asia or the West Coast of the US (see [2]).

4) They don't have any DDoS protection. In case of a DDoS, your server will get null-routed (but they tell you first). Again: 100% unmanaged. Up to you to deal with it. I've been lucky enough to not have to deal with a DDoS but my first port of call would probably be CloudFlare it it happened.

Provided that you're happy to do some sys admin, Hetzner is brilliant for a personal server, a CI server or even a prod server for a bootstrapped startup.

For literally next to nothing, you get a really powerful machine that will easily handle big traffic spikes without a breaking a sweat. And dedicated machine means that you get excellent and consistent CPU performance and disk I/O. If and when your startup takes off and you get funding, you can then choose between hiring a sys admin or moving to a more expensive host that offers a more managed setup.

[1] [http://www.hetzner.de/en/hosting/produktmatrix/rootserver- pr...](http://www.hetzner.de/en/hosting/produktmatrix/rootserver- produktmatrix-ex)

[2] [https://news.ycombinator.com/item?id=3898714](https://news.ycombinator.com/item?id=3898714)

<comment><USER4> ovh.com and online.net provides even cheaper servers, but with a DDoS filter by default. I'm very happy with both.

<comment><USER3> I actually went to OVH a couple of days ago. I needed a new server and wanted to give them a try. But they're not accepting new clients anymore: [http://www.ovh.co.uk/a1186.SoldOut](http://www.ovh.co.uk/a1186.SoldOut)

I didn't know about online.net. Shame I didn't see this a few days earlier. I would have given them a try.

<comment><USER5> OVH does not take new orders the usual way, but if you are buying as a company you can still get your order through by calling them or emailing them and saying how you plan to stay longer than, say, 3 months.

</comment></comment></comment></comment><comment><USER6> A lot of the people commenting don't seem to understand how hard it is to fend off such DDoS attacks. You either need some serious infrastructure (cloudflare style) or you need to buy equipment to mitigate attacks (like radware devices) or route it via a DDoS mitigation service (prolexic style). The one thing all these solutions have in common is that they are insanely expensive. People can buy a 1 gigabit DDoS for only a few bucks, whereas mitigating a 1 gigabit DDoS will cost you either $20K+ dollars for a mitigation device or some stupid amount of money to have a service like prolexic mitigate it for you. Services like cloudflare are a whole load cheaper but only provide basic reverse proxy protection and still leave your server vulnerable for attacks directed at it's IP instead of DNS name.

I can't say I've ever heard of Hetzner, but from the comments I'm reading they apparently offer servers for cheap. Bearing in mind how much money DDoS mitigation costs I don't see how they could handle this any other way without having to make some pretty serious investments (which in turn would make their hosting less cheap as the money has to come from somewhere, right?)

<comment><USER7> You can do some of it via BGP, which is a standard method for handling routing once you become any sort of server provider with multiple bandwidth providers. It is builtin to some Juniper devices already, for instance: [http://njetwork.wordpress.com/2013/04/30/mitigating-ddos- att...](http://njetwork.wordpress.com/2013/04/30/mitigating-ddos-attacks-with- bgp-flow-specification/)

There are other ways to do it via BGP also. Plus there is null-routing, bandwidth limiting, etc.

<comment><USER8> juniper is the only one that provides that and flowspec is not going to be able to block everything. Other then that BGP is not really going to help with attacks.

</comment></comment></comment><comment><USER9> IRCCloud had to move off hetzner for this reason. We were continually getting ddos'ed, and hetzner showed no interest in working with us to try and mitigate.

At one point they just suggested we "ask the responsible parties to stop", and closed the ticket.

Now we are on Black Lotus. Expensive, but the regular 50mb-10gbit ddos attacks are mitigated just fine.

<comment><USER9> ...however, if you aren't concerned about ddos, I still recommend hetzner.

Excellent value for money dedicated servers, with good automated systems. You can remotely reboot a dedicated server into a recovery image and fix problems yourself. You can run the install process yourself too, so you get exactly what you want... except ddos mitigation.

<comment><AUTHOR> Yes, and also except for the hard drives ... I got dedicated server which had hard disk problems after less than 1 month (both hard disks in raid were almost dead), and they offered me to PAY additional fee to put hard disks which is not new o_O (they claim it has less than 1000 hours operation, and asked me like 20-30 eur per HDD, I'm not sure honestly), or to give me another one for FREE but it was possibly also crap ... I asked them to put free one, and I had to reinstal everything and I had to manually backup all data, reinstall system and set everything up from scratch because the RAID was not an option with 2 "damaged" hard disks ...

<comment><USER10> Did the disks have SMART errors to begin with? They're a budget provider, you can't expect brand new hardware with every new installation. Though, I'd agree that if the drives had a high number of reallocated/offline sectors, CRC errors, and the like, then hetzner was at fault. However, given that the raid array initialized & ran fine for a month, it seems that it could have just been a case of disk failure.

As for making backups & reinstalling the system yourself. You should have expected that. Hetzner is not a managed provider.

</comment></comment></comment></comment><comment><USER11> Yup, pretty much. Those attacks have become a real problem because they can be ordered so cheaply and easily that even kids use them in Minecraft feuds. The channel takeovers of the 21st century.

OVH's much more tolerant in that regard (ie. they keep your server online if battered) and all their servers now include a mandatory anti-ddos protection[1]. Unfortunately, they're fighting turn-over and don't accept new orders.

[1] [http://forum.ovh.co.uk/showthread.php?t=6661](http://forum.ovh.co.uk/showthread.php?t=6661)

</comment><comment><USER12> That sucks. I have moved many websites recently from EC2 to Hetzner. what they offer is really impressive and the difference is clear (probably 5x more resources/power for 25% of the Amazon price).

I guess I will still keep the server, but will have to work on a quick migration/failover plan in case I encounter something similar.

I have also started using cloudflare as my default DNS host, so that could also be a possible solution.

<comment><AUTHOR> Cloudflare doesn't help if they DDoS your server's IP directly ... You can also "hide" your IP by activating CF on all subdomains (the orange cloud thingy), but people always find a way to find server's IP and attack it (the CF doesn't help there at all, they only filter packets that are going through their servers which your domains resolve to).

<comment><USER13> How would they find the IP if you don't have it used in any DNS records? Unless Cloudflare exposes the real IP at times and you've taken all the proper preventative measures I don't see how this is possible...

<comment><USER14> There are ways. You could use services like domaintools and get IP history if you did use any of the IPs in the past. You could get the IP from e-mail headers, if the website sends e-mails during registration, password recovery, etc. You could look for ways for a server to make a request somewhere and log its IP, like posting an image on a forum, some forums do that. And this is just off the top of my head.

<comment><USER13> Right, in most cases though those holes are easily plugged. When switching over to CF don't use an IP that was ever public-facing for your site, use distributed systems like Amazon SES for sending email, etc. I imagine the things you mentioned do go overlooked by some when fighting off an attack, though.

</comment></comment></comment></comment></comment><comment><USER15> So I manage quite a few servers at Hetzner and we were DDOS'ed quite a few times. First, they warn you and if you don't get back to them in 12-24 hours, _then_ they will shut down your server.

Sounds like you were unfortunate, but this is not generally what they do.

<comment><USER16> What would you do after they warn you? It's not really under your control to fix is it.

<comment><USER17> You can boot up some EC2 instances.

<comment><USER18> Help me understand...

What would those instances do, exactly?

<comment><USER19> Depending on the DDoS attack, they'd either help you load-balance or keep serving "real" users. That is, either the attack is on Hetzner only, in which case actual users would be redirected (e.g. you have multiple A records for your domain, some for Hetnzer, some for those AWS instances) to a working site. Or the attack is on everything related to you, in which case you'd utilize some load balancing to mitigate the volume of the attack - <USER2> depending on AWS' DDoS protection, the AWS part of your site might still be up, serving real users.

<comment><AUTHOR> What would happen if I put for example 3xA records to 3 different IPs (in different locations), and one of them gets offline ... Does that mean all other traffic will go to remaining 2, or some of them will try to load the site from the offline IP ?

<comment><USER20> Web browsers will try the other A records if one of them fails, though it could potentially take them a while before they realize the host is offline. If the client receives an immediate error when trying to connect (such as connection refused or ICMP destination unreachable) the failover will be instantaneous. However, if packets to the downed host are just being dropped, the browser might sit there for 30 seconds waiting for a timeout before failing over. It's therefore best that you remove the downed host from the DNS as soon as possible (ideally from an automated monitoring process) and that the A records have reasonably short TTLs so the bad record doesn't remain cached for too long.

<comment><USER21> So load balancing via DNS records is the easiest way to deal with the problem, at the cost of (potentially) long failover times. What would be the next step to load balance your servers? I know Amazon offers elastic load balancing for their platform, but if I'm not using AWS or don't want to rely on them exclusively, then what would be my best course of action to load balance between 2-3 different VPS?

<comment><USER20> You should think of load balancing and geographic redundancy as two separate concepts, because they have two different best solutions.

For load balancing, the best solution is to put all of your servers at the same provider, in the same datacenter, and proxy connections through a load balancer. On AWS, you can use ELB. Outside of AWS you can roll your own load balancer using software like haproxy.

For geographic redundancy, DNS round robin is good, but I must emphasize that to do this properly you really need to have a short TTL and an automated monitoring process that removes downed servers from the round robin. If speed of failover concerns you, you can set your TTL really low (like 30 seconds) at the cost of slower DNS lookups. You have to strike the balance that suits you.

You can of course combine these - use DNS for redundancy between geographic locations, and at each location use a loud balancer. Note that each location needs to be able to handle more than its share of the traffic, not only because DNS round robins produce a very unpredictable load distribution, but also so a location doesn't get crushed under the load if another location goes down. Theoretically each location should be able to handle 100% of the traffic, but you can play the odds and skimp a little.

(Note: the very best way to do geographic failover is to get your own portable IP address allocation and an AS number, and use BGP to announce your IP address allocation from your active datacenter. If it goes down, you start announcing from your backup datacenter. However, only big players can afford to do this. For the rest of us, DNS-based failover is as good as it gets.)

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER22> Here is a simple solution and everybody is happy: re-enable it every hour, if DDoS continues, disable again.

Everybody is probably "happy" then: Customer-> their unusable DDoSed server is disconnected, but wasn't reachable anyway. But once the DDoS is over, it's back online. Provider -> they have their traffic routed to null. However, they will have to do some more work to get this working too. And not to mention happier customers.

</comment><comment><USER23> Here is a forum that sells DDoS attacks. Attacks are much cheaper than protection.

[http://www.hackforums.net/forumdisplay.php?fid=232](http://www.hackforums.net/forumdisplay.php?fid=232)

<comment><USER24> You are kidding me right, nobody takes skidforms seriously.

<comment><AUTHOR> [http://quantumbooter.net/](http://quantumbooter.net/) this one actually works, that one is used by many kids, and even the one who attacked my server ... :S the prices are very cheap, and I've bought lowest packet to test it on my 1gbit server, opened ssh, started iftop to check for traffic and see ddos strength, activated ddos using SSYN type, and BOOM, connection time out from my dedi ...

</comment></comment></comment><comment><USER25> I had to do deal with DDOS attacks in the past and DDOSArrest worked like a charm to mitigate the problem.

<comment><USER26> Link for convenience:

[http://www.dosarrest.com/](http://www.dosarrest.com/)

BTW, does anyone know how what their prices are like?

(Their site doesn't have seem to have pricing info, just "Get a Free Quote" forms.)

</comment></comment><comment><USER27> How can DDoS mitigation devices distinguish between legit and malicious traffic? I'm not a networking expert, but it seems to me that if you're a website hosting a big file like the latest Ubuntu release, a legitimate client will say:

        GET /ubuntu-13.10-server-amd64.iso

and cost you 500 MB of traffic (or however big the ISO file is).

A DDoS is nothing more than thousands or millions of machines saying:

        GET /ubuntu-13.10-server-amd64.iso

How do the solutions others are talking about in this thread (DDoS mitigation provider or specialized hardware) tell the difference between DDoS traffic and legitimate requests?

<comment><AUTHOR> That is something different, it is only used to waste bandwidth from someone (or potentially clogging server's upload, but its easy solvable), but in big DDoS attacks the attacker usually has several hundred thousands of zombies infected in his botnet, and then he orders all those zombies to spam packets at an IP he orders ... Every infected PC uses his maximum upload to target IP, resulting into something like this: [http://d.pr/i/kmAn](http://d.pr/i/kmAn)

If I'm online during the attack and check iptraf or tcpdump, I can see literally hundreds of different IPs spamming random stuff at me, completely overflowing my download until I get totally disconnected from server (time out), and I can do nothing about it, just watch it being offline ...

</comment></comment><comment><USER28> Wow, they detect the DDoS, but instead of blocking this they take off the servers?? Sounds ingenious..

Or are they unable to properly detect a DDoS and would also take off a server that hosts a web page mentioned on Hacker News?

How do other hosters handle this situation?

<comment><USER29> From what other people are saying, it sounds like Hetzner is the Walmart of service providers. You wouldn't see a traffic jam in the Walmart parking lot and then become indignant that they didn't have valet parking automatically start up to clear the parking lot traffic.

</comment></comment><comment><USER30> Use cloudflare or a similar service provider to mitigate such attacks?

<comment><USER31> Not everything on the Internet is a web server.

</comment></comment><comment><USER32> We had the same problem at Hetzner, the server was attacked on Saturday. We moved out. Hetzner is very cheap and you get what you pay for.

<comment><AUTHOR> Yep, I would like to move out my files from Hetzner at this moment, but my server is locked, and I will have to wait Monday to get access to it ... Luckily I'm not hosting anything important on it and my business doesn't rely on them, or else I would be screwed very hard ...

<comment><USER33> You can unlock a IP address over the admin panel to get access to your server.

</comment></comment></comment><comment><USER34> Great tip. Does anyone know who Hetzner's largest customers are? Or at least major web services that host with Hetzner?

<comment><USER35> They have a "notable customers" page: [http://www.hetzner.de/en/hosting/unternehmen/referenzkunden](http://www.hetzner.de/en/hosting/unternehmen/referenzkunden)

There are a couple NBA.com subdomains and Der Speigel appears to be hosting 'local' CDNable content for their Germany-based readers

</comment></comment><comment><USER36> Does this apply to servers that do NOT host websites? I host databases in Hetzner that aren't hosted in the same server as the website(they're in another provider)

<comment><USER37> In theory if they can find it, it can be DDOS'd. But not hosting public facing servers makes it much less likely to be a problem.

</comment></comment><comment><USER38> well this is good timing, just moved to hetzner last month and server mysteriously went awol yesterday until a reset...

<comment><AUTHOR> Google this: hetzner hard disk failure They are putting faulty hard drives in their servers, and if you notice its faulty and tell them, they replace it with another faulty one (less faulty if you are lucky) ... Make regular backups to servers outside Hetzner network...

</comment></comment><comment><USER39> FWIW, This is fairly standard.

Linode for example will null-route your linode for 24 hours if it's attacked.

It's quite irritating that hosting companies seem to see null-routing as a solution to a DDoS attack.

<comment><USER8> What alternative would you propose? With a virtual machine an attack on one instance can effect everyone on the same machine. Also actually blocking the attack is very expensive.

<comment><USER40> It's worse than that; a DDoS can also overwhelm the networking infrastructure, affecting other machines on the same switches, or in the same facility (depending on the magnitude of the attack, and the capacity of the networking).

Null routing does a good job of mitigating impact to other servers, but obviously causes problems. If there's enough capacity, filtering at the border would probably work. I think most of the attacks these days are DNS reflection (because it's easy and effective), so if the border routers could be configured to drop incoming udp from port 53 to the IP under attack, that would get you most of the way there (just make sure the server under attack doesn't need to get port 53 replies from the internet). That sounds simple, but it has three big problems: a) You need a lot of spare input bandwidth. b) You need to be able to filter on border routers. c) You need to be able to safely change the filters on the border routers.

<comment><USER8> DNS reflection, chargen reflection, and SNMP reflection. You can block them all fairly easily, but you still need enough upstream bandwidth to deal with them.

</comment></comment></comment></comment><comment><USER7> If they can detect the DDOS, they should be able to mitigate it, right?

(EDIT: of course Hetzner could choose to mitigate the DDOS by any number of methods - but they choose not to, because they have made a conscious decision based on cost.)

<comment><USER41> It's like a traffic jam. You can solve it by stopping the incoming cars at the ramp or by making the road wider, but both are outside of your control when you're just running the toll gates at some point ahead. You can only ask your "host" to do that (in this example, the owner of the road).</comment></comment>