How can someone go off-web and anonymize themselves after a life online? - <AUTHOR> http://security.stackexchange.com/questions/47293/how-can-someone-go-off-web-and-anonymise-themselves-after-a-life-online <comment><USER2> I've become increasingly concerned with this topic. Looking back I feel incredibly stupid.

When I was 16, I was 'paranoid' about computer security. I used gpg, would never dream of giving my real name to any website. I distrusted every website.

Now I'm 31, and you can easily find 5 years of reddit comment history linked to my almost unique real name. Facebook has algorithms that can link my face to my real identity. I carry a location tracking device and microphone in my pocket. I sit in front of a camera all day. Google has all my search history. My ISP knows which sites I visit.

It's just incredible how comfortable I became with sharing this data about myself online. I don't know how it happened. My 16 year old self imagined that I'd have had some awesome technological setup that made me immune to tracking. But the truth is I just got used to the possibility of being tracked, and then forgot about it altogether.

But these NSA revelations have woken me up and I'm now back in that paranoid state and actively seeking ways of protecting myself.

I think many people have had the same experience.

For all the talk about the generations of the future being tracked and recorded from birth, I have to wonder if we're the dinosaurs and our kids will be saying:

"Shit Dad, why did you share all this stuff about yourself, are you stupid or something?"

<comment><USER3> You are oversimplifying. The thing is that it's _good_ to be part of a community. Feels good to use your real name and it's only natural.

Facebook is an awesome service to find old friends, find new friends with the same interests, exchange visual info (sharing your baby's picture with your family) etc.

Being able to sync devices across is awesome. But try that outside Google/Apple and you're against a wall. Of course it can be done, but not for all applications etc.

Thing is that today's technology offers one hell of an options compared to ten years ago. But more than half of those comes with sharing your _real_ data. Not just out of cunning CEO's or the NSA but out of pure functionality: If you don't use your real name in Twitter/Facebook how are other people suppose to find you?

Try keeping contact with friends via email. It's impossible, especially for the tech-unsavvy, while with facebook everything comes easy.

I would like to erase some of my online past, but it's not important. Not that much. What matters to me is:

* Remove all LEGAL powers for someone to harvest, sell, explore your data without me explicitly knowing about it.

* Prevent and heavily punish data abuse.

* Know the system[1]

[1] If you know how a system works (the internet, surveillance, etc) you can circumvent it, anytime. Tor will not help you if you use it to login into facebook. I mean, your _enemy_ will know instantly that you're using TOR, your exit node etc. Being anonymous online, even without someone tailing you it's very hard but it can be done.

The problem with _true_ online anonymity, is that it's not a one-time thing. It's a policy that needs you to be strict. No more forums, new fake emails all the time, registrations, etc. All of us can do that, but are we willing to do so??? I'm not.

We can't turn the clock back in time and undo what we did (except maybe if we are able to travel at light-speed... but that's not the point I guess). We have to focus on how to make governments and companies much more accountable about what they are doing. Many laws need extra elasticity (copyright) while others need to get much tougher (abuse of power by any authority).

Also some advancements at protocol level like user-predefine packet (path) routing, DNS and internet decentralization (away from USA), would not be a bad start.

<comment><USER2> It's impossible not to over-simplify without writing a 10,000 word essay on the topic.

I'm not advocating disappearing altogether. There's nothing wrong with keeping in touch with my friends on facebook, sharing the odd picture.

I do disagree that it feels good to use my real name. I wish I'd been more careful peppering my various accounts with references to my real name. I'd really love to be able to press a button and have zero results come up on google for my name.

Why on earth should you, random internet stranger, be a few clicks away from finding photos of my baby son on facebook? Who gave you the authority to know so much about me? Well, I did by sharing it in the first place.

Not that I have anything in particular to hide, but I feel very strongly that it should be _me_ who's in control of that data.

I don't think it can be wholly solved by technology or by laws.

I think it can only be solved by behavioural changes in the way we use the web.

<comment><USER4> In my opinion, user24, I like being publicly linked to everything I say or do online. It means that the "me" that exists online is in perfect harmony with the "me" that exists in the physical world. Anything I do online has repercussions for my offline life, and vice versa. It makes me more connected with the world.

<comment><USER2> You don't fear that one day you might be denied entry to the UK for this tweet? [https://twitter.com/bovermyer/status/398163555742396416](https://twitter.com/bovermyer/status/398163555742396416)

<comment><USER4> Nope. I'm pretty content to speak my mind and let others figure out whether what I'm saying chimes with their values or not.

<comment><USER2> That's very brave of you, and I respect that, and only hope that you never live to regret it.

I perhaps have slightly more fringe interests than you, and in the past have been somewhat indiscreet, albeit on obscure forums.

<comment><USER4> I have to lie in the bed that I make, for better or worse, heh. I'm not trying to sell a philosophy here, though - only state mine. You're perfectly welcome to your own, and I'm not going to think any worse of you for it.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER5> I've being trying to make this a HN hit for a long time now, and I will try again as it is related to the topic.

The "name-your-price" web comic The Private Eye: [http://panelsyndicate.com/](http://panelsyndicate.com/) is a great story on a future world where online anonymity became impossible, then people went for real world anonymity, all of us wearing masks and pseudonyms. It is really interesting and entertaining to follow.

I have no connection of any kind with the authors, just a fan of their work. And I really thought HN crowd would love it. But I submitted it a few times and not a single upvote. Ever. Hope to find them some love now.

<comment><USER6> I'm guessing it's because the only thing I see when I follow that link is the long list of "buy now" buttons. No teaser for the comic, nothing to let me gauge the comics worth... just a poorly designed webpage that only seems to want to part me and my money.

I closed the website after about 3 seconds, and wouldn't even have commented except for your seemingly genuine puzzlement at why someone wouldn't want to read it.

<comment><USER7> You name your price, so just pay nothing for the first one, then if you like it, buy the second.

<comment><USER8> It requires thought for the customer to realize they can do that, and thought equals friction in a transaction, it costs you customers.

You could get the same effect without the friction just by offering the first issue free and then putting the "Buy Now" buttons on the following issues. That lets you put the word "FREE" next to one issue, which will indicate to people that it's safe to click on if they're uncommitted. No thought on their part required, which should translate to more downloads.

</comment></comment></comment></comment><comment><USER9> Overheard in conversation at a security conference a few years ago: for many people, it's probably likely that the fingerprint you leave just by the _top ten websites you visit_ is quite distinguishing.

Given that - on top of all of the 'obvious' linkages that are out there (friends, email contacts, identity cards/numbers, payment methods), it's imaginable that for a 'suitably advanced' adversary, you have to be extremely vigilant if you want to detach your new identity.

The reality for many people (including in modern witness protection, I'd imagine) is that it simply isn't humanly feasible to be that perfect - so you have to trust your benefactors to a certain extent (and assume that no-one will get access to any correlation information they might build).

<comment><USER10> I read that in the New York Times during the glory years of The War Against Terror. So yes, our friends at the NSA are on to that one already. So you could go to random cyber-cafes in a big city and still be stalked.

Obviously there is no evidence of _them_ having apprehended an al-qaeda terror operative using this technique, mostly because there is no such thing as an al-qaeda terror operative.

However, there is writing analysis too, 'I'd' imagine that they know you use 'I'd' and you might want to start using 'I would' from now on just to foil their systems. Me, I am going to stop using the phrase 'The War Against Terror'...

</comment></comment><comment><USER11> Reminds me of this video, from The Onion: "Google Opt Out Feature Lets Users Protect Privacy By Moving To Remote Village" [http://www.theonion.com/video/google-opt-out-feature-lets- us...](http://www.theonion.com/video/google-opt-out-feature-lets-users- protect-privacy,14358/)

</comment><comment><USER12> Of course the answer is: you can't.

You could ten years ago. You could give it a good shot now. But in another ten years it's impossible.

We are living in the last age of anonymity.

We're also living in the last age of personalized history. When I tell a story from my teenage years, in the 1980s, I get to tell the story any way I like. After all, it's my memories. Each of us are able to own our version of reality, which was very human and very comforting.

For kids going through their teenage years in the 2020s? Their entire life is just a big dataset: where they went, who they talked to, what they said, how their opinions evolved. Not only can they never leave their old life because of hueristics and fingerprints, they can't even enjoy their own version of how their life went. That's a major change in the nature of what it means to be human.

</comment><comment><USER13> I think it's best to create the public persona you want, and then just have a bunch of stuff out there. People mostly won't sort through all of it.

This isn't me, by the way: [http://www.selfgrowth.com/experts/david_welton.html](http://www.selfgrowth.com/experts/david_welton.html) :-)

</comment><comment> k-mcgrady The biggest problem is probably going to be preventing other people from putting you back on it (tagging you in photos, writing about you etc.).

I recently went through the Google listing for my name and found dozens of profiles and sites I couldn't even remember signing up for. I was able to get most closed successfully. One site I had particular issues with was FriendFeed. I believe I signed up with my Twitter account, which I have since deleted. In otherwords I can't sign in to shutdown the FriendFeed profile. It also seems impossible to get in touch with them so that profile is stuck on the web when I would rather it wasn't.

Once you remove yourself from the first few pages of Google (time consuming but not difficult) your online life is invisible to most people. The only people that will find anything are those that really dig.

P.S. If anyone knows how to get my profile removed from FriendFeed when I can't login please let me know :)

</comment><comment><USER14> The interesting thing about the answers given there, is that they seem to assume you can only have one identity at a time. Why not simply have multiple identities?

Anonymising yourself then simply becomes a matter of picking a new name for yourself and never telling anyone what it is and never doing anything under that name. It's a pretty useless form of anonymity of course, but it does suggest that anonymity itself isn't really particularly useful.

Perhaps anonymous _acts_ are what are really desired? The question of how to act anonymously might provide different answers.

<comment><USER15> The problem is the ease at which the various identities can be identified as belong to the same person.

Very few people have the kind of processes that really protect that online personas from being uncovered.

It's non-trivial to ensure that you always associate cookies , IP addresses, browser signatures, behaviour... with one persona and yet never ever leak that to another.

Bear in mind that computers are so precise in their work here that you only need screw up once to burn that persona forever.

Pseudonymity is achievable, but unless you say nothing and participate in nothing anonymity is incredibly hard to achieve for a single persona.

Anonymity is exponentially harder to achieve when you have multiple personas.

And this is before you get into face recognition of photos, language analysis, gait analysis etc all attempting to to the real you from the very things that are harder to fake.

</comment></comment><comment><USER10> Here is how to do it:

[http://en.wikipedia.org/wiki/Disinformation](http://en.wikipedia.org/wiki/Disinformation)

> Disinformation is intentionally false or inaccurate information that is > spread deliberately. It is an act of deception and false statements to > convince someone of untruth. Disinformation should not be confused with > misinformation, information that is unintentionally false.

> Unlike traditional propaganda techniques designed to engage emotional > support, disinformation is designed to manipulate the audience at the > rational level by either discrediting conflicting information or supporting > false conclusions. A common disinformation tactic is to mix some truth and > observation with false conclusions and lies, or to reveal part of the truth > while presenting it as the whole (a limited hangout).

> Another technique of concealing facts, or censorship, is also used if the > group can affect such control. When channels of information cannot be > completely closed, they can be rendered useless by filling them with > disinformation, effectively lowering their signal-to-noise ratio and > discrediting the opposition by association with many easily disproved false > claims.

<comment><USER16> yes, that's the way I would do it, use pictures of people that looks like you, start tagging yourself in such pics, also change your online behavior gradually, create several online alter egos, and try to erase your old online presence, after a while the online information about you should be worthless.

</comment></comment><comment><USER17> I was scared while reading the best answer:

> Shops know what you buy in what amounts, because nobody buys all the same > brands you are getting fingerprinted constantly. This is used for targeted > advertising, but it can also theoretically be used to track you.

I live in france and I wonder if this is even legal here.

Even if you change you name, if you start buying some sort of the same combination of grocery article, it can lead back to you if they do some statistics on every people shopping on the globe. Even if you use cash, they can still know that someone bought the same combination of articles somewhere.

Not using a smartphone will reduce how you're located. A classic GSM phone can still be located being around a cellphone tower (and more precisely by using triangulation, but it's painful), but smartphones have a tendencies of communicating your GPS coordinates, which allows to track everyone's location at the same time. The amount of hardware required to store all that information must be quite astronomical, but still, a latitude and longitude will be precise and just take 16 bytes at most, which is ridiculous.

</comment><comment><USER18> This is quite an interesting topic. For me, the thing is, I have no idea what I am going to be up to in the coming decades. What if I want to serve in some local comminity position? Or create my own company? All of the sudden, that hasty from-the-hip comment I posted to a forum years ago might actually matter. I find it quite tiring to stress over my online "brand". Locking down my Facebook profile has helped, I will say. Using my real name on HN, especially as someone recently trying to pivot into web dev, has definitely affected the way I post. I am more considerate, which is good. But I also self censor more.

The never-forgetful nature of the web definitely changes things.

</comment><comment><USER19> It all really comes down to how far you want to take anonymity. I think the first thing to do is dox yourself, and methodically go through the websites and both change all info to random unrelated information, and then delete or deactivate, whatever is possible. The next step without taking drastic measures such as new identity, is to simply stop using the internet. And remember to not be included in digital pictures. The abstinence from the internet is what I think is the hardest part, especially if the internet has become an integral part of your life.

<comment><USER20> That means not owning a cell phone. Maybe not using a phone at all. Even possessing a land line or using pay phones might not work.

You also need to use cash. But possibly only make withdrawals directly from a human bank teller, and not an ATM. You'll still show up on bank cameras though. Maybe check cashing and pawn shops is the way to go. Just having a bank account could be problematic, because oft he transaction register.

Using paper cash, you'll still have to consider serial numbers on bills as clues in the long run. Like it or not, the paths of exchange that bills follow can be traced to a degree, using the same imaging software that post offices use on mail, and banks use on paper checks. It's not as reliable for the smaller bills, but for larger denominations, they tend to take short paths exiting the bank (mostly via ATM) and then immediately returning to the bank through business deposits.

So even if you float through space, absorbing all light like a black hole, your gravity will still influence the light-emitting bodies around you.

</comment></comment><comment><USER21> Even if someone can successfully go "off the grid" never using a cellphone or computer, and paying for everything with cash you keep stuffed under your mattress, mentions of your name and images of you will inevitably appear in others' emails, Facebook posts, and whatever else.

In addition, eventually with facial recognition and the high density of security cameras, going anywhere other than the middle of the nowhere will be building a database of your movements.

This is all more than enough to build an accurate profile of you.

</comment><comment><USER22> I think that the question is "How can I _mitigate_ the risk my online identity poses?"

Once on the web I dont think you can get off, the tag I use on HN is one I have used for years, but in those past years I was always a lurker almost never ever commenting (until recently). Now when I search my tag on google HN comments come up, my NSA comments could label me as anti-govt, but thats not the case.

Without going RMS I dont think you can really get off-web if you still want to use the internet in any fashion.

</comment><comment><USER23> You know the answer might be to, you know, not worry about it? I mean it's a two part process (1) are people coming after you with the internet? If no, then (2) pay some attention to your local politics, be wary of advocating for discriminatory policies when it suits you, and remember no man is ever an island.

If yes, well, you kind of did the wrong thing posting on StackExchange about it.

</comment><comment><USER24> The hard part is that in the next decade or so, there is going to no longer be an "off-web".

Your face, walk, voice and probably the pattern of places you go and the smell of your breath will be just another set of tracking cookies for which you will be unable to switch browsers to be rid of.

Say goodbye to "incognito mode".

</comment><comment><USER25> stop using the internet.

then get a fake identity in the same ways that have been possible for hundreds of years now...

that big trail of data on the internet is useless if it points nowhere.

what i never understand is why anyone wants to do this unless they want to do something wrong and get away with it. i also think we all collectively need to get over privacy - if its not about being caught out doing wrong things then all you get is embarassment potentially? i'm not sure i really understand...

i had to fill out a voter registration form recently, its a legal requirement and there was a big part of me that wanted to not do it on principle. however, weighing up the advantages of being a legal member of society over not quickly changed my mind...

<comment><USER26> This brings up an interesting question: who are you on the web anyways? Take my case: I have never signed up for a social networking account on the likes of facebook, myspace, etc. I just don't like to give up that much control about my personal information. To me, just putting it out there, even without the govt snooping on it is giving up WAY too much control over it _. Many websites I used to visit have become "social" over the years though but no matter, I don't use my real name there, or real personal email account. On my android phone I purposely created another gmail account, not linked to my real personal one. I have avoided google+ like a plague and I have seen first hand how nuclear they have gone in order to make people enable the goddamn + account_ _. But the question still remains, who am I online? I 'm still signing up to sites and email accounts that even if they don't represent the whole picture they still have parts of it. It struck me that it may not matter if those accounts don't particularly link to my real name: it is still me doing all that stuff online. Am I just my name or what I do? (Google probably doesn't care that much about my "name", they're still getting a lot of information from "me". To them all of us are probably just a unique ID on their servers.

_I'm not a tinfoil hat kind of guy or anything like that. My efforts are still limited by the convenience of the internet once you sign up to the many services available. People somewhat skilled in tracking people online can probably still track the real me: just because I haven't made much effort in actively learning all I would need to do in order to really hide myself online. I still think that I at least have mitigated it a bit so that not any Joe Schmoe can track me. __First I couldn 't review apps on the play store, now I can't comment on youtube wtf?? but still, you will have to do more than that to get me on, google. I still think that they will end up forcing everyone to g+ in the end, at that point I will seriously consider just leaving google altogether.

</comment></comment><comment><USER27> Only by removing the motives (profit and control _) of those who wish to track and monitor everything. Good luck with that.

_ By "control" I mean enforcement of laws, security(government, corporate, etc. internal/external), oppression, and the like.

</comment><comment><USER28> You can't erase what you've already done online. Moving forward, you could live like a person from 1950. No mobile phone, no internet, pay cash for stuff. That's about as close to anonymous as you can get, if it's worth the pain.

<comment><USER2> I've moved over to cash for 95% of things that aren't bills, it's quite liberating.

</comment></comment><comment><USER29> It's probably easier to change your offline identity. Bureaucracy can be dealt with, plastic surgery results are in continuous improvement... And you can even start a new family, damnit.

But The Internet never forgets.

</comment><comment><USER30> Easier to change your name.</comment>


Benefits of Rising Early, and How to Do It - <AUTHOR> http://zenhabits.net/2007/05/10-benefits-of-rising-early-and-how-to-do-it/

<comment><USER2> As with most Zenhabits posts, it's just filled with feel-good hokey (come on, the Dalai Lama? How is he an expert on good working hours? I'd rather get sex tips from the Pope). No actual evidence. And if you've got flexible hours (which a lot of programmers have), most of it is moot anyway. Late risers can have their breakfast and prepare for the day at 12 o'clock if they want to...

<comment><USER3> The guy is clearly not a programmer. Programmers' most productive time is at night - everyone knows that.

<comment><USER4> I actually get a lot more done early in the morning. All the distractions are gone. No one is sending me email or trying to chat with me on IM.

<comment><USER5> Some people are morning people and some people are creatures of the night. My mind is much sharper in the mornings. Most of my co-workers are better at night.

</comment></comment></comment></comment><comment><USER6> The article is OK, but in my opinion, it gets a few things wrong. And I should know: after struggling with getting up my whole life, I became an early riser (pre-5am). Rather than rehash it, here's a previous comment on the subject:

<http://news.ycombinator.com/item?id=525334>

</comment><comment><USER7> Agree with ojbyrne that this could all be replicated on a late night schedule as well.

How about this: 10 benefits of conforming to your bodies natural rhythm? I've read plenty of research on innate, physical inclination to rise early or late.

Regardless of whether I work late or early, I tend to rise at the same time (unless acted upon by an outside force). Working late means I need less sleep and am more productive.

Of course the downside of this is that the world is on a morning schedule. I am currently conforming to this due to children. I look forward to reverting to a night schedule.

[http://www.time.com/time/health/article/0,8599,1903838,00.ht...](http://www.time.com/time/health/article/0,8599,1903838,00.html)

</comment><comment><USER8> The benefits of rising early are so profound that he decided to do the opposite: <http://zenhabits.net/2009/04/how-to-become-a-late-riser/>

</comment><comment><USER9> Another Benefit - reading an article like this leaves you with a feeling of acknowledgment and pride.

As opposed to my current feeling of guiltily agreeing with all of these points and knowing I won't do anything about it.

<comment><USER10> There are benefits to rising late as well. Some of them are similar to the ones listed in the article. If you are in your room by yourself, 1AM to 2AM tends to be a quiet and peaceful time. A lot of social activity also happens around midnight, and if you are an early riser you may find yourself burnt out and unable to participate by the time midnight rolls around. If you are forced to stay up late a few days in a row and you naturally wake up early as well, you may start accumulating a sleep deficit. A sleep deficit will eventually catch up to you and you'll stop appreciating those morning benefits.

There was also an article posted a while back of a study done that showed people who tend to wake up late performed better on some cognitive tests at certain times of the day, while there was no similar advantage for people who naturally woke up early at any other times.

I do think _this_ article does point out some of the practical advantages that the experiment in that other article wasn't interested in; like how a sunrise makes you feel, the value of beating commute traffic, the value of getting a head start on daily activities, etc. (I like rising early and have tended to for the past 5-6 years. In my case, natural light and no drugs[1] is all it takes)

[1] Alcohol, caffeine, as well as ibuprofen and other painkillers can all help me sleep through a sunrise.

<comment><USER11> Additionally, working (and therefore staying up) late can be out of your control. If you have a job where you have "fire drills" with some frequency then your sleep early/wake up early schedule is going to be hard to maintain. However, if you have a sleep late/wake up late schedule then a last minute fire drill might cut into dinner, social life, etc... but its less likely to cut into sleep.

</comment></comment></comment><comment><USER12> Wow that putting alarm clock away from the bed is ingenious. I've gotta try that.

<comment><USER13> Don't worry, you'll learn to sleep through the alarm clock =P

<comment><USER14> You think you jest...

I have to have 2 alarm clocks on normal ok-if-I'm-late days, and 3 if I absolutely have to wake up by x:00 AM. Each a progressively further zombie walk away from the bedside.

I have 3 for the important days because walking to the kitchen is apparently not enough to wake me.

The problem now is that my third is my cell phone, and if I forget to turn my ringer way up, I can sleep right through that since it's so far away.

I am not a morning person. -_-

</comment></comment></comment><comment><USER15> Total irrational BS. Just like mhd said, that whole site is full of feel- goodey things - doesn't belong on HN. The comments clearly reflect that.

</comment><comment><USER16> If I'm healthy and exercising 3 times a week and eating right, I can wake up whenever I want and be productive the entire day.

If I'm not, it's simply impossible.

This stuff really isn't that complicated, people.

</comment><comment><USER17> When I was younger I used to wake up at 4pm and code till early 6am, pure zen.

Now that I am older I try to go to bed the same day I wake up.

The older I get the earlier I want to wake up, the less I want to sleep.</comment>


Ghostery deploys AI in the fight against ad trackers - <AUTHOR> https://www.wired.com/story/ghostery-deploys-ai-in-fight-against-ad-trackers/ <comment><USER2> Wired must have an article template that looks like this: Company X claims breakthrough-level use of AI to do Y and obtain result Z. This is just a pile of bullshit with buzzwords flying around it.</comment>


Lock-Free Bugs - <AUTHOR> https://research.swtch.com/lockfree <comment><USER2> Here's a fun related issue: A common Linux mutex-implementation strategy to deal with this issue (waker-waiter races) is by releasing a mutex before futex-waking any threads blocked on it. Since the waker doesn't know for sure if a waiter is present at the time of waking, the locking thread may fail its futex-wait attempt, acquire the mutex, and destroy it, freeing the mutex's memory back to the allocator. In the end the waker issues a wakeup call to an address no longer occupied by a mutex.

As a result, the mutex implementation might spray futex-wake calls into arbitrary addresses not owned by the implementation. This means that other libraries in the same process can't rely on accurate wakeups, even if it would otherwise be algorithmically correct to do so (unless it has some way of guaranteeing that its synchronization locations never share an address with other libraries).

If you've ever wondered why pthreads-style condition variables allow spurious wakeups, this is one reasonable situation in which they can occur.

</comment><comment><USER3>
        "By June 1949, people had begun to realize that it was
         not so easy to get a program right as had at one time
         appeared. … It was on one of my journeys between the
         EDSAC room and the punching equipment that the
         realization came over me with full force that a good
         part of the remainder of my life was going to be spent
         in finding errors in my own programs." - Wilkes

That cuts deep

</comment><comment><USER4> Interesting read. It's about a lock;free bug (freeing a mutex while it is locked) in code using a custom mutex implementation.

</comment><comment><USER5> This is not a lock free bug. But it does illustrate how hard it is to get thread synchronization correct. The only bug that ever beat me was a lock free bug in a concurrent skip list. I spent two months of evenings and weekends trying to find it. I wrote special logging tools and visualization tools to try and gain insight. The bug would happen roughly once every hundred billion operations. In the end I decided it is easier to change the entire algorithm than to fix the bug.

<comment><USER6> It's not a lock-free bug, but it is a lock, free bug.

My favorite lock-free bug involved a concurrent ring buffer. At one point the code had to figure out how much data to read, which was either the size of the buffer passed in, or the amount of data available in the ring buffer, whichever was smaller. So the code did something like:

        toRead = min(requestSize, availableData)

But, surprise! The min macro was a custom job using the typical bad implementation of:

        #define min(a, b) ((a) < (b) ? (a) : (b))

But this is no problem as long as the two expressions have no side effects, right? Wrong! Every so often, another thread would write to the ring buffer between evaluating a<b and evaluating a or b the second time. And every so often, this would happen when availableData was smaller than the request size, but then became larger after writing the additional data. In those rare circumstances, the read function would use that too-large availableData as the amount of data to copy, and write past the end of the passed-in buffer. Hilarity then ensued.

I deleted the custom min macro and used the (properly implemented) system- provided one, and all was well.

<comment><USER7> Your bug doesn't seem to be related to the 'min' macro, but to the fact that your lock-free algorithm is not actually lock-free: if one thread can update an 'availableData' variable while another thread can read it, make a choice based on it, and then act on that choice, then that's not lock-free; that requires a lock to keep the read, decision, and action "atomic".

I'm skeptical that using some other 'min' implementation is a real fix to the root of your problem, but without knowing more, I can only speculate.

<comment><USER6> The algorithm is sound given a proper min() implementation. The reader just needs to read at most however much data is available at the moment or however much data the passed-in buffer can hold. If the available data increases during this operation, it's no problem, because the reader is allowed to read less than all of the data available.

For some concrete numbers, if the ring holds 4 bytes, the caller passes in a 6-byte buffer, and a concurrent writer adds 4 more bytes, it's OK if the reader only returns 4 bytes. It would also be OK if the reader sees the new data and returns 6 bytes. It is not OK if the reader returns 8 bytes, which the incorrect min macro could cause.

In this scenario, the incorrect min macro could return 4, 6, or 8, despite the fact that one of the parameters was a constant 6 and the result should never exceed 6. The correct min macro always returns 4 or 6.

<comment><USER7> I see. So you are relying on single-reads of your shared variables. They must be suitably defined to be atomic on your architecture (so no word-tearing, out-of-order visibility), etc.

I would think that if you had all the proper barriers in place to do the reads and writes atomically, though, that you would never have been able to use the min() macro in the first place. You would have had to do your barrier reads into local variables first, and then the min() macro on those local variables would have produced the right result.

Unless I'm missing something. Like I said, it all depends on your implementation, but in general, this sure sounds like the min() macro wasn't at fault. Perhaps your specific scenario is much more constrained (single architecture, no barriers required, non-portable, etc).

<comment><USER6> Most lock-free code I've seen is pretty architecture dependent and often non- portable, and this was no exception. I should also note that it was only intended to be correct for a single reader and a single writer.

Given those constraints, it's not all that difficult. The amount of data in the ring buffer gets stored in a properly aligned word-sized field which the architecture guarantees will have atomic reads and writes. You just need a write barrier after writing new data and before updating the amount of data available, and a read barrier after reading the amount of data available and before actually reading the data. The read code would look something like:

        amountToRead = min(outBufferSize, availableData)
        barrier()
        memcpy(outBuffer, ringBuffer, amountToRead)

(Ignoring the tricks you have to play to deal with wraparound, just to keep things simple.)

This works fine as long as min() actually evaluates to the smaller of the two numbers passed in. The moment it evaluates to a number larger than outBufferSize, you have a buffer overrun.

<comment><USER8> Eh, the portability is actually getting a lot better nowadays. Most consumer and server processors being released in the last few years support double- world compare and swap. Atomic increment, load, store, etc. are also pretty widespread now.

The key is like someone said above that you cannot make your code ambiguous. In C++ especially, the atomic primitives can compile to different instructions depending on how you use them so it's important to be very explicit.

One thing that can be a pain is memory fences. The guarantees can be different on different CPUs so you can run into weird bugs. In my case, generally I compile to target particular CPUs on servers we own so it's not a big deal. But I can imagine for consumer software this can be a huge pain.

<comment><USER6> I haven't seen too much recent stuff. I made this fix probably around 2004, and the code was at least a couple of years old at that point.

We initially targeted PPC Mac, then it eventually expanded to x86 Mac and ARM iOS. All fairly similar when it comes to simple stuff like this, the barriers are straightforward when needed, and no need for DCAS or similar fanciness. (Indeed, no need for any CAS at all.)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> mods (and russ..): perhaps better to change title to "lock;free" or "lock+free" bugs (or something else entirely), T he current title "Lock-Free Bugs" is confusing since the article has nothing to do with lock freedom.

<comment><USER10> I read it as an intentional play on words.

</comment></comment><comment><USER11> I really enjoyed reading it! System programming is hard :)</comment>


China's Surveillance and Social Credit System Alive and Kicking in Berlin - <AUTHOR> https://firstname.lastname@example.com/chinas-surveillance-social-credit-system-alive-kicking-in-berlin-6c2b3b10b197 <comment><USER2> "your use of the app constitutes your consent". Well. That's why there's the, um, _permission_ system in Android. Don't want to give app X rights to Y? Well don't then. Yeah, it might break the app, tough beans.

The other points are valid, though.</comment>


Keyless SSL: The Nitty Gritty Technical Details - <AUTHOR> https://blog.cloudflare.com/keyless-ssl-the-nitty-gritty-technical-details/ <comment><USER2> Interesting how they specifically refer to yesterday's discussion here on HN and over at Reddit, but don't go into the frequently mentioned complaint that despite not having access to the PK itself, Cloudflare can still intercept and modify all cleartext sent between the client and server, which for most intents and purposes means pretty much the same. Maybe not intentionally so, but it comes off as slightly misleading, because many people I spoke to who briefly skimmed the original announcement thought "keyless SSL" would mean this is no longer the case.

Let me ask again: why not offer a setup where Cloudflare only acts on the network layer instead of on the application layer and proxies the still- encrypted HTTPS packets to the destination server? This would mean a customer could not use Cloudflare's caching (CDN) features, but still enjoy their DDOS protection without them being able to see all my customer's private details. If I were the CISO at one of the world's largest banks, that is exactly what I would want.

<comment><AUTHOR> Because DDoS protection isn't just something you do at the network layer. It's important to be able to apply filtering at the "application" level.

<comment><USER2> I understand that there are many different types of DDOS attacks, and that for some of them having access to the content on the application level makes your job of mitigating them a lot easier.

But even without that, would comparing the encrypted traffic patterns for an individual client to other client's patterns (or to that client's traffic to one of the other 2 million Cloudflare websites); and analyzing the encrypted traffic as in [1] not still give you a wealth of information to use? It might have made for a much more rewarding two years of development: even if such a setup would be slightly less effective at mitigating DDOS', it would be infinitely better in terms of privacy and trust.

[1] [http://arxiv.org/abs/1403.0297](http://arxiv.org/abs/1403.0297)

<comment><AUTHOR> In some ways what you propose sounds even scarier, i.e. figure out the best way to look inside SSL encrypted data without having the key.

But in terms of privacy and trust, CloudFlare is already in the position of terminating SSL connections and establishing new ones to origin web servers. We take the protection of that data and the associated keys very seriously. There will be more announcements over the coming weeks and months about how we secure things.

<comment><USER2> I'm not quite following how having just enough insights about the encrypted data to perform DDOS mitigation would be scarier than having full read- and write access to the cleartext. Thanks for the responses though, and definitely looking forward to those blogs.

<comment><USER3> You are implying something fundamental: that the encrypted traffic could be adequately analysed for insight without the need for decryption.

Yet to do so would be to defeat SSL itself, or at least to declare it as insufficient to adequately protect secrets.

What CloudFlare is doing isn't defeating SSL or any kind of attack on it. They are merely working around some prior limitations on requiring access to an organisation's private key.

As a proxy that is charged with DDoS protection (and other types of protection and performance improvements), they _are_ being asked _to_ terminate and work on the unencrypted data to a very strict set of complience by the end organisations, but they need to do this in a way that does not involve possessing or having access to the private key.

Their solution works extremely well given the multiple constraints (technological and legal) that they have.

<comment><USER4>
      > You are implying something fundamental: that the encrypted traffic could be adequately analysed for insight without the need for decryption.
      > Yet to do so would be to defeat SSL itself, or at least to declare it as insufficient to adequately protect secrets.

This is possible through HTTPS traffic analysis, see [0] and [1] for starters. Of course, it's much easier for Cloudflare to do analysis for DDoS protection if they have access to plaintext.

Whether this means that SSL is, as you say, "insufficient to adequately protect secrets" is an interesting discussion to have.

[0] [http://arxiv.org/pdf/1403.0297.pdf](http://arxiv.org/pdf/1403.0297.pdf)

[1] [http://blog.ioactive.com/2012/02/ssl-traffic-analysis-on- goo...](http://blog.ioactive.com/2012/02/ssl-traffic-analysis-on-google- maps.html)

</comment></comment></comment></comment></comment></comment></comment><comment><USER5> To save people a long read, there's nothing particularly surprising in these details. The key is held by the origin; when a computation using the key is needed the origin is asked to perform it over a secure connection. They have improved support for session resumption in a distributed environment (which is even more important now that key computations are even slower), this is commonly done as closed-source, and CloudFlare have promised to open-source it.

Good improvements, and session resumption is important to implement, but nothing groundbreaking.

<comment><USER6> Oh well. Guess we'll have to wait for Homomorphic encryption:

[http://en.wikipedia.org/wiki/Homomorphic_encryption](http://en.wikipedia.org/wiki/Homomorphic_encryption)

</comment></comment><comment><USER7> The Keyless SSL server is open source and available on GitHub - [https://github.com/cloudflare/keyless](https://github.com/cloudflare/keyless) \- although I'm wondering if its dependence on OpenSSL was a good choice.

<comment><USER8> Open source but a rather restrictive license!

<comment><USER9> The source is published but it wouldn't count as a OSI open source licence.

Also note the patent mentioned in yesterday's discussion.

</comment></comment></comment><comment><USER10> I don't get it. What's the point in keeping the key secret from cloudflare whilst providing a key server signing everything it is asked to sign? Isn't this like "Sorry I don't trust you enough to provide you a key to my apartment. But if you need something, just ask the janitor, he'll open the door any time you want"?

<comment><USER11> If you no longer trust the person, it's easier to tell the janitor not to admit him than to rely on his returning the key (and any copies) or to change the lock on the apartment.

<comment><USER10> Is "changing the locks" (revoking the certificate) really so complicated that this "janitor-solution" is easier/cheaper/safer?

<comment><USER11> The CA can revoke the certificate, but since revocation checking in browsers is neither universal nor reliable under attack, revocation isn't a completely effective way to recover from a compromised private key.

</comment></comment></comment></comment><comment><USER12> Why should all of these have a marketing spin to them? By that I mean a catchy name. Keyless SSL is very misleading. I guess after the stupendous success of the heartbleed bug marketing every one now wants a catchy name.

Having said that is it solving any security problems?

1\. Companies don't have to give out their private key to Cloudflare but they still give them the encryption and hmac keys. So nothing really changed from a privacy or data protection point of view.

2\. Now they opened up a new public API on the company's server for cloudflare to use the private key.

3\. Is it simply to mitigate against another heartbleed kind of bug? Whats preventing the same bug to appear in the company's server?

</comment><comment><USER13> Interesting read. Still no information how it compares to other HSM implementations however, and why they didn't use the more well studied PKCS protocols in theirs.

I would imagine the problem was more with existing implementations rather than some problems with HSMs per se, as they probably have much larger latency than you normally have so things like blocking reads start to matter.

<comment><USER14> I guess you mean PKCS#11, but that isn't a protocol. It's a C API. Trying to funnel it over a network would be both extremely complex, and non-standard.

<comment><USER13> No, I was thinking of PKCS#7 which you can run over IP if you want. But it's a complicated set of standards and I'm not sure how they inter-relate. That's why I summarized them as PKCS-style protocols.

This is not some theoretical thing. You can buy these devices off the shelf. If you have worked with PKI you have seen them, or some variant thereof.

<comment><USER14> Yup, I know. My day job used to be writing firmware for nCipher/Thales HSMs :)

<comment><USER13> Cool! We had those at a previous job. Good stuff.

</comment></comment></comment></comment></comment><comment><USER15> Is this all the software necessary to run your own little keyless-like infrastructure with a couple of nginx server not having a key and talking to this reference key server over the Internet?

BTW I probably should have asked yesterday but which CloudFlare plans will include the keyless feature?

</comment><comment><USER16> Sorry if I'm missing something, but shouldn't this be called "Keyless TLS" as CloudFlare uses TLS and not SSL for this program?

<comment><USER17> From the article:

"This may seem confusing at first, but makes sense since TLS is just a minor update to SSL 3.0. Subsequent versions of TLS have followed this pattern. Since TLS is an evolution of the SSL protocol, people still use the terms TLS and SSL somewhat interchangeably."

<comment><USER16> Oops, must have missed that. Thanks! I'm personally a fan of using the new official name whenever possible but I understand the tradition/ingrained nature of SSL.

</comment></comment></comment><comment><USER18> So from a non-technical business owner perspective this means I can offer SSL for free without buying certs or making changes to my server?

<comment><USER19> No. This allows a content delivery network (such as cloudflare) to host content on your behalf without having to have direct access to your private key material.

The main use cases you may be interested in are to 1) make it so that users connecting to you get a faster experience - since they can connect to a geographically nearby cloudflare server, rather than your distant server and 2) make it so that cloudflare can absorb large denial-of-service attacks that your server couldn't otherwise cope with.

<comment><USER18> Ok thanks. So this basically makes it easier and more secure to get set up with cloudflare?

So why couldn't Cloudflare set up what I'm suggesting? Are there technical reasons, or just lack of demand?

<comment><USER20> To serve SSL for your domain, Cloudflare needs a certificate for your domain issued by an accepted CA, which _someone_ needs to buy.

The only way for them to set that up would be to become a CA, get browsers to accept their master certificate and then issue their own certs for their clients.

<comment><USER21> They actually do offer a service where they proxy your plain-text HTTP to end users as HTTPS and provide a certificate. It's extremely convenient.

<comment><USER20> Yes, but you still need to buy and give them a certificate, right? The question included "can offer SSL for free without buying certs", which is not possible, AFAIK.

<comment><USER21> No, they essentially add your domain to a list of domain names in their own certificate.

<comment><USER20> Really? And their CA allows that? That's amazing.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER22> It's one thing to keep SHA1 in old services, but why do they keep pushing SHA1 into _new ones_?

<comment><USER23> From the user POV, nothing is changed. The decision to deprecate SHA1 or not isn't specific to Keyless SSL (ie, Keyless SSL isn't a new product for the users)

From the bank's point of view, where we can say that Keyless SSL is a new product, only 2 cipher suites are allowed:

        ECDHE-ECDSA-AES256-GCM-SHA384
        ECDHE-RSA-AES256-GCM-SHA384

which, as you can see, don't include sha1.</comment></comment>


Algorithmic Trading: The Play-at-Home Version - <AUTHOR> http://www.wsj.com/articles/an-algo-and-a-dream-for-day-traders-1439160100? <comment><USER2> Here's the problem with trying to create your own trading system.

How do you back test it to know that it works. If you back test over the past 5 years then you are only testing your model against a huge bull market.

If you back test over the past 20 years then I'm not sure it helps much as the market of 20 years ago didn't really have any of the major market drives of today's markets, HFT's, huge numbers of hedge funds and the money they bring, and huge passive investing via ETF's, all those factors were there 20 years ago, but they weren't the major market drivers that they are now.

So for back testing you are damned if you do and damned if you don't:(

I think I've said this before but in my experience, a decent trading system is 70%+ market insight. And there are only really 3 ways to get this insight, hard earned experience, applying new streams of data to the markets and applying math to existing streams of data to unlock new insights.

All 3 are being done every day by scores of MIT Phd's.

So if you are writing your own trading models you should start by answering the question of what's your alpha when compared to everyone else....

<comment><USER3> One clarification - these mom & pop prop shops aren't necessarily competing against big baskets of MIT PhDs. If your book size is only $200K, the big players (i.e. scores of MIT PhDs) won't even bother competing with you on the same strategies. Alpha from these strategies may very well be orthogonal to hedge fund alpha.

That said, you're still playing a zero sum game with other mom & pop shops and the general large-scale movements of the market. You need to be confident that you have better information / forecasting ability than the others, or technology infrastructure that enables strategies no one else can execute.

Source: I work in the industry.

<comment><USER4> Yeah, most people don't realize that finance is a zero-sum game. That leads to arms races which over time remove the lion's share of the profit (companies will spend money on a better solution to a problem until such time as a better solution costs more than the value of the opportunity).

I expect this to replicate itself on the low-end as well.

<comment><USER5> The 'zero-sum game' part is meaningless and adds no insight here. It doesn't matter whether you believe trading/gambling is zero-sum or not.

For instance, imagine you have a poker-playing bot at a poker table, where the house takes no cut of the stakes. Zero-sum? Yes. But your bot has to be better than the other players, and win more than its running costs in order to profit. Now imagine the same situation but the house _does_ take a cut. Zero- sum? Not any more. But your bot still has to be better than the other players and its running costs in order to profit. The house 'rake' just increases your operating costs.

I think you are confusing the issue with large scale HFT (where companies pay more and more for slightly faster comms), which is a world away from the trading strategies the article is talking about.

<comment><USER6> Think you missed the point. All he was saying (I believe) that no matter how good your models are it's still a race between the players to get the biggest share of the pot. The more players the less chance you have

<comment><USER5> That's true of many endeavors, and is unrelated to any 'zero-sum' argument.

</comment></comment></comment></comment></comment></comment><comment><USER7> Interesting article but for a different take on a statistical approach to the market, curious if any peeps on HN are into volatility trading?

From what I understand a lot of the "DIY vendors" cater to the equity crowd, meaning people who build their models on technical indicators (MACD, RSI, advancers/decliners ratio, Fibonacci golden ratio retracement, MA); you build your model of some combined signals, back-test it with historical data on some tickers to see if it's promising.

I trade options, specifically selling 1.8-2 sigma calls and puts (with 90%-95% probability of not being in the money) on the major market indices (SPX, RUT, NDX) 45 days out from expirations; so kinda of like an insurance company underwriting policy or a bookie taking bets from both sides.

When the market moves against me (e.g., Grexit), I try to hedge my exposure to that side of the market by buying a longer dated call and put neutralizing the delta of my overall book (kind of like going to the re-insurance market to hedge my book); and also adjust the spread or close it entirely for a loss if risk/reward no longer works out.

But I never trade directional or have any market outlook; and do this systematically and only thing I change is sizing my book depending on how high VIX is (insurance premium/market fear).

This is a well-documented and traded strategy for peeps who follow, say TastyTrade, volatility arbitrage funds or just options in general. Just curious if anyone here also trades this way?

<comment><USER8> If all those TastyTrade strategies work, why isn't it being automated and backed by significant funding? Serious question.

<comment><USER9> Because significant funding requires significant liquidity and strategies that work for an individual trader can exploit being small to find opportunities too small for big money to bother chasing. Small players can enter and exit trades instantly with little slippage, big players move the market and have to play a very different game to open large positions.

</comment></comment></comment><comment><USER10> It absolutely can be done. I know a handful of people who are doing well, or have spun a small hedge fund out of their DIY trading system.

Having said that, it's hard. Most people simply don't have the bandwidth to do it properly. Parsing daily Yahoo prices and having a R script or two somewhere will absolutely not put you in the "I do successful algo trading" camp. Or running TradeStation with a handful of pair strategies or whatever.

So I differentiate between them and people who write more serious automated algos. I'm sure in the first camp there are some that make profits, but as someone said here already, It's usually a zero sum game.

Who are the more serious DIY traders? Hard to define but if they do most of those things I'd say they are in the small camp of home traders that often make money.

\- Willing to fork up the money for good historical intraday data (10 years of US intraday data from Nanex cost about USD 30K). I'm super serious about this, if people even blink an eye here, I'd walk away. They don't understand how important data is.

\- Build a proper security master database, or buy a security master feed. This is a project of blood sweat and tears. Every day securities need to be resolved. Again if building this out of, e.g., Bloomberg data, quite some money has to be invested.

\- Have a sophisticated back-test system where they can simulate models against MANY traded stocks concurrently. Meaning, they can back-test making decision tick by tick, or second by second, against a large universe of instruments, for example 3000 US stocks. This can be slightly eased by doing some daily preselection, but usually good strategies still scan hundreds of stocks real-time.

\- Have the ability to execute this stuff live. IMHO, the article is right to mention Interactive Brokers for this, they are one of the best in this kind of setup.

(Only now are we where most people try to start) \- Have the stats, math and dev chops to find and implement a profitable model/strategy. I have to say though, while still difficult, there are many untapped possibilities to make some money. But once decided on a general direction the "time to market" is still often at least half a year. Many give up before that...

There are some more things I'd look for but this is the main list I'd say.

<comment><USER11> > Or running TradeStation with a handful of pair strategies or whatever.

You can now buy trading strategies from their TradingApp Store. Supposedly, they've been written by professional algorithmic traders.

[https://www.tradestation.com/trading- technology/tradestation...](https://www.tradestation.com/trading- technology/tradestation-platform/extend/tradingapp-store)

[https://tradestation.tradingappstore.com/search/all/Rating](https://tradestation.tradingappstore.com/search/all/Rating)

<comment><USER12> >Supposedly, they've been written by professional algorithmic traders

And discarded by them as not profitable enough for their own use.

</comment></comment></comment><comment><USER13> How is this new? The first time I setup my own linux server was at my parent's house, so I could write a crawler to collect market data in the hopes of applying some AI technique for market predictions. This was 10 years ago. It's a lot of programmers dream to write a little program that can print money. Afterwards I went and worked on wallstreet, and realized how foolish this was.

<comment><USER14> If you learn something while writing it, and it sits and earns a little bit of money without much work, then it's not really foolish. I bet you learned more doing that project, things that probably helped get you that Wall Street job, than you seem to estimate now.

<comment><USER13> Sure there's always value in doing. What I believe is foolish after learning more about what the industry does is the belief that a DIY program can actually make money.

<comment><USER14> You probably can't beat the market. But if you can beat a savings account earning 2% interest it's probably worth it for the learning experience.

<comment><USER13> Well, you'd have to beat 2% + transaction fees + capital gains which would mean you'd have to beat the market.

<comment><USER15> Not quite -- even if you match the market or slightly under-perform the market, it might still make sense if you achieve these results with lesser volatility and more consistent earnings. I do not necessarily want to beat a market which returns 7% on average but loses 40% every 6 years...

</comment></comment></comment></comment></comment></comment><comment><USER16> That's been going on since the late 1970s, when personal computers first became available. It worked back then, because few were doing algorithmic trading at the time. Now, everybody is.

There's new interest in systems which look at external data sources such as news items, rather than just at market data.[1] They have a chance of picking up something before the market reacts. But that's already widely deployed; about 15% of HFT operations were using it by 2013.[2]

[1] [http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2326414](http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2326414) [2] [http://www.automatedtrader.net/online-exclusive/67056/fit- to...](http://www.automatedtrader.net/online-exclusive/67056/fit-to-fade-news- driven-algorithmic-trading-strategies)

<comment><USER5> Algorithmic trading is more than just HFT, where every microsecond counts, and where these small-scale day traders cannot compete.

Your example of analysing new items is a good one. If you can spot a better way to predict stock or currency movements by a cunning new way of processing news feeds, then you could potentially profit from it without having to rely on speed of execution.

People have been using computers to do stock analysis, as you say. But this doesn't make it a solved problem. There's money to be made if you can do it better than the others. Of course, the competition is very tough.

</comment></comment><comment><USER17> I've always been confused why so much of algorithmic trading centers on technical indicators. What about fundamentals? Wouldn't it be easier to cobble together a system that checks for healthy companies that are low in their PEG ratio historical range, and then buy-and-hold? There are super-boring companies with reliable earnings history out there, with stock prices that go up and down throughout the year. So you'd be trading a few times a month, not many times a day (or minute).

Then there are the simple algorithms like 'Just buy index funds', although I haven't found a good resource on how to sector-balance various index funds. Simplest I guess would be 'Just buy VFINX' but that can be volatile and scary sometimes.

<comment><USER18> How easy is for a bot to read fundamentals? I do not know if there is some place to gather this kind of data.

I mean, in "The intelligent investor" the author points out that lots of companies bury important information in side notes in their annual reports, that can totally change their attractiveness.

I wonder how easy is to check these things nowadays.

<comment><USER17> That data set is harder to come by, which really sucks... morningstar has something like that - historical fundamentals - and they charge a ton. But I think Quantopian gives access to it? Not sure if others do. And there might be ways for other services to bundle it somehow across many subscribers.

</comment></comment></comment><comment><USER19> Whenever people tell me they're going to go into day trading, I usually say, "How many people do you know who've become wealthy day trading?" They say, "Zero." I also know zero people who got rich doing at-home algorithmic trading. But hey, there are probably some! That's the nature of randomness.

<comment><USER20> This seems like nonsense to me. How many Phd's are awarded every year? Doesnt mean its a random occurrence... I know quite a few independent day traders who do quite well. I think its about who you know-- i.e if you are working in Finance you'll know more day traders.

<comment><USER21> Have you seen their trade logs?

<comment><USER20> Just a few of them.

</comment></comment></comment></comment><comment><USER22> I once used a Bitcoin trading bot with some luck- called 'Butter-Bot'. It had the ability to amplify gains vs. holding during the times of the upward market. Did quite well- it paid for itself in a month IIRC. Then the market started going sideways & its limitations began to show. It doesn't work at all anymore. I think the devs went out of business after pouring all their resources into a newer version which had better strategies for the evolving market. It's too bad, I was a big fan.

<comment><USER23> Some googling came up with this reddit thread [1] and a quote:

> It's a simple EMA crossover bot, don't buy this stuff. There's a chrome > plugin that does the same for free.

Rhetorical questions to the comments:

What chrome plugin? Have you used it? Is it better/worse than Butter-bot?

Even other people in the reddit thread question why people aren't mentioning the name of the open source project - what is this fight club rules?

[1] [https://www.reddit.com/r/BitcoinMarkets/comments/1nc12p/has_...](https://www.reddit.com/r/BitcoinMarkets/comments/1nc12p/has_anyone_here_heard_of_or_used_this_butter_bot/)

</comment></comment><comment><USER24> If your starting stake is $10K (as in the story), you're wildly better off taking any job, even programming on elance/odesk than working on algorithmic trading...

<comment><USER25> Yeah, that seemed bizarre to me. You'd need to have a huge edge (extremely unlikely) to make money after fees with only $10k. Plus you're very limited in what you can actually trade with that little capital. For example, many options contracts trade for more than $10k.

<comment><USER9> 10k is plenty, you're forgetting leverage; equities I think is generally around 3x, futures usually offer around 10x leverage, forex in U.S. 50x, forex outside of the US, 200x plus.

<comment><USER24> Calculate out what the projected hourly rate might be, after discounting the gains on a broad-based index fund.

If you can get 30% annually instead of 10%, that extra 20% on $10K would be $2000. If you worked just 200 hours in a year (seems a low estimate), that's a $10/hr "job" and you have a non-trivial risk of a substantial drawdown during the year that could put you near out of "business" (as happened in the article).

Borrowing on margin, you're going to pay at least 7.5%, so if you're borrowing to 3x, you're paying 15% on your account balance. (Small margin loans are higher rate.) - [1]

Forex doesn't have the margin loan problem, but you can't generate much income on a $10K yen/USD carry trade.

Round trip commissions will be a much higher drag on a $10K portfolio than a $500K portfolio.

If you have the talent to develop an algorithmic trading system that can 3x the market, you can make more than $10/hr in any number of easier and riskless manners.

1 - [https://us.etrade.com/investing-trading/pricing- rates#margin](https://us.etrade.com/investing-trading/pricing-rates#margin)

<comment><USER9> What it calculates per hour as a job has nothing to do with anything; you said 10k wasn't enough to overcome fees, that's absurd, 10k is plenty to invested and get a healthy return from and far better than 30% annually, hell I made 35% last month and I didn't work an hour for it. Beat the market is a meaningless scare tactic used by those that don't understand being small has enormous advantages over being large and "the market" returns are what large institutions struggle to beat, small traders destroy the market constantly because being small is easier. Returns decrease as capital increases so it doesn't scale up.

<comment><USER24> > What it calculates per hour as a job has nothing to do with anything; you > said 10k wasn't enough to overcome fees, that's absurd

I agree that is absurd, but I said no such thing.

I don't think it's a sensible use of time, but of course you can beat just the fees.

<comment><USER9> You said

> You'd need to have a huge edge (extremely unlikely) to make money after fees > with only $10k.

No you don't. 10k is plenty to make money, equities aren't the only market and paying the spread is trivial. Beyond that you're overestimating the time it takes; what do I care how many CPU cycles my computer spends watching the market? For a programmer, algo trading is trivially easy and a fun hobby so that time isn't work anyway and it's only a few hundred lines of code to work up a strategy, it's not exactly a big investment of time.

<comment><USER24> No, _I_ didn't say that. jeffreyrogers said that.

I said you were better off taking a side job and I still believe that to be true from an economic point of view.

<comment><USER9> My bad on the mistaken identity.

Side jobs require my time, robots don't, they aren't comparable. I'm better off working no side job and letting a robot earn money on the side because that's simply better even if the bot earns less than I would on a side job. Selling labor is not economically preferable to free money.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER26> The latest craze? I guess WSJ has been out to lunch for the past decade.

<comment><USER27> Knowing their track record on climate change...

</comment></comment><comment><USER28> Algorithmic trading from home isn't new at all, it has been around for a while. You can even buy backtesting and live trading software with a DSL that makes it easier to do, e.g. Tradestation or Ninjatrader or as was mentioned, InteractiveBroker provides solutions.

For the most part, people doing this are not on the scale of people doing HFT. The net latency from home to market is much higher and the software lower performance but I do know some people that have done it successfully. Their bots trade anywhere from maybe dozens of times per day to once or twice per week. It is a difficult business though.

<comment><USER29> The primary activity of HFT and algorithmic traders is fundamentally different. Whereas algos are trying to "beat the market", that is find alpha, HFT is trying to "beat the market _makers_ ". They trade dozens of times every second to find the liquidity distribution and arb the flows, not take advantage of technically or fundamentally driven moves.

Needless to say HFT requires several orders of magnitude more resources than a hobbyist algo trader, though there are also giant algos in hedge funds.

</comment></comment><comment><USER30> Request for startup: Make the AWS for algorithmic trading. Strategies are not that hard, but it would be awesome if someone did hard infrastructure work and provide nice API.

So the only thing you need to do is to upload some code and wire some money.

Bonus point for matching money. E.g. if you have your own system you can allow to match your funds. So instead of $20k you can get additional $80k from other investors. Likely it will be regulatory nightmare... but maybe there is some legal workaround.

<comment><USER31> It already exists: [https://tradingmotion.com](https://tradingmotion.com)

There you'll find a marketplace of 3d party trading strategies ready to use (once you pay the monthly fee that goes straight to the developer)

The API if you want to roll your own strategy is: [https://sdk.tradingmotion.com](https://sdk.tradingmotion.com)

</comment></comment><comment><USER32> Probably have better luck writing your own poker-bot.

<comment><USER33> I remember someone making a go at this circa 2006. Didn't work for whatever reason.

Isn't gambling online illegal now?

<comment><USER12> >Didn't work for whatever reason.

The online platforms check for it so its a cat & mouse game.

</comment></comment></comment><comment><USER34> I did a course over coursera "Computational Investing" and tried building my own trading model, but it's hard . you win some and loose some. Needs 100% of effort as it can't be done in parallel with regular job . I need some motivation here .

</comment><comment><USER35> Alpha aside, someone learned the value of unit tests.

</comment><comment><USER36> What're your opinions on algorithmic swing trading based on fundamental analysis / trends / other market measures?

</comment><comment><USER37> Are there any free or cheap sources for historical stock data in a computer- friendly format?

<comment><USER38> I do not endorse trading with it, but for many types of uses, you might take a look at the Yahoo finance CSV "API." It has per-day resolution going back about 15 years or so.

<comment><USER39> I'd take Yahoo's data with a grain of salt. Was tinkering with options prices there, and the data wasn't clean which lead to a lot of false positives (e.g. a lower strike call was selling for 1/2 of what a higher strike call sold for).

Checkout TradeKing. You have to open an account to use it, but (to my knowledge) you don't have to fund it.

</comment></comment></comment><comment><USER40> So... anyone want to share some of these algorithms for predicting stock prices?

<comment><USER9> You don't have to predict the market to trade, you need only follow the market and ride the waves carefully. If the market is going up, buy, going down, sell, you're betting that it'll continue in that direction. More than a few people are rich off strategies as simple as buy when price breaks a 20 day high and sell when price breaks a 20 day low.

<comment><USER41> _" More than a few people are rich off strategies as simple as buy when price breaks a 20 day high and sell when price breaks a 20 day low."_

More than a few people are broke off such strategies, too.

While momentum strategies manifestly _can_ work out, there is no guarantee that the "momentum" is actually present. As my econ prof put it, "The market isn't going; it went."

<comment><USER9> > More than a few people are broke off such strategies, too.

Of course they are, it takes more than an entry strategy.

> While momentum strategies manifestly can work out, there is no guarantee > that the "momentum" is actually present.

There doesn't need to be. Price moves up, down, or sideways, if it doesn't go up/down as expected, get out and look for another opportunity or wait for the market to move, it will move.

<comment><USER41> _" Of course they are, it takes more than an entry strategy."_

If all that is is their "entry strategy" then the strategy they are using is more complicated, which runs counter to your earlier assertion.

 _" There doesn't need to be. Price moves up, down, or sideways, if it doesn't go up/down as expected, get out and look for another opportunity or wait for the market to move, it will move."_

It will move. But it needs to move in your favor more than against you, or you're losing money. There's no particular reason to expect that to be persistently the case.

<comment><USER9> > If all that is is their "entry strategy" then the strategy they are using is > more complicated, which runs counter to your earlier assertion.

Not really, you need to have a position size strategy as well as an exit strategy, all are fairly simple and only a few lines of code. Given a bankroll, how much do you bet on any one trade (2% is trader standard, or if you're really aggressive half kelly), and given an exit strategy, i.e. a stop if you're wrong and an exit if you're right, just compute how much to trade based on the amount you're willing to risk on the trade to the stop position.

So with a 10k roll, you place trades that will lose $200 bucks if you're wrong and you let them ride until trend change hopefully netting $600 or more by the time it's over, perhaps lots more depending on how long the trend runs. Once that trade is safe, i.e. you're risk off, you start looking for another one and you build up your position size without ever risking more than your initial risk.

> It will move. But it needs to move in your favor more than against you, or > you're losing money. There's no particular reason to expect that to be > persistently the case.

Sure there is, markets aren't random, they trend persistently for longer than they should if they were actually random. The EURUSD was on a 6 month long downtrend this last year; that is not a random occurrence, that is a result of long term economic trends. If you're selling breaks of a running low, you'd have been shorting that all the way down; you need only adjust for volatility so you don't get knocked out on normal sized retracements and ride the trend down. You have to size your stops to adjust for normal volatility and allow the trend to carry you to a win and you have to tune your entry so you're already near an extreme against the trend so your odds are better, you can't enter blindly, but it doesn't have to be dead on accurate if your stops are wide enough. But wider stops require more capital to profit because you have to stay in the market longer to see your profit point hit if you're keeping a favorable risk to reward.

When the market isn't trending, don't trade it, you'll lose your ass in random price movements.

</comment></comment></comment></comment></comment></comment><comment><USER42> Math.random might do quite well?

</comment><comment><USER43> It's funny I didn't get an article about my algo, my returns aren't to shabby...

[http://www.strategic-options.com/trade/](http://www.strategic- options.com/trade/)

<comment><USER44> I think you would be more successful at attracting attention if you fixed your website.</comment></comment>


RaiBlocks is designed from the ground up for scalable instant txs and 0 fees - <AUTHOR> https://raiblocks.net/ <comment><AUTHOR> Github:

[https://github.com/clemahieu/raiblocks](https://github.com/clemahieu/raiblocks)</comment>


Becoming popular on web is hard, staying popular harder - <AUTHOR> http://www.cbc.ca/technology/story/2008/09/19/web-changing.html?ref=rss <comment><USER2> That's pretty obvious. The barrier of entry for making a websites is pretty low, and getting lower. It's quite easy, for example, for someone to go to CNN to get news instead of Yahoo News. All that's required is a different url.</comment>


PG: Would you be willing to upload a more current version of news.arc - <AUTHOR>

The one posted here: https://raw.github.com/nex3/arc/042990a55d8c880a0a88476eb52b169e47f35d32/news.arc is from 2006, it'd be nice to see a newer version that includes some of the changes the site had made since then. <comment><USER2> The latest pg-release is at <http://ycombinator.com/arc/arc3.1.tar> . It was released "only" two years ago (<http://arclanguage.org/item?id=10254> ) and runs on the latest version of racket. The repository you're looking at is an unofficial one that is much more active than pg's releases.

</comment><comment><USER3> Have you tried looking at the github repo, here: <https://github.com/nex3/arc/blob/master/news.arc> the latest commit on that was in November 2011

</comment><comment><USER4> I think he has stated before, or it has been implied by others, that releasing the source for hacker news might do more harm than good. There is a significant problem with voting rings now from the sounds of it. Hacker News has become a force that can greatly affect business and startups. If the voting ring people had the source, they might be able to find an exploit more easily that allows them to game the system. Clearly that would hurt Hacker News as a whole.

Basically, there are more black hats than white hats who would take a look at the source. YC is too powerful now.</comment>


Discovering less-known Postgres v12 features - <AUTHOR> https://www.cybertec-postgresql.com/en/discovering-less-known-postgresql-12-features/ <comment><USER2> It was added in 11, but I want to mention again how great of an addition the websearch_to_tsquery() function was. You can now get a respectable search system immediately in PostgreSQL without having to implement any sort of query-parsing or special data indexing yourself. All you need to do is add a column/index for fulltext search, which you can even do now using the generated columns added in 12 (instead of a trigger).

Then just pass user-entered queries through websearch_to_tsquery() and have access to multiple search capabilities easily. It supports searching for alternatives with "or", phrases in quotes, and excluding terms with a minus sign, which is all syntax that a lot of people are already used to. For example it can handle a query like:

        blizzard or overwatch -"hong kong"

That would find anything with "blizzard" or "overwatch" in it, but exclude anything with "hong kong" (if you wanted to avoid results about that recent controversy).

It's really useful, and makes it so that you can add reasonable search functionality with almost no work at all.

<comment><USER3> That is really cool and definitely a massive timesaver (I've written my own primitive query NLP for SQLite fstext indices and can definitely appreciate this feature!)

That said, (and only because your example reminded me) I absolutely hate the NLP `or` operator. I feel like any language to offer an `or` operator needs to require the simultaneous usage of a grouping operator (parentheses, etc.) to clarify the order of operations. A space is an implicit `and` so I never know if the `or` is working on directly adjacent keywords or the query as a whole.

</comment></comment><comment><USER4> Inlining CTEs is a big deal but they also finally have a way to see the progress of an index build without a PhD in postgresology!

Check it out here: [https://www.postgresql.org/docs/12/progress- reporting.html](https://www.postgresql.org/docs/12/progress-reporting.html)

<comment><USER5> My #1 request for indexes from PostgreSQL or 3rd party tool is concurrently rebuilding indexes.

I have had several cases where query performance dropped significantly (unfortunately in managed AWS Aurora DB, so I have limited inspection abilities).

Index rebuilds is the solution but it locks the entire table.

You can get a concurrent rebuild by creating a new index concurrently, then transactionally drop the old one and rename the new one. It's error prone though.

<comment><USER6> You will need to pick a new #1 request since PostgreSQL 12 finally delivered REINDEX CONCURRENTLY. :)

<comment><USER5> !!!!!

Wonderful! (Now I only have to wait 12 months for AWS Aurora to upgrade.)

</comment></comment></comment></comment><comment><USER7> SQL/JSON path expressions look very powerful for making queries easier to understand, they're a part of the SQL spec I hadn't seen yet.

I'm a little sad though to see yet another JSON location mini-language appear. We've got JSONPointer (IETF standard for the web, used by jsonschema), JSONPath, the more expressive but cross language JMESPath, the CLI favorite jq, and presumably a billion more. It's wearing me out.

</comment><comment><USER8> One nice thing about the CTE inlining its that materialised CTEs were stopping parallelism from being used. WITH is useful in big OLAP queries, where parallelisms should really help.

<comment><USER9> OTOH, it slowed down one of my queries from a few ms to 3s just by upgrading. So it's not some magic absolute benefit.

I had to go and add "materialized" to a few places.

<comment><USER6> Yeah, if the planner estimates are bad the inlining can produce a worse query plan.

</comment></comment></comment><comment><USER10> Are there any good advanced SQL books out there?

<comment><USER11> I found Dimitri Fontaine's [https://theartofpostgresql.com/](https://theartofpostgresql.com/) advanced. Learned lots of new tricks out of it.

<comment><USER10> Looks nice and it's on discount because of Black Friday. Thanks!

</comment></comment></comment><comment><USER12> Wasn't aware of log_transaction_sample_rate - should be useful for not spamming the logs.

</comment><comment><USER13> Site blocking the page with an empty overlay named "BorlabsCookie". No thanks.

<comment><USER14> The cookie acceptance on this site is as good as it gets IMO. Clear, detailed, privacy policies linked, with the ability to accept a reasonable, minimal subset.</comment></comment>


Ask HN: Tell me about your non-traditional entry into IT? - <AUTHOR> <comment><AUTHOR> For me, it all started the day my brother purchased a computer for my mom. My family didn't have much money, and as a result, I would help my mother clean bathrooms at a fruit and produce junction for $40 on Thursdays.

At the time, my brother had received money from an accident he had as a child. A dog tore off his eyelid, and the money was supposed to be for plastic surgery. Instead of incurring any more pain, he opted not to have the operation.

I discovered AOL and immediately was chatting with strangers, one who sent me a virus that would delete my hard drive. I spent days befriending him by first re-installing my operating system and rerunning the virus. Why? The virus would put me in a private chatroom in which I could speak with him.

After spending days talking to him, he finally agreed to share what happened. All I had to do was find this piece of software called Visual Studio. It took me 7 straight days to download it over a dialup.

I started to play with Visual Studio, but I wasn't good. Since my trial account on AOL was running out, I had to find a way to stay online. This is where I uncovered the power of Hex Editors. All I had to do was locate a password stealer, use the hex editor and change the password embedded in the program. That's exactly what I did!

I was able to stay on AOL for a year by "borrowing" the accounts of other people using this password stealer. That wasn't enough for me, and I had to write my own. This leads me down the path of reading source code of others, DosFx, and eventually attending trade school to make my software better.

At this time, I had no clue that people get paid to write software. During my time attending a trade school, I graduated with a 4.0 and my teachers told me I should write software for a living.

Six months after trade school I was still working at Home Depot. One day a customer came in running their mouth about the color of the paint they ordered. This experience made me very upset, and I wanted to snap, but I kept my composure in front of the customer. I went down the aisle to find another base to remake their paint. In a moment of rage, I punched a hole through a box and expressed how I felt out loud. That's when I heard someone from a distance say my name followed by "Why are you working here?" It was the teacher at trade school who told to get a job in IT.

Words can't express how disgusted and frustrated I felt with myself. He and I talked for a few minutes about what I was doing. It calmed me down enough and encouraged me to make a change.

There are a lot of small details I left out which are equally important in my journey, but I don't want to make this post too long.

<comment><USER2> But how and when did you enter IT?

<comment><AUTHOR> So after the encounter with my teacher at Home Depot, I posted my resume on dice.com. A trade school contacted me.

I went for the interview, and the founder asked me "Do you think you could teach?" I replied with confidence and said "Yes." I had no clue if it were something I would do right. All I knew was I would give it _everything_ I had.

I ended up working at a trade school as a teacher making $25/hr. I had to travel ~2 hours in one direction and would get home around 2 am. I was too afraid to quit working at Home Depot so I kept both jobs.

It wasn't until I got a job at a start-up company making $30k a year during the .com boom. I was severely underpaid, but I was happy that I finally had a full-time job in IT! Until the company went belly up a year later and everyone got laid off.

So what happened next? I was on unemployment for six months and refusing to work in IT. I was angry that after all of my dedication and hard work, I would be tossed aside. I read JavaScript articles by candlelight because my electricity was turned off for not being able to pay the bills early on. So I felt violated and used.

A manager from that startup company asked me to come interview at a company where he was working. I decided to give IT one more change. That job paid $40k a year. Granted I was still underpaid, but I had hope. Although I was underpaid, I was now able to pay bills and not struggle. I rented an entire home for $400/month. Side note, I was renting a house from an ex-drug dealer who didn't own the home. Instead, he took an abandoned home, cleaned it up and said he owned it. I didn't know this until years later.

By 2005, I was only making $48k a year, a far cry from what I make now. I learned how to negotiate and obtain my worth. I make it a point to teach people about how to achieve their proper worth. e.g., Never negotiate with yourself! You know that mental process when you want to ask for $$$ but decide on asking for $$? Learn how to value being told "No" during an interview vs. doing and accepting anything to "Yes."

Hope that answers your question.</comment></comment></comment>


Show HN: We built a SXSW radio guide to make exploring music at SXSW simple - <AUTHOR> http://unofficial-sxsw.earbits.com

<comment><USER2> thanks! I was just wondring why the south-by folks didn't make such a thing themselves.

I think there's an opportunity out there for an iphone app that did sxsw music previews, to help browse through acts when meeting up with friends.

<comment><AUTHOR> Our iPhone and Android apps will support these same channels by the time SXSW arrives. Also, you can click on the show info in the mobile apps and email the information to yourself or someone else easily.

<comment><USER3> Will there be calendar integration, i.e., add an artist's SXSW showtime+venue to my calendar when I discover something I like? If so, I can stop working on this myself and move on to something else :)

<comment><AUTHOR> There is not. That's a great idea, though.

</comment></comment></comment></comment><comment><USER4> Checking it out now - good stuff so far!

Edit: Do you guys have an API?

<comment><AUTHOR> Thanks! No, no API at this time.</comment></comment>


The Clock Is Ticking on Florida's Mountains of Hazardous Phosphate Waste (2017) - <AUTHOR> https://www.sarasotamagazine.com/articles/2017/4/26/florida-phosphate <comment><USER2> This really is everyones problem.. for such a critical product needed by everyone, this is what governments should be leaders in finding a solution.</comment>


EPUB 3.1 Standard Update - <AUTHOR> https://rdeltour.github.io/xmlprague2016-epub31/#1 <comment><USER2> I'm a big fan of EPUB because it's a pretty neat example of the Unix philosophy "worse is better".

A proprietary or custom format might have been more suited to creating e-books (drop-caps, for example, would probably have been easier), but using HTML and CSS as the basis means it's open, easily understandable, and pretty simple to make corrections to e-books or even begin making your own.

The fact that a teenager with a little bit of interest and a small amount of knowledge can play with an e-book, then view the results on real devices, is hugely appealing. I realise that they could also be making a website, or doing `real' programming, but to some people the ability to correct typoes in _Ender's Game_ or reformat _Dune_ to their preferred layout is just so much more appealing and empowering. When I started playing making websites at age 14, I'd have killed to be able to use those skills to make an e-book.

<comment><USER3> In spanish there are communities that take paper books, digitalize them and release them in epub format. Books that probably will never see an official digital release, which surprisingly, is the fate of many classic books (it's improving but the spanish digital book market isn't as vibrant as the one Amazon created for english titles). The best community I've found has a huge manual that in essence works as an editorial blueprint of how your epub must be constructed and formatted in order for them to accept your digitalization of a book and release them under their label, which in effect makes this site a clandestine digital _publisher_ of sorts. After you submit a title they have editors that review it, etc. and all of this is done by volunteers online. It's just awesome. There's no need to debate the legality of this site of course but if I was 14 today, with the free time I had back them, I'd definitely be geeking the hell out of that manual and digitalizing some of my many paper books. If there is a heaven, a special place must be reserved for all those guys that make that community work, if you ask me. Anyways, I haven't found similar communities in english although I assume there must be some.

<comment><USER4> I'm sure there are some Spanish-speaking people that would appreciate knowing the URL to this hub, and a few more who wouldn't mind being aware of its existence so they can reference it, etc.

</comment></comment></comment><comment><USER5> EPUB is, without any doubts, my preferred ebook format. I knew it was an open format standard, but I have never taken a look at what's under its hood. After I took a look at this presentation, I'm going to dedicate this evening to exploring the format, its advantages and its disadvantages.

Unfortunately, I am not able to find any reasonably good looking Linux client for it, so I usually use ebook-convert command that comes with Calibre to convert my books from PDF to EPUB and vice versa.

<comment><USER6> During your exploration, did you find something about why amazon's kindle doesn't work with EPUB?

<comment><USER7> Practically speaking it matters not. Calibre can do pretty lossless (practically speaking) conversions between formats. The limitations/features of ebook readers have a much greater influence on how a book looks than the format.

I love the status-quo where Amazon pretends that their DRM is not hopelessly compromised and I get to strip the DRM from the vast quantities of ebooks I purchase there. I would much rather they keep turning this blind eye than adopt epub behind a more effective DRM scheme. That would be a huge loss to me.

When my e-ink kindle can support high-resolution color images, smooth and fast LCD-like scrolling etc. then I might care about epub features, but while I don't want or expect more from my kindle than a paperback, I am quite happy.

</comment></comment></comment><comment><USER8> FWIW: navigation is done using the arrow keys on your keyboard or swiping when on mobile.

<comment><USER2> It never occurred to me to use either, so I used the spacebar and backspace to move forward and backward.

</comment></comment><comment><USER9> I dabbled with converting a Japanese text to an ebook and adding furigana to it. However, I couldn't find a free tool that supported generating 3.0 epubs, so I thought, it's mostly HTML, how hard can it be to just do it manually.

Very hard, I must say. The are all sorts of meta files that doesn't seem to ad anything to this format, but are somehow required.

<comment><USER10> So did you complete the book? I always appreciate some helpful furigana.

<comment><USER9> No. I could generate / write the html, but I couldn't for the life of me figure out the xml (I think there's another type of files required too) meta data. I looked at a few books and tutorials and it just looks like gibberish.

</comment></comment></comment><comment><USER11> I have a dream ... that some day there will be a _standard structured document format_ for publishing. And ePub is very, very, very nearly it. May well be, I've only been kinda keeping tabs. And yes, I'm a huge fan.

My one biggest frustration (noted earlier here) is that the Linux ePub client, fbreader, _doesn 't have a way to specify a maximum content line length_. Margins, sure, but not _content_. Result is that reading full-screen on a 16:9 display is ... kinda pessimal. Small issue, but one of those "pay attention to details" items that matters.

From the preso (nice layout there, by the way), Metadata: I'm in love. This is something that I'm familiar with via hNews and Readability. _Structured information in documents_ would be really, really peachy.

I've got a long-standing rant on the browser world, which is entirely flipping falling down as a _content presentation tool_ , for numerous reasons. Among the reasons for _tab_ proliferation, I'm realising, is _really piss-poor content management tools_. As I'm in the midst of a large research project, with some 1800 articles curated and another thousand-plus books and documents (mostly PDF, but a mix of other styles), _being able to reliably extract title, author, publisher, date, etc., information_ would be quite valuable.

[https://redd.it/256lxu](https://redd.it/256lxu)

Some things HTML is rather pointedly missing:

1\. A footnote / endnote / sidenote native capability. I've mucked around creating a sidenote implementation, but as a native element it'd be useful.

2\. A native hierarchical threaded discussion structure. The fact that every last online Web forum (HN included) is re-implementing tin / mutt (and doing it _wrong_ ) is ... depressing.

3\. A hell of a lot more thought to accessibility. From font/colour/size choices to support for text-to-speech and speech-control systems, I'm developing a growing awareness through the community I'm currently supporting. ePub with its simplified presentation may be a win here.

4\. Equations. Not always needed, but support for something at _least_ as powerful as LaTeX would be quite useful.

5\. Annotations. A lightweight mode of interacting with content would also be helpful.

That said, very happy to see progress.</comment>


Cutting Through Conflict and Tough Conversations - <AUTHOR> https://firstround.com/review/our-6-must-reads-for-cutting-through-conflict-and-tough-conversations/ <comment><USER2> Great read. Thank you.</comment>