Send Messages by Carrier Pigeons - <AUTHOR> https://flypigeon.co/ <comment><USER2> Does this implement the full spec from RFC 1149?

[https://tools.ietf.org/html/rfc1149](https://tools.ietf.org/html/rfc1149)

</comment><comment><USER3> So I don't blame anyone for taking this seriously (it took me 5 minutes of reading their blog before I finally convinced myself it was a joke), but this is definitely a parody. I mean, just look at this from their YC application:

> Our initial market research suggests Pigeon can become a major player in the > pigeon-as-a-service (PaaS) space.

> It's difficult to explain to non-pigeon users (NPUs), but the excitement is > real...

EDIT: When I posted this most of the comments in this thread seemed to take the company at face value. The multiple references to RFC 1149 and the "fleet" page since then have rather changed the tone. :)

<comment><USER4> I'm pretty sure it is, but it's still fun to play along with it :)

</comment></comment><comment><USER5> This is all in good fun, but let's also recognize that there is serious utility for this form of communication - especially in complex electromagnetic environments like wartime.

To that end, as recently as 2011 Chinese PLA have 10s of thousands of trained carrier pigeons for secure message transport.[1]

1: [http://www.telegraph.co.uk/news/worldnews/asia/china/8356921...](http://www.telegraph.co.uk/news/worldnews/asia/china/8356921/China- trains-army-of-messenger-pigeons.html)

</comment><comment><USER6> Is this compatible with existing RFC1149 deployments?

</comment><comment><USER7> Opportunity: "Get Early Bird Access"

<comment><USER8> I can't believe we didn't use this. Would you like a job in our puns department?

<comment><USER9> Can I apply for a position as COO?

<comment><USER10> Unfortunately, that position is already occoopied. However, you can check out our careers page to learn more about the available positions: [https://flypigeon.co/careers](https://flypigeon.co/careers)

<comment><USER11> Perhaps once your customer base warrants it, you might consider adding the "Chief Operating Officer for Puns" (COOP) role?

</comment></comment></comment></comment></comment><comment><USER12> Their YC application contains a few additional details; [http://blog.flypigeon.co/our-application-to-y- combinators-w1...](http://blog.flypigeon.co/our-application-to-y- combinators-w16/)

<comment><USER13> _55000 lines of code_

for what?

<comment><USER14> 2 different frameworks, 22 libraries and 100 domain models. For a CRUD app.

</comment></comment></comment><comment><AUTHOR> I read once that carrier pigeons go only one way (back home). How does this work?

<comment><USER15> I live in City A and want to send messages to my friend in City B. We both have a coop of carrier pigeons on our respective roofs. We meet up one day, and swap a number of pigeons. Now, if I want to talk to my friend, I just go up to my pigeon coop, select one of his pigeons, attach my message, and let the bird fly home.

It's like swapping private keys, except it can only happen in person.

<comment><USER16> When you exchange the number of pigeons, do they need to observe the travel back to City A to set directional bearings, or do you need to keep them in darkness so they will return to City B out of pure instinct?

Apologies for my ignorance, asking for myself and a friend....

<comment><USER15> According to this [1] nature article, homing pigeons have an innate sense of where 'home' is and can use the Earth's magnetic field to point themselves in the right direction, thanks to iron in their beaks. They don't need to 'learn' the path, like some migratory birds - they have a built in GPS. Which is kind of neat, when you think about it.

[1] [http://www.nature.com/news/2004/041122/full/news041122-7.htm...](http://www.nature.com/news/2004/041122/full/news041122-7.html)

</comment></comment></comment></comment><comment><USER17> I honestly can't tell if this is a joke or not...

</comment><comment><USER4> I would like to use this. Would make a great and very special birthday gift. Any plans to expand this to other countries?

<comment><USER18> The next generation of pigeons will have genetically engineered trans-atlantic and trans-pacific flight endurance capability.

<comment><USER4> Nice, though i'm sure recruiting from other cities wouldn't be a problem. Lot's of pigeons in every major city and tourist spots.

<comment><USER18> Exactly, and most of them are out of a job right now. Employment opportunities as Pigeon-Of-Piece have gone down drastically over the past couple of years. The Pigeon-as-a-Service industry has an opportunity here to step up in the name of social responsibility and do its share to increase and distribute wealth in one of the most ignored sectors of the populace.

</comment></comment></comment></comment><comment><USER19> I would pay money for such a delivery system, but it would only work with pidgeon hotels/stations in each city and the more specific local distribution by for example volunteer or paid cyclists who know where to go. It would be valuable carbonfree messaging supporting the local economies (and health of pidgeons)! You could also do solar powered fidonet-like raspberry pi asynchronous national network and print-outs (or handwriting) for last mile delivery but that would perhaps be more difficult and expensive to setup. Not sure if either would be a viable business plan.

</comment><comment><USER20> Wait until you see my startup to send passengers by passenger pigeon.

</comment><comment><USER18> Indeed, if we also cut out the browser and the cloud, then NSA and colleagues will just have to train their own under-cover stool pigeons. This should be fun to watch.

<comment><USER6> The problem with using pigeons for transferring private data is the automatic route tracing (as described in RFC1149). Any person will notice any pigeon routing that is being performed and this may provide enough information for future packets to be captured in transit. However carefully designed dietary controls can reduce the risk, for example, by limiting fiber intake.

</comment></comment><comment><USER21> In a similar vein, I'm working on an app that maps the location of hot pies cooling on windowsills next to open windows in your locality. ;)

<comment><USER8> The real question is how much VC will you need to make it a reality?

</comment></comment><comment><USER22> I love how in all of their "meet the fleet" headshots, the birds are totally out of focus, only the grass in the background is clear

</comment><comment><USER23> I don't get it, are these _actual_ pigeons? I would have expected a photo if so...

<comment><USER24> Yes. [https://flypigeon.co/fleet](https://flypigeon.co/fleet) \-- The service only operates in Florida (USA) so far, but they bring an actual bird to the door when delivering the message.

<comment><USER25> So print the message, and arrive holding a pigeon. No-one will know the difference.

</comment></comment></comment><comment><USER26> Their startup is actually upstair in my building since a couple months. I'll go talk to them to figure out what's going on. I guess it's a way to get attention and that they will provide some sort of secure messaging system.

<comment><USER10> No other messaging than this exists. We've been waiting for a couple weeks, squirt. Come up here, and I'd be happy to show you the press kit and everything else. Taco Tuesday's tomorrow! God bless.

</comment></comment><comment><USER27> I love how they're touting their in-office kergerator to potential hires.

The "heigene" typo on their "Junior Caretaker" job offer is weird though.

<comment><USER18> The typo clearly qualifies its author as a beneficiary of the U.S. education system.

</comment></comment><comment><USER13> how do the pigeons know where to go to?

<comment><USER14> This is a question I have as well...

As I understand carrier pigeons, is that they don't know where to go, only how to get home so usually, you'll give them a letter and they take it home, where the person then can collect the letter. But this seems different.

Someone have any idea how it actually works?

<comment><USER28> If you read the blog posts and YC app it's pretty obviously a parody.

</comment></comment></comment><comment><USER29> This is great. Is expanding to Alaska and Hawaii on the roadmap?

<comment><USER8> We're hard a work developing North Face jackets and water wings to make this a reality.

</comment></comment><comment><USER30> Is it April First already?

</comment><comment><USER31> will it scale?

</comment><comment><USER32> What is the latency like?

<comment><USER33> If you're willing to trade very high latency for high bandwidth, could be worth it, see [https://what-if.xkcd.com/31/](https://what-if.xkcd.com/31/).</comment></comment>


Show HN: BIND9 Denial of Service Exploit Checker - <AUTHOR> http://scan.patrolserver.com/bind/CVE-2015-5477

<comment><USER2> The problem with checking version.bind is that SOP for configuring BIND securely has been to disable or spoof it for some time now.

<comment><USER3> Yes, that is a problem. 'Hardened' configurations cannot be checked. But there are a lot of non-hardened configuration. Actually there are more that returns there version than not.</comment></comment>


Ask HN: Can programming languages unify at all? - <AUTHOR> Imagine, as a long term goal if all the major programming languages moved towards using standard terms for well established concepts.
In computing there are many concepts that have multiple terms.  Hash table and dictionary come to mind.
Consider also if basic actions had unified terms.  In one language currently you add an item to an array with "push", and in another language you "append".  What if all languages used the same term for the same action?
Thoughts? <comment><USER2> Different languages are different. That's mainly what makes them different.

/s

And actually, the terms you mention are subtly different as well. In part because of implementation details that will be different from otherwise similar languages.

Push and append for example, seem to be adding elements to the opposite ends of the thing being pushed or appended to. Push goes to the head. Append goes to the tail.

In Lisp for example, you push onto the head of a list while append concatenates the contents of multiple lists returning it all as one list.

Arrays, which are not the same as lists in Lisp, have vector-push which does what Python's append method does.

These things can't be unified because beneath the covers they use very different data structures.

And some things would get lost in translation anyway.

The word "macro" for example. C has macros, but C's macros are to Lisp's macros what a photograph is to a 3D movie. And Lisp's Condition System is unlike the closest thing in any other language I'm aware of.

And returning multiple values. The only languages that I know of that do it are Assembly and Lisp. Other languages may return compound values -- but it's not the same.

I don't know, brother -- you just have to do a little translation in your head. Like whenever those whacky Brits call trucks lorries, and bongs hubbly bubblies (actually, I dig that one).

</comment><comment><USER3> Is push and append really the same action? Append to me clearly indicates that it does at the end and I would probably expect to use it last. Push might mean it goes at the front and I will use it first.

Same goes for hash table and dictionary. A hash table describes the (probable) implementation but a dictionary usually just means a group of key-value pairs. In C that could be an array of pairs of pointers to C strings.

</comment><comment><USER4> Many of the terms and concepts are abstract and thus difficult for humans to fully understand (much less agree upon).

Some of the terms and primitives go way back too (1980s). Thread research is a good example. And then the concrete implementations come along (pthreads, etc.) that may keep some names/concepts, but change others:

1989 - An Introduction to Programming with Threads

[https://birrell.org/andrew/papers/035-Threads.pdf](https://birrell.org/andrew/papers/035-Threads.pdf)

1992 - Beyond Multiprocessing: Multithreading the SUnOS Kernel

[https://www.usenix.org/legacy/publications/library/proceedin...](https://www.usenix.org/legacy/publications/library/proceedings/sa92/eykholt.pdf)</comment>


Ask HN: What do you use to protect your home network? - <AUTHOR>

What software, hardware, protocols, or methods do you use to protect your home network? <comment><USER2> OBSD and a tailored pf, running on a Soekris 5501. I'd probably do OpenWRT today on a commodity router, but I was having too much fun back then rolling my own.

<comment><AUTHOR> I'm curious how you tailored pf and how you keep it up-to-date with repo patches, without breaking your changes?</comment></comment>


Benedict Cumberbatch: Julian Assange tried to talk me out of WikiLeaks film - <AUTHOR> http://www.theguardian.com/tv-and-radio/2013/sep/13/benedict-cumberbatch-julian-assange-wikileaks

<comment><USER2> This film may be, in part, be a vehicle for paying off Daniel Domscheit-Berg, who stole and destroyed a lot of material, including the Bank of America leaks and the no-fly list. I will avoid it.</comment>


"The Chaos" poem catalogues English word irregularities. Can you read all of it? - <AUTHOR> http://www.spellingsociety.org/journals/j17/caos.php#caos

<comment><USER2> The poem actually misses a word with multiple pronunciations, that's already embedded in the poem: console (line 9). I can console you if your console gets broken.

</comment><comment><AUTHOR> I think it gets more difficult as the poem progresses. The beginning is definitely the easiest part.</comment>


Privacy Plug-In Fakes out Facebook - <AUTHOR> http://www.technologyreview.com/printer_friendly_article.aspx?id=23405&channel=web&section=

<comment><USER2> The examples given are cases where people fail to understand the implications of posting data to a public forum. A browser plugin is not going to magically make people start thinking about the implications of their blabber-mouthing on Facebook.

Another problem is lack of understanding of the privacy policy. Over the weekend I was talking with a family member who mentioned that she messaged something private on Facebook to a friend without realizing that it would be publicly visible. A plugin wouldn't have helped her one bit, she mistakenly assumed her conversation was being kept private by Facebook. She learned her lesson and now doesn't use that medium for that kind of information.

Twitter's policy is that all of your posts are public (unless they are all private that is) which I think is less surprising to people.</comment>


Benford's Law - <AUTHOR> http://phpir.com/benfords-law

<comment><USER2> This is a new posting, with some interesting data. Even so, Benford's Law has been posted here many, many times:

<http://searchyc.com/benford>

</comment><comment><USER3> There's also sorting algorhitms based on this law iirc. Can';t find the link in 5 mins googling. Anyone?

</comment><comment><USER4> one of the many faces of scale-invariancy :)</comment>


The most surprising Unix programs - <AUTHOR> https://minnie.tuhs.org/pipermail/tuhs/2020-March/020664.html <comment><USER2> For me, the most surprising one was paste.

paste allowed me to interleave to streams or to split out a single stream into two columns. I'd been writing custom scripting monstrosities before I discovered paste:

        $ paste <( echo -e 'foo\nbar' ) <( echo -e 'baz\nqux' )
        foo     baz
        bar     qux
        $ echo -e 'foo\nbar\nbaz\nqux' | paste - -
        foo     bar
        baz     qux

I wonder what other unix gems I've been missing...

<comment><USER3> Recently I found myself wanting to stop a program after a certain timeout. I found myself thinking "sure there's a UNIX program to do this in shell?" There is. And it's called... brace yourself... "timeout".

20+ years in UNIX, never encountered it. Part of GNU coreutils.

<comment><USER4> Sometimes good to use for cron jobs so that the previous one won't keep running on top of the current run.

You should debug for it to not happen though but probably better than seeing your system choke with dozens of instances even slowing the jobs more.

There's also "run-one" that allows you to control which job should survive instead of just killing the old one.

[http://manpages.ubuntu.com/manpages/trusty/man1/run- one.1.ht...](http://manpages.ubuntu.com/manpages/trusty/man1/run-one.1.html)

<comment><USER5> If you're using systemd timers, you get this behavior for free because those timers only activate service units. If the service is already active, nothing happens.

(Of course, this can be surprising in a different way when you expect a new job to start, but it does not because a previous one is still lingering. Not saying it's better or worse, just pointing out for those who don't know.)

</comment></comment></comment></comment><comment><USER6> > Hidden inside WWB (writer's workbench), Lorinda Cherry's Parts annotated > English text with parts of speech, based on only a smidgen of English > vocabulary, orthography, and grammar.

Writer's Workbench was indeed a marvel of 1970's limited-space engineering. You can see it for yourself [1]: the generic part-of-speech rules are in end.l, the exceptions in edict.c and ydict.c, and the part-of-speech disambiguator in pscan.c. Such compact, rule-based NLP has fallen out of favor these days but (shameless plug alert!) Writer's Workbench inspired my 2018 IOCCC entry that highlights passive constructions in English texts [2].

[1] [https://github.com/dspinellis/unix-history- repo/tree/BSD-4_1...](https://github.com/dspinellis/unix-history- repo/tree/BSD-4_1_snap-Snapshot-Development/.ref-BSD-4/usr/src/cmd/diction)

[2] [https://ioccc.org/2018/ciura/hint.html](https://ioccc.org/2018/ciura/hint.html)

<comment><USER7> This _Writer 's Workbench_ seems really cool. The wikipedia page indicates there were quite a few more programs in the suite: [https://en.wikipedia.org/wiki/Writer%27s_Workbench#Package_c...](https://en.wikipedia.org/wiki/Writer%27s_Workbench#Package_contents)

Do you know where I could be able to find the source for all of these?

I'd be interested to "revive" these utils, possibly rewriting as python or bash for easy hacking. I have some basic scrips for that, and they are already proving to be useful even though they simply call grep [https://github.com/ivanistheone/writing_scripts](https://github.com/ivanistheone/writing_scripts)

<comment><USER8> I don't think it has all of them, but [0] is a tarball from Research Unix that has some Writer's Workbench source code.

The files are in cmd/wwb.

Other tarballs may have more Writer's Workbench code but I haven't looked at them.

[0] [https://www.tuhs.org/Archive/Distributions/Research/Dan_Cros...](https://www.tuhs.org/Archive/Distributions/Research/Dan_Cross_v10/v10src.tar.bz2)

</comment></comment></comment><comment><USER9> One of the useful applications of trigram-based analysis I have done is the following: for a large web-based application form where about 200000 online applications were made, we had to filter out the dummy applications - often, people would try out the interface using "aaa" as a name, for example.

Since the names were mostly Indian, we did not even have a standard database of names to test against.

What we did was the following: go through the entire database of all applications, and build a trigram frequency table. Then, using that trigram table, do a second pass over the database of names to find names with anomalous trigrams - if the percentage of trigram frequency anomaly in a name was too high (if the name was long enough), or the absolute number of trigrams in name was too high (if the name was short), we flagged the application and examined it manually. Using this alone, we were able to filter out a large number of dummy application forms.

Of course, it is not a comprehensive tool since what forms a valid name is very vague, but I think this kind of a tool is useful and culture-neutral.

<comment><USER10> The problem with these methods is that you exclude everybody that deviates from the norm. Yes, it might make your life (as a developer) a little bit easier, but it makes the lives of some of the applicants a lot harder.

<comment><USER11> I think the "manual review" phase makes this OK, in a way that simply autobanning Mr Null from your system isn't.

<comment><USER10> I'm not sure if that would work either.

Why not send them an email to verify? Or use captcha tech developed by big companies that actually has some science behind it.

<comment><USER9> These are high school students in India. Many of them are from rural backgrounds and do not have personal email accounts. From our experience, the forms are many times filled up by employees at cyber cafes who fill their own email addresses and mobile numbers instead of the students.

The only reliable means of communications back to students is by a government approved website, or newspapers, or official media. (The process also has to stand up in court in case some student says that (s)he did not get the communication, and newspaper ads are a documentable evidence of communications on a specified date.)

(BTW, the forms do have captchas, the spurious forms are manually filled in by mischievous/malicious/curious applicants.)

</comment></comment></comment></comment></comment><comment><USER12> It's surprising that Doug McIlroy still reads and writes about UNIX.

For those who don't know, Dough is the guy that invented pipes.

<comment><USER13> Also interesting:

> _Originators of nearly half the list--pascal, struct, parts, eqn--were > women, well beyond women 's demographic share of computer science._

<comment><USER14> In the 40s, computing was seen as primarily women's work (similar to the stereotype of switchboard operators). Into the 60s, women still comprised up to half of the computing workforce. In 84, they peaked at 37%. So demographically speaking, the ratio was not as bad as it is today.

(Source: [https://en.wikipedia.org/wiki/Women_in_computing](https://en.wikipedia.org/wiki/Women_in_computing))

<comment><USER15> _Into the 60s, women still comprised up to half of the computing workforce._

My life experience corroborates this.

When I was in school, girls were taught to type, and boys weren't. Because of that, many of the girls from my school went into computing, while the boys went to more "manly" pursuits. It's also why Catholic nuns were over- represented in early computing.

<comment><USER16> "It's also why Catholic nuns were over-represented in early computing."

Would you mind expanding on that?

<comment><USER15> Highly educated, and able to type. The first woman to earn a PhD in Computer Science was a nun in Ohio.

Catholic nuns have done much for the business world that is overlooked.

From memory, so no citations:

The first female CEOs were Catholic nuns — in the early 1900's, a time when women in the American workplace was uncommon, Catholic nuns founded over 800 hospitals in the United States. They also ran schools, colleges, and universities.

The concept of just-in-time delivery was invented by nuns running those hospitals.

The first health insurance company was founded in Missouri by nuns to care for railroad workers.

There are others. I once ran across a list of them on the intarwebs, but these are the ones that stuck with me.

<comment><USER17> Fascinating, thanks. This all makes Sister Celine's contribution less surprising. I came across her in the classic _A=B_ "about identities in general, and hypergeometric identities in particular, with emphasis on computer methods of discovery and proof." An amazing and wonderfully-written book.

[https://www.math.upenn.edu/~wilf/AeqB.html](https://www.math.upenn.edu/~wilf/AeqB.html)

[https://mathworld.wolfram.com/SisterCelinesMethod.html](https://mathworld.wolfram.com/SisterCelinesMethod.html)

[https://en.wikipedia.org/wiki/Mary_Celine_Fasenmyer](https://en.wikipedia.org/wiki/Mary_Celine_Fasenmyer)

</comment></comment></comment></comment></comment></comment></comment><comment><USER18> I didn't knew about typo. One surprising unix program I discovered this year is cal (or ncal). Having a calendar in your terminal is sometimes useful and I wish I knew earlier I could type things like _ncal -w 2020_

<comment><USER19> A similarly flavored one I've always appreciated is the man page for ascii, which shows the octal, decimal, and hex values for each character in the ASCII space.

Most unixes have one, although the format differs.

<comment><USER20> How have I been googling ASCII codes for two decades with this right under my fingertips?!? Thank you!

</comment></comment></comment><comment><USER21> And people say theoretical computer science isn't useful in "the real world"…

I am curious about this one, though, has anyone used it?

> The syntax diagnostics from the compiler made by Sue Graham's group at > Berkeley were the mmost helpful I have ever seen--and they were generated > automatically. At a syntax error the compiler would suggest a token that > could be inserted that would allow parsing to proceed further. No attempt > was made to explain what was wrong.

On the surface it sounds a lot like it would produce error messages like "expected ';'" that most beginner programmers come to hate: was it any better than this, or was that the extent of its intelligence and everything else at the time was even worse?

<comment><USER22> > On the surface it sounds a lot like it would produce error messages like > "expected ';'" that most beginner programmers come to hate

Do people really come to hate these? I'd expect the opposite -- that people would start off hating messages like "expected ';'", but fairly quickly become accustomed to what they almost always mean.

As long as you can look at the message and have a good idea of what's wrong, it's not a bad message.

<comment><USER23> Once you've used a compiler like Rust or Elm that actually provides suggestions for common solutions to these errors (effectively building the tribal knowledge of what the error "really means" into the compiler itself), it's hard to tolerate these cryptic errors that only really make sense to machines.

<comment><USER24> I often found Rust's errors completely confusing, even after chasing down the '\--explain CrypticNNNNN' follow-up explainer. This was in 2019 — not some ancient version of Rust.

<comment><USER21> Yeah, Rust's compiler errors are decent if you make simple mistakes but degrade to being about as bad as any other modern compiler's once you start doing complicated things. Which isn't horrible, but --explain isn't really useful so it's just wasting space on my screen.

<comment><USER25> Please file bugs for any message that is confusing! We track them like any other bug, and there's some folks actively working on them.

<comment><USER21> Hmm, I wouldn't call them confusing per se, they're just not useful, and I don't think any compiler has really solved this problem (but then again, generation of compiler error messages is not something I'm an expert in). Let's say I forgot to put a "*" in front of something: the compiler's error might be something like "xyz does not implement SomeTrait, here is a page explaining what traits are". I'd be more than happy to file bugs for things like these but I have generally refrained from doing so because I am unsure if this is something that is possible to fix. If you'd like, I could file issues for things like this, but I'm genuinely curious to hear if there's any strategies on improving these or work done in this area.

<comment><USER25> Let us determine if it's possible or not. The person who currently works on errors is of the opinion that any time the error isn't useful, it's a bug.

> I'm genuinely curious to hear if there's any strategies on improving these > or work done in this area.

It's just a ton of work. You look at what the compiler produces, look at what information you can have at that point, see if you can craft something better, and then ship. And then look at the next error.

<comment><USER24> > Let us determine if it's possible or not. The person who currently works on > errors is of the opinion that any time the error isn't useful, it's a bug.

That's a fantastic attitude and I really appreciate that someone is working towards that goal, thanks.

> It's just a ton of work. You look at what the compiler produces, look at > what information you can have at that point, see if you can craft something > better, and then ship. And then look at the next error.

Exactly. That person is a saint.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER26> The author is THE Doug McIlroy. It's wonderful to learn that he's still around and spreading the good word.

[https://en.wikipedia.org/wiki/Douglas_McIlroy](https://en.wikipedia.org/wiki/Douglas_McIlroy)

</comment><comment><USER27> « Typo was as surprising inside as it was outside. Its similarity measure was based on trigram frequencies, which it counted in a 26x26x26 array. The small memory, which had barely room enough for 1-byte counters, spurred a scheme for squeezing large numbers into small counters. To avoid overflow, counters were updated probabilistically to maintain an estimate of the logarithm of the count. »

This sounds like something from the same family as hyperloglog

Wikipedia traces that back to the Flajolet–Martin algorithm in 1984. When would typo have been written?

<comment><USER28> Probably not related.

Sounds like it's just doing something like replacing `counter++` with `if(rand() % counter == 0) counter++`, so that the counter will increase slower and slower the larger it gets.

<comment><USER29> Absolutely related! This is essentially the same observation that makes Flajolet-Martin and HyperLogLog work - that when comparing counts, the exact low bits of large numbers "matter less" than the low bits of small numbers, so you can store the logarithm of the count. They differ in how they calculate the "incremental log" without storing the real values, based on what they are counting (high-dimensional events vs. high-cardinality sets).

</comment></comment></comment><comment><USER30> How about GNU parallel? [https://www.gnu.org/software/parallel/](https://www.gnu.org/software/parallel/)

<comment><USER31> wow! You just saved the future me thousands of hours.

<comment><USER21> I hope you don't mind the citation nags ;)

</comment></comment></comment><comment><USER31> What about "comm" \- compare two sorted files line by line. You can easily get occurrences only in file 1, in both files, only in file 2.

Super powerful and saved me hours of work.

<comment><USER32> comm is a really useful tool, with one big caveat — you must make sure your input files are all sorted the exact same way. If not, you can get unexpected results, and worse, might not even realize it.

This may seem obvious, but there are many tiny ways that sorts can differ between locales, operating systems and programs (e.g. Excel), especially when dealing with Unicode. It may look the same 99% of the time, and you may not realize until later that you've accidentally filtered out values.

<comment><USER33> My advice is to sort the files just-in-time using the shell:

        comm <(sort fileA.txt) <(sort fileB.txt)

</comment></comment></comment><comment><USER34> There's a very simple system tool that clicked on about 50 simultaneous lightbulbs in my brain after only 10 minutes of playing with it: mkfifo

<comment><USER35> The man page for it is awful, 0 explanation of what it actually does.

It allows you to create pipes as files! So you can do:

`echo 'hello world' > mypipe` on one terminal and `cat < mypipe` on another!

Very neat, I'm sure I'll find uses for it in the future.

</comment></comment><comment> ur-whale The fact that dc does (or at least tries to) guarantee error bounds on the _result_ is news to me.

And if that does indeed work, that's pretty cool.

<comment><USER36> I doubt the modern GNU or BSD versions of it that you are likely using do. Noone uses the original anymore.

<comment><USER37> Is scale factor the same as error bounds in [http://man.openbsd.org/dc](http://man.openbsd.org/dc) ?

<comment><USER36> I believe it's just the number of digits when the printing cuts off

</comment></comment></comment></comment><comment><USER38> sl

```

                              (  ) (@@) ( )  (@)  ()    @@    O     @     O     @      O
                         (@@@)
                     (    )
                  (@@@@)

                (   )
             ====        ________                ___________
         _D _|  |_______/        \__I_I_____===__|_________|
          |(_)---  |   H\________/ |   |        =|___ ___|      _________________
          /     |  |   H  |  |     |   |         ||_| |_||     _|                \_____A
         |      |  |   H  |__</comment><comment></comment><comment></comment><comment>--| [___] |   =|                        |
         | ________|___H__/__|_____/[][]~\_______|       |   -|                        |
         |/ |   |</comment><comment>-----I_____I [][] []  D   |</comment><comment>=|____|________________________|_
       __/ =| o |=-O=====O=====O=====O \ ____Y___________|__|__________________________|_
        |/-=|___|=    ||    ||    ||    |_____/~\___/          |_D__D__D_|  |_D__D__D_|
         \_/      \__/  \__/  \__/  \__/      \_/               \_/   \_/    \_/   \_/

```

<comment><USER39> During college my friend and I kept an innocent prank going for a couple of years: every time one of us left our laptops unlocked the other would jump in and type 'alias ls=sl' in the prompt and then clear the screen. Good times.

<comment><USER40> Put it in their bashrc ;)

</comment></comment></comment><comment><USER29> > _struct - Brenda Baker undertook her Fortan-to-Ratfor converter against the > advice of her department head--me. I thought it would likely produce an ad > hoc reordering of the orginal, freed of statement numbers, but otherwise no > more readable than a properly indented Fortran program. Brenda proved me > wrong. She discovered that every Fortran program has a canonically > structured form. Programmers preferred the canonicalized form to what they > had originally written._

We could've had prettier et al instead of style linters 40(+?) years ago. :(

<comment><USER41> I had to look up 'Ratfor' because I'd never heard of it — apparently it's a FORTRAN preprocessor that added control structures.

<comment><USER42> One of the books that most influenced my coding was Software Tools by Brian W. Kernighan, P.J. Plauger [0]. Even though I never used Ratfor, the clear descriptions were immensely useful.

[0] [https://www.goodreads.com/book/show/515603.Software_Tools](https://www.goodreads.com/book/show/515603.Software_Tools)

<comment><USER41> I have that book. I have read snippets of it. Evidently I should read all the way through it.

</comment></comment></comment></comment><comment><USER43> I've written a few useful scripts that everyone should have.

histogram - simply counts each occurrence of a line and then outputs from highest to lowest. I've implemented this program in several different languages for learning purposes. There are practical tricks that one can apply, such as hashing any line longer than the hash itself.

unique - like uniq but doesn't need to have sorted input! again, one can simply hash very long lines to save memory.

datetimes - looks for numbers that might be dates (seconds or milliseconds in certain reasonable ranges) and adds the human readable version of the date as comments to the end of the line they appear in. This is probably my most used script (I work with protocol buffers that often store dates as int64s).

human - reformats numbers into either powers of 2 or powers of 10. inspired obviously by the -h and -H flags from df.

I'm sure I have a few more but if I can't remember them from the top of my head, then they clearly aren't quite as generally useful.

Anyone else have some useful scripts like these?

<comment><USER44> I work with csv files a lot. I have a short awk script which truncates/pads each column to a fixed width which I can specify at runtime. It also repeats the top column (headers) every 20 rows in a different ANSI color. I pipe the output to less -SR for interactive use so I can scan delimited data in a scrollable grid, with all columns aligned and labeled.

I understand there's vim plugins for this, but, ehh.

<comment><USER45> There's also the likes of console-flat-table-viewer . One would have to convert the comma-separated stuff into one of the table types, but that's what Miller is for. (-:

* [http://jdebp.uk./Softwares/nosh/guide/commands/console-flat-...](http://jdebp.uk./Softwares/nosh/guide/commands/console-flat-table-viewer.xml)

* [http://johnkerl.org/miller/](http://johnkerl.org/miller/)

</comment></comment></comment><comment><USER46> _"To avoid overflow, counters were updated probabilistically to maintain an estimate of the logarithm of the count."_

Stuff like this really makes me love what the pioneers of CS did in the past. In the past, they were counting every byte and every register while nowadays, programmers make things without considering the impact it will have on the HW.

</comment><comment><USER47> > The math library for Bob Morris's variable-precision desk calculator used > backward error analysis to determine the precision necessary at each step to > attain the user-specified precision of the result.

I wonder if compilers could do this today? If you can bound values for floating point operations, you might be able to replace them with fixed point equivalents and get a big speedup. You might also be able to replace them with ints or smaller floats if you can detect the result is rounded to an int.

CPU's also have the possibility to do this since they know (some of) the actual values at runtime, and could take shortcuts with floating point calculation in places where not needed for the result.

<comment><USER48> Replacing floats with fixed point isn't usually a meaningful optimization on modern CPUs. The FPU runs in parallel to the integer units, so you can easily end up idling the FPU while the integer units are too busy doing both the math and the necessary state management (counters, pointer arithmetic etc.)

This could make sense for SIMD however, but then the problem is getting the array data in the right format before the computation — if you're converting from float to int and back within the loop, it destroys any performance gain.

<comment><USER47> Fixed point uses a lot less power though, and many use cases are effectively power limited rather than functional-unit limited, since if you really do fill all functional units on every cycle you'll soon need to throttle back your clock speed...

Perhaps a good example of that is video encoding, which is mostly fixed point, despite it looking like a pretty close fit for floating point maths.

<comment><USER48> A very good point. My worldview of performance is highly biased towards "full steam ahead" desktop graphics.

Video encoding is a bit of a special case though because the common algorithms are carefully designed for hardware acceleration. For most rendering, it doesn't make sense to go out of your way to avoid the FPU.

</comment></comment></comment></comment><comment><USER49> What's surprising about eqn, dc, and egrep? I'm using the latter two all the time, and have used eqn (+troff/groff and even tbl and pic) in the 1990's for manuals and as late as (early) 2000's to typeset math-heavy course material. Not nearly as feature-rich as TeX/LaTeX, but much more approachable for casual math, with DSLs for typesetting equations, tables, and diagrams/graphs. I was delighted to see that GNU had a full suite of roff/troff drop-in replacements (which I later learned was implemented by James Clark, of SGML and, recently, Ballerina fame).

<comment><USER50> I had never heard of eqn and was surprised to find that the binary is still there on my Linux box.

With regard to roff in general, when I got into Linux-based typesetting around the turn of the millennium, that was already seen as antiquated tech, superseded by LaTeX which was undergoing a frenzy of development and improvement around that time. So, anyone under the age of 30 will probably be hearing of such *roff stuff for the first time (and sadly even familiarity with LaTeX has waned).

<comment><USER49> Ok I'm probably showing my age here then :) Back in the 1980 and 1990s, the roff suite, and most definitely egrep and classic Thompson DFA construction and DFA->NFA conversion was definitely Unix folklore/taught in Uni. Manpages are still rendered using roff/groff today, so probably many of us are using it regularly. Whereas GNU's texinfo has matured less well I'd say, or wasn't even very useful in practice to begin with due to lack of content.

I'm also using TeX/LaTex, but it's still a programming language whereas roff/eqn etc are non-Turing DSLs and renderers for particular narrow purposes. I get your point, but saying these are "antiquated" is like saying HTML is obsoleted by JavaScript.

<comment><USER51> > and most definitely egrep and classic Thompson DFA construction and DFA->NFA > conversion was definitely Unix folklore/taught in Uni

I think you mean "Thompson NFA construction" and "NFA->DFA."

Regardless though, this is not what the OP is pointing out. 'egrep' (or just GNU grep these days) is doing something more clever (emphasis mine):

> Al Aho expected his deterministic regular-expression recognizer would beat > Ken's classic nondeterministic recognizer. Unfortunately, for single-shot > use on complex regular expressions, Ken's could finish while egrep was still > busy building a deterministic automaton. To finally gain the prize, Al > sidestepped the curse of the automaton's exponentially big state table by > _inventing a way to build on the fly only the table entries that are > actually visited during recognition._

Russ Cox talks about this a bit in part 3 of his articles on regex matching[1]. Its implementation in RE2 is here: [https://github.com/google/re2/blob/master/re2/dfa.cc](https://github.com/google/re2/blob/master/re2/dfa.cc)

[1] - [https://swtch.com/~rsc/regexp/regexp3.html](https://swtch.com/~rsc/regexp/regexp3.html)

<comment><USER49> > _I think you mean "Thompson NFA construction" and "NFA->DFA."_

Yep, only noticed it later, then left it in to see who's paying attention :)

</comment></comment></comment></comment></comment><comment> ur-whale First time I hear of typo ... it's not on my standard Linux install ... where can I find the source code?

<comment><USER21> It's not quite the original, but Rob Pike wrote an implementation in Go: [https://github.com/robpike/typo](https://github.com/robpike/typo)

<comment><USER52> In go? This is a job for rust!

...there goes my weekend.

</comment></comment></comment><comment><USER53> I would add bc to the list, very useful to make occasional calculations from command line using "human readable" syntax.

<comment><USER54> Fun fact, the first version of bc was just a frontend to dc. It converted the structured input to dc's stack-based form and let dc do the math.

<comment><USER53> Did not know that, thanks. I searched for dc inside bc and could find a reference to /usr/bin/dc, so I think bc still is just a wrapper.

% uname -a

FreeBSD skyrocket 9.3-RELEASE FreeBSD 9.3-RELEASE #1: Fri Nov 27 20:28:19 UTC 2015

<comment><USER54> GNU bc isn't though they share the bignum code, I am not surprised that the BSDs are following the older implementation more closely!

</comment></comment></comment></comment><comment><USER55> I have found it useful to survey the existing unix utilities (maybe every several years). I'm no genius but I find things I will use. One way of course is simply to review the names in wherever your system stores manual pages, and read (or skim) those where you don't know what they do, trying out some things, or trying to remember at least where to look it up later when ready to use it. Another is by browsing to [https://man.openbsd.org/](https://man.openbsd.org/) , then put a single period (".") in the search field, optionally choose a section (and/or other system, not sure how far the coverage goes), and click the apropos button.

</comment><comment><USER56> Doug McIlroy is regularly active in the groff mailing list [https://lists.gnu.org/archive/html/groff/](https://lists.gnu.org/archive/html/groff/)

</comment><comment><USER57> What does he man by "record structure in the file system" in re to Multics?

<comment><USER58> Unix files are simply a stream of bytes and outsource concern of file structure to userland. There's nowhere to set/get a type, no mechanism to create schema in the file like fields, lengths, constraints, etc. You simply can seek to a place in the file (if it's seekable) and read/write the bytes. What they mean is up to the programs/user/convention.

Earlier filesystems were trying much more to be like databases.

</comment></comment><comment><USER59> Crabs seems likes a really cool program.

Here is a paper from Bell Labs

[http://lucacardelli.name/Papers/Crabs.pdf](http://lucacardelli.name/Papers/Crabs.pdf)

</comment><comment><USER60> I didn't find egrep surprising - I use it quite often. The thing I didn't know about it was that it was Al Aho's creation. I only knew about him from awk.

</comment><comment><USER61> killall5 is the most bizarre command that I learned recently.

Read manpage before trying it.

<comment><USER62> Sys5 killall: Bane of all regular linux administrators that also sometimes administered solaris boxes.

Once after blowing up an in production database server during the day, I suffered the unfortunate difficulty of having to explain why running a command "killall" on a critical server that killed everything was an innocent mistake and that I didn't have any reason to expect it to kill everything.

It's extremely difficult to not sound like a moron when explaining that you didn't expect "killall" to "kill all".

</comment></comment><comment><USER63> Hadn't heard of most of these.

The peoples' names were more recognizable.

</comment><comment><USER64> I found GNU parallel to be very useful/cool.

</comment><comment><USER65> sed awk tr egrep for processing making special greeting lol converting images

all are so exciting!!

</comment><comment><USER52> sed & awk for life

</comment><comment><USER66> both rename and mmv are pretty handy

</comment><comment><USER67> xargs parallelizes with -P _n_</comment>


Europe's New Privacy Rules Favor Google and Facebook - <AUTHOR> https://www.wsj.com/articles/how-europes-new-privacy-rules-favor-google-and-facebook-1524536324 <comment><USER2> The true cost of GDPR doesn't come from compliance itself. It comes from trying to protect oneselef from the legal vulnerability it creates for businesses.

Like any regulation, once enacted, it creates yet another reason to harass, burden and encumber businesses - no matter what business model they have or if they are shady or not.

Large companies truly love this. Their already considerable legal departments chew any new regulation the moment it appears while the newly erected walls keep those pesky startups away.

Small business do not even pay the price - they simply cease to exist, to be founded. It's a chilling effect. The prospective founder says "why bother with this shit when I can simply get a job"?

The visible end result: an Europe more and more hostile to startups staying on the sidelines while the US wins the innovation race for the future. Fewer startups means also fewer large business (just look at the most valuable companies in the world, how many of those are European?) and thus fewer well paid jobs in the future (just check out the difference between an IT salary in USA vs Europe).

<comment><USER3> Yes, let's allow companies process personal data indiscriminately because it will help small businesses. /s

Please provide actual evidence that it will do so and that small data processing businesses actually had to endure big legal fees due to privacy protections.

The lack of start-up scene in most of Europe is caused by very different factors, mostly more conservative and less wealthy investors.

</comment></comment><comment><USER4> Not related to the ads industry but GDPR compliance in general for smaller companies (which is discussed in various other comments to this article), I would like to add these remarks:

I think "GDPR compliance" is not a binary thing - there are levels of compliance. No smaller normal company or Google or Facebook will be able to "fully comply" at any point in time. There will always have to be incremental improvements over time, hopefully in the right direction.

Also recognize that the obligations in GDPR "scales" with the size of operation/amount of data that is processed. For example smaller companies:

\- are exempted from some of the record keeping obligations,

\- may not have to appoint a data protection officer,

\- may often not have to do data protection impact assessments,

\- may not have to set up and implement some policies relating to data protection

\- may also have less obligations when it comes to Privacy by Design (since reasonable costs are a recognized factor)

\- may generally have fewer data processing activities that requires consent (most normal personal data processing does not)

I would also forget about scares of 4% fines unless you are doing something really bad or do not care at all even after a fine, repremand or warning from a Data Protection Authority.

My guess is that the "normal level of fine amount" will not change with GDPR for data protection violations in the EU (compared to current UK levels). This seems to be what the UK:s Data Protection Authroity (ICO) is going around and telling everyone as well (if I have not misunderstood).

<comment><USER5> _No smaller normal company or Google or Facebook will be able to "fully comply" at any point in time._

Which makes it ripe for abuse. By making a regulation that is impossible to fully comply with, they have created a situation where any (or all) of the 28 countries in the EU can fine any company at will, because they're never fully in compliance. Further, individual users can seek compensation under GDPR - companies will be deluged with such claims.

Just having to depend on the goodwill of regulators who you think will just be attempting to enforce the spirit of the law is a terrible position to put companies in. When these countries and individual citizens are starved for cash, all of a sudden the letter of the law will be the only thing that matters, as is always the case with regulations that put all market participants in violation by default.

<comment><USER4> The level of fines looks like it will be harmonized by EDPB (a board of all Data Protection Authorities) so individual DPAs will probably not be able to set arbitrary levels. There are also the one-stop-shop rules so a company will normally only have to deal with one DPA.

When it comes to compensation, you have to show what material or non-material damages you've suffered. Class actions are generally not a big thing in the EU. Sure, there is an exposure here but in practice I think it will be very limited, especially for smaller companies.

<comment><USER5> _When it comes to compensation, you have to show what material or non-material damages you've suffered_

You realize that "material or non-material damages" can mean just about anything right? If it were limited to "material" damages, that would at least impose some form of sanity. The way this is written, it's a free-for-all. Saying _" I have had alot of trouble sleeping since I found out that only 45 of 46 tracking companies were listed on the disclosure"_ is all it would take for an individual to obtain money from a company under this absurd regulation.

<comment><USER6> They'd have to prove in court (to the balance of probabilities) that they have insomnia and that the insomnia was caused by the non-disclosure. They would also have to show that this is an expected result of non-disclosure.

Assuming they do this - how much do you think a compensation payput for insomnia is worth? How many thousands of pounds do you think people would get?

<comment><USER5> I don't know, but if you multiply the cost of defending one of them by the millions of such nuisance claims that will be filed by people, and it will quickly add up.

<comment><USER6> You're saying millions of people will be willing to commit several criminal offences in order to attempt to get a payout of about £1000, in a system that routinely doesn't pay for psychological harm?

I mean, those people can already do this, and they don't appear to do it, so I'm not persuaded GDPR will trigger a bunch of claims.

<comment><USER5> Is it a criminal offense though? How can it be disproven that you haven't been sleeping as well as you normally do, or that you've been having anxiety because an externally loaded PNG served a cookie that wasn't disclosed (because the website operator didn't know about it)? They'd have to disprove that to jail you for making a false claim.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><AUTHOR> As suspected, smaller companies seem to be hurting a lot more than the giants. From the article:

 _A digital-advertising firm called AdUX recently closed a service that harvested location data from people's smartphone apps to show them targeted ads, said CEO Cyril Zimmermann, because his firm had little hope of asking for—much less getting—consent from users. Instead, AdUX will aggregate data from bigger companies. He said the shift has cut into revenue. "For them, it's easy," he said. "The problem is, who knows AdUX?"_

<comment><USER7> How does GDPR affect selling personal data? Once I've given permission for a company to collect my data, can they just sell it to whoever they want? Is there any way to get rid of that data once it's been sold? Is there any way to find out who's bought it?

<comment><USER8> That's a lot of questions. To sum up, the GDPR is forcing:

\- granular consent (like you might opt into a newsletter, but not retargeting ads)

\- right to request what a company knows about you

\- right to get that data in digital format (json, xml)

\- right to know when your data is exposed in a data breach

\- right to request that your data be deleted

\- right to request that an organization stop processing data about you

fwiw - I wrote a bunch more about it (link) and am happy to answer questions.

[https://blog.varonis.com/gdpr-requirements-list-in-plain- eng...](https://blog.varonis.com/gdpr-requirements-list-in-plain-english/)

<comment><USER7> So as long as there are 1000s of companies like AdUX that purchase my data without telling me or acquiring my consent, I don't really have any hope of actually having my information deleted.

I suppose someone could make a service that lets you request every known company delete your info.

<comment><USER9> If they want to comply all those 1000's of companies will have to get you to reaffirm your consent under the new GDPR rules. Didn't you notice already a lot of the services/sits you use asking you to re-agree ('We've updated our privacy policy etc.')

<comment><USER7> Are you sure that all applies to companies that _purchase_ personal information and not just companies that _collect_ it?

<comment><USER3> They also _process_ the data which is explicitly mentioned in GDPR.

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> This strikes me as a fluff PR piece to try to head off similar legislation in the US.

</comment><comment><USER11> I've posted this before: GDPR is good independent of any effects it has on competition.

Governments already have the tools to deal with market dominance: antitrust enforcement. The EC has three antitrust cases against Google. One decision has already been given, and the Android decision will be made sometime within the next few months. The Federal Cartel Office in Germany has opened a probe into Facebook [1][2].

[1] [https://www.politico.eu/article/facebook-data-collection- cou...](https://www.politico.eu/article/facebook-data-collection-could-be-an- antitrust-abuse-in-germany/)

[2] [https://www.reuters.com/article/us-facebook-privacy- germany/...](https://www.reuters.com/article/us-facebook-privacy- germany/facebooks-hidden-data-haul-troubles-german-cartel-regulator- idUSKBN1HU108)

<comment><USER12> A $100+ million revenue company has the capital to comply with GDPR. The 1-5 man bootstrapped start up does not. They will be in the 3 felonies a day type of non-compliance and hunted by bureaucrats looking for easy targets. Other countries have a philosophy of exempting small business from complicated regulations because of this. The EU tends to not do this.

That is the fundamental issue and it has nothing to do with monopoly.

Just like you need progressive taxation to not choke out the poor, you need progressive regulation to not choke out the small business.

<comment><USER13> I did not see anything in GDPR which does not make sense to me, it looks like common sense applied to data management.(Edit: It's rare enough to have some legal text in IT which makes sense in real life, I'm surprised). I don't see why small businesses should be exempt for basic regulation, small businesses are not exempt from fire safety regulation either.

> The 1-5 man bootstrapped start up does not.

In my current company we are only two developers and we will comply with GDPR, it's going to take us a week or two worth of technical work, it's really not a big deal.

<comment><USER12> The GDPR seems easy, but I get a feeling you haven't tried to actually properly implement the GDPR regs, talking to lawyers and everything. If you have you'll see there is far more involved than you think.

I wish it was easy as just manually looking for user data after an email, deleting it, and keeping that email request as part of the 'audit log'. And getting affirmative consent during signup.

Most small businesses would be fine with something that casual.

EDIT: If you guys winged it by actually just reading the regulations and winging it, you probably did something wrong.

<comment><USER14> The GDPR is inconvenient with many of the shady business models popular among web-centric companies.

In the long run, it's good for all of us if those business models are discouraged.

<comment><USER12> It's inconvenient for the non-shady, non adtech, we give you a service in exchange for money businesses too!

I think the best thing to take away from GDPR compliance is 'it's not as easy as you think it is', 'it's not as obvious as you think it is', and small businesses who totally respect privacy will probably still be breaking GDPR.

<comment><USER14> I'm OK with that and I accept that cost, because in my eyes stopping the savage victimization of users is worthwhile. For you (or the WSJ) to persuade me that the GDPR is problematic, I'd have to believe that first, that's something that those opposed to the GDPR actually care about, and that it's something that could be achieved through other means.

<comment><USER15> Except this does nothing to really stop the savage victimization of users by Facebook or Google. If anything it solidifes their dominance.

The abuses are happening from these big companies, not the thousands of tiny startups that live and die within months that have a few hundred thousand users.

<comment><USER14> So your proposal is a return to the status quo ante and unlimited exploitation of users? Not good enough. I will take the imperfect good of the GDPR as an incremental improvement over the option of abject surrender.

<comment><USER12> No the proposal is to apply the GDPR to the large organizations that can afford to implement it properly. It's not black and white ;)

<comment><USER3> GDPR is not applied indiscriminately you know. It has various responsibilities indeed depending on company and/or database size.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER16> "It is paradoxical," said Bill Simmons, co-founder and chief technology officer of Dataxu , Boston-based company that helps buy targeted ads. "The GDPR is actually consolidating the control of consumer data onto these tech giants."

<comment><USER17> he probably haven't heard the term "Revolving door". Simple study of antitrust policies would reveal him that this is the case.

[https://en.wikipedia.org/wiki/Revolving_door_(politics)](https://en.wikipedia.org/wiki/Revolving_door_\(politics\))

<comment><USER3> I'm pretty sure previous privacy laws did little to EU businesses. (Some local more restrictive ones like German neither.) LThis one will not do anything much like you said either. There ate much mote critical laws that are being tampered with such as tax code and employment laws.

</comment></comment></comment><comment><USER18> I still don't understand how this is going to be enforced. If I'm a website or app provider in China or Philippines or Russia or India, and I collect information on EU residents who visit/use my website/service, how is GDPR going to be enforced? How are they going to fine me 4% of my revenue?

<comment><USER12> You arrest company officers flying into the EU (even for a layover, even for emergency landings) for default judgements.

You block their companies from doing business in the EU. You confiscate bank accounts set in the EU, you confiscate shipments to/from the EU.

<comment><USER19> For a public company, sure. How would you find company officers of private offshore companies located in Cayman/Seychelles/BVI?

<comment><USER20> Of course you can always do illegal stuff. GDPR won't change that. You can still buy e.g. stolen credit cards on the black market. But personal information? Doesn't seem worth the hassle, especially when you can anonymize it and keep using it. Or get consent. If you're smart enough not to get caught, you're probably smart enough to come up with a business model that isn't shitty.

</comment></comment></comment></comment><comment><USER21> Currently going through GDPR certification/signoff as a primarily US based company.

As a hacker, I enjoy the spirit of the thing. It's nowhere near as dumb as the cookie pop-up law. Categorically I would prefer default third party services to flush logs rather than retain.

As a founder, the cruel business logic very much favors removing support/refusing to onboard European users.

Your exposure is in a fine for 4% of global revenue or 20 million euros. Whichever is greater. No big deal. Europe may make up 10% of your revenue. The effort is going to decrease resources by 20%+ in the short term, add unknown long term complexity and overhead, and longer term employing a compliance officer. Also can we go back to the 20 million Euro fine thing for a second? Any potential acquirer is going to want to double down on diligence before assuming risk like that.

The requirements initially seem innocent enough - ability to hard delete/export user history. Easy right. Some notes from my meeting with legal:

Well what about backups? Do you retroactively go through your backups and remove R2BF users? Do you only keep backups for 6 days? What happens if you delete the user and restore from a 5 day old backup? Do you keep a second list of users who requested you delete their data - and if so how do you store that? How do you now represent activity that materially effected other users still on the platform? Is this subbed out with a "blank shadow" user? The legislation very much does not solve for the recursive logic of "how do you track users who've requested deletion". Counter-intuitively we would now have to log more user activity so we can gracefully handle rendering deleted data. Why do we have to file with a third party that we've deleted info on a user that requested we remove all trace of them?? What if a user enter's PII in table entries owned by other users through that other users account? What if they do it through a third party API? Speaking of, how do you ensure your third party API partners are in compliance? What if a user enters in PII in a field that you thought should have no PII in it? How do you treat EU citizens using your service outside the EU vs inside it? What if they use it in both places? What if you have an American user who happens to be located...

Sure most of these are solvable along with the another hundred edge cases not illustrated, but for bigger apps at 50+ person companies almost every one of those points is a bullet point in a meeting that requires nuanced development from multiple stakeholders. Don't think it's hard? Try roll into a meeting a Twitter and explain to them how easy it would be to add a button that lets you edit tweets. Some requirements are a BFD - you can imagine how things went over in a meeting where we half-jokingly suggested we no longer make backups rather than take on the burden of implementing a backup policy that may or may not be compliant with a new law that has no precedents in court to guide drafting a spec.

The net result is implementing it with the least effort and suddenly it's just this dogpile of cludgey UI popover screens that people click through without reading and TOS updates and mandatory log outs and a lot of other things besides that add overhead to make it "non-trivial."

It's a big deal because pre-series A startups will choose to avoid these waters rather than navigate them.

<comment><USER22> While I'd love to say I sympathise, I do not. The end goal is not to make life easy for small businesses. The end goal is protecting the consumer.

All your remaining points seem more easily solved than the backup bit, but I do not understand what could be so hard about deleting a user from a backup. Perhaps I'm dumb, but just go in there and delete it retroactively? If it affects other users, that's ok. They see a [deleted] a la Reddit.

You're not responsible for ensuring the API partners are in compliance. Those companies should have to fend for themselves.

When you delete _everything_ you stored about a user, you also delete _all_ potential personally identifiable information, so I again do not see what the problem is.

It just sounds like a bit of whining from where I am standing. It's like a small bridge builder saying "but the standards are too damn hard!". We start with the end goal of safe Bridges and safe users and then go backwards. Not the other way around.

<comment><USER23> One problem with backups is that you don't want them to be always online (store them offline and off site for disaster recovery). Backups can also be helpful in case of bugs that delete your whole database so that you can recover the data after the bug is fixed. Imagine the follow query `delete from users where id = 372936 or status = inactive`. Unfortunately, due to a bug, all users have been marked as inactive. Now you destroyed the whole database. No problem you think, just restore it from a backup. Due to the new law, this script ran also on your backup. Congratulations, you have now lost all your users. Hence, you never want to "change" backups. You only ever want to restore from them.

The easiest way to comply with it would be have a list with all user deletion requests stored somewhere else and every time you restore from a backup, you also apply this list from the backup. But have you really deleted users if you keep backups?

<comment><USER22> It's not a shell script that you will write each time a user asks for a deletion though, would it? It will be a carefully tested and automated procedure.

It certainly sounds like the possibility of deleting all your users might actually make us all more careful, less gun-slinging, shoot-from-the-hip programmers.

<comment><USER23> It will probably be an SQL query and if software development has thought us anything is that testing is not enough. Proper testing can be very helpful and prevent many errors but it can never prevent all because you just cannot test for all circumstances. This is why backups are helpful and why backups should not be changeable. It is a reason why backups were made on magnetic tab back in the day so it wouldn't be overridden by mistake. Backups were just as important back when programming wasn't all about that fast iterating and push updates and fix later – or gun-slinging programming as you call it – as they are now.

</comment></comment></comment></comment></comment><comment><USER24> By looking through the comments it seems many seem to think that smallest GDPR violation will immidiately see 20M fine. Isn't it the last method if previous methods are ignored like giving a warning, smaller fines etc.

So you don't have to do every single perfectly, just that you the basics done properly and your filling to fix things as they are shown to be broken. Well, unless your broken system leaks millions of users data into the wild I'd guess you're pretty safe.

</comment><comment><USER25> I believe GDPR is like a pre-annoucement of GFoC but for Europeans.

</comment><comment><USER26> How will GDPR interact with the goals of intelligence agencies and their contractors? Is there going to be a wave of problems for intelligence contractors and their possession of data or are European countries going to let it slide?

<comment><USER12> Isn't what most intelligence agencies do illegal in other countries already?

<comment><USER26> Probably. Although there is the Five Eyes style arrangement where you don't spy on your own citizens and instead spy on citizens of other countries and then share the data between each other so it all becomes legit. This is described here: [https://en.wikipedia.org/wiki/Five_Eyes#Domestic_espionage_s...](https://en.wikipedia.org/wiki/Five_Eyes#Domestic_espionage_sharing_controversy)

<comment><USER12> I think those agreements come from the fact that the agencies cannot 'spy' inside their home country, but can get information from other agencies that happen to have info about their country.

I still think it is illegal for those other agencies to spy within the home country. Not to mention countries they don't have agreements with.

The home agency does not do anything illegal itself, but the other agency does illegal stuff for them.

</comment></comment></comment></comment><comment><USER27> >Unlike the giants, the ad tech firms have no direct relationship with consumers. They say Google's and Facebook's response pressures publishers to seek consent on behalf of dozens of ad tech firms that people have never heard of.

This is exactly what I said back in November.

[https://news.ycombinator.com/item?id=15769544](https://news.ycombinator.com/item?id=15769544)

<comment><USER28> Target the site, not the users. Simple as that, arguably you'd have much better accuracy targeting an interest rather than trying to profile each IP anyway.

</comment></comment><comment><USER29> Due to the GDPR Facebook added all kinds of new controls. I looked through them today.

Stuff Facebook has tracked, despite not using their app on my phone (preloaded but completely disabled) and using their website only with uBlock Origin):

a) everywhere you ever used Facebook to login to a website. I thought I never did this but apparently I still had 5+ websites linked to this. They seem to use this to profile you. Facebook mentions about this _"These are active apps and websites. This means that you recently logged in to them using Facebook, and they can request information you've chosen to share with them."_ I never wanted to share anything, but Facebook often changes settings or pretends I did

b) every time you connected an app to your Facebook account. Apparently I again did at a few times (to easily upload some photos). I also saw old phones in here, Runkeeper (don't recall using that), Tinder (haven't used in years). All of those seem to forever be able to get information from you.

c) topics (yet another new name). Not sure how they figured things like that out. Fortunately you can completely disable the personal tracking, though you need to do that in various different places.

d) advertisers whom added their contact list to Facebook. Really!! So some other company shared my details with Facebook. Which means they've also shared details of people not using Facebook with Facebook.

e) used websites and apps. This is surprisingly accurate and complete (hundreds of sites+apps!), despite having the Fb app disabled plus always using uBlock Origin. I really wonder if they somehow are able to retrieve the browser history. It seems Facebook also tracks you using "apps using Facebook technology", which I guess is a library to show ads or something which uses the app permissions to further track the user.

f) it uses your profile (relationship status, employer, job title, education) to targets ads

g) somehow puts you into categories. E.g. "uses a mobile device (xx to xx months)". I disabled this crap yesterday, it updated it today with more information. I assume through Whatsapp/Instagram or maybe some kind of Facebook ad library? Again, I don't want it to track me, yet they seem to be easily able to do this. I turned off all the settings but apparently still missed a few.

i) ads topics. Not sure how topics differs from the 3 other ways they call this.

j) your location. It doesn't show what it does with this. You cannot disable the location tracking; it suggests to turn it off in the phone settings (which is a cop-out). I'll need to download my data to further determine this. I'm pretty sure Facebook is big enough to figure out additional ways of location tracking than just what Android/iPhone allows.

k) websites visited. I assume any link clicked. This is somehow different than all the other ways it already showed websites.

l) the existing "Who can contact you using email address / phone number". It says "Who", but it is used for the API as well, that's not a user so I find the usage of "Who" misleading.

Despite doing all of this, still not sure how to prevent something like Cambridge Analytica from downloading my data through others ("friend"). Further as I already mentioned it seems to continue tracking me despite turning everything off. Not cool.

</comment><comment><USER30> why do you keep posting paywalled content?

</comment><comment><USER31> Why do people keep posting WSJ paywalled content here?

<comment><USER32> They often have quality content.

<comment><USER28> This sure doesn't seem like it.

</comment></comment></comment><comment><USER33> Fine by me. Do we really need more "small ad firms"?</comment>


Show HN: I made a physics-based Windows Phone 8 Game – Happy Birds - <AUTHOR> http://www.windowsphone.com/en-us/store/app/happy-birds/f99c6c82-3a93-4331-897a-ea1b68399f8c

<comment><AUTHOR> I'm a big fan of HN (I don't post much though) but I just wanted to show off my new WP8 game, cheers!</comment>


Yahoo's new TOS: no class action lawsuits; third-party arbitration - <AUTHOR> https://policies.oath.com/us/en/oath/terms/otos/index.html <comment><USER2> I love the way they frame "binding arbitration and waiver of legal rights" in their friendly update mail:

"We've added a mutual arbitration clause. Hopefully, disputes will never be an issue, but in the case of one, this allows a third-party arbitrator to help us resolve them. We've also added a class action waiver."</comment>


5G May Never Live Up to the Hype - <AUTHOR> https://motherboard.vice.com/en_us/article/59xnw8/5g-may-never-live-up-to-the-hype <comment><USER2> 5G has a compelling use case for dedicated point to point as Verizon demoed and will be rolling out commercially [https://www.fiercewireless.com/wireless/verizon- ceo-5g-trial...](https://www.fiercewireless.com/wireless/verizon- ceo-5g-trials-going-extremely-well-expectations-to-deliver-1-gbps)

For handsets? Not so clear.

</comment><comment><USER3> For one thing the talk has to be about getting across the finish line as opposed to "who is winning the race".

5G and AI are both stuck in a pernicious narrative where nobody can say "This is going to create X amount of value" but they can speculate "Maybe X is greater than Y or maybe Y is greater than X"</comment>


Ask HN: How to patent my idea and codes? - <AUTHOR> I added a new feature to my chatbot startup that we are literally the only chatbot startup that did it.
I need to patent the idea and code if possible.
I need to know the cost and where to start?
And does patent really would make a market entry barrier for my competitors who might try to copy/streal the idea?
Thanks in advance <comment><USER2> When you start to worry about someone copying your idea then you have already lost. If you have something that is worth while, be the best at doing what it is that you want doing. If others copy your idea then use that as a marketing point (My idea is so good that others want to copy it).

Onec you go down thae path of patents, it is going to cost your dearly - lawyers, patent office fees and there is no guarentee that you will ever get any return on your time and effort.

Unless you are a extremely cany business person who is able to manage all the various aspects of the entire process, stick with being the best there is in your very specific field and use that for sales and support.

You have built on what others have done before you and there will be others who build on what you have done. Trying to stop this is like trying to stop the tide coming in.

In my own personal case, any new ways of doing something or developing something are given way to anyone who wants them. I get the kudos of that work and I get further work out of my abilities, not that that is any consideration today as I am effectively no longer in any industry.

</comment><comment><USER3> The advice about jurisdictions here is sound with the caveat that you will probably be able to limit yourself to some jurisdictions as being really important for you to focus on.

As far as patents I would only find them interesting if they can be used to secure you more funding. Some sources may consider it a plus if you have something patentable.

Probably for a startup however the cost of patents will be more than they're worth, just like for a startup having a dedicated devops team and onsite servers probably isn't worth it.

</comment><comment><USER4> Getting the patent is only a start and doesn't stop anyone from copying. You have to actively search and litigate which costs even more money. You also have to do this in each jurisdiction you care about which is hard when Internet services can choose to locate anywhere with global reach.

</comment><comment><USER5> Patent law varies wildly from one jurisdiction to the other.

In most jurisdictions "ideas" are not patentable and a lot of jurisdictions place strict limitations on software patents.

</comment><comment><USER6> The standard advice applies here:

Consult a patent attorney.</comment>


Show HN: Estimating Service That Ensures You're Profitable - <AUTHOR> http://www.penroseapp.com/

<comment><USER2> Hi everyone,

We built Penrose to help freelancers and other small businesses emphasize their profitability while estimating projects. The tool is flexible enough for virtually any service-based business--not just web-based.

We're currently in open beta and are anxious to hear what users think of the toolset we've put together, and more importantly what we can do better.

If you've got any questions or suggestions, please let us know here.

</comment><comment><USER3> Neat!</comment>


Pokemon Go Gets a Huge Revenue Boost - <AUTHOR> http://www.geek.com/games/pokemon-go-gets-a-huge-revenue-boost-thanks-to-halloween-event-1677654/ <comment><USER2> It's interesting that a simple themed update can result in a revenue spike, no update to core functionality, just a few tweaks to the interface. It makes me wonder in what other applications an update like this can produce similar results.

</comment><comment><USER3> Like a lot of my friends and coworkers, everyone was playing Pokemon Go and then they weren't.

It's a shame because it gave everyone a chance to be glue to their phones and be social in public. A game changing update like pokemon battles or PVP would rekindle my interest immediately.

It's possible these kinds of features are too risky for Nintendo given that Pokemon Sun and Moon are coming out in just a few weeks.

<comment><USER2> I think Ninantic's main issue with losing users is with not rolling out significant enough updates. There's no real sense of achievement in the game, and so when levels get particularly hard to move on from users lose interest in the game.

<comment><USER4> The main flaw is the inability to battle other players. The only battles are in the gyms and it becomes boring or too hard (the only gyms around me are protected by kids with way too much time in their hands and their pokemons are too strong for a filthy casual like me).

<comment><USER5> The same problem exists in Ingress as well if you live in even a moderately populated area. Ninantic's core issue (in my mind) is that the longer their games run, the less approachable they become. They haven't figured out a solution for Ingress, so I'm skeptical that they'll figure out one for Pokemon Go.

Granted, Ingress is a little more "hardcore" in the sense that it's not really focused at a more casual market like the Pokemon franchise generally is.</comment></comment></comment></comment>


Scholarpedia extends $10K article writing contest - <AUTHOR> http://blog.scholarpedia.org/2013/01/23/10-reasons-to-participate-in-the-brain-corporation-contest-now-that-the-deadline-has-been-extended/

<comment><USER2> Great news, particularly for PhD students! Some Scholarpedia articles get a lot citations (ex. the page on the NEST simulator has been cited 186 times) and can be a great way to get your name out there.</comment>


Did Alexa Die and No One Notice? - <AUTHOR> http://www.inquisitr.com/did-alexa-die-and-no-one-notice <comment><USER2> _crickets_

</comment><comment><USER3> Maybe they're having problems with their new system that they implemented? I wouldn't worry too much anyway, it's not like Alexa is particularly accurate, our site is getting more traffic than ever before but somehow Alexa doesn't think so...

</comment><comment><USER4> Seems to fall behind for days/weeks sometimes without getting updated. I think they should rename "yesterday" to "Some day in the last month"

</comment><comment><USER5> for being so off on stats for any site smaller than top 1,000 and being way off for those as well ... I don't understand how the service ever was bought for $250mm

<comment><USER6> It was 1999.</comment></comment>


College Enrollment Skids for 8th Year in a Row in 2019 - <AUTHOR> https://wolfstreet.com/2019/12/16/college-enrollment-skids-eighth-year-in-a-row-in-2019-but-student-loans-skyrocket/ <comment><USER2> ...causing skid row

</comment><comment><USER3> The article answers its own question: men are increasingly not enrolling in colleges.

This shouldn't be terribly surprising given how anti-male the environment has become.</comment>