Deep Learning's Impact on Image Processing, Mathematics, and Humanity - <AUTHOR> https://sinews.siam.org/Details-Page/deep-deep-trouble-4 <comment><USER2> I question this meme that deep learning has less mathematical elegance or interpretability than other machine learning model. It's simply not true. At this point, it reads like a mindless copypasta with the possible political intention to reduce AI funding and use, such as the EU regulation which requires all algorithms to "explain" their reasoning. Either that, or the author is just more comfortable with familiar old-school methods which makes them think that those methods are more elegant. This push towards elegance is unscientific and counterproductive, much like the mistaken push Einstein and co. made for deterministic interpretations of quantum mechanics. We can now understand that its motivation was simply due to a biased notion of elegance due to familiarity with classical mechanics.

Examples of elegance in deep learning: The gating mechanisms of LSTMs (also adds to interpretability); structure of convolutions, i.e. weight sharing, translation invariance; the policy gradient theorem (Sutton 1999); dropout, its relationship to biology and variational inference; Generative Adversarial Networks, and connections to game theory, split-brain; the reparametrization trick; the log-derivative trick; connectionist temporal classification. These examples are only the surface. Depending on your specialization inside deep learning, you'd find many more.

Examples of interpretability: Nguyen et al.'s Synthesizing preferred inputs to hidden neurons, Zeiler et al. convnet visualization, Guillaume Alain's linear probes of hidden layers, attention readouts in attentional models for machine translation or speech synthesis, and many more. Ultimately, deep learning methods are probabilistic, and decent deep learning engineers would be able to tell you why a model is doing what it's doing by printing probabilities and activation statistics, much like any other probabilistic machine learning model.

<comment> maze-le >> This push towards elegance is unscientific

I think you are right on that one. The Standard Model for example is absoluteley not elegant, it is a huge convoluted mess of parameters and constants (just google for "Standard Model Lagrangian"). Yet, it is the best we came up with, explaining the dynamics of fields and particles[0], that surround us. Correct answers don't have to be elegant per se, its nice when they are, but it's not a prerequisite.

But, consider this: You can train a Neural Network to predict the distance of an object, thrown with velocity v_0, an angle of α, under the influcence of gravitational acceleration g. After a few hunderd rounds of training, a suited NN can reasonably predict the outcome of said experiment, with the input: v_0, α and g. And, as you pointed out, a researcher can explain to you why this is the case based on activation, feedback-loops, learning algorithm and other parameters. But neither the NN nor the researcher will be able to give you a rule like: "F=(G m_1 m_2)/(r^2)" to explain the underlying reasons for the object-trajectory dynamics.

A Neural Network can give you answers and predictions, yes, but you are not able to incorporate them into a wider theory, since the output is always numerical in nature. It is also always tied to one specific fact you are interested in and cannot give you a generalization.

[0]: In the model of QFT particles are also fields of a different kind.

<comment><USER3> > But neither the NN nor the researcher will be able to give you a rule like: > "F=(G m_1 m_2)/(r^2)" to explain the underlying reasons for the object- > trajectory dynamics.

Come on, you can just fit the NN numerical outputs with Excel and come up with tentative explanations for both the underlying elegant rule and the science. The point with NN-based science being you are not left guessing much, instead retrofit the NN numerical output and build a coherent narrative with other accepted results.

<comment><USER4> Without that equation, you cannot build the rocket that goes to the moon, however.

<comment><USER3> Feeding the retrofitted "blurred" equation to mathematicians and let them come up with tentative exact models, while being almost there already?

<comment><USER5> Anything that fits within the error bars is viable if you ask me. Whatever works works.

<comment><USER6> A lot of investors agree with you. Often they get very rich, especially if they head for the beach before the fit breaks down.

On the other hand...

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> This is an interesting article. It made me think of the socio-political consequences of the move towards neural networks.

Many of us, children of the personal computer revolution, were attracted to computers for its empowering and democratizing effects. Just with a computer you could build anything! You are as powerful as any of "them"! The proprietary software model was developed to monetize end consumers by enchaining them, but the free software model and the fertile communities of the early Internet showed that the model of personal computation could not simply be displaced by legalese.

But if we take the author's position on the impact of deep networks, the model of computation is changing again.

If teaching deep networks is the new way to write useful programs, our brains and personal computers are not enough... they are obsolete. We now need clusters and, most importantly, massive access to data! This gives extraordinary leverage to the hoarders of data and servers of the Internet, the googles and the facebooks.

I'd like to think that maybe we are just at the beginning, the early "mainframe neural networks" era. That we just have to wait until there is enough technology and new markets discovered to build the "personal neural network". That consensual, open and distributed ways of sharing data will emerge and that new massively parallel computers will become affordable by the masses. That the models of neural networks will become well divulged and simplified and kids will be able to program them with their "Neural Basic"...

But at the moment the prospects don't look good. The Internet is becoming more and more centralized, personal computers harder and harder to program and there is a general "war on general computation" [1]. Even universities seem to bedisplaced by the Internet Lords in driving neural network development...

Maybe it is already a good time to start thinking about what "Libre Neural Networks" look like. And how can we get there.

[1] [https://www.youtube.com/watch?v=HUEvRyemKSg](https://www.youtube.com/watch?v=HUEvRyemKSg)

</comment><comment><USER8> There seems to be a lot of angst in the responses about how deep nets et al "are too elegant; are too interpretable!".

It's important to remember that the intended product of science is not tools, it is understanding. Deep nets may produce very elegant and interpretable tools, from a very elegant and interpretable theory, but that is not what the scientists are looking for. They are generally creating and assessing theories _in their subject domains_ , which are then evaluated by the tools they have built.

Deep learning does do a great job of providing a baseline ("your theory must be this accurate to matter"), but it seems to do a much less great job at extracting new understanding.

<comment><USER2> Machine learning has departed from the venture of understanding-based science much before deep learning [1][2]. Thus, singling out deep learning reeks of motivated reasoning or just lack of awareness of the history of machine learning.

Having said that, there still exists the lunatic fringe (it's a compliment) inside deep learning who continue to work on the task of generative modeling in the hope of "understanding the world" rather than "solving a task". Yoshua Bengio and Yann LeCun don't miss an opportunity to impress on the world how important unsupervised learning is.

Physics too has abandoned understanding for predictive power. If you take philosophy of science seriously, you either have to pick "science is whatever the elite society believes in (Kuhn)" or "science gives us tools for prediction (Post-Kuhn)". Claiming that science helps in finding truth or understanding, is going to put you in a very indefensible slippery slope.

"Understanding" is a subjective human concept, and not worth pursuing. Humans evolved to run from tigers, and struggle with harder tasks like understanding quantum mechanics or understanding the brain. The directionality of scientific progress as evidenced by QM is not "understanding", but tooling and predictive power. I hope Deep Learning is going to follow the path of predictive power, rather than regress into the pseudoscientific mess of elegance and understanding.

[1]: [http://www2.math.uu.se/~thulin/mm/breiman.pdf](http://www2.math.uu.se/~thulin/mm/breiman.pdf)

[2]: [http://norvig.com/chomsky.html](http://norvig.com/chomsky.html)

<comment><USER9> "Understanding" is a subjective human concept, and not worth pursuing."

Isn't human understanding of how our environment works precisely how you're able to sit in your chair and write comments on HN for others to read?

<comment><USER10> Sure but I might say that if I were to create a chair by generative methods, would I need to understand how it was created for it to be useful?

<comment><USER9> If you have ever crafted a chair, table, or anything else that is built by craftsperson, you will soon realise why such a profession exists, because it's very difficult.

You might slap a few pieces of wood together and by brute force, create something like a chair and that might do, but it's very unlikely to be a good, safe, attractive, comfortable long lasting chair. Building quality furniture is complex and difficult work which takes a lot of skill, experience and understanding in order to do.

I understand your point, I never said there is anything wrong with experimenting with NNs, I just don't agree with the OPs sentiment.

</comment></comment></comment></comment></comment><comment><USER11> I echo the feelings of the author of this piece, and I'm very wary of a world where predictive power always trumps deeper understanding.

(Edit: I knew I'd expressed this feeling before: [https://twitter.com/copingbear/status/825098385548009472](https://twitter.com/copingbear/status/825098385548009472))

</comment><comment><USER12> I have no doubt that some day (soon?) neural networks will be able to explain themselves (using language). But the explanation they give will be less than satisfactory to our aesthetic criteria. Occam's razor is a great aesthetic rule, but it doesn't guarantee beauty. We will most likely find out that the computational vision of the past had managed to find some rules covering some cases, but it missed many more, most of which will be higher-order and not that elegant. And it may turn out denoising was a rather messy problem, as was probably speech recognition.

Drugs work but we don't know how a surprisingly large number of them work. Yet chemists don't complain about them. Theoreticians should not complain either. Deep nets give them a huge subject matter that probably hides many interesting insights in it. I mean, don't convolutional layers look like gabor filters?

The question of what can be understood from them can go deep, to the limits of the ability of language/math to express ideas.

<comment><USER13> I don't see a problem in splitting all these domains to an "applied" school, that focuses research into evolving solutions that currently seem effective in approximating answers for certain engineering problems, and a more theoretical school that has the potential to carve new revolutionary paths while pursuing some (abstract?) ideal such as mathematical elegance. One shouldn't exclude the other and apparently there are enough scientists around that are attracted to either one of the two approaches.

</comment></comment><comment><USER14> Deep learning/neural networks -- as well as a number of other "machine learning" methods -- is fitting a mathematical model with a huge number of fitted (tunable) parameters and component functions to data. It has been known for a long time -- arguably back to the discovery of the Taylor series expansion in the early days of calculus -- that any function can be approximated arbitrarily well by the composition of an arbitrarily large composition of other functions.

If the task is interpolation between the data points this can be highly accurate. It the task is extrapolation such as prediction or designing a truly new machine or system in an engineering application, the approximation will often fail. The about one percent error rate in predictions of planetary motions from epicycles is one of the earliest cases of this problem.

The simple example is approximating data with a polynomial with an arbitrary number of terms such as in the Taylor series expansion. With enough terms a polynomial model can always approximate any data set arbitrarily well. Polynomial models can interpolate very well unless the data has some generally unusual behavior -- changing unpredictably at successively finer scales for example. However, finite polynomial models almost always extrapolate grossly incorrectly. As you move away from the data set in the space of independent variables such as X, the largest power N in X^N dominates and the polynomial approximation function blows up to either plus or minus infinity which is rarely physical.

What we think of as "understanding" corresponds conceptually in part to the ability to make accurate predictions. "Understanding" or "explanation" corresponds mathematically not to some arbitrary super-complex function with large numbers of arbitrary parameters but rather to a mathematical object such as as system of differential equations (e.g. Maxwell's Equations for Electromagnetism or the General Theory of Relativity) that express interrelationships among the variables and data points.

<comment><USER15> Machine learning researchers are well aware of the problem of out-of-sample predictions, in fact most of the work goes into making systems appropriately generalizable by making architectures robust through dropout or regularization. The resulting systems might not correspond to our intuitive definition of "understanding", but they look more similar to what we know about the human brain than an elegant function with a small number of interpretable parameters.

</comment></comment><comment><USER16> First comment I make on HN, but I feel compelled to say something. This post reeks of the author piping on about the "good old days" of mathematical elegance.

Firstly, mathematical elegance is not an objective metric by any means. Many mathematicians disagree on what results are most elegant, though there are many commonalities as well.

Secondly, all of machine learning is quite elegant. I'm by no means an expert since I only took two courses in my fourth year that were surveys of many methods (quite mathematically rigorous btw) but I've taken enough math to be able to judge what is elegant or not.

As the author himself says, he has slightly modified his research methods. These days you'd be a fool to ignore deep learning, whether you have a deep understanding of it or not.

<comment><USER17> I felt the article was quite fair in acknowledging the intriguing results of deep learning, and decidedly arguing against ignoring them.

"Elegance" can indeed not be defined, but I believe the misunderstanding here is more about the point of view: Deep neural networks are plenty elegant when looked at from an ML POV. But, if you're an expert in, for example, image processing, using a DNN as a tool, the solution it may give you won't be "elegant" by anybody's definition. It is, after all, a repetitive formula with X million arbitrary floats.

Previous ML methods usually resulted in models, or formulas, that were small enough to "grasp" intuitively. The "beauty", or "elegance" was that often, you could find connections from terms in your model to the real world.

<comment><USER18> > Previous ML methods usually resulted in models, or formulas, that were small > enough to "grasp" intuitively.

Which is not true in practice. SVM in reality is an instance based method, with thousands to 10 thousands high-dimensional vectors as its 'parameters'. Random forests, which is famous for its ease of interpretability is no easy meal either when you end up with 100s of trees with outrageous branches. Not mention that real world model are embarrassingly complex ensemble of smaller but still complex models.

</comment></comment></comment><comment><USER19> Deep learning is a perfectly useful engineering solution.

As a tool for basic research, particularly in the biomedical realm, it's awful. You get a system that performs pretty well most of the time, but tells you nothing of interest.

That's fine, as far as it goes, but "performs pretty well" is so much more useful in industry than "we know how X biological system works" that entire fields that are interested in the mechanics of how perceptual biological systems work and may be fixed are getting eaten alive. To our collective detriment, I think.

</comment><comment><USER20> Author mentions style transfer as a product of NN research.

Yet this paper [1] from 2014 works extremely well, no neural networks. Here's one from 2008 [2] that worked extremely well too, better than most early NN techniques.

Although, deep learning is far from the intuitive approach that existed 10 years ago. It's clearer now how to reason about models, layers, activation functions etc. As much as there's no mathematical foundation, I do not believe it helps that much in the case of SVMs or whatever else. There still has to be experimentation, approximation and proper testing.

[1]: [https://dspace.mit.edu/handle/1721.1/100018](https://dspace.mit.edu/handle/1721.1/100018) [2]: [http://link.springer.com/chapter/10.1007%2F978-3-540-88690-7...](http://link.springer.com/chapter/10.1007%2F978-3-540-88690-7_3?LI=true)

<comment><USER21> To claim that those techniques are even in the same ballpark as deep learning based style transfer is just silly. They are extremely limited in their application, as you can clearly see by looking at the examples. And the entire approaches are overfit to the particular applications that they show, so they can't be readily extended.

These are actually excellent examples of how deep learning has totally changed our expectations of what can be done with computer vision by using deep learning.

<comment><USER20> Deep learning based style transfer that got viral a year or two ago was worse than this 2008 result, especially worse for that hone application to portraits.

The hype was nowhere near just because there's no deepness.

I admit that [https://arxiv.org/abs/1705.01088](https://arxiv.org/abs/1705.01088) this beats everything and is extremely powerful and simple but the hype for deep things seems a bit too strong.

</comment></comment></comment><comment><USER22> > To put it bluntly, your grandchild is likely to have a robot spouse.

I'll take the other side of that bet.

<comment><USER23> I was already on the fence about the article being standard deep learning click bait when I saw that :)

</comment></comment><comment><USER24> In my opinion, this push towards "mathematical elegance " and "interpretability" is hurting us and is unscientific. About interpretability, I'll quote Yann LeCun here, you don't really care too much about whether or not your taxi driver is interpretible, he's a black box. Not only that, but there are a variety of new methods such as attention mechanisms, gradient masks and auxiliary explainer networks that give really good interpretability. And it's funny how this criticism never comes up for SVMs...

I also think it's unscientific to expect mathematical elegance from everything. Math is ultimately a logical descriptor tool for AI, a means, not an end in itself. Besides, there's actually a host of involved mathematics underlying the seemingly simple SGD of deep nets. For example, tropical geometry has been used to analyze the loss surfaces of ReLU networks and random matrix theory has been employed to analyze the loss surfaces with respect to the quality of local minima.

Tl;dr: deep nets are far more interpretible than they've been given credit for and they're also more mathematical than some (such as this author ) would have you believe.

</comment><comment><USER25> Good Computerphile episode on understanding what is going inside a neural net, [https://www.youtube.com/watch?v=BFdMrDOx_CM](https://www.youtube.com/watch?v=BFdMrDOx_CM) . Looking at high information neurons to see what they pick up can yield a lot of insight into the problem.

</comment><comment><USER26> But this is true of neural nets as well, so I'm not sure why the author was trying to contrast them. Essentially it is a question of do you want a system that works well but that no-one understands or one that works not so well but is understandable? But if you meta it, it's the same.

<comment><USER2> I don't think even this dichotomy is real. Deep learning methods are both performant and elegant.

<comment><USER27> They're elegant only if the sentence 'consider a model with a billion features' doesn't make you feel disgusted, I'm afraid.

<comment><USER28> elegant, _adjective_ : (of a scientific theory or solution to a problem) pleasingly ingenious and simple.

One easily-described approach that is state-of-the-art in a massive number of domains? That's pretty f'ing elegant.

</comment></comment></comment></comment><comment><USER29> Many methods in machine learning and statistics benefit from two types of theory:

1) Optimization theory - which says, if I repeat this iterative method N times I will find a (nearly) globally optimal solution

2) Statistical theory - which says, if I observe this process N times I can accurately estimate a population quantity with high probability

Deep learning does not benefit from the same theoretical guarantees.

For the most part the response from the community is "but it works really well!" which is a fair and valid response especially since what most practitioners care about is predictive accuracy.

Personally, I find applying neural networks extremely annoying at times due to the amount of twiddling of hyperparameters, slow convergence, etc.

</comment><comment><USER30> " The facts speak loudly for themselves; in most cases, deep learning-based solutions lack mathematical elegance and offer very little interpretability of the found solution or understanding of the underlying phenomena."

I don't agree with this statement, a simple look at cs231n lecture series will show you how much math is involved. A lot of articles/people claim its a black box, but while writing a small network architecture you realise it's not. Methods like stride, padding, activations, learning rate, optimisations, drop outs etc. give you "aha" moment which is followed by a mathematical explanation. One should study the topic thoroughly before criticising it.

<comment><USER31> Having a lot of math involved does not mean that it is mathematically elegant. As a mathematician, I ask myself several questions. First of all, what is really a neural network? Is it an approximating function? Is it a geometric separation on a space, such as SVM? Is it a manifold classificator? (see [http://colah.github.io/posts/2014-03-NN-Manifolds- Topology/](http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/), which is very interesting)

Also, what are we approximating? Continuous functions? Non-continuous functions? Are they even functions and not probability measures? Are those functions arbitrary or do they represent something like a manifold?

And the most important: how well are we approximating whatever we want to approximate? The universal approximation theorem gives uniform convergence for measurable functions, but do not specify at which rate or depending on which parameters. It is a strong theorem but not that surprising from the mathematical standpoint, where you already know that you can approximate any function by continuous, compactly supported functions.

Finally, how do you mathematically define the problems that arise in neural networks? What is overfitting? How does the learning algorithm affect the results?

The fact that some techniques are justified by mathematical explanations does not mean that it is mathematically elegant. For it to be mathematically elegant you should have at least clear definitions of the objects of study and the problems you want to solve. I don't think this is the case in neural networks.

<comment><USER30> "Having a lot of math involved does not mean that it is mathematically elegant" I agree to your point, but at the same time how much effort is being made to understand the intricacies of deep learning is my question. In my opinion, those who dont understand these techniques bluntly say its a black box but its not entirely a black box. I am confident if more reearchers start to peel it off layer by layer , a lot more insights will be generated given the field is relatively new .

</comment></comment></comment><comment><USER18> > deep learning-based solutions lack mathematical elegance and offer very > little interpretability of the found solution or understanding of the > underlying phenomena

I disagree. Elegance and interpretability comes with one big flaw: assumption, lots of them. While deep learning based methods assume few or not. If they are so elegant why would them underperform? Probably it is because the problem we are trying to solve doesn't follow the assumptions, like convexity, etc.

In that sense, one can say deep learning methods are even more elegant, because they can work end-2-end. In that sense, NN can be a blessing, because now we got the answer, we don't need to assume anything, but just to DECODE it.

<comment><USER32> There is a no free lunch theorem in learning, which says that without limiting your hypothesis space, you cannot learn. That means, you need to have some assumptions.

No doubt that DNNs are subject to the same theorem, the only question is, how does their hypothesis space looks like? Does anyone have idea about this? I suspect we don't really know what the DNNs assumptions are.

<comment><USER33> The DNNs assumptions are "The true model can be fit by this network architecture". If you use convolutions, then an additional assumption is translation invariance.

<comment><USER32> > The DNNs assumptions are "The true model can be fit by this network > architecture"

Yeah, but that's tautological. What I mean can somebody sit down and for a given DNN architecture, write down (at least approximately) the set of functions that it can learn? Or more importantly, what functions it cannot learn? Or at least, how many bits are assumed and how many bits have to be learned?

I think that is what bothers people about DNNs. I personally think they are sometimes even inefficient - we are learning them much more parameters (bits) than the actual hypothesis space requires.

<comment><USER33> > What I mean can somebody sit down and for a given DNN architecture, write > down (at least approximately) the set of functions that it can learn?

For a two-layer architecture with ReLU activations and n units in the hidden layer, this is the set of piecewise linear continuous functions with n kinks.

<comment><USER34> How many of those do you need to tell the difference between cats and dogs in random internet pictures?

</comment></comment></comment></comment></comment></comment><comment><USER35> > In its feed-forward architecture, layers of perceptrons—also referred to as > neurons—first perform weighted averaging of their inputs, followed by > nonlinearities such as a sigmoid or rectified-linear curves.

I have a really hard time keeping reading after this passage.

</comment><comment><USER36> You can do regression

Exam score= cat_a + cont_b + err

but how do you say something like cf male female has 10 score advantage and family income impact is ...

Can Galileo invent physics by doing nn

</comment><comment><USER37> > _" But what about us scientists? What is the true objective behind the vast > effort that we invested in the image denoising problem?"_

This is the telegraph operator lamenting all the time spent learning Morse Code when the telephone was invented.

I think scientists can be motivated by looking for new ways to solve real problems. Don't weep over the time spent understanding the classic models. Rejoice that we have a much better tool to further mankind.</comment>


An inside look at Quantum DOM Scheduling - <AUTHOR> https://hacks.mozilla.org/2017/06/an-inside-look-at-quantum-dom-scheduling/ <comment><USER2> I'm still waiting for native delayed rendering solution with transactional and atomic DOM operations from browsermakers.

In one talk with a Mozilla guy from Vancouver we went over it and it came to "framework pushers in WHATWG will never agree to that as it obviates the need in Angular/React/others and their Shadow DOM peg leg workarounds"

<comment><USER3> Is that true though? The shadow DOM diffing isn't the point of React. The point was to turn rendering into a function that takes application state and returns the view. The shadow DOM diffing stuff is just an implementation detail that made this model fast enough to be practical.

<comment><USER4> You're mixing up react's virtual dom with the native shadow dom.

<comment><USER3> Yes, sorry, I meant virtual DOM. But other than that, my point still stands.

</comment></comment></comment></comment><comment><USER5> We have SharedArrayBuffer (shared memory-ish) and Atomics (in latest Firefox) so we have multi-threading in JS (whether it's a good idea or not that's a completely different story) but we still don't have the ability for multiple threads to work with the DOM. We need a SharedDOM construct with Atomics. Well, just because multi-core, right? What am I missing?

<comment><USER6> [https://community.oracle.com/blogs/kgh/2004/10/19/multithrea...](https://community.oracle.com/blogs/kgh/2004/10/19/multithreaded- toolkits-failed-dream)

</comment></comment><comment><USER7> we need to stop calling things quantum that are not quantum - its confusing

<comment><USER8> I got confused too.

We can't stop people from naming their projects/products the way they want. However, I think headlines in HN should be editoralized a bit to provide context, followed by the actual headline from the webpage.</comment></comment>


Show HN: Material UI Color Selector - <AUTHOR> http://www.materialui.co/colors

<comment><AUTHOR> Our Team has developed this.

We will be adding various component like Material UI code snippets, sample themes, etc. soon.

Please provide your valuable feedback.

</comment><comment><USER2> This is awesome!</comment>


Ask HN: How to monetise helmetrex.com? - <AUTHOR> We started HelmetREx project http://helmetrex.com in order to help developers in creating UI for Web, and not only JavaScript developers.
We want to create a place where developers can find and share boilerplates for Web applications which they can use immediately as starter prototypes of own application.
HelmetREx is tightly integrated with visual tool for prototyping React UI Builder https://www.npmjs.com/package/react-ui-builder, which allows to prototype UI of any complexity using any set of ReactJS components, and helps to generate starter clean source code for new components.
This project is really big, and we need resources.
So, we want to ask how to monetise all that ? <comment><USER2> can you create a market place like a standard theme website? those things make $$$$

<comment><AUTHOR> Yes we can, and actually now we started to implement this. But my question is more about how to attract the investors to our project.

<comment><USER3> If your goal is to attract investors, you have to show them how they are going to make money backing you because that is the most important thing. Also, you should reframe your initial post if the question is about attracting investors..:)

<comment><AUTHOR> Thanks, here is my reframing: [https://www.quora.com/How-do-we-can-monetise- ReactJS-Compone...](https://www.quora.com/How-do-we-can-monetise-ReactJS- Component-Exchange)</comment></comment></comment></comment>


The Limits of Correctness (1985) [pdf] - <AUTHOR> https://www.student.cs.uwaterloo.ca/~cs492/11public_html/p18-smith.pdf <comment><USER2> Holy crap, I knew that name ringed a bell. At first I confused him with Brian Cantrill (dtrace et al. fame)... but this is Brian C. Smith of 3-Lisp reflective tower interpreter.

[https://en.wikipedia.org/wiki/Brian_Cantwell_Smith](https://en.wikipedia.org/wiki/Brian_Cantwell_Smith).

I didn't know any of his other papers. Beautiful.

</comment><comment><USER3> As a primer on the many and significant difficulties of analytical verification, this may have some merit; as an argument for the futility of attempting to do so, it depends on a false all-or-nothing dichotomy.

</comment><comment><USER4> This is a beautiful text.</comment>


US Admits It Uses Predictions to Blacklist Flyers - <AUTHOR> http://www.wired.com/2015/08/security-news-week-us-admits-uses-predictions-not-data-blacklist-flyers/

<comment><USER2> Why would US law enforcement do this? I mean, I understand their economic incentives, but predictive blacklisting seems to violate a number of sacred principles, like presumption of innocence, due process of law and if the algorithm is set up right, freedom of speech. Don't law enforcement officers take an oath to defend the US constitution? Doesn't this bother anyone involved other than Snowden?

<comment><USER3> US Law enforcement does what it's told (more or less) and acts on the orders of the various governmental entities, from the Federal level on down to the tiniest incorporated entity with it's own budget and cop. Some politician or bureaucrat decided how to do this, not a cop.

Do you really still think that the US government pays attention to any of the constitutional stuff?

I wish with my whole heart that they did. All that civics crap from the schoolbooks is little more than propaganda. It's been roundly ignored since way before I was born.

Too many lawyers, too many overlapping laws created simply to make a politician seem busy for his sheep before for election time, too many politician's bought and sold.

</comment></comment><comment><AUTHOR> > Why do this ?

Preventative measure.

> violate sacred principles

Algorithmic blacklisting is a standard of proof.

Some others are :

      Reasonable suspicion
      Probable cause for arrest
      Beyond reasonable doubt

Any preventative ( rather than curative ) measure will work only if it operates before the fact, and to do so, it works to operate under a standard of proof that is less than that required for an after the fact measure, being less than required for a curative.

So a custodial sentence is a curative ( restrain offender from doing more harm ), after the fact, and after the facts of the offense have been proven beyond reasonable doubt.

Arrest and detain, or prevent from flying, is a preventative, before the fact, based on a standard of suspicion.

The balance here is that the inconvenience caused to a detainee ( or a blacklisted flyer ) is less grievous, is better, than the possibility of a harm caused by not detaining or preventing that suspect. This balance is guided by the sacred principle of harm minimization ( protecting people ), and the preventative measure is justified.

Innocence until proven guilt does not make regards to suspicion. A person can be innocent ( and is by definition ) until they are proven guilty, and at the same time they can still be a suspect. Innocence until guilt does not equate with a person being beyond suspicion.

If suspicion was not a sacred principle, a lot of investigations would lead nowhere, and a lot of preventable harm, would occur. In fact, it's reasonable to argue that the courts often err on the side of devaluing suspicion, resulting in offenders who, once they are released and commit a crime, find themselves in the middle of a community backlash of people asking, "Why wasn't this person prevented from doing this?"

In the view of the courts, a reasonable suspicion, or a probable cause, or an algorithmically determined "black hand", is a workable balance in effecting harm minimization. Its carriage is the very due process of law, and by implication, a defense of the US constitution.

Another way of seeing this whole thing, is that any curative is by definition insufficient, by virtue of a crime having been committed, how can any punishment ever change that ? So even a curative, can be seen to be really a preventative on the possibility of future offenses carried out by that person.

I don't imagine Snowden would be troubled by this, as his whole crusade seemed to be founded on the idea, that if he releases the documents now, perhaps he can prevent the possibility of future abuses occurring, and create what he views as improvements. Snowden seems to believe more profoundly in the idea of a preventative measure even than the courts.

Finally, it's worth mentioning that algorithmic blacklisting is actually a step in the right direction. The lesser standards of proof ( such as reasonable suspicion and probable cause for arrest ) are by their nature and undertaking highly discretionary, and subjective, and also unscalable. An algorithmic means of administering assessments of suspicion, is something that you accept each time you pay the requested amount for your mortgage or use your credit card. The assessment of risk, carried out by machines and algorithms, is at least more fair and just, than if someone could deny your startup loan application, not because the algorithm said your idea was unlikely to work, but because the person simply didn't like you.

Algorithmic blacklisting provides the means to assess every person in the same manner, being the very definition of, "equality under the law."

To get reflexive, every time you find yourself suffering the inconvenience of being stopped at the airport for a random check, it works to remind yourself that this inconvenience is lesser than if you were dead, because someone hijacked your plane. You can't cure your way out of that, no one can. So while you champion that people are innocent until proven guilty, and deny the validity of suspicion, every day you judge people based on what you suspect them of being capable of. You don't let your child sit next to that man on the bus you think is strange looking, or you don't give money to that homeless girl, because, in each case, you make a judgement about what they person may do. You suspect your colleagues at work, you judge the people you just meet based on what they say, and you suspect your spouse, and yet when you find yourself in the position of being judged on suspicion, you claim the double standard for yourself, that the very thing you do, should not be done to you.

If you want to think clearly about things, it works to see clearly that the double standards you employ are a contradiction, and how useful suspicion is, both from a personal, and a legal, point of view.</comment>


In Ben Garrison, Alt-Right Found Favorite Cartoonist–and Almost Ruined His Life - <AUTHOR> https://www.wired.com/story/ben-garrison-alt-right-cartoonist/ <comment><USER2> _Through Patreon, he 's even become a modest financial success, though he's quick to point out that the platform doesn't reward his worldview as much as others'. "Pro-tranny cartoons get 15 to 20 grand a month, but conservatives, we don't get that kind of break."_

Since every Patreon supporter supports who they want, isn't that something of a free market? There's very little coercion - I've seen Youtubers and so forth saying I could support them on Patreon, but nothing more.

I think Garrison is indulging in some of the comspiracy mongering he uses to inspire his cartoons, or, at the very least, he should be noting how the Patreon market rewards him, rewards others, and emulate the winners without complaining.</comment>


Tanzanian woman presented as a man for 10 years so she could work a mine - k-mcgrady http://www.bbc.com/news/magazine-39705424 <comment> k-mcgrady NB: Despite the click bait headline it's an interesting article on succeeding as an entrepreneur in difficult circumstances.

Edit: Post title has been updated (thankfully), I originally had the article title.

<comment><AUTHOR> She is awesome. I thought from the title the story would end badly for her, but it ended awesomely.

<comment><USER2> Yeah. Is it correct to say that the only reason she could prove her innocence from the rape charge though is that she at that point revealed she was a women?

</comment></comment></comment><comment><USER3> Women disguising themselves as men to do "men's work" is probably as old as time. Here's documentation of women doing so to fight in the US Civil War:

[https://www.archives.gov/publications/prologue/1993/spring/w...](https://www.archives.gov/publications/prologue/1993/spring/women- in-the-civil-war-1.html)

[http://www.smithsonianmag.com/history/the-women-who- fought-i...](http://www.smithsonianmag.com/history/the-women-who-fought-in- the-civil-war-1402680/)

</comment><comment><USER4> I had no idea miners could actually hit the jackpot like she did and find enough gems to become rich with.</comment>


A plane flying non-stop for five years - <AUTHOR> http://www.flightglobal.com/articles/2008/03/03/221956/darpa-pushes-limits-of-unmanned-aircraft-capability-to-extremes.html

<comment><USER2> The business applications of such aircraft will be far more valuable than the military applications.

<comment><USER3> If you could mount a powerful cell phone repeater it would be more valuable than an Aerostat as whenever a cell tower is taken down, you simply need to fly one or more over an area to cover it.

They might even be a cheaper more efficient way of covering densely populated areas than paying exorbitant land fees and the constant battle to get them placed. I really don't see how a plane that can fly for 5 years wouldn't benefit the Cellular industry.

It would also help wildlife studies. I mean simply program one to follow a high density of radio transmitters, and tag as many birds as humanly possible before they migrate and then you've got perminant tracking and monitoring of the birds flight positions. The plane could be equipped to transmit the locations through a satellite telephone link or other such device.

I'm sure there are thousands of uses. Terrain mapping, you just strap a camera to the bottom and GPS and you could have it build up a high detailed map of an area. I mean it would be a lot more useful for any industry that requires land surveys as getting a high detailed map with a wide angled lense would allow you to build up a topological map.

Hell you might even be able to spot useful resources, or really anything visible from altitude. Police could use it to find recent graves with infrared and optical sensors and it'd be cheaper and faster in open areas than using people.

As I write this I realise the list is literally nearly endless, so I think I'll save myself the time and STFU now before I get carpel tunnel!

<comment><USER4> The possibilities are endless, but not all the scenarios you describe require planes that can stay in flight for 5 years. For a lot of the scenarios you describe it would suffice to have a fleet of planes that take off, fly, land, resupply and take off again completely autonomously.

</comment></comment></comment><comment><USER5> Okay, I was going to deride this (60000+ feet for a 7 horsepower propeller aircraft seems _ludicrous_ , a typical 200 horsepower light propeller aircraft can't climb above 16000 feet, and the thin air increases required propeller RPM as you climb). However, it seems that Boeing has already managed to send a solar-powered ROV to 67500 feet. Admittedly only for a few minutes, but it doesn't any longer seem like such a stretch.

Amazing what concepts a huge budget and no requirement to satisfy the public market can give you. I wish I could work on something like this...

</comment><comment><USER6> s/^(.*)$/\1\?/</comment>


Elon Musk Teases Second Part of 'Top Secret Tesla Masterplan' - <AUTHOR> https://techcrunch.com/2016/07/11/elon-musk-teases-second-part-of-top-secret-tesla-masterplan/ <comment><USER2> Any idea why my two submissions about this yesterday were flagged? But this one is getting through?

<comment><AUTHOR> Since the article from techcrunch was posted 3 hours ago, the submissions you posted may have contained a URL that already existed on HN.</comment></comment>


WoW No Longer World's Biggest MMO? - <AUTHOR> http://gigaom.com/2008/06/26/warcraft-no-longer-worlds-biggest-mmo/ <comment><USER2> Raph Koster did a presentation at GDC2007 that touched on this: <http://www.raphkoster.com/gaming/gdcprime2007.shtml>

The slides that struck me were this one: <http://www.raphkoster.com/gaming/gdcprime2007/Slide61.JPG> followed by this one: <http://www.raphkoster.com/gaming/gdcprime2007/Slide62.JPG>

Note that even back then Habbo had more active monthly users than WoW.

Of course Habbo and Runescape are free to play (at the basic level at least), but the costs of running them are so much smaller. Last I heard, Blizzard's WoW team numbered close to 1000 full-time staff. Jagex has a very small team that has only very recently started to expand.

</comment><comment><USER3> (WoW) paying subscribers vs (Habbo) free active user accounts

Not even a contest: WoW wins. Also, according to the Habbo site:

        * 2262 Habbos online now!
        * 2,845,022 visits in the last 30 days

How do these numbers translate to 9.5MM active accounts?</comment>


Gary Vaynerchuk Apologizes: Cannes Party Invite Seeking'Attractive Females Only' - <AUTHOR> http://www.adweek.com/news/advertising-branding/gary-vaynerchuk-apologizes-cannes-party-invite-seeking-attractive-females-only-172163 <comment><USER2> Real life imitating satire? Ali G did it before, in the movie "Ali G Indahouse".

 _I know how to solve the himmigration problem! We need more FIT WIMMEN!!_</comment>


Show HN: Alphabeat – Make music from text - <AUTHOR> https://epiqueras.github.io/alphabeat/ <comment><USER2> Did not understand what I could do, you could maybe add a tutorial or tooltips.

</comment><comment><USER3> Beware. When you open this page it emits horrible sounds.</comment>


China Spreads Propaganda to U.S. On Facebook, a Platform It Bans at Home - <AUTHOR> https://www.nytimes.com/2017/11/08/technology/china-facebook.html?rref=collection%2Fsectioncollection%2Ftechnology&action=click&contentCollection=technology&region=rank&module=package&version=highlights&contentPlacement=2&pgtype=sectionfront <comment><USER2> It's not surprising. China just adds itself to the list of countries with "you do it over there, but you don't do it here" laws.

* China fabricates propaganda on Facebook, but bans it at home.

* Germany manufactures and exports firearms, but tightly regulates them at home.

* The US manufactures and exports cigarettes and pharmaceuticals, but tightly regulates them at home.

* Israel markets binary option trading scams to the rest of the world, but bans them at home.

* Russia produces malware, but only punishes anyone who unleashes it at home.

There's a lot to be said about the credibility of products and services that governments won't let their own people have access to but freely sell to the rest of the world.

<comment><USER3> This should be a WTO principle or something: you're only allowed to sell products/services to foreign countries thay you allow to circulate internally and on similar conditions.

<comment><USER4> Why?

<comment><USER5> Because that's the globalist answer to everything. Find some organization that you feel is responsible for fixing some moral failing you perceive in the world. Lobby them, claiming that they are responsible for the moral failing and immoral if they don't fix it. Get some lame piece of regulation out of it that causes more harm than good. That harm becomes your next moral failing and the process repeats. But everyone involved gets to project the outward impression that they are making the world a better place.

<comment><USER6> How do you propose we better address perceived moral failings?

</comment></comment></comment></comment></comment><comment><USER7> I'm so glad my government, the United States Government, has definitely never spread propaganda in foreign countries with the express interest of creating a more favorable political climate there for their interests.

<comment><USER8> I think this opinion is very narrow minded. Who is the person you're trying to convince with this thought? No one is doubting that we are doing this overseas, that doesn't make it any less concerning that it is happening here. It is a weapon that doesn't result in war, I hope my government, the United States Government, uses this more as an early option instead of going to war. If anything, I'm upset they are not more effective at this.

When a foreign government disrupts my free elections though, I'm going to be troubled by it. What you're saying is effectively belittling the importance of hardening our own systems from this foreign influence.

<comment><USER9> > _Who is the person you 're trying to convince with this thought?_

Anyone who reads the article, and thinks "this is outrageous!" instead of treating it as a chess move by an opponent.

<comment><USER10> It's outrageous to the extent we're completely unprepared and maybe unwilling to deal with it.

Radio Free Europe was at least just using the airwaves. These efforts are using US corporations against the US.

<comment><USER11> European airwaves, on European radios.

<comment><USER10> So, is that confirmation on 'unwilling'?

</comment></comment></comment></comment></comment></comment><comment><USER12> China didn't ban Facebook b/c they don't think it should exist, they banned it b/c its a propaganda engine that they didn't want to compete with.

In the world of international jockeying and competition it would be naive to assume they would use a tool like facebook against their economic rivals.

<comment><USER13> > they banned it b/c its a propaganda engine that they didn't want to compete > with

That's a good way to put it. Since China is so in-love with AI, I wonder if we could use AI with the same purpose - spreading "freedom" information in China.

Something like this actually happened recently:

[http://www.independent.co.uk/news/world/asia/china- chatbots-...](http://www.independent.co.uk/news/world/asia/china-chatbots- communist-party-ruling-critics-peoples-tencent-babyq-little- bing-a7875601.html)

<comment><USER14> Or maybe it'll just end up spreading racism and denying the Holocaust.

Something like this actually happened recently:

[https://techcrunch.com/2016/03/24/microsoft-silences-its- new...](https://techcrunch.com/2016/03/24/microsoft-silences-its-new-a-i-bot- tay-after-twitter-users-teach-it-racism/)

</comment></comment></comment><comment><USER15> At what point is "propaganda" just annoying advertising?

I mean, I've seen these silly promotion ads where they interview random people who say China is great. They're annoying, but not unlike random mattress ads or ebook ads I see everywhere on the timeline. Most of the time i just ignore them always.

Besides, they're no where near the level of nastiness of local political ads that denounce other sides as rapists and child molester (at least in my area, where they basically show latinos as gang members)

</comment><comment><USER16> How is this news? Every global and regional power does this sort of thing to further their economic and political interests. They do it against both friend and foe. They've been doing it since newspapers first became popular.

And there's nothing wrong with it. Sure it leaves a bad taste in your mouth, but Facebook posts filled with misinformation, sponsored by foreign agents disguised as your countrymen are vastly preferable to the alternatives which have historically included sponsored coups, astroturfed bloody protests, economically crippling sanctions, and assassinations.

[1] China, Russia, the US, the UK, Israel, KSA, India, Germany, etc. Yes, even Germany which many people on HN seem to think miraculously stopped spying in 1989 and ascended to ethical heaven. I wouldn't be surprised if Canada used it during the Keystone pipeline debate.

<comment><USER17> It's news because China runs by far the largest overseas propaganda campaign, spending over $10bn per year on it. They spend 15x more than the US does on overseas propaganda. What's worrisome is not the fact that they do it, but the sheer scale of their program.

<comment><USER18> Where did you get that digit?

$1.6B figure was leaked on the state budget in 2009

<comment><USER17> $10bn estimate from 2015 [https://www.foreignaffairs.com/articles/china/2015-06-16/chi...](https://www.foreignaffairs.com/articles/china/2015-06-16/china- s-soft-power-push)

</comment></comment></comment></comment><comment><USER19> I mean, it bans Facebook exactly because it's easy to spread propaganda through it.

What the hell is up with this title? "Country attacks with weapons, the same thing it bans internally". Obviously?

<comment><USER20> No, not obviously. Why isn't it banned in every country that's not a US ally? Just China...

</comment></comment><comment><USER21> For all the talk about democracy, freedom, diversity and dissent the country seems extremely uncomfortable and insecure in the face of diversity and dissent it has no control over, unlike tightly controlled narratives in traditional media and is now seeing gremlins everywhere. This is paranoia.

This is a new form of McCarthyism and its a double edged sword. Other countries can use the exact same logic to ban hollywood and western media for propaganda and you can't then lecture them on free press or accuse them of authoritarianism.

It should be taken for granted there is going to be fair share of propaganda and rumour mongering everywhere and always has been in human society.

The only way to 'control' this is to establish a 'ministry of truth'. But given the level of double speak and paranoia I won't be surprised to hear folks advocating a tightly controlled 'ministry of truth' to 'protect' our fragile democracies.

<comment><USER22> Diversity of opinion is one thing. Dissent from official US positions, OK. But very effective manipulation from very skillful people is something a bit beyond "diversity of opinion". You can present a diverse opinion without trying to manipulate, and without being dishonest about who you are.

This is something above mere diversity of opinion, or even dissent. This is dishonest, covert propaganda.

<comment><USER23> >But very effective manipulation from very skillful people is something a bit beyond "diversity of opinion".

You've just described the whole advertising industry.

<comment><USER22> Yes, I noticed that. Honest advertising is also something beyond "diversity of opinion", but it's one that we accept. Dishonest advertising is another step beyond that, one that we don't accept, and that we try to keep out of society. This (China/Russia Facebook propaganda) is more like dishonest advertising; we should seek to eliminate it.

</comment></comment></comment></comment><comment><USER24> I was wondering when someone was finally going to get around to asking if countries other than Russia spread memes on Facebook. To the surprise of no one except apparently Democrats, they do. Now the million dollar question, will anyone investigate _China's_ involvement in influencing the election? Of course not, that would weaken the narrative that Russia acted alone in this endeavor (with Trump, of course).

<comment><USER25> > Now the million dollar question, will anyone investigate _China's_ > involvement in influencing the election?

If at all, China would have helped Clinton. An isolationist like Trump is not good for China which depends on exports to the US and other western countries to fuel its growth, which China needs in turn to prevent social unrest caused by poverty.

No one will waste resources in tracking down who helped the loser of the election.

</comment></comment><comment><USER26> Before someone points out that nations have always done this:

It has never been this effective, targeted or difficult to uncover.

<comment><USER24> > It has never been this effective

Debatable, unless you're taking the claims that $100K of meme-tier ads on Facebook changed the outcome of the election at face value.

> or difficult to uncover.

Is it really? Facebook certainly seemed very sure of every little ad that was purchased by a Russian party.

<comment><USER26> > Debatable, unless you're taking the claims that $100K of meme-tier ads on > Facebook changed the outcome of the election at face value.

It's not just Facebook, or Twitter, or 4chan, or Reddit, or HN, or IRC, or email, or TV ads, or superpac ads, etc....

Ads can be targeted moreso than ever, celebrities are more easily roped in willingly or as useful idiots. Content has wider possibilities ("verified" emails). People are much more primed for disinformation than they were in the past.

> Is it really? Facebook certainly seemed very sure of every little ad that > was purchased by a Russian party.

They paid in rubles. They didn't seem to even think it was terribly important to hide.

</comment></comment></comment><comment><USER27> "New York Times Spreads Propaganda to U.S. as it Realizes Their Monopoly on U.S. Propaganda is Coming to and End"

</comment><comment><USER28> Perhaps Zuckerberg can ask another member of the Chinese government to name his next child to smooth this over.

<comment><USER20> [https://twitter.com/MgtowExpat/status/927593680277348352](https://twitter.com/MgtowExpat/status/927593680277348352)

</comment></comment><comment><USER29> Honestly curious - does the US spread propaganda to the rest of the world? If I had to guess it would be that hollywood historically did most of the work - exporting "American" values and culture to other nations, but I don't know if that's still the case.

Anyways, it would be interesting to know if there is any kind of US presence on Chinese/Russian social networks (like, is there some bizarro mirror world /r/the_donald in Russian with posts pushing some kind of US agenda?)

<comment><USER30> House of Cards? Brazil faced a coup d'etat with the same script of the series, at the same time it was aired. I'm skeptical so I don't believe in coincidences.

<comment><USER31> There was no coup.

And really. People joked about it being equal to House of Cards. But the real facts were way too outrageous to get into a fictional script.

<comment><USER30> This is what we call Propaganda, my friend.

<comment><USER31> Do you honestly the Brazilian Justice and Congress joined forces to do a coup and used a US TV series to manipulate the people into supporting them?

Really?

<comment><USER30> Do you genuinely believe that what I said is about a plot between a TV Series and the corrupt congress and brazilian justice? Really?

I bet you are these kinds that think Brazil was been "left-wing" brainwashed and the federal universities are all bolivarian. But a TV Series is just a TV Series.

</comment></comment></comment></comment></comment></comment><comment><USER32> I don't understand why the media obsess that much about who spreads which propaganda where.

If people uncritically eat up whatever they are fed by their newsfeeds, then we are hopeless. If, however, people can consume news critically, then propaganda may merely be annoying, hardly harmful.

<comment> wu-ikkyu >I don't understand why the media obsess that much about who spreads which propaganda where.

Maybe because they're part of the problem?

</comment></comment><comment><USER33> At Quora there is a very strong presence of commentators who only focus on answering geopolitical questions about China. It is an area of inordinate size (quantity of questions + answers) compared to other topics of interest.

</comment><comment><USER34> Oh, and this is an ad that ran in the U.S. airwaves.

[https://youtu.be/OTSQozWP-rM](https://youtu.be/OTSQozWP-rM)

</comment><comment><USER35> I think they probably banned Facebook about 20 milliseconds after they realized it could be used as a weapon like this.

<comment><USER36> They banned Facebook in 2009 because the 2008 Olympics were over and they no longer had to pretend to be opening up to the rest of the world (they used Uighur separatism discussion as an excuse). They'll probably ban it before the 2022 Olympics to redo that show for face purposes only to reban it again afterwards.

</comment></comment><comment><USER37> CNN spreads propaganda to U.S. On Facebook, CNN earns awards as fairness and freedom fighters.

</comment><comment><USER38> lol this is like some propaganda inception</comment>


Windows 3.1 Flash Edition - <AUTHOR> https://sakamoto.pl/weblog/07-windows-3-flash-edition/ <comment><USER2> Even Visual BASIC 3.0 because it is useful to create 16-bit apps. Thank you for making this. Your comment editor has black on black text on your site.

</comment><comment><USER3> Screenshots appear to be windows 95 though. Recycle bin, start button, wallpapers ...

<comment><USER4> Yeah, he's using Calmira, a ProgMan.exe replacement that has a windows 95 look and feel</comment></comment>


What version controls don't solve: Semantic Conflict - <AUTHOR> http://martinfowler.com/bliki/SemanticConflict.html

<comment><USER2> There is actually a third way to spot conflicts like this. The author is even aware of it - his first example is captured by the type system. Is it possible that a stronger type system can solve more of these problems?

 _Let's imagine I'm looking at that calculateBill method and realize that as well as calculating the bill, it also sends off accounting entries to the accounting system. I don't like the side effect, so I pull it out into a separate notifyAccounting method._

Before:

        calculateBill: BillEntries -> Writer (AccountingSystem) Bill
        ...Jed's code:...
        produceBillEntries >>= calculateBill >>= billProcessor

After:

        calculateBill: BillEntries -> Bill
        notifyAccounting: BillEntries -> (Writer AccountingSystem) ()
        ...Jed's code:...
        produceBillEntries >>= calculateBill >>= billProcessor

This yields a type error. calculateBill needs to have type a -> M b, instead it has type a -> b. If your type system can express concepts like "removed side effects" then the compiler will catch mistakes like "forgot to put the side effect back".

</comment><comment><USER3> _Often people try to justify DVCSs based on how they make feature branching easy. But that misses the issues of semantic conflicts. [If your features are built quickly, within a couple of days, then you'll run into less semantic conflicts (and if less than a day, then it's in effect the same as CI). However we don't see such short feature branches very often.]_

Like anything else, feature branches can be abused, and require some discipline on the part of the developer to use correctly.

On git.git every feature is developed on its own branch. However, the maintainer integrates all the features into a single branch for testing and to prevent these sorts of semantic issues from creeping up. It's really only an issue where the feature branch is allowed to diverge too far from the main branch.

As a developer maintaining your own feature branch, there are generally two good practices to follow:

\- Regularly integrate the main branch with your feature branch, review what's coming in from main, and test. Having a feature branch does not relieve you of being aware of what else is going on in the code base. Note that you'll generally do this on a throw-away branch, as you don't want to clutter your feature branch with test merges.

\- Rebase your feature branch as needed if the changes in main invalidate any of your work so far. As part of rebasing, you'll have to account for those changes.

You can have a long-lived feature branch, the important thing is not to go into a cave with it. You need to keep it up-to-date wrt to mainline development.

For a shared feature branch, rebasing is a bit harder (but not impossible), so instead you will typically merge the main branch into your feature branch as needed. It is important to do these merges only when necessary if you wish to maintain a relatively clean development history, something that is important if you later need to track down the source of a bug or backout the feature.

</comment><comment><USER4> Does "semantic conflict" really have anything to do with VCS? It seems obvious to me that the same situation can happen during normal development on the head branch. A bit less obvious but still clear in my mind is that semantic conflicts can occur anytime a developer's assumptions about the code base are wrong.

Yes, static typing fixes problems with changing function signatures but that's only the first step of the way. The next step is to start thinking about and defining modules (files, classes, whatever) in semantic terms.

I haven't heard it discussed in a few years but the discussions around the Liskov substitution principle and Liskov/Wing semantic typing highlights the difficulties of adding semantics to everyday software development practices.

<comment><USER5> > Does "semantic conflict" really have anything to do with VCS?

I think his point is precisely that VCSs cannot automatically avoid semantic conflict. He's not blaming the VCS, but rather saying that Feature Branch should be avoided even if you're using a modern VCS.

<comment><USER6> The reason he is not fond of feature branches seems to be that they allow a single developer to work on a feature for too long before integrating with the master branch.

This only becomes apparent in the footnotes where he states that if your feature branches live for only a day, then your basically doing what he calls continuous integration.

This feels more like something that can be remedied through having good practices. Just like you shouldn't go without committing for days when working with Subversion, it is probably a good idea in a DVCS to:

\- integrate the main branch with your feature branch often (at least daily)

\- don't do semantic changes in a feature branch. Instead, switch to the main branch (or a very shortlived new branch) and do the semantic change.

This should avoid 2 feature branches doing semantic changes and then having a merging hell after a week or so of development.

</comment></comment></comment><comment><USER7> Title's a bit off. cvs/svn/bzr and any version control system doesn't solve it either, and they weren't meant to. Fowler's article doesn't have that in the title, so I don't think he was even trying to imply that people think version control solves this issue in the first place.

<comment><USER8> Agreed. Title is pure link-bait.

The only VCS I see mentioned is git in one of the footnotes regarding rerere.

<comment><USER9> Just to note: it isn't the author's title, it's a Hacker News users' title.</comment></comment></comment>


How we built a GDPR-compliant website analytics platform without using cookies - <AUTHOR> https://usefathom.com/anonymization/index.html <comment><AUTHOR> We are incredibly open to any ideas, comments or concerns on how we're doing this. This is a big step up from what we had previously, but there's always room for improvement. Happy to hear thoughts in the comments.

<comment><USER2> Hi Paul, thanks for being open about this. I have a big, important question.

ICO, the agency in charge of enforcing GDPR and related legislation in England, released guidance earlier this month on the topics of cookies. One of the most notable parts of this guidance is that "device fingerprinting" is treated the same as a cookie[1]. And also that website analytics requires consent to use cookies or similar technologies[2] ("similar technologies" including device fingerprinting).

Now, the above guidance is related to PECR rather than GDPR, which is what your post is about. But, given the above, do you think that your software is compliant/exempt from PECR or do you think that organizations will still have to take extra steps to be compliant with privacy legislation?

[1] [https://ico.org.uk/for-organisations/guide-to- pecr/guidance-...](https://ico.org.uk/for-organisations/guide-to- pecr/guidance-on-the-use-of-cookies-and-similar-technologies/what-are-cookies- and-similar-technologies/#cookies5)

[2] [https://ico.org.uk/for-organisations/guide-to- pecr/guidance-...](https://ico.org.uk/for-organisations/guide-to- pecr/guidance-on-the-use-of-cookies-and-similar-technologies/how-do-we-comply- with-the-cookie-rules/#comply15)

<comment><USER3> I think that's a fair question.

AFAICT, v1 of PECR awkwardly applies whenever the cookie is not functionally directly necessary for the service that the user is using. PECR applies even if, like here, the cookie is just for counting unique numbers of visitors, and is not used for fingerprinting individuals.

The draft v2 of PECR contains an exemption for first party analytics. I think this maybe strikes a nice balance: explicit consent would still be required for the more-harmful third party analytics.

Not sure when v2 of PECR will happen. It is years overdue. Perhaps it is a priority for the newly elected European Parliament and the new Commission?

<comment><USER4> What makes analytics first-party? When the first party serves them, or only if the data never leaves machines under the direct and exclusive control of the first party?

<comment><USER3> I don't know the answer to that.

FWIW, my guess would be that the definition is fairly strict.

Now, I don't think that would prevent the first party using data processors, but I suspect the first party would have to exercise a lot of control over the processor. This would be in contrast to a service like Google Analytics, where the company's control and choice is simply limited to take it or leave it.

<comment><USER4> A data processor agreement is not usually negotiated all that hard and this is indeed not really possible with truly large companies (and let's face it, that's where most companies get their enterprise software). Therefore I feel that expecting a lot of control being exercised is a bit of a pipe dream.

</comment></comment></comment></comment></comment></comment><comment><USER5> Why all the trouble with hashes - can't you just do it on the client and not having to store any data at all?

"For tracking unique page views"

      if(!sessionStorage[location.href]) {
        sessionStorage[location.href]=1;
        navigator.sendBeacon("/unique-pagehit?" + encodeURIComponent(location.href));
      }

"For tracking unique site views"

      if(!sessionStorage["Hi!"]) {
        sessionStorage["Hi!"]=1;
        navigator.sendBeacon("/unique-sitehit");
      }

"For tracking previous requests"

I'm not sure I fully understand what is being measured (is it session-only?). For the duration someone watched a page, you can use sendBeacon in onBeforeUnload. To detect a bounce, set a Math.random() in a session variable, send it at the start of the page, and have every page load send the previously stored random variable. Then count the unique random keys you received on the server - those are the bounces.

I know, in practice you'll need to trim sessionStorage, sanitize URLs, use something less-colliding than Math.random, dealing with new tabs, some polyfills and other robustness, etc... but I don't yet see why the tracking mentioned needs any user ids or hashing at all.

<comment><USER6> Great idea but we can't use sessionStorage under PECR, which is why we made this move. Plus we got rid of anything being stored on the user's machine.

</comment></comment><comment><USER7> As a user of the open source version of Fathom, I'm a little concerned by the lag in publishing this update to the community edition. I assumed development work was happening in the open on Github, but I guess that's not the case?

<comment><USER6> Whole new language & codebase. Old developer left, we don't write Go.

<comment><AUTHOR> So it's not as easy as just pushing the update to the repo. We are still committed to open-source, but we also have a business to run and need to make a living here (we're two dudes who care about privacy, not a huge company with deep pockets) :)

The community version is getting a full update soon. We just have to focus on profit a bit (this keeps us in business and able to update the repo).

<comment><USER8> Speaking as someone who built an analytics business while still maintaining an open source version of the code, I can confirm that it's hard. Sometimes it starts off as one small patch that only makes sense for the hosted service, but you can't push to the OS branch until the code is refactored to protect secrets.

I cannot give you any tips on managing your time between the two, but you may want to consider raising your prices. Back in 2011, we ran our basic account at $50/mo, Business at $150/mo and Enterprise at $CallMe/mo. We could probably have upped that after a year with all the new features we'd put in, but we were acquired before the 2y mark and dropped the first 2 plans.

We still maintain the open source version of the code, but neither of the original founders work on it (for me it's gone back to being a hobby because there are now people paid full time to work on it). We still get questions about the lag in updates. We typically do quarterly bulk pushes to the open source version now.

<comment><USER6> Thanks for sharing your story. It is hard and we have to fight to not resent OSS because of 0.001% of the OS community. Open-source software has contributed significantly to our lives, and we love it, so we're going to be pursuing OS Version 2 regardless of a few angry people.

<comment><USER9> Armchair CEO here but charge a fair rate and if that means double do it. People will respect it and pay it if the product or service is helping them make money and I'd rather pay a higher amount and know I'm helping fund a tool that I use to help me be productive than get something cheap.

</comment></comment></comment></comment></comment></comment><comment><USER10> This is a common solution to the problem of PII, but without any information on returning users I would argue that it's value as an analytics platform is limited. Few are the tools where you can grow the business without knowing the difference between a first-time and return user which is the reason cookies were invented in the first place.

However, since such businesses already need to collect personal info as part of your account creation it shouldn't be hard to build analytics on top of that existing PII. If they are already collecting PII it doesn't seem to save much to have their analytics tool avoid it?

<comment><USER6> Fathom Analytics is intentionally limited, and the limitations you point out are 100% intentional. There are many businesses who can't use our product, but millions that can :)

</comment></comment><comment><USER11> Most schemes of this kind are just more complicated cookies that people hope will avoid the GDPR provisions by dint of being obfuscated.

What the article is discussing looks (at first brush) to be a sensible way of aggregating users up-front before it hits the database, rather than later. So no personal data is stored.

Does this meet the requirements for a site to avoid notifying users under the GDPR? I have no idea.

Even with the best of intentions, if you use a service like this then you are relying on them a) doing what they claim, and b) not screwing up (by leaving logs around, etc).

If I use this service and data from my users gets leaked by Fathom, who gets blamed? The users were on my site, so I guess it is I that gets fined. Maybe the risk is worth it, maybe it isn't.

<comment><USER12> In response to your final question, this[1] document from the UK's ICO has some interesting info. Essentially you're either a Data Controller (that would be your site in this example) or a Data Processor (Fathom, in this case -- probably?!).

"64\. The ICO cannot even take action directly against a processor who is entirely responsible for a data breach, for example by failing to deliver the security standards the controller has required it to put into place. However, in these cases the ICO may decide not to take any enforcement action against the controller if it believes it has done all it can to protect the personal data it is responsible for and to ensure the reliability of its processor, for example through a written contract. However, whilst the ICO cannot take action against the processor, the data controller could take its own civil action against its data processor, for example for breach of contract."

Though it goes on to say that in some circumstances, the processor can _become_ a controller, in which case the ICO can go after it.

[1]: [https://ico.org.uk/media/for- organisations/documents/1546/da...](https://ico.org.uk/media/for- organisations/documents/1546/data-controllers-and-data-processors-dp- guidance.pdf)

<comment><USER6> And even if there was a data breach where myself & Paul were held at gunpoint and told export the database, there's no personal data to do anything with. Not even in our Redis queue! Our database is very boring, anonymous and simple, and we like it that way.

</comment></comment></comment><comment><USER13> If visits expire after 30 min, why not rotate the salt every 30 min? Keep current and previous salt, update as needed.

I would have more faith in privacy, if you didn't store the salt in the DB or permanent storage. If you manage to statically load-balance the users (e.g. hash site, ip, user-agent, don't forget site), the hash could be in-memory only. Sessions would break on server restart, but that's more of a feature.

To move thing further, you might not even need to store the hashes in the DB. Keep them in server memory only and (real-time) update aggregate data in DB.

<comment><USER6> The visit expires 30 minutes after the visitor lands. The expiration isn't generic.

It's an interesting idea. We have multiple servers under the load balancers, so we'd be able to store them in Redis, but that is no better than permanent storage, since Redis could still be breached and you'd see it with ease.

</comment></comment><comment><USER14> Hi Jack and Paul - love what you're doing! This solution is so needed.

I wondered whether you could explain what makes your hashing different from the hashing used by Facebook for their custom audiences tool which was deemed unsuitable for anonymisation as per [https://www.spiritlegal.com/en/news/details/e-commerce- retai...](https://www.spiritlegal.com/en/news/details/e-commerce-retail- facebook-custom-audience-not-allowed-without-consent.html)

</comment><comment><USER15> Couldn't people just parse the log files from their webservers?

<comment><USER16> Definitely. Helps with people who disable JS as well. I've used this before [https://goaccess.io](https://goaccess.io)

<comment><USER6> bashy from laracasts? Hey there! And I've heard the disabled JS piece before. How many people are disabling javascript?

<comment><USER16> Yes that's me. Hi there.

I use fathom on one of my sites and I like it. I was just replying to someone who said people could just use access logs instead. Some don't like adding JS to their site I guess.

Ad blockers will also block requests to the tracker.js if you use analytics.domain for example.

</comment></comment></comment></comment><comment><USER17> Looking at their live demo ([https://stats.usefathom.com/#!p=1w&g=hour](https://stats.usefathom.com/#!p=1w&g=hour)), I can see a lot of traffic is coming from ycombinator. So...

(I mean, I don't have a point here but I find it pretty interesting. xD)

<comment><USER18> Duck Duck Go traffic is 10% of the Google traffic.

Is that typical for privacy-focused tech sites? I would have expected a lower percentage.

(DDG fan here, but people look at me funny when they see me using it...)

</comment></comment><comment><USER19> sorry but, isn't that a (unnecessarily complex) cookie?

<comment><USER20> Look, entire industries exist for being complaint with the letter but not the spirit of the law so I'm sure that this in no way meets the definition of a cookie as far as the GPDR is concerned.

 _However_ , this is absolutely a cookie. Scraping just enough information from the browser to create a unique but stable hash and then having the browser compute it every request isn't at all different from that browser information acting as the cookie.

</comment></comment><comment><USER21> The data is not anonymous. Anonymity is actually very hard to claim (read k-anonymity, differential privacy, etc).

Nevertheless, the chances of identifying someone are probably pretty low, and it´s a good effort to make analytics more privacy friendly.

<comment><USER6> So it's practically anonymous. Nobody has enough computing power to brute it & the data is deleted, typically, in 30 minutes.

<comment><USER22> To de-anonymize the data you don't actually need to brute force the 256-bit hash. If the other pieces of data are known (salt, site, page, day of year) and you can make a shrewd guess at the user agent then you'd only need to brute force the 32-bit IP address( _).

(_) Assuming IPv4. Obviously IPv6 addresses would be much harder to brute force, but still easier than a 256-bit hash.

<comment><USER6> I do take your point but when we get into this area, it becomes a big question of trust. Because if I gave you a hash, you would need to guess the 256 bit salt in addition to all the other possibilities.

I mean, hey, here's a hash: 26246226167b9f190d3a1ce726efe07ae18bbf0480a78d19390b9aaf13f25cb0

Imagine you just got hold of it through a data breach. I'll give you $1,000 if you can get it de-hashed before midnight Chicago time ;)

<comment><USER22> It's true that you'd need to know the salt first, but the server _does_ know the salt, and if you obtained the hash through a data breach then you probably obtained the current day's salt as well.

<comment><USER6> So if someone had unlimited access to our servers, that would be a problem. One piece to note is that page views get deleted after around 30 minutes.

So one of the reasons we posted to HN was for conversations like this. Reading what you put makes me think we need to do more when creating the PageRequestSignature and the SiteRequestSignature, because if someone had access to our server, got the hash, stopped all cron jobs from processing data, then they could work it out after a huge amount of time / computing power. But to be honest, at this point, they could also add log($_SERVER) and get the entire request body of a user, so we would have much bigger problems in that scenario.

Anyway, you've given me a few new thoughts on hardening from data breaches. Because, yes, if they know the hash & have full access to our server then it becomes easier. So we almost need to move it to the point where they won't know the salt that was used for a particular user. So we'd not recycle a single salt, we'd recycle multiple salts that are based on, perhaps, the first 2-3 digits of a users IP address combined with the last 2-3 digits.... Then it would be much harder to break without first knowing the $_SERVER dump (the user's IP etc.). Obviously this wouldn't stop a "complete control of server" attack where they could just start logging everything but it would really ruin a brute forcers day because they wouldn't know what salt to start with.

What do you think of that idea? I'm running on little sleep so be nice ;) Also thanks so much for all your feedback so far, it's so appreciated!

<comment><USER22> The case I would be (slightly) concerned about would be where an attacker has obtained limited, read-only access to your database, so they have the current salt and the hashes related to the last 30 minutes of activity but not the ability to simply log whatever data they want. At that point brute-forcing the 32-bit address space and a small set of common user agent strings would allow them to determine the IP addresses for specific page views.

Of course, if an attacker has access to the salt and is interested in the pages viewed by a specific _known_ IP address then this all becomes much simpler. Then the only unknown is the user agent, which is relatively low- entropy.

> So we'd not recycle a single salt, we'd recycle multiple salts that are > based on, perhaps, the first 2-3 digits of a users IP address combined with > the last 2-3 digits....

If an attacker is already brute forcing the IP address and has the ability to obtain the salt then they would just use the correct salt for each IP address. Making the salt depend on other data already included in the hash doesn't change the size of the search space.

<comment><USER6> I think what's been helpful for us with posting this here is to hear of different ideas for how someone might hack Fathom. When we came up with it, our starting point wasn't "you have the salt and IP address", go break a hash. It was on the assumption that you don't have the salt or IP. I think we can improve what we've built. 720 salts improves resilience in a few areas but not in the scenario you are painting here. The scenario you're painting here has made me think of additional ideas though.

If they had the salt, IP and user agent, they'd have to also brute force every possible hostname and pathname, which would be insane. But I suppose they'd only have to do a few million based on the data we have on hostnames / pathnames...

Lots of ideas for improvement are popping into my head and I love how this community keeps challenging you to improve things. We had feedback on Reddit but it was much angrier!

The next step is to take your feedback and look into how we would defend against the scenario you've provided. Thank you!

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER23> Maybe I'm missing something but (1) I don't think this is GDPR compliant, and (2) why so complicated?

Regarding (1),

 _> Brute forcing a 256 bit hash would cost 10^44 times the Gross World Product (GWP). [...]

> We have rendered the data anonymous to the point where we could not identify > a natural person from the hash.

> It's possible that GDPR does not apply to Fathom since data is made > completely anonymous. Even if GDPR did still apply, we reiterate the stance > that there is legitimate business interest to understand how your website is > performing._

This seems to imply a profound confusion between the difference of hashing vs. anonymity. Just because it's hashed doesn't mean it's anonymous! You don't need to "brute-force" the hash, you just need to find a user that matches your hash... which is 1 in 7 billion (or so), much more tractable. This is also the principle e.g. MD5 rainbow tables are based on...

They claim to change the hash every 24 hours, so it's equivalent to having a session cookie with 24-hour expiration (session cookies are "anonymous" by their definition, they don't have any user information and they're impossible to "brute force", they "just" _enable tracking_ ). I've no idea if 24-hour session cookies are GDPR-compliant...

Regarding (2), given that this seems (again, I might be misunderstanding) equivalent to a 24-hour session cookie, why not just do that? However, then you're ... drumroll ... giving control to the user. Why not just _give control to the user, period?!_ For example, by storing the list of pages visited in Local Storage, and only pinging the server once for each page(view) every 24 hours?

<comment><USER6> For GDPR compliance notes please see: [https://usefathom.com/data/](https://usefathom.com/data/)

> You don't need to "brute-force" the hash, you just need to find a user that > matches your hash... which is 1 in 7 billion (or so), much more tractable. > This is also the principle e.g. MD5 rainbow tables are based on...

Not quite. We use a SHA256 hash as our salt, and that changes each day, so you'd need to brute force that.

In terms of how many possible combinations there are for this salt, please see: [https://stackoverflow.com/a/49520766](https://stackoverflow.com/a/49520766) \- you would need to brute force it and try each possible combination with every single possible IP / User Agent / Site combination to break a hash. This is why it's not theoretically impossible but it's practically impossible.

We would love to approach things in an easier way but PECR doesn't want cookies, even anonymous ones.

Now, one thing that we have uncovered thanks to someone on here is that we need to increase our resistance to data breaches. If someone had complete, unlimited access to all our data / servers, including the daily salt, then they could de-hash page views from the last 30 minutes. I have no idea how long that would take. There are 4,294,967,296 (?) possible IP addresss, and then over 3M (?) user agents, so it'd be an absurd, pointless exercise.... Anyway, we're going to be bringing in multiple salts that depend on the user IP address, meaning that, in the event of a data breach, a hacker won't know which salt has been used for a hash :) Perhaps we base the salts on the first 3 digits of an IP address? That would mean we'd have 720 possible SHA256 salts!

<comment><USER18> You can get an order of magnitude on hash collision resistance by rolling every two hours. Maintain "two" backend databases to gracefully track sessions between roll overs.

Also, for non GPDR IP blocks, maybe just store a per client salt in a cookie(!) and then xor it with the rotating server salt.

<comment><USER6> I'm not worried about hash collision with sha256. Any reason why I should be? :)

And we can't use cookies because of PECR!

</comment></comment></comment></comment><comment><USER24> > Tracking page views alone, without visits, is completely useless and means > that you won't have insight into how many people visit your site / pages > each day.

What's the difference between a page view and a visit?

<comment><USER6> If I came onto your website and refreshed one of your pages 500 times, that would count as 500 page views but only 1 visit.

</comment></comment><comment><USER25> At first glance this appears to be a well thought out solution. It will never be able to give you some of the stuff that GA can give you but that is by design. The problem that I see is that as long as GA is able to claim they are GDPR compliant there will be very few websites that will see this as a necessity and so adoption will be relatively low. But, and this is just an idea, one of the things that company could do is to proudly present a 'zero retention' button or logo assuming they do not have other trackers on their pages. That way it might become a distinguishing factor for the adopters and that might drive further adoption.

Thanks for building this, I will promote it.

<comment><USER6> A fantastic idea. We've recently designed a button for websites to show users that they care but I really love this idea of a 'zero retention' button :)

</comment></comment><comment><USER26> Looks decent, but pricing is insanely high for the extremely limited set of stats.

<comment><AUTHOR> Totally fair, as that's an opinion :) Luckily our customers are happy with the price, and other folks use the open-source (100% free) version. Cheers!

<comment><USER27> I actually think your prices are pretty reasonable for business use, but I wonder if you've considered a "personal" plan of, say, $6/month for up to 10k hits, non-commercial use only.

I've been looking for a privacy-first tracker for some personal sites for a while, but nobody is offering pricing that makes sense at the lower end.

</comment></comment></comment><comment><USER28> I think the GDPR was enacted into law not to prevent cookies, but to prevent collecting data on regular people. This seems to circumvent the technicalities of the law but not the spirit. The risk is that they enact a new law that puts even further restrictions on website operators.

I'm not sure this is a good idea.

<comment><USER6> Thanks for the concern here. We are GDPR compliant (and may be exempt from it). See here: [https://usefathom.com/data/](https://usefathom.com/data/)

<comment><USER29> You might like to edit the line on that policy page that refers to "the most privacy-focused manor"... while a privacy-focused manor is an interesting idea, I suspect you meant "manner". :)

<comment><USER14> Equally, I'm not sure what it means to be GDPR "complaint" but I'm thinking it's probably supposed to be "compliant" ;)

</comment></comment></comment></comment><comment><USER30> It's so exciting that thanks to GDPR we are now seeing innovate analytics solutions that respect privacy.

<comment><AUTHOR> Agreed!

</comment></comment><comment><USER31> I am not sure what exactly they did here. How do they persist the hash between requests?

My guess is they use localStorage and sending the hash to their servers with each request.

So we are talking about a mechanism that's just like a cookie.

As long as they don't have any PII and can't figure out who the user was, then I think the GDPR gives them an exception.

But "without cookies" claim is dubious!

<comment><USER6> We have been GDPR compliant for many months but our aim here was to meet E-Privacy demands.

We don't use localStorage... Read the blog post, we don't use cookies.

<comment><USER31> Oh, apologies, you are right! You don't use any kind of cookie mechanism. I just read it again:

      Random SHA256 String (daily regenerated)
      IP Address
      User Agent
      Site ID
      Day of the year

My only question is about this "Random SHA256 String", where is it stored between requests?

<comment><USER6> Redis Cache. It's a Fathom-wide random string that is used to prevent rainbow table attacks. The salt is refreshed at midnight every day.

<comment><USER31> Thanks, makes total sense.

So basically the only drawback I see is that all employees from behind one corporate NAT using same browser will count as one user.

But don't see a way around that if you can only use IP and User Agent strings.

</comment></comment></comment></comment></comment><comment><USER32> why are you calling it an analytics platform when it isn't one?

</comment><comment><USER33> This is still logging everything the GDPR says you can't without asking for consent, but you made your search convoluted (but not less efficient if you have all the pieces) to (suggest|lie?) that you need to break the hash and that's why you don't need consent.

None of the information you are using on the hash wouldn't be in the search query itself! ip, user agent, path, date, etc. So there is no way to reverse the hash. You just hash your search query and compare in O(1) time.

The _only_ piece of information that realistically makes the hash slightly difficult to get is the random number refreshed every day. But either you store it (and i have no reason to believe you do not) or it make the brute force effort trivial as I only need to generate the hash with that variable now.

<comment><USER6> You're focused mostly on Recital 26, which was only a theory of mine, outside of that we are GDPR compliant anyway. I likely shouldn't have included it since that isn't our primary ground for processing. Please see: [https://usefathom.com/data/](https://usefathom.com/data/)

And yes the daily hash gets stored until midnight. But what are you talking about with 'search query' containing IP, user agent etc.?

<comment><USER33> If a search query on your data would contain all the components of the original hash, i don't have to walk backwards and break the hash. i just have to hash my query terms in the same way.

Also I suggested you store the daily hash forever. But even if you really erase it every day, as you say, If you or an attacker makes the same request every day at a predetermined time, when you/they get your logs, you/they can use that predictable request to get the daily secret too.

I consider the information to be stored in plain text, and that you would have to have requested permission just the same. You pretty much have an identifiable user (via IP/UA/access time) stored in your logs.

Anonymization is removal of information, not encoding it in a convoluted hash.

<comment><USER6> So that needs to be our next target point (access logs). We want to move to a position to keep no access lgos.

And a hacker could indeed "win" if they broke into our system, got the salt and exported the DB. We didn't focus on this in our article, as it's unbelievably unrealistic, but it's still possible. Our next step is to address that.

Without the hash, it's practically impossible to brute force.

<comment><USER33> Not talking about a hacker. I am stating that the described hash dance offers no exclusion from GDPR as saying "we promise we won't look" would do.

My point about brute forcing being useless, is that you hold all the information needed to re-create the hash. All but one tiny piece that is the random number. so brute force is a very effective O(<tiny piece size>). And since it is stored in your locally available data, there is no rate constraints.

<comment><USER6> > I am stating that the described hash dance offers no exclusion from GDPR as > saying "we promise we won't look" would do.

Under your logic, you would never trust us because we could just add $log->write(UserIp, UserAgent, Hostname, Path) in plain text. Trust is very important and what you do with the data is important under GDPR.

And we don't hold all the information to re-create the hash, that's the thing.

I thought a lot about "Oh but you could just do this, this and this" but, no, that argument doesn't hold. Our obligation under GDPR is what we actually do with data.

</comment></comment></comment></comment></comment></comment><comment><USER34> This is very weak reasoning, because you cannot identify an individual by IP either. This project looks like trying to exploit loopholes. The idea behind GDPR is to make sure companies log only data they need. This project looks into logging the data but without expressing why this is even necessary. Therefore I don't think this is compliant with GDPR.

<comment><USER35> > because you cannot identify an individual by IP either

Yes you can, particularly if you correlate across different websites.

<comment><USER34> You are conflating identification of a person by behaviour analysis with matching an ID. What is the ID is irrelevant here - may as well be a hash. That just proves my point that this project is not compliant.</comment></comment></comment>