A Programming Idiom You've Never Heard Of  - <AUTHOR> http://prog21.dadgum.com/121.html <comment><USER3> This idiom is even more useful in functional programming where the last step is an inverse of the first step in the applicative sense rather than in the side-effects sense.

Mathematicians call it conjugation, and it can be usefully thought of as a temporary change of coordinates or change in vantage point. Problems often become simpler, even trivial, when working in the right coordinates.

In practical programming, it is commonplace that the "inverse" that occurs in conjugation idioms is a genuine right-inverse but only partially a left- inverse. This is true for his example of squares and square roots. For another instance, consider

        reverseWords = unwords . reverse . words

The composition words . unwords :: [String] -> [String] is the identity, but unwords . words :: String -> String is the identity only for strings whose words are separated by a single space rather than multiple spaces, tabs, etc. This property can actually be put to good use:

        normalizeWhitespace = unwords . words

If you like, you can think of unwords . words as an orthogonal projection onto the subspace of strings on which unwords is a genuine left-inverse of words (cf. Moore-Penrose pseudoinverses).

That's cool and all, but the effect on reverseWords is that it normalizes interword whitespace in the process of reversing words, whether you like it or not.

Incidentally, it's not too hard to abstract out this pattern with invertible functions:

        data Invertible a b = Invertible (a -> b) (b -> a)

        invertible = Invertible
        apply (Invertible f f') = f
        unapply (Invertible f f') = f'
        inverse (Invertible f f') = Invertible f' f
        compose (Invertible f f') (Invertible g g') = Invertible (f . g) (g' . f')
        conjugate i f = unapply i . f . apply i
        on = flip conjugate

Now you'd have to set up invertibles for all the usual foo/unfoo pairs of functions. Assume that has been done. Our example only needs words/unwords:

        import Data.String as S
        words = invertible S.words S.unwords

With those preliminaries out of the way, you can then reimplement reverseWords very neatly:

        reverseWords = reverse `on` words

In fact, it's now so concise and self-explanatory that you hardly need the named definition anymore and can just refer to reverse `on` words directly.

<comment><USER4> One of the interesting properties of the Invertible datatype is that it can be made an instance of the Haskell Category type class, which makes invertible functions fully composable using the (.) operator.

<comment><USER5> Yes, composable data structure editors (lenses) with an overloaded (.) make my day to day work much more pleasant.

</comment></comment></comment><comment><USER6> A similar pattern that blew my mind when I first saw it was ruby's scoped open().

instead of

      f = open "test.txt"
      print f.read
      f.close

You can write

      open "text.txt" do |f|
        print f.read
      end

The file is automatically closed at the end of the block, and it also handles exceptions and so on. I find it also lines up better semantically with how I think about opening handles to things.

I believe Python also added something similar via the 'with' syntax. Not sure about other languages.

<comment><USER7> Yeah, Python has "context managers" that let you do the same thing. Here's the Python equivalent:

        with open("text.txt") as inf:
            print(inf.read())

User defined classes that implement special methods __enter__() and __exit__() can be used as context managers. More info: <http://www.python.org/dev/peps/pep-0343/>

For just file output/input Scheme has the "with-output-to-file" and "with- input-from-file" macros which redirect output and input to a given file. I vaguely remember Common Lisp having a similar thing, but I can't remember the name off hand.

<comment><USER8> I often use:

      for line in open("name.txt"):
          print line # Or do something else with it.

<comment><USER9> Me too when I'm lazy but we shouldn't; it happens to work fine on CPython but on implementations use garbage collection instead of reference counting (e.g. Jython) it leaks file handlers. The "with" statement is the right cross- platform way to handle resources.

<comment><USER10> I'd imagine it's an implementation bug. Once the iterator finishes, the file should be closed and the file handler freed.

<comment><USER11> Perhaps in this case - but how about a case where you partially iterate over the file, but then the file object goes out of scope? CPython would close (and free) that immediately, but other implementations probably wouldn't.

<comment><USER10> While editing the answer to masklinn comment, this case came to me. You both are, of course, right. There is no reliable way to close the file from the for iterator.

:-/

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> Interestingly, Python has the "with" statement for this basic genre of operation (executing a block of code while some particular resource is held), although it doesn't require that the third, "undo" step be automatically derived from the first step - it's just a protocol where a 'start' and 'end' method are called on the object representing the resource.

It sure makes it easy to be sure you're closing your file handles/database connections/mutexes, though.

<comment><USER13> I find it very handy with GUI programming under python too.

with wait_cursor(wxwindow):

       ... change cursor to hourglass whilst process runs...

I have similar context processors for locking controls or other temporary GUI changes

</comment></comment><comment><USER14> This idiom is common in mathematics, where it's called a conjugate.

Any time you have an element, y, of a group, the action of pre-composing with any other element, x, and post-composing with x's inverse to get xyx^{-1} is called conjugating, and the result is a called a conjugate of y.

<http://en.wikipedia.org/wiki/Conjugation_(group_theory)>

</comment><comment><USER15> Constructor / Destructor pairs are how you do this in C++. This is so fundamental to the object system that it's a bit steep to claim that "you've never heard of it". You have. The author's example of "get the rake - use rake - put away rake" maps exactly to [construct - call methods - destruct].

Given the jmp-iness of exception handling in C++, this is the recommended (i.e. the only safe) way to clean up after yourself.

<comment><USER16> This is just for resource allocation.

The pattern the article talks about is FAR more general -- it's about all reversible processes, even things that have nothing to do with allocation.

<comment><USER17> Sentry classes, which are a generalization of RAII, are used idiomatically in C++ for non-allocation tasks, such as whitespace skipping, logging, and profiling.

<comment><USER18> Also mapping state to scope e.g. mutex objects that set/clear on ctor/dtor

</comment></comment></comment></comment><comment><USER19> _Is there a programming languages where functions have inverses?_

How about a computer architecture where every instruction has an inverse, and therefore all functions and even programs are invertible? (Quantum computing, for example.)

<http://en.wikipedia.org/wiki/Reversible_computing>

<comment><USER20> Interesting. I'm sure you could do a VM where all the opcodes are reversible. Definining inverses of functions as inverses of opcodes and other functions would then be feasible. Might make a cool project.

</comment></comment><comment><USER21> Go handles this type of situation nicely using deferred function calls:

"Go's defer statement schedules a function call (the deferred function) to be run immediately before the function executing the defer returns. It's an unusual but effective way to deal with situations such as resources that must be released regardless of which path a function takes to return. The canonical examples are unlocking a mutex or closing a file." <http://golang.org/doc/effective_go.html#defer>

</comment><comment> v-yadli Seems that everybody is talking about files. :-) Open, operate, close. I think the author tries to bring us more, mathematically. Like a matrix operation: Move to origin, rotate, move back. But this is a lot harder than just "resource guard", imo. Any ideas?

<comment><USER22> Exactly. So far I'm kind of disappointed by the HN debate, it seems like many people only read half the article and are then quick to point out that their own favorite language has a way to release resources when done.

I liked the _sum under square_ better than the _read under open_ example (although the latter may be more relatable). _Sum under square_ squares items, sums them up, and then performs the "inverse-square" operation on the sum. I don't think this is similar to _using X_ or RAII in C++.

</comment></comment><comment><USER23> Bidirectional programming is somewhat related.

[http://www.cis.upenn.edu/~bcpierce/papers/lenses- etapsslides...](http://www.cis.upenn.edu/~bcpierce/papers/lenses- etapsslides.pdf)

e.g. read a file, parse it into an AST, modify AST (i.e. rename a variable), unparse the new AST, close the file.

<comment><USER20> Awesome slides. Would you happen to have a transcript for it as well. The slides themselves are pretty clear, but there's probably stuff missing.

</comment></comment><comment><USER24> That's in many cases what a LET with a dynamic binding does in Lisp.

Set origin draw Reset Origin

is

        (let ((origin (move 50 50)))
          (draw something))

Similar in the Common Lisp object system we use :before, :after and :around methods.

        (defmethod draw :before () (move 50 50))

        (defmethod draw () (draw-something))

        (defmethod draw :after () (move -50 -50))

<comment><USER25> > _Similar in the Common Lisp object system we use :before, :after and :around > methods_

Ditto for Moose (in Perl)...

      before draw => sub { shift->move( 50, 50 )};

      sub draw { shift->draw_something }

      after draw => sub { shift->move( -50, -50 )};

Also you can package up these method modifiers into a role which not only can be composed into a class but can also be applied just to an object...

      {
        package DrawRole;
        use Moose::Role;

        before draw => sub { shift->move( 50, 50   )};
        after  draw => sub { shift->move( -50, -50 )};
      }

      my $d = Draw->new;
      DrawRole->meta->apply( $d );

</comment></comment><comment><USER26> People have posted a lot of examples in a lot of different programming languages. What they show is that all you really need to build this idiom is first class functions (or some approximation thereof). (It also shows that the title is not correct; many, many people are familiar with this idiom.)

When you need to wrap up a common block of code, you use a function. This lets you call the same block in a number of different contexts. When you need to fix the context in place and let the block in the middle vary, you write a function that takes another function as a parameter. An example in JavaScript (might be a little off):

        var withFile = function (path, action) {
            var f;
            try {
                f = File.open(path);
                action(f);
            } finally {
                File.close(f);
            }
        };

        withFile('foo.txt', function (f) { ... });

This can be generalized (to some extent and depending on language) so you don't need to write the same structures every time:

        var context = function (prolog, epilog) {
            return function (action, ...args) {
                var o;
                try {
                    o = prolog.apply(args);
                    action(o);
                } finally {
                    epilog(o);
                }
            };
        };

        var withFile = context(File.open, File.close);

This example uses JavaScript' hypothetical future "rest" parameters, but there are other ways to manage the same goal.

(Other comments have mentioned the author conflates context management with applying reversible functions. I've only dealt with one of those things here, but similar techniques apply in solving both problems.)

<comment><USER27> I think that few people are actually aware of this as an idiom. Rather, many people are aware of just a few special cases, but have not considered this in the more general form.

All the examples of various languages providing this for open files also kind of misses it. The point is not whether a particular language supports this for open files, but whether people recognize the idiom and the language supports doing this yourself wherever it fits. For a pattern this incredibly common it's a point well worth making.

<comment><USER26> Yeah, you may be right that the full generality of the pattern is not appreciated.

For what it's worth, I think most commonly used languages can implement some form of the context function in my example.

</comment></comment></comment><comment><USER28> C++'s variant is RAII: [http://en.wikipedia.org/wiki/Resource_Acquisition_Is_Initial...](http://en.wikipedia.org/wiki/Resource_Acquisition_Is_Initialization)

<comment><USER29> What RAII and Ruby's block statement lack here are automatically doing the reversing of functionality.

Few languages have any procedural way to _do_ that.

If Ruby had some way to state "the opposite of opening a file is closing a file" and then use it everywhere, we could do that. The block-syntax open is _nearly_ that, but not quite. You'd need to have no non-block-scope way to use the resource for it to work.

RAII would also give the same benefit, _if_ there were no non-RAII way to declare the resource.

<comment><USER30> Hmm. I don't agree with this conclusion. Take for instance a file class in C++ which opens the file descriptor in the constructor and closes it in the destructor. Now, for any instance of this class, it is impossible for the open fd to outlive the instance!. That is an as strong guarantee as I need! Now, if I leak the instance the fd will leak too, but that is more an issue of manual lifecycle management if you ask me.

This very significant property is what makes me love RAII. I don't count C#'s using-statement and/or rubys block statement as RAII since it relies on useage patterns. C++'s automatic destruction of graphs of objects also makes it outshine all other methodologies, and the guaranteed destruction is the key here.

<comment><USER31> RAII _is_ a usage pattern: You must stack-allocate an object or wrap it in a suitable smart pointer. It's just as easy to do things differently as it is with Ruby's blocks or C#'s using statement. The reason it has a name is because it is a pattern that's talked about so much exactly because following the pattern is so intrinsic to modern C++ programming.

Once you're passing a block to a method like this in Ruby, the execution of the cleanup code is just as guaranteed as it is in C++.

<comment><USER30> I don't agree completely. As I said, you cannot leak the fd without leaking the memory of the object. This means that in C++ you get the correct behavior as long as you take care of the lifecycle of your objects correctly. You don't need to do stack allocation or smart pointers as long as you de-allocate the object when your are done with it. So the lifetime of the fd is tightly bound to the lifetime of the wrapper instance. If you stack allocate without deallocation you have a bug _anyway_ in your C++ program, thus I would like to argue that it is a problem of manual lifecycle management, not a flaw in the RAII implementation.

You do however have a point when you say that you can miss deallocation and that is just as easy as forgetting to put a variable in a block. In an ideal world I would like to be able to decouple de-allocation time from destruction time, as someone mentioned above they are completely orthagonal issues. If this would be possible, one could guarantee the destructor callback, but defer deallocation until later. This would probably give leeway to more effective memory allocation patterns while keeping RAII alive.

And another thing about blocks: how do you deal with trees and graphs of objects? That is not trivial when using blocks. You can get around it, but continuing that way would most likely lead you to something like Haskells monads (the "programmable semicolon" aspect of them).

</comment></comment></comment></comment></comment><comment><USER32> It is a common idiom in C++ RAII [https://en.wikipedia.org/wiki/Resource_acquisition_is_initia...](https://en.wikipedia.org/wiki/Resource_acquisition_is_initialization)

Context managers in Python:

      with fly('Seattle'):
          see_sights()

Real-world example from Fabric <http://fabfile.org>:

      with cd('/to/path'):
          do_stuff()

</comment><comment><USER33> Python has this with the "with" statement. It's often used for files, but you can write your own with classes and an __enter__ and __exit__ method.

However you can use the ``contextlib`` module to make really simple ``contextmanger``'s with the yield statement

e.g.:

        from contextlib import contextmanager

        @contextmanager
        def glcontext():
          glBegin()
          yield
          glEnd()

(This is an example where someone wanted to do OpenGL in python)

</comment><comment><USER34> The OP seems to be conflating two different things: undoing an action (closing a file, etc), and performing the inverse of a reversible function.

The first is amply covered by the RAII/unwind-protect etc idioms than most languages do have.

The second is not, as most programming languages don't have the concept. The closest I can think of is the ability to use a predicate in either direction in Prolog (or logical programming in general)

</comment><comment><USER35> This reminds me of my friend's project for a recent hackathon where he implemented a language that guaranteed every function in it was reversible. It would let you do something like this for _any_ function.

Unfortunately, I don't think he found it particularly useful and gave up on it after the hackathon. Still, it was a cool project.

<comment><USER36> This sounds fun. Is there a link anywhere?

<comment><USER37> There are a pair of research languages called Harmony and Boomerang [1] that does this with what they call "lenses". There are also some easy-reading introductory slides [2,3].

[1] <http://www.seas.upenn.edu/~harmony/>

[2] [http://www.cis.upenn.edu/~bcpierce/papers/lenses- etapsslides...](http://www.cis.upenn.edu/~bcpierce/papers/lenses- etapsslides.pdf)

[3] <http://www.cis.upenn.edu/~bcpierce/papers/harmful-mfps.pdf>

</comment></comment></comment><comment><USER26> C#'s 'using' statements allow this kind of pattern for types implementing IDisposable. It looks something like this (might be a little off):

        using (var file = File.Open("foo.txt"))
        {
            ...
        }

<comment><USER38> This is what I thought of immediately except to implement the end with behavior you have to override the IDisposable shut down sequence? (does that make sense?)

</comment></comment><comment><USER39> LISP has had this for years:

(with-open-file (output "foo.txt" :direction :output :if-exists :supersede) (print-foo output))

<comment><USER20> That's not a general op/inverse thing like J apparently has, though. Also, which lisp?

<comment><USER40> That looks like Common Lisp, which also has a general version named unwind- protect: [http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec...](http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/Body/speope_unwind- protect.html)

<comment><USER41> Not really. unwind-protect is just Lisp's version of Java's "finally" statement.

<comment><USER40> Oh you're right, it'd need to take a param with the setup form(s) and somehow look up the corresponding cleanup form(s), sort of like getter and setter code registered with setf.

</comment></comment></comment></comment></comment><comment><USER42> The language Nickle (<http://nickle.org>) has a "twixt" control structure:

        twixt (expression1, expression2)
               statement1

If expression1 is true, then statement1 will be executed followed by expression2 in that order. The twixt statement guarantees that expression2 is always executed if expression1 is true, similarly to how "finally" works in java.

<comment><USER43> > If expression1 is true, then statement1 will be executed followed by > expression2 in that order. The twixt statement guarantees that expression2 > is always executed if expression1 is true, similarly to how "finally" works > in java.

Am I missing something here? How is that different from, say:

        if (expression1) {
            statement1
            expression2
        }

    ?

<comment><USER42> It's different in that expression1 is always evaluated whenever control enters the twixt and expression2 is always evaluated whenever it leaves. A simple example is that if statement1 throws an exception, then expression2 is evaluated before the exception is handled. This is how finally works in Java.

More interestingly, Nickle has support for continuations, so if statement1 generates a continuation and makes a non-local exit, then expression2 is evaluated before that exit, and expression1 is evaluated before the continuation is resumed.

In other words, statement1 is _always_ bracketed by expression1/expression2, even in the presence of non-local control flow.

<comment><USER43> Aha! Thanks for clearing that up.

</comment></comment></comment></comment><comment><USER44> If there was a way to mark functions and their inverses consistently in language, then functions with inverses could be used in (semanticaly) pure functional code, even if they were destructive.

For (stupid) example:

        def reverse(x) as inversible with(reverse): #special case - reverse is itself inverse function
            x = x.reversed() #I'm too lazy to code this by hand

        def lastFive(x) as pure: #let's say language checks if pure function only uses pure and inversable functions
            return x[:5] under reverse

Here lastFive is pure function even thought reverse is destructive.

This assumes everybody sticks to one definition of reversing something, in this case functions f and g are inverse of each other, when after invoking f(x), doing only pure or reversable things to x, and invoking g(x), x is unchanged.

And I don't know if this will be useful for anything - the only point seems to be avoiding making copies and still making pure functions. But resulting code, while semanticaly pure, is not threadsafe. So I don't know if it's worth anything.

</comment><comment><USER45> You could dispense with the 'you've never heard of'.

</comment><comment><USER46> IIRC, we use this Idiom in SICP while doing symbolic algebra.

For instance, when we have x+y=z and know only x and z, we need to reverse the equation to have y=z-x. The system was built on nodes which would propagate change (a-la events).

IMO, python's with statement with __enter__ and __exit__ looks mostly like this.

</comment><comment><USER47> I refuse out of principle to click on links with headlines that are so obviously trying to trick me into going to a site.

Make the headline informative instead, if you have great content people will read and share it anyway.

Better version of this headline:

XXXX: A programming idiom that helps you do YYYY

</comment><comment><USER48> Other posts have already mentioned that specific macros of this form exist in several Lisps. I've seen more than one macro-writing guide that talks about the with-something macro pattern for managing resources, and I've seen it applied in libraries that provide macros like with-database and with-query- results.

CL and Clojure could easily support a generic with macro requiring that there be an open and close method for the resource in question.

</comment><comment><USER49> In C# this is "using". using(new something as something) { Do something with it. } //it goes away now.

Typically this idiom is the function call itself. The act of sipping coffee is the point of the function, and picking up the cup and putting the cup back down are the whole point of the function call.

</comment><comment><USER50> This reminds me of Suzuki's idiom, "Leave No Trace," found in the book _Zen Mind, Beginner's Mind_.

</comment><comment><USER51> Python has this idiom for a while now: it's known as a "context manager", also known as the "with" statement. Here's the PEP: <http://www.python.org/dev/peps/pep-0343/>

</comment><comment><USER52> The BETA language does this with methods (you can't override a method, but you can add pre-and-post operations that wrap it).

I was never convinced this was a great idiom. BETA didn't become very popular, in any event.

<comment><USER53> Beta lives on in the Racket class system as (inner).

[http://docs.racket- lang.org/reference/createclass.html?q=inn...](http://docs.racket- lang.org/reference/createclass.html?q=inner#\(form._\(\(lib._racket/private/class- internal..rkt\)._inner\)))

</comment></comment><comment><USER54> An interesting approach to this is the one used in Go, where the "defer" keyword schedules code to run on function exit. It's simple and doesn't require adding more nesting to your code.

<comment><USER31> Ruby has "ensure" if you want something to be guaranteed to run on exit - <USER2> can be used at the end of methods, or by explicitly wrapping code in begin .. end.

<comment><USER55> (Assuming it works the same way as Smalltalk's ensure...) a really nice aspect of this is that the resource is held open while you're in the debugger, but cleaned up when you finish debugging.

</comment></comment></comment><comment><USER10> "Never heard of" is quite a stretch.

During my dark past as a VB programmer (early-mid 90's) I even indented the code for pairs like this.

</comment><comment><USER56> Oh, c'mon! All people with _with_ and _RAII_ and files, please reread the article. At least the last example.

<comment><USER57> I was going to argue that point, but glance at the J docs here:

<http://www.jsoftware.com/help/dictionary/d202n.htm>

Obverse is defined in a table and some special cases, and there's a built in "this is the obverse of that" instruction.

You could easily tie square/sort together as a pair in Python and it wouldn't be significantly different.

Under(square, sum, mylist) or whatever.

</comment></comment><comment><USER58> Pretty sure Ruby has this.

File.open(name, mode) receives a block, executes it, and then closes the file.

<comment><USER59> Ruby has a specific instance of that pattern, but to my knowledge it lacks a way to say, "do some arbitrary operation _that has a side-effect_ , do this, then undo the operation". The function taking the block still has special-case code to open and close files.

<comment><USER60> You still have to implement the inverse of the arbitrary operation, at some point, somehow.

A programming language can't figure out what the inverse of open() is without someone somewhere telling it to close(). The point of the idiom is to have a library figure that out so that users don't have to. Therefore, Ruby supports it fully (there where API developers are sane, that is).

<comment><USER59> Having that built in for _every side-effect in the language_ is pretty amazing, though.

Case by case, the block scoping idiom is a good way to handle it.</comment></comment></comment></comment>


Discovery of scholar's notes shine light on race to decipher Rosetta Stone - <AUTHOR> https://www.theguardian.com/culture/2020/aug/25/discovery-of-scholars-notes-shine-light-on-race-to-decipher-rosetta-stone <comment><USER2> Seeing the Rosetta Stone in person, right at the entrance to the British Museum was one of the most impressive things I've ever seen in my life.

No reproductions or photographs can do it justice or compare to seeing history right in front of you.

I highly recommend visiting the British Museum even if it's only to see this one incredible artifact, though there are countless others there that are also amazing.

<comment><USER3> I'm wondering how the stone is presented to the viewing public. Did you happen to see the sides where the British desecrated this ancient relic by carving an account of their "confiscation" of the Stone from the French?

What may be more impressive than the stone itself is how modern politics gets reflected in the stone like it's a mirror. As one might expect Egypt has been entrenched in a Decades long battle to have the stone returned as a object of their national identity. And yet to your point that no reproduction/photo can do it justice, at one point the British actually produced a replica and gifted in to Egypt in lieu of the real thing.

<comment><USER4> The real politics is actually on how they quickly realised calling it "Egyptian" was a mistake due to ownership claims, and proceeded to start calling their looted artefacts by their dynasty (i.e "Babylonian" or "Sumerian") rather than the name of the country it was stolen from ("Ancient Iraq") in a way of stripping the country away from its heritage.

The same tactic has now been employed in dismissing land claims or pretending that ethnic cleansing isn't ethnic cleansing by hostile groups in the region since the 50s.

<comment><USER5> Are they wrong? Why does a modern person living in Egypt have any more claim to something like Imhotep's toothbrush any more than anyone else? Just by possession of the land where it was left? I don't believe in real estate staking an ownership claim. I have more in common with my worst enemy than I do with my ancestor in 4000 BC.

<comment><USER6> Yes, they are clearly wrong? Modern Egyptians obviously share more links (ancestry, bloodline, culture, geography, etc.) with ancient Egyptians than you do. Whether they have more in common with modern Brits or their ancestors has nothing to do with anything.

<comment><USER5> Sure they have more links. But why does that give them ownership of Imhotep's toothbrush? If possession equals ownership for land (i.e. nobody is really claiming that the land that Egyptians live on should be given to someone other than Egyptians), why not for artifacts?

</comment></comment></comment></comment></comment></comment><comment><USER7> The article just brushes over Champollion but if this story sparks your interest, do yourself a favour and have a look at Champollion's life.

He not only deciphered hieroglyphs and its rules as a language but he was a linguistic genius, with amazing abilities in ancient languages.

[https://en.wikipedia.org/wiki/Jean- Fran%C3%A7ois_Champollion](https://en.wikipedia.org/wiki/Jean- Fran%C3%A7ois_Champollion)

<comment><AUTHOR> Young was also a remarkable polymath, defining _Young 's Modulus_ for characterizing material rigidity/elasticity, and performing the _Double Slit_ experiment demonstrating diffraction, interference and the wave theory of light:

[https://en.wikipedia.org/wiki/Thomas_Young_(scientist)](https://en.wikipedia.org/wiki/Thomas_Young_\(scientist\))

[https://en.wikipedia.org/wiki/Young%27s_interference_experim...](https://en.wikipedia.org/wiki/Young%27s_interference_experiment)

</comment></comment><comment><USER8> If you can find it, A historical introduction to archeology by [https://en.wikipedia.org/wiki/C._W._Ceram](https://en.wikipedia.org/wiki/C._W._Ceram) (who it needs to be noted has a nazi past, which he repudiated but still) is fascinating.

You would want to read more modern work as well, but this is a really good readable story of the history of archeology, and it covers the complexities of the rosetta stone quite nicely.

</comment><comment><USER9> There is a great podcast called The Ancient World that has an episode on the Rosetta Stone here:

[https://ancientworldpodcast.com/2014/04/episode-r1-broken- st...](https://ancientworldpodcast.com/2014/04/episode-r1-broken-stone/)

The whole set of R-prefix episodes on rediscovery of ancient artifacts is fascinating.

</comment><comment><USER10> No reproductions or photographs</comment>


W3C Working Group Just Killed Web SQL Database - <AUTHOR> http://www.w3.org/TR/webdatabase/ <comment><USER2> Probably a good thing. The most complex part of Web SQL amounted to "Whatever SQLite implements", neither Mozilla nor Microsoft supported it making the standard effectively irrelevent, and I remain thoroughly unconvinced that providing an extremely thin JavaScript API to an SQL-driven database is an elegant solution to this particular problem.

Presumably the W3C have decided to favour IndexedDB instead, which solves many of the same problems in a much more elegant, simple and well-specified fashion.

<comment><USER3> _The most complex part of Web SQL effectively amounted to "Whatever SQLite implemented"_

Which is one of the most tested, most utilized, most known pieces of code used among modern software. Saying "reference it" is entirely rational. And really the ludicrous notion that documenting its behaviors is unfathomably difficult is so asinine that it demolishes that line of argument.

 _neither Mozilla nor Microsoft supported it_

Microsoft doesn't REALLY support either: Does IE9 have IndexedDB? Not that I'm aware of.

Microsoft's angle regarding Web Databases was nothing more than "I don't know...that looks mighty uppity for a web app..."

Mozilla is taking a very myopic, "high-road" approach that is similar to, and less justifiable, the whole h.264 debacle: Everyone loses, but someone gets to hold their head up high.

Further, Chrome, Safari, Opera, other...none of them support IndexedDB, so as a standard it is effectively irrelevant.

 _Presumably the W3C have decided to favour IndexedDB instead, which solves many of the same problems in a much more elegant, simple and well-specified fashion._

Ha ha ha!

The IndexedDB spec right now has _incredible_ gaps that make it close to useless. The braintrust at Mozilla that are responsible for it can't stop themselves from saying "CouchDB" long enough to actually work on it, it seems.

I think they got caught up in the whole NoSQL thing. So much so that they now seriously advocate that libraries should pop up inefficiently implementing SQL on top of IndexedDB, which is just saddening.

But aside from that, rah!

<comment><USER2> The truth is the reason they pointed to SQLite is because the proposed API was so obscenely complicated that it was easier to point to a reference implementation than to accurately specify a standard, which points to a rather fundamental problem with the proposed standard: SQL is too damn complicated, and embedding it in browsers is a bad idea.

For a web standard to succeed, you need either Mozilla or Microsoft support, preferably both. They have 75% of the browser market between them, so without either any standard is dead on arrival. Chrome, Safari and Opera individually do not have enough marketshare to be a significant influence on the takeup of web standards and still follow in the footsteps of the others. Incidentally, Chrome does support IndexedDB ([http://peter.sh/2010/10/indexeddb- milestone-8-es5-strict-mod...](http://peter.sh/2010/10/indexeddb- milestone-8-es5-strict-mode-and-progress-on-print-preview/)).

Mozilla have made it clear they are supporting IndexedDB and not WebSQL. Microsoft have publicly posted remarks of support of IndexedDB on the IE blog ([http://blogs.msdn.com/b/ie/archive/2010/03/09/working- with-t...](http://blogs.msdn.com/b/ie/archive/2010/03/09/working-with-the- html5-community.aspx)) and Pablo Castro, an architect involved with web standards has publicly favoured IndexedDB over WebSQL ([http://blogs.msdn.com/b/pablo/archive/2010/01/04/html5-does-...](http://blogs.msdn.com/b/pablo/archive/2010/01/04/html5-does- databases.aspx)). It's not always easy to judge the inner workings of Microsoft, but I think it's a pretty strong hint as to what they are working on.

The IndexedDB spec seems relatively well-specified to me, and much more detailed than the WebSQL spec. Incidentally, the original authorship of WebSimpleDB was Nikunj Mehta, who was an Oracle employee at the time. The standard didn't originally come from any "braintrust at Mozilla".

<comment><USER4> Why not just use SQLite? Write a small wrapper around it to isolate each website off on it's own database or set of databases. Each site corresponds to a SQLite user with a password that ONLY the browser knows.

<comment><USER5> Because if everyone used SQLite there would be no second implementation, which is something that is _necessary_ for a proposed standard to become an actual standard.

<comment><USER6> I believe you, especially as it makes a number of lines I've heard around this make more sense... but why? Especially in this case, where there _is_ a standard and it's already been implemented by several? Is it just pedantry because they don't get to call the shots?

<comment><USER5> > where there is a standard

There is no standard. There is a _proposed_ standard, but it's failed because it didn't have multiple implementations.

> and it's already been implemented by several

No, it has exactly one implementation: SQLite. SQLite-in-Safari and SQLite-in- Chrome obviously don't count as different implementations.

<comment><USER6> It has one implementation implemented identically in more than one browser. SQLite. Which _is_ a standard _with_ an implementation to compare against.

Anyway.

Yes, there is only one implementation, one standard, one non-W3C-spec. Why does it matter to them that there's no competing spec, when they _could_ just support something that is already out there, and can be solved by pointing to <http://www.sqlite.org/docs.html> ? The API for accessing it is mindlessly simple and small, documenting _that_ would take a day at worst.

It seems they're trying to run it into the ground _because they don't control it_ , because it didn't go through their proper procedure, whereby it must have a competitor.

Why? In what way is this helpful? And if they're not being helpful, what's their purpose? To be dogmatic and prevent good ideas they didn't come up with from spreading?

\---

edit:

> _The World Wide Web Consortium (W3C) is an international community where > Member organizations, a full-time staff, and the public work together to > develop Web standards. Led by Web inventor Tim Berners-Lee and CEO Jeffrey > Jaffe, W3C's mission is to lead the Web to its full potential. Contact W3C > for more information._ (from <http://www.w3.org/Consortium/>)

How does trying to prevent it aid in "lead[ing] the Web to its full potential"?

<comment><USER5> _Why does it matter to them that there's no competing spec_

Because _ALL_ web standards are standards only because of the fact that there are _MULTIPLE_ implementations. This is a basic principle behind the Web, and making even a single exception leads to a dangerous slippery slope.

 _It seems they're trying to run it into the ground because they don't control it_

No.

 _because it didn't go through their proper procedure, whereby it must have a competitor._

Yes. In the long term, not having another implementation is a dangerous sign -- it is a sign that developers are incapable of coming up with another implementation. What if SQLite can't be ported to some hypothetical architecture/stack in the future -- for example, a fully managed OS? What is the guarantee that someone will be able to come up with another implementation of the same spec which can run on that OS? At least with multiple implementations you know that the problem is tractable.

 _Why? In what way is this helpful?_

It is helpful because it demonstrates long-term thinking.

 _And if they're not being helpful, what's their purpose? To be dogmatic and prevent good ideas they didn't come up with from spreading?_

No, to think in the longer term than you do.

<comment><USER6> And the multiple implementations of IndexedDB are...? (don't actually know. If you do, I'd be interested in links)

Similarly, there are plenty of implementations of SQL out there. They don't count?

<comment><USER5> _And the multiple implementations of IndexedDB are...? (don't actually know. If you do, I'd be interested in links)_

There's an early one in Firefox 4 beta right now [1], and an upcoming one in Chrome [2]. Microsoft will probably implement IndexedDB in a future version of IE [3]. (IndexedDB isn't a standard right now either, but it'll be on track to be one once two implementations are complete.)

[1] <https://wiki.mozilla.org/Firefox/Projects/IndexedDB>

[2] [http://www.chromium.org/developers/design- documents/indexedd...](http://www.chromium.org/developers/design- documents/indexeddb)

[3] [http://blogs.msdn.com/b/ie/archive/2010/03/09/working- with-t...](http://blogs.msdn.com/b/ie/archive/2010/03/09/working-with-the- html5-community.aspx)

 _Similarly, there are plenty of implementations of SQL out there. They don't count?_

No, since none of them share all the peculiarities of SQLite. For example, SQLite doesn't have type checking, while most other SQL databases do.

<comment><USER6> I don't think _any_ browser shares all the peculiarities of any other browser. That's hardly an argument against there being a W3C spec for HTML, CSS, the DOM, or JavaScript. Why pick on SQL?

<comment><USER5> By "peculiarities" I didn't mean minor implementation differences but the major ideas behind the implementation. Be reasonable here -- the differences in implementation in the DOM between Gecko and WebKit (say) are trivial compared to "type checking" vs "lack of type checking". There's a bit of subjectivity involved, agreed, but I think any reasonable person would agree about the differences.

<comment><USER6> There are indeed differences. Much greater with SQLite vs (most) other SQL engines.

I still see it as killing something out of some ulterior motive rather than supporting something highly effective. SQLite exists because it has survived against _many_ competitors for a _long_ time. It's not like this is some new, untested thing that cropped up out of nowhere.

edit: just realized there's a _much_ stronger argument here. And a bigger threat.

By not supporting WebDatabase, because the implementers all used SQLite, they've essentially _guaranteed_ that they will never support an SQL-based database. Few are going to want to compete with SQLite on its own turf as an embedded database when it's already so optimized. _Especially_ if they all have to agree on a spec similar to SQLite.

SQLite isn't even all that poor of a match for a JS-interfaced database. JS is nearly as fast and loose with its types as SQLite, and the types that _do_ exist are very close to what JS has.

<comment><USER5> You still haven't answered what you're going to do if the current SQLite code simply can't run on a particular OS stack. There's no guarantee that every future stack will have a C compiler, or even allow C programs.

<comment><USER6> All that matters is the behavior, which SQLite has very well defined because it's a bit wonky out of maintaining backwards compatibility. Even the fundamental algorithms will still translate, so all that's needed is to compile it or something logically similar for the system with whatever compiler it allows. Heck, it'd probably be _easier_ than implementing off an abstract spec, especially when you consider most of the optimizations will still translate but do not need to be discovered.

If such a system exists, it won't be able to run the vast majority of stuff out there. Re-implementing SQLite will be the least of its concerns.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> Not quite, methinks:

> _This document was on the W3C Recommendation track but specification work > has stopped. The specification reached an impasse: all interested > implementors have used the same SQL backend (Sqlite), but we need multiple > independent implementations to proceed along a standardisation path._

That notice has been up for quite a while now. They're just making it plainer that they've given up, because there's already a spec out there that they don't have control of.

Also, to perform a join in IndexedDB, you've gotta do it by hand, iterating over the results and merging data programmatically[1]. While WebDatabases have a PITA async-only API for reasons which are entirely beyond my comprehension, SQL is an _incredibly_ capable language for interacting with a database, and has _very_ mature and efficient implementations. Losing that would count as a _massive_ loss to me. Why can't we have both? WebKit / Chrome is already going this route, as they have WebDatabase and recently implemented IndexedDB as well.

[1]: [http://hacks.mozilla.org/2010/06/comparing-indexeddb-and- web...](http://hacks.mozilla.org/2010/06/comparing-indexeddb-and-webdatabase/) example 4

<comment><USER7> _While WebDatabases have a PITA async-only API for reasons which are entirely beyond my comprehension_

That's because they want to support responsive applications. Javascript doesn't really have native concurrency operations, so the best (only?) way to do it is via async operations.

It's like XMLHttpRequest: Async only, which can be harder to program, but makes it a lot more robust.

<comment><USER8> > Javascript doesn't really have native concurrency operations

Except for Web Workers, also part of HTML5. The way Google Gears handled this was to allow synchronous database operations only in worker contexts.

<comment><USER9> Worker<->UI communication is asynchronous, and has additional limitations (only copying, only JSON-able data), so synchronous worker API in many cases would just shift complexity elsewhere.

<comment><USER6> In part. It makes conditional, nested queries much simpler, however, when you don't need to nest async callbacks 20 layers deep.

I understand that having a synchronous API would mean more people would use it. But honestly, I see that as their problem. A fair number of queries, such as "select * from your_table where id = ?", are _useful_ to have synchronous as an option, and should perform quickly enough to be invisible. And synchronous can easily be made asynchronous, but the other way around is essentially impossible without any Thread.join.

</comment></comment></comment></comment></comment><comment><USER10> The W3C didn't kill it, it was Mozilla and Microsoft who decided they didn't want to implement it (for a variety of reasonable reasons) and me who decided to not continue working on a spec that didn't have all the browser vendors on board.

</comment><comment><USER11> No one has mentioned yet that both Android and iPhone implemented SQLite as described in the draft spec. There's a tremendous number of developers -- including myself -- who have built apps against it. The described spec may not become commonplace in desktop browsers, but it's an entirely different story on advanced mobile devices where Mozilla / Microsoft have nearly zero influence.

I definitely don't see this just suddenly evaporating or going unmaintained in those platforms.

</comment><comment><USER12> This spec isn't the only game for offline storage. IndexedDB is the favored replacement: <http://www.w3.org/TR/IndexedDB/>

I like it because the API is much more web-like than SQL. Others have other reasons to prefer it.

<comment><USER13> For someone unfamiliar with the term, what does "web-like" mean?

<comment><USER6> I'll be obtuse, and claim it means "JavaScript-like". You know, multiple "truthy" values, the inability to detect the difference between arrays and objects, a weird mashup of OO and functional abilities, `=== undefined` but `isNaN()`, and no real difference between integers and floats, and all the math weirdness that goes with them. Browsers implementing the same thing different ways, interacting with a system (the DOM) that's the same but different everywhere, and patched together by multiple, usually-mutually- exclusive libraries that everyone uses instead of the actual language.

(disclaimer: I've come to quite like JS, it's made some of my experiments phenomenally simple. but it's weird.)

<comment><USER14> isNaN() only exists because the real way to detect NaN is so counter intuitive. Lots of people will see "if (x !== x)" and will try to "optimize".

</comment></comment></comment></comment><comment><USER15> Just great, it just seemed like we will have something this decade, but sadly that will not be the case.

I have read the comments in Mozilla's thread [http://hacks.mozilla.org/2010/06/beyond-html5-database- apis-...](http://hacks.mozilla.org/2010/06/beyond-html5-database-apis-and-the- road-to-indexeddb/) they clearly don't give a damn about what the users want.

They should have just agreed on a simple subset of SQL instead of going NoSQL, but apparently that wasn't good enough for them, waiting a decade more for something to store data in is probably better.

I personally love the concept of NoSQL, it's just that IndexedDB needs a few more years, which means it will probably even miss IE10.

</comment><comment><USER16> Why, again, was this needed? HTML5 is _already_ looking like a security and management shitstorm; anything that makes it simpler is fine by me.

<comment><USER6> How is a domain-locked local SQL database a security shitstorm?

<comment><USER16> I suppose I'm just jumpy because of the lack of granularity in user control over local storage, and wary of the eptitude of the browser vendors, and in favor of fewer, smaller features. After reading the inept IndexedDB thing, however, it seems like the _wrong_ local storage mechanism is going to get it in the neck.

<comment><USER6> You've got just as much control. Grab the console and 'drop table' - all that data, gone. You can even trash all the databases in most browsers pretty simply. And restrict their size. The difference is that client-side JavaScript becomes _useful_ with a large-ish local storage.

Arguably, it's significantly safer than cookies, which are sent with every request (ie: Firesheep).

<comment><USER17> The issue would largely be storing sensitive data (I presume), namely personally identifiable information. Imagine all the user data being stored in a simple and quick API that advertisers (et al) can grab at, rather than having to scrape through HTML bits on a page. Now imagine malware spreading across the internet and frictionlessly gathering this data.

Although some might argue we're already there and that there's nothing stopping the deluge of user information.

</comment></comment></comment></comment></comment><comment><USER18> I think that's a great news. HTML5 adds lots of complexity in the browser, and yet don't address most of the fundamental problems of HTML: security, complexity. Web SQL was another step in the wrong direction.

Local storage wont go away. I hope Web SQL will be replaced by something "better" and more straightforward, A simple key-value database would be great.

<comment><USER19> > A simple key-value database would be great.

That's exactly it:

<http://www.w3.org/TR/IndexedDB/>

<comment><USER20> IndexedDB is anything but simple.

</comment></comment></comment><comment><USER21> Am I the only one that's sad about this?

<comment><USER22> Yeah, me too. I think it's just politics, in the best traditions of big corporations, that doesn't do any good to actual web developers.

From practical perspective we would be better off with Web SQL DB. Functionally it addresses everything a developer needs to create offline HTML apps. API is not perfect but you can always improve that with small wrapper libraries (like JQuery improves DOM API). You can always easily implement IndexDB-like API on top off SQL if you really want that. The opposite is probably possible but much more difficult.

Also I think IndexDB might turn out to be yet another compatibility headache because everybody will be creating their own implementation (I'm pretty sure there will be at least 3 - Microsoft, Mozilla & Webkit). It's a speculation but most likely this wouldn't be an issue with Web SQL DB because everybody would just use the same SQLLite backend.

And worst of all, this is delaying progress of web applications. Lack of solid cross-browser client-side storage solution is the last big gap for next generation of web applications. If everybody had just agreed on SQLLite we could be there today... now it's still unknown when we will get there.

<comment><USER5> _Also I think IndexDB might turn out to be yet another compatibility headache because everybody will be creating their own implementation (I'm pretty sure there will be at least 3 - Microsoft, Mozilla & Webkit). It's a speculation but most likely this wouldn't be an issue with Web SQL DB because everybody would just use the same SQLLite backend._

For a proposed web standard to become an actual web standard there need to be at least _two_ implementations. This is a _good_ , healthy thing. There was going to be no second implementation of Web SQL, so it was never going to become a standard.

</comment></comment></comment><comment><USER23> So is IndexedDB a beefier version of localStorage or does it serve a different purpose altogether?

<comment><USER12> Local storage is just key/value queries, IndexedDB allows you to make range queries against indexes you define, so you can get eg: recent blog posts, etc.

<comment><USER24> yeah. to give an example:

if I want to store persist objects using localStorage, I'd convert them to JSON and store key-value. However I cannot store whole object hierarchy as it'll soon become an overkill.

so I store separate objects, and recreate relations upon retrieval manually. But I also need to maintain my own index - to know which objects I have stored under what keys.

</comment></comment></comment><comment><USER24> The heading is misleading imho.

As you see written in the spec, it was on the track to W3 Recommendation, but as every vendor who implemented it chose SQLite, there was little to do about it.

In other words, it isn't a "kill", just end of active maintainence. But spec as it is, is pretty solid. And WebSQL is here to stay in WebKit - there are just too many apps dependent on it to kill it. And no good reason to kill as there's no viable alternative on the horizon.

Summing up - WebKit (so iOS, Android and others) has it and will have it for quite a while, IE won't, FF won't too.

</comment><comment><USER25> SQLite is already baked into Firefox and available to Firefox itself and to extensions/addons, although not available to web page authors:

<https://developer.mozilla.org/en/Storage>

</comment><comment><USER26> So reading all the comments I see that most "against" are because "Because if everyone used SQLite there would be no second implementation, which is something that is necessary for a proposed standard to become an actual standard." WTF? Do we need to implement MySQL, PGSQL or MSSQL to get a chance of having SQL in our browsers? Does that sound stupid only to me? Is Mozilla trying to follow the "success" of IE? Who pays W3C? Why do they play like the God of Web? And why can't we have both? I certainly don't want to ever use IndexedDB if I have and SQL option.

</comment><comment><USER27> So what does this mean for browsers that support web SQL and apps that are built on it? Is it likely that Safari, Chrome, etc. will drop support for it before a viable alternative is avaliable?

<comment><USER9> It will be maintained until IndexedDB becomes viable.

Perhaps someone will write JS library that implements IndexedDB on top of WebSQL (or vice-versa).

</comment></comment><comment><USER28> Lets hope it stays dead, seems that we just dodged a pretty nasty bullet. IndexDB seems to also be ridiculously complex, but as wicked as most standards are anything is better than SQL.

</comment><comment><USER29> why would you write SQL with JavaScript? don't think that JavaScript yet at a point it can run heavy GUI element, handle AJAX, maybe Video or 3D _and_ Query a database. JavaScript likes JSON very much, so all you need is just export your data with a language like PHP in JSON format, and handle it with JavaScript.

</comment><comment><USER30> There's enough security/privacy issues with cookies, and those are simple text strings. Imagine the issues a nefarious mind could cause with a DB.

<comment><USER4> Just imagine how scary it wouldn't be if you understood the security implications and risks of cookies compared to the lack there of for a local browser db storage....

</comment></comment><comment><USER31> If the SQL guys were serious about their shit they should have patch some JavaScript at the syntax level.

Instead they pushed/lobbied for the usual broken binding that is the very reason SQL is so contreversed: impedance mismatch.

Of course, hacking a javascript parser to introduce some SQL semantic in it is not exactly "easy"

Basically, the initiative was doomed from day one, what a waste of time...

<comment><USER32> I'm not sure what you're trying to say. SQL works quite well in just about every language there is without introducing new syntax. Are you talking about something like LINQ? Nice, but unnecessary.</comment></comment>


Elon Musk Keynote Presented by Oracle - <AUTHOR> http://sxsw.com/elon-musk-keynote-presented-oracle

<comment><USER2> He framed the Mars problem quite powerfully. Seems his primary reasoning is quite practical, compelling and admirable at that: hey, humanity has the ability to travel off-world to other planets. Let's do it now while we still can...

we obtained this ability and can equally and quite potentially lose it just as easily. Our sun is not getting any younger.</comment>


We're ready for foldable phones, but are they ready for us? - <AUTHOR> https://techcrunch.com/2019/02/27/were-ready-for-foldable-phones-but-are-they-ready-for-us/ <comment><USER2> phones with moving parts always had problems in the past, now there will be same issues with foldables too, however, i like they are pushing limits to make better foldable phones.</comment>


New broadband users shun UK porn filters, Ofcom finds - <AUTHOR> http://www.bbc.co.uk/news/technology-28440067

<comment><USER3> I love how they repeatedly frame the internet censorship as "porn filter".

Even in the face of (their statistics)6 out of 7 people opting out, it's still desperately trying to be presented as the feature that will save you from the bad parts of the internet. The implication being that the majority of people (everyone who has opted out) are morally corrupt, bad people with no interest in "child safety" or "family friendly" features.

I imagine Malcolm Tucker from The Thick Of It screaming down to the BBC offices and making sure they use the official party line regarding "porn filter" vs "Great Firewall of Cameron".

<comment><USER4> Personally this is just reassurance that Brits are sensible people. That most understand filtering to be annoying and shamelessly opt out.

And I find it hard to ciriticise parents that leave the filtering active. It is perfectly reasonable to try and shelter a child from porn. This tech may not do that particularly well but that is a different issue.

<comment><USER5> Is it necessary to shelter children from porn? As a child of the internet in a time when porn filters didn't even exist and most of the sites on the internet were mainly porn sites I find that argument without legs.

Don't get me wrong you're well withing your right as a parent to postpone the issue as long as you like but is it really as necessary as most people tend to think?

<comment><USER6> I think it would be more reasonable to monitor your child's search history and talk to them about pornography than to try and hide it from them and treat sex like its dirty and gross.

</comment></comment></comment></comment><comment><USER7> The ISP provided optional filters all differ from each other and offer varying levels of granularity. This article does a poor job of explaining exactly what people are opting out of - are they opting put of all filtering? Or is thos article only talkin about porn filters?

And just for clarity: the UK has three sets of filtering. There's the default- on but optional filters offered by ISPs (and similar has been offered by mobile providers for years); there's the IWF system which is specifically images of child sexual abuse and criminally obscene content. As far as I can tell only one ISP doesn't use that filter - A&A; then there are court ordered filters used to prevent access to pirate bay and KAT. Personally, I feel this is sub-optimal. The government said that users were clamoring for filtering to be provided by ISPs. I guess this shows that most of them were not.

<comment><USER8> Indeed the filters are very unpopular and the uptake figures haven't significantly changed from July, as far as I know?

Claire Perry MP was campaigning for porn censorship by default. She's only got that from Sky now, a huge reversal from what they said before, and when you see that not even 10% of their customers who were asked wanted it, it's clear their brand manager's made a huge mistake making it opt-out.

I thought then, and think now, that the TalkTalk bias is due to the demographics of the users of that ISP. Everyone with the technical skill to visit SamKnows or ThinkBroadband knows TalkTalk are poorly-rated and thus tends to avoid choosing them, I'd venture.

As said before, censorship infrastructure itself presents a very real vulnerability, although this doesn't seem to be fully appreciated by many of the stakeholders.

<comment><USER9> Plus the filters break websites all the damn time. I cannot count the number of times on both hands that Reddit has "disappeared" from the internet, and if you traceroute it it disappears into an IWF box. This has been almost common on Virgin Media.

Then there was that time that they "accidently" knocked all of Wikipedia offline because of a single IWF-flagged album cover (and then blocked the page, and then finally removed the block completely). It did spawn an interesting discussion on the difference between "child porn" and "art."

<comment><USER8> The jQuery CDN is also a semi-frequent victim of Sky's filter.

I pause here simply to note that it's quite possible to use the infrastructure's distinctive behaviour as an oracle to scry a complete list of all blocked sites, and to monitor that over time, given just a few volunteers on each major ISP.

</comment></comment></comment></comment><comment><USER10> In other news: hot water still hot, thermometers find.

I wonder: what was (and is) the economic cost of this whole charade? How many man-hours were spent writing memos, how many meetings were called, how many expensive developers and sysadmins were forced to waste their time implementing stuff that nobody wanted -- likely not even the proponents: I'd be surprised if Cameron and his entourage were actually using these filters.

<comment><USER11> Spare no expense if it will create a wedge issue that gets people elected.

<comment><USER9> In the UK? I highly doubt it.

The UK has a lot of wedge issues, like austerity, capitalism Vs. socialism, privatisation, classes, illegal wars, women's issues, train ticket cost, anything touching the NHS or the roads...

However they are rarely what I'd call puritanical like in the US and Ireland. I'm sure it does happen, but I cannot think of any particular moral-based wedge issues from the last few elections.

UK politics is very pragmatic.

<comment><USER12> Immigration!

Apart from that, not too many wedge issues. Certainly not train ticket costs or anything like that (where are the people who want it to go _up_?).

<comment><USER13> >> "Apart from that, not too many wedge issues. Certainly not train ticket costs or anything like that (where are the people who want it to go up?)."

In charge, it would appear.

<comment><USER12> Touche

</comment></comment></comment></comment></comment></comment><comment><USER14> To the surprise of exactly nobody, it turns out that people don't mind being able to access porn.

<comment><USER15> Are you sure that its not that the first site they hit which is a false positive causes them to opt out?

ThinkPad wiki is the one comedic false positive I get when I was testing Sky's offering (to determine workarounds in case this shit becomes mandatory one day).

<comment><USER14> I'm sure that's what everyone will claim :P

Seriously though, a huge proportion of people access porn even though an equally huge proportion of people would deny it. And that's the whole point of a private life - people should be free and comfortable to do things in private that they might not want other people to know about. You shouldn't HAVE to tell anyone about your online habits, it's none of their business.

It's not the Government's job to even ASK you about that sort of thing, let alone TELL you to have these filters. It's patronizing and paternalistic and intrusive.

Of course, it's also a thinly veiled excuse to force the installation of traffic monitoring and filtering equipment across the whole of the UK internet - since obviously you can't block sites without inspecting the traffic. It certainly makes GCHQs job easier - why bother hacking servers and tapping fibre when you've got ISPs forced to monitor every connection and paying for it too.

</comment></comment></comment><comment><USER16> Censorship should always be opt-in, not opt-out.

</comment><comment><USER17> This report is from July of last year.

</comment><comment><USER18> While I wouldn't be surprised if most people in the UK don't want the filters, this data by itself is not really representative.

I presume most households already have Internet. New customers will to a large degree be people that are moving out on their own, and young adults living by themselves are less likely to want a porn filter than somebody with children. (And people with children are less likely to have available time to rurn things on.)

<comment><USER15> Actually we like to switch providers a lot to snag the best deals.

</comment></comment><comment><USER19> Does this apply to men and women equally?

<comment><USER20> With numbers like four percent vs gender ratios around half, I don't think it can be blamed on gender.

Astroturfers try to tell us that in the UK everyone in the general public loves police surveillance, but when given a choice, it appears not to be as loved and appreciated.

<comment><USER7> These filters have nothing to do with police surveillance.

And you're wrong about how the UK public feels about surveillance. The UK has more CCTVs than most countries and we didn't get here by most of the public hating cctv.

Here is a story from _this year_ where police request men to give a DNA sample, and 500 men voluntarily give police a DNA sample. Are all these men "astrotrufers"?

[http://www.bbc.co.uk/news/uk-30920277](http://www.bbc.co.uk/news/uk-30920277)

[http://m.bbc.co.uk/news/uk-england- sussex-30972723](http://m.bbc.co.uk/news/uk-england-sussex-30972723)

We have CCTV everywhere. People in the UK object when they think it's misused - councils checking whether people are using the right bins for example - <USER2> even then (councils checking whether people claiming to live in a school's catchmet area actually live there) you don't have to look hard for people defending that.

If yu want to have any hope of changing the surveillance culture you need to accept that most people in UK just don't know much about surveillance and wouldn't care much if they did know.

<comment><USER12> >And you're wrong about how the UK public feels about surveillance. The UK has more CCTVs than most countries and we didn't get here by most of the public hating cctv.

It helps that CCTV is pretty fucking useless. It's hardly the same thing to take a grainy video of you walking down the high street as it is to slurp up every email you've ever written.</comment></comment></comment></comment>


Ask HN: How do you keep track of meeting notes, to do lists, random ideas? - <AUTHOR> Do you use an app (like Evernote, Notion etc) on a computer or use digital pen apps (Notability, Goodnotes) or just plain paper/pen? What has worked for you? <comment><USER2> I use Roam Research ([https://roamresearch.com/](https://roamresearch.com/)) since 3 months ago. I used Evernote for many years, but ended up with lots of notes that I never read again, because it wasn't simple to link them. Roam is better because instead of a WYSIWYG HTML editor like Evernote, it is built on top of graphs and bidirectional links.</comment>


Show HN: My wife and I built a location infographic generator - <AUTHOR> http://cartoscope.io/ <comment><USER2> Really interesting much more consumable than most census data infographics I've seen. Can you speak to what some of the "private sources" of data are?

On a more technical note, I can't get any reports to render in Firefox 56 (working fine in Chrome). Also, it seems Angular is running in development mode on the live site?

<comment><AUTHOR> Hey @nprescott,

We have purchased some data (database of business locations, database of consumer trends) that we integrate into our "premium" reports. You don't see any mention of those because we haven't rigged up an e-commerce system. But some users have more reporting options than what you see.

Regarding dev mode, I'm having trouble getting the 'ReportViewer' component, which depends on JIT compilation to jive with angular-cli's built-in AOT compilation functionality. Thus `ng build` works when I build it in dev mode, but not prod.

Let me know if you'd like to chat more, email@example.com

</comment></comment><comment><USER3> Super cool. Is US Census data easy to work with as a developer? How much of your work was spent on parsing out the data to be a useful format?

<comment><AUTHOR> Hey Godot!

Census data is _relatively_ easy to get used to once you know how the geo formats work and you are willing to work with their geographies. They have a nice API to work with (api.census.gov) and I think that it is going from its current array of arrays format to full JSON later this year.

That said, the app does two cool things that I have not seen anywhere else:

(1) Arbitrary geographies, like drawing a circle or a polygon of your own choice, requires significant additional work. The app mostly does this with procedural SQL and PostGIS on the back-end.

(2) Longitudinal reporting. The Census bureau changes the geographies every 10 years to some degree, so geography with id "foo" might not be the same area in 2010 as it will be in 2020. This is solved by "virtualizing" the geography and is relevant to the same concepts for handling arbitrary geographies.

Happy to chat more if you have questions!</comment></comment>


Obama promises new era of scientific innovation - <AUTHOR> http://www.newscientist.com/channel/opinion/dn15130-obama-promises-new-era-of-scientific-innovation.html?feedId=online-news_rss20 <comment><USER2> NO MORE POLITRIX. Please.

Edit: Submit when he's actually done it instead of _talked_ about doing it. Politics drive me (and most of HN) insane.

</comment><comment><USER3> > Politics drive me (and most of HN) insane

But you voted in whatever election(s) are appropriate in your jurisdictions, right? And maybe donated time or money to causes you believe in (as they're often political as well)?</comment>


Google was overwhelmingly against Trump, now it's trying to win him over - <AUTHOR> http://www.cnbc.com/2017/01/17/google-was-against-trump-now-its-trying-to-win-him-over.html <comment><USER2> AT&T, through Bell Labs, tried to be indispensable to the US Govt throughout the Cold War so that it would allow its monopoly to continue to exist. Google is following the same game plan. The problem with this game plan is that by Google seeming too cozy with USG it makes it harder for it to operate in foreign markets. AT&T did not have this problem as foreign markets during the Cold War were much smaller and more closed.</comment>


Hacker News doesn't pay the bills - <AUTHOR> https://blog.reifyworks.com/hacker-news-doesnt-pay-the-bills-fdeb5c08096a#.auuopd7m8 <comment><USER2> First line of article: _In B2B SaaS, value MQLs over traffic._

Has anyone seen my Babel Fish?

<comment><AUTHOR> Even worse than that, it's the subtitle!

In companies that sell to other companies, not to individuals (B2B -- business to business), value people who perform specific actions to get to your app (marketing qualified leads -- MQLs) over traffic.</comment></comment>


Go server starter - <AUTHOR> https://github.com/AquiGorka/go-server-starter <comment><USER2> If it meets the guidelines, this might make a good 'Show HN'. Show HN guidelines: [https://news.ycombinator.com/showhn.html](https://news.ycombinator.com/showhn.html)

<comment><AUTHOR> I will definitely share it that way as soon as I fix one issue.</comment></comment>


In  2017, UK water companies still rely on "magic" - <AUTHOR> https://medium.com/@sallylepage/in-2017-uk-water-companies-still-rely-on-magic-6eb62e036b02 <comment><USER2> In engineering you've got to do whatever it takes to solve the problem.

If you ever have a really tricky bug, try printing out the entire source repo on origami paper and laying out the sheets in a Hilbert curve pattern on the floor. Then throw out some silicium sticks -- one for each thread you were running -- and 63% of the time, one will land on the buggy line of code.

<comment><USER3> The most strange bugs have the strangest solutions. Just an hour ago I was trying to debug a queueConstructor (it returns a malloc'd pointer to a queue data structure), right after I finished implementing the operations for a stack data structure. Malloc was inexplicably causing the program to hang (I don't know if it's a segfault, I was using Windows). printfing around the constructor and inside it didn't work. I had included all the libraries I needed to, syntax was good, no warnings, I even changed the declarations from int _var to int_ var.

It turns out the problem was that in main() just before I called queueConstructor, I added the item 633346 as an integer element in my stack data structure. Whether this overflowed the integer, I don't know. But I don't know why it would stop malloc for a different data structure stored in a different variable from working (perhaps it was occupying some memory that my queue wanted?).

This isn't the first time bugs like this happen, and as a novice programmer I really wish tools were better, or I knew how to use the tools. GDB available via CodeBlocks wasn't helpful.

<comment><USER4> That sort of magic isn't right and if you don't find the source then you'll probably end up with more and more weird bugs down the line, to the point that nothing will work and you won't know why.

My first suggestion would be to familiarize yourself with valgrind to make sure that you aren't accessing memory that you shouldn't be. Next would be writing tests so you can isolate problems like this.

When you've got 58 different systems interacting you'll occasionally come across some weird problems that you can't replicate, but they should be few and far between.

</comment></comment></comment><comment> castle-bravo I like to think that dowsers are really a secret guild of hydrologists who make a show of using magic to account for the uncertainty inherent in finding things underground.

</comment><comment><USER5> reminds me of this fake bomb detectors episode, also sadly from real life and not some farcical dark comedy - [https://www.theatlantic.com/international/archive/2016/07/ir...](https://www.theatlantic.com/international/archive/2016/07/iraq- fake-bomb-detectors/490088/)

<comment><USER6> Also Michael Shermer: Why people believe weird things [http://go.ted.com/E35Jfg](http://go.ted.com/E35Jfg)

</comment></comment><comment><USER7> the magic happens in the brain of the user not the rods.

the brain can sense a lot more than we think.

<comment><USER8> Yup, I mostly think that once an area dowser gets experience in where past pipes have been laid, then they get better at guessing on their next call.

<comment><USER7> hunch detector

<comment><USER8> Hunch detector yes, but also human experience with how past pipe laying decisions went. The past people making those decisions weren't just making randomized decisions - they were likely thinking about property lines, and about of running through difficult to dig areas or likelihood of areas that might get dug into unintentionally. Humans interpreting humans, sometimes using talismans to focus an unconscious thought process.

It would probably be made into a better performing process if they got rid of the tokens, and someone thought about it, wrote up what they thought past layment decisions were based upon, and taught others as well as evaluated how well that worked, but maybe not worth that level of formalization/optimization either.</comment></comment></comment></comment>


PostgreSQL Setting up Streaming Replication walk thru - <AUTHOR> https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/ <comment><USER2> Weird that they decided to use the controversial master/slave terminology when the Postgres community at large has been calling it master/standby since at least 2010 [1].

I don't have strong opinions on the issue, but it's a curious choice to go with terminology not only controversial, but non-standard for the product.

[1] [https://www.postgresql.org/message- id/4BFE7213.80603@enterpr...](https://www.postgresql.org/message- firstname.lastname@example.com)

<comment><USER3> Master slave has been a term in computer science forever. I don't see what the problem is.

<comment><USER4> When those words first became common the people for whom they are painful were not as plentiful in the field. The painful meaning of those words far predates the existence of computers.

The lack of foresight of the coiners is not an excuse for current callousness.</comment></comment></comment>


Watch German Typhoons Intercept a Boeing 777 That Lost Contact with the Airport - <AUTHOR> http://foxtrotalpha.jalopnik.com/watch-german-typhoons-intercept-a-boeing-777-that-lost-1792530997 <comment><USER2> Semi-related: There was a TV event/movie in Germany/Austria/Switzerland last year about a (fictitious) court decision about an airforce pilot that shot down such a plane. The plane was taken hostage by terrorists and took course to a football stadium. The (non-fictitious) audience in Germany could vote if the pilot is guilty for killing the people in the aircraft or not. 86,9% in Germany voted that the pilot is not guilty.

[http://www.imdb.com/title/tt5680442/](http://www.imdb.com/title/tt5680442/) [https://de.wikipedia.org/wiki/Terror_%E2%80%93_Ihr_Urteil](https://de.wikipedia.org/wiki/Terror_%E2%80%93_Ihr_Urteil)

<comment><USER3> To give a bit more background: The law strongly disagrees with the court of public opinion here. It is illegal to shoot down an aircraft in such a situation in Germany. The German government at one point tried to change this and our constitutional court told them that their new law violated the constitution and has to be scrapped.

<comment><USER4> There was a new judgement a few years ago:

[http://www.zeit.de/politik/deutschland/2013-04/verfassungsge...](http://www.zeit.de/politik/deutschland/2013-04/verfassungsgericht- terrorismus-flugzeug)

Shooting down an aircraft is legal, but only if the cabinett gives the order. Which is still unrealistic, as the Minister of Defence can't order it alone even in an emergency. The judges recognised that this is suboptimal, but a change would need a change in the constitution. Some former ministers of defence are though on record they would still order it and deal with the legal consequences fallout later.

Then there is the question if the pilot alone (I guess that what the tv movie was about?) could make a decision? For example if the aircraft is flying towards a stadium or nuclear plant and the jet pilot has to make a decision in a split second.

<comment><USER5> The nuclear plant worry is an unrealistic trope. Most Western nuclear plants have a very robust containment dome that can handle the impact from an airliner.

[https://www.nei.org/News-Media/Media-Room/News- Releases/Anal...](https://www.nei.org/News-Media/Media-Room/News- Releases/Analysis-of-Nuclear-Power-Plants-Shows-Aircraft-Cr)

</comment></comment></comment></comment><comment><USER6> > In the meantime, we have some pretty mind-blowing video of what such an > incident actually looks like from the air.

Mind-blowing is a bit of an exaggeration. it's just a video of three planes flying along side one another.

</comment><comment><USER7> Very cool video. I wonder what the fighter jets could actually achieve by pulling up alongside like that? I don't suppose they could do any kind of visual inspection or signaling at that distance - even though one did seem to inch closer at one point.

The wording from the airline statement was a bit funny: _" the German Air Force deployed its aircraft to ensure the safety of the flight and its guests"_ \- considering that their main purpose was presumably to blow up the jet if it posed a threat.

On a side note, I had never seen a video like this before and it really drives home what massive amounts of exhaust these machines put out.

<comment><USER8> > and it really drives home what massive amounts of exhaust these machines put > out.

It just looks this massove, but that's mistly due to condensation of water vapor. Sure, those engines have a massive throughput of air, but the visible "exhaust" is barely the result of burning kerosene.

<comment><USER9> And water vapor from the burnt fuel. Hydrocarbon + o2 = co2+h2o. Those contrails are burnt fuel, just like car exaust on a cold day. The water is a product of burning.

</comment></comment></comment><comment><USER10>
      [Contact] was briefly lost while flying over German airspace. Communication was safely restored within a few minutes.

That sounds like the whole incident didn't take more than 15 minutes. Does somebody know how fast the Typhoons were with the airliner?

I know there are some guidelines for the Luftwaffe like "a fighter needs to be able to get to any point in german airspace within 15 minutes" or so, but that all sounds VERY fast.

Edit: Formatting

<comment><USER11> Interesting that they intercepted after only 15 minutes. Maybe the aircraft had crossed the border without making contact. In the US, airliners accidentally lose contact for close to this time period quite frequently. To reestablish, the controllers will sometimes have a fellow company ship relay a message with the correct frequency via ACARS, or the lost aircraft can listen on 121.5 until someone calls with the correct frequency. I suppose in Europe this would be considered a more critical issue due to the frequent border crossings.

<comment><USER12> It's dependant on the airspace you are flying through and when are you supposed to make contact.

Try flying over NYC out of contact for 15min and lets see how it goes.

<comment><USER11> An airliner at that altitude would be well outside of any city's airspace.

<comment><USER12> I know, there are still different communication requirements for different airspaces.

When you file your travel plans you'll have to check in with different ATCs at different points, you are also likely to be contacted by ATCs depending on the flight plan, conditions and various other factors during the flight.

If you do not check in on schedule e.g. when there is a handover of ATCs or you do not respond to ATCs that trying to make contact with you, there will be a flag and an alert will be likely sent. If this goes on for minutes then they'll assume something was wrong and follow their procedure to make contact and investigate which might involve scrambling jets to make visual contact.

It's not like pilots can be too busy to respond to ATCs yes they might delay a response if they are in a middle of a sentence with a member of the air crew but that's a 5 seconds delay. Even a 1 minute delay would be flagged and you'll likely to be asked why you haven't responded or made contact on time.

</comment></comment></comment></comment></comment><comment><USER13> Quite the LOL at the chatter between the the British Airways pilots filming this saying _I expect they love the opportunity to do this. Any excuse_ and the other one _yeah_.

</comment><comment><USER14> Could anyone comment on why both lower fighters keep to the left side of the airliner? For better visibility, I would have expected them to take it between them. Additionally, the sun seems to be on the left, so they would be hard to make out.

<comment><USER15> It's all standardized interception behaviour. The flight leader is trying establish contact with the plane on the left side, the other jet is just on standby.

Edit: And they picked the left side because that's where the pilot in charge of the plane is supposed to sit.

<comment><USER16> Plus, I would imagine, the interceptors are leaving the right side clear: should an additional _aviation_ emergency develop, the plane is still safe to maneuver in 5 directions (the sixth being blocked by the interceptors).

</comment></comment></comment><comment><USER17> Everything is so beautiful, up there.

</comment><comment><USER18> In the northern hemisphere these warplanes are called "Tornado".

<comment><USER6> Eurofighter Typhoon and Panavia Tornado are different aircraft.

<comment><USER19> It was an obvious joke relating to regional names for weather events.

<comment><USER18> Of course; it doesn't even make sense for Europe to have a "Typhoon" aircraft due to the geographical location.

I guess Europe has some "Penguin" submarines, too.

<comment><USER20> We'll consider renaming the plane when US changes the Apollo program to something that doesn't refer to European heritage.

</comment></comment></comment></comment></comment><comment><USER21> > As a precaution, the German Air Force deployed its aircraft to ensure the > safety of the flight and its guests.

sure, safety of the flight and guests, you mean humanely killed by rocket instead of hitting some target on ground?

<comment><USER12> Making visual contact and making sure that the passengers and the crew are not in distress.

Could've been cabin preassure loss and all of them are passed out all dead just as much as hijacking.

Shooting down can be just as achieved with surface to air missiles.

We send jets to establish contact and use short range and if needed LOS communication to figure out what is going.

<comment><USER22> > cabin preassure loss and all of them are passed out

What could the fighter jets possibly do to "save" the passengers in such a situation?

<comment><USER14> Nothing: [https://en.wikipedia.org/wiki/Helios_Airways_Flight_522](https://en.wikipedia.org/wiki/Helios_Airways_Flight_522)

<comment><USER23> Observe and report.

[https://en.wikipedia.org/wiki/1999_South_Dakota_Learjet_cras...](https://en.wikipedia.org/wiki/1999_South_Dakota_Learjet_crash)

</comment></comment></comment></comment></comment><comment><USER22> > As a precaution, the German Air Force deployed its aircraft to ensure the > safety of the flight and its guests.

Does anyone still believe this bullshit? Clearly there's nothing that the fighter jets could do to ensure the safety of the flight/passengers - the most they could do is to shoot the plane down in a safe area to avoid casualties _on the ground_ , if it were e.g. controlled by terrorists.

And the media/institutions wonder why people don't trust them...

<comment><USER14> It would be illegal in Germany to shoot down a hijacked airliner and ordering the pilot to do so would be an illegitimate order the pilot would be forbidden from following. The most anyone could do would be the Minister of Defence of the Chancellor talking directly to the pilot and telling her that they would take the fall if her conscience allowed them to shoot down the plane. The pilot would still face a trial with uncertain results.

So, please stop with the alternative facts and the conspiracy stuff. Not every country works like the US of A, unbelievably.

<comment><USER24> > It would be illegal in Germany to shoot down a hijacked airliner and > ordering the pilot to do so would be an illegitimate order the pilot would > be forbidden from following

Yeah we're going need a source on that. Don't even think of pulling up that vague Grundgesetz about human dignity, the legality of refusing an order would be established AFTER the fact. To call such an order illegal before any court judgement is the very thing that you are branding as an alternative fact.

<comment><USER14> Decision 1 BvR 357/05 ([https://www.bundesverfassungsgericht.de/SharedDocs/Entscheid...](https://www.bundesverfassungsgericht.de/SharedDocs/Entscheidungen/DE/2006/02/rs20060215_1bvr035705.html)):

>The armed forces' authorisation pursuant to § 14.3 of the Aviation Security Act (Luftsicherheitsgesetz – LuftSiG) to shoot down by the direct use of armed force an aircraft that is intended to be used against human lives __is incompatible with the right to life under Article 2.2 sentence 1 of the Basic Law in conjunction with the guarantee of human dignity under Article 1.1 of the Basic Law __to the extent that it affects persons on board the aircraft who are not participants in the crime.

<comment><USER24> It sounds incredibly dumb and the fact that nobody has challenged and changed this is testament to the pace of public official life. If somebody were to get a nuclear bomb on a civilian plane, the german government would be constitutionally obligated to let them fly to wherever they want. I imagine that in real life the german government would ask France or Great Britain to quickly fly over and do the job for them. Afterwards it's nobody's fault and we can all keep pretending that the Grundgesetz is perfect.</comment></comment></comment></comment></comment>


Fighting spam with BotMaker - <AUTHOR> https://blog.twitter.com/2014/fighting-spam-with-botmaker

<comment><USER2> What I'm missing the most is any concrete information on false positives. They state the want near zero false-positive, which is great, but what's the actual rate? and how is this measured?

It's easy to reduce spam by X% if you don't care about increasing your false- positives rate.

I'm also curious what ways users have to flag false-positives on twitter. It's easy to 'block or report', but is there a spam box to inspect and mark things that were wrongly classified? If there isn't an _easy_ way, then it's going to be much harder to even measure false positive rates, let alone reduce it.

<comment><USER3> I can myself attest to the fact that Twitter has plenty of false positives. I use a custom url shortener url and on several occasions I have been flagged for spam.

What I'd really like to see is the option to submit a report for the false positive. I'm sure the data mining is picking up on not only my initial submission, but also reattempts with altered content to try and bypass their greedy algos.

Needless to say it's a bit of a bummer when you curate a tweet only to be denied sending. I think they have it out for .CO url shorteners :)

<comment><USER4> What reason was given you in the spam flag notification?

<comment><USER3> I haven't had one trigger in a week or two so I can't recall the exact response. I've had it happen on separate accounts using url shorteners as well as when attempting to send DMs containing shortened urls.

If it's of any use, I use Tweetdeck.

</comment></comment></comment></comment><comment><USER5> "Key reduction in a spam metric". Stupid question: if they know what spam is well enough to chart it, then why not just use that to fight it?

<comment><USER6> They're fighting spam in realtime, but they can analyze their accuracy later.

One example: if you have a rule "sending the same URL 100 times in a row is spamming" then you'll _let 99 spams through_ before you identify that they were spams.

</comment></comment><comment><USER7> I was hoping to hear more about what is considered spam. High scores being posted from a game through the API at a high level? What about mass favoriting tweets to get more followers? Or are we just talking about 'Sex pills, free rolexes, getting girls' links to malware sites?

</comment><comment><USER8> Would be interesting if they would have share some more details on the rule engine/framework. How does it compare with drools (apart from it being probably faster).

</comment><comment><USER9> I wonder how these rules came to be known as "bots" within Twitter. It has an nice symmetry to it though: use bots to fight bots.

</comment><comment><USER10> TLDR

Twitter built a DSL for writing spam detection algorithms.

</comment><comment><USER11> To publish 'a how we did it' article like this suggests to me that the Twitter engineering team are either justifiably confident in their spam fighting creation (because they are confident it is infallible), or they are naively supremely over-confident; thus publicizing how this works will come back to bite them in the butt.

Only time will tell I guess. From the article it would appear that they aren't doing anything magically different with regards to classification of twitter- spam, but they have found a way to deal with the volume of classification tasks in a pertinent manner. It also gives them a way to quickly respond to new types of spam attacks.

Very interesting. They should consider opening it up as an Askimet competitor. The difference between a blog comment and a twitter post is negligible.

<comment><USER12>
      > publicizing how this works will come back to bite them in the butt

I didn't see any rules or heuristics published. Merely that they employ a multi-stage filter (as any engineer could imagine), and that they codify their ruleset with a human-readable DSL (which is kind of interesting, but also kind of weird).

      > The difference between a blog comment and a twitter post is negligible.

The difference in textual content between a tweet and blog comments falling within a similar character length are perhaps negligible, though there is often no such length constraint in place for blogs. Comment spam strategies are free to vary string length to optimize for evasion, click through, proliferation, and other criteria. I would also argue there is a different demographic distribution between Twitter and blogs with regard to readership and participation.

That said, Twitter just outlined a number of reasons why their case is special. They are a high-availability, high-volume, low-latency service. Twitter's spam solution was designed to handle their very special set of constraints. I think a multi-stage filter complete with asynchronous post- processing jobs would be a bit much for your average Wordpress blog. People just starting out with (say, PHP) probably can't fathom a multiprocess deployment architecture. Not to say they couldn't, but the journey is a long one.

<comment><USER13> > I think a multi-stage filter complete with asynchronous post-processing jobs > would be a bit much for your average Wordpress blog. People just starting > out with (say, PHP) probably can't fathom a multiprocess deployment > architecture. Not to say they couldn't, but the journey is a long one.

That's why he suggested an Akismet competitor - in case you haven't used it, Akismet is a SAAS solution that filters your comments on their servers, not on your own, so you don't have to worry about the architecture or deployment.

<comment><USER12> My mistake. Thanks for the correction. :)</comment></comment></comment></comment>