What's So Bad About Posix I/O? - <AUTHOR> https://www.nextplatform.com/2017/09/11/whats-bad-posix-io/ <comment><USER3> Tldr

"POSIX I/O is simply not what HPC needs." but it's ok for almost any other use case and "There are many ways to move beyond the constraints imposed by POSIX I/O without rewriting applications, and this is an area of active research in both industry and academia." The problem is the lack of a standard solution yet.

The post does a good job at explaining why HPC doesn't need the strong guarantees provided by POSIX I/O and why they slow down those kind of applications. They are perfectly fine and desirable for most workloads on single user workstations, such as web development (save a file in the editor and the compiler gets its bytes exactly as you wrote them.)

<comment><USER4> I'd have thought MPI-IO is the relevant standard for HPC i/o (perhaps under something like HDF5). I'd expect HPC applications with serious i/o requirements to be using that anyway.

<comment><USER5> It is and people do use it, but it ultimately still exists as software that sits on top of POSIX in the vast majority of cases. It can clean up really bad POSIX I/O by buffering and reordering, but doing so induces latency, memory overheads, and consumes other resources that don't always pay dividends at all scales.

<comment><USER4> ROMIO unfortunately requires global locking on Lustre, but I don't see how that's relevant to the use of standard or other high-level (HDF, NetCDF, ...) non-POSIX interfaces. You can swap in PVFS, or something else with a different architecture. (So what if object stores are on _local_ POSIX-y filesystems?)

I'd have thought the salient feature is collective i/o, not cleaning up silly POSIX i/o of some sort. Nothing's perfect, but what should replace the non- blocking collective MPI-IO under5, for instance? I wonder what are the major NERSC applications that don't do sane i/o and should. The published workload characterization doesn't help.

</comment></comment></comment></comment><comment><USER6> The author brings up a lot of good points, but they're mostly good within the context of HPC. Out in the broader universe where most data lives, many people do require the full suite of POSIX namespace, metadata, and permission semantics. Yes, even locking, no matter how many times we tell them that relying on that in a distributed system is Doing It Wrong. I know because I support their crazy antics at one of the biggest internet companies.

The author's on much firmer ground when he talks about POSIX consistency semantics. While we can't control misguided applications' (ab)use of locks, we can certainly offer models that don't require serializing things that the user never wanted or expected to be serialized, or making them synchronous likewise. As I've also written before[1], we need more ways to describe the user's _desired_ level of ordering and durability, separately, giving the implementation maximum flexibility to optimize I/O while preserving the guarantees that users really care about. The setstream primitive in CCFS[2] is a good example of the sorts of things we need here. I'm not at all convinced that object-store-centric approaches are the way to go here, but some evolution is clearly needed.

[1] [http://pl.atyp.us/2016-05-updating- posix.html](http://pl.atyp.us/2016-05-updating-posix.html) section under "fsync"

[2] [https://www.usenix.org/system/files/conference/fast17/fast17...](https://www.usenix.org/system/files/conference/fast17/fast17_pillai.pdf)

<comment><USER5> The context was definitely HPC, but that's mostly because it's the last place where this problem persists in a very big way. The WORM workloads that serve up most of the data we consume globally just rejected POSIX entirely and created the scalable object interfaces we know and love. It was very obvious that having every Roku in the world mount a Netflix file system over the WAN was insane, so nobody had to write an article calling for the end of that bad behavior.

Updating POSIX, whether it be through more expressive consistency options or just giving middleware developers the knobs they need to implement such mechanisms themselves, has come up before[1], as you say. My worry is that many of the serious efforts from years past were ahead of their time, and hyperscale has gone their own way so the ship has sailed on POSIX I/O.

[1] [http://www.pdl.cmu.edu/posix/](http://www.pdl.cmu.edu/posix/)

</comment></comment><comment><USER7> I'm surprised there is no mention of the lack of non-blocking filesystem reads and opens. In other words, I should be able to submit a read request, and then select()/poll()/epoll() for when the data is available like network I/O.

<comment><USER8> Most unixes (including Linux) always report files to be readable under select()/poll()/epoll() so this doesn't actually help read from disks (even when they're backed by network/shared storage). That's why you often see this kind of seeming "nonsense" in HPC startup scripts:

        for x in *; do cat "$x" & done | pipeline ...

Furthermore, in a networked-environment, select()/poll()/epoll() actually decrease concurrency compared to threads because _now_ you need another system call to go fetch the data from the kernel. If I've got the CPUs I'd rather use threads.

In the Windows model (in contrast), the programmer says "when this data is available (from this socket, file, etc), put it here (into this memory buffer)" and then waits for the kernel to do _something_. As a result, you skip that extra syscall -- but you also skip any other read you might have needed to do for data that's arrived at (roughly) the same time. AIO[1] is potentially similar, but it never seems to benchmark well in my experience.

[1]: [http://man7.org/linux/man- pages/man7/aio.7.html](http://man7.org/linux/man-pages/man7/aio.7.html)

<comment><USER9> > Most unixes (including Linux) always report files to be readable under > select()/poll()/epoll()

There is no other choice really. Let's say you want to read a specific page from a file. No page is currently in the kernel buffers. If poll where to return non-ready (and remember that poll has no idea which pages you are interested in), what action, within posix, would you take to change it?

poll and friends just do not make sense for plain file descriptors. In linux, splice add an interesting twist, but it is still awkward to use.

<comment><USER8> > Let's say you want to read a specific page from a file.

pread() is an unusual case. I'd be happy if poll() simply indicated read() would succeed, and I'm happy doing an lseek() before every read().

However a pread() with O_NONBLOCK could return EAGAIN and poll() would _then_ know what pages I'm interested in.

> what action, within posix, would you take to change it?

The general (POSIX-compatible) strategy (as I indicated) has been to fork a thread off to handle reading so that my application is notified when the data is in the fifo. This works for me when I'm reading (say) five files from a slow (or remote) storage, but it's not ideal since it causes a copy.

<comment><USER9> poll can't guarantee that the next read will succeed because first of all doesn't know how many pages you are interested in reading and any relevant page in the page cache might be evicted between the poll and the read.

Also having pread, which is designed for concurrency and scalability, carry state around for poll (an unbounded amount of state btw: you can have multiple threads issuing concurrent preads on the same fd) makes it completely pointless.

There are linux patches floating to add flags to preadv2 so that on a page not present it returns EAGAIN, so that you can optimistically do a sync read an fall back to a thread pool in the slow case. It still doesn't involve poll and friend.

<comment><USER8> poll() can't guarantee that the next read will succeed anyway, because _something else_ might drain the socket or fifo (e.g. another thread). Programmers simply use O_NONBLOCK anyway and use poll() is wait until the process should try again.

</comment></comment></comment></comment></comment></comment><comment><USER10> Note that the title is a little click-baity. Specifically, this article is about what is wrong with POSIX I/O _in high-performance, exascale computing_. This article is about how the POSIX I/O model breaks down in a pretty extreme use case, not that it's fundamentally broken.

<comment><USER11> the atomic write business is pretty painful in any distributed setting, and only comes into play with concurrent writers with overlapping byte offset regions.

I've always kind of struggled to imagine a case where an application would be correct with this guarantee and faulty otherwise. i guess some sort of cleverly designed structure with fixed length records where a compound mutation could be expressed in a contiguous byte range?

<comment><USER12> Imagine two processes appending rows to a csv file. If writes were non-atomic, they might insert partial rows, or overwrite each other.

<comment><USER11> the append case is actually different, it requires serializing the end of file pointer.

in the write atomicity case we're asking the file system to order all the writes, and make sure the resulting file shows each of them being completed in their entirety before the next is applied, without any interleaving even if they refer to the same byte offsets in the file

</comment></comment></comment></comment><comment><USER13> > A typical application might open() a file, then read() the data from it, > then seek() to a new position, write() some data, then close() the file. > File descriptors are central to this process; one cannot read or write a > file without first open()ing it to get a file descriptor, and the position > where the next read or write will place its data is generated by where the > last read, write, or seek call ended.

pwrite() doesn't have this problem at alll, and my copy of the "POSIX.1-2008 with the 2013 Technical Corrigendum 1 applied" man pages (type "man 3p pwrite" on your local up-to-date Linux box) mentions a function called pwrite(). I'm pretty sure that this makes pwrite() an example of POSIX I/O.

> Because the operating system must keep track of the state of every file > descriptor–that is, every process that wants to read or write–this stateful > model of I/O provided by POSIX becomes a major scalability bottleneck as > millions or billions of processes try to perform I/O on the same file > system.

This makes no sense. If you have millions or billions of processes on one machine trying to perform I/O on the same file system, you probably have scalability problems, and those problems have nothing whatsoever to do with POSIX I/O.

If, on the other hand, you have millions or billions of processes performing I/O to the same _networked_ filesystem, then you certainly need to think carefully about scaling, and POSIX semantics may well get in the way, but the problem you face here has nothing to do with the fact that POSIX has file descriptors. This is because file descriptors are local to each machine, and they very much don't synchronize their offsets with each other.

(Unless you use O_APPEND. Having millions of processes O_APPENDing to the same file is _nuts_ , unless your filesystem is designed for this, in which case it's probably just fine.)

<comment><USER14> _type "man 3p pwrite" on your local up-to-date Linux box)_

Or, of course, type "man 3p write" into Google, there are many man-page sites.

I typically use die.net [1] which is among my top search hits. It could do with some cleaning of the web versions, but it's fine.

[1] [https://linux.die.net/man/3/pwrite](https://linux.die.net/man/3/pwrite)

</comment></comment><comment><USER15> > the guarantee that data has been committed to somewhere durable when a > write() call returns without an error is a semantic aspect of the POSIX > write() API

No, that's just flat out wrong.

> POSIX I/O is stateful

This is fundamental to the authorization model. Authorization happens at file open time. It's also what enables the stream abstraction.

The title of the article is really bold. POSIX I/O solves a common problem just fine (it's not perfect, but not for the reasons given in the article, and we don't know how to do it much better). I don't know anything about the domain the author talks about (HPC), but it seems what he needs is basically direct access to the block device. Or writing away through network sockets / using a database.

<comment><USER16> >> the guarantee that data has been committed to somewhere durable when a write() call returns without an error is a semantic aspect of the POSIX write() API

> No, that's just flat out wrong.

from "man 3p write":

           After a write() to a regular file has successfully returned:

            *  Any successful read() from each byte position in the file that  was
               modified  by  that  write  shall  return  the data specified by the
               write() for that position until such byte positions are again modi‐
               fied.

            *  Any  subsequent successful write() to the same byte position in the
               file shall overwrite that file data.

>> POSIX I/O is stateful

> This is fundamental to the authorization model. Authorization happens at > file open time. It's also what enables the stream abstraction.

This is very true, but in the workloads the author is talking about, there are often times that a stateless API would enable a more efficient implementation. Think about what is going on in your file server when you have 100k clients all accessing the same open file.

> . I don't know anything about the domain the author talks about (HPC), but > it seems what he needs is basically direct access to the block device. Or > writing away through network sockets / using a database.

The author is talking about (possibly distributed) networked filesystems backing clusters with extreme levels of parallelism (minimum 100s of nodes with 10s of processors on each node, and it gets much bigger). As far as "using a database" that falls under the category of a user-space I/O stack, where the (userspace) database is proxying the I/O to reduce state.

The title of the article isn't at all bold in context, because it is well accepted in HPC that POSIX I/O is the bottleneck for certain types of loads, and the author is clarifying to those not familiar with the details why this is true.

<comment><USER17> For this ...

        Any successful read() from each byte position in the file that  was
        modified  by  that  write  shall  return  the data specified by the write()

... perhaps I can re-open() and re-read() the same byte value written by another process for the same file, but the file contents may not have been fully flushed all the way to disk. The file contents may be "durable" across processes on the same running OS that mount the same filesystem ... but if the OS happens to die before the data is flushed, then perhaps after reboot the open()/read() will return an older value previously written.

The semantics of "durability" are a squishy concept.

<comment><USER16> yes, the term "durable" was perhaps a poor choice of words, but the paragraph that followed made it clear that they were aware of the specific requirements (specifically mentioning making dirty caches available to all processes)

<comment><USER18> _Durable_ has a very specific meaning for I/O and is just not correct here.

POSIX does intentionally not specify any durability at all (e.g. a no-op fsync is explicitly permitted).

<comment><USER19> And in practice it is close to that, due to cheating hardware with a cache.

</comment></comment></comment></comment></comment></comment><comment><USER20> I've spent most of my career specifically dealing with HPC I/O from a systems and applications perspective. I worked in academia for a while, then in the DOE (almost indistinguishable from academia except worse bureaucracy; shocking, I know), and finally in private industry.

I don't think I've ever read a more spot-on description of the problem. Once I left the DOE, I realized that the problem was far more acute than I had thought. At least on the real big systems, most of our work was with a small number of research groups basically doing the same workflow: start your job, read in some data, crunch, every 15 minutes or something slam the entire contents of system memory (100s of TB) out to spinning disk, crunch, slam, your job gets killed when your time slice expires, get scheduled again, load up the last checkpoint, rinse, repeat. Because of the sheer amount of data, it's an interesting problem, but you could generally work with the researchers to impose good I/O behavior that gets around the POSIX constraints peculiarities of the particular filesystems. You want 100,000,000 cpu hours on a $200M computer? You can do the work to make the filesystem writes easier on the system.

Coming into private industry was a real eye-opener. You're in-house staff and you don't get to say who can use the computer. People use the filesystems for IPC, store 100M files of 200B each, read() and write() terabytes of data 1B at a time, you name it. If I had $100 for every job in which I saw 10,000 cores running a stat() in a while loop waiting for some data to get written to it by one process that had long since died, I'd be retired on a beach somewhere.

The problem with POSIX I/O is that it's so, so easy and it almost always works when you expect it to. GPFS (what I'm most familiar with) is amazing at enforcing the consistency. I've seen parallel filesystems and disk break in every imaginable way and in a lot of ways that aren't, but I've _never_ seen GPFS present data inconsistently across time where some write call was finished and it's data didn't show up to a read() started after the write got its lock or a situation where some process opened a file after the unlink was acknowledged. For a developer who hasn't ever worked with parallel computing and whose boss just wants them to make it work, the filesystems is an amazing tool. I honestly can't blame a developer who makes it work for 1000 cores and then gets upset with me when it blows up at 1500. I get grouchy with them, but I don't blame them. (There's a difference!)

But as the filesystems get bigger, the amount of work the filesystems have to do to maintain that consistency isn't scaling. The amount of lock traffic flying back and forth between all the nodes is a lot of complexity to keep up with, and if you have the tiniest issue with your network even on some edge somewhere, you're going to have a really unpleasant day.

One of the things that GCE and AWS have done so well is to just abandon the concept of the shared POSIX filesystem, _and_ produce good tooling to help people deal with the IPC and workflow data processing without it. It's a hell of a lot of work to go from an on-site HPC environment to GCE though. There's a ton of money to be made for someone who can make that transition easier and cheaper (if you've got it figured out, you know, call me. I want to on it!), but people have sunk so much money into their parallel filesystems and disk that it's a tough ask for the C-suite. Hypothetically speaking, someone I know really well who's a lot like me was recently leading a project to do exactly this that got shut down basically because they couldn't prove it would be cheaper in 3 years.

</comment><comment><USER12> It seems like the OP really wants to use a database instead of a file system.

<comment><USER20> I don't think so; HPC is a different beast There are very few if any organizations on earth that have a database that can write a sustained 1TByte/second+ from as few as 300-400 node to 100s of PB of stateful media for years on end.

Ken Batcher (maybe of OSU?) wrote a quote that the rest of us have been using for years: "Supercomputers are a tool for converting a CPU-bound problem into a HPC-bound problem."

The filesystems start to look more like databases over time, but it's not like they can throw down a nice Cassandra cluster and have it pick up the slack. I'm not saying it will never happen, but I don't think it's am option at the moment.

<comment><USER20> Rather "into an I/O-bound problem". Apologies for ruining Ken's great joke.

</comment></comment></comment><comment><USER21> Now working on my second parallel file system, I am still amazed how the (POSIX) file system interface has stood the test of time and has been reimplemented in so diverse forms, from local file systems to file systems that are essentially modern large-scale fault-tolerant distributed systems. Completely different languages, platforms, technologies.

Sure, it is not easy to implement a file system and the details are subtle, but the abstraction gives you the necessary freedom to build very scalable and high performance systems and at the same time provides applications with a well-defined set of mechanisms to solve persistence.

It might be the most stable and versatile interface that we have in computing?

</comment><comment><USER22> Basic message: big parallel I/O systems need to shift responsibility for maintaining file state, moving away from the operating system and towards the application.

POSIX API has pretty simple view of file state with open(), read(), write(), and close(), but this interface does not scale well concurrency-wise.

<comment><USER6> That part scales fine. There are other problems with the POSIX open/read/write model, such as being too strict wrt consistency and too loose wrt ordering and durability, but it _scales_ OK. It's the metadata/namespace operations that are hard to scale. I've been working on distributed filesystems for a long time. If all I had to deal with was reads and writes I'd be a happy (OK, less angry) man.

<comment><USER22> I trust your experience over my own in this area; just reading this piece with curiosity. That makes sense, metadata and namespace operations would be difficult when distributed. But isn't the graph in the article specifically about file handle state maintained by the OS? I guess the graph is weird though it wants to be logarithmic but the power of the series keeps decreasing...what type of graph is that?

</comment></comment></comment><comment><USER23> The POSIX aio functions. [http://man7.org/linux/man- pages/man7/aio.7.html](http://man7.org/linux/man-pages/man7/aio.7.html)

</comment><comment><USER24> What's so bad about POSIX IO? People not knowing that async write APIs exist. Additionally, that mmap exists to bypass all the atomicity guarantees. Both are in POSIX.

</comment><comment><USER25> Some of the claims in this article are total nonsense.

The article points out that file descriptors are stateful - they have a current offset associated with them, and I/O syscalls update that offset - then claims that this "stateful model" could be a "scalability bottleneck" with "billions of processes" in parallel filesystems. Except that each of those billions of processes will have its own file descriptor with its own offset! The only case where this could cause contention is if multiple threads in a single process are trying to read from the _same_ file descriptor, which would be a really bad idea in the first place, precisely because file descriptors are stateful. Just open multiple descriptors - or use pread/pwrite, which have existed for a long time. Perhaps the process-wide statefulness of many POSIX APIs is a bad design in a world with threads, but it has nothing to do with the concurrent-file-open benchmark in the article, or really any other performance problems with parallel filesystems.

Anyway, file descriptors are just per-process handles used for communicating with the kernel. At least in principle, there's no reason that remote filesystems should know or care about file descriptors on the client end, unless clients are using file locks (well, except those aren't file- descriptor-based anyway, although they should be).

Later, the article claims:

> While the POSIX style of metadata certainly works, it is very prescriptive > and inflexible; for example, the ownership and access permissions for files > are often identical within directories containing scientific data (for > example, file-per-process checkpoints), but POSIX file systems must track > each of these files independently.

[..]

> Supporting the prescriptive POSIX metadata schema at extreme scales is a > difficult endeavor; anyone who has tried to ls -l on a directory containing > a million files can attest to this.

POSIX access bits are literally 15 bits per file. uid and gid are a few bytes. The overhead of storing these for each file definitely isn't what's making ls -l slow. Perhaps there's some system where time spent _checking_ them all, at access time, is a bottleneck, but I'd be very surprised if modern Linux was such a system; that kind of problem sounds easy to solve with some basic caching.

The article calls out created and modified times as another part of the metadata, thereby amusingly missing the only one of the three POSIX file timestamps - access time - that actually can cause big scalability issues (if left enabled).

Also, apparently the author has not heard of either ACLs, which are in POSIX, or xattrs, which are pseudo-POSIX (multiple systems have roughly compatible implementations based on an old POSIX draft) - both of which try to improve flexibility compared to classic POSIX metadata. There are problems with both of them, but you'd think they'd at least deserve a mention in the list of alternatives.

<comment><USER6> > Except that each of those billions of processes will have its own file > descriptor with its own offset!

The offset is only a tiny part of how POSIX is stateful. The very fact that each read or write is associated with a particular fd, therefore with a particular authorization and lock context, is more of an issue at the servers. Even more of an issue is the possibility of still-buffered writes, which POSIX does require be visible to reads on other fds.

> At least in principle, there's no reason that remote filesystems should know > or care about file descriptors

Untrue, and please don't try to "correct" others with your own inaccurate information. As I just said, each file descriptor (or file handle in NFS) has its own authorization and lock context, which must be enforced at the server(s) so knowledge of them can't be limited to the client.

> POSIX access bits are literally 15 bits per file. uid and gid are a few > bytes.

Also mtime and atime, and xattrs which can add up to kilobytes, but more importantly what the author was really talking about was _namespace_ information rather than per-file metadata. It's a common mistake. Even as someone who writes code to handle both of these separate concerns, I'm not enough of a pedant to whine every time an application programmer gets my domain's terminology wrong.

> the only one of the three POSIX file timestamps - access time - <USER2> > actually can cause big scalability issues (if left enabled).

Untrue yet again. Mtime can be a problem too, as can st_size and st_blocks. In an architecture where clients issue individual possibly-extending writes directly to one of several data servers for a file but other clients can then query these values through a separate metadata server, that creates a serious aggregation problem. That's why I think the separate ODS/MDS model (as in Lustre) sucks. People resort to it because it makes the namespace issue easier, but it makes metadata issues harder. In the particular use cases where people have to stick with a filesystem instead of switching to an object store, it's a net loss.

</comment></comment><comment><USER26> Relevant quotes from [http://github.com/globalcitizen/taoup](http://github.com/globalcitizen/taoup) ...

 _Optimization considered harmful: In particular, optimization introduces complexity, and as well as introducing tighter coupling between components and layers._ \- RFC3439

 _Design up front for reuse is, in essence, premature optimization._ \- @AnimalMuppet

 _To speed up an I /O-bound program, begin by accounting for all I/O. Eliminate that which is unnecessary or redundant, and make the remaining as fast as possible._ \- David Martin

 _The fastest I /O is no I/O._ \- Nils-Peter Nelson

 _The cheapest, fastest and most reliable components of a system are those that aren 't there._ \- Gordon Bell

 _Safety first. In allocating resources, strive to avoid disaster rather than to attain an optimum. Many years of experience with virtual memory, networks, disk allocation, database layout, and other resource allocation problems has made it clear that a general-purpose system cannot optimize the use of resources._ \- Butler W. Lampson (1983)

 _Crowley 's 4th rule of distributed systems design: Failure is expected. The only guaranteed way to detect failure in a distributed system is to simply decide you have waited 'too long'. This naturally means that cancellation is first-class. Some layer of the system (perhaps plumbed through to the user) will need to decide it has waited too long and cancel the interaction. Cancelling is only about reestablishing local state and reclaiming local resources - there is no way to reliably propagate that cancellation through the system. It can sometimes be useful to have a low-cost, unreliable way to attempt to propagate cancellation as a performance optimization._

 _Optimization: Prototype before polishing. Get it working before you optimize it._ \- Eric S. Raymond, The Art of Unix Programming (2003)

 _Before optimizing, using a profiler._ \- Mike Morton

 _Spell create with an 'e'._ \- Ken Thompson (referring to design regrets on the UNIX creat(2) system call and the fallacy of premature optimization)

 _The No Free Lunch theorem: Any two optimization algorithms are equivalent when their performance is averaged across all possible problems (if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems)._

 _An efficient program is an exercise in logical brinkmanship._ \- Edsger Dijkstra

 _Choose portability [high level] over efficiency [low-level]._ \- Mike Garcanz: The Unix Philosophy

 _Laziness is the mother of efficiency._ \- Marian Propp

 _Jevons Paradox: As technology progresses, the increase in efficiency with which a resource is used tends to increase (rather than decrease) the rate of consumption of that resource._

 _It brings everything to a certainty, which before floated in the mind indefinitely._ \- Samuel Johnson, on counting

... and the kicker...

 _Those who don 't understand Unix are condemned to reinvent it, poorly._ \- Henry Spencer

<comment><USER27> And this is how we get incredibly inefficient programs like atom.</comment></comment>


Time-Tracking via a Real Device - <AUTHOR> http://www.gettiller.com/ <comment><USER2> For programmers, it can be automated to not need a manual physical timer on your desk.

For ex: [https://wakatime.com](https://wakatime.com)</comment>


Ask HN: What OSS has the best written (clean) code you have seen? - <AUTHOR>

With some topics [1] talking about how bad the OpenSSL code is to read, I was wondering what would be an OSS project that is a good contrast to it.<p>I'm looking for code that you think is enjoyable to read/easy-on-your-eyes that's a major part of open software.<p>[1]: https://news.ycombinator.com/item?id=7556407 <comment> Sean-Der I have found the code from the suckless project, to be a great read. When I was working on a toy HTTP server I learned a lot from [http://git.suckless.org/quark/](http://git.suckless.org/quark/)

Another code base I enjoy reading is SBCL. When I M-. into functions and it takes me to something defined in SBCL it always is a great read, and I have learned a lot from it.

FreeBSD is a great one as well, if you checkout SRC it is really illuminating to browse around the source. I have learned a lot by seeing the layout of such a large project.

</comment><comment><USER2> i wish that there wasn't a double standard where open source is expected to be well-documented or elegant (etc). (especially for smaller scale projects that don't have the infrastructure / time)

for OSS projects that are hard(er) to read, the question goes - why hasn't the community helped clean it up / document / etc?</comment>


Oregon Shakespeare Festival commissions translation of plays to modern English - <AUTHOR> http://www.wsj.com/articles/a-facelift-for-shakespeare-1443194924 <comment><AUTHOR> John McWhorter, the Columbia linguist who pinned the article, makes a conniving argument.

However, the counterargument is that Shakespeare plays are just that - plays, which are meant to be experienced and watched, and not read. The fact that Shakespeare's prose are full of alliterative sound echoes and half-rhymes means that translations to modern English will likely see nuances rid of or at least diminished. A good band of actors can conveys Shakespeare effective beyond just the texts: via delivery, body motions, intonation, and all of this aids in the audience understanding and appreciating his plays.</comment>


Better sleep with sunset emulation and relaxing light - <AUTHOR> http://blog.svet.io/post/128705636514/better-sleep-with-sunset-emulation-and-relaxing <comment><USER2> This is a blatant PR post on SVET's own corporate blog, however, I am up- voting this because I have considered building a similar system myself using RGB arrays.</comment>


Georgia election server showed signs of tampering, expert says - <AUTHOR> https://www.nbcnews.com/tech/security/georgia-election-server-showed-signs-tampering-expert-says-n1117441 <comment><USER4> I have been voting in GA since 2008 and there hasn't been a paper trail since at least then. The voting systems they still use today were purchased and put into service in, I believe, 2002. There is literally no reason not to also have a paper trail.

I have no hard data, and this is from 2017/2018, but it is very sketchy when the secretary of state, the office which oversees elections in GA, runs for governor, doesn't resign ahead of time, doesn't allow any investigations into the electronic voting systems, doesn't push forward any paper trail requirements, and subsequently wins the election for governor. Everything Kemp did and didn't do was suspect as it relates to elections.

I have zero faith in GA election results. And until the State moves off of these electronic systems and/or installs a paper trail, and brings a sense of integrity back into the election system, I will continue to feel the elections in GA are without integrity and potentially illegitimate, no matter who wins in any given election. This is outside of any political affiliation. This is the core of our country not being properly governed and protected.

EDIT>> After some research, it looks as if GA is getting a paper trail after a court order required the state to move off the previous electronic system. Now it looks like they're installing BMDs which can be verified by the voter prior to scanning to commit their vote. Of course, there are plenty of articles relating to how hackable BMDs are as well.

<comment><AUTHOR> _> There is literally no reason not to also have a paper trail._

There is _one_ reason, which is that if you give the voter a paper record stating who they voted for, that voter can be shaken down upon leaving their polling place by goons hired by one political machine or another demanding to see that record. That sort of thing used to happen all the time back in the heyday of machine politics, as the dominant machine would very much want to know if you actually voted for their candidates or not -- particularly if they'd paid you before election day to do so.

<comment><USER5> Why would the voter remove the paper ballot or paper receipt from the polling place?

The paper would be kept by poll workers so the electronic count can be audited.

My state has paper ballots that are counted electronically. I feel this is a good compromise and allows much easier vote total auditing.

<comment><USER6> > My state has paper ballots that are counted electronically

This is really not much better.

<comment><USER7> Paper ballots counted electronically allows for a hand count. In case of close elections (as measured by the electronics) a hand count is usually required. You can (and some jurisdictions will) also do a hand count of some number of ballots to confirm the electronic count is at least plausible.

Given the number of questions voters are voting on (at least in CA and WA, there's usually 30-50 things to vote on), hand counting everything would be immensely time consuming.

</comment></comment></comment></comment></comment><comment><USER8> I'm really glad Germany's supreme court (BVerfG) banned electronic voting ([0]) with the common sense reasoning that elections have to be understood by everyone, not just computer experts.

[0] [https://www.ccc.de/de/updates/2009/wahlcomputer-urteil- bverf...](https://www.ccc.de/de/updates/2009/wahlcomputer-urteil-bverfg)

<comment><USER9> I wish Canada would adopt some of that common sense.

<comment><USER10> ....we use paper ballots in Canada. What are you referring to?

<comment><USER9> When I last voted (most recent federal election), my ballot was fed through an electronic reader of some sort on its way into the ballot box, no doubt an electronic vote counter.

<comment><USER10> Huh, interesting. I'm in Montreal and we dropped them into boxes through a slot, same as before.

I wonder if you had a pilot? Afaik, elections canada runs things the same way all around the country.

</comment></comment></comment></comment></comment><comment><USER11> Deleted logs and a server wiped clean days after a lawsuit was filed by election integrity activists, that is terrifying.

If we can't agree on a bipartisan basis that this isn't acceptable, what does that mean for our future?

<comment><USER12> Means certain states like Georgia won't be democratic. What the handful of people in power say goes. Doesn't matter what the voters actually try to vote.

<comment><USER13> Good case for the electoral college, localize the corruption to the state- level.

<comment><USER11> The electoral college is the sole reason 2 out of the last 3 presidents lost the popular vote while winning the general election.

In no universe does the electoral college mitigate corruption on the federal level. In fact, one could argue that it makes corruption worse, because a candidate only has to win a few key states to tip the election, regardless of the popular vote.

<comment><USER14> _it makes corruption worse, because a candidate only has to win a few key states to tip the election, regardless of the popular vote._

The alternative is only counting the popular vote, which only serves to disenfranchise the majority of states.

Without the electoral college, a candidate would only have to campaign in and win three or four large states, and the rest of the nations and its concerns can be ignored.

That concern is the whole reason we have the House/Senate system: So that large states with lots of people in the House of Representatives don't run amok. The Senate balances that out.

<comment><USER15> > The alternative is only counting the popular vote, which only serves to > disenfranchise the majority of states.

The majority of red states are also the majority of the population, so this doesn't check out. (Trump states ~180mm to Hillary's ~140mm.)

<comment><USER16> This is false.

[https://www.cnn.com/election/2016/results](https://www.cnn.com/election/2016/results)

>Trump 46.4% votes 62,984,825

>Clinton 48.5% votes 65,853,5165

</comment></comment></comment></comment></comment></comment></comment><comment><USER17> Electronic voting, in addition to security concerns, isn't even a user experience improvement.

I remember working at a polling station as a teenager during the 2004 election. One elderly lady needed help understanding how the machines worked, so I showed her how you tapped the buttons on the screen.

Well, this woman had lost a lot of fine motor control. After selecting her choice for president, she went to hit the Next button. Her hand shook so badly that she skipped about five screens worth of races.

I asked her if she's like my help going back to choose candidates for all of the races she accidentally skipped. "No," she replied, "I got the important one."

I'm thankful that this experience, and many others from working the polls, eroded my faith in the system early on in life.

<comment><USER18> Not to mention the negative impact on parallelization! Hundreds of paper ballot station can easily be setup in a room, but setting up an additional voting machine requires a ton of other things (budgets, purchasing, techs, etc).

</comment></comment><comment><USER19> A question to Americans, is there nothing a common citizen can do about it? In India, in a similar situation anyone would be able to file a Public Interest Litigation (PIL) with High Court (highest court at state level) or Supreme Court (highest court at central/federal level). If any of the courts decided to take up the PIL, they could ask any investigative body, including law enforcement or ex-judges, for investigation.

<comment><USER20> Try to make electronic voting illegal? Not sure how though - call your representative, raise awareness, that sort of thing

<comment><USER21> Phone calls and visits to your representatives on all levels are taken very seriously by their offices

</comment></comment></comment><comment><USER22> Is there a term for the increased credulousness with which people sometimes seem to approach increasingly, obviously problematic topics? The more blatant the irregularities in this race, where a black woman had a serious chance of becoming governor, the less seriously everyone (including her opponent, _who oversaw the election in his official capacity_ ) took them.

I mention race because it is clearly a likely central motivation for malfeasance in this case, an election for a state with a history of disenfranchisement of its large black population.

</comment><comment><USER23> Here's the actual declaration from PACER, which has some interesting details:

[https://tmp.tonyb.xyz/lamb-decl-pacer.pdf](https://tmp.tonyb.xyz/lamb-decl- pacer.pdf)

<comment><USER24> Could you please post this somewhere else too, site is blocked for me here.

<comment><USER23> That's strange that my site is blocked, is it an overly broad rule, like "lol, if it's not .com or .gov, it's not a website that upstanding gentlemen go to"?

Anyway, here's something:

[https://tonybox.net/tmp/lamb-decl-pacer.pdf](https://tonybox.net/tmp/lamb- decl-pacer.pdf)

[https://moveything.com/tmp/lamb-decl- pacer.pdf](https://moveything.com/tmp/lamb-decl-pacer.pdf)

<comment><USER24> Thank you very much, the message had no details, sorry, just that it was blocked. I assume overly broad rule not liking xyz domain.

</comment></comment></comment></comment><comment><USER25> I think it's safe to assume that any election that's done with software has been hacked unless proven otherwise. Of course there was foul play. Election fraud is currently a huge issue with a ton of players competing to be the ones rigging elections. When a secure, proven technology like mailed paper ballots exists, I question anyone who pushes computer solutions and immediately suspect them of wanting to commit fraud or incredible stupidity and incompetence. After all, if someone suggested to replace armored cars carrying money with unarmored convertibles without a top, what other conclusion can one draw?

<comment><USER26> Isn't that all speculation? I'm not aware of any verified election fraud, in the US. It's like Halloween candy, which everybody knows isn't safe but there are never any verified cases of a problem.

<comment><USER27> If you are not aware, then it is because you have not looked.

[https://www.heritage.org/voterfraud](https://www.heritage.org/voterfraud)

<comment><USER28> The parent poster was referring to election fraud, not voter fraud. They're different. Voter fraud is when someone casts a vote they should not have, e.g. because they moved or they've cast multiple votes. Election fraud is when election officials tamper with the election.

There are documented cases of both, but the Heritage website, documenting about 1,200 over a period of at least 20 years (I didn't check every state, but the earliest I saw was 2000) is an insignificant amount of voter fraud over 20 years and hundreds of millions of votes cast. Even if they underreported by a factor of 100, that's no threat to democracy.

<comment><USER27> Ah, election fraud like illegal voter suppression? Like [https://en.wikipedia.org/wiki/Florida_Central_Voter_File](https://en.wikipedia.org/wiki/Florida_Central_Voter_File)?

Or how about tampering with ballots after they have been cast like [http://nymag.com/intelligencer/2019/02/election-fraud-is- rea...](http://nymag.com/intelligencer/2019/02/election-fraud-is-real-and-it- involves-a-republican.html)?

(Of course if you go back a little ways in history, the most egregious was probably Tammany Hall. Election fraud has a rather long history.)

</comment></comment></comment></comment></comment><comment><USER29> This story seems to now invalidate the line that we have been hearing for about 3 years now that says that no evidence of tampering with the 2016 national election has been found. Of course, they were careful to not say that evidence of no tampering had been found, or even that anyone behind the no- evidence assertion was sufficiently skilled to find and recognize tampering. If no one can vouch for the correctness of the Georgia results now, what evidence is there of no error in the national results; how many other states might have had similarly overlooked anomalies?

</comment><comment><USER30> Because, electronic voting is still a bad idea: [https://www.youtube.com/watch?v=LkH2r-sNjQs](https://www.youtube.com/watch?v=LkH2r-sNjQs)

</comment><comment><USER31> Highly recommend following election security writer Jenny Cohn for more on this topic. GA is a highly egregious recent example, but the entire industry is a mess and very few are paying attention.

[https://twitter.com/jennycohn1](https://twitter.com/jennycohn1)

<comment><USER24> Also Kim Zetter, she's been reporting on this with a greater level of technical expertise and less activism for at least a dozen years. In fact she was the one that scooped this last night [https://twitter.com/KimZetter/status/1046429196182941696](https://twitter.com/KimZetter/status/1046429196182941696)

</comment></comment><comment><USER32> The benefit of extra speed in ballot count does not completely offset the risks of electronic voting. Not only the risks of massive tampering in electronic ballots and servers, but also the lack of clarity on how the process works under the hood for the median voter.

</comment><comment><USER33> for USA folks, you can in each state request a paper ballot. visit your State's voting details page at heir statename.gov site to find out how to request one.

</comment><comment><USER34> Oregon gets voting pretty right. Do it by mail, automatic registration, and of course it's all paper.

There are some risks about not having the privacy of a voting booth, but in general it seems to work pretty well.

<comment><USER35> Voting by mail is not at all secure. For example, a post-office worker could easily skew the results by throwing away letters from a partisan district.

<comment><USER28> Oregon has had vote by mail for at least 20 years and there is almost no irregularity in the process, either for voters or election officials.

In the 2016 election, out of 2 million votes cast, they found something like 54 cases of voter fraud, mostly people voting after they'd moved out of their district.

Election fraud in Oregon is equally rare -- the big one in 2016 was a Republican volunteer in Clackamas County attempting to toss out a box of dem voter ballots. I don't remember the specifics but I think they were caught almost immediately.

The security thing is absolutely a red herring to discourage more states from adopting vote-by-mail, pushed by anyone who benefits from low turnout, because vote-by-mail significantly improves turnout without threatening election security in any meaningful way.

<comment><USER35> This is such a non-sensical argument. Obviously nobody gets convicted for voter fraud if the system cannot detect it.

<comment><USER28> They do detect it and have found 50-some cases in the 2016 election alone.

It's not a problem.

</comment></comment></comment></comment></comment><comment><USER24> two source articles and thread by original journalist on this - <USER2> exploited (ha, remember that one?) and Drupalgeddon was left unpatched - <USER3> is clown-car stuff

[https://threadreaderapp.com/thread/1218055746534817792.html](https://threadreaderapp.com/thread/1218055746534817792.html)

</comment><comment><USER13> *Security expert for the plaintiffs in the case

<comment><USER36> And the only one outside of the FBI with access to images of the server and the non-incriminating ability to comment.

</comment></comment><comment><USER37> [https://xkcd.com/2030/](https://xkcd.com/2030/)

[Summary to remind you: "Don't trust voting software and don't listen to anyone who tells you it's safe.]

</comment><comment><USER6> It is beyond me how naive implementations of electronic voting can be trusted so easily by individuals.

</comment><comment><USER21> $1 says that hack originated from within the state of Georgia, perhaps routed through Russian servers and back, but domestic nonetheless</comment>


The Midas List 2018: Top Tech Investors in the World - <AUTHOR> https://www.forbes.com/midas/list/ <comment><USER2> The list misses Koos Bekker which would hands down be on top for turning the initial $30m investment in Tencent into > $160bn.</comment>


That can be your next Tweet - <AUTHOR> http://yes.thatcan.be/my/next/tweet/

<comment><USER2> Neat idea. I'd recommend trying to ignore Foursquare checkins in the users' tweetstreams.

</comment><comment><USER3> that was more than useless. Its like having one of those disorders that makes you say random words. I would rebrand it "Your tweets after two weeks on meth"</comment>


Show HN: Catalan Election results with D3.js - <AUTHOR> https://politibot.io/asi-ha-votado-cataluna/ <comment><USER2> That is a cool visualisation! One thing is that the page is in spanish (thanks google translate), I would assume not everyone will understand it.</comment>


Standard Cognition raises $40M to replace retailers' cashiers with cameras - <AUTHOR> https://techcrunch.com/2018/11/15/standard-cognition/ <comment><USER2> Cool tech. How exactly does this "humanize retail" as they claim? Does "removing conveyor belts" really humanize the process if the cashier behind the conveyer belt is removed too?

I'm generally a big skeptic of company missions (which this 'tagline' appears to be) in the first place, but completely non-sensical ones like this rub me raw. What's wrong with the mission/tagline being "reducing costs and improving shopper experience"? I'm sure their customers (retailers) would still love the product

/rant</comment>


Ask HN: What to read after "Thinking in Systems"? - <AUTHOR> I recently finished thinking in systems and i want to learn more.So i am looking for book reccomendation.If possible any books related to programming and math and systems thinking. <comment><USER3> Take a look through Brian Castellani's Map of the Complexity Sciences (2018):

[https://www.art-sciencefactory.com/complexity- map_feb09.html](https://www.art-sciencefactory.com/complexity-map_feb09.html)

<comment><AUTHOR> the map looks magnificent.

</comment></comment><comment><USER4> The Fifth Discipline by Senge. Also a great introduction to systems thinking.

Also everything written by Goldratt.

In particular: The Goal and It's not luck Both are business novels, they read lightly but the lessons are profound.

<comment><USER5> The Goal is both insightful and hilarious.

The system-oriented lessons/examples are great. The 80s style dialogues, integrations, and throwbacks are fun sideshows.

</comment></comment><comment><USER6> Depends, do you have ~$500 to spare?

I ask because the single best book on cybernetics (and I mean it, it feels a tad bit absurd how much better than other works on the topic it seems to me) costs roughly that, isn't available on Libgen, and nobody dares mention it anywhere in fear of driving prices on used copies up, because nobody wants the publisher to get any money for it, cause while they sure charge a lot for it, they actively hampered efforts by the original authors for a new edition of the book. (The rights to it eventually did end up in the right hands - <USER2> decade too late.)

If you wanna know why we don't see widespread adaption of systems thinking, it seems almost singularly caused by the nega-butterfly effect of that book not sitting on Libgen & having a price tag on it so gigantic people don't even wanna talk about it.

So ironic, makes me want to puke.

<comment><USER7> Are you talking about the 2nd edition of "Design for a Brain"?

There are embarassingly many cybernetics books this could refer to (particularly from the midcentury)...

<comment><USER8> Probably not: [https://openlibrary.org/books/OL5791995M/Design_for_a_brain](https://openlibrary.org/books/OL5791995M/Design_for_a_brain)

<comment><USER6> [https://news.ycombinator.com/item?id=20229045](https://news.ycombinator.com/item?id=20229045)

</comment></comment></comment></comment><comment><USER9> Hey I just bought this book, at the back she gives a great suggested reading list, I encourage you to take a look and pick one that sparks your interest :).

<comment><AUTHOR> yes I have seen that but as said in question I am looking some mathy and programming of systems,if not that then something different according to the reader liked.

<comment><USER10> Sterman's "Business Dynamics" may be of interest to you, but it's pricey. You may be able to find a PDF online. Morecroft's "Strategic Modelling and Business Dynamics" is available on Safari Books Online if you have a subscription.

<comment><AUTHOR> both books look good.Thanks

</comment></comment></comment></comment><comment><USER7> I'll do you one better, and give you a whole bibliography: [http://pcp.vub.ac.be/CSBOOKS.html](http://pcp.vub.ac.be/CSBOOKS.html)

Some of the books on that list aren't readily available, because as cybernetics went out of style they went out of print.

</comment><comment><USER3> New Books in Systems and Cybernetics podcast is also a good resouce.

[https://newbooksnetwork.com/category/systems-and- cybernetics...](https://newbooksnetwork.com/category/systems-and-cybernetics/)

</comment><comment><USER11> "The Origin of Wealth" is an excellent take on macroeconomics from a complex adaptive systems view.

<comment><USER11> Also recommend this course by Scott Page from the Santa Fe Institute: [https://www.thegreatcourses.com/courses/understanding- comple...](https://www.thegreatcourses.com/courses/understanding- complexity.html)

<comment><USER3> SFI generally is an excellent source of complexity science research and writing.

[https://www.santafe.edu](https://www.santafe.edu)

</comment></comment></comment><comment><USER12> The Model Thinker is excellent.</comment>


New Research on Overcoming Loneliness - <AUTHOR> http://wsj.com/articles/new-research-on-overcoming-loneliness-1442854148 <comment><USER2> Old research on loneliness: E. Fromm, _The Art of Loving_. To paraphrase:

(1) For humans the fundamental problem in life is doing something effective about feeling alone.

(2) The first recommended solution is a good romantic relationship.

(3) In such a relationship, the couple (A) gives knowledge of themselves to each other, (B) cares about each other, (C) respects each other, and (D) responds to each other.

The couple has love -- compassionate, committed, affectionate, romantic, intimate, passionate, and faithful. They are honest with each other and can trust each other.

Exercise: Look at other couples and estimate how many of these features of love their relationship has.

Hint: May find that only a small fraction of the couples have more than four of these features.

Result: A large fraction of people have no or poor romantic relationships and are lonely.

</comment><comment><USER3> dupe [https://news.ycombinator.com/item?id=10260029](https://news.ycombinator.com/item?id=10260029)

<comment><USER4> It doesn't count as a dupe on HN unless the post has had significant attention in the last year, which this one hasn't. See [https://news.ycombinator.com/newsfaq.html](https://news.ycombinator.com/newsfaq.html). Reposts of solid stories are a good thing, because otherwise /newest is too random.</comment></comment>


Ask HN: Have you built apps with Vue and Rails? Any opinions/learnings? - <AUTHOR> I'm building a site using vue and rails and I'm torn between building separate apps and then just treating rails as an API, versus building the vue frontend inside rails. I feel with the latter there may be some synergies in wiring up things, whereas the former would mean I can build the Vue app as strictly idiomatic as possible. <comment><USER2> I've used React and Rails together. Unless your app is highly interactive, I would do the frontend in Rails. You can use helper methods to render the React / Vue / etc components alongside your regular markup. In some cases, you can even pass the data directly into the component avoiding an API call at all. To me, this is the best of both worlds. You let Rails do the heavy lifting and bring in framework components when you need to do something complicated / interactive in the frontend.

</comment><comment><USER3> Although I don't have experience with Vue and Rails specifically, I've always found it beneficial to have the front & backend separate. It's really nice have the backend API completely decoupled from eachother.

It also depends on what your goals are - for a personal project, do what you like! Whatever is quickest & easiest is best.

For more public-facing services, I prefer to have the decoupled structure.

</comment><comment><USER4> I am currently wrangling with exactly this and after trying both ways I would always prefer using Vue CLI completely separated. The thing is though, after going through all of this I would completely skip the frontend JS mindset and just use server rendered templates. Development speed is so much faster and Turbolinks + some JS here and there is more than enough for 99% of projects.</comment>


Ask HN: Which piece of knowlege was most surprising for you? - <AUTHOR> In terms of information theory the most valuable pieces of information are those that surprise us most [1][2].
For example I wasn't really surprised while reading HN featured article "The healthiest people in the world don't go to the gym" [4], because the results of research presented there were close to my expectations (due to the model that emerges from all my previous knowledge).
On the other hand Libet's experiment [3] regarding consciousness was very surprising for me - because it's results are counterintuitive and if I was asked to guess them I probably would't be right.
Which facts are/were most surprising for you?
[1] https://en.wikipedia.org/wiki/Information_content [2] https://plus.maths.org/content/information-surprise [3] https://qz.com/quartzy/1452630/the-healthiest-people-in-the-world-dont-go-to-the-gym/ [4] http://www.informationphilosopher.com/freedom/libet_experiments.html <comment><USER2> That we can influence and direct the electrical energy that flows through our fascial planes via intense meditative practices.</comment>


Comcast Sued for Turning Home WiFi Routers into Hotspots - <AUTHOR> http://www.sfgate.com/business/article/Comcast-sued-for-turning-home-Wi-Fi-routers-into-5943750.php

<comment><USER2> i remember filing this under "dubious but probably legal b/c of some ridiculous ToS bullshit" when i first saw comcast's announcement about this a while back. the electricity angle seems pretty interesting-- no matter what arguments comcast could plausibly make about it not affecting your personal bandwidth it's pretty much impossible for them to argue that they're not essentially "stealing" electricity to subsidize their wifi network.

ianal, but the two keys here appear to be that the customers affected seem only to be people leasing equipment from comcast (ie the customers don't own the routers) and the opt-out nature of how the network is currently set up, as compared to a (kinder) parallel universe where such a network would be opt-in.

as to the latter point, i'm not sure if there is any difference in terms of legal doctrine but it definitely speaks to comcast's greed / customer hostility that they felt entitled to make the customer subsidize _them_ , instead of, say, giving the customer a discount on their bill if they chose to opt in to providing a base station for the network.

<comment><USER3> I'd also argue that charging you a rental fee for the modem and then turning around and selling access to the device again is shady as hell.

</comment></comment><comment><USER3> So many things wrong with this:

-Comcast provides a device to me _and charges mea rental fee_ then turns around and profits off of it at _no_ benefit to me. The modem even requires batteries which I have to pay to replace for God's sakes.

-Legal implications of other users potentially accessing illegal materials over a device located in my home.

-The Modem/Router combo supplied by Comcast is a piece of junk, and barely works for home use, let alone handling traffic created by strangers connecting.

I can confirm that Comcast does indeed turn this on without your permission. Furthermore, unless you pay attention you may not know that it is even happening. You can't disable it through the router administration either, you have to go through Comcast's website to find the setting and opt out.

All in all it's kind of a slap in the face after suffering through years of sub-par service and borderline abusive business practices.

</comment><comment><USER4> BT have been doing this for years in the UK.

[http://en.wikipedia.org/wiki/FON#BT_Fon_Community](http://en.wikipedia.org/wiki/FON#BT_Fon_Community)</comment>


Uber Driver scam, need your help asap - <AUTHOR> Hi, yesterday i took Uber in Colorado to drive me to the airport. Today i looked at the price Uber charged was about $150 instead of $45 for the trip because of cleaning fees. The driver accused me of leaving the car super dirty even though i know for a fact this wasn't me. I'm trying to prove this wasn't done by me. I have attached the pictures Uber sent me as "evidence". I have found a few discrepancies in the pictures taken. I need your help, let me know what i could say or if you see something i did not.
Here are the pictures:
https://imgur.com/a/rOXCh
Here is what i've found so far:
1) The seat colors are not identical: One is dark grey, the other is light blue.
2) The lights and shadows are strong in the photos of the grey seats while the lights and shadows in the blue seat pictures are non-existent, showing they were not taken at the same time at all. How could this driver get them cleaned on-time if she took the pictures at different time?
3) One of the picture depicts snow, but it did not snow for more than 4 days, all the snow was gone. This picture must have been taken days earlier if not weeks.
4) The pictures do not have data associated to them. The driver phone was a Samsung Galaxy S series. If she did take the pictures, she should be able to provide the original pictures with metadata associated to them, such as date, time, and location of the pictures taken. Please ask for these, i'm sure you'll get a negative response.
Thank you all ! <comment><USER2> Contact Uber via Twitter directly, also contact their customer support. As for the driver's accusation it should be verifiable since they have cameras running.

On another note, this looks like old stains.

<comment><AUTHOR> thanks for the feedback. Yea, definitely looks old</comment></comment>


Why not to use (f)lex, yacc or bison - <AUTHOR> https://tomassetti.me/why-you-should-not-use-flex-yacc-and-bison/ <comment><USER2> Is it not the case that most serious parser implementations are hand-written? In part because it makes it so much easier to provide good diagnostic messages. I feel like every other project moves from parser-generators to hand-rolled parsers and levels.

<comment><USER3> For general-purpose languages, yes, but there are some exceptions. Ruby uses yacc, and Python uses a custom LL(1)-ish parser generator, which is in the process of being replaced by a custom PEG parser generator [0].

[0] [https://github.com/gvanrossum/pegen](https://github.com/gvanrossum/pegen)

<comment><USER4> Ruby's yacc parser is my usual go-to for telling people how not to do it. That thing's terrifying.

<comment><USER5> To be fair, ruby is not a fun language to machine-parse.

<comment><USER4> Oh absolutely. As a _user_ , I love the ruby syntax. As a potential contributor to the codebase? NOPENOPENOPE.

</comment></comment></comment></comment></comment><comment><USER6> This whole article is very polemic and promotes an ANTLR book. It is futile to comment in detail because nearly every statement is false.

<comment><USER7> ..wow, this is a bit strong. I would encourage you to be more respectful of persons who work in this very specific field and share their ideas. There are always persons behind some work you insult so easily.

You know, there are over 2M persons who read our articles. A few hundred also bought a book or a video-course from us, but the vast majority just got some information from free, and we like it in this way. I do not think we got so many persons interested in what we do by lying.

If you have a different professional experience I would happy to learn from it.

<comment><USER8> It is irrelevant how many people read your articles; the critics will also show up as readers in the statistics.

Let's start the professional experience with another person's view:

[https://research.swtch.com/yyerror](https://research.swtch.com/yyerror)

"Seibel: And are there development tools that just make you happy to program?

Thompson: I love yacc. I just love yacc. It just does exactly what you want done. Its complement, lex, is horrible. It does nothing you want done.

Seibel: Do you use it anyway or do you write your lexers by hand?

Thompson: I write my lexers by hand. Much easier."

I happen to like both bison and flex, which are relatively easy to use and bug-free in my experience. Yet your article spreads hundreds of lines of FUD about these tools, a strategy that many ANTLR people use.

I have used ANTLR. It is not intuitive, the documentation is horrible, if you happen to find some advice on Stackoverflow it is likely to be for another one of the incompatible versions.

I suppose if you use ANTLR long enough, these problems go away. But bison or Menhir don't have these problems in the first place.

</comment></comment></comment><comment><USER9> It's been a while, but I had some fun using the "lemon" parser generator[1] used by SQLite. Worked alright for something where regexes weren't enough and I couldn't be bothered to write a custom parser. Probably has the same cons that lex/yacc would get, according to the post.

I do have the ANTLR book, but to be honest, if I would have the task of doing something more involved and where integration isn't as important (i.e. it could be an isolated command, not a module for huge Java/C# app), I'd probably be more inclined to get deeper into SML/Ocaml than ANTLR for this.

[1]: [https://www.hwaci.com/sw/lemon/](https://www.hwaci.com/sw/lemon/)

</comment><comment><USER10> I have seen multiple projects that run into maintenance problems due to the large grammar with thousands lines and antlr 3/4 incompatibility.

Generally parser generator adds a layer of complexity/constraint which may be significant if you need full control of lexing, parsing, semantic processing, error recovery of your language.

After working on a few language projects (including core language, web-based editor with syntax coloring and auto-completion) myself I would strongly suggest hand-rolled parser for any serous language endeavour(general purpose or DSL).

<comment><USER7> I am a bit surprised by this, as I had the opposite experience. I find ANTLR grammars much more maintanable than the hand-written parsers I encountered. Indeed I was asked to port hand-written parsers to ANTLR for maintanability. Also, ANTLR4 seems to me to produce grammars which are clearer than ANTLR3.

The weak point is error recovering, in my opinion. While ANTLR offers a sort- of-decent error recovery strategy for free, one has to customize it if they want to get great error messages.

I think that many maintainability issues are due to poor usage of ANTLR. What we do is to: 1) Limit semantic actions 2) For complex languages do tree- transformations, after parsing

With this approach we got pretty decent results.

Personally I find hand-rolled parsers too costly to build and harder to maintain, but I have limited experience with them. I guess it also depends on the context: if you are designing a DSL while writing the parser you want to be able to evolve it very quickly and in that context I think that a parser generator is very useful. If instead I would need to build an industrial grade parser for a general purpose language, having a large budget, then I would go for an hand-rolled parser

<comment><USER11> My experience has been similar to yours. The company I work at has very complex business logic written as a custom DSL parsed/executed by a crazy pile of Perl code.

Recently we had requirements to come up with a way of providing on-the-fly validation in a web editor. This was only possible by ditching the old implementation and re-writing the grammar using ANTLR. While the old implementation is unmaintainable and (probably, who knows?) buggy, the ANTLR implementation is trivial to work on, test and add new features to.

If you're working with a limited time budget, which is common when your main job isn't to maintain the language, then parser generators such as ANTLR are a godsend. ANTLR even enables you to generate parsers in different languages depending on where it needs to be executed. Need something to run client-side, in the users browser? Generate a JavaScript parser and you're done. Need it to run in the Java backend? Generate it in Java and call it a day.

While it's true that error handling isn't the best, it's already better than nothing, and, as you say, can probably be improved with a bit of customization.

</comment></comment></comment><comment><USER12> > Can you even imagine the quality of code that was written and fixed for 30 > years?

Total nonsense. Bison is very reliable, Flex can be a bit finicky to set up but is also reliable.

PostgreSQL uses bison ...

<comment><USER13> Indeed PostgreSQL is a good example of fairly old software (>20 years) that is highly reliable and well used. How many people needing a database complain that they _have to_ have something that has been written, and re-written, in recent years?

</comment></comment><comment><USER14> "ANTLR instead is more actively developed. It has been re-written from scratch a few times during the years, so the code has is of good quality."

... rewrites improve code quality ... right ... right???

<comment><USER15> I first learned Antlr way back about 20 years ago and found it fairly impressive. So when I go to write a new parser or lexer I always go take a look again...

But every time I take a new look at Antlr there's a huge amount of issues related to the fact that they have a huge new rewrite deprecating old versions while the new version is not ready: entirely new ways of doing things with missing documentation and examples, incomplete or missing language backends, and missing packaging for various Linux distributions.

What I get from it is: Antlr is a very powerful project... if you're in the Java ecosystem and if you're willing to use old no longer supported versions.

Otherwise, questionable choice.

<comment><USER16> Your experience mirrors my experience quite a bit. I remember a few years ago playing with ANTLR precisely as an alternative to flex/bison for a project I was working on. Just recently I had an embedded project (quite resource constrained, like 4kB of RAM) that needed to parse some data coming in a serial port, and I thought "hey, I should look at ANTLR again".

I fire it up, play around a bit, and did a calculator example or something like that. The next step, I figure, is to get it to generate some C code so I can test it on the device. Turns out ANTLR4 has dropped the C backend entirely! C++ is, currently, a no-go for the project, so... I guess I'm out of luck there.

Pleasantly, in a discussion with a friend, he asked the silly question: "couldn't you parse those strings with sscanf()?". I blinked in disbelief, wrote the tiniest parser to split the input on newlines, and sscanf() did the trick.

<comment><USER17> Yeah sscanf can do quite a bit. But it doesn't have any informative error messages, can't recover (skip to the next semicolon or whatever), doesn't keep a symbol table, can't do any parsing more complex than "the next line looks exactly like this pattern" etc.

So after years of rolling my own, I finally decided I would never do that again. All the off-by-one errors, difficult-to-diagnose failures and endless fiddling is out of my life now. I just use a parser tool.

</comment></comment></comment></comment><comment><USER18> Last I checked, the interface between lex and yacc (resp flex and bison) is by mutating static global variables. Is that still the case? It always struck me as exceptionally bad design.

<comment><USER19> I don't think you are particularly restricted to doing this, but lex/yacc was conceived and designed at a time (1970s) when memory was limited, and speed and size of code that could be parsed was valued much higher than structured code and being able to provide good error messages. You simply can't do a recursive descent parser with an in memory AST in 64KiB of RAM while being able to parse any meaningful size source file. You had to serialize the problem and output a sequential AST/intermediate opcodes on the fly. FSA based lexers and parsers are beautiful in how little RAM they actually require and how fast they are, albeit clunky and very low level by modern standards.

</comment></comment><comment><USER20> I tried to read this article, but unfortunately the lack of grammar was too distracting and I just couldn't get through it.

A lot of these are things that Google Docs or Microsoft Word will notice and ask you to change. I highly recommend using one of those to write (especially if English is not your first language) and trying to understand the suggestions it makes. Your articles will come out much more readable to others.

<comment><USER21> Counterpoint: I used to "and I stopped reading" when I ran into grammar issues. I gradually realized that it's worth the effort to focus on the underlying ideas and evaluate _that_ , rather than someone's 2nd language ability. People are smart, even when they talk funny, y'all.

(The worth/validity of the underlying ideas is a separate matter.)

Counterpoint 2: Automated grammar corrections are risky unless you're a native speaker. Ironic.

<comment><USER22> I think it's helpful to recall that the language is called English, not American, and the people who are experts on it think we don't talk good.

<comment><USER23> Most English speaking people everywhere speak with some variation on "formal" English. Even the received pronunciation in South england and london is relatively modern.

If anything, some American accents are closer to the English of the late 18th century than many British ones are, and the "official" received pronunciation is not without its fair share of critics.

</comment></comment></comment></comment><comment><USER24> Maybe I was holding it wrong but in my experience ANTLR's performance (on JVM) was abysmal.

<comment><USER25> Do you mean the performance of ANTLR, or the performance of ANTLR-generated parsers?

<comment><USER24> I mean ANTLR-generated parsers. Apologies for the ambiguity.

</comment></comment></comment><comment><USER26> I would have to check, but the resources on lex/yacc that I remember would make a point of their usability at scale. Generally it would be unlikely to wrote a faster lexer than what lex could do, but very likely to write a better parser than yacc. However writing a small lexer and/or parser _quickly_ would be easier with those tools than hand rolling one almost every time. Its great parser generators are still being developed and improved, and its fine to point out issues, but this article misses the point by a wide margin.

</comment><comment><USER27> Last time I checked ANTLR, it sucked, and after a quick skim now, it still sucks.

\- no way to specify operator precedence explicitly (instead, it's based on ordering of alternatives)

\- no explanation of conflicts / ambiguities in the grammar (again, ambiguities are silently resolved based on ordering of alternatives)

\- might, or might not, properly handle left recursion (by "handle" I mean, that the parser always halts)

TL;DR: LL parsing is a dead end, don't use it.

</comment><comment><USER28> Just a few points that I'd like to answer to this article: \- One of the primary concerns of Bison is the speed of the generated parsers. In the section about performance, they say that ALL is 135x faster than GLR. Digging into the original paper, we find that: \- 135x is from their best case against the worst GLR they could find \- they compare different tools, not different algorithms in the same tool \- the grammar they use is optimized for their tool \- the best GLR tool they're testing against isn't even running on the same OS \- moreover, they mention that the biggest chunk of their performance comes from the fact that they cache temporary results, so theoretically including a cache in a GLR parser could make it even faster. \- since there is no grammar analysis, we get back to the classic "static vs dynamic" language debate, where Bison tells you at build time if you have conflicts, and ANTLR relies on your good test coverage. \- They only have a passing mention of the fact that their grammar is strictly less powerful than GLR since they don't handle non-direct left recursion.

</comment><comment><USER29> I have used ANTLR to write a parser for a simple language and found it kind of confusing, but put it down to being new to ANTLR and to parser generators in general. The reason I chose it was for its ability to output parsers in multiple implementation languages for a single grammar - lemon et al seem to be tied to C. Are there any other systems the HN crowd would recommend for this use case?

<comment><USER30> I've been poking at coco/r recently and it seems to have a nice collection of backends.

Not sure how maintained it is, spent an afternoon getting the Python version ported over to py3 (which wasn't really all that hard) to learn how it works.

As TFA states you can't reuse grammars for multiple languages because the actions are declared inline but a few of the other complaints aren't an issue due to the way the generated parser does it thing -- quite well designed IMHO.

</comment></comment><comment><USER31> This is just an advert for ANTLR. I have no real experience with either but the article is really grating to read beyond the introduction.

<comment><USER7> I am sorry we had this impression. Sure, we use ANTLR a lot, for commercial and open-source projects, and we had to work with parsers written in (flex), yacc, and bison, so we shared our experiences as people who work all day long with parsers. We offer a lot of free resources on ANTLR and we have not specific interest in advertising ANTLR. For us it is a tool we use and love, just that

<comment><USER32> Almost anything fares better than flex/bison, this is shooting the proverbial fish in a barrel.

It comes across very strangely that your brother writes as if y'all are unaware of the existence of parser generators other than ANTLR. This makes me sad. In a fair comparison, ANTLR really is not a great tool; its capabilities are eclipsed by its marketing.

<comment><USER7> We have several articles comparing tens of other tools. That said of all the tools we worked with, ANTLR was the one we were most efficient with. It is true, there is a learning curve, but in our experience learning it was a great investment. I can understand other could have different preferences

</comment></comment></comment></comment><comment><USER33> I think they are learning tools for people to start and to play around. It is hard for an application developer with no background knowledge to understand what a parser does because it is quite abstract, for example lexer, tokens and expression tree.

However, to build a realistic and comprehensive compiler, I believe to build it manually is a much better option because it is less error prone, more flexible to tweak around and favours unit testing. Perhaps parser generator is better? I am investigating these kind of tools for work because we need to make our own unique formula implementation. I did use yacc for a school project, and I know its limitation. Since I am busy at my staff so I just let my colleague to make their decision and they decide to use jison. It turns out the product owners want see a much better error message in invalidating the formula, also we need to define our own function so we have to switch to another implementation in the next phrase.

</comment><comment><USER34> The paper:

[https://www.antlr.org/papers/allstar- techreport.pdf](https://www.antlr.org/papers/allstar-techreport.pdf)

Has the important detail why it scored good in benchmarks against other parsers:

"7.3 Effect of lookahead DFA on performance

The lookahead DFA cache is critical to ALL(*) performance. To demonstrate the cache's effect on parsing speed, we disabled the DFA and repeated our Java experiments. Consider the 3.73s parse time from Figure 9 to reparse the Java corpus with pure cache hits. With the lookahead DFA cache disabled completely, the parser took 12 minutes (717.6s)."

I'd still most probably hand-roll, at least when making something for a long- term project, and not some "demo" \-- there the stability and ease of maintenance is more important than fast availability of initial results.

</comment><comment><USER35> What I don't like about any of these parsers, including ANTLR is that for real languages you get not a clear EBNF grammar, but terrible mix of declarative and imperative statements. I was pretty sure PEG is the modern way to go, but looks like it's not. So, what is then?

<comment><USER32> [https://news.ycombinator.com/item?id=22494419](https://news.ycombinator.com/item?id=22494419)

</comment></comment><comment><USER36> I recollect using lex/bison for writing compiler for COBOL on mainframe using SAS/C to list variable names in COBOL programs (remember Y2K!). Later in life used ANTLR do something fun in Java. Never thought of comparing them. They both have their best part. If you are coming from Lex, Bison world - you will understand the philosophy behind ANTLR. But there is little more learning curve with ANTLR. Felt it is slightly complicated.

</comment><comment><USER37> If the new thing is actually better, you should not have to spend the bulk of an article tearing down the old thing to make the new thing look better in comparison.

</comment><comment> spider-mario That ANTLR example with the sum does little to convince me that the code should be separate from the grammar. It being separate means that you need to look at the grammar to know what tokens you have access to anyway.

(And, really, returning something of type "Any"?)

</comment><comment><USER3> > ANTLR uses an algorithm [...] called ALL.

Why is the tool called ANT _LR_ if it uses an LL algorithm? Did earlier versions use LR, or is it just a confusing name?

<comment><USER6> I think officially it is "ANother Tool for Language Recognition", unofficially it is "Anti LR", which usually is the mindset of ANTLR adherents.

<comment><USER2> Do they have a reason for this? I have written small LL and LR parsers and the latter seem to make much more sense to me.

</comment></comment></comment><comment><USER38> Tldr: use ANTLR instead.</comment>


Debugging Behind the Iron Curtain (2010) - <AUTHOR> http://archive.is/O70Rg <comment><USER2> Wow. I've discovered many bugs in my life. But none caused me to fear for my life.

>There were often significant food shortages in the Soviet Union, and the government plan was to mix the meat from Chernobyl-area cattle with the uncontaminated meat from the rest of the country. This would lower the average radiation levels of the meat without wasting valuable resources. Upon discovering this, Sergei immediately filed immigration papers with any country that would listen. The computer crashes resolved themselves as radiation levels dropped over time.

<comment><USER3> Check on the Therac-25 case, whis is particularly scary for me, having worked for 10 years on medical software (on the sysadmin side).

A race condition which killed people: [https://en.wikipedia.org/wiki/Therac-25](https://en.wikipedia.org/wiki/Therac-25)

> It was involved in at least six accidents between 1985 and 1987, in which > patients were given massive overdoses of radiation. Because of concurrent > programming errors, it sometimes gave its patients radiation doses that were > hundreds of times greater than normal, resulting in death or serious injury.

</comment></comment><comment><USER4> I love reading these bug-hunting war stories, there's no breakpoint for radioactive cows in GDB! Another favorite of mine is the 500-mile email bug[1], if there's a bigger collection of these stories anywhere on the internet, I'd love to read them

[1] [http://web.mit.edu/jemorris/humor/500-miles](http://web.mit.edu/jemorris/humor/500-miles)

<comment><USER5> [https://github.com/danluu/debugging- stories](https://github.com/danluu/debugging-stories) [https://github.com/danluu/post-mortems](https://github.com/danluu/post- mortems)

<comment><USER6> Well... I guess I didn't need to do much work this morning. Thanks for the links.

</comment></comment></comment><comment><USER7> I kind of doubt this story. What kind of radiation would you have from Chernobyl?

My understanding is that bit flips (SEEs) are typically caused by energetic particles like cosmic rays. To simulate upset damage on Earth, you typically need to decap the IC and go to a testing facility that directly hits the wafer with particles. It seems unlikely that somewhere near Chernobyl that there would be particles energetic enough that didn't get stopped by the IC packaging.

Gamma radiation gives total ionizing dose effects that degrade chips until the point they no longer work. Those shoot right through the chip packaging. But I don't think it typically causes upsets.

<comment><USER8> Neutrons from the various fission products?

[https://www.isis.stfc.ac.uk/Pages/ChipIR.aspx](https://www.isis.stfc.ac.uk/Pages/ChipIR.aspx)

</comment></comment><comment><USER9> Having issues loading the text on this page. Seems to keep "blinking" out for me. Is there anyway we can get a printer friendly or text only version? Super interested in this article.

<comment><USER10> [https://pastebin.com/raw/NR0SzmiL](https://pastebin.com/raw/NR0SzmiL)

</comment></comment><comment><USER11> archive.is banned in Russia. Haha, the Irony.

<comment><USER12> allow me to call the parent story bullshit.

<comment><USER13> Why? The error seems reasonable (radiation can and could have flipped bits, I've done it on sensitive hardware with a camera flash).

<comment><USER12> 1\. anonymous Sergey as a story source, without any verificable identity.

2\. "he went drinking with a few military personnel", so it's retelling of rehearsal, even more trust.

3\. "government plan was to mix the meat from Chernobyl-area cattle with the uncontaminated meat", that's seems plausible. contamination of meat was "up to 1,0*10-6 Ci/kg" (according to widely cited secret "annex 10 of protocol #32"), it's 300 times of banana equivalent. from the same scary stories fansites, there were 34 Кtons of bad meat (it's one thousand of railway carriages btw). so not so dramatic, definitely not a near nuclear blast.

4\. [computer was] "located in a building close to the railroad tracks". inverse-square law, walls, exposure time from a moving train etc. i would more believe in vibration caused by trains as a cause of problems.

5\. the SM-1800 computer: KR580VM80A cpu (8080 clone) is 6 mkm and 2 MHz (compare to modern 10 nm and 2 GHz), K565RU3A ram (4116 analog, like in ZX Spectrum), and other components. even the computer case was solid metal. i can not tell you precisely, but there should be a powerful source of radiation to crash such computer reproducibly.

so no, i don't buy it. sorry for hurting your radio- and russo-fobias.

<comment><USER13> > sorry for hurting your radio- and russo-fobias.

That's a mighty big chip you have on your shoulder there. Sorry to disappoint your ego but I was curious as to your objections. They're... not very good.

The inverse square law one is the worst - you make tons of assumptions about the building and then screw up the basic science. A few inches of lead is enough to block gamma rays -- which is why you wear a lead apron when you get x-rays -- but at least 3 meters of concrete are needed to stop them.

Assuming a standard brick or breeze block building and normal computer metal case plus a meter or two of separation (common at railways in Europe at least, when stuff is going by slowly) that wouldn't stop much gamma radiation at all.

Other than "it's a semi-anonymous story" you really don't have much in the way of objection here.</comment></comment></comment></comment></comment>


Ask HN: Any thoughts on our site Tweetflix.com - <AUTHOR>

So after about 4 months of hacking away our little web app is ready for some much needed HN dissection.<p>Any and all thoughts welcome and encouraged. http://www.tweetflix.com <comment><USER2> Clickable <http://www.tweetflix.com> .

IMO I think this is neat, but that being said, I don't know if I would actually use your service. I think maybe I am distrustful of the review algorithm. I looked at the reviews for "Orphan" and it had "reviews" like

"gah. boredom. havent tweeted in a while. no school more yay. watching orphan double yay :d"

and "excited to see orphan tonight! :d" and "@questlove some teens brought not one but two babies to see orphan. i got mgmt to escort out. enough is enough."

none of which are actually reviews, so how do you use data from that to tell me whether I would like it?

Maybe you are not using those to calculate your result, but if you are not, why show it in the "reviews" section.

I'm not trying to be overly critical, I like the idea, I think the site itself looks great, but I think the implementation might need some work if its going to be useful.

<comment><AUTHOR> Excellent Points!

And actually this is where we are having a little bit of difficulty.In that we are actually excluding many tweets, like the ones above. Out of 9.1 million total tweets on Tweetflix.com we are actually only counting 30% towards the approval rating and the rest are more of the "pulse on the movie".

Does it make more sense to split them out into "chatter" and "reviews"?

Thanks for the once over, very helpful! Also as far as the design goes, have to thank @kurtwojda for the time put in!

</comment></comment><comment><USER3> Very cool site!</comment>


The Presidential Records Act of 1978 meets web-based social media of 2009 - <AUTHOR> http://www.whitehouse.gov/blog/Reality-Check-The-Presidential-Records-Act/ <comment><USER2> Not getting political, but it always struck me as absurd when: _Perino says she "could not rule out" that as many as 5 million emails are missing but stressed the White House does not know the extent of what maybe lost. Twenty- two current White House staffers hold RNC email accounts -- 50 during the course of the two terms. About 1,000 people work for the Executive Office of the President. All White House official records are to be preserved indefinitely under the Presidential Records Act

Regarding Karl Rove -- who is in Portland, OR today doing fundraising -- the White House says that in 2004, his email accounts at the RNC were exempted from the RNC's practice of an automatic 30-day purge that applies to other users. In January 2006, Rove's ability to use a "double delete function" was removed so he could no longer manually delete and then empty his delete file. Officials say this applied only to Rove as far as they know now._

edit: source: <http://www.salon.com/politics/war_room/2007/04/13/e_mails/>

</comment><comment><USER3> Very interesting. I remember reading about this when Obama first took office, I believe. The article was discussing the painful process of updating the whitehouse home page. Apparently, any change to the page requires that the current page is saved in its entirety, archived, then a the updated page can be posted.

I've been wondering how this requirement effects Obama's web 2.0 push.

<comment><USER4> Probably very little--any version control system should meet that requirement.

</comment></comment><comment><USER5> It makes me physically sick that they have to rebut the accusation that archiving blog comments and other UGC is "a secret, sinister plan to catalog the activity of individuals on all social networks and capture personal or private information about individuals without their consent."

<comment><USER2> I agree the _secret, sinister_ stuff is nonsense, BUT I think there are legitimate privacy concerns for citizens. The Presidential Records Act is being, appropriately I believe, extended to social media. The average citizen probably had no idea the act existed; probably more so for anyone who is buying into the wildest exaggerations.

As the post explains, 'The PRA also extends to communications from people outside the White House. So if a lobbyist or CEO emails the White House, that communication is a presidential record and will eventually be made public. The same holds true for correspondence received by the White House from the public. Because of the PRA, Presidents Reagan, Bush, Clinton, and Bush all treated letters, faxes, and other communications from the public as presidential records.'

'Instead of sending an email, people often now post on someone's profile or comment on a video or photo that's been uploaded. When people want to tell the White House what they think, they'll often do the same thing on our social media pages. A lot of times, we solicit this feedback because we want to hear from you.

These new types of communications from individuals to the White House, even though they take a different form, are governed by the PRA.'

So yes, in the past if you sent a letter to the White House it was archived, but it is an entirely different mindset for someone posting a comment, or sending an @whitehouse on Twitter. We're talking about archiving, for public record, something you probably quickly typed, not a letter you sat down, wrote, and sent via the mail.

Is there anything sinister? No. But I think it makes perfect sense that this is something that needs to be explained to the general public who are a) woefully ignorant of how their government works, and b) rightfully suspicious of how laws written decades ago, which weren't designed for social media, let alone the Internet, are being interpreted and applied on the fly.

</comment></comment><comment><USER6> A few years ago, I wouldn't have believed this URL would exist one day: www.whitehouse.gov/blog/Reality-Check-The-Presidential-Records-Act/</comment>