Next generation Unix pipe by Alex Larsson - <AUTHOR> http://blogs.gnome.org/alexl/2012/08/10/rethinking-the-shell-pipeline/ <comment><USER2> I find the tendency to repeat Microsoft's mistakes deeply disturbing. Even if, in this case, the author acknowledges PoweeShell goes too far, his own idea goes too far.

I'd be all in with flags that make ps or ls spit JSON or XML, but this typed nonsense? What when I want to output a color? Will I need a new type?

Oh... and the sort thing... its not hard to sort numerically.

<comment><USER3> >I find the tendency to repeat Microsoft's mistakes deeply disturbing.

What, in this case, you consider "Microsoft's mistake"? I thought that PowerShell was commonly considered conceptually sound, but flawed in the implementation, mostly for it's verbosity making it unwieldy for interactive use. If this project can solve that, then I don't see it "repeating Microsoft's mistakes". Instead it would be correcting them.

</comment></comment><comment><USER4> I wonder where this comes from. There's no need for next generation.

People have, for thirty years or so, successfully printed their data into a suitable textual streams for processing with programs glued together with pipes, and optionally parsed the results back into some native format if so required.

Meanwhile, none of the "next generation" pipes have gained any momentum. Obviously they solve something which is either not a problem or they do solve some problems but create new ones in greater numbers than what they solved, tipping the balance into negative.

Any object or intermediary format you can think of can be represented in text, and you're back to square one. For example, even if there's XSLT and XQuery, you can just serialize trees of XML elements into a row-based format of expression and use grep on the resulting stream to effectively make hierarchical searches inside the tree.

<comment><USER5> I have the opposite experience, actually. Its so damn annoying to have programs communicating structured data with each other over pipes that more complicated things inevitably diverge into a) monocultural programs made with a single programming language and that don't communicate with the outside world or b) Some form of exchange format (JSON, XMl, etc) that needs to be explicitly supported by every participant.

And unix utilities suck at handling structured data. If your file format is line based you might have a chance of it being easy to work with but don't even ask what happens if you insert a newline inside a textfield then.

</comment></comment><comment><USER6> That is a terrible idea: sometimes the app can take advantage of a constraint to minimize work done.

In your example, if we just wanted to filter for a particular user, dps would have to print out ALL of the information and then you could pick at it. This doesn't seem bad for ps (because there's a hard limit) but in many other examples the output could be much larger than what is needed. That's why having filtering and output flags in many cases is more efficient in generating everything.

As a side note: To demonstrate a dramatic example, I tried timing two things:

        - dumping NASDAQ feed data for an entire day, pretty-printing, and then using fgrep
        - having the dumper do the search explicitly (new flags added to program)

Both outputs were sent to /dev/null. The first ran in 35 minutes, the second in less than 1 minute

<comment><USER7> If "sometimes the app can take advantage of a constraint" is an argument here, you should be against all usage of pipes.

<comment><USER6> That's not true. So in the case of `ps`, there is a known limit to the number of processes, and it is fairly small, so the performance hit is limited.

As another example in this context, if the original data source is gzip'd, it's faster to gunzip and then pipe rather than integrating the gzip logic into the app itself.

<comment><USER7> I still disagree. I think you are arguing for the inclusion of, at the least, grep, cut, head and tail in cat.

I do not claim that is a bad idea (conceptually, pipes do not require multiple processes, and those tools could be dynamically linked in) but why stop at those tools? Some people would argue that sed and awk also should be in, others would mention perl, etc.

I also do not see why it would be faster to use an external gzip tool through a pipe. If it is, the writer of the 'tool with built-in unzip' could always, in secret, start an external unzip process to do the work.

</comment></comment></comment></comment><comment><USER8> For certain types of unix pipeing, I have found it useful to pipe from tool to CSV, and then let sqlite process the data, using SQL statements. SQL solves many of the sorting, filtering, joining things that you can do with unix pipes too, but with a syntax that is broadly known. Especially the joining I have found hard to do well with shell/piping.

I think a sqlite-aware shell would be awesone, especially if common tools had a common output format (like csv with header) where that also included the schema / data format.

<comment><USER9> very clever

</comment></comment><comment><USER10> My preference for a "next generation pipe": Shared file descriptors. (Sort of)

It would work virtually the same as a standard pipe; the difference being you could control whether it was read, write, or both, and every application you 'piped' to would have access to the same file descriptors as the parent, unless a process in the path of the pipe closes one.

The end result will be the equivalent of passing unlimited individual arbitrary bitstreams combined with the ability to chain arbitrary programs. In fact, you could simplify things by simply passing the previous piped command's output as a new file descriptor to the next program, so you could easily reference the last piped program's output, or any of the ones before it.

For example:

 _cat arbitrary_data.docx | docx --count-words <$S[0] | docx --head 4 <$S[0] | docx --tail 4 <$S[0] | docx --count-words <$S[2] | echo -en "Number of words: $STREAM[1]\n\nFirst paragraph: $STREAM[2]\n\nLast paragraph: $STREAM[3]\n\nNumber of words in first paragraph: $STREAM[4]\n"_

STREAM[0] is the output of 'cat'. STREAM[1] is the counted words of STREAM[0] ($S[0] is an alias). STREAM[2] is the first 4 lines of the doc. STREAM[3] is the last 4 lines of the doc. STREAM[4] is the counted words from STREAM[2] (note the _" <$S[2]"_). And STREAM[5] is the output of 'echo', though since it's the last command, it becomes STDOUT.

There may be a more slick way of doing this, but you can see the idea. Pass arbitrary streams as you pipe, and reference any of them at any point in the pipe to continue processing data arbitrarily in a one-liner.

...

Actually, it looks like this is already built into bash (sort of), as the _Coprocesses_ functionality. I don't know if you can use it with pipes, but it's very interesting.

</comment><comment><USER11> I like the idea of processes dumping structured objects: pipes are rather often used for the processing of structured data, and while tabulated output certainly makes it easier, we still end up effectively using constants: cut to the third column, sort the first 10 characters, and print the first four lines.

This method is fragile when given diverse input: what if the columns could themselves contain tabs, newlines, or even nul bytes?

Passing objects as binary blobs, on the other hand, doesn't allow for ease of display or interoperability with other tools that don't support whatever format they happen to be. This, of course, can be rectified with a smart shell with pretty-print for columnar data (insofar as a shell could be charged with data parsing; you may imagine an implicit |dprint at the end of each command line that outputs blobs).

I'd also be interested in seeing a utility that took "old-format" columnar data and generated structured objects from it, of course, with the above format caveats.

<comment><USER12> Something like a cut, only we call it dcut? Actually sounds like a pretty good idea - that way those who don't want to switch to the new format don't have to, and you can pipe it through this program to create the new style structured output...

<comment><USER13> The inverse of dtable? Yeah, that would be very nice.

</comment></comment></comment><comment><USER14> What would be ideal to solve first is some sort of initial format negotiation on pipes. Otherwise you will end up with the wrong thing happening (eg having to reimplement every tool, spewing "rich" format to tools that don't know it, or regular text to tools that could do better).

We've already seen something like this - for example ls does column output if going directly to a screen, otherwise one per line, and many tools will output in colour if applicable. However this is enabled by isatty() which uses system calls, and inspecting the terminal environment for colour support.

Another example is telnet which does feature negotiations if the other end is a telnet daemon, otherwise just acts as a "dumb" network connection. (By default the server end initiates the negotiations.)

However the only way I can see this being possible with pipes is with kernel/syscall support. It would provide a way for either side to indicate support for richer formats, and let them know if that is mutually agreeable, otherwise default to compatible plain old text. For example an ioctl could list formats supported. A recipient would supply a list before the first read() call. The sender would then get that list and make a choice before the first write() call. (This is somewhat similar to how clipboards work.)

So the question becomes would we be happy with a new kernel call in order to support rich pipes, which automatically use current standard behaviour in its absence or when talking to non-rich enabled tools?

I would love it if grep/find/xargs automatically knew about null terminating.

<comment><USER15> man grep:

       -Z, --null
          Output a zero byte (the ASCII NUL character) instead  of  the  character  that  normally
          follows  a  file  name.   For example, grep -lZ outputs a zero byte after each file name
          instead of the usual newline.  This option makes the output  unambiguous,  even  in  the
          presence  of file names containing unusual characters like newlines.  This option can be
          used with commands like find -print0,  perl  -0,  sort  -z,  and  xargs  -0  to  process
          arbitrary file names, even those that contain newline characters.

       -z, --null-data
          Treat the input as a set of lines, each  terminated  by  a  zero  byte  (the  ASCII  NUL
          character)  instead of a newline.  Like the -Z or --null option, this option can be used
          with commands like sort -z to process arbitrary file names.

man xargs:

       --null
       -0     Input  items are terminated by a null character instead of by whitespace, and the quotes
          and backslash are not special (every character is taken literally).  Disables the end of
          file  string,  which  is treated like any other argument.  Useful when input items might
          contain white space, quote marks, or backslashes.  The GNU find -print0 option  produces
          input suitable for this mode.

man find:

       -print0
          True;  print  the  full  file  name on the standard output, followed by a null character
          (instead of the newline character that -print uses).  This allows file names  that  con‐
          tain newlines or other types of white space to be correctly interpreted by programs that
          process the find output.  This option corresponds to the -0 option of xargs.

<comment><USER13> Yes, in other words, the parent is right that zero termination is currently _not_ automatic.

<comment><USER15> He did not say automatic, he said "knew about" nulls. When you talk about automagically detecting nulls I have this image of an ascii-art Clippy with a cowsay bubble that says "I see you are using null terminated data, I have enabled --null for you."

<comment><USER14> I did exactly say automatic. It is the previous word to "knew about" you quoted!

And yes, I would expect that find detects that when it is talking to xargs then null termination should be used without the user having to go and fish out what the options are for each tool. And if you used ps with another tool that prefers json then ps can automatically do that, again without having to find and maintain flags.

</comment></comment></comment></comment></comment><comment><USER16> At the risk of stating the obvious - this won't take off for a simple reason of being too complex _by Unix standards_.

<comment><USER17> I'm not sure, it's probably not too complex by GNU standards.

</comment></comment><comment><USER15> _"Even something as basic as numerical sorting on a column gets quite complicated."_

        sort -g -k field_num

<comment><USER18> Two problems: the header is sorted along with the fields, and you have to look up the field number. Insurmountable? No; but somewhat complicated.

<comment><USER15> `grep -v` or `tail` seems like a lot easier workaround than desiging a brand new shell piping system. Maybe there are other use cases but sorting numeric fields is definitely not worth the effort. A lot of black magic can be easily conjured up with `col`, `tr`, `cut`, `column`, `head`, `tail`, `grep`, `pr`, and `sort`; and that's without ever even touching `sed` and `awk`.

<comment><USER13> Obviously a lot can be done, but its hardly easy, you yourself call it black magic. But its _very_ easy to do with typed data, and only the beginning of what you _can_ do.

<comment><USER15> When I said black magic[1] the last thing I was trying to convey was that using coreutils/bsdmainutils was complicated. Typed data is easy to work with, but creating a sophisticated unix pipes 2.0 is not. No matter how complicated you think coreutils/bsdmainutils mastery is, you have to admit its a lot easier than building unix pipes 2.0.

If you throw in numutils and moreutils you can go nuts with columns of data. What tasks would you like to accomplish on the command line with columns of typed data and pipes 2.0?

[1] On a side note I was surprised that we had different conceptions of what black magic. I was going for evil, nefarious, and/or unorthodox. Have I been using the term wrong? That's an honest question, it would not surprise me if I have been oblivious.

<comment><USER13> I don't expect every user to create unix pipes 2.0, so the difficulty of that is not really what needs to be compared. It will only have to be done once.

And once this is done any user can avoid having to painstakingly construct pipelines that try to cut out the right columns to treat as numbers, or avoid all the problems parsing strings that may contain spaces or other control characters. You can do an operation like:

filter out all processes with %cpu > 20 with uid > 1000 and sort by second cmdline arg as:

dps | dfilter pcpu ">" 20 uid ">" 1000 | dsort "cmdvect[1]"

Obviously a made up example, but something like this is easy to read and write, whereas something working on tabular ascii data would be quite long and complicated.

As per black magic, I have about the same interpretation as you. I didn't really misunderstand it to be about how complicated it was. However, "black magic" certainly has a feel of "you should not do this", and arguing that you can then use that in order to do something which could instead be simple and obvious in a typed system seems kind of weird.

<comment><USER19> Your example would be about the same length with awk and sort, with the only caveat that you need to figure out the field numbers, and the upside that I can trust the tools are available pretty much everywhere.

<comment><USER13> Its doable yeah, but its a lot more work.

First you have to handle the header specially (want it in the result but not in the comparisons).

In order to compare by uid you need numeric uids (-n), but that means you can't also get the readable username, so you need a custom output format.

Then you need to ensure the output format is such that nothing with possible spaces or control chars can end up in a column before the data you're looking at, as then finding the right column is hard.

Even then, extracting the first command line arg like in the example will fail in the case of a binary name that has a space in it (as there is no way to know which spaces in the commandline corresponds to actual spaces in the arguments or just delimiters).

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER20> Neat! I've commented about this very problem before on several of the many threads regarding "object pipes", ie. REPLs.

<http://news.ycombinator.com/item?id=1033623>

<http://news.ycombinator.com/item?id=1566325>

<http://news.ycombinator.com/item?id=2527217>

Since that last comment, I've been working a bunch with Clojure, which has a far more expressive variant of JSON, as well as some heavy duty work with Google's Protocol Buffers.

A few points:

1) Piping non-serializable objects is a _BAD IDEA_. That's not a shell, that's a REPL. And even in a REPL, you should prefer inert data, a la Clojure's immutable data structures.

2) Arbitrary bit streams is, fundamentally, unbeatable. It's completely universal. Some use cases really don't want structured data. Consider gzip: you just want to take bytes in and send bytes out. You don't necessarily want typed data in the pipes, you want typed pipes, which may or may not contain typed data. This is the "format negotiation" piece that is mentioned in the original post. I'd like to see more details about that.

3) There seems to be some nebulous lowest common denominator of serializable data. So many things out there: GVariant, Clojure forms, JSON, XML, ProtoBuf, Thirft, Avro, ad infinitum. If everything talks its own serialization protocol, then none of the "do one thing well" benefits work. Every component needs to know every protocol. One has to "win" in a collaborative shell environment. I need to study GVariant more closely.

4) Whichever format "wins", it needs to be self-describing. A table format command can't work on field names, unless it has the field names! ProtoBufs and Thrift are out, because you need to have field names pre-compiled on either side of the pipe. Unless, of course, you start with a MessageDescriptor object up front, which ProtoBufs support and Avro has natively, but I digress: Reflection is necessary. It's not clear if you need header descriptors a la MessageDescriptor/Avro, or inline field descriptions a la JSON/XML/Clojure. Or a mix of both?

5) Order is _critical_. There's a reason these formats are called "serializable". Clojure, for example, provides sets using the #{} notation. And, like JSON, supports {} map notation. Thrift has Maps and Sets too. ProtoBufs, however, don't. On purpose. And it's a good thing! The data is going to come across the pipe in series, so a map or set doesn't make sense. Use a sequence of key-value-pairs. It might even be an infinite sequence! It's one thing to support un-ordered data when printing and reading data. It's another thing entirely to design a streaming protocol around un-ordered data. Shells need a streaming protocol.

6) Going back to content negotiation, this streaming protocol might be able to multiplex types over a single stream. Maybe gzip sends a little structured metadata up front, then a binary stream. ProtoBufs label all "bytes" fields with a size, but you might not know the size in advanced. Maybe you need two synchronized streams on which you can multiplex a control channel? That is, each pipe is two pipes. One request/response pair and the other a modal byte stream vs typed message stream.

Overall. This is the nicest attempt at this idea I've seen yet. I've been meaning to take a crack at it myself, but refused to do it without enough time to re-create the entire standard Unix toolkit plus my own shell ;-)

<comment><USER13> Regarding order. The dtools approach uses a stream (i.e. potentially infinite) of variants. Each variant is a self contained typed data chunk which is by itself not "streamable" (i.e. you have to read all of it). The data chunk is strongly typed and the type is self-described.

The supported primitive types are: bool, byte, int16, uint16, int32, uint32, int64, uint64, double, utf8 string (+ some dbus specific things).

These can be recursively combined with: arrays (of same type), tuples, dicts (primitive type -> any type map), maybe type, and variant type

In my dps example I generate a stream of dictionaries mapping from string to variant (i.e. any type). The type of each item in the map differs. For instance cmdvec is an array of strings, whereas euid is an uint32.

<comment><USER20> Thanks. I looked at the GVariant page a bunch too.

It seems like the encoding is a steam of _{type, value}_ pairs, where values can contain per-type headers as well.

Protobufs, on the other hand, use _{field, wire-type, value}_ where _field_ is required to have an externally known type to parse _value_ , but _wire-type_ is sufficient to determine the length of value, so you can skip unknown fields (used for backwards compatible protocols). In theory, required fields could omit field and wire-type, but Protobufs deemed it more complexity than justifies the space and performance impact.

Primitive values like integers are totally expected in any such format like this. Their salient feature being that they're of known length. I'm a little more leery about "arrays" or other data structures of variable length which are encoded with a known length. Consider Pascal strings _{length, [chars]}_ vs C strings _{[chars], NULL}_. The later lends itself much better to streaming protocols, but the former is far simpler to work with when you have a complete dataset.

I ran into this situation with a Protobuf I was designing where the first attempt had a message with a repeated field, but it became obvious that I wanted a begin message, a repeated message of singular fields, and then an end message, to allow a fast-start on the send, which didn't require to know the full data set length up front.

There are, however, situations where you _do_ want the length up front. For example, if you need to allocate space to put things. You can get faster parsing if you know the total message size immediately. In general, however, I don't think it matters all that much with modern languages and hardware.

This is one reason why Clojure has both lists and vectors. Lists are lazy head/tail pairs and (count some-vector) is a constant time operation. Unfortunately, Clojure's reader doesn't seem to offer streaming reads of lists (I may be wrong about this).

The bigger issue with unbounded values is that they are more difficult to work with in most languages. Haskell, Lisps, and other functional languages fair far better than most, but once you start mixing fixed-sized messages with known fields, with variable-sized sequences, you wind up with a situation like _{x, [ys], z}_ where a piece of code wants to look at _z_ before looking at _ys_. If that tuple is represented as an associative structure _{:x 1, :ys [2 3], :z 4}_ then it's suddenly very confusing that it's an ORDERED map and all sorts of assumptions go out the window.

Even more fundamentally: Source code is a serialized protocol. You write down text and the order of the characters on the page have meaning. Sometimes, that order may be over-specified, but regardless, humans see order and make assumptions from it, even when order doesn't matter.

</comment></comment></comment><comment><USER21> I've quickly jotted some thoughts here: <http://damnkids.posterous.com/rich- format-unix-pipes>

Regarding this version, standardizing on a particular transfer format is a bad idea. If history has shown anything, it's that we like to reinvent this stuff and make it more complicated than necessary (see also XDR, ASN.1, XML, etc. :) pretty much on a 5 year cycle or thereabouts.

Do the bare minimum design necessary and let social convention evolve the rest.

<comment><USER13> Having to many different formats is also a problem though, as incompatible formats means you can't combine two apps in a pipepine.

The negotiations in dtools is made using a F_GETLK hack with a magic value offset. That approach could easily be extended to support multiple formats.

<comment><USER21> The way I see it there's two distinct problems being coupled together, kinda like inventing HTTP but defining it only to be used with (html, gif, png) or something. The reality is that if your solution gets even some adoption, conventions will quickly emerge based on actual use rather than _expected_ use, which almost never goes well. Additionally when some after-market use is discovered that wasn't part of the original spec, yet makes fabulous sense, existing implementations may be better positioned to deal with it (instead of suddenly finding they're being fed PNG files which are actually base64-encoded XML, or something mad like that, typical shoehorning crap).

Similarly, producing a big ecosystem of utilities to go along with it will probably result in a bunch of 1970s style compatibility commands that nobody actually uses any more (say, in 2030).

But feel free to bake a glib-specific serialization in and I'll feel free to pass it up. ;)

Love your fcntl() hack. My kind of hack!

</comment></comment></comment><comment><USER22> If I can do a slight PG impression, "what problem does this solve?"

<comment><USER18> Among others, this problem:

[http://www.dwheeler.com/essays/fixing-unix-linux- filenames.h...](http://www.dwheeler.com/essays/fixing-unix-linux- filenames.html)

find -print0 is a lame hack, and even filenames with spaces (not newlines) are somewhat messy to work with on the Unix shell.

Or a little recurring problem I have: How do I grep the output of grep -C (matches showing multiple lines delimited with a "--" line)? I wrote a custom tool to do it, which does the job, but really it would be nice if I could use all the normal line-based Linux tools (sort, uniq, awk, wc, sed) with a match as a "line".

<comment><USER23> > find -print0 is a lame hack, and even filenames with spaces (not newlines) > are somewhat messy to work with on the Unix shell.

This problem is simply a flaw in sh (and its descendants), other shells handle it much better, see for example Tom Duff's rc shell: <http://rc.cat-v.org>

Also note that Plan 9, the successor to Unix (and which uses the rc shell as its main shell) doesn't even have a find command, find's design is not really very unix-y.

As for your second questions, the answer might be structural regular expressions: <http://doc.cat-v.org/bell_labs/structural_regexps/>

<comment><USER18> > This problem is simply a flaw in sh (and its descendants)

Indeed, although it's not _just_ sh; if you want to, say, make a table of filenames and some attributes of each file, you're in trouble if the filenames contain spaces (awk, cut, sort don't work as easily) and screwed if they contain newlines.

What does Plan 9 use instead of find?

> As for your second questions, the answer might be structural regular > expressions:

I've actually been meaning to write a clone of the command line portion of sam, tack on some slightly more powerful features, and try living with it... it would be able to solve much of that use case, but I think it would be cleaner if all the normal tools just knew that the output of grep -C is, in fact, a list of multiline strings.

<comment><USER24> The Plan 9 approach is to avoid creating problems for yourself by not using spaces in file names to begin with. The file server initially disallowed spaces in file names just as nulls and slashes are disallowed. That restriction has since been relaxed,† but everyone still avoids spaces. If you cannot avoid files with spaces in their names, there exists trfs,†† a file system that transparently replaces spaces with something more convenient.

Instead of find, I run /bin/test on a list of files. For anything more complicated than what test can handle, I use Inferno's fs program.†††

† [http://swtch.com/cgi- bin/plan9history.cgi?f=1999/0323/port/c...](http://swtch.com/cgi- bin/plan9history.cgi?f=1999/0323/port/chan.c;v=diff;e=1)

†† <http://a-30.net/inferno/man/4/trfs.html>

††† <http://www.vitanuova.com/inferno/man/1/fs.html> It has a misleading name. It is not a file server.

</comment></comment></comment></comment></comment><comment><USER25> Also see <https://github.com/unconed/TermKit/>

</comment><comment><USER26> I've been playing around with a similar idea - using plain JSON as the message format, you can make a set of pipeable command line utilities for manipulating data from many web APIs.

<comment><USER27> Have you seen RecordStream? It's some good cli tools based on streams of json which might be handy.

<https://github.com/benbernard/RecordStream>

I've used it a lot and it's a godsend for a most "record-y" manipulation.

</comment></comment><comment><USER28> I've noticed that the NUL-termination problem [1] has come up a number of times in these comments. If you want a solution to this that isn't so drastic as an object system, perhaps take a look at Usul [2], non-POSIX 'tabular' Unix utilities which use an AWK-style $RS.

[1]: <http://news.ycombinator.com/item?id=4369699>

[2]: <http://lubutu.com/soso/usul>

</comment><comment><USER29> Similar idea: <http://code.google.com/p/recordstream/>

</comment><comment><USER12> What about providing a filter that converts to whatever format you can think of? e.g. outputs in JSON or XML

<comment><USER30> Because what are you converting _from_? It can't be turtles all the way down, at some point there must be a defined system that everything speaks. Adding output formats after that is relatively simple.

<comment><USER12> The output is typed, in a standard format. He defines in his post (cf output of the dfs program).

</comment></comment></comment><comment><USER31> Actually I prefer Powershell's approach to transfer objects, as it is more flexible than standardize in a specific transfer format.

But I do concede that it has the downside that if the object lacks the properties you want to access, then it might be painful in some cases.

<comment><USER32> But it is not like you cannot do normal string processing using cmdlets like "Select-String". And an object missing a property is almost same a column missing in the returned text output right?

<comment><USER31> Good point, I've forgotten about that.

</comment></comment></comment><comment><USER33> I can't believe this. Just 2 or so weeks ago I set about writing exactly something like this in Haskell [1]. It's by no means complete or even working at this point, but basically what I had in mind was something like:

        yls | yfilter 'mdate = yesterday && permissions.oread = true' | yformat -ls

Every tool emits or consumes "typed" JSON (i.e. JSON data with an additional JSON schema). Why typed? Because then the meaning of things like _mdate = yesterday_ can be inferred from the type of _mdate_ and mean different things depending on whether _mdate_ is a string or a date. In the case of a date, the expression _mdate = yesterday_ can automatically be rewritten to _mdate >= 201208110000 && mdate < 201208120000_ etc. In the case of a string we do string comparison. In the case of a bool we emit an error if the compared-to value isn't either _true_ or _false_ , etc.

Basically, I wanted to build a couple of standard tools inspired by the FP world, like filter, sort, map, fold (reduce) and have an universal tool for outputting formatted data in whatever form is desired - be it JSON, csv files, text files or custom formats. Every tool would support an -f parameter, which means that its output is automatically piped through the format tool, so that something like

        yls -fls

is functionally equivalent to

        yls | yformat -ls

which would output the JSON data from _yls_ in the traditional ls way on a unix system.

        yls | yformat -csv

would output csv data. Some more examples:

        yls | yfold '+ size' 0

prints out the combined size of all files in the current directory.

        yls | ymap 'name = name + .jpg' | ymv

would append .jpg to all files in the current directory.

        ycontacts | yfilter -fcsv 'name = *John*'

would print out all Google contacts containing _John_ in their name as a csv file.

        yps | yfilter 'name = java*' | yeval 'kill name'

would kill all processes whose names start with 'java'.

The cool thing about this is that this approach conserves one of the main selling points of FP: composability. I.e. you can throw something like _yfold '+ size' 0_ in a shell script and then write:

        yls | size.sh

This way people would be able to build an ever growing toolbelt of abstracted functionality specifically tailored to their way of doing things, without losing composability.

[1] <https://github.com/pkamenarsky/ytools>

<comment><USER34> Personally, I'm not feeling the quotes and would prefer parens since they're nestable.

</comment></comment><comment><USER35> So JSON?</comment>


Distilling JRuby: The JIT Compiler - <AUTHOR> http://www.realjenius.com/2009/10/06/distilling-jruby-the-jit-compiler/ <comment><USER2> Not sure why they don't simply compile to Java bytecode to start with. It doesn't seem like they're making use of runtime type profiles here a la Self.

It still seems to me that JIT compilation is needless, unless your compiler is as smart as Self's. One should simply compile to the desired target language and avoid warmup costs.

<comment><USER3> The JRuby developers have spent a lot of time doing performance measurements, implementing optimizations and trying out different approaches. You're just throwing in your two cents based on second-hand knowledge of an obsolete language implementation...

<comment><USER2> Indeed, I don't know about JRuby. I know about Scheme. There is no fast Scheme that JIT-compiles, that I know of. All the fast Lisp compilers that I know compile ahead-of-time.

If you know why they did this, pray tell :) As a language implementor yourself, you probably have a good feel for the cost accounting.

</comment></comment></comment><comment><USER4> i like the domo kun.</comment>


How To Build A Site That Looks Great On Every Screen - <AUTHOR> http://www.readwriteweb.com/archives/how_to_build_a_site_that_looks_great_on_every_scre.php <comment><USER2> Honestly, the sites I find most usable on my iPad and iPhone are normal desktop sites that don't interfere with pinch-zoom and double-tap. That's how the device and browser were designed to work.

Sites that try to deliver an "optimized experience" are usually crippled at best. At worst they are completely broken. The really fancy ones like OnSwipe crash Mobile Safari after a couple of page views.

<comment><USER3> Not sure if this is the same thing you're talking about, but I've found a handful of WordPress blogs have some sort of mobile theme that tries to mimic the iOS "bounce" when scrolling all the way up or down. Ironically, most of the time it breaks MobileSafari on my iPad.

<comment><USER2> That is OnSwipe

</comment></comment></comment><comment><USER4> This is no tutorial or even an article on responsive design. It is a small collection of links, that seems to confuse HTML5 or iPad magazines with responsive design.

I think this is a fluff piece, with a catchy title, build around a buzzword, not around expertise or original research. We designers get a few links to the basics of HTML or irrelevant frameworks, could you even imagine the same happening for a serious programming article?

</comment><comment><USER5> Good in theory, bad in practice. I will fully admit that this is how it should work, but the soapbox site immediately points to the trouble you get from it. From simple things such as an email field that doesn't say it's an email field, and running out of the viewport boundary because it's too long, to issues like the top banner cutting the photo wrong on the Stay Connected page, and having to scroll down deep to get to content. I find this site looks a lot more unusable on a mobile device.

I applaud the Boston Globe for doing this so well, but I have yet to see other good implementations on proper big websites.

<comment><USER6> Another thing I always notice on these "responsive" sites is some huge picture that gets downloaded even if you are on a tiny screen. Here it is 25*(224x224px) + one bigger header image. That's quite good actually - I saw a site where I would get (as in download and render resized) a 2Mpix picture in the header even if I resized the tab to some 200px in width.

<comment><USER7> Reminds me of the good old days of

        <IMG SRC="Logo-From-Branding-Guidelines-2000x2000.gif" HEIGHT=100 WIDTH=100>

... It is pretty easy to avoid doing this with CSS background images: just use something like:

        .banner {
            background-image: url(banner-big.jpg);
        }

        @media screen and (max-width: 999px) {
            .banner {
                background-image: url(banner-small.jpg);
            }
        }

... of course, this kind of thing doesn't translate well into a "framework".

</comment></comment></comment><comment><USER8> I'm really not happy with the current situation of web design. Especially the "looks great on every screen AND device" part. To create a truly responsive Design you have to incorporate Orientation, Width, Height, Pixel density and much more. (which usually nobody bothers with since this would be way to complicated) People are using only subsets of these, so the sites often break on different devices, etc.

Then there's the problem with the content. Mobile users won't necessarily need all the things Desktop users do, since they usually have a very clear picture of what they want to do on the site. There needs to be an easy server-side implementation for such things. (would love to hear about it if someone knows any related projects)

PS: Please allow your users do "default" to a standard-version. I've seen so many sites break on mobile phones, making them completely useless...

<comment><USER9>
      > Mobile users won't necessarily need all the things Desktop users do,
      > since they usually have a very clear picture of what they want to do
      > on the site.

That's backwards. Desktop users want exactly the same that mobile users do. Desktop version has more only because someone thought "hey we have some space there why not to fill it with something?". That something is often most usless stuff.

      > PS: Please allow your users do "default" to a standard-version. I've seen
      > so many sites break on mobile phones, making them completely useless...

The thing is: the default version will soon be the mobile one. Take a look at Luke Wroblewski's "Mobile first" it's a good insight what's going on.

<comment><USER8> The first point is actually the best thing that has come from Mobile Design thus far. It significantly reduced the amount of unnecessary distracting Ads on the page. Since, as you've said, they are usually used to fill the space larger Environments offer.

I've used the mobile first approach for my last project and I think a site isn't truly responsive if it hasn't a liquid(%) layout. Building those layouts is a pretty messy thing though :(

</comment></comment></comment><comment> alok-g It displays link URLs when printing or converting to PDF.

"kit called Foundation" changed to "kit called Foundation (<http://foundation.zurb.com/>)

See also: [http://davidwalsh.name/optimize-your-links-for-print- using-c...](http://davidwalsh.name/optimize-your-links-for-print-using-css- show-url)

</comment><comment><USER10> But why does the Foundation layout go directly from 4 columns to only one at smaller sizes? For example, the soapbox page has 4 photo columns that at smaller sizes go to directly to one large column. Why not have two columns as a transition before one?

</comment><comment><USER11> That site was unreadable from my mac as there were ads _everywhere_.

<comment><USER12> It was useless on my MacBookPro (with Chrome) because the menu bar was white text, on a white background. And some white text had over-written the title. And a bunch of other stuff.

</comment></comment><comment><USER13> This is typical 'naive' solution which thinks all mobile devices are just "PC with smaller screen and slower CPU". Majority of my potential mobile market still use BlackBerry OS 5 browser which is not WebKit-based, and even more who use feature phone. Oh, and China phone.

<comment><USER9> So they will became your real market when they upgarde to the more capable smartphones, right? There should be a reason why they are "potential market" and I bet that not enjoyable user experience with those old-schoold devices is a big part of that.</comment></comment>


A Nashorn Starter for Gradle Users - <AUTHOR> https://github.com/psanders/nashorn-gradle-starter <comment><USER2> Your site contains 5 tiny files, each written in a different syntax, i.e.

      README.md - 8 lines of Markdown
      app - 6 lines of bash
      build.gradle - 14 lines of Apache Groovy
      log4j.xml - 18 lines of XML
      main.js - 8 lines of Javascript

There's something wrong with the state of the ICT industry when so many different syntaxes are required for something so simple as getting Gradle to build some Javascript for the JVM.</comment>


Inside Kdenlive: How to fuzz a complex GUI application? - <AUTHOR> https://kdenlive.org/en/2019/03/inside-kdenlive-how-to-fuzz-a-complex-gui-application/ <comment><USER2> Previous submission: [https://news.ycombinator.com/item?id=19337689](https://news.ycombinator.com/item?id=19337689) (3 days ago)</comment>


Ask HN: How to do equity split when a founder can and puts in more money? - <AUTHOR> Hey guys!<p>I have more cash than others on my founding team. I've already been putting more of it in than the rest of the team. What's the fair / standard way of handling this situation so that the team continues to congeal? I'm pretty confident that we have a smash hit on our hands. Is there a fair way of doing this long term and short term? Thanks! <comment><USER2> It's best to put in the cash on the same terms as an outside investor would, and split the founder shares evenly. The SAFE investment documents [0] are standard and battle-tested.

[0] [http://www.ycombinator.com/documents/#safe](http://www.ycombinator.com/documents/#safe)

<comment><AUTHOR> Perfect! Thank you.</comment></comment>


Ask HN: Are there any alternative query languages? - <AUTHOR>

I am going to do my research proposal on query language. Could anyone give me any references on alternative query languages especially those have very different format than SQL. <comment><USER2> There's Object Graph Navigation Language (OGNL), and OMG Object Query Language used by most OODB manufacturers. And Db4o uses SODA query language.

Then MongoDB has a form of query language with JSON syntax.

<http://www.opensymphony.com/ognl/>

<http://en.wikipedia.org/wiki/Object_Query_Language>

<http://www.db4o.com>

<http://www.mongodb.org/display/DOCS/Advanced+Queries>

</comment><comment><USER3> K programming language for kdb from kx systems.

<http://en.wikipedia.org/wiki/K_%28programming_language%29>

</comment><comment><USER4> LINQ is a query language for .NET

</comment><comment><USER5> Look at XPath and XQuery.

</comment><comment><USER6> I am surprised this isn't mentioned yet, but Prolog makes a very acceptable query language. Although I think in that version it is called 'Datalog'.

<http://en.wikipedia.org/wiki/Datalog>

It is obviously much different than SQL. There are actually quite a few implementations of it if you look at that wikipedia page.

The queries end up being much more declarative than anything you'd do with SQL. Pretty much you ask the database a question, and it returns an answer (which is awesome).</comment>


My boss asked me to use Orbit UI and I said No - <AUTHOR> http://www.gorkem-ercan.com/2010/10/my-boss-asked-me-to-use-orbit-ui-and-i.html <comment><USER2> No, his boss asked him if they -should-. He didn't tell him to use it. He did what a boss should.

Sensationalist (and untrue) headline.</comment>


Why Finland's Basic Income Experiment Isn't Working - <AUTHOR> https://www.nytimes.com/2017/07/20/opinion/finland-universal-basic-income.html?smid=fb-nytopinion&smtyp=cur&_r=0 <comment><USER2> The article authors' headline "Why Finland's Basic Income Experiment Isn't Working" is misleading. It suggests that they are commenting on information about how the experiment is going. Instead, they criticize only the trial's approach, which was known before it began.

There is a comment about the sample size having been reduced to a fifth of the original proposal. But my previous sentence is the entire content in the article relating to how the experiment is going.

Everything else in the article is commentary about the original proposal and unrelated to the experiment's progress.

</comment><comment><USER3> Although i believe that Universal basic income will one day be available to everyone, we simply are not ready for it yet. The full automation of jobs is still years and years away. When we start running out of jobs for people then our government might start considering it.

As i see it this experiments will have little effect in the whole idea of the UBI. If someone guarantees you an extra income for only a year, then you wont really change your whole life. You might do an extra trip, purchase something that you wanted but never could but nothing life changing.

</comment><comment><USER4> This article is complete shit. No actual data thet proves the now misleading title. Some Yank decided to take a shit on an idea that is eventually going to be reality.

<comment><AUTHOR> I am pro-UBI but this experiment only is I. So "as an experiment" it has failed.

<comment><USER5> > I am pro-UBI but this experiment only is I. So "as an experiment" it has > failed.

You might want to read TFA. The experiment hasn't started yet and the authors criticise the reduction in scope.

<comment><AUTHOR> Err, yes that was my point. It's not going to be U or B. So there's no point in calling it a UBI experiment.</comment></comment></comment></comment>


Justice Dept. defends secret rulings in new spy court filing - <AUTHOR> http://gigaom.com/2013/07/08/justice-dept-defends-secret-rulings-in-new-spy-court-filing/ <comment><USER2> The actual article, without the blogspam and inaccurate title: [http://gigaom.com/2013/07/08/justice-dept-defends-secret- rul...](http://gigaom.com/2013/07/08/justice-dept-defends-secret-rulings-in- new-spy-court-filing/)

The gist of the DOJ's argument is sound. The key thing to understand is that the FISC court is not making law. It is interpreting the government's powers under existing law. This is a process the executive regularly engages in: it asks itself, is this activity legal? This is the only way it can act, indeed the only way anyone can act, within a federal court system that does not allow courts to issue advisory opinions. You interpret the law to determine whether your course of action is constitutional.

Without secret FISC rulings, you'd have secret NSA legal interpretations. The FISC can't grant the executive any power it didn't otherwise have--it exists instead as a check on the discretion the executive would otherwise have in using that power. Nothing forces the government, or any other entity, to publicize its interpretation of law until it is challenged in public court.

EDIT: I should note that I don't think the DOJ's argument is necessarily the "right" one. Or the "best" one. But I think it's sound. It's based on a conservative interpretation of the scope of the executive's powers, but not one that's outside the mainstream.

<comment><USER3> The FISC court has repeatedly made the wrong decisions. When their decisions have been brought to light, there's been massive public outrage, and an effort on their part to to arrest the "leaker". These are all signs of a corrupt court, that needs to be dissolved.

<comment><USER2> What "massive public outrage?" A pitiful 400 people showed up to the San Francisco "Restore the 4th" protests.

<comment><USER4> Just because people didn't give up their hotdogs to risk getting their heads smashed in doesn't mean there's not outrage.

<comment><USER3> Agreed. National holidays might not be the best day to hold protests.

</comment></comment></comment></comment></comment><comment><AUTHOR> Looking at history it would seem that when law disappears from view, people disappearing isn't far behind.

</comment><comment><USER5> The DOJ's brief says on p. 16, footnote 5: "Sixteen members of the House of Representatives as amici curiae raise two concerns regarding information sharing between the Executive Branch and Congress: (1) that because much of the information they receive is classified, they cannot discuss it in congressional debate..."

I had thought that the Constitution guaranteed Congress the right to say anything when they are talking aboiut legislation ("...and for any Speech or Debate in either House, they shall not be questioned in any other Place.", the Speech or Debate clause.) Any Constitutional lawyers here? How do these two things jibe?

<comment><USER6> You might think so, but that immunity excludes felonies, and felonies are cheap these days.

<comment><USER5> Does it? That's not how I read it, but I'm not a Constitutional Lawyer. Here's the full clause

"The Senators and Representatives shall receive a Compensation for their Services, to be ascertained by Law, and paid out of the Treasury of the United States. They shall in all Cases, except Treason, Felony and Breach of the Peace, be privileged from Arrest during their Attendance at the Session of their respective Houses, and in going to and returning from the same; and for any Speech or Debate in either House, they shall not be questioned in any other Place."

I guess the semicolon can be interpreted in two ways, although I interpret the Speech or Debate clause as being only after the semicolon; the use of "and for any..." also does not seem to exclude felonies. Logically the clause would also seem to lose any bite it might have if things the legislature declared a felony were then excluded from Speech or Debate. But again, IANACL.

</comment></comment></comment><comment><USER7> Erm, so the US is a democracy, right? So, all these muppets in government are servants of the US people, right? So, who exactly are they to tell Americans what they have the right to know and not know?

<comment><USER8> Nobody. If you disagree, organize their ouster. Just be warned that by all appearances most Americans don't see this as a voting issue.

<comment><USER9> So voter turn out in 2012 was around 57% or so and Obama had 50%-51% of the popular vote. So, basically 25% of America decides the election ignoring the electoral collage which reduces it to 270 people, not all of which are required to vote with their districts popular vote. This pretty much marginalizes the middle ground voter.

So yeah... what can you do.

The only way I see any change coming is if you can get around 10% of congress recalled in their given states. Unless people are upset enough to do that, you wont see any change. Even if incumbents are thrown out during normal election cycles, congress doesn't seem to get the point. As a whole their approval rating is <15% which is terrible. But their individual approval rating amongst their own constituents is much much higher... There is something wrong with that logic. How do you convince voters that they are part of the problem.

<comment><USER10> No, no, if you think voting alone can change things then you're missing the point. These policies would have been virtually the same under _any_ candidate with a nonzero chance of winning. It's not like choosing Obama in particular makes a difference. Same thing for most congressional candidates.

</comment></comment></comment></comment><comment><USER11> Somebody hook a generator to Kafka, he's probably putting out enough RPMs to power New York.

</comment><comment><USER12> It is inconceivable that a democratic society ruled by law would even have a notion of secret law, secret courts or secret interpretations of law as it is inconceivable that I would ever get a constitutionally guaranteed fair hearing. We are falling down the rabbit hole.

</comment><comment><USER13> A law that cannot be known is no law at all.

</comment><comment><USER14> It seems that the reason for the ACLU's request being declined is jurisprudence, which is a weak argument. The court did not directly explain why it is unreasonable to request such information (claiming national security doesn't necessarily mean it is so). Since 2007 and 2008 many things have come to light, including how deep their operations go and which companies (in part) are being used for surveillance. Therefore more evidence has come to light and the ACLU is pretty much just being dismissed in my opinion without justification or at least without a valid reason (even if it is justified).

Legally, I think the court is "sound," but that does not mean they could not throw the ACLU a bone by just releasing the most basic of their law interpretations (since we already know them).

</comment><comment><USER15> "The Justice Department filing also, however, suggests the Obama administration is taking steps to make the court more transparent."

Wait, it's a court. How does the executive branch have any sway over how a court manages itself? Unless it's a captive court ...

<comment><USER16> Or a secret court</comment></comment>


Tell HN: I'm giving away a sample dataset -- URI scheme frequency - <AUTHOR>

Hi everyone,
I've released a small, sample dataset based on a 7.5MM page crawl. The dataset lists the frequencies in which URI schemes appear in the anchor element's href attribute. The list was truncated at 300 URI schemes (for this sample) and is available in TSV format. The columns are: uri_scheme, frequency, unique_pages, unique_domains.
http://webscaled.com/static/samples/uri_scheme_part-00000.tsv
Please keep in mind that this is rough data. This file came straight from a MapReduce job. The values have not been classified (as they will be in the full version of the dataset), and a few are just people attempting to write "http:" but failed.
You are free to do whatever you please with this dataset. I just ask that you reference http://webscaled.com/ somewhere. Also, send me an email from the Webscaled contact form if you're interested in the sample dataset for top Doctypes.
Edit: Probably best to release under a license. Creative Commons Attribution-Share Alike 3.0 United States License.
Thanks for your time <comment><USER2> I've looked at the data, it seems that 'rough' is a pretty good description, cleaning it up shouldn't take more than a couple of minutes though.

Most of the misspellings are for http and for mailto (with matilto as the funniest), and there are a bunch of entries that are clearly nonsense and can be lost without any penalty.

aol: is listed 75 times, one could only wish that it would be that easy to get rid of aol :)

<comment><AUTHOR> Certainly, I could have eyed it and removed a ton of the crap. I figured most people would be interested in the top 30 or so anyway, and this also serves to give people an idea of just how dirty the Web really is. ;)

The final dataset will contain MUCH less noise.</comment></comment>


Ask HN: How did you significantly reduce your AWS cost? - <AUTHOR> One of the most obvious things is to better utilize your Reserved Instances, but what are some things that are not so obvious? Do you use any tools? Thanks! <comment><USER4> I'm a consultant in AWS, mainly doing cost optimization.

My favorite that most people miss is the S3 VPC endpoint. Putting an S3 endpoint in your VPC gives any traffic to S3 it's own internal route, so it's not billed like public traffic. I've seen several reductions in the 20-50k a month range with this one trick.

Otherwise, stop doing baremetal processes on the cloud. It's dumb. For instance, I see people operating a file gateway server (with 5 figure a year licensing costs) on ec2 using EBS storage. This is a perfect use case for replacement with S3, with lambda code running in response to whatever file gateway event you're monitoring.

Lastly, you need to really challenge people's conceptions about when resources need to be in use. Does a dev/test site need to be running the 16 hours that people are not at work? Of course not, except when people are staying late. So you create incentives to turn it off, or you run it on something like ECS or kubernetes with software to stop containers if they're not actively used in a 15 minute window (and then the cluster can scale down).

<comment><USER5> Great post. Something I'd add is if you can use tags and reporting to incur chargeback pain in internal budgets, it's a great motivator to not waste internal resources. We've saved six figures a month in AWS spend by doing this.

</comment></comment><comment><USER6> Tactically: spot instances, reserved instances, auto-scaling groups, deleting data that we didn't need.

Strategically: I've only ever see it work by making it somebody's job to save money on Amazon, ahead of feature velocity. Find a tool that analyses where you're spending [0], pick the biggest bucket of spend you don't really understand, and drill down until you're sick of looking at it. Make some optimizations around how you're spending that money, rinse, repeat.

Every org I know has done a first pass that's pretty effective, where you buy a bunch of reserved instances for the compute power you need. A couple of the companies I've worked with stalled out at that point. The others figured out real cost / benefit models for real projects and did things like move data from S3 to Glacier, or from hosted MySQL to RDS, or from Cassandra to DynamoDB.

There are only so many free-lunch cuts you can make, so it's worth your while start down a path that allows you to consider trade-offs, which includes empowering somebody to lobby for them.

[0] Seems nice, haven't used it: [https://github.com/Teevity/ice](https://github.com/Teevity/ice). I know the folks at [http://cloudhealthtech.com](http://cloudhealthtech.com), they're building a pretty solid business if you're willing to spend money to save money.

</comment><comment><USER7> The HFT Guy has a good blog and did some posts on this. Basically Google Cloud is much cheaper than AWS so you should look at that. There are also other alternatives including IBM SoftLayer which I've never looked at before.

"Run an entire tech company in the cloud, or run only a single [big] project requiring more than 10 servers? Google Compute Engine

Run less than 10 servers, for as little cost as possible? Digital Ocean

Run only beefy servers ( > 100GB RAM) or have special hardware requirements? IBM SoftLayer or OVH"

[https://thehftguy.com/2016/06/08/choosing-a-cloud- provider-a...](https://thehftguy.com/2016/06/08/choosing-a-cloud-provider- amazon-aws-ec2-vs-google-compute-engine-vs-microsoft-azure-vs-ibm-softlayer- vs-linode-vs-digitalocean-vs-ovh-vs-hertzner/)

[https://thehftguy.com/2016/11/18/google-cloud- is-50-cheaper-...](https://thehftguy.com/2016/11/18/google-cloud- is-50-cheaper-than-aws/)

<comment><USER8> I like his summary however, on the low end, I found OVH is more affordable than DigitalOcean depending on your needs. Digital Ocean is $20/month for 2 GB of RAM while OVH is $28/month for 7 GB with roughly similar specs for the rest (SSD, etc). Note that OVH also offers VPS SSD with 8 GB for $13.50/month however it doesn't automatically renew (at least in Canadian region) so you have to manually pay your bill each month. The $28/month option is under their public cloud offering and supposedly does auto renew.

You can also scale up memory at OVH without scaling up the rest of the server. So you can get to 15 GB of RAM for $34/month (at Digital Ocean, that is closer to $120/month or $160/month with much better specs for rest of server). But if all you need is more RAM...

Digital Ocean is a lot more user-friendly though than OVH. If you need good support, DO is probably a better bet.

<comment><USER9> On the low end, Linode is also an option: 2GB/$10.

<comment><USER10> Don't forget Vultr

</comment></comment></comment></comment><comment><USER11> We stopped storing our IoT raw data in databases. We still need to search it, but now we store only metadata in the database (we know what we will search by, so we can make appropriate metadata) and store the raw data in S3. So any searches are in the DB, using the DB for what it is good at. This means that our storage / database cost approaches just the S3 cost, because our metadata is ~0.001 the size of the raw data. It took our product from "we are going to shut this thing down" to making money.

Got the idea from an AWS talk here [https://youtu.be/7Px5g6wLW2A](https://youtu.be/7Px5g6wLW2A). Blew my mind. I coded up all the changes in a day. Took way longer to move the data than to code it.

<comment><USER12> How slow is it to retrieve the data from s3 though?

<comment><USER11> It is slow, but most of the time, we don't use the raw data - we use the metadata. It worked great for our app. We got a excited and tried to implement some on the fly data summaries where we would have to touch a bunch of files every time we wrote anything. That just didn't work because of the speed.

Let's be specific on speed. Most of the time, the MB or two needed for a plot is a fraction of a second, which our customers can deal with. That retrieval is a single object from S3, the way we organize things.

Having said that, the talk I linked to has some great advice - use the "folder" structure to write data so you don't search, you just use the naming scheme to do a direct object read. In addition, we can keep most reads to a single object, which is fairly fast.

As is always true, you will need to test to see if it fits your speed needs. But even with just the naming scheme and meta data in the database to eliminate all searches on S3, the speed works for us.

</comment></comment></comment><comment><USER13> After cleaning up all our unneeded instances and optimising our architecture, we finally realised that our developer stacks and test environments didn't need to run 24/7, which has so far been the biggest single cost saving we've implemented. By only running stacks from 8am-8pm then stopping everything, we're getting a 50% saving on our instance costs. If a developer needs a stack they can still launch a new one or start their existing one. We also moved a few test stacks to use single AZ modes instead of multi AZ for another significant saving.

Dialing down our snapshots in non production environments was also a great help to cut our costs.

</comment><comment><USER14> We wrote about how we analyzed our AWS usage [https://segment.com/blog/spotting-a-million-dollars-in- your-...](https://segment.com/blog/spotting-a-million-dollars-in-your-aws- account/), and then some of optimizations we made to cut down costs [https://segment.com/blog/the-million-dollar-eng- problem/](https://segment.com/blog/the-million-dollar-eng-problem/).

<comment><USER15> This is an awesome breakdown!

</comment></comment><comment><USER16> I had lots of DNS zones in Route 53 that didn't really need to be there. So I moved a bunch of them to ClouDNS, which supports ALIAS records to have apex domains point to CloudFront distributions. Seems to work.

Also I realized that DynamoDB autoscaling is relatively expensive for a large number of tables, because it sets up multiple CloudWatch Alarms for each table. So I turned off autoscaling for tables that don't really need it.

I have started to replace DynamoDB tables with Cloud Directory instances in cases where I don't want to have the fixed cost of tables sitting idle (Cloud Directory is charged by request). But this takes a lot of work to do retroactively and Cloud Directory doesn't have a nice CRUD UI, and of course it only fits projects that work well with a graph database.

These are not big costs in absolute $$$, but for personal projects they can be significant when you want to avoid piling up fixed monthly costs to keep services running.

</comment><comment><USER17> For RDS, it is much cheaper to pay for more storage than more IOPS, at least if you are under 3k IOPS.

IE, you will save a bundle if you are currently buying 3k provisioned IOPS vs just paying for 1 TB of storage.

More here: [http://blog.textit.in/why-buying-provisioned-iops-on-rds- may...](http://blog.textit.in/why-buying-provisioned-iops-on-rds-may-be-a- mistake)

<comment><USER10> It's for EBS in general, if your application is not IOPS intensive, use large gp2 volumes instead. We've had stress test wrong and chose provisioned volumes then have it to revert back to general volumes when in production.

<comment><USER18> You may find FittedCloud automated/transparent EBS optimization useful - [https://www.fittedcloud.com/solutions/aws-ebs- optimization/](https://www.fittedcloud.com/solutions/aws-ebs-optimization/)

</comment></comment></comment><comment><USER19> Shameless plug here. Working on an side hustle to help you monitor and measure your AWS spend: [http://www.cloudforecast.io](http://www.cloudforecast.io)

You'll get a daily report that breaks down your cost for you via inbox with weather symbols indicating the health of your cost. Think of this like the weather you check every morning before you start your day. Our forecasting in our report is pretty accurate too!

Hit me up if anyone is interested with helping us and provide product feedback. We're offering a 60 day free trial for our first users so we can evolve from a reporting tool to a tool that help you save on AWS cost.

Would love to hook you all up: firstname.lastname@example.com

This post has been super helpful to figure out what we should start thinking about and what to build next!

</comment><comment><USER20> I think the first option is to go to AWS and tell them that. They can discount prices + have staff to help you reduce your bills. I'd start here actually

<comment><USER21> How come every time I try, the response from AWS is abysmal? I've been told this many times, and try to reach out, but it is always a pitiful case of broken telephone.

<comment><USER20> It depends on your budget. They start caring about >10K$/month I think.

Also if you're a startup, you can get usually 100K$ of free credit

</comment></comment></comment><comment><USER22> Rather than spot instances, use spot fleets and structure your bid such that you arbitrage across different node types and data centers that are equivalent for you.

We cut our spot expenses by 20% , which were already cheaper than on-demand by about 70%.

No matter how we played with the price calculator, Google was twice as expensive than our spot fleet algorithm

<comment><USER23> Spot fleets are really powerful but they take a lot of parameters and are quite hard to get configured right.

But running standalone spot requests is indeed much worse.

Autoscaling groups come much more natural to people, it's just that the native spot implementation is unreliable.

I mentioned before in this thread that I implemented a tool called autospotting that allows your on-demand autoscaling groups to be automatically converted into a sort of spot fleet, by replacing their members with the best available spot instances.

You get the best of both worlds: easy configuration based on your group settings, so you don't have to worry about bid prices, instance types and weights: the bid price is automatically set to the on-demand hourly price of your original instances, the instance type is also automatically determined based on the original instance type, so you can get any other type that's at least as large, even from a different generation, the selection is currently based on the lowest price, so you will pretty much automatically get various types without explicit configuration. When no spot instance types are priced lower than the on demand price the group will happily run the initial on demand instances until eventually some become available for a better price.

Gradually these spot instances are launched and attached to the existing group, and on demand instances are terminated to keep the capacity constant.

Unlike the spot fleets, this supports out of the box anything that is backed by autoscaling groups, such as Elastic Beanstalk, Kubernetes clusters built using kops, environments managed by CodeDeploy, and so on.

And unlike spot fleets, the migration to spot and back is a matter of setting a tag on the group, so you can easily revert back to the original group configuration if you decide to.

</comment></comment><comment><USER7> Buy your own hardware. Esp for dev work having your own rack in the office really is cheap and its kinda nice. The other benefit is it limits the budget to what we have, rather than allowing anyone to create more servers...

</comment><comment><USER24> 1\. Use RI's (like you said) and make sure your utilization is high.

2\. Look at any under utilized EC2 instances and make them smaller. If they are very under utilized, consider moving what they do to Lambda, which is very cheap.

3\. S3 for everything storage. Only use EBS volumes when you need to.

4\. Someone mentioned multiple availability zones and it's a great tip. Turning it on is 2x the price for RDS, and it's probably not needed for anything but critical instances.

5\. Make sure unused instances are terminated, not stopped.

6\. For very expensive systems, use spot instances to strategically reduces your costs when ec2 instances are cheaper.

7\. Try to keep your instance types down to a few types. This will give you fewer headaches when you go to purchased reserved instances.

</comment><comment><USER25> What helped when we first got started is more quickly terminate test/development environments. We'd often launch a full environment to test out a theory or do a POC without touching the regular development, staging and production environments. It would often happen that those environments would run a little longer than supposed to. We've put automated systems in place that detect inactivity and notify us.

This only helped cut our bill before we launched and started spending more and more money on our production environment when these kind of things still made up a significant amount of our a bill.

</comment><comment><USER26> I use AWS and online.net. I use the latter for 8 core bare metal servers that cost me 16euro a month. AWS for the rest (at work)

We need a couple of things:

1\. A nice easy way to distribute load through cheap bare metal providers with peak load taken by cloud. Kubernetes is nearly there.

2\. Somebody to build a LCD display that sits in the office and shows the projected cost for cloud deployment at current usage. It's so easy to throw up a 40 node cluster and not think about the cost - especially where I currently work, a government funded project, where the taxpayer will pony up.

</comment><comment><USER27> Over the years we have done a number of things, probably the most impactful was auto-scaling on our core engines. Whenever we scale down it's just like scooping money out.

</comment><comment><USER16> Amazon Lightsail instances can provide cheaper bandwidth. The $5/month subscription includes 1TB, while 1TB of outgoing data on EC2 would cost $90 ($0.09 per GB).

(This is such a big difference that I don't really understand it.)

</comment><comment><USER28> __Blueprint your Footprint __

Observe your AWS footprint. Dump it as CloudFormation, or hook up a tool like[http://hava.io](http://hava.io) or [http://www.visualops.io/](http://www.visualops.io/) to get a spatial representation of your infrastructure.

They both have powerful costing tools. Use this data in conjunction with your baseline performance metrics (memory/CPU/disk consumption) and see if there are any broad savings to be made by refactoring your platform.

If you have been steadily adding components (and their SG dependencies), its likely there is no clear understanding of the End-to-End scope of your AWS footprint.

From all this data comes your Blueprint. You choose what's necessary; VPC, ELB, AZ, RDS, S3, R53 and so on.

Use something that is not the AWS console to define and capture a new, blueprinted environment and deploy it. See if it does the job by running your availability and performance suite on it.

Start doing Blue/Green Deploys so your costs are reflective of your application.

</comment><comment><USER23> Shameless plug: if you are using autoscaling and have stateless/12factor components, you would like to try spot instances but don't feel like spending time and much effort with a migration to spot fleets, just give autospotting a try.

It will continuously replace your group's on-demand instances with the best priced compatible(at least as big and identically configured) spot instance types, with minimal configuration changes performed by simply tagging the group. The group configuration is unchanged so it would keep launching on demand when scaling out or when replacing outbid spot instances.

The tool is open source and available on github and can be set up in a few minutes: [https://github.com/cristim/autospotting](https://github.com/cristim/autospotting)

</comment><comment><USER29> > How did you significantly reduce your AWS cost?

We moved to Google Cloud Platform.

Snark aside, we're continuing to be pretty happy about that decision, in addition to paying a lot less.

</comment><comment><USER30> AWS Trusted Advisor, gives great metrics on underused resources... Sadly, part of Premium.

[https://aws.amazon.com/premiumsupport/trustedadvisor/](https://aws.amazon.com/premiumsupport/trustedadvisor/)

</comment><comment><USER31> By switching to Hetzner GPUs (99 euro/month for a server with an NVIDIA 1080)

</comment><comment><USER32> Serverless / Zappa, hosting my API's for 10x decrease in monthly costs.

[https://github.com/Miserlou/Zappa](https://github.com/Miserlou/Zappa)

<comment><USER33> We are in the process of converting over to this as well

</comment></comment><comment><USER34> Being extremely aggressive about cleaning up/terminating/scheduling dev environments have led to tremendous cost savings. Also, monitoring those instances and downsizing them has helped a lot too.

These are usually the quickest and most common solutions to cost problems that I've seen. After that, you look for "minor gains:" storage (S3 instead of EBS, Glacier instead of S3 for infrequently accessed data), using reserved instances, moving to RDS (though reliances on underlying OS can make this difficult and many databases have them), autoscaling, running ALBs/ELBs instead of EC2 instances running HAproxy, CloudWatch alarms for things that exceed cost parameters, etc.

One of the biggest cost issues that I've seen is people treat AWS like an internal PaaS cloud (like vSphere) and move their ways of maintaining bare- metal hardware/software into AWS (i.e. "lift and shift"). It saves time in that you don't have to actually think about how your application works in a cloud world and gives managers that "I did the impossible and moved us to the Cloud in three months" badge/bonus, but it costs SO MUCH MORE to run and really does a disservice to what AWS can do. (Cynically, I guess this creates another carrot: "I did the impossible and saved us tons of money by fixing the cost explosion from lift and shift!")

Breaking that culture down is paramount to efficient usage of any cloud in a way that won't break the bank.

</comment><comment><USER35> There's no one silver bullet, as always, you just have to get really good at using Cost Explorer.

Project 1: We reduced our EC2 bill by about $10k a month by moving to spot instances for our Continuous Integration servers

Project 2: We saved another $10k a month by moving to CloudFront for our downloads, instead of using S3 directly

Project 3: Finding and eliminating waste from years of tech debt. Saved $5k a month so far.

As we go we need to spend more and more time to find the same amount of cost savings, it's a law of diminishing returns.

</comment><comment><USER36> Spot instances. Spot instances. Spot instances.

<comment><USER37> This, this, and this again.

I'll also add, do not keep large EBS volumes around. If you need to store data forever, get it moved to S3.

</comment></comment><comment><USER38> If I had to significantly reduce my AWS cost, I'd turn to Corey Quinn of "Last Week in AWS" ([https://lastweekinaws.com](https://lastweekinaws.com)) who fixes AWS bills for a living ([https://www.linkedin.com/in/coquinn/](https://www.linkedin.com/in/coquinn/))

</comment><comment><USER39> Docker. ECS, Kubernetes, Mesos, whatever floats your boat. Everything on RIs. Then start playing with spot-instances.

</comment><comment><USER40> I didn't see this one mentioned.

Use VPC end points for S3 to save cost.

If you are using NAT gateways and S3 (probably applies to Dynamo as well) and your app is using S3 then you could be saving $$ by having that traffic run through a VPC endpoint which is free bandwidth and not using the NAT gateway which is paid.

</comment><comment><USER41> This is totally just an idea, and not something I have ever seen put into practice (not worked anywhere that used AWS), but depending on your expense, hiring someone who is specifically skilled at objective performance measurement and optimization could pay for itself I would imagine.

</comment><comment><USER42> We developed a continuous scalability testing technology that lets you meet the concurrency and throughput requirements of your microservices build, before they ship to production.

Our community uses this information to reduce the waste at the software, middleware and AWS level. e.g. configure Auto Scaling groups to scale upon reaching a custom ELB Target Trigger equal to your measured concurrency.

I wrote an article about this specific ELB use case a couple weeks ago, at [https://medium.com/@stacktical/how-to-lose-money-with-the- ne...](https://medium.com/@stacktical/how-to-lose-money-with-the-new-aws-elb- network-load-balancer-e87e670c05ea)

Maybe it can give you some ideas.

</comment><comment><USER43> \- For consumer compute consider serverless (api gateway and lambda) - <USER2> addresses the problem of idle/pre-prod environments. \- Data at rest: consider s3, retention policies and glacier. \- Consider aggressive retention policies on cloudwatch logs, watch out for unnecessary cloudwatch metrics (I still prefer statsd because of the percentiles) \- If you use Kinesis - <USER3> In/Out bytes and provision shards as needed, control retention. \- Watch out for cross region/hybrid (on prem) stacks - data transfer is very expensive and not obvious. \- Use reserved instances. \- Remove unused EBS volumes. \- Watch out for high IO EBS volumes. \- Use Cloudfront to cache static content.

</comment><comment><USER44> Shameless side project plug

I'm working on [https://www.dormantbear.com](https://www.dormantbear.com) which is an aws scheduler to turn instances on and off for certain times of the day.

I use it for my own VPN box which I only require during office hours.

The main commercial use case is for turning off staging environments when not needed during out of office hours. For setups that are fixed and don't spin up test environments on the fly.

For security you should provide aws creds that can access your staging boxes with permission to list instances and turn them on and off. Even though they are symetrically encrypted.

The service is free as I have not written billing or marketing pages yet.

</comment><comment><USER45> * Use spot instances whenever possible, especially for development. This can sometimes cost more time than it's worth, but it can turn out quite useful, especially for broad QA testing across many nodes.

* We (Userify[1] plug: SSH key management) identified our most expensive activity (in our case, ~70% of our bill was simply bandwidth) and then switched to external or third-party resources where possible, but..

* _After_ we went through all of that effort, we mentioned it to our account manager who worked with us to resolve it; wish we'd just started there and saved a good bit of time!

* Switched historical real-time data storage to archival mode (ie a quick script to extract old data, compress, and save to S3). This saved a lot in Redis/Elasticache memory fees which were growing extremely quickly.

* Disabled detailed monitoring on autoscaled instances where we didn't need higher-resolution data.

* Removed NAT instances from VPC's where they weren't needed.

* Found and retired old snapshots and volumes (those are hard to figure out, but this exercise is worth pursuing, since they seem to multiply like rabbits). Far less expensive but easier to track down are unused Elastic IP's and Route 53 zones.

* Don't launch an instance without tagging it, if at all possible. You should use tags to group instances (and other resources) and provide some way to track down the instance owner. Often people are reluctant to turn something off because they don't know if it's still in use. _If it 's critical, it should be clearly labeled so._

* Look through seldom-used accounts and regions for left-over instances that are still running. Turns out there's often a lot of these.

* Look at CPU/RAM/IO etc for larger instances and decide if it can be reduced without affecting user experience. This is especially effective with dev or internal boxes.

* Stop (but do not terminate!) all dev/non-critical boxes when outside normal business hours. This can be done with a route 53 record for the box (to prevent needing EIP for each one) and a tag that indicates the box can be stopped when needed. It's amazing how many project boxes just keep running and costing $. This is also the case if you're not sure if something looks unused and can be terminated safely; stop it and see if anyone screams and add a tag with your contact info. (corollary: make sure critical boxes have uptime monitors, cloudwatch, etc!)

* Don't front-end static resources with Cloudfront if latency and scalability are not going to be an issue for them, to reduce bandwidth/tx/region costs by at least half.

* Try to avoid cross-region (or cross-account in the same region) or even cross-AZ bandwidth charges. Avoid multi-AZ builds unless critical infra.

* Watch the per-transaction fees on especially things like Lambda, but also frequently-updated or written items on S3

* Pay attention to those tiny fees that come along with certain types of AWS technology that can add up huge over time. (esp DynamoDB and Lambda; however, these can be awesome for fast-to-develop low-utilization projects)

* Use classic ELBs instead of ALB's if you have a lot of short sessions instead of fewer long-running sessions. Also become familiar with NLB's[2], which are very useful in some circumstances.

* NAT's can be a very significant (esp in bandwidth) and hard-to-surface cost... Delete them if you don't need them.

* Learn CloudFormation[3] and use it to build clearly labeled dev and production stacks. The great thing about this is that when you're done, you can just delete the dev stack (which cascade deletes all of the dependencies) and launch a prod stack in a different account. CloudFormation has gotten a LOT easier to use with YAML support, and a LOT faster at spinning up. Start with something small and work up from there.

* Keep the prod account separate (and backed up as offline as possible, preferably on your premises, or in another cloud). Only let a few trusted people manage prod, and anything outside of prod is fair game to get turned off. Ideally, give each dev their own sandbox account, and use tools like Userify (plug!) and third-party IAM roles.

* Lastly, if costs are really too high for you, also look at AWS Trusted Advisor, third-party tools like Cloudability or Teevity Ice, and of course talk with your account manager after gathering data about where your biggest costs are. They have a lot of power to help you and they often _really_ do have your best interests at heart.

1\. [https://userify.com](https://userify.com)

2\. [http://docs.aws.amazon.com/elasticloadbalancing/latest/netwo...](http://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html)

3\. [https://aws.amazon.com/cloudformation/](https://aws.amazon.com/cloudformation/)

</comment><comment><USER15> The thing with AWS is that you have to first measure your costs in order to reduce them - which isn't always obvious before the bill arrives.

What are some of the big frustrations people have found when it comes to monitoring these costs?

</comment><comment><USER46> That depends a lot on your application, that said, i'd like to add CloudFront. The more you respond from the CDN cache, the less requests hits your backend and that can be huge, it is not unusual to offload 80-90% of traffic.

When it hits the backend, if it is Lambda functions you can also save a lot by ensuring you don't have wasted resources. Same idea for databases, DynamoDB is very cost effective. Usually the less you manage or provision by hand, the more you save.

</comment><comment><USER47> You don't use AWS in the first place. Most projects are better off going with bare metal and use spot instances for spikes. Will cut your costs dramatically.

</comment><comment><USER48> This blog post is straight forward and useful: [https://www.datawire.io/4-simple-strategies-for-managing- you...](https://www.datawire.io/4-simple-strategies-for-managing-your-aws- costs-with-reserved-instances/)

tl;dr: Configure your reserved instances to be region-scoped; buy in a popular region; pay-as-you-go not upfront; sell your excess reserved instances (like you mentioned)

</comment><comment><USER49> You may want to check out FittedCloud (www.fittedcloud.com). It identifies potential cost savings opportunities using machine learning and provide optimization recommendations in form of actionable advisories for most of the AWS resources. Actionable advisories are one click to automation for the provided recommendation. They have a free version with limited capabilities and a paid version for full capabilities.

</comment><comment><USER50> Maybe 'engage' with the AWS sales team?

Like I am doing...I have £300K of annual cloud and dedi hosting costs spread over several providers so I reached out to the sales team last week. I'm sure they'll be rushing to call me any day now - I just hope it's not this afternoon as I'll be on a booked call with Google.

</comment><comment><USER51> Depending on your spend, the specialists over at www.quinnadvisory.com should be able to sort you out quickly.

</comment><comment><USER52> When you can't use reserved instances... switching to GCP (with their automatic discount) saved us ~50%.

</comment><comment><USER53> So, aside from using reserved instances...

1\. Autoscale with triggers based on utilization. 2\. Don't use a full OS if you don't need, consider containers that are extremely targeted with your needs only. 3\. Services help, you don't need to build everything on your own.

</comment><comment><USER54> 1\. Using Cloudflare as a CDN of s3 to reduce traffic cost. 2\. Using Cloudflare Firewall instead of AWS WAF. Cloudflare only charge for passed traffic. AWS WAF charge for both passed and blocked traffic (I have confirmed this with aws billing department)

</comment><comment><USER55> Cross-AZ fees can add up!

</comment><comment><USER56> Big instances w/ lots of containers and putting everything in cloudformation/terraform so that you can nuke non-prod environments at certain times.

</comment><comment><USER57> Check your cloudfront and data transfer usage. We had bingbot overcrawling a number of our urls resulting in high cloudfront costs.

</comment><comment><USER58> Small gain: Deleted all snapshots that were accumuated over 6 months across 100 instances. (Generated by Cloudwatch rules).

</comment><comment><USER59> I compressed some data stored in S3 in a lossy way to 2% of its original size, saved a bundle.

<comment><USER60> What did you lose? We've seen some interesting things like lowering precision of timestamps and gps have large impact because zip doesn't work as well on basically random numbers.

<comment><USER61> Never thought of that, something to keep in the very (very) far back of my mind for future usage

</comment></comment></comment><comment><USER62> Easiest, quickest and most flexible way is with www.parkmycloud.com or similar tools.

</comment><comment><USER63> Use Windows EC2 instances only as a last resort.

</comment><comment><USER64> Have a clear retention policy. You must differentiate between hot and cold data, it will help you minimize the size and thus cost of your high- performance cluster and the lower performance one.

</comment><comment><USER65> Re-architect for the cloud.

</comment><comment><USER66> I hate how people think this kind of garbage is worse than maintaining your own stack...

</comment><comment><USER67> Some great bits on here about effective cloud strategies

[http://www.venturi-group.com/podcast/](http://www.venturi-group.com/podcast/)</comment>


Farmville for Dummies - <AUTHOR> http://www.amazon.com/FarmVille-Dummies-Angela-Morales/dp/1118016963

<comment><USER2> Never before have I seen a book title so accurately describing the individuals who would buy it.

<comment><AUTHOR> Yep, it's like titling one "book for readers".</comment></comment>


Ask HN: Anybody live in government housing? - <AUTHOR>

I'm about to start a company and will draw little to no salary for 1-2 years.  I believe this qualifies me for government housing.  Has anyone gone down this route before? I live in Massachusetts, USA which apparently has many social programs. <comment><USER2> It's up to the local government to see if you qualify — although my guess is that's based on your previous income and other factors (like if you're a citizen or have a family). Even if you qualify there may be a waiting list — and my bet is that you couldn't really count on the quality:

<http://www.massresources.org/public-housing.html>

I once had a friend who was a blogger who lived in the projects here in NYC and the quality of housing was pretty terrible...

</comment><comment><USER3> I grew up in the projects which is government housing but I guarantee you don't want to move there. There are several types of government housing i.e.: Section 8, Projects etc... Did you want to know about any specific form of Gov. Housing?

<comment><USER4> Are they hard to get into?

<comment><USER3> Yes, and I am sure there will be a long waiting line.

<comment><USER5> I also think they don't open up applications frequently.

[http://www.ajc.com/news/atlanta/housing-crisis-reaches- full-...](http://www.ajc.com/news/atlanta/housing-crisis-reaches- full-589653.html)</comment></comment></comment></comment>


Ask HN: Any Hackers Love Sports? - <AUTHOR> First time, long time--do any hackers here love sports? I do. As a sports fan, I don't feel like there are many quality Apps on mobile devices. Twitter has enhanced how I watch television, sports included, but there's a lot of noise. There are a plethora of scoreboard apps, but where's the App that will revolutionize the game day experience for the fan? Is anyone interested in brainstorming, prototyping, and developing the home screen app for sports fans with me? If so, respond here and we can exchange contact information.<p>I have a strong development background in C++ / MySQL / PHP / JavaScript / some Objective C. I'm also somewhat networked with new sports media. An ideal co-founder would have a strong iOS and Objective C background, and loves sports. <comment><USER2> I love To Do Sports.

Every now and then a story on "Exercise/Sleep/Meditate well and you will reach the Nirvana of Human 2.0" comes up here at HN. But the thing is, that Sports are quite fundamental. According to Wikipedia: "Sport (or sports) is all forms of usually competitive physical activity". It's analogous to all "entrepreneurs" here doing "competitive mental activity", at least if you are in the same industrial sector. It's not exercising only, sports teaches you about sportsmanship, respect and fairness. That's what I look up to.

Heck, Hacking itself is even some kind of sport to me.

So, if you need somebody in order to raise the experience of doing sports: I'm all yours. ;-)

</comment><comment><USER3> I definitely love sports and am currently working on a side-project in that area. However, it is not an in-game experience. Hoping to do a Show HN soon to share it.

My feeling is that the "in-game" experience will be tough to crack. Call me old fashioned, but when I sit down to watch a game, I may have a second screen open, but it's rarely used to augment watching the game, it's mostly for multi-tasking.

</comment><comment><USER4> We have a community of Founders / Investors that started this year that love Ultimate Frisbee. Invite only events where we play / mastermind / do deals / hang.

Current members include Dave McClure of 500 Startups, Elliott Bisnow of Summit Series, Evan Reas of Circle and others.

[http://UltimateFounders.com](http://UltimateFounders.com)

</comment><comment><USER5> Big sports fan here. I enjoy watching the game and participating. I've been working on a side project for managing the social sport/rec league teams I play on.

The idea you raised is an interesting one. There are lots of really cool visualizations out there these days. I've always wondered how we could make them more useful during the games.

</comment><comment><USER6> We're a small team of football fans and we recently made a game in that area for iOS devices. It's called Football Heroes. We'd love your feedback!

[https://itunes.apple.com/us/app/football- heroes/id543009156?...](https://itunes.apple.com/us/app/football- heroes/id543009156?mt=8)

</comment><comment><USER7> I'm a big fan of sports, mainly soccer and basketball, but I don't have any background on iOS and Objective C.

During games I usually use a second screen to debate with my friends what's happening in the game or to share some betting tips.

In case you need something more feel free to ask me.

</comment><comment><USER8> Sounds interesting to me. I have a couple of side projects involving visualization and comparison of data from track and field events. My technical background includes Python, Java, Postgresql, Javascript. Unfortunately no iOS.

<comment><USER9> I run d1 track in college right now. Would love to talk about this with you if you want. firstname.lastname@example.com

</comment></comment><comment><USER10> I'm learning to code and building stuff at [http://www.insidefpl.com](http://www.insidefpl.com), a site dedicated to Fantasy Premier League Football.

</comment><comment><USER11> "love sports" like in "couch potato watching sports on TV getting fatter with beer, pizza and chips"? no thanks

</comment><comment><USER12> Saw this few days ago: [http://www.rivalrygames.com/](http://www.rivalrygames.com/)

</comment><comment><USER13> Wait until the Word Cup comes this year, I know many hackers that love Soccer.

<comment><USER7> A real soccer fan watches games during all year, not only in the World Cup :)

</comment></comment><comment><USER14> Some "hackers" love sports, the pope is catholic and water is wet.

</comment><comment><USER15> Sports are for idiot working class men.

<comment><USER16> If this sort of response is what you consider a "contribution" to this community, we're better off without you.</comment></comment>


Timeline of recent events at Wikimedia Foundation - <AUTHOR> http://mollywhite.net/wikimedia-timeline/ <comment><AUTHOR> GitHub repo of the page: [https://github.com/molly/wikimedia- timeline](https://github.com/molly/wikimedia-timeline)

</comment><comment><USER2> [http://i.imgur.com/b8N3qy3.png](http://i.imgur.com/b8N3qy3.png)</comment>