Execute SQL against structured text like CSV or TSV - <AUTHOR> https://github.com/dinedal/textql <comment><USER3> It is importing the data into SQLite and running the query there. I don't see any attempt at type detection.

SQLite has virtual tables where app provided code can provide the underlying data and you can then use SQL queries against it. This is by far the easiest approach for foreign data formats. [http://www.sqlite.org/vtab.html](http://www.sqlite.org/vtab.html)

I'm the author of a Python wrapper for SQLite. It provides a shell compatible with the main SQLite one invocable from the command line (ie you don't need to go anywhere near Python) - [http://rogerbinns.github.io/apsw/shell.html](http://rogerbinns.github.io/apsw/shell.html)

That shell has an autoimport command which automatically figures things out for csv like data (separators, data types etc)

sqlite> .help autoimport

.autoimport FILENAME ?TABLE? Imports filename creating a table and automatically working out separators and data types (alternative to .import command)

The import command requires that you precisely pre-setup the table and schema, and set the data separators (eg commas or tabs). In many cases this information can be automatically deduced from the file contents which is what this command does. There must be at least two columns and two rows.

If the table is not specified then the basename of the file will be used.

Additionally the type of the contents of each column is also deduced - <USER2> example if it is a number or date. Empty values are turned into nulls. Dates are normalized into YYYY-MM-DD format and DateTime are normalized into ISO8601 format to allow easy sorting and searching. 4 digit years must be used to detect dates. US (swapped day and month) versus rest of the world is also detected providing there is at least one value that resolves the ambiguity.

Care is taken to ensure that columns looking like numbers are only treated as numbers if they do not have unnecessary leading zeroes or plus signs. This is to avoid treating phone numbers and similar number like strings as integers.

This command can take quite some time on large files as they are effectively imported twice. The first time is to determine the format and the types for each column while the second pass actually imports the data.

<comment><USER4> I've written a log file viewer ([http://lnav.org](http://lnav.org)) that exports log messages as SQLite virtual tables. I was hoping this project also leveraged virtual tables since they're one of the coolest features in SQLite.

lnav tries to push things a little further and extract data from unstructured log messages. For example, given the following message:

      "Registering new address record for 10.1.10.62 on eth0."

It will create a virtual table with two columns that capture the IP address (10.1.10.62) and the device name (eth0). Queries against this table will return the values from all messages with the same format (Registering new address record for COL_0 on COL_1). See [http://lnav.readthedocs.org/en/latest/data.html](http://lnav.readthedocs.org/en/latest/data.html) for more info.

<comment><USER5> Very cool! Looking forward to trying this out today.

</comment></comment></comment><comment><USER6> Straight from 1998! Perl's CPAN Module DBD::CSV [http://search.cpan.org/~hmbrand/DBD- CSV-0.41/lib/DBD/CSV.pm](http://search.cpan.org/~hmbrand/DBD- CSV-0.41/lib/DBD/CSV.pm)

<comment><USER7> I actually wrote a small data collection script that used this to parse the data some time in 2008. Using DBD::CSV along with dbish: [http://search.cpan.org/~tlowery/DBI- Shell-11.95/lib/DBI/Shel...](http://search.cpan.org/~tlowery/DBI- Shell-11.95/lib/DBI/Shell.pm) would give you pretty much the same functionality.

</comment></comment><comment><USER8> Perl has a CSV driver for DBI: [http://search.cpan.org/~hmbrand/DBD- CSV-0.41/lib/DBD/CSV.pm](http://search.cpan.org/~hmbrand/DBD- CSV-0.41/lib/DBD/CSV.pm)

</comment><comment><USER9> Surprisingly, we've been doing this on the Microsoft side of things for years with ADO and csv data sources!

[http://msdn.microsoft.com/en- us/library/ms974559.aspx](http://msdn.microsoft.com/en- us/library/ms974559.aspx)

(article dated 2004)

<comment><USER10> Even better is LogParser, which lets you do SQL queries against just about any data format you'll find in a file on Windows. It's been around for a decade and continues to be improved.

[http://technet.microsoft.com/en- us/scriptcenter/dd919274.asp...](http://technet.microsoft.com/en- us/scriptcenter/dd919274.aspx)

<comment><USER11> Microsoft LogParser is not only an established utility, it's extremely fast, too.

<comment><USER12> I remember LogParser from years back. I used to be very involved in that nascent community, and I developed a plugin for it that let you parse an input file using a regex with capture groups so you could use SQL queries on log files that did not have records with simple delimited columns.

</comment></comment></comment></comment><comment><USER13> If you use PostgreSQL, consider also the "file_fdw" Foreign Data Wrapper module:

[http://www.postgresql.org/docs/current/interactive/file- fdw....](http://www.postgresql.org/docs/current/interactive/file-fdw.html)

</comment><comment><USER14> An alternative approach is Microsoft Log Parser. While it is old in internet time (2005), it works brilliantly on very large files. I've successfully used it on 10+ GigaByte files. The documentation is a little sparse but once you get the hang of it it's pretty simple. Clearly it only works on windows boxen.

Example: #Find all unique from col1 logparser -i:csv -o:csv -stats:off -dtlines:2000 -headers:off "select distinct col1 from input.csv" >out.csv

<comment><USER15> In general Microsoft provided an ODBC driver over flat files that would at least give you limited SQL query capabilities.

</comment></comment><comment><USER16> We use a format called Tabix[1] frequently in bioinformatics. It doesn't have a full query language, but is optimized for extracting regions from a text file and supports compression. It's pretty clever — it's based on a block compressed (BGZF) tab file with virtual offsets. An index allows for fast random access.

[1][http://samtools.sourceforge.net/tabix.shtml](http://samtools.sourceforge.net/tabix.shtml)

</comment><comment><USER17> I can currently do this:

        sqlite> create table mydata (id integer, value1 integer, description text);
        sqlite> .separator ","
        sqlite> .import mydatafile.csv mydata

I assume textql is basically just running a similar SQL query, but perhaps autodetecting column types for you? (Haven't looked at the code.) Is that the value proposition, or does it do something more?

<comment><USER18> [https://github.com/dinedal/textql/blob/master/main.go](https://github.com/dinedal/textql/blob/master/main.go) around line 154: appears to make TEXT cols for everything. SQLite is pretty liberal about types so some math works anyway:

      sqlite> select "3"+"3";
      6

though note:

      sqlite> select "24" < 300;
      0
      sqlite> select "9" < "10";
      0

(see [http://www.sqlite.org/datatype3.html](http://www.sqlite.org/datatype3.html) for more about SQLite typing.)

<comment><USER3> Your examples should be using single quotes for strings not double quotes. The latter are for quoting identifiers (eg if you named a table "select") and through an accident of SQLite history degrade to a string if no matching identifier is found.

\+ is the SQL numeric addition operator. || is string concatenation.

        sqlite> select 3 || 3 ;
        33
        sqlite> select 'a'+'b' ;
        0

<comment><USER18> Interesting, thanks.

</comment></comment></comment></comment><comment><USER19> I was doing this with MS Query and the CSV ODBC driver 15+ years ago.

<comment><USER20> Coincidentally that was the first thing I thought of when I read the title.

</comment></comment><comment><USER21> This is awesome, but please make that gif into a html5 video so the playback may be controlled.

EDIT: here's gfycat version: [http://gfycat.com/FlusteredFarApatosaur#](http://gfycat.com/FlusteredFarApatosaur#)

</comment><comment><USER22> I do something similar with Excel files using jq ([http://stedolan.github.io/jq/](http://stedolan.github.io/jq/)) and j ([https://npmjs.org/package/j](https://npmjs.org/package/j)):

        $ j -J sample_data.xlsb tbl | jq 'length'
        3
        $ j -J sample_data.xlsb tbl | jq '[.[]|.id]|max'
        3
        $ j -J sample_data.xlsb tbl | jq '[.[]|.id]|min'
        1
        $ j -J sample_data.xlsb tbl | jq '[.[]|.value]|add'
        18

jq syntax is very terse compared to the equivalent SQL

<comment><USER23> At the expense of being nearly unintelligible (or writable) without the documentation open. I don't think anyone was trying to make SQL more terse.

</comment></comment><comment><USER24> I do a great deal of work in DevOps and I have a text file full of scripts which come handy to do work like this during production issues.

Nearly every sysadmin/firefigher I know has command line perl/awk/sed magic text files full of this.

Having said that I would appreciate if something like this was available for XMLs. Having to deal with SQL designs force fully shoe horned into NoSQL databases so often these days. I frequently run into situations where I have to query data from XML dumps. Most of the time, I have to invent a Perl one liner that essentially does the job of a select query.

<comment><USER25> I've done this a few times with XPath from an ipython shell, using the lxml module to parse the XML and handle the XPath queries.

30-second XPath pitch:

        //foo/bar

This gets the 'bar' children from every 'foo', regardless of where it occurs in the hierarchy.

        //baz[@name='Alice']

Gets everything like: <baz name="Alice">

There's lots of more complicated things you can do, but these are the most frequent types of things I've had to do.

<comment><USER26> I have a command-line thing put together with HtmlAgilityPack that represents HTML in an XPath-navigable document. It lets me apply the same XPath queries to HTML, for screen scraping.

<comment><USER25> Oh, that reminds me: You can use XPath in Chrome with $x(). To get all of the links on the current page, open up the console and type:

        $x('//a')

</comment></comment></comment></comment><comment><USER20> Good to see that it's not using SQLite's own CSV parser, which the last time I checked didn't seem to support the standard convention for quoting fields that contain commas themselves ( _very_ common if your CSV stores text strings in some columns):

[https://www.sqlite.org/cvstrac/wiki?p=ImportingFiles](https://www.sqlite.org/cvstrac/wiki?p=ImportingFiles)

(They also seem ignorant of the existence of RFC4180 when they claim there is no CSV standard.)

</comment><comment><USER27> I end up parsing a lot of excels at work (not my choice). I saw this and thought it looked interesting - export Execl to a CSV and run a query there.

At the end of the day the problem is that non technical users don't fill in things properly. They have a space before a number so it gets treated as text. Or in the same column Excel spits out an integer as a float. Basically the things that a real database keeps in order for you, so unfortunately I don't see it solving my problems. Shame.

<comment><USER28> I was going to write something like this. From my experience parsing a lot of CSV files, what's really tough isn't parsing data out of a nicely formatted CSV file, but cleaning and correcting the junk that is written into it. Sometimes I'd have to pass 3 or 4 different "correctors" and checkers in order to get an acceptable CSV file.

<comment><USER27> I assume this is a common problem to be solved. Are there any best practices, or libraries, or techniques that I could look at to help.

At the moment I parse and update the parser every time a new error appears. Most of the time it seems to be excel presenting things to the user that appear correct when they are not.

I have even been tempted to write my own "data import grid". It would look like excel, have very little of the functionality, but could allow a developer to set rules on what can / can't be entered in each column.

</comment></comment></comment><comment><USER29> [http://hackage.haskell.org/package/txt- sushi](http://hackage.haskell.org/package/txt-sushi)

<comment><USER30> Also, the Haskell version parses the SQL, so no sqlite required.

</comment></comment><comment><USER31> On a related note there is a recently published paper exploring how to load CSV data into a database efficiently[1].

The authors get a loading throughput (including index creation!) of over 1.5 GB/S with multithreading. They don't seem to do any dirty tricks when loading the data as they immediately can run hundreds of thousands of queries per second on the data.

There also seem to be more interesting papers on the research project page [http://www.hyper-db.de/index.html](http://www.hyper-db.de/index.html)

[1] "Instant Loading for Main Memory Databases" [http://www.vldb.org/pvldb/vol6/p1702-muehlbauer.pdf](http://www.vldb.org/pvldb/vol6/p1702-muehlbauer.pdf)

</comment><comment><USER32> I suggest another approach:

textql -query "select count(*) from persons.csv"

...because csv files are basically tables. This would also allow you to do joins on multiple files:

textql -query "select p.name, c.name from persons.csv p inner join companies.csv c on p.companyid = c.id"

<comment><USER33> [https://github.com/dinedal/textql/issues/6](https://github.com/dinedal/textql/issues/6)

</comment></comment><comment><USER34> What's the benefit (or tradeoff) of doing this over some variation of "command line data science" of the type outlined in Greg's post [1]?

Would it theoretically be faster than the unix join/grep commands for small CSV files, the kind you get usually via email?

I love SQL and prefer it for data work of this type, but I'm not entirely sure of the benefit of this, since it looks like you're just reading it into SQL, which means having to format and normalize anyway?

[1][http://www.gregreda.com/2013/07/15/unix-commands-for-data- sc...](http://www.gregreda.com/2013/07/15/unix-commands-for-data-science/)

<comment><USER35> I think the benefit is that a lot of people are proficient with SQL and are not very proficient with awk. I think whether it will be faster or not depends on how complex your query/queries are relative to the amount of data that you need to import.

</comment></comment><comment><USER36> The R package sqldf together with backend packages RSQLite (the default), RH2, RMySQL and RPostgreSQL can be used in a somewhat similar manner automatically generating create statements, uploading referenced tables to a database and performing the specified SQL. Tables referenced in the SQL statement are assumed to be R data frames or external files depending on whether an R data frame or file description object of the same name exists in the R work space. The result is returned as an R data frame. See [http://sqldf.googlecode.com](http://sqldf.googlecode.com)

</comment><comment><USER37> Reminds me of something I wrote to import Apache logfiles into a temporary SQLite database:

[http://www.steve.org.uk/Software/asql/](http://www.steve.org.uk/Software/asql/)

</comment><comment><USER38> I do this with the unix commands 'join' and 'awk'

</comment><comment><USER39> H2 SQL has this functionality as well. You can specify CSV file at table creation. It can even mix regular tables and CSV. And it has nice web interface build in.

</comment><comment><USER40> Cool stuff, and it's written in go!

I made a simple ruby version of this a while back: [https://github.com/dergachev/csv2sqlite](https://github.com/dergachev/csv2sqlite)

Subsequently also I discovered CSVKIT, a python implementation which is probably more robust:

[https://github.com/onyxfish/csvkit](https://github.com/onyxfish/csvkit)

</comment><comment><USER41> I'd use this if it was in Python. For go, I don't have a compiler handy on any machine I might need this.

<comment><USER42> I created a very similar project in Python a few weeks ago, based on csvkit[1]. Check out the Gist[2].

[1]: [https://pypi.python.org/pypi/csvkit](https://pypi.python.org/pypi/csvkit) [2]: [https://gist.github.com/jdp/8447221](https://gist.github.com/jdp/8447221)

</comment></comment><comment><USER43> if anyone is interested there is a similar tool (called "comp") but the queries are expressed with list comprehension syntax. It also allows to join data from json and xml.

It has two modes: 1) as a commnad line

./comp -f commits.json,authors.txt '[ i.commits | i <\- commits, a <\- author, i.commits.author.name == a ]'

2) as a service to allow querying of the files through simple http interface

./comp -f commits.json,authors.tx -l :9090

curl -d '{"expr": "[ i.commits | i <\- commits, a <\- author, i.commits.author.name == a ]"}' [http://localhost:9090/full](http://localhost:9090/full)

or through an interactive console on [http://localhost:9090/console](http://localhost:9090/console)

disclosure: I'm a co-author, and happy to get a feedback or answer any questions :)

<comment><USER43> well and here is the link - [http://github.com/ostap/comp](http://github.com/ostap/comp)

:)

</comment></comment><comment><USER44> I'd be interested to see how well this scales with a multi megabyte CSV.

<comment><USER45> From what I can tell, this creates an SQLite instance each time it's executed, and creates it in memory unless told not to do so. So I'd guess it wouldn't scale well. Still really neat though!

<comment><USER46> Even then, it should scale reasonable well if one added a repl, and kept the database in memory. Probably would be fine then up until at least a couple gigabytes, depending on how well sqlite handles in-memory tables (probably very well).

</comment></comment></comment><comment><USER47> I have been doing this with the unix commands 'join' and 'awk'. For larger files I import it to a database (ex sqllite). I am not sure what advantages this tool provides.

</comment><comment><USER48> I use Linq with a few addon assemblies to do stuff like this.

</comment><comment><USER49> Looks like a light version of Optiq. [https://github.com/julianhyde/optiq](https://github.com/julianhyde/optiq)

</comment><comment><USER50> See also [https://news.ycombinator.com/item?id=7176323](https://news.ycombinator.com/item?id=7176323)

</comment><comment><USER51> A related project is this:

[http://www.steve.org.uk/Software/asql/](http://www.steve.org.uk/Software/asql/)

</comment><comment><USER52> Offtopic: is there a better implementation of 'cut' that handles field reordering and multi-character separators?

<comment><USER53> awk. :-)

</comment></comment><comment><USER54> You can do this with external tables in Oracle (and probably other db servers)

</comment><comment><USER55> Perhaps something similar could be implemented more directly in Bash.

</comment><comment><USER56> Slightly OT but how can I create an animated gif like that?

<comment><USER57> Screencast, then convert screencast to gif?

</comment></comment><comment><USER58> One can also import csv to R and use sqldf package.

</comment><comment><USER59> txt-sushi does this too. It doesn't use SQLite, but processes the files directly, and supports joins.

</comment><comment><USER60> Ever used Google Refine?

</comment><comment> guard-of-terra Why not simplify the syntax to allow textql 'select * from sample-data.csv where i = 0'?

Otherwise - already checking out, that's what I wanted for a while.

<comment><USER61> It seems that textql doesn't parse the query at all, it imports the CSV into an sqlite3 db and then runs the query against it. What you ask for would be doable but it would be much more complex (parsing SQL is not that trivial, and people would want to do more complex queries like JOINing multiple CSV files etc., so you can't just look for "from <filename>").

</comment></comment><comment> guard-of-terra Can we please have -tab instead of -dlm="\t"? Because really.

Also, without support for multiple tables & joins this is not very useful.

<comment><USER62> _Also, without support for multiple tables & joins this is not very useful._

Not even slightly true. Can we stop crapping on v1 releases? I'd rather a developer release early and often than hold their code back until it has every possible feature.

<comment> guard-of-terra It's a wonderful concept but I don't know if its development would just sit there. I guess I should offer pull requests.</comment></comment></comment>


Become a better programmer with programming challenges - <AUTHOR> http://macgyverdev.blogspot.com/2014/04/become-better-programmer-with.html <comment><USER3> I agree with several already-existing comments that programming challenges do not help you become a better programmer (assuming that you're hireable in the first place).

I will note, however, that they DO HELP with interviewing. The skillset is quite overlapping... artificial time-pressure, artificially contrived problems, cleverness is rewarded, etc. (But it's not entirely the same, because programming contests don't give points for half-working answers).

Also, IMO the best coding contest site is [http://code.google.com/codejam/](http://code.google.com/codejam/), because:

1) GCJ has unusually clear problem statements. UVa in particular is terrible for this.

2) GCJ supports EVERY programming language (unlike all others I know of except Project Euler)

<comment><USER4> I disagree strongly with that.

First, in all programming helps you become a better programmer, just in the same way as playing the piano makes you a better piano player. Even if you already know how to play the piano, you still need to put in the hours to improve (or even to maintain your skills). The more you program, the more parts of the act become total routine that you don't have to pay any attention to.

Of course programming contests or challenges might not be the optimal thing to do to increase your programming skills. But the comparison isn't to programming something else, it's most likely to somebody wasting time by playing a video game, reading blog posts, watching TV, etc. If a contest acts as a extrinsic motivator for somebody to program something, how can that be a bad thing?

In my experience it's incredibly common for someone learning a new language to lose steam quickly due to not having anything to write. We'll always give the advice of just writing some program they need. But amazingly enough many people just don't think of the world like that. To them a well structured problem set is just what's needed. It'll allow them to learn the new language, and learn something else on the side.

But you're also making the assumption that a coding challenge can't teach you anything useful which is blatantly untrue. I learned a lot about writing efficient data structures for a certain class of problems during the Netflix Prize. Likewise for solving some of the harder ITA software puzzles. I'd known the theory of stack smashing attacks on a very vague level, but having to properly figure out how to do (microcorruption) it was a completely different thing, incredibly fun and helpful. Likewise the Matasano crypto challenges -- learning to crack badly applied crypto made me acutely aware of ways in which I've badly applied crypto in the past.

I'll admit that there's probably nothing of practical value to learn from e.g. doing high level Perl Golf.

<comment><USER5> "First, in all programming helps you become a better programmer, just in the same way as playing the piano makes you a better piano player."

I think it's a good comparison. Programming contest would be a very specific part of piano playing, like playing scales super fast. It's useful, but still a very small part of what it takes to making music.

<comment><USER6> I agree that doing these challenges helps you play the scales super fast. But I think the important segue to acknowledge is that the fastest pianists don't necessarily make the "best" pianists in terms of the quality of the compositions they're able to churn out. I'm saying that being a super fast pianist is not a necessary and not a sufficient condition to become a great pianist. Does it help? Sure it does! It helps to move up and down the scales and play rachmaninov for the evening's entertainment effortlessly, but at the end, it becomes one of many ingredients required to actually "make" good music.

The question is, is there a really good balance that we could engage in? What part of lives are best spent practicing heavily (and perhaps "composing" lesser music) and what's the best indicator to transition into a part where you're not practicing as heavily, but you're composing more music.

</comment></comment></comment></comment><comment><USER5> Those programming challenges are certainly interesting and they make a great learning tool but they use a very specific set of skills. Usually, the difficult part is to find an algorithm that solves the problem within the given constraints. Implementation is comparatively straightforward.

To me, they are more "problem solving contests" than programming contests. The code is just there to validate the solution.

<comment><USER7> But that IS the gist of development - figuring out how to solve a problem efficiently. Writing code is usually the easier part.

<comment><USER8> Depends on task. I found out that when dealing with large systems, it is not an algorithm that is the biggest problem. The gist is usually to find out and keep in head how various parts interact and what exactly they do. The hardest and most important is to figure out how to make each change in such a way that above will be as simple as possible (eg. keep the system clean and maintainable without hacks and dirty tricks).

Both parts (figuring and writing) are figuring out, but both are entirely different from programming challenges. First requires you to be able to decipher someone elses code reliably and fast. Second requires you to figure out how to do things as little fragile as possible.

Algorithms are more about solving isolated more math-like problems. Once you got solution, you do not need to consider anything then making it work, so it is different kind of code. Even if you would plug in it into some large system, it is would be isolated so you can actually afford to be more dirty.

<comment><USER9> Do you see this being analogous to where systems testing versus unit testing are most appropriate?

</comment></comment></comment></comment><comment><USER10> And also [http://codingbat.com/](http://codingbat.com/) somewhat basic. [http://www.pyschools.com/](http://www.pyschools.com/) (mostly basic python). [http://www.hackerrank.com](http://www.hackerrank.com) all levels , many ML/AI challenges. [http://www.codewars.com/](http://www.codewars.com/) quite large set of problems.

</comment><comment><USER11> I actually run through the first few Project Euler [https://projecteuler.net/](https://projecteuler.net/) problems whenever I am looking to play with a new language. It was really eye opening comparing my answers in Haskell to my answers in the procedural languages.

<comment><USER12> And that's the winner in my book - this is not about doing a test or challenge once, and passing some "exam". Its about stretching me and discovering more about the strengths and weaknesses of certain languages and environments - <USER2> should do more little practises like this, but in different ways.

</comment></comment><comment><USER13> There are many sites like this, with different focuses:

Project Euler (mathematical problems): [https://projecteuler.net/](https://projecteuler.net/)

Rosalind (bioinformatics): [http://rosalind.info/](http://rosalind.info/)

Are my two favorites. There are others, of course.

<comment><USER14> Codility has a good choice of languages; [https://codility.com/train/](https://codility.com/train/)

<comment><USER15> Interesting, I've been enjoying [https://www.codeeval.com/](https://www.codeeval.com/) which also has the headhunter based model and supports a few more languages, but doesn't have the time limits like a real code interview usually would.

</comment></comment></comment><comment><USER16> I've really been enjoying codewars.com because you get to see the solutions of others after you pass all tests. I've learned a lot of little javascript tricks this way.

</comment><comment><USER17> You'll become a much better programmer working on real project. Believe me.

<comment><USER18> I disagree. A real project will teach you a set of problems that are useful at most day jobs but these coding challenges will help you learn a class of skills and techniques that are very hard to learn/figure out on your own. These are much more centered around Computer Science problems than typical software craftsmanship problems. You need both (CS and software craftsmanship based skills) if you want to a complete well rounded Engineer.

<comment><USER9> I have 11 years in the industry in various jobs. I was very good in the algorithms and data structures part of my course when I studied it, but the frequency that I actually get to use any of that stuff is maybe a couple of weeks per year.

I would say that learning libraries and how to use them will be far more use, as most common problems have been solved, implemented, put in a library and tested for you. A good programmer will use that library rather than reinvent the wheel.

</comment></comment></comment><comment><USER19> The submission deadline for the 2014 contest is today, but otherwise Hello World Open ([https://helloworldopen.com/](https://helloworldopen.com/)) would have been an interesting programming challenge to participate in.

Positioned as a sort-of World Championship for programmers, it is a virtual slot car race. The challenge is to create an AI and drive a slot race car on a track against other racing bots. Very entertaining... Finals will be played on June 10th in Helsinki, Finland.

</comment><comment><USER20> For beginners, easy programming challenges are very practical, since they let them practice getting their loops and conditions right.

Afterwards, programming challenges become a little less practical since they are more challenging than practical programming. However, I think one can still make challenges that are practical, and I'm thinking of doing this on my site [http://www.learneroo.com](http://www.learneroo.com).

</comment><comment><USER10> I'm looking to practise my SQL skills, are there any website that offer SQL challenges and not only regular math/comp-sci questions ?

<comment><USER21> Hey, sorry for the self-plug, but try [http://pgexercises.com/](http://pgexercises.com/) . It lets you try SQL exercises in the browser and offers decent explanations to go with it. It's focused on PostgreSQL but the large majority is standard SQL.

There should be a pretty decent range of difficulties for you to test yourself on.

<comment><USER22> thank you so much! this is quite helpful, as I'm definitely a person whose memory solidifies through doing, rather than by reading.

<comment><USER21> Hope you enjoy it :)

</comment></comment></comment></comment><comment><USER23> I can also recommend [http://codeforces.com/](http://codeforces.com/). They hold regular contests where you can compete against others and then see how you improve over time. It is popular in Russia and sponsored by VK.

</comment><comment><USER24> UVA is just one of many online judges. See [http://qr.ae/r9bsW](http://qr.ae/r9bsW) and [https://github.com/andreis/interview](https://github.com/andreis/interview)

</comment><comment><USER25> I really like this one [http://oj.leetcode.com/problems/](http://oj.leetcode.com/problems/) have been doing them for a while and it's a very nice exercise to the mind.

</comment><comment><USER26> I feel like I've found the problems thrown up by trying to build open-source (and commercial ones where I have a lot of autonomy to change things) projects to be more informative and useful.

</comment><comment><USER27> There is checkio.org for Pythonistas. As I know, there is no language choice for now but there will be on the future.It motivates coders with gamification and also level of problems.

</comment><comment><USER28> I'll stick with something more modest, something that appeals to the masses and allows for crowdsourced feedback.

[http://codecondo.com/coding-challenges/](http://codecondo.com/coding- challenges/)

You get more than enough.</comment>


Ask HN: How can I generate side income? - <AUTHOR> Hi HN,
I work in a company with very low income. I find myself often struggling to buy any luxuries/pay off debt after my paycheck comes in.
I have free time which I just spend surfing YouTube, Facebook, and watching videos. I need to turn this fun time into money time.
What are some ways I could turn my time into money? <comment><USER2> Frustratingly vague question that is asked very often.

What are you good at? What skills do you have? What are you passionate in? How much spare time do you have? How much are you willing to invest ($)? Physical or digital? Craft or service?

The list goes on. Ask yourself the above, narrow down the scope of the question.

</comment><comment><USER3> Make something people want and charge them for it.</comment>


Airbnb Listings Mostly Illegal, NY State Contends - <AUTHOR> http://www.nytimes.com/2014/10/16/business/airbnb-listings-mostly-illegal-state-contends.html <comment><USER2> I wondering if we are living in a short era of cheap and semi-legal sharing economy and eventually all these sharing economy companies will get relegated and become too expensive to make sense. I'm saying this because from what I experienced using Uber and Airbnb, none of people who provided the service were "sharing" their own stuff. They were doing business. The Uber driver usually is a full time driver. The Airbnb landlord had multiple apartments to rent. So they are the same old taxi drivers and motel owners. We are just paying less because it's illegal.

<comment><USER3> Another perspective is that it's cheaper because it lowered the barrier to entry so that the smallest of entrepreneurs could enter the market. Rather than purchase a hotel, you can rent out a studio apartment. You can start with one studio and then expand over time. You don't need much (if any) additional capital. Airbnb is your entire marketing department and you only pay them when they deliver a client. Handy cleans for you and fixes the shower if it breaks.

Brands are very lucrative assets. Maybe part of this shift is an Airbnb brand tax of 6% in place of a Hilton, Marriot, or SPG tax of 30%.

How much of the old model was there to protect monopolies and how much of it was there to provide for consumer safety? How do we incorporate the latter and eliminate the former? That's the real question in my mind.

<comment><USER4> >How much of the old model was there to protect monopolies and how much of it was there to provide for consumer safety?

when medallion costs $200K-$600K is it to protect monopoly or to provide for consumer safety? Tough question :)

<comment><USER5> the medallions are distributed in auction, to align with the great american myth that those who can pay the most deserve it the most.

Granted, alternatives would probably involve taxi companies wining and dining to get preferred treatment.

Once we've decided on having some sort of system to limit the number of taxis, there's going to be some form of selection. Apparently people thought auction was the "least worst" way (compared to some other selection method).

<comment><USER4> >Apparently people thought auction was the "least worst" way

Auctions only confirmed the intrinsic value of medallion in that system - $2K/month rent pay for a medallion in SF before the auctions corresponds to something like $200K value of a rent-generating asset.

>Once we've decided on having some sort of system to limit the number of taxis

exactly, protecting monopoly as such a limit has nothing to do with consumer safety.

Edit: just googled - the medallion price in NY is $1M and in SF - $300K. Sorry for using obsolete data (holy macrel what a nice exponentially looking price ride of recent 3 years Uber has killed : [http://www.aei- ideas.org/2014/06/chart-of-the-day-nyc-taxi-m...](http://www.aei- ideas.org/2014/06/chart-of-the-day-nyc-taxi-medallion-prices-are-flat-uber/) )

<comment><USER5> >exactly, protecting monopoly as such a limit has nothing to do with consumer safety.

There's other reasons to want that. Congestion control is one of them. No comment on the validity of such a fear, though

<comment><USER6> I believe that's historically what caused the first introduction of such a system, in London some centuries ago: residents complained there were too many hackney carriages driving around the streets, and wanted them limited to a fixed number. (They also wanted them better maintained and more safely driven, which led to additional rules.)

<comment><USER7> But funny enough, there is no fixed number of cabs in London now.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> The right answer seems to be somewhere in the middle but neither side has put forth anything reasonable.

For one thing, HOAs and landlords should have pretty liberal abilities to forbid short-term rentals and pretty much all have.

Renting out a room where the owner/tenant is present should be allowed on a limited basis (perhaps several weeks/months per year).

Renting out a full residence in a non-zoned area should be allowed on a more limited basis (perhaps up to 1 month per year).

Just like on Ebay, et al, "casual sellers" should be able to execute some limited amount of quasi-business activity before triggering all the business rules.

Finally, AirBnB-style rentals should be regulated more like BnBs (duh!) than hotels.

The question is, what's the right answer for these situations? And no kudos to AirBnB, et al for offering up no suggestions.

<comment><USER9> If you own a property (or have the owner's permission), why does it matter whether you actually live in it or rent it out to other people? Should we be prohibited from lending our physical books to other people just because publishers would rather we didn't?

Why does it matter who you rent out your property to? Why does it matter whether you rent it out for 30 days a year or 300 days a year? Why does it matter whether you rent it out for 30 days to one person or one day each to 30 people? The last one is especially arbitrary, and the only reason it means anything at all is because the hotel industry uses it as a threshold for market segmentation.

Is it safety that matters? If so, let's drop all the "up to X days" bullshit and actually focus on safety. How can we check whether an AirBnB listing is "safe", whatever that means in this context, how do we compensate victims of "unsafe" listings, and how do we come up with a good incentive structure to encourage "safe" listings and weed out "unsafe" ones?

Is it taxation that matters? Of course people should pay taxes on all of their income. In fact, AirBnB income should be easier to track down than the proceeds from that MacBook you just sold on Craigslist. AirBnB keeps excellent records of all of your income (and their fees), after all. Then it's just a matter of getting those records into the hands of the IRS.

Is it zoning that matters? If so, why do we care about zoning at all? Over the last few decades, the only thing that zoning achieved in the United States is the complete segragation of residential areas from all other areas of modern life, so that everyone needs to drive a dozen miles to work and another dozen miles to get their groceries. Many cities are now beginning to toy with the idea of vibrant mixed zones, where people can live, work, shop, and play all within easy walking or cycling distances. A hostel next door suddenly doesn't seem like a bad idea after all. (Oh, it's not properly managed and people are shooting crack up their arms in the backyard? Well, that's a problem with poor management, not a problem with the mere fact that it's a hostel. Would you ban houses with basements because some basements have meth labs in them?)

So I refuse to believe that the right answer has anything to do with how long you should be able to rent out a property. It's your property, you should be able to do whatever you want with it within reasonable bounds (like paying taxes and keeping your property safe for other users).

As long as we keep focusing on peripheral issues that are framed by existing business models with conflicts of interest, we'll never get to ask the really pressing questions, such as reasonable safety and insurance standards for small-time landlords.

And yes, I agree with you that AirBnB's reactive (as opposed to proactive) attitude toward this issue isn't helping, either.

/rant

<comment><USER8> I'm guessing you don't own a home or condo because your arguments all violate common sense. First, do you understand why owners don't like renters? Do you understand why short-term renters are worse than long term renters? Do you understand why owner-present is more desirable than owner-unpresent? Do you understand why a residential owner/renter would prefer not to have commercial activity next door? Do you understand why zoning exists? Do you understand why citizens are granted certain abilities to conduct business-like activities before triggering various business rules?

You're fighting a pretty losing argument here.

<comment><USER9> You're right, I don't own any real estate, and I do not plan to do so at any time in the foreseeable future. I don't like being tied down to any arbitrary segment of the Earth's crust, and I'll gladly pay more for the freedom to move somewhere more exciting at a few weeks' notice. (Yeah, so that's part of my motivation for defending AirBnB.)

The answer to the first four of your rhetorical questions all seem to involve some version of "I don't want my property to lose value". Since I have little sympathy for that kind of sentiment, it seems selfish to me for you to force (i.e. legally require) someone else to do this or that with _his_ property just because it might negatively affect the value of _your_ property. As long as your neighbor doesn't produce loud noises or obnoxious smells that cross into your property, you have no right to interfere with his use of his property. Would you also seek to ban African-American tenants from your neighborhood? Because, you know, the racial makeup of the neighborhood does have an impact on property values.

Note that I'm not saying everyone should bear with a hostel next door. If you want to live in an all-residential neighborhood, you are absolutely free to go and live in a place where everyone voluntarily agress to HOA rules against business-like activity. All I'm saying is that this should be voluntary, not a legal requirement, and there are already plenty of places in the U.S. where such voluntary agreements exist. If Manhattan doesn't happen to be one of those places, well, too bad.

As for the other two two questions, yes, I do understand the rationale for such policies, but I think that they have been corrupted to serve a narrow range of entrenched interests, and that they are badly in need of more flexible rethinking.

</comment></comment></comment></comment><comment><USER10> Everyone knows this, and has known this. Airbnb knew it too. Hopefully they used their runway to develop a better plan than the denial that's been ongoing up to this point, since it's been obvious since its inception that Airbnb would have to deal with these regulatory problems if it ever reached critical mass.

</comment><comment><USER11> Suppose it is mostly "professionals"

1\. is it bad for individual "professionals" to invest capital on a small scale to provide services that people want at a greater or lesser price? Be it cars or houses.

2\. If it is bad, why, and who for? Is it bad for consumers or bad for existing businesses?

<comment><USER12> It's complicated.

On one hand, traditional regulation keeps small or part-time players out of the market, because regulatory overhead is too high. The relative burden of regulation on large companies is much less, because they can amortize the cost across more business.

On the other hand, this only works if these smaller players get relief from regulations -- which means that they are operating outside the laws and regulations. For instance, turning an apartment building into a hotel via Airbnb sidesteps rules around zoning, tax collection, housing policy, etc. You could decide that these regulations are just annoying red tape, but they were put in place for legitimate purposes in the past and it seems like we ought to at least consider the impact before we throw the baby out with the bath water.

A third argument is that peer to peer marketplaces are changing the balance of power between producer and consumer. As a consumer, this sounds great -- but the person at the other end of the line is now an individual, not a giant corporation. When I have a bad cab ride, I go on Facebook and bitch about it, and life goes on. When I have a bad Uber ride, I give the driver one star and they may very well get fired -- and Uber will just replace them with one of the teeming masses hoping to make it rich driving strangers around for a living.

<comment><USER6> _On one hand, traditional regulation keeps small or part-time players out of the market, because regulatory overhead is too high._

I suppose it discourages some, but it's not like there aren't small or part- time players in property rental. There are a _lot_ of them! Small-time landlords who own a few rental units are everywhere; I know two such people in my circle of friends who do that. You can either do it as a part-time job, or hire a management company for a fee to oversee the actual management. The difference is that they do it legally. They own apartments in buildings where it's legal to rent out the units, and they rent them out legally.

</comment></comment></comment><comment><USER13> Interesting that the AG's report is titled _AirBnB in the City_ , likely a reference to the classic NYC tv show _Sex and the City_ [1].

Interesting, in that many people incorrectly call that program _Sex IN the City_ , which the producers object to. They point out that the _AND_ is deliberate - the City of New York is an integral character in the story, not just a location where the story (and sex) occurs.

In this dispute, the government is making a similar gambit - arguing that they need to be part of the conversation, via regulation and enforcement etc, not just the location in which the lettings take place.

In other words, the report title seems to mis-quote the name of the TV show. And in doing so it missed the opportunity to position the State more strongly as a central actor in this drama.

[1] And lest anyone observe the title may be a coincidence, note that Eric Schneiderman is the AG of New York State, not just the city.

</comment><comment><USER14> With the Internet, you can't stop people with spare rooms and people who need a place to stay from finding each other. We now stand at a crossroads.

Down one road we recognize this is a new kind of interaction enabled by technology, which leads to legalization, with light regulation to deal with new patterns of abuse that aren't handled effectively by existing rules. Reasonable taxes and fees so the new industry covers the additional costs it's imposing on government and pays its fair share to the city, state, and country.

Down the other road is heavy-handed regulation, and punitive taxes and fees that drive small entrepreneurs and the websites that enable their business model out of the legitimate market and into the shadows. AirBNB is legitimate, but if they stop servicing an area due to regulation, I'm sure a sketchier copycat will try to take over that market. The government misses out on taxes and fees from the grey/black-market transactions. Hosts / guests that have been abused or scammed will be reluctant to seek recourse through the legal system if it will mean admitting their own illegal conduct, and will have to either endure their suffering or try to self-help.

It's really the same story as abortion, drugs, or Bitcoins.

</comment><comment><USER15> > But the attorney general's report says rentals in three areas in Manhattan — > Lower East Side/Chinatown, Chelsea/Hell's Kitchen and Greenwich Village/SoHo > — accounted for 40 percent of private stay revenue, or $187 million.

This surprised me not at all. I live in the Lower East Side and see an almost non-stop parade of groups of tourists with suitcases waiting to get let into apartments here. I know friends-of-friends who have four or five apartments that they rent out for this purpose.

People who operate these illegal hotels are attracted to neighborhoods like LES/EV/Chinatown, aside from them being trendy, is because there are a lot of non-doorman buildings that have relatively inexpensive rent. Unfortunately, those buildings are a critical source of semi-affordable housing in Manhattan, and more and more of it is getting snatched up by hoteliers.

I'm glad that NY is not taking the same approach as SF. People should absolutely be allowed to rent out a room or a couch, but the illegal full- time, full-apartment listings are terrible for residents who aren't aspiring hoteliers.

</comment><comment><USER16> AirBNB also provides a product that was very hard to find before. Although most of the NYC housing market consists of units kitchens and living spaces, it was very hard to find a two-bedroom apartment to rent short term before AirBNB. As someone with a family, the ability to rent a traditional apartment for a short time is a significant improvement in the market.

</comment><comment><USER17> It must be nice to have friends in high places. I know if the NY Attorney General said 75% of my business was illegal, I'd be shut down tomorrow. Yet, Airbnd can fight these charges (mostly in the court of public opinion and somewhat in the court system).</comment>


Ask HN: How does a developer learn a non formal language? - <AUTHOR> I'm a developer and I want to learn German. There is a plan, a deadline, motivation and hopefully time to do it. Basically, I'll try to reproduce what I  did to learn English (native Spanish speaker) but in a shorter period of time. I have to said, my English is still far from perfect but I can watch movies and the like and understand about 80% of them, I can communicate effectively in my area of domain (software development)  and in other common areas (but cannot talk about politics or geology, for example). I would say my level of English is something between B2 and C1 (https://en.wikipedia.org/wiki/Common_European_Framework_of_Reference_for_Languages)
So, this is my plan:
July, August and September (2016): Learn the basics, the grammar, watch Youtube videos with German subtitles. I have bookmarked a couple of grammar books and good Youtube channels.
October (2016) to July (2017): I'll be in Germany, this means I'll have to speak! Although what I'm going to do there only requires me to speak English. Will keep improving the grammar and will keep watching, not only videos, but also the TV and the radio and of course I'll talk with German people.
The level I want to reach is the same as my level of English I have now. I know it's a hard task to accomplish, but boy, I'm motivated.
Suggestions, tips and personal experience learning non formal languages are welcome. Thanks! <comment><USER2> My rate of improvement in Japanese really picked up after starting spaced repetition. I use an app called "Nihongo", which is for Japanese only, but I think Duolingo has German.

<comment><USER3> Duolinguo definitely has German.

I suggest supplementing active practice with passive exposure (switch to music with German lyrics, stop watching tv and movies not in German).

<comment><AUTHOR> That's something I did/do with English. I usually listen to podcasts while going to work (about 1h commute) and it's kinda difficult sometimes to "hear" what it's playing. I mean, I get distracted and suddenly I have no idea what the podcast is about. But I guess that's part of the "passive" exposure.</comment></comment></comment>


Microsoft won't tell you what its Windows 10 updates do - <AUTHOR> http://bgr.com/2015/08/22/windows-10-software-updates/ <comment><USER3> When Gates ran Microsoft they got away with lighter stuff than this. This Post-Gates Microsoft seems to be trying to get away with more privacy things and not telling you what its updates do.

Someone told me that Windows 10 uploads files on your hard drive to a Windows Server to see if any of them have been pirated and downloaded from torrent trackers. That Windows 10 can remove pirated software, and even delete pirated files on your hard drive.

Windows 10 has created such paranoia that some Private Trackers ban Windows 10 clients: [https://torrentfreak.com/torrent-trackers-ban- windows-10-ove...](https://torrentfreak.com/torrent-trackers-ban- windows-10-over-privacy-concerns-150822/)

When I upgraded to Windows 10 I turned off the data sharing settings. I am considering going to Linux and running a copy of Windows in Virtual Box. I'd use the Steam client for Linux and only play the video games that got ported to Linux. I haven't done this yet, but I have to turn off SecureBoot to get GRUB to load.

<comment><USER4> Give it a try. There are over 1,000 games on Steam that run on Linux (most can also be purchased from GOG.com if you're wary of Steam).

<comment><USER3> I might just do so when SteamOS comes out in November.

If not I'll try Mint or Lubuntu.

</comment></comment></comment><comment><USER5> Not surprising considering their target audience. A worrying headline none the less.

</comment><comment><USER6> here, have some mystery executables. No, we won't tell you what they do - <USER2> would ruin the surprise!

(or, alternatively - why download mystery executables from the internets, just let Windows 10 do that for you!)</comment>


VisiCalc during the early days [w/pics] - <AUTHOR> http://www.bricklin.com/history/saiearly.htm <comment><USER2> I truly miss those times. Well... my computer is much more powerful now, but, still, those heroic days when I could fully understand the Apple II ROM will not be forgotten.

At least, not by me ;-)

</comment><comment><AUTHOR> I always love reading stuff like this. It's really inspiring to learn about other inventors who have had great success and started out in the same fashion as you. (I'm currently working out of an attic in Arlington Ma, not too far from that house).

</comment><comment><USER3> Stories of the early days of personal computing are ones I find particularly inspiring because these entreprneurs did what they did in an industry that didn't exist yet and in a climate where there was far less of a startup community in terms of investors and other entreprneurs to show you the ropes. These guys didn't surf the wave, they made it.</comment>


The People Who Grind Out Movie Trailers - <AUTHOR> https://www.theringer.com/movies/2018/7/23/17601024/movie-trailer-editors-marvel-pixar-how-made <comment><USER2> Funny story:I use to work for a tech company that buid pay-to-view systems for hotels (small atom box with nice screen in each hotel room). Thennnn the guys wanted porn movies... fine we got it and it was great (ppl buying porn with their breakfast). But the porn movies didn't have trailers so we just took any 30 sec of it.... BIG MISTAKE... no one was buying the movies anymore... We quickly realised our random 30sec were too graphic People were only watching the trailers on repeat (yes this is the one time your mind-in-gutter is right) ! lol We had to make new trailers but show nothing below the naval :D

</comment><comment><USER3> I watch a lot of trailers just for fun, mostly from this site: [http://davestrailerpage.co.uk/](http://davestrailerpage.co.uk/)

I've gotten very picky about what constitutes a 'good' trailer. Any trailer that has any of these:

* an overly dramatic deep manly narrator

* is actually a 'clip' rather than a trailer

* is actually a 'making of' rather than a trailer

* has a pre-trailer to the trailer

and a bunch of other things, gets deleted immediately. Thought I would share ;) I also have a dave's trailer page scraper if anybody is interested/obsessed like I am.

<comment> B-Con What about summarizing the entire movie? That's an instant NO from me. I've literally not watched movies because I've seen the trailers and feel there's really no wiggle room left plot-wise, I know it all.

I very actively avoid all trailers for movies that I'm going to see. Modern blockbuster trailers just ruin the movie.

<comment><USER3> Trailers don't summarise the whole movie. And even if you feel they do, knowing the outline of a movie isn't going to ruin the actual movie experience itself. I avoid heaps of crap movies simply because I dismiss them out of hand by their trailer. Obviously there is the occasional movie whose trailer was crap but was actually exceptional. These come around again.

There is so much drek out there a strong filter and upturned nose is essential to carefully picking something that won't waste your time and money. Otherwise you're saddled with Transformers 12: Re-revengance the pre-sequel. Or Star Wars 8 ;)

It's how the same old variation of a story is told, how well did they pull it off, was it charming, was the dialogue and it's delivery any good, etc.

And watching a movie with another person changes the movie again. Or watching it at a cinema surrounded pigs at their troughs.

So many ways to appreciate movie watching that a trailer can't ever touch.

<comment> B-Con > Trailers don't summarise the whole movie.

Some do. If they contain:

* the protagonist's set-up

* the antagonist's set-up

* an explanation of the protagonist's challenges

* the protagonist overcoming the challenges

* the nature of the twist

* (sometimes the twist itself is hinted at)

* any romantic relationship of significance is both shown and the direction it ultimately goes is implied

* half the big awesome SFX scenes

* half the funny scenes

* almost every character of significance to the plot

then there isn't much left, I know almost all the key checkpoints, who has to be where and do what and what action has to occur. It's pretty much sketched out. There's minimal suspense because I know who can't die and who has to be where at the end of the movie.

> And even if you feel they do, knowing the outline of a movie isn't going to > ruin the actual movie experience itself.

It definitely can. Part of a story is experiencing it for the first time, watching it unfold and not knowing what will happen. Trailers take that away from you. "Oh yeah, I've seen this fight sequence and heard that joke before" just doesn't carry the same weight. And when I have a checklist of people with places to be and action scenes to happen, I can predict things. "Well, I know there's going to be a twist because X has to fight Y in the streets and it looks like they're getting along OK as we approach 1h 20m mark."

There's minimal surprise left, which really hampers my ability to empathize with the characters and engage in the story.

<comment><USER4> A movie can work even if you know how these things. The how is just as important as the what in a movie, even if, for example, you know the twist of a movie, it can be just as fun to watch how you get there, how all the scenes and narration from the trailer is strung together into the movie.

<comment><USER5> I don't believe anyone here was suggesting that the movie becomes literally unwatchable.

But a movie is just a 90 minute experience. It's nice to maximize it. And one simple approach is to avoid giving in to curiosity by watching its 30 second summary for the same reason you don't read plot outlines on Wikipedia before reading books or tv show episodes.

<comment><USER4> I do actually go to Wikipedia and other places to spoil the plot outline to myself. Makes it much easier to focus on the implementation of the story rather than having to focus on both story and implementation.

<comment><USER6> If there is one comment that embodies the HN spirit, it's this one...

<comment><USER4> I would disagree there a bit, I do it because I enjoy taking things apart and seeing how it all ticks together.

Like taking apart a running grandfather clock to see it tick.

Movies and TV series or even books are similar to me. I do enjoy digging deep into TVTropes and AllTheTropes so I can whlie watching a movie, take it apart and see how it ticks.

How the movie achieves the suspension of disbelief, how the heroes journey is being implemented in lockstep with the villain (if there is even one) or how a character is built, what parts make them what they are.

I think it's more of a hackers spirit rather than HN. I read the manual of the movie so I can take it apart like some hacker might read a device manual to be able to take it apart more easily.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> My friend decided to never watch trailers or teasers, and know as little as possible before watching a movie.

It worked out really well for him. The less expectations he had, the more he appreciated the work of art for what it really was.

It's a great philosophy, but it is tedious to time it late to a movie showing exactly right so that you miss the trailers.

<comment><USER5> I just close my eyes. And if it's a movie I think I will particularly enjoy, I also plug my ears.

Might look goofy but I'm not exactly the center of attention.

</comment></comment><comment><USER8> Huh what a coincidence I just watched this video [https://www.youtube.com/watch?v=a_jjzzgLARQ](https://www.youtube.com/watch?v=a_jjzzgLARQ) which covers the trailer industry. Pretty eye opening to see all the tricks behind the trailers, a almost-must watch.

<comment> B-Con I think people are watching movie trailers because they're almost self- contained movies. Very frequently the protagonist and antagonist are set-up, the protagonist's challenges are laid out, the protagonist is shown overcoming the challenges, the nature of the twist is shown (and sometimes the twist itself is hinted at), any romantic relationship of significance is both shown and shows the direction it will go in the movie, half the big awesome SFX scenes are included, half the funny one-liners are included, and every character of significance is revealed.

Sometimes you literally don't even need to watch the movie, you just got the tl;dw in the trailer.

I refuse to watch trailers for movies I might see, because they ruin the experience since I know everything that's coming.

</comment></comment><comment><USER9> If you like these, you might like the annual trailer mashups like this one: [https://youtu.be/YU8CvjIwAlY](https://youtu.be/YU8CvjIwAlY)

</comment><comment><USER10> I think the real challenge would be creating a trailer for a movie that is not very good so that you don't give away the few good bits.

<comment><USER11> In most cases the rule of thumb is you can hide the good bits from the trailer, in which case no one will see the good bits because no one will go to the film, or you can show the good bits in the trailer and people will go to theaters and pay to watch the rest of the film.

<comment><USER12> I'm not entirely certain you need to go that far. My issue is that most trailers, especially the ~two minute ones, show the majority of the story arc, and either contain spoilers, or hint heavily at those spoilers. That removes dramatic tension and any desire for me to watch.

A made up example, but I've seen similar: let's take an ensemble superhero action piece. You see a clip of a scene in the trailer where superhero A is in mortal peril, set against a desert backdrop. Compelling, but when I see the movie in theaters, there's an action set piece where superheroes A and B are fighting against villains C and D. B is temporarily incapacitated, and it looks like C and D have the drop on A. Unfortunately, this fight has an ocean backdrop. As such, all dramatic tension is gone, since I know A makes it to a desert I haven't seen yet.

I've had some success making a strict rule of watching only teasers (~30 seconds), or the first 30 seconds of a normal trailer, where they have time to set up the basics, but not enough to ruin anything. If I'm at a movie, I'll just close my eyes after that point, since I can't fast forward, and removing visuals is usually enough.

I wish movie makers would keep that in mind.

<comment><USER13> > You see a clip of a scene in the trailer where superhero A is in mortal > peril, set against a desert backdrop. Compelling, but when I see the movie > in theaters, there's an action set piece where superheroes A and B are > fighting against villains C and D. B is temporarily incapacitated, and it > looks like C and D have the drop on A. Unfortunately, this fight has an > ocean backdrop. As such, all dramatic tension is gone, since I know A makes > it to a desert I haven't seen yet.

I don't follow your logic. You already know A makes it out OK, because it's a superhero movie. Why doesn't that ruin the dramatic tension equally?

</comment></comment></comment></comment><comment><USER14> The worst is when there's something in the trailer that's not in the actual movie.

<comment><USER15> I have a bad memory, so I would not remember this happening. Do you have an example? Also: why would they do that? If they already have the footage, and it is 'impressive' (for the trailer), why NOT put it in the movie?

<comment><USER16> Payback (1999, Mel Gibson) trailer contains a scene not in theatrical release. The main poster for the movie shows Mel Gibson in a scene which never made it to theatrical release either.

Usually I avoid trailers. It's marketing, spoilers, trying to sell you the movie. For me, they might be clever, but that's all. The Payback trailer is clever because it has no dialog... [https://youtu.be/ji6aaZRACew](https://youtu.be/ji6aaZRACew)

Drama behind the scenes on Payback saw the director fired by the studio and replaced, a completely new ending written and shot.

Normally when studios mess with the writing, the film suffers, but not in this case. The theatrical release of Payback is better than the "director's cut" released on video a few years later. The director's cut has a boring ending, just generic gun fights and a mess of scenes. The theatrical release has a good twist, more substance and style.

</comment></comment></comment><comment><USER17> It's really sad that you can't read this article on mobile Chrome without getting trapped into an annoying "you have won" ad page that also clears the browser history.

<comment><USER18> Why don't you install an ad-blocker?

<comment><USER19> Chrome on Android doesn't support extensions so you can't install its ad- blockers.</comment></comment></comment>


Announcing etcd 3.1 - <AUTHOR> https://coreos.com/blog/etcd-3-1-announcement.html <comment><USER2> Congrats to the coreos/etcd team!

With that said, I feel really conflicted about v3. I understand why it was introduced and understand why it is objectively better than v2, however the reason I fell in love with etcd was because of v2. It was a simple, highly available, not very fast key store that exposed an easy way to view and traverse the store (via the /v2/keys API endpoint). Likewise, the client api was simple, quick to pickup and very easy to work with.

With v3, while the performance has increased dramatically (and many other things), the "ease" of etcd seems to have gone as well.

To this point, all of the software I've written that makes use of etcd is tied to etcd v2. Should I make a conscious effort to migrate to etcd v3? How much longer is v2 going to be kept around?

<comment><USER3> > the "ease" of etcd seems to have gone

Can you explain this more? One of the goals of the new API is to make etcd easier to interact with.

Can you provide a few examples where v3 is more difficult than v2 to work with? (etcd-dev or github might be a better place to discuss though) We would love to hear and improve.

<comment><USER4> Agree with sibling comment, gRPC is nice as an optimization but terrible as a default or exclusive means to speak to an API.

<comment><USER3> etcd3 provides a HTTP+JSON API as a gateway: [https://github.com/coreos/etcd/blob/master/Documentation/dev...](https://github.com/coreos/etcd/blob/master/Documentation/dev- guide/api_grpc_gateway.md).

There are users we know successfully using it. If users are unhappy about the HTTP API, or it is too hard to use, we might consider to provide a native HTTP API.

</comment></comment></comment></comment><comment><AUTHOR> Much of the focus on this release is continuing to support Kubernetes with the best possible consistency database. Xiang Li, one of the leads of etcd, included a couple of notes in his email to Kubernetes dev:

etcd v3.1 improves the read performance by a factor of 10 for Kuberentes use cases (unless quorum read is disabled).

etcd's v3 API will be the default backend in Kubernetes 1.6 release. But now you can also run Kuberentes 1.5 with etcd3 by providing `--storage- backend=etcd3` to the API server.

[https://groups.google.com/forum/#!topic/kubernetes- dev/wgYXb...](https://groups.google.com/forum/#!topic/kubernetes- dev/wgYXbyHKLEw)

</comment><comment><USER5> Tangentially, has anyone tried using etcd as a database for content?

etcd was obviously designed for consensus, coordination and discovery, but there seems to be nothing to prevent it from being perfectly adequate as a data store for other things. From what I can tell, it's pretty fast, strongly consistent, replicating, and has the building blocks to perform atomic transactions across multiple keys. It might work really well as the primary store of truth with something like Elasticsearch on top.

<comment><USER6> There are a few things preventing it from being an effective database for general-purpose content:

\- "bespoke" data storage [1]

\- originally optimized for small amounts of data [2]

v3 made progress on both of these fronts but see [3][4]

[1] [https://github.com/coreos/etcd/issues/2647](https://github.com/coreos/etcd/issues/2647) [2] [https://groups.google.com/forum/#!searchin/etcd- dev/store/et...](https://groups.google.com/forum/#!searchin/etcd- dev/store/etcd-dev/ATZm7sRJy60/aqeQcrIB7NcJ) [3] [https://coreos.com/etcd/docs/latest/faq.html#deployment](https://coreos.com/etcd/docs/latest/faq.html#deployment) [4] [https://coreos.com/etcd/docs/latest/op- guide/hardware.html](https://coreos.com/etcd/docs/latest/op- guide/hardware.html)

<comment><USER5> Hm. The first two are pre-3.x, and that issue (#2647) is closed, seemingly because everything has been implemented? Not seeing any huge limitations mentioned in 3 and 4.

The "small amounts of data" issue apparently no longer applies to 3.x, which now persists efficiently to disk.

Not sure what "bespoke" data storage means? 3.x has pluggable backends.

<comment><USER6> I thought they might've meant to add pluggable backends but I can't find any reference to them? Do you have a link?

<comment><USER5> Oops, I might be confusing it with something else, sorry.</comment></comment></comment></comment></comment>


Welcome, cryptanalyst - <AUTHOR> http://126.96.36.199/ <comment><USER2> Last time I visited, it had a message like "You can only log in if you have logged in before." Is this part new, or is it just a different way to enter?

<comment><USER3> This is completely different—the last site was targeting python devs.

<comment><USER4> No, it's the same. This is just a test that lets you access foobar.

</comment></comment></comment><comment><USER5> If anyone involved is reading this -- please make the timer a bit more obvious. I opened the link, went back here, then made a quick wikipedia safari and when i got back to it timed out with no indication that i can restart it. :(

edit: The console could use a little error checking, if the request dies (i had referrer disabled), it prints "requesting xxxx..." and then nothing. Tsk, tsk :)

<comment><USER6> Delete the cookies from ig-game.appspot.com.

</comment></comment><comment><USER7> I completed a few of these foobar challenges, and I'd love to keep working on them, but it seems to have stopped working for me. I can no longer save files in the editor, and I get a "Not Found" error when I run the verify command. Anyone else run into this bug (and maybe figure out a fix)?

<comment><USER7> FYI: I tried deleting cookies again this morning, and everything is working again. The challenge I had finished (but was unable to save or submit) was showing as completed. Not sure if someone at Google took action on my bug report, or if everything just worked itself out.

</comment></comment><comment><USER8> This game is not working correctly on my browser (FF33). I tried several times and waited until the end because nothing was happening... then tried with chromium and it did work :-/ (btw noscript is turned off)

<comment><USER6> FF33.1.1 (on Archlinux) is working fine here. In addition to NoScript, you probably need to disable any extension that forbids cross-domain cookies as the relevant cookies are registered for [https://ig- game.appspot.com/](https://ig-game.appspot.com/) and not a raw IP.

</comment></comment><comment><USER9> This is entrance to foobar. It is pretty easy to figure out crossword using either a calculator to convert bases or (wolfram alpha for the lazy) and [0] for regex. I have not used regex in years, but got in easy.

[0] [http://www.regexr.com/](http://www.regexr.com/) . Hint mouse over regex after you type it for explanation of what will match, try it out in the text editor below. Fairly trivial.

<comment><AUTHOR> all is trivial but regex, what data to use for those hints ?

<comment><USER9> It does not really matter. Either characters to use would be included in regex itself, such as [j-g] or you could use any.

For example "\D{2}\S{3}" means match 2 non-digit characters followed by 3 nonspace characters. Some things that match "aaaaa", "bbbbb", "ALUSD", "AA123"...

Site I linked really explains things quite well on mouseover. Once you complete first challenge (make sure you open this in chrome, I've had exciting quest of delete elements in firefox until I could save my solution) you can register and come back anytime later.

</comment></comment></comment><comment><USER10> Kinda surprised to be challenged like that suddenly, but managed to pull through with my (t)rusty regex-foo.

Foobar is fun and well made! Passed the first challenge in a few minutes and will continue tomorrow. Try to start challenges on foobar only when you want the timer to start (e.g., not when you need to go to bed like me..). Thou shan't worry much on the first challenge though, it allows for 48 hours.

<comment><USER6> FYI, the time alloted to each challenge seem to increase with the difficulty (the first couple of problems are 48 hours, then 72, then 96, etc.).

</comment></comment><comment><USER11> What does something like \D{2}\S{3} mean? Out of context it seems a bit... cryptic ;)

<comment><USER12> It's a regular expression, probably Perl syntax:

[http://perldoc.perl.org/perlrebackslash.html](http://perldoc.perl.org/perlrebackslash.html)

<comment><USER11> Thank you. I thought it could be a regex, but I had never seen perl syntax before.

</comment></comment></comment><comment><USER13> I completed this puzzle. It then takes you to the foobar puzzle which was posted a while ago. It has a series of challenges. After completing the first challenge, it says "Max active challenge count reached"

</comment><comment><USER14> I passed the crossword! If you fill in the blanks with, say, "a" you can check your answers as you go.

I'm now facing a small coding challenge: 48 hours to come up with code with Python or Java (my choice).

<comment><USER14> So level 1 has only one problem, 48 hours. Level 2 has two problems, 72 hours apiece.

Edit: Level 3: 3 puzzles, 96 hours apiece. How deep does this go?

</comment></comment><comment><USER15> Doesn't work.

Upon completing the crossword the page auto-submits the solution and gets "403 Forbidden" in response. Something got broken somewhere or someone has pulled the plug.

<comment><USER16> tried it out, still works. Try clearing cache and cookies and restart. Also use a mainstream browser

<comment><USER15>
      <<
      POST / HTTP/1.1
      Host: ig-game.appspot.com
      ...

      >>
      HTTP/1.1 403 Forbidden
      Alternate-Protocol: 443:quic,p=0.02
      Cache-Control: no-cache
      Content-Length: 0
      Content-Type: text/html; charset=utf-8
      Date: Tue, 25 Nov 2014 21:32:12 GMT
      Server: Google Frontend
      Set-Cookie: session=(snip); Path=/; HttpOnly
      X-Firefox-Spdy: 3.1

Nope. Might be geo-locked or something.

<comment><USER5> Look at the response body, the 403 i got had a nice html error message attached.

<comment><USER15>
      Content-Length: 0

:-|

</comment></comment></comment></comment></comment><comment><USER17> Okay, that was an exercise in frustration. By the third timeout i was pretty confident i had the answers right, but nothing happened. :(

<comment><USER18> same. claimed i had errors, checked multiple times, did not work.

if this is a recruiting tool, they should make sure they code it properly. unless part of the game is to debug it ... but i do enough of that already at work, so screw it.

</comment></comment><comment><USER19> Neat puzzle. There are coding challenges if you manage to pass the crossword. Guessing it's some kind of broad net recruiting?

</comment><comment><USER20> I tried 3 times on Firefox and it was always saying 4/5 correct. Then I tried on Chrome and I passed to the foobar terminal.

<comment><USER19> Odd what's FF version? It worked fine for me on v32.0.2.

edit: It might be this suggested by Kripke:

"kripke 0 minutes ago | link

FF33.1.1 (on Archlinux) is working fine here. In addition to NoScript, you probably need to disable any extension that forbids cross-domain cookies as the relevant cookies are registered for [https://ig- game.appspot.com/](https://ig-game.appspot.com/) and not a raw IP."

</comment></comment><comment><USER21> Hmm, "This cipher machine is jammed. Come back and try to crank it up later."

I suppose that is some sort of HN effect.

</comment><comment><USER22> I managed to get 2/5 correct. I guess I'm still far too much of a noob for these things :P

<comment> _-__--- Was it the hex/oct -> dec conversions?

...yeah, that's all that I got too.

</comment></comment><comment><USER23> 4/5.... will any correct regex match count as accurate or are they hints at a specific answer?

<comment><USER24> Looks like any matching regex is accepted.

<comment><USER25> Odd. That wasn't working for me. Might be because I'm using Safari.

Edit: Yep, works on chrome. Lame. I wasted some time on that.

</comment></comment></comment><comment><USER26> What happens if you make an incorrect submission for the coding challenges?

<comment><USER27> Nothing. At least at level 1, I made an incorrect submission multiple times without apparent reprisal.

</comment></comment><comment><USER16> Looks pretty easy, I wonder if google recruits like this?

<comment><AUTHOR> This isn't for google, it's some marketing thing for upcoming movie "The Imitation Game" (based on Alan Turing).

<comment><USER28> The footer has links to Google. Unless Google is now doing movie marketing.

This is also part of another game that was submitted a couple weeks ago which was a recruiting game for GOogle.

<comment><AUTHOR> I got to that site through this movie trailer [https://www.youtube.com/watch?feature=player_detailpage&v=sX...](https://www.youtube.com/watch?feature=player_detailpage&v=sXKl3qu10-s#t=3) if you look at the paper at 3 second mark you can see the address "18.104.22.168". But now i wonder if its somehow connected with google

<comment><USER16> Wow! You clearly pay attention... I wonder if I would have noticed this. Thanks for sharing!

Somehow, google must have to do with this, either google paying weinstein for displaying this or weinstein paying google for making this a social media booster.

<comment><AUTHOR> Well Benedict asked to pay close attention :D

</comment></comment></comment></comment></comment></comment><comment><USER29> 6 lines of pythoncode for 1. puzzle. anyone less?

<comment><USER27> I believe we get puzzles randomly selected from a pool. Mine was about date parsing. Yours?

<comment><USER29> comparing two datafiles with suffled numbers to find matching pairs. all pairs have the same quotient. task was to find the quotient and return it as "improvement-ratio"

<comment><USER30> Can confirm they're both different from mine. I got a palindrome problem.

<comment><USER19> Yep mine was summing the digits of a number until you reach a single digit number (eg 455 -> 5).

</comment></comment></comment></comment></comment><comment><USER31> It restarts with new answers if you time out.

</comment><comment><USER32> I will sell the answers for $15. Contact me at firstname.lastname@example.com

<comment><USER19> There's only one constant answer and that's not that hard to get from the references just on the page itself. Shoo.</comment></comment>