Why Learn Awk? (2016) - <AUTHOR> https://blog.jpalardy.com/posts/why-learn-awk/ <comment><USER2> I use awk because there's an almost 100% chance that it's going to be installed on any unix system I can ssh into.

I use awk because I like to visually refine my output incrementally. By combining awk with multiple other basic unix commands and pipes, I can get the data that I want out of the data I have. I'm not writing unit tests or perfect code, I'm using rough tools to do a quick one-off job.

For instance, "mail server x is getting '81126 delayed delivery' from google messages in the logs, find out who is sending those messages".

# get all the lines with the 81126 message. Get the queue IDs, exclude duplicates, save them in a file.

cat maillog.txt | grep 81126 | awk '{print $6}' | sort | uniq | cut -d':' -f1 > queue-ids.txt

# Grep for entries in that file, get the from addresses, exclude duplicates.

cat maillog.txt | grep -F -f queue-ids.txt | grep 'from=<' | awk '{print $7}' | cut -d'<' -f2 | cut -d'>' -f1 | sort | uniq

Each of those 2 one-liners was built up pipe-by-pipe, looking at the output, finding what I needed. It's not pretty, it's not elegant, but it works. I'm sure there's a million ways that a thousand different languages could do this more elegantly, but it's what I know, and it works for me.

<comment><USER3> I know you're not asking for awk protips but you can prefix the block with a match condition for processing.

... | grep foo | awk '{print $6}' | ...

becomes

... | awk '/foo/{print $6}' | ...

If you start working this into your awk habits you'll find delightful little edge cases that you can handle with other expressions before the block (you can, for example, match specific fields).

<comment><USER4> To pile on :-) you often want -w (match word) flag to grep.

In awk, I couldn't find how to do this. I tried /\bfoo\b/ and /\<foo\>/ but neither worked. I don't know why and don't care enough which brings me to my major awk irritation ...

It doesn't use extended or perl REs, which makes it quite different to ruby, perl, python, java. Now, according to the man page it _does_ ; at least on OSX (man re_format) but as mentioned it didn't work for me.

Details

       $ echo fish | awk  '/\bfish\b/'

gets nothing, vs

       $ echo fish | perl -ne  '/\bfish\b/ && print'

fish

<comment><USER4> UGH! Found the problem; it simply doesn't work. Assuming the OSX awk is the same as the freebsd awk there is a very old open bug on this:

awk(1) does not support word-boundary metacharacters [https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=171725](https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=171725)

<comment><USER5> GNU awk supports \< and \> for start and end of word anchors, which works for GNU grep/sed as well

GNU awk also supports \y which is same as \b as well as \B for opposite (same as GNU grep/sed)

Intererstingly, there's a difference between the three types of word anchors:

        $ # \b matches both start and end of word boundaries
        $ # 1st and 3rd line have space as second character
        $ echo 'I have 12, he has 2!' | grep -o '\b..\b'
        I
        12
        ,
        he
         2

        $ # \< and \> strictly match only start and end word boundaries respectively
        $ echo 'I have 12, he has 2!' | grep -o '\<..\>'
        12
        he

        $ # -w ensures there are no word characters around the matching text
        $ # same as: grep -oP '(?<!\w)..(?!\w)'
        $ echo 'I have 12, he has 2!' | grep -ow '..'
        12
        he
        2!

<comment><USER4> Sure, but a fair bit of the value of the tool is it's consistency across platforms.

There's no point in awk if perl etc are ubiquitous and more consistent.

</comment></comment></comment></comment></comment></comment><comment><USER6> Because for some bizarre reason, "cut" doesn't ship with any decent column selection logic that is the equivalent of awk's $1, $2, etc., even in 2020.

That's like 90% of my use of awk right there. I don't know of any easier equivalent of "awk '{ print $2 }'" for what it does.

Posted partially so the Internet Correction Squad can froth at the mouth and set me straight, because I'd love to be showed to be wrong here.

<comment><USER7> > I don't know of any easier equivalent of "awk '{ print $2 }'" for what it > does.

Does `cut -f2` not work? My complaint with cut is that you can't reorder columns (e.g. `cut -f3,2` )

Awk is really great for general text munging, more than just column extraction, highly recommend picking up some of the basics

Edit to agree with the commenters below me: If the file isn't delimited with a single character, then cut alone won't cut it. You need to use awk or preprocess with sed in that case. Sorry, didn't realize that's what the parent comment might be getting at.

<comment><USER8> It does not. Compare:

        $ echo '   1     2   3' | cut -f2
           1     2   3
        $ echo '   1     2   3' | cut -f2 -d' '

        $ echo '   1     2   3' | awk '{print $2}'
        2

"-f [...] Output fields are separated by a single occurrence of the field delimiter character."

<comment><USER9> Also, the field separator (FS) can be a regular expression.

        FS = "[0-9]"

<comment><USER6> IIRC, there is an invocation of cut that basically does what I want, but every time I try, I read the manual page for 3 or 4 minutes, craft a dozen non- functional command lines, then type "awk '{ print $6 }'" and move on.

<comment><USER10> > IIRC, there is an invocation of cut that basically does what I want

I don't think there is, because cut separates fields strictly on one instance of the delimiter. Which sometimes works out, but usually doesn't.

Most of the time, you have to process the input through sed or tr in order to make it suitable for cut.

The most frustrating and asinine part of cut is its behaviour when it has a single field: it keeps printing the input as-is instead of just going off and selecting nothing, or printing a warning, or anything which would bloody well hint let alone tell you something's wrong.

Just try it: `ls -l | cut -f 1` and `ls -l | cut -f 13,25-67` show exactly the same thing, which is `ls -l`.

cut is a personal hell of mine, every time I try to use it I waste my time and end up frustrated. And now I'm realising that really cut is the one utility which should get rewritten with a working UI. exa and fd and their friends are cool, but I'd guess none of them has wasted as much time as cut.

</comment></comment></comment></comment></comment></comment><comment><USER11> Awk, like shell, is a fraught programming environment, full of silent failure and hidden gotchas. Even for one-liners.

I use shell because I have to, not because I like it. I dread maintaining shell scripts which have a bunch of awk and sed in them.

The Unix ideal of small single-purpose tools and text processing is separable from these old warhorses.

<comment><USER12> scripts are the antithesis of modern software development. No Unit testing, No CI, no monitoring, no consistency, often no source control. Appeals to me as a hacker but only my scripts are intelligible - which is a bad sign. :)

<comment><USER13> I'm not actually convinced unit testing is all that valuable, unless the unit under test has a very clear input -> output transformation (like algorithms, string utilities, etc). If it doesn't (and most units don't), unit tests just encumber you.

<comment><USER14> I'm really glad to read that.

I never understood the whole religion around unit tests. Integration tests are often far easier to write and far more valuable.

Like you said, unit tests are really nice when testing for a known, expected output.

Unit tests that are effectively testing mocks and crazy stubs because your method has side effects? Not for me.

<comment><USER15> It is easy to explain: unit tests give a documented proof that you care about code robustness. It is used more for social and psychological effect than for its advantages. In fact, outside a few domains, unit tests make it harder to evolve software because the more tests you write, the harder it is to make changes that move the design in a different direction. This is, by the way, my main problem with verbose techniques of programming: the more you have to code, the harder to make needed changes.

</comment></comment></comment></comment></comment><comment><USER16> I gave awk a sincere attempt, but I have to say that it wasn't worth it. As soon as one tries to write anything bigger than a one liner the language shows its warts.. I found myself writing lots of helper routines for things that should be part of the base language/library, e.g. sets with existence check. I also had to work around portability issues, because awk is not awk, unlike this post claims. E.g. matching a part of a line is different among different awks.

And some language decisions are just asinine, e.g. how function parameters and variable scope works, fall through by default although you almost never want that, etc..

But hey, you have socket support! Sounds to me like things have developed in the wrong direction.

And of course no one on your team will know awk.

I found the idea of rule-based programming interesting, but the way it interacts with delimiters and sections (switching rules when encountering certain lines) doesn't work well in practice.

I also found the performance to be very disappoinging when I compared it to C and python equivalents.

<comment><USER17> You realize AWK is about 1/100th the size of Python, right? That's like comparing a Leatherman multi-tool to a Craftsman 2000 piece tool set that weighs 1,000 lbs. This matters significantly when addressing compatibility and when building distros that are space constrained.

Awk is there for a reason: to be small. That's why the O'Reilly press book is called "Sed & Awk", because they were orignally written to work together in the early days of unix dating back to the late 70's. Sed (1974) & Awk (1977) are in the DNA of unix, Python is something totally different.

<comment><USER16> First of all I'm not a distro maintainer. I also doubt that people would use awk for seriously space constrained environments. And distros ship both awk and python anyway. And again, I don't understand why they'd support networking but not basic data types/functions.

The only reason I could've seen to use awk was to throw code together more quickly in a DSL.

However this is much less the case than I had hoped. For the one liners there are usually specialized tools like fex that are easier to use and faster (for batch processing).

When I compared my C/python/awk programs the difference was msec/sec/minutes. As soon as I use such a program repeatedly it starts to hurt my productivity. And the development time is not orders of magnitude slower in non-awk languages.

<comment><USER18> > I also doubt that people would use awk for seriously space constrained > environments. And distros ship both awk and python anyway.

Python is absolutely not available everywhere one can find Awk. I've never seen a system with Python but not Awk, but have seen many systems with Awk but not Python (excluding the BSDs, where Python is never in base, anyhow).

Actually, not many years ago I used to claim that I never saw a Linux system with Bash that lacked Perl, but had seen systems with Perl that lacked Bash. (And forget about Python.) This was because most embedded distros use an Ash derivative, often because they used BusyBox for core utilities or a simple Debian install. Perl might not have been default installed, either, but invariably got pulled in as a core dependency for anything sophisticated. Anyhow, the upshot was that you'd be more portable, even within the realm of Linux, with a Perl script than a Bash-reliant shell script. Times have changed, but only in roughly the past 5 years or so. (Nonetheless, IME Perl is still slightly more reliable than Python, but variance is greater, which I guess is a consequence of Docker.)

One thing to keep in mind regarding utility performance is locale support. Most shell utilities rely on libc for locale support, such as I/O translation. Last time I measured, circa 2015, setting LC_ALL=C resulted in _significantly_ improved (2x or better, I forget but am being conservative) standard I/O throughput on glibc systems.[1] I never investigated the reasons. glibc's locale code is a nightmare[2], and that's more than enough explanation for me.

Heavy scripting languages like Perl, Python, Ruby, etc, do most of their locale work internally and, apparently, more efficiently. If you don't care about locale, or are just curious, then set LC_ALL=C in the environment and test again. I set LC_ALL=C in the preamble of all my shell scripts. It makes them faster and, more importantly, has sidestepped countless bugs and gotchas.

For the things I do, and I imagine for the vast majority of things people write shell scripts for, you don't need locale support, or even UTF-8 support. And even if you do care, the rules for how UTF-8 changes the semantics of the environment are complex enough that it's preferable to refactor things so you don't have to care, or can isolate the parts that need to care to a few utility invocations. In practice, system locale work has gone hand-in-hand with making libc and shell utilities 8-bit clean in the C/POSIX locale, which is what most people care about even when they care about locale.

[1] The consequence was that my hexdump implementation, [http://25thandclement.com/~william/projects/hexdump.c.html](http://25thandclement.com/~william/projects/hexdump.c.html), was significantly faster than the wrapper typically available on Linux systems. My implementation did the transformations from a tiny, non-JIT'd virtual machine, while the wrapper, which only supports a small subset of options, did the transformation in pure C code. My code was still faster even compared to LC_ALL=C, which implied glibc's locale architecture has non- negligible costs.

[2] To be fair, it's a nightmare partly because they've had strong locale support for many years, and the implementation has been mostly backward compatible. At least, "strong" and "backward compatible" relative to the BSDs. Solaris is arguably better on both accounts, though I've never looked at their source code. Solaris' implementation was fast, whatever it was doing. musl libc has the benefit of starting last, so they only support the C and UTF-8 locales, and in most places in libc UTF-8 support simply means being 8-bit clean, so zero or perhaps even negative cost.

<comment><USER19> There was a long period of time where it was easy to find a non-Linux Unix with Perl installed but not Bash: SunOS, Solaris, IRIX, etc., admins would typically install Perl pretty early on, while Bash was more niche. Like, maybe 1990 to 2000. Now we're getting into an era where lots of Unix boxes run MacOS, and although they have Bash, it's a version of only archaeological interest. But they do have Perl.

</comment></comment></comment></comment></comment><comment><USER20> Honorary mention of Taco Bell Programming. (fits this genre).

[http://widgetsandshit.com/teddziuba/2010/10/taco-bell- progra...](http://widgetsandshit.com/teddziuba/2010/10/taco-bell- programming.html)

Someone ought to write - Zen and the art of Unix tools usage.

<comment><USER21> >> suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it?

I don't know enough about the 'real way' or the 'taco bell way', but interested to know --- is this doable the way Ted describes in the article via xargs and wget?

<comment><USER22> Yes, absolutely. This is absolutely how ~~we~~ many (most?) of us used to scrap web pages in the Dark Ages.

</comment></comment></comment><comment><USER23> Skip learning of sed and awk and jump straight to perl instead.

      $ perl --help
      ...
      -F/pattern/       split() pattern for -a switch (//'s are optional)
      -l[octal]         enable line ending processing, specifies line terminator
      -a                autosplit mode with -n or -p (splits $_ into @F)

      -n                assume "while (<>) { ... }" loop around program
      -p                assume loop like -n but print line also, like sed
      -e program        one line of program (several -e's allowed, omit programfile)

Example. List file name and size:

      ls -l | perl -ae 'print "@F[8..$#F], $F[4]"'

<comment><USER24> Because syntatically as a language/tool it is super easy to remember. Writing one liners with awk feels more intuitive to me.

Awk example:

ls -l | awk '{print $9, $5}' or ls -lh | awk '{print $9, $5}'

Seems a whole lot simpler. To me. I find if you have to write exhaustive shell scripts then maybe you can look for something more verbose like Perl, I guess.

<comment><USER23> Yep, but you have bug in your awk one-liner.

<comment><USER18> If you mean the lack of quotations, then the behavior is well-defined and is presumably what was intended. Per POSIX,

> The print statement shall write the value of each expression argument onto > the indicated output stream separated by the current output field separator > (see variable OFS above), and terminated by the output record separator (see > variable ORS above).

The default value for OFS is <space> and for ORS, <newline>.

<comment><USER25> > If you mean the lack of quotations,

No, lack of commas in output and broken filenames with spaces.

<comment><USER24> I see your point regarding spaces in names. Suppose I could use FILENAME. But I think my point was made.

</comment></comment></comment></comment></comment></comment><comment><USER26> My favorite Awk reference is this:

[https://ferd.ca/awk-in-20-minutes.html](https://ferd.ca/awk- in-20-minutes.html)

Also, a handy trick is to combine awk and cut. For example I had a log line that had a variable amount of columns just in one field, but immediately after the field was a comma. I cut based on the comma:

cut -d, -f1,2

and then awk'd the last column:

cut -d, -f1,2 | awk '{ print $2" "$5" "$NF }'

So, sometimes awk and cut can help each other.

<comment><USER27> I've been procrastinating learning awk for a while now. Thanks for this, I read it and it's just what I needed.

</comment></comment><comment><USER28> Another good resource is "Why you should learn just a little Awk: An Awk tutorial by Example"[0]

[0] [https://gregable.com/2010/09/why-you-should-know-just- little...](https://gregable.com/2010/09/why-you-should-know-just-little- awk.html)

</comment><comment><USER29> I've been parsing some documents converted from PDF (using the Poppler library's "pdftotext" command with the "\--layout" option).

I found that reading these -- sort-of half-assed structured data, but with page-chunked artefacts and idiosyncrasies -- was difficult on a line-by-line basis, and thought idly "this would be a lot easier if I could process by page instead".

Text was laid out in columns, and the amount of indenting (the whitespace between columns) was significant. So preserving this somehow would be Very Useful.

Suddenly those pesky '^L' formfeeds were an asset, not a liability. Let's treat the _formfeed_ ("\f") as a _record_ delimiter, and the _newline_ ("\n") as a _field_ delimiter. We can parse out the actual columns based on witespace, for each line:

        BEGIN { RS="\f"; FS="\n" }
        {
            pageno = NR
            lines = NF
            for( line=1; line<=lines; line++ ) {
                ncols = split( $line, columns, " {2,}", gaps )
            }
        }

This gives me:

\- The running tally of pages.

\- Each _line_ of the page as an individual _record_.

\- Via the split() function, an array of _columns_ separated by two or more spaces, which are saved as an array of _gaps_ so I have the whitespace to play with.

Edge cases and fiddling ensue, but that's the essential bit of the code there.

Since the lines are an array, I can roll back and forth through the page (basically being able to read forward and backwards through the text record), testing values, finding out where column boundaries are, etc., and then output a page's worth of content, transposing to a single-column format, with appropriate whitespacing, when done.

In testing and debugging the output (working off of 20+ documents of 100s to ~1,000 pages), a lot of test cases, scaffolding, diagnostics, etc., have been created and removed to make sure the Right Things are happening. Easy with awk.

<comment><USER29> And to be clear: After realising this ... and knowing what to look for ... I found this documented in the GNU Awk User's guide:

[https://www.gnu.org/software/gawk/manual/html_node/Multiple-...](https://www.gnu.org/software/gawk/manual/html_node/Multiple- Line.html)

</comment></comment><comment><USER30> I love awk, but I'm still waiting for the structural regex awk: [http://doc.cat-v.org/bell_labs/structural_regexps/se.pdf](http://doc.cat-v.org/bell_labs/structural_regexps/se.pdf)

<comment><USER31> I just learned about SE a month or so ago, and it is indeed pretty awesome. I tried out the "vis" editor, using SE to create multiple cursors that I then manipulated with vi-like commands. That was a pretty sweet use case.

</comment></comment><comment><USER32> Awk is a command I turn to time and again. For me it's the single most valuable command for enabling the piped single-purpose pattern.

As an example, if I want a sorted list of all open files under the home directories on CentOs I can do this:

lsof | awk '{ print $10 }' | grep ^/home/ | sort | uniq

<comment><USER33> Don' stop there. You've solved a real problem in your life, and you might want that information another day. Make a tiny script that encapsulates it. Generalize it a tiny bit, and give it a memorable name (perhaps lsof-tree). That done, you can stop worrying about the mechanics of the solution and build on it.

      #! /bin/bash
      # lsof-tree: list open files in a given directory tree (default /home)

      NAME=9 # set to 10 for CentOS
      BASE="${1:-/home}"

      lsof | awk -v NAME=$NAME '{print $NAME}' | grep "^$BASE" | sort -u

<comment><USER34> YMMV, but I find it easier and faster to know the basic utilities and how to compose them with pipes than remembering the name of a zillion such scripts.

I guess it depends on how often you need that particular pipeline. Every day? Sure, make a script. Every few months? Nah, I won't remember it anyway,or probably I remember that I've made a script like that but then I have to start searching my bin directory in the end using more time than just writing the pipeline in the first place.

</comment></comment></comment><comment><USER35> It is fast, robust, and frequently far more performant than a lot of modern tools that can be overkill for most data manipulation. I use it all the time in our ETL processes and it always works as advertised.

<comment><USER23> Perl is much faster[0], with much more features, with bunch of ready to use libraries, with package manager (CPAN), and similar syntax to awk.

Why you use awk?

[0]: [http://rc3.org/2014/08/28/surprisingly-perl-outperforms- sed-...](http://rc3.org/2014/08/28/surprisingly-perl-outperforms-sed-and-awk/)

<comment><USER36> Perl was my second programming love, but awk is much shorter and easier to remember for the simple cases where I need it.

Remembering which Perl command-line arguments simulate awk's line-by-line processing is harder than just remembering awk.

<comment><USER37> I use Perl similarly to awk if I need to use regex rather than white space delimited fields.

I think if you know Perl really well and can remember the command line arguments - particularly -E, -n, -I and -p - then it's a good swap in substitute for grep, sed, awk, cut, bash, etc when whatever 5 min task you're working on gets a tiny bit more complex.

Similarly a decent version perl 5 seems to be installed everywhere by default.

I'm curious to know if anyone would say the same about python or any other programs? I'm not particularly strong in short python scripting.

<comment><USER36> I would say Perl's native support for regular expressions makes it more useful on the CLI than Python, but Python is also very low on my preferred languages list.

I do, however, use it for JSON pretty printing in a pipeline: python -mjson.tool IIRC.

</comment></comment></comment></comment></comment><comment><USER38> Last week I threw out AWK and replaced it with Ruby (Could've been Python, Perl or PHP even).

Because AWK is not suited for CSV. Please prove me wrong!

I had to parse 9million lines. Some of which contain "quoted records", others, same column, are unquoted. Some contain comma's, in the fields, most don't. CSV is like that: more like a guideline than actual sense.

Two hours of googling and hacking later, I gave up and rewrote the importer in Ruby, in under 5 minutes.

Lesson learned: I'll stay clear of AWK, when I know a oneliner of Ruby (or Python) can solve it just as well. Because I know for certain the latter can deal with all the edgecases that will certainly pop up.

<comment> F-0X > I had to parse 9million lines.

Awk would chew through that no problem.

> Some of which contain "quoted records", others, same column, are unquoted.

In which case, there is the FPAT variable which can be used to define what a field is. FPAT="\"[^\"] _\ "|[^,]_", which means "stuff between quotes, or things that are not commas", would probably have worked for you. (EDIT: Looks like formatting has gotten hold of my FPAT and I don't know how to stop it... hopefully it is still clear where asterisks should be)

> Some contain comma's, in the fields, most don't. CSV is like that: more like > a guideline than actual sense.

Well, I would say that's absolutely false. You can't just put the delimiter wherever you fancy and call it a well-formed file. Quoting exists for the unfortunate cases your data includes the delimiting character (ideally the author would have the sense to use a more suitable character, like a tab).

This is just a retort to prevent your post from dissuading readers from awk, which is a fantastic tool. If you actually sit for half and hour and learn it rather than google to cobble together code that works, it is wonderful. I also don't think it is valid to base your judgement of a tool on what was apparently garbage data.

<comment><USER39> Garbage and poorly specified csv files are a fact of life and people have to deal with them all the time.

But if you want to be in a world where people only deal with well specified files like RFC 4180 (for some definition of well specified), your quick field pattern doesn't conform. It doesn't handle escaped double quotes or quoted line breaks. If you're using your quick awk command to transform an RFC 4180 file into another RFC 4180 file you've just puked out the sort of garbage you were railing against.

While awk is a great tool if you're dealing with a csv format with a predictable specification, and probably could be made to bend to the GP will with a little more knowledge, it gets trickier if you're dealing with handling some of the garbage that comes up in the real world. What's worse is the programming model leads you down the path of never validating your assumptions and silently failing.

I love awk for interactive sessions when I can manually sanity check the output. But if I'm writing something mildly complex that has to work in a batch on input I've never seen, I too would reach for ruby.

</comment></comment></comment><comment><USER40> Parenthetically, since there are a bunch of UNIX greybeards on HN: if anyone has artwork of the AWK t-shirt I will happily pay any reasonable price. The shirt has a parachute-wearing bird about to jump from an airplane and is captioned with AWK's most famous error message: awk: bailing out near line 1.

Contact information is hn handle @ yahoo.com.

</comment><comment><AUTHOR> Here are some clever one-liners for awk [1] Please be sure to add your own.

[1] - [https://www.commandlinefu.com/commands/matching/awk/YXdr/sor...](https://www.commandlinefu.com/commands/matching/awk/YXdr/sort- by-votes)

<comment><USER5> I have a repo dedicated for some of the cli text processing tools like grep/sed/awk/perl/sort/etc. Here's my one-liner collection for awk [1]

[1] [https://github.com/learnbyexample/Command-line-text- processi...](https://github.com/learnbyexample/Command-line-text- processing/blob/master/gnu_awk.md)

<comment><AUTHOR> That is a very well written set of examples. Clear and concise. Thankyou!

</comment></comment></comment><comment><USER41> Hmm. I'm sure this question will induce a flamewar of practical "#NeverAwk"-ers fighting toolbelt bloat, versus tech-hoarding AWK apologists arguing against throwing something out given if fills <niche>.

Here's the thing: these arguments all too commonly focus on subjective notions of "simplicity", and toy examples divorced from _actual_ common practise, and or solid comparable _benchmarks_.

Show me a range of practical examples, for each competing env (awk, sed, perl, python, ruby, maybe bash).

Include:

\- time it takes to teach a total novice (the time it takes to learn whatever is needed _that_ example, not the entire language)

\- how easy it is to recall said knowledge at a later date

\- how fast example is multiplied by how much you are likely to use it = actual time saved in terms of execution. for small, fast examples the difference is irrelevant, a 10x speedup that is 0.1s vs 0.01s is meaningless.

\- how _extendable_ an example is. Hence the original example should include a series of extensions to the original task, to demonstrate how flexible / composable they are: e.g. task 1) count lines in a file; task 2) count lines in a file, then add 42 to it;

I suspect awk falls behind in practicality vs perl (which can do simple one- liners, but also more complicated constructs), but perhaps has a hidden virtue wrt speed in more expensive tasks, ala [https://news.ycombinator.com/item?id=17135841](https://news.ycombinator.com/item?id=17135841) or [https://news.ycombinator.com/item?id=20293579](https://news.ycombinator.com/item?id=20293579)

<comment><USER42> > Here's the thing: these arguments all too commonly focus on subjective > notions of "simplicity", and toy examples divorced from actual common > practise, and or solid comparable benchmarks.

I think your missing the point of awk. The O'Reilly sed and awk book has some complex examples, but when I look at my own usage they are all toy examples within a much larger scope. It's more like a special DSL extension for my shell than something I'd pick to build the entire solution, so a comparison to perl, python and ruby don't really make sense, they are general purpose languages but awk just has a couple of features that make it a very specialized yet useful one.

As an example, a have a system for importing and parsing log files that mostly done from a shell script, awk is used in two parts. The first is to transform a structured and easy to read file (records '\n\n' separated) into a csv easier to consume for bash, there's probably quite a few options to do this from tr to bash and it's done inline. The second is to filter the results down to what I need, so I have scripts like:

      #!/usr/bin/awk -f
      /some common error I don't care about/ { next } #skip line
      /other common error/ { next }
      /Error/ { print $0 } #this prints error lines, alternatively:
      /Error/ && !errors[gensub($1", "", "g", $0)]++ { print $0 } #print each error once
      {next} #skip everything else

Apart from one single line which wasn't in the original that's something you could teach a total novice in minutes, the /pattern/{action} syntax is about as simple as programming can be. Execution speed could probably be improved with a specific program but I suspect the bottleneck would be the spinning disk anyway, I run this over hundreds of MB every few minutes and it's not a problem, when I run it manually it's near instantaneous, I spend longer waiting for the desktop calculator to open up these days.

<comment><USER41> Can you give an example of when you'd choose AWK over perl?

perl is general purpose, but that doesn't mean it can't be used for one- liners.

</comment></comment></comment><comment><USER43> do you know that feeling when a problem is too much for the shell but too little for C? that's where Perl was supposed to fit.

in everyday life however, many small problems are a bit too much for the shell but too little for Perl/Python/whatever.

awk fits very nice in there.

</comment><comment><USER44> > _Imagine programming without regular expressions._

I live in this future and it's beautiful.

Steps to programming without regular expressions:

1) find a PEG library for your language of choice

There is no step 2.

<comment><USER45> Sorry for asking such dub question, but what's a PEG library?

<comment><USER46> Parsing expression grammar;

It is a recursive decent parser with the tiny tweak that productions are ordered (not a set) and short circuit.

A 90's language you did not have to imagine that saved you from regex in this way was[is] REBOL with it built in `parse` dsl.

examples here [http://www.rebol.org/search.r?find=parse&form=yes](http://www.rebol.org/search.r?find=parse&form=yes)

[0][https://en.wikipedia.org/wiki/Parsing_expression_grammar](https://en.wikipedia.org/wiki/Parsing_expression_grammar) [1][http://www.rebol.com/](http://www.rebol.com/)

</comment></comment></comment><comment><USER47> So why for example Awk instead of let's say Perl? The argument of "it's likely to be installed" isn't very compelling.

<comment><USER29> If you're versed in both, it comes down to taste, though there really _are_ cases where you'll have awk (usually via busybox) but _not_ Perl. OpenWRT comes to mind (just verified it's not present by default, though yes, packages are available).

For a huge number of simple tasks, awk is available and sufficient. It's largely a subset of Perl, so yes, there's some skills overlap, but there are times where knowing awk is the right tool and the _available_ tool will pay off.

<comment><USER47> I like Perl regexes more though, especially aliases. Using \d is a lot neater than [0-9] or [[:digit:]].

<comment><USER29> I hear Perl may support those ;-)

(Use the right tool for the job.)

</comment></comment></comment></comment><comment><USER48> Because you often don't need anything else to deal with a surprisingly large class of tasks in IT.

</comment><comment><USER49> After a decade of writing application software in C family languages, I am now working on a big devops effort in a Linux environment. A lot of the syntax is janky but it's pretty amazing what you can accomplish with a shell and the Linux command line tools.

</comment><comment><USER50> In the early aughts, fresh out of college, one of my first projects was to take semi-structured text from database blobs and convert it to XML. I quickly realized it was not a task to be fully automated, too many edge cases that really required human judgement because the it was only very loosely semi- structured. I turned to sed, awk, and pico. Sed & awk did the 99%, and dumped into pico when it didn't know what to do and I would resolve the issue. Doing it semi-supervised in this way was 100 times faster than doing it all manually, and 10 times faster than full automation, and probably more accurate.

But it was the ability to string together these types of command line tools that made it possible.

</comment><comment><USER51> The author forgot one: C-like syntax

I believe this was intentional by the authors.

In the early days of UNIX, I think more of its users knew C. Today, it is probably a much smaller number. However, learning AWK today, IMO, can help someone who also intends to learn C.

</comment><comment><USER52> "Imagine programming without regular expressions."

Best of all worlds. I wish there would be a way to get back the time I spent with debugging edge cases of different regex implementations on various OS's.

</comment><comment><USER53> I have used awk for decades, but I simply stoped and dropped the habit of using it along with sed and perl. Nowadays I would rather write a program than a script just to avoid memorizing all these tricks and hacks and glitches

</comment><comment><USER54> It would be so great if awk had a csv mode. For whatever reason (Excel), CSV seems to be the default text format for field oriented data.

Maybe I'm dumb but I've never come up with a separator regex that is quite right.

<comment><USER29> The simple case is:

        FS = ","

or:

        awk -F ',' <program>

If you're working with CSV data that has quoted strings with embedded commas, FPAT is your friend:

        FPAT = "([^,]+)|(\"[^\"]+\")"

See: [https://www.gnu.org/software/gawk/manual/gawk.html#Splitting...](https://www.gnu.org/software/gawk/manual/gawk.html#Splitting- By-Content)

<comment><USER54> just want to say thanks for this, I haven't had to deal with CSV for years and now only a week after you posted this I needed it.

brew install gawk

good to go

:)

</comment></comment></comment><comment><USER55> There used to be a website shared on HN a lot, which was a table of: Task, SED, Awk. It was really useful but I haven't seen it for years.

</comment><comment><USER56> Instead, learn fex:

[https://www.mankier.com/1/fex](https://www.mankier.com/1/fex)

</comment><comment><USER57> Does someone know how I have to invoke awk to make sure that the awk code I wrote is POSIX compliant?

</comment><comment><USER21> Needs a '(2016)' at the end of the title.

<comment><AUTHOR> [updated] ty.

</comment></comment><comment><USER58> Why not just use Emacs for simple scripts and text manipulation? Way easier, lots of one liners as well, just as expressive.

<comment><USER59> Too big and slow. Also I disagree with the expressiveness, (emacs-has-chosen- a-very-longwinded 'way (to (express things))) that would be shorter in languages with more syntax.

</comment></comment><comment><USER60> (2016)

</comment><comment><USER17> When writing a shell script the robustness/reliabilty can be inferred from it's scope:

1\. Uses shell-only commands (echo, for) - most robust; but things like basename/dirname and regex's vary by shell (sh, bash, zsh, ksh)

2\. Uses /bin - might run into missing a binary but not likely, still robust and allows a richer set of tools (e.g., uname, chmod and admin-ish things live in /bin)

3\. Uses /usr/bin - runs risk if missing packages, likely not very robust (packages drop things in here, like gzip, yacc, gcc)

4\. Uses /usr/local/bin or /opt/local/bin - definitely requires package installs, least robust</comment>


Best-selling introductory psychology books give misleading view of intelligence - <AUTHOR> https://digest.bps.org.uk/2018/03/08/best-selling-introductory-psychology-books-give-a-misleading-view-of-intelligence/ <comment><USER2> Gottfredson's paper is a paper that I have read, and I see it is being used as the analytical framework for criticizing psychology textbooks. I do research for writing popular writings on psychology, so I have a whole bunch of introductory psychology textbooks in the office where I am typing this, and I have to agree that introductory psychology textbooks leave a LOT to be desired in representing the consensus of modern psychology research. That's true about research on human intelligence and true about any other psychology topic: the introductory textbooks only do a so-so job.

That said, one might wonder where to find good information about current psychology research. Sometimes there are review articles that update practitioners on current research, which are incidentally read by scientists in other disciplines. I'll note for the record that not all psychologist agree EITHER with the review article I will link here, but it is a good readable account of current issues in the psychological research on human intelligence and well worth a read for Hacker News participants who are curious about these issues. It refers to many of the most important papers in the field, most of which I have read over the last three decades.

Nisbett, R. E., Aronson, J., Blair, C., Dickens, W., Flynn, J., Halpern, D. F., & Turkheimer, E. (2012). Intelligence: New findings and theoretical developments. American Psychologist, 67, 130-159. doi:10.1037/a0026699

(Disclaimer: I have met many of the researchers on human intelligence, including Gottfredson, at professional conferences, but my views of what the overall research says and who has the best leads on open research questions are my own.)

[http://people.virginia.edu/~ent3c/papers2/Articles%20for%20O...](http://people.virginia.edu/~ent3c/papers2/Articles%20for%20Online%20CV/Nisbett\(2012\)%20.pdf)

</comment><comment><USER3> Intelligence studies : Left :: Climate science : Right

The Left views the expert consensus on intelligence in much the same way that the right views the expert consensus on climate change. The consensus view is rejected not because of evidence, but because of what it does to their model of the world and fear that it will open the door to great abuses in the future (increased government regulation for climate change, and increased racism and discrimination for intelligence studies).

<comment><USER4> I'm on the far left, and while I am comfortable with whatever the scientific consensus on intelligence is at the moment and how it develops in the future, I am very recalcitrant about allowing that understanding into politics, policy or most discussions I want to have about anything.

Looking towards the past on how (pseudo)science about intelligence has been leveraged to commit atrocities, and looking at the reasons _why_ so many on the right really really really like bringing up IQ and whatnot, I'll pass. At best the conversation dissimulates the urge to justify hierarchies of dominance-- the same fucking reason why the right likes to bring up anything from biology. At worst you get enthusiastic eugenicists; meeting such people at, say, conferences or other programmer venues (i'm sure programmers skew both towards higher IQ and views that the strong should crush the weak) is like stepping on a cow pie.

Not to mention, I know plenty of people who would score high on intelligence tests, perhaps much higher than I would. Some of them seem to leverage that intelligence to be far stupider than anyone dumber than them could manage (us programmers again...).

The point being-- the gap between scientific research on intelligence and what it means for A) society, and B) my personal life is a chasm I have no reason to bridge.

<comment><USER5> > The point being-- the gap between scientific research on intelligence and > what it means for A) society, and B) my personal life is a chasm I have no > reason to bridge.

The problem is that well-meaning people on the left come to the table full of assumptions and intuitions about how reality works and what causes inequality, and base their policy positions on these. People on the right bring up biology because it can expose weaknesses in these assumptions and beliefs.

> Looking towards the past on how (pseudo)science about intelligence has been > leveraged to commit atrocities

You can also look at how blank slatism and Lysenkoism were leveraged to commit atrocities in the USSR. Science and pseudo-science have been use to justify all kinds of horrible regimes. That isn't a license to ignore reality and form policy based on unfounded beliefs.

<comment><USER4> > That isn't a license to ignore reality and form policy based on unfounded > beliefs.

Did I say anything about unfounded beliefs? I didnt. I even said I accept research on intelligence. Not basing policy on X is not the same thing as basing policy on X.

But I'll bite. Tell me a way that intelligence research should inform policy that isn't immediately morally repugnant.

<comment><USER5> > Did I say anything about unfounded beliefs? I didnt. I even said I accept > research on intelligence.

I wasn't commenting on your personal beliefs. I was commenting on your position that certain true facts should not be spoken in a political context. To me, that is the same as saying that people should be free to make policy without being challenged if their beliefs are incorrect.

> But I'll bite. Tell me a way that intelligence research should inform policy > that isn't immediately morally repugnant.

No Child Left Behind in the USA is (I think) widely regarded as a failed piece of legislation. It is based on the idea that the only reason kids don't succeed in school is because schools are failing them. It heavily penalized schools that can't get their achievement scores up. Intelligence research can provide one leg of a counterargument that kids are different in all kinds of ways, and proscriptively mandating equality of outcome makes for bad policy.

</comment></comment></comment></comment></comment><comment><USER6> It seems like there's a subset of psychological researchers that want so bad to debunk the years of research into IQ. I get that there are some uncomfortable implications that arise from accepting IQ as a measure of overall intelligence, but, from my understanding, IQ is an excellent predictor of income, social status, and academic & job performance.

My opinion is that we are doing society a huge disservice by dismissing IQ.

<comment><USER7> Yes, but then again, I'll posit that having Blue Eyes is correlated with income. But I'd like to see someone to try to defend that position scientifically.

In reality, all I've done is created an artificial group which is heavily skewed towards richer countries. The color of the eyes had nothing to do with IQ, and yet it's correlated.

So what, if anything, is the IQ test measuring? How do you know you haven't simply written a test which measures how much time the person has to think about brain teasers? Researchers should ask these questions to determine if the correlation is actually caused by anything real, or if it's better explained by some other underlying factor.

<comment><USER8> >How do you know you haven't simply written a test which measures how much time the person has to think about brain teasers?

Come on man, do you really think a century's worth of psychologists never considered such trivial questions?

<comment><USER9> This kind of response is, in my opinion, anti-academic.

A better response would be too try to find some literature on the subject.

<comment><USER8> One aspect of IQ is your ability to solve certain kinds of abstract problems. OP asked what if the IQ tests just test your ability to solve certain kinds of abstract problems.

OP's post didn't seem like a genuine academic inquiry but instead a weak emotional attempt to dismiss something they don't like.

</comment></comment></comment></comment></comment><comment><USER10> Two comments:

\- Lay people seem to have a very good intuition about the actual heritability of intelligence (taken to assess the genetic and environmental determinants of intelligence). [0]

\- If you'd like to read an up-to-date and thorough literature review on human intelligence, I highly recommend "Intelligence: All That Matters" by Stuart Ritchie.

[0] [https://news.ycombinator.com/item?id=16468694](https://news.ycombinator.com/item?id=16468694)

</comment><comment><USER11> Psychology as a field is one that is constantly becoming out of date. General consensus amongst psychologists often doesn't reach academic text quickly, because of the bureaucracy that exists inside the APA (American Psychological Association). See the controversy around the most recent version of the DSM.

I've observed that academics in the field will often take theory as most likely fact, if only because the actual published academic material hasn't caught up yet, but they've all been to the various conferences that are relevant to their particular field (social psychology, cognitive, etc).

Any study that is citing materials from over 20 years ago is grossly outdated. Many of the materials from even 10 years ago are becoming outdated. And now, more recently, we've started to examine the racial bias that exists in our systems of research, so I expect past material to come under even further scrutiny.

<comment><USER12> Some people have researched this.

I remember reading "The Half-Life of Facts: Why Everything We Know Has an Expiration Date" by Arbesman, and IIRC psychology papers had some of the lowest half-lives of all the sciences. In math the half-life of a paper was something like 85 years, but in pysch it was closer to 10. I forget the exact statistics and don't have the book here but it was pretty interesting.

Scientometrics and Eurekometrics are two areas of research that delve into this sort of thing more.

</comment></comment><comment><USER13> The most remarkable thing BY FAR: That some expert(s) are actually looking at and critiquing introductory textbooks!

This type of issue is present in nearly every field. Education would be dramatically improved if more actual experts reviewed more of the introductory materials (ideally all the way down to elementary school level).

</comment><comment><USER14> >> To identify factual inaccuracies, Warne's team used as a benchmark a consensus statement on intelligence research published in 1997 by Linda Gottfredson et al

Is the best way to define 'fact' really a twenty year old study?

Also, even if the results are believed, it doesn't seem to be a big deal. Afterall, they're reporting an average of 1 factual inaccuracy per book. When each book presents hundreds or thousands of truth claims, is 1 factual inaccuracy per book noteworthy? I'm willing to bet a much larger percentage of the 'scientific consensus' they define as 'fact' is in fact wrong.

<comment><USER2> I know eminent IQ researchers back at the time who declined invitations to sign on to the Gottfredson consensus statement (with good reason, in my opinion). On the other hand, it is still a reasonably fair statement of a lot of the consensus among IQ researchers, but as you say now twenty years out of date. (I read the consensus statement right after it was published.)

</comment></comment><comment><USER15> Link to the actual paper: [http://psycnet.apa.org/fulltext/2018-07714-001.pdf](http://psycnet.apa.org/fulltext/2018-07714-001.pdf)

<comment><AUTHOR> Thanks! I wasn't sure whether to link the summary blog post or the original paper. Having both is definitely good for the discussions here.

<comment><USER16> BPS is hit and miss. Sometimes they're great; sometimes they really mangle the science.

[https://www1.bps.org.uk/networks-and-communities/member- micr...](https://www1.bps.org.uk/networks-and-communities/member- microsite/division-clinical-psychology/understanding-psychosis-and- schizophrenia)

First response from 2014: [https://www.nationalelfservice.net/publication- types/report/...](https://www.nationalelfservice.net/publication- types/report/understanding-psychosis-and-schizophrenia-a-critique-by-laws- langford-and-huda/)

Newer response to updated document from 2017: [https://www.nationalelfservice.net/mental- health/psychosis/u...](https://www.nationalelfservice.net/mental- health/psychosis/understandingpsychosis/)

It's a bit worrying that they get so much wrong.

</comment></comment></comment><comment><USER17> This may be a bit off the deep end, but in 2014 the Journal of Intelligence had a special issue on the state-of-art for intelligence research (in psychology). It's open access :).

[http://www.mdpi.com/journal/jintelligence/special_issues/int...](http://www.mdpi.com/journal/jintelligence/special_issues/int_where)

<comment><USER18> Just to make sure I understood, that link is a collection of hand-picked papers? I expected it to be a review article, summarizing a bunch of past work. Just wanted to be sure that I was looking at the right thing.

<comment><USER17> Yeah, that's right--it was a special issue of the journal, where they invited many people to write review articles on intelligence.

</comment></comment></comment><comment><USER19> * Intelligence is a statistical construct defined based on a group of rather arbitrary tests.

That one is true.

" _This contradicts the 1997 consensus statement which tackles this issue and concludes that "intelligence tests are not culturally biased"._ "

Well, glad we got that out of the way.

<comment><USER15> >* Intelligence is a statistical construct defined based on a group of rather arbitrary tests.

>That one is true.

In what sense are they "rather arbitrary"? It sounds to me that you might be suggesting that there is little correlation between performance on different IQ tests or different parts of the same IQ test. But the authors of the paper claim (with cited evidence) that "a common _g_ factor accounting for about half of variance on cognitive tasks has been found across many human cultures (e.g., Carroll, 1993; Dolan, 2000; Dolan & Hamaker, 2001; Frisby & Beaujean, 2015; Gurven et al., 2017; Reuning, 1972)".

<comment><USER19> "Arbitrary" in the sense of not being chosen based on an understanding of what "intelligence" is.

Please correct me if I'm wrong, but _g_ is defined by noting that there is a correlation in performance on loosely defined cognitive tasks and assuming that there is a single important variable explaining that correlation.

<comment><USER20> What you call an assumption actually sneaks in at a slightly different place. When you design multiple tests that can only correlate positively or not at all, you get a matrix with nonnegative entries. Of course when you factor it you find a dominant factor, there has to be a dominant factor (or some, if matrix has zeroes, unlikely in a noised case). Frobenius–Perron theorem guarantees that dominant factor for you. You can make up any set of traits, test for positive correlations this way and always get some dominant factor. Mathematics guarantees there will always be a _g_ factor whether from real influence or as an artifact of factorization, if your tests can't correlate negatively.

<comment><USER19> How does one choose between a single _g_ factor and multiple _g_ factors?

<comment><USER20> They do not appear in the real data, you have one g stronger than all others. This g has as much explanatory power as noise. Frobenius–Perron guarantees it would always appear whether there is one strong underlying influence on the data or hundreds unrelated ones.

Factor analysis used to establish g doesn't and can't have the power to tell. There are numerous and intense attempts to paper over this mathematical fact by circumstantial evidence, I'm not up to date with all the contortions. I don't think they can make linear algebra a causal theory: when you want to replace your data with linear combination of factors there will happen to be a strongest factor.

To somewhat answer the direct question: how to check factors. There is Confirmatory Factor Analysis that checks how factors decay. You take out factors you see as accounting for most of the variance in the data and check if indeed not much partial correlation is left. This has most power to negate an influence but next to none or very little power to confirm it not being spurious.

These wikipedia pages don't look any good, but I don't want to evaluate that right now or chase better material [https://en.wikipedia.org/wiki/Confirmatory_factor_analysis](https://en.wikipedia.org/wiki/Confirmatory_factor_analysis) [https://en.wikipedia.org/wiki/Partial_correlation](https://en.wikipedia.org/wiki/Partial_correlation)

</comment></comment></comment></comment></comment></comment><comment><USER21> One of the most shocking things for me when I went through highschool a decade ago was how new things we learned about maths/physics showed the previous things we learned were not accurate. Things like relativity and time dilation, integrals/derivatives, etc.

This was further proved (but not that shocking then) when I went through Engineering in the University with fluid mechanics et al.

I think when learning a whole new topic, specially ones experienced at a practical level, we need to start at a point with some broad oversimplifications. Not saying that is what happens here since I know little about psychology, but I would expect the same to apply here. Then we will be ready to dig deeper.

<comment><USER22> When I was in elementary school my dad constantly pointed out that they were teaching me nothing but lies. Sometimes he'd ask me what we were learning and then he'd give me questions to fluster the teacher. Teachers either loved it, or hated it (most hated it).

For some topics it makes sense to start with a simplified view, but I don't understand the general trend of insisting it's actually correct while they do it. It's already hard to convince some kids to learn, when they later find out that most of it was lies it has to sap their motivation further.

The simple solution, when you're dealing with a simplified/incorrect version of something for educational purposes, admit that's what you're doing, and why.

An example of this madness: in my high school physics/chemistry class the teacher asked how many states of matter there are are, I raised my hand and said 4. She said, "Wrong, there are three." I said, "Solid, water, gas, and plasma. Four." Her response, "Look in your book, what does it say? THREE: water, solid, and gas."

I told her that plasma is the most common form of matter in the universe and a textbook wouldn't change that.

I got detention. I got detention for insisting plasma exists, in a high school science class.

<comment><USER23> This is a general problem with organization and counting of abstract categories. All of my science, history, and social science courses in middle/high school were pretty much garbage, and as far as I can tell that is the norm in the US (and many other places; not through special fault of the teachers, but because the curriculum/textbooks are insipid).

"State of matter" is not a perfectly defined property, but only a simplified model of high-level aggregate behavior/interactions; individual particles have simpler interactions which are not categorized the same way. Scientists are happy to admit a new "state of matter" whenever they manage to observe a situation that doesn't behave according to their previous models, and are happy to admit that the states of matter are just a simplified model, but some petty bureaucrats (like, sadly, many teachers) can't handle this.

You can similarly get into trouble arguing about how many "kingdoms" there are of eukaryotes, or how many Romance "languages" there are, or how many "continents" there are, or the difference between crops and weeds or a fetus and a person or plagiarism and research.

Quizzing students on how well they can memorize and recall labels (and even worse, punishing them for knowing additional labels) is not just an incredible waste of human time and attention, but is actively harmful insofar as it disempowers students to ask and investigate their own questions and discourages them from seeking a deeper understanding of what they are learning.

I really appreciate the beginning of the MIT electronics textbook [https://amzn.com/1558607358](https://amzn.com/1558607358) which explicitly describes the abstractions used and their limitations, up front. Rare economics textbooks also do a decent job of this.

<comment><USER21> Oh yeah, I got that in University. If I remember correctly from thermodynamics, high pressure and heat above the triple point [1] makes the change from liquid to gas to be a continuum and not a discrete jump.

[1] [https://en.m.wikipedia.org/wiki/Triple_point](https://en.m.wikipedia.org/wiki/Triple_point)

</comment></comment></comment></comment><comment><USER24> It's probably worth noting that the "consensus statement" cited here was only signed by 52 of the 131 researchers it was sent out to.

[https://en.m.wikipedia.org/wiki/Mainstream_Science_on_Intell...](https://en.m.wikipedia.org/wiki/Mainstream_Science_on_Intelligence)

<comment><USER25> It's also worth noting that judging from the examples provided, the methodological approach of Warne et al. is to suggest textbook authors are guilty of a specified fallacy if they make any statement which appears to conflict with Gotfredsson's listed points in any way (even if that statement is itself cited, quite possibly true and not actually presented as a refutation - fallacious or otherwise - of any claims made about intelligence)

</comment></comment><comment><USER26> Hard right: everybody should take an IQ test Libertarian right: I take drugs to increase my IQ Libertarian left: there is no IQ Hard left: everybody has the same IQ

</comment><comment><USER27> When laypeople misuse science and maths, (I am in this context a layperson btw) the consequences depend on the applicability of the subject to ordinary discourse and events.

Because I don't work in bomb science, power engineering or nuclear medicine, If I misuse facts about nuclear fission making decisions to hire a new staffer, the consequence is low. Nothing about nuclear fission informs what happens in the workplace. So the colloquial lack of understanding on my part as to what science actually says about nuclear physics is low.

When I mis-use understanding of I.Q. and g to make a hiring decision, the consequences are huge for the person hired and the organisation. Assumptions about normative behaviour, likely success, future hiring, application of applied knowledge, ability to learn, all these things have direct meaning and importance. The impact on the hire, or non-hire is also immense. Future promotion, reward, positional authority can all be affected by somebody in the psych space, working in H/R, mis-applying domain-specific knowledge around IQ and intelligence.

Its not nuclear physics, but its being mis-applied. Thats the problem. (that it's being mis-applied: not that it isn't nuclear physics. Although a quality of its mis-application _is_ about it not being _hard_ science: its guesswork, rei-fied through G and other statements of single-value measure against an imprecise measurement. I don't actually think social science is well named.)

</comment><comment><USER28> This doesn't make any sense. He's taking a study over 20 years old designed to measure academic consensus and then using it to try and disprove 29 textbooks authored by leading psychologists. Couldn't the latter be said to be a (much more recent) form of academic consensus? Why should a measure of academic consensus over 20 years old be used as some kind of evidence against newer ideas?

</comment><comment><USER29> Kinda related to this, probably a good idea to re-evaluate EQ while we're at it. Although, my gut feeling is that a healthy mix of IQ and EQ makes good worker bees (if that's the metric we're interested in).

[https://en.wikipedia.org/wiki/Job_performance#Emotional_inte...](https://en.wikipedia.org/wiki/Job_performance#Emotional_intelligence)

</comment><comment><USER30> Those books are written by psychologists, maybe they have a good reason to be ... imprecise, underspecific? Be honest: nobody "knows"; Instigate thought: don't serve the truth on a silver tablet. That seems to be true for every advanced topic. Reverse psychology is still psychology.

</comment><comment><USER31> I'm so confused now. Who is right about this?

<comment><USER32> For decades, the media have been misreporting what is known about intelligence, because it is politically incorrect. A good book documenting this is "The I. Q. Controversy: The Media and Public Policy" by Stanley Rothman and Mark Snyderman (1988).

<comment><USER33> That "political correctness" as you call it is masking a much deeper problem.

If you believe that there are innate differences in intelligence then what do you do about it?

If we are not going to discriminate based on innate differentiators like gender, race, etc, then how can we discriminate based on intelligence, if it is innate?

The idea of social justice for people with innately low intelligence is very disruptive so better to pretend they don't exist.

<comment><USER32> It is essential to discriminate in hiring on the basis of qualities that are relevant for the job. If a person's race or sex does not affect his or her ability to perform as a programmer, but his or her intelligence does, then of course you use perceived intelligence as a criterion. Athletic ability is heritable. How can sports teams not discriminate on the basis of athletic ability?

<comment><USER33> > How can sports teams not discriminate on the basis of athletic ability?

What applies to a few thousand people playing games does not necessarily apply to society at large.

> It is essential to discriminate in hiring on the basis of qualities that are > relevant for the job

Yes, the job has to fit the person, but what about the salary? If a person can only qualify for a certain position based on their innate intelligence then they should not earn less because of that.

Jobs might be distributed based on innate intelligence but distributing income based on innate intelligence would seem to be counter to the anti- discrimination norms that our society lives by.

</comment></comment></comment></comment></comment><comment><USER34> Sweet! Time to print new editions of all those textbooks! Time to buy a boat.

</comment><comment><USER35> On one hand is, "A researcher in human intelligence at Utah Valley University". On the other are, "the 29 best-selling introductory psychology textbooks in the US - some written by among the most eminent psychologists alive".

Why would I believe the former? I'm not saying that Utah Valley U. researchers shouldn't raise questions, but that the standard for what I'm going to take at face value is much higher than one study that contradicts everyone else. One study is just the start of the start of a discussion. I think I'll let the body of evidence accumulate and let experts weigh in.

One psychological concept I'm familiar with is confirmation bias. This article confirms beliefs held by some at HN about political bias, authority, and accuracy in science, (and unfortunately, an effort to rationalize and justify racism by finding some basis for it). That doesn't make this report any more likely to be accurate. Apparently, 29 leading textbooks contradict it; if not for the bias, why not take 29 publications at face value instead of 1?

EDIT: a bunch of edits; sorry for the mess.

EDIT: I'll add that this one study was published "in an open-access article in Archives of Scientific Psychology". It's not even peer-reviewed [EDIT2: It is, in fact, peer reviewed; sorry. I still expect the textbooks to be reviewed much more carefully than one paper.]

<comment><USER36> Well, there is nothing to believe or not believe in this study; it simply points out that most of the books disagree with what's widely accepted in the field[0]. Which do you believe, the widely accepted theory or 29 books?

[0]: [https://en.wikipedia.org/wiki/Cattell%E2%80%93Horn%E2%80%93C...](https://en.wikipedia.org/wiki/Cattell%E2%80%93Horn%E2%80%93Carroll_theory)

<comment><USER35> > it simply points out that 29 books disagree with what's widely accepted in > the field

The study claims that. The textbooks claim otherwise. Why do you believe the study's claims? (I'd believe Wikipedia less than either.)

<comment><USER37> > I'd believe Wikipedia less than either.

That's where you make a mistake. Whatever the reason, the Wikipedia articles on the _g_ factor, the intelligence quotient and Cattell–Horn–Carroll theory represent better what is widely accepted in the field than any other source. This somehow also turns out to be true for other academic fields.

Anyway, when source A says explicitly that source B is wrong but source B makes no such claim about source A, you should usually believe source A. But in this case you can know that the study is right by just learning for yourself what does happen to be widely accepted in the field and why it is widely accepted.

<comment><USER35> > represent better what is widely accepted in the field

How do you know this? Are you a practitioner in the field?

> when source A says explicitly that source B is wrong but source B makes no > such claim about source A, you should usually believe source A

Yikes. That is scary: "You are wrong." You'd better respond or else that proves it.

"Newton is wrong." "Darwin is wrong about evolution." "Climate change scientists are all wrong." "Popper is wrong about postpositivism." "The legal system is wrong about the right to cross-examination."</comment></comment></comment></comment></comment>


After 104 rejections in tech, I figured out what I was really good at - <AUTHOR> http://qz.com/680209/it-took-me-104-rejections-in-tech-to-figure-out-what-i-was-really-good-at/ <comment><USER2> I think the biggest skill for "engineers" is the patience to sit and work on a problem until you figure it out. or look it up on stack overflow.</comment>


Highly advanced shader that simulates almost every aspect of the CRT screen - <AUTHOR> http://emulation.gametechwiki.com/index.php/CRT-Royale <comment><USER2> How comes I never even noticed this 30 years ago?</comment>


Aussie censors implement six degrees of separation policy - Links to links now banned - <AUTHOR> http://www.theregister.co.uk/2009/05/07/oz_link_ban/ <comment><USER2> This looks like an easy one to protest. Just get a huge bunch of people to post the link: on personal websites, message boards, whatever.

<comment><USER3> A series of redirecting pages would be fun. Get a few hundred people to form a link chain..

</comment></comment><comment><USER4> Sadly, the Aussie govt even issued a "link removal" notice to EFA, who complied by removing their link to a link to an offending site. They may not have deep enough pockets to battle the govt in court.

<comment><USER5> If a lion is to beat a crocodile, it must stay out of the water.

In this case they did the right thing. You comply with the governments legal request, then dig through their trash and find ugly stuff you can throw at them.

Then with that info, you go to a friendly journalist and cry censorship and talk about how evil the politicians are. Then you put the content back up again and wait for the next sucker - they will stop trying pretty quickly.

It continues to annoy me how many people who try to play it clean with the politicians. The reason you have to shower is that the real world is dirty; pretending otherwise will mean you end up being the one who stinks.

</comment></comment><comment><USER6> what kinda koolaid is the aussie government drinking? you would think that in 2009 people would know how the interweb works. tubes, ya know?

how can anyone exert control over what someone else links to? government forcing you to accept responsibility for what others link to basically chills linking at all. what if you link to a blog which has public commenting and someone then comments with a link to 'censored' materials. where is the list of censored materials? who knows what is censored? how do i know that something is censored and that i should not link to it?

this quote was taken from an article linked off the op, ( [http://www.itnews.com.au/News/98679,conroy-requests-faith- in...](http://www.itnews.com.au/News/98679,conroy-requests-faith-in-net- filter-scheme.aspx) ):

"Communications Minister Stephen Conroy has urged detractors of the Government's proposed Internet Filtering scheme to have faith in their elected parliamentarians to pass the right legislation."

here is a news flash: stop censoring! then i wont have to worry about elected parliamentarians passing the right legislation. this is like right out of a tolstoy novel or something.

<comment><USER7> _what kinda koolaid is the aussie government drinking?_

Let me try and explain what koolaid they're drinking: I believe the real reason the Australian Government is pushing this plan so hard in spite of massive opposition from industry and the general public is they need support from a couple of minor party/independent senators in order to pass basically any controversial legislation. These senators represent social conservative special interest groups who have strong objections to things like gambling and adult (legal) pornography. In this case, we're talking about the online versions of these things.

It has little do to with protecting the children (that's just the cover) or anything else other than good old political manoeuvring. The whole situation is probably not helped by the fact that the current left-wing Australian Government is by and large, about as conservative as you can get and still be called left-wing. But I think it's mainly that they need support from outside their own party if they really want to get anything done.

<comment><USER8> _they need support from outside their own party if they really want to get anything done._

I thought Rudd had already broken the record for most time spent overseas for any PM in Australia's history.... Seems like he's getting plenty done[1].

Oh wait, you mean getting stuff done in the country? Nevermind.

[1] - for those who don't know, our current PM was elected after it was brought to light he used a tax payer funded overseas trip to visit a strip club in New York. Since becoming PM, he has spent more time overseas than any other Australian PM in history...

</comment></comment></comment><comment><USER9> By this definition don't they have to block Google?

<comment><USER10> Yes they would if the title of the article was accurate. But it isn't. The title is a poor piece of sensationalistic journalism. They are only issuing 'take down' notices for direct links. This is stupid enough in itself and should be ridiculed. But adding false information to the mix is counter productive.

</comment></comment><comment><USER8> The ironic thing in this whole internet censorship debacle here in Australia is that this bullshit is being led by the left wing side* of our political system which one would typically associate with progressive ideals and liberal thinking.

The same party also whole heartedly opposed any plans for Internet censorship in 2003 at the ISP level, which they are now trying to implement themselves.

[http://www.smh.com.au/news/technology/biztech/web- censorship...](http://www.smh.com.au/news/technology/biztech/web-censorship- plan-heads-towards-a-dead-end/2009/02/26/1235237810486.html?page=fullpage) (last 2 paragraphs)

Go figure.

*(I can't say the liberal side, because the Liberal Party here in Australia are the conservatives/right wingers, whereas the Labor Party are the liberal/left wingers and I wouldn't want to confuse the two)

<comment><USER11> I think authoritarian vs. non authoritarian is more useful than left-right. For instance in the US both of the big parties are authoritarian and are pushing big government. To me they are very similar.

That it is leftist doing censor ship is not surprising at all, just look at the history.

In the US both the term "liberal" has been hijacked by socialists. In the US the term libertarian is used for what in the rest of the world is still called "liberal".

<comment><USER12> To be fair, the original usage of the term "libertarian" in a political context was to refer to socialist-anarchists.

</comment></comment></comment><comment><USER13> As far as I know, this isn't the only instance of a law banning certain hyperlinks. If I remember correctly, in Germany it is currently illegal to link to software which can be used to remove copy protection. (for example, DVD-ripping software that breaks the CSS encryption)

Copyright holders can bill you (together with a cease and desist) or sue you if you don't comply.

</comment><comment><USER10> I believe this is inaccurate. After reading through the entire article and the page on EFF there is no indication of a take down notice for anything but a direct link to the 'offending material'. If I've missed something please let me know.

</comment><comment><USER14> Wow this is embarrassing. I thought Australians were progressive.

<comment><USER15> We are... as an Aussie living overseas I am still amazed by these laws, and the fact that there doesn't seem to be a lot of protests/action against them. Maybe protests have been banned as well?

<comment><USER7> I think most people don't realise what's at stake. Many just don't care, many others hear the talk about protecting the children, and swallow it, hook line and sinker.

I think in places like this forum, where most people are pretty web/tech savvy, it's easy to forget that we are still the minority.

When this latest round of censorship stuff reared its ugly head, there was some talk from parts of the pro-censorship side that the country should instate a policy of relegating certain web content to different ports. So rather than the standard port for HTTP being port 80, instead certain web content could only be served on a particular port and of course this would make it easy to block said content by blocking its associated port.

Yes, you read that right. So like you'd have a porn port (lets say port 69) and a gambling port and eventually I guess you'd have a political decent port before too long.

This is the level of understanding we're dealing with. Born in an earlier time, I'm sure these people would be spending their days burning witches after making sure they were witches by determining that they weighed the same as a duck.</comment></comment></comment>


Google Earth didn't update its image of a military installation for eight years - <AUTHOR> https://motherboard.vice.com/en_us/article/gye79x/google-earth-stopped-mapping-this-military-installation <comment><USER2> > [...]We immediately assumed censorship at a federal level[...]

This is just sensationalism. How about:

a) These areas are restricted airspace, and most of the aerial imagery Google et al buy is taken from airplanes.

b) Even if it wasn't restricted airspace, what's the commercial interest for a company like Google? To sell mapping services to the armed forces within their own bases? It's not like they're all buying Android phones and getting turn- by-turn directions in their Humvees.

c) There's also satellite imagery, but no imaging company is imaging the entire globe at a resolution Google and friends are interested in. You need to pay for specific areas, which brings us back to b), what's the commercial interest?

d) They point out that other restricted areas had updated imagery, but none are the size of the Nevada site, so they might have been imaged as a side- effect of imaging other nearby areas via satellites.

Maybe that's not what's going on, but these sound like reasonable explanations that don't require us to assume a bunch of men in black and a shadowy cabal of CIA agents coercing SV companies.

<comment><USER3> >> a) These areas are restricted airspace, and most of the aerial imagery Google et al buy is taken from airplanes.

Just replying to point out that this is true. I worked 3 summers for "Aerometric", a company that does exactly this. We flew flight lines from 10-20 thousand feet with a huge camera rig. Those photos were transmitted back to corporate, where they are stitched back together to form large high- resolution aerial maps. Our customers were usually timber companies, gas/oil companies, and occasionally FEMA.

Until that job, I naively assumed everything was a satellite image with some Bond-like zoom capabilities.

<comment><USER4> What's your career path?

<comment><USER5> Coming from a background in remote sensing, and having been to several conferences where companies desperately try to sell destitute students instrumentation worth hundreds of thousands of dollars...

A lot of remote sensing folks come out of geography degrees into specialist Masters' programs. The use cases are all the various things you'd use high resolution aerial maps for. To be clear most of the scientists are _users_ of this data, rather than the creators. Historically there were more computer vision scientists involved, but we've mostly solved the big problems like image stitching and camera calibration. Then GPS came along and solved the geo-referencing problem as well.

Most academics just buy their data or get it from a space agency, do some analysis in GIS software and get on with answering their research questions. There is still a healthy research field in remote sensing, but fewer people worry about the collection side. There are also plenty of people involved from the space side of things, but again, unless you're on an instrument team you're a consumer not a creator.

So the career path depends. Typically degree in geography with a specialism in remote sensing, then into a career with an appropriate company. Most of the airborne cameras are produced by large orgs like Leica Geosystems. Also look at Hexagon AB who have bought up just about everyone else.

The people that generate this data tend to be small companies who own a plane (or three) kitted out with expensive surveying cameras. They are then contracted to fly over places and they provide calibrated (ortho-rectified, geo-referenced, etc) data to the client. Depending on the budget and requirements, this can include (multispectral) imagery and/or LIDAR mapping.

If you're interested in camera development that spans the engineering gamut from electrical (sensors) to optical. There are also aeronautical engineers and of course pilots. Nowadays if you have some machine learning knowledge, you're probably useful.

</comment></comment></comment></comment><comment><USER6> Without getting into the conspiracy-ish side of this, I'd like to point out that this is changing. Until recently, I worked at Planet Labs where our goal was to image the entire landmass of the Earth, at 3m resolution (not quite as good as Google's most zoomed-in imagery), every day and make it available to everyone. The monitoring system design means, amongst other wins, we don't need to know "something is about to change" to get the before/after pictures, and it also prevents the "checkbook shutter control" the article talks about.

You can sign up for an account[0] with just an email address and start digging through imagery right now (unfortunately, purchasing it at full quality is a different story). Make sure to select an Area Of Interest to get down to daily imagery, I'll start you off with one for the Tonopah Test Range[1] which has coverage on 21 different days so far in October.

[0] [https://www.planet.com/explorer/](https://www.planet.com/explorer/) [1] [https://www.planet.com/explorer/#/area- coverage/0.32,1/mosai...](https://www.planet.com/explorer/#/area- coverage/0.32,1/mosaic/global_monthly_2018_09_mosaic/center/-116.851,37.723/zoom/11/geometry/POLYGON\(\(-116.8971+37.6131,-116.5308+37.6131,-116.5308+37.8955,-116.8971+37.8955,-116.8971+37.6131\)\)/items/PSScene4Band%3A20181023_180533_100c,PSScene4Band%3A20181023_180532_100c,PSScene4Band%3A20181023_180531_100c,PSScene4Band%3A20181023_180530_100c,PSScene4Band%3A20181023_180529_100c,PSScene4Band%3A20181023_180528_100c,PSScene4Band%3A20181023_180425_1009,PSScene4Band%3A20181023_180424_1009,PSScene4Band%3A20181023_180423_1009,PSScene4Band%3A20181023_180422_1009,PSScene4Band%3A20181023_180421_1009,PSScene4Band%3A20181023_180420_1009,PSScene4Band%3A20181023_174521_0f49,PSScene4Band%3A20181023_174520_0f49/interval/1%20day)

</comment><comment><USER7> > _The contract we signed with Apollo states that we may show "image > derivatives" of image #103001000EBC3C00 to "authorized users" within our > company. Thus all attendees will have to sign a document temporarily making > them independent contractors of Eyebeam._

That's the best darn r/MaliciousCompliance I've ever seen.

</comment><comment><USER8> Vice is being very sensationalist. There are very real abuses of government power related to 9/11 and the PATRIOT Act, but this is a distraction.

<comment><USER9> > Vice is being very sensationalist.

FWIW, they've always been investigative editorialists, so I'm not surprised. Although, I do often enjoy their style, even if I don't always agree with their opinions.

</comment></comment><comment><USER10> I thought that many, maybe even most US military facilities (at least the more secretive ones) were blanked out altogether on Google Earth. I thought it had always been that way. Maybe I'm wrong.

I'm more surprised that it was there in the first place, if there were people who considered it sensitive. Not updating the imagery of a particular military site for 8 years seems more like expected behavior than some monocle-popping revelation of malfeasance.

<comment><USER11> You might be surprised. You can't find it by searching, for instance, but if you put in the coordinates you can see the notorious Groom Lake airfield (aka "Area 51") isn't blanked out: [https://goo.gl/maps/UHEneL7nJbN2](https://goo.gl/maps/UHEneL7nJbN2)

<comment><USER12> There's even an easter egg around there when you try to use the Street View.

<comment><USER13> Who needs streetview when the nearest highway is 'Extraterrestrial Highway'?

Also, possibly spoiling the easter egg - apparently military bases are a) boring b) not as strict as expected (see [1], which specifically says not to photograph and yet... though there are also no connected street views.)

[1] [https://goo.gl/maps/eRP4dhtVK3R2](https://goo.gl/maps/eRP4dhtVK3R2)

ETA: Apparently google has added photo spheres to its map, so that's not strictly street view. However, it's still interesting.

</comment></comment></comment></comment><comment><USER14> I'm probably overly cynical but I think it highly unlikely any of the publicly viewable 'military installation' mapping would be accurate or up to date. Basic military strategy requires maps for planning - why share location details with your potential enemies?

<comment><USER15> The vast majority of our enemies have satellites of their own for high resolution images that we can't stop them from taking, right?

<comment><USER16> The concern was mainly over terrorists, who don't have their own satellites and have to rely on commercial imagery.

</comment></comment></comment><comment><USER17> Seems like if there were really some mustache-twirling villainy going on here, there would be hi-res imagery that was completely inaccurate/faked.

<comment><USER18> This is the case in South Korea where imagery of sensitive locations is illegal. If you compare Naver maps to Google maps you'll see military installations are missing and are replaced with home and trees. Interestingly this also applies to the presidential palace, which is a huge building in the center of the capital which has stood for decades. I'm still not sure why they did that.

<comment><USER19> Simply due to North Korea. Despite all talks of normalization between South and North Korea, it's still not safe to let your guard down. Such talks of normalization took place few times in the past, going back as far as 1960s I believe.

<comment><USER2> There's been good discussion on some episodes of the Arms Control Wonk podcast[1] about how bad the South Koreans are at this, although I unfortunately can't remember on which episodes.

Things like military installations being obvious from aerial imagery because it's the only areas where someone incompetent at using the Photoshop clone tool has copied a bunch of trees around.

Other examples are "secret" missile silos being discovered because a bunch of buildings are seen in the exact size / configuration as other known military sites. The South Koreans having re-used the plans and architectural drawings.

They said that in some cases the most plausible explanation for some sites still being secret is to save face. Nobody wants to own up to how much of an open secret they are due to opsec failures over the years, even though everyone, and certainly the North Koreans, know exactly where they are and what they're for.

1\. [https://www.armscontrolwonk.com](https://www.armscontrolwonk.com)

</comment></comment></comment></comment><comment><USER20> Except that it was refreshed in 2017 (per the article itself).

What is even the point?

<comment><USER21> Slightly misleading headline and first paragraphs yes, but then they focus on the idea that the location was not refreshed for 8 years and are curious about this gap.

<comment><USER22> I think it is much more likely that the mapreduce shard responsible for updating the tile was stuck for 8 years, than there is a massive government conspiracy to prevent the update.

</comment></comment></comment><comment><USER23> Here is the Google Maps link to the satellite imagery of the Tonopah Test Range site [1].

[1] [https://www.google.com/maps/place/Tonopah+Test+Range+Airport...](https://firstname.lastname@example.com,-116.7295302,49155m/data=!3m1!1e3)

</comment><comment><USER24> Isn't it possible for this to be happening on a smaller scale and in plain sight? That is, for example, if Google is not always the source of the images, can't the supplier / vendor doctor the images it provides to Google? Who would know?

</comment><comment><USER25> Google used to blur many sensitive commercial sites. Fuel depots, airports, and critical manufacturing plants would just be indistinct smudges. They seem to have abandoned the practice after switching to primarily aerial imagery for urban areas.

<comment><USER26> A more accurate description is that they used to buy satellite imagery from third parties such as DG, which were responsible for the blurring. Nowadays, a lot of images come from planes, many of which are operated by Google, which does not do the blurring.

</comment></comment><comment><USER27> It's a military installation unless it's an illegal millatary installation I don't see what's wrong with any government from any country asking any public satellite company not to show details publicly.

</comment><comment><USER28> Commenters annoyed by sensationalism, use of the term "censorship", or the lack of a substantive argument, should bear one thing in mind: It's an art installation.

A successful show needs a message, and needs press. It's very accepted _within the art community_ for the message to be a bit blurred around the edges. Take these "cyberlibertarian progenitors", for instance. Who are they and how are they making people believe the internet is a "flat, endless digital plain"? Any sources? No? Well, it doesn't really matter, because this isn't really journalism. It could be very important to the artists – of course it would be very important to them! – but it should not be judged by journalistic standards.

Sadly, this does mean that Vice is something less than a home for journalism proper – but why would anyone ever have started to think that in the first place?

</comment><comment><USER29> They have not updated many places in NC for over two years now. I think they are going to phase it out like all their other pet projects.

</comment><comment><USER30> > join us at Eyebeam, at 199 Cook Street in Brooklyn, at 6pm on Thursday > October 25th to view image #103001000EBC3C00 yourself.

Cool - I'll be there!

</comment><comment><USER31> > _That this gap occurred for eight years without any acknowledgement from > Google, Alphabet, or the federal government suggests that it can happen > again with no warning or oversight. This is Google's Earth, not ours._

Reading this sentences, I gained a new understanding of what my dog feels when he tilts his head and looks at me with his big, questioning eyes.</comment>


"Greedy" developers: the end of indie app developers? - <AUTHOR> https://medium.com/@rcorin/greedy-developers-a4c84fd5ee14#.ijpgznt7z <comment><USER2> I'd like to be able to buy software conventionally. That if I buy your app, 1.0, I can assume I'll get a year or two maybe of updates as necessary, but when you have a big release, like a 2.0, I am free to use the 1.0 version, but can upgrade to 2.0 at a new cost.

Then, my app doesn't go away because I stopped paying you. And I don't get free features forever. If I use your app and I want the new things, then I can pay you again for them.</comment>


Facebook's political ad tool let us buy ads "paid for" by Mike Pence and ISIS - <AUTHOR> https://news.vice.com/en_us/article/wj9mny/facebooks-political-ad-tool-let-us-buy-ads-paid-for-by-mike-pence-and-isis <comment><AUTHOR> Original title: "Facebook's political ad tool let us buy ads "paid for" by Mike Pence and ISIS"</comment>


Ask HN: Quora Down? - <AUTHOR>

<comment><USER2> not for me in SF</comment>


Mark Shuttleworth: Merging is the key to software developer collaboration - <AUTHOR> http://www.markshuttleworth.com/archives/126 <comment><USER2> I've asked this over on programming.reddit also: any of you dudes using a distributed version control system like bzr? I'm looking at that and Mercurial.

<comment><USER3> Mercurial

Read the docs and hang out on the lists and you'll see a world of difference. Of course, trying it out for yourself makes it pretty clear, too. :-)</comment></comment>


The Louder the Monkey, the Smaller Its Balls, Study Finds - <AUTHOR> http://motherboard.vice.com/read/the-louder-the-monkey-the-smaller-its-balls-study-finds-42361364663309 <comment><USER2> > Naturally, Knapp warns against anthropomorphizing these findings, and she is > completely right. There is a world of difference between howler monkeys and > humans, especially with regard to sexual selection.

Then of course the author has to go there.

</comment><comment><USER3> I think we have an early leader for next year's Ig Nobel prize in biology.

As a follow up project, they might want to examine the correlation between tree climbing altitude and tail visibility of monkeys.

</comment><comment><USER4> Is it surprising? Don't gorillas have small balls too?

The sexier you are, the more assured of reproductive success you are, so you don't need to compete so much on "reapplication".</comment>


10 places to learn geeky stuff online for free - <AUTHOR> http://blog.geekli.st/post/29086373998/10-places-to-learn-geeky-stuff-online-for-free

<comment><USER2> <http://programming-motherfucker.com/become.html></comment>


Improve your servicenow incident management - <AUTHOR> https://store.servicenow.com/sn_appstore_store.do#!/store/application/da6bd9fadb827600360eda11cf961926/1.0.0?referer=sn_appstore_store.do%23!%2Fstore%2Fsearch%3Fq%3Ddesktop%2520central <comment><AUTHOR> ServiceNow administrators may feel a sense of need to troubleshoot remote computers directly from their incident window, deploy business application to remote users, shutdown, restart, lock, and hibernate remote computers all from your incident window. If you are servicenow user please download the app and feel the benefit for yourself.</comment>


Journey from Israeli military intelligence to VP R&D - <AUTHOR> https://linearb.io/blog/7-secrets-learned-on-my-journey-from-israeli-military-intelligence-to-vp-of-rd/ <comment><AUTHOR> Love this part... "Early on, I figured out that while we have awesome technologists, probably among the brightest in the world, no one knew how to manage development, release, and write good code. So I took charge. Part of leading, especially when you don't have the title, is putting in the work. Initially, at Cymmetria, I was doing my job plus creating process documents, writing training guides, building automation, helping everyone… It worked for me in the army so I did it again." Don't wait to be asked to solve the problem, just solve it!</comment>