Ripgrep – A new command line search tool - <AUTHOR> http://blog.burntsushi.net/ripgrep/ <comment><USER3> Meh, yet another grep tool.... wait, by burntsushi! Whenever I hear of someone wanting to improve grep I think of the classic ridiculous fish piece[0]. But when I saw that this one was by the author of rust's regex tools, which I know from a previous post on here, are quite sophisticated, I perked up.

Also, the tool aside, this blog post should be held up as the gold standard of what gets posted to hacker news: detailed, technical, interesting.

Thanks for your hard work! Looking forward to taking this for a spin.

[0] [http://ridiculousfish.com/blog/posts/old-age-and- treachery.h...](http://ridiculousfish.com/blog/posts/old-age-and- treachery.html)

<comment><USER4> Another burntsushi project was recently posted, but didn't get much attention:

[https://news.ycombinator.com/item?id=12559515](https://news.ycombinator.com/item?id=12559515)

<comment><USER5> Looks interesting. But I did not find binaries for it and do not want to setup a rust env to try it out.

<comment><USER6> The releases are right there on GitHub: [https://github.com/BurntSushi/ripgrep/releases](https://github.com/BurntSushi/ripgrep/releases)

<comment><USER7> I think the GP was asking about xsv, not ripgrep. There are binary releases for xsv though: [https://github.com/BurntSushi/xsv/releases](https://github.com/BurntSushi/xsv/releases)

<comment><USER5> Awesome. Thanks

<comment><USER5> PS: You might want to revise this verbiage in the README markdown file:

        Installing xsv is a bit hokey right now. Ideally, I could release binaries for Linux, Mac and Windows. Currently, I'm only able to release binaries for Linux because I don't know how to cross compile Rust programs.

<comment><USER7> Ah how embarrassing! I will fix that soon. Thanks :-)

<comment><USER5> No need for any embarrassment. I should have looked for releases instead of only looking at the doc.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> I'm the author of ag. That was a really good comparison of the different code searching tools. The author did a great job of showing how each tool misbehaved or performed poorly in certain circumstances. He's also totally right about defaults mattering.

It looks like ripgrep gets most of its speedup on ag by:

1\. Only supporting DFA-able Rust regexes. I'd love to use a lighter-weight regex library in ag, but users are accustomed to full PCRE support. Switching would cause me to receive a lot of angry emails. Maybe I'll do it anyway. PCRE has some annoying limitations. (For example, it can only search up to 2GB at a time.)

2\. Not counting line numbers by default. The blog post addresses this, but I think results without line numbers are far less useful; so much so that I've traded away performance in ag. (Note that even if you tell ag not to print line numbers, it still wastes time counting them. The printing code is the result of me merging a lot of PRs that I really shouldn't have.)

3\. Not using mmap(). This is a big one, and I'm not sure what the deal is here. I just added a --nommap option to ag in master.[1] It's a naive implementation, but it benchmarks comparably to the default mmap() behavior. I'm really hoping there's a flag I can pass to mmap() or madvise() that says, "Don't worry about all that synchronization stuff. I just want to read these bytes sequentially. I'm OK with undefined behavior if something else changes the file while I'm reading it."

The author also points out correctness issues with ag. Ag doesn't fully support .gitiginore. It doesn't support unicode. Inverse matching (-v) can be crazy slow. These shortcomings are mostly because I originally wrote ag for myself. If I didn't use certain gitignore rules or non-ASCII encodings, I didn't write the code to support them.

Some expectation management: If you try out ripgrep, don't get your hopes up. Unless you're searching some really big codebases, you won't notice the speed difference. What you will notice, however, are the feature differences. Take a look at [https://github.com/BurntSushi/ripgrep/issues](https://github.com/BurntSushi/ripgrep/issues) to get a taste of what's missing or broken. It will be some time before all those little details are ironed-out.

That said, may the best code searching tool win. :)

1\. [https://github.com/ggreer/the_silver_searcher/commit/bd65e26...](https://github.com/ggreer/the_silver_searcher/commit/bd65e2691d6ce673e026842d7e7d02bc25adfd26)

<comment><USER7> Thanks for the response! Some notes:

1\. In my benchmarks, I do control for line numbers by either explicitly making it a variable (i.e., when you see `(lines)`) or by making all tools count lines to make the comparison fair. For the most part, this only tends to matter in the single-file benchmarks.

2\. For memory maps, you might get very different results depending on your environment. For example, I enabled memory maps on Windows where they seem to do a bit better. (I think my blog post gives enough details that you could reproduce the benchmark environment _precisely_ if you were so inclined. This was important to me, so I spent a lot of time documenting it.)

3\. The set of features supported by rg should be very very close to what is supported by ag. Reviewing `ag`'s man page again, probably the only things missing from rg are --ackmate, --depth, some of the color configurability flags (but rg does do coloring), --passthrough, --smart-case and --stats maybe? I might be missing some others. And Mercurial support (but ag's is incomplete). In exchange, rg gives you much better single file performance, better large-repo performance and real Unicode support that doesn't slow way down. I'd say those are pretty decent expectations. :-)

Thanks for ag by the way. It and ack have definitely triggered a new kind of searching. I have some further information retrievalish ideas on evolving the concept, but those will have to wait!

<comment><USER8> In terms of core features, ripgrep is totally there. It searches _fast_. It ignores files pretty accurately. It outputs results in a pleasant and useful format. If a new user tries rg, they'll be very happy.

My warning about the feature differences was meant to temper ag users' expectations. There are lots of little things that ag users are accustomed to that are either different or missing in ripgrep. Off the top of my head: Ag reads the user's global gitignore. (This is harder than most people think.) It detects stdout redirects such as "ag blah > output.txt" and ignores output.txt. It can search gz and xz files. It defaults to smart-case searching. It can limit a search to one hardware device (--one-device), avoiding slow reads on network mounts. And as a commenter already pointed out, it supports the --pager option. Taken together, all those small differences are likely to cause an average ag user some grief. I wanted to manage expectations so that users wouldn't create annoying "issues" (really, feature requests) on your GitHub repo. Sorry if that came off the wrong way.

On a completely unrelated note: I see ripgrep supports .rgignore files, similar to how ag supports .agignore. It'd be nice if we could combine forces and choose a single filename for this purpose. That way when the next search tool comes along, it can use the same thing instead of .zzignore or whatever. It would also make it easier for users to switch between our tools. I'd suggest a generic name like ".ignore" or ".ignores", but I'm sure some tool creates such files or directories already.

Edit: Actually, it looks like .ignore can work. The only examples I've found of .ignore files are actual files containing ignore patterns.

<comment><USER7> You raise good points, thank you. I hope to support some of those features, since they seem like nice conveniences.

In principle I'd be fine standardizing on a common ignore file. We'd need to come up with a format (I think I'd vote for "do what gitignore does", since I think that's what we're both doing now anyway).

Adding files to this list is kind of a bummer though. I could probably get away with replacing `.rgignore` proper, but I suspect you'd need to add it without replacing `.agignore`, or else those angry users you were talking about might show themselves. :-)

I do kind of like `.grepignore` since `grep` has kind of elevated itself to "search tool" as a term, but I can see how that would be confusing. `.searchignore` feels too long. `.ignore` and `.ignorerc` feel a bit too generic, but either seems like the frontrunner at the moment.

<comment><USER8> I also vote for "do what gitignore does". My plan is to add support for the new file name, deprecate .agignore, and update docs everywhere. But it'd be a while before I removed .agignore completely.

I really like .ignore, and I like it _because_ it's generic. The information I want it to convey is:

 _> Dear programs,

> If you are traversing this directory, please ignore these things._

Of course, some programs could still benefit from having application-specific ignore files, but it'd cut down on a lot of cruft and repetition.

<comment><USER7> Let's do it. [https://github.com/BurntSushi/ripgrep/issues/40](https://github.com/BurntSushi/ripgrep/issues/40)

<comment><USER8> …and merged: [https://github.com/ggreer/the_silver_searcher/pull/974](https://github.com/ggreer/the_silver_searcher/pull/974)

I'll tag a new release in a day or two. Also, it looks like the sift author is getting on the .ignore train: [https://github.com/svent/sift/issues/78#issuecomment-2493352...](https://github.com/svent/sift/issues/78#issuecomment-249335277)

This worked out pretty well. :)

<comment><USER7> Same! [https://github.com/BurntSushi/ripgrep/pull/41](https://github.com/BurntSushi/ripgrep/pull/41)

Agreed :-)

<comment><USER9> This is probably the best case of out-in-the-open open source developers of similar-but-different tools collaborating on a new standard and implementing them in record time that I have ever seen.

Keep it up all (rg/ag/sift)!

<comment><USER10> I completely agree. That was one of the most reasonable and level-headed discussions between strangers I have _ever_ seen on the Internet!

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> _In contrast, GNU grep uses libc's memchr, which is standard C code with no explicit use of SIMD instructions. However, that C code will be autovectorized to use xmm registers and SIMD instructions, which are half the size of ymm registers._

I don't think this is correct. glibc has architecture specific hand rolled (or unrolled if you will lol) assembly for x64 memchr. See here: [https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86...](https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_64/memrchr.S;h=840de30cd71ba96b3ae43540e6ac255c28906cc5;hb=HEAD)

<comment><USER7> Drats, you're totally right. It's easy to mess up that kind of thing.

Thankfully, it looks like my analysis remains mostly unchanged. I don't see any AVX2 in there (and indeed, I didn't when I looked at the profile either, in contrast to Go's implementation).

I updated the blog, thanks again for the clarification.

</comment></comment><comment><USER12> Nice! Lightgrep[1] uses libicu et al to look up code points for a user- specified encoding and encode them as bytes, then just searches for the bytes. Since ripgrep is presumably looking just for bytes, too, and compiling UTF-8 multibyte code points to a sequence of bytes, perhaps you can do likewise with ICU and support other encodings. ICU is a bear to build against when cross- compiling, but it knows hundreds of encodings, all of the proper code point names, character classes, named properties, etc., and the surface area of its API that's required for such usage is still pretty small.

[1]: [http://strozfriedberg.github.io/liblightgrep](http://strozfriedberg.github.io/liblightgrep)

<comment><USER7> I hadn't heard of liblightgrep, nice. It's on my short list for looking more closely.

I doubt I'd ever be comfortable with Rust's regex engine growing a dependency on libicu, but it's still worth understanding your implementation. Some questions, if you don't mind. The big one is: does your regex engine use finite automata, and does it put the text decoding into the automaton itself? For example, when you compile the `.` regex, you end up with an automaton that inlines UTF-8 decoding itself. It looks like this: [https://gist.github.com/anonymous/8fbe170bfcca5d7475b59299fa...](https://gist.github.com/anonymous/8fbe170bfcca5d7475b59299fabbd0a3)

Does you regex library do that for each type of encoding? Or is there a transcoding step?

<comment><USER12> Reply fail, see above.

</comment></comment></comment><comment><USER13> I wish more people actually took steps to optimize disk io though; my current source tree may be in cache, but my logs certainly aren't. Nor are my /usr/share/docs/, /usr/includes/, or my old projects.

Chris Mason of btrfs fame did some proof of concept work for walking and reading trees in on-disk order, showing some pretty spectacular potential gains: [https://oss.oracle.com/~mason/acp/](https://oss.oracle.com/~mason/acp/)

Tooling to do your own testing: [https://oss.oracle.com/~mason/seekwatcher/](https://oss.oracle.com/~mason/seekwatcher/)

</comment><comment><USER14> "Anti-pitch

I'd like to try to convince you why you shouldn't use ripgrep. Often, this is far more revealing than reasons why I think you should use ripgrep."

Love that he added this

</comment><comment><USER15> It would be interesting to benchmark how much mmap hurts when operating in a non-parallel mode.

I think a lot of the residual love for mmap is because it actually did give decent results back when single core machines were the norm. However, once your program becomes multithreaded it imposes a lot of hidden synchronization costs, especially on munmap().

The fastest option might well be to use mmap sometimes but have a collection of single-thread processes instead of a single multi-threaded one so that their VM maps aren't shared. However, this significantly complicates the work- sharing and output-merging stages. If you want to keep all the benefits you'd need a shared-memory area and do manual allocation inside it for all common data which would be a lot of work.

It might also be that mmap is a loss these days even for single-threaded... I don't know.

Side note: when I last looked at this problem (on Solaris, 20ish years ago) one trick I used when mmap'ing was to skip the "madvise(MADV_SEQUENTIAL)" if the file size was below some threshold. If the file was small enough to be completely be prefetched from disk it had no effect and was just a wasted syscall. On larger files it seemed to help, though.

<comment><USER7> One thing I did benchmark was the use of memory maps for single file search (cf. `subtitles_literal`). In that case, it saved time (small, but _measurable_ ) to memory map the file than to incrementally read it. Memory maps were only slower in parallel search on large directories.

Thankfully, ripgrep makes it easy to switch between memory maps and incremental reading. So I can just do this for you right now on the spot:

        $ time rg -j1 PM_SUSPEND | wc -l
        335

        real    0m0.406s
        user    0m0.350s
        sys     0m0.293s

        $ time rg -j1 PM_SUSPEND --mmap | wc -l
        335

        real    0m0.482s
        user    0m0.380s
        sys     0m0.317s

Note that this is on a Linux x64 box. I bet you'd get completely different results on a different OS.

<comment><USER15> Interesting that user time went up as well.. not sure if that's significant.

I guess it's not too surprising that mmap isn't much of a win these days for anything... SIMD can copy a memory page pretty fast these days.

I just installed rg from homebrew and it's quite impressive... about 2.5x faster than ag on my macbook pro. Interestingly I get another 25% improvement by falling back to -j3 even though I'm on a quad-core machine. Not sure what is bottlenecking since it's all in cache.

<comment><USER7> Yeah, figuring out the optimal thread count has always seemed like a bit of a black art to me. I can pretty reliably figure it out for my system (which has 8 physical cores, 16 logical), but it's hard to generalize that to others.

-j3 will spawn 3 workers for searching while the main thread does directory traversal. It sounds like I should do `num_cpus - 1` for the default `-j` instead of `num_cpus`.

</comment></comment></comment></comment><comment><USER16> To build a static Linux binary with SIMD support, run this:

        RUSTFLAGS="-C target-cpu=native" rustup run nightly cargo build --target x86_64-unknown-linux-musl --release --features simd-accel

<comment><USER7> That's an awesome demonstration of how easy it is to swap out the libc in Rust. :-)

Note that I also distribute statically compiled executables with musl and SIMD enabled (using target-feature=+ssse3 instead of target-cpu=native): [https://github.com/BurntSushi/ripgrep/releases](https://github.com/BurntSushi/ripgrep/releases)

<comment><USER16> > That's an awesome demonstration of how easy it is to swap out the libc in > Rust. :-)

Now I have to look up how I use cargo to build a static binary on FreeBSD, where I don't have to swap out libc.

> Note that I also distribute statically compiled executables with musl and > SIMD enabled (using target-feature=+ssse3 instead of target-cpu=native): > [https://github.com/BurntSushi/ripgrep/releases](https://github.com/BurntSushi/ripgrep/releases)

I took the flag from your blog post, thanks for pointing out the explicit feature flag. That will allow the binary to run on more cpus.

</comment></comment></comment><comment><USER17> Very nice. Not only fast, but feels modern.

Tried it out on a 3.5GB JSON file:

      # rg
      rg erzg4 k.json > /dev/null  1.80s user 2.54s system 53% cpu 8.053 total

      # rg with 4 threads
      rg -j4 erzg4 k.json > /dev/null  1.76s user 1.29s system 99% cpu 3.059 total

      # OS X grep
      grep erzg4 k.json > /dev/null  60.62s user 0.96s system 99% cpu 1:01.75 total

      # GNU Grep
      ggrep erzg4 k.json > /dev/null  1.96s user 1.43s system 88% cpu 2.691 total

GNU Grep wins, but it's pretty crusty, especially with regards to its output (even with colourization).

<comment><USER7> My guess is that since you ran `rg` first, the file wasn't in memory, and you ended up benchmarking disk IO. (Notice the sys time decrease from your first run to the second run.) Subsequent commands then run faster with the file already in memory.

This is one of many reasons why assembling the benchmarks in my blog post was so difficult. For example, on every command I benchmarked, I ran them 3 times for "warmup" and didn't record any measurements. I then ran them another 10 times in which I recorded them. You can see the raw output here: [https://github.com/BurntSushi/ripgrep/blob/master/benchsuite...](https://github.com/BurntSushi/ripgrep/blob/master/benchsuite/runs/2016-09-20-ubuntu1604-ec2/raw.csv)

In any case, on my underpowered Mac, here are some results on a 1.2 GB file (notice how much the time fluctuates until its fully in cache):

        mac:~ andrew$ ggrep --version
        ggrep (GNU grep) 2.25
        Packaged by Homebrew
        mac:~ andrew$ time ggrep 'Bruce Springsteen' foo.jsonl > /dev/null

        real    0m5.447s
        user    0m0.600s
        sys     0m0.350s
        mac:~ andrew$ time ggrep 'Bruce Springsteen' foo.jsonl > /dev/null

        real    0m1.247s
        user    0m0.549s
        sys     0m0.264s
        mac:~ andrew$ time ggrep 'Bruce Springsteen' foo.jsonl > /dev/null

        real    0m0.803s
        user    0m0.542s
        sys     0m0.259s
        mac:~ andrew$ time ggrep 'Bruce Springsteen' foo.jsonl > /dev/null

        real    0m0.805s
        user    0m0.544s
        sys     0m0.260s

And now for rg:

        mac:~ andrew$ time rg 'Bruce Springsteen' foo.jsonl > /dev/null

        real    0m1.062s
        user    0m0.339s
        sys     0m0.333s
        mac:~ andrew$ time rg 'Bruce Springsteen' foo.jsonl > /dev/null

        real    0m0.640s
        user    0m0.337s
        sys     0m0.302s
        mac:~ andrew$ time rg 'Bruce Springsteen' foo.jsonl > /dev/null

        real    0m0.637s
        user    0m0.336s
        sys     0m0.300s

Oh! And check this out, on a Mac, not using a memory map for single files is faster. My goodness---memory map performance is all over the place.

        mac:~ andrew$ time rg 'Bruce Springsteen' foo.jsonl --no-mmap > /dev/null

        real    0m0.445s
        user    0m0.170s
        sys     0m0.274s

If I do this on my Linux machine on the same file, I get timings of 0.275s for rg, 0.398s for rg with no memory maps (opposite direction for Mac) and 0.708s for GNU grep (v 2.25).

Benchmarks are fun, eh?

<comment><USER18> I ran each test four times and picked the best result — not my first rodeo — but for the first result I picked the wrong time from my output, which obviously didn't make use of the cache. Here's it again, complete result, added --no-mmap:

        # ggrep
        zerogravitas$ for x in {1..4}; do (time ggrep erzg4 k.json >/dev/null); done
        ggrep erzg4 k.json > /dev/null  1.96s user 0.67s system 99% cpu 2.641 total
        ggrep erzg4 k.json > /dev/null  1.95s user 0.68s system 99% cpu 2.660 total
        ggrep erzg4 k.json > /dev/null  2.00s user 0.66s system 99% cpu 2.672 total
        ggrep erzg4 k.json > /dev/null  1.96s user 0.67s system 98% cpu 2.662 total

        # rg
        zerogravitas$ for x in {1..4}; do (time rg erzg4 k.json >/dev/null); done
        rg erzg4 k.json > /dev/null  1.76s user 1.40s system 99% cpu 3.180 total
        rg erzg4 k.json > /dev/null  1.77s user 1.31s system 99% cpu 3.088 total
        rg erzg4 k.json > /dev/null  1.74s user 1.36s system 99% cpu 3.128 total
        rg erzg4 k.json > /dev/null  1.76s user 1.41s system 97% cpu 3.265 total

        # rg --no-mmap
        zerogravitas$ for x in {1..4}; do (time rg erzg4 k.json --no-mmap >/dev/null); done
        rg erzg4 k.json --no-mmap > /dev/null  0.98s user 0.75s system 99% cpu 1.743 total
        rg erzg4 k.json --no-mmap > /dev/null  0.99s user 0.75s system 99% cpu 1.740 total
        rg erzg4 k.json --no-mmap > /dev/null  1.01s user 0.76s system 99% cpu 1.772 total
        rg erzg4 k.json --no-mmap > /dev/null  0.99s user 0.75s system 99% cpu 1.754 total

        # rg -j4
        zerogravitas$ for x in {1..4}; do (time rg erzg4 k.json -j4 >/dev/null); done
        rg erzg4 k.json -j4 > /dev/null  1.75s user 1.35s system 98% cpu 3.134 total
        rg erzg4 k.json -j4 > /dev/null  1.75s user 1.44s system 98% cpu 3.224 total
        rg erzg4 k.json -j4 > /dev/null  1.80s user 1.38s system 99% cpu 3.204 total
        rg erzg4 k.json -j4 > /dev/null  1.80s user 1.35s system 99% cpu 3.164 total

        # rg -j4 --no-mmap
        zerogravitas$ for x in {1..4}; do (time rg erzg4 k.json -j4 --no-mmap >/dev/null); done
        rg erzg4 k.json -j4 --no-mmap > /dev/null  0.98s user 0.75s system 99% cpu 1.740 total
        rg erzg4 k.json -j4 --no-mmap > /dev/null  0.97s user 0.74s system 99% cpu 1.721 total
        rg erzg4 k.json -j4 --no-mmap > /dev/null  0.99s user 0.75s system 99% cpu 1.752 total
        rg erzg4 k.json -j4 --no-mmap > /dev/null  0.98s user 0.76s system 99% cpu 1.748 total

Sounds like "alias rg=rg --no-mmap" is a good idea on a Mac.

<comment><USER7> Wow. Those are awesome results, thank you.

> Sounds like "alias rg=rg --no-mmap" is a good idea on a Mac.

I will fix that in ripgrep proper by making --no-mmap the default on Mac. :-) It should be an easy one to knock off: [https://github.com/BurntSushi/ripgrep/issues/36](https://github.com/BurntSushi/ripgrep/issues/36)

> not my first rodeo

Right, sorry about that. :-) Just had to cover all my bases!

(Also, `-j` on a single file won't do anything, and ripgrep should try to use multiple threads by default when searching multiple files.)

<comment><USER17> I suspected that -j wouldn't do anything on a single file. For three large (6.5GB in total) files I'm getting good performance, about 1.6x of what GNU Grep does, best case.

</comment></comment></comment></comment></comment><comment><USER19> Compiling it to try right now...

Some discussion over on /r/rust: [https://www.reddit.com/r/rust/comments/544hnk/ripgrep_is_fas...](https://www.reddit.com/r/rust/comments/544hnk/ripgrep_is_faster_than_grep_ag_git_grep_ucg_pt/)

EDIT: The machine I'm on is much less beefy than the benchmark machines, which means that the speed difference is quite noticeable for me.

</comment><comment><USER20> Nice work. But your story is incomplete if you don't include comparison with icgrep (parabix.costar.sfu.ca). Although icgrep is still an active research project, it is faster in many cases and has broader Unicode support (full Unicode level 1 of UTS #18, plus many level 2 features). For example, try the '\N{SMIL(E|ING)}' search that finds lines containing emoji characters with SMILE or SMILING in their Unicode name. icgrep also correctly applies Unicode character class intersection with expressions such as [\p{Greek}&&\p{Lu}], while ripgrep fails to meet UTS 18 level 1 requirements by interpreting '&&' as literal characters.

Nevertheless, we appreciate the challenge that ripgrep presents to our performance story. We definitely see some cases in which rg achieves better performance taking advantage of fixed strings somewhere in the pattern. We'll have to work on that... But for patterns based on Unicode classes (e.g., \p{Greek}, icgrep can be much faster, especially on large files. (We are only focused on big data applications -- icgrep has significant dynamic compilation overhead). It also does very well in cases involving alternations and ambiguity.

The icgrep performance story is based primarily on a new bitwise data parallel regular expression algorithm working off the Parabix transform representation of text. See our ICA3PP 2015 or PACT 2014 papers.

It might be fair to say that icgrep is not yet polished enough for inclusion in your study. I just added our first implementation of -r/-R flags last night and we certainly haven't yet handled .gitignore, etc. But if you want to understand the truth about regular expression performance, I think that the data point represented by icgrep (and its continuing development) needs to be included.

<comment><USER21> Speaking as the leader of the Hyperscan project ([https://github.com/01org/hyperscan](https://github.com/01org/hyperscan)), I'd say you might not be the only project feeling a bit neglected in the performance comparison here.

It's nice to be mentioned - and even called out by name as the inventor of Teddy (!) - but we're always even more pleased when someone else measures Hyperscan, as it minimizes the prospect of us embarrassing ourselves by posting some self-serving and/or outlandish microbenchmark.

Also it saves everyone time reading the obligatory Intel legal disclaimers...

<comment><USER20> We are very interested in tackling the multiple-pattern regular expression problem and will definitely want to use Hyperscan as a comparator.

Any help in setting up a study with both patterns and data sets would be most appreciated!

<comment><USER21> This sounds interesting. I suspect there are many alternate approaches to multiple regex.

Sadly, regex benchmarking is a sewer. There are two classes of multiple regex benchmarks: public ones and good ones, and not much intersection between the two. Synthetic pattern generation can be manipulated to say whatever you want it to say, Snort patterns aren't intended to be run simultaneously (so putting a big pile of them into a sack and running them is of arguable use), and most vendors guard proprietary signature sets closely (we have thousands, but all the good ones are customer confidential).

That being said, there are some paths forward here. Let's talk.

<comment><USER20> Certainly. I dropped an e-mail at the hyperscan account.

By the way, if you try out icgrep and use the -DumpASM option, you may notice a very unusual characteristic: the generated code is almost completely dominated by AVX2 instructions!

</comment></comment></comment></comment></comment><comment><USER22> Rust is really staring to be seen in the wild now.

<comment><USER23> I agree, and I'm already using both ripgrep and rust-parallel ( [https://github.com/mmstick/parallel](https://github.com/mmstick/parallel) , a gnu-parallel replacement which should probably get another name ).

I am really happy to see Rust apps actually been written -- Rust programmers seem to be actually trying to replace the existing code lying around, instead of just insulting it and telling us how much better the world would be if we used their language.

<comment><USER24> This would have been a lot more exciting if it were designed to actually be even slightly compatible with grep (or maybe had some core in there that was, while leaving the other UI parts he wants to change on top) and were then approaching GNU and saying "hey, this is something I've been working on: would you consider making grep the first standalone tool to _move_ to Rust, and what would it take from me to make this happen?" as opposed to writing an article about how "I am smarter than the GNU grep people for these reasons and have built a tool named after how my tool is going to kill their tool" which almost seems to go out of its way to set up an air of competition rather than collaboration. Maybe you value pure technical chops, and the difference between "insulting it and telling us how" is much worse than "insulting it and actually writing code", but to me they both start with "insulting it" and demonstrate an almost tragic inability to work with others. For people like me, people who actually want to see a language like Rust get used en masse and entirely replace languages like C, the attitude in this blog post is extremely depressing. Even if I now we're to myself take the time to go to the GNU grep authors and try to talk to them about this, the mere existence of this blog post is going to make that slightly more taxing and slightly more of a battle for everyone involved :/. (I mean: seriously... "ripgrep"?! This developer is clearly going _out of their way_ to be combative. What ever happened to the open source spirit of collaboration? What happened to actual communication between teams? Why do projects seem to just assume "the design decisions, or even specific implementations, or _even accidental mistake_ of existing tools are set in stone, and so the right way to talk about changing them is to discuss competition between entire projects or at best hard forks rather than working with other people"? :/)

<comment><USER7> > (or maybe had some core in there that was, while leaving the other UI parts > he wants to change on top)

But I did! Have you looked at the dependency list of ripgrep? It's utterly filled with tons of tools that you can pick out and use in other projects for any purpose you like. I didn't mention this in the blog post because there's already too much there, but sure, here they are:

memchr - Fast single byte search: [http://burntsushi.net/rustdoc/memchr/](http://burntsushi.net/rustdoc/memchr/)

walkdir - Recursive directory iterator: [http://burntsushi.net/rustdoc/walkdir/](http://burntsushi.net/rustdoc/walkdir/)

utf8-ranges - Generate utf8 automata: [http://burntsushi.net/rustdoc/utf8_ranges/](http://burntsushi.net/rustdoc/utf8_ranges/)

regex-syntax - A regex parser (including literal extraction helpers): [https://doc.rust-lang.org/regex/regex_syntax/index.html](https://doc.rust- lang.org/regex/regex_syntax/index.html)

regex - The regex engine itself: [https://doc.rust- lang.org/regex/regex/index.html](https://doc.rust- lang.org/regex/regex/index.html)

grep - Line-by-line search (as a library). This is where all of the inner literal optimizations happen, for example. [http://burntsushi.net/rustdoc/grep/](http://burntsushi.net/rustdoc/grep/)

And this is only the stuff that _I_ did. This doesn't count all the other wonderful stuff I used that other folks built!

And sure, I could do better. There's more stuff I could move into the `grep` crate, but this is only the beginning, not the end.

> "hey, this is something I've been working on: would you consider making grep > the first standalone tool to move to Rust, and what would it take from me to > make this happen?"

But I didn't want to do that. I don't understand why that's a problem. Doing this requires being POSIX interface compatible, and that's not a hole I care to dig. I don't mean any disrespect, it's just not what I want to spend my time doing.

> as opposed to writing an article about how "I am smarter than the GNU grep > people for these reasons and have built a tool named after how my tool is > going to kill their tool" which almost seems to go out of its way to set up > an air of competition rather than collaboration

I'm really sorry if I came across that way. It was of course not my intention. My intention was to write about what I had learned, which seems like a pretty fundamental component of collaboration. I'm sure five years from now, when I've moved on to other problems, someone will come along and beat ripgrep, and I can only hope that they write about it. :-)

I mean, if it weren't for the innumerable people who wrote about their experience with this kind of stuff, I never would have been able to get here. It only makes sense for me to write if I think I have something valuable to share.

> (I mean: seriously... "ripgrep"?! This developer is clearly going out of > their way to be combative.

I'm not. "rip" was supposed to mean "rip through text." I'm sorry it came off as combative. I wasn't even aware of the "rest in peace" interpretation until someone else pointed it out.

Honestly... I was trying hard to find a way to justify the binary name `rg` because I liked that `r` could stand for Rust. But `rustgrep` seemed a bit too in-your-face, so I started searching for small relevant words starting with `r`. That's it.

<comment><USER24> > I'm sure five years from now, when I've moved on to other problems, someone > will come along and beat ripgrep, and I can only hope that they write about > it. :-)

I contend that the world would be a much better place if instead of someone building a new tool which beats yours, they worked to improve your tool (which, at the point where you moved on, would hopefully have been granted to a body of separate maintainers, whether people you find or a group such as Apache which specializes in maintaining valuable open source projects). Sure: a world where people write about what they do is better than a world where they don't, but that's a really depressing thing to be "hoping for".

I am apparently becoming extremely unpopular in these circles for expressing this opinion, but a really important aspect of open source was about people collaborating towards a common effort to build high quality software: to avoid working together, to even expect that people will or should continually build new projects from scratch that "compete" with each other, defeats many (if not even most) of the benefits of open source software, as it relegates us to the same process by which closed source software improves.

> But I didn't want to do that. I don't understand why that's a problem. Doing > this requires being POSIX interface compatible, and that's not a hole I care > to dig. I don't mean any disrespect, it's just not what I want to spend my > time doing.

Providing some of the parts, or an offer to help, goes a long way: you don't have to do all of the work (and others wouldn't expect that); also, it is worth noting that GNU grep has on occasion added alternative backends (whether "extended" expressions or later using PCRE), and often have made improvements or added features. The assumption that grep is a tool which works the way it does and which will always do what it does, and that improvements should come in the form of competition, is demotivating to contribution.

<comment><USER7> > I am apparently becoming extremely unpopular in these circles for expressing > this opinion

To be really clear: I don't think your opinion is necessarily the problem. When I first read your comment, I typed up a response that I wasn't proud of. It wasn't nice because your comment wasn't nice. I had to step away from the computer and take a moment to put things back into focus to give you the response I did. It wasn't easy.

And really, my reaction to your comment had nothing to do with your opinion that we should try to collaborate more. That's a completely reasonable thing to hope for. But some of the things you said, or implied (about me personally), were really way way off, and I personally found them pretty insulting.

I get that you took my blog post as combative, so maybe you think the same about me. But you didn't ask for clarification, you just kind of dove right into the insults and assumptions and bad faith, and personally, I think that is just a really awful way to interact with other humans.

It's clear that we have different valuations on how to spend our time, and I really don't appreciate your implicit condescension. I also don't appreciate you telling me how I should spend my time. My free time is precious, and I want to spend it doing the things I find interesting. I don't want to work on a legacy code base, in C and spend _enormous social resources_ pushing on one of the most established C projects _in existence_ to switch to a new programming language. That does not sound like fun to me, and I want to work on something _fun_ in my free time. `ripgrep` happened to be it. (N.B. Fun is not the only criterion, but it's a big one.)

> but a really important aspect of open source was about people collaborating > towards a common effort to build high quality software

I've spent a huge portion of my free time in the past 2.5 years contributing to the Rust ecosystem. If that's not collaborating towards a common effort, then I don't know what is. `ripgrep` itself is barely a blip in that effort. All the stuff that went into building `ripgrep` that is freely available as other libraries? Yeah, that took a while.

</comment></comment></comment></comment></comment></comment><comment><USER25> I love the layout of this article. Especially the pitch and anti-pitch. I wish more more tools/libraries/things would make note of their downsides.

I'm convinced to give it a try.

</comment><comment><USER26> "if you like speed, saner defaults, fewer bugs and Unicode"

Warning - Conditional always returns true.

</comment><comment><USER27> When I use grep (which is fairly regularly), the bottleneck is nearly always the disk or the network (in case of NFS/SMB volumes).

Just out of curiosity, what kind of use case makes grep and prospective replacements scream? The most "hardcore" I got with grep was digging through a few gigabytes of ShamePoint logs looking for those correlation IDs, and that apparently was completely I/O-bound, the CPUs on that machine stayed nearly idle.

<comment><USER7> > Just out of curiosity, what kind of use case makes grep and prospective > replacements scream?

Unicode? Check out the subtitle benchmarks in the blog post. In the best case, grep is a little slower. In the worst case, grep is orders of magnitude slower.

ripgrep achieves speed by building UTF-8 decoding straight into its DFA regex engine (well, strictly speaking, this is Rust's regex engine, not ripgrep).

The other case where grep users might scream is when you're searching large code repositories. A `grep -r` might catch a large binary file, or search your `.git` or whatever. Both `ag` and `rg` will look at your `.gitignore` so that the results you see have higher relevance. (Of course, this is just a default, you can always "search everything" with ripgrep too!)

<comment><USER28> Maybe I'm unique to this, but that sort of default would drive me batshit insane. Not all of us are programmers by trade who use git. It's just another gotcha to keep track of. I'd seriously recommend removing that as a default.

That said, this is an overall awesome project.

<comment><USER7> The default isn't going to change. The Silver Searcher has, IMO, proven that it's a good default. I've spoken with so many people that love it, myself included.

If you want to not respect .gitignores, it's easy: `alias rg="rg -u"`.

(To be clear, I agree that the default is a trade off. I'm not so gung-ho on this that I think it should never be anything else. But for my project and its goals, I think it's the best fit.)

</comment></comment></comment></comment><comment><USER29> I'm never sure whether or not I should adopt these fancy new command line tools that come out. I get them on my muscle memory and then all of a sudden I ssh into a machine that doesn't have any of these and I'm screwed...

<comment><USER30> I have to switch back and forth between ack and grep all the time. Sometimes I use ack, sometimes I use grep. I wrote ack, but I've never stopped using grep, and ack has never been intended as a replacement for grep.

For most of the common ack flags, they are the same as grep. -i, -l, -v, -w, -A, -B, -C, etc. That was intentional, to minimize that whiplash.

One other suggestion is that you drop ack into your ~/bin directory that you sync between machines. It's a single Perl file, and that portability is a feature. As long as your machine has a Perl on it, you can use ack.

</comment></comment><comment><USER31> Thanks for the detailed comparisons and writeup.

I find this simple wrapper around grep(1) very fast and useful:

[http://www.pixelbeat.org/scripts/findrepo](http://www.pixelbeat.org/scripts/findrepo)

<comment><USER7> Thanks for the kind words!

Note that the key thing that `findrepo` doesn't support is respecting your .gitignore files. For example, in the Rust ecosystem, we often have a `target` directory in our projects that contains a lot of stuff we probably don't want to search. In fact, running `cargo new` will add that directory to your `.gitignore` automatically!

Tools like The Silver Searcher and ripgrep will ignore that directory (and _all_ others like it) automatically.

There are other advantages to ripgrep. For example, every other tool that supports Unicode as well as ripgrep (that's `git grep` and GNU grep) experience a substantial slow down when trying to use more advanced Unicode features (like \w, -i, etc.). This is one of the things my benchmarks show.

<comment><USER31> Yes ignoring .gitignore contents is a very useful feature. That could be added quite easily to findrepo though would probably have to be an opt as it is often useful to search intermediate build files etc. that aren't checked in. Whereas `git grep` handles the other use case of only searching checked in files.

Interesting info wrt efficient unicode processing for \w and -i.

cheers

</comment></comment></comment><comment><USER21> I'm glad to see this work get written up. If anyone wants a good project, there are some optimizations we (the Hyperscan team) have done in our "Teddy" SIMD string implementation that aren't captured (yet) in the Rust implementation to my knowledge. We're very happy to see techniques from our library get used in other projects as one of the points of open sourcing it ([https://github.com/01org/hyperscan](https://github.com/01org/hyperscan)) was to share how we do things with the community.

<comment><USER7> You guys have _so many_ amazing optimizations that I haven't captured yet. :-)

I personally love bragging about your Teddy algorithm everywhere I go! Thank you so much for opening up the Hyperscan project. It has been a huge boon!

</comment></comment><comment><USER32> Anyone have any suggestions regarding how to best use Ripgrep within Vim? Specifically, how best to use it to recursively search the current directory (or specified directory) and have the results appear in a quickfix window that allows for easily opening the file(s) that contain the searched term.

<comment><USER7> I'd like to get this working too, since I know a lot of folks are happy with it for ag and ack. rg does have a --vimgrep option that should make it as easy as ag to use, but I don't think there is a proper integration just yet.

<comment><USER33> I like ripgrep quite a bit from trying it today, and am hoping to find time to work on a Vim plugin this weekend. No promises, but I'll share as soon as I have something usable.

<comment><USER7> That's fantastic! Please don't hesitate to file an issue if you run into problems.

</comment></comment></comment></comment><comment><USER34> I am not sure how excited I am ... I readily accept this to be faster than ag -- but ag already scans 5M lines in a second for a string literal on my machine. Not having to switch tools when I need a recursive regexp is win enough to tolerate a potential .4s vs .32s second everyday search.

</comment><comment><USER35> nice, but does it compile and run on armhf? I don't see any binaries

<comment><USER36> I was able to build ripgrep from source for ARM, cross-compiling from my laptop running Debian, following the directions here:

[https://github.com/japaric/rust-cross#tldr-ubuntu- example](https://github.com/japaric/rust-cross#tldr-ubuntu-example)

(I haven't actually run it because I don't have an ARM linux device handy.)

<comment><USER37> Cross-compiled and ran a couple basic searches on an armv7l device. So at least the basic functionality works just fine.

</comment></comment></comment><comment><USER38> Why not make --with-filename default even for e.g. "rg somestring" ? That seems like it could hinder adoption since grep does it and it's useful when piping to other commands.

Is it enabled when you specify a directory (rg somestring .) ?

<comment><USER7> It should be the default whenever you search more than one file.

</comment></comment><comment><USER39> That is really cool. Although I think this is a case where Good Enough will beat amazing, at least for me (especially given how much I use backrefs).

</comment><comment><USER40> Does it use PCRE (not the lib, the regex style). If not, ack is just fine. My main concern with grep are Posix regular expressions.

<comment><USER41> It uses the re2 style, as provided by the rust regex library.

</comment></comment><comment><USER42> On a somewhat related note.

There does not appear be a popular indexed full-text search tool in existence.

Think cross-platform version of Spotlight's _mdfind_. Could there be something fundamental that makes this approach unsuitable for code search?

Alternatively, something like _locate_ , but realtime and fulltext, instead of filename only.

<comment><USER43> AFAIK mdfind heavily depends on file system events in macOS, it would be painful or impossible to implement such system in cross platform way with support for file systems like FAT etc.

<comment><USER42> sure, FAT. OTOH some people think it's not a totally intractable problem

 _A cross-platform file change monitor with multiple backends: Apple OS X File System Events,_ _BSD kqueue, Solaris /Illumos File Events Notification, Linux inotify, Microsoft Windows and a stat()-based backend._

[http://emcrisostomo.github.io/fswatch/](http://emcrisostomo.github.io/fswatch/)

[https://github.com/emcrisostomo/fswatch](https://github.com/emcrisostomo/fswatch)

<comment><USER43> nice.

</comment></comment></comment></comment><comment><USER44> Superb work, and a superb writeup. It's really great to see such an honest and thorough evaluation.

</comment><comment><USER45>
      Mega-Thanks to the authors of grep (longtime user)
                                     ack (nice innovation)
                                      ag  (outstanding work)
                                      rg  (outstanding work)

</comment><comment><USER46> Impressive! It works really well. Has anyone set it up with ctrlp in vim? I have `rg %s --files --color=never`, which works, but shows an empty white line at the prompt, so I need to use a cursor to jump to the desired file.

</comment><comment><USER47> Great tool. Does there exist a faster implementation of sort as well? I once implemented quicksort in C and it was faster than Unix sort by a lot, I mean, seconds instead of minutes for 1 million lines of text.

<comment><USER7> I've never waited more that a few milliseconds to sort a million lines.

I expect GNU sort does quite a bit, including handling data sets that don't fit into memory.

Anyway, this post is about regexes, which really has nothing at all to do with sorting. They are two very different problems. :)

</comment></comment><comment><USER48> rg is harder to type with one hand because it uses the same finger twice. :)

<comment><USER49> It does?

I guess if you use a rigid fingering system. I never bothered to learn that back in elementary school, so my fingering is ad hoc based on what I'm typing. To type "rg" (using QWERTY), I'd just bring my middle finger up to R while using my index finger for G. This would probably be a little slower than "ag" because my hand is more likely to already be in place to type the latter without movement ("home row"), but not as slow as reusing a finger would be. It's not something I'd have to think about; this is already what I do whenever I have to type "rg" as part of a word.

I'm curious whether an ad-hoc approach is more or less efficient overall. Fingering customized per word clearly has the potential to optimize finger movement, but my error rate is relatively high - mostly timing-related - <USER2> might be exacerbated by an ad-hoc system because there are more (and more complex) unfamiliar transitions between words.

Anyway, I upvoted you for mentioning typing. It might seem trivial - well, if you use a lot of custom aliases, it is trivial - but if a command runs fast enough (and ag is already very fast on small source trees), the time spent typing its name can become a significant bottleneck. The author of Pijul, for instance, a version control system meant to compete with Git, seems not to recognize this... the command is 'pijul', which is essentially impossible to type on QWERTY without reusing a finger at least once.

<comment><USER48> I'm not super rigid with all keys but I do seem to have "R with index finger" ingrained. R with the middle finger does work nicely in this case if I think about it and force myself to do it.

</comment></comment></comment><comment><USER50> It looks very good and I'd like to try it. However I'm lazy and I don't want to install all the Rust dev environment to compile it. Did anybody build a .deb for Ubuntu 16?

<comment><USER51> If you don't want to compile it yourself, the blog post has links to binaries. Since they're entirely self-contained, you don't need a full .deb to compile them; just delete the binary when you want to get rid of it.

<comment><USER50> I didn't notice that. Thanks.

It works. The binary is 11 MB, or 1.5 MB stripped. ag is 69176 bytes. Shared libraries were a good invention :-) Sooner or later rg will be packaged properly too.

<comment><USER52> I'd much rather have a self contained binary than 11 more MB of free space.

</comment></comment></comment></comment><comment><USER53> Looks like every tool has its upsides and downsides. This one lacks full PCRE syntax support. Does one have to install Rust to use it?

<comment><USER54> No. binaries: [http://blog.burntsushi.net/ripgrep/#installation](http://blog.burntsushi.net/ripgrep/#installation)

</comment></comment><comment><USER55> Nice writeup! Any chance you'll support macports for those of us who never jumped ship to homebrew?

<comment><USER7> I'm not a mac user, so I'm terribly unfamiliar with the ecosystem. In principle, I have no problem with supporting macports, but I haven't looked into it.

One thing that would be a huge help is if someone briefly wrote up what would be necessary to get ripgrep into macports: [https://github.com/BurntSushi/ripgrep/issues/10](https://github.com/BurntSushi/ripgrep/issues/10)

</comment></comment><comment><USER56> > We will attempt to do the impossible

Oh well. Waste of time then.

<comment><USER57> Maybe the author hasn't had breakfast yet and has only done 5 impossible things so far?

</comment></comment><comment><USER58> Tragically the news that LLVM is switching to a non-Copyfree license (see copyfree.org/standard/rejected) has ruined everything... Nothing written in Rust can be called Free Software. :(

<comment><USER59> According to that site, Free Software (as defined by the FSF) is not Copyfree.

As far as I can tell, Copyfree is like the BSD license, except you can only restrict changes to the license text itself; the license itself may not be changed or added to, but everything else in the project may be used and changed in any way. So, effectively, it seems like a license where the license itself is copyrighted with all rights reserved, but the rest of the project is public domain.

So, just like BSD, anyone can take a Copyfree project proprietary, turning it non-free.

The GPL is as important today as ever.

</comment></comment><comment><USER60> 1\. Ag have nice editor integration. I would miss emacs helm-projectile-ag

2\. Pcre is good regexp flavor to master. It is have good balance of speed, power and popularity. In addition to Ag, there are accessible libraries in many languages, including python.

I think it would be good if everyone settled on Pcre, rather than each language thinking they will do regexps better.

<comment><USER7> PCRE suffers from worst case exponential behavior, so it's not suitable for all tasks.

For the most part, the syntax supported by ripgrep is a strict subset of the syntax supported by PCRE.

But yes, I can agree that supporting PCRE can be considered an advantage if you use advanced features heavily (backreferences and lookaround come to mind).

<comment><USER61> Meh. Back-references and lookaround both take too much brainpower to use at an interactive shell. I've used them few times in programs that I was writing, but just to find some text in some files on my disk? Never.

</comment></comment></comment><comment><USER62> >It is not, strictly speaking, an interface compatible "drop-in" replacement for both, but the feature sets are far more similar than different.

</comment><comment><USER63>
        ...
        $ rg -uu foobar  # similar to `grep -r`
        $ rg -uuu foobar  # similar to `grep -a -r`

I knew it. The name is absolutely ironic. I cannot just drop-it-in and make all my scripts and whatever scripts I download work immediately faster (nor is it compatible with my shell typing reflexes). New, shiny, fast tool, doomed from birth.

<comment><USER7> It fundamentally can't be interface compatible, sorry. I think I was pretty clear about this in the blog. :-)</comment></comment>


Carl Sagan on the Meaning of Life - <AUTHOR> http://www.brainpickings.org/index.php/2013/07/08/carl-sagan-meaning-of-life/?utm_source=buffer&utm_campaign=Buffer&utm_content=buffer817dc&utm_medium=twitter <comment><USER2> I love this guy, very inspirational</comment>


Yakety - <AUTHOR> https://yakety.co.uk <comment><AUTHOR> Hi everyone, I have managed to get my platform, Yakety, into a state where it is just about usable and I'm looking for Beta users to put it through its paces and inform me of the direction it should go next. If have project that is suitable and you're interested then get in touch with me at email@example.com to discuss the potential of us developing your system collaboratively. All the best, Matt</comment>


Understanding JavaScript module formats and tools - <AUTHOR> https://weblogs.asp.net/dixin/understanding-all-javascript-module-formats-and-tools <comment><USER2> Nice summary, the only pain left is that commonjs is still the module that dominates in the nodejs ecosystem.</comment>


Let's Give Visas to Startup Founders - <AUTHOR> http://www.businessweek.com/technology/content/dec2009/tc2009121_842902.htm <comment><USER2> "The idea and the founder's résumé would then need to pass muster with a government or industry-appointed board of venture capitalists, financiers, or technology experts"

This is the fatal flaw in the program. The whole _reason_ startup culture exists is that we don't have a way of knowing in advance what ideas are really worth pursuing, short of giving them a try. But this program would create a board of experts whose job it would be to try to predict which ideas had a reasonable chance at success.

The obvious outcome would be a huge distortion in the kinds of startups foreign nationals would found. The value of a green card is far higher than the expected value of a startup, so the process would turn into a bad example of grantsmanship - trying to tailor your proposal to what you think the committee wants to see, and then making sure your startup stays solvent long enough for the permanent residency to come through.

Rather than taking real risks and trying to innovate, the role startups are supposed to take in our economy, we would be encouraging an extremely conservative startup culture, whose real purpose would be to secure green card status for the founders lucky enough to get in the program. How likely are you to bet the company if the consequence of failure is getting kicked out of the country?

<comment><USER3> To be fair, this was not at all part of the original proposal; it's one of those idiotic riders tacked on by politicians after they get their hands on something. The only "approval" required was in the form of investors stumping up cash. Any accreditation should be of the _investors_ , not the the founders.

<comment><USER4> This is the problem area that bugs me about the founder visa concept. The concept is great. Embodying it into law requires it go through the sausage factory.

I jokingly told a doctor last week I could write health care reform in 5 pages. I'm pretty sure I could. But it wouldn't get through congress in that form.

<comment><USER5> Well, there is potential for abuse: a startup founder gets an "investment" of $500k from an accredited angel and a permanent resident visa. A little while later, the startup takes an exit (e.g. dissolves the company and returns the assets to the investor), netting the founder a greencard in the process.

<comment><USER6> If you've got somebody willing to wire you $500k, you can basically do that already.

<http://en.wikipedia.org/wiki/E-2_visa>

<comment><USER5> Well, that requires that you leave the country immediately after the business has been concluded; this proposal says that "... the founder would be granted a permanent resident visa", which presumably wouldn't expire.

</comment></comment></comment></comment></comment></comment><comment><USER7> A better solution would be to just fix immigration. Adopt something similar to the Commonwealth point system [http://en.wikipedia.org/wiki/Highly_Skilled_Migrant_Programm...](http://en.wikipedia.org/wiki/Highly_Skilled_Migrant_Programme), where you get points for being young, educated, in a useful profession, etc. Get enough points and you're automatically allowed in.

A system like this would reduce the distortion, and allow talented people into the country to bootstrap or e.g. work at Google.

<comment><USER8> Also removing worldwide taxation of non-resident income of non-resident citizens. (As in, if I make a million a year living in peru, I don't want to have to pay taxes for the income above $80'000 to the IRS that I made in Peru while I lived in Peru). The US is the only (or one of a very small number) of countries that do this. I do not want to become a US citizen expressively for that reason.

</comment></comment><comment><USER9> The idea is a good one, but I'm wrestling with the dollar amount that needs to be invested before the founder can get a visa. Specifically, how likely is it that a US venture capitalist will invest $250,000 or more in a startup that's not located in the US (or Silicon Valley for that matter) vs. another startup that is?

Even if the non-US startup will move to the US once they receive funding, the largely intangible advantages bestowed on a startup in the Valley (which I've witnessed in the month that I've lived here) give it a better chance.

To be successful in facilitating the immigration of founders to the US, the founders' Visa should be available to any startup that has received any funding from a "qualified US investor" over $15K, which would include non-US startups accepted into YC.

And to avoid creating a huge immigration loophole (which seems to be the author's rationale for suggesting a "real money" cutoff), perhaps the Visa could have a clause that limits it to two years unless additional capital is secured and the startup is deemed to have "passed muster", at which point a permanent resident visa could be granted.

<comment><USER10> _Specifically, how likely is it that a US venture capitalist will invest $250,000 or more in a startup that's not located in the US (or Silicon Valley for that matter) vs. another startup that is?_

It might be more likely than you think. If the startup and the founders are solid, and if there is a clear (and most importantly, fast) visa path like the one proposed, I don't think most investors would have a problem. If anything, $250,000 is way too low for a VC, so you might have to find an angel investor instead. Some countries would have an advantage over others, of course, but I don't think it's a showstopper - my first startup, based in Canada, received an offer for funding which would've put us over this threshold.

 _Even if the non-US startup will move to the US once they receive funding, the largely intangible advantages bestowed on a startup in the Valley (which I've witnessed in the month that I've lived here) give it a better chance._

Can you elaborate? What is the difference, once both startups are in the Valley, and assuming there are no language issues? So many startup founders are already originally from other states or even countries.

<comment><USER9> It may be more likely than I think, but what matters is if it is so much more likely that P(Silicon Valley startup getting funding) < P(Canada/foreign startup getting funding), all other things being equal. Do you think that this is the case?

I don't think that every startup needs to be in the Valley to get funding, but what I've witnessed here is a prime example of "it's not (only) what you know, it's who you know". If you have a viable startup, knowing people that have made names for themselves in the startup scene, many of which are based in Silicon Valley, can go a long way towards getting introduced to potential investors, whether they be angel investors or VCs.

<comment><USER10> I don't think we're disagreeing on much. Yes, of course, a startup with better connections in the US has a better chance to get funding from US investors, and it's easier for an US startup to make these connections.

My point is that P(Canada/foreign startup getting funding) is greater than zero even with today's visa situation, and is still large enough to make a visa program worthwhile, even if it is smaller than P(Silicon Valley startup getting funding).

</comment></comment></comment></comment><comment><USER11> Current waiting time for China and India-born employment-based Green Card application is likely to be 4-5 years.

Many of those applicants are potential startup founders. Sadly they cannot pursue their ambitions due to the long immigration process. Some of them will choose to leave US and found their startups in their home countries.

<comment><USER12> Speeding up the Green Card process is actually a better solution & also has the greater probability of happening vs granting a founders visa.

</comment></comment><comment><USER13> Why can't the venture capital go there. Every article on Venture these days says American VCs are looking overseas more than here. Surely they'll find the best and brightest with great ideas and can invest at a better exchange rate by keeping entrepreneurs local. We have more than enough innovation in this country. Our problem is once the idea is proven we can't make an industry out of it without being undercut.

</comment><comment><USER14> So my question is - if the startup has $250k in funding, why can't they sponsor for an H1B for the founder(s)?

<comment><USER15> Yearly quota. Only 65k applications for H1B visas are accepted every year. To be able to sponsor someone out of the quota, the institution has to be either a non-profit or educational - universities and such.

<comment><USER14> Good point.</comment></comment></comment>


"An attacker can listen to _any_ UK Vodafone customer's phone call." - <AUTHOR> http://thcorg.blogspot.com/2011/07/vodafone-hacked-root-password-published.html

<comment><USER2> This is big. But at the same time I bet that many other wireless carriers have similar flaws in their micro cells, where their infrastructure blindly trusts them to have not been tampered with.

They are products developed by teams independent of those who designed the security of the backbone of the phone network.</comment>


Ask HN: How do I contact 65,000 people? - <AUTHOR> Hi,
My first submission to HN, hope I've done it right.
I have a web application that has 65,000 signups over a 4 year period. I want to launch a new version of the application and tell all of the users about it.
How do I go about emailing 65,000 people? I'm pretty sure some are going to hit the 'spam' button and that is my biggest concern from sending them out on the server.
What does everyone recommend?
Thanks! <comment><USER3> Yes, MailChimp, ConstantContact, AWeber, CampaignMonitor are all good ideas.

But...

Here's the first thing you need to consider: why do people use these services instead of writing their own? You already answered - you want it to reach the person's mailbox instead of being marked as spam. That means that these services (and others like them) depend on their reputation as good mailers for delivery. They've built that reputation over millions/billions of interactions.

So here you come with a list of 65,000 - that's a lot of folks. You can't tell them/us how many of these are still active. You don't know how many will hit the spam button.

Where I'm going with this is to consider this question: Why would any of these services risk harming their reputation by sending 65,000 emails for a first- time bulk emailer like yourself?

I suspect (but don't know for sure) that you will get resistance from the companies. Let's just say that 15,000 of your addresses are gmail accounts and that, after they send your email, 1000 of them fail as "Invalid email account", another 1000 mark your email as spam, another 12,000 do not even open your email, and 1000 open it. Of the 1000 who open it, 100 click through.

How will Google react to that? I'd guess they would start lowering their trust for that bulk mailer.

Anyway, I'm rambling. I remember years ago signing up for ConstantContact and they made me do a phone interview and a test run of 1000 emails. They told me that if I ever received more than 1 spam report per 1000 emails, I would likely be asked to leave. I don't know how serious they were and, since it's been several years, I don't know if they still do it.

Just something to consider!

<comment><USER4> >They told me that if I ever received more than 1 spam report per 1000 emails, I would likely be asked to leave. I don't know how serious they were and, since it's been several years, I don't know if they still do it.

They do, but it's not an automated process. I was sending to a customer list of about 20,000 emails on a monthly basis, and got a phone call when I added an older (3 year old) list to the mix. They asked me about the source of the list, walked me through best practices (e.g. don't send to old lists - <USER2> customers might not remember who you are), and things were fine after that. So, unless it's a blatant situation, I suspect that you get fair warning.

I think in this case, the greatest concern -is- going to be the age of the list. I'd be wary of sending mail to anyone you haven't had some contact with in the last six months.

</comment></comment><comment><USER5> The low impact version would be to use the existing application to communicate with the users. You can either tell them there's a new version coming (with a link to a landing page with a call to action) or you can ask them if they want to opt in to announcements regarding the new application.

Alternatively you could use a service such as mailchimp to send emails to all 65,000 users. If this is the only time in 4 years that you've emailed them, then I don't think many people would react badly as long as you introduce yourself, let them know why they're getting the email and give them the opportunity to opt out (which they'll get through a service like mailchimp). It also helps if you keep things completely on topic (i.e. "you're receiving this because you signed up to our app and we wanted to tell you about some of the new features we're rolling out in the next version").

If you're getting a 1-2% conversion rate from mail shots you're doing well, so you might want to consider both (I'd probably do the mailshot first, then the page rather than the other way around).

<comment><AUTHOR> Wow thanks for the quick reply. I've had a look at some of the rules attached to CampaignMonitor and looks like I can only send to the last 2 years worth with them -

<http://www.campaignmonitor.com/anti-spam/>

</comment></comment><comment><USER6> It's nowhere near the same scale, but when I wanted something to email all the people registered for the MathsJam (this weekend - geting nervous and excited!) I just wrote a script that connects to my SMTP server and sends an email. That way each email is personally addressed and only has the single recipient, making it less likely to be considered spam. The text is personalised as well, populating various fields from the data I have on that individual. Because the email goes only to them, that's OK. I don't believe that would be possible with a bulk emailer.

I'm only emailing about 200 people and it takes about 2.5 seconds for each email. To hit 65000 people would take about 2 days, but at least each person would get an email sent specifically to them.

<comment><USER7> But how do you know that the individual recipient's mail client/service doesn't categorize your message as spam? If you're baking your own e-mail solution, you need a 'good' IP address, correct SPF/reverse DNS/headers/obscure SendMail tweaks, and much other magic.

<comment><USER6> I'm using the SMTP gateway that my well-known ISP provides, just as if I'm sending each one, lovingly hand crafted, via a standard email client from a regular desktop.

So here's the question: how do my regular emails not get classed as spam?

<comment><USER8> It's fine unless you have many recipients on one particular domain (or one particular MX). Then it'll notice.

For example, I recently did an email shot for someone. Of 3936 registered users, 924 are @hotmail.com and 357 are @gmail.com.

(these were sent from a server which has built up its reputation over time; I assume there would be a problem with an order of magnitude more emails arriving from a new location)

</comment></comment></comment></comment><comment><USER9> <http://www.mailchimp.com> and <http://www.campaignmonitor.com> are probably two best options

</comment><comment><USER10> Try Madmimi. I use them constantly and they are solid. Their stats are a bit weak, but they are more relaxed as to the origins of the email list.</comment>


Ostrich Pillow - take a nap anywhere - <AUTHOR> http://www.kickstarter.com/projects/ostrich-pillow/ostrich-pillow

<comment><USER2> Does this really belong on Hacker News? This is the stuff of your standard infomercial or Sharper Image catalog. Kind of gimmicky concept that tries to convince you that really, it's cooler and more useful than it looks. Not sure why this is on HN.

Also, doesn't this violate Kickstarter's new "Kickstarter is not a store" rules, such as no having the rewards simply be "1 of x, 2 of x, 5 of x, 10 of x"? And $75 plus $15 shipping? That seems pretty steep for... a pillow, even a fancy one.

<comment><USER3> I think you might be right:

"Offering multiple quantities of a reward is prohibited. Hardware and Product Design projects can only offer rewards in single quantities or a sensible set (some items only make sense as a pair or as a kit of several items, for instance). The development of new products can be especially complex for creators and offering multiple quantities feels premature, and can imply that products are shrink-wrapped and ready to ship."

Who the hell wants ten ostrich pillows anyway? A ten-headed narcoleptic hydra?

<comment><USER4> The guideline only affects projects started after September 20. This one was started on the 18th.

</comment></comment></comment><comment><USER5> I thought there would be at least some sort of acknowledgment in the video of how absurd the thing looked - a little self awareness might go a long way towards connecting with people otherwise put off my looking like an insane person in public spaces.

<comment><USER6> _Queues PGs post on Segways_

<comment><USER7> Unless there are several of them, queuing them before cueing them is probably unnecessary. ;)

</comment></comment></comment><comment><USER8> Unfortunate name in that wearing it in public will definitely get you ostracized - I have almost no doubt about that.

I think Id be genuinely afraid to wear this thing out in that it probably suffers from the same thing mascots suffer from - there is an urge to tackle you or bop you as if your a stuffed animal. I just feel like inevitably you'd be awoken by a whap to the back of the head by some teenager on a dare.

<comment><USER9> I think you mean ostrichized, no?

<comment><USER10> England here and we think you both mean: ostracised.

</comment></comment></comment><comment><USER11> I thought this was a nicely put together spoof of kick starter, I'm flabbergasted how that has raised so much money. The people using it look like the heaven's gate cult photos from years ago.

<comment> chao- I don't really understand the cult connection, even after Googling Heaven's Gate (I am too young to remember much about them directly).

But on the topic at hand, I would buy one of these, or at least, I definitely would if I were still in High School. On state standardized test days (normally a half-week occurring a few times per year), I (and many peers) would inevitably finish in about 2 of 4 hours. Of course we were obligated to stay in the room and remain completely silent for the entire 4, so I would fold a jacket such that it fashioned a very poorly-functioning version of this.

The main issue while using a jacket was having proper respiration between my face-down head and the desk. If this product solves this when it comes out (maybe I can try it out somewhere? Brookstone or some such?), I can see myself considering it.

<comment><USER11> They all committed suicide with cloth bags over their heads. It's an extremely creepy sight and I won't be linking to the photos for many obvious reasons, but...they look just like they're all wearing this product.

</comment></comment></comment><comment><USER12> Dying watching this video over here. Watched it twice and couldn't stop laughing ...

Some missing FAQs:

* How fricking hot is it in that pillow-hat?

* How do you prevent people from messing with you when sleeping on the train?

On a regrettably serious aside, I _am_ curious if there is a natural advantage to sleeping with you hands over your head - opens your lungs up, but would make circulation more difficult ...

<comment><USER13> I too wondered about the heat. That is usually the first thing that affects my ability to sleep.

</comment></comment><comment><USER14> The design is based on a terribly unergonomic position, it's ugly, and it looks perfectly possibly to suffocate yourself with it.

<comment><USER15> I backed the project and I kind of agree that the armholes are kind of unnecessary because that's not the most ergonomic way to use it, at least for me. However that doesn't preclude using it the way I plan on using it, which is arms crossed on the table in front of me outside the ostrich pillow with my head in the pillow resting on top of my crossed arms. That's perfectly ergonomic and a position I've slept in multiple times.

This project is particularly interesting to me because I'm not physically able to sleep reclining or on my back because I snore so loudly and with such vigor that my uvula swells up to the point that I can partially swallow it into my throat. I welcome anything that makes sleeping face down, but still breathe. I also see this useful on those cold winter nights where you want to keep you head warm as well when in bed.

This is one of those project that either makes total sense to you or doesn't. For me, it'll be a godsend. There are tons of times I've wanted something like this, looking like a goofball be damned.

The pillow reduces not only light but sound too, while allowing you to breathe. This is already better than most other sleep aids for sleeping in public such as eye masks.

</comment></comment><comment><USER16> Using one of these in a public place will be an invitation to get mugged.

<comment><USER13> I'm guessing this would get used in very loud public places, like airports or train stations. I'm also guessing that muggings would be rare because of all the bystanders.

However, wearing one of these in public is an invitation to get your picture taken and posted on twitter.

<comment><USER17> I think airports and university libraries are their big markets. Public space would be off limits in my opinion.

</comment></comment></comment><comment><USER18> I'm sorry, but this thing looks downright ridiculous.

<comment><USER19> If you're willing to sleep in public then you already don't care about looking ridiculous.

I find this product highly appealing.

</comment></comment><comment><USER20> Who knows, snuggies (I think that's the name, the backwards bathrobe/blanket looking thing) have made lots of money...

<comment><USER21> I agree, by looking at the percentage already funded, I think it has to the potential to follow Snuggie's success. Infact, I found Snuggie to be a ridiculous invention when it first came out and have good doubts about its success. Well, its just a blanket with sleeves, who wouldn't see that through! right? but boy was I wrong. It even made it to the people's top holiday shopping list. So when I have a chance I snagged the misspelled variation of its domain <http://www.snugies.com> when it expired from a domain squatter and did it referral redirect to the product page on Amazon.

<comment><USER22> Just curious, how meaningful was the monetary return on the referral redirects from snugies? Wondering if we should do the same and get ostrichpillows or something.

<comment><USER21> Don't have an exact figure since I use the same referral tag on different sites but my estimation is about $100 per month in commissions for this one page static website. Domains are inexpensive and they are the cheapest investment you can have, so I would suggest getting it. However make sure you get the SEO right. If you Google snugies, the first result is the link to the above site. Try to have yours listed within the first 10 results for the prime keywords.

Good Luck!

</comment></comment></comment></comment><comment><USER23> Kind of expensive, I'll wait for a cheap Chinese knock-off on eBay.

<comment><USER24> You suck at life

</comment></comment><comment><USER15> This with an inflatable cheese wedge pillow would be the best thing on flights.

Something like this:

[http://www.travelersjoy.com/images/item_images/archive/00000...](http://www.travelersjoy.com/images/item_images/archive/0000007364.jpg)

Leaning forward on flights like that is my favorite flight sleeping position.

<comment><USER25> I don't know how anyone could actually use the wedge pillow on a flight. There is never enough room to use it.

</comment></comment><comment><USER26> I don't care how ridiculous it looks, I think this is an amazing idea. I do not do well in public places and have to find a way to detach every once in awhile or I gradually get more and more irritated. This pillow is a godsend. The next $75 I get is going straight to this.

</comment><comment><USER27> I don't care how ridiculous it looks. If it solves the red-eye and airport sleeping problem for the rest of us, its worth it.

</comment><comment><USER28> Basically it's like sleeping with an Half-Life headcrab in your head.

</comment><comment><USER29> Might make a hilarious sex toy. Absurd for any other purposes.

</comment><comment><USER30> It make me laugh! Is it a "joke" ala <http://en.wikipedia.org/wiki/Sokal_affair> ?

</comment><comment><USER31> Sleeping in a public place? With a bag on your head (and you hands tucked into it)? Isn't that just asking to be molested or robbed?

</comment><comment><USER32> I wonder if it includes flame retardant chemicals like other pillows do in some countries...</comment>


French Presidential hopeful Melenchon appears as hologram at rally - <AUTHOR> https://www.youtube.com/watch?v=EejJoePprsA <comment><USER2> He menaces to punch on the table of EU. How much noise a hologram punching on the table does?</comment>


Show HN: uMOOC, an online tutoring platform for Harvard's CS50 and MIT's 6.00.1 - <AUTHOR> http://www.umooc.org <comment><AUTHOR> I'm Ragheed Al-midani, one of uMOOC's two co-founders. I met my co-founder, Dwayne Kennemore, when we were both in CS50 at Harvard Extension School. uMOOC was his idea (he will probably share his inspiration here shortly), and I offered to help. The first prototype of uMOOC was built as our final project for CS50, but the scope of the project has grown tremendously since then.

If you're wondering what our stack looks like, we're using (among other things) Python 3, Django, Django-Rest-Framework, Ember-CLI, Emblem, Sass (the original, indented version), Bootstrap, CoffeeScript, and EasyRTC.

If you have any questions or concerns, technical or otherwise, please don't hesitate to ask!

<comment><USER2> This is very cool! Have you considered implementing group sections.

That way, students can pay less and tutors can earn more. If 10 students pay $10/hr, that's $100/hr for the tutor, and the students would still benefit immensely.

<comment><USER3> Hello, I'm Dwayne Kennemore, the other founder of uMOOC.

A group section would be terrific for another reason - in academic contexts, while students DO learn from their professors, they learn from ONE ANOTHER as much or more. There are a few reasons for this, but one of them is, I think, that they're more at ease with one another and don't have to worry about looking bad - this openness helps ideas flow more freely.

Group functionality would make that dynamic possible, further replicating the network that occurs in real life, in cyberspace.

You are already anticipating v2.0.

<comment><USER4> Won't it then just become an online class? How will you differentiate?

<comment><USER3> Think of it like a breakout session of an online class rather than an online class itself. Why is that useful, you may ask... Having a group and sense of connection helps keep people motivated.

</comment></comment></comment></comment></comment><comment><USER5> This is an interesting idea: online course + tutor. The tutor is what was missing from the original formulation. Just solving at online problems and quizzes doesn't work except for very few.

A person needs another person to see his/her efforts, to appreciate, in order to maintain motivation. That's the main role of the teachers in the brave new world of MOOCs - to witness. The actual teaching and testing is automated. So the teacher would be more like a coach. He would offer guidance, boost your confidence, and push you over the limits.

<comment><USER3> Yes, that is exactly the idea. We know it is not perfect of course. We invite any input for suggested changes or additions. This project is not for our egos, but rather, to help an online community.

</comment></comment><comment><USER6> Can you talk a little about your experience at the Harvard Extension School? What it is, what the experience was like, what courses you took, the cost, and the job prospects post graduation.

<comment><AUTHOR> Harvard Extension School (HES) is an open-enrollment school at Harvard that lets you take individual courses, as well as pursue certificates and undergraduate and graduate degrees.

My experience with HES has been very positive. I've taken 2 courses there so far: CS50 and Calculus 1. I took both as a remote student. This is nice if you're busy with other things like family and work, because you can watch the materials when you're free, alert, and most ready to learn.

I intend to pursue the Masters in Software Engineering, but I'm taking things one step at a time.

Undergrad courses in most subjects are currently $1,550, while graduate courses are $2,700. I believe only Computer Science and Digital Media courses are priced at the graduate level even if you take them for undergraduate credit.

You actually get a Harvard transcript, and if you earn a degree, you get to participate in the annual commencement in May. To get a degree, you have to complete at least one course by going in-person to Cambridge (more for some degrees).

I don't have any first-hand information on job prospects, but I would imagine the more you can make it to Cambridge, the more classmates and professors you can network with.

For more information, please check out the HES website:

[https://www.extension.harvard.edu/](https://www.extension.harvard.edu/)

<comment><USER7> > I believe only Computer Science and Digital Media courses are priced at the > graduate level even if you take them for undergraduate credit.

It depends on the course. I've been very happy with HES and expect to graduate with my ALB May 2018. Besides the list of courses on the HES web site, I've taken advantage of many additional opportunities: a Teaching Fellow for the Intro Data Science class where I helped HES, regular college and grad students, and Head TF for Advanced Data Science class this Spring (CS109b).

With a certain GPA and number of credits, you can get Special Student status opening up virtually the entire Grad level course space for HES credit. This past Spring I took CS280, a grad level AI class, where I was one of two non traditional students in a class of 20 consisting of mostly CompSci Phd candidates.

HES opens up many opportunities. Like anything, it is what you make of it that counts.

</comment></comment></comment><comment><USER8> I dunno if this is any good honestly.

All this monetization of learning is the wrong approach.

In a world where software is moving towards FOSS it's depressing to see people still trying to sell education instead of trying to teach it.

Education is a journey, teaching is a joy, and if it's not, no amount of money can make you want to teach anyone anything.

I think that's why institutes with great researchers have great teachers and students, it's an environment feeding itself in a positive feedback loop.

It's depressing to see this here.

<comment><AUTHOR> I think we need to distinguish between charging people money for access to knowledge and charging them for personalized guidance in seeking that knowledge.

I personally believe all textbooks and courses should be free. But if a tutor is going to dedicate an hour of her time to sit down with me and help me learn, it is not unreasonable for her to demand compensation.

Universities, of course, are only able to release MOOCs for free because they make money from tuition, alumni donations, endowments, etc.

</comment></comment><comment><USER9> A MOOC is a massive online course, so why calling your service uMOOC if it provides 1:1 tutoring? It is not a class, it focus on a single learner and it is not free (which is expected as well from MOOC, otherwise it is traditional remote education) so it seems very far from the spirit of MOOC.

<comment><USER3> The idea behind the name is that rather than just tutoring generally (which is provided by sites like wyzant), these tutors are specially selected for their competence with certain MOOCs - they've taken these courses before, typically as a student in a university. The "u" represents the learner - while MOOCs are a depersonalized learning medium, this service puts "u" back into it. We don't offer MOOCs, but a service used parallel to taking a MOOC to enhance the student's likelihood of success.

</comment></comment><comment> barry-cotter You need to change the signup process as at present it's needlessly complicated. Until you add more courses remove the search, institutions and subject and just let people click on the currently available choices. Try and build a community on those two courses, then expand to other computer science courses, then branch out. You only have two options, offer them and only them.

<comment><AUTHOR> Great observation and suggestions. Thank you!

We initially wanted to list courses involving many subjects and institutions (basically the entire edX catalog). Later, we decided to offer two initial courses, so we could focus our tutor recruitment efforts.

We will definitely put your suggested change on our short-term agenda.

</comment></comment><comment><USER10> Plenty of students are more than willing and able to help other people, _free of charge_. I personally think it would be a better use of time cultivating that and structuring it to be of maximum benefit to the tutor and the reader/person receiving knowledge.

E.g.

For a given topic, "Topic A", there may be subtopics, "Subtopic A", "Subtopic B" and "Subtopic C." One possibility may be allowing different students to explain, in their own words, the varying subtopics. People can vote on which one was the most helpful/clear as well as help correct people who may have misunderstood.

As a result of this, "Topic A" may have a personalized, "user perfected" version consisting of the crowd's own vetted explanations.

Paying tutors only helps the elite, who wouldn't need this to begin with. People falsely believe that help should come at a cost. Plenty of people post comments and give advice on this very site for free. As soon as you introduce money you're going to change the incentives to maximize money, not benefit to the person receiving the help. This is not bad, but it runs counter to the ideals most people think of when they think of "education" (or maybe not, who knows).

</comment><comment><USER11> The name reads like micro-MOOC to me, which a) is kind of funny (micro- massive-open...) and b) reminds me of this: [https://en.wikipedia.org/wiki/Micromoog](https://en.wikipedia.org/wiki/Micromoog)

How is it supposed to be read? you-mooc?

<comment><AUTHOR> Yes, you-mooc sounds right; we're putting "u" back in the MOOC, by connecting you with tutors who can help you on an individualized basis.

MOOCs are great; I personally have completed 5 of them, and plan to complete many more. But not all learners enjoy or are able to learn in solitude--many want someone to point them in the right direction and give them feedback, like you would get when taking an actual college/university course. That's the gap we're trying to fill.

</comment></comment><comment><USER12> What differentiates you from other similar platforms that offer tutoring (e.g. Pluralsight or codementor.io to name two random ones)?

<comment><AUTHOR> tl;dr uMOOC tutors are qualified to immediately and directly help you in a specific MOOC.

In the case of CS courses (which is what we are focusing on for our initial launch), it's not enough for someone who wants to be a tutor to know C or Python. They must have taken the course on-campus, or a very similar one, or completed the MOOC itself. In the case where a tutor is not familiar with the material in a specific course, we gently push him/her to enroll in the MOOC's current run.

To take things a step further, one of our tutors completed the CS50 MOOC back in 2013 or 2014, when the second half of the course was taught with PHP. He said he was going to enroll in the MOOC again and do the later assignments, which now involve Python and Flask.

</comment></comment><comment><USER13> Nice.

From the title I thought this was something like 'self service hosted mooc platform', which I was also really excited about.

<comment><USER14> You might want to check out these two projects:

[https://github.com/learningequality/ka- lite](https://github.com/learningequality/ka-lite)

[https://github.com/learningequality/kolibri](https://github.com/learningequality/kolibri)

I'm not associated with either, I just heard about them recently and was similarly excited by the potential.</comment></comment>


An appeal to all package managers - <AUTHOR> http://nelm.io/blog/2012/05/an-appeal-to-all-package-managers/

<comment><USER2> This is a great idea. A protocol/standard to handle front-end asset packaging. There will likely be suggestions to use existing tool X or Y; however, a non- language specific standard (as this article suggests) would be the most useful to all.

Excellent proposal.</comment>


Ask HN: Open Source Self-Hosted FaaS Software - <AUTHOR> Can anyone recommend a well-regarded open source FaaS solution that I can host myself on Linux? I've ran into some limitations with AWS Lambda (primarily IPv6 support) [1][2] so I'm looking for alternatives and I want to break away from AWS a bit as well, so this would be a good experiment. The ability to schedule the functions to execute periodically (minute, hourly, daily, etc.) and to run tens of thousands (or more) would be required. I primarily use Go and some Python 3 functions in Lambda.<p>Thanks for any feedback.<p>[1] - https://forums.aws.amazon.com/thread.jspa?messageID=771807<p>[2] - https://stackoverflow.com/questions/41694477/how-to-call-a-ipv6-rest-service-over-aws-lambda <comment><USER2> I haven't used it, but [https://openwhisk.apache.org](https://openwhisk.apache.org) seemed the most promising when I was looking for a self-hosted lambda equivalent.

Not due to any limitations of lambda, but just for personal projects where I already have a server at home anyway and it doesn't make sense to pay extra.

(In the end I just went with Docker and Jenkins because I didn't have anything complex enough to justify something heavier)

</comment><comment><USER3> [https://www.openfaas.com/](https://www.openfaas.com/)</comment>


Apple Used the DMCA to Take Down a Tweet Containing an iPhone Encryption Key - <AUTHOR> https://www.vice.com/en_us/article/pkeeay/apple-dmca-take-down-tweet-containing-an-iphone-encryption-key <comment><USER2> >Kilpatrick Townsend & Stockton

Same guys wanted a Youtuber to remove a video where he shows location of a fuse on macbook motherboard.

</comment><comment><USER3> I wonder why they backtracked and asked for the Tweet to be restored. I don't think key material is copyrightable. Besides, once it's posted the cats out of the bag.

I'd rather see Apple continue to strengthen iOS security and leave the legal bs for other problems.

<comment><USER4> They probably backtracked because Sony attempted the same thing in 2007 and in addition to being futile it actually incited a revolt on a very popular Reddit peer from that era.

[https://en.wikipedia.org/wiki/AACS_encryption_key_controvers...](https://en.wikipedia.org/wiki/AACS_encryption_key_controversy)

> At one point, Digg's "entire homepage was covered with links to the HD-DVD > code or anti-Digg references."

[https://i.kinja-img.com/gawker-media/image/upload/s-- 4VwaveO...](https://i.kinja-img.com/gawker-media/image/upload/s-- 4VwaveOK--/c_fill,fl_progressive,g_center,h_900,q_80,w_1600/18rbvd4atqu8cjpg.jpg)</comment></comment>


Bitcoin: The real great emancipator - <AUTHOR> https://medium.com/@jwweatherman/bitcoin-the-end-of-human-slavery-664ad39aee89 <comment><AUTHOR> This is part 1 in my series, Bitcoin, the real Great Emancipator.

The series is a survey of the major types of human slavery, and how Bitcoin disrupts these repulsive institutions. In Part 1 we will establish a solid definition for slavery</comment>


Ask HN: Interview questions for identifying employer red flags? - <AUTHOR> <comment><USER2> Asking about working hours is likely to provide some useful information. e.g. "How often do people stay late at the office?" or "When was the last time that you had to stay late to fix something?".

Employers may not be completely honest with their answers, but if they value work-life balance, then the answer will probably reflect this.

</comment><comment><USER3> In the last unsuccessful interview I went through, the interviewer mentioned something about "being a family" or similar.

I have major issues with this statement because it conveys a lot of unvoiced & unreasonable expectations or workplace dysfunctionality. It also seems that a lot of folks become really defensive when asked to explain what they mean by this and they try to pass it off as some sort of common decency and camaraderie when what they really mean is tolerating a lot of unprofessional conduct, extending a lot of undeserved and unreciprocated forgiveness, and forgoing duly deserved and legally owed recompense.

I am not a sex worker. Nor am I any sort of phycologist or psychiatrist. I also do not usually earn those sorts of wages. So I am absolutely 100% unwilling to pretend that anyone at work is related to me or engage in any of those things I described above.

</comment><comment><USER4> Why did the person who was previously in my role leave? What's the average time someone is employed here? How many employees have kids? How old are the other employees? What's the usual vacation time someone actually takes here?

I probably wouldn't ask all of these, but a cross-section of them usually gives me an indication of how much churn they're having, the type of employees they have, and what the work style is

</comment><comment><AUTHOR> e.g. at a previous employer, asking "have you written your own in-house Perl ORM and Object Model?" would have saved me a lot of mental anguish and pain.</comment>