How is GNU `yes` so fast? - <AUTHOR> https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes_so_fast/?st=j3v3iw3c&sh=5651ea3c <comment><USER3> Speed vs readability:

[https://github.com/coreutils/coreutils/blob/master/src/yes.c](https://github.com/coreutils/coreutils/blob/master/src/yes.c)

[https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c](https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c)

<comment><USER4> One thing to keep in mind when looking at GNU programs is that they're often intentionally written in an odd style to remove all questions of Unix copyright infringement at the time that they were written.

The long-standing advice when writing GNU utilities used to be that if the program you were replacing was optimized for minimizing CPU use, write yours to minimize memory use, or vice-versa. Or in this case, if the program was optimized for simplicity, optimize for throughput.

It would have been very easy for the nascent GNU project to unintentionally produce a line-by-line equivalent of BSD yes.c, which would have potentially landed them in the 80/90s equivalent of the Google v.s. Oracle case.

<comment><USER3> This is the first revision:

[https://github.com/coreutils/coreutils/blob/ccbd1d7dc5189f46...](https://github.com/coreutils/coreutils/blob/ccbd1d7dc5189f4637468a8136f672e60ee0e531/src/yes.c)

<comment><USER5> And here is the madness: :)

[https://github.com/coreutils/coreutils/commit/35217221c211f3...](https://github.com/coreutils/coreutils/commit/35217221c211f3116f374f305654462195aa634a)

<comment><USER6> And the motivation: it's used as test inputs. Not sure I agree with completely destroying the readability of a perfectly readable file - but ok.

<comment><USER7> The motivation, as far as I see, was only that it "may be used."

But see the post of belorn where he argues that the error handling seems to be good.

</comment></comment></comment></comment></comment></comment><comment><USER8> The /r/programming discussion of this is interesting [1].

Someone does a Go version and gets the same speed as GNU yes. Someone else tries several languages. This person got the same speed in luajit, and faster in m4 and php. Ruby and perl about 10% slower, python2 about 10% slower still, and python3 about half that. The code is given for all of these, and subsequent comments improved python3 about 50% from his results, but still not up to python2.

[1] [https://www.reddit.com/r/programming/comments/6gxf02/how_is_...](https://www.reddit.com/r/programming/comments/6gxf02/how_is_gnus_yes_so_fast_xpost_runix/)

<comment><USER9> I had to smile, as this thread is a microcosm of programming language stereotypes: with the python programmers tweaking code to get that extra 10% (nobody mentioned pypy..), someone trying to get javascript to work in a long running program, of course a rust implementation that isn't working well just yet (but we're all rooting for it.. set that compiler flag and rerun).. One comment about perl (turns into a one liner..). Nobody bothering to redo the Lua, Ruby and PHP. And for fun somebody throws down some fortran.

what is going on with the m4 code?

<comment><USER10> One night for fun, I wrote a fizzbuzz (without actual printing) in almost all of the languages mentionned here (except fortran), and benchmarked it with 10000 executions each.

<comment><USER11> Pretty sure none of your implementations will beat the FizzBuzzEnterpriseEdition [1].

[1]: [https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...](https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition)

<comment><USER10> That link is always marked as already visited and yet I always forget it exists.

</comment></comment></comment></comment></comment><comment><USER12> The recent commit that sped up GNU yes has a summary of the perf measurements

[https://github.com/coreutils/coreutils/commit/3521722](https://github.com/coreutils/coreutils/commit/3521722)

</comment><comment><USER13> If anyone, like me, is wondering what "yes" is used for. You can use to pipe "y" to commands that require interactivity, so if you just want to say "y" to all the inputs, you can use "yes" to do this:

      yes | rm -r large_directory

      yes | fsck /dev/foo

<comment><USER14> I know its only an example but why

       yes | rm -r <>

and not

        rm -rf <>

<comment><USER15> I think rm -rf would also suppress errors in addition to confirmation messages, so presumably if you want a "soft" force...

<comment><USER16> No it doesn't.

      > rm -rf foo/bar
      rm: foo/bar: Permission denied

<comment><USER17> If foo/bar does not exist, it should be silent. Via [http://pubs.opengroup.org/onlinepubs/007904975/utilities/rm....](http://pubs.opengroup.org/onlinepubs/007904975/utilities/rm.html) on the -f option:

> [...] Do not write diagnostic messages or modify the exit status in the case > of nonexistent operands. [...]

<comment><USER16> Yes, but that's the only error it suppresses, and that's because "not existing" isn't really an error for a tool that's trying to delete something. In my example foo/bar existed but I didn't have permissions to delete it, and as you'll see it printed out that error.

</comment></comment></comment></comment></comment></comment><comment><USER18> Back when I worked at the Genius Bar at Apple Stores I saw a customer come in and talk to a 'Genius' about their MacBook being "slow". After a quick bit of troubleshooting, he just opened up 4 terminal windows an ran yes in all of them, and did some hand wavy explanation about diagnostics.

<comment><USER19> I'm curious why he did it. To impress people who work at the apple store?

OK, may be I misunderstood, who ran yes, the customer or the genius?

<comment><USER20> I just read this on Wikipedia ([https://en.wikipedia.org/wiki/Yes_(Unix)#Uses](https://en.wikipedia.org/wiki/Yes_\(Unix\)#Uses)):

> In 2006, the yes command received publicity for being a means to test > whether or not a user's MacBook is affected by the Intermittent Shutdown > Syndrome. By running the yes command twice via Terminal under Mac OS X, > users were able to max out their computer's CPU, and thus see if the failure > was heat related.

<comment><USER21> Now if I could just prove to Apple that my computer randomly shuts down all the time... problem is, it doesn't appear to be heat related.

<comment><USER22> Mine just freezes, the last 0.3 second of sound is just repeated until it reboots by itself.

<comment><USER23> For reference, I had that exact same issue on my MBA - went on for years. Many times while watching youtube or doing something Garageband related.

At the time I had 10.11 on it, but that OS had itself been upgraded from 10.10.

When 10.12 came out, I decided to install fresh - so I backed everything up and installed onto a new SSD drive. The issue has now gone away completely....

</comment></comment></comment></comment></comment></comment><comment><USER24> _the limit isn 't the processor, it's how fast memory is. With DDR3-1600, it should be 11.97 GiB/s (12.8 GB/s)_

I don't understand this reasoning. Why is it being limited to main memory speed? Surely the yes program, the fragments of the OS being used, and the program reading the data, all fit within the L2 cache?

<comment><USER25> In general, only the CPU itself sees the L2 cache. Anything you see on another device (screen, disk, NIC etc) has been flushed out of cache.

<comment><USER24> Sure, but this is a pipe between two tiny processes, hopefully with very little else being run on the computer at the time (otherwise all bets are off for any benchmarking). There's no kind of 'real' I/O going on (in terms of stuff like screen, disk, NIC, and so on)

There's no reason that the L2 cache _needed_ to be flushed at any point - <USER2> caches are all dealing with physical memory, rather than virtualised address space, so the fact that there are two processes here shouldn't stop the caching from working.

<comment><USER25> I am probably way behind the current state of the CPU judging by the downvotes I got so if you are saying there is no reason and the data can be written into a device without leaving the CPU I will just concede my ignorance.

<comment><USER24> Don't fret about the downvotes, these magic internet points aren't redeemable anywhere :)

It's completely possible for the data to not (all) leave the CPU. If the caches are large enough, then the pages full of "y\n" will be still resident in the cache when the next iteration of the program overwrites the same pages again. Then the CPU has no need to send the original page out to main memory.

<comment><USER25> If you did for(;;) buffer[(i++)%size] = 'y'; then you'd be correct. However, you do i/o. And the 'y's appear at the i/o driver and have to be made visible for the device it's driving, which can be anything, including a process on another CPU core in another socket. If they remained in the issuing CPU cache, I fail to see how the destination device could possibly see them. There are some devices which can snoop the cache (like SoC GPUs and other CPU sockets on some archs) but the snooping is much slower than memory bus. Writing to memory is the only way which a) guarantees the data is available elsewhere and b) is the fastest.

</comment></comment></comment></comment></comment></comment><comment><USER26> You could make "yes" faster with the tee() syscall. Keep duplicating data from the same fdin (doesn't actually copy) and it becomes entirely zero-copy.

<comment><USER27> Actually yes should use vmsplice().

And pv on the other side of the pipe should use splice().

Now that would be a complete zero-copy I/O path, purely limited by the CPU, not by memory bandwidth. It would benchmark at hundreds of GB/s :)

<comment><USER28> Someone posted an implementation which uses vmsplice on the reddit thread:

[https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...](https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes_so_fast/diu73y0/)

<comment><USER27> His code can further be optimized by having only 1 large element in the iovec (vs. many 2-byte "y\n" elements).

<comment><USER29> from what i can see it's already in a large iovec buffer

<comment><USER27> Correct, I misread the code. However bumping the pipe size buffer bumps the speed from 28 to 74 GB/s on my Skylake CPU (i5-6500).

Edit: I optimized further and now get 123 GB/s See [https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...](https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes_so_fast/diua761/)

<comment><USER28> next step is playing tricks with memory mapping so that the buffer is large in virtual memory but only takes a single page in physical memory, fitting in L1 (at least on the read side).

</comment></comment></comment></comment></comment></comment></comment><comment><USER30> FreeBSD's yes has just been updated because of this.

[https://github.com/freebsd/freebsd/commit/1d61762ca37c20ab6f...](https://github.com/freebsd/freebsd/commit/1d61762ca37c20ab6fa4f5de6e179fa530ec49f2)

It's about twice as fast as GNU yes now on my FreeBSD system here.

<comment><USER12> Looks like that may drop some data if you get a short write, possible when writing to pipes etc.

Update: They fixed that issue with this follow up [https://github.com/freebsd/freebsd/commit/2592fbb8](https://github.com/freebsd/freebsd/commit/2592fbb8)

</comment></comment><comment><USER31> Let's not forget the most crossplatformest, purest `yes` of them all: [https://www.npmjs.com/package/yes](https://www.npmjs.com/package/yes)

        # /usr/local/bin/yes | pv > /dev/null
        11.5MiB 0:00:09 [1.02MiB/s] [                             <=>]

        # /usr/bin/yes | pv > /dev/null
        1.07GiB 0:00:09 [ 142MiB/s] [                             <=>]

JavaScript wins again!!

<comment><USER31> It's come to my attention that lower numbers are not better here. I have filed a bug, we'll get to the bottom of this shortly. I want to apologize to all our users, this issue does not reflect the values and principals we at Pure JavaScript `yes` hold dear [https://github.com/Sequoia/yes/issues/3](https://github.com/Sequoia/yes/issues/3)

</comment></comment><comment><USER32> I was not going to post this because hacker news has this ethic (?) of down voting anything that seen as not positive. Perhaps we should have discussion about that, I'm not sure that's a good thing but I'm not in charge here.

The top comment is:

"It's a shame they didn't finish their kernel, but at least they got yes working at 10GiB/s."

which as an OS guy, someone who has been working on Unix for 30+ years, as a guy who was friends with one the QNX kernel guys (they had perhaps the only widely used microkernel that actually delivered), that's hugely amusing and spot on. The GNU guys never really stepped up to being kernel people. Bitch at me all you want, they didn't get there. It's a funny comment, especially coming from reddit.

<comment><USER33> > hacker news has this ethic (?) of down voting anything that seen as not > positive

We must not be reading the same Hacker News...

Anyway, the comment you're quoting is just a shallow jab that belittles the GNU developers' work without contributing anything new or meaningful. It's telling that you had to spend two paragraphs to justify cross-posting it here.

<comment><AUTHOR> You say that and then immediately become an example of what hes talking about. This "shallow jab that contributes nothing new or meaningful" is, in some circles, known as a "joke." I'm continually frustrated by people who think that misinterpreting comments as harmful is a useful activity.

<comment><USER33> Not really -- I didn't downvote that comment. And I still dispute the notion that negativity is rare or always shunned on this board: to the contrary, it's so commonplace that an actual rule [0] had to be added to try to sway things in the other direction.

Jokes have their place, but bringing up the failure of Hurd in every GNU- related post is banal. And saying they "never really stepped up" to your level as a mighty kernel developer, as if the people who brought us glibc and coreutils lack an understanding of OS internals, just seems rude and curiously out of touch.

[0] [https://news.ycombinator.com/item?id=9317916](https://news.ycombinator.com/item?id=9317916)

<comment><USER32> So negativity here is not rare, when people don't like something they are fast to jump on it.

What I was trying to get at is this, if you care about your upvotes, hacker news promotes a sort of hive mind. Which is somewhat like "say only nice things unless you are clearly swatting down something that is obviously wrong".

Which is mostly fine, fantastic in fact. I'm fine with it, hacker news is really pleasant because of the (more or less _) quality of the posts and especially the quality of the comments. I 'd much rather have it be this way than a free for all, those go bad pretty fast.

So I'm for the hive mind, I was just pointing out that you can't make jokes and be upvoted. The joke wasn't banal at all IMO. GNU has done a lot of good, I've been there since the beginning and paid attention along the way. They have also been pretty self serving with their choices, every project has to sign away their rights and then GNU takes full credit for the project even though they had nothing to do with it other than it being GNU troff for example. Given their tendency to take credit for stuff that they didn't do, and their claim that they can do an OS but clearly can't, that joke is funny as heck. If you don't get that, sorry, you haven't been paying attention.

(_) The quality of the posts in the area of programming, especially systems programming, is spotty. Some stuff is great, stuff I didn't know (there is a lot of that here and I'm very grateful for it, it's why I stick around), some stuff is meh, and then there is stuff like "wow, look at $OBVIOUS, isn't that cool?" that gets upvoted. That last one I just don't get, but whatever, the good stuff is good. The signal/noise ratio here is better than any other programmer oriented site I've found.

</comment></comment></comment></comment></comment><comment><USER34> Years ago I read a similar experiment about max. CPU data flow. Guy was testing how much data can his CPU pass in a second. He was writing it in C, using some Linux optimization, optimizing code for CPU caches, using some magical C vectors that are optimized for such purpose. He got some help from someone working at Google. I tried to find that post but never succeeded. Does anyone here know it?

</comment><comment><USER35> `yes` (with the backticks) is my favorite "bring the system to its knees right now" shell command.

<comment><USER36> What does `yes` try to do?

<comment><USER37> This will attempt to open a child shell process with whatever output of "yes" is, interpreted as shell script.

But before that, parent shell has to buffer until EOF. With "yes" output being unbound, this means unbound buffering / memory growth. That is, until OOM killer take notice and shut it down.

Whole thing will probably take few seconds to a minute (depending on how much free RAM there is vs. how fast it is), peg a single CPU core in the process, and will recover cleanly (except for shell in question being terminated).

...unless system in question has single CPU and large+slow swap. Then yes, bring-it-to-its-knees.

<comment><USER38> It may also push everything in memory to swap in the process, what is the real speed killer.

<comment><USER39> Make sure to have a sane memory limit set in PAM and switch vm.swappiness to lower value than 60.

</comment></comment></comment></comment></comment><comment><USER40> And the question is, do we need yes to be so optimized?

Not complaining, I like this kind of analysis

But it seems you won't be limited, in a shell script, by the speed you can push y's

<comment><USER41> That kind of question doesn't make sense for open source code.

Somebody wanted to optimize 'yes', so they did. There doesn't need to be a good reason, just like there doesn't need to be a good reason for a person to read a certain book or watch a certain movie, other than they want to do it.

<comment><USER40> The question was "do we need it? (for a specific use case)" rather than "should we do it?".

And it does make sense for Open Source code because the resources are limited hence other features and/or bug fixes might be more important than pushing data at full speed

<comment><USER41> > And it does make sense for Open Source code because the resources are > limited hence other features and/or bug fixes might be more important than > pushing data at full speed

That's not how open source works, though. There's not a group of people obligated to work on the GNU utilities. There's not a central project manager ordering people around telling them what changes to make.

Declaring that 'yes' is generally fast enough already doesn't imply that it was fast enough for the person who spent their time optimizing it. Somebody needed (or just wanted) 'yes' to be really fast, so they did the work and submitted the changes back.

<comment><USER40> > There's not a central project manager ordering people around telling them > what changes to make.

No, but there is review and approval of patches (also the most frequent contributors know where the project is going and there is bug tracking)

If one just makes it faster without major benefits that patch is likely to be rejected

<comment><USER41> > If one just makes it faster without major benefits that patch is likely to > be rejected

 _IF_ the project were overwhelmed with incoming patches, then I would agree that there's probably higher priority changes than optimizing 'yes'.

But I'm almost certain the GNU project is not being overwhelmed with changes to the core utilities. Everything else being equal (code quality, readability, test coverage, etc.) there's really no reason to reject a patch that demonstrably improves the project, even in some silly way like making 'yes' really fast.

</comment></comment></comment></comment></comment></comment><comment><USER42> I would just pre-allocate a static array of "y\n" of size BUFSIZ, write it out in a loop, and call it for the day, skipping the whole malloc and filling loop business.

Make the static array BUFSIZ * 1024 to trim the syscalls by a factor of 1000.

<comment><USER43> Real yes accepts a string to print (so you can have it spit out a full "yes" or "Y" rather than a hardcoded "y").

<comment><USER42> Have a few pre-written arrays for the common cases: "y", "Y", "n", "N", etc. Those are the fast cases (or the benchmark optimized cases, like, what Volkswagen did). Have another pre-allocated static array to fill in with other input.

<comment><USER44> > _(or the benchmark optimized cases, like, what Volkswagen did)_

This is amazing. Maybe we can turn this into a verb? "I Volkswagened the common cases with precalced buffers".

<comment><USER45> That's how you get sued. Especially because now Ford and GM are also in court, for having done the same as VW.

<comment><USER46> Let me check that I have correctly understood the scenario you're anticipating.

1\. Someone uses "volkswagen" as a verb meaning "cheat in benchmarks".

2\. Volkswagen takes them to court, on the basis that it is slanderous or libellous to associate Volkswagen with cheating in benchmarks.

3\. Counsel for the defence reminds the court that Volkswagen were in the news for a protracted period for a benchmark-rigging that saw them hit with a multi-billion-dollar fine, slashed a third off their stock price, saw their CEO resign, led to there being a "Volkswagen emissions scandal" page on Wikipedia, etc., etc., etc.

4\. [THIS IS THE BIT I'D LIKE YOU TO FILL IN FOR ME.]

5\. Volkswagen wins the case.

How exactly does step 4 go?

<comment><USER45> By volkswagen arguing that this is industry standard behaviour, and the entire media campaign is already libel and slander.

And they'd have a pretty good case with that, considering the original report all this media campaign was based on called out 6 carmakers, but the media (and the poster) only call out Volkswagen.

</comment></comment></comment></comment></comment></comment></comment><comment><USER47> Why is it so slow(compared to the post) in the macbook air. Native yes runs at 26 MiB/s, and GNU yes at 620 MiB/s.

<comment><USER48> It isn't! yes runs at 7.2 GiB/s on my macbook air. Though I have Linux installed on it instead of Mac OS :)

<comment><USER49> Are you trolling? The commenter obviously meant the macOS version of "yes".

</comment></comment></comment><comment><USER50> With that malloc overhead, I expect GNU yes to be slower when only a few bytes are read from it.

So, what's the distribution of #bytes read for runs of 'yes'? If we know that, is GNU 'yes' really faster than the simpler BSD versions?

Also, assuming this exercise still is somewhat worhtwhile, could startup time be decreased by creating a static buffer with a few thousand copies of "y\n"? What effect does that have on the size of the binary? I suspect it wouldn't get up much given that you can lose dynamic linking information (that may mean having to make a direct syscall, too).

<comment><USER51> Unless you are statically linking, one malloc doesn't significantly affect your startup time. When only a few y's are read, time is going to be dominated by ld.so, by a large margin.

</comment></comment><comment><USER52> Measurements are really noisy, but I seem to get significantly better numbers than that when I use fsplice() on a pre-generated few pages of file data instead.

<comment><USER39> Yes, splice can bypass the pipe buffer in some cases.

</comment></comment><comment><USER53> I thought this was a fascinating read but it left a serious question lingering in my mind, which is a little out-of-scope for the article, but I hope someone here can address.

Why did the GNU developers go to such lengths to optimize the yes program? It's a tiny, simple shell utility that is mostly used for allowing developers to lazily "y" there way through confirm prompts thrown out by other shell scripts.

is this a case of optimization "horniness" (for lack of a better word) taken to its most absurd extreme, or is there some use case where making the yes program very fast is actually important?

<comment><USER54> The stated use case for the perf improvement was "yes(1) may be used to generate repeating patterns of text for test inputs etc., so adjust to be more efficient."

Source: [https://github.com/coreutils/coreutils/commit/35217221c211f3...](https://github.com/coreutils/coreutils/commit/35217221c211f3116f374f305654462195aa634a)

I've personally used it for generating repeating text and filling disks in system testing, so I appreciate it being faster at those tasks. I also sometimes use it as a signal generator for a hacky load generator, like so:

      yes | xargs -L 1 -P NUM_PROCESSES -I {} curl SOME_TARGET_URL > /dev/null

This doesn't benefit from being _faster_ per se, but I appreciate it using less CPU since I want to give curl as much system resources as possible.

</comment></comment><comment><USER55> But doesn't this make the typical use case (just a few "yes"s needed) slower, since first it has to fill a buffer?

I would write() the buffer each time it gets enlarged, in order to improve startup speed.

Also: The reddit program has a bug if the size of the buffer is not a multiple of the input text size.

And it's increasing the buffer by incrementing one at a time, instead of copying the buffer to itself, reducing the number of loops needed (at cost of slightly more complicated math).

<comment><USER56> If your only overhead is filling an 8K buffer, I don't think your user is going to care. Taking one microsecond instead of one nanosecond doesn't matter all that much when you're going to lose way more than that in pipes, the kernel, the program you're piping it to, etc.

<comment><USER57> But what's the use case for a large volume of continuous output? It feels like we're optimizing for the wrong use case

<comment><USER58> Maybe filling a disk or flooding a network connection as in

        yes | ssh server "cat > /dev/null"

But yes take arguments so there might be more use cases:

        $ man yes
        NAME
               yes - output a string repeatedly until killed

        SYNOPSIS
               yes [STRING]...
               yes OPTION

        DESCRIPTION
               Repeatedly output a line with all specified STRING(s), or 'y'.

</comment></comment></comment></comment><comment><USER59> GNU yes is fast because it is coded with the assumption that it's not answering any real question, such as _" can I combine this free code with a proprietary program?"_ or _" Would you accept the following monstrous patch to GNU Coreutils /bin/true without a copyright assignment?"_

</comment><comment><USER60> I think we can do better.

How about a /proc/bin/yes for this? Like most /proc files, it would appear to be empty. Executing it would involve a fs/binfmt_proc.c file in the kernel source, which would be a handler for this sort of executable. That would get the job done entirely in the kernel.

</comment><comment><USER61> man, I just spent like 8 minutes today writing a python script to use up all the disk space on some servers (part of ops readiness testing) when I could have just used this trick.

`yes` will help me on the "see what happens when something uses all the CPU and memory" test case. Thanks Reddit/HN!

<comment><USER62> dd can also be useful for that kind of thing since you can use a source like /dev/urandom to generate random bytes if you're trying to avoid compression and the adjustable block sizes can be optimized for the underlying storage system.

<comment><USER61> I did actually use dd to begin with but it didn't work as I wanted it too right away (can't remember why, probably something SUSE related) and by then I'd used up 6 mins of the 15 mins I gave myself for the job.

I also had to do the same test on windows, so python won

</comment></comment></comment><comment><USER63> Did you notice PHP outperforms any other scripting languages? Some report that it event beats the GNU yes implementation.

After reading here so many unfair critics and pedantic dislike over PHP[1][2][3][4][5][6], I just want to say: STFU.

[1] [https://news.ycombinator.com/item?id=12706136](https://news.ycombinator.com/item?id=12706136)

[2] [https://news.ycombinator.com/item?id=3825227](https://news.ycombinator.com/item?id=3825227)

[3] [https://news.ycombinator.com/item?id=3824881](https://news.ycombinator.com/item?id=3824881)

[4] [https://news.ycombinator.com/item?id=1823022](https://news.ycombinator.com/item?id=1823022)

[5] [https://news.ycombinator.com/item?id=1819517](https://news.ycombinator.com/item?id=1819517)

[6] [https://news.ycombinator.com/item?id=1819413](https://news.ycombinator.com/item?id=1819413)

... Just to name a few.

<comment><USER64> It's almost funny that you try to refute, or rather, dismiss, criticism of PHP with "but it's fast", when none of what you cite even mentions that. I just want to say: STFU.

<comment><USER63> I don't dismiss criticism. I dismiss unfair critics. What I cite is full of that. And, btw, [6] implies PHP is slow, read on:

[quote]

It's hard to find a worse example than php, because when many "bad" languages fail in one or the other category, php seems to fail in most categories.

* Java has a really repetitive syntax, but it's libraries might make up for that.

* C++ with all it's features has bazillion ways to shoot yourself, but it's arguably fast and the syntax is bearable. ...

[/quote]

<comment><USER46> It doesn't in fact imply that PHP is slow. What it says about PHP (besides "php seems to fail in most categories" which doesn't imply any single specific failure because most != all) is this: "But php has both an ugly syntax, horrendous stdlib, and fame of security issues." You will notice that none of those specific complaints is about speed.

</comment></comment></comment></comment><comment><USER65> I run the command `yes | pv > /dev/null` on my MacBook Pro, it's only 37m/s, is this normal? I am not familiar with the command.

<comment><USER66> From the OP:

>OS X just uses an old NetBSD version similar to OpenBSD's >NetBSD's is 139MiB/s, FreeBSD, OpenBSD, DragonFlyBSD have very similar code as NetBSD and are probably identical,

It really depends on how fast your memory and processor are. Everything there was benchmarked with an i7-4790 with DDR3-1600, and very very little running in the background.

<comment><USER67> Two orders of magnitude difference can't easily be blamed on CPU, in this case it depends on how optimized _yes_ is. I have some older version of _coreutils_ on one machine here and performance is similarly abysmal.

</comment></comment></comment><comment><USER68> Well now I know what `yes` does (And pv)

</comment><comment><USER69> Why is he using backticks to quote "yes" in the title?

<comment><USER70> It is common in markdown formatting to indicate a command line with backquotes. It tells the markdown compiler to apply special css and possibly syntax highlighting. So, a lot of people who write about shell commands and setup tutorials, or use markdown on something like github will automatically identify it as a command.

</comment></comment><comment><USER71> clearly, we just need /dev/yes

<comment><USER72> Sounds like a reasonable feature request for systemd.

</comment></comment><comment><USER73> yes | write <USERNAME> "Don't you hate dialup connections?"

<comment><USER74> I think you mean yes "Don't you hate dialup connections?" | write username

</comment></comment><comment><USER75> The proprietary Oracle Solaris 11.2 yes really slowed down when they added DRM and Verified Boot support...

</comment><comment><USER76> [https://m.xkcd.com/619/](https://m.xkcd.com/619/)

</comment><comment><USER77> tl;dr someone who doesn't understand how i/o works gets a small insight into how memory and a cpu work and decides _" Buffering is the secret"_ and _" You can't out-optimize your hardware"_

Can we have a new flag for posts by people who don't know what they're doing so I can skip them? I am serious.</comment>


Totally amazing hack: Use an elevator without stopping - <AUTHOR> http://garrysub.posterous.com/totally-amazing-hack-use-an-elevator-without <comment><USER2> I remember when this was a meme back in 2005 (<http://benmetcalfe.com/blog/2005/08/elevator-hacks/>)

I guess it's time for it to go round again for the next generation!

</comment><comment><USER3> Seems like only a few hours ago that <http://news.ycombinator.com/item?id=1883748> was asserting that the door- closing button in most elevators is a no-op.

</comment><comment><USER4> Coming from someone who spends a lot of time in Hotels... This hack works in _very_ few elevators.

</comment><comment><USER5> If this worked and everyone did it, elevator throughput would plummet and you would wait a lot longer when calling one. Tragedy of the commons: why we can't have nice things.

</comment><comment><USER6> because your couple of extra minutes are more important than everyone else's who may be waiting - same type of problem as the people who hold subway doors delaying hundreds for the sake of a few

</comment><comment><USER7> I'll definitely try this tomorrow morning at work. It would be great if it works on the way down as we're on the top floor!

<comment><USER8> I'm going to try this in about 5 minutes, hopefully I will have phone reception in the elevator to update if I get stuck.</comment></comment>


Major Proposed Changes to Linux From Scratch - <AUTHOR> http://lists.linuxfromscratch.org/pipermail/lfs-dev/2020-June/073815.html <comment><USER2> If you're not ready for the LFS plunge, install and configure an Arch Linux system with a graphical desktop environment. I learned more about Linux in that 8 (non-consecutive) hours than I had in the couple of years preceding.

Then study for the RHCSA and RHCE v7 (v8 is all Ansible now). Arch Linux and RHCSA and RHCE taught me a ton about Linux and made me super comfortable with my Linux desktops and servers.

<comment><USER3> [https://knowyourmeme.com/memes/install- gentoo](https://knowyourmeme.com/memes/install-gentoo)

Goes even deeper than arch. The desktop profiles make it almost set-and-forget now.

<comment> yjftsjthsd-h I'd say Arch, then Gentoo, then LFS. Arch provide you with usable binary packages and leaves you to know what to install and how to configure it. Gentoo makes you actually build those packages from source and gives you more control over how the process works. Then LFS takes away even what portage was doing for you and makes you do the build process yourself by hand from scratch.

<comment><USER4> It's kinda sad that no one recommends Slackware Linux anymore.

Back in the day there we used to say: "when you know slackware you know Linux. When you know red hat all you know is red hat".

Effectively when I got into slackware my computer wasn't 100% working all of the time but I learned a lot. I then moved on and years later I got rhcsa certified and that's very worth it too.

Not sure about arch, I never really used that, but I can vouch for slackware, Gentoo and the RedHat/centos/rhcsa thing.

<comment><USER5> Absolutely.

If you're looking to just learn about Linux, you can't go much further than Slackware. It's as Linux as you can get: you can learn distros (which admittedly is easier now that many are built on a few different platforms - GNOME, xfce, systemd, etc.), but Slackware gives you a kernel, core-utils, and a bunch of packages - can't get much simpler than that.

Also once you track -current and get around SlackBuilds you can install almost anything you want, it's a full-fledged and modern distro. Without dependency management to be sure, but if you've got a well-made distro and you compile most of the software you install, who needs it?

</comment></comment></comment></comment></comment><comment><USER6> I can't recommend Linux From Scratch enough. If you want to understand the composition of the Linux user space, this is one of the best exercises to follow.

<comment><USER7> i feel like linux from scratch is kind of overkill

1\. format a disk a bootable active partition with ext4

1\. download the kernel, make defconfig && make

1\. copy the kernel to the partition with syslinux mbr.bin and probably extlinux.conf (probably copy modules too)

1\. copy busybox over

1\. voila

<comment><USER8> Is this documented properly somewhere, so that I can give it a go?

I've tried and failed with LFS so many times, I can't count - I tend to just run out of time, and by the time i go back to it, I've totally lost where in the process I was.

<comment><USER7> I don't like giving my real name out online but I have a GitHub repo that does this _automatically_ as a CI/CD job and a single shell script. It's really neat. Reach out to me (my username @ gmail.com) and I'll share it with you/

</comment></comment></comment></comment><comment><USER9> This brings me back to my years of running Gentoo as my daily driver. All fun and games until you update your system over the weekend and spend the rest of the weekend fixing broken networking. But it is all worth it if your goal is learning the ins and outs of an OS and how it all fits together.

It was fun booting to a graphical desktop in a few seconds, opening activity monitor and seeing < 70mb memory used. This was with XFCE around 2010. Now it is 2020 and Task Manager on my daily Windows 10 machine shows 27gb memory used.

<comment><USER10> Every time I read comments about Gentoo it's either about systems unexpectedly getting broken after an update, which supposedly requires considerable amount of effort to fix, or I read about how extremely stable Gentoo systems are and require almost no maintenance.

It seems like there's some dissonance here and I'm not sure where this comes from.

<comment><USER11> I was a Gentoo user about 5 years ago. With Gentoo you can have both, if you only use stable packages you shouldn't broke your system. But if you are using bleeding edge packages thinks can break.

<comment><USER12> Current Gentoo user. I find the opposite. Using unstable, things just work. Sometimes package updates fail to compile but your system is still fine, and the package will get fixed eventually. Never had an update break my system on unstable. Stable often had issues, which I presume is due to devs mostly working on unstable.

</comment></comment></comment></comment><comment><USER13> LFS, or gentoo from a stage 1 tarball, is something that is extremely valuable to do.

Once. Once is usually enough.

<comment><USER14> I ran Gentoo in college for a couple of years. Oh god the compilation.

<comment><USER15> I ran Gentoo for about a week a long time ago. On a 400 MHz Powermac G4. By which I mean, I installed Gentoo once and then decided I didn't like it.

<comment><USER16> It still works for PowerPC Macs :)

</comment></comment></comment></comment><comment><USER17> I find Linux From Scratch a bit masochistic. Embedded Linux development is equally educational while actually being useful. There are good frameworks like Buildroot: [https://buildroot.org/](https://buildroot.org/)

<comment><USER18> I think a bit of masochism is the point. I did it once to fill in all of the gaps and cement all the dependencies when putting together a system. That kind of knowledge helps when you're trying to do server maintenance but try and keep the system up or do crazier things but be fairly confident certain parts won't be affected.

I can't see myself using a LFS system in production.

<comment><USER17> I ran Gentoo for a few years, and it was a great balance of being low level and customizable but practical for day to day use. When you have set things up from scratch, you can fix it when it breaks. The Gentoo install disk helped me recover many a server. I got tired of the endless compilation, though.

</comment></comment></comment><comment><USER19> After seeing this headline I was just thinking it would be a lot of fun to dive back into Linux 15 years after I was last a desktop Linux user. It would be fun to try Linux From Scratch, or Gentoo if that's still a thing. Then I realized I don't even own any normal PC hardware that I could do that with without some wrangling (I could probably technically get it working on my old Intel iMac or my Synology NAS, but that's not really the challenge I'm looking for). It's wild to think that 15-20 years ago I was obsessed with Linux and spending my free time trying to get graphics drivers and dual monitors to work, and now I barely even have any traditional PC hardware.

<comment><USER20> > I barely even have any traditional PC hardware

Can you elaborate?

Are you using a Macbook or something instead?

<comment><USER21> Probably tablets, phone, or something similar..?

</comment></comment></comment><comment><USER22> I have tried LFS before because I thought it would teach me more about Linux (been a desktop Linux user since 2006, having tried a number of distributions, including Arch, and more recently back on Ubuntu, but never dug deep into internals).

However, I found I didn't really learn much and was just blindly executing steps. It's fun for a while to watch the magic, but maybe there is some better way to consume the material? I found that I wasn't really getting enough context around what was happening to appreciate what was being done and why at every stage.

Maybe someone can suggest something?

<comment><USER23> Interesting - my experience was that I learned a lot from it not so much by blindly executing the steps, but by things not working the way they were supposed to. This forced me to dive into the details and understand what was going on. (This is probably 15 years ago though - maybe the instructions have gotten better :) )

</comment></comment><comment><USER24> What I retain from all of this...

I salute the fact he did work on the hint I gave on their "weirdly handled" irc chat on freenode... (you may be able to find some logs...). And they succeeded! I'm very happy that now the public is gonna have the clean way to produce an operating system with Gnu tools & Linux.

While I was very kind sharing with them they didn't appreciate I wasn't giving solution "directly" and clearly thrown stuff to me like I'm a noob speaking too much of thing I don't master... well, it was very surprising to me to end in this situation... I even asked bdubbs itself to support me with that... he thrown me that "he doesn't handle irc chat only book writing...". I was kind of meh I just left their chat...

The result of this weird context pushed me into this view:

THEY ARE BUILDING GNU/LINUX SINCE 20years! and 20 years later they are still waiting information to drops from sky... and then acts like pioneerrs...

This went far the edges limits! go in hell NOOBS!

</comment><comment><USER25> I'm 30 and I can't believe I didn't do this pilgrimage yet.

When will this be ready?

<comment><USER26> I'm almost 31 and I said the same thing to myself this weekend.

Ironically coincident with this announcement, I ended up going through LFS and managed to get a kernel compiled using an entirely bootstrapped toolchain.

About halfway through I noticed the development version they're discussing, which is available here: [http://www.linuxfromscratch.org/lfs/view/development/](http://www.linuxfromscratch.org/lfs/view/development/)

I believe this version is likely able to be run-through but please keep in mind it looks like it's been changing every few days!

To the point of other commentators mentioning it can get a little tedious, especially in chapter 6- I think the new revised edition they're working on could help with that somewhat as it looks like avoiding the /tools buildout will help minimize the number of packages that need to get rebuilt.

<comment><USER27> We can start a study group.

"Uncles doing LFS"

</comment></comment></comment><comment><USER28> Lots of comments so far about using Linux From Scratch to learn about the userspace/kernelspace interaction, and while that's all feasible you can do this on basically any distro.

This week I've been iteratively rebuilding the kernel and glibc and the vDSO on my Ubuntu Server 20.04 install and learning a ton about the interactions, while also having the Ubuntu apt repositories to fall back on for when I want to switch from "learning mode" to "just get this task done mode."

</comment><comment><USER29> We really need to fix GCC so that the libraries and compiler can be built separately (like Clang).

</comment><comment><USER24> Am glad to have taught them something before they thrown me out \o/

Tools to empower robbery... AND ACT LIKE PIONEER :D

I swear what you do... SINCERELY!

</comment><comment><USER30> Is there a reputable startup / company that employs Linux kernel developers?

<comment><USER31> Red Hat?

<comment><USER32> Google?</comment></comment></comment>


Complete Introduction to git - <AUTHOR> http://students.ceid.upatras.gr/~sxanth/ungit.html <comment><USER2> "One caveat in version control that often programmers fall into is trying too much to have a perfect versioning history. In this situation, the developer spends too much time trying to separate commits in semantically well described changes and attach too well written description messages."

I plead guilty!</comment>


The Importance of Realism in Startups - <AUTHOR> http://www.bothsidesofthetable.com/2013/08/10/the-importance-of-realism-in-startups

<comment><USER2> For those who prefer to watch at a faster speed, the link to the video: [http://avideos.5min.com/371/5178371/517837087_4.mp4](http://avideos.5min.com/371/5178371/517837087_4.mp4) (138Mb)

</comment><comment><USER3> Earlier discussion:

[https://news.ycombinator.com/item?id=6194136](https://news.ycombinator.com/item?id=6194136)</comment>


Rediscovering the Small Web - <AUTHOR> https://neustadt.fr/essays/the-small-web/ <comment><USER2> I enjoy reading this website. The color tune is very comfortable.

<comment><AUTHOR> Author here. Thank you. It's a bit of an experiment, glad you enjoy it.</comment></comment>


[Law] to require Brazilians to [use real name and SSN-like] to post [in the web] - <AUTHOR> https://translate.google.com/translate?sl=auto&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=http%3A%2F%2Folhardigital.uol.com.br%2Fnoticia%2Fdeputado-quer-obrigar-brasileiros-a-informar-o-cpf-para-postar-em-redes-sociais%2F50148&edit-text=&act=url

<comment><AUTHOR> I changed the title to give some context to international headers and stay within the 80 character limit, I hope it stills within the spirit of the rule concerning editorial changes. A literal translation would be "Representative to require Brazilians to give CPF to post in social networks", but the text makes clear the proposed law will affect any website and, in Brazil, CPF is something alike a SSN originally used for taxes but now required for a lot of things, including financial operations, boarding a plane (sometimes even an interstate bus!), opening bank accounts, getting a job, going to school, etc.

The Google-translation is pretty good for an machine translation, but follows a few fixes:

\---

Representative to require Brazilians to give CPF to post in social networks

The House of Representatives will discuss a bill that seeks to end anonymity on the Brazilian Internet.

Authored by Congressman Silvio Costa, from the Social Christian Party from Pernambuco, PL 1879/2015 wants to change the Internet Civil Mark to include an obligation to store the full name and CPF number of Internet users who want to publish something on the network.

Any site that allows exposure of ideas would have to store this information of its users. The obligation would be set by adding a paragraph to the Civil Mark as it would read: "Internet application provider, as defined in the head, which allows posting of public information by third parties, as in blog comments, forum posts, status updates on social networks, or any other method to insert information in the Internet, shall keep, in addition, records of these users including at least their full name and their Natural Person Record number (CPF)."

For the deputy Silvio Costa, "this simple requirement will certainly curb the attitudes of those who cowardly hide behind anonymity to disseminate criminal messages in the network. Also," he continues, "individuals who insist in this type of conduct will be more easily identified and properly prosecuted."

The project is still in the House for analysis from commissions of Science and Technology, Communication and Informatics, and Constitution and Justice and Citizenship.

<comment><USER2> > _For the deputy Silvio Costa, "this simple requirement will certainly curb > the attitudes of those who cowardly hide behind anonymity to disseminate > criminal messages in the network. ["]_

And also those who hide behind anonymity to disseminate legal, even helpful messages across the network. Having an online alias allows many shy, repressed, abused, and frightened individuals the opportunity to chat, interact, and discourse about their lives without fear this information will be used against them.

> _[ "]Also," he continues, "individuals who insist in this type of conduct > will be more easily identified and properly prosecuted."_

More of the "If you've got nothing to hide, then you've got nothing to worry about" attitude. This suspicion is so hostile to normal traffic that its chilling effect is undeniable. It's very sad to see.

<comment><AUTHOR> > More of the "If you've got nothing to hide, then you've got nothing to worry > about" attitude. This suspicion is so hostile to normal traffic that its > chilling effect is undeniable. It's very sad to see.

It's worse -- these CPF numbers have nothing special but a simple checksum at the end. It is trivial to generate them and, if we get a used one, check the actual owner from our IRS website.

So we can assume everyone with bad intentions will simply use one from anyone else and the law will have no effect on criminals __but __will terrorize people who need /want to keep their PII private. Also, I suppose everyone will infringe that law at some level, opening the door to selective prosecution of "undesirables" or someone opposing the powers that be.

<comment><USER2> So there's no requirement that the ISP verify you're using your own CPF? That's a hole in the system which is worthy of ridicule.

</comment></comment></comment></comment><comment><USER3> Heavy on the order and light on the progress</comment>


Rejected By Apple, iPhone Developers Go Underground - <AUTHOR> http://www.wired.com/gadgetlab/2009/08/cydia-app-store

<comment><USER2> Great to see Jay, who's been there since the very beginning of iPhone development, get some press here. From its humble beginnings when it was battling Installer for jailbreak directory king, it's cool to see how far Cydia (literally an apple eating moth) has come.

</comment><comment><USER3> IANAL and haven't looked hard into the issue at all, but surely the Jailbreak developers are in danger of being sued for some DMCA infringement?

(ref xbox mods for ppl who want to play Japanese games)

</comment><comment><USER4> But seriously, anyone know what theme that is (in the article photo)?

<comment><USER5> It looks like a hand-rolled combination of Deep and Illuminous, with some extra icons thrown in.</comment></comment>


New iPhones this June: 4" & 4.8" Californian series (iPhone 5S, iPhone Math) - <AUTHOR> http://translate.google.com/translate?hl=en&sl=auto&tl=en&u=http%3A%2F%2Fnews.chinatimes.com%2Ffocus%2F501012675%2F122013012100104.html

<comment><USER2> Honestly, how is a story from the China Times calling the next iPhone the "iPhone Math" considered a credible story?

</comment><comment><USER3> Is there a more accurate translation of iPhone "Math"? I don't see Cupertino naming a phone that.

<comment><USER4> The original Chinese actually uses English for the model names in this article, so we have nothing more to go on...

</comment></comment><comment><USER5> Maybe they are using a mattress designation, California King for the 4.8inch iPhab.

</comment><comment><AUTHOR> iPhone Math: designed specifically for GOP from Cupertino. Upcoming: iPhone Biology (Creationists); iPhone History (truthers).</comment>


Hello Firefox! Mozilla's browser gets built-in WebRTC video chat - <AUTHOR> https://gigaom.com/2014/10/16/hello-firefox-mozillas-browser-gets-built-in-webrtc-video-chat-through-telefonica-partnership/

<comment><USER2> This project is known as Loop: [https://wiki.mozilla.org/Loop/Try_Loop](https://wiki.mozilla.org/Loop/Try_Loop)

a free full-stack WebRTC solution, with backend servers, clients, API.

* Mozilla provides the OAuth 2.0 authentication, the signaling server, and the clients

* Tokbox (Telefonica) provides STUN/TURN server [http://tokbox.com](http://tokbox.com)

* Nexmo provides SMS service / MSISDN gateway [http://nexmo.com](http://nexmo.com)

<comment><USER3> Thanks, your mozilla wiki link answers my group video calling question: Loop / Hello is primarily meant for 1-to-1 video calling.

Although this is an awesome development, group video would have made it a killer application.

(for now, you can use [https://opentokrtc.com/](https://opentokrtc.com/) to try out OpenTok's group video calling support)</comment></comment>


Ask HN: New passwords - <AUTHOR>

I just got an email from "firstname.lastname@example.com" giving me a new password with no other explanation. What's this all about? <comment><USER2> You perhaps used clickpass to login. PG was discussing removing it here <http://hackerne.ws/item?id=3085168>, and if you depended on it, it would make sense to email you a password so you can log in.</comment>


Opportunistic encryption and authentication methods - <AUTHOR> https://moderncrypto.org/mail-archive/messaging/2014/000767.html

<comment><USER2> I wonder how much

> A small number of users performing "stealthy" authentication could protect > other users by creating uncertainty about which connections can be > undetectably attacked

is worth in our post-Snowden world. We already know that the governments of the US, China, Syria, etc, are carrying out pervasive surveillance. They could just start doing MITM attacks against _every_ connection, and we'd be in the same situation as we were before implementing opportunistic encryption?

<comment><USER3> That would be expensive. The goal is to generate asymmetric costs.

<comment><USER4> Be careful what you wish for. NSA's biggest policy goal isn't total awareness of all information on the Internet. It's _maximizing their budget_.

</comment></comment></comment><comment><USER5> Wouldn't protect against passive eavesdropping though would it? They'd just mim everything

<comment><USER6> A passive attack is an attack that can be done without interfering with the signal.

A MITM attack requires you to hook into the signal and interfere with it, replacing Alice's and Bob's keys with your own and decrypting/re-encrypting the communication to keep the illusion up.

So a MITM attack is not a passive attack, but an active one.

<comment><USER5> Hmmm. I don't see the distinction. They are collecting a ton of data by a small army of corrupt proxy servers. Without authentication it's just like a new protocol. And hardly a challenge when we know they are already doing it on ssl connections.

<comment><USER7> You don't see a cost (an/or effort) difference between deploying an army of trivially simple packet sniffers and and deploying an army of of _stateful_ (RAM requirement), _fast_ (to deep-packet-inspect in realtime) MitM proxies? That notable difference in cost makes the endeavor worthwhile on it's own, as long you remember what it buys you (still need real authentication).

A better comparison - which is often ignored in these discussions - is that the test for something being "useful" doesn't rely only on if it solves _every_ aspect of the problem. A method should also be compared to the _current_ situation. For encryption, way too often the current situation is "still using plaintext". In that situation, even _rot13_ is an improvement because it require an attacker to take at least some minimal step.

Sometimes you can't solve every part of the problem in one step. Adding an authentication feature later to improve something that already exists is probably easier than waiting for a perfect, complete product to happen.

Historically, the crypto community has taken a hard line approach that any weakness shouldn't be allowed, because they correctly deduce that an attacker can use any weak link to break past all your security efforts. The result is we got pgp and everybody still using plaintext. The "all or nothing" attitude lead to web browsers complaining about self-signed certs when they should be _automatically_ generated by Apache by default with HTTP retired permanently. Instead, browsers conflate authentication with encryption and scare people back to plaintext.

Even if hardware gets so cheap that is zero cost difference between passive eavesdropping and running a MitM proxy or deep-packet {inspection,injection}, non-authenticated crypto is _still_ worthwhile: it creates a culture where encryption is normal instead of being seen as suspicious, and it can be an educational or traditional tool.

//of course, if we _can_ get authentication as well, that would usually be a better option. Even if we have to wait for "v2.0" to get it

<comment><USER5> It's not that I disagree, so much as I consider dropping auth completely to be pretty much a complete waste of time - it solves virtually none if not none of the problems.

What would be a minimum imho is a "persistence" authentication.

i.e. I don't need to know that Anonymous123 is really joe bloggs of Boston. What I do need to be reasonably sure of is that Anonymous123 is the same Anonymous123 that I was talking to last week.

I don't even really care if someone sits and listens in to the conversation - the weak link in encryption imho is that it isn't anonymous.

And in that context - tor solves most of these problems already.

<comment><USER8> > it solves virtually none [of the problems]

Consider this: in today's HTTP, the communication between the user and the server is for everyone to see. With HTTPS, the communication is private between the user and _a_ server. I believe only the NSA has accurate numbers, but I guess a whole lot of the servers users are talking to are the correct one.

Authentication comes _on top of that_ so we can predict this number with more accuracy. But you may not need to; maybe your conversation really is public, maybe you don't care. The thing is that we actually _can_ make it so that everyone can be happy with that: those who expect a minimum of privacy and those who don't.

We can provide encryption; it's easy and cheap. It doesn't mean the communication is 100% secure (it never is), but it means it's fare better than what we have today. We must do it. Again, it doesn't mean communications will be secure, and browsers should certainly not display unauthenticated encrypted communications with the reassuring green lock; it means they should show _all_ HTTP connections as completely insecure and unauthenticated HTTPS connections as mildly secure.

By the way, your model is called Trust On First Use or TOFU [0] and is already used in some crypto domains, most notably ssh.

[0] [https://en.wikipedia.org/wiki/User:Dotdotike/Trust_Upon_Firs...](https://en.wikipedia.org/wiki/User:Dotdotike/Trust_Upon_First_Use)

<comment><USER5> Or just use tor browser...</comment></comment></comment></comment></comment></comment></comment>


Ask HN: How to motivate friends - <AUTHOR> So I've started a project with two friends. This project consumes me to that degree that I'm working 40 hours on the project + 40 hours on my normal 9-5 per week.
Now my two friends, one is a marketing professional and the other is a developer joined the project.
I finished a MVP (the other developer works on a different product) and would like to push it out. This is where the trouble started. I thought when we had an MVP the marketing guy would start creating content on a blog, get some social media traction, and create videos to showcase the product.
I've tried to talk to him about it, that he just needs to put in some hours, that I don't care if it's not perfect, I trust him with what should be done. Whenever we have a drink together he seems passionate about the project. He always have a positive attitude. I try to get him involved by asking what should be done next on my side or what goalposts he has.
I kind of understand him, he has a normal, good life. But right now he down-prioritises the product constantly, he might put in one hour per week iff we meet up over coffee.
When I speak with him I just get the response that I work too much and that he will get to it in due time or deflects the issue.
So my question is: how do I motivate a friend that says he is passionate but not putting in enough effort? I mean right now, even admitting to not having time would be fine. <comment><USER2> I think you are wasting your time on him. Friends may or may not have same ambition and priorities as you. Better pay some one to get this job done otherwise you may loose close friend too.</comment>


Barcode recovery using a priori constraints - <AUTHOR> http://www.windytan.com/2016/02/barcode-recovery-using-priori.html?m=1 <comment><USER2> previous discussion

[https://news.ycombinator.com/item?id=11037524](https://news.ycombinator.com/item?id=11037524)

<comment><USER3> There is no discussion there? limited reposts are explicitly allowed if a link hasn't been discussed before. Please do not flag as dupe in these cases.

<comment><USER2> Hi detaro. I'm sorry if it frustrates you, but I've been asked to mark dupes by the HN site admins. If there is already an up-voted story, particularly if the previous story is sitting on the front page of HN (i.e. /news), then additional submissions of the same story are dupes, even if it's the same story from a different source or via a slightly different url.

If you could please 'unvouch' the two dupes, it would be appreciated. It will save others the unnecessary work of re-hiding them. thanks.

<comment><USER3> > _If there is already an up-voted story,_

This seems to be a different interpretation of the rules than is practiced otherwise, which interprets attention as "has been discussed". If there is a new official interpretation IÄd appreciate a link to it.

In this case, the timestamps confused me: I saw the submission yesterday night (17 hours ago), clicked here on "past", only saw this old submission, but didn't see that it was apparently bumped by HN, because if I click on the link it shows "4 hours ago". Given that it is very close I will unvouch these two submissions, but I won't flag potential further submissions of the link (and maybe even resubmit it myself, with enough distance)

<comment><USER4> This was confusing because it's a borderline case, so you were both right. 28 points is significant attention, but 0 comments is no discussion. That's unusual, and right on the boundary. A case like that probably needs to be called by the ref.

In this case, it's probably a good article because the site is known to be excellent, so we could probably have let it go through.</comment></comment></comment></comment></comment>


Show HN: A single pass, self-hosting compiler for the Oberon-07 language (JVM) - <AUTHOR> https://github.com/lboasso/oberonc <comment><USER2> Awesome project, Wirth is a hero.

What was your inspiration? How fare are you planning on taking this project?

<comment><AUTHOR> Niklaus Wirth is an amazing engineer/computer scientist, unfortunately he is often unknown by new generations (especially later achievements, like Oberon). I was inspired by his work and never ending quest for simple and effective solutions.

If enough people are interested in this project, I am planning to write a blog post series dissecting and explaining the code base (given the small size it should be feasible). Another idea is to port the whole Project Oberon to the JVM like this project [https://github.com/schierlm/OberonEmulator](https://github.com/schierlm/OberonEmulator), with the difference that the code will run at full speed since there is no emulation involved.</comment></comment>


Trauma Counselors Pressured to Divulge Confidential Info About Facebook Mods - <AUTHOR> https://theintercept.com/2019/08/16/facebook-moderators-mental-health-accenture/ <comment><USER2> I saw a heartbreaking YouTube mini-documentary on Facebook moderators.[1] The American workers seem to get the "lighter" flags like animal abuse, which is more than enough to cause trauma with daily exposure. I shudder to think what the offshore moderators go through when dealing with human/child abuse flags.

In my view, Facebook clearly wants this to be a temporary evil, so they can use the data from human moderators as a training set for automated ML-based moderation. But I wonder how long people will have to endure this for those models to reach an acceptable efficacy.

[1] [https://www.youtube.com/watch?v=bDnjiNCtFk4](https://www.youtube.com/watch?v=bDnjiNCtFk4)

<comment> munk-a > In my view, Facebook clearly wants this to be a temporary evil, so they can > use the data from human moderators as a training set for automated ML-based > moderation. But I wonder how long people will have to endure this for those > models to reach an acceptable efficacy.

Alternatively I wonder if it's worth it to put any number of human beings through this kind of suffering for something as worthless as social networking.

<comment><USER3> You, too, can walk away from Omelas.

<comment><USER4> It's a beautiful story.

[https://www.utilitarianism.com/nu/omelas.pdf](https://www.utilitarianism.com/nu/omelas.pdf) The Ones Who Walk Away from Omelas By Ursula LeGuin

</comment></comment></comment></comment><comment><USER5> _moderators could previously count on 45 minutes every week with a counselor, or two hours a day for those viewing images of child sexual abuse, with a minimum quota of one visit per quarter. Today, moderators find themselves barred from even this scant mental health care unless "their productivity was high enough for that day,"_

</comment><comment><USER6> I kind of think if you need hundreds of people looking at horrible content all day, then maybe your business is messed up and shouldn't exist. Humans can survive without Facebook. It's not like they're hospice workers or criminal investigators. The only goal is to make Facebook shareholders rich.

<comment><USER7> I knew someone who did content moderation for BlackBerry (née RIM) back when BBM was huge. Any platform that enables communication can be used for illegal and horrible content. What's changed is the fidelity. The worst I heard of on BBM at the time was images such as illicit/child pornography and human trafficking (via text). The ability to share high res video means the horror today is more visceral for those that have to watch it.

Having said that, there's no excuse for how Facebook and their subcontractors treat the people doing this job. They can easily afford to do better.

</comment></comment><comment><USER8> If it looks to an outsider like you work for company F, but you actually work for and are managed by company A, then company F has deemed you not worth the trouble of direct responsibility. They don't even see you as expendable, they just don't see you.

</comment><comment><USER9> For what its worth, I believe that the most egregious offenders (think violence, murder, rape, child porn) should be directly referred to the police with all data Facebook has on them.

This at least should slow down the torrent of shit that has to be moderated over the time...

<comment><USER10> I mean in the US pictures of murder aren't illegal.

<comment><USER9> Showing them to children is, at least in Germany, I do hope there are similar public decency laws in the US.

In any case, people sharing murders who are not journalists or citizens trying to get attention for an issue (think gang violence or political oppression) should be investigated for their sanity.

</comment></comment></comment><comment><USER11> Heard a story a while back on NPR about how Youtube is outsourcing the trauma of moderating terrible content to the Philippines. It was a glimpse into a world I had no idea existed.

<comment><USER12> Maybe that's just a convenient source of desperately poor Catholic workers.

But I wonder, maybe they could recruit people who enjoyed viewing horrible stuff. At least, if they were self-aware enough to know that the stuff they liked was horrible.

<comment><USER13> To further that thought, they can just wire up sensors to these people's brains, and flash the images for then to see. If the sensors notice arousal: mark photo as explicit.

In the dystopian future, they're going to not have enough psychopaths for this kind of work, so they're going to wire up prisoners in lawless parts of the world and modify the software to detect distress... 20 hours a day, 7 days a week.

<comment><USER14> This sounds horrific, but realistic.

We already have prisoners making license plates as slave labor. All it takes is a couple congressmen in the pockets of a big social network to make it mandatory for "rehabilitation" to moderate social networks.

</comment></comment></comment></comment><comment><USER15> >" _Facebook, Accenture, and WeCare may try to feign ignorance or implement common liability limiting language in their response. We hope all parties do not succumb to these common and repeated trends, and instead do what is right instead of what you are legally allowed to get away with._ "

I admire their optimism.

</comment><comment><USER16> Well here's an interesting thought. Let's say you train a generative network on illegal imagery. Is the combination of network and weights illegal to posses? What about images it generates?

<comment><USER17> Depends on your nation's gun laws. Is a gun legal? What about shooting someone?

</comment></comment><comment><USER18> Hopefully a small group of licensed psychologists and therapists can set up a small office across from the FB office in Austin and start serving the moderators privately.

<comment><USER19> Yeah I'm sure the therapy will be covered by the amazing mental health benefits these moderators receive through the no-cost health insurance provided by Accenture. If not, they'll have no trouble at all paying out of pocket with that cushy $14.50/hr they're making!

<comment><USER20> There is something (in the US) askew about mental health coverage. I think that regulations were passed to try to get mental health conditions treated more like others, but it seems like there is a shortage of counselors, social workers, and doctors, much more than other kinds of medical professionals.

I have this feeling that there's something about the incentive structure under the ACA that is choking off the availability, but I'm not sure what exactly it is. I'm very curious about the economics of running a mental health practice these days.

If you can't pay $70-140/hr to talk to a social worker every week or two, they may well offer you the option of online chat for $150/month. But I feel like there is a mystery there, because why _should_ it cost that much? And yet, the people who charge that much seem desperately overworked and barely functional. Where does the money go? Not to doctors, because everything is done by non-MDs these days. Not to insurance, because deductibles are in the thousands.

<comment><USER19> The issue is that mental health providers are grouped separately from other doctors and health care professionals on the insurance side. Different insurers, accordingly, have dramatically different networks of mental health professionals which vary in size and quality mostly depending on the reimbursement rate. Low-paying insurers have worse networks. Some don't even manage their own mental health benefits. In the bay area, for example, Blue Shield's benefits are managed by another insurer named Magellan with famously shitty rates and a correspondingly shitty network of available professionals.

<comment><USER20> Perhaps, but specific problems of the bay area or some insurance networks being insufficient are beside my point. The providers that exist are generally in-network with the insurance plans I'm aware of in my area, it's just that they seem to be overtaxed and mostly not taking new patients.

I'm saying, in an area where $50K a year is a decent living, and people take for granted that an actual MD is not available in most cases, why is $70+/hr not translating into supply?

It reminds me of my former employer, which was not healthcare, but they were billing clients something like five times what they paid staff, and yet their response was never to try to increase the hours billed and hire more people, but only to squeeze and squeeze costs and shrink through attrition as though they were losing money.

As a matter of fact, I really should ask someone I know who works for NAMI.</comment></comment></comment></comment></comment>


How should I calculate compensation between rounds? - <AUTHOR> I'm an early employee (2) at a pre Series A startup. I've been there a year (joined after seed round) and have been doing fairly well in my role (driving a decent amount of measurable value for the company). The situation I've run into is that, in part because of our collective success, we've delayed raising our A until later this year. As a result of that, our founders have said that they won't be changing anyone's compensation until the A happens.
I totally understand not increasing salaries until the raise, but something one of the founders said sounded wrong to me. He said that my vesting schedule is effectively a raise for me because the company has become more valuable.
Before he said that, I had thought that my first year total compensation equation was:
salary_value + total_equity * company_valuation_last_year
If that's not the case, then last year my compensation was:
salary_value + 0.25 * total_equity * company_valuation_last_year
which is quite a bit lower than even some pure salary offers that I had. Perhaps this is a bit long-winded, but what's the right way to think about compensation at this stage? Should I be asking for additional equity based on both the fact that we haven't raised salaries yet and that my value has increased in the last year? Or am I missing the point entirely?
My guess on the equation would be that my first-year total cost to the company was $X and that this year, even if I didn't increase in value, they should provide additional equity along the equation:
(X - current_salary) / company_current_valuation <comment><USER2> Compensation and equity are not the same, vesting over time is not equal to a raise (or "effectively a raise"). Compensation is what you were paid and what you were taxed on (typically) during the year. Equity is never compensation until/unless the company's shares are marketable, then it can count as compensation (kind of a general statement, there are nuances of course). I like to explain compensation this way, if you can deposit what the company gave you into a traditional bank then it is compensation, everything else is either a future promise or a potential liability.

I could write a ton on this, but let me say this. A company that delays their A is either knocking it out of the park so they don't need to take the dilution and see a path possibly of not raising an A, OR they have issues and are trying to figure it out. If they are doing so well, the company may delay their Series A but good founders will recognize they cannot delay taking care of employees just because they delayed the Series A. They may limit the number of increases and make them smaller than they originally planned, but they will take care of the employees that are making them successful, because if you lose those employees you lose momentum and possible the business if it is still only a few people.

Your founder is either just inexperienced and talking out his/her ass trying to justify their position or they are being deceitful for a reason. I'd always err on the side of a young founder being inexperienced over deceitful, but only someone involved can figure that out.

Equity is the bonus (lottery ticket) used to convince people to take a chance on an unknown company with unknown founders, it is not compensation initially. 98% (a number totally pulled out of my ass but based on my experiences) of the time equity will be worthless. That said, if things are still fun & worth it then you join and forget about the equity unless you fully vest and the company has an exit or the equity becomes marketable in another way (secondary market etc). Also, that doesn't mean you shouldn't try to get more equity as you go, especially if things are going well, extra grants is a way to help make it more of a real event for you in that rare 1-2% chance it works out. Also extra equity is the only way to offset the dilution each raise will bring you.

I have seen founders try to say your compensation is your (salary + (vested equity * current valuation)) or some variation of that. That is just a way to try and convince inexperienced people that they are being compensated fairly. No one has a clue what the valuation will be when the equity becomes marketable or at an exit, it may even be negative (liabilities > assets). You have no way of paying bills by telling the bank you own 1% of "ABC SaaS startup", no one is loaning you money against it etc. And even if someone did loan you money that way, that is still not compensation it is trading on your credit/reputation not the companies.

If you got paid $80k this year by the startup, that's your compensation, period. Only you can say whether you feel what is going on is fair. If it is fair and you like the place, stick it out, if it isn't be ready to find your next gig and use this experience to set yourself up differently next time.

BTW -- I am definitely not negative on startups, I just believe people should know the facts and likely outcomes so there isn't a misunderstanding or someone feeling taken advantage of at some point.</comment>


Ask HN: What motivates you in your work? - <AUTHOR>

What is the breakdown of your motivation in choosing jobs, contracts or building startups? How much of it is money driven and what are the other factors? Did it change in the course of your career? <comment><USER2> It's all money driven. Nothing else, money, the more money, the more motivated I'm. I don't subscribe into the idea that work is fun or should be fun. If it is, great! If it's not, whatever!

<comment><USER3> what would you do if you didn't need money ?

<comment><USER4> Most people would be satisfied with their hobbies, I think. I know I wouldn't do anything I didn't like anymore.

</comment></comment></comment><comment><USER5> I think my motivation for day-to-day work is different from the motivation that goes into choosing a job.

In no particular order:

Money: There's a certain threshold of that I need to achieve to maintain the lifestyle that I want. I'll take as much over that threshold as I can get, but below it is a no-go.

Remote: I move a lot, my average lease anywhere is 6 months. I want to live in every state in the U.S. and perhaps a couple countries in Europe and Asia before I die, being able to be remote lets me make that happen.

Professional Satisfaction: I like the problems that I'm solving to be overall interesting, that is, I would like them to have some novelty and perhaps be challenging my existing skill set. I am not interested in just doing route work where I am stagnating by just exercising the same skills or challenges over and over again. If not, it becomes difficult to be motivated for the day- to-day work.

Ethical Behavior: I set a high standard for the ethical treatment of clients and client's clients and I expect my employer to as well. This means not collecting or sharing private user data if we do not have the resources to do it securely. This means turning down feature requests or projects because the client is asking for behavior that is legally dubious, abuses privileged information, or would be do more harm to the client or the client's clients then good. I do not believe in selling a project to someone if I cannot stand behind the service. That is will be overall beneficial and has the possibility to succeed. I don't want to waste someone else's money on a project that has no or little chance of success or promise them false hope in order to make a sale.

</comment><comment><USER6> In general professional service firms aim for a balance between three goals: profitability, professional satisfaction, and client service. As a freelancer, I try to satisfy the same goals. I want a good income. I want my work to be enjoyable, rewarding, and make me grow. And I want my clients to be happy.

All three of those are genuine ends in themselves. If any is missing, I'll be unhappy. If I'm not earning enough, I'll be worried. If I'm making a fortune but doing horrible work, I'll yearn all day for something else. If I'm eagerly diving into the latest tech but my clients are dissatisfied, I'll feel like I don't deserve their business.

Each goal is also a means to the other two, but it's important to keep that in the background. Of course I'll earn more if I keep my clients happy, but if that's my only motivation, things will eventually break down. One reason I freelance is that I value the interpersonal side of software development, meeting with clients and helping them achieve their goals.

There are projects I've really sought that didn't pay as well I could make but were interesting or would let me develop new skills. There are projects I've sought because of who I could work with. I'm willing to give up some cash for those ends. I'm still financially motivated, but I like my job and I take pride in it, so money isn't everything.

</comment><comment><USER7> 100% money based motivation. I don't particularly like programming but I'm good at it. I just want more money to buy things I enjoy using/doing.

</comment><comment><USER8> I run a small company so might be a different perspective than you're looking for. But in general my motivations are:

1\. Working with people that challenge me, make me laugh, and who do great work. And who I can have a beer with :) That goes for the team here and our customers.

2\. Doing projects or products that are challenging and somehow "meaningful" to me. I like to enter unchartered territory in my projects where possible.

3\. Earning enough money to be happy day to day (I don't need much), but having the slightest opportunity to earn LOADS of money :)

It did change my career. I started out earning good money contracting ($200K) and got bored after 4 years. Then started a business working with good people but bad clients and non-meaningful project, the business failed after 4 years. Now I'm building products and projects with great clients earning reasonable money, and working with a fun team.

</comment><comment><USER9> I'm a product manager. So, working and re-working solutions to problems and watching the users interact with each iteration really gets me going.

It's all about playing with technology and using it to solve problems.

</comment><comment><USER10> The women! They go crazy over programmers...

</comment><comment><USER11> Primarily opportunities to do things I haven't done before.

</comment><comment><USER12> Office snacks. It's all about the snacks.</comment>


The State of the Couch - <AUTHOR> http://jan.prima.de/plok/archives/180-The-State-of-the-Couch-The-Invited-Talk-at-the-Erlang-Workshop-at-ICFP-in-Edinburgh-in-2009.html

<comment><USER2> "The big thing coming up in October is the inclusion of CouchDB into the Ubuntu Linux distribution"

\- ah, I wondered why I saw erlang being installed while updating the latest ubuntu alpha (5)... and here I was thinking that all those erlang articles a couple of weeks ago started a new trend...

</comment><comment><USER3> Nice update. CouchDB is constantly being stacked up against the big key value storages. It's really quite different. It may not be the answer to your web site scaling needs but it has interesting things going for it nevertheless.</comment>


'Do you ever think about me?': the children sex tourists leave behind - <AUTHOR> https://www.theguardian.com/society/2019/mar/02/children-sex-tourists-leave-behind-fathers-visited-philippines <comment><USER2> The article mentions the exorbitant cost of DNA/paternity testing for sexy workers; I wonder what happens if sites like Ancestory and 23andme were made more available for paternal identification (or even familial searching, "close enough" genetically).

Also curious if and how you petition across countries for child support once you have evidence in hand.

<comment><USER3> Interesting if they offered this service on a charitable basis as a detriment to abusive sex tourism.

<comment><USER4> On the other hand, it would probably kill the sex tourism industry there. You might think that's a benefit, but I'm not sure how good it would be for the people living there. It's not like people pick that type of life if they have _better_ options.

</comment></comment></comment><comment><USER3> Heartbreaking</comment>


Inside the Volkswagen emissions cheating - <AUTHOR> http://lwn.net/Articles/670488/ <comment><USER2> The line about how the engineers view the tests as an optimization problem really resonates with me. When the results of tests administered in an artificial setting becomes the primary method of assessment, such a trend will inevitably emerge.

Take school examinations for example, in highly competitive environments where test scores alone are the major determining factor, students begin to consider the benefits of forgoing comprehension of the material for rote learning.

<comment><USER3> These are both good examples of Goodheart's law — "When a measure becomes a target, it ceases to be a good measure."[1]

[https://en.wikipedia.org/wiki/Goodhart%27s_law](https://en.wikipedia.org/wiki/Goodhart%27s_law)

<comment><USER4> Thanks for that, I remembered that this was a "law", but I never remember the name, and whenever I try to tell anyone I get it wrong or express it in a confused manner.

</comment></comment></comment><comment><USER5> > He then disassembled the firmware, which looked quite a bit different than > disassemblies of normal code. It looked somewhat like electronic schematics > had been turned into code, rather than spaghetti-like output from other > kinds of programs, he said.

The actual logic diagrams and definitions of this data for Bosch ECUs (sans the emissions cheat, obviously) can be found in the "Funktionsrahmen," leaked but easily available from many VW tuning forums online. It's quite fascinating and worth a look for anyone interested in how modern ECUs work (it's a far cry from the oldschool "fuel table" style of engine management).

<comment> semi-extrinsic Wow, that's an acronym soup if I ever saw one:

[http://s4wiki.com/wiki/Tuning](http://s4wiki.com/wiki/Tuning)

<comment><USER5> Wait until you see some of the logic diagrams (this is the one for intake air temperature sensing):

[http://imgur.com/KWob8Xf](http://imgur.com/KWob8Xf)

<comment><USER6> These look like diagrams from Bosch Otto ECUs (ME series).

FWIW, the ECUs in question (EDC17) use much more descriptive names:

Epm_nEng - engine speed; InjCt1_qSetUnBal - set quantity for first injection (roughly); VehV_v - vehicle speed

I much prefer the EDC17 style, but your mileage may vary.

<comment><USER5> Oh, you're right! I've never owned a diesel so I've only looked at EDC17 in passing. :)

The variables certainly aren't as cryptic... I think even a native German speaker would have trouble with the ME7 ones.

</comment></comment></comment></comment></comment><comment><USER7> The statement that the cheating was to reduce AdBlue usage is not the complete answer to "why", as many of the affected cars in the US didn't use after- treatment at all.

</comment><comment><USER8> Interesting quote:

"he noted that there is a 12KB block of code that is used to ensure the tachometer always shows 780 RPM when the car is idling. Even though the engine is not that steady, car owners want to see that value hold steady at idle, so car makers effectively lie to satisfy them. Domke said that it is clear that lots of different kinds of cheating is going on in the ECU and noted that the speedometer doesn't really show the speed of the vehicle, just something related to it."

</comment><comment><USER9> previous discussion [https://news.ycombinator.com/item?id=10870518](https://news.ycombinator.com/item?id=10870518)</comment>


Ask HN: What Do You Consider Non Trivial Portfolio Project Ideas? - <AUTHOR> Hi,
I will appreciate the guidance of Senior Software Engineers/ Technical Hiring managers, with regards, what makes a good project, one could share with prospective employers, in seeking a Development role. I have been self learning Python programming, in conjunction with Flask the Python web framework, off and on for about 2 years and will really like to clench a development role. so I set out with the idea of building a CMS system based on Python and Flask, in order to both aggressively push my learning and programming savvy and ultimately clench that dream developer role. What are your thoughts? Too big of a project? have any other projects you might suggest? I will appreciate your feedback <comment><USER2> It depends where you want to go with your career,

If you're looking to get just any job, then a CMS is a good idea because it will teach you CRUD, and there's lots of jobs out there that are doing mostly CRUD.

If you want to get into Data Engineering / backend engineering job, then write a crawler. Pick a subject, e.g. German music, or Podcasts, or Electronic components, or anything like that, and write a crawler from scratch. Pull the HTML, parse them, store it and build a query dashboard on top of it. Building something like this end-to-end will give you lots of insight into systems engineering as a whole, and I used a project like this to get my job.

<comment><USER3> I have a little more experience than op. I see you have experience about specific domains of programming. May I ask how do I find a list of domains from which I could choose what else I want to do ?

<comment><USER2> I dont know if such a list exists - try extrapolating something you love.

Love fishing? What about using tidal information, public available fishing trawler data, weather patterns to suggest optimal fishing spots?

Love electronics? Can you index the top 10 global electronics companies and make a super slick product search with automatically updating pricing.

Love cooking? The BBC is shutting down their recipe website, mirror it, index it and make a cool search tool.

What do you love?

</comment></comment></comment><comment><USER4> Funnily enough, I was recruiting for a Python/Flask engineer not long ago. The best applicant didn't have side projects, but had a blog with over 150 articles on Python and Flask. He was rejected because they couldn't afford him.

I've written about side projects, and two best ideas are usually:

\- Contribute to open source. I would contribute to the Flask project if that's what you want to use professionally. Either improve the framework directly, or write a useful extension.

\- Make something that people will use, and preferably makes you money. People want to see evidence of production quality, not hobby projects. If that means shrinking the scope, do it.

<comment><AUTHOR> Thank you, appreciate your time and comment

</comment></comment><comment><USER5> Just one guys opinion...

I think a CMS is pretty boring unless it fits into your personal story/narrative.

__WHY__ did you create this CMS? What was lacking in the current offering and prompted you to tackle this problem? Perhaps it is industry/vertical/niche specific? Have you been charging for it? Without that part your are just another guy with another half-baked CMS.

Rather, it is much more interesting to see people go deep on their own interests, pursuits and passions, which (sometimes) lead to something with a broader appeal. Even if they don't, it still much more interesting.

\-- Rumor has it that Elasticsearch started when the guy wanted to make a recipe database for his girlfriend....

My latest side projects involve stitching together various DockerFiles and docker-compose.yml files to create super simple services like favicon grabbers, website-screenshot-as-a-service, and web-crawlers.

Anyway, that is my two cents.

Good luck

<comment><AUTHOR> Hi Sheraz,

I thank you for your candor, most appreciated.

</comment></comment><comment><USER6> I think you mean clinch, not clench :-).

I'd suggest you write a project that uses a sql database, with or without an ORM. While there are plenty of NoSQL databases out there, sql syntax is fairly standard and universal for basic operations, so picking up some knowledge of it will set you apart. Also in a production environment you will almost always have multiple python servers running and feeding into a central database where any request might be received by any server, so thinking about data persistence is important.

Maybe pick up a front end framework and a little javascript, enough so that you can make api calls to rest endpoints. Get your free year of aws and put it up, because learning to deploy to production and get something running is a good experience to talk about.

<comment><AUTHOR> I thank you, yes I meant to say clinch :)</comment></comment>


You Can Now Earn Real College Credit with Coursera - <AUTHOR> http://www.fastcoexist.com/1680899/you-can-now-earn-real-college-credit-from-online-education-provider-coursera

<comment><USER2> This is awesome! Glad to see they're paving the way! There's nothing like the face-to-face interaction of the university classroom, but for many the MOOC is a viable, and often preferred, alternative for education.</comment>


Ask HN: How do PassPath possibilitate 12 quintillion combinations? - <AUTHOR>

I feel gleefully ignorant again –guess I'm about to learn something.
How is it possible to have 12 quintillion combinations with 4 strokes? Like in this example http://www.simplyzesty.com/mobile/passpath-app-gives-you-12-quintillion-password-combinations
I get that the first stroke is one of 36 possible, all connections in any direction. And the following 3 strokes are then one of either 8, 5 or 3 choices. Not? How does that become quintillions?
Besides, 12*10^18 combinations or not, this lock is frightfully insecure in the real world. Take one look at the video and notice if you can remember the correct code. In fact one month ago, someone in my vicinity unlocked their phone this way.
I still remember that code too. <comment><USER2> Checking the screen for smudge marks is also a good way to break this system quickly.

As for the 12 quintillion combinations... I have no idea where that number came from either.</comment>