Ask HN: Are some signup CAPTCHAs becoming a bit ridiculous (yes you HN)? - <AUTHOR> My friend showed me this after trying to sign up for HN.  There's now a Capcha when you sign up which is more than difficult to answer... mostly because the text is so difficult to read it doesn't look like english.
Not sure if the intent is trying to stop bots or trying to stop anyone who doesn't want to spend 10 guessing characters. Oh, and think the text CAPTCHA is hard… listen to the audio puzzle (get your pen and paper ready). <comment><USER2> It seems to me that any captcha worth breaking is already "broken", given the existence of captcha farms - and I put broken in quotes because _technically_ , with captcha farms, the captcha is working as designed, you're just paying human beings to break them. The assumption, of course, that there is necessarily a difference between a "spambot" and a "human being" is not as true as it used to be.

Even so, the constant war of escalation between captchas and anti-captcha measures should eventually lead to the necessity to create a captcha which is impossible for most humans to decipher, once the capability of software to decipher them passes baseline human ability. At that point, just being able to solve the captcha would more or less prove you're probably not a human being. So the basic model of "text a human can read but a computer can't" is probably obsolete, and only still works due to the inertia of programmer laziness, and the fact that breaking captchas probably doesn't have a ROI worth the trouble for most sites.

Constructing more subtle captchas present their own problems, in that they can make cultural assumptions about the user. If you're also using the captcha as a community filter, this may be a feature though (for instance - having a site about anime set up a quiz about anime as a captcha, or having users solve complex programming puzzles.) Even so, any process which a human can perform through rote UI can be automated, so even those tests will fail. Most captchas are poorly designed and leak their solutions one way or another anyway. I've even seen a few posted here which seem to add their solutions in plaintext to the form as a hidden field or something.

I haven't got a clue what Recaptcha can be replaced with once it's thoroughly useless, but i've come to believe that captchas are one of those things it's impossible to do correctly, just adequately most of the time.

<comment><USER3> This is like saying we should stop locking our doors because every door can be broken by a locksmith anyway. The purpose of captcha is not to break any attempts of automation but only to make it more harder. Considering that automating a script to break captcha with human intelligence / automation is not trivial by any means I think, they serve their purpose.

<comment><USER4> The issue is that, given a little more time, we'll see a "one-size-fits-all" captcha breaker, one that needs little to no modification to break the captcha on new sites.

</comment></comment></comment><comment><USER5> The captcha on HN is temporary. We put it up to mitigate an attack while implementing a longer-term solution. Sorry for the inconvenience.

If we ask nicely, perhaps kogir will show up to say more.

<comment><USER6> There's a much better solution than a CAPTCHA, but when SN/HN was started, it wasn't as simple or easy to do. The answer is to simply require a tiny donation like one to five dollars to a charity (like Watsi) or non-profit organization (like the EFF).

YC now has (at least) two payment processors who could handle the transactions, either in real currency (Stripe.com) or virtual currency (Coinbase.com):

[http://stripe.com](http://stripe.com)

[http://coinbase.com](http://coinbase.com)

The payment processors will always have upstream fees that need to be paid, but if they're feeling generous, they could waive their profits on the donations.

If for some reason a person objects to donating to the EFF, then just give them a choice of charities, including the YC funded Watsi.org:

[http://ycombinator.com/watsi.html](http://ycombinator.com/watsi.html)

[https://watsi.org/](https://watsi.org/)

The altruism would also be good promotion for YC, Stripe, CoinBase, Watsi, and whoever else is involved.

If you want to give a break to the starving university students, then let them by-pass the donation requirement with an *.edu email address.

To keep everything fair between new and old users, the tiny donation could be an annual requirement. I sincerely doubt the kinds of hackers who want to contribute to technical discussions will have any problem with making a small contribution to one of many known-good charities.

<comment><USER7> I like this idea a lot, but it won't work in practice without killing off serendipitous contributions from people associated with stories here that happen to find out about HN and join just to answer questions. That happens a lot, and it'd be a shame if we killed it off.

<comment><USER6> That's a huge and important point. We all love to see the contributions from the people associated with the stories (well, I should only speak for myself, but it's seriously one of the best parts of HN).

Using an eventual required donation in conjunction with an initially usable account (e.g. allowing "X" days/comments/submissions before requiring a donation) might be able compensate, but it would not solve the underlying CAPTCHA, drive-by trolling, or spam problems.

It seems safe to assume the increased friction of requiring a tiny $1-5 donation (micro-payment) to a good cause will reduce the desired and beneficial contributions from people associated with a submission.

The tough question is, "By how much?"

It would be difficult to measure when associated people show up to add their contributions to relevant submissions. It would take just about all of the data on HN, and some very elegant code. The results would still be imperfect, but the results might still be useful. Even if actual measuring proves to be too difficult, a rough guess (opinion) of better/worse for before/after could still be useful. (Heck, removing per-comment karma scores "worked" to reduce hostility/competition even though we don't have any solid measurement data to prove it)

And the tougher question is, "Are we sure?"

Contentious comments (dumb/mean) have always been a problem in open discussion systems, and similar is true for abusive manipulation (spam, ring-vote, ring- flag, etc.). Having a gate of an act of altruism _could_ also both assure intent/interest and improve quality enough to actively encourage associated people to consider commenting to be even more worthwhile than it currently is.

In other words, there's also a second less obvious safe assumption; we're far more inclined to join a good discussion on a topic of interest than a bad discussion on the same topic.

Without testing, we just won't know if the added friction of a required donation would be overall harmful, or overall helpful, or roughly even. Prior to Stripe/CoinBase/Watsi/... the idea of requiring a tiny donation just wasn't feasible, but now, it might be an experiment worth running.

But the toughest question is, "How much will it hurt to try?"

If it really did kill off contributions from associated people, then it would certainly need to stop, but other than the possible temporary reduction of associated contributions (which could be reversed), I'm unable to see any other real or lasting harm. Even if requiring a tiny donation turned out to be a totally failed experiment in forum design, it would still do some good in the world.

You might be totally right and it might fail in practice, but until we run the minor risks of actually testing it, we'll never really know.

</comment></comment></comment></comment><comment><USER8> Okay, you win HN, I created an account because the anti-captcha crowd is missing the point.

There's a current bug in Recaptcha.

IPs that successfully solve too many captchas get given progressively more difficult challenges, which is fine, but currently Recaptcha is using the IP of the web servers, not the client. This means that the difficulty ramps up for all users quite quickly. It seems the iframe Recaptcha is permanently affected, if you use AJAX its fine after the first reload (I wrote a simple JS hack that makes it reload the first time, see www.mPoll.me)

Only noticed it because I was previously proxying Recaptcha through the server and it run its successful solutions up too high, so when the new bug came in it was immediately obvious what had happened when the first challenge is "wthdyjikhgfyijv" and on reload its "fluffy bunny 18".

On my website I'm currently overwriting the Recaptcha callbacks to allow multiple captchas, just put in a simple check to reload it the first time:

        var reloaded = false;

        function reloadCaptcha(challenge) {
            $(':input[name=recaptcha_response_field]').val('');
            $('img.recaptcha').attr('src', '//www.google.com/recaptcha/api/image?c='+challenge);
            $(':input.recaptcha').val(challenge);

            if(!reloaded)
            {
                reloaded = true;
                Recaptcha.reload();
            }
        }

        Recaptcha.finish_reload = function(challenge,b,c){
            reloadCaptcha(challenge);
        }

        Recaptcha.challenge_callback = function(){
            reloadCaptcha(RecaptchaState.challenge);
        }

        Recaptcha.create(recaptchaKey);

</comment><comment><USER9> For a throwaway account recently it took me >20 tries to register an HN account.

<comment><USER10> Did it on my second try -- I, personally, don't think it's so bad. I thought it was a bit crazy when I first looked at it but if you look hard enough, you can make the letters out. I am not sure what the idea is behind the "challenge text" though.

</comment></comment><comment><USER11> Most CAPTCHA's nowadays I find unnecessarily complex. Use tricks like timing form completion (<50ms? Bot), hidden fields, etc, before ruining the UX with CAPTCHA.

Then again... does HN really care about UX? Token expiration after x time when browsing through the listings, ancient unresponsive design, etc.

There comes a point where it'll be more cost effective for spammers to just farm out the solving of CAPTCHA's to people in third-world countries. It just depends if there is enough value in spamming HN for them to bother (probably not, given the user-curated-and-rated content model.

<comment><USER12> Timing form completion also rejects legitimate users that have form filling extensions (e.g., LastPass, 1Password, Roboform, etc.) In my experience hidden form fields were only marginally effective, a lot of bots run headless browsers capable of detecting if a field has been hidden by JS/CSS

<comment><USER13> Between a simple captcha and a CSRF token, I've found that nearly all spambots are defeated. Hidden form fields don't do much, since bots are almost never paying attention to anything other than the DOM.

What can work well -- if you're willing to give up on fallback for non-JS users -- is inserting a required form element into the page with JS on submit.

</comment></comment></comment><comment><USER14> The Facebook account delete CAPTCHA was literally impossible. I had to give up and use the audio option. Every step of the process made me happier I was doing it.

<comment><USER15> Here is the best (most readable) Facebook captcha I had.

[http://imgur.com/P7nln3d](http://imgur.com/P7nln3d)

</comment></comment><comment><USER16> I don't know that the research says about their effectiveness (hard to find through all the stuff on google) but I've liked the slider-based captchas I've seen:

[http://www.3dcaptcha.net/](http://www.3dcaptcha.net/)

Seems promising given that human visual processing and pattern recognition are lightening fast, and the slider is intuitive and kinda fun.

Anyone know how effective they are at stopping bots?

<comment><USER17> If you can end up getting a polygon list from it I'd think it wouldn't be very effective at all. it looks like it decomposes the triangles onto multiple planes so that they only line up when rotated one way, but you can calculate that rotation based on the rotation of the planes that the polygons are on. Even if they correct that, you'd still likely be able to find whatever image has the lowest amount of white pixels (or whatever the background is) and probably do a reasonable job at least with the example on the page.

</comment></comment><comment><USER18> There have been several websites recently where I've given up because after 10 attempts or so I still can't get the capcha right.

<comment><USER19> Perhaps you're not human after all…?

<comment><USER20> HN gave me this image just now:

[https://www.google.com/recaptcha/api/image?c=03AHJ_Vutw8XwYq...](https://www.google.com/recaptcha/api/image?c=03AHJ_Vutw8XwYqiyhWsv8G5nYgad0R0mGi9ZTyQmbJpoSzJv0tJwspYPGVVYjzwMlOKL37wzUw3SSlUKrhIO0hvmYNmY7iIaaWQyCEXprzNGYFDvpbJiu3JW6-L0p4xXRO6wUUG- LiyxuJ1YwqFSTZadbTasR8zpVkkTLmNNZR9-EUdARTyHwk84biJh6RQSWNG5cAAiMiT75SuHh45bc3v7zsQWkFWa9kQ)

</comment></comment></comment><comment><USER21> I sometimes think that people don't really give much thought to some of the things that they do where they simply copy what others do.

My own personal pet peeve is people, on HN, who obscure their gmail address so that it can't be slurped by bots.

I mean why not just use a dedicated gmail account, just for HN, rather than "use my hn handle at that email service that everyone else uses generally". The dedicated account has spam protection and you can forward mail to your primary account as a filter if you want.

I have a couple of web forms with no spam protection at all. The amount of bots that I get isn't so great that I need to trouble people to figure out a captcha. Much less a really difficult one.

<comment><USER22> >why not just use a dedicated gmail account, just for HN, rather than "use my hn handle at that email service that everyone else uses generally". The dedicated account has spam protection and you can forward mail to your primary account as a filter if you want.

Because starting, configuring, and checking ANOTHER gmail address takes more time and effort than obscuring my current dev-related email address to be human parse-able only. Seems like a strange thing to have a pet-peeve about, as it could only barely affect your ability to get in touch with these people. Unless, you've been trying to slurp email addresses from HN profiles... I could see it being bothersome in that case.

<comment><USER21> "and checking ANOTHER gmail address"

If you forward it it's not another account to check though. Sure you have to set it up.

Also, obscuring it doesn't prevent someone who has written to you having a situation where your email ends up in the wrong hands (for sure you've received, if you don't have spam protection, those emails where someone who has emailed you has had some virus which gets all their email contacts, right?)

If you setup a dedicated gmail you can periodically change that as well "refresh" using your own email you can't do that.

To be sure though the pet peeve is really more just thinking that people are being overly hygienic about protecting their email address I guess. You are right that it's not a huge burden on someone who is sending you an email.

When I used to post my gmail account to my HN profile gmail caught all the spam it wasn't a problem (maybe this was for a year or year and a half that it was in the profile..)

<comment><USER22> I'll concede, I'm a little under-protected when it comes to spam in my email, and sectioning my usage out to different emails would probably be a very big improvement upon that.

Still, this ought be more a pet-peeve for myself. I can't see how obscuring my email would bother you, except perhaps in the line of thought that you think it does NOTHING to avoid spam, and I am just wasting a tiny amount of everyone's time by having them parse it while not benefiting personally. I figured if there was ever a profile a geek would write a scraper for, HN would probably be it. It was a single line of defense I saw others use, and adopted.

Appreciate the conversation. The mostly undiscussed behavior of email format preference in HN profiles is something that has always strangely held my attention; I consider myself a geek for micro-behaviors in this and most any community though.

</comment></comment></comment></comment><comment><USER23> Just today I found this:

[https://www.sublimetext.com/forum](https://www.sublimetext.com/forum)

Incredibly frustrating. A CAPTCHA that requires you to email for the code

</comment><comment><USER24> We're just using the standard reCAPTCHA ([https://www.google.com/recaptcha/intro/index.html](https://www.google.com/recaptcha/intro/index.html)). Maybe they're harder because I don't trust third party javascript and use the iframe version instead?

It sucks, and will shortly go away for most users. When previously our code would refuse or tell you to try again in a few hours, the captcha will be required instead.

</comment><comment><USER25> > Not sure if the intent is trying to stop bots

CAPTCHA does not stop bots. Captcha solving can, at the very least, be automated away. CAPTCHA's do not work.

<comment><USER26> google security did a nice blog post supporting the statement that CAPTCHAs aren't as effective as they used to be:

[http://googleonlinesecurity.blogspot.com/2013/10/recaptcha-j...](http://googleonlinesecurity.blogspot.com/2013/10/recaptcha- just-got-easier-but-only-if.html)

</comment></comment><comment><USER27> Some random ideas which I know are not perfect just putting it out here :

Taking an image and turning into a jigsaw puzzle.

Using a proof of work scheme similar to bitcoin.

Do a google image search for say fish . Take 5 of those images and put them on one side put two on the other alongside images of 10 other random objects. Ask the user to pick the two on the right similar to the ones on the left.

</comment><comment><USER28> Try the audio version, it's even worst!

</comment><comment><USER29> May I suggest an alternative: paying a small fee to avoid/replace captchas (say $1-10, or higher than the captcha farms pay...)

Also it seems like HN is using the older recaptcha (without numeric signs), I didn't know you could choose your recaptcha "version" though.

<comment><USER30> You may if you feel like paying every person on earth $500 a month to cover their captcha fees. Unless your goal happens to be excluding most of the world from Internet services and random web forums.

<comment><USER29> I don't know where you got that number from, that's extreme. This is a one- time fee we're talking about here. Also why should I pay anybody? Strange reply.

<comment><USER31> I think it was a strange way of saying that $1-10 may not be cost-prohibitive to financially-comfortable residents of 1st world countries, but could certainly be cost-prohibitive to users with less financial resources or from 3rd world countries. Such a fee could be a serious deterrent on a free and open internet with diverse participants.

</comment></comment></comment></comment><comment><USER32> HN uses reCAPTCHA, it can be bit frustrating sometimes. but its one time thing to get in .

<comment><USER33> I tried to solve the HN captcha. It's almost impossible and you have to get _really_ lucky that you get an easy image. The audio alternative is pretty messed up too.

<comment><USER34> are you using TOR or any other IP masking solution? The more reCaptchas get solved or requested by a particular IP the harder they become, so if you are behind a IP that has been used for solving/requesting a lot of reCaptchas it is close to impossible to get a readable one (took me about 80 tries till I got one right when I used TOR). I don't know what exactly determines this difficulty but it is definitely there and Google talks about it here: [http://googleonlinesecurity.blogspot.de/2013/10/recaptcha- ju...](http://googleonlinesecurity.blogspot.de/2013/10/recaptcha-just-got- easier-but-only-if.html)

</comment></comment></comment><comment><USER35> If you like the image, just wait until you try the audio! Even more fun.

</comment><comment><USER36> Is the actual link missing? Not sure where I'm supposed to look, anyway.

<comment><USER37> [https://news.ycombinator.com/logout](https://news.ycombinator.com/logout) then [https://news.ycombinator.com/newslogin](https://news.ycombinator.com/newslogin) I guess

</comment></comment><comment><USER38> IMHO visual/audio recognition based CAPTCHAs are a dead end.</comment>


Bitcoin mining likely uses more energy than it takes to keep NZ's lights on - <AUTHOR> http://www.abc.net.au/news/2017-12-12/bitcoin-mining-likely-uses-more-energy-than-nz-electricity/9246888 <comment><USER2> Not this discussion again. Yes, proof-of-work wastes energy. No, you do not have to compare the number with each single type of energy-usage in the world.</comment>


Supercharge your Computer Vision models with the TensorFlow Object Detection API - <AUTHOR> https://research.googleblog.com/2017/06/supercharge-your-computer-vision-models.html <comment><USER2> Their repository is pretty neat! It includes three state-of-the-art architectures in object detection: Faster-RCNN, RFCN, and SSD. It is missing YOLO [1][2], though, which shares some similarities with SSD. Another detector is the recently released Mask-RCNN [3], which of course wouldn't be possible to be included in this publication as we can't travel through time yet.

[1]: [https://arxiv.org/abs/1506.02640](https://arxiv.org/abs/1506.02640)

[2]: [https://arxiv.org/abs/1612.08242](https://arxiv.org/abs/1612.08242)

[3]: [https://arxiv.org/abs/1703.06870](https://arxiv.org/abs/1703.06870)

<comment><USER3> There are already newer versions, Yolov2 and DSSD. See [http://github.com/sbrugman/deep-learning- papers](http://github.com/sbrugman/deep-learning-papers)

In practice Faster R-CNN worked better for me than YOLOv2 as it, in contrast to what is reported in the paper, had a higher recall for the detect task I used it for.

<comment><USER4> "Speed/accuracy trade-offs for modern convolutional object detectors" seems to establish that Faster R-CNN beats R-FCN and SSD-type architectures in accuracy, however YOLOv2 can beat Faster R-CNN and R-FCN in speed, while retaining high accuracy.

</comment></comment></comment><comment><USER5> So, could you use this to solve the image recognition captcha's that ask you to select all images that contain [object]?

<comment><USER6> Maybe they will make you do a Captcha before you access the API?

<comment><USER7> So then you bootstrap that using another api key :)

<comment><USER8> It's captchas all the way down. :)

<comment><USER9> Mutual recursion ;)

</comment></comment></comment></comment></comment><comment><USER10> Is this a new Google API for use through their cloud offering or is it a set of tensorflow artifacts one can download and use freely without ever contacting Google Cloud?

<comment><USER11> Clicking through the two layers of links, it is a GitHub repository containing pre-trained models, training scripts, and scripts for running the models on Google Cloud: [https://github.com/tensorflow/models/tree/master/object_dete...](https://github.com/tensorflow/models/tree/master/object_detection)

<comment><USER12> Holy moly, I can't believe I didn't know about [https://github.com/tensorflow/models](https://github.com/tensorflow/models)

</comment></comment></comment><comment><USER13> So they are launching all of these frameworks targeted to mobile but what's happening to Tensorflow Lite ? I'm beginning to think that these things that they are releasing are scaffolding for this . I really hope it's not going to be vaporware from google I/O

<comment><USER14> I missed I/O -- what's particular to Tensorflow Lite? Is that distinct from the CPU target?

<comment><USER15> Mobile focused version of tensorflow.

<comment><USER16> Lol. Parallel data computations across resource (including battery) constrained devices? Good news, the owner of the device is now the product. The device is also the product. Can't wait.

<comment><USER17> I'm going to guess that Google knows a thing or two about mobile devices and their performance characteristics. Also, feeding something through an already trained NN can be pretty darn performant. I'll wait and see what this ends up looking like, but I am hopeful.

</comment></comment></comment></comment></comment><comment><USER18> Finally I'm getting the results for all those traffic sign CAPTCHAS I've been solving.

(And I just noticed I should not have include the post as part of the sign–sorry for any inaccuracies I may have caused)

</comment><comment><USER19> Anyone know of a sample app that uses this?

Say to detect if something is or isn't a hot dog?

<comment><USER20> Yes, here you go: [https://itunes.apple.com/us/app/not- hotdog/id1212457521](https://itunes.apple.com/us/app/not-hotdog/id1212457521)

</comment></comment><comment><USER21> This would be great to run a security camera still feed through. It could completely eliminate false positives.

<comment><USER15> I wonder if Nest is using it with their new cam[0], as it has person alerts now (with face detection).

[0] [https://nest.com/camera/meet-nest-cam-iq/](https://nest.com/camera/meet- nest-cam-iq/)

<comment><USER22> They would probably use FaceNet[0] then, if they only want to detect faces, as that should give better results.

[0] [https://arxiv.org/abs/1503.03832](https://arxiv.org/abs/1503.03832)

<comment><USER23> Except they wouldn't be able to detect people wearing masks, which is probably an important thing for a security camera to do...

</comment></comment></comment></comment><comment><USER24> I recently came across a company that's built a ML model to track feet (for footfall observations). It seems that if you had an appropriate training set (labelled feet) you could re-create what they have done with this technology. Perhaps not achieving state-of-art but close. Thoughts?

</comment><comment><USER25> They need some kind of context input.

-GPS position, intent/goal, domain etc.

I'm at a dog show I would want breed etc.

I'm on the street I just want it come back dog maybe dangerous dog, friendly dog.

Also, would be cool/scary to just get back movable object 1, person 1, living movable object 3 etc. and if I give it multiple scenes from a video it knows person 1 is the same person 1 and if I name (them) Tony it keeps tracking tony.

<comment><USER26> > I'm on the street I just want it come back dog maybe dangerous dog, friendly > dog.

Most autonomous humans ship with this capability.

<comment><USER27> But not all.

[https://www.google.com/search?q=blind+man+bitten+by+dog](https://www.google.com/search?q=blind+man+bitten+by+dog)

</comment></comment></comment><comment><USER28> Would it be able to detect textual regions in an image as it depics kite/persons in the example image?

<comment><USER29> Yes, if you train those models using a dataset with box annotations. A more relevant model if you want to transcribe the text : [https://github.com/tensorflow/models/tree/master/attention_o...](https://github.com/tensorflow/models/tree/master/attention_ocr)

</comment></comment><comment><USER30> Just spent the last 6 months making anpr camera. Now just need to put Python on it. Fun times.

<comment><USER31> My sentiment exactly. For my full-time startup, we've been trying, testing (many), and productionizing (one) object detection network for the past nine months. It was a tedious effort of implementing papers from last year's CVPR conference. This makes some of our MOJO go away, but in the scheme of things we can focus more closely on our business. Mixed bag.

</comment></comment><comment><USER32> What's the hype here. It's a curated model zoo, or?

<comment><USER4> The researchers have created a framework for object detection such that one can easily experiment with using different feature extraction networks, separated from the "meta-architecture" such as Faster R-CNN, R-FCN, or SSD, used to handle the object detection task. They compare many models using this framework, described in [https://arxiv.org/abs/1611.10012](https://arxiv.org/abs/1611.10012) \- and they were able to construct the winning entry of the COCO 2016 detection challenge based on this research.

</comment></comment><comment><USER33> This doesn't seem to include training scripts ?

</comment><comment><USER34> I can't find the license, anyone have better luck?

<comment><USER35> The root of the repo has Apache license 2.0.

<comment><USER36> Basically everything Google releases is Apache 2.0. It was company policy when I was there.

</comment></comment></comment><comment><USER37> Anyone know what license this is under?

</comment><comment><USER38> Direct link [https://research.googleblog.com/2017/06/supercharge-your- com...](https://research.googleblog.com/2017/06/supercharge-your-computer- vision-models.html)

<comment><USER39> Admins update submission please

<comment><USER40> [https://news.ycombinator.com/item?id=14562314](https://news.ycombinator.com/item?id=14562314)</comment></comment></comment>


Wickr Inc – When Honesty Disappears Behind the VCP Mountain - <AUTHOR> https://www.vulnerability-db.com/?q=articles/2016/10/27/wickr-inc-when-honesty-disappears-behind-vcp-mountain <comment><USER2> The gist is they claim Wickr failed to pay promised rewards offered by its bug bounty program, even though Wickr patched the issues reported by the researcher.

<comment><USER3> The company is taken over by the VCs. Perhaps the new executive thinks it is a waste of time/money to engage in VCPs.

<comment><USER2> Doesn't really matter what they think of it, new management has to honor existing commitments.

</comment></comment></comment><comment><USER4> Please excuse my ignorance. I searched on the webs but couldn't find what VCP means in this context. Can someone please decode that for me (and maybe other readers like me)?

<comment><USER5> "Vulnerability contributor program" \-- i.e., bug bounty program. These folks seem to be using it to mean 'the people at wickr in charge of the VCP.'

<comment><USER4> Ah, it makes sense now. Thanks for clarifying!

</comment></comment></comment><comment><USER6> Much prefer to use Whisper Systems Signal App...

<comment><USER7> Not sure why people trust a closed source app to keep their coms private. Signal is much better.

</comment></comment><comment><USER8> Btw, Wire [1] messenger just implemented timed messages which expire, like Wickr offers. Combined with a non-requirement for registration with a phone number and Google Play Services makes it even more ideal.

[1] [https://medium.com/wire-news/safe-and-tidy-with-timed- messag...](https://medium.com/wire-news/safe-and-tidy-with-timed- messages%EF%B8%8F-4f26ff17b11b#.8heq745dt)

<comment><USER9> Signal has that too [https://whispersystems.org/blog/disappearing- messages/](https://whispersystems.org/blog/disappearing-messages/) And using GCM is only a problem for people running a custom Android ROM without Google Play Services. They can use MicroG instead. For the vast majority of people who do have Google Play on their phone this is completely irrelevant. Using GCM doesn't make Signal less private.

> _Google doesn 't see any data via gcm, it's just a tickle. If you want push > messages, you gotta use a push network._

[https://twitter.com/whispersystems/status/695399112833761283](https://twitter.com/whispersystems/status/695399112833761283)

Those who want to use Signal without GCM can help out with code [https://github.com/LibreSignal/LibreSignal/issues/43](https://github.com/LibreSignal/LibreSignal/issues/43) or money to anyone who does the work [https://www.bountysource.com/issues/35722527-create- proper-p...](https://www.bountysource.com/issues/35722527-create-proper-pull- request-to-add-libresignal-s-websocket-support-to-ows-signal)

> _I 've also seen first hand how difficult 3rd party clients can be on large > networks with actual client logic, and unfortunately we simply don't have > the resources to deal with that._

> _I hope that everyone here who prioritizes federation above all else moves > to federated products that support their goals, and I hope that those > projects can demonstrate that I 'm wrong about the inability to build > competitive user experiences over the long term._

> _If the only thing that the remaining people here want out of LibreSignal is > a websocket-only solution and gmscore isn 't an option for whatever reason, > I would consider a clean, well written, and well tested PR for websocket- > only support in Signal. I expect it to have high battery consumption and an > unreliable user experience, but would be fine with it if it comes with a > warning and only runs in the absence of play services. However, I also > realize that still won't help people that are trying to build a Google-free > experience on Google's platform, since we still don't have the things we > need to be comfortable distributing software outside of Play._

[https://github.com/LibreSignal/LibreSignal/issues/37#issueco...](https://github.com/LibreSignal/LibreSignal/issues/37#issuecomment-226646872)

Here's a great comment about Wire [https://www.reddit.com/r/privacy/comments/57s7qw/wire_messen...](https://www.reddit.com/r/privacy/comments/57s7qw/wire_messenger_tos/d8wd77c/)

> The thing is, Wire is developed by a for-profit company that has yet to > discover a sustainable business model. They seem to be in a hurry to gain > users, boasting about their own app's security and privacy before it has > ever been independently audited.

> In December 2014, when they launched Wire, they claimed they could not read > their users' messages. They were forced to retract their statement when a > journalist asked about it [https://motherboard.vice.com/read/wire-built-by- > ex-skype-emp...](https://motherboard.vice.com/read/wire-built-by-ex-skype- > employees-retracts-promise-not-to-read-your-messages) , and didn't add end- > to-end encryption until March 2016 [http://www.reuters.com/article/us- > dataprotection-messaging-w...](http://www.reuters.com/article/us- > dataprotection-messaging-wire-idUSKCN0WC2GM) . Contrary to popular belief, > the protocol they now use is also not the Signal Protocol, but a custom > protocol that Signal's developers have said they don't recommend > [https://twitter.com/whispersystems/status/774482849609031680](https://twitter.com/whispersystems/status/774482849609031680) > .

> Wire's privacy policy states that they log metadata: > [https://wire.com/legal/#privacy](https://wire.com/legal/#privacy)

> __Using the Service to communicate by chat, our servers store your encrypted > messages and other encrypted content and log other information such as the > time and date of your conversations, and the other user or users with whom > you are communicating. When using the Service to make or receive calls, our > servers log and collect time and date of your calls, and the other user or > users with whom you are communicating. __

> Meanwhile, Signal's developers have said that "there are no "safe" > jurisdictions anymore, only safe services" > [https://twitter.com/whispersystems/status/783318001349070849](https://twitter.com/whispersystems/status/783318001349070849) > . Concerning metadata, Signal's privacy policy states: > [https://whispersystems.org/signal/privacy/](https://whispersystems.org/signal/privacy/)

> __Certain information (e.g. a recipient 's identifier, an encrypted message > body, etc.) is transmitted to us solely for the purpose of placing calls or > transmitting messages. Unless otherwise stated below, this information is > only kept as long as necessary to place each call or transmit each message, > and is not used for any other purpose. __

> This was put to the test in the "first half of 2016", when Signal's > developers received their first subpoena. According to the documents that > were published by the ACLU and OWS > [https://whispersystems.org/bigbrother/eastern-virginia- > grand...](https://whispersystems.org/bigbrother/eastern-virginia-grand- > jury/) , the Signal servers only store the number you register with (which > can be anonymous [https://yawnbox.com/index.php/2015/03/14/create-an- > anonymous...](https://yawnbox.com/index.php/2015/03/14/create-an-anonymous- > textsecure-and-redphone-phone-number/) ), the time you registered and the > last time you connected to the Signal server (the precision of which is > reduced to the day).

Here's a great comment about Wire [https://www.reddit.com/r/privacy/comments/57s7qw/wire_messen...](https://www.reddit.com/r/privacy/comments/57s7qw/wire_messenger_tos/d8wd77c/)

> The thing is, Wire is developed by a for-profit company that has yet to > discover a sustainable business model. They seem to be in a hurry to gain > users, boasting about their own app's security and privacy before it has > ever been independently audited.

> In December 2014, when they launched Wire, they claimed they could not read > their users' messages. They were forced to retract their statement when a > journalist asked about it [https://motherboard.vice.com/read/wire-built-by- > ex-skype-emp...](https://motherboard.vice.com/read/wire-built-by-ex-skype- > employees-retracts-promise-not-to-read-your-messages) , and didn't add end- > to-end encryption until March 2016 [http://www.reuters.com/article/us- > dataprotection-messaging-w...](http://www.reuters.com/article/us- > dataprotection-messaging-wire-idUSKCN0WC2GM) . Contrary to popular belief, > the protocol they now use is also not the Signal Protocol, but a custom > protocol that Signal's developers have said they don't recommend > [https://twitter.com/whispersystems/status/774482849609031680](https://twitter.com/whispersystems/status/774482849609031680) > .

> Wire's privacy policy states that they log metadata: > [https://wire.com/legal/#privacy](https://wire.com/legal/#privacy)

> __Using the Service to communicate by chat, our servers store your encrypted > messages and other encrypted content and log other information such as the > time and date of your conversations, and the other user or users with whom > you are communicating. When using the Service to make or receive calls, our > servers log and collect time and date of your calls, and the other user or > users with whom you are communicating. __

> Meanwhile, Signal's developers have said that "there are no "safe" > jurisdictions anymore, only safe services" > [https://twitter.com/whispersystems/status/783318001349070849](https://twitter.com/whispersystems/status/783318001349070849) > . Concerning metadata, Signal's privacy policy states: > [https://whispersystems.org/signal/privacy/](https://whispersystems.org/signal/privacy/)

> __Certain information (e.g. a recipient 's identifier, an encrypted message > body, etc.) is transmitted to us solely for the purpose of placing calls or > transmitting messages. Unless otherwise stated below, this information is > only kept as long as necessary to place each call or transmit each message, > and is not used for any other purpose. __

> This was put to the test in the "first half of 2016", when Signal's > developers received their first subpoena. According to the documents that > were published by the ACLU and OWS > [https://whispersystems.org/bigbrother/eastern-virginia- > grand...](https://whispersystems.org/bigbrother/eastern-virginia-grand- > jury/) , the Signal servers only store the number you register with (which > can be anonymous [https://yawnbox.com/index.php/2015/03/14/create-an- > anonymous...](https://yawnbox.com/index.php/2015/03/14/create-an-anonymous- > textsecure-and-redphone-phone-number/) ), the time you registered and the > last time you connected to the Signal server (the precision of which is > reduced to the day).

And two comments from Hacker News [https://news.ycombinator.com/item?id=12149642](https://news.ycombinator.com/item?id=12149642) [https://news.ycombinator.com/item?id=11726188](https://news.ycombinator.com/item?id=11726188)

<comment><USER8> At the end of the day Signal doesn't transfer messages realiably, which is also often repeated problem on the Google Play reviews. So I can't put my confidence in a messenger which can't reliably deliver instant messages. Not to mention using it without submitting to Google (Chrome "app") and which also happens to be OWS customer.

<comment><USER9> You can see if your message was sent to the server and if the message was sent to your friends phone. I haven't really had any problems delivering messages apart from one time when they had servers problems.

No need to use Chrome if you don't want to, Chromium also works.

</comment></comment></comment></comment><comment><USER10> i prefer telegram

<comment><USER9> Telegram isn't secure. Use Signal if you want a proper secure messenger.

[http://www.gizmodo.com.au/2016/06/why-you-should-stop- using-...](http://www.gizmodo.com.au/2016/06/why-you-should-stop-using- telegram-right-now/)

> _" Encryption works best if it's ubiquitous and automatic. The two forms of > encryption you use most often -- https URLs on your browser, and the > handset-to-tower link for your cell phone calls -- work so well because you > don't even know they're there._ > > _Encryption should be enabled for > everything by default, not a feature you turn on only if you 're doing > something you consider worth protecting._ > > _This is important. If we only > use encryption when we 're working with important data, then encryption > signals that data's importance. If only dissidents use encryption in a > country, that country's authorities have an easy way of identifying them. > But if everyone uses it all of the time, encryption ceases to be a signal. > No one can distinguish simple chatting from deeply private conversation. The > government can't tell the dissidents from the rest of the population. Every > time you use encryption, you're protecting someone who needs to use it to > stay alive."_

[https://www.schneier.com/blog/archives/2015/06/why_we_encryp...](https://www.schneier.com/blog/archives/2015/06/why_we_encrypt.html)

Pavel himself admits security isn't a priority here [https://twitter.com/durov/status/678305311921410048](https://twitter.com/durov/status/678305311921410048) in response to this:

Thomas H. Ptacek [https://twitter.com/Snowden/status/678274362609426432](https://twitter.com/Snowden/status/678274362609426432) _By default Telegram stores the PLAINTEXT of EVERY MESSAGE every user has ever sent or received on THEIR SERVER._

Edward Snowden [https://twitter.com/Snowden/status/678274362609426432](https://twitter.com/Snowden/status/678274362609426432) _I respect @durov, but Ptacek is right: @telegram 's defaults are dangerous. Without a major update, it's unsafe._

[https://twitter.com/Snowden/status/678274362609426432](https://twitter.com/Snowden/status/678274362609426432) _To be clear, what matters is that the plaintext of messages is accessible to the server (or service provider), not whether it 's "stored."_

Moxie Marlinspike [https://twitter.com/moxie/status/678219238394298372](https://twitter.com/moxie/status/678219238394298372) _It 's just how Telegram works and is self-documented to work: Only their marketing copy suggests otherwise._

[https://twitter.com/moxie/status/678277776391077888](https://twitter.com/moxie/status/678277776391077888) _If you 're on an iPhone, they also send a plaintext copy of every msg you receive to Apple's servers. So not even in transit._

[https://twitter.com/moxie/status/678309008789258240](https://twitter.com/moxie/status/678309008789258240) _For iOS push notification previews. They didn 't do the work to make them privacy preserving._

It's the least of Telegrams problems but let's not forget their home made crypto even though there are better alternatives. See the take-home message here:

> _" We stress that this is a theoretical attack on the definition of security > and we do not see any way of turning the attack into a full plaintext- > recovery attack. At the same time, we see no reason why one should use a > less secure encryption scheme when more secure (and at least as efficient) > solutions exist._ > > _The take-home message (once again) is that well- > studied, provably secure encryption schemes that achieve strong definitions > of security (e.g., authenticated-encryption) are to be preferred to home- > brewed encryption schemes. "_

[https://eprint.iacr.org/2015/1177](https://eprint.iacr.org/2015/1177)

And the conclusion here:

> _" Abstract: The number one rule for cryptography is never create your own > crypto. Instant messaging application Telegram has disregarded this rule and > decided to create an original message encryption protocol. In this work we > have done a thorough crypt analysis of the encryption protocol and it's > implementation. We look at the underlying cryptographic primitives and how > they are combined to construct the protocol, and what vulnerabilities this > has. We have found that Telegram does not check integrity of the padding > applied prior to encryption, which lead us to come up with two novel attacks > on Telegram. The first of these exploits the unchecked length of the > padding, and the second exploits the unchecked padding contents. Both of > these attacks break the basic notions of security, and are confirmed to work > in practice. Lastly, a brief analysis of the similar application TextSecure > is done, showing that by using well known primitives and a proper > construction provable security is obtained. We conclude that Telegram should > have opted for a more standard approach._ > > _Conclusion: TextSecure is > based on strong primitives that have withstood crypt analysis from the > crypto community for years, and these are combined in a way that proven > provides authenticated encryption. Telegram on the other hand has crafted > its own encryption scheme and deployed it in an unproven state, and prior to > any scrutiny from other cryptographers. We have seen this done time and time > again, and rarely with good results. Take for example the smart grid meters > that were shown to use terrible crypto back in April this year. Furthermore, > the DH Ratchet is a very nice way of providing forward secrecy on a per- > message basis with little overhead, which is an improvement over Telegram 's > one key per 100 messages approach._

[http://cs.au.dk/~jakjak/master-thesis.pdf](http://cs.au.dk/~jakjak/master- thesis.pdf)

<comment><USER11> After seeing your comments in this thread and looking at your comment history, I have to ask: What is your affiliation with Signal/Open Whisper Systems, if any?

<comment><USER9> No affiliation, and never talked to Moxie or any other dev as far as I know.

<comment><USER11> Just very enthusiastic about Signal then, I guess?

Often when someone is so outspoken about a product it's because they have a vested interest in its success (and they don't always disclose that fact)... that's why I asked. Thanks.

<comment><USER9> More enthusiastic about privacy and free software. I often see worse apps recommended for privacy reasons, so why not bring up the flaws in those and what's better about this. If something better comes along I'll switch to that and recommend that instead.</comment></comment></comment></comment></comment></comment>


Stupid Unix Tricks - <AUTHOR> https://sneak.berlin/20191011/stupid-unix-tricks/ <comment><USER2> Regarding the point of "SSH: Faster Crypto", you should not enforce only one single specific cipher for ssh. The reason also seems wrong, modern hardware should be capable of achieving better performance with an AEAD cipher (such as AES-GCM or ChaCha20-Poly1305) instead of AES-CTR, as the latter also requires an additional HMAC.

If there really is a slow (or insecure) cipher you do not want to use, remove it by prepending a minus sign, for example `Ciphers -3des-cbc`, which keeps all other default ciphers. Otherwise, you will miss out on better ciphers as they are added and would be stuck on this one forever.

<comment><USER3> > _Otherwise, you will miss out on better ciphers as they are added and would > be stuck on this one forever._

Is that actually a real concern at this point vs the risk that comes from not white listing a few known reliable ones? It seems like old-style security, favoring modularity and "what if we need to change this someday soon?", but in practice it's turned out to be a lot less valuable than expected and raise significant risks of accidentally using something bad. We seem to be past the point where new ciphers being "better" is actually an event to be expected with any real frequency, prime/elliptic curve seems pretty mature now. Post- quantum could be a new frontier at some point, but may well require significant other changes as well.

WireGuard for example decided to just flat out say that any cipher changes will be tightly coupled with a full on version change. If you're using it you know exactly what you're getting and that's it.

I don't disagree that white listing means care with what is chosen, and picking AEAD-based seems like a better idea anyway (WG is Curve25519/ChaCha20-Poly1305/SipHash/BLAKE2s). Plus there is server compatibility to consider in some cases. But I'm not sure the logic of "you might miss out on better ciphers as they are added" is convincing either, vs setting yourself an alarm to recheck your SSH setup every year or two. Shouldn't any cipher additions/deletions arguably be something you actively consider, rather than have automagically added?

<comment><USER4> > what if we need to change this someday soon?

Git was created in 2005 and its hash algorithm is already outdated.

Additionally, software and hardware support continue to develop for better performance.

<comment><USER5> SHA-1 was already broken (Feb 2005) when git was first published (April 2005). But Linus decided that git doesn't need a collision resistant hash function. [https://marc.info/?l=git&m=115678778717621](https://marc.info/?l=git&m=115678778717621)

<comment><USER4> SHA-1 was not broken until 2017.

[http://shattered.io/](http://shattered.io/)

If you know an earlier instance, go ahead and take the crown from the shattered folks.

\---

The choice to use SHA-1 was a trade-off of security, size, performance. If Linux invented git today, I imagine the choice would have been different, because those parameters are now different.

<comment><USER5> In cryptography broken means "known attack significantly faster that brute- force", which was published in 2005. And cryptographers were advocating for deprecating it several years before that, because the security margin was clearly insufficient. [https://www.schneier.com/blog/archives/2005/02/sha1_broken.h...](https://www.schneier.com/blog/archives/2005/02/sha1_broken.html)

The time between a theoretical attack and practical demonstration of an attack should be considered a grace period we can use to migrate to a secure primitive. Choosing SHA-1 for an application which relies on collision resistance after the 2005 papers is plain incompetence.

Git chose SHA-1 because Linus did not consider collisions a problem. The downsides of SHA-256 were pretty small even then (32 instead of 20 bytes, and somewhat slower performance which is still faster than most IO).

</comment></comment></comment></comment></comment></comment><comment><USER6> > Don't try to install things with brew if brew is not installed:

       if which brew 2>&1 /dev/null ; then
            brew install jq
       fi

This just hides a useful error message (brew not installed). I would rather just see that error message (either interactively or in a log) and have the script fail. Hiding the error message just leads to an eventual failure down the road when jq is invoked.

<comment><USER7> In bash, there is the build-in command named "command". I have used it for the same purpose, like in:

        command -v brew >/dev/null && brew install jq

<comment><USER8> Wow. It's impossible to google, and on MacOS, `man command` is useless.

<comment><USER9> $ help command

command: command [-pVv] command [arg ...] Execute a simple command or display information about commands.

        Runs COMMAND with ARGS suppressing  shell function lookup, or display
        information about the specified COMMANDs.  Can be used to invoke commands
        on disk when a function with the same name exists.

        Options:
          -p    use a default value for PATH that is guaranteed to find all of
                the standard utilities
          -v    print a description of COMMAND similar to the `type' builtin
          -V    print a more verbose description of each COMMAND

        Exit Status:
        Returns exit status of COMMAND, or failure if COMMAND is not found.

<comment><USER10> Thank you.

I never knew about help to describe a builtin shell command. normally searching for something like "command" in the bash man page would be very tedious.

also: help echo

<comment><USER11> i do think 'man command' should bring up the specific man page for this, so there is no need to search into the bash man. this works with most builtin shell commands. e.g. 'man ls' is a thing

<comment><USER12> On (at least) Debian systems (, and probably many more) `ls` is not a built- in, but an external binary (`/usr/bin/ls`) is used.

`which ls` and `help ls` will show you, if it is similar on your system.

<comment><USER11> It does't need to be a built-in to have a man page (many first party and third party libraries come with their own man pages).

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> _Before we begin, first note that bashrc refers to something that runs in each and every new shell, and profile refers to something that runs only in interactive shells (used by a user at a keyboard, not just a shell script, for example). They aren't the same and you don't want them to be the same._

I'm not an expert on bash exactly though I am a heavy shell user (POSIX shell for scripting) but this part doesn't sound right. When is .bashrc ever executed when bash isn't interactive? And as far as I can tell when I open new shells .profile isn't read. I am using Linux and tmux and the reason I mention tmux is that it opens bash as a login shell and therefore .bash_profile is also loaded. Is this a mac OS thing or the version of bash mac OS comes with, which I believe is really old due to some license issues.

<comment><USER14> _as far as I can tell when I open new shells .profile isn 't read_

True, iff you have a .bash_profile. Bash only reads the first of ~/.bash_profile, ~/.bash_login, ~/.profile. It will ignore the rest of the list once it's found an existing file.

<comment><USER15> This explains so many of my problems. Thank you.

<comment><USER16> [https://hackernoon.com/bash-profile-vs-bashrc-vs-bin-bash- vs...](https://hackernoon.com/bash-profile-vs-bashrc-vs-bin-bash-vs-etc-bash- tutorial-help-example-not-found-command-linux-mac-1c7f98c11186)

Note that in macOS every new Terminal (tab) opens a login shell, while most GUI Linux environments don't. (In Tmux/screen it depends on the configuration).

</comment></comment></comment></comment><comment><USER17> This inspired me to (quickly) write down my own Stupid Bash Tricks: [https://gist.io/@tsutsu/2c11fc0a36000a46566e9fd62c60dea4](https://gist.io/@tsutsu/2c11fc0a36000a46566e9fd62c60dea4)

Most of it might be somewhat obvious; but scroll down to "Bash Hooks" for a cool trick I've never seen anywhere else.

<comment><USER18> I found the OSX specific XDG section helpful, because I've been setting it to ~/.cache for years! Good to know there is a better way that the OS understands (a bit a least).

</comment></comment><comment><USER19> This got me to raise an eyebrow:

        if [[ -d "$HOME/dev/go" ]]; then
            export GOPATH="$HOME/dev/go"
        fi

Using a directory named "dev" not to store device files, but development tools. I am so stuck with conventions.

<comment><USER20> This is a point of conflict for me. I use "~/dev" for all of my development work, but I don't want to because of the convention set by "/dev".

My problem is finding an alternative to "dev" that is just as convenient. Do any of you have any suggested alternatives?

<comment><USER21> I use ~/ops

...but not in the "operator" sense. Rather, the "operations" sense...and that is because i used to use ~/projects...But i got lazy to type that out.

<comment><USER22> sometimes I use ~/opt

</comment></comment></comment></comment><comment><USER23> "Docker desktop for mac is closed source software, which is dumb for something that asks for administrator permissions on your local machine. This lameness aside, it runs the docker daemon..." ...Proceeds to run MacOS...

<comment><USER24> Like every time someone points this out: there are an unreasonable amount of companies that don't let you pick anything else. It's either a macbook, or a macbook, and if you want a different machine go buy it yourself and use it outside of work hours "because we can't afford writing three different copies of the same internal documentation for more than one OS when we're the ones paying for the machine you work on".

And in an unfortunate twist, if you _have_ to run Docker, macos is actually the best choice for that. Installing it on linux or windows is an exercise in "you know what, maybe I should buy a mac for this instead".

<comment><USER25> > Installing it on linux or windows is an exercise

...? Am I missing something obvious? Installing docker on Ubuntu is as simple as

        sudo apt install docker.io

and you're done. Installing from the official repos is simple too if you want the latest version [1]

[1] [https://www.digitalocean.com/community/tutorials/how-to- inst...](https://www.digitalocean.com/community/tutorials/how-to-install-and- use-docker-on-ubuntu-18-04)

</comment></comment></comment><comment><USER26> The author configures lots of syncing by setting the location of config files, I do it by setting up a whitelist based gitignore in my homedir:

      *
      !.gitignore
      !.bashrc
      !.ssh/authorized_keys

It's fast (comparative to a blacklist based 'git status' scan) and less work :)

On a sidenote, I'm curious about the security implications of the git repository - if the git host the service is breached, as far as I known there's really nothing stopping the actor leveraging access to achieve code execution on my host right?

I'm aware of commit signing but in the context of a raw git directory synced over ssh an attacker could create and use any valid signature key to commit to the repo. Hosting on Gitlab/Github would require a breach or significant abuse of security controls, but is still possible, too.

<comment><USER27> So you have a gitignore that says "ignore everything except these three files" \-- what does that do? Is it supposed to replace the line where he curls those files to github? Isn't it awkward having all your other git repos in your home dir be under that git repo?

<comment><USER26> > what does that do? Is it supposed to replace the line where he curls those > files to github

No, it is for synchronization. See the article sections titled 'SSH: Move Your SSH Config File Into a Synced Folder' and 'Extra Credit'.

> So you have a gitignore that says "ignore everything except these three > files" \-- what does that do?

My gitignore is upwards of 100 files, but allows me to track changes and synchronize configurations across hosts, which I do often as I often work in short-lived graphical VMs and across multiple hosts. Using a whitelist of tracked files means 'git status' wont take seconds/minutes to scan the entire directory tree under my homedir which seemed to be the case when using a blacklist when I initially configured it.

> Isn't it awkward having all your other git repos in your home dir be under > that git repo?

It breaks stuff like `git add -A` which I haven't fully solved, but don't really feel the need to - most of my commits are 2-3 files at most and I'd prefer to be aware of exactly what's being committed for the additional minor overhead.

There's other alternatives, like rsync, which solve entire tree synchronization but that's not what I'd normally like to do as my ultraportable has a 128gb SSD and my daily driver is a 2TB laptop. I'd be open to hear other suggestions, but at this point git is a convenient and flexible solution that works well in my environment :)

</comment></comment></comment><comment><USER28> With docker >= 18.09, you can connect directly over SSH by setting DOCKER_HOST=ssh://<user>@<host>

</comment><comment><USER29> Is there any advantage/difference to using ProxyCommand with netcat vs just saying "ProxyJump bastion.example.com"?

<comment><USER30> ProxyJump was added in OpenSSH 7.3 so systems from a few years ago might not support it. It does the same thing as ProxyCommand with -W %h:%p but you can't set custom options for the jump connection with ProxyJump.

<comment><USER12> OpenSSH 7.3 was released on 2016-08-01, so more than 3 years ago. Nobody should be running SSH versions this old (hopefully).

<comment><USER31> CentOS 6 lives on in enterprise, unfortunately. (don't know exact OpenSSH version but didn't support -J when I checked earlier)

</comment></comment></comment></comment><comment><USER32> I've had my shell startup scripts modularized for many years now, but I do a much more complicated system where the scripts actually have a function `require` exposed to them that can be used to express dependencies. If you `require` something that's already been loaded it does nothing, so each script is still only just loaded once.

</comment><comment><USER33> > I use Mac OS X (pron: "ten").

To be extra pedantic: unless you're running an eight-year-old OS, you are likely running either OS X or macOS.

<comment><USER34> Thank you!

</comment></comment><comment><USER35> author should probably be using: command -v

instead of: which

<comment><USER36> Fighting the good(?) fight for POSIX shell portability.

[https://pubs.opengroup.org/onlinepubs/9699919799/utilities/c...](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/command.html)

[https://shellhaters.org](https://shellhaters.org)

</comment></comment><comment><USER37> His tip about running "make clean" on ~/Desktop is okay, assuming people actually use Desktop the way he does. Which I observe that many, possibly most, users don't. Apple says that "The desktop is where you do most of your work", but I see a lot of desktops with no files whatsoever.

[https://support.apple.com/en-gb/guide/mac- help/mh40612/mac](https://support.apple.com/en-gb/guide/mac-help/mh40612/mac)

<comment><USER38> Very true because I don't want any file icons on my wallpaper. Btw, i use this [https://irvue.tumblr.com/](https://irvue.tumblr.com/)

<comment><USER39> You can also hide all the desktop icons with a defaults setting:

        defaults write com.apple.finder CreateDesktop false

<comment><USER38> Aha, thanks for sharing!

</comment></comment></comment></comment><comment><USER40> > "always ssh as root" example

urggh... I've seen devs do this too many times and it breaks a bunch of stuff when configured incorrectly. You almost never want "Host *" to have a user entry.

also completely ignores the "don't ssh as root" opsec best practice. I would strongly argue that even for dev environments, it's worthwhile building the opsec muscle memory and spending the effort at the start of a project.

</comment><comment><USER41> _The best way to store SSH private keys is in a hardware security module, or HSM._

I have three Yubikeys. Would I need to add three ssh keys to every one of my ssh accounts?

<comment><USER42> Yes.

If one Yubikey were lost, stolen, or damaged, you would then revoke its access by removing the corresponding entry in ~/.ssh/authorized_keys.

</comment></comment><comment><USER43> Thanks for the cached ssh connections thing, didn't know that was possible. Not that useful when I'm on my environment with TMUX (typically have each connection in its own tmux pane) but massively useful on all the jump boxes where I don't have a sensible shell environment :)

</comment><comment><USER44> Using a tor hidden service for emergency ssh access is pretty sweet and I'm going to have to go set that up for myself now. Maybe with an extra bit to auto-publish the hostname so I don't have to write it down every time something changes.

</comment><comment><USER45> I find it weird that an article about "Unix tricks" requires the GNU bash shell.

</comment><comment><USER46> Instead of which it's better to use command -v.

</comment><comment><USER47> Ssh ProxyCommand: make sure to use ssh -W option rather then depending on netcat.

</comment><comment><USER48> I was expecting something like

        cat file | less

Stop abusing cats!

</comment><comment><USER10> another interesting site is:

[https://dotfiles.github.io/](https://dotfiles.github.io/)

</comment><comment><USER49> Bash is not Unix.

<comment><USER50> But it does run on my Unix boxes.

<comment> yjftsjthsd-h Although amusingly, not exclusively; it also runs on Haiku and NT. (This isn't at all to say that it doesn't fit in an article about Unix, since it's very popular there (just like talking about Firefox on Unix wouldn't be out of place), just a fun side note.)

<comment><USER51> Haiku is a UNIX, though. :)

</comment></comment></comment></comment><comment><USER52> Well, I had to stop reading when I scrolled a bit and some pop up blocked the content with (I presume) the button to close it off screen on mobile.

<comment><USER53> There should have been an X at the top right corner.

Also, do you have reader mode on your device? That would fix it too.

<comment><USER54> Or you know, people could not put popovers on blogs.

<comment><USER24> If you're not browsing with an adblocker installed, then this complaint is basically moot: it's 2019, the ad-free internet died years ago; use a decent adblocker.

If they show up even with that active, though, now you have a valid complaint.

<comment><USER55> While this is common on desktop, do you have suggestions for how to do this on mobile?

<comment> yjftsjthsd-h Firefox mobile supports extensions, including ublock origin (and umatrix, if you're into that).</comment></comment></comment></comment></comment></comment>


Steve Jobs on iPhone Lock-In: Sorry, but we like it just the way it is. - <AUTHOR> http://waffle.wootest.net/2008/06/29/writing/

<comment><USER2> wow, an email from jobs himself.

it's still early days in all of this. apple is in a tricky situation. they don't have free rein here like they do with the mac. they have to cooperate with the wireless carriers at least a little bit.

as they develop more and more clout in this space, i think the situation will get friendlier.</comment>


Why the RDBMS is the future of distributed databases - <AUTHOR> https://www.citusdata.com/blog/2018/11/30/why-rdbms-is-the-future-of-distributed-databases/ <comment><USER2> So far Larry Ellison has had the last laugh, claiming that Amazon Does Not Use AWS to Run Their Business and that Amazon's databases trail Oracle by 20 years:

"Larry Ellison: Amazon uses Oracle, not Amazon to Run Their Business… Because AWS is Not Good Enough"

[https://www.webpronews.com/larry-ellison-amazon-oracle- datab...](https://www.webpronews.com/larry-ellison-amazon-oracle-database- aws/)

</comment><comment><USER3> So many implementations of this out there already, like tidb.

[https://github.com/pingcap/tidb](https://github.com/pingcap/tidb)

So yes i can believe it!</comment>


Mojolicious 1.0 released - A new Web Framework for Perl - <AUTHOR> http://blog.kraih.com/mojolicious-10-released

<comment><USER2> For a web engineer, <http://mojolicio.us/> is as hard-hitting a brochure page as I've seen in some time. Built-in long-polling combined with full-stack- style templates and simple views combined with that one line install. Good luck and thank you for this contribution!

</comment><comment><USER3> As the article says, Mojolicious was funded in part by a grant from The Perl Foundation.

If I remember correctly, the vote for it was almost unanimous. (I'm on the grant committee.)

</comment><comment><AUTHOR> For more see also the website at <http://mojolicio.us>

</comment><comment><USER4> Highly intriguing. Perl is a solid language that has really held firm ground for scripting but has been over taken by the likes of ruby in the web. Its nice to see the emergence of an mvc framework to help in its popularity and further growth. I for one will definitely be trying this and comparing with my current implementations in ruby.

<comment><USER5> you know there are several mature mvc frameworks already right? catalyst for example..

<comment><USER4> Yea I'm aware of catalyst but havent really spent any time with perl as of late. A lot of web development that I do is with ruby/sinatra which is the kind of framework I enjoy working with. Its just nice to see new development still going on for perl in this fashion.

<comment><USER6> you know there's lots of new development going on for perl aside from Mojolicious, right? :)

<comment><USER4> I know of the perl 6 development. I've not really seen anything else get noticed. Anything noteworthy to mention here?

<comment><USER6> Oh man, far too much to list in a post.

Just for starters though: perl 5.12 language features, Moose, Plack, perlbrew, Task::Kensho, the Modern Perl and Enlightened Perl movements, the Padre IDE, Strawberry Perl. That's maybe 0.0001% of what's going on.

</comment></comment></comment></comment></comment></comment><comment><USER7> I've been looking at Perl frameworks recently, and Dancer in particular. Dancer looks good to me - but so does this. I'd love to see it compared and contrasted to Dancer and Catalyst. Great work!

<comment><USER8> Adam Kennedy did a comparison between Mojolicious and Dancer a while ago - "Pitting Mojo vs Dancer in a competition to build Top100 2.0" (<http://use.perl.org/~Alias/journal/40270>, <http://use.perl.org/~Alias/journal/40292>, <http://use.perl.org/~Alias/journal/40312>)

</comment></comment><comment><USER9> How well has the installation process been tested for userland-only (non root) installs?

<comment><AUTHOR> In userland you actually don't have to install at all, there are no prerequisites to resolve and there is no build process. Just unpack the tarball from <http://latest.mojolicio.us> and start playing. :)

<comment><AUTHOR> It is of course also available as a CPAN module, so you could use the normal toolchain too. <http://search.cpan.org/dist/Mojolicious/>

</comment></comment></comment><comment><USER10> Very awesome!

</comment><comment><USER11> Grats!

</comment><comment><USER12> YAAY</comment>


Ask HN: Is comma.ai snake oil or they are actually delivering something? - <AUTHOR> I know that comma ai is a persona-driven company. Have you ever experienced their product in wild? <comment><USER2> I work in the self driving car industry. Building a level 5 self-driving car is not possible with a small team. Period.

What they are building is more of advanced cruise control. This I believe they can succeed at. ACC is orders of magnitude less difficult than level 5 AV.

</comment><comment><USER3> It's an open source company you can find all their releases on GitHub and try it yourself if your model is supported [https://github.com/commaai/openpilot/blob/devel/README.md#ha...](https://github.com/commaai/openpilot/blob/devel/README.md#hardware)

</comment><comment><USER4> Yeah it's pretty amazing for $700 and time if your car isn't already supported

check YouTube videos

</comment><comment><USER5> They definitely ship code, and people seem to like them. I really like their model, the hacker culture, the holy crap we built a self driving car for <1k, but I don't know if they'll be a real contender in industry.

<comment><AUTHOR> I share your sympathy towards hacker culture but I would like to hear actual experience rather than wishful thinking.</comment></comment>


How Does an Intel Processor Boot? - <AUTHOR> https://binarydebt.wordpress.com/2018/10/06/how-does-an-x86-processor-boot/ <comment><USER2> This part is a bit of a mess:

"16-bit Real Mode with insruction pointer pointing to address 0xffff.fff0, the reset vector. In this initial mode, the processor has first 12 address lines asserted, so any address looks like 0xfffx.xxxx. This fact combined with how addressing using segment selector (SS) register works, allows the CPU to access instruction at reset vector address 0xffff.fff0"

SS is the Stack Segment register. CS is for code.

In real mode, the instruction pointer is only 16-bit and can not hold a value in excess of 0xffff.

Ignoring those issues, the explanation still doesn't match what I've seen before in the documentation. If things have changed, when did that happen? The old explanation:

The CS base is set to something like -16, that is with all but the lower 4 bits set. This covers all of physical address space, with any higher bits just being ignored. The instruction pointer is set to 0. The result is execution that starts 16 bytes below the first address that is beyond the end of the physical address space. For example, with 44-bit physical addresses this would be at 0x00000ffffffffff0.

<comment><AUTHOR> yes CS not SS. i should fix that.

regarding how the CPU addresses 0xffff.fff0 is not exactly specified in the post. actually CS register is loaded with 0xf000 and normally this would yield a segment selector address of 0x000f.0000 (CS left-shifted by 4 bits). but on a reset, like the post mentions, first 12 address lines are asserted so the base address ends up being 0xffff.0000. these address lines remain asserted until a long jump is made, after which the first 12 address lines are de- asserted and normal CS segment selector calculation resumes.

instruction pointer contains -16 as you mentioned, the resulting address is:

base address + IP = 0xffff.0000 + 0xfff0 = 0xffff.fff0

i am not sure if this is worth adding to the post but it is definitely useful.

<comment><USER3> I recall reading that it's not that those 12 bits are explicitly asserted, but rather that the CS descriptor after reset is in an "unreal mode". After all, x86 segment descriptors consist not just of their numeric value, but also of a base address, segment size, and privilege information.

So at reset, CS is set to a descriptor whose numeric value is 0xf000 and whose base address is 0xffff0000, or something to that effect. All the rest follows naturally -- there's no special case logic that asserts lines of the address bus until the first long jump, it's simply that the reset value of the CS descriptor is rather magical, and that long jumps by their nature load a new CS segment descriptor which isn't magical.

<comment><AUTHOR> this is interesting! i am not aware of how this logic is implemented, i.e. the logic of initial state where 12 most significant bits but thanks for enlightening

<comment><USER4> I looked it up:

[https://software.intel.com/en-us/articles/intel-sdm#nine- vol...](https://software.intel.com/en-us/articles/intel-sdm#nine-volume)

Get volume 3A and read chapter 9.1.4 at pg 315. The text is quite readable:

      The address FFFFFFF0H is beyond the 1-MByte addressable range of the processor while in real-address mode. The
      processor is initialized to this starting address as follows. The CS register has two parts: the visible segment
      selector part and the hidden base address part. In real-address mode, the base address is normally formed by
      shifting the 16-bit segment selector value 4 bits to the left to produce a 20-bit base address. However, during a
      hardware reset, the segment selector in the CS register is loaded with F000H and the base address is loaded with
      FFFF0000H. The starting address is thus formed by adding the base address to the value in the EIP register (that
      is, FFFF0000 + FFF0H = FFFFFFF0H).

Any change to CS reverts this to normal real mode operation. So near jumps are OK, far jumps or interrupts are not.

<comment><USER5> Speaking of readability, here's a readable copy of that quote:

> The address FFFFFFF0H is beyond the 1-MByte addressable range of the > processor while in real-address mode. The processor is initialized to this > starting address as follows. The CS register has two parts: the visible > segment selector part and the hidden base address part. In real-address > mode, the base address is normally formed by shifting the 16-bit segment > selector value 4 bits to the left to produce a 20-bit base address. However, > during a hardware reset, the segment selector in the CS register is loaded > with F000H and the base address is loaded with FFFF0000H. The starting > address is thus formed by adding the base address to the value in the EIP > register (that is, FFFF0000 + FFF0H = FFFFFFF0H).

(Don't use indentation to format a block quote, only use it for code listings.)

</comment></comment></comment></comment></comment></comment><comment><USER6> Hey, thanks for the great article on an intricate topic. It is nice to see the resurgence of WordPress blog posts recently. Back in the day it was the primary source of sacred knowledge for me and many other people around the world.

</comment><comment><USER7> It's a bit light on details in the ME and pre-real mode stages. For example, it omits to mention the 'Authenticated Code Module' which now executes before the BIOS reset vector.

<comment><USER8> Lots of those details are vendor-specific. Really the specifified behavior exists in the modern world not to enable "boot" (which as you point out is mostly the job of firmware running on the ME and not the application CPUs), but to clearly define the initialization sequence for SMP startup (i.e. how do the auxilliary CPUs start -- Yes, that's right, they start in real mode) and some other minutiae like mode switching for legacy interaction with BIOS calls.

</comment></comment><comment><USER9> Doesn't this article confuse BIOS and UEFI? UEFI starts the bootloader in 32-bit mode, however, the classic BIOS still jumps through the bootloader in 16-bit mode.

<comment><USER4> That's a hard question to answer. On the original PC, ROM BIOS (Basic I/O System) was the name for both the (EEP)ROM-based firmware sitting at address F0000 and the INT based interface it provided to the OS.

This included how the system handed of control to the OS. BIOS just loaded the first disk sector and executed whatever it found there. MBR-based partition tables were a DOS-convention, the BIOS couldn't care less what the first disk sector did once it was in control.

When UEFI, the new boot interface, was invented, we needed a name for the old boot conventions. So we called them BIOS.

It's hard to claim the article confuses anything if the original word BIOS has such a confused meaning to begin with. If you say BIOS=PC firmware except the option ROMs, the article is correct.

<comment><USER10> The use of "BIOS" to name the PC, PC/AT, PC98, and suchlike firmwares long pre-dates the existence of EFI, and was _not_ a reaction to it.

And the name is even more confusing than you paint it to be. The "BIOS" was _also_ the bottom-half of MS/PC/DR-DOS, contained in IO.SYS in (pre version 6) MS-DOS and in IBMBIO.COM in PC-DOS and (post version 3) DR-DOS.

<comment><USER4> To clarify: The name BIOS was used for the very first PC, but there was no clear distinction made (or necessary) between firmware, boot protocol, interface, ...

When the old boot protocol needed a name, it got BIOS. This situation arose as reaction to the existence of the new UEFI boot protocol.

AFAIK, IO.sys/IBMBIO.COM was the interface used by DOS to the hardware (some kinde of HAL). Microsoft seemed to think every PC-clone vendor would implement its own firmware, and they would have to port IO.SYS for every platform. Happily, after I thinc Compaq reverse engineered the IBM BIOS, this turned out to be unnecesary: IO.sys was only ever implememented for BIOS.

<comment><USER10> Repeating that this was a reaction to EFI does not make it the case. That remains an ahistorical explanation.

And this despite the change of tack from from naming the firmware to naming the boot mechanism. The name used for the boot mechanism, also adopted long before EFI existed, was a "boot record" or "boot sector", subdivided by type into _volume_ boot records and _master_ boot records, terminology that goes back to the 1980s. The boot protocol was not actually named "BIOS" at all.

IO.SYS/IBMBIO.COM was the Basic Input/Output System, nomenclature (alongside the names of the other parts of the operating system: the BDOS, the command processor, and the housekeeping utilities) that MS-DOS got from CP/M. It was one of two things called that, the other being the machine firmware.

Neither was in any way influenced by something that did not exist until decades later. And although there was confusion between the two, it was not some generalized confusion about parts of the system in general. "BIOS" was not a name for a boot sector/record, even though one of the things _contained within_ a boot record was a BIOS Parameter Block, which people had to regularly explain meant "the _other_ thing that is called a BIOS". And people regularly distinguished in the 1980s between such things as "BIOS services" and the "ROM BIOS".

<comment><USER4> Let's try and find out:

[https://www.pcjs.org/pubs/pc/reference/ibm/5150/techref/](https://www.pcjs.org/pubs/pc/reference/ibm/5150/techref/)

First external PDF, pg 169 first talks about the ROM resident Basic I/O System (BIOS). From that point on, the text refers to this as BIOS, and says that a complete listing is provided in appendix A -- which has nothing concerning to IO.SYS. So it seems IBM considered only the ROM based part BIOS, without the IO.SYS part on the diskette. Even ROM BIOS seems to be a misnomer according to this document. But, as the BIOS sits in ROM and provides services, it is easy to see people talking about ROM BIOS and BIOS services.

As the 5150 has no hard disk, we need the 5160 manual for the HD boot protocol: pg 417 and 419 of [https://www.pcjs.org/pubs/pc/reference/ibm/5160/techref/](https://www.pcjs.org/pubs/pc/reference/ibm/5160/techref/). It reads the first sector from the HD, and checks if the last 2 bytes have a specific value. If yes, it executes whatever it found there.

This de facto defines the MBR as that first sector, without ever calling it MBR. No partitions or Volume boot records exist at the BIOS level, these were purely DOS conventions. See

The Bios Parameter Block was a DOS structure in specific file systems like FAT16. It told DOS how to translate BIOS disk layout (e.g. Cylinder/Head/Sector) to DOS disk layout. See [https://en.wikipedia.org/wiki/BIOS_parameter_block](https://en.wikipedia.org/wiki/BIOS_parameter_block) and note how details vary with DOS versions. Despite its name, the BIOS did not know or care about this structure.

UEFI is a new firmware standard for the PC, intended to replace the BIOS. At this point, the BIOS was used mainly for booting the OS and communicating PC parameters to the OS. Applications did not directly talk to the BIOS anymore - this was a habit from the DOS era. Seeing UEFI and BIOS today as only a boot mechanism is therefore not a 'change of tack'.

I found no references calling IO.SYS part of the BIOS. Maybe this was a CP/M convention?

Of course, the word BIOS did not time travel to after the release of UEFI. But if you need to describe the non-UEFI way with 1 word, what are you going to do? Call it "that stuff the BIOS did when it wanted to load an OS" is a bit long, so boot mode={UEFI,BIOS} seems reasonable to me. This nomenclature is regularly used in the references provided by [https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_In...](https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface)

</comment></comment></comment></comment></comment></comment><comment><USER11> It would be interesting to understand the fundamental difference why ARM systems can even boot without a BIOS and a BIOS chip. Of course they also need to have their DRAM configured, but when compiling U-Boot I don't recall any complication like stackless code. Just that your address space is extremely small in the beginning.

<comment><USER12> IIRC from my Embedded Linux course, ARM has a small amount of SRAM embedded in the chip that is available during boot and is later disabled in favor of the bigger DRAM chips once the controller is initialized.

Uboot is responsible for this in some chips.

<comment><USER11> Yes, ARM has this small amount of SRAM. Having a bit of memory available was such a trivial concept that I did not even think of that the much more complicated x86 processor could just lack it.

But I guess this might not be the only fundamental difference.

(I have thought to utilize the SRAM for some optimization later at system runtime because it should be incredibly fast. At least if you don't have to care about power consumption that should be possible or does the specification require to turn it off?)

<comment><USER12> Well, as the article mentions, Intel uses Cache during Boot as an early SRAM memory, difference seems to be (probably for legacy reasons) that this is not the default.

Most of the boot process is just swinging from one legacy mode to the next until you hit the modern parts.

</comment></comment></comment></comment><comment><USER13> I have a suggestion and a question. The article makes numerous references to "memory initialization" but I didn't see anywhere where its explained what it means to "initialize" DRAM hardware. If I overlooked this I apologize.

My understanding has always been that initializing DRAM consisted of two things:

The BIOs had to enumerate how much physical memory that motherboard had installed. And then to test that that memory is working by writing a bit to each location and reading it back.

Would this be accurate?

Would also be worth noting that many BIOSes allow you to hit the space bar to skip memory initialization presumably because it somewhat time-consuming.

<comment><AUTHOR> no you didn't overlook. the article doesn't discuss actual mechanics of DRAM init, so thank you for adding this info :) i know there is a process of memory training whose aim is to arrive at the right parameters for that DRAM. the way i see it, it is sort of in-field caliberation. boot firmware can then store those parameters inside BIOS chip and then on next reboot just use those parameters, because memory training is a time-consuming process.

</comment></comment><comment><USER14> If the memory initialize code is released as a binary blob, it is called as FSP (which usually is used with Coreboot). Why not just call it as the original name - memory reference code(MRC)?

<comment><AUTHOR> you're right, MRC is a major part of FSP but i think FSP does more work than just initialise memory. it also performs some CPU init and also ICH.</comment></comment>