Why Is This Website Port Scanning Me? - <AUTHOR> https://nullsweep.com/why-is-this-website-port-scanning-me/ <comment><USER2> It's why Tor Browser restricts access to localhost by default. This problem was already predicted and considered by Tor developers back in 2014, see ticket #10419 - _Can requests to 127.0.0.1 be used to fingerprint the browser_ [0] and has been fixed since then. Scanning localhost is a dangerous way to fingerprint the user if there are local open ports.

If you are not using Tor Browser and want to fix the security hole without disabling WebSocket completely, running the web browser in a separate network namespace is a workaround - you get a loopback interface which is independent from the main namespace, and you create a NAT interface within the network namespace to allow outgoing traffic. It's also a possibility for a website to probe other machines, such as the setting page on your router. For better protection, you should block all the local addresses defined by RFC1918 via netfilter/iptables as well.

For developers who needs less restrictive blocking for debugging, you can run multiple Firefox processes in different profiles ( _firefox -P --new-instance_ ), each running in a different network namespace - to make it easy, you can code everything in a shell script and create desktop icons for them. I normally use an ad-blocked and 3rd-party-cookies-blocked profile for web browsing, but a naked Firefox profile for development.

[0] [https://trac.torproject.org/projects/tor/ticket/10419](https://trac.torproject.org/projects/tor/ticket/10419)

<comment><USER3> > It's why Tor Browser restricts access to localhost by default. This problem > was already predicted and considered by Tor developers back in 2014, see > ticket #10419

Sorry to invoke the meme, but Opera did it first[0], in Opera 9.50 (2008). I don't have a good reference to hand, but [1] is a developer complaining about this. [Edit: [2] covers the feature in some detail.]

Opera also blocked access to private IP addresses (so there were three tiers: public IPs, private IPs, localhost; higher tiers could communicate with lower ones, so the block was only unidirectional).

IE10+/EdgeHTML-based-Edge (and I know there was some talk about blocking this in Chromium-based Edge) also blocks it, so that too is prior art to the Tor change.

[0]: [https://w3cmemes.tumblr.com/post/62942106027/if-you-can- thin...](https://w3cmemes.tumblr.com/post/62942106027/if-you-can-think-of-it- its-999-likely-opera)

[1]: [https://stackoverflow.com/questions/1836215/access- to-127-0-...](https://stackoverflow.com/questions/1836215/access- to-127-0-0-1-by-default-in-opera-10)

[2]: [https://web.archive.org/web/20140302021701/http://my.opera.c...](https://web.archive.org/web/20140302021701/http://my.opera.com/securitygroup/blog/2012/07/03/operas- cross-network-protection)

<comment><USER3> To add more about why current browsers don't do this:

One is clearly that you need to communicate the requesting IP deep enough into the network stack to the point where you get the DNS response (if there is one), which means there's a fair bit of work to ensure this is done everywhere;

Another is it's known to break corporate websites ([https://internal.bigcorp.com/](https://internal.bigcorp.com/) on a public IP expecting to be able to access private IPs on their intranet), and smaller browsers are reluctant to lose corporate users by changing long-standing behaviour;

Then there's the WebRTC question: if two users on the same local network start a WebRTC call, do we want all this traffic to go to via first point where the public IP is known? For the majority of users the answer is no. And as long as that's possible, there's a (very limited) means of local communication. (But if it's limited to only things that respond to the WebRTC handshake, we're already in a much better place!)

<comment><USER4> In Kazakhstan we have e-government website. This website allows users to use crypto-tokens to access government services (every citizen can get a digital certificate representing his identity).

This website used to run Java applet. This applet was signed and it could access restricted APIs to access USB device. So website talked to applet and applet talked to USB device to sign data.

After major web browsers disabled Java applets, they implemented another approach. Now user must install a software which runs a web server on 127.0.0.1. This webserver listens on a specific port and uses web socket to communicate.

So government website now uses JavaScript to connect to 127.0.0.1:12345 using websocket. And then it uses that connection to interact with USB device.

So an ability for external website to connect to 127.0.0.1 actually is essential for this use-case.

My guess is that there are plenty of other websites which use local web server to interact with locally installed software. I know at least one another such a website: Blizzard website. It runs web server in its game launcher and website can communicate with it.

PS also they have to install custom trusted certificate because browser requires wss from https and there's no easy way to get a legitimate certificate for that kind of use.

<comment><USER5> > So government website now uses JavaScript to connect to 127.0.0.1:12345 > using websocket.

It sounds like random other websites (Ebay, etc) would be able to interact with people's USB devices this way too. Maybe without people knowing?

<comment><USER6> Yes, if this is programmed badly (missing security or a security hole).

The browser connecting to the government website accesses two servers: the original one and the second local one you install yourself on your system. The local server runs natively and therefore can access the USB device. Like all servers it should be programmed such that misuse by hackers is prevented.

<comment><USER7> That's already a security hole.

The only thing missing is a rogue website abusing it.

There's no guarantee you will never connect to any rogue website that abuses this government mandated backdoor.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> The greater issue is that browsers are allowing code executing from the public Internet _scope_ (scope meaning security domain) network access to the localhost scope or the Intranet scope (RFC1918 addresses.)

If anything, this should require very explicit permission granting from the user. I'd prefer it be something more like an undocumented toggle accessible solely to developer types.

<comment><USER9> There are legitimate reasons for port scanning, but I'm not sure most websites out there are using it for noble purposes. I guess browsers could allow it based on explicit permission from the user, just like it's already done for microphone and camera.

<comment><USER10> I'm curious, what would be a good reason to do this? I'm not creative enough to think of anything this enables a site to do that isn't malicious. If I'm running a service on localhost, and that service needs to communicate with the site I'm browsing, surely I could just direct that service to communicate with the site itself.

For instance, if I'm running a local chat application and need it to communicate with the web version, why does the website need to be able to port scan to accomplish this? I can think of other ways to accomplish this that are a lot more secure.

<comment><USER11> Ubiquiti routers have a fairly magical browser SPA that can run on their domain and talk to local routers. It involves webrtc connections to local addresses.

But I think if same-origin were enforced more strictly, they could have found another way.

<comment><USER12> Huh, I never looked but always assumed this was proxying through the controller.

<comment><USER11> It does this most of the time, either through the cloud or direct to the controller. But during setup of the first device on a network it does something direct from the browser to get it connected to the cloud.

</comment></comment></comment></comment></comment></comment><comment><USER13> > Port Scanning is Malicious

Though port scanning can be (and maybe even frequently is) done with malicious intent by looking for misconfigured/bugged servers, I disagree that it's inherently malicious. Port scanning is just about checking to see what services a host is offering you. It's like going to a random shop at a mall and asking what services they provide. Would asking about their services be malicious?

It feels like the reason asking about services is considered malicious is because shops frequently give out info to the public that they shouldn't have. It's like:

client: What services do you provide?

shop owner: Well, I can provide you with a list of all my clients along with their personal information they entrusted to me.

So, is the client being malicious for asking or is the shop owner the one that was in the wrong for mistakenly providing that info to the public?

I feel the only reason we don't blame the shop owner is because even though he's the one that mistakenly discloses private info, sometimes he's just following a script written by a random programmer unassociated with him. Maybe the response was a mistake on the programmers part, maybe it was a mistake in how the shop owner used the script (a configuration error). In the end, it's simpler to blame the client for asking out-of-the-box questions (after all, most clients just come in to ask if you're giving out flyers/pamphlets because that's what everybody does) and so they don't feel responsible for the response that results.

I can provide a shop that also offers things different than http(s) with open access to the public. It shouldn't be a crime/violation to ask me if I offer them.

<comment><USER14> I think the dynamics of the Internet have shifted from the early days. Basically, HTTPS on port 443 is pretty much the only service that anyone intends to make publicly available. This is different from 30 years ago, when those same sites had HTTP, FTP, Gopher, a public Telnet server, a public NTP server, etc. and they wanted you to use them. It was very reasonable to look around back then, but nowadays anything that is available publicly is probably an accident.

<comment><USER13> Exactly! And do we want to continue on that trend? Personally, I don't.

I dislike the growing idea that HTTP is a core part of the internet, and not just the most popular part. The difference lies in if we're going to see legislation that dictates proper use of the lower networking layers like TCP/IP by stuff of the upper layers like HTTP. I'd really hate to see something along the lines of "it's illegal to use a TCP port unless it was specified as available to the public in some (possibly js-rendered) part of an HTTP response."

<comment><USER14> I don't think it's worth getting caught up on which data framing protocol everyone is using. Everything that Gopher, IRC, FTP, etc. did are perfectly expressible as any other RPC protocol; these things were just RPCs before we invented the term RPC. Now we have protocols that can generically transport any RPC, and so we don't need to think about these things in terms of port numbers or running services.

</comment></comment></comment></comment><comment><USER15> This raises the question: Is port scanning without consent a violation of the CFAA? Either it is legal, and researchers should face no repercussions for doing so, or it isn't and eBay is non-compliant with CFAA. I recall hearing about someone either being arrested or convicted due to port scanning a courthouse, but it was many years ago and I can't find the case with a cursory Google search.

I have to wonder what value eBay would get from port scanning its customers. Is it part of an attempt to detect bots/attackers? Is malware running on their server trying to determine if the client is likely vulnerable to some propagation method?

<comment><USER16> Was it this? [https://news.ycombinator.com/item?id=21023023](https://news.ycombinator.com/item?id=21023023)

<comment><USER15> No, I think it was probably close to a decade ago, but I likely am misremembering some of the details. Could've been a police department, but I'm not sure.

That one you linked is a messed up case. There is a phenomenal podcast that interviews those guys and walks through their engagement. [https://darknetdiaries.com/episode/59/](https://darknetdiaries.com/episode/59/)

</comment></comment></comment><comment><USER17> > Furthermore, when I installed and ran a VNC server, I didn't detect any > difference in site behavior - so why is it looking for it?

Not an eBay employee, but used to work in fraud detection. Two very obvious related guesses from my experience:

1\. Fingerprinting a user to help identify account takeover (ATO). Open port signatures is probably a pretty good signal for that kind of thing (and it doesn't seem to be measured in [https://panopticlick.eff.org/](https://panopticlick.eff.org/)).

> However it is also a valid tool used by administrators for remote access to > machines, or by some end user support software, so the presence of VNC is a > poor indicator of malware.

2\. In a Bayesian sense, this probably isn't right. I don't know what eBay's traffic looks like but I'm willing to bet that all other things being equal, traffic coming from a machine with an open VNC port is riskier. Fraud detection is a game of probabilities, so the existence of a valid user showing a particular characteristic doesn't mean that the characteristic isn't useful in a fraud model. The example I always give is that when I was doing this (quite some time ago), we could have had a 99% accuracy rate for a simple rule banning IPs from Turkey, Ghana, Nigeria, and Vietnam. It's not because there weren't any valid users from those countries, it's just that the fraudsters where overwhelmingly likely to be using IPs from those countries.

<comment><USER18> Those four are still considered untrustworthy, and I've had to add India, Ukraine, and Brazil to the list of nations I filter entirely.

</comment></comment><comment><USER19> Port scanning from a web page, combined with DNS rebinding, can present a really nasty attack, and can effect an entire private network, not just localhost.

Some more info here: [https://medium.com/@brannondorsey/attacking-private- networks...](https://medium.com/@brannondorsey/attacking-private-networks- from-the-internet-with-dns-rebinding-ea7098a2d325)

Example code: [https://github.com/brannondorsey/dns-rebind- toolkit](https://github.com/brannondorsey/dns-rebind-toolkit)

A malicious DNS rebind server: [https://github.com/brannondorsey/whonow](https://github.com/brannondorsey/whonow)

Disclaimer: I performed some of this research a few years ago. So those resource suggestions are my own, but they feel very relevant here.

</comment><comment><USER20> First of all, fraud detection seems like a legitimate use case here. And WebSockets has many valid uses.

 _HOWEVER_ \-- how the hell is localhost port scanning allowed to happen _without my permission_?!

This feels no different from a website trying to check the existence of named directories on my file system or something.

Does WebSockets not require permission to function at all, or shouldn't it be limited to some kind of CORS-type policy or _something_ to connect without a permissions dialog? Or _even_ if it's allowed to port scan the entire public internet, at least block your local machine and network without explicit permission?

<comment> ryan-allen If you find a way to prevent this in Chrome/Edge please let me know.

Edit: [https://defuse.ca/in-browser-port-scanning.htm](https://defuse.ca/in- browser-port-scanning.htm)

There doesn't seem to be a way to access anything locally, just test for open ports. I use SSH tunneling a lot and was having a minor freak out.

</comment></comment><comment><USER21> This use doesn't seem to be covered by eBay's privacy policy [https://www.ebay.com/help/policies/member-behaviour- policies...](https://www.ebay.com/help/policies/member-behaviour- policies/user-privacy-notice?id=4260)

</comment><comment><USER22> Lots of chat in the comments about how this is all websockets' fault, but don't forget you can portscan localhost with pure JS as well.

[https://portswigger.net/research/exposing-intranets-with- rel...](https://portswigger.net/research/exposing-intranets-with-reliable- browser-based-port-scanning)

<comment><USER23> Ach! That's diabolical.

<comment><USER3> Timing attacks make it very hard to prevent port/host probing generally, sadly, with the sheer number of things that are observably loaded cross-origin (iframes in that example, but also images, scripts, stylesheets…).

(In the private/loopback IP ranges we should really just make those requests always fail, but I addressed that in another comment as to why that's not trivial.)

<comment><USER22> Private and loopback space should really be outside the sandbox, or at least in a permission. I'm happy with mycorp.net accessing 10/8 space, but not ebay.

</comment></comment></comment></comment><comment><USER24> Every time I hear about some shiny new feature being added to a browser, I think...

1) Will I ever actually use this

2) How is this gonna screw me over

WebSockets, WebBluetooth, WebAssembly, Web-You-Can-Access-my-Accelerometer- and-Battery, haven't ever wanted to use those. Ever. For anything. For any reason. (Edit 3: Oh yeah, I forgot! WebRTC!)

Edit: Fantastic. You can't disable it in Firefox. So what, does Firefox need a freaking iptables implementation now? [1]

1 - [https://bugzilla.mozilla.org/show_bug.cgi?id=1091016](https://bugzilla.mozilla.org/show_bug.cgi?id=1091016)

"The only theoretical reason for the WebSocket pref these days is the possibility to disable it easily in case there is a security issue found in the protocol itself or so."

The protocol itself is the security issue. ALL OF IT.

Edit 2: So I don't have the time to investigate every new fad when it comes out. I originally thought WebSockets were raw sockets, but they aren't. Firefox blocks access to port 22 -- I was hoping all privileged ports, but it seems just those. Opening a WebSocket to netcat dumps out a HTTP request, so it seems unlikely that you'd be able to talk with anything that doesn't talk HTTP and WebSockets. Firefox also seemingly blocks access to 192.168/24 and 10/8.

This makes me less angry. But what STILL make me angry is that I have to sit and research about some stupid thing that I don't want and can't turn off. Sooner or later, some web dev is gonna argue that all sites should be loaded over WebSockets because his bloated javascript stack performs marginally better, and then WebSockets _won 't_ be something I can turn off. Websites will just whitepage.

Edit 4: Done researching this now. I went to ebay on Firefox, and wasn't getting websocket scans. But I've got a stack of uBlock and NoScript... maybe that's interfering with it some how? Opened up a stock config for google- chrome -- that's my browser for "some dumb new web tech that isn't working in Firefox" \-- not seeing any scans when I open up inspector and click "WS".

Regardless, his point still stands. You can totally use WebSockets as a port scanner for localhost, assuming the Content Security Policy allows it. Now I gotta go update my nginx configs...

<comment><USER25> Websockets are nice for some things. I hack on Mastodon and it uses WSS for streams and they're very helpful.

But WebBluetooth, ASM, etc are all fairly insane. WebRTC feels like a massive security issue (I've seen a demo of someone using WebRTC to find computers on an internal network at a security conference years ago. Even if that hole is fixed, it's still a hacky solution to video streaming behind NAT).

I agree; most of this stuff needs to have ways to disable it, in the base configuration screen of the browser (not hidden somewhere in about:config).

<comment><USER26> Why is ASM insane? Are you talking about WASM? That's got the same security model as JavaScript.

<comment><USER27> WASM is great peace of tech but I can't help to think it would be abused a lot in the future. For example, right now we can use ad blocker to block ads and analytics by blocking its js from loading. Imagine when wasm gains mainstream popularity and ad companies begin to ship their ads and analytics product as a libaries to be linked at compile time. How do we block something like that? Sure the adblocker can hide the relevant dom contents, but the code is still run and doing whatever it want on your browser.

<comment><USER26> These kind of complaints are based on a misunderstanding of how JS works or how the browser works.

You can do the same exact thing in JS _right now._ In fact, if anything, JavaScript makes this way easier than WASM. With JS, you can just use something like Rollup or Webpack to put your analytics code in the same code.

<comment><USER27> Yeah but using webpack is not how majority of websites deployed, so it's probably not worth the effort for ads companies to support it. They will consider this when webpack/wasm become mainstream enough (approaching 50% web), which may or may not happen. Probably won't happened but the thought always linger in my mind.

<comment><USER26> WASM is way less popular than Webpack.

Almost every site with some kind of front-end framework like React will use a bundler of some kind.

</comment></comment></comment></comment></comment></comment></comment><comment><USER28> Browsers should be blocking this by default.

"This website is trying to access services on your local PC, do you want to allow?"

Or at least as blockers should have a rule for it.

</comment><comment><USER29> My kids complained today that Google Classroom isn't working. After a quick investigation, I noticed that Snort on my firewall blocked the relevant Google server due to incoming TCP port scans. Sigh.

<comment><USER30> Be careful with automated rules - unless it's a full TCP handshake, you can't conclusively identify the source of a port scan as the IP may be spoofed. If someone port scanned you and spoofed eg the IPs of your DNS servers, you've self-DoSed yourself.

</comment></comment><comment><USER31> To my knowledge, a lot of effort has been put into the design of CORS (and related APIs) to specifically prevent misuse like that. A well-behaved Websocket implementation should not give the calling script any indication _why_ a connection failed.

I know timing oracles are difficult to avoid in many cases - but the technique shown here seems to actually exploit different kinds of exceptions being thrown by the browser.

This seems like a straight-up bug and pretty serious security vulnerability to me.

<comment><USER32> cors/csp allow the webpage owner to control what servers the javascript running on on their webpage can access and allow web servers to control what 3rd party websites make requests to them.

Notice the missing piece? Neither of those allow the _user_ to control these things.

At the end of the day, it is reasonable to assume that localhost access is a valid security barrier in the general networking sense. Making an exception for certain types of networked applications is just adding a pitfall for some dev to fall into. Good process design has to take into account the inevitability of human error, and leverage things like "forget safe" rather then "remember safe" (forgetting a step should fail safely, with an error or incorrect but still safe behavior, then unsafely, with an exploit or an explosion)

Using websockets or XHR to transverse internet firewalls is browsers transversing security barriers as a feature, and needs to go the way of the mic access, with a per-site prompt.

</comment></comment><comment><USER33> If anyone thinks of implementing this, don't forget to guard against reflection attacks[1]

EDIT: revisiting my comment (and the wikipedia article linked), a reflection or amplification attack in this context is sending traffic and generating (perhaps much more) traffic from a different source than yours as part of an attack. For example, you could spoof the IP address of the HTTP packets and cause the server to port scan another machine -> little traffic (HTTP request) causing a lot of traffic (port scanning). As part of a DDOS attack, a botnet for example could use this to amplify their attack and masquerade the source.

[1] [https://en.wikipedia.org/wiki/Denial-of- service_attack#Refle...](https://en.wikipedia.org/wiki/Denial-of- service_attack#Reflected_/_spoofed_attack)

</comment><comment><USER34> Why is localhost / 127.0.0.1 allowed from a remote JS file without any permissions?

</comment><comment><USER35> Potentially you can do more than just port scan; it's possible to use/access the servers that you have running on your local machine if they're left open. See my post about this: [https://jameshfisher.com/2019/05/26/i-can-see-your- local-web...](https://jameshfisher.com/2019/05/26/i-can-see-your-local-web- servers/)

</comment><comment><USER36> From the title I assumed this was going to be something else. I remember some sites used to port scan you on registration. This was to check if registrations were from an open proxy, which was a very strong bot indicator. I might be misremembering but I think Slashdot used to do it. There were also some plugins for phpBB forums that did it too. I used one back in the day and it helped quite a bit with spam registrations.

<comment><USER37> How did port scanning work back then before WebSockets?

<comment><USER30> These were external scans - see if the visitor's IP is running an open SOCKS or HTTP proxy for example. Many IRC networks still port scan you on connect for the same reason.

</comment></comment></comment><comment><USER38> Is there a way to block this at the browser level? Ex: block access to localhost for all domains (except from localhost itself)?

<comment><USER39> Ublock allows you to block websocket requests. eg.

        *$websocket

will block all websocket connections. You probably want to operate on a whitelist on a site by site basis. Blocking localhost or 127.0.0.1 isn't reliable because sites can use dns rebinding attacks to bypass your filters.

<comment><USER40> Ublock origin says it supports ABP filter rules which allow for whitelisting sites which seems like it should allow something approximating:

      ~site.com$websocket
      *,~site.com$websocket

However this seems like it's invalid syntax because switching your example to this opens all websocket use back up tested via [https://websocketstest.com/](https://websocketstest.com/)

<comment><USER41> That is not valid ABP filter syntax. This is what you want:

        *$websocket,domain=~site1.com

For more than a single site:

        *$websocket,domain=~site1.com|site2.com|...

I would personally suggest people to just enable advanced user mode and create rules such as:

        * 127.0.0.1 * blocked

To block all request attempts to 127.0.0.1.

<comment><USER40> Thanks for the information and all that you do. I had found the linked section from UBO to [https://help.eyeo.com/en/adblockplus/how-to-write- filters#el...](https://help.eyeo.com/en/adblockplus/how-to-write- filters#elemhide_domains) and assumed that *$websocket was <all_domains><separator><rule> similar to how other filters seem to work but I haven't delved too deeply in the rule syntax. Appreciate the correction.

</comment></comment></comment></comment></comment><comment><USER42> This is bad and should be blocked IMO, at least by default, but can a site do anything other than find out which ports respond to a websocket request? AFAIK they can't send arbitrary network packets. The websocket will only open if the port they are trying to talk to speaks websocket back. This is mentioned in the article.

I'm not saying that's okay. I still don't want them scanning ports on my machine. There might be some services that offer a websocket connection like Plex for example, or the Kinect driver, or Leap Motion. I also don't want them cataloguing ports that are open.

</comment><comment><USER43> I can't believe of the 363 comments no one has mentioned Samy K and his awesome Poisontap project. Parts of which did this local scanning and connecting to your internal router management page.

[https://github.com/samyk/poisontap](https://github.com/samyk/poisontap)

See also, [https://www.theregister.co.uk/2010/01/05/geo_location_steali...](https://www.theregister.co.uk/2010/01/05/geo_location_stealing_hack/)

</comment><comment><USER44> If the bank is checking on my security then its reasonable for me to check the security of the bank, right?

<comment><USER45> Check the bank EULA. You might have agreed to the scans without knowing.

</comment></comment><comment><USER46> > Furthermore, when I installed and ran a VNC server, I didn't detect any > difference in site behavior - so why is it looking for it?

I think behind the scenes they keep log of some sort of fraud risk, e.g. geoip different from billing country, suddenly a new operating system, vnc/teamviewer running would probably flag your account (even for benign purposes, e.g. you can get your money back or purchase cancelled if that info can prove your transaction was actually unauthorized).

I worked on a ecommerce where the previous developers implemented a rudimentary "score" system like that so that suspicious orders would be put in queue for phone verification (this was pre gdpr)

<comment><AUTHOR> That makes a lot of sense. I assumed it was somehow for anti-fraud, though I still don't like it.

</comment></comment><comment><USER47> Port scanning localhost from a webpage has been possible for a long time and does not require websockets.

[http://jsscan.sourceforge.net/](http://jsscan.sourceforge.net/)

</comment><comment><USER48> I don't follow what this has to do with websockets specifically; they just go over HTTP, so why couldn't you do this with a regular HTTP request?

Either way it seems easy to mitigate at the browser level: block all requests to localhost that don't originate from a page served on localhost. It's not that different from the CORS policy.

</comment><comment><USER42> Yes, a drive by web page shouldn't be able to do this but similarly a native app shouldn't be able to do this and yet I suspect some not insignificant percent of native apps, especially on mobile on both OSes are doing this either directly, the app dev is doing it deliberately, or via one of the many 3rd party libraries they included but aren't aware of the behavior.

I really want the OS to prevent this by default and require permission from the user. I want apps (probably only possible on iOS/Android) to have to list the sites they'll connect to, that list will have to be reasonably small 10-30 sites with special exceptions for browsers

This would have 2 positive affects. #1 it would prevent the apps from scanning the network. #2 it would effectively force apps to launch the user's browser for external links instead of an embedded browser in which they can spy on all activity.

</comment><comment><USER49> I don't think motivation is malware detection, I am assuming this is sort of fraud detection (like carding)

</comment><comment><USER50> Interesting, port scanning is illegal in some countries as it's classified as security testing, it can be only performed with permission.

How would you feel is someone was walking on busy car parking and checking if doors of the cars are open? It' what port scanning is, checking if the car has open door.

<comment><USER51> > How would you feel is someone was walking on busy car parking and checking > if doors of the cars are open? It' what port scanning is, checking if the > car has open door.

More like sending a "hi, can I enter?" signal to a self-driving taxi that has been left waiting in a public arena.

Don't put a server online with a public IP if you don't want to receive those signals. Don't send "hi yes you can enter!" responses when you get the query, if you don't want to let people in.

<comment><USER52> Except in the case described by the article, the port scanning is being done not on servers on the public internet but on clients of certain websites.

</comment></comment></comment><comment><USER53> Curious what HN thinks about this hypothetical: Imagine you have a web app designed to talk to a specific backend server API. It's also common for users to run instances of the server on their local machine. How would you feel about the app checking a (single) well-known port to see if there's a local server running, and prompting the user: "we detected you're running a local copy of the server, do you want to connect to it?"

This doesn't seem to be done very often, and the public cases usually seem to be pretty ugly (Zoom). But I could see it being useful. Imagine for example an app for browsing S3 directories, that could also detect if you're running a minio server and allow you to connect to it, and transfer data back and forth between your different backends.

<comment><USER54> I don't think the case that you're describing is unethical, but I also don't see it as beneficial enough to outweigh the security risks of it being possible.

</comment></comment><comment><USER55> This slideshow by an NSA dude seems to go into this, from 2016.

[https://datatracker.ietf.org/meeting/96/materials/slides-96-...](https://datatracker.ietf.org/meeting/96/materials/slides-96-saag-1/)

</comment><comment><USER56> There is an open Chromium bug for this: [https://bugs.chromium.org/p/chromium/issues/detail?id=378566](https://bugs.chromium.org/p/chromium/issues/detail?id=378566)

I hope they consider it still valid and not close it.

These are the blocked ports: [https://github.com/chromium/chromium/blob/83.0.4103.53/net/b...](https://github.com/chromium/chromium/blob/83.0.4103.53/net/base/port_util.cc#L22)

Accessing localhost and LAN addresses works perfectly fine, except for those ports.

I am going to patch Bromite so that it doesn't allow any access to localhost nor private networks.

<comment><USER56> Interestingly enough they are already blocking these attacks for background requests, see [https://github.com/chromium/chromium/blob/83.0.4103.53/third...](https://github.com/chromium/chromium/blob/83.0.4103.53/third_party/blink/renderer/modules/background_fetch/background_fetch_manager.cc#L251)

Perhaps they simply forgot to cover also the WebSockets case, or the discussion on the related bug was not allowing for expanding the coverage.

</comment></comment><comment><USER57> This does suggest to me that browser websocket requests against localhost should at least:

1) return the same error message for all failures (unless some opt-in / launch flag is set)

2) fiddle with the timing slightly to make timing attacks less useful? (how long is a localhost TLS connection? 100ms? I think devs can wait a handful of frames for their failure response.)

I have no idea how many legitimate apps are leveraging some kind of localhost connection -- it sounds like an unusual use case but I can certainly imagine some enterprise app that ties into desktop services or programs by that route.

EDIT: Of course banning them outright or requiring specific user whitelisting of domains would work as well. Just trying to get away with the smallest change.

</comment><comment><USER58> Many questionable Russian sites do full port scans not only on localhost but on all the private subnets. I had to block all access to ports above 1024 for all local subnets. Usually people don't have firewall rules for that.

</comment><comment><USER59> Allowing ws connections to local addresses can be pretty useful in many cases (admittedly, many of these could be better solved with WebExtensions' native messaging) so disallowing it would not fly.

But since this is pretty rare, a message saying: "this website is trying to connect to services running on your computer - allow/deny?" would be pretty easy to implement and solve this for good. Sites that need this already require you to jump through hoops, so one more popup would be fine, but sites that do this for other reasons would probably not want to risk a popup.

</comment><comment><USER60> See also: BeEF[1]

Theres lots of scanning/attacks you can do using the web browser as your scanning tool. Its troubling that major sites are starting to use some of these techniques, but these techniques have been readily available to attackers with open source tooling.

I think it's long overdue for browser to find a way to mitigate these sorts of attack vectors. If the security folks can't justify it due to BeEF, maybe the privacy folks can using articles like this.

[1] [https://beefproject.com/](https://beefproject.com/)

</comment><comment><USER61> When you do this:

        new WebSocket("ws://127.0.0.1:8080")

An application listening on 8080 is indeed getting a packet delivered.

Run this to see the packet:

        nc -lp 8080

And the page can figure that out via the error returned.

I wonder if that is in line with the same origin policy.

On the other hand, maybe the same is possible by creating an image with src="[http://127.0.0.1/hello.jpg"](http://127.0.0.1/hello.jpg") and looking at the onload/onerror event?

<comment><USER62> But note that a simple

        <img src="http://127.0.0.1:8080">

should do so as well, ie this is just the newest iteration of an old problem that the major browser vendors never chose to properly address - with the exception of Opera, see eg this stackoverflow question [1] and corresponding answer from 2011.

[1] [https://stackoverflow.com/questions/5464599](https://stackoverflow.com/questions/5464599)

<comment><USER61> The page could never read the contents of the image, right?

If the user has a web-socket-server running (for example because he is a developer) could the page read from it?

Can a page read from _any_ web socket server on the internet?

<comment><USER62> _If the user has a web-socket-server running (for example because he is a developer) could the page read from it?_

Only if that server chooses to accept the request, which it can decide based on the Origin header.

Personally, I was more concerned with getting spurious requests on ports bound to 127.0.0.1 (which I've been using for IPC), but that issue already existed before the introduction of WebSockets.

WebSockets of course do make things like port scanning easier, but as others have pointed out, you could already do that with a bit of ingenuity eg through tracking response times.

</comment></comment></comment></comment><comment><USER63> hmmm, so the conclusion is:

"Whether the port scan is used as part of an infection or part of e-commerce or bank "security checks", it is clearly malicious behavior and may fall on the wrong side of the law."

Though I really don't know what ebay or banks or any site might be doing, it seems like it's almost certainly a defensive thing looking for signs of trouble. I don't know if I'd call it malicious. Isn't this totally harmless in this case? That is, eBay portscans me, how is this malicious?

<comment><USER64> You don't know whether they're collecting the data and running analysis on it. What services you're running may already reveal something about you.

</comment></comment><comment><USER65> This guy Just has it wrong when he calls port scanning an adversarial technique. It's Just a way to discover Services. You can then use the result to do malicious things but it's not like the only or even main purpose. I humbly refer to this: [https://koeln.ccc.de/ablage/portscan- policy.xml](https://koeln.ccc.de/ablage/portscan-policy.xml) (Google translate can help with the german)

</comment><comment><USER66> This is scary. I've always left locally running services unprotected for convenience given they can't be accessed from outside. I can imagine a lot of people running local apps, servers or databases without any auth that could contain sensitive information. Would a webpage be able scrape data from such services? Any way to disable this completely in Firefox and Chrome?

<comment><USER67> No, webpage javascript is limited to using websocket protocol [1] for connections. That means your database or IP camera, or VoIP phone or router are safe for now. Though the websocket connection establishment seems to allow the javascript to differntiate between a closed and an open TCP socket and a TCP socket that speaks websocket.

[1] [https://en.wikipedia.org/wiki/WebSocket](https://en.wikipedia.org/wiki/WebSocket)

<comment><USER66> So if a local service allows WS connections, can data be scrapped off such a service?

<comment><USER67> Yes, the primary (or only) reason to even implement a WS server connection is exactly to allow data to be scrapped off using a web-browser.

E.g. Asterisk nowadays allows enabling SIP protocol access over websocket so that you can run a javascript VoIP client from inside a browser [1] (and WebRTC for the media layer).

[1] [https://wiki.asterisk.org/wiki/display/AST/Asterisk+Builtin+...](https://wiki.asterisk.org/wiki/display/AST/Asterisk+Builtin+mini- HTTP+Server)

<comment><USER66> Sure but such servers would always implement authentication when deployed to the web but a lot of such services could run locally unauthenticated to serve local apps. For example services to power electron apps. This tells me data from such services can be stolen very easily.

<comment><USER67> Electron apps have much more access permissions than a normal website. I don't know technical details of how electron apps interact with the bundled chrome, however this post [1] suggests that electron apps can just talk low-level TCP protocol, no need to wrap everything in an additional websocket protocol layer.

[1] [https://stackoverflow.com/questions/41674063/is-it- possible-...](https://stackoverflow.com/questions/41674063/is-it-possible-to- create-a-tcp-client-with-electron)

</comment></comment></comment></comment></comment></comment><comment><USER68> I can think of a legitimate use case for this. If you watch some of these scammer youtube videos, one common thing they seem to do is get on a screensharing application, and have the user log into their bank account. From there, the scammer inspects the html, and manipulates the values to trick the victim.

A bank knowing if someone else is watching your screen is a decent security measure.

</comment><comment><USER69> This is why the websocket implementation does not have meaningful error codes, so people can not abuse it. But they still do.

</comment><comment><USER70> in firefox it seems you can disable websocket with network.websocket.max- connections = 0

Firefox and the illusion of privacy

(1)[https://www.remembertheusers.com/2018/03/0455-firefox-and- th...](https://www.remembertheusers.com/2018/03/0455-firefox-and-the-illusion- of-privacy.html)

</comment><comment><USER71> How does it work in practice? It seems in Chrome these errors cannot be caught try-catch blocks.

        try {
          var socket = new WebSocket('ws://localhost:808');
        }
        catch (ex) {
          console.log(ex) // control does not reach here
        }

<comment><USER72> It probably fires as an async error, I'd expect this would log it (if inserted at the end of the try block):

        socket.onerror = (...args) => console.log('async error', ...args)

<comment><USER71> socket.onerror event happens for both cases. But there does not seem to be any difference between the error object that is passed to the these handlers.

</comment></comment></comment><comment><USER73> Actually , I'm pleased ebay is doing this. It wasn't a new issue but now ebay doing it, it took a lot of attention. It's like disclosing a security issue in WebSocket protocol. Now I'm sure next releases of the most browsers will fix it.

</comment><comment><USER74> Javascript was a mistake, seriously. The benefit-cost ratio from the user's point of view is disastrous. I'd rather slog through less fancy data entry forms than suffer endless tracking, privacy and security attacks.

</comment><comment><USER75> Please ELI5 why it doesn't happen for www.ebay.com using Firefox in Debian. I see no websocket connections to localhost in Network Monitor or iftop.

</comment><comment><USER76> I've noticed many of the survey for cash sites use websockets to scan for running services on your machine Personally I think its straight up evil

<comment><USER77> Those are natural abuse (automation) targets so always used a lot of tracking.

</comment></comment><comment><USER78> I see the port scanning behavior in Firefox and Chrome but not in Brave, even with Brave shields down. Anyone else use Brave?

</comment><comment><USER79> So if disable javascript - it will not be possible?

</comment><comment><USER80> Now scanning for open ports I can do on Linux.

But how would I go about monitoring which ports are being scanned on Linux?

No tool doing this comes to mind

</comment><comment><USER81> See also, [http://localrouter.net/](http://localrouter.net/)

</comment><comment><USER82> Did they check the source of the request? My guess would have been an extension doing this instead of the site.

</comment><comment><USER83> I just tried myself and could not find any such thing. MAy be a bug or removed after seeing it featured on HN?

</comment><comment><USER84> Is there a setting or something in either chrome or FF to block websites from being able to port scan you?

<comment><USER85> Supposedly you can disable websockets in Firefox by setting network.websocket.max-connections in about:config to 0.

</comment></comment><comment><USER86> How do I prevent this? Is there something I can do to block localhost (RFC 1918) port scanning?

</comment><comment><USER87> Apropos past issues with Zoom installing a local server, this is important to consider.

</comment><comment><USER88> Damn. Yet another example of "This is why we can't have nice things."

</comment><comment><USER89> Poorly written article: mixes facts and opinions

> it seems many sites are port scanning visitors for dubious reasons.

Claims that in the intro, but then admits ebay is scanning for VPNs, which probably means it's doing fraud detection, which is definitely not a dubious reason, and is probably actually beneficial to the customer.

</comment><comment><USER90> I downloaded Brave for the first time today after reading this.

</comment><comment><USER91> anyone know the config or flag in Chrome to disable any requests to localhost? Ideally excluding origin=localhost, but if not possible i can dev on a different account

</comment><comment><USER92> Any suggestions of ways to block this? Any Safari extensions?

</comment><comment><USER93> eBay must have some logic to determine who to scan. I can't get it to trigger on my windows desktop or mac.

</comment><comment><USER90> Downloading Brave Browser right now...

</comment><comment><USER94> If you want analogies, this is like walking into a bank to do business and the security guard checking to see if you're wearing a mask.

<comment><USER95> People are good at inventing analogies that support their point of view.

</comment></comment><comment> awinter-py TLDR because you have javascript enabled

</comment><comment><USER96> port scanning is fine and should not be illegal. It's just "looking" at a house to see if there is a door and what type of key (protocol) it uses.

Trying to open a connection on the other hand it's like trying to open the door. That should be considered as a violation.

<comment><USER97> Don't try to understand this with metaphors. It's a trap. Port scanning isn't enough like anything in the real world for the analogies to apply.

I'd strengthen pfundstein's claim; port scanning intrinsically works by trying to open connections. That isn't enough "like" any particular physical thing to make it a correct analogy; it isn't knocking, or walking in, or opening, or anything else. But one thing we can say, without using analogies, is that it is definitely an _active_ effort, an action deliberately taken, not something passive like "looking" is in the real world. That is not on its own proof that it is wrong... I am merely saying, it is certainly _active_ , not passive.

<comment><USER98> I think your final objection fails by bad analogy too: one has to _actively_ direct ones gaze, in general, if one is to notice a type of lock or other security arrangements.

<comment><USER97> I'm not analogizing to "actively looking"; I am saying, it is an active action. You can tell it's active because if they don't make deliberate decisions to write code that performs this scan, no scan will happen. They have had meetings about this functionality, and implemented it, and tested it, and management has signed off on it, and in a place like eBay quite likely their legal department has signed off on it. It is an action they have taken, with deliberation and intention; it is not a thing that just suddenly started happening to them one day, like, Firefox shipping a new browser that has a new default font or something.

I'm referring to the literal, probably-hundreds-of-person-hours actions taken to create this functionality. This is relevant to both ethical and legal analyses. No analogy.

</comment></comment></comment></comment><comment><USER99> Port scanning isn't malicious behavior. Port scanning is about equivalent to walking down the street and looking at the architecture of the buildings.

<comment><USER100> Your analogy might apply to a port scanner running over the Internet and looking at what ports are open on Internet-facing servers. (Though I would still argue the analogy is flawed there: port scanning Internet-facing servers is more like going up to each locked door on the street and writing down what kind of lock it has, in case you want to try to pick it later.)

But a port scanner running inside the browser on my local machine is equivalent to someone sneaking into my house and going through each room seeing what valuables are there.

<comment><USER99> I agree. It's foolish to give websites permission to do that kind of thing. There's a very simple solution: don't give them permission. Turn off javascript. Yes, ebay will complain but it'll still work. The power is in your hands. No one is forcing you to use eBay either.

Giving arbitrary websites the ability to run arbitrary code on your machine is just asking for trouble. It's like someone who opens and executes every email attachment they receive. Try out NoScript temp-whitelist only mode that blocks by default and requires manual permission giving.</comment></comment></comment>


Electric car threatens European car industry – ING – Kennis over de economie - <AUTHOR> https://www.ing.nl/zakelijk/kennis-over-de-economie/uw-sector/automotive/electric-car-threatens-european-car-industry.html <comment><USER3> I think there is the potential for massive disruption and huge downward pressure on retail price with the transition to electric vehicles - <USER2> about the complexity of an internal combustion engine + drivetrain versus a battery, some electrics, and motors. There are also potentially huge impacts on all the allied industries i.e. fuelling, servicing, and then extending it to self-driving cars the impacts are potentially even greater - who cares about driving experience when the thing drives itself? How can the sexy aspect of speed etc. be used to sell when all cars max out at a limited speed based on the software driving them? Why own a car if you can expect an autonomous vehicle on demand? Interesting times.</comment>


Tinypy 1.1 - <AUTHOR> http://groups.google.com/group/tinypy/browse_frm/thread/405e7993d2a5664a

<comment><USER2> Really cool... I hadn't heard about this small (64k!), alternate implementation of Python; it has much the same appeal to me as Lua originally did. The home page:

<http://www.tinypy.org/>

</comment><comment><USER3> What a relief to have an alternative to Lua. For games I would appreciate Lua's incremental GC, of course. But far better to have something tiny that you can add on to than something big you have to rip things out of!

</comment><comment><USER4> Maybe we will once again get a modern Python interpreter on the Palm OS?</comment>


Google to pay largest fine in FTC history for bypassing Safari privacy settings  - <AUTHOR> http://appleinsider.com/article/?id=20377CachedYou

<comment><USER2> It's like.. 5 minutes of revenue for Google. I guess that fine will let them remember to respect the privacy of the users. Oh wait.

Irony aside, I thing this fine is too small for Google to learn something useful.

</comment><comment><USER3> From August 9.

No clue why this one is making the rounds again, nothing has happened recently to make it relevant and I've seen it in multiple places.

</comment><comment><USER4> The headline is a bit sensational — a $22.5M fine? A pin-prick to a Fortune 500 company.

<comment><USER5> It is an accurate fact.</comment></comment>


Facebook changes the rules for the public web - <AUTHOR> http://petewarden.typepad.com/searchbrowser/2010/06/facebook-changes-the-rules-for-the-public-web.html

<comment><USER2> couldn't you just use their API?

<comment><USER3> probably not, but that's irrelevant. the point is facebook's robots.txt says you can crawl it but their TOS says you can't. facebook have already shown they are willing to sue anyone who tries to crawl without permission. this new model that facebook is trying to use isn't scalable, it favors the big guys and is bad for the open web.</comment></comment>


Show HN: I built a directory to find freelancers in your city - <AUTHOR> Https://lancerlist.co <comment><AUTHOR> Hey Hacker News!

I'd like to share with you my latest 'startup', Lancerlist.

Lancerlist is a simple directory for finding freelancers in your city.

This is the first version of continually evolving project so would love to hear any feedback you may have!</comment>


LibreOffice 7.0 "Personal Edition" Branding Dropped - <AUTHOR> https://blog.documentfoundation.org/blog/2020/07/20/update-on-marketing-and-communication-plans-for-the-libreoffice-7-x-series/ <comment><USER2> Bravo...that would have been a terrible decision.</comment>


The missile knows where it is [video] - <AUTHOR> https://www.youtube.com/watch?v=bZe5J8SVCYQ <comment><USER2> Sir Humphrey Appleby is now a missile man!</comment>


When Anti-Pirates Sue Each Other Over Pirating Each Other's Technology... - <AUTHOR> http://techdirt.com/articles/20100521/1529489535.shtml

<comment><USER2> "Making things even more amusing? In the lawsuit, MPV mis-identifies its own patent, incorrectly using the title of a patent.. held by Warner Bros! MPV's patent 7,187,633 is titled "Marking of a data medium material for information intended for reproduction." Warner Bros., though, has a different patent, 7,206,409, titled "Motion picture anti-piracy coding." But in the filing... MPV refers to the title of its own patent as "Motion picture anti-piracy coding" and never mentions Warner's actual patent by that name. "

I think it's time they switched lawyers, or at least the copy editors..</comment>


NATS (High Performance Messaging System) Elixir Client Available - <AUTHOR> You can download the Elixir client via the 'Clients' list - it is right under the C client: http://nats.io/download/ <comment><USER2> This might make a good "Show HN".[1]

Is there an interesting story behind the project?

[1]:Guidelines: [https://news.ycombinator.com/showhn.html](https://news.ycombinator.com/showhn.html)

</comment><comment><USER3> fwiw, [http://nats.io/](http://nats.io/) and [https://github.com/nats- io](https://github.com/nats-io)</comment>


Automakers Must Deliver on the Promise of Self-Driving Car Technology - <AUTHOR> https://www.nytimes.com/interactive/2018/07/30/opinion/self-driving-car-accessibility.html <comment><USER2> >As the industry matures, the primary obstacles to wide adoption of these vehicles are regulatory and political.

The primary obstacle is technological, there really hasn't been much in the way of strong political/regulatory resistance. At a recent talk, when queried as to whether upcoming mass transit projects should be shit-canned in lieu of the imminent arrival of robotaxis, Waymo CEO John Krafcik cautioned listeners to not get too excited, widespread availability is still a decade or more away.

I fully support designing robotaxis that can accomodate the wheelchair bound among those with other disabilities, but give the companies a chance to get to a functioning business model for the dominant use case first. Henry Claypool, and myself, a resident of a northern city with harsh winters will have to rely on human driven taxis for the forseeable future.</comment>


The Disadvantages of an Elite Education - <AUTHOR> http://theamericanscholar.org/the-disadvantages-of-an-elite-education/

<comment><USER2> This guy spent 24 years at Ivy League schools, and "never learned that there are smart people who don't go to elite colleges, often precisely for reasons of class"?!

The fault, dear OP, lies not in the stars but in yourself that you are an arrogant, academic douchebag.

> There is something wrong with the smugness and self-congratulation that > elite schools connive at from the moment the fat envelopes come in the mail. > From orientation to graduation, the message is implicit in every tone of > voice and tilt of the head, every old-school tradition, every article in the > student paper, every speech from the dean. The message is: You have arrived. > Welcome to the club.

False. That was the message _you decided to take away_ from all the pomp and circumstance of the occasion. Ever been to orientation at a less-than-Elite institution? It's full of the same self-congratulatory, self-important crap. The problem is that OP decided to eat all the BS they were serving up, because he already saw himself as better than other people.

Then he goes from being simply wrong to being meta-wrong, that is, he negates his incorrect generality that "Elite Education Produces Better People", hoping that two wrongs make a right:

> Graduates of elite schools are not more valuable than stupid people, or > talentless people, or even lazy people.

False, or incoherent. If OP refuses to define "valuable", or defines it in some utterly wishy-washy subjective way, then this statement is non-sensical. If OP defines the "value" of a person in common terms such as utility of services rendered to others in society, then this statement is demonstrably false.

If 24 years at Yale and Columbia can't give him pause when committing this sort of logical blunder, then I'm not sending my kid to Yale or Columbia.</comment>


Offline Google Docs starts playing peek-a-boo - <AUTHOR> http://news.cnet.com/8301-30685_3-20072507-264/offline-google-docs-starts-playing-peek-a-boo/

<comment><USER2> There is an offline Google Docs tool. It's called GDriveSync ( www.gdrivesync.com). It's a sync tool between local computer and Google Docs. When offline - it uploads/ syncs changes when back online. Of course the local computer editor has to be MS Office, Open Office anything other than Google Docs... but you get your changes in Google Docs - automatically.</comment>


Debuggex: visual regex tester – now with PCRE - <AUTHOR> http://www.debuggex.com/?flavor=pcre <comment><USER2> If you like tools like this one, here's some more. There's a lot of 'visually see' the regex type tools.

As one who always forgets regex after 3 months of not using it, I find them helpful :)

[http://pineapple.io/resources/tagged/regular- expressions?sor...](http://pineapple.io/resources/tagged/regular- expressions?sort=all_time)

I particularly find this one fascinating, even though I don't fully grasp: [http://jkff.info/articles/ire/](http://jkff.info/articles/ire/)

</comment><comment><USER3> I wish there was a decent set of tools for using regexes as list generators. I've seen a few tools and libraries, but they make weird assumptions and never quite do what I want.

Here's what I'd like:

\- Infinite repetition operators are automatically rewritten as constrained repetition operators based on some global parameters e.g. * = {0,2}, + = {1,2}, {n,} = {n,n+3} or some such

\- Character classes can be rewritten as well so . can be kept as "just printable ASCII codes" instead of veering off into all sorts of unicode bits

\- Exhaustive list generation, generate every possible match using the rules above. Too many tools seem to go about some sort of random walk against the FSM and you never know what you're going to end up with

\- Decent coverage of operators. Too many have weird limits like no {,} operator or no [] character classes.

Getting exhaustive lists like this can be a better way of showing what a regex will match than trying a bunch of examples. A Perl and/or Python library would be extra awesome.

<comment><AUTHOR> Can you give me a use case for this? It is not very hard to do, given that random matches are already being generated, but I'm skeptical of its usefulness.

<comment><USER3> Use-cases:

1) regex search against an indexed database: use a regex to generate a search list, search a well indexed database with this list instead of linearly scanning data in the database with the regexes. AFAIK there's no way to index in a way that is regex friendly, only wildcard friendly...so most database systems that offer regex search just do a more traditional regex match against the contents of the table elements. In most cases, even searching tens of thousands of terms will be faster than the typical regex alternative.

2) education - you might use the generated lists while learning regexes to verify that you've accounted for various edge cases

3) data verification - you're using a regex to verify data, something is slipping through, see if it exists in the edge of generated matches (surprisingly common problem)

4) software testing & quality assurance. Define the possible inputs to a method via regex and generate all possible inputs instead of just testing at boundary conditions etc.

5) security/penetration testing, like #4 but to test cryptographic systems, passwords etc.

there's tons more...

Generating random members isn't sufficient as the runtime can grow extraordinarily long as you keep generating hoping you get all possible members, and you may not be certain when you can halt.

Some tools like tempting, like Microsoft Research's Rex [http://research.microsoft.com/en- us/projects/rex/](http://research.microsoft.com/en-us/projects/rex/)

but the member list it generates is incomplete for example: ^(rich|richard|dick)( j\\.)? th?om{1,2}[ae]s{1,2}[eo]n$

generates: "rich\sj.\sthommesson" "richard\sthomasen" "dick\stomasen" "rich\stommeson" "dick\sj.\stommassen" "dick\sj.\sthomessen" "dick\sthomessen" "rich\sthommeson" "richard\stommasen" "rich\sthomesen"

which misses a whole host of other matches

<comment><AUTHOR> Thanks for the feedback! I'll take this into account for future development plans.

</comment></comment></comment></comment><comment><USER4> While not a visual tool I highly recommend txt2regex[1]. It is a regex wizard that helps you build regexs for a number of different environments. It supports:

      a) -awk                                m) -mysql
      b) -ed                                 n) -ooo
      c) -egrep                              o) +perl
      d) -emacs                              p) +php
      e) -expect                             q) +postgres
      f) -find                               r) -procmail
      g) -gawk                               s) +python
      h) -grep                               t) +sed
      i) -javascript                         u) -tcl
      j) -lex                                v) -vbscript
      k) -lisp                               w) -vi
      l) -mawk                               x) +vim

[1] [http://aurelio.net/projects/txt2regex/](http://aurelio.net/projects/txt2regex/)

</comment><comment><USER5> Also check perl's Regexp::Debugger module (which provides an rxrx command) for a nice console regex debugger.

</comment><comment><USER6> Maybe somebody can give me of workflow tip:

I have collection of regexes, they each have several test cases that they should/should not match. Is there any good app that can do something like:

1\. Manage Regexes

2\. Create, add and modify test cases

3\. Run test cases on change

4\. Set different implementation (Javascript do not have lookbehind)

(5.) Sharable Urls

I will be grateful for any tips.

<comment><AUTHOR> Debuggex already covers 4 and 5.

Real-time unit testing is actually the next major feature planned (I expect the release to be in about 4-6 weeks). That will cover 2 and 3.

1 will be improved in the _very_ near future.

<comment><USER6> That sounds promising :) I also managed to find some mockup I got when I was thinking about working on something like that

[http://jsbin.com/axojax/2](http://jsbin.com/axojax/2)

some UI tips:

1\. Make the font bigger for inputs + add padding for input elements.

2\. Make difference between _ and space and other escaped characters more noticeable.

Great work!

<comment><AUTHOR> Thanks for the feedback! There will be a ui overhaul once all the major features are done. That will be in a few months.

</comment></comment></comment></comment><comment><USER7> When you need wizards and debuggers to formulate a question your grandmother would understand then your API is clearly broken. Regular expressions are a failure. A new interface to 'pattern matching' is long overdue. Smart young people, it's your turn!

<comment><USER8> Complicated stuff is complicated.

<comment><USER7> Simple stuff need not be complicated.

</comment></comment></comment><comment><USER9> Took pretty well to this URI regexp from Ruby. Impressive!

[http://www.debuggex.com/r/y47J4d_L02NwJrXq/0](http://www.debuggex.com/r/y47J4d_L02NwJrXq/0)

</comment><comment><USER10> The RFC822 email address regex : [http://www.debuggex.com/r/-Izy_lmZmdCAITvQ/0](http://www.debuggex.com/r/-Izy_lmZmdCAITvQ/0)

Very pretty indeed

<comment><USER11> Wow!

Is there a way to export this as a JPEG|PNG?

<comment><AUTHOR> Click "Export to StackOverflow", then you can save the image.

</comment></comment></comment><comment><USER12> Great work. This is exactly what I want. Thank you so much for putting your time into making this.

The syntax diagram is wonderful!

</comment><comment><USER13> I saw your presentation at the Velocity Pitch Night, I loved it then and loving it now. Keep up the good work! :)

<comment><AUTHOR> Thanks! I'm very flattered that you created an account to leave a comment :)

</comment></comment><comment><USER14> My favourite is [http://www.regexper.com/](http://www.regexper.com/) which seems to be simpler and more visually appealing (no affiliation).

</comment><comment><USER15> there's a program called 'grep' that does this, too... >_>

<comment><AUTHOR> grep has a very different purpose. Here's just one use case that it's not good at:

I see a regex in some code that I don't own. What does the regex do?

</comment></comment><comment><USER16> regexr is another of these - better, arguably</comment>


Looking Back at Three Months of afl-fuzz - <AUTHOR> http://lcamtuf.blogspot.com/2015/01/looking-back-at-three-months-of-afl-fuzz.html

<comment><USER2> > _Since then, afl-fuzz helped to squash hundreds of bugs, in part due to a > community of folks who found the tool to be fun to use._

I wonder whether a tool as unexpectedly successful as this presents the security community with a weird dilemma: If so many people have begun to use afl-fuzz, find problems, and report them, can't we expect that just as many people find problems and _don 't report them_?

Now, my security expertise goes as far as "don't roll your own", so maybe all the bugs found were, in practice, relatively difficult to exploit. But could afl-fuzz have helped scores of blackhatters to find and abuse the next shellshocks? If so, in hindsight, was it actually a good move to release afl- fuzz so openly and enthusiastically?

<comment> vezzy-fnord I don't see why you're singling out afl-fuzz when you can say the exact same thing for every automated penetration testing tool.

There's scores of Linux distributions dedicated to bundling as many security- related scripts as possible. If we're going to be talking about "utility to blackhatters", there's plenty of tools that have been around for longer and have been far more influential.

<comment><USER2> I'm singling out afl-fuzz because it seems to be so spectacularly successful. In fact, this blog post is all about how spectacularly successful it is. Maybe it isn't actually, compared to all those others tools I don't know about, but then maybe you could've just said that and skipped the sneering? I've been pretty forthcoming about my lack of security expertise, I'm just asking people like you an honest opinion.

<comment><USER3> Yes it was. It's safe to assume the "bad guys" have this stuff already, now the public gets to catch up.

</comment></comment></comment></comment><comment><USER4> The more I have heard of this guy's work, the more disturbed I am by his skill, breadth, and depth in InfoSec.

Not to mention his insane CNC and robotics work. And that is just a freaking hobby to him.

[https://duckduckgo.com/html?q=lcamtuf%20cnc](https://duckduckgo.com/html?q=lcamtuf%20cnc)

</comment><comment><USER5> I'd like to see the SQLite SQL statements, are there any links available?

<comment><USER6> These are the crashing statements:

      SELECT n()AND+#0;
      SELECT strftime()
      DETACH(SELECT group_concat(q));
      DROP TABLE IF EXISTS t0; CREATE TABLE t0(t);
      INSERT INTO t0 SELECT strftime();
      SELECT quote(t) FROM t0

See [https://www.sqlite.org/src/info/fe578863313128](https://www.sqlite.org/src/info/fe578863313128) for the patch.

<comment> f- Also SELECT c.* FROM (a,b) AS c;</comment></comment></comment>


Ruby 1.9 Encodings: A Primer and the Solution for Rails - <AUTHOR> http://yehudakatz.com/2010/05/05/ruby-1-9-encodings-a-primer-and-the-solution-for-rails/

<comment><USER2> _Take note of this article, don't think it doesn't apply to you just because you don't have an internationalized rails app._

i totally respect the need in the ruby community to support different charsets, but i wish ruby 1.9.1 had a mode where we could say DEFAULT_CHARSET_IN_CASE_OF_INCOMPATIBLE_ENCODING= "utf-8" and then it would convert to that encoding when in doubt. Would totally solve our issues.

details:

We just moved a fairly large Rails 2.3.4 app to 1.9.1 this week and have run into a number of 'incompatible charset encoding' issues.

My client is running a site that sends middle american students to middle american colleges. There's not a foreign character or thought in sight (which is a separate issue). However, our content writers occasionally use MS Word and a "smart quote" slips into the uploaded content.

When that happens, our rails HAML layout is in one encoding, and the partial that renders the smart-quote-bomb text from the database are now in different encodings, and the page errors.

The solution is to write a custom sanitize() method on string that forces encodings and gsubs away smart quotes. The consequences of missing one in 1.9.1 are just higher than in 1.8.7.

</comment><comment><USER3> It is said:

> However, this solution does not work very well for the Japanese community. > For a variety of complicated reasons, Japanese encoding, such as SHIFT-JIS, > are not considered to losslessly encode into UTF-8. As a result, Ruby has a > policy of not attempting to simply encode any inbound String into UTF-8.

> This decision is debatable, but the fact is that if Ruby transparently > transcoded all content into UTF-8, a large portion of the Ruby community > would see invisible lossy changes to their content. That part of the > community is willing to put up with incompatible encoding exceptions because > properly handling the encodings they regularly deal with is a somewhat > manual process.

Does someone has any sources about this?

Esp, why is SHIFT-JIS important? Is Unicode not capable of encoding the full Japanese language? What information would you loose by encoding with Unicode?

And why can't Unicode be extended to encode this missing information?

This seems to be a much better solution than this encoding nightmare. Esp in a language like Ruby, I want to have it simple and straightforward.

<comment><USER4> > Does someone has any sources about this?

Try this for example: <http://tclab.kaist.ac.kr/~otfried/Mule/unihan.html>

> Esp, why is SHIFT-JIS important? Is Unicode not capable of encoding the full > Japanese language? What information would you loose by encoding with > Unicode?

The problem is that SHIFT-JIS, and other regional east asian encodings, codify variations of characters that Unicode combine into one. The Japanese in particular dislike this conflation of characters that they feel should be separate. This process of combining characters was called Han Unification:

<http://en.wikipedia.org/wiki/Han_unification>

Now imagine that you're a Japanese user who feels that three similar ideograms are in fact sematically different, but a standards body has decided that the differences are merely stylistic and has combined them into one form: I'm sure you wouldn't be as willing to lay your language at the feet of Unicode and swear allegiance to the one true encoding.

Also consider that UTF-8 is optimized for efficient encoding of Western languages: a japanese text may be in practice four to eight times the size of a sematically similar version in a western language. SHIFT-JIS presumably does not suffer from this problem.

> This seems to be a much better solution than this encoding nightmare. Esp in > a language like Ruby, I want to have it simple and straightforward.

Except that languages are not really that simple or straightforward. I appreciate the flexibility of Ruby 1.9's encoding system. The only thing really broken right now are the tools that wycats mentions in this post.

<comment><USER3> Thank you for the information and the links.

But it still seems to me that extending Unicode to also include those variatons of characters is a better and more clean solution than these workarounds.

In your link, the author speaks also about some fonts being incomplete. The right solution about this would be to fix the fonts, not to switch to another encoding with its own font.

Having one encoding (Unicode) that is able to encode just everything would simplify everything.

Btw., you are speaking about space efficiency. Afaik, most Asian characters can be encoded as 2 bytes in UTF-8. You cannot get it much better. And the space required for text is in most cases much smaller compared to other media. Also, if there is much redundancy in it, it can easily be compressed (also in some transparent way if needed, like all Ruby strings with more than 32kb are automatically compressed internally or so).

<comment><USER4> > Afaik, most Asian characters can be encoded as 2 bytes in UTF-8

I overstated my case for sure. The Unified ideograms are four words wide, but the non-unified extensions are larger, iirc. And while disk space is hardly a problem anymore, you might see why programmers from 10 years ago may have made different choices.

> But it still seems to me that extending Unicode to also include those > variatons of characters is a better and more clean solution than these > workarounds.

I certainly can't argue with that. But Unicode is a standard, and real problems don't have time to wait around for standards bodies. In the eyes of many East Asian organizations, Unicode is broken now, and so the burden falls on the programmer.

Even here in the US, there are tons of data sitting around in tables encoded in Windows-1251 and ISO-8859-1. Having had to deal with UTF-8 and Latin-1 mismatches in the past, I don't find ruby1.9's encoding all that onerous myself.</comment></comment></comment></comment>


Why All the Speakers at Schnitzelconf are White Men - <AUTHOR> http://unicornfree.com/2010/all-schnitzelconf-speakers-are-male-white-have-that-flippy-do/ <comment><USER2> It looks like you're trying to manufacture conflict to market your conference.

<comment><AUTHOR> Nope. The conflict is going to come, and I wanted to head it off by adding a sane, cool-headed voice to it.

Three female organizers, and one male organizer, contacted me privately to thank me for being brave enough to say what I said -- because they, too, have been harrassed and harangued in public about the "diversity" of their speakers.

I am not surprised (though I don't consider myself brave).

If you don't read about conferences, or run one, you might not realize how nasty people are about this one thing -- an accusation you can't defend against. If you are accused of being racist in your speaker selection, what can you say?

"Well I happened to invite a really awesome african woman and an indian dude but they both turned me down"? Because then, of course, you're suggesting that you DO think in terms of race - ergo you are a racist! Then people will list women to you who have nothing to do with your conference, and you are obligated to say "I don't invite speakers who aren't on-topic for my conference" and then you are accused of hating women. Because if you loved women, you'd invite off-topic women speakers to a conference where everybody else would see the women didn't fit, and would assume that they got picked as window dressing.

These are charges you can't defend against, and the people applying it are -- without fail -- privileged caucasian men, who use their power and their smugness to make other people feel small and less righteous than they are. It's some kind of Look, I May Be A Famous White Dude But I'm So Sensitive And Feminist! signalling.

I, for one, am sick of it! It's selfish, divisive, and more importantly, useless, assuming the people trying to kick up a fuss _actually care about having more non-white-men in tech and business_.

I prepared this essay in advance - because I knew it would happen. Then it did happen. An internet-famous white man (!) reached out to my very slightly involved male co-organizer (!) and not me (even though he knows me, and he knows it's my conference), and accused us of being "non-diverse." As if we had to answer to him.

Note that he never answered our calls to our network for people from different places and walks of life who were bootstrapping.

These brutal attacks have got to stop, and I figured if I could do it while also reaching out to women bootstrappers, and non-white bootstrappers, to convince them to step forward, so much the better.

At the same time, you don't make attacks stop by shrinking away from them. You also can't do it by pointing out that the attackers are hypocritical assholes just in it for the attention. The only way to do it is to shine a light on just how silly the whole thing is.

So that is what I attempted to do.

I tried to make it _too embarrassing_ for anyone else to attack me publicly.

And now there is a resource that those other conference organizers, who don't feel as if they can speak freely, can link to without fear of direct reprisals.

In short, you could say I'm trying to bury non-existent controversy and then tamp down its grave.

<comment><USER2> Gah. I had no idea. Sorry about my snarky comment then. I didn't realize you were _already_ fielding that kind of drama.

This is really stupid though... hey world, guess what, software development is a field dominated by men, get over it!

I'm not a fan of forcing diversity for diversity's sake. If you give a spot to a somewhat qualified person, forsaking a more qualified person, that is wrong. If there are minorities that are talented enough to rise to the top, they will. Agree?</comment></comment></comment>


Manual Work is a Bug – Always be automating (2018) - <AUTHOR> https://queue.acm.org/detail.cfm?id=3197520 <comment><USER2> This article needs a giant warning on it. Automating is _brilliant_ , but an automated process is a liability. If no one is regularly checking the output of something that's been automated then you can't know if it's broken, and that could be catastrophic. Every story of "the backups had failed", "payments had been missed for months", or "inventory wasn't where it was supposed to be" is the story of an automated process no one was checking.

If you automate _anything_ you need robust error reporting (which is not an email someone will ignore).

<comment><USER3> > which is not an email someone will ignore

Every developer should take this advice to heart. If a customer asks for 'a daily email report', you should strongly advise against it.

\- Most reports the customer asks for will be meaningless vanity metrics anyway

\- If you only do error reporting (not sending an email if everything goes smoothly), you won't notice it if email is not being delivered.

\- If you report everything (not just the errors), people will stop reading them

\- A mailbox history is not a log

\- you'll get a request just about every week to change the recipient list of the report.

\- Who reads the reports on weekends?

I could go on with arguments for a while, but the short story is this: email is not suitable for logging and reporting. In fact: email is not suitable for a lot of things. But customers will ask you anyway, because that's the tool they know (if you have a hammer, all problems will look like nails).

My advice:

\- Log everything to disk when possible (syslog, pipe)

\- use a logging aggregator to filter and archive (fluentd, ELK, graylog)

\- use an exception tracker for exceptions (sentry, raygun)

\- make incidents actionable by creating tickets automatically (jira, zendesk, slack)

\- if needed, use an incident response service (pagerduty, opsgenie)

<comment><USER4> >make incidents actionable by creating tickets automatically

not really a serious concern, but what happens when ticket automation goes down? If all the problems are auto-reported, in a form of tickets or tasks, then if that somehow breaks down it might take a moment before anyone notices, especially if reported exceptions are rather rare. If it is normal to have tens of tickets every day, then problems with ticket creation will be noticeable almost instantly, if they are very rare - then not so much

The second issue with automated exception tracking is that you loose the "huh, this is weird" mechanism that works when actual humans go through logs or reports. While any tool will of course be orders of magnitude faster and also probably more accurate, by relying solely on such automation an opportunity to notice some "weird"/not-typical entries or rare/unexpected sequences of those might be missed. Then again in most cases - I guess - simple statistical analysis might be a good substitute. And that can be automated. (edit: formatting)

<comment><USER5> The problem is, for every automated reporting system, you need another system to check that that system is working. And so on, all the way down.

<comment><USER6> Don't let the perfect be the enemy of the good. 99.9XXXX reliability is good enough. Eventually have a enough nines and your risks are things like "nuclear war", "dinasour-killer sized asteroid hitting the earth", et cetera.

<comment><USER4> Agreed, there is absolutely no point going further after reaching a certain reliability level. However, one thing is eliminating risks, the other is limiting the consequences of said risks. I strongly prefer 99.9% reliability where that 0.1% means some insignificant problem over 99.99% reliability where the remaining 0.01% means total disaster. My point is that doing "too much" automation gives diminishing returns (which is not bad in itself), but might also disproportionately increase the consequences of that 0.xxxx1%

</comment></comment></comment></comment></comment></comment><comment><USER7> I've been thinking about writing an essay about the flip side for a while, but maybe it only needs to be a toss-off HN comment, so...

In political/economic debates, people like to say: "if you tax something, you get less of it." And in turn "if you subsidize something, you get more of it."

You could think of automation as a labor subsidy. Or, on a rough level, as a tax on attention. That's not strictly correct, but it gets you in the right frame of mind: automation is, among other things, _designed_ the achieve labor with diminished (or even absent) attention.

The problem is that diminished attention can have overlooked tradeoffs. If you know those tradeoffs, you may be able to automate around them. If you're not paying attention, you probably can't. If you've automated away attention, you might not notice until you've hit a problem at the scale of your automation.

Automation _is_ valuable, absolutely. You probably want more of it. I do. And yet...

<comment><USER8> Relevant

[https://www.researchgate.net/publication/49400362_Attention_...](https://www.researchgate.net/publication/49400362_Attention_and_automation_New_perspectives_on_mental_underload_and_performance)

</comment></comment><comment><USER9> I particularly love the suggestion to just document what you're doing as the first step to proper automation. It's not that much work or cognitive effort. Sometimes a decent description of your procedure is all you will need and it tends to prove very valuable later. It can be also gradually improved. I was doing it often already, and this will remind me to persist.

<comment><USER10> This is very true. I stand up a server every now and then and just define the setup as a single-file Ansible playbook— pretty much no matter what, it'll be some combination of packages to install, users to create, config files to modify, etc. Why write it down when I can just define it as code and check it in somewhere?

But then the IT department comes along and hates it; they make a wiki page for each server with a list of instructions for what was done. Their approach is more flexible, as not every tool/step can be easily or robustly automated, and if you can just screenshot the config webpage and paste it into your wiki, that might be just as good as (and a lot faster than) spending an hour trying to bypass that step.

Plus, their approach assumes aggressive use of VM-level tools. Where I'm like "oh, I need N build slaves? Let me add them to my inventory file and re-run", whereas they're like "N build slaves? Sure, I'll set up the first one manually and the clone the machine for the others."

I know there are real actual advantages to version controlled configuration, but there is something pragmatic about just doing the thing and keeping human- readable notes as you do it.

</comment></comment><comment><USER11> Reading this makes me realise on my team I'm the automation guru. That's my culture through and through. The rest of my team's culture is the opposite. I always want us to play a little more defense, so that we can over time get a lot more done. They always push back that "we're too busy to do it". Yet I continue to automate and things continue to improve. Sucks because we could be significantly more effective if it were the team culture.

<comment><USER12> This may be a healthy dynamic. There's always a risk both of over-automating and under-automating and people tend to sway one way or the other.

<comment><USER11> I sort of feel it's the inverse. The last place I worked was heavily into automation and they had the one manager who always cautioned about relying on too much automation. I think that was a healthier dynamic tbh.

Too much vs. not enough are just different sets of problems.

Too much = you risk no one on the team being able to troubleshoot when things go wrong because no one has ever had to run through the process themselves before so no one understands it.

Too little = a non-trivial percentage of your bandwidth is eaten up by trivial things you keep doing over and over again that are in the vain of servicing requests rather than solving problems.

In practice I find I'd rather have the first problem and I can think of solutions to it's problems that I'm happy enough with.

</comment></comment></comment><comment><USER13> I've recently been on a learn-about-economics spree, so I have a handy lens nearby with which to view this advice: you must not allow your gains to grow merely linearly (by constantly learning but not automating). The economic imperative is to allow gains to compound (by automating). Manual workflows cannot compound; automated workflows can.

Sam Altman covers the same sentiment in [http://blog.samaltman.com/how-to-be- successful](http://blog.samaltman.com/how-to-be-successful) .

<comment><USER14> in theory[1], yes.

[1]:[https://www.xkcd.com/1319/](https://www.xkcd.com/1319/)

(I may have been doing a lot of automation recently because, I think, bosses have paid a lot of money for an automation tool and feel the need to use it, and therefore feeling a little cynical about the whole thing)

<comment><USER10> Ha, I expected a link to this one: [https://xkcd.com/1205/](https://xkcd.com/1205/)

</comment></comment></comment><comment><USER15> I've struggled with this. I'm part of a smaller local faculty IT and our systems are created by a central IT.

Some problems I struggle with:

1\. How do I automate in an environment of so many disparate systems. Some things are done through web interfaces, some RDP'ing to an AD server, email, spreadsheets sitting on shared drives, etc.

2\. How do I handle credentials. Central IT won't create generic accounts for us, even for testing, so all automation would have to be done through my credentials. How do I set up an automated system that uses my credentials safely?

3\. Where do I run this system from? From my own laptop, a box under my desk, somewhere in the cloud where I'd have to beg security to allow it to do anything non-trivial? And how do I pass this on to the next person?

I've looked in to RPA type solutions (expensive), headless browsers (so unreliable), Microsoft Flow (limited and confusing).

I'm at a loss but I did recently find TagUI which looks very interesting [https://github.com/kelaberetiv/TagUI](https://github.com/kelaberetiv/TagUI)

<comment><USER16> > How do I automate in an environment of so many disparate systems.

1\. Using disparate tools if need be. Fortunately we have PowerShell these days which covers a whole lot of stuff including being ok at various web-based interactions. Anymore, if it can't be done with PowerShell (in a Windows environment that is) then I think it's probably too fragile to leave to automation anyway... ok, that's not entirely true. There is at least one important production system in our environment I have to automate using AutoHotKey and that works pretty well.

2\. I have no idea why anyone would think not allowing generic accounts for this sort of thing would be a good idea, but under your own account shouldn't be a big deal as long as you leave appropriate documentation about what needs to happen in case you're hit by a bus. Many sins are forgivable if there is adequate documentation.

3\. I usually just run automation scripts on the system I'm scripting. If I'm dropped into an environment and asked to figure out how some system is automated, where's the first place I'm going to look?

</comment></comment><comment><USER6> A better way to do step 2 in the process: "Do-Nothing Scripting". Discussed on HN recently: [https://news.ycombinator.com/item?id=20495739](https://news.ycombinator.com/item?id=20495739)

</comment><comment><USER17> Automation gives you consistency. If you automate the incorrect process, you'll get it done consistently wrong.

It still is a win because you'll have a better chance on diagnosing and improving something that fails the same way as opposed to some manual process that has been done in a dozen different incorrect ways and you learn nothing from mistakes because there are no automated controls to feed back into.

Most people and organizations have a strong negative gut reaction when automation does something wrong but are okay when a manual process yields mistakes because we largely accept humans as fallible (at most, someone gets fired, and more bureaucracy is introduced), but the long-term economics of automation look ultimately better.

I've seen time and time again apparently innocent manual processes that, when not recognized as a liability, rapidly turn into entire departments (and entire management hierarchies), at which point it's impossible to improve, because changing culture is harder than changing code. Having enough instances of that is what creates corporate whales without competitiveness.

</comment><comment><USER18> I never heard any excitement from people when we told them we would test some automation with their request, but otherwise I love being reminded of this. We tend to slack with that in our team, all the reasons mentioned in this post.

What we have found helped us quite a bit is, after writing the initial runbook for a task, to write a second script (usually pretty short, maybe two dozen lines) that asks you questions and then generates customised steps. We then later use these functions in the actual automation.

<comment><USER19> I don't know if you already saw the other HN post on the front page about this, or just coincidence, but:

> Do-nothing scripting: the key to gradual automation

[https://news.ycombinator.com/item?id=20495739](https://news.ycombinator.com/item?id=20495739)

</comment></comment><comment><USER20> Always improve, but manual work is the craft that gets you to that improvement. Once the wheel has been created, no need to recreate it, but brakes might be nice, too.

</comment><comment><USER21> I would argue not all manual work is a bug nor does all manual work need to be automated. For example, if you have a manual task that needs to be done once per quarter, is it really worth investing time and money to automate it? In contrast, if you have a manual task that you do multiple times per day, then perhaps that's worth investing resources to automate that task.

</comment><comment><USER22> I applaud this sentiment. Automate all the things.

<comment><USER23> Automate things that you sufficently understand. Do it first 20 times manually and then when you know what it really does automate.

</comment></comment><comment><USER24> This article is short-sighted at best. I am a system administrator which sadly cannot do much automation.

The reason is simple: I manage systems not installed by me, where problems are often a one-off issue and more important, every system is installed a bit differently (differently enough that there is no portability for automation between systems).

Automation is awesome. But sometimes it just cannot be done, because of constraints put in place by others.

</comment><comment><USER25> [https://xkcd.com/1205/](https://xkcd.com/1205/)

<comment><USER11> Better yet - Terrence Parr's famous quote - "Why program by hand in five days what you can spend (twenty) five years of your life automating?"</comment></comment>


Tesla is suing oil industry exec for impersonating Elon Musk - <AUTHOR> https://electrek.co/2016/09/14/tesla-is-suing-oil-industry-exec-for-impersonating-elon-musk-to-get-delivery-numbers/ <comment><USER2> For impersonating Elon Musk... to get confidential information.</comment>