The End of Moore's Law and Faster General Purpose Computing, and a Road Forward [pdf] - <AUTHOR> https://p4.org/assets/P4WS_2019/Speaker_Slides/9_2.05pm_John_Hennessey.pdf <comment><USER2> We've built up layers and layers and layers of inefficiencies in the entire OS and software stack since the gigahertz wars took us from 66Mhz to multiple GHz in the 90s.

The software industry is awful at conserving code and approaches through the every-five-years total redo of programming languages and frameworks. Or less for Javascript.

That churn also means optimization from hardware --> program execution doesn't happen. Instead we plow through layers upon layers of both conceptual abstraction layers and actual software execution barriers.

Also, why the hell are standardized libraries more ... standardized? I get lots of languages are different in mechanics and syntax... But a standardized library set could be optimized behind the interface repetitively, be optimized at the hw/software level, etc.

Why do ruby, python, javascript, c#, java, rust, C++, etc etc etc etc etc not have evolved to an efficient underpinning and common design? Linux, windows, android, and iOS need to converge on this too. It would be less wasted space in memory, less wasted space in OS complexity, less wasted space in app complexity and size. I guess ARM/Intel/AMD would also need to get in the game to optimize down to chip level.

Maybe that's what he means with "DSLs", but to me "DSLs" are an order of magnitude more complex in infrastructure and coordination if we are talking about dedicated hardware for dedicated processing tasks while still having general task ability. DSLs just seem to constrain too much freedom.

<comment><USER3> Correct me if I'm wrong, but isn't this exactly the problem LLVM was designed to tackle?

<comment><USER4> If you're targeting non-CPU designs--such as GPUs, FPGAs, systolic arrays, TPUs, etc.--it is very much the case that you have to write your original source code differently to be able to get good speedups on those accelerators. It has long been known in the HPC community that "performance portability" is an unattainable goal, not that it stops marketing departments from trying to claim that they've achieved it.

LLVM/Clang makes it much easier to bootstrap support for a new architecture, and to add the necessary architecture-specific intrinsics for your new architecture, but it doesn't really make it possible to make architecture- agnostic code work well on weirder architectures.

<comment><USER5> True. If you want performance - you have to re-write the code for the new architecture otherwise it is pointless to develop the new core.

The problem with developing a good processor architecture is you have to _always_ maintain legacy compatibility without sacrificing performance - because you know software.

This adds layers of extra HW with each passing generation of the processor lying around for some legacy code.

</comment></comment></comment></comment><comment><USER6> For those who have not looked yet: John Hennessey presentation. Argues -- with a lot of detail -- that Moore's law has closed out, that energy efficiency is the next key metric, and that specialized hardware (like TPU) might be the future.

When I buy a machine, I am now perfectly happy buying an old CPU, and I think this shows you why. You can buy something from as far back as 2012, and you're okay.

However, I do look for fast memory. SSDs at least, and I wish he had added a slide about the drop in memory speed. Am I at inflection?

Perhaps the future is: you buy an old laptop with specs like today and then you buy one additional piece of hardware (TPU, ASIC, Graphics for gaming etc).

<comment><USER7> _> energy efficiency is the next key metric, and that specialized hardware (like TPU) might be the future._

This is nonsense pushed forward by large corporations who want to own all your data and computational capacity.

<comment><USER8> Admittedly knowing little about how multi-core CPUs work, I've always thought the next breakthrough in CPU tech would be a hardware-based scheduler, a chip that effectively load balances threads to ensure all cores are used equally and simplifies the writing of software. The dev writes thread-safe code and the hardware does the rest. I wonder how feasible that really is.

<comment><USER9> That sounds perfectly reasonable to me, but John Hennessy literally wrote the book on computer architecture. Towards the end of the deck he has a slide that we shouldn't expect large gains from improved architecture (on general purpose chips) in the future. I'm inclined to believe him, although I would be interested in hearing a deeper proof/disproof of the architecture you proposed.

</comment></comment></comment></comment><comment><AUTHOR> This ties in nicely with chiplets: [https://semiengineering.com/the-chiplet- race-begins/](https://semiengineering.com/the-chiplet-race-begins/) \- a way to integrate dies in a package, where the dies can use specialized processes for different functions - e.g. analog or digital or memory or accelerators or CPUs or networking etc. This would make it easier to iterate memory/CPU/GPU/FPGA/accelerator designs at different rates, and reduce development costs (don't need to support/have IP for every function, just an accelerated set of operations on an optimized process within each chiplet). But it will need progress on inter-chiplet PHY/interface standardization.

</comment><comment><USER10> So yes, if you compare matrix multiply in Python vs SIMD instructions, you will find a big improvement. Much harder to do that for more general purpose workloads.

And it doesn't scale: [https://spectrum.ieee.org/nanoclast/semiconductors/processor...](https://spectrum.ieee.org/nanoclast/semiconductors/processors/the- accelerator-wall-a-new-problem-for-a-post-moores-law-world)

And in many cases, if you normalize all the metrics, e.g. precision, process node, etc. You'll find that the advantage of ASICs is greatly exaggerated in most cases and is often within ~2-4X of the more general purpose processor. E.g. small GEMM cores in the Volta GPU actually beat the TPUv2 on a per chip basis. Anton 2, normalized for process, is within 5x ish of manycore MIMD processors in energy efficiency.

In other cases, e.g. the marquee example of bitcoin ASICs, that only works because of extremely low memory and memory bandwidth requirements.

</comment><comment><USER11> A possibly stupid question from a neophyte: what was the driving force behind Moore's law when it was in operation? Did it become a self-fulfilling prophecy by becoming a performance goal after becoming enshrined in folklore, or is there an underlying physical reason?

<comment><USER12> [https://en.wikipedia.org/wiki/Transistor_count](https://en.wikipedia.org/wiki/Transistor_count)

The transistor can only get so small before it stops working. There are many issues with required extreme ultraviolet light sources (lasers) and allowed amount of impurities in silicon waffer. And R&D cost for each iteration of lithography is getting higher while bringing less benefits.

<comment><USER11> Yes, the existence of an upper bound on transistor count follows easily from the atomic nature of matter. The Wikipedia article on Moore's law lists multiple disparate "enabling factors" which do not seem to have much to do with one another. The conjunction of which comprise an explanation of sorts, but I'm wondering whether there's a simple observation or fact that ties them all together, apart from my Sociological theory.

</comment></comment></comment><comment><USER5> Slide 36 compares the TPU with a CPU/GPU. This is apples to oranges comparison. One uses an 8bit Integer multiply while the other uses a 32b Floating Point multiply which inherently uses at least >4X more energy[1]. If you scale the TPU by 4, it is not an order of magnitude better. The proper comparison should be between the TPU and an equivalent DSP doing 8b computations. That would show if eliminating the energy consumed due to the Register File accesses is significant.I suspect most of the energy saving comes from having a huge on chip memory.

[1] From slide 21

Function Energy in Pj

8-bit add 0.03

32-bit add 0.1

FP Multiply 16-bit 1.1

FP Multiply32-bit 3.7

Register file *6

L1 cache access 10

L2 cache access 20

L3 cache access 100

Off-chip DRAM access 1,300-2,600

</comment><comment><USER13> Big chipmakers are turning to architectural improvements such as chiplets, faster throughput both on-chip and off-chip, and concentrating more work per operation or cycle, in order to ramp up processing speeds and efficiency [https://semiengineering.com/chiplets-faster-interconnects- an...](https://semiengineering.com/chiplets-faster-interconnects-and-more- efficiency/)

Scaling certainly isn't dead. There will still will be chips developed at 5nm and 3nm, primarily because you need to put more and different types of processors/accelerators and memories on a die. But this isn't just about scaling of logic and memory for power, performance and area reasons, as defined by Moore's Law. The big problem now is that some of the new AI/ML chips are larger than reticle size, which means you have to stitch multiple die together. Shrinking allows you to put all of this on a single die. These are basically massively parallel architectures on a chip. Scaling provides the means to make this happen, but by itself it is a small part of total the power/performance improvement. At 3nm, you'd be lucky to get 20% P/P improvements, and even that will require new materials like cobalt and a new transistor structure like gate-all-around FETs. A lot of these new chips are promising for orders of magnitude improvement—100 to 1,000X, and you can't achieve that with scaling alone. That requires other chips, like HBM memory, with a high speed interconnect like an interposer or a bridge, as well as more efficient/sparser algorithms. So scaling is still important, but not for the same reasons it used to be.

</comment><comment><USER14> It is not that I disagree with Hennessy, but I think it is premature to conclude that general-purpose processors have reached the end of the road. There is a healthy middle in between specialized and general-purpose design. Exploiting that middle is what I think will deliver the next generation of growth. That is exactly what naturally occurred with SoC and mobile design.

The raw computational capabilities of the TPU don't really prove anything. Of course co-design wins. Whether it is vison or NLP -- NN training has dominant characteristics. The arithmetic is known: GeMM. The control is known: SGD. Tailoring control and memory-hierarchy to this is a no-brainer and of course the economic incentives at Google push them in this direction and of course the expertise available at Google powered this success. For other applications it is not so clear.

Finding similar dominance in other applications is trickier. To accelerate an application with a specialized architecture you need dominating characteristics in the apps memory-access, computational, and control profiles.

</comment><comment><USER15> It's odd that the presentation doesn't discuss alternatives to using silicon. Ultimately, this is akin to saying that there are limits on how small a vacuum tube we can make. We already know of a number of other potential computing platforms such as graphene, photonics, memristors, and so on. These things have already been discovered, and they have been shown to work in the lab. It's really just a matter of putting the effort into producing these technologies at scale.

Another interesting aspect of moving to a more efficient substrate would be that power requirements for the devices will also lower as per Koomey's law [https://en.wikipedia.org/wiki/Koomey%27s_law](https://en.wikipedia.org/wiki/Koomey%27s_law)

<comment><USER16> Well...no. what it says is there are limits on how small of wires we can make, and how small of layers make a material functional (about 5).

Wires can't get smaller without compromising RC (and thus speed). Quite horrifically: this is way more an issue than the transistor.

Graphene and photonics don't help this. At all. It isn't a matter of how small a tube. You physically need 5nm to insulate, and 5nm for a functional material. So a 5nm device with a 5nm spacer and a 5nm space to the next device is about it. The smallest pitch of any physical device is 20nm. The critical pitches in wafer are about 30nm and 40nm, so in an ideal world, we can reach 3x, ever. It doesn't matter which material you choose.

And yeah, you can stack up, but not in quite the way you dream, and thermal and processing issues make this hard in most domains. When I build, I deposit at temperatures, which affect underlying layers. So stacking doesn't quite work as you might expect. Again, real materials Ina real flow are actually different, and not in a trivial 'just make it work' reducible fashion.

Memristors may not really exist, and are useful in the context of high speed memory. That has real physical challenges. And people.have spent billions for decades on this problem.

Anyway, this is missing some background, but the presentation is great.

<comment><USER15> We already know you can use individual atoms as transistors [1]. So, clearly we can go a lot smaller than 5mm here. Obviously there are challenges in scaling these new substrates up to create useful chips, and creating the infrastructure to put them into mass production. My point is that we know this is possible, and an inflection point will come where investing in these new substrates starts being more lucrative than trying to squeeze more out of silicon.

[1] [https://www.sciencedaily.com/releases/2018/08/180816101939.h...](https://www.sciencedaily.com/releases/2018/08/180816101939.htm)

</comment></comment></comment><comment><USER17> "WASTED WORK ON THE INTEL CORE I7", slide#12 (page 13 in pdf) is fascinating to me. But I want to know how the data was collected, and what the % wasted work actually means.

40% wasted work, does that mean that they checked the branch-predictor and found that 40% of the time was spent on (wrongfully) speculated branches?

It also suggests that for all of the power-efficiency faults of branch predictors (aka: running power-consuming computations when it was "unnecessary"), the best you could do is maybe a 40% reduction in power consumption (no task seems to be 40% inefficient).

<comment><USER18> > ... INTEL CORE I7

When someone says Intel i5 or i7, I immediately wonder if they're talking about 2008 i7 or 2019 model.

Intel would be smart to retire whole i3/i5/i7/i9 branding. People seem to think every i5 or i7 is the same.

<comment><USER17> > People seem to think every i5 or i7 is the same.

Unfortunately, this is a feature, not a bug. Intel wants their branding to have this effect... the lay-person isn't supposed to understand Sandy Bridge (i7-2700k) vs Skylake (i7-6700k)

<comment><USER18> So Intel wants laypersons not to realize there's something faster available and to upgrade their x86 based systems?

Before that era people didn't know much about the details either, but they did understand 800 MHz was faster than 533 MHz.

</comment></comment></comment></comment><comment><USER19> Still be too early to call the end of the march of microprocessors though.

[https://www.scienceabc.com/humans/the-human-brain-vs- superco...](https://www.scienceabc.com/humans/the-human-brain-vs- supercomputers-which-one-wins.html)

The limits they are running up against are indeed crisises, but they're probably going to be able to find that they can copy whatever it is that biology is doing and squeeze out quite a bit more. The tradeoffs will get a lot weirder though.

<comment><USER20> Humans are not good at general purpose computation. Your linked article states the brain achieves 1 exaflops, and cites [http://people.uwplatt.edu/~yangq/csse411/csse411-materials/s...](http://people.uwplatt.edu/~yangq/csse411/csse411-materials/s13/cs_1/brostm_AI- in-commercet.doc) for this number. That document states the value with no citation or rationale.

I can do far less than 0.0001 single precision floating point operations per second, so whatever the context for "1 exaflops" is, it isn't general purpose computation.

EDIT: this seems sort of like saying that throwing a brick through a window achieves many exaflops because simulating the physics in real time would require that performance. I'd like to read more about this value and how someone came up with it, but googling just gives me that same scienceabc article and stuff referencing it.

<comment><USER21> Nahh, you could do more than 0.0001 floating point operations per second. To beat that you need to do a single floating point operation in two hours, which is quite achievable with paper and pencil ;-)

0.01 floating point operations per second seems harder, but perhaps humanly doable.

<comment><USER20> I'm easily distracted.

</comment></comment></comment></comment><comment><USER22> Amin's keynote is relevant here: [https://onfconnect2019.sched.com/event/RzZl](https://onfconnect2019.sched.com/event/RzZl)

The basic form of computing is becoming distributed. More are coming.

</comment><comment><USER23> I'm amazed that it's less than a picojoule to do an 8 bit add.

<comment><USER24> The Landauer limit is about a billion times smaller than this, so there's room for power savings before we hit any physical limits.

</comment></comment><comment><USER25> So what's the name of the metric flop/sec/USD because that keeps on growing exponentially thanks to GPUs/TPUs, a paradigm shift predicted by Ray Kurzweil.

</comment><comment><USER26> Is there a video of this talk available somewhere?

Also can someone tell me what p4 is? Looks like almost every company and a bunch of universities are "contributors" there.

<comment><USER27> P4 is a domain-specific language for specifying packet forwarding pipelines, i.e. the hardware that takes packets in one port, decodes their headers (e.g. destination MAC or IP address), munges them somehow (e.g. updating TTL, destination MAC, and checksum), and sends them out another port. This enables you to build all sorts of network devices from Ethernet switches to IP routers to RDMA fabrics, etc.. You can compile P4 onto a CPU, a smart NIC, an NPU, a programmable ASIC, an FPGA, etc.. It can also be used a bit like EBPF and compiled into a pipeline in the Linux kernel.

Basically P4 allows you to (re)program your network data plane to do whatever you want, and you can create new network protocols or change the way existing ones work without having to change your hardware and without losing line rate performance.

It's also somewhat like EBPF, but it compiles to hardware as well as software.

</comment></comment><comment><USER28> One of the more interesting things I've read on HN in awhile. Seems like this will result in a large paradigm shift for the computing industry.

<comment><USER29> I think we've already seen the shift with cellphones.

I think consumer facing performance processors will fade.

Data centers will continue to push for more performance. It could mean less rack space, less power consumption, and less to manage.

Cell phone/tablet focused processors will become powerful enough to handle the majority of daily tasks while enjoying extended battery life.

</comment></comment><comment><USER30> There's an internet meme about "Imminent death of Moore's law predicted".

All Moore's law talks about is the density of transistors on a chip, and it's never been a linear progression of numbers. Recently I've seen news articles about some research into 5nm processes and other methods for increasing density of components on silicon, so it seems Moore's law (really Moore's rule of thumb or Moore's casual observation) isn't done yet.</comment>


Let's Encrypt now fully supports IPv6 - <AUTHOR> https://letsencrypt.org/2016/07/26/full-ipv6-support.html <comment><USER2> So unfair! Comodo once, a while ago, also thought about using IPv6!

But seriously: letsencrypt is doing excellent work. It's a great case study in how inefficient a mostly-free market can be: SSL adoption doubled within a year. All that was previously deadweight loss.

<comment><USER3> There is no such thing as a free market when certificates were only created and issued by those who had the political clout to get their authority in the Microsoft / Google / Mozilla trusted keyring or in the trust network of another trusted provider.

A free market is one where you can compete independently.

If anything, it is a demonstration of why faux-markets with monopoly control (or in this case, the collusion between established and trusted authorities to prevent competition) are dangerous.

<comment><USER4> What? Anyone can become a certificate authority, so long as you'll take the time to document your procedures, store the private key securely, and be externally audited. You probably want an insurance policy as well.

The only program that charges money to be included is/was Oracle, as far as I know. Everyone else is free.

<comment><USER5> Everyone else charges indirectly. Inclusion is free, you "only" need to go through someone like webtrust who will charge you $100k+ for an audit.

<comment><USER6> So? Relying on commercial auditors to verify compliance with published industry standards is a pretty common free-market solution to problems of trust.

<comment><USER5> I'm not disagreeing with it. I just think the parent post was weird: anyone can do it, just spend some time to obtain things, only Oracle charges for the program.

</comment></comment></comment></comment></comment></comment><comment><USER7> According to conversation on [https://github.com/letsencrypt/boulder/issues/593](https://github.com/letsencrypt/boulder/issues/593) they couldn't support it because one of their datacenters didn't support IPv6 traffic.

<comment><USER8> Which sucks because a data center that doesn't support IPv6 is the 2016 version of a data center that only does Token Ring networking.

<comment><USER9> They are unfortunately and shockingly common.

<comment><USER10> (Cough) AWS? (cough)

What's up with that anyway?

<comment><USER11> As someone who is responsible for protecting networks for large data centers the biggest issue I see with adopting IPv6 is the overhead required. I took a job with another company this last year and while I thought the asset tracking was bad my previous company, I found it can get much worse. I have clients whom we host servers for that I have no clear method of determining what their IP space is.

System Admins routines make mistakes with complicated host names and trying to acquire an accurate inventory is an absolute nightmare. This ties into IPv6 because why would anyone take that disfunctional system, which barely works with 'easy' IPv4 addresses, and make it even more complex? We would have to support both IPv4 and IPv6 simultaneously and firewall rules would get much more complex initially and they are already a huge issue for me to get changes made.

At my old job this was similar. Even though it was in the financial industry and that particular company was rolling in profits it couldn't keep enough network engineers around to save it's life. The turnover was high, documentation was horrible and projects to make things better languished on in the ether.

No no no, forget the whole IPv6 thing, just run IPv4 for all things internal and gratuitously support IPv6 outside if you really have to. I jest of course, but that is the reality of my corporate life in the last 6 years in two fortune 500 companies.

<comment><USER12> The way I think about IPv6 is that it's an entire second network which happens to frequently coexist on the same layer 1 and layer 2 equipment.

That said, I think there's often a strong argument for only using IPv6 for the internal parts of a network. IPv6 actually simplifies things, and where IPv4 remains needed, it can be encapsulated and routed over a v6 network.

But it takes a team which understands the v6 world and is able to take advantage of the benefits for this to become a reality.

</comment></comment></comment></comment></comment></comment><comment><USER13> For any Go users out there, I'd recommend Russ Cox's package: [https://godoc.org/rsc.io/letsencrypt](https://godoc.org/rsc.io/letsencrypt). It automatically acquires certificates and keeps them up to date.

</comment><comment><USER14> someone play devils advocate and tell me reasons I might __not __want to use LetsEncrypt? (aside from potential issues from short-lived certs).

<comment><USER15> Contrary to popular belief, it might be quite hard to configure it. I spent few days trying to make it work, but I didn't succeed yet. Though my requirements might be a bit atypical. I'm not going to run their software which does too much for my taste, therefore I'm using letsencrypt.sh, which I briefly inspected and I feel comfortable to have full control over process. I don't want to perform domain validation using HTTP, I want to use DNS validation, so I have to write an additional software layer to integrate letsencrypt.sh and my DNS provider API (vultr). And that turns out not so easy.

For me StartSSL was the best offering, but now I need few 3-rd level domains, so letsencrypt seems the only free choice.

<comment><USER16> To be completely honest, if you're not using the tool Let's Encrypt is creating with the specific intent of making configuration easier, it sounds like a moot point to state that it "might be quite hard to configure it". I'm not saying that your use case is invalid, but starting your comment off with stating that it is hard to configure might throw people off.

<comment><USER17> I don't think it's too much to expect people to read a single paragraph before jumping to conclusions.

</comment></comment></comment></comment><comment><USER18> Famous question - intranet.

We can do dns-01 verification, on intranets (like valid domain). But the downside is our domain would be logged in the certificate transparency log. What is the downside of being on the log?

<comment><USER19> Whether you like it or not, all certs in the near future from all providers will be logged anyway.

Most sysadmin don't like their intranet adresses being in the log so as to not provide intel to intruders.

<comment><USER18> Ah. I didn't realize we will eventually. So if I get a cert for *.dev.example.com I am exposing just dev.example, but not foo.dev.example.com?

<comment><USER20> Yes, the log is static. It only contains the subject name if the certificate.

But there's little to fear from exposing internal domain names. DNS names are more or public knowledge - they are transmitted unencrypted, end up in plenty of caches etc. Attackers can probably brute force it or the PTR records anyway.

</comment></comment></comment></comment><comment><USER21> Awesome! This was the second of only two steps to remain until I can fully turn on HTTPS. Now they just need support for IDNs (which they've also announced) and LetsEncrypt will be functionally complete from my point of view.

</comment><comment><USER22> What's this mean? If a site only has an AAAA record it can now get a cert?

<comment><USER23> I think you could always get a cert via the ACME DNS challenge for an IPv6-only/AAAA-only domain. But you could not talk to the ACME/API endpoint from an IPv6-only system, so actually requesting and retrieving the cert would have to happen on another, IPv4, system.

(Just as a sidenote: you _never_ need to request and retrieve the cert on the system that the domain name points to. That is just the easiest way and the workflow most clients suggest, since it also makes a lot of sense.)

<comment><USER24> Would it be possible to set up a somewhat isolated VM that has the sole purpose of requesting certs? I get stuck on needing either the webroot or standalone method of certbot-auto.

<comment><USER25> What's your specific problem with the webroot method? I'm using it on my systems, and contrary to popular belief, the certbot can easily run as a non- root user when using the webroot method. (My configuration is at [https://github.com/majewsky/system- configuration/blob/master...](https://github.com/majewsky/system- configuration/blob/master/hologram-letsencrypt.pkg.toml) .)

</comment></comment></comment></comment><comment><USER26> Does anyone know if Let's Encrypt supports DNSSEC validation? I mean, do their data center recursive DNS servers do DNSSEC validation?

I'm wondering how easy it would be to forge DNS responses to their servers checking that I control a domain name.

<comment><USER27> DNSSEC is enforced at the resolvers.

<comment><USER26> Thanks. Since my zones are secured with DNSSEC that makes me feel a bit safer.

</comment></comment></comment><comment><USER28> > We're looking forward to the day when both TLS and IPv6 are ubiquitous.

Kudos to Lets Encrypt for their great work on the former.

A single sad tear for the state of the latter.

</comment><comment><USER29> Expected but when is Tor support coming? I read a forum thread indicating it would be nigh on impossible due to .onion tld status.

<comment><USER30> I'm asking out of ignorance because it'd never occurred to me before just now: would you _need_ HTTPS on a Tor site? I thought Tor itself would handle trusted encryption for you. Or is that as a layer of defense against malicious nodes?

<comment><USER17> It does for hidden services, but HTTPS allows the browser to know the connection is secure, which lets it apply different rules, like mixed content blocking (if, say, you're browsing a .onion forum and someone links to an image hosted on a non-onion address).

<comment><USER31> Then perhaps browsers should whitelist .onion as "secure" regardless of protocol?

I'd also like to see whitelists for the reserved-for-private-use IPv4 ranges and a .local or .home TLD, since those are circumstances where HTTPS doesn't give you much either, and where getting a certificate is unreasonably difficult.

<comment><USER32> This is a terrible idea, since nothing stops the onion TLD from being spoofed. The way the browser requests resolution of .onion names is no different than any other; to visit them you need to be communicating through some sort of proxy (possibly on your own computer) that intercepts both your DNS requests and the HTTP requests to the returned address. HTTP does not provide any way to validate that your are connecting to the intended proxy instead of a malicious one.

These same thing applies to .local, .home, and private IPv4 ranges, which can all be spoofed depending on where an attacker is in your network.

<comment><USER31> > This is a terrible idea, since nothing stops the onion TLD from being > spoofed. The way the browser requests resolution of .onion names is no > different than any other; to visit them you need to be communicating through > some sort of proxy (possibly on your own computer) that intercepts both your > DNS requests and the HTTP requests to the returned address. HTTP does not > provide any way to validate that your are connecting to the intended proxy > instead of a malicious one.

Presumably you wouldn't be visiting a .onion address if you're not already connected through a Tor instance you know about.

> These same thing applies to .local, .home, and private IPv4 ranges, which > can all be spoofed depending on where an attacker is in your network.

Which would be exactly the point, those are OK to spoof, since you'd only be visiting them through a trusted network, where nothing can be externally verified anyway.

<comment><USER32> > Presumably you wouldn't be visiting a .onion address if you're not already > connected through a Tor instance you know about.

How about a link? Or even more problematically, using its now trusted status to load inside an HTTPS page!

> Which would be exactly the point, those are OK to spoof, since you'd only be > visiting them through a trusted network, where nothing can be externally > verified anyway.

What? How is it okay for safe-place.home to be trusted when an attacker can spoof the DNS resolution upstream (like ISPs already routinely do to point you to ads)?

The whole point of distinguishing HTTPS connections is that they provide some way of guarding against spoofing of name resolution/packets and snooping. Nothing about how .local, .home, .onion, or local-reserved IP ranges are handled by browsers prevents these from being attacked, in many cases even from outside your network. If you curl 192.168.80.1 (assuming that's not within your subnet), your router will happily shoot some packets at your ISP. The situation for the others is even worse.

<comment><USER31> > What? How is it okay for safe-place.home to be trusted when an attacker can > spoof the DNS resolution upstream (like ISPs already routinely do to point > you to ads)?

I guess I was unclear, my point was that I think some TLD should be dedicated for home networks, with ICANN and especially browsers recognizing that.

ISP spoofing wouldn't be an issue because if you used these TLDs then legitimate requests would never reach that far anyway. If not, well, you wouldn't be visiting such domains anyway and there would be nothing to spoof.

> If you curl 192.168.80.1 (assuming that's not within your subnet), your > router will happily shoot some packets at your ISP.

But that's not an issue, because if it's not on your subnet then you wouldn't be visiting it in the first place. Any snooping ISP could just as easily make you visit some other address instead that actually did have a TLS certificate, as they could make you visit that. In the worst case, you could make the browser check your subnet mask. But since the contents on those IPs will be unique from local network to local network anyway, I really don't see the point in bothering.

<comment><USER32> > I guess I was unclear, my point was that I think some TLD should be > dedicated for home networks, with ICANN and especially browsers recognizing > that.

If it's for home networks, .home resolution would usually occur at the DNS server on your router. How does the browser know that your router follows the new rules and won't route that DNS request up to your ISP, and therefore should trust the request?

> But that's not an issue, because if it's not on your subnet then you > wouldn't be visiting it in the first place.

Unless your attacker can get you to click a link? That's a pretty easy thing to get users (especially the unexperienced) to do. Or they can sneak it into a secure page and monitor requests/serve malicious assets.

> In the worst case, you could make the browser check your subnet mask. But > since the contents on those IPs will be unique from local network to local > network anyway, I really don't see the point in bothering.

This ignores the case where your local network is either (a) infiltrated (b) a coffeeshop. The second being _super_ common, and would need to be guarded against by the browser having some sort of Windows-style public/private network distinction, which users would remember to configure correctly.

> But since the contents on those IPs will be unique from local network to > local network anyway, I really don't see the point in bothering.

I'm not seeing the connection. If someone with control of your public internet connection (i.e. what HTTPS is designed to guard against) sends a response when your browser requests something from that address, what does it matter what that address does in another local network?

Everything I've described here has been an element of a real attack where something somewhere was more trusted than it was supposed to be. This would add a massive array of attack vectors, and at best would indicate to the user trust in something that has no reason to be trusted.

If you're doing something on your local network, it makes a lot more sense to just create a self-signed CA and put the root on your devices. In the onion case, you should use HTTPS between you and your proxy (e.g. with a *.onion wildcard cert) to make sure you actually connect to your proxy.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER23> This is in no way criticism against LE, where I work _nothing_ is IPv6 and we do not even have it on any agenda.

But when an "we are going to change the future of the internet"-project makes IPv6 a Prio-2 feature (to be added later, not native from the start) it just shows that we are really not there yet.

<comment><USER33> It's getting to be time for us to admit what the holdup is—IPv6 hasn't been deployed because IPv6 NAT ("NAT66") isn't a thing.

There are a million reasons NATs are terrible for the internet. But they're _used_ on IPv4, and IPv6's technical goal of increasing the address space is tied up into the technical goal of killing NAT, immediately, and changing the way a lot of people think about networking. For instance, end-user ISPs are expected to give you a /64 or more instead of a single IPv6 address so that you don't need to NAT, but many of them don't, because that's not how people think about addressing. If you have a NAT-using site and you want to switch to IPv6, you have to pursue the political goal of convincing your ISP to think differently about addressing.

Meanwhile, IPv4 and IPv4 NAT _works_. I'm typing this from behind a NAT, you're probably reading it from behind a NAT. It's not ideal, but, rough consensus and running code.

As soon as we all put our collective feet down and insist on IPv6 NAT implementations, such that IPv4 sites can move without rearchitecting their environment (whether or not that rearchitecting would be a good thing), IPv6 will get deployed quickly.

<comment><USER30> > For instance, end-user ISPs are expected to give you a /64 or more instead > of a single IPv6 address so that you don't need to NAT, but many of them > don't

Name one? I've been on Comcast and Sonic, and both natively provide /64 networks. I've never heard of an ISP providing a /128.

> Meanwhile, IPv4 and IPv4 NAT _works_.

No, it _doesn 't_. It breaks a million things more than it solves and it makes the Internet worse (and vastly more asymmetric, but that's repeating myself). NAT needs to die in a fire, and there is zero political or technical motivation to inflict its brokenness on a new protocol that absolutely does not need it. Evidence: that many ISPs _are_ providing native, un-NATed IPv6 to their customers. Perhaps some don't, but someone will manage to screw up _any_ given feature. They need to fix their shit, not coerce the rest of the Internet to break itself for their convenience.

<comment><USER34> OVH's cheap dedicated servers, Kimsufi, only provides a single /126 IPv6 address, instead of the recommended /64 block :(.

<comment><USER12> I have heard rumours that, although this is what they state, they actually in practice do assign the entire /64 to the machine. Not sure if this is true and I have not tested it myself.

I see it largely as an attempt to do market segmentation and limit the usefulness of Kimsufi to push people towards their other brands. Unfortunate, but...

<comment><USER34> Well, it does work (just have to allocate the IPs statically) but since you're kinda not supposed to do that, I assume it will either stop working one day, or get my machine voided as a TOS violation or whatnot.

It really is unfortunate. Not having to use a proxy for the sole purpose of sharing the 80 port would be nice...

</comment></comment></comment></comment></comment></comment><comment><USER35> Great news.

</comment><comment><USER36> You mean it didn't?

<comment><USER37> Does that mean that you are not using IPv6? I know I am not...</comment></comment>


Simple Software Manifesto - <AUTHOR> https://arewesimpleyet.org/ <comment><USER2> Every new generation of programmers starts a crusade to simplify without first trying to understand why we're not "simple" despite the previous generation having done the same. Of course, one possible explanation could be that without a constant crusade for simplicity we'd be in a far worse place; the "permanent revolution" explanation. But is this the right explanation? Turing Award winning computer scientist Fred Brooks had this to say:

> The complexity of software is an essential property, not an accidental one. > Hence, descriptions of a software entity that abstract away its complexity > often abstracts away its essence. (No Silver Bullet, 1986)

> Complexity is the business we are in, and complexity is what limits us. (No > Silver Bullet Refired, 1995)

The reason "complexity is the business we are in" is that software doesn't live in a vacuum. Like the economy and like bureaucracy, it is built to serve a certain social purpose. Of course, there needs to be a constant struggle not to _over_ -complicate, but it should be done with the understanding that over- simplification is just as problematic.

<comment><USER3> I think that all else being equal, simplicity is to be preferred over complexity. You're pointing out that all else is never equal; ok, cool; but it's hard to imagine anyone finding value in deliberately complicating things. Simplicity seems to be an inherently valuable trait in system design/maintenance. Can you oversimplify? Sure. Does that justify not striving to reduce to the simplest functionally complete, scalable, and maintainable design? In my opinion, no.

<comment><USER2> My points are:

1\. Even software with no "accidental" complexity is necessarily complex (in fact, that's why it's useful).

2\. If the manifesto's point is that we should stop _over_ -complicating things, then wouldn't it better to first examine whether, and if so why, things are overcomplicated now, given that the previous generation also went on a crusade to abolish (over)complication? If the point is to declare "don't overcomplicate!" then, if you believe things are overcomplicated now, why is that declaration useful if it's clearly not effective?

<comment><USER4> Yep, now that we have lots of version controls histories publically available there is no excuse for this.

</comment></comment></comment></comment><comment><USER5> > There is only one correct way to do something .... so take great care in > doing it right the first time.

I hear this a lot, in various contexts, and I even used say it a lot. But this has actually stopped making sense for me. I don't think I've ever genuinely seen a case where there is only one obvious way to do something, or that there's a way that is obviously the "right" way. Everything non-trivial is chock full of trade offs and hidden corners that could make it obviously wrong in hindsight.

Does someone disagree? Can you share your perspective.

<comment><USER6> Completely agree. I'd change this whole line to something like:

"There is never only one way to do something. You can't know which is correct, so go for the simplest option that is easy to change later"

<comment><USER7> > go for the simplest option that is easy to change later

And once customers get their hands on your simple thing, management won't let you change it to something better...

(Also known as, "The demo becomes the product.")

Your simple solution is embedded in a complex social reality. If you're not taking that into account, it probably won't stay simple for long: i.e. simple to maintain, simple to operate, simple to build new features on.

</comment></comment></comment><comment><USER8> > We target C99 or C11

C is anything but simple. It looks simple on the outside, then beats you to death with undefined behavior and either overexplicitness (if you don't use macros) or nested-macros hell (if you do). Their choice of C is routed more in the authors coming from the Unix community, not in their desire for simple software.

If anything, they should start with a truly simple programming language. Something like Go minus huge runtime and maybe plus basic generics.

<comment><USER9> I was going to say something similar. If you really want a simple system, maybe avoid C and stick with something memory-safe. Overall, I'm not sure what to make of this page. C is simple, git overdid it, etc. Oh ok.

</comment></comment><comment><USER10> Yet more suckless-style posing. Ho fucking hum.

Software is complex because the real world is messy. That messiness cannot ever be "abstracted away", it can only be hidden.

You can't derive a better software stack from first principles. What we have is the tireless work of many people, quite a few of whom are smarter than you. Rust and LLVM, for instance, are much more fit for purpose than "my pet compiler backend + my favorite C subset" because even the most disciplined C subset is still full of UB traps and Rust has had years going on a decade of work, by many hands, addressing the very cases where C falls flat on its ass. The same with LLVM vs. whatever brain fart J. Random Hacker thinks up as an ideal abstract machine,

<comment><USER11> > That messiness cannot ever be "abstracted away", it can only be hidden.

How would you hide something without hiding it behind an abstraction?

> You can't derive a better software stack from first principles.

What do you consider "first principles"?

</comment></comment><comment><USER12> I have come to believe that we are dealing not with a domain specific problem but with the human condition itself. I call this my gradient theory.

Consider human learning as an optimization in the mathematical sense. Note that humans are good at learning things along a gradient. Now note that this is not just about how the learning works but also how far we go. For example, if a human thinks it is too cold, they turn the knob not where they want the temperature to be but much farther up, and then when the desired temperature is reached, they turn it back. Or consider how you find a page in a book. You overshoot and then go back. Or how you choose the speed when driving. You go as fast as you feel comfortable, and you find that by overshooting and correcting.

I think this is inherent in humans. Evolution made us this way.

The downside is that humans are bad in environments where there is no clear limit or where you can't correct after overshooting, for example driving too fast and then having an accident, or substance abuse.

Now take this thought and apply it to programming.

If you find me a way to work with complexity, for example modularization, abstracting it away, etc., it will not lead to us writing a good version of the software we had before. It will lead us to writing larger software that will then again be at the limit of what we can do with the technology available.

So far it is just a theory, but I can see in my software selection that all the programs that I consider good and simple in the sense that this manifesto describes, all of those are old programs that were written when the functionality they offer was the limit of what was achievable with technology back then. The reason they did not metastasize into something unwieldly is that a) they did but it was still comparatively small or b) they already contain all the features anyone could think of.

Try comparing two programs of your choice that do essentially the same. You will find that if there is an old and a new one, the newer one will be bigger and slower and usually work in less environments and have less features.

It also turns out that no two people have the same idea of what "simple" means. For me, with software, simple means it does all I really need it to do, and nothing else. Other people usually need it to do other things and they don't need some of the stuff I need.

<comment><USER11> > If you find me a way to work with complexity, for example modularization, > abstracting it away, etc., it will not lead to us writing a good version of > the software we had before. It will lead us to writing larger software that > will then again be at the limit of what we can do with the technology > available.

So basically Parkinson's law applied to complexity rather than time. I think it's definitely true. More complex software is (ideally) more versatile, handles more cases of the problem domain, and so there's competitive pressure to make software as complex as is feasible to solve as many problem as possible, and thus, provide as much value as possible. The ceiling is exactly as you said: what level of complexity is manageable given the other constraints that must be satisfied (performance, robustness, etc.)?

Edit: to be clear, I mean "essential complexity" above, not accidental complexity from poor designs. Being able to handle more essential complexity is a good thing.

</comment></comment><comment><USER13> Just from some discussions from the last weeks, I don't think simple is possible anymore: just Unicode and Font Rendering by themselves have a complexity which boggles the mind. If you then consider stuff like authentication/authorization, i18n and support for blind people its clear that "simple" just doesn't cut it.

<comment><USER14> A common excuse is that they target "expert users" and not everyone. While I can imagine a specific set of "expert users" that this excuse will work, not every "expert user" will agree to that.

</comment></comment><comment><USER15> Every software has a reason to be complex. Often software authors desire to make simple software by removing such features regarded 'complex' without understanding why those features were introduced (mostly because they don't use them). The consequences is that people who are in the minority can't use the software.

As a CJK person, the most prominent example is multilingual support in the FOSS community; I still have not seen any Linux GUI app with proper out-of- the-box IME support (and that includes DEs, WMs, etc...) mostly because for some reason (hint, most are from the western world) the devs of Linux distributions turn off IME support by default. Is it really 'simple' to turn off some features that you don't use, resulting in total breakage of apps when the feature is in use? Really?

<comment><USER16> I tihnk that's just a symptom of how FOSS devs think about GUIs. GUI is largely not something FOSS developers actually use. The design of FOSS GUIs makes it seem as though "GUI" is something they've read about in text books and personally only see value in as a fancy tmux. Consequently, the only mental model they have for someone who actually uses a GUI is the kind of person who only needs a web kiosk.

</comment></comment><comment><USER7> Ugh, I don't share this sentiment.

Each new generation of programmers creates new tools to replace the 'complex' ones that came before without even bothering to understand them.

In my 25 year career I've seen it multiple times. We are all going downhill.

Attention spans in the smartphone era might be preventing people from reading the f*ing manual.

This means that any system or library which has an operating model that must be understood before it can be effectively used¹ simply can't be understood by the majority of programmers today.

1: The currently popular things in software world are mostly the type of system that lack such a model and which can be 'mastered' by asking discrete questions on stack overflow. This lack of complexity causes a loss of expressive power.

<comment><USER17> And no one even writes the manual anymore. Software project documentation is a disaster.

</comment></comment><comment><USER18> Don't #2 and #3 contradict?

> Constraining the user to simple tools encourages creativity and allows the > system to be more quickly understood

> There is only one correct way to do something, because two ways would be > more complex

When you encourage creativity with combinations of simple tools, creative people will come up with multiple ways of doing things.

<comment><USER19> I read the "only one correct way" to be a per-tool guideline. When looking a tool to do a specific task then (ideally) the tool should just process input with minimal futzing with configuration options. If you need to do something special then you should have a special tool.

<comment><USER20> That seems reasonable. Though in the context of packages etc, i would like to then introduce the term "toolset". Just because a piece of software is a single package doesn't mean it has to be a single tool. It is much more _simple_ to have a bunch of tools for related tasks collected in a single toolset rather then making seperate apps for each one- never mind the overhead that the last approach would create.

</comment></comment></comment><comment><USER21> My related observation is that the phrase _" simple yet powerful"_ is the biggest cliché in developer/techie-facing marketing of software:

[https://github.com/search?q=%22simple+yet+powerful%22&type=C...](https://github.com/search?q=%22simple+yet+powerful%22&type=Code)

Is there any camp that DOESN'T consider their preferred way of doing thing to be simple yet powerful?

</comment><comment><USER22> It shouldn't be just about software. Hardware has become astonishingly complex as well. Forget about a high quality superhet radio containing 15 transistors, or a CPU with a manageable instruction set and an assembler that is a pleasure for a human to write in. No doubt modern systems are better than old ones, but I miss those times somehow...

<comment><USER23> Is that a bad thing though? Modern CPUs seem to be incredibly reliable. I think much of the complexity has to do with the the fact that all the low hanging performance improvements have long since been implemented. To continue to get faster, sometimes the only option is to increase the complexity of the design.

</comment></comment><comment> GR-O-ND This also reminds me of OpenBSD, in particular Ted Unangst. He's famous for deleting features that add complexity without any significant benefit or are only useful in a small number of edge cases. For instance, the replacement of sudo (with its massively complicated configuration syntax) with doas.

</comment><comment><USER24> My proposals, under high level programming something derived from:

\- thou shall learn how to use the wheel, thou shell not reimplement it instead

\- best software barely satisfies task it was made for

\- antipasta statement: no lasignas, no spaghetti and no raviolli

\- the code forming interface will be at max 20% of actual code written

\- no interface will be designed to solve all humanity problems for next millenia

I think that all those were hugest pains I had within 23+ years of development.

What were yours?

</comment><comment><USER25> "Simple Made Easy", a talk by Rich Hickey [https://news.ycombinator.com/item?id=4173854](https://news.ycombinator.com/item?id=4173854)

</comment><comment><USER26> What's wrong with Perl?

Also, with no C macros there are no C generics. I wish the author luck in maintaining that mess.

<comment><USER27> > What's wrong with Perl?

Looking at their central axioms, I would guess:

1 - simple software breaks in simple ways: weak, dynamic typing means a lot of problems trivially solved by static and/or strong typing.

2 - simple tools: no idea. I remember cpan being fairly great and perldoc being excellent.

3 - one correct way to do thing versus perl's TMTOWTDI central theme

4 - dependencies are part of the system: lots of CPAN modules, so there's a tendency to import black box modules instead of writing it yourself and retaining ownership.

5 - standards are useful but full of cruft: perl has a much larger standard lib than, say, Go or Python as far as I understand.

As to why it would be what's wrong with git: after a cursory glance at the git repo (0), I would guess the problem is that it still contains .perl files?

[0]: [https://github.com/git/git/search?l=Perl](https://github.com/git/git/search?l=Perl)

</comment></comment><comment><USER28> Simplistic software manifesto.

1\. All software breaks, so let's avoid failure by not even trying.

2\. Constraining the user to simple tools, because the user is an idiot.

3\. There is only one correct way to do something, and it's mine. Coming up with one mediocre way to do things is already a great accomplishment; thinking about design tradeoffs is unthinkable. YOLO!

4\. The dependencies of a system are not fun, so they are your responsibility. Good luck.

5\. Standards are a source of inspiration, idiots (see #2 above) aren't going to run into interoperability problems, missing features, and other consequences of implementing only the "simple" parts of the standard.

The cproc missing features list is a brief but very insightful example of this way of thinking (I mean, this way of _not_ thinking).

</comment><comment><USER29> This reminds me of suckless.

[https://suckless.org/philosophy/](https://suckless.org/philosophy/)

<comment><USER22> There is something weird about trying to put a layer of simple software on top of an astonishingly complex system kernel. (RiscOS, for example, looks more balanced in that regard.)

<comment><USER29> I'm tempted to suggest that the suckless people are on Linux because they can't handle OpenBSD or Plan 9, but that would be cruel.

</comment></comment></comment><comment><USER30> I suggest the biggest problem we have in the industry is a laziness towards engineering and an unwillingness by business to accept the law of fast, good and cheap.

Very few organizations use real data and the scientific method to make decisions on architectures and design. Most rely on anecdotes, gut feel, politics and software cults to make these decisions.

Consequently, there's rarely real proof on the advantages of one approach over another. The rigor, and thus cost, needed to make those determinations is perceived to be too great but the reality is, those perceptions are often unfounded when considering the total cost of software.

This all hit me years ago when working for company that had to go through CMMI certification. As part of the effort, a consultant would come out quarterly to coach us along and monitor our progress. During that time period I noticed a significant portion of my time was now spent on the implementation of the process. When the consultant was out for a visit I told him I had concerns because we did not have a charge number for the extra time it was taking to implement the process. He asked why that would be necessary and I asked, how else could you make an objective determination on the benefit/cost ratio of implementing the process being greater than one. Disappointingly, the blank stare ensued.

</comment><comment><USER31> I believe the complexity is inherent, and we haven't yet found a way to handle it gracefully.

I'd like to note that "simple software" means a different thing to the developer, and to the user. And in the grand scheme of things, the developer and all the code he wrote is highly irrelevant. We'd like to think we are, but how much does it really matter what kind of trowel the mason used, and how clean he kept it? What matters is the cathedral he built, the feel of the space within, the durability of the walls.

In my view, the complexity is closely related to scale. I'd like to draw a parallel with nature, where things scale linearly up to some critical point, at which a system goes through reorganization and only then continues to grow. I think mimicking this phenomenon can be a path towards software simplicity.

</comment><comment><USER32> > Constraining the user to simple tools encourages creativity and allows the > system to be more quickly understood. This supports the removal of > complicated features.

Oh god, no. This horrible idea escalated so far that there is now a billion dollar industry (RPA) automating braindead tasks just because software can not be arsed any more to include API, code/CLI based access or, to steal the wording, complex tools at all.

IMO this is another round of weirdly religious adherence to "UX" schemes which ends in massive damage, like the unconditional love for whitespace everywhere.

Please stop using "simplicity" as a cop out to prevent having to build software that, for example, has basic things like stack deletion. The result will just suck.

</comment><comment><USER33> > POSIX should probably be replaced when the time is right.

Everyone loves to bash POSIX. The idea of replacing it lingers on, but has anyone considered the idea of updating POSIX to remove the gnarly parts? This is a somewhat incompatible change and would have to be versioned appropriately.

</comment><comment><USER34> simple for developers != simple for users

</comment><comment><USER35> I would suggest sinit for init / supervisor: [https://core.suckless.org/sinit/](https://core.suckless.org/sinit/)

</comment><comment><USER36> > Simple software manifesto

Excellent!

> There is only one correct way to do something

!!! The correct way to do something depends on what else you're doing at the same time.

> Simple software in practice > C

Dear God

</comment><comment><USER37> Why use C? Constrained with no macros? What about undefined behavior?

Rewrite in Zig!

</comment><comment><USER38> apk truly is a great package manager. Glad to see it mentioned here. It's such a bummer going back to aptitude after using it in my lighter weight VMs.</comment>


What are the legal issues around web scraping? - <AUTHOR> http://evan.law/2020/09/09/what-are-the-legal-issues-around-web-scraping/ <comment><USER2> This article is just a teaser to get you to hire the author (who is an IP lawyer) to explain the issues raised. It's not very informative.

<comment><USER3> This is useless. It's just an ad and IMO inappropriate for Hacker News.

<comment><USER4> Definitely, however HN the comments are insightful

</comment></comment></comment><comment><USER5> 1\. A website operator must capture your IP address to issue a cease-and- desist or come after you legally.

2\. It is 100% legal to mask your IP address when accessing websites.

3\. Always mask your IP address and you will never hear from the website owner.

(I'm not condoning anything illegal, and I have public-facing web-servers so I know all about the pain from aggressive bots and scraping. I'm just pointing out the facts around this instead of some vague lawyer's blog post with a "who knows, contact us" at the end of the article.)

From the website owner's perspective:

1\. I get nailed with bots and scraping all day. If you do something outstandingly stupid and become the signal within the noise: I'm just going to harden my bot detection and mitigation instead of calling up my lawyer.

2\. Almost all websites published today are running blind. They have no idea you are a bot, and they are more likely to get excited in the bump in traffic because "of course it's from the money we poured on that marketing effort".

3\. If a website has a bot problem, they will recognize your scripts and give you guidelines on how not to be a jerk when scraping their content. This is usually backed up by IP based throttle limits.

4\. I'm more afraid of this lawyer's hourly price, not the possibility of damage caused by web-scraping against my servers.

<comment><USER6> what do you mean by masking your IP address? Tor?

<comment><USER5> Tor is great, but they also publish a list of IP addresses for their exit nodes, so website owners can block that easily.

Random open web proxies are harder to defend against.

You can also spin-up your own server from a cloud provider and proxy your web traffic through that.

</comment></comment></comment><comment><USER7> When it comes to web scraping, it feels like what is considered a crime is highly dependent on how eager prosecutors can be to charge you with unauthorized access. It is common to read tech articles about security vulnerabilities where web pages were left unprotected, the flaw being verified by a journalist or the person reporting the issue.

This is what was used against weev[1] when he crawled an AT&T website and dumped subscriber e-mail addresses, to sentence him to 41 months in prison. Don't get me wrong, weev is a terrible person that society should be protected from, but the severity of this sentence felt completely out of place for both the supposed "crime" and the banality of what he did.

[1] [https://en.wikipedia.org/wiki/Weev#AT&T_data_breach](https://en.wikipedia.org/wiki/Weev#AT&T_data_breach)

<comment><USER8> Correct me if I'm wrong but it wasn't publicly available in the sense that it was linked to from some publicly facing website. I'm understanding that they found some URL scheme that spit out these records. So quite an edge case. People have been prosecuted for stealing data from unsecured s3 buckets too.

If you could reasonably infer that you're not supposed to have access, try to obtain it and then go on to share that data with the media (which only acknowledges you know you're not supposed to have it) then I don't see a huge problem with prosecuting it.

Usually by scraping I understand accessing something that's already being accessed by clients and you're merely automating what's already happening.

Now if someone inadvertently obtained this kind of data without trying (say part of a bigger scrape) and didn't use or distribute it then obviously I don't want to see that prosecuted. And I doubt it would be.

</comment></comment><comment><USER9> This article offers almost no information at all. I'm fine with someone publishing an info piece to drum up business, but usually the author makes an actual attempt to educate the reader. This article provides essentially no substantive information, or info a reader could take action on.

And if the author is going to bring up ToS and CFAA violation, it's negligent to not mention that recent high-profile precedents like LinkedIn vs HiQ exonerated the scraper of ToS and CFAA liabilities. Other precedents may say different for specific situations (pages behind a login?), but bringing up these legal obstacles with no indication that the best precedent we have makes them non-issues for scraping public information feels like the author just wants to scare the reader into thinking they need to call for a consultation.

</comment><comment><USER10> There are lots of potential legal issues that may arise depending on your use case. Check out the webinar we did at Scrapinghub on legal compliance in web scraping to get an overview of some of the potential legal issues you may run into. [https://info.scrapinghub.com/webinar-legal-compliance-in- web...](https://info.scrapinghub.com/webinar-legal-compliance-in-web-scraping)

</comment><comment><USER11> > What are the legal issues around web scraping?

delusional.

rhetorical language device intending to pretend you can make something public then dictate what public means to you.

</comment><comment><USER12> Step 1 to stoping scraping is to block all traffic from AWS.

</comment><comment><USER13> 100% legal.

If you're Google.

<comment><USER14> So true... so many sites have a clause in their T&C's saying that scraping is not permitted but they're perfectly happy to let Google do it. Google should totally blackout any site with this kind of garbage in their T&C's just to raise awareness and get people to remove it.</comment></comment>


10 years stuck at minimum wage - <AUTHOR> http://qz.com/240827/ive-worked-at-mcdonalds-for-10-years-and-still-make-7-35-an-hour/

<comment><USER2> "As fast-food workers, we prepare burgers and fries, not balance sheets. We struggle to survive from paycheck to paycheck, without million-dollar annual bonuses or second homes. We often work behind the scenes—or counters, getting little of the credit for billions of dollars in company profits."

I feel bad, but why is a mother of four still working a minimum wage job at Mcdonalds? Minimum wage was never meant to support an entire family (especially a family of four).

I'm not sure what more credit these workers want. They don't have that much responsibility. Their job is to either put pre-cooked food in the fryer or accept money at the register. They get a paycheck on time for a job completed. This should be enough.

"and many of those barely pay more than minimum wage."

So Mcdonalds is STILL paying more than minimum wage in many instances, and it's not enough? I like how this is buried in the article..even though the title would make you believe that they only pay minimum wage.

"We'll discuss how to escalate our fight for $15 an hour and a union."

I feel like basic economics has been lost.

An increase in only Mcdonalds wages, might work, but Mcdonalds would never do this because the prices would increase and couldn't stay competitive anymore.

An increase in minimum wage would only decrease the spending power for everyone because goods and services that rely on minimum wage workers (which is pretty much everywhere), would be forced to increase their prices.

You can see examples of this in any country with a high minimum wage. I went to Australia a few months ago and the cost of a combo meal at Mcdonalds was almost $10. At this point, your spending power doesn't really increase that much.

</comment><comment><USER3> There is a risk (for fast food workers) that as wages rise, there is more of an incentive for employers to explore ways of replacing workers. For example, most people are capable of placing orders on a touchscreen device and seem comfortable with this process (as evidenced by self-checkout lanes at grocery stores). A fast food company who used to pay $8/hr for cashiers but is now paying $15 and maybe will pay $20 soon might find it more economical to replace 4 cashiers with touchscreen ordering stations.

<comment><USER4> That's why they want a union. Raise the minimum wage and reduce or eliminate the risk of the job being eliminated.

<comment><USER3> All the more reason for this hypothetical company to want to replace them now, before they unionize.</comment></comment></comment>


Getting Things Done, Applied - <AUTHOR> http://www.wired.com/techbiz/people/magazine/15-10/ff_allen?currentPage=all

<comment><USER2> What I did starting in 1967

1\. Learn speed reading 2\. If it can be acted upon immediately, do it! 3\. If it requires delayed action, put it on the todo list. 4\. Else discard it. If its important, it will come back again. 5\. Check todo list, for next thing to do and do it or delete it!

My in box was empty and my todo list was short 98% of the time AND I got things done.

The rest of his crap is a con to justify the cost of his clinics.

<comment><USER3> I wouldn't say it's all crap I really didn't think he was pushing his clinics too much in his book. And I did get some decent advice about dealing with email overload.

</comment></comment><comment><USER4> I knew there were some New Age overtones in GTD, but I hadn't read this article, (September 2007) so thanks for that.</comment>


Ship hits new Panama Canal wall, sparking design concerns - <AUTHOR> https://www.theguardian.com/world/2016/jul/26/ship-hits-new-panama-canal-wall-sparking-design-concerns <comment><USER2> I'm confused. An expanded canal, that allows for ships three times the size, is harder to navigate? How does that work?</comment>


How long until Apple is bigger than Microsoft? - <AUTHOR> http://blogs.zdnet.com/hardware/?p=2850 <comment><USER2> Actually, I would hate to see Apple in MS position. Let it remain small(er), yet churn out innovative, compelling products as they have been doing in past few years, and remain profitable!</comment>


Building Linus Torvalds' PC Build (video) - <AUTHOR> https://www.youtube.com/watch?v=Kua9cY8q_EI <comment><USER2> To save you from YouTube ui, here's the article mentioned in the video:

"You can build Linus Torvalds' PC: Here's all the hardware and where to buy it": [https://www.zdnet.com/article/you-can-build-linus- torvalds-p...](https://www.zdnet.com/article/you-can-build-linus-torvalds-pc- heres-all-the-hardware-and-where-to-buy-it/)</comment>