LCFS: A New Container Filesystem for Modern Datacenters - old-gregg https://portworx.com/lcfs-new-filesystem-modern-datacenter/ <comment><AUTHOR> How many more file systems are we going to see before we realize that the file system is the wrong layer of abstraction to be solving the deduplication problem?

<comment><USER2> I think the filesystem is a fine place to do it, I just wish more filesystems would give us the option to do deduplication offline.

ZFS's dedup uses grotesque amounts of RAM since it insists on holding the dedup table in it and only doing dedup in realtime.

btrfs however allows you to scan the filesystem and do dedup of existing data later (even while it is mounted), using no RAM at all. This is one of a couple of places where I think btrfs is actually becoming a better-designed filesystem than ZFS.

<comment><USER3> Again, the focus of this blog is not about a de-duplication story, but about a new docker storage driver.

<comment><USER2> Well thankfully open discussion is a feature of Hacker News, not a bug.

</comment></comment></comment></comment><comment><USER4> Hi, I am one of the contributors on this project, and happy to answer any questions... LCFS will also be supporting the snapshot driver discussed here - [https://github.com/docker/containerd/pull/484](https://github.com/docker/containerd/pull/484).

Currently, the install experience is not the greatest... We rely on the Docker v2 plugin interface and that is still in beta. We are working with the Docker team to help us smooth out some of these rough edges.

LCFS will also support other container formats and any help from the community is much appreciated.

<comment><USER5> This looks amazing. One of the biggest pains of using docker is that the storage driver inevitably explodes the system.

Is anyone using LCFS in production? Why did you get started building it?

<comment><USER4> LCFS is not yet being used in production as far as I know

We started building this because there are a lot of issues running containers in production, specifically around image space and inode management and also with page cash usage.

This file system was specifically designed to address memory and storage management of container images

</comment></comment></comment><comment><USER6> Really nice to see the new file systems addressing this issue, but, as many including me hurt by AUFS, I am being cautious. Do you think it is battle tested enough? As it is different than other merging ones, it is a little more dangerous than them as well since it works on block level? Not trying to be negative, I really liked it, and I am not even happy about even a docker load command performance over my overlay2, and will defintely be trying LCFS.

<comment><USER3> It is a mini-file system, working directly on top of a block device. Having something else in between is not a good idea. Much of the data handled by docker storage driver is temporary and does not even have to hit disk before thrown away.

This project is currently in experimental mode. We would appreciate any feedback you may have.

</comment></comment><comment><USER7> Does anybody know of a good deduplication script / tool that runs on every filesystem and can be used as a nightly cronjob?

<comment><USER8> Here's a quick one if your filesystem supports hard links:

        #!/bin/ksh

        typeset -A hashes

        find . -type f  -exec shasum -a 512 {} \; |
            while read hash filename; do
                if [[ -z ${hashes[$hash]} ]]; then
                    hashes[$hash]="$filename"
                else
                    echo ln -f "'${hashes[$hash]}'"  "'$filename'"
                fi
            done

Pipe to /bin/sh if you're happy with the results.

<comment><USER9> Good answer, but unless you mark the files read-only/immutable, this ought to have big red flags all over it. Userland just can't do this safely in the general case.

<comment><USER8> Good point. This is for data that's not in use.

</comment></comment></comment></comment><comment><USER10> It's fuse based however, so that has some caveats. Interesting project though, could see a native port one day maybe.

<comment><USER3> This was done using _fuse_ to make it portable across different distros. We have not seen any major issues testing it as a docker storage driver, compared to the graphdrivers in use today.</comment></comment>


Ceph storage system - <AUTHOR> http://www.techbar.me/2013/05/ceph-storage-system/ <comment><USER2> Not sure if much has changed, but the last time I tried to install Ceph it wouldn't work under CentOS. That, and it was far too complicated to set up.

GlusterFS, on the other hand, was incredibly easy, although I am not a fan of FUSE due to the high CPU usage.

<comment><USER3> I still see Ceph as competition to supercomputer filesystems and not really as competition to GlusterFS. For example, the design of it directly attacks the problem of centralized metadata (especially useful after the DARPA project to bring it to Lustre failed).

I was working on an unrelated project with one of the designers of Ceph (UCSC's Scott Brandt) and in conversation he also seems to concur that Ceph was really built as a replacement for PanFS or Lustre (but still may be useful for other things of course).

Using it to replace GlusterFS still seems odd to me. It feels like they're both solutions to different problems.

<comment><USER4> Recent gluster vs ceph debate at LCA.

<http://www.youtube.com/watch?v=JfRqpdgoiRQ>

<comment><USER3> Nice talk, but I think I was basically trying to say that Ceph _really_ shines when you compare it to Lustre and PanFS.

It's not a clear winner when you compare it to GlusterFS because the original design wasn't intended to replace GlusterFS (although it may do a good job of this anyway).

<comment><USER5> Ceph is still a very long way behind Lustre for streaming bandwidth, so to say it shines would be a little much. Lustre's weakness is in scaling to support large file counts, but in real deployments this can be mitigated by using an MDS server with a lot of grunt. Ceph can't compete with Lustre for HPC deployments until it supports RDMA, and even with that it's still going to take a long time to reach Lustre's performance (which is close to line rate at this point)

</comment></comment></comment></comment></comment><comment><USER6> FLOSS Weekly recently interviewed the Ceph project. Link to the show: <http://twit.tv/show/floss-weekly/250>

</comment><comment><USER7> Here is a summary of how Ceph works. [http://muratbuffalo.blogspot.com/2011/03/ceph-scalable- high-...](http://muratbuffalo.blogspot.com/2011/03/ceph-scalable-high- performance.html)

<comment><USER4> Inktank is the professional services company backing Ceph. Here is their youtube channel with a whole lot of up-to-date videos:

<http://www.youtube.com/user/inktankstorage>

<comment><USER8> Glad to see Sage commercializing this and happy to see research activities from UC Santa Cruz's (where I went for grad/undergrad) Storage Systems Research Center making it into commercial use.

More SSRC work and some of the original CEPH papers can be found at:

<http://www.ssrc.ucsc.edu/index.html>

</comment></comment></comment><comment><USER9> The guys behind Ceph are very smart, I'd keep an eye on this one as it's going to be awesome!</comment>


USA Liberty Act of 2017 has passed - <AUTHOR> https://www.congress.gov/bill/115th-congress/house-bill/3989/actions <comment><USER2> Has the definition of "passed" changed recently?</comment>


Don't use JPEG-XR on the Web - <AUTHOR> https://calendar.perfplanet.com/2018/dont-use-jpeg-xr-on-the-web/ <comment><USER2> From time to time I've run an analysis on the use of alternative image formats on the web and it never comes out to be worth it.

New formats are not supported on all browsers so you wind up having to support the mainstream formats as well as one or more new formats. Rather than benefiting from reduced storage costs, your storage costs get multiplied.

Performance is also a problem, particularly when people add "yet another polyfill" to get an image to work without native support.

Once I get into eyeballing images closely I also end up questioning if the compression gains are real. For instance, one I read a comment on the article here

[https://calendar.perfplanet.com/2018/is-avif-the-future- of-i...](https://calendar.perfplanet.com/2018/is-avif-the-future-of-images-on- the-web/)

and looked closely I couldn't unsee that the AV1 had very different artifacts than the other compressed images. Not necessarily worse, but definitely different. Unlike the poster I liked the way the suit rendered but definitely there was a lot of blocking around edges that the poster interprets as "pixelation on the flag".

Like many things, the costs of switching image formats are definite, but the benefits are questionable.

<comment><USER3> > From time to time I've run an analysis on the use of alternative image > formats on the web and it never comes out to be worth it.

I just checked the Google homepage, and even the New Year's Eve Google Doodle currently being shown is a good ol' GIF, rather than a modern format such as WebP. And this is with me using a desktop Chrome browser that definitively supports WebP.

> Like many things, the costs of switching image formats are definite, but the > benefits are questionable.

One easy way to get some of the benefits without incurring the switching costs is to leverage Cloudflare Polish[1]. I've used it at a few places, and have had good experiences. Although the majority of gains came from optimizing images that were poorly optimized to begin with, rather than the incremental improvements that occurred from enabling WebP support.

[1] [https://support.cloudflare.com/hc/en- us/articles/36000060737...](https://support.cloudflare.com/hc/en- us/articles/360000607372-Using-Polish-to-compress-images-on-Cloudflare)

<comment><USER4> > I just checked the Google homepage, and even the New Year's Eve Google > Doodle currently being shown is a good ol' GIF, rather than a modern format > such as WebP. And this is with me using a desktop Chrome browser that > definitively supports WebP.

new-years-eve-2018-4995722058399744.2-law.gif

299 972 bytes

new-years-eve-2018-4995722058399744.2-law.webp

293 636 bytes

new-years-eve-2018-4995722058399744.2-law.apng

251 516 bytes

Interesting...

<comment><USER5> Meanwhile using ffmpeg's default settings it's 63k as h.264 and 45k as h.265.

<comment><USER6> Starting a video in most browsers is pretty expensive (it usually loads up various GPU features for hardware video decoding).

On some old and buggy machines, that might cause the browser to crash too.

Not something Google wants to risk for a doodle - that's why all non-trivial doodles require a click to load.

<comment><USER5> > Starting a video in most browsers is pretty expensive (it usually loads up > various GPU features for hardware video decoding).

OTOH it might be cheaper (relatively) on phones as they can offload the work to hardware components and ramp CPU down (or avoid ramping it up).

</comment></comment></comment></comment></comment></comment><comment><USER7> It's great to see a summary at the top of the article.

However, the summary (and the article) leaves out a relevant question that immediately comes to mind: Are other image formats decoded _off_ the main thread or is it just that decoding JPEG-XR is just more taxing compared to other formats, thus negating its advantages?

<comment><USER8> Yeah, this was my thought as well.

I'm fairly certain the article may be poorly worded. Certainly standard JPGs are decoded on the CPU side as well, are they not?

To the best of my knowledge, I don't think any browsers are doing JPG decoding in hardware; I think they use the GPU for primitives and compositing, etc.

I would look forward to being corrected if I'm wrong!

<comment><USER9> The CPU portion of JPEG decoding is relatively cheap (just Huffman + RLE) and easy to parallelize (DCT blocks are independent). Conversion and upsampling of YCbCr can be done on the fly on the GPU, and this even saves memory compared to using raw RGB/RGBA bitmaps.

Other formats use more expensive entropy coding, more block sizes, smoothing filters, and inter-block prediction which forces them to be decoded mostly serially.

<comment><USER10> Isn't the JPEG instruction set built into the CPUs now anyways? I would be shocked to find out they are not seeing as MPEG2 has been part of the CPU since the Pentium days.

<comment><USER11> There is no "JPEG instruction set" as such --- and what you may be referring to by MPEG2 is MMX, which is just a SIMD extension that certainly helps with JPEG and MPEG decoding, among other things.

<comment><USER10> Interesting. I just remember the days of the add-on hardware boards for real- time MPEG2 encoding for DVDs that were pretty much obsoleted when some sort of upgrade to the CPU was made. I officially went 100% software/CPU encoding in 2005. However, it looks like something more specific from Intel called Quick Sync came out in 2011.[0] This is a dedicated core on the die. Do browsers get access to that level of hardware?

[0][https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video](https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video)

</comment></comment></comment></comment></comment></comment><comment><USER12> I have several somewhat orthogonal thoughts on this:

1\. This is exactly what I've feared and suspected about JPEG-XR, webp, and JPEG-2000 (which Safari ever so quietly supports). I especially wondered about JPEG-XR after reading how Microsoft had done some great things with GPU-based hardware acceleration for JPEG decoding in IE11: [https://blogs.msdn.microsoft.com/ie/2013/09/12/using- hardwar...](https://blogs.msdn.microsoft.com/ie/2013/09/12/using-hardware-to- decode-and-load-jpg-images-up-to-45-faster-in-internet-explorer-11/)

I was puzzled by the fact that they never mention JPEG-XR and have never published a similar post about hardware acceleration for the more modern format. They've been incredibly lazy and unfocused on XR for years. It's effectively dead now.

2\. The author of the don't use it post keeps saying it gets "software decoded". This implies that this is unusual. But it isn't. Does he think JPEGs are not software decoded? They are in Firefox, and probably every other browser but IE11 and Edge. Someone correct me if I'm wrong. Browser makers have been surprisingly inert on using GPUs for image decoding. libjpeg-turbo is used by Chrome and Firefox, and possibly Safari. It's pure CPU, albeit with some SIMD. That's "software decoding". Hardware acceleration normally means GPU, or a fixed function unit/ASIC like you see for some video codecs.

3\. One percent? This was all about a -1% drop in some metric? What's the error margin?

4\. He implies that there's a lot of JS in this SPA. This seems like a possible confound that won't necessarily apply to rigorous web developers who don't overload on JS (who are admittedly extremely rare in 2019). Sure, they showed that XR was using more CPU, but so do all post-JPEG formats.

5\. Which leads to my last point. What about webp? He strangely never mentions it. It's like only JPEG and XR exist. webp performs very poorly in CPU efficiency compared to JPEG. What are trivago and Cloudinary seeing on that front? Is webp less taxing than XR?

<comment><USER13> A few clarifying points: * the author specifically said -1% Conversion. If this were AWS, that would mean a daily reduction of $6million in revenue! * the author specifically says this is for IE users. This implies two things: a) webp is not an option and b) likely we are also talking about lower class of hardware (mom & dad). I would expect WebP had older hardware probably has the same kind of performance tax, but it's harder to identify because at least with IE/Edge there is an implied age-of-hardware

</comment></comment><comment><USER14> > -1% negative impact on core business metrics including conversions in our > test of JPEG-XRs [...] Let's not use JPEG-XR on the Web.

Is this an overreaction, or a subversive call to improve the decode perf of JPEG-XR? I'm certainly on the performance-is-a-feature bandwagon myself, but 1% ux impact is pretty small. And shouldn't we assume that decode perf will improve in the near future? Is there any reason to assume that this problem is permanent and unfixable?

<comment><USER15> > but 1% impact is pretty small

The best you can say here is that it's within the margin of error, in which case the effect is so small it's literally unmeasurable. In that case the null hypothesis still wins, and you should not switch.

> Is there any reason to assume that this problem is permanent and unfixable?

IE11 as tested in the article is not getting anything other than security patches. The users who use IE11 are probably also the users most affected by poor performance since they most likely have the oldest hardware.

<comment><USER14> Interesting response, thanks. Your comment gives me even more questions, though!

> The best you can say here is that it's within the margin of error

What is preventing JPEG-XR from becoming faster and a net ux positive, tomorrow or in the future? Isn't the best I can say that today's results might be a bug or fluke or a quick and dirty implementation, and not a permanent problem?

> IE11 as tested in the article is not getting anything other than security > patches.

The title and conclusion both said we should avoid JPEG-XR on the web, not that we should avoid it on IE11.

You're right, the article focused on the problem with IE11. Is it already a net positive on Edge? (The article didn't say.) Should the shim to make it work in the old browser that fewer people use _really_ override the potential benefits on the newer browser that more people are already using?

Does it seem reasonable or unreasonable to make future web decisions depend on an old browser that's on life support today and has small and declining market share?

</comment></comment></comment><comment><USER16> Apparently only IE and Edge support JPEG XR, so the format not being used on the web will happen on its own anyway.

<comment><USER17> Well... With Edge phasing to Blink noone will support JPEG XR

<comment><USER18> The Edge team could still decide to integrate their JPEG XR implementation into Blink. That would mean Chrome would gain support for JPEG XR.

<comment><USER19> What makes you think the changes would be accepted into mainline?

<comment><USER20> They don't have to be; Microsoft could maintain that as a separate fork/add-in to Blink. In theory it sounds like it should be highly modular, so that shouldn't be a huge ongoing maintenance burden.

I don't know why they wouldn't just switch over to WebP in preference to that, though. It's better to have fewer of these formats around and the simplest thing to do is use the one that already has support.

<comment><USER21> > They don't have to be; Microsoft could maintain that as a separate fork/add- > in to Blink

Note that parent said "Chrome would gain support for JPEG XR", which I believe is what 05 is replying to.

</comment></comment></comment></comment></comment></comment><comment><USER22> This piece gives a hint where Trivago as a company went with Holacracy. The lack of focus is so obvious and seems to let individual developers create some work which is not just poorly worded but also nonsense:

Why should anyone test a format which is used by just two browsers with a very small market share and which will be never adopted by the company with the biggest market share (because Google has WebP). JPEG-XR is DOA and nobody should care and waste time on this (except the OP gone astray).

I like that there is a tl;dr right at the beginning though.

<comment><USER23> I'm not sure I'd be quite so blunt, but I do agree with your argument in the main: this is so low on the list of priorities that I can't help but wonder if someone at Trivago thinks they've run out of any actual revenue-generating development work.

It indicates to me that either the inmates are running the asylum - that devs are being given too much leeway to pursue personal interests - or (and?) that Trivago's day to day work is proving so dull that developers in some teams are inventing R&D work just to stay sane.

<comment><USER22> Yeah my tone was a bit to harsh and I didn't want to insult anyone but I am still wondering that nobody at Tribago gave him this feedback. However and hopwfully this useless piece of work leads him to some other more relevant stuff.

</comment></comment></comment><comment><USER24> Hardly surprising. JPEG is a very mature technology with deep support and optimizations going all the way to hardware support built into many (most?) cpus. If the real world savings are ~ 20%, it probably won't be worth it.

</comment><comment><USER23> Exotic file formats, like exotic JavaScript language constructs, tend to perform poorly because they are not optimisation targets. And they are not optimisation targets because - obviously - no-one uses them. To my knowledge there is no particular intrinsic reason that JPEG-XR decoding cannot be moved off-thread.

The solution to this problem is to start using these new technologies, so the browser vendors can prioritise work to make them performant. It is not to write articles that entrench that neglect and tie us to flawed, ancient technologies from the 1990s.

It gets worse. If there's one kind of meme that survives a long time in the development world, it's performance memes. Rumours about poor perf invariably end up haunting technologies with FUD long after any issues have ceased to be relevant.

<comment><USER25> Yep, it's kinda like Github and SSL/TLS. When they went SSL only it was a bit painful, but things have largely caught up and improved overall.

</comment></comment><comment><USER26> Going a little further than just JPEG-XR - definitely do performance tests for new releases. We had a bug that affected one page in our app. It made people cringe using the feature set that invoked it, but essentially something in angular was firing, and the thing it did caused a re-fire. It just kept looping when on these very specific pages in the app. I finally found it and killed it. But definitely don't go to production with a client CPU murdering bug. It can't be measured the same as network latency, etc... so you really need to use your apps.

I was surprised to see a post from Trivago - I might have to book my next hotel through them.

</comment><comment><USER27> If the compression is really much better than with JPEG, there is probably a point where the connection is so slow, that JPEG-XR would be faster than JPEG, because the file-size would out-weight the decoding performance.

</comment><comment><USER28> This is a little bit wrong in that the other CPU-bound tasks during loading are generally in the main thread, whereas image decoding is usually on a different thread (so, depending on how many images you have, and how many CPUs, it may not increase overall page load time in reality).

Now, overall time to decode still matters. If it's a gigantic hero image above the fold, then it could take a noticeable amount of time to decode, but it's probably still a win.

With the only good JPEG 2000 encoder, its quality at small file sizes is easily better than webp, but in practice it seems virtually nobody uses JPEG 2000, because the open source encoders are not really that great (probably because the patents are a guessing game, although technically the known holders are willing to grant royalty-free licenses).

</comment><comment><USER29> At this point in time I suspect we do need to use anything not invented by Google, if only to slow down arrival of googlenet.

</comment><comment><USER30> Regardless of the findings, I think it's funny that they think they were able to "statistically verify" a -1% impact on conversions...

That's a hilarious thought to me. How can a 1% change in a sales funnel be considered statistically relevant?

</comment><comment><USER31> The author says that JPEG-XRs are decoded on "the software-side on the CPU" a couple of times that it seems like it might be a term of art but it sounds very awkward to me.

Is the TLDR actually "jpeg xr implementation is poorly optimized and negatively affects browser performance which decreased conversions."</comment>


Ask HN: iPhone apps: same user id across apps? - <AUTHOR>

I am not an app developer and tried to find an answer for this online without much success.
When storing information about users, would two autonomous apps identify the user by the same id in each respective database?
I install app_1 in my iPhone, how is my user id created in app_1's user db?
Then I install app_2, would the iPhone use the same user id?
Or does the user id completely depends on how it was programmed on each app?
What if those apps use social networks to allow me to sign-in? and I use facebook in both? <comment><USER2> Apple provides an ID that can be used to track users so you can tell the same user is still using your service.

This quickly got abused for tracking so a few years ago Apple changed the way it worked. Now the ID is specific to that install of the application. Different apps get different IDs and if you delete an app and redownload it that app will see a different ID.

I don't know if games can find out your GameCenter ID.

If you sign into Facebook, you've now told that app who you are. They could compare notes with other apps to see if the same Facebook IDs have been used and tie you together that way.

<comment><AUTHOR> Thanks! This helps a lot. So, based on the second part of your answer, comparing facebook ids, and other social networks, emails, etc. is what brings Oracle and other Data-as-a-service providers to talk about identification. Sort of trying to connect the dots across platforms, joe.black in fb, jblack in instagram, @jb1974 in twitter, etc.</comment></comment>


Tamgu (탐구) a FIL Programming Language - <AUTHOR> TAMGU (탐구) is a programming language that combines functional, imperative and logical paradigms into a single formalism. The language has also been specially designed to simplify automatic annotation and data augmentation for Data Programming.
Pre-comiled versions for Windows, Mac OS and Linux are available at: https://github.com/naver/tamgu/releases. <comment><USER2> If it meets the guidelines, this might make a good Show HN.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)</comment>


C as a Portable Intermediate Language - <AUTHOR> http://cybertiggyr.com/ick/

<comment><USER2> He might have considered citing any of the myriad academic projects that did exactly this (the brain-dead obvious cite would be Cfront and ObjC, but there are better examples in the literature).

One of the reasons this approach isn't more popular is that it limits what you can do at runtime in ways a virtual machine doesn't; C (and the basic blocks emitted by a C compiler) are very hard† to work with, whereas VM instructions almost by definition aren't.

† _But not impossible; see, for instance, Dynamo/RIO_.

<comment><USER3> To be fair, Objective-C doesn't use C as an intermediate language, it actually is a strict superset of C, and only the objc calls end up being "translated" to C for the runtime to execute.

<comment><USER4> Nowadays, and the last 2.5 decades, yes, but originally it used to use C as an intermediate language.

</comment></comment></comment><comment><USER5> Pretty light on information.

For a better, somewhat Scheme-specific intro, check out <http://matt.might.net/articles/compiling-scheme-to-c/> .

For an ML-flavored one, "No Assembly Required: Compiling Standard ML to C" by Tarditi et al ([http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.43.7...](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.43.7510)), and its bibliography.

</comment><comment><USER6> _That's it. There's no magic. You compile L to native code via C._

This is how Squeak is portable across so many different platforms. (Altered to suit a dynamic language.) The VM is implemented in a statically-compiled Smalltalk subset which is emitted as C. I understand that Rubinius uses this trick as well.

</comment><comment><USER7> GHC (Haskell) compiles to a Haskell program modelling C--, a subset of C. From there, it is rendered as actual C and compiled using GCC.

I believe there's more recently an option involving LLVM, though I'm not sure if this replaces the C-- step or is an alternative to GCC.

<comment><USER8> Man, it is really hard to google for '-f-via-c.'

I believe that it has been deprecated now that the LLVM backend is more mature. I'm not 100% sure, though.

EDIT: found it! [http://www.haskell.org/ghc/docs/7.0.1/html/users_guide/relea...](http://www.haskell.org/ghc/docs/7.0.1/html/users_guide/release-7-0-1.html) (in the 1.5.6 section)

> The registerised via-C backend, and the -fvia-C flag, have been deprecated. > The poor floating-point performance in the x86 native code generator has now > been fixed, so we don't believe there is still any reason to use the via-C > backend.

<comment><USER9> -fvia-c was not the main mode of operation even in 6.x, it had an NCG long before the LLVM backend.

</comment></comment></comment><comment><USER10> This article seems out of date given the existence of llvm.

<comment><USER11> Out of date not just as a suggestion, but in the very specific sense that it was exactly what people did before LLVM and co.

<comment><USER12> Article is from 2002, updated 2006.

Doesn't HIPHOP use this approach though?

<comment><USER13> A similar one, it compiles down to C++.</comment></comment></comment></comment>


Raspberry Pi automatically tweets at Comcast when ISP speeds drop - <AUTHOR> https://www.reddit.com/r/technology/comments/43fi39/i_set_up_my_raspberry_pi_to_automatically_tweet/ <comment><USER2> Great idea! I can think of a crowd sourced site that publishes these hourly stats from all ISPs and at different locations.

</comment><comment><USER3> Very Clever.</comment>


Nine-year-old bug in the Go standard library enables DoS - <AUTHOR> https://github.com/ethereum/public-attacknets/issues/12 <comment><USER2> The issue:

"Certain invalid inputs to ReadUvarint or ReadVarint could cause those functions to read an unlimited number of bytes from the ByteReader argument before returning an error. This could lead to processing more input than expected when the caller is reading directly from a network and depends on ReadUvarint and ReadVarint only consuming a small, bounded number of bytes, even from invalid inputs.

"With the update, ReadUvarint and ReadVarint now always return after consuming a bounded number of bytes (specifically, MaxVarintLen64, which is 10). The result being returned has not changed; the functions merely detect and return some errors without reading as much input."

What's a varint? This: [https://developers.google.com/protocol- buffers/docs/encoding](https://developers.google.com/protocol- buffers/docs/encoding)

You have to use varints to be affected, but of course you could be using them in a library without knowing it.

<comment><USER3> So is it Golang's varint or protobuf/grpc varint that had the bug? It's not clear to me.

Edit: [https://go-review.googlesource.com/c/go/+/247120/](https://go- review.googlesource.com/c/go/+/247120/)

<comment><USER4> Proto has a different structure with a fully unrolled loop. It could never read more than 10 bytes. encoding/binary just looped as long as the more bit was set, which is a hazard of more-bit encodings (which suck; unary-length- prefixed codes like UTF-8 are strictly better in every way).

<comment><USER3> Woops. Sounds like an big mistake. Fuzzing or even protest would have caught that.

<comment><USER5> I think it's unlikely this would have been caught by fuzzing. I think you'd have to write assertions that it only read as many bytes as necessary, which is basically saying just don't write the bug in the first place.

<comment><USER3> Not sure but couldn't a fuzzer have created an input that would have timed out the fuzzer? I guess it also depends on the timeout you set when running it.

<comment><USER6> You would also have needed to have a slow unboundedly large data stream on the other end. I fuzzed a bunch of varint-using code last month and never found this, because I was always working with in-memory buffers. The loop is simple enough that anything reasonably sized (e.g. under a GB) would probably not hit any reasonable timeout.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> Misleading title. Reads like the bug was discovered 9 years ago but was not fixed until recently. However, the bug seems to have been fixed pretty fast. Thanks to the Go team!

<comment><USER8> Another way to read it is that 9 years worth of software could be susceptible. It's patched now but how long will it take that all executables have been updated? How long for all projects to update their dependencies to one with a fix? Generally software deployment also varies wildly in terms of quality and speed.

<comment><USER9> And, since people tend to link Go statically, it's not just like `apt-get upgrade libgo`, one has to do a full recompile. (Which, to be fair, Go makes a lot less painful than many languages.)

</comment></comment></comment><comment><USER10> 9 years may seem like a long time, but its also just a DoS vuln. Which is a real security issue in many contexts, but its not exactly a high severity one

<comment><USER11> It wasn't a 9 year old reported bug that took 9 years to fix. It was a 9 year old undetected bug that took... I'm not sure, it looks like a 1 week from discovery by the Ethereum team to a fixed released by the Go team.

<comment><USER12> Interesting that you and another commenter interpreted "9 year old bug" to mean "bug discovered 9 years ago". I'd have thought it was unambiguous that the bug was introduced 9 years ago, but apparently it isn't!

From the article, "this attack was due to a 9 year old bug in the Go standard library. During the "post-mortem" @protolambda, @prestonvanloon, @raulk and I uncovered this bug," so at least they are quite clear that it is a recently discovered bug.

<comment><USER13> _> I'd have thought it was unambiguous that the bug was introduced 9 years ago, but apparently it isn't!_

It is unambiguous. Neither interpretation says that the bug is newer than nine years vintage.

 _> Interesting that you and another commenter interpreted "9 year old bug" to mean "bug discovered 9 years ago"._

In an unusual moment of reduced cynicism, I too interpreted it as that the bug was newly (or at least recently) discovered whcih does appear to be the case this time.

It could easily have been a bug discovered that long ago that was for some reason marked WONTFIX (the danger not appreciated, that bit of code was at that time due to be deprecated RealSoonNow, ...) or CANTREPRO.

 _> From the article_

Yep. The fault lies with whoever named this submission. It isn't as if they've just used the title of the page linked to, as that differs completely, so a thought process happened and a wording decision was made.

<comment><USER12> > as that differs completely

There must still be confusion, as when I read the title it means exactly what is said in the GitHub issue.

</comment></comment></comment></comment></comment><comment><USER14> Reading an oversized int from the network just keeps reading bytes unbounded eh? Do you have to provide an insanely long int or could it be combined with a slow loris?

<comment><USER15> It's basically slowloris-- send 0xFF every 20 seconds to keep the varint reader blocking waiting for the end of the number.

</comment></comment><comment><USER16> Aiee, code is hard. Great work to all involved. Glad the good guys found this one first.

</comment><comment><USER17> Quite an interesting read!</comment>


Ask HN: Which job criteria is important you when searching a new job? - <AUTHOR> When you search a new job, apart from the regular job description (position, technology, Joel test) which criteria are important for you?
What makes a good job for you?
I feel that a lot of companies still underestimate softer criteria like "remote possible", "deep work possible" or "development plan from day". <comment><USER2> \- How long after joining does it take to find out who your line manager is?

\- How long does it take to stick a debugger into a particular place in the codebase and run a single automated test to trigger it?

\- How long does it take to get a diagram of the schema (or implicit schema) of the relationships among data entities in their main applications?

\- How long does it take to find out who the business stakeholders are for my team or projects?

\- How long does it take to find out what the goals are for the engineering and product team are overall?

\- Can the product manager explain the relationship between technical debt and engineering velocity?

——————

Is it weird that I find myself utterly distressed that these can be so uncertain?

<comment><USER3> It's certainly depressing.

Are you having a hard time finding this?

<comment><USER2> I once joined a company where I didn't know where my line manager was.

My current company uses MongoDB and I _really_ wish there was a schema I could read.

</comment></comment></comment><comment><USER4> I want to speak to my possible peers. I join my current company, a huge fintech company everyone has heard of, hoping to find amazing people here. The team I originally joined is full of junior developers who have learnt horrible coding habits from idiot architects and seniors. I changed my team within 6 months because I couldn't fight the system anymore.

I am not joining a new place unless I can discuss work with my possible peers. They need to be deserving of me just as I need to be deserving of them.

</comment><comment><USER5> Culture. I'm no longer interested working for companies where technology is treated as a disrespected cost center.

There's ways to identify such companies. One of the more obvious ones tends to be having a strict dress code (i.e. must wear biz casual).

</comment><comment><USER6> I won't work for a company that doesn't have revenue or that has revenue that doesn't come from charging money for a product or service.</comment>


Why It's a Good Thing People Are Shallow (and 13 Ways to Exploit It) - <AUTHOR> http://blog.kissmetrics.com/shallow-people/ <comment><USER2> I disagree with a lot of the points made in the article. For example, the "raise your rates", and "put yourself out of reach" suggestions would be suicide for anyone targeting the mobile application space. There, the majority of users are only willing to pay a nominal cost to get their app. Raise your price above a dollar or two, and you're sunk.</comment>


Defining the Undefinedness of C (2015) [pdf] - <AUTHOR> http://fsl.cs.illinois.edu/FSL/papers/2015/hathhorn-ellison-rosu-2015-pldi/hathhorn-ellison-rosu-2015-pldi-public.pdf <comment><USER2> Here's something I'd love to know about undefined behavior in C: is this something specific to C, or is it something that _any_ similar language would have to contend with?

It seems like problems crop up when you combine a fairly low-level language with an emphasis on performance, a specification that explicitly calls out implementation-defined and undefined semantics, and very highly optimizing compilers.

None of the "safer C" languages seems to have gained much traction. Is that because the problem really is intractable, or just because maintaining a decent level of compatibility with C is too hard? The major selling point of a safer C is being able to reuse lots of existing C code, after all.

Note that other languages probably have the same pitfalls, just latent for now -- their specs are silent about undefined operations, or they don't have specs at all, and their compilers don't optimize as aggressively as GCC or Clang.

<comment><USER3> I think it was an unintended consequence of the wording chosen by the ANSI C89 committee. The reality of pre-ANSI C was that if you wrote '+', the expectation was that you'd get the target machine's 'add' instruction, no more and no less. It might overflow, it might not; it might crash or hang your machine on overflow or trap values — but that is all your problem, not the language's or compiler's. I worked on a commercial compiler at the time, and at the time, people took 'undefined behaviour' as acknowledgement that sometimes the effect of generated code could be surprising to people unfamiliar with the particular system, not as encouragement to screw the programmer.

<comment><USER4> That's a good explanation for invalid memory accesses and divisions by zero but I'm not aware of many architectures where addition overflow traps (it _can_ trap on MIPS but there's an other instruction that simply wraps around). That being said I don't have an encyclopedic knowledge of instruction sets.

I could be wrong though, after all compilers back then were a lot less clever than now so maybe they couldn't really make use of this information. I'd be curious to get a first hand testimony as to why certain UB exist in the first place.

<comment><USER5> This is not first hand, but the commonly given reason for undefined signed integer overflow is that it allows for mathematically correct reasoning: x < x + 1 is true for all x. Or rather, the compiler may assume that it is true and optimize accordingly, since the programmer is trusted to avoid the undefined overflowing case.

<comment><USER2> "Mathematically correct" doesn't seem like the right way to think about it, since the whole problem is that it isn't correct!

<comment><USER5> Yes, but it wouldn't be correct the other way round either. Wraparound on signed overflow is a well-defined operation on CPUs, but it is nonsensical. In almost no computation will it give a result that is meaningful in the context of what the program is trying to do. If you try to compute the average of two positive numbers as (a + b) / 2 but get a negative result, it would be of no use if this were "well-defined".

So from the point of view of a C compiler that exploits undefined signed overflow, if your code could overflow, it was _probably_ already buggy, and _you_ should have fixed it.

(One problem with this line of reasoning is that it kind of also applies to unsigned numbers, but there C goes the other way.)

</comment></comment></comment></comment></comment></comment><comment><USER6> That team's awesome. One of few groups in formal methods using rewriting logic (Maude) instead of things like Coq or Isabelle/HOL. They seem to move faster on semantics as a result. They also build their own modified logic called matching logic on top that they claim is better than separation logic.

[http://www.kframework.org/index.php/Main_Page](http://www.kframework.org/index.php/Main_Page)

More interesting, their use of these tools allowed them to make their C semantics executable in a style similar to GCC:

[https://github.com/kframework/c-semantics](https://github.com/kframework/c-semantics)

I want this group to do one for Rust and SPARK as a reference spec for certifying compilers for those. Also useful for the diverse compilation concept to counter Karger's compiler/compiler subversion. On a side note, they also have a company that they use to fund and apply their work:

[https://runtimeverification.com/](https://runtimeverification.com/)

<comment><USER5> > using rewriting logic (Maude) instead of things like Coq or Isabelle/HOL. > They seem to move faster on semantics as a result.

This statement is close to nonsensical. What are you comparing it to? Where is a comparable executable semantics project in Coq or Isabelle?

This is great work, and they are definitely using the right tool for the job. But the field of "formal methods" is enormous, and you seem to be saying that apples are better than oranges.

<comment><USER7> Here is a Coq formalization of C11: [http://robbertkrebbers.nl/research/ch2o/](http://robbertkrebbers.nl/research/ch2o/) It only covers a large fragment of C11, but they formalized the operational, axiomatic, and executable semantics and proved that these correspond to each other.

<comment><USER5> Yes, and there are other formalizations as well. But none of them (as far as I am aware) formalize the _undefined_ parts in the way the featured article does. I think comparing projects with different goals and then saying "the differences must be due to the tools used" isn't solid reasoning.

</comment></comment></comment></comment><comment><USER8> My good god. Do you actually believe that this scenario reflects a weakness in C or a C compiler?

<comment><USER9> Yes?

In the grandparent's solution, I should be able to e.g. collection.map, the type system and/or hints from myself should inform the compiler whether or not there are side-effects, if runtime bounds checks are needed, if this is something that can and should be turned into SIMD or concurrent threads. OR I should be able to write some mildly-portable assembly-type algo using intrinsic functions that represent precisely what I want the machine to do, and expect that is precisely what will be done (and deal with porting this to whatever platforms I support). In neither case is there a concept of undefined behavior.

Modern C/C++ is at a level of abstraction where I'm writing explicit instructions for a hypothetical machine, the compiler will consider explicit details in my implementation as implicit suggestions in order to map what I've asked it to do to what the hardware is capable of, and it's up to me to have to be aware of the consequences.

It's not the worst problem of course, but it's not a non-problem.

<comment><USER8> Your approach is emblematic of programming as a culture of production versus a culture of understanding. I don't agree with any approach which characterizes a language as good or bad based on it's ability to read minds and make decisions based on the unknown.

When you mandate safe behavior based on any expert rules-based or heuristic system you have to leave the unsafe option open which is _just_ as bad.</comment></comment></comment>


As Uber Grew Hastily, Diversity Took a Backseat - <AUTHOR> https://www.bloomberg.com/news/articles/2017-03-24/as-uber-grew-hastily-diversity-took-a-backseat <comment><AUTHOR> I believe the new information in this article is that Uber didn't allow its HR department access to their diversity data that they collected.

</comment><comment><USER2> Note: article has new infornation/anecdotes compared to previous articles about this topic.</comment>


What I Learned Juggling Five Projects at Once - <AUTHOR> https://medium.com/@jessehorne/what-i-learned-juggling-five-projects-at-once-94fef6bf017d <comment><AUTHOR> I've took this article out of the MPP so that it's free to read. Enjoy!</comment>


Julian Assange rape charges dropped - <AUTHOR> http://www.sfgate.com/cgi-bin/blogs/abraham/detail?entry_id=70624

<comment><USER2> The article says the women were afraid of his power. That's a laugh. His power as Wikileaks founder is the ability to publish leaked documents! I'm pretty sure individuals are safe from him.

</comment><comment><USER3> This is an even worse PR disaster, unless they have an ace up their sleeve. Suddenly, the CIA (doesn't matter if it even was them in the first place most people would just assume it), the scary CIA, of conspiracy theorists dreams whimpers and drops charges in a smear campaign?

This is why I doubt it if it's over yet. No one trying to run a smear can be _this_ stupid. It simply doesn't make sense.

</comment><comment><USER4> Its really interesting to see this play out. One of the big questions is this a ploy to discredit WikiLeaks? If it isn't then why are these people coming up with these allegations? Is it because he is a celebrity?

</comment><comment><USER5> Looks like somebody did get raped and it wasn't those women.</comment>


Show HN: Caption Generator, A 7 day project - <AUTHOR> http://captiongenerator.com/

<comment><USER2> Functionality seems good on the key component, but when I went to submit my fifteen minutes worth of work, I didn't supply a title and it kicked me back and CLEARED all my captions. You just lost me.

The captioning mechanics are good (awesome even), work on the form entry and general UI.

<comment><AUTHOR> Wow, thanks for drawing that to my attention. I was calling "formatTime" on the text parameter on accident, so when it was trying to run the function on just text, it was returning NaN.

I've fixed the problem now, very sorry you lost your work :(

<comment><USER2> I'm sorry for complaining. Bad form. Nice work on it. I like it.

</comment></comment></comment><comment><USER3> Awesome promo video :D</comment>


The patent to invalidate software patents - <AUTHOR> http://www.techdirt.com/articles/20110822/13094215621/what-idiot-wrote-patent-that-might-invalidate-software-patents-oh-wait-that-was-me.shtml

<comment><USER2> I don't know if this has been mentioned previously, but I've not seen it in any of the previous debate about software patents.

Quoting from the article:

 _Patents are meant to foster innovation by protecting the inventor and then upon expiry providing a library of information for others to build on. To this end, going forward, if a patent doesn't actually tell you enough information to understand and build the invention, it shouldn't be valid._

Given that a software patent is...well, software, why not require that _actual working code_ be submitted as part of the patent application/grant? The code is held in escrow/confidence by the patent office for the term of the patent, and is released to the public domain when it expires.

There's probably some issue there with copyright of the code that might complicate the public domain part, but at the very least you could clean-room reimplement it. An alternative might be that you are required to renounce/void copyright on the code in order to obtain a patent - the short term monopoly of the basic idea comes at the cost of no long/indefinite-term copyright on the specific implementation.

That opens up all sorts of new possibilities, like requiring/permitting the patent claims to be machine-readable, in the sense of a conformance test- suite. That both ensures that the provided implementation is a true instance of the patented invention, and gives the holder/patent office/court an easy way to check for violation.

<comment><USER3> "Actual, working code" might be prohibitively context-dependent. Working code to detect credit-card fraud would be pretty dependent on your financial infrastructure. In many cases though, that would not be as big a problem, and in others it might be satisfactorily overcome. I think this is a good idea.

<comment><USER2> Yeah, it did occur to me after commenting that it might be difficult to extract the specific patentable invention chunk from the overall application.

I can't think of a single approach that would work everywhere, but logically isolating it as a library or service might work, similar to the way in which you can implement an IPC/RPC shim for interfacing to GPL'd code from your proprietary app.

Inventions that don't fit that model (Operating Systems, Embedded Firmware, Massive Scale, etc) complicate the matter, but I imagine a set of disclosure options could be devised, and the inventor/patent office would select the most appropriate.

</comment></comment></comment><comment><USER4> > That said, it's the world we live in. So, like every other Silicon Valley > entrepreneur, I file patents.

This is patents-as-positional-good, a zero sum game that consumes resources and produces effectively no net value.

<comment><USER5> Worse, its almost a pure text-book case of the tragedy of the commons.

<comment><USER6> Excuse me? Could you please explain, as I fail to see how a tragedy of the commons arises.

(Of course patents are problematic, but the reasons that I know of are not related to the tragedy of the commons in a way that I'm aware of.)

<comment><USER5> The best thing for everyone at this point is for individuals (and especially for small start-ups) not to file for patents at all.

This is not the best thing to do for the individual, especially if he wants to be acquired. The best thing to do in that case is to file as many broad and unspecific patents as possible to boost your perceived value. Good for you if you get acquired (or go bust and fire-sale everything), bad for everyone else who now has to live with even more bogus patents that are likely on their way to troll-land.

Its a case of "see, this is why we can't have nice things". It is conceivable for the patent system to work if there weren't so many bad actors maximizing their own gain at the multiplied expense of everyone else. Even those with good intentions feel that they have to do this a little just to get by.

 _"That said, it's the world we live in. So, like every other Silicon Valley entrepreneur, I file patents."_ Everyone feels that they have to graze their sheep to survive, but its making the common grounds into desert.

<comment><USER6> Oh, so it's an externality, not necessarily a tragedy of the commons (which is a very specific type of externality).

</comment></comment></comment></comment></comment><comment><USER7> The points he made about how to improve the patent system are really good ones. I have always though a short lifespan on patents would be better for inovation.

I think patents are good for certain things but over all it slows inovation. I think we could be another 25 years farther down the road with technology if there where not so many bad patents out there.

<comment><USER8> The problem with merely shortening the lifespan of patents is hinted at by the logic in the argument for doing it. When we grant a monopoly to a party in return for a patent, what we are in essence saying is _"If you hadn't filed here then we would not know of this invention until later."_

This argument made sense when the system was first conceived because the norm for that time was to keep inventions as family secrets and innovators themselves were rare. But these days the norm is for some openness and there are so many innovators that there can be very little margin between first to invent and second to invent. Software patents in particular are problematic because it is far more open by its very nature -- the existence of a solution highlights the search space. The problem with merely shortening patents in this light is that it just adjusts the cost/benefit calculation on the basis of this being a sudden jump when this development is the result of a trend that has no signs of stopping.

Fortunately, the revision seems straightforward: revamp patents to make the calculation explicit. By the time software patents are completely useless, they will have a 0 year life (and be dead). Meanwhile if other fields end up as amendable to optimization as software is, we can gracefully scale down the patents there without having the same conversation all over again.

</comment></comment><comment><USER9> Before anybody gets too excited, know that this case doesn't actually open the door to invalidating software patents in general. It clarifies things a little bit is all.</comment>


Never-Slow Mode a.k.a. Slightly-Fast Mode - <AUTHOR> https://github.com/slightlyoff/never_slow_mode <comment><USER2> This seems a bit like a complication that browsers shouldn't be handling? Like really, do we need this feature baked into the web forever, just to make it easier for websites to audit their performance?

A cynical part of me notes that adding more complexity to browsers just makes Chrome monoculture easier: I have no doubt that Google can win any war of "throw developers at it". But it feels like they're building a complexity debt nightmare that we'll all suffer from.

Edit: I guess the other and more direct angle for why Google would push this is that they can give "fast" sites a ranking boost in search, i.e. dictate what web pages do via soft power to make their job easier.

<comment><USER3> I think the real motivation is to allow Google to use guaranteed never- slowness as an unspoofable ranking signal, essentially creating an alternative to AMP that doesn't require special tags or libraries.

<comment><USER2> Yea, that's kind of what I meant to say, although I wasn't thinking about AMP. Can Google really not use the vast amounts of data it ingests to deduce which sites have bad user experiences and are slow, though? Shouldn't that be their core competency?

<comment><USER3> Yes, Google's crawler already calculates page speed yet they've observed that few popular sites are fast. Another aspect of AMP and NSM that I didn't mention is that they are binary; this prevents sites from adding just a little more bloat and then just a a little more after that until they're quite slow.

</comment></comment></comment></comment><comment><USER4> This does not remotely belong in a HTTP header. DevTools, CI is more like it.

In any case, it is impossible to prove that your JS code will not somehow exceed these limits on a rarely exercised execution path. So I recommend turning this on in production as soon as the halting problem is solved.

<comment><USER5> Isn't it also impossible to prove that your code won't violate CSP on a rarely exercised execution org?

<comment><USER6> CSP had the report-uri that helps mitigate....and it's pretty clear how something violates CSP.

<comment><USER5> I am pretty sure I saw the same reporting behavior with this idea.

</comment></comment></comment></comment><comment><USER7> What's the likelihood that a mode like this will lead to sites starting to change their design to encourage more scrolling/tapping/clicking, simply so they get a larger resource budget? "Oh we could just show this information, but if we put it behind a 'tap here to see more' link we'll be able to load 10 more images!"

Also, if scrolling increases the budget, doesn't that penalize sites for being loaded in larger browser windows as the user has to scroll less?

</comment><comment><USER8> Why have a standardized definition of what a "fast" website is, when Google can just impose and publish its own standards for Search, or even implement its own nontransparent standards in the back end if its worried about metrics becoming targets? I don't mean that as a sarcastic question, I don't see any problem at all with a private approach.

This isn't really what people like me meant when we said we wanted Google to use performance metrics in their ranking. We wanted Google to come up with its _own_ internal metrics, not standardize them across the entire web. Because performance isn't standard, it's extremely context-specific, and it's important that sites like search engines and browsers be able to iterate quickly on their metrics and to change them depending on what the ecosystem currently looks like.

I genuinely just don't understand what value standardization has here other than to make the system easier to game and less flexible in the future.

What positive, consumer-friendly, non-scammy things are possible for us to do with a universally standardized set of performance criteria that we couldn't do with privately maintained, non-standardized sets of criteria?

</comment><comment><USER9> The austerity approach to page performance works for Google because they control their whole stack, they have world class devs to can devote time to fine tuning products, _and_ their bread and butter is in small, focused tools (search is an input and a list).

Other big tech companies like Facebook or Netflix don't follow Google's approach, because most products are sprawling, and built by many teams and many engineers. Keeping a small, tight, codebase under those conditions doesn't work.

Contrast Never Slow mode with what the React folk are doing with React Fiber. They have rewritten their engine to allow JS execution that doesn't block the main thread. This is much more pragmatic than setting budgets and simply telling devs to try harder.

</comment><comment><USER10> I think this proposal looks good.

Too many people shovel on slow JavaScript libraries and make the user experience worse. Speed is a tradregy of the commons issue - it's time serious action was taken to resolve it.

</comment><comment><USER11> It's unfortunate that, under this proposal, one may end up splitting an application into multiple iframes to get "enough" budget (or, somewhat more nasty, an embedded iframe could decide to add a second internal iframe).

I could imagine an <iframe allow-slow-performance-shares=33%>, and leaving the iframe to figure out how to subdivide this among itself and its children. (And something similar for the main document.)

Deploying this will be hard, since the slowest devices also tend to run outdated browsers (and thus won't enforce limits anytime soon.)

</comment><comment><USER12> This proposal looks pretty great; it's what AMP should have been!

</comment><comment><USER13> I think such featurse should be instead configurable only by the user, per URL, with customizable thresholds, and that no reporting should be implemented.

</comment><comment><USER10> What about a revised proposal:

No line of JavaScript may execute more than 200 milliseconds after the last user touch/tap.

That would still let video sites play videos, but everything else would have to be 'ready to view' within 200ms. Let the developer decide if they want to achieve that by making resources smaller, having fewer resources, getting rid of js frameworks, etc.

Sites which _need_ more time could demand the user hold a finger on the screen while they load up their heavy frameworks, but that would be a clear "we are slow" indicator the site would want to avoid.

<comment><USER14> > No line of JavaScript may execute more than 200 milliseconds after the last > user touch/tap.

This is how Opera Mini "implements" JavaScript. The Opera Mini server will run whatever JavaScript code there is in the page for ~2 seconds. Whatever can be seen on the screen at that point is sent to the Opera Mini client.

It works good enough for things like SPAs and "click here to show more".</comment></comment>


Ask News.YC: How much time and effort do you spend on SEO techniques to gain traffic? - <AUTHOR>

...and if you've had successes in the past what did you do? Did you hire an SEO firm or do it yourselves?<p>I've got a startup that's facing a fair amount of keyword competition. Ranking in search engines is going to take some time. Purchasing advertisements is possible, but our budget is limited. Any advice? <comment><USER2> Since SEO and organic search favors older links, I started the blog for my startup before I even started coding! I'm still very early on so I can't report any results, but if you write blog entries that are more useful than anything else, people will find them. But it takes time and LOTS of writing (see Paul Graham, Jeff Atwood, Joel Spolsky, Reg Braithwaite, etc). Little mediocre posts get quickly lost and forgotten because they're not worth pointing out (see Steve Yegge's latest post).

SEO prominence comes not from what words you put in your posts but from content good enough that other people link to you. Become prominent! (there, that's enough armchair quarterbacking)

<comment><USER3> Yes, most of true SEO (read: not blackhat tricks that will get you burned in a few months) are just standard practice.

Like starting a blog thats heavily linked to about your topic of expertise.

</comment></comment><comment><USER4> Make something so good that it spreads by word of mouth.

<comment><USER5> Yeah, I don't think Facebook depends on SEO.

I guess it depends on the context of the site though. Good SEO generally always important (can be a kickstarter). Wordpress blogs are great 'out of the box' for SEO - maybe a company blog with Wordpress could be a good start.

<comment><AUTHOR> We do have a blog and are posting nearly daily. I see our blog as being the absolute minimum. It's something we have to do, but it's not going to be enough. Good suggestion, though. It's a good start.

<comment><USER5> I hear that blogs on wordpress.com have great SEO from the get-go and can be custom domained too.

</comment></comment></comment></comment><comment><USER6> What does your startup do?

<comment><AUTHOR> Our startup is in the merchant account services industry.</comment></comment>


The True Identity of Andy's Mom Makes 'Toy Story' Even More Epic - <AUTHOR> http://moviepilot.com/posts/2014/02/24/the-true-identity-of-andy-s-mom-makes-toy-story-even-more-epic-1247874

<comment><USER2> [https://news.ycombinator.com/item?id=7292387](https://news.ycombinator.com/item?id=7292387)

<comment><AUTHOR> Sorry - I wonder why the anti-duplicator code didn't catch it?

<comment><USER2> I used the bookmarklet to submit it and forgot to remove all the tracking crap on the end of the link. Largely my fault.

</comment></comment></comment><comment><USER3> I thought this was immediately obvious on first watching, and am surprised to find it's considered an Easter Egg.</comment>


Bitcoin's Enormous Energy Costs May Prove to Be Its Biggest Risk - <AUTHOR> https://www.bloomberg.com/news/articles/2017-11-09/bitcoin-s-exorbitant-energy-costs-may-prove-to-be-biggest-risk <comment><USER2> "So, this is the next attack vector? Bitcoin is not green? The energy put into Proof-of-work is what makes the system immutable. People who complain about bitcoin's energy use, don't seem to grasp what makes the bitcoin Blockchain immutable. In order to change the Database, you have to go back > and do the "work" which becomes practically impossible after approx. 10 blocks. You take that away and you'll make the system Tamper evident, and not Tamper Proof as it is right now. It's the only secure data base ever invented, and thats because of the energy used to secure it.

I don't see these people complaining how much energy Las Vegas wastes. Or Hollywood. What about professional sports? How much energy does going to war cost? All of these are less important than the only sovereign moneys system in the history of man-kind. So, please, stop complaining about the energy use, i left my violin at home."

\- daggerhashimoto

In addition, if Bitcoin disrupts the banking industry and a portion of wall street, we'd have to subtract their energy consumption.

It's just logically lazy to not consider if the benefits outweigh the costs, and sound immutable finite hard money out of reach of central authorities brings with it enormous benefits.

<comment><USER3> Yes. The people who hate bitcoin -- despite all it does for them-- generally are hating it for political reasons.

They may hate sound money, they may worship at the alter of neo-keynsianism and think inflation is necessary (how else will you pay for bombing brown people?)

But they will use the "its anti-environment" claim because that has gotten traction.

We already have a massive global propaganda campaign spreading leftist views of the environment.

When bitcoin defeats these people, I will be very happy.

<comment><USER4> bitcoin is currently inflationary -- what do you think is paying the miners, but new bitcoins created out of nothing?

<comment><USER5> It's mostly new BTC but there's also ~1-2 BTC per block of transaction fees now too. Some spikes up to around 4.

[https://www.smartbit.com.au/charts/transaction-fees-per- bloc...](https://www.smartbit.com.au/charts/transaction-fees-per-block)

</comment></comment></comment></comment><comment><USER6> "Even as bitcoin flirts with $8,000, the price required for mining to be marginally profitable stands at a jaw-dropping $300,000 to $1.5 million by 2022 based on current growth trends and electricity use, according to Christopher Chapman, an analyst at Citigroup Inc. At that pace, the implied consumption would match that for the whole of Japan over the long haul."

This Citi analyst either doesn't understand Bitcoin or doesn't understand economics: because the protocol has difficulty adjustment and because marginal revenue will always approximate marginal cost, the cost to mine 1 BTC will always approximate its value.

Therefore, projecting the cost of mining is _equivalent_ to predicting the price of Bitcoin. If the cost to mine 1 BTC was $1MM USD (and so $1MM/BTC), that would imply the network value of the Bitcoin ecosystem would be well over the USD M1 supply (tens of trillions, in fact). Bitcoin would have effectively replaced fiat in that world and perhaps all the cost it eliminated would be worth the mining.

In any event, from what he said, I doubt this (equity) analyst understands that he can't project $1MM BTC mining cost without at least $1MM per BTC price.

<comment><USER7> It's probably worth noting that the 2022 calculation is probably taking into account the fact that the amount of BTC gained per block will probably have halved once or twice by then, which doubles the cost of mining each time. And eventually the amount of BTC per block will go to zero, meaning miners make nothing directly off of the block directly.

The network already has a system in place to combat this issue - fees. So it can technically cost $1MM to mine 1 BTC while the price of BTC is only $8,000, as long as the transactions provide enough fees to offset the costs. In practice I think that's pretty unlikely to work out that well, but that is the intended system either way.

Edit: Reading the article again, it's not clear to me they're actually taking halving into account in their calculation, and I'm also not extremely confident they're not confusing bitcoins and blocks. There are no direct quotes detailing that part of the article, and I wouldn't really expect the person writing the article to know the difference.

And it also occurred to me that something else to take into account is that miners themselves are betting on the price of bitcoin going up. So just because it cost them, say, $200,000 to mine $100,000 worth of bitcoin, if the price of bitcoin eventually doubles from the point that they mined them then they would still break even, though not immediately. So it can make some sense to mine bitcoins at a bit of a loss. I don't think it would make sense to mine at anywhere near the losses they're describing though.

<comment><USER8> If you expect Bitcoin to increase in value, wouldn't it be a better idea to just buy bitcoins rather than buy computers and electricity to mine at a (current) loss?

I think the only reason such people would prefer mining is if it's difficult for them (legally, practically, etc.) to directly buy bitcoins in the area they live or at the scale they desire to invest.

<comment><USER7> > If you expect Bitcoin to increase in value, wouldn't it be a better idea to > just buy bitcoins rather than buy computers and electricity to mine at a > (current) loss?

Estimating how much it will cost you to mine X Bitcoin is not an easy thing to do, especially now that each block almost always has at least an extra Bitcoin worth of fees, with the highest having over 4, meaning the amount of Bitcoin you get per block can vary by approximately an extra 1/3 and this extra value is pretty random.

The power to run the hardware is also constant cost, not a one-time cost like buying a bunch of Bitcoin at market price, so while it's easy to say "Just buy when it's at X value and turn off your hardware", the miner has no way of knowing what that value is, and without knowing the future values of Bitcoin they have no idea if waiting and spending all their current savings on Bitcoin at a low price will actually net them more then just waiting it out and continuing to mine the whole time. It's a crapshoot, but the miners presumably know what they're getting into.

My point was more that it is probably acceptable to risk mining at a bit of a loss if you're confident the value is going to go up, you already invested in the hardware anyway, and you think there's a good chance you may make more then if you had just bought some at what you think will be the lowest price. That said, my example is probably more extreme then would be acceptable.

<comment><USER6> It's absolutely easy to do. The fact that it's probabilistic doesn't by default make it hard, and in fact, over long time frames, it's much easier to forecasts (expected) costs and profit for BTC miners than in most businesses (say, e.g., Walmart).

Miners have continuous liabilities in USD and so most only hold Bitcoins as long as as is necessary by the protocol (100 blocks). Further, expected fees are easy to estimate over medium-to-long time frames. I basically disagree with everything you've said here and below.

</comment></comment></comment></comment></comment><comment><USER9> People are always quick to point out that this is just what it takes to build a trustless and decentralized currency, which is probably correct, but it's not clear whether such an extreme level of trustlessness is even necessary (if you are not a crypto-libertarian on principle).

The benchmark for Bitcoin are normal financial transactions as they exist today (and work quite well, most of the time). So the question really is: Are you ready to have humanity spend the additional energy consumption of a country (or many countries if Bitcoin were to scale up) to run a system that offers only very limited throughput (relative to established systems) and hasn't even proven yet that it's even a viable alternative to fiat money from an economic perspective? Not to mention the scaling issues that come with such broadcast protocol.

I don't really see the widespread use, at least as envisioned by Bitcoin proponents. If it's just the decentralized aspect and transparency that you want, you can relax your trust assumptions and implement something like a distributed ledger with known identities which doesn't need proof-of-work to prevent sybil attacks (identities which of course would have to be certified in some way, unfortunate).

Oh well, it's still a very cool technology, but I hope for the planet's sake that cryptocurrencies stay in their niche of anonymous transactions for those who actually need it.

<comment><USER3> You think we should be blocked from donating to wikileaks?

You think venezuelans should be forced to use their local currency?

You think Zimbabwe's people should go thru hyper inflation again? They are about to, and bitcoin is booming there as an alternative.

Sorry, you seem to think you know better than other people what they should value and what they should do.

Bitcoin is completely voluntary, you don't like it, don't participate.

It's also not your energy getting spent.

At the risk of sounding like a libertarian: Other people are not your property.

<comment><USER10> Oh come on. We should be charitable in our discussions. The comment clearly mentions that there are some useful applications, which cover the censorship and collapsing government use-cases. The important thing to recognize is that the large majority of people don't have these use-cases. This means that BTC becomes a niche system rather than supplanting world banking.

It isn't my energy being spent, but it is CO2 that harms our planet. Until we have zero-emissions energy I think it is reasonable to criticize systems as energy-inefficient.

<comment><USER11> Why are we blaming the individuals who use the energy rather than the individuals who make the energy dirty in the first place. Place the blame where autonomy is.

<comment><USER12> There is plenty of autonomy to blame when someone chooses to use an absurdly large amount of energy when a trivial amount would solve the same problem.

</comment></comment></comment></comment></comment><comment><USER13> By reading through the comments. I don't think people realise how much energy it actually uses.

One bitcoin transaction already now uses as much energy as your house in 1 week. [https://motherboard.vice.com/en_us/article/ywbbpm/bitcoin- mi...](https://motherboard.vice.com/en_us/article/ywbbpm/bitcoin-mining- electricity-consumption-ethereum-energy-climate-change)

I'm sorry but I just fail to see how this can work going forward.

<comment><USER14> Why measure it in energy/transaction? The energy is not used for transactions, but for securing the ledger. With second-layer scaling solutions you will have no limit of transactions per block.

<comment><USER15> Why bother with a second layer when new blockchain designs improve upon the designs of Bitcoin? The ledger is "secured" by about 6 mining pools which might even be controlled by the same people.

Second layer goes completely against Satoshi's primary design in the first sentence of the white paper - which is removing the need for third parties. The lightening network design is a siphoning power grab by Blockstream.

After all, if you look at Bitcoin from a technical perspective the services Bitcoin offers are available in Litecoin, Monoero, etc.

<comment><USER16> >Why bother with a second layer when new blockchain designs improve upon the designs of Bitcoin?

We can argue about how to scale layer one, but layer two always provides scalability benefits that layer one can't provide. So choose your favorite layer one scaling scheme and then use layer two on top of that.

>Second layer goes completely against Satoshi's primary design in the first sentence of the white paper - which is removing the need for third parties.

Satoshi in the whitepaper proposed SPV nodes which introduce semi-trusted third parties for the sake of scalability.

Layer two does not introduce trusted third parties. It, under certain conditions, is less censorship resistant that layer one, but that is why you want a scalable layer one as well so that transactions which need censorship resistance can fall back to layer one. There is no reason to bloat layer one with transactions that layer two can handle. Layer two helps layer one scale.

>The lightening network design is a siphoning power grab by Blockstream.

Even if this is true, and I very strongly doubt it, that shouldn't influence your view on the technology. Either it works or it doesn't.

</comment></comment></comment></comment><comment><USER17> This doesn't even make sense.

If it isn't profitable to mine, then people won't mine. Difficulty will fall, which will mean that fewer cycles (and thus, less energy) is needed to find the next block.

The author literally took the "total cost of all energy in the network" and divided it by "total number of transactions". That's not how it works. There is effectively zero incremental cost in terms of energy usage for a transaction.

<comment><USER18> The point is to measure the system's efficiency and energy usage.

Bitcoin's Proof of Work algorithm using SHA-256 and power hungry ASICs with a bottle neck on 4-7 transactions per second (average, as latency is in 10 minute intervals) or even in the case of the controversial Bitcoin-Cash with 56 transactions per second; what Bitcoin as a service does with the type of infrastructure required for the software to run can be analyzed and defined in terms of cost of operations versus alternative services which offer similar or identical functionality.

</comment></comment><comment><USER19> Just so this isn't forgotten, I think we have to ask _in reference to what exactly?_ Paper money and its supporting electronic banking systems cost energy and money too e.g. delivering cash to banks and other institutions, building and powering banks and ATMs, software costs in building and maintaining electronic payment systems, etc. I think it would be instructive to see how the costs compare.

</comment><comment><USER20> These pieces are starting to get pretty funny.

`If it keeps growing, it'll be so expensive, it can't possibly work.` Also assumes bitcoin as we know it today is fully immutable, but that isn't the case, and neither is energy production, or technology used to run the whole thing.

But you know what gets tricky, rewriting history in bitcoin's blockchain. `You'd almost have to spin up an alternate time line, and those alternate time lines are expensive, and use a lot of energy, it cant possibly work.`

The value in the future of bitcoin may be that it has a long and very public chain... and that alone is probably enough to hold reasonable utility in the future. Just like silver/gold and other scares things provided utility for quite a while. Only now of course, if it gets too expensive, another slightly different public fork will persist (or several of them).

<comment><USER10> Isn't the immutability of BTC one of the major things that proponents point to? Wasn't this why there was such pushback against increasing block sizes by just 2x? Why should we assume that a community that is so based in political rather than technical ideals will just come up with some unnamed technical solution in the future to improve efficiency?

</comment></comment><comment><USER21> Another perspective (from 2015): "Nothing is Cheaper than Proof of Work" [1]

[1] [http://www.truthcoin.info/blog/pow- cheapest/](http://www.truthcoin.info/blog/pow-cheapest/)

</comment><comment><USER22> Well, let's clean up the energy grid and stop blaming consumers?

<comment><USER12> At the current estimated energy consumption, if everyone on Earth performed an average of one transaction per month then Bitcoin would use more electricity than is being generated.

<comment><USER23> That's not how Bitcoin works. The amount of energy used to secure the blockchain is not proportional to the number of transactions.

<comment><USER12> It doesn't matter that it's not _supposed_ to be proportional, it's a simple scaling up from current energy usage and current transaction frequency to the world population.

All that tells us is that bitcoin _as it currently exists, and as part of the way it interacts with the real world_ has a flaw — it _doesn't_ tell us if that flaw is an oversight in the design or if it's a gold-rush and/or a price bubble because humans are being irrational.

</comment></comment></comment></comment><comment><USER24> Bram Cohen just announced Chia to address this, using storage and time as the proof-of-work.

[https://techcrunch.com/2017/11/08/chia-network- cryptocurrenc...](https://techcrunch.com/2017/11/08/chia-network- cryptocurrency/)

previous conversation here: [https://news.ycombinator.com/item?id=15655840](https://news.ycombinator.com/item?id=15655840)

</comment><comment><USER25> demand and offer, I wander why the elitist hacker rank "liberal" people don't get it, the difficulty is so high because there is a lot of hash power, the network would be fine with much less, I think around 15% of the current one.

On the logic of banning stuff for power saving, we should ban video games and movies.

<comment><USER3> Most of the people on this site are hard left, and not very much into hacking... there are junior developers and wannabe CEOs with MBAs here. You're not going to find cypherpunks on hacker news, generally.

So, anything that fits the hard left agenda is promoted, without regard to its scientific or technical accuracy, even when it's against the startup ethos that created silicon valley in the first place.

<comment><USER26> Please stop your continuous stream of personal attacks and your rants against the 'left', which besides lowering the general level of discourse is factually incorrect in many ways.

</comment></comment></comment><comment><USER27> The problem isn't the energy cost, but the fact that the energy cost is going towards work that has no utilitarian purpose. Ethereum shows a potential way to fix that, and bitcoin can adopt a similar approach once the core problems of the approach are solved.

</comment><comment><USER28> These arguments also seem to confuse energy consumption with the resulting pollution. I'd give it another 10-20 years at most until the majority of energy is coming from renewable sources, making this a non-issue as far as Bitcoin is concerned

</comment><comment><USER29> This seems like a easy way to explain a Bitcoin ban, should any country wish to do so.

</comment><comment><USER14> How much energy and toxic waste is generated by gold mining?

</comment><comment><USER30> What is the most energy efficient way to get rid of Bitcoins? Does it matter if I sell them all at once or if I spread it out over more transactions?

</comment><comment><USER31> So, this is the next attack vector? Bitcoin is not green?

The energy put into Proof-of-work is what makes the system immutable. People who complain about bitcoin's energy use, don't seem to grasp what makes the bitcoin Blockchain immutable. In order to change the Database, you have to go back and do the "work" which becomes practically impossible after approx. 10 blocks. You take that away and you'll make the system Tamper evident, and not Tamper Proof as it is right now. It's the only secure data base ever invented, and thats because of the energy used to secure it.

I don't see these people complaining how much energy Las Vegas wastes. Or Hollywood. What about professional sports? How much energy does going to war cost? All of these are less important than the only sovereign moneys system in the history of man-kind. So, please, stop complaining about the energy use, i left my violin at home.

<comment><USER22> Once you tie the benefits of bitcoin to something as tangible as the "benefits" of a weekend in Vegas, people will back off the application as the problem.

If grid energy is renewable, bitcoin is not a problem, and actually helps push increasing energy infrastructure. In the ideal world...

</comment></comment><comment><USER32> Proof-of-Work is supposed to cost energy and there is no other way, see: [http://nakamotoinstitute.org/mempool/proof-that-proof-of- wor...](http://nakamotoinstitute.org/mempool/proof-that-proof-of-work-is-the- only-solution-to-the-byzantine-generals-problem/)

<comment><USER33> I'm hesitant to call that a proof, there are various points in the proof that might not be false, but are certainly non-obvious.

Firstly, it seems to be enough for any alternative consensus to be impossible to maintain, rather than impossible to produce.

Secondly, to make it practically (economically) impossible to maintain an alternative consensus should requires Proof-Of-Resources, not Proof-Of-Work.

Thirdly, there's no reason to assume those resources can't be stakes in the consensus itself.

Finally, if the only way to maintain consensus is by building massive expensive specialised buildings, then congratulations you've just rediscovered a bank.</comment></comment>