Apple Open-Sources its Compression Algorithm LZFSE - <AUTHOR> https://www.infoq.com/news/2016/07/apple-lzfse-lossless-opensource <comment><USER2> With energy efficiency as a primary goal I was expecting way more use of explicit SIMD instructions.

The InfoQ post mentions xcodebuild, but there is also a Makefile. I really appreciate the presence of a no-nonsense Makefile. No autoconf, no pkgconfig, just plain and simple make. Also, because nobody mentioned it: yes, it compiles on Linux out of the box.

<comment><USER3> _> I really appreciate the presence of a no-nonsense Makefile._

Indeed, the current version of their Makefile is a great example of how to write a simple yet portable Makefile:

[https://github.com/lzfse/lzfse/blob/33629bc65f4b356072c9a7d5...](https://github.com/lzfse/lzfse/blob/33629bc65f4b356072c9a7d5b6321d3b360ed5ec/Makefile)

 _> No autoconf, no pkgconfig, just plain and simple make._

While I agree with your sentiment, I believe that your statement about pkg- config goes a bit over the top.

Yes, the LZFSE project doesn't use pkg-config, but it also doesn't have any library dependencies. There's not a single "-l" argument in the linker flags.

If it had, I would prefer pkg-config over any other mechanism, as that is right now the best "simple yet portable" method of defining library dependencies.

Pkg-config is especially handy when it comes to cross-compiling, or when you have a special need for a static build instead of shared libraries.

<comment><USER4> > Indeed, the current version of their Makefile is a great example of how to > write a simple yet portable Makefile:

Yet it forgets the MOST important thing: make uninstall.

Nothing worse than software where I have to reverse engineer a makefile in order to uninstall!

<comment><USER3> I don't trust "make uninstall" anyway. Instead, I usually put self-compiled packages into separate directories, such as:

        make install INSTALL_PREFIX=/opt/lzfse

or

        make install INSTALL_PREFIX=$HOME/.../lzfse

Uninstalling is then as simple as:

        rm -r /opt/lzfse

For convenience, I either add _/ opt/lzfse/bin_ to _$PATH_ , or create a symlink from _/ opt/lzfse/bin/lzfse_ to _/ usr/bin/lzfse_.

More generally, I believe that uninstalling, upgrading and related operations are the task of a package manager, not a build script.

<comment><USER4> Good luck with getting pkg-config to recognize your include files automatically, for example. Or the good old "man" utility.

Alternative: maintain a HUUUUUGE list of xPATH env variables, and update them every time you recompile something and change the directory in the process. Oh, and hope that other people's Makefiles are intelligent enough to not mess up shit (e.g. use headers from system, and library .so files from your own compile)...

<comment><USER3> If you are alluding to cross-compiling, rest assured that I wrote my comment being fully aware of the plenty of pitfalls, which is why I started the MXE (mingw-cross-env) project some time ago:

[http://mxe.cc/](http://mxe.cc/)

<comment><USER4> Cross-compiling is yet another pile of dungheap in addition to the dungheap I mentioned.

I usually set up a Debian chroot with qemu and compile "natively" (e.g. for RPi). It's dog slow, yes, but at least it works reliably in contrast to cross compilation.

The only way I ever got CC to work is with the buildroot toolchain, which has the downside that it isn't Debian.

</comment></comment></comment></comment></comment></comment></comment><comment><USER5> I feel like LZFSE is too little, too late. It would be great to have a proper comparison, but Zstd is stable, and offers a superior compression ratio with compression and decompression speeds that seem on par with LZFSE.

And Zstd is not proprietary. (This issue is relevant in this regard: [https://github.com/lzfse/lzfse/issues/21](https://github.com/lzfse/lzfse/issues/21))

[https://github.com/Cyan4973/zstd](https://github.com/Cyan4973/zstd)

Edit: here is a quick comparison I did on Linux with Project Gutemberg's webster ([http://sun.aei.polsl.pl/~sdeor/corpus/webster.bz2](http://sun.aei.polsl.pl/~sdeor/corpus/webster.bz2)).

      $ time ./lzfse-master/build/bin/lzfse -encode -i webster -o webster.lzfse
      real    0m1.885s
      user    0m1.860s
      sys     0m0.024s

      $ time ./zstd-master/programs/zstd webster -8 -f -o webster.zstd
      webster              : 25.98%   (41458703 =>10772836 bytes, webster.zstd)
      real    0m1.700s
      user    0m1.660s
      sys     0m0.036s

      $ ls -l
      -rw-r--r-- 1 tyl tyl 12209496 Jul  7 16:26 webster.lzfse
      -rw-rw-r-- 1 tyl tyl 10772836 Jul  7 16:31 webster.zstd

      $ time ./lzfse-master/build/bin/lzfse -decode -i webster.lzfse -o /dev/null
      real    0m0.127s
      user    0m0.112s
      sys     0m0.012s

      $ time ./zstd-master/programs/zstd -d webster.zstd -o /dev/null
      webster.zstd        : 41458703 bytes
      real    0m0.116s
      user    0m0.112s
      sys     0m0.000s

LZFSE's -h option doesn't show a flag to tweak compression. Zstd's default -1 compression is super-fast, but obviously not optimal. Its -8 is the closest I got to LZFSE's compression speed; its -4 was the closest to LZFSE's compression ratio, with a speed of 0m0.527s real compression, 0m0.101s real decompression.

<comment><USER6> I'm not that comfortable with Xcode, but I just built two versions of this tool, one direct from GitHub, one (called lzfse2) that calls the Compression library on Mac OS X. Typical timings on my system are:

        > time ./lzfse -encode -i webster -o webster.lzfse

        real    0m0.945s
        user    0m0.864s
        sys     0m0.062s

        > time ./lzfse2 -encode -i webster -o webster.lzfse2

        real    0m0.803s
        user    0m0.715s
        sys     0m0.072s

        > time ./lzfse -decode -i webster.lzfse -o /dev/null

        real    0m0.133s
        user    0m0.091s
        sys     0m0.036s

        > time ./lzfse2 -decode -i webster.lzfse2 -o /dev/null

        real    0m0.083s
        user    0m0.053s
        sys     0m0.025s

So, the version that ships on Mac OS X seems to be faster (10% at encoding, 35% at decoding) than what this source and makefile produce. I don't think that has to do with my way of building them, as I used the makefile (which uses -Os) to build the original tool, and any compiler flags will not have much effect on the one using the Mac OS X library.

Worryingly, the two versions also produce different files (12,209,496 bytes for the GitHub code, 12,234,159 bytes for the library on Mac OS X 10.11.5), but they can decompress each others files and produce the original file.

<comment><USER5> > _Worryingly, the two versions also produce different files_

That is actually pretty common. I would guess the version you have in macOS is an older version. They may have tweaked the algorithm to deliver slightly better compression at the expense of speed, which is always the tradeoff in this field.

On the other hand, I would not be surprised if they prepared special tweaks in their internal version to better support arm64. Strategically, Apple seems to believe that people will stick with building for their App Store if they are pushed to write non-cross-platform code.

(Oddly enough, LZFSE/LZVN seems well-suited for file system compression on hard drives, but here again, Yann Collet wins with its LZ4's superior compression speeds, which impact write speeds.)

</comment></comment></comment><comment><USER7> If you want to see some crazy C code, check out this file from the GitHub repo: [https://github.com/lzfse/lzfse/blob/master/src/lzvn_encode_b...](https://github.com/lzfse/lzfse/blob/master/src/lzvn_encode_base.c)

<comment><USER8> Excerpt from the link :

          if (D == D_prev) {
            if (L == 0) {
              *q++ = 0xF0 + (x + 3); // XM!
            } else {
              *q++ = (L << 6) + (x << 3) + 6; //  LLxxx110
            }
            *(uint32_t *)q = literal;
            q += L; // non-aligned access OK
          } else if (D < 2048 - 2 * 256) {
            // Short dist    D>>8 in 0..5
            *q++ = (D >> 8) + (L << 6) + (x << 3); // LLxxxDDD
            *q++ = D & 0xFF;
            *(uint32_t *)q = literal;
            q += L; // non-aligned access OK
          } else if (D >= (1 << 14) || M == 0 || (x + 3) + M > 34) {
            // Long dist
            *q++ = (L << 6) + (x << 3) + 7;
            *(uint16_t *)q = D;
            q += 2; // non-aligned access OK
            *(uint32_t *)q = literal;
            q += L; // non-aligned access OK
          } else {
            // Medium distance
            x += M;
            M = 0;
            *q++ = 0xA0 + (x >> 2) + (L << 3);
            *(uint16_t *)q = D << 2 | (x & 3);
            q += 2; // non-aligned access OK
            *(uint32_t *)q = literal;
            q += L; // non-aligned access OK
          }

<comment><USER9> I've never seen this code base before, but this makes straightforward sense to me as I do keep my hand in with compression software, and I don't anticipate others who works with compression algorithms in general would have any trouble.

It looks just like the style of code in all the other fast LZ codebases. They are all in this style.

The "non-aligned access OK" comment litter is presumably to silence an LLVM performance sanitizer.

<comment><USER10> When implementing complex algorithm, this kind of code is usually easiest to understand.

When you look at the code, you use the paper that describes the algorithm as documentation. Using same short one letter variable names in the code and paper makes understanding much easier.

The thing I hate most is when the the paper uses 1-based numbering and the programming language uses 0-based numbering. We should settle for 0-based numbering when describing algorithms.

<comment><USER11> Arithmetic coding often looks just as dense, because C is not a good vehicle to describe algorithms.

Look at eg [https://www.cs.ox.ac.uk/jeremy.gibbons/publications/arith.pd...](https://www.cs.ox.ac.uk/jeremy.gibbons/publications/arith.pdf) to see a cleaner alternative.

(This is about describing algorithms in papers. Optimizing for performance after the big-O has been taken care of is a different matter.)

</comment></comment></comment></comment></comment><comment><USER12> [https://news.ycombinator.com/item?id=11944975](https://news.ycombinator.com/item?id=11944975)

<comment><USER13> And even earlier: [https://news.ycombinator.com/item?id=11922453#11925386](https://news.ycombinator.com/item?id=11922453#11925386)

</comment></comment><comment><USER14> Here's the github link [https://github.com/lzfse/lzfse](https://github.com/lzfse/lzfse)

</comment><comment><USER15> It's nice to see that we will be able to at least decode these archives. Though I think for new software people are better off using zstd if they're looking for this set of performance characteristics.

</comment><comment><USER7> Interesting to see a benchmark against .zip and .gz in terms of file reduction and uncompress time.

</comment><comment><USER16> _> LZFSE is only present in iOS and OS X, so it can't be used when the compressed payload has to be shared to other platforms (Linux, Windows)._

So now it will be cross platform?

<comment><USER17> You can bet that there is a Linux version soon, and if it's good enough, it will end up in the default repo. For Windows, it's a different story, but maybe because of the iPhone and iPad, and many MS employees using them, will it be supported somewhere in the not so near future. Anyways, 7zip and Winzip will probably support it soon enough.

<comment><USER18> It compiles in Linux as is. There is a generic makefile.

</comment></comment></comment><comment><USER19> If anyone on FreeBSD wants to try this I just added a port under archivers/lzfse

[http://www.freshports.org/archivers/lzfse/](http://www.freshports.org/archivers/lzfse/)

</comment><comment><USER20> I'm not fully up on the latest and greatest in compression technologies but my go to format these days is usually 7zip which I believe is just a container that uses LZMA. For whatever reason *nix people seem to hate it even though I get much better compression with it than tarballs, zlib/zips. Is there a similar container format that will or does use LZFSE? And how much better is it than 7zip/LZMA?

<comment><USER21> The Unixy version of LZMA is called XZ. Unix people prefer tar over zip, probably for historical/tribal reasons.

There are really three different compression "markets"; LZMA/XZ provides strong compression while LZFSE provides moderate or light compression so they don't really compete.

<comment><USER22> > Unix people prefer tar over zip, probably for historical/tribal reasons.

To this day I refuse to use the z argument to tar. Packaging and compressing are two different operations, and this is why the UNIX gods created pipes.

And also I'm a stubborn old fart.

</comment></comment></comment><comment><USER23> Just a side-comment, it would be _really_ nice if we could get the browser vendors to support a newer compression algorithm beyond gzip and deflate. I know there have been a couple others implemented, but nothing that has been implemented by multiple browsers that has stuck. Really need to get MS, Google, Apple and Mozilla to come together on this. Should be patent free.

<comment><USER24> I think Brotli is getting there, with support in Firefox and Chrome so far.

[https://samsaffron.com/archive/2016/06/15/the-current- state-...](https://samsaffron.com/archive/2016/06/15/the-current-state-of- brotli-compression)

<comment><USER23> Thanks for this... the last I checked was a few months ago, as I coulldn't believe we were still limited to deflate/gzip... I'm somewhat surprised lzma didn't get broader support earlier on though.

mental note, setup a new dokku box, and try getting this setup...

</comment></comment></comment><comment><USER25> For anyone wondering, I've tried throwing afl-fuzz at this. It's only been a few hours, but as yet, nothing has turned up.

</comment><comment><USER26> Sorry, I think this comment is out of topic. I don't know why Apple put this separated with their open source projects on [https://github.com/apple](https://github.com/apple).

</comment><comment><USER27> A quick test resutl(zip a 1.5GB file):

        lzfse:
        real	1m44.481s
        user	1m17.956s
        sys	0m2.852s

        lz4:
        real	0m28.136s
        user	0m1.200s
        sys	0m2.240s

lz4 is much faster somehow. The final size are very close.

<comment><USER28> It's not terribly useful for most applications to test compression speed. The only applications I can think of where this is relevant is data backup and archiving.

There are two typical speed benchmarks you want to do. For the "compress once, decompress many times" situation, benchmark the time it takes to _decompress_ and ignore compression time. For the "compress once, decompress once" situation, add the compression and decompression times.

The first situation is common for distributing packages and static assets, the second situation is common for distributing dynamic assets.

<comment><USER29> For the second situation, I prefer the compress/transmit/decompress metric (as a function of transmit pipe speed), as described here:

[http://fastcompression.blogspot.com/p/compression- benchmark....](http://fastcompression.blogspot.com/p/compression- benchmark.html)

</comment></comment></comment><comment><USER30> Waiting for first guy who take this implementation and run it through emscripten so we can actually use it in client -> server communication, eg sending compressed json payloads to the server.

<comment><USER31> Is running (not to mention downloading/parsing/compiling) emscriptened decompression code really that much faster than just transferring the data?

I think a better strategy would be to add support for LZFSE to the browsers themselves.

<comment><USER30> Native support in browser would be appreciated. But I was also thinking about nodejs server side decompressing via native or emscripten transpiled module.

<comment><USER32> A quick reading of the license[0] shows that there's no legal stumbling blocks to the code showing up in browsers, but the license doesn't mention anything about patents.

[0] [https://github.com/lzfse/lzfse/blob/master/LICENSE](https://github.com/lzfse/lzfse/blob/master/LICENSE)

</comment></comment></comment></comment><comment><USER33> Open source; but is it patent-free?

</comment><comment><USER34> Huh, I hope macOS code looks better.

</comment><comment><USER31> The article is essentially a link to [https://www.infoq.com/news/2016/07/apple-lzfse-lossless- open...](https://www.infoq.com/news/2016/07/apple-lzfse-lossless-opensource) with a bunch of ads on top. Maybe someone could update it to point there instead?

<comment><USER35> Ok, we changed to that from [http://www.appleworld.today/blog/2016/7/6/apples- lzfse-compr...](http://www.appleworld.today/blog/2016/7/6/apples-lzfse- compression-algorithm-goes-open-source).

</comment></comment><comment><USER36> It's 2016. How can you launch a reasonably high profile open source project with code that looks like this? This fulfills all the TODO list for unreadable code. One character variable names, one character parameter names, full of magic numbers...

Yes. This is very performance critical code and I completely see the need to write very optimized code. That's fine. But optimizing code for speed shouldn't imply also optimizing it to use as few characters as possible.

Compression code is code that often runs at boundaries to the external world and thus is a very significant attack surface. To release compression code in a non-safe language is risky enough but then using what amounts to practically write-only code is, IMHO, irresponsible.

<comment><USER37> > To release compression code in a non-safe language is risky enough

At the moment, what's their real alternative? Rust is the only memory-safe language I can think of that could hope to meet their performance requirements, but even the Rust runtime would be a lot of overhead for this application.

That said, I agree this isn't acceptable C code for something that runs on untrusted data while using tons of pointer arithmetic.

<comment><USER36> You're right about C. C in general, I would find acceptable, because, yes, there aren't that many good alternatives around for this kind of code.

But there's nothing stopping you from writing readable C code. That's where my concerns come from.

<comment><USER38> I don't really understand where the downvotes come from? I find the readability concerns legitimate, and would like to understand why compression algorithm developers feel like this is OK? Is it just the math heavy background? Can't think of any real benefits to this style.

<comment><USER39> If you don't understand the underlying mathematical algorithms its using, no amount of explicit varible names are going to help you. If you do, the concise structure makes things straightforward. The code is not meant to be read alone and understood, the papers published along with it need to be understood first.

<comment><USER40> Exactly. Not all code can be understandable to layman with zero effort.

</comment></comment></comment></comment></comment></comment><comment><USER41> Well, what's the Weissman score?

<comment><USER42> The Weissman score is the most moronic compression "metric" ever devised. I put "metric" in quotes because from a mathematical perspective, it is practically gibberish. I'm tired of seeing it mentioned in every HN post on compression.

<comment><USER43> I also see this kind of response a lot in situations the score is brought up. I don't have a background in compression, what is the current method for measuring compression efficiency? Is it a balance between size reduction and speed to unpack or is measuring compression efficacy far more nuanced?

<comment><USER42> Usually you compare on a 2D graph, with one axis being decompression speed and the other axis being compression ratio. Sometimes decompression speed is replaced by decompression + compression speed, for round-trip data. Very rarely you only consider compression speed, for "write-many read-seldom" data (like backups).

Normally, any sensible 1D metric would be visualized as isolines in that 2D plane. But the "Weissman score" doesn't even make that much sense, it's a discontinuous, non-monotonic function with singularities right in the middle! It doesn't even make sense from a dimensional analysis perspective… the score will fluctuate wildly based on the _units_ you choose for time and size (minutes? seconds? octets? bits?). There is no conceivable real-world application of the Weissman score. It is just a bit of bad math that appeared on TV once.

</comment></comment></comment></comment><comment><USER44> So... basically a clone of LZ4?

<comment><USER45> Basically an Apple-specific reimplementation of Zstd: [https://github.com/Cyan4973/zstd](https://github.com/Cyan4973/zstd)

<comment><USER44> So, worse than LZ4 for what Apple seems to be using it for. Why didn't they just use LZ4? Confusing company, they are.

<comment><USER16> They are very much into NIH, seemingly out of paranoid fear of patent attacks (though not sure how it can protect them).

<comment><USER12> This is a baseless speculation. First of all, Apple provides LZ4 in libcompression. Secondly, LZFSE uses Lempel–Ziv algorithm and ANS coder invented by Jarek Duda ([https://arxiv.org/abs/1311.2540](https://arxiv.org/abs/1311.2540)): [https://developer.apple.com/library/ios/documentation/Perfor...](https://developer.apple.com/library/ios/documentation/Performance/Reference/Compression/index.html)

<comment><USER46> Well, many people nowadays pretend that, as if they read Jarek's paper, understood it and came out with a genuine implementation of their own.

But Jarek's ANS paper was first out in 2007, and almost no one paid attention to it, because it was plain inscrutable.

Many years later, it took an individual to create FSE ([https://github.com/Cyan4973/FiniteStateEntropy](https://github.com/Cyan4973/FiniteStateEntropy)), to prove that it could be transformed into something actually useful and competitive. Since then, the paper has been updated a few times, borrowing a few points from FSE in order to become more readable. But it's still very hard to read.

In contrast, FSE code can be copy/pasted.

And all of a sudden, lot of versions have popped out over Internet. By pure chance, they all look like derivatives of FSE or Fabian Giesen's rANS, but they pay tribute to Jarek's ANS paper, because quite clearly it is the source of their work, and prior existence of an actual open source implementation which works and looks pretty damn close to theirs was purely accidental.

This is not paying tribute where it's due.

<comment><USER47> Very well said. Its a stark contrast to how Yann Collet acknowledges others inspiration and collaboration ([http://fastcompression.blogspot.fr/2013/12/finite-state- entr...](http://fastcompression.blogspot.fr/2013/12/finite-state-entropy-new- breed-of.html))

</comment></comment></comment></comment></comment></comment></comment><comment><USER48> I just quickly tested it, in terms of highest compression ratio it still does not beat xz, e.g. `tar -cf -FILE | xz -c9e > FILE.tar.xz` [https://blog.benmarten.me/2016/04/01/Compress-Files-With- Hig...](https://blog.benmarten.me/2016/04/01/Compress-Files-With-Highest- Compression/)

<comment><USER6> That's not surprising, given that they went for compression and decompression speed and for energy usage.

Their goal seems to have been to be at least as good as zlib at compressing stuff using less energy and doing it faster (that often correlates quite well with energy use on modern CPUs, as it allows them to drop to low energy states faster)

<comment><USER49> Could you give me some pointers on the actual numbers? My searches came back with nothing. I'm especially interested how they benchmarked the energy consumption.

<comment><USER6> [http://asciiwwdc.com/2015/sessions/712](http://asciiwwdc.com/2015/sessions/712) is the best pointer I know, but it does not give details.

My guesses would be that they have a simulator that computes/estimates power usage, and that they have CPU setups where they measure power usage directly. I doubt they regularly do the "compress things till you run out of battery" thing that that talk mentions. That takes too long, and cannot be used to measure small changes in power usage.

<comment><USER49> Do you mean by extrapolating from executed instructions? It would be very much dependent on the architecture then. It would be nice to read some independent studies and experiments, after the recent events I take any manufacturer claim on consumption with a pinch of salt.

</comment></comment></comment></comment></comment><comment><USER50> Kind of weak licence, what are you actually allowed to do with this code? Change it? Distribute your changes?

[https://github.com/lzfse/lzfse/blob/master/LICENSE](https://github.com/lzfse/lzfse/blob/master/LICENSE)

<comment><USER51> It's the BSD 3-Clause :

[https://opensource.org/licenses/BSD-3-Clause](https://opensource.org/licenses/BSD-3-Clause)</comment></comment>


A Revolution Sweeping Railroads Upends How America Moves Its Stuff - <AUTHOR> https://www.morningstar.com/news/dow-jones/TDJNDN_201904035791/a-revolution-sweeping-railroads-upends-how-america-moves-its-stuff.html <comment><USER2> I do a lot of work with the rail industry in the UK (both heavy and light rail) - but the UK is much more passenger focussed.

Reducing cost can take a lot of tweaking to get right, and is likely to take more than one cycle to do. But new schedules do take some bedding in until they are running smoothly. Once that happens, you can then look for the next round of optimisations - tweaks to the way the trains are assembled in the yards which then start building the incremental gains.

If you have a better scheduling system, you build resilience into the schedule. You open up more time for maintenance, you keep your assets in better condition, so you have fewer breakdowns. One thing to remember is that (in general) there is a Pareto on train breakdowns - 20% of the locomotives cause 80% of the problems. It will be the worst locomotives that are taken out of service, so the reduction in service affecting failures should be quite significant.

You make some money on the OPEX side by doing more scheduled maintenance, and less emergency maintenance (overtime is really expensive!). You also make savings on the CAPEX side, because your assets last longer.

[edit]. Changed para order

<comment><USER3> Can you please explain why "new schedules do take some bedding" ?

Cant u sort of simulate the expected scenarios and get sort-of expected performance?

I realize there is unexpected events, but these will occur anyway in old and new schedules

Also - does real train software has nice graphics or do u get a boring excel ? ;(

<comment><USER4> In theory there is no difference between theory and practice...

As for software, the optimization of rail slot allocation is far beyond spreadhseets - for the whole French network it is a math-heavy process for which a whole team harnesses a bunch of specialist software. More about the process (in French): [https://tel.archives- ouvertes.fr/tel-01423840/document](https://tel.archives- ouvertes.fr/tel-01423840/document)

<comment><USER5> > In theory there is no difference between theory and practice...

This is really good.

</comment></comment></comment></comment><comment><USER6> The customer perspective:[1] The metrics grain shippers use all got worse.

[1] [http://www.grainnet.com/article/160328/precision- scheduled-r...](http://www.grainnet.com/article/160328/precision-scheduled- railroading-operating-ratios-and-grain-service-metrics)

</comment><comment><USER7> > CSX spokesman Bryan Tucker said the company could have done better > communicating the changes to customers, but he defended the actions by > pointing to its financial results.

I think the most telling part of the article

</comment><comment><USER8> I don't understand: how/why is service worse? The article makes it clear that service is worse, but the details on why are important.

I can come up with a number of different scenarios, with different faults and restrictions. Without knowing deeper reasons I don't know if this is a bad idea (as BNSF claims) that needs to be scrapped or a great idea that just needs to get over growing pains.

</comment><comment><USER9> This seems to be an argument that one scheduling system is worse than the previous. However, one of the major objectives of a scheduling system is to reduce cost, which this seems to have done. What is the downside?

<comment><USER10> You are missing the key pay of the question: reduce costs _for whom_? Since railroads are effectively a (regulated) monopoly reducing costs for Norfolk Southern or CSX can mean Wall Street profits combined with worse service for customers at the same price.

<comment><USER11> Wouldn't that be a question for regulators then in terms of how much profit can be extracted without lowering costs or using the savings on other investments?

</comment></comment></comment><comment><USER12> I'm not sure I get the problem. Doesn't this just call for modeling a network of branches, nodes, with set of realistic max/min/stddev timings to traverse each, and an inventory + desired times of trains to be released into the system?

Is it that railroads don't have the expertise to do this? Hard to imagine.

Why are railroad execs huddled around a physical table + map trying to propose timings for trains to run through? Maybe the article is simplifying things that makes it sound really inefficient.

</comment><comment><USER13> Is precise scheduling of trains really a revolution?

<comment><USER14> For cargo yes. Trucks arrival times are predictable within a few hours, trains are within days.

<comment><USER13> If trains are already running precise schedules on those tracks, then it's not that revolutionary really?

</comment></comment></comment><comment><USER15> So, basically a routing/allocation problem? You could throw SAT solvers at it, right?

</comment><comment><USER16> The layoffs are going to hurt. Fewer good jobs for people with those skills.

</comment><comment><USER17> Behind paywall unfortunately.

<comment><USER18> Try 'web' link.

<comment><USER19> Has that ever worked for you with the WSJ paywall? Not for me.

<comment><USER20> One just worked for me for this specific article.</comment></comment></comment></comment>


Ask HN: Advertising on Reddit - US/CA/UK Credit Card required - <AUTHOR>

Now - I know some off you might consider this totally offtopic, but this is the easiest way I could think off to get this job done. I want to advertise on reddit but I'm a swiss credit card holder. I paypal you 60$ and you spend 50$ on reddit ads - your account, you just send me the stats. I'm posting this here, because I don't know anybody from the US, UK or CA that I could ask and because I trust the HN community.
Contact me on email@example.com
Thanks,
johnny <comment><USER2> I'm curious, what your results were from reddit ads? I did a few days worth of ads a month or so back with mixed results. How did yours go?

<comment><AUTHOR> so here you go:

        Impresions unique	total  | Clicks u  t  | click-thru (%) u   t

                   11,488	74,563	       243 286	              2.12% 0.38%

Click-thru is rather poor, but that is probably because an online notepad isn't the hottest thing right now. Anyway, I think the reddit audience still classifies as a rather receptive for these kinds of web apps, you probably would have gotten an even lower click-thru rate on other sites.

<comment><USER2> 16,330 76,30 293 308 1.79% 0.40%

<comment><AUTHOR> thanks. what were you advertising? <http://hangwith.me>? And what kinds of threads - are you posting links on forums?

<comment><USER2> yeah for HangWith. I will post a link to HangWith occasionally along with a comment when I see a post where someone is seeking someone to hangout with or 'forever alone' or something relavant to what the site does.

I don't do it very often. Only when I think the link and comment in appropriate and genuine to the thread. However, when I do and, I think, because the link it relative to the OP I always get amazing click-throughs. Again, the key is being genuine and relative to the thread. I've probably generated 20k+ site visits and well over 1k signups from reddit alone from maybe 3-4 comments posted in relavant threads.

<comment><AUTHOR> just did the same on some forum I've been a member of for quite a while. the traffic is definitely very good - and I think in general probably a lot better than what you get from reddit.

<comment><USER2> Just as an additional comment; we just started sponsoring very small Starcraft2 tournaments and we are seeing excellent returns in clicks but also and hopefully in brand and loyalty.

There really is something to say for finding, building and participating in community.

</comment></comment></comment></comment></comment></comment></comment><comment><AUTHOR> Ad has been paid and set up. Thanks a lot - this is a great community!</comment>


Show HN: Real-Time Monitoring with WebGL for PM2 Downloads - <AUTHOR> http://map.keymetrics.io/ <comment><AUTHOR> Author of this hack speaking - I've built this interface 3 years ago to track downloads of PM2 ([https://github.com/Unitech/pm2](https://github.com/Unitech/pm2)), a process manager for Node.js I've authored. I'm really proud to see this dashboard here on HN!

In term of UI, the map is inspired from the impressive Tron Movie interface (Gmunk artwork [http://gmunk.com/TRON-Board-Room](http://gmunk.com/TRON-Board- Room))

The dashboard is very hypnotic and it asked a lot of meticulous tuning to give this effect. Hope you like it!

</comment><comment><USER2> Best web interface i've ever seen when it comes to design !

That is why PM2 will always be my favorite production process manager :)

</comment><comment><USER3> Great tool. I've been using it for over a year, never let me down. Would recommend for any Node.js dev</comment>


Intent to implement: HTML5 by Default - <AUTHOR> https://groups.google.com/a/chromium.org/forum/m/?_escaped_fragment_=searchin/chromium-dev/HTML5$20by$20default/chromium-dev/0wWoRRhTA_E/__E3jf40OAAJ <comment> n-gauge I'm surprised the Scratch programming language which is currently all flash based isn't on the whitelist ?</comment>


Openmoko starts to take orders - <AUTHOR> http://www.openmoko.com/

<comment><USER2> This company is really cool.. I respect their commitment in promoting the mobile phone as an open development platform. While the hardware may not be as sexy as the iPhone, I hope it still gains market share among hackers.

The GTA01 is the device on sale now, but the GTA02 is the version you want (WiFi, 3D graphics, etc) and it comes out in October.

</comment><comment><USER3> I've been following the Openmoko mailing list. While the hardware doesn't LOOK shiny/sexy like the iPhone, they're packing in lots of stuff, especially in the october release (as you mentioned), which will have the wifi that the people on the list have been clamoring for.

The devs on the list are pretty wary of making a good UI since that'll make or break the device as well. I expect some fairly pretty apps to be on the device.

They get new ideas they bounce around too. The latest was someone talking about making ripple effects on the screen when you touch it, as feedback and a coolness factor.</comment>


Secret Space Plane Can't Hide From Amateur Sleuths - <AUTHOR> http://www.wired.com/dangerroom/2011/03/secret-space-plane-cant-hide-from-amateur-sleuths/ <comment><USER2> I love reading this kind of stuff. I love being reminded of how complicated reality is, given my own high levels assumptions. You think "change orbit, that's easy" or "take space pictures" but the nitty gritty filled reality is so much more amazingly complex, it's mind boggling and impressive.

</comment><comment><USER3> [http://www.smbc-comics.com/index.php?db=comics&id=1522](http://www.smbc- comics.com/index.php?db=comics&id=1522)

<comment><USER4> I'm amused that geeks, to the extent hey notice this at all, think that the space program is _science_ masquerading as _military_. Are people under the impression that Lockheed Martin makes microscopes and centrifuges when they're not building shuttle components? Or that it is coincidental that outgrowths of the billions spent on the space program include Tang, Velcro, and the ability to turn Moscow into glass?

<comment><USER3> Good point. Makes me wonder how much further spaceflight would have progressed if SDI had got off the ground. On one hand weaponizing space is bad, but on the other it could push better space technology forward. A faustian bargain.

</comment></comment></comment><comment><USER5> A previous mention on HN: <http://news.ycombinator.com/item?id=1918368>

In at least two articles about the X-37B, I've read about it changing orbit. But isn't that an energetically expensive operation? In a previous article ([http://www.csmonitor.com/Science/2010/0824/Secret-Air- Force-...](http://www.csmonitor.com/Science/2010/0824/Secret-Air- Force-X-37B-space-plane-changes-its-orbit)) the change seemed to be one of altitude/elevation (sorry if I'm using the wrong orbital mechanics terms ), rather than a plane change. But isn't that still relatively expensive, even with a Hohmann transfer or something? I can't imagine the X-37B can do very many orbit changes - can anyone provide more information on the physics behind this?

<comment><USER6> Excellent question, fascinating answer: <http://www.orbitermars.co.uk/stdorbit.htm>

Trivia: this was the first spot googling "energy cost change orbital plane". I almost didn't click the link "Standard Orbit, Mr Sulu!" but was very glad I did. Bing had Wikipedia as the first spot, not nearly as engaging.

<comment><USER7> I highly recommend the simulator Orbiter. Getting the space shuttle to LEO with this level of realism is _hard_

</comment></comment></comment><comment><USER8> At large observatories (in the US anyway), the military will actually require that the telescope avoid observing certain regions of the sky at certain times so that enterprising astronomers can't figure out the orbits of their spy satellites. Of course, most of the "forbidden zones" are randomly chosen patches of sky so that you can't figure out the satellite's orbits based on the positions of the forbidden zones alone.

<comment><USER9> Source?

<comment><USER8> Hmm...I can't find a source in print, so all I can tell you is that I'm an astronomer who has observed at some of these observatories (e.g. LBT).

</comment></comment></comment><comment><USER10> And according to real time tracking, it's about to fly over Libya:

<http://www.n2yo.com/?s=37375>

No surprise that the US is flying it's secret space spy plane over the country.

[http://en.wikipedia.org/wiki/2011_military_intervention_in_L...](http://en.wikipedia.org/wiki/2011_military_intervention_in_Libya)

<comment><USER11> It's hardly a secret craft - if amateur astronomers can find it, you can bet most of the world's intelligence agencies know quite a bit about it - and now that the West has total air control over all of Libya a space plane that'll pass over the area only once every couple orbits would be a pretty stupid way to do recon.

Fly a few dozen Predators or use the existing air assets already patrolling over Libya and you'll get better data.</comment></comment>


Threader – Compile Your Twitter Threads - <AUTHOR> https://threader.app/ <comment><USER2> Hi, creator of Threader here. Show HN is a way to share your own products. Can you remove the "Show HN" in your title? [https://news.ycombinator.com/showhn.html](https://news.ycombinator.com/showhn.html)

<comment><USER3> You can also contact the mods directly via the Contact link in the footer: they can update the title as well.

<comment><USER2> Thanks, I will do that.

<comment><USER4> Removed!</comment></comment></comment></comment>


Flying frog discovered during NASA rocket launch - <AUTHOR> http://instagram.com/p/eKfsSLIaB9/

<comment><USER2> Pretty much related text: "NASA'S SUCCESSFUL QUANTIFYING OF COMEDY TIMING"

[http://symftr.tumblr.com/post/5987695109/nasas-successful- qu...](http://symftr.tumblr.com/post/5987695109/nasas-successful-quantifying- of-comedy-timing-by-penn)

</comment><comment><USER3> OK, game idea for someone: Space Frogger. Mr Toad tries to dodge orbital debris & marauding aliens on his way to Venus.

<comment><USER4> I wrote this exact game as a high school project, but instead of Venus, I think you had to get to a Stargate / wormhole

</comment></comment><comment><USER5> Why the heck are we launching rockets on top of a wildlife refuge? [https://www.google.com/maps/preview#!q=nasa&data=!1m4!1m3!1d...](https://www.google.com/maps/preview#!q=nasa&data=!1m4!1m3!1d37571!2d-75.4579581!3d37.9241459!4m10!1m9!4m8!1m3!1d75168!2d-75.4496325!3d37.8973297!3m2!1i1440!2i1146!4f13.1)

<comment><USER6> I remember reading that launch sites are often nature reserves _because_ they're launch sites. An exclusion zone enforced by armed guards does wonders for biodiversity.

The artillery training ranges I grew up near to were similarly lush.

<comment><USER7> True, oddly enough. As is, famously, the demilitarized zone in Korea.

<comment><USER8> And indeed are the verges of motorways (highways) in the UK, where stopping is prohibited unless in an emergency and a strip of land leading up to the hard shoulder is fenced off. They are excellent hunting grounds for hawks and other raptors.

</comment></comment></comment></comment><comment><USER9> Was the flying performed on the frog's part, or was it assisted by the 360,000 pounds of thrust coming out of the rocket?

<comment><USER10> Probably: 0.1 lbs thrust from the frog, and 359,999.9 lbs thrust from the rocket

</comment></comment><comment><USER11> Best line in the caption: "...the U.S. Fish and Wildlife Service has an agreement with NASA to use the NASA-owned portion of Wallops Island for research and management of declining wildlife in special need of protection."

Hope that frog wasn't one in special need.

</comment><comment><USER12> Hopefully NASA works to make certain the launch area is clear of small critters in the future. It's really not good PR to be blasting frogs (or anything) with a few hundred thousand pounds of rocket exhaust.

</comment><comment><USER13> How does this go with the space bat?

</comment><comment><USER14> what if one of it's ancestors once thought, i would like to fly like one of those things. a few generations of evolution later you have a flying frog. sadly that's not how evolution works.

<comment><USER15> But there actually are "flying" frogs (they can glide).

</comment></comment><comment><USER16> Looks like someone misunderstood the cooking directions for the bullfrog.</comment>


Voice controlled mouse for those with mobility problems - <AUTHOR> http://spamtech.co.uk/software/voice-mouse-for-mac-os-x/ Software that allows you to control the mouse using your voice. A perfect companion for those with mobility issues.<p>It's a work in progress and makes use of Applescript, python and C++ <comment><AUTHOR> Any feedback would be greatly appreciated :)</comment>


NewtonVR: A free physics-based interaction system for Unity and the HTC Vive - <AUTHOR> http://vrinflux.com/newtonvr-a-free-physics-based-interaction-system-for-unity-and-the-htc-vive/ <comment><USER2> Pretty terrible they are using .NET's System.Collections instead of System.Collections.Generic.

You only ever use System.Collections if you don't want to use generics which is basically never these days. What on earth are they thinking? Or is this yet another shitty limitation of Unity and it's butchered ancient version of Mono?

<comment><USER3> Not sure what you're talking about. I'm using list<> and dictionary<> all over the place.</comment></comment>


Towards deep symbolic reinforcement learning - <AUTHOR> https://blog.acolyer.org/2016/10/12/towards-deep-symbolic-reinforcement-learning/ <comment><USER2> Based on a quick skim, what they're describing already exists in the form of a feature extractor + traditional RL algorithm. State -> features/symbols -> actions as opposed to state -> actions.

So, sure if you split up your decision making into more pieces, then you can specialize those pieces better to your specific task, then it will probably perform better at that task. But the whole point (and success) of deep learning is the feature extractor and decision model should are one network, where the early layers are acting as feature extractors (symbolic component) and the later layers are acting as the decision makers or value learners.

In my mind, this partitioning strategy runs counter to the greater goals of machine learning, which to me is a generic learner that can 'take raw data in' and 'get smart stuff out'.

<comment><USER3> You're definitely right that we've found joint training to be highly useful and having a neural net simply act as a feature extractor is unsatisfying, but I think that success at unifying neural nets with symbolic or even non- learning systems would be a big step forward. This isn't it, but even the alphago work relied on Monte Carlo tree search, so I wouldn't dismiss this out of hand.

<comment><USER4> But MCTS works in AlphaGo because the 'world' can be fully represented by a tiny state vector which fits a billion times into primary memory these days. The real difficultly of real world problems is the ill-defined, mutable, high- dimensional and partially observable state space.

<comment><USER3> I'm merely pointing out that "all neural everything" is not always the best approach despite joint training being very appealing.

</comment></comment></comment></comment><comment><USER5> It seems like a really interesting approach but it needs to be validated on a larger set of Atari games. Considering even a Linear learner outperforms DQN on several games such as Double Dunk. It's possible that the DQN underperforms because the game is more amenable to the symbolic representation used. [0]

[0] Figure 3 [http://home.uchicago.edu/~arij/journalclub/papers/2015_Mnih_...](http://home.uchicago.edu/~arij/journalclub/papers/2015_Mnih_et_al.pdf)

</comment><comment><USER6> In addition to the intro that I liked a lot, I find interesting, or even important, the usage of an unsupervised method to generate the visual symbolic dictionary, a convolutional autoencoder here.

In arxiv:1412.6856 unsupervised object detectors emerge as byproducts in a CNN for another kind of task, for more interesting, photorealistic objects.

</comment><comment><USER7> I feel that the use of moving objects weakened the case in the example, such as requiring object persistence priors. The paper also did not report theoretical maximum scores. Instead, I would have liked to see a planning scenario:

1\. Objects are randomly placed such that all circles are reachable without hitting exes.

2\. Objects do not move.

3\. The objective function is the total points collected, discounted by time.

In this case, which is essentially the traveling salesman problem[1], it is possible to analytically compute the optimal strategy using discrete optimization methods (e.g. dynamic programming[2]). The optimal strategy gives us an absolute basis for algorithm evaluation. Next, divide the playing space into groups (maybe via spectral clustering[3]). Use dynamic programming to determine the order in which to visit groups, and automated planning for the sequence of moves to reach the groups. After group planning comes step planning. Build a state predictor that maps a the current state and a set of steps to a new state, where the steps have been determined through automated planning to reach an object. Again, use something like dynamic programming to determine the optimal series of plans to visit all the objects within the group.

This would be a hybrid system for using NN-derived features to fuel discrete optimization and automated planning algorithms. Such an approach should learn faster than DQNs and generate good solutions faster than dynamic programming.

In the more general case of an Atari game, I can see clustering frames into game states, mapping moves to those states, and using automated planning to determine the series of states, and therefore moves, to optimize score.

[1]: [https://en.wikipedia.org/wiki/Traveling_salesman_problem](https://en.wikipedia.org/wiki/Traveling_salesman_problem)

[2]: [https://en.wikipedia.org/wiki/Dynamic_programming](https://en.wikipedia.org/wiki/Dynamic_programming)

[3]: [http://scikit- learn.org/stable/modules/clustering.html#spect...](http://scikit- learn.org/stable/modules/clustering.html#spectral-clustering)

</comment><comment><USER8> Disclaimer: I work in the lab that created Spaun.

This seems kind of similar to Spaun [0]. You use Deep Learning to get the features you want and then you do symbolic like operations on those feature vectors to accomplish reinforcement learning. At least Spaun used neurons for the rule manipulation too. I also don't believe the transfer learning claim at all, because the tasks are way too similar. Still, it's a step in the right direction as far as I'm concerned [1]. That being said, good luck find a set of general prior rules and scaling them.

[0] [http://science.sciencemag.org/content/338/6111/1202](http://science.sciencemag.org/content/338/6111/1202)

[1] [https://medium.com/@seanaubin/why-does-ai-still-suck-and- can...](https://medium.com/@seanaubin/why-does-ai-still-suck-and-can-it-suck- less-9db36be294dc#.akb82s8o9)

</comment><comment><USER9> See also [http://www.neural-symbolic.org/](http://www.neural-symbolic.org/)

</comment><comment><USER10> Isn't this approach similar to or used in DeepMind's AlphaGo where the policy network is corresponding to high-level representational knowledge(kind of expert system in traditional AI), and the value network is corresponding to the decision making part(reinforcement learning)?

</comment><comment><USER11> Automated planning systems and languages have some interesting ideas you could look into. I worked on a similar idea with yours for resource allocation a while ago, even before deep learning and automated planning had a lot to offer.

</comment><comment><USER12> The symbolic component can be seen as programming so this is a mixture of machine learning and classical programming.

<comment><USER13> The name "symbolic learning" also applies things like the Google's Page Rank.</comment></comment>


User gets fed up with bug; cracks program to fix it himself. - <AUTHOR> http://forum.skype.com/index.php?s=17fbdf08801503eebf66d315f03d14b6&showtopic=310121&st=20&p=1633781&#entry1633781 <comment><USER2> I actually had a similar frustration for Half-Life 2 single player. Valve originally hard-coded the field of view (fov) to 75 which caused intense motion sickness for me (for some reason). I enjoyed the game immensely, but I still had to go lay down for 20 minutes for every 20 minutes of play.

In the first few weeks of the game (before they patched it to be less restrictive), it ignored and reverted changes to the game fov with some sort of bizarre level of meanness. They _really_ wanted the fov to stay at 75 for some sort of game experience reason.

I edited the game binaries to remove the forcing of that field of view back to 75 (using VC6's binary editor after using SoftICE to monitor the code that was changing the value in memory) so that I could play the game without getting sick.

<comment><USER3> If not 75, what # did you play at?

<comment><USER2> 90 degree fov was what worked for me (and most shooters I know used at the time). Sorry I didn't clarify that above.

They eventually allowed changing it as a cheat, but it was frustrating for a while .

The community talks a little bit about the differences/issues here: <http://developer.valvesoftware.com/wiki/Field_of_View>

</comment></comment></comment><comment><USER4> I has a player of one of my games pull a similar stunt on me.

He was sick and tired of other people using bots to cheat the game. He made some correct assumptions about my architechture, Memcached in front of MySQL, and determined a way to find pages that would guarantee a cache miss. He then wrote a bot to hit as many of these pages as possible, essentially blackmailing me into fixing the bot problem.

A simple captcha and a few bot honeypots later, the problem was fixed.

<comment><USER5> I don't think that is a fair comparison.

Discussions about the ethical implications of patching a DLL aside, it is quite a different thing to patch a DLL to prevent an application from crashing when you use it than to hammer someones server in an attempt to force them to fix a problem.

</comment></comment><comment><USER6> It reminds me of TTDPatch. It started with minor modifications to Transport Tycoon, but eventually rewrote large parts of the game itself.

Summary: <http://www.ttdpatch.net/features.html>

Details: <http://www.ttdpatch.net/Manual/ttdpatch.html#The%20Patches>

<comment><USER7> They finally rewrote the game - OpenTTD - <http://openttd.org>

<comment><USER8> Oh why did you guys link to this, do you know how many hours I'm going to loose now?

</comment></comment></comment><comment><USER9> Can someone explain how one might do something like this? I get that you can use some sort of disassembler to read the code, but how did he insert his check without upsetting the machine code's memory offsets?

<comment><USER10> There are two (easy -- there are others) ways to pull it off. One is to replace instructions in the area with smaller ones, and take advantage of the open space. This is generally fairly easy to do, as compilers are terrible at generating code that's both fast and small.

The other way to pull it off easily is to replace an instruction with a jump to another part of code that's empty/unused, where you put the original instruction, your new ones, and a jump back to the patch location.

<comment><USER11> I was wondering that myself, thank you for the great reply -- I don't have that much experience modifying disassembled code.

</comment></comment></comment><comment><USER12> As linux user I cannot be anything but glad to see skype developers a little bit pwnd. What is wrong with skype? They detain the monopoly on VoIP and yet they have not been able to keep up with the development of the program in years, now. It seems it is all about selling headphones on the website.

I think the linux version is something like 2 years old now.

<comment><USER13> They don't have a monopoly. You are free to use other VoIP apps or create your own.

<comment><USER14> According to Wikipedia, a monopoly "exists when a specific individual or enterprise has sufficient control over a particular product or service to determine significantly the terms on which other individuals shall have access to it.".

Given Skype's market share (not Voip in general, but free worldwide voip calls), I would say they do have a monopoly.

</comment></comment></comment><comment><USER15> As far as I understand, Skypemate is a hardware phone made by company named Yealink and designed to work with Skype. It is not made by Skype, and that can possibly explain why it is not so polished and bugs are left unaddressed for long time.

If the user really cracked Skype, as the title made me believe, this would be big news. I have been told that Skype's protection is really tough.

<comment><USER16> Skype voice encryption is really though, meaning that agencies can't tap Skype. Change the machinecode of a program on your own pc is something entirely different.

</comment></comment><comment><USER17> Like the resume of a friend I read once (Stanford, Microsoft, etc.), I like how the guy talked about what he did (a fairly difficult debug/patch) in a low-key manner.

<comment><USER18> The excel multithreading thing, right?

<comment><USER17> No, I think you're probably thinking of someone else. His resume had no fluff. Sometimes it's like you look at normal people's resumes and they try to fluff it up. His was like: "Proficient: MFC, Win32, C++, C, Japanese...Stanford CS GPA 3.93 etc..MS SoftDev Intern". And it didn't take up the whole page.

I'm curious what the Excel multi-threading issue is/was though.

</comment></comment></comment><comment><USER19> Is it wrong to bump up a post based solely on it's title? Title love. Now to read the article.

</comment><comment><USER20> Title is very misleading. A better title could be

'User got fed up with the bug and finds a workaround.'

<comment><USER21> He patched the DLL to prevent a bad pointer dereference - that qualifies as cracking it in my book.

<comment><USER10> Why is that cracking? It's just a hack; patching a binary to fix a bug is by no means a crack IMO. A clever hack? Sure, but calling it a crack just serves to make it sound like a bad thing.

<comment><USER22> That was my big take away too and my sense was that in another forum, he would have used the term hack. But it's easier to explain it away as a crack to convey that users should observe caution, share it quietly, and also to piggy back on a commonly held notion that it won't break or otherwise harm any system (such as the service being used).

</comment></comment></comment></comment><comment><USER23> That's amazing. Consider him hacker certified lol

<comment><USER24> Damn - I was expecting to see something horrific in your comments judging from all the downvotes (had to highlight to read). What was so annoying about this comment? Relax folks!

<comment><USER25> Writing "lol" means almost automatic downvote in HN. And that's usually a good thing.</comment></comment></comment>


Facebook Takes Away Eclipse IDE's Page and Hands It Over to Twilight - <AUTHOR> http://twitter.com/ralph_mueller/status/128560326798082048 <comment><USER2> This is a very good object lesson in "Don't hang your brand on Facebook" (or any other service that you don't control or have an SLA with). Eclipse existed long before the Twilight series, and here it is ousted because it has less mass appeal than a movie about horny vampires. Obviously seniority and first- to-field isn't a factor with Facebook.

It really sucks for the Eclipse project, though. I hope they're able to get it back, though knowing Facebook, I'd be genuinely surprised if they did.</comment>


4 Most Popular Static Site Generators You Should Know - <AUTHOR> https://bejamas.io/blog/guide-into-static-site-generators/ <comment><USER2> FWIW, Jekyll, Hugo, Next.js, and Gatsby.

There are so many SSGs because they are so easy to write. I did one for my own web site, many years back.

Now I want to migrate, because I forgot how my own code works. I keep a version of Python 2.6 around just to keep it going. As per paradox of choice, there are so many options around that I don't know where to start.

I want Python, just because that's what I'm used to. I tried Pelican, but seemed slow and too blog oriented. Next I'll try Nikola (because of the doit- based build system) and then perhaps also Lektor (to try out the CMS).

So very many choices.</comment>


The Itch (2008) - <AUTHOR> http://www.newyorker.com/magazine/2008/06/30/the-itch?currentPage=all

<comment><USER2> _> Aquagenic pruritus is recurrent, intense, diffuse itching upon getting out of a bath or shower, and although no one knows the mechanism, it's a symptom of polycythemia vera, a rare condition in which the body produces too many red blood cells._

Oh wow, so that's what that is. I get that. I always assumed that was typical; it's a pretty horrid sensation, it's like my eczema crossed with pins and needles, and it happens when I get in or out of a shower or bath, especially quite hot ones (but even just warm can do it).

</comment><comment><USER3> I had to stop reading this article because it was making me itch all over.

<comment><USER4> Likewise. Fascinating and amazing.

</comment></comment><comment><USER5> Oh man, I remember reading this right after it was published.

How such a mundane thing as an itch can completely wreck your life and your body -- I still count it as one of the scariest things I've ever read.

</comment><comment><USER6> There was another New Yorker story about a man (or multiple people) with some disorder that caused them to be massive assholes to everybody as well as try to harm themselves despite not wanting to. I remember one guy carried around a little business card that explained it and it described another person who would "accidentally" say "eat shit and die" as a greeting. It was around the same time period. Anybody here remember it and can link it? I've been looking for years.

<comment><USER7> I think the disease is Tourette's Syndrome. That could help with the search for the article.

<comment><USER8> South Park did an episode on that: [http://en.wikipedia.org/wiki/Le_Petit_Tourette](http://en.wikipedia.org/wiki/Le_Petit_Tourette)

Cartman fakes to have tourette so he can be asshole to everyone</comment></comment></comment>


Ask HN: Where are you from? - <AUTHOR> I have always been curious where everyone is from.  I am sure there are plenty of people from Silicon Valley on here, but I would also guess there are many people from many different places.<p>I am from the New York area. <comment><USER2> Hradec Králové, Czech Republic, but I currently live in Prague and frequent London

</comment><comment><USER3> Mission BC (In the Fraser Valley/65k east of Vancouver)

</comment><comment><USER4> From NYC. Now in Manila, Philippines.

</comment><comment><USER5> Moscow, Russia

</comment><comment><USER6> Columbus, OH

</comment><comment><USER7> Near Bristol, UK

</comment><comment><USER8> North East UK.

</comment><comment><USER9> Buffalo, NY</comment>


Names and Locations of the Top People Killing the Planet - <AUTHOR> https://decolonialatlas.wordpress.com/2019/04/27/names-and-locations-of-the-top-100-people-killing-the-planet/ <comment><USER2> So let's say a genie grants your wish and lets you remove all of those people and companies. It wouldn't change a thing as long as consumer demand remains.</comment>


Ask HN: Books you would recommended for starting out in product management? - <AUTHOR> <comment><USER2> These books apply to pretty much any type of product:

"The Mom Test" by Rob Fitzpatrick will teach you how to figure out what your customers really want, because it is almost certainly not what they are asking for.

"How to Make Sense of Any Mess" by Abby Covert. How to organize and present information.

"To Sell is Human" by Dan Pink. This is how you should look at sales and marketing.

Assuming you are going to be building software:

"Seductive Interaction Design" by Stephen P. Anderson.

"Planning Extreme Programming" and "Extreme Programming Explained" are must- reads for working with software teams.

I've worn both product and engineer hats on my path to being a sort of rentable CTO, and am always happy to help out people that are getting started as a PM -- email is in the profile.

Good luck!

</comment><comment><USER3> I've been going through this list:

[https://medium.com/@noah_weiss/50-articles-and-books-that- wi...](https://medium.com/@noah_weiss/50-articles-and-books-that-will-make- you-a-great-product-manager-aad5babee2f7#.3gopwkxkk)

Includes books and articles. Been very helpful so far.

</comment><comment><USER4> "Badass" by Kathy Siera: how to make products that succeed by making your users be badass

"Design for How People Learn" by Julie Dirksen: instructional design, but any product will involve a bunch of teaching.

Will return with more if I remember any.

</comment><comment><USER5> How to Win Friends and Influence People.

Being able to get people in your corner will help you.

Don't Make me Think, Zero to One, The Lean Startup, and anything about capturing metrics and data driven decisions is great too.

</comment><comment><USER6> No book. Build a product from scratch. If you've never built a product from A to Z, including having at least 1-2 users you're communicating with, then you won't land a PM job.

<comment><USER7> Yep, build and learn... the best way to do it

</comment></comment><comment><USER8> "inspired, how to create products that people love" by cagan, his blog on svpg.com is good too.

"Lean startup"

"The hard thing about hard things"

</comment><comment><USER9> Rework, Learn startup

</comment><comment><USER10> Cracking the PM Interview</comment>


China Government: "China Needs Special Agents to Gather Secrets from the US" - <AUTHOR> http://ipvm.com/forums/video-surveillance/topics/china-government-declares-china-needs-special-agents-to-gather-secrets-from-the-us <comment><USER2> This is one of those things that exists in a continuous feedback loop. China is spying on us, so we have to spy on them; the US is spying on us so we have to spy on them.

Thus the digital cold war continues. It's no wonder that our government wants more control over tech and encryption. China has control over their companies and a firewall around the whole country.</comment>


Blank business cards - <AUTHOR> http://thestartuptoolkit.com/blog/2013/08/business-cards-can-slide/?utm_source=feedburner&utm_medium=twitter&utm_campaign=Feed%3A+TheStartupToolkit+%28The+Startup+Toolkit%29

<comment><USER2> I've always preferred cards that were smooth white on at least one side. Made it much easier to jot down some extra info.

I give out so few cards I decided to hand-print a batch using a rubber stamp and a small stencil I used with some flat paint. No two are alike. Has a nice texture.

The back is blank for any additional stuff to be added.

Key info (URL, E-mail, phone) was pre-printed, though, since I do not want to have to trust my handwriting skills if I'm rushed.

I've had people hand me cards, tell me the E-mail address printed on it is wrong, and then they scribble down their correct address; later I find that I cannot read it.

</comment><comment><USER3> I do this too. Strangely that's one of the things that actually make business cards stand out. I have a stock of regular cards, but they seem pompous by comparison.

I first got those perforated cards made for inkjet printers you'll find in stationary stores, but I can't always find them. So I got some blank card stock and used a paper cutter to get them down to size. Sadly my handwriting is rubbish, but I try to write slowly. Seems to leave a bigger impact and it feels more "personal".

...I'd still avoid shorts though ;)

</comment><comment><USER4> I hate to admit how awesome this is. It's specific, bold and fits the need at hand.</comment>


Washington Post Ran 16 Negative Stories on Bernie Sanders in 16 Hours - <AUTHOR> http://www.commondreams.org/views/2016/03/08/washington-post-ran-16-negative-stories-bernie-sanders-16-hours <comment><AUTHOR> Confirmed: Bezos is doing everything he can to prevent Bernie Sanders from winning.

</comment><comment><USER2> You can search Google News for Daily Telegraph (UK) articles about opposition leader Jeremy Corbyn, to see a similar campaign. Around the time of vote for the new leader of the Labour party, this newspaper went crazy trying to smear him with some epic headlines :D</comment>


Jquery.com uses only 34% of jQuery - <AUTHOR> http://blog.bolinfest.com/2011/04/jquerycom-uses-only-34-of-jquery.html

<comment><USER2> I really feel the headline should be "Non-issue used for article to drive advert for my FireFox addon". There was really nothing of interest there; the whole point of the kind of toolkit (jQuery) is to have good coverage across as many common tasks as possible, but if you need very specific functionality you end up with something of a scatter-gun. Not being able to compile you own intensely specific version is a trade off between ease of setup over results, personally I would say it is the correct choice (for standard web use, I wholeheartedly disagree with they way that jquery mobile is heading).

</comment><comment><USER3> The benefit of using the whole jQuery library on a single page is that it will likely already be cached, particularly if you are using the Google version.

The largest time expenditure is on the loading from an external server, not the processing once it is local. Don't think the author understands this.

<comment><USER4> actually, i think the author _does_ understand this and in the case of stackoverflow, i think he's obliquely making the case that many of the effects could otherwise be achieved with css without creating the performance and memory hits you might get from using $().

in most cases, js performance is a non issue, but when it is, it's interesting to see how and where you can optimize, which is really what his add-on (again, obliquely the real point of the post) seems to target.

<comment><USER3> Some what ridiculous that we must figure out what the point of his post is instead of him just telling us.

</comment></comment></comment><comment><USER5> From the Article:

"From these preliminary results, I believe that most sites that use jQuery could considerably reduce the amount of JavaScript that they serve by using Closure. As always, compiling custom JavaScript for every page must be weighed against caching benefits, though I suspect that the Compiler could find a healthy subset of jQuery that is universal to all pages on a particular site."

This is the interesting point - is there a subset of jQuery that includes enough for most sites? That subset could potentially be served alongside the main jQuery release, and if enough sites use it, it might be worth it.

Otherwise, right now, caching means jQuery bloat isn't too big a problem, I believe.

<comment><USER6> Sizzle.js maybe? I've tossed CDN jQuery into a couple projects to get Sizzle and jQuery's AJAX wrappers, but usually ignoring the visual effects capabilities of jQuery entirely.

Honestly though, I've never done anything at large enough scale that serving something like 14kb instead of 29kb mattered to me or my users.

</comment></comment><comment><USER7> I think picking jQuery as an example wasn't the best choice since it probably is already cached for sites that use google's or other popular CDNs. This is true for StackOverflow (one of the examples in the article).

The main point of the article still stands though. Closure and similar tools make a lot of sense because they can reduce the size of js quite a bit. Especially for huge apps that require several libraries but only use parts of them it can make a huge difference.</comment>


Should the 27 Club Be a "Thing"? - <AUTHOR> http://blog.pbutler.com/2011/07/should-the-27-club-be-a-thing/

<comment><USER2> No, it's just another coincidence. Tons of famous people have died at the same age as a bunch more. It's like noticing when a streetlight goes out when you walk past, you wouldn't notice it any other time, or be around when it occurs without you but people seem to think they have some mystical connection with them...

Honestly think that the whole 27 club thing is just another way to pad out some celeb news...</comment>


Types of Engineers - <AUTHOR> http://omar.io/2017/01/10/the-types-of-engineers.html <comment><USER2> This reads like a horoscope. I can tut-tut at the over engineers and under engineers while considering myself _just right_.

Anyone to the left or right of me on the spectrum will also think they're _just right_ and I'm one of the former or latter. This recursive confirmation bias then sends the article straight to the front page.

This is the most smug article I've read on here in a long time. Why can't we just be professionals?

<comment><USER3> I think a far more useful way to divide up software engineers is by how good they are at gathering requirements/understanding the problem they have been tasked with.

That said, and this might be crazy... but sometimes it is nice to have people good at different things on your team :)

With a good lead you can have a person with overengineering leanings on your team and have it be a strength as long as you balance that with other ways of thinking.

<comment><USER4> I find that the only acceptable way to divide engineers is with a chainsaw. It might get messy, and is hard to hunt them down especially if they are of the runner kind, but afterwards you get the satisfaction of a work well done.

<comment><USER3> Slow Romero style coders can be sawed 10x faster than other normal ones.

</comment></comment></comment></comment><comment><USER5> Sorry ahead of time for going slightly meta and off topic. I am a software engineer myself but...

I find the fact that this blog refers to software engineering practices as if they apply to all engineering disciplines to be a little jarring and best and misleading at worst.

I wouldn't expect a mechanical engineering blog to have an article "Types of Engineers" that only discusses mechanical engineering.

I refuse to not call software an engineering discipline (I believe it is or at least strives to be) but even so I make sure to quantify my statements by putting the word "software" in-front of "engineer."

<comment><USER6> Author here.

I won't get into whether or not it's fair to call call "developers" "engineers", but I do make the distinction in the first paragraph that the article refers to software engineers. In addition, velocity, bug count, and lines of code are all software terms. I can't imagine anyone thinking this is referring to any other type of engineering.

<comment><USER7> Velocity? How is that a software term?

<comment><USER8> It's a pretty well known term in agile software development.

<comment><USER9> It is also a pretty well known term in other fields, and was long before agile software development came around.

<comment><USER10> In this case, other context clues tell us that the article means 'velocity as in software development'.

Agile, for example, is another general term that has a fairly specific meaning in this context.

If you overhear someone in a park say "Oh, you guys are agile? What's your average velocity?" would you assume they are talking about athletics or software development?

</comment></comment></comment></comment></comment></comment><comment><USER11> From what I can tell almost everyone is an "over engineer". I fight this mindset in myself everyday. How many times have I seen interfaces and dependency injection and kinds of patterns used (besides MVC) on crappy little web sites that get re-written every 3 years anyway.

No we don't need an interface for something that is not a plug-in or will never be released as a stand-alone library! Coding for a future that will never happen is a fools game and none of the real bosses will care unless the UI or something more tangible (performance?) changes drastically.

<comment><USER12> Finding the balance is one of the most difficult things in software. On the one hand there's YAGNI (You ain't gonna need it) which you described, and on the other end are "hacky" prototypes that take the most direct approach but often ignore edge cases and are hard to build upon.

</comment></comment><comment><USER13> This reads like a vast oversimplification of types of Engineers and it paints certain design goals as "good" and "bad" .

<comment><USER14> Seems a pretty naïve description of Engineering types. Not only is there no data to back this up, but it makes it seem like hard divisions: I can anecdotally tell you about junior engineers who over-engineer in the wrong areas, or experienced engineers who `under-engineer` due to time|pressure|issue-critical requirements.

Nonsense.

<comment><USER6> Author here. Your point is valid. I do try to make this clear in this portion:

> However, an over-engineer can sometimes perform like an under-engineer or an > engineer. Underperformance and overperformance could be influenced by a > number of factors including work environment, context, intellectual > horsepower, and even personal life.

I personally continue to write under-engineered and over-engineered code because there are so many internal and external factors that go into solving a problem.

<comment><USER14> I encourage you to keep writing, but it seems you need more research into developer patterns if you want to contribute to an already highly discussed topic.

For instance, last month this article[1] gave some seriously good insights to developer checkin frequency based on hard data. It provides a lot more value than the anecdotal, muddy definitions of over/under/engineer.

[1] - [https://blog.gitprime.com/check-in-frequency-and-codebase- im...](https://blog.gitprime.com/check-in-frequency-and-codebase-impact-the- surprising-correlation/)

<comment><USER6> I appreciate the feedback.

That's a very interesting article. However, it does seem suffer from the same issues Joel outlined, namely, the measure of attributes of a commit can be gamed and worked towards to achieve a "prolific" or "senior" status.

What the article you linked does reinforce is the value of iterative development. As opposed to doing large infrequent commits, doing small bits of work more frequently causes a success higher or "impact".

I would say that the article provides a data driven approach to justifying the effectiveness of good iterative planning. However, it doesn't address whether the commits of the "senior" engineer or the "prolific" engineer are of high quality.

</comment></comment></comment></comment></comment><comment><USER15> All three of these categories are served by a product team that makes economical constraints explicit.

Too often business comes into the conversation with e.g. "We need a disaster recovery system" and no figures on desired RPO or RTO, no figures for what a minute's worth of business data, or a minute's worth of business activity costs or expected incidence or duration of outages from which even the True Engineer could derive suct figures. Too many organizations, when an individual engineering contributor starts asking questions about these business figures put on a face as if a dog started speaking to them instead of having answers.

It's somehow not seen as inevitable that we, in this vacuum of evidence, reach for all we have -- assumptions based on individual personal prejudice, or just whatever seems easy/fun/cool to implement.

Why should I do the effective thing when I don't get a bonus when the solution works, but I do get overtime pay (or reduced feature load in the sprint) when it fails?

Why should I do the simple thing when I could do the thing that I can put on my resumé and hope I can be a member of a team that either doesn't have a role called "Business Analyst" (because isn't that everyone's job?), or has such a role that actually performs analysis instead of just being a mouthpiece for management.

Sure, a good engineering manager will know this is what has happened, and will push for business to consider the economics of potential solutions and facilitate the decision to implement the solution with the lowest expected costs in the face of risks, but the common level of skill here, and all I've ever been privileged to work with can't process figures like these (everything is either unknown, a feeling, or - rarely - a single precise number, it is literally unthinkable to have a distribution of possible numbers or do maths on such figures), and won't notice their absence or seek them out.

<comment><USER6> Your points couldn't be more accurate. In any organization, the quality of the function that dictates the work to be done (be it Product Managers or Business Analysts) heavily impact the success of an engineer's work. Some organizations choose to remove this level of management [1], others have ensure only former engineers fill this role.

It's very important but broad topic. I chose to not explore it in the post because it deservers its own discussion.

[1] [https://www.quora.com/Does-Stripe-have-product-managers- or-d...](https://www.quora.com/Does-Stripe-have-product-managers-or-do- engineers-manage-the-products-themselves)

</comment></comment><comment><USER16> Eh, it's pretty much the same stuff over and over again that gets posted here when people try to categorize engineers into types.

Maybe saying the same thing over and over again has merit to nail in the idea but eventually it becomes stale.

</comment><comment><USER17> No. Stop putting people in boxes. People don't fit into boxes. They're people, not rats.

<comment><USER18> I don't get it. If rats fit into boxes, surely people fit into bigger boxes?

You can certainly categorise people as well as rats. Some are faster / slower, (not-)curious, low/high-energy, ...

<comment><USER19> It's a totally shallow classification. Slower and Faster are relative terms, curious is a relative thing, energy is a relative thing. The problem is that engineers create these generic "boxes", place their staff in it, and divvy out work/rewards/responsibilities based on it. It's just like designing an object oriented framework. Your original classes are probably not quite right.

The problem is your boxes are almost certainly overly generic. You might say Joe is just a slow developer, put him in that box and move on. Maybe you talk to Joe, and he doesn't quite tell you exactly why he's slow... he might not know why. You have to dig deeper into it. Maybe he has weaknesses he is afraid to tell. For example, he might be slow because he has 10 years of doing C++, and now he's on a Node.JS team. All his prior knowledge is nearly obsolete. Maybe he's slow because he's not interested in tech problems, instead he's more interested in learning the domain... so he's just not motivated to work fast. Maybe he's slow because he wants to move up in his career, but there's no clear path where that is possible. Maybe he's slow because he has a new born baby, and he doesn't sleep at night.

If you just put him in a box, you're not going to work to try and find his strengths, and play to them, you're not going to work with him to find solutions. It is YOUR job as manager to learn what makes up your team as individuals, and to help them grow. If you just put him in a "box" you're going to call him weak, and your team will work at a crippled pace.

<comment><USER18> I don't understand even the first paragraph in your argument. Are you saying you can't pick some rats which are faster than some other rats?

Yes, it's relative - everything is relative - but what does that have to do with shallow?

If you are looking for rats to win a race, clearly some rats will be much better to use than others.

What you just wrote feels like a rant without a coherent basis.

<comment><USER19> Your comment is unnecessarily combative, so i'm not going to further explain myself. The issue with your line of reasoning is that your analogy doesn't hold up. An engineering team building a product is not a rat race.

<comment><USER18> Maybe you are unaware, but the term "rat race" is a colloquial expression that is used to describe the type of situation that many office workers are in:

[http://www.urbandictionary.com/define.php?term=rat%20race](http://www.urbandictionary.com/define.php?term=rat%20race)

That isn't what I was talking about but it was an interesting choice of words :)

</comment></comment></comment></comment></comment></comment><comment><USER20> As an Under-Engineer, I enjoy writing code like this:

         for (i=0;i<n;i++) {
            printf("Please enter the %d%s array item\n",i,i==1?"st":"th");
            scanf("%lf",d+i);
         }

<comment><USER21> There's nothing in this world quite like entering the 2th array item :P

</comment></comment><comment><USER22> The old tongue in cheek classification of SW Engineers is something like

smart & eager, smart & lazy, stupid & lazy, stupid & eager

stupid & eager does the worst damage and smart & eager doesn't exist (;

</comment><comment><USER23> > Mid-level engineers who recently discovered software design patterns tend to > fall into this category.

A mid-level engineer who only just discovered Design Patterns. So, a first or second year university student is a mid level engineer?

<comment><USER24> In theory yes, in practice no. Generally speaking university students learn, prove-knowledge-of, then move on to the next course/subject.

University projects also tend to be much smaller in scope than any sort of realistic product that a company can be built on. Therefore, the level of technical debt created by under-engineering and the codebase comprehension learning-curve created by over-engineering is not something most university students have exposure to.

Therefore, I think the article's intent hints more at experience than knowledge. It is possible that once a graduate gets enough experience with an under-engineered project, a lightbulb moment will occur where they see how they can apply their university knowledge by refactoring everything to be more testable, scalable, etc. Depending on the opportunities offered, they may well choose to over-engineer their next task.

Given the theoretical and academic nature of most course work though, it's more likely that an over-engineered solution is inspired by a HN post than by someone recalling their university education.

<comment><USER23> Good point! There is of course a difference between just studying for an exam and retaining the information long enough to pass the test, and actually retaining the information for a longer period and being able to apply the learned principles in bigger, industrial-scale projects.

</comment></comment></comment><comment><USER25> The idea that engineers are abstract thinkers is completely laughable. An abstract thinker wouldn't ask you to show them use cases or test cases. An abstract-minded programmer:

(0) Views a program as a relation between the initial and final states of a process controlled by it.

(1) Understands that the sheer size of this relation makes it impractical to reason about it by enumeration of its members.

(2) Develops calculational machinery to minimize the amount of intellectual effort required to establish that a program meets a specification.

</comment><comment><USER26> Unlike what the article suggests, I was an over-engineer when I was junior but now (I hope) I'm closer to an engineer. Could be a personal thing for me but I often preach to the juniors that they shouldn't build "platforms", just solve the problem and see what emerges. That's difficult, though, because you have to solve the problem in a future-proof way, which I only learned after many years of practice.

</comment><comment><USER21> Random thought:

>Software organizations tend to reward programmers who (a) write lots of code and (b) fix lots of bugs. The best way to get ahead in an organization like this is to check in lots of buggy code and fix it all, rather than taking the extra time to get it right in the first place.

What are HN's thoughts on a system that rewards developers that fix a lot of bugs, alongside rewarding devs that produce bug-free code?

<comment><USER24> I think a key metric that is often missed about developer contribution to a project is lines of code in production and life-span of committed code.

Although these metrics are also prone to manipulation as people reformat code to get their name on git blame, or write perfectly redundant code rather than improve on existing code.

</comment></comment><comment> pow_pp_-1_v These classification may or may not make sense. But why group people into these arbitrary categories? We already have our own biases and prejudices that fairly or unfairly judgde people the moment we meet someone. Why add more?

</comment><comment><USER27> are under-engineers just another word for Morlock?

</comment><comment><USER28> As throwaway2016a alludes to, title should be changed to "The Types of Software Engineers"

<comment><USER29> I second that, the title is misleading.

</comment></comment><comment><USER30> What a load of __*. This post was under engineered.</comment>