The NES Homebrew Scene - <AUTHOR> https://tedium.co/2018/04/10/nes-homebrew-scene-history/ <comment><USER2> A while ago I wanted to try some NES ROM hacking but I couldn't find any tools for it that were really convenient - I ended up typing 6502 into some online assembler to copy the output into HxD and other processes like that. So, I wrote something more to my liking that was more like an IDE.

I couldn't think of a good name, but I was modding Kid Icarus at the time, and it's in .NET, so I just called it IcarusNet. It was primarily for ROM hacking, but as far as I know, it should be suitable for writing something from scratch in conjunction with tile editing programs and maybe other things you'd need.

Not nearly as advanced as NESMaker (very impressive from the looks of it) seems to be, but maybe a middle ground for someone who wants to write a game more to the metal but also wants something more graphical with the same "one click assemble/run" mindset.

Comes with instructions for a sample project to make an edit to a freely available ROM in less than 5 minutes.

[https://github.com/ldyeax/IcarusNet](https://github.com/ldyeax/IcarusNet)

Since then I've heard that license might have some problems for software, so I'll change it if the need arises.

<comment><USER3> As someone who simply wanted a NES assembler, I've found cc65 [0] to be great, for example wrapped as a docker image in [1].

[0]: [https://github.com/cc65/cc65](https://github.com/cc65/cc65) [1]: [https://hub.docker.com/r/philhsmith/cc65/](https://hub.docker.com/r/philhsmith/cc65/)

</comment></comment><comment><USER4> Nowdays, it's impossible to do modern programming without relying on millions of lines of other people's code. It's frameworks and libraries all the way down.

On the NES however, every line of code is yours. There's no such thing as libraries, operating systems, or frameworks. They don't exist. It's refreshing.

One of the big problems of modern programming is how easy it is to add complexity. As Charles Moore once said:

>"Simplify the problem you've got or rather don't complexify it. I've done it myself, it's fun to do. You have a boring problem and hiding behind it is a much more interesting problem. So you code the more interesting problem and the one you've got is a subset of it and it falls out trivial. But of course you wrote ten times as much code as you needed to solve the problem that you actually had."

It's easy to add complexity on modern systems, but it's hard on the NES. The programming too clunky; it's too much work to anything but simple things. For that reason, it's far more common to simplify the problem and reduce features rather than add them. It's a wonderful thing.

<comment><USER5> I agree with you in general, but I think it's worth pointing out that programming the NES's PPU is actually very similar to programming with a 2D graphics library today.

NES games never wrote their own scrolling logic, or there own graphics logic. They wrote tile values into a background table and then told the NES how much to scroll the playing field, and the NES rendered and scrolled everything automagically.

NES games never drew their own sprites. They just said "Hey NES, write the sprite at this x and y location" and the NES did it.

Audio worked the same way.

Honestly, coding for the NES would be similar to working with a simple 2D graphics library. So I don't know if it would be accurate to say "every line of code is yours" on the NES. It's more just that the "libraries" used were implemented in hardware instead of software.

<comment><USER6> > NES games never drew their own sprites. They just said "Hey NES, write the > sprite at this x and y location"

Yeah, but the hardware is so limited with weird corner cases that writing a bitblt might be easier. For example, some games have to reconfigure their sprites at the beginning of every horizontal scan line to overcome the hardware limitation on simultaneously displayed sprites (8).

<comment><USER7> Which games? To my knowledge this is at best _exceptionally rare_ because there are only ~114 CPU cycles in a scan line. The reports I've heard are that if you do any OAM changes mid-frame, you get something like four lines of glitches. More common is to just do a full OAM DMA during vertical blank, at least from the ROMs I've seen.

<comment><USER6> I can't find the original link where I read about this, it may have been more theory than about specific games. These links seem to be relevant: [http://wiki.nesdev.com/w/index.php/Sprite_overflow_games](http://wiki.nesdev.com/w/index.php/Sprite_overflow_games) [http://forums.nesdev.com/viewtopic.php?t=16299](http://forums.nesdev.com/viewtopic.php?t=16299)

</comment></comment></comment></comment></comment><comment><USER8> Somewhat related:

I'm always thoroughly impressed with how complicated and knowledgeable the people who work on these topics are.

I don't even use the product, but I love reading the Dolphin blog [0] because they write about fairly complicated topics in a way that's pretty approachable, and it always makes me appreciate how much I don't know about these topics.

While I feel like it would be difficult-but-not-impossible to be a contributor to say, the linux kernel (this is probably me grossly oversimplifying the process), I think I could spend the next year diving into graphics programming and basic hardware topics and still not contribute in any meaningful way to a project like Dolphin.

Articles like this and the Dolphin blog definitely make me realize how much I don't know.

0: [https://dolphin-emu.org/blog/](https://dolphin-emu.org/blog/)

<comment><USER9> The difference is domain knowledge, it takes time. You might feel that way about kernel because the high level concept of resource management is pervasive in programming. So you might feel you can easily dive into it, but it will take you equally long to say port the kernel to a new architecture/write drivers for a new platform.

</comment></comment><comment><USER10> I recently started work on an NES project, and wrote the first part of it up: [https://www.moria.us/blog/2018/03/nes- development](https://www.moria.us/blog/2018/03/nes-development)

It's a lot of work. I've managed to get data from Tiled into my ROM image, and I'm working on figuring out an architecture for scrolling and displaying a status bar.

Implementing scrolling and a status bar sounds simple but is actually a total pain. You only have two screens of tiles to work with, which can be arranged horizontally or vertically, and anything more complicated than scrolling in a single direction (including status bars) requires adjusting the scrolling registers between scanlines, and making sure that your status bar is in a different part of the tilemap (called "name tables") than your world.

A common way to accomplish this is to lay out your graphics memory so a certain address line turns on and off once every scanline, and then putting a chip (called a "mapper") on the game cartridge which wires that address line to a counter, which drives a CPU interrupt line. Simpler games like Super Mario Bros. don't need this extra chip, but you only get a single status bar + scrolling in one direction only. Games with scrolling in both directions often display graphical glitches on the edge of the screen, like Super Mario Bros. 3 (look at the right edge of the screen).

This is unnecessary on newer consoles, and less painful even on the Sega Master System.

<comment><USER11> But programming with restrictions like that will make you a better programmer. Nothing brings out creativity better than a limiting environment.

<comment><USER12> > Nothing brings out creativity better than a limiting environment.

This is so true as a life principle!

I once read a book about the development of laser guided bombs [1], and in it it talked about how Texas Instruments made the first viable prototype [2] on a shoe-string budget even though they were competing against a large government contractor with a huge budget.

Because of the budgetary constraints, TI couldn't afford multi-million dollar wind tunnels for testing, so they built scale models of the bomb and dropped them into swimming pools, performing mathematics transforms on the results using the drag coefficients of water vs air. They came up with a bunch of other innovations as well, such as the shuttlecock seeker head and made a superior product that vastly outperformed the high-budget contractor.

[1] [https://www.amazon.com/Weapons-Choice-Development- Precision-...](https://www.amazon.com/Weapons-Choice-Development-Precision- Munitions/dp/0817315322) [2] [https://en.wikipedia.org/wiki/Paveway](https://en.wikipedia.org/wiki/Paveway)

<comment><USER13> I'd find completely awesome if it weren't for tools aimed at killing people. I'm not sure I enjoy advances in that area.

<comment><USER14> It's just as easy to say it is a tool designed to save lives. Ultimately it's a tool, it's how you use it that's important.

<comment><USER15> Just as easy maybe, but fundamentally incorrect. Its mechanism of action is killing people -- choosing to believe the claim that killing those people will save lives alters neither the purpose nor MO of the tool.

<comment><USER16> The goal of the mechanism is to kill fewer people than other available methods.

Tools are, simply put, devices that increase a user's leverage.

We live in a world full of tools capable of leveraging against life. We also live in a world with individuals and groups who - for whatever reasons - choose to use that leverage.

We can't get rid of tools like this. They are very straightforward inventions, especially as technical knowledge increases. The simplest nuclear fission reactor is a bomb. That does not make nuclear fission fundamentally worthless, and whether you agree that knowledge about it is a "good" or not, that knowledge is not going to simply disappear.

> choosing to believe the claim that killing those people will save lives > alters neither the purpose nor MO of the tool.

How a tool is used does not change what it is or what it does. It does, however, define the purpose. If the purpose of killing someone is to prevent that individual from taking another life - or several other lives - then the purpose of the tool used is to save lives. The goal for perfecting a tool that is to be used for that purpose is to minimize the amount of lives taken, and damage done. That means a "better" bomb takes fewer lives, saving more.

If you choose to believe that an individual taking action against the life of another has not forfeit his/her own right to live, then you might consider this tool to have no legitimate purpose. Frankly, I disagree, and hope you will reconsider.

</comment></comment></comment></comment></comment></comment></comment><comment><USER17> My own personal story with the NES didn't last that long, because I was a kid without money, terrible at platformers, and my parents wouldn't buy me a lot of games. You can only play Super Mario Bros so many times before you're tired of not making it past the 2nd level. So I eventually switched to computers (Apple II vintage), and never looked back to consoles.

This past year, however, I decided to take up a little retro project based on the NES. Its best described as a "video game music player alarm clock", using some authentic hardware. It uses the CPU from an actual NES to faithfully synthesize music from actual NES games, while building the surrounding system out of modern components.

First prototype in action: [https://www.youtube.com/watch?v=izMFPKmD5ZU](https://www.youtube.com/watch?v=izMFPKmD5ZU)

Blog posts detailing the project: [http://hecgeek.blogspot.com/2018/02/nestronic-1.html](http://hecgeek.blogspot.com/2018/02/nestronic-1.html) [http://hecgeek.blogspot.com/2018/03/nestronic-2.html](http://hecgeek.blogspot.com/2018/03/nestronic-2.html)

</comment><comment><USER18> I was a bit turned off by the way assembly is described as "tricky" and "tedious"... All programming languages can be tricky and tedious. Assembly (specially on the 6502) is conceptually _very_ simple and, while it may not be trivial to translate higher-level concepts to its simplicity, as long as what you want can be readily expressed in it, it's trivially easy.

You will, of course, need to know what bits to set at what addresses to make the NES auxiliary chips work and do what you expect them to do and that may represent some work but that complexity is from the platform, not the language. You can have a library (or a set of macros) that deals with it written in any language you can compile for the platform.

<comment><USER19> >Assembly (specially on the 6502) is conceptually very simple and, while it may not be trivial to translate higher-level concepts to its simplicity, as long as what you want can be readily expressed in it, it's trivially easy.

That's precisely why I would qualify ASM as "tricky" and "tedious". You're bogged down in the tiny details that nowadays a compiler would probably solve as well (or even better) than you do. If you want to express "I want to make a function that adds two integers" then yeah it's trivial. If instead it's "I want to iterate through the leaves of a binary tree and compute its standard deviation" then have fun; good luck.

"This is a multiplication by 5, it's pretty expensive, probably better off shifting by two and adding once. Oh but then I need an intermediary register, do I have one available?"

"I guess I could inline this bit of assembly here instead of making a function call. Oh but wait now I need to re-do my register allocation to match."

"I need to reserve a bit more stack space to bank this register, let's modify my function's prelude and prologue to match."

Ain't got no time for that.

I think writing assembly is a skill any coder should have, especially if you usually deal with close-to-the-metal languages like C, C++ or Rust for instance. I think having a good grasp of assembly will make you a better coder, if only because it'll let you read the assembly output of your compiler to figure out what gets optimized and how (and also maybe what you could change to speed things up). There are also a few situations where writing assembly is the right solution.

That being said you'd have to pay me a lot to write a full application in assembly, regardless of how many layers of macros and pre-processing steps I'm allowed to use (unless gcc is allowable as a pre-processing step...)

<comment><USER20> You've captured exactly what it feels like -- this is the constant tension among asm developers. I love really high performance code, but I also like accomplishing more than one trivial thing per hour. It's hard to even remember the time before I knew C, when coding in assembly was all there was. When it was "normal." But I do remember the first time I wrote for the M68000 chipset and finding a multiply and divide instruction. "What? I don't have to write my own divide routine?" Tears of joy!

I will typically build something out in a high level language, profile to see where the time is going, and take a closer look at the algorithm first to see if it's sensible or some different approach should be taken. If it's an appropriate algo, then it's time to look at the implementation to see where we can shave cycles.

For the most part, even if you're talking to hardware that requires exceedingly precise timing (interface/bus protocols, certain chips), C will probably do the job. (It's no coincidence it was referred to as "universal assembly language".) Only where one is absolutely starved for resources (as the NES, and many 8-bit systems were/are) is ASM necessary.

"high level when you can, assembly when you must"

<comment><USER21> > "high level when you can, assembly when you must"

I was already following that on MS-DOS with its 640 KB, with all Basic, Pascal, C and C++ compilers from Borland and Clipper.

Although I did spent one year still focused on using TASM for everything and playing how much I could cram into a COM file.

</comment></comment></comment></comment><comment><USER22> The appeal of retro console/computer dev is essentially an exercise in constrained programming. Systems of these era had extremely constrained specifications, and making the most of them is an art in itself.

<comment><USER23> Sometimes I wish we had more constraints because I fear we've all forgotten how to write fast and lean code in favor of semi-fast but very bloated code.

I remember back when I first used Windows 95, opening programs was very snappy and I was impressed. I imagined Windows of the future being even snappier, but somehow the opposite has happened. Now every application takes a long time to boot up, check for updates, etc just because they use orders of magnitude more resources. I long for apps with instantaneous startup times, no loading screens, and small footprints

<comment><USER24> We haven't forgotten, we choose to put our time and energy into features rather than performance because that's what end users care more about (at least, it's what they vote with their feet for).

<comment><USER25> Some things have been lost, though, and I'm not even referring to the usual claim about constraints facilitating creativity. Games of the 8- and 16-bit console generations, as well as all arcade games up until that point, were written with CRT monitors in mind and often used 'racing the beam' techniques. This resulted in games with extremely low latency, something you never find in a game of any sort today.

The difference can be felt when comparing a game like Super Mario Bros running on an emulator with a typical LCD screen to running on original hardware with a CRT monitor. The emulated version is so sluggish it feels like you're playing underwater!

<comment><USER24> > Games of the 8- and 16-bit console generations, as well as all arcade games > up until that point, were written with CRT monitors in mind and often used > 'racing the beam' techniques. This resulted in games with extremely low > latency, something you never find in a game of any sort today.

The techniques haven't been lost - if anything there are probably more people who know how to do those things today than there were in the past.

Typical LCD monitors are undeniably worse in many ways than the CRTs they replaced, but again, most customers turned out to prefer the convenience of the LCD on the whole. Nowadays with the rise of 144Hz monitors those rare users who care a lot about gaming latency do have an option.

<comment><USER25> This article [1] has been posted on HN before but not everybody has read it so I thought I'd mention it here. If you scroll down there's a paragraph where Dan talks about 144 Hz gaming monitors. Shocker of Shockers (not surprising to me, actually) the Apple ][ has far lower latency than a modern PC, even with one of these monitors. When actually measured with a high speed camera, a CRT refreshing at 60 Hz still has lower average latency than a 144 Hz gaming LCD.

I would expect the NES to perform similarly to the Apple, since it has the same CPU (6502) and also uses a CRT display.

[1] [https://danluu.com/input-lag/](https://danluu.com/input-lag/)

</comment></comment></comment></comment></comment></comment><comment><USER26> I literally just wrote my first ROM that displays a green box on a black background working on my retropie machine last night.

The NES homebrew scene is pretty cool and the tools and languages have come a long way. One interesting avenue that might be work exploring is to use code generation and CSPs to generate and shrink code.

It's a lot of fun. It's why I like the PICO8. Constraints are interesting.

I just wish I still had my old BASIC code for games I made on my Amiga back in the day...

<comment><USER27> > _Constraints are interesting._

Constraints are also helpful when working towards a sense of mastery.

Look at a modern operating system (or console) for that matter. So much capability, so much complexity. You'll never learn it all, never be able to master it.

Heck, just truly understanding the machine code of a modern processor, and knowing everything it does and can do is quite a daunting task.

These older 8-bit systems (NES, C64, etc.) in contrast are much easier to completely understand. If you are willing to live within the limitations, you can create something really awesome, and be able to leverage the full extent of the system's capabilities.

That provides enormous satisfaction to some.

<comment><USER26> Just understanding the intel instruction set is basicallly impossible for a mere mortal. How thick is that reference manual now?

</comment></comment></comment><comment><USER28> Slightly related, but for GameBoy Homebrew, I've been working on a Javascript implementation of GBDK, a C library for GameBoy dev, and am using Emscripten to build a React debug tool allowing you to check out the state of the different graphics buffers and registers and print to the browser console. Bit early days but it's really helping speed up a game I'm working on.

[https://www.gbdkjs.com/](https://www.gbdkjs.com/)

Got an example project at

[https://www.gbdkjs.com/examples/shooter/web/](https://www.gbdkjs.com/examples/shooter/web/)

</comment><comment><USER29> It's also a matter of seeing what you can achieve on aging hardware. Can you do better than a team of professionals could in the 80s? Could you make something that pushes the console to its limits? Those types of questions are very inviting for both homebrew and ROM hack developers alike. Or perhaps before them, for those in the demoscene.

</comment><comment><USER30> Love the NES and love retro gaming, but the article completely contradicts itself. It's not just driven by nostalgia, but...

 _It says something about the legacy of the NES that most homebrewers continue to make games for the system for reasons other than nostalgia. It's mostly because they love the NES.

"I've always wanted to create NES games, ever since I was a kid. I remember sitting around drawing maps on paper, daydreaming of what my designs would be like to play," explained Nathan Tolbert, a relatively new figure to the NES homebrew scene who's already built a couple of titles for the annual NESDEV competition._

So...nostalgia.

 _One creator, Antione Fantys, wants to bring back some of the elements from retro games that don't seem to exist in the modern generation. Because the NES is the most iconic of these systems, he enjoys programming with it, citing the limitations of the system as a means to creating the most accurate representation of a retro game as possible._

More nostalgia.

The article pretty much proves these homebrews are the product of nostalgia. Not that there is anything wrong with that.

<comment><USER31> I used to do a lot of dev against the NES. 6502 for the NES was my first language, wrote my own assembler, built games, cartridges, etc. etc. Hell, I even have a PCB, minus the PRG and CHR ROMs, from an old Ice Hockey cartridge on my desk in front of me as I write this.

Never had an NES growing up, never played one until I got into NES dev. I was turned onto it by a classmate. We're both too young to have had the NES growing up, so we simply couldn't be a nostalgia play. There's a lot of folks who get into because it's surprisingly approachable, and scratches the itch of "make a game." TFA just had a bad quote...

<comment><USER30> Nostalgia doesn't have to be purely about one's childhood interests, although it frequently is. In your case, your interest still seems a product of nostalgia.

I don't really understand why people view interests based on nostalgia as a weakness. Being nostalgic doesn't have to mean that life isn't good or that things were better once upon a time.

Like what you like, do what you do, and enjoy what you enjoy without apology!

<comment><USER32> If we want to get pedantic; "Nostalgia: a wistful or excessively sentimental yearning for return to or of some past period or irrecoverable condition", where there seems to be an emphasis on 'wistful', 'excessive' and 'irrecoverable' parts of the definition, which in turn can infer a negative connotation about it all.

However, I agree with you, trying to capture past enjoyments and in cases like this, continue to make new memories and fun times, go for it I say!

</comment></comment></comment></comment><comment><USER33> [https://twitter.com/clearvus](https://twitter.com/clearvus) streams at 9pm EST on Mondays and Thursdays on Twitch, he has been working through making an NES game from scratch. He is off this week, but people who are interested in seeing this type of work should definitely check out his series, he deserves way more viewers! I've learned a lot from it.

[https://www.twitch.tv/clearvus](https://www.twitch.tv/clearvus)

recordings are posted on youtube: [https://www.youtube.com/watch?v=XwGj1ciSAtw](https://www.youtube.com/watch?v=XwGj1ciSAtw)

</comment><comment><USER34> Maan, I envy the homebrew scenes in the US so much. In Japan there are a few people who're doing homebrew, but because of the constant intimidation by Nintendo and marginalization by the society ("they're technically criminals.") they're not as open or organized as the US. I think we're killing our own talents.

</comment><comment><USER35> I haven't read the article yet but, judging by the title, I thinks this is relevant:

[https://www.kickstarter.com/projects/1316851183/nesmaker- mak...](https://www.kickstarter.com/projects/1316851183/nesmaker-make-nes- games-no-coding-required)

</comment><comment><USER36> The existence of phenomena like this and the demoscene made me realize that there are levels of programming skill that I will never be able to reach. Kind of disheartening, but still neat to watch other people do it.

<comment><USER37> I felt like that for a long time, and eventually it prompted me to build an Apple II emulator, and implement the compiler from the Coursera Compilers course.

Simple versions of these things are not as insurmountable as you think: just put your foot on the first step :-)

</comment></comment><comment><USER38> Very nice! One thing I always loved about Nintendo games is that you can usually just pop them in and start playing. No dragged out tutorials, setups, etc.

</comment><comment><USER39> The article namedrops Shovel Knight. I highly recommend the game. It's one of the best games I've ever played, even putting nostalgia aside.

</comment><comment><USER40> I've had a lot of fun trying homebrew games on my modded NES Classic. If you're a fan of the original Metroid, you should definitely try Rogue Dawn. It plays like a true sequel to NES metroid and it adds features from later Metroid games.

</comment><comment><USER41> Are there any uniquely impressive homebrew NES games worth playing?

</comment><comment><USER42> Can anyone recommend any particular enjoyable home brew games? I can see a few already listed in comments, but it would be nice to gather up a short list.

<comment><USER43> Blade Buster, fast paced modern time attack shmup; very pretty, don't forget to hit select to increase speed.

My top score in 2min is 500,000+.

<comment><USER42> That's great! (and difficult).... 100,000 is initial best.

Having recently discovered OpenEmu, I'm trying to track down good homebrew/freeware games.

</comment></comment></comment><comment><USER44> I got to the "and these are the reasons why" section before I really had to give up. The reasons why you remember the NES over every other video game system of the era was that Nintendo were monopolizing bastards at that time, and if you (as a developer) made games for the NES to be released in the USA, you were essentially forfeiting the right to release those games for other systems in the USA.

Honestly, I'm glad Nintendo got knocked down a peg on their last few systems. I'm just not happy about what they taught their competitors.

</comment><comment><USER45> The best games age well. People still play pac man. People still play Mario. That said, the aesthetic is absolutely nostalgic.

<comment><USER46> The aesthetic serves a purpose though. Hi-fi 3D game assets take orders of magnitude more work to create than low-fi 2D ones. I agree that nostalgia and familiarity are what allows gamers to accept NES-style graphics as a valid form of low-fi 2D art. The creator of Stardew Valley mentioned he wouldn't have been able to finish the game had he had to create 3D assets that looked as good to people as the current 2D art.

Having people accept NES-style pixel art is a boon to indie devs. Games like Super Meat Boy have higher res 2D graphics but is still a similar aesthetic. Low-fi poly is also a retro art style. However I've noticed that low-fi poly games are still rendered in high resolution and using more advanced lighting techniques so still requires more work than low-fi 2D.

</comment></comment><comment><USER47> My favorite genre of games are still platformers. 20xx (A Megaman X roguelike) is my favorite game I've played this year.

</comment><comment><USER48> Fascinating that these scenes are still thriving, just recently saw "8-Bit Guy"'s postmortem video on writing and shipping a game for C64.

</comment><comment><USER49> Of course it isn't just driven by nostalgia. It also requires a constant supply of coffee, for one thing.

</comment><comment><USER50> Ah nostalgia. Everyone here should look up the original meaning of nostalgia though to really understand this. It's so apt yet so crippling.</comment>


Show HN: Koyeb – Simple serverless processing workflows, on any cloud - <AUTHOR> https://www.koyeb.com/ <comment><AUTHOR> I'm Yann, one of the founders of Koyeb. Koyeb is a platform for developers and businesses to run serverless data processing apps in minutes.

We provide an easy to use platform to build production-grade workflows for all your data, including image, video, audio, or document processing.

To provide a little bit of context, we previously developed Scaleway ([https://scaleway.com/](https://scaleway.com/)), a European Cloud Service Provider, and started Koyeb initially around multi-cloud object storage ([https://news.ycombinator.com/item?id=21005524](https://news.ycombinator.com/item?id=21005524)) We are now going a step further: we are trying to also provide an easy way to process data and to orchestrate distributed processing from various sources.

Currently, we provide an S3 compliant API to push your data, you can implement processing workflows using ready-to-use integrations ([https://www.koyeb.com/catalog](https://www.koyeb.com/catalog)) and store results on the cloud storage provider of your choice (i.e. GCP, Azure Blob, AWS S3, Vultr, DigitalOcean, Wasabi, Scaleway, or even Minio servers).

We're working on adding support for Docker containers and custom functions to let our users combine catalog integrations with their own code in workflows. We will also add support for new data sources to send, ingest, and import data from different services.

We of course take care of all the infrastructure management and scaling of the platform.

The platform is in early access phase and I'd love to hear what you think, your impressions and feedback.

Thanks a lot!

<comment><USER2> I wasn't able to find a reference for the yaml schema faster than I can open the "add comment" page, so apologies if this is address by some docs somewhere:

Given that `steps:` is a list, isn't having `after: video-clipping` redundant, since it already comes after the video-clipping step?

<comment><USER3> The `after` attribute is present to let you implement your processing logic. The steps list contains the processing actions but it is not a sequential execution. The workflow can have multiple processing branch and perform a series of processing on the result of a specific step.

</comment></comment></comment><comment><USER4> YAML is the new XML. Monstrous configurations used by some magic runtime to glue pieces together.

Dependency injection frameworks were there already and decided against it at the end. I wish CI, data pipelines, config management, IaC tools and all other modern users of YAML-fits-everything approach learned from the past.

What we need is an ease of creating mini languages, but apparently it is not something industry/academia is aiming at, so this skill in not existent among common engineers, which often come up with ideas of new products.

<comment><USER5> >Dependency injection frameworks were there already and decided against it at the end.

How did dependency injection frameworks solve this problem then?

<comment><USER6> By declaring dependencies in code (usually with annotations or decorators)

<comment><USER5> What do you do when you want to choose the implementation for dependencies and optionally configure them at runtime?

I think that's why most of the popular dependency injection frameworks I see support configuration through XML or JSON.

<comment><USER7> Firstly, you barely ever do.

Secondly, unless you're selling compiled software for others to configure, there really is no difference between config in config files and config in code, except that the former does not let you build such meaningful and useful abstractions. You're presumably delivering it all into production via CI/CD anyway, so what difference editing a config file vs. some actual source code, besides the lack of real type safety and IDE assistance?

Pulumi have figured this out.

(And I suspect that if your end users need to configure your software's use of Spring's Dependency Injection in XML config then you are probably doing it wrong.)

<comment><USER8> What exactly has Pulumi figured out?

FWIU, their path is not the right one for DevOps.

Dhall and Cue have the correct starting philosophy, i.e Turing incomplete

</comment></comment></comment></comment></comment></comment><comment><USER9> How does this compare to minio ?

<comment><USER10> Aside from having an s3-compatible layer, that's where the similarities end. Minio is self-hosted, and doesn't provide the additional workflow processing, etc.

</comment></comment><comment><USER11> so node-red on the cloud? I've been waiting for something like this

<comment><AUTHOR> Kind of, completely managed with Docker containers support and multi-cloud storage :)</comment></comment>


Law to target aggressive fundraising by charities - <AUTHOR> http://www.bbc.co.uk/news/uk-33490964

<comment><USER2> The one time I dumped a £3 donation via SMS my number was basically call bombed afterwards. After the 15th call in under 24 hours, some at unsociable hours, I took the SIM out and threw it away and signed up for another giffgaff account (I have a few SIMs lying around). This was for one of those "international crisis" ones.

I tend to tell them all to bugger off now. If you're going to shit on me for being nice, I don't want to know.

My mother was a victim as well. Had someone call her all sorts of names for cancelling a direct debit that they'd "miscalculated" on purpose. I managed to get the caller's identity and company and sent a nasty email to their directors. Got a cheque back for the lot. Sod the charity.

<comment><USER3> yeah, it gets frustrating when you're bombarded with calls by them as well as those PPI/whatever robocalls.

I've cancelled my donations to several charities because of their harassment and their pushiness makes me wonder if they are actually using their money on the people they're claiming to help :/

<comment><USER4> In many cases, they aren't: [https://www.propublica.org/article/how-the-red- cross-raised-...](https://www.propublica.org/article/how-the-red-cross-raised- half-a-billion-dollars-for-haiti-and-built-6-homes)

<comment><USER2> This doesn't surprise me. Back in about 2002 the company i worked for built a shitty intranet for a major high street charity in the UK. Cost them £100,000...

<comment><USER3> I'm in the wrong job... :p

<comment><USER2> Everyone's in the right job. You just have to find customers who don't know how much everyone else charges :)

<comment><USER3> Haha, cynical but true. Reminds me of this: [http://www.news24.com/SouthAfrica/News/Free-State-govt-We- pa...](http://www.news24.com/SouthAfrica/News/Free-State-govt-We- paid-R40m-for-website-20130304) (R40m is ~$3,211,291)

<comment><USER2> Wow now that's special!

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER5> Shame, I'm just trying to have the red cross stop calling my house about blood donations.. ill give blood, just stop calling me.

</comment><comment><USER6> The fundamental problem here is that the charitable sector is packed full of moral hazard, broken incentivisation, inefficiency, lack of transparency, and ignorance of customer needs.

There are very few markets where I'm willing to go as far as using the word "broken". One is insurance (which is why I helped found Guevara) the other is charity.</comment>


Terry Pratchett's unfinished novels destroyed by steamroller - <AUTHOR> https://www.theguardian.com/books/2017/aug/30/terry-pratchett-unfinished-novels-destroyed-streamroller <comment><USER2> Good to see his wishes fulfilled. In the end, it was his choice and no-one else's.

In comparison, Douglas Adams' remaining incomplete notes were turned into a book after he died (The Salmon Of Doubt). As a fan, it was frustrating to read - you could see his moments of genius, but the gaps in the story and the unrefined parts of the text were a let down. I gladly bought the book, but after finishing it, I'm not sure it was a good choice to publish the unfinished writings.

If an author had a very-nearly complete book, I reckon it might be worthwhile to let someone else do the final tidying up and publish, but if Terry Pratchett's remaining works were lots of partially complete writings, I think it would be a poor reflection on the author to let someone else work on them.

<comment><USER3> On the other hand, if it weren't for the _Salmon of Doubt_ , I wouldn't have realized Adams' genius writings on other (non-fiction) topics besides THHGTTG and Dirk Gently. In fairness, I could have found that out in other ways if I'd have looked, but thanks to Salmon of Doubt I've come to appreciate the man's writings and personality in a much broader way.

So I say it was a good thing overall. Even though it was also sad because it was the unfinished writings of a dead author I love.

<comment><USER4> You might like [0]. It's a lecture he gave about his nonfiction book Last Chance to See.

[0] [https://m.youtube.com/watch?v=_ZG8HBuDjgc](https://m.youtube.com/watch?v=_ZG8HBuDjgc)

</comment></comment></comment><comment><USER5> I have mixed feelings about this. On the one hand this was his wish, but on the other hand I am sure that future generations of pratchettologists will mourn the loss.

<comment><USER6> It's interesting the parallels I see between this and right to die.

He didn't want to destroy his work whilst he could still work. But if he waits too long and misses the opportunity to take action whilst in good health he may lose control over it. Without the other limiting factors - this feels like a very easy call. I'm thrilled that he trusted people to see his wishes through - and as importantly that they've shown themselves to be deserving of said trust

<comment><USER7> I'm really annoyed that Pope John Paul's II wishes were not respected by his life long friend and right hand - Cardinal Dziwisz - Pope asked him to burn all his notes after his death, which the cardinal promised, and then released all the notes in a book after pope's death. I can "sort of" get his reasoning - that these notes were extremely valuable to the church and people in general - but a personal wish, especially such as this, should be respected in my opinion.

<comment><USER8> That's an interesting example of where personal wishes cross over the 'corporate' boundary. The person is subsumed by the job and unless they display huge self-restraint it will be difficult to separate personal notes from their corporate thinking.

Presidential diaries would be another example, or a CEO's Rolodex.

<comment><USER9> Here's the story: [https://www.nytimes.com/2014/02/05/world/europe/entrusted- to...](https://www.nytimes.com/2014/02/05/world/europe/entrusted-to-burn-john- paul-iis-notes-cardinal-publishes-them-instead.html). It isn't just The Pope's notes, but notes that predate his inauguration as Pope, and it isn't just about retention, but about publication.

Both retention and publication go against the wishes of the deceased, but I would see putting them in the Vatican Archives for a few researchers to peruse as a lesser transgression than putting them in a book for all to see.

I agree with you that there is probably a legal or moral requirement to retain the notes of Pope John Paul II, and that may override his personal wishes (though, again, in an archive, rather than in all good bookshops).

You are also correct that it may be difficult, or even impossible to separate JP2's notes from Karol Wojtyla's notes after 1978, but what about before? Are the notes of Bishop of Ombi or the Archbishop of Krakow also as important? How far back does the corporate entanglement go?

<comment><USER10> In addition to the moral questions, wouldn't this be a straightforward copyright violation?

<comment><USER11> My (somewhat oblique) question is why copyright would apply to a dead guy's works. ...It's not like respecting copyright is going to get him to write more.

<comment><USER10> The standard answer is that it incentivizes living authors to know that their heirs will be able to continue to benefit from their work.

In any case, the question of why it is that way doesn't much matter for this particular question. Regardless of why copyright applies to a dead guy's works, it does.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER12> This is dramatic and certainly renders the drive itself useless, but this is not how you securely destroy data. The bits on those platters are still largely recoverable. Do Terry a favor and run that drive over a degausser (or at least a really big magnet) before putting it in a museum. And let everyone know you did it. Remove the temptation for someone to steal it, attempt a forensic recovery, and try to sell unfinished Terry Pratchett novels that the author never intended to see the light of day.

<comment><USER13> That sounds like a lot of money to spend to get fragments of notes for an unfinished book that you can't even sell as "Terry Pratchett's Unfinished Book" because his estate would sue you. Not to mention the legal risk of stealing the hard drive fragments then publicly admitting that you did it when you publish your book.

<comment><USER12> Well when step one is "steal hard drive from museum", you can assume that the sale and (initial) distribution are unlikely to be conducted above-board. Depending on one's access to sophisticated forensic tools, this may still be a difficult undertaking to extract any profit from. But money isn't the only possible motive.

</comment></comment></comment><comment><USER14> Before anyone else gets enraged, this was Terry's will being honored. However, there's a debate to be had regarding whether it would have been worthy to release his final works even if incomplete and against his wishes.

<comment><USER15> meh, I dont care what "debate" there is about it. It was his dying wish. Not for filling that is disrespectful in the extreme.

He wants his legacy to be as static as possible and not pimped out and whore'd to the highest bidder. Fair enough I say.

<comment><USER16> Kafka also wanted everything burned. Imagine what a loss that would have been.

<comment><USER17> The world would have been fine. We've probably lost uncountable number of beautiful works to destruction and geniuses to crib death that would have otherwise completely changed the (cultural) state of the world, but we don't miss them because we never knew about them. The path history took is as valid as all other paths it could have taken.

<comment><USER18> Sure the world is fine. But that's a really low bar of argument. If someone was going to go back in time and destroy great works, would you not strongly prefer them be stopped? (All else equal and no secondary consequences from time travel or whatever.) Do they have literally no utility to the world in your mind?

<comment><USER17> No, because that's a paradoxical requirement. I can't care for things which I never knew (could have) existed, and interfering with reality so that they would exist would require me to care enough for them to exist so as to interfere in the first place.

Every action we make chooses one from (infinite) branches of reality. Favoring one over the other is meaningless. The result of things existing will always be other things not existing. The atoms that I'm using to exist could have been used to compose other entities instead. You're assuming a sort of situation where everything that could exist would exist. A Borgesian library where not only every possible work of Kafka is written, but in every possible variation. The act of writing by itself destroys every alternative work. Are we certain that the Kafka we have is the _best_ Kafka we could have?

One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a horrible vermin.

One morning, when Georg Samsa woke from troubled dreams, he found himself transformed in his bed into a disgusting bug.

One day, when Georg Valta woke from troubled dreams, he found himself awake in his bed transformed into a horrible vermin.

In the morning, when Georg Sanka woke from dreadful dreams, he found himself awake in bed in the body of a nightmarish vermin.

...

<comment><USER18> You can assign value to different world states. All of life is about trying to steer reality into branches we prefer! Every decision you will ever make is to increase the probability of world states you desire more. If you really didn't have any preferences, you wouldn't do anything at all.

And yes, I absolutely would prefer to live in the world state with optimal Kafka. You really wouldn't? If Kafka came up to and asked him to help you with his wording or something, you wouldn't assist him at all?

<comment><USER17> Yes, you can assign values to different world versions, but each version would have their own values. It's a meaningless (to others), subjective categorization. There is no solution to the trolley problem where both branches agree to let one branch die.

I didn't mean that I do not have any preferences, but that my preferences aren't special or better than others' preferences, in the same way that another version of the world (where Pratchett's wishes were overruled) is not a better world than the current one. I think I'm skirting close to Leibniz here. Yes, the world could be better, but it would not be our world, and it would only be better to some of us, and worse for others.

It's highly unlikely that your best Kafka is my best Kafka. As such there is no Best Kafka. The world in which I help Kafka finish The Castle is one in which (for want of a nail) a trillion years from now the Zormulons exterminate the Qdhraxians. I doubt the Qdhraxians are happy I at least got an ending to a story.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER19> I've found Ursla Vernon ([http://www.redwombatstudio.com/](http://www.redwombatstudio.com/)) to be a witty and lucid writer much in the spirit of Sir. Pratchett.

<comment><USER20> I throughly enjoyed the webcomic Digger [1]. I didn't realize the author had so many other works published on the internet and in print. Thank you for this pointer!

[1] [http://diggercomic.com/](http://diggercomic.com/)

</comment></comment><comment><USER21> Franz Kafka had similar wishes, that his friend was to destroy all his work. I can't say I'm "glad" per se that he didn't since such emotions are antithetical to Kafka, but I suppose it is for the best, culturally that his friend went ahead and published them ...

I suppose it's a little different given Pratchett's stuff here was "unfinished" tho Kafka could have argued all his stuff was unfinished too ...

<comment><USER22> I like to compare Kafka to James Joyce.

With Kafka, we got great novels out of the postumous publication. The publication of Joyce's letters to Nora Barnacle, however, paint him as someone who "[fastens] intently on peculiarities of sexual behaviour, some of which might be technically called perverse"[1]. No one has really benefited from their publication, except perhaps gossip columns.

I think I would stick to the author's desires. As great as Kafka's work is, I don't think we are "entitled" to them. But I can understand the opposing point too.

[1] Richard Ellmann, Selected Letters of James Joyce, Introduction, Faber & Faber, London, 1975

</comment></comment><comment><USER23> Would this qualify as an addition to this list?

[https://en.wikipedia.org/wiki/Lost_work](https://en.wikipedia.org/wiki/Lost_work)

<comment><USER24> I was amazed to not see the handle my scrollbar reduced to a tiny pixel. This list could be orders of magnitude larger.

<comment><USER25> It's on Wikipedia, so get editing! :)

</comment></comment></comment><comment><USER26> Well to be honest his more recent books started to be more dull in a way, although I respect deeply his work, especially with the earlier books, but maybe the alzheimer started to have an effect also on his writing. I've read most of his works until Snuff or so, but those 3 - 5 books before that just really didn't had the same magic anymore, so maybe it's for the best.

<comment><USER27> Agreed; 'Raising Steam' made me so sad I couldn't finish it (I thought it was a much better story when titled 'Making Money', and even better when it was called 'Going Postal').

<comment><USER26> Good to hear, Going Postal was still good in my books, sometime I've pondered should I make an attempt to read those I didn't feel the call to, now I can assure I have no need to read those.

<comment><USER27> Making Money is worth reading, still.

</comment></comment></comment></comment><comment><USER28> That's one way to clear your internet history after you die.

</comment><comment><USER29> GNU Terry Pratchett

<comment><USER30>
        curl -s -D - https://www.ceondo.com -o /dev/null | grep Terry

I think we are many fans having this running since 2015.

<comment><USER31> There's a list of sites 'broadcasting' over at [https://www.reddit.com/r/GNUTerryPratchett/](https://www.reddit.com/r/GNUTerryPratchett/)

</comment></comment></comment><comment><USER32> That's a very old looking HDD. It seems odd that his most recent work would be on that.

<comment><USER33> Could easily have been a drive that had been in his main work machine for quite some time. He no doubt had the good sense to have backups on other local drives and remote services[1], but this could still be his primary store. It isn't like holding the text of books in progress is a task that needs the speediest of SSDs so if that drive just kept working why would he replace it?

[1] presumably these have also been purged if his wishes were so written or interpreted, probably by more conventional means[2]

[2] the bulldozer being used for this one drive for ceremonial purposes

<comment><USER34> SATA drives replaced IDE drives (which this appears to be) more than ten years ago, somewhere around 2004-2006.

<comment><USER33> PATA drives were still common enough ten years ago, maybe not in brand new machines but certainly in active ones and readily available as replacements/upgrades. Using a machine from ~10 years ago for document authoring isn't unusual. A good keyboard screen is more important than the rest of the machine for that sort of job - you don't need something new, fancy, & powerful, and many authors _prefer_ an older machine because they can't run distractions well!

</comment></comment></comment></comment><comment><USER35> Terry Pratchett was one of my most favourite authors, it's rare to see someone keep building up a universe that both my elderly teachers and me as a student could appreciate during high school. Moreover, I wish there were more movie adaptations than what they have now.

Also seems so long ago, I used to play Discworld MUD and relive the stories on there as well. And now Terry has passed away (RIP, hope you're having fun with Death) and Discworld is off on its own, floating on a turtle somewhere. I have to be honest, I would read some Discworld fan fiction just to see those familiar characters again.

</comment><comment><USER36> I respect this a lot - if he hadn't, relatives or others would eventually publish his unfinished works (either in an unfinished state or completed by another author). The money is too alluring.

</comment><comment><USER37> I'd like to imagine that the steamroller was driven by Nomes.

</comment><comment><USER38> To save you the click - That's what he wanted.

</comment><comment><USER39> i'm sure it could still be recovered through forensic means from the pieces if one really wanted to

</comment><comment><USER40> pRest pIn Peace, pTerry.

</comment><comment><USER41> My Will specifies that my warp drive design be destroyed. Earthlings are not ready for it yet.

<comment><USER41> That does it. I'm not sharing my Hodge Conjecture solution, either!

</comment></comment><comment><USER42> Damn those Hyperspace bypasses!

<comment><USER43> You've got the wrong author. You are thinking of Douglas Adams.

<comment><USER44> Eh, same vein of humor. I'll allow it.

</comment></comment></comment><comment><USER45> !!!!!

<comment><USER45> sigh...

[http://wiki.lspace.org/mediawiki/index.php/Multiple_exclamat...](http://wiki.lspace.org/mediawiki/index.php/Multiple_exclamation_marks)

</comment></comment><comment><USER46> If anyone believes that the REAL HD was destroyed then he/she doesn't know how business work.

PS: I would have destroyed the shit out of that HD.

<comment><USER47> What makes one HD more real than any other? Terry was fairly tech savvy. Would he not have kept backups?</comment></comment>


A Treatment for Heroin Addiction - <AUTHOR> http://projects.huffingtonpost.com/dying-to-be-free-heroin-treatment <comment><AUTHOR> A couple more articles about suboxone. First, a doctor speaking about making medical therapy more available:

[http://www.hazeldenbettyford.org/articles/16jan/greenagel- su...](http://www.hazeldenbettyford.org/articles/16jan/greenagel-suboxone)

A writer who became addicted to suboxone without having used other opiods:

[https://www.thefix.com/content/suboxone-addict-you-never- kne...](https://www.thefix.com/content/suboxone-addict-you-never-knew-existed)

And an article that discusses prescriptions being diverted to street use:

[http://www.villagevoice.com/news/is-suboxone-a-wonder- drug-t...](http://www.villagevoice.com/news/is-suboxone-a-wonder-drug-that- helps-heroin-addicts-get-clean-or-just-another-way-to-stay-high-6439885)</comment>


Could You Raise Your Rates? - <AUTHOR> http://www.lifehack.org/articles/management/could-you-raise-your-rates.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+LifeHack+%28lifehack.org%29 <comment><AUTHOR> I don't necessarily agree with the "package" deal but the first point is extremely valid.</comment>


Keydown event doesn't fire on iOS when arrow keys pressed on external keyboard - <AUTHOR> https://bugs.webkit.org/show_bug.cgi?id=149054 <comment><USER2> Is it possible this was done for a reason? Users on the linked page report that it's not just keydown but also keypress and keyup. If the first two were disabled but keyup still worked that decision may have some (limited) justification because keyup is the only one that necessarily fires once per physical input by the user.

I don't want to give Apple more credit than they deserve but it is equally foolish to assume that everything that is not how I expect it to be is a mistake. Then again, maybe they just don't want people using iOS when they "should" be using OSX, which is the kind of judgemental decision-making for which Apple is known. From the linked bug report:

> Would be a huge benefit to the iPad community if arrow key events fired. > That explains the lack of support by sites like Cloud9, plnkr, etc... I do a > lot of web development in those IDEs to teach people and to brainstorm when > I'm not at my computer so it would be awesome if the team got this in and it > made its way to IOS.

<comment><USER3> > it is equally foolish to assume that everything that is not how I expect it > to be is a mistake

This humble attitude saves me a lot of time on a daily basis.

<comment> yjftsjthsd-h It also saves face; it is _quite_ embarrassing to realize you've been complaining about something and suddenly discover that it works fine and _you_ are wrong.

</comment></comment></comment><comment><USER4> Finally! I _knew_ that the whole modal editing thing and the `hjkl` commands would become useful some day!

</comment><comment><USER5> Definitely remember some issue where I wasn't getting events for one of the arrow keys (can't recall which off-hand) in my iOS app. Also, there's no public UIKit API for getting key up events.

</comment><comment> time-domain0 Maybe this is related: I tried using Codeanywhere on iOS with a first gen Magic Keyboard. Most modifier keys and arrows weren't working properly, which made Vim, tmux and readline a PITA.

Anyone else have problems also with Ssh terminals or iVim on iOS?

<comment><USER6> That's a common complaint with AWS Cloud9. The editor and everything about the technology supports an iPad just fine, but the keyboard arrow keys don't work so it's basically pointless.

</comment></comment><comment><USER7> It's not a bug, is a feature...

</comment><comment><USER8> Well, Steve Jobs did try to discourage us from using the arrow keys back in 1984...

</comment><comment><USER9> I hate iOS: [https://developer.mozilla.org/en- US/docs/Web/Events/click#Sa...](https://developer.mozilla.org/en- US/docs/Web/Events/click#Safari_Mobile)

</comment><comment><USER10> I know there's some kind of joke around here about how the real error is using the iPad like a computer...

<comment><USER11> Whats a computer?</comment></comment>


KDB bindings for Lua - <AUTHOR> https://github.com/geocar/qlua <comment> ah- _You can use this to link with nginx+lua /openresty/whatever to get a Q-powered web service that can do more than a few thousand connections per second._

Cool stuff! I can see a single machine running this being loads faster than a whole elaborate cluster of php/perl/python/node servers even with some caching.

Could this also work with just the integrated web server in q? I never tried it, but I wonder why lua is involved? Is it supposed to run parts of the logic in lua? Or does just using nginx as a reverse proxy not work well?

<comment><USER2> Using kdb as an nginx backend isn't going to work well at high loads because you'll run out of TCP port numbers to use for connections. You might try keepalive, but it still requires clients to behave.

It's in lua because nginx doesn't have a module system: getting a lua module available on all our edge nodes caused less friction than a bunch of nginx patches and an nginx deploy.

</comment></comment><comment><USER3> My kdb is behind haproxy; just using kdb's built-in HTTP/1.0 webserver for now. I am still experimenting with stored procedures and haproxy's regexp filtering/rewriting.

Has anyone ever compared the performance of kdb+ on Linux versus BSD? It works under compatibility mode under the latter but I have never benchmarked it against some other OS. Apparently there is Kx customer demand for a Solaris version but none for BSD?</comment>


Smart vibrator syncs with audio fantasies - <AUTHOR> http://www.wired.co.uk/news/archive/2013-07/30/vibease

<comment><USER2> How about an "open" version of the protocol, that can be switched to allow people on the same wifi network find the device and activate it? (With a user- limited scale, of course.) Then you'll have people playing teledildonic games in public spaces with these. How about protocols to enforce consequences of wagers? Eventually, you'll have people saying that they have one, but faking it.

<comment><USER3> There are 5,590 results in Google for "teledildonic". Wat

</comment></comment><comment><USER4> I was hoping the term 'faptics' would catch on for this sort of thing.

</comment><comment><USER5> The sexism on HN is obvious. It would be impossible for a male sex toy to end up on the HN front page. Prove me wrong, I dare you.

<comment><USER6> Is it sexism when stories that are more interesting (titillating?) to heterosexual males get more votes on a site where most users are heterosexual males, or is it just math? All stories that get up-voted enough to be on the frontpage get there because they appeal to a majority of users; is it sexism that stories that appeal more to women don't get as many votes on HN, or just simple math again?

There's a lot of sexism out there, including on HN. But I'm not sure this is a good example of it.</comment></comment>


Dilbert's Scott Adams on Betting on the Bad Guys  - <AUTHOR> http://online.wsj.com/article/SB10001424052748704025304575285000265955016.html

<comment><USER2> Warren Buffett has said many times that being so large (and eg able to buy billion dolar companies) actually makes it more difficult for them to make money - because there aren't many companies large enough to make a difference, _and everybody already knows about those companies_.

He's also been quoted as saying that if he only had a million dollars, he could double it in a year. I haven't seen him explain this specifically, but the idea seems to be to research companies that are not well known, because that's where you're more likely to find stocks that whose price are mis- valued. And this is the work side: you have to research all those stocks (not just the top 300 or so that have analysts following them), and you have to be sure (within a margin of safety) of your evaluation of their value (especially in terms of enduring competitive advantage). If they're selling cheap, with respect to your evaluation, you buy. It takes a lot of work, and self- confidence.

<comment><USER3> "you have to research all those stocks"

That sounds like actual work. I thought the idea of investing is to let the money work for you.

<comment><USER4> A fool and his money are easily parted. If you do no work to find out if something is a good investment, you are that fool.

</comment></comment></comment><comment><USER5> Funny how everyone talks about BP but doesn't mention CAM, RIG, ATPG, and others. If you're going to invest, find the companies that make better drilling equipment because their services are going to be needed for the relief wells and fresh mandates for higher quality drilling.

Funnier still is how we think BP is the "bad guy" when the U.S. hasn't developed a transportation policy that doesn't focus on driving your SUV 30 miles to a grocery store. It's hypocrisy at its finest. We've had a blowup in coal, oil, and shale in the past 2 months but we still need cheap energy.

<comment><USER6> IE, Don't join the gold rush, but sell picks and shovels to those who do.

</comment></comment><comment><USER7> Why not become a contract killer? For the price of a handgun, you can make over 50 grand a day! Why bet on bad guys when you can _be_ the bad guy?

Interestingly, this half-sarcastic rant has brought up a question in my mind. Why not charge betters on crime for the crime? Why not charge shareholders for the crimes corporations commit?

<comment> daniel-cussen Because of limited liability?

<comment><USER7> If I paid someone to drain a bunch of oil into the ocean, I'd be in trouble. But if I "invest in" BP and it does the same, I'm OK. Why?

<comment><USER8> The difference isn't anything to do with you paying (or owning stock in them) but rather intention. If you were paying someone to refill boats with oil, and they _unintentionally_ spilled a bunch (just as BP did), you'd be liable in the same way as BP is now.

Note that BP _is_ in trouble... There will be lawsuits all over the place, and not just from shareholders.

<comment><USER7> My question is, why not go after the shareholders in addition to BP?

<comment><USER9> simply enough: it's. not. their. fault.

<comment><USER7> How many shares do you have to sell before it's not $crime_for_hire?

<comment><USER10> Mu. Buying a hitman lunch does not make you an accessory to murder.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> Under "invest in companies you love", he seems to define such companies as those with good sales pitches. But what about the more sensible definition, companies with passionate users? Anyone who invested in the obvious ones (Amazon, Google, Netflix, and Apple, for example) would have made a killing. On the other hand, take Comcast, the Consumerist's "Worst Company in America" for 2009. Comcast's stock has been stagnant at best.

<comment><USER12> Exactly. Teehan+Lax tested that theory, and while I'd probably disagree with some of their choices, the results are pretty impressive.

<http://teehanlax.com/uxfund/>

</comment></comment><comment><USER13> True. I realized this a while ago and my life is a lot less confusing. I've dabbled in media long enough to know it's more magic than reality. When you need to find, shoot, and edit a piece in less than 24 hours on a limited budget the truth stretches as much as your dollars do (this varies on whether your producer is Lucifer himself or not). Long story short I'm not involved in media anymore, you should spend your time on things you hate loving versus things you love hating, and read HN.

</comment><comment><USER14> If you really want to bet on the bad guys, invest in the companies behind BP and this oil spill. Such as Halliburton. Some people are speculating that BP will fold from this disaster due to the enormous projected cost of cleanup (10-30 times the cost of Valdez).

</comment><comment><USER15> Funniest line:

"CEOs are highly skilled in a special form of lying called leadership"

</comment><comment><USER16> wrong + wrong + irony != right

</comment><comment><USER17> Consider the relativity of hatred. He gives as one of his examples, Apple Inc., which he claims to hate, and therefore uses that to "prove" his point (buy stock in companies you hate). By his reasoning I should have avoided Apple's stock because I like Apple.

Really though, this is a fantastic article, and I think the best bit of advice is at the end:

      Again, I remind you to ignore me.

<comment><USER18> I think it's simpler than that: just go for the companies that you (and other people) develop strong feelings about.</comment></comment>


Show HN: Did I Ship Today? - <AUTHOR> https://didishiptoday.com/ <comment><AUTHOR> Hey HN, for some reason I decided last night I needed to launch and build a project. I wrote a log as I was doing it, took me a few hours I started going further down the rabbit hole till I just decided to up and launch it, you can read what I was doing as it was going along here, [https://medium.com/@nadermx/did-i-ship- today-b6696ed04abe](https://medium.com/@nadermx/did-i-ship-today-b6696ed04abe)</comment>


LinkedIn Group for 2013 Startup School Attendees to Keep In Touch - <AUTHOR> http://www.linkedin.com/groups?home&gid=6530249

<comment><AUTHOR> LinkedIn Group for people who wants to keep in touch and maintain their professional relationships on LinkedIn. Personally, I want to add a lot people as LinkedIn connections.</comment>


Ask HN: Anyone using Paddle in Germany? How do you deal with taxes? - <AUTHOR> I recently started a saas business in Germany and decided to use Paddle.com as a payment processor.
Then my tax consultant asked to provide all invoices sent to clients as PDF files.
Paddle doesn't support pdf export out of the box and their advice was to go through invoices and save as pdf manually.
Another issue was that Paddle doesn't provide country of transaction when exporting balance. Tax consultant needed countries as well.
So anyone using Paddle in Germany having the same issues? How does it work for you? Should I switch Paddle or tax consultant? <comment><USER2> You can keep both. You probably have not told your tax consultant that Paddle acts as the merchant of record. That means you don't actually have any relationship to your customers when it comes to taxes. That's Paddle's role. Instead, your single "customer" is Paddle, whom you will probably get a single transaction from per month.</comment>


The Science of Asking What People Want - <AUTHOR> https://blogs.scientificamerican.com/observations/the-science-of-asking-what-people-want/ <comment><USER2> Interesting, the author writes:

 _"Over the past few years, I've conducted about 300 market research surveys while working in a creative advertising agency. As I was reading about Charles Booth it occurred to me that there are two types of survey questions: those that extract and those that evoke. Questions that extract involve collecting information; this is Booth on the street asking people about their income. Questions that evoke involve eliciting a reaction. Think about a focus group moderator versus a standup comedian. Both care about what you think. But the stand-up comedian goes one step further to provoke a response."_

I can see this argument, though I wish the author offered more empirical evidence on how successful it is in metrics like economics.

Some could argue that survey questions deliberately evoking and eliciting a reaction could be leading the respondent.</comment>


How Vipassana meditation breathes life into Anirban Lahiri's golf game - <AUTHOR> http://www.espn.com/golf/story/_/id/20284690/how-ancient-practice-vipassana-meditation-breathes-life-indian-pga-tour-pro-anirban-lahiri-golf-game <comment><USER2> Vipasanna is really a mind opener for any individual. I recommend every individual to experiment this methodology. It is just a mental exercise and practice makes it perfect. I regularly practice it and it have made a really a big meaningful impact in my life and work.</comment>


People with broad smiles seen as warmer but less competent - <AUTHOR> http://www.dailymail.co.uk/sciencetech/article-4153674/People-broad-smiles-seen-warmer-competent.html <comment><USER2> Ohh my god!! I am less competent then!!

<comment><USER3> People with broad smiles are only seen as less competent, but it doesn't change their actual competence. Reading the Daily Mail however...

(I know, correlation does not imply causation. Also, it's "less competent than" \- seems your handle is appropriate :P)</comment></comment>


Ask HN: Is the Bastion Host Security Pattern Outdated? - <AUTHOR>

Following on to this article:
http://programming.oreilly.com/2014/01/is-the-jump-box-obsolete.html?cmp=tw-prog-na-article-pr_is_the_jump_box_obsolete
The article proposes that the Jump Box / Bastion Host pattern is obsolete for many cloud deployments.  I've been using a 'bastion host' pattern to access Amazon VPCs - and while the security / IP infrastructure is somewhat simpler to maintain, there really is no way to audit user access; the AWS EC2 private key infrastructure and bastion host pattern pretty much ensure every user runs as 'root'.
Are there better alternative for AWS (or other IaaS services in general) than the bastion host pattern? <comment><USER2> Well, that article was written by the head of this company [https://www.jumpcloud.com/about/](https://www.jumpcloud.com/about/) that sells an SaaS solution that builds on Chef and Puppet. More generally speaking, each DevOps framework has its own integrated solution for user account provisioning / security. Also see: 13 Practical and Tactical Cloud Security Controls in EC2 [http://www.tuicool.com/articles/NbIz6z](http://www.tuicool.com/articles/NbIz6z)</comment>


Neural Networks, Manifolds, and Topology - <AUTHOR> http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ <comment><USER2> This is a very interesting perspective on neural networks, is it novel? I've never seen a geometric interpretation of NNs.

In particular this:

> (Apparently determining if knots are trivial is NP. This doesn't bode well > for neural networks.)

Is there any theoretical research on obstructions to NN learning? Not that it would change much the practice (for instance, MLE learning of gaussian mixtures is NP-hard, but everybody does it anyway), but it could shed some light on the geometry of the "hard" instances.

EDIT: for example, to get a sense of how ill-conditioned is deep NN learning, a recent paper [1] shows that if we feed to an object classifying NN an image with crafted but unnoticeable perturbations the predictions change completely.

[1] [http://arxiv.org/abs/1312.6199](http://arxiv.org/abs/1312.6199)

<comment><USER3> I think the author has confused NP with NP-hard (or NP-complete). Indeed, it is extremely unlikely that the unknottedness problem is NP-complete, as it is already in NP ∩ co-NP (assuming the generalized Riemann hypothesis). Furthermore, we know that the problem of computing many knot invariants is "only" BQP-complete, so it's possible that unknottedness is in BQP (i.e., efficiently solvable by quantum computers), though I don't know the expert opinion on this.

<comment><USER2> Indeed I read that as NP-complete. That's _very_ interesting, do you have any references on it? Even if knots are ruled out, there might be hard topological decision problems that can be reduced to linear separability under homeomorphism, maybe in higher dimension.

</comment></comment></comment><comment><USER4> Bloody brilliant. I have longed for a ML methodology that breaks out of the float vector representation because I don't think it describes naturally occurring problems that well, most ontologies are semi-structured. However, this really gets to the nub of the problem, real, but solvable, problems are manifold complexes. I don't think anyone has really built a learner for complexes though, persistent homology is a stab in the right direction but its not there yet IMHO.

</comment><comment><USER5> I don't like his proof that a 1-2 hidden unit system can't handle the red circle in the blue torus. Feels too handwavy.

My proof: Suppose the matrix W is singular. Then there exists a nonzero vector v for which W v = 0. Find a point x in the red circle, then find a different point y=x+av (for some scalar a) in the blue torus. Wy+b=W(x+av)+b=Wx+aWv+b=Wx+a0+b=wX+b. Fin.

</comment><comment><USER6> Wouldn't this only apply to neural networks as used for classification? I mean the general paradigm of deforming curves until they're separated by a hyperplane seems pretty obvious now that I see it in front of me, but what about neural networks used to approximate continuous functions?

<comment><USER7> I'll take a stab at this (I'm a decade out from my last machine learning class, so no guarantees on correctness): The only reason it's fitting a hyperplane is because one class is being mapped to the continuous value -1.0 and the other class is being mapped to the continuous value 1.0 and there's a thresholding step (the hyperplane perpendicular to the line onto which the continuous values are being projected) at the end to determine the class. If you're doing regression instead of classification, your training data will be fed in with more output values than just 1.0 and -1.0 and you'll omit the thresholding at the end, but otherwise the behavior and intuition should be the same as in the article.

</comment></comment><comment><USER8> The article is exceptionally clear and well presented. We need more of this level of writing about machine learning.

For me, it is most important to note what the article touches only at the end. If you have good features and representation, a linear model may be enough. Real world data and tasks usually don't require knot untying. In my opinion, the model (e.g., neural net) should be kept simple, and the challenge is in finding the right features and representation.

Coates & Ng (2012) wrote:

Many algorithms are available to learn deep hierarchies of features from unlabeled data, especially images. In many cases, these algorithms involve multi-layered networks of features (e.g., neural net- works) that are sometimes tricky to train and tune and are difficult to scale up to many machines effectively. Recently, it has been found that K-means clustering can be used as a fast alternative training method. The main advantage of this approach is that it is very fast and easily implemented at large scale. On the other hand, employing this method in practice is not completely trivial: K-means has several limitations, and care must be taken to combine the right ingredients to get the system to work well.

[http://www.stanford.edu/~acoates/papers/coatesng_nntot2012.p...](http://www.stanford.edu/~acoates/papers/coatesng_nntot2012.pdf)

</comment><comment><USER9> This is very interesting. I wonder if anyone knows what can be gained in terms of separating out different topologies by allowing for a restricted family of conformal transformation on top of the tanh functions, of the sort that allows taking the insides out and so forth. Comments?

<comment><USER2> Why conformal? That seems a little too rigid.

<comment><USER9> To get the effect I was referring to, yes it is too strict a class. I wasn't thinking very hard about it, but my idea was that conformal transformation have nice compositional properties, which could be exploited in developing the method and its analysis.

</comment></comment></comment><comment><USER10> A bit of an aside: Are there any methods for compressing/collapsing/simplifying a neural network.

What I mean is imagine you've built and trained a neural network, as per the article, it is hard to ascertain exactly what it is doing. I was wondering whether there is work in this area, and it occurred to me a possible first step would be to collapse the neural network to a simpler but functionally equivalent structure.

I imagine this is far more difficult than it sounds, but I just wondered.

<comment><USER11> The Universal Approximation Theorem[1] asserts that you only ever need one hidden layer, which at least asserts that "an (approximate) simplification exists".

But I can't say off the top of my head how you'd collapse an ANN just two hidden layers into 1. It's not obvious how sigmoid functions compose, but I suppose I should give it more thought...

\</comment></comment><comment>---- 1: [http://en.wikipedia.org/wiki/Universal_approximation_theorem](http://en.wikipedia.org/wiki/Universal_approximation_theorem)

<comment><USER12> Although one hidden layer suffices theoretically, most of the cutting-edge work involves multilayer networks. Perhaps paradoxically, those become more "auditable" from what I've heard. (I've spoken to people with real-world experience of applying ANNs to unsolved problems, but I am not one.) What I've heard is that, with one layer, you get too much overloading (i.e. different stuff ending up at the same hidden node) to understand what it is doing, but that deeper networks are more legible (with appropriate visualization tools).

UAT says that a solution exists, but it doesn't put a limit on the number of nodes required, so it would have you doing an optimization in a space that is not just large, but of arbitrary finite dimension. It can be pretty nonconstructive (in the sense of proving "there exists" without showing how to find something) insofar as it's often non-trivial to get convergence to a working solution in reasonable time.

As for how sigmoids compose, imagine how bell-shaped curves would compose, just as you can make a painting out of bell-shaped "points" if allowed arbitrary precision/steepness. Now, the difference of two sigmoids can be bell-shaped, e.g. [http://www.wolframalpha.com/input/?i=plot+y+%3D+1%2F%281%2Be...](http://www.wolframalpha.com/input/?i=plot+y+%3D+1%2F%281%2Be%5E-x%29+-+1%2F%281%2Be%5E-%28x-1%29%29+from+x+%3D+-5.5+to+6.5) . I don't know how much this means in practice, but it establishes the possibility.

<comment><USER13> With respect to the deep learning networks ( as well as more traditional with just a weight matrix and bias), we can look to matrices for this. I will offer a compressed representation, not so much a way of pruning, but I explain why below.

Each single layer neural network is made up of 3 matrices, a weight matrix (connections), visible bias, and a hidden bias.

In theory, this can be represented as a flattened array.

This is what I do in deeplearning4j[1] for optimization (note: I'm the author)

The problem with pruning neural nets so to speak is this isn't really a search problem we're solving like A*. Both are graphs in a sense, but each neuron in a neural net when we have backpropagation gets updated to correct for error that it caused, rather than pruning like in Alpha Beta Pruning for Game playing AI.

I will offer one last thing and say that the way neural nets learn (especially if you stream data in to it for training rather than training all at once via online/mini batch learning) each neuron also tends to learn different components of an overall solution and some will activate more on certain feature vectors when you train them on an overall data set.

A solution to this is to set the neurons relative to the input size[2].

[1] [https://deeplearning4j.org/](https://deeplearning4j.org/)

[2] [http://www.cs.toronto.edu/~hinton/absps/guideTR.pdf](http://www.cs.toronto.edu/~hinton/absps/guideTR.pdf)

</comment></comment></comment><comment><USER14> It is always good to reference SVMs ([http://en.wikipedia.org/wiki/Support_vector_machine](http://en.wikipedia.org/wiki/Support_vector_machine)) when talking about neural networks.

The advantage of SVMs is that after re-mapping the data so it becomes separable, they also guarantee to separate it as clear as possible.

Not sure if they can solve all the type of problems as well but in some cases it is considered a better, more analytic approach.

<comment><USER15> "after re-mapping the data so it becomes separable" This is sort of the hard part.

<comment><USER14> Not saying it isn't that is the main bulk of the algorithm, just saying it there is an additional benefit of SVMs in the end.

<comment><USER16> SVM is great, it's often stacked on top of neural networks to do classification.

But it can't really deal with extracting features, especially from things like images and sounds. If you fed images ( pixel by pixel ) straight to the SVM and tell it to classify them, it would do extremely poorly - especially with big images.

A better thing is to have a deep ( layered ) network that will get pixel data from the input and pass it through layers that will then extract some useful features from it - for example, learn to recognize lines.

Then, after this information passes through the whole network, you are finished with smaller amount of features that are more useful for classification. Then you an send this information to SVM and it will deal with it a lot easier than if you tried to feed it straight pixels.

The thing is, we don't really have problems with classification algorithms - SVM does extremely well. What is the biggest hurdle is simply extracting features that then can be fed to well-working classification algorithms.

</comment></comment></comment></comment><comment><USER17> Cascade correlation networks are known to be good at solving the spirals problem. They are also very fast to train.

</comment><comment><USER18> > One could learn the vector field at fixed points (just take some fixed > points from the training set to use as anchors) and interpolate in some > manner.

One such interpolation that has nice properties (smooth everywhere, invertible, can be constructed for any set of pointwise correspondences) are diffeomorphisms.

Unfortunately, the mathematical objects are infinite-dimensional, which makes computation with them somewhat involved, even if they are derived from a small-dimensional set of point correspondences. See, e.g., <[http://www.cs.unc.edu/Research/MIDAG/pubs/papers/Joshi_TIP_2...](http://www.cs.unc.edu/Research/MIDAG/pubs/papers/Joshi_TIP_2000.pdf>) for one method of constructing them.

</comment><comment><USER19> Off topic: I wish, there would be a practical way to program for neuromorphic chips, which are inherently analog. Anybody have some idea or links regarding this field?

</comment><comment><USER20> So happy to see a friend on the front page! Go Colah!

<comment><USER21> w00t!

</comment></comment><comment><USER22> The typeface and layout are beautiful! As is the work itself. I love it!</comment>