Your Version Control and Build Systems Don't Scale - <AUTHOR> http://chadaustin.me/2010/03/your-version-control-and-build-systems-dont-scale-introducing-ibb/ <comment><USER2> A friend of mine expressed the problem described here in a more formal manner in this paper: <http://gittup.org/tup/build_system_rules_and_algorithms.pdf>

He ended up creating his own build system, "tup" < <http://gittup.org/tup/> >, based off of it. It also has the property desired in this article that, "No-op builds should be O(1) and instantaneous, and most other builds should be O(WhateverChanged)".

<comment><USER3> tup looks very cool, but I don't see how to fetch the source.

<comment><USER3> this seems to work: git clone git://gittup.org/tup.git

</comment></comment></comment><comment><USER4> I strongly disagree with the assertion that C compilation is I/O bound. If you have optimizations turned off, this is probably true, but if you are using a decent optimizing compiler, this is not the case.

We ran some tests where I work on a pretty good sized code-base and found that we were CPU bound, even on an 8-core system.

[edit] There is an exception, if you are using networked file systems of some type (especially clearcase dynamic views) then you are almost certainly I/O bound.

<comment><USER5> If C compilation were I/O bound then ccache would not speed things up. It does. I've used it on my NFS mounted filesystems and it speeds things up there too.

There is clearly a point where if your filesystem is slow enough that the process does become I/O bound. I have no experience with ClearCase but from what I've heard it's molasses slow and might be to that point.

<comment><USER6> Years ago I implemented a network file cache using a file-modification-count returned in every open handle. At that point it became an order of magnitude faster to build over the network than local, because the local file system had no such cache. Anyway this race between speed, space and network performance has been around a while.

</comment></comment></comment><comment><USER7> This is the one thing I miss from working with Java in Eclipse. There, you're always in a built state so every time you save the file you're working on, you instantly get all the compiler errors if any. There's also a large amount of errors the editor can detect without invoking the compiler, and you get constant feedback about those as well.

My current main environment is Visual Studio for C#, and there you get a lot of errors detected while editing, but not all of them, so you have to continually press the rebuild button, and the time of that rebuild just keeps growing...

Then again, I recently bought a SSD, that helps a bit with build times. :-)

<comment><USER8> If only there were some sort of tool you could use which would automatically detect which dependencies have changed and compile only those files...

I swear, I read "press the rebuild button" and wanted to cry.

 _Edit: the sarcasm-free version of the above amounts to this: there was a time when the idea of "writing software to help you write software" was a standard notion, something that everyone did. It seems like the modern world is training a generation of programmers who don't understand this, and who see the "end result" software as the only software worth writing. The idea of tool creation and integration is alien to them. That's what the IDE buttons are for._

<comment><USER7> I meant the build button, of course you don't have to do a full rebuild every time, that would be insane. Did you honestly think Visual Studio or the cs compiler is that retarded?

However, it doesn't change the fact that you still have to perform it. If you're changing a file that has a lot of dependencies, all of those have to be re-built, unlike if you're sitting in Eclipse coding Java, where you don't.

<comment><USER9> The only reason incremental compilation is possible in Java is that the language completely lacks abstractions like macros and overloading which can globally affect the meaning of existing code. I already spend more time reading and writing code than recompiling, so making recompiling faster by making the code more verbose, repetitive, and error-prone is not a tradeoff I'm happy about.

<comment><USER10> I don't see how macros should make everything depend on everything else. When you want to use a macro you (should) still need indicate somehow where it comes from.

<comment><USER11> They make compilation of the macro's clients depend on the implementation of the macro, and therefore those clients must be recompiled when the macro changes. In Java, there's no expectation that changing the implementation of a method in one java file would cause another java file to be recompiled.

<comment><USER5> But if you add an parameter to a function then don't all the files that have calls to that function have to be recompiled (so that you can see the error)?

I don't know Java so I'm assuming it's similar to other compilers. Please correct me if I'm wrong.

<comment><USER10> Or how about changing types? Javas weak type system should make it necessary to re-compile (and even change the sources).

<comment><USER11> If you change a parameter, then you are changing the interface, and it makes sense that that affects recompilation.

Changing the type is an interesting one. If you change the backing class from one type to another, but you were using an interface to access it, then the Java compiler doesn't need to recompile that code -- the compiled InvokeInterface bytecode for that method invocation doesn't change. However -- I feel like I may have read this somewhere -- there are optimizations which might cause it to replace InvokeInterface with static invocations when it can determine at compile time what class is used. If that's the case, then it would have to recompile the client class too.

<comment><USER10> I was thinking about changing the type of a parameter and return type in parallel --- where the application only treats them as block boxes and just gives and takes those objects, but never looks at them.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER12> Good article about large builds. I know of at least one outfit with a large legacy C++ code spend the effort to move to C#, in no small part due to unmanageable build times. This would be under the "constant reduce" leg of this journey, but with C# you cut your file count in half all other things being equal.

A minor nit: _most other builds should be O(WhateverChanged)_ \-- consider a C++ header file used by 10 c++ files. Those 10 would need to be recompiled. The agony there is that you wish for some less-than-full-file dependency analysis so that only the files that use that constant you just changed need to recompile.

On the whole a good start to those massive builds you have.

<comment><USER13> Also worth point out is the ability to modify the MSBuild file to use parallel build paths - assuming the dependency tree is fairly flat.

<comment><USER14> Which, in case you didn't know, the MSFT build stack was able to do with C++ projects long before it could with C# projects. I wonder if wglb's referenced company would have liked to know that. ;)

<comment><USER13> True, last I checked MSBuild support for VC++ projects was lacking. I think their solution was to run the build through the visual studio command interface...hopefully it's gotten better.

</comment></comment></comment></comment><comment><USER15> Here's a mirror (don't know how well it will hold up): <http://hildr.luminance.org/ibb/>

<comment><USER16> Apparently, neither does his hosting provider...

</comment></comment><comment><USER17> The idea of optimizing build performance for huge monolithic chunks of code seems somewhat misguided. Isn't it better to break your product up into a set of reasonably sized libraries that can be built separately? That way when you change something in one module you can just rebuild that one library, and then perhaps relink it with the others. That general approach can usually deliver fast development cycles with any language or build tool.

</comment><comment><USER18> I like the idea of in-memory change detection... there's a definite use there. I don't agree with the idea of storing all files in memory (especially given their complaint that Git is slow on 20GB of files...), though I admit grepping that is fast.

But what if it were simplified to _just_ a live ls-diff based on when a command was last run / a timestamp? If Git / Make would hook into something like that it'd be really useful.

<comment><USER19> ibb, at it's core, is just an in memory tree of the filesystem metadata. Search is implemented as a plugin that stores the file contents in memory.

And ibb is proof of concept, showing that O(1) is possible, desirable and useful. Having said that, I've been using it daily, because we have > 200MB of PHP/JS/CSS/HTML at IMVU.

<comment><USER18> Gotcha, thanks. That known, I quite like the idea. Any way a makefile could be twonked to use it? (haven't looked at all, don't really know where to start)

</comment></comment></comment><comment><USER20> I agree with the general thrust of the article in that iteration time is one of the most important factors of any kind of creative work, programming included. The longer you go without feedback or corrections, the less confident you can be that you're on the "right track."

That said, build systems and VCS aren't the only ways to optimize the process. For example, a system using scripts processed at runtime lets you reduce the amount of compiled code and iterate quickly on the one section by pressing a "reload script" button, or, even more conveniently, monitoring the source file and auto-reloading when changes are saved. If you're daring enough you could probably even do this with machine-compiled languages by abusing dynamic linking, though the ease of doing that would ultimately depend on the runtime linker capabilities.

Doing this kind of data-driven thing will cost runtime performance since less can be compiled and optimized statically, but if you treat it as the "scaffolding" work that it is and also include ways to reclaim some of the static factors for release builds, you can get a better overall result than you would if you were just suffering through long builds.

</comment><comment><USER21> Whilst I hated clearcase as a version control system (and I mean 'fire and stakes' hate, not just 'crossO the road') - it did have one good trick up its sleeve:

Since the dynamic views into the source were implemented as a custom file system, they could track all the file access that went into building each object. They supplied a custom version of make that pulled that info. out to construct complete dependency descriptions. If you had the toolchain in clearcase as well, that would include all the compiler bins, libs and includes.

Once that info was cached, and as they had code at filesystem level, it could easily be invalidated as files were touched.

Another product of this mechanism was that if you tried to build something that someone else had already (all the same inputs), it would just grab it over the network. In practice this meant that the nightly build would cache the bulk of the object files.

</comment><comment><USER8> FWIW: I'm not sure I buy that 21 minute Mozilla build, unless he was working on a cold disk, or unless the build is doing a bunch of stuff with broken dependencies. If the build was just done, the file metadata is in the page cache, and you can read a _staggering_ number of dents in 21 minutes.

<comment><AUTHOR> I've got a slow Pentium 4. I assure you, it's a real timing... A no-op build on my Core 2 Duo laptop is ~7 minutes.

Either way is ridiculous.

<comment><USER8> 7 minutes following a successful build? That just can't be due to stat/getdents() calls, it can't. Check the log, no doubt there's a broken dependency in there causing something to happen (e.g. an unconditional copy done, and things that depend on the copied file). Again, you should be able to get hundreds of millions (!) of files stat'd in those seven minutes.

There may well be something wrong with the mozilla build, but I swear, it's not the metadata reads from make.

</comment></comment></comment><comment><USER22> Disable make's implicit rules with -r or explicitly in the Makefile:

        .SUFFIXES:
        %:: %,v
        %:: RCS/%
        %:: RCS/%,v
        %:: s.%
        %:: SCCS/s.%
        %.c: %.w %.ch

This gave me a significant speedup for no-op builds (0.7s → 0.1s, ~500 goals in the Makefile).

Of course you can only do this if you do not rely on these implicit rules.

</comment><comment><USER23> My version control and build systems scale better than your blog.

<comment><USER24> Really? If you mean 'your version control and build systems can handle more load than his blog,' perhaps you are correct. But the expression 'scale' refers to the rate at which increases in load places increased demand on resources. Unless you have a very specialized infrastructure for VCS and builds, it probably doesn't scale at a lower rate than his blog infrastructure scales.

<comment><USER23> Do you know what his blog infrastructure is? Over 3 hours later its still not working. Git is decentralized and can and does scale. Builds can be spread across multiple machines on EC2.

<comment><AUTHOR> From 2003 until a few months ago, I hosted my website off of a Pentium 2 in my apartment. Behind slow DSL, no less.

Once my blog posts started reaching hacker news I thought "Oh, I'll just move site out of the apartment and into the cloud!" and bought an account at prgmr.com (which I highly recommend, by the way).

However, I serve WordPress via Apache on a 256 MB VM, which clearly thrashes under load.

Tonight I will purchase an upgrade to 512 MB of RAM and play with nginx.

I'm sorry for the inconvenience.

p.s. I do have WP-Supercache enabled and a PHP bytecode cache. I _could_ just host at wordpress.com, but I might as well learn nginx while I'm at it...

<comment><USER25> Such a good problem to have, Chad. ;)</comment></comment></comment></comment></comment>


A proposal for removing obsolete Emacs packages - <AUTHOR> http://comments.gmane.org/gmane.emacs.devel/198007 <comment><USER2> I'm not opposed to removing stuff which is really, truly, obsolete, but I am concerned about removing useful functionality and never adding it back (c.f. gnome-screensaver, which destroyed the functionality xscreensaver offered and _never_ restored it — it's been _over a decade_ and gnome-screensaver is _still_ not a replacement for xscreensaver!).

A good part of my concern hinges on the definition of 'obsolete.' Code for handling multi-byte characters on black-and-white Macintoshes? Probably obsolete. BBDB? Stagnant but not obsolete (it's sad, incidentally, that it _is_ stagnant, but that's a post for another time).

<comment><USER3> The problem is such code can kill a long-running system. Bugs pile up in code no-one understands, or uses anymore. People are afraid to make large-scale refactorisations / modernisations to code bases as they will have to figure out how to upgrade/fix old stale code.

In the case of emacs I can imagine this situation is particularly bad -- lots of old code and limited developer time.

<comment><USER4> Anecdote is not data, but for Emacs has been pretty bug free. My impression is that things that get used tend to be robust and my belief is that bugs in code that isn't used isn't a problem. From the discussion on the proposal, my guess is that the people who tend to use obsolete packages are people who have been using Emacs for a really long time, and my feeling is that such people know what they are getting into.

That said, I guess my premise is that Emacs is a bit unlike a lot of software in that new users encounter problems with features and only very experienced users are likely to experience bugs. YMMV.

</comment></comment></comment><comment><USER5> I like what Mercurial does: never remove anything, no matter how obsolete. In Mercurial, "obsolete" just means it won't show up in the docs or the wiki or anything. The code is still there, but nobody talks about it, and nobody has to use it. Also, nobody has to fix it. Sure, it's a bit annoying as an hg dev that we can't get rid of any public functionality, but hg users are much happier that the scripts they wrote ten years ago against hg still work today. It doesn't mean that progress can't happen; it just means that progress can't trample other people's habituation.

Emacs has been around for over 30 years or even 40 years, depending on how you count. People have come to rely on all sorts of behaviour since the dawn of time. Changing that behaviour just to make the developers a little less annoyed is completely at the expense of the users.

I reiterate some of Steve Losh's here,

         Allow me to illustrate this with a helpful Venn diagram:

                          </comment><comment>-
                         /3333333\
                        |333333333|
                         \3333333/
                          </comment><comment>-

        11 -> People who give a shit what a program's codebase
        11    looks like.

        22 -> The authors of said program.
        22

[http://stevelosh.com/blog/2012/04/volatile- software/](http://stevelosh.com/blog/2012/04/volatile-software/)

<comment><USER6> This attitude leaves the project hamstrung with old misfeatures. Sometimes externally-visible functionality isn't outright broken, but badly designed in some way, like exhibiting a nonsensical corner case or whatever. People "out there" depend on that behavior, though. Their scripts might break if it is changed. You really want to fix it.

There are ways forward, like supporting a backwards compatibility mode which emulates the old behavior. For instance --emulate-version=6 could make version 7 behave like 6 with regard to changes like this. At some point, you cut off support for --emulate-version values of 6; the program then fails with a message saying "Version 17 can only emulate quirks from version 8 and newer." and bail.

If people want the old behavior, they can use an older version of the program.

<comment><USER7> I think the situation is different for Mercurial. Since every hg client has to be able to inter-operate with any hg client from any other machine, regardless of whether it's been updated or not, it makes sense to retain old functionality, so that newer clients can still talk to older ones. I believe rsync also does a similar thing, where it has some kind of system to figure out the highest version of the rsync protocol supported by both sides, and newer rsync versions support all the old versions of the protocol. Emacs is not in this position. Multiple versions of Emacs can always inter-operate as long as they can all read and write text files.

<comment><USER6> Two things: I suspect that not every feature of the client has to do with protocol or repo format interoperability. Secondly, things can change in that regard, too. Secondly, formats and protocols can be subject to versioning and obsolescence also. In short, the situation is not different.

> _Multiple versions of Emacs can always inter-operate as long as they can all > read and write text files._

What? No; Emacs is a language implementation, changes in which can easily break old code. Sure it can _read_ some elisp file you wrote in 1991, but will it work?

</comment></comment></comment></comment><comment><USER8> Would it be easy to include suggested replacement packages? This would help people bring forced to upgrade.

My .emacs is ancient, and I'm pretty sure there's tons of obsolete elisp packages.

Thx!

</comment><comment><USER9> [https://xkcd.com/1172/](https://xkcd.com/1172/)

</comment><comment><USER10> We should also remove obsolete wiki.

cf [http://batsov.com/articles/2012/03/20/die- emacswiki/](http://batsov.com/articles/2012/03/20/die-emacswiki/)

And try to make an awesome replacement, with an attractive website. Like [http://spacemacs.org/](http://spacemacs.org/) ;)

(I saw some initiatives such as [http://nicolas-petton.fr/ressources/emacs- website/](http://nicolas-petton.fr/ressources/emacs-website/))

come on, we are in #b11111100000 !! =)

<comment><USER11> If you look at the edits in the past month for emacs wiki and wikemacs, emacswiki is still the place to go:

[http://wikemacs.org/index.php?title=Special:RecentChanges&da...](http://wikemacs.org/index.php?title=Special:RecentChanges&days=30&from=&limit=500)

[http://www.emacswiki.org/emacs?action=rc;days=28;all=0;showe...](http://www.emacswiki.org/emacs?action=rc;days=28;all=0;showedit=0)

</comment></comment><comment><USER12> It sounds like a reasonable plan to me. I didn't even know longlines-mode was obsolete.

Would suggest that obseletion messages should also have optional space to include suggestions about what packages are not obsolete that may fill the same space. I had to google for "visual lines mode" being the successor.

</comment><comment><USER13> Seems like a change that would break a lot of configurations for users.

<comment><USER7> Those users would be warned about the breakage well in advance under this proposal. They can choose not to upgrade Emacs, or choose to switch to a non- obsolete replacement package for their desired functionality, or choose to complain that the feature should not be removed because it has no replacement.

<comment><USER11> and when they complain, what would the answer be?

Would there be a complain-button next to the deprecation warning?

</comment></comment></comment><comment><USER14> They can't even remove obsolete code (c.f. [http://emacshorrors.com/posts/forget-me- not.html](http://emacshorrors.com/posts/forget-me-not.html)) why do we think they'll be able to remove obsolete packages.

<comment><USER7> That obsolete code has been removed for decades. It's only left in the file in a commented-out form because it is the only documentation of the behavior of the original function. Are you complaining about the poor documentation practices of an open source project in the early 1980s?</comment></comment>


Why Python 3 Exists - <AUTHOR> http://www.snarky.ca/why-python-3-exists <comment><USER2> IMO one of the reasons for all the angst is that .encode() and .decode() are so ambiguous and unintuitive which makes them incredibly confusing to use. Which direction are you converting? From what to what? The whole unicode thing is hard enough to understand without Python's encoding and decoding functions adding to the mystery. I still have to refer to the documentation to make sure I'm encoding or decoding as expected.

I think there would have been much less of a problem if encode and decode were far more obvious, unambiguous and intuitive to use. Probably without there being two functions.

Still a problem of course today.

<comment><USER3> Hm, I never saw this as ambiguous at all, except for a few weird encodings that Python has as "convenience" method.

Here's how you remember it: "Unicode" is _not_ an encoding. It never was, it never will be. Of course, the data must be encoded in memory somehow, but in Python 3, you cannot be sure what encoding that is because it's not really exposed to the user. From what I understand, there are different encodings that string objects will use, transparently, in order to save memory!

You always "encode" something into bytes, and "decode" bytes back into something. There should be exactly two functions, because the functions have different types: "encode" is str -> bytes, "decode" is bytes -> str. Explicit is better than implicit.

        output = input.decode(self.coding)

With Python 3, I instantly know that "input" is bytes and "output" is str.

<comment><USER4> Indeed, the Python 3 Unicode string object is fascinatingly clever. Code worth reading:

[https://github.com/python/cpython/blob/master/Objects/unicod...](https://github.com/python/cpython/blob/master/Objects/unicodeobject.c)

<comment><USER5> Also, it's incompatible with UTF-8 strings stored in C, which means that when you cross the Python/C API boundary, you have to re-encode all strings. This is a large performance penalty right at the time when you can least afford performance penalties.

IMNSHO, most modern languages should be storing strings as UTF-8 and give up on random access by characters. You almost never need it; in the most frequent case where you do (using indexOf or equivalent to search for a substring, and then breaking on it), you can solve the problem by returning a type-safe iterator or index object that contains a byte offset under the hood, and then slicing on that. Go, Rust, and Swift have all gone this route.

<comment><USER6> This design doc from DyND (a possible NumPy alternative) has some useful references on this point: [https://github.com/libdynd/libdynd/blob/master/docs/string-d...](https://github.com/libdynd/libdynd/blob/master/docs/string- design.md#code-unit-api-not-code-point)

<comment><USER7> Nope. This is just a normal short-string implementation, inlined with 64bit systems, with a tag bit. Very common on every better VM.

This has nothing to do with (inefficient) encodings, and their bad historic namings, probably derived from perl.

Of course any name like byte2utf8 or just to_byte or b2u8 would have been better than encode/decode.

And of course cache size matters nowadays much more than immediate substring access for utf8, so nobody should use ucs-2 or even ucs-4 internally. This is easily benchmarkable.

<comment><USER8> "Byte2utf8" is a pretty confusing name for a method, considering utf8 is a byte encoding of unicode... :)

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> Unicode worked just fine by Python 2.6. I had a whole system with a web crawler and HTML parsers which did everything in Unicode internally. You had to use "unicode()" instead of "str()" in many places, but that wasn't a serious problem.

By Python 2.7, there were types "unicode", "str", and "bytes". That made sense. "str" and "bytes" were still the same thing, for backwards compatibility, but it was clear where things were going. The next step seemed to be a hard break between "str" and "bytes", where "str" would be limited to 0..127 ASCII values. Binary I/O would then return "bytes", which could be decoded into "unicode" or "str" when required. So there was a clear migration path forward.

Python 3 dumped in a whole bunch of incompatible changes that had nothing to do with Unicode, which is why there's still more Python 2 running than Python 3. It was Python's Perl 6 moment.

From the article: _" Obviously it will take decades to see if Python 3 code in the world outstrips Python 2 code in terms of lines of code."_ Right. Seven years in, Python 2.x still has far more use than Python 3. About a year ago, I converted a moderately large system from Python 2 to Python 3, and it took about a month of pain. Not because of the language changes, but because the third-party packages for Python 3 were so buggy. I should not have been the one to discover that the Python connector for MySQL/MariaDB could not do a "LOAD DATA LOCAL" of a large data set. Clearly, no one had ever used that code in production.

One of the big problems with Python and its developers is that the core developers take the position that the quality of third party packages is someone else's problem. Python doesn't even have a third party package repository - PyPI is a link farm of links to packages elsewhere. You can't file a bug report or submit a patch through it. Perl's CPAN is a repository with quality control, bug reporting, and Q/A. Go has good libraries for most server-side tasks, mostly written at Google or used at Google, so you know they've been exercised on lots of data.

That "build it and they will convert" attitude and the growth of alternatives to Python is what killed Python 3.

<comment><USER10> > That "build it and they will convert" attitude and the growth of > alternatives to Python is what killed Python 3.

Well said.

</comment></comment><comment><USER11> > _We have decided as a team that a change as big as unicode /str/bytes will > never happen so abruptly again. When we started Python 3 we thought/hoped > that the community would do what Python did and do one last feature release > supporting Python 2 and then cut over to Python 3 development for feature > development while doing bugfix releases only for the Python 2 version._

I'm guessing it's not a coincidence that string encoding was also behind the Great Sadness of Moving From Ruby 1.8 to 1.9. How have other mainstream languages made this jump, if it was needed, and were they able to do it in a non-breaking way?

[https://news.ycombinator.com/item?id=1162122](https://news.ycombinator.com/item?id=1162122)

<comment><USER12> C and C++ are so widely used that transitions like this are made not at the language level but at the level of platforms or other communities. Some parts of the C/C++ world made this transition relatively seamlessly, while others got caught in the same traps as Python.

The key is UTF-8: UTF-8 is a superset of 7-bit ASCII, so as long as you only convert to/from other encodings at the boundaries of your system, unicode can be introduced to the internal components in a gradual and mostly-compatible way. You only get in trouble when you decide that you need separate "byte string" and "character string" data types (which is generally a mistake: due to the existince of combining characters, graphemes are variable-width even if you're using strings composed of unicode code points, so you don't gain much by using UCS-4 character strings instead of UTF-8 byte strings).

My theory is that the python 3 transition would have gone _much_ smoother and still accomplished its goals if they had left the implicit str/bytes conversion in place but just made it use UTF-8 instead of ASCII (although in environments like Windows where UTF-16 is important this may not have worked out as well).

<comment><USER3> You are correct that there's no real benefit in UTF-32 over UTF-8, which is why Go and Rust (and others) have worked with UTF-8 in memory just fine. However, the actual encoding of a str object is irrelevant, and it's not the point. The whole point of the str/bytes difference in Python is that you make Python keep track of whether you've done the conversion or not. In Python 2, you can be sloppy, and the programs are buggy as a result!

You're putting the cart before the horse here in terms of the Python 3 transition. The str/unicode fix was one of the driving factors for Python 3 to exist in the first place, and if you removed it, then what's the point?

Again, look at Go or Rust. Both of them have separate types for strings and bytes, even though they have the same representation in memory, and as a result we don't have that kind of bug in our program.

<comment><USER12> > The str/unicode fix was one of the driving factors for Python 3 to exist in > the first place, and if you removed it, then what's the point?

The reason python 3 exists is that most python 2 code had latent unicode- related bugs that would only manifest when they were exposed to non-ascii data. The backwards-incompatible barrier between str and bytes was the solution the python 3 team chose for this problem; adopting utf-8 as the standard encoding would have been another solution which I claim would have been more backwards-compatible (essentially moving to the go/rust model, which prove that you don't necessarily need separate byte and character string types for correct unicode handling).

<comment><USER13> But not every 8-bit byte string is valid UTF-8 so that could still cause a world of pain.

</comment></comment></comment></comment></comment><comment><USER14> In the Reddit discussion of this, someone linked to this criticism [1] of Python 3's Unicode handling written by Armin Ronacher, author of the Flask framework.

I am not competent to say whether this is spot on or rubbish or somewhere in between [2], but it seemed interesting at least.

[1] [http://lucumr.pocoo.org/2014/5/12/everything-about- unicode/](http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/)

[2] Almost all of my Python 2 experience is in homework assignments in MOOCs for problems where there was no need to care about whether strings were ASCII, UTF-8, binary, or something else. My Python 3 experience is a handful of small scripts in environments where everything was ASCII.

<comment><USER15> It's wrong. The whole point of cat is to concatenate files. But if you concatenate two files with different encodings, you end up with an unreadable file. So you _want_ cat to error out if one of the files that was passed has a different encoding from the encoding you told it to use, _which is exactly what the python cat will do_.

<comment><USER16> > The whole point of cat is to concatenate files.

Yes.

> So you want cat to error out if one of the files that was passed has a > different encoding.

No. I expect it to read bits from stream A until it is exhausted and then read bits from stream B until it is exhausted. All the time just writing the ones and zeros read to the output stream (of bits). And no, a byte does not have to be "8 bit" ([http://www.lispworks.com/documentation/HyperSpec/Body/f_by_b...](http://www.lispworks.com/documentation/HyperSpec/Body/f_by_by.htm)).

<comment><USER15> And concatenating a file of 9-bit bytes with a file of 8-bit bytes will produce something useful? No. If you don't know what your bits represent then you will corrupt them. Python does not need to faithfully reproduce all the historical oddities of unix.

<comment><USER16> It might - depending on what I intend to do with the file (I still have the offsets of the individual files because I know the original file sizes).

API-wise, in my humble experience, it's hell to deal with operations that are supposed to work on bit-streams but try to be smart and ask me for encodings of those - this is information I might not even have when building on those basic operations. The "oddities", how you call them, are the result of not over-abstracting the code to handle yet-unknown problems.

You want to concatenate text files with different encodings? _Convert_ them. Expecting a basic tool to do this for you (or carp on "problems") quintessentially leads to cat converting image formats for you and demanding to know how to _combine_ those images: (addition, subtraction, append to left/top/right/bottom of first image etc).

<comment><USER15> > API-wise, in my humble experience, it's hell to deal with operations that > are supposed to work on bit-streams but try to be smart and ask me for > encodings of those - this is information I might not even have when building > on those basic operations. The "oddities", how you call them, are the result > of not over-abstracting the code to handle yet-unknown problems.

That's how C ended up as the security nightmare that it is. There are a lot of things you can do in C that you can't do in Python - you can reinterpret strings or floats as integers, you can subtract pointers from integers, you can write to random memory addresses.... Sometimes these things are useful, but most of the time they just lead to your program breaking in an unhelpful, nonobvious way.

Python is not that kind of language; it will go to some pains to help you not make mistakes. If you want to do bit-twiddling in Python there are APIs for it, and you could implement a "bit-level cat" using them, but it's never going to be the primary use case. Arguably there should be better support for accessing stdin/stdout in binary mode, but that would make it very easy to accidentally interleave text and binary output which would again result in (probably silent) corruption. (Writing a "binary cat" that concatenates two _files_ of bytes would not lead to any of the problems in the linked article - it's only trying to use stdin/stdout that's causing the trouble in the link).

<comment><USER16> > That's how C ended up as the security nightmare that it is.

And that's how I ended up completely wasted after a friend had to throw a party after a GNU version of a common unix tool was able to accept his first name as valid parameters.

And no, C's problem are _not_ based on encoding issues. Those are not even a first-class symptom.

> Python is not that kind of language

Maybe. I don't care much, even though I do speak Python fluently. Though, I _do care_ about minimal functional units whose documentation I can grasp in minutes, not hours.

> [Python] will go to some pains to help you not make mistakes.

This is not specific to python, this is specific to [language] developers. If all you do is text processing, you will think in characters and their encoding. I don't. A lot of programmers don't - because they deal with real world data that is almost never most efficiently encoded in text.

To reiterate: We are all dealing with bit-streams. Semantics of those are specific to their context. If your context is "human readable text" \- deal with it. But please don't make me jump through hoops if I actually just want to deal with bit-streams. If you need magic to make your specific use-case easier, wrap the basic ops in a library and use it.

Last but not least: This all is completely off-topic when the question is about "why python3" \- it's great for your use-case, but from an abstract point of view, python3 was just the rational continuation of python2, cleaning up a lot of inherited debt. Though it might fit your world-view, it's not necessarily what it was about.

</comment></comment></comment></comment></comment></comment></comment><comment><USER17> IMO the biggest reason to use Python3 is its concurrency support via async + await.

Fixing the unicode mess is nice too of course, but you can get most of the benefits in Python2 as well, by simply putting this at the top of all of your source files:

from __future__ import unicode_literals

Also make sure to _decode_ all data from the outside as early as possible and only _encode_ it again when it goes back to disk or the network etc.

<comment><USER18> So much this! asyncio was the main selling point for me, but in general, why not follow the language?

I never really understood the "rather stay with py2.7" thing. I get it with big old monolithic applications. You don't "just" rewrite those, but _every new python project_ should be done with the latest stable release.

Is anyone starting their PHP projects on PHP4? Any new node projects in 0.10? Of course not, that would be moronic.

<comment><USER19> Because you don't know what libraries you may depend on in the future when you start a new project. I was a fervent Python 3 supporter, and wrote every of my fresh projects in Python 3 instead of 2, until one day I found I needed LLVM in my project, yet the python port at that time was for 2 only.

I mostly avoid writing Python 3 nowadays, because I don't want to rewrite my project or find painstakingly an alternative solution when I could have just imported a module that runs fine under Python 2.

</comment></comment></comment><comment><USER20> I chose to port from CPython2 to PyPy4, rather than to CPython3. It just made more sense. I for one see no value in Python3 (unicode has been supported since 2.6). My reasons for migrating to PyPy4 instead of Python3-

1) It was easier than porting to CP3.

2) It gave me a tangible benefit by removing all CPU performance worries once and for all. Added "performance" as a feature for Python. Worth the testing involved.

3) It removed the GIL. If you use PyPy4 STM, which is currently a separate JIT. Which will be at some point merged back into PyPy4.

So for me, Python3 can't possibly compete, and likely never will with PyPy4 once you consider the performance and existing code that runs with it. PyPy3 is old, beta, not production-ready, based on 3.2 and Py3 is moving so fast I don't think PyPy3 would be able to keep up if they tried.

Python3 is dead to me. There's not enough value for a new language. I'm not worried about library support because Py2 is still bigger than 3 and 2.7 will be supported by 3rd party libraries for a very long time else choose irrelevance (Python3 was released in 2008, and still struggling to justify its existence...). My views on the language changes themselves are stated much better by Mark Lutz[0]. I'm more likely to leave Python entirely for a new platform than I am to migrate to Python3.

PyPy is the future of Python. If the PyPy team announces within the next 5 years they're taking the mantle of Python2, that would be the nail in the coffin. All they have to do is sit back and backport whatever features the Python2/PyPy4 community wants into PyPy4 from CPython3 as those guys run off with their experiments bloating their language. I believe it's all desperation, throwing any feature against the wall. Yet doing irreparable harm bloating the language, making the famous "beginner friendly" language the exact opposite.

I already consider myself a PyPy4 programmer, so I hope they make it an official language to match the implementation. There's also Pyston to keep an eye on which is also effectively 2.x only at this time.

[0][http://learning-python.com/books/python- changes-2014-plus.ht...](http://learning-python.com/books/python- changes-2014-plus.html)

<comment><USER21> There's just a bit of hyperbole in your comment. Most major libraries have been ported to Python 3. I wonder if the opposite of what you're saying will happen--i.e, the libraries that don't support Python 3 will be left behind. Fabric is an example of that for me.

<comment><USER20> Of course you may be right, but obviously I don't think so at this time. I feel I made a smart, safe bet. I'm still writing valid 2.7 code that could be migrated to 3.x at any point if for any reason my current plan would fall through. It would be just as easy for me to migrate to 3.x as it would anyone else with 2.x codebases.

So far, solving CPU performance in Python and removing the GIL is pretty much the holy grail. Python2 would have to completely collapse (no signs of this) for CPython3's ecosystem to outweigh the long-tail of libraries only on 2.x and the truly next-level dynamic language CPU performance.

While I am enjoying "performance as a feature" and GIL-free Python at the moment, I can still migrate to Python3 at any time if I lose my safe bet with PyPy4/Py2.

To me it looked and still looks like a no brainer.

<comment><USER21> > I'm still writing valid 2.7 code that could be migrated to 3.x at any point

Well, that's kinda true, although there is somewhat of a learning curve doing 2-to-3 migrations. Depending on your situation, this may or may not matter, but if you need to "move fast" on this at some point, it'd at least be beneficial to know what's involved (even though it's not that onerous IMO).

<comment><USER20> I've used (tested) Python3 many times over the years and releases. I'd consider the changes trivial. I'm not anti-Python3, I'm pro-PyPy4.

In my attempts to test Python3 releases though I've ran across bugs and performance regressions from 2.x. After I continually ran into this in a few releases I eventually threw my hands up. The last release I ran tests on was 3.4 and I'm no longer interested in later releases until something as substantial as PyPy's CPU performance and removal of the GIL (PyPySTM) shows up to beat what I have now.

It's slick to have the entire Python2 ecosystem, major performance boost to your code, and still leave the door open for Python3 if they ever stop bloating the language.

Other than being dramatically slower than PyPy4, Python3 is also feature soup. I strongly dislike technical churn rather than true technical innovation (which is what PyPy represents). I'm more in line philosophically with Go, I'd prefer to remove features until you're down to a very concise and stable core. Python3 has many negatives, but from my perspective they keep piling on more.

</comment></comment></comment></comment></comment><comment><USER22> I love when people with native english skills write monsters like this: "If people's hopes of coding bug-free code in Python 2 actually panned out then I wouldn't consistently hear from basically every person who ports their project to Python 3 that they found latent bugs in their code regarding encoding and decoding of text and binary data."

This should be under penalty ;)

Anyone to divide it into few simpler sentences?

UPDATE: And another one from our connected sentences loving author: "We assumed that more code would be written in Python 3 than in Python 2 over a long-enough time frame assuming we didn't botch Python 3 as it would last longer than Python 2 and be used more once Python 2.7 was only used for legacy projects and not new ones."

<comment><USER23> The first one:

> If people's hopes of coding bug-free code in Python 2 actually panned out

Python2 developers wanted to write bug-free code _.

_ code = for the purpose of processing text and binary data

> then I wouldn't consistently hear from basically every person

Python2 developers could not write bug free code. So they complained _.

_ complained = complained about their algorithms having bugs when they rewrote those algorithms in Python3

> that they found latent bugs in their code regarding encoding and decoding of > text and binary data.

Python2 code written by the same developers had bugs that they did not know about.

When the same developers rewrote their code in Python3, they found the bugs.

(If Python3 did not exist, then it would be very hard to write bug-free code in Python2.)

The second one:

> We assumed that more code would be written in Python 3 than in Python 2 over > a long-enough time frame assuming we didn't botch Python 3 as it would last > longer than Python 2 and be used more once Python 2.7 was only used for > legacy projects and not new ones.

If we designed Python 3 correctly, then we expect Python 3 to live longer than Python 2. We also expect more code to be written in Python 3 for the same reason. We also expect only old projects will be written in Python 2.7.

</comment></comment><comment><USER24> Since python3 is not backwards compatible with python2, why didn't the python devs leverage the opportunity for creating a more performant non-GIL runtime for python3?

<comment><USER25> Removing the GIL is a difficult problem: [http://python- notes.curiousefficiency.org/en/latest/python3/...](http://python- notes.curiousefficiency.org/en/latest/python3/multicore_python.html)

<comment><USER26> I think the question was if you aren't going to be backwards compatible, why not unshackle yourself completely and design a new language without the GIL and make it as pythonic as possible?

A scripting language that support multi-threading is possible, right? I think TCL does it.

<comment><USER27> I guess they were happy with breaking backwards compatibility a little bit, not not as much as simply removing the GIL and not adding back any implicit synchronisation at all.

</comment></comment></comment></comment><comment><USER28> So Python 2 did not have super obvious string handling. One of the odd things that they seemingly could have fixed pretty easily is to change the default encoding from 'ascii' to 'utf8'. That would have fixed a bunch of the UnicodeDecodeErrors that were the most obvious problem with strings: [http://www.ianbicking.org/illusive- setdefaultencoding.html](http://www.ianbicking.org/illusive- setdefaultencoding.html)

If they had to make Python 3 anyway, I think the main thing they were missing is that they should have added a JIT. That makes upgrading to Python 3 a much easier argument. If the only point of the JIT was to add a selling point to Python 3, that probably would have been worth it.

</comment><comment><USER29> It seems to me if bytes/unicode was the only breaking change we would probably be over the transition by now.

There are a lot of other subtle changes that makes the transition harder: comparison changes and keys() being an iterator for example. These are good long term changes, but I wish they weren't bundled in with the bytes/unicode changes.

</comment><comment><USER30> We migrated to Go from Python 2, since instead of incompatible Python 3 we needed faster Python 2 replacement.

</comment><comment><USER31> Str is tip of the iceberg. Python before 2.7 and current Python is completely different language semantically; methods, functions, statements, expressions, Global interpreter lock behavior.. This is sad that this blog post and discussions around it didn't mention anything about it.

<comment><USER32> The article isn't covering all the differences between Python 2 and Python 3. Based on this article as well as other articles I've read in the past, the Unicode issue was the original reason they decided it was necessary to break backward-compatibility, but once that decision was made, there was no reason not to make any further backward-incompatible improvements.

</comment></comment><comment><USER33> The reason, I still did not port to Python 3:

(and yes, Unicode in Py2 is a mess ...)

They just broke to many things (unnecessarily!) internally. Particularly they changed many C APIs for enhancement modules, so that all of them had to be ported, before they could be used with Python 3. They did not even consider a portability layer ... why not??

Some (not all) of the bad decisions (like the u"..." strings) they did change afterwards, but than it was a little late.

So many modules are still not ported to Python 3 -- so the hurdle is a little to high -- for small to nil benefits!

So, the problem (from my side) is not Unicode at all ... just the lack of reasonable support from the deciders side.

\---

Maybe, some time later, when I have to much spare time.

<comment><USER34> Agreed. Also this:

"So expect Python 4 to not do anything more drastic than to remove maybe deprecated modules from the standard library."

But _why_ break all the existing libraries that use those modules, even if there's now "better" ways. In every comparison I've ever seen on performance, robustness, etc python always loses to the other big languages. Except in one area: the availability of user-land submitted packages and extensions. So why break them for a little perf boost?

<comment><USER33> Sounds really a bit weird to me.

I hope, that they really think hard, which modules really must be removed.

I would only consider security reasons to really remove a library module. Or at least mark it deprecated for a long while until it is used only really rarely.

</comment></comment></comment><comment><USER35> This is a pretty good explanation of unicode in Python: [http://nedbatchelder.com/text/unipain.html](http://nedbatchelder.com/text/unipain.html)

</comment><comment><USER36> I like Python3 personally. It's new and better but a _different branch_. I'm annoyed by people abbreviating it as "Python" and treating it as a substitute for Python2. In my opinion, the "Python" name should be exclusively used for Python2, and Python3 should've been always used as one word. The whole Python3 situation caused unnecessary confusion to the outside (non-Python) people, which I think could be avoided.

</comment><comment><USER37> Since I'm trying to keep a small footprint, I rely on the system version of Python on Mac OS X, which is 2.7.10 now.

To use anything newer, I'd have to ask users to install a different interpreter, or bundle a particular version that adds bloat. There's no point. The most I've done is to import a few things from __future__; otherwise, my interest in Python 3 begins when Apple installs it.

</comment><comment><USER38> The Go authors have solved this problem thoroughly. When working in Go, I usually never have to think about this.

[https://blog.golang.org/strings](https://blog.golang.org/strings)

<comment><USER39> Go came out in 2009. What's your point? I'd sure hope they'd look at languages older than them.

</comment></comment><comment><USER40> How long is the transition going to take? Serious question. Because I'm rather tired of starting new work and finding some module that drags me back to 2.x.

<comment><USER41> Technically, "forever", since there will be people who never port their code. If you're depending on one of those holdouts, it's time to find a new dependency, because if they haven't ported by now they won't and that's your problem because...

in practical terms, the transition is about to be over, since now the Linux distros are all-in on converting to Python 3 for their current or next releases and that will forcibly move the unported libraries in the bin of obsolescence.

</comment></comment><comment><USER42> From the outside, Python 3 seems like a much better language. I don't have strong views of its object system (I avoid OOP as much as I can) but it seems like the string/bytes handling is much better, and I'm also a fan of map and filter returning generators rather than being hard-coded to a list implementation (stream fusion is a good thing). Also, I fail to see any value in print being a special keyword instead of a regular function (as in 3).

What I don't get is: why has Python 3 adoption been so slow? Is it just backward compatibility, or are there deeper problems with it that I'm not aware of?

<comment><USER43> What makes me snarky is the replacing of

        '%s %s' % ('one', 'two')

With

        '%s %s'.format('one', 'two')

The latter is just more annoying to type. Stupid argument I know but I find myself grumbling to myself every time...

<comment><USER44> Simplified to this in Py 3.6:

        f'{one} {two}'

<comment><USER45> Do you have a reference for that? Wouldn't that be introducing a huge security hole in all programs?

<comment><USER44> PEP 498. Literals only, does not add any _additional_ security problems.

<comment><USER45> Ah, right; I missed the 'f' prefix. And since it's only done when parsing the expression, it is not a security problem. Thanks!

</comment></comment></comment></comment></comment></comment><comment><USER46> Ok, fine. Can we have the print statement back?

<comment><USER47> Genuinely curious, why do you prefer `print` to be a statement rather than a function? I've heard a lot of criticisms of Py3, but this is the first time I've heard this one.

<comment><USER48> I think a function is "better", but the print statement should have been preserved.

Print is used either for:

1) Writing to stdout/stderr 2) Debugging the hacky way

The print function is better for the former (though more often than not, I use Armin Ronacher's click.echo for compability)[0], but I fastly prefer the print statement for (2). Not dealing with parentheses is always a plus; I can add and remove print statements far more quickly.

I find it to be an increase in friction, and I don't see any real downside in leaving it in.

[0] [http://click.pocoo.org/5/api/#click.echo](http://click.pocoo.org/5/api/#click.echo)

<comment><USER47> I agree with you on 1), I much prefer `print(..., file=sys.stderr)` to `print >>sys.stderr, ...`.

For 2), though, I have always used auto-inserting parens in my vim, so I never really experienced any pain w.r.t. parentheses. I can see how that would be a pain, though.

</comment></comment></comment></comment><comment><USER49> I think this is the primary reason why some scripting languages end up in the education space (as a tool for learning), while others go mainstream and ubiquitous in the commercial space. Breaking stuff between versions is a headache and expense for everyone except the most superficial users.

The 'there should be one – and preferably only one – obvious way to do it' rule sounds like another reason. It's like being asked to choose between a perfect general use knife or a Swiss army knife.

<comment><USER50> This would _almost_ be a reasonable post if Python weren't mainstream and ubiquitous in the commercial space.</comment></comment>


Socialhome: a federated personal profile with social networking functionality - <AUTHOR> https://github.com/jaywink/socialhome <comment><USER2> I feel like GNU Social/Mastodon and the like is going to be the most successful in the space these days than Diaspora-based sites.

I spend some time on Diaspora back in the day, and it felt like it's community was solely people who had likely been banned or pushed out of more mainstream sites. It lacked a lot of the appeal of a modern social network.

I feel like the ActivityPub/OStatus crowd has done a better job reaching wide appeal, federated communities are small enough without multiple standards.

It does look like Socialhome wants to include ActivityPub in the future, and that sounds like a big win.

<comment> dane-pgp I feel like the most useful feature a distributed social network needs is the ability to grant/deny access to people based on decentralised identifiers (such as the person's email addresses, although people may prefer to use a dedicated email address for this purpose).

If such a distributed ID system were available, then it should even be possible to move away from one web host to another host, and still have access to your friends' pages, and have them find you. Alternatively you could instruct your current host to broadcast an "I'm about to move" message to your friends before you move your data to the new host.

Perhaps the biggest hurdle standing in the way of this is the spam / privacy issue, since people don't want to have to prove they own a specific email address in order to read your posts on the niche site where you're hosting your profile. Also, if your posts are being pushed out to the sites where your friends have accounts, then you have to implicitly trust the privacy policies and security of all those sites.

<comment><USER2> The current system the fediverse uses is, more or less, the same as email addresses... with the same downsides. I'm firstname.lastname@example.com ...My account at the server I'm at, just like email. And yeah, email migration is quite a frustrating process, and the best way to take control of it is to own your own domain name.

Perhaps the solution is, like an MX record, for me to be able to assign my social traffic from my domain name to route to a different server, so I can assign it to the server I'm using, and move it at my leisure? This is why I have my personal domain pointed at FastMail for mail, it means I can move it without giving everyone a new address.

<comment> dane-pgp Nice ideas, and I use a similar setup for my personal domain. For people who don't want the hassle of setting up a domain name, maybe there should be a standard by which an email provider could be queried to find someone's social network account provider.

If your email address were, for example, ocd AT trekk DOT ie, an email contact of yours should be able to make a DNS (or SMTP?) request to trekk.ie and find your social network account email@example.com, where they could send a friend request from their own social network provider. The link between email account and social network account could be checked periodically based on the TTL.

There is a somewhat analogous IETF standard for looking up identity data from mail servers, here:

[https://tools.ietf.org/html/rfc7929](https://tools.ietf.org/html/rfc7929)

<comment><USER3> > an email contact of yours should be able to [...] find your social network > account firstname.lastname@example.com [...] The link between email account and > social network account could be checked periodically

This is how the keybase sigchain works. (And I do mean "works"—as in, it's available now and real people have been using it.)

PS: Your HN profile doesn't contain any contact info.

</comment></comment></comment></comment></comment><comment><USER4> A detailed text description of what it's like to use it might be nice. There's a lot of technical documentation but it's hard for me to want to dive into that before I understand what the payoff is.

<comment><USER4> Also that poster is too hard for a guy with bad eyes to read.

<comment><USER5> Just in case anyone isn't aware of the feature, clicking onto the source image, then holding down control and scrolling up with the middle mouse button in Firefox will magnify the image (making it closer to readable, but still suffers from a lack of pixels).

<comment><USER6> Instead of posting this here, you could have this written above the poster. This is called usability and it's a real thing!

</comment></comment></comment></comment><comment><USER7> [https://github.com/jaywink/socialhome](https://github.com/jaywink/socialhome) is a link to the Github while the site seems to be struggling under load.

<comment><USER8> Ok, we'll change to that from [https://socialhome.network/content/1198667/socialhome-080-re...](https://socialhome.network/content/1198667/socialhome-080-released- this-release-focus/) for now.

We also changed the title to a description taken from that page, since the project doesn't seem to have been discussed on HN before.</comment></comment>


Browserify-loader - <AUTHOR> https://github.com/island205/browserify-loader

<comment><AUTHOR> What is browserify-loader

browserify-loader is another CommonJS loader for browserify workflow. With BL, You don't need any tools like watchify, browserify-middleware to auto build and serve bundle *js in development env.

browserify-loader is similar with requirejs, but:

follow Modules/1.1.1 like Node get rid of wrapper code like define() be compatible all npm package and all bower components witch support CommonJS. like underscore, backbone, jQuery and so on.</comment>


An open source GitHub-like platform as an alternative for AWS Lambda - <AUTHOR> https://github.com/1backend/1backend <comment><USER3> There is also [https://hook.io/](https://hook.io/) which is a similar open- source platform that's been running for three years and has over 142 million deployments.

I'm using this service everyday, also I created it.

<comment><AUTHOR> Hi Marak!

Hats off to you. I remember when talking with Asim Aslam about him going and creating micro ([https://github.com/micro](https://github.com/micro)), a microservices framework, I tried to advise him to go high level, as high level as possible and create a complete platform.

I used hook.io as the example of a project which is on the right path. He said the world is not ready for it. I said some day someone will make a tool like hook.io and promote it well and get incredibly popular.

Turns out we were both right. Seems like the world is ready for these kind of solutions now, and you were ahead of the curve with hook.io.

<comment><USER3> Greetings! Thank you for the kind words.

It's really good to see more developers getting into the space of writing open-source platforms for services. I believe there is room for much growth in this field. I'll take a deeper dive into 1backend and check out how it works.

hook.io has pretty much a zero dollar budget and maybe $150 spent on marketing. It's been slowly and steadily growing for the past years and I'm happy with it's progress. I'm currently working on building integrated hardware solutions with raspberry pi to offer an open-source local version of Amazon Echo or Google Assistant type devices.

<comment><USER4> > I'm currently working on building integrated hardware solutions with > raspberry pi to offer an open-source local version of Amazon Echo or Google > Assistant type devices.

Any info on the software side of your project? I've been looking for exactly that (FOSS programmable digital assistant totally under my control). Closest I've seen so far is Jasper[1] but it's still pretty raw.

[1]: [http://jasperproject.github.io](http://jasperproject.github.io)

<comment><USER3> The raspberry is running a local copy of the same software which powers the production hook.io site. Code is at: [https://github.com/bigcompany/hook.io](https://github.com/bigcompany/hook.io)

I don't have any turn-key solutions available for setting up the software, but we are working on it.

I wish I could offer you more at this point. You can signup for updates on our site and we'll send an email when we have something ready.

I'd suggest hacking away on your own solution and post a write-up on what you ended up using. The hardest issue I've encountered so far is local voice detection. Still trying to figure out the best solution for that.

</comment></comment></comment></comment></comment><comment><AUTHOR> Hi guys!

One of the authors here! We hope you enjoy this much loved project we are building with our tiny team.

This is a love project of a team who failed at a startup. We decided to build something we love before getting back to our normal lives :)

I have worked in the faas/microservices space for years before but never wanted to build something like this (I thought it was too early to explain what we are doing).

Now with the advent of faas/serverless/lambda perhaps it's a bit easier.

Our focus is building something that enables service reusability and composability: no more writing the same login service multiple times.

We would also like to host other peoples low traffic services for free if we can (right now we can't because we are on a small server, fixing that).

Thanks

<comment><USER5> .

<comment><AUTHOR> You don't have to delete your comment, it's nothing to be ashamed of. The angular versioning situation is really not that simple.

And you are right, the ReadMe likely should say Angular 5 and not Angular 2.

<comment><USER5> For some reason I am getting downvoted quite heavily for it so I thought deleting it might stop that. But apparently now it is just getting more downvotes. Oh well.

<comment><USER6> Why do you care?

<comment><USER5> I don't know about you but I do feel really bad when I get downvoted, especially when I don't understand why and have no way to find out.

<comment><USER7> I wouldn't worry so much about it. This is an opinionated group, and having a different opinion is sometimes not taken well, but it is good to challenge the status quo, and good to not be part of groupthink. If you stand by what you said, take your downvotes with pride.

On the other hand, claiming something that isn't true, or being flippant or not contributing to the conversation will also result in downvotes. And those are probably a good guide of how to adjust your communication style to work better within the group.

I have no idea what your original comment was, so I don't know which of those scenarios is the case here.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> Stuff like this is really useful, although does miss some of the key benefits of Lambda-like services... the connection between billing and usage on a per- function basis, the ability to monetise your functions, the ability to scale genuinely on demand rather than with the lead time required to spin up new containers or VMs.

<comment><AUTHOR> Some of that is coming, but right now it's only 3 of us who are building this. It also doesn't aim to be a carbon copy of anything - we have a bunch of ideas we would like to explore ourselves (we think it's better for the community this way).

<comment><USER9> Good stuff in GP list... plus I'm looking for rate limiting to curb abuse (in your service offering).

</comment></comment></comment><comment><USER10> Also checkout Fission ([https://github.com/fission/fission](https://github.com/fission/fission)). A Kubernetes based FaaS platform in which you can define functions in any language and link them together using a DSL.

<comment><USER11> Also OpenWhisk, Kubeless, Fn, OpenFaaS, Riff (I work on this) and probably a dozen others by now that I've forgotten.

<comment><USER9> Is there a list of hosted FaaS offerings somewhere?

<comment><USER12> [https://github.com/anaibol/awesome-serverless#hosting-and- co...](https://github.com/anaibol/awesome-serverless#hosting-and-code- execution-in-the-cloud)

</comment></comment></comment></comment><comment><USER13> Everyone here should check out [https://github.com/apex/up..](https://github.com/apex/up..). No exaggeration: it's the serverless platform we've all been dreaming of.

Also, it is VERY easy to migrate to for many projects, and has great language support... and it's developed by TJ Holowaychuk.

<comment><USER14> Working link: [https://github.com/apex/up](https://github.com/apex/up) Docs: [https://up.docs.apex.sh/](https://up.docs.apex.sh/)

</comment></comment><comment><USER15> Shame it doesn't support Python nor PostgreSQL

<comment><AUTHOR> It's coming as it's rather easy to add support for running Python apps. However, we don't have the python know how to implement automatic Python client generation for services.

</comment></comment><comment><USER16> Checkout SanFran - Lambda like serverless functions platform for Kubernetes [https://github.com/dosco/sanfran](https://github.com/dosco/sanfran)

The main focus of its design was fast function startup time and horizontal scaling.

Fast function startup is done with a combination of pool of warm containers and instant container recycling.

As for horizontal scaling functions can themselves scale horizontally in high QPS situations and for large deployments every micro-service that makes up SanFran can itself be scaled horizontally.

Currently working on using machine learning to predictions to manage warm container pools.

<comment><USER17> eh... why should I need a "serverless functions platform for Kubernetes" especially since I actually need servers for kubernetes. and I can probably do most stuff way easier than having a framework on top of another deployment/app scheduling framework.

<comment><USER16> A serverless functions platform like SanFran can be deployed on your existing Kubernetes cluster. It will sit besides all your existing apps on there using resources only when needed. You can have many more services (functions) running / deployed than the size of your cluster.

It can be a way for anyone within the organization to deploy a function they need. For example marketing needs a quick endpoint to handle Mailchimp registrations or for Stripe payments. The function will not take up resources on the cluster until its needed. Functions will run in a isolated, devops free environment.

Technically you're not running on top of another framework, your javascript function is just running in a NodeJS instance thats managed there are no extra layers. Your NodeJS instance is running on Kubernetes but then why do I need Kubernetes is an entirely different discussion.

</comment></comment></comment><comment><USER18> Great for people stuck with some ancient hosting provider or on prem solution. Not say great for cost as literally the first you need to run this is serverless solution is...a server! Still super nice job.

<comment><USER19> I'm cringing so hard at the idea of a tech guy somewhere explaining to his boss that "we can't go serverless because we don't have enough servers for that". You know that conversation _is_ happening in an office somewhere.

Godspeed, poor engineer.

</comment></comment><comment><USER20> What does "GitHub-like" mean?

<comment><AUTHOR> It means we want to build a similarly accessible platform for microservices/faas as what github did with git repos. We use them as basis of comparison because we have a lot of respect for their product/ideas and we are inspired by them a lot (well sometimes we just simply steal ideas from them, haha).

I guess we are trying to convey the idea that it's a complete product from user management to charging - you could install it for your company or launch a web hosting company with it.

<comment><USER21> Interesting launch. Last month for Producthunt hackathon I was planning to create something similar by using openfaas, but did not(just like many other ideas/side projects we all have here :-)). One questions that bothered me a lot was how to define pricing for such product?

Would you be able to throw some light on how you/your team came up with the sustainable plan and pricing you have?

<comment><AUTHOR> Hi Sharma!

The basic idea in terms of pricing was low cost and low maintenance for low traffic things: when someone is not running anything substantial even 10 quid a month feels like a waste.

And then when you start to have multiple side projects, you might need multiple servers (or set up more complex things like port forwarding based on domain etc)

So that's how it came about. But we are sure it's not sustainable entirely yet! We would just like to provide low cost and low hassle hosting. Let's see how it goes.

</comment></comment></comment></comment><comment><USER22> You should also check out [https://webtask.io](https://webtask.io) by the Auth0 people.

They have an in browser code editor that's fantastic.

<comment><USER3> Webtask is a solid and well designed product. Kudos to Tomasz Janczuk and his team at Auth0.

The only thing I don't like about Webtask is that it's not an open-source project, so there is no way for me to run their backend on my own hardware ( like a raspberry pi ).

</comment></comment><comment><USER23> FWIW: I tried to register and couldn't. The red error box appeared in bottom right corner, but was blank so I don't know what failed. Nothing crazy in my submission that I can see. Even when inspecting error box, no indication of why it failed was apparent.

Maybe it's because you're on the frontpage and getting lots of requests?

<comment><AUTHOR> It's very likely because of the HN hug! We will have a look and fix it, sorry.

Good news is perhaps on your own machine it will run flawlessly :)))

</comment></comment><comment><USER24> I am currently looking for something along the lines of a distributed CPU - <USER2> Amazon Lambda, but enabling me to run sets of interdependent computation steps "in the cloud". Also - supporting redundancy and replication (so that a running program would never crash if one of the servers it was running on has crashed).

Any ideas?

<comment><USER16> I've used Google App Engine + Cloud TaskQueues to build pretty solid pipelines.

</comment></comment><comment><USER25> Fantastic to see open source alternatives. At a previous job, I was repeatedly frustrated using AWS lambda's extremely closed-source environment.</comment>


No One Is Supporting the Doctors - <AUTHOR> https://www.theatlantic.com/ideas/archive/2020/04/doctors-already-manage-alone/610249/ <comment><AUTHOR> This piece from Slate is also good: [https://slate.com/technology/2020/04/nyc- emergency-log-coron...](https://slate.com/technology/2020/04/nyc-emergency- log-coronavirus.html)

 _Less great: Two more of my residents are sick. COVID, of course. Even without a test, we know. Their symptoms are familiar. My first impulse is to be frustrated. Didn't I tell them to keep the mask on? Didn't I say to wipe down the phone, or to not eat at work? I hate that impulse. It's the same one that leads us to ask, "What were their underlying medical conditions?" when we hear about a young person dying. If someone was careless, that's why they got sick. Therefore, if I am not careless, I won't get sick. If someone had underlying conditions, that's why they died. I don't have that condition, so I won't die. It's comforting, maybe, to blame a victim. It gives a sense of control. But it's not true._

 _My residents worked so hard to stay healthy. I know that. I wonder if I had gotten more PPE sooner, would that have made a difference? Maybe it really is just a matter of time till we all have it. Maybe it will be a different kind of relief once that's happened._</comment>


Review My App: Goggles -- Graffiti for the Web - <AUTHOR> http://goggles.sneakygcr.net/

<comment><AUTHOR> Hi!

Hosted on the cheapest prgmr server available, so sorry if it doesn't last.

My tribune to _why's hoodwink'd project, Goggles is a bookmarklet that lets you scribble on any webpage you like. Drag bookmarklet to bookmarklet bar, click, enjoy.

Left mouse = draw. Right mouse held down = erase (really stupid logic but it works well enough).

Hosted on nodeJS hiding behind nginx. We use long JSONP polling but I want to switch to something more dignified like socket.io for that.

Please don't draw rude pictures -- this is my final for art class. I'll restore it to yesterday's backup when I present.

Source: <http://github.com/gcr/goggles>. BSD licensed; feel free to fork away!

It's a really stupid bandwidth hog right now; all points are sent to the server when you draw a shape and all points are sent when you delete. This is more of a proof-of-concept than anything. Let me know what you think!

Tested in firefox 3.6 and Chrome. I've been told it doesn't work in Fx 4.

Who knows how long this will last before the vandals arrive...

For future work: bandwidth savings, vandalism mitigation, different pen sizes (it's there but no UI for it).

</comment><comment><USER2> hi, nice app ;) i found little bug - if u draw and u drag mouse out of window and return to window app will draw without need to have left mouse button pressed:) tested it on chrome

<comment><AUTHOR> Thanks.

That happens because the canvas' onmouseup event never fires (it landed on another window). Does anyone know how to fix?

<comment><USER2> u could check onmousemove function if button is pressed... ;)</comment></comment></comment>


N95-Mask Donation Locations - <AUTHOR> https://findthemasks.com/ <comment><USER2> Thanks for making this, I'll ask my wife to fill out.

Between, does anyone know where small independent medical practices can get the N95 masks, or any other masks?

My wife is a primary care who does medicare home visits in Texas and she doesn't have any masks.

There are a lot of doctors who would gladly pay.

What even more scarier are the patients who don't take this seriously. A colleague's wife had a patient 9 days ago who neglected to mention travel to Italy, California and is now diagnosed with COVID-19. The travel history came out yesterday after the patient's condition got worse.

</comment><comment><USER3> Sorry, this is bullshit. Government needs to contract 3m and the like to make as many masks as they can, as fast as they can, ASAP.

They needed to have done this months ago. Instead of running a propaganda campaign that masks are supposedly not effective against coronavirus and that we should not buy them anyway because we need to save them for the hospital workers who really need them. Completely disingenuous.

It's beyond frustrating to be told to conserve face masks in the face of a pandemic rather than to purchase contracts to ensure companies produce what we need.

China is currently making 200 million masks a day. In addition, China and South Korea attribute much of their success at containment of coronavirus to the use of masks.

Making masks available for all (like South Korea who is giving every citizen and resident an allotment of 2 face masks/week) + a mandate for all to wear face masks would do so much to curb this pandemic. But instead we're running this crazy experiment of in essence shutting down our economy via mandatory business and company closures. Not thinking through the real long term damage and consequences of such actions. Unemployment. Social unrest. Destruction of business.

Some more food for thought. Jack Ma was able to secure and send 1 million face masks to the US already. If government won't act, private citizens and foundations (I'm looking at you Bill Gates Foundation) can.

<comment><USER4> That'd be fantastic in a week or a month. ICUs in Washington and New York are running low on supply as we speak, any help now could buy a medical professional in these areas precious days.

<comment><USER3> That's why they should have done this months ago. Still have yet to do so as far as I know.

<comment><USER5> crying over spilt milk won't help right now.

</comment></comment></comment></comment><comment><USER6> > _Don't make us orphan our two toddlers._

I mean, if that were to happen it would be more due to the parents' altruistic choice and less about the actions of others, no? They've made the socially right choice but they can't blame the consequences on someone else.</comment>


Show HN: Jottedly, track your job search in one place - <AUTHOR> http://jottedly.com/ I've been learning how to code, and I'm building Jottedly as a personal project to help people track their job search.  Welcome any feedback!  Thanks.  -rob <comment><AUTHOR> Hi - I'm currently learning how to code, and I'm building this web application to familiarize myself with various tools and languages (Python, Flask, MySQL, JQuery, Twitter Bootstrap, etc.). If you're looking for an application to help you manage your job search, I'd welcome you to try it out, and look forward to your feedback. Thanks! - Rob</comment>


Workers are ghosting their employers - <AUTHOR> https://www.washingtonpost.com/business/2018/12/12/workers-are-ghosting-their-employers-like-bad-dates/ <comment><USER2> In my opinion, it's happening due to low unemployment rates AND low wages. Wage growth has been weak, and that has a lot to do with the decline in labor- force participation.

In addition to that, there's certainly been a cultural shift in the United States that I personally think is great. Workplaces are more casual and bosses can't necessarily demand undying support from their employees anymore.

I think the message is clear-- treat employees fairly, or they will ghost you.

If employees stil ghost you, then well, "It's Just Business, Nothing Personal."</comment>


Ask HN: Review my prototype - http://cluud.in - real time local search for SF - <AUTHOR>

To scratch an itch, I built this prototype (http://cluud.in) for a local search system that leverages twitter. It allows you to follow "locations", rather than trends or users, like twitter and other services currently offer. What do you think - the design, idea et al? and what would make it more useful and make you come back to use it? <comment><AUTHOR> Oh and its pronounced "Clued in"! Cheers. PS. Direct link: <http://cluud.in>

<comment><USER2> Am I the only one who completely discounts a .in domain as being spam from India? I certainly would not have ever clicked this link outside of an "Ask HN" style post b/c I would have expected it to be as such.

<comment><AUTHOR> Hmm, not necessarily. Something like .us was made popular by delicious, but there's still outside.in and other well named sites that use the postfix "in" as a verb.. something like justin.tv not being spam from Tuvalu :)

<comment><USER2> I think you perhaps I wasn't clear: I'm not in any way suggesting that, because it is from India, it is automatically spam. I am suggesting that, in my experience, the overwhelming majority of .in websites that I've been to, been sent to, or seen are either (a) nefarious places, (b) content theives (i.e. screen scrapers), or crappy blog sites who do nothing but re-post the same "funny pic!" over and over.

I mean seriously - if you saw a link today for, "imbored.ru", would you (a) chuckle at how clever the name is and then click the link, or (b) immediately think, "It's either content theft or a nefarious site"? It's just what it is.

I could make the argument that it's not that .us was made popular by delicio.us; it's that other companies haven't made the .us domains "seedy" (yet?).

<comment><AUTHOR> I see what you mean.. In fact, I wasn't actually aware that the proliferation of .in sites was so large that they might be termed spammy/seedy.. I for one have never actually seen the seedier ones!

</comment></comment></comment></comment></comment><comment><USER3> Neat. Where'd you get the location definitions? Yelp?

</comment><comment><USER4> PHP and mootools? Not into Django/jQuery? :)

<comment><AUTHOR> What gives you the impression its the former? :) Its actually Django with Prototype..</comment></comment>


The Gray-1, a homebrew CPU exclusively composed of memory - <AUTHOR> http://2x-1.net/ob/gray1/ <comment><USER2> Ah, memories of one of my very first lab assignments in digital circuits engineering class (back when computer science was still busy splitting off from applied mathematics and electrical engineering). We were given nothing but a breadboard, some chips with basic logic gates (NANDs, multiplexers/demultiplexers), an EPROM, access to an EPROM burner (w/ hex keypad!), some LEDs, switches, resistors, jumper wires and a power source.

The assignment was to build a traffic light simulator, set the whole thing running, change the traffic lights as a result of switch inputs acting as sensors and a simulated interval timer.

Some students were baffled by this (lectures hadn't caught up with lab assignments at that point): how could you build a small processor using only logic and an EPROM? There's no memory or registers to keep state!

This is what differentiates combinatorial logic from sequential logic: feedback. Use some of the EPROM's data outputs along with logic gates and switch outputs (using the multiplexers / demultplexers) as address inputs to the same EPROM.

Sweet memories of solving Karnaugh maps, Quine-McCluskey minimization, logic hazard mitigation, etc. Good times.

<comment><USER3> I took a similar class. I loved it. we started out with a project that would make leds perform a cylon style pattern [1]

For the final project we had to implement a multiplier (booths algorithm) using nothing except breadboards, wires, and a few very basic ICs that had Ands nors etc.

I loved that class and it really helped me understand how a processor actually works.

[1][https://68.media.tumblr.com/46f8f8a9ffd3f4bf900e1710c4e08fd0...](https://68.media.tumblr.com/46f8f8a9ffd3f4bf900e1710c4e08fd0/tumblr_inline_o0ceh739Xm1ru1o4q_500.gif)

<comment><USER4> > I loved it

You misremember ;)

I took a similar class, and like for everybody else I ever talked to, half of the wires were somewhat broken and caused intermittent connections. Solving the puzzle of building the system was fun, and quickly done - finding the &#(@ broken cables and replacing them was weeks of tedium.

I suppose in a way it was perfect preparation for a career in software engineering. Solve the core problem, and then spend significantly more time on actually making it work in a world the adamantly refuses to be pure and perfect.

<comment><USER5> My solution? Dumpster diving for wire.

</comment></comment></comment></comment><comment><USER6> Tangential to the article itself: webrings! I forgot that those once existed!

Homebuilt CPUs webring's home: [http://members.iinet.net.au/~daveb/simplex/ringhome.html](http://members.iinet.net.au/~daveb/simplex/ringhome.html).

<comment><USER7> If search engines had not gotten better as fast as they did I suspect webrings would have been a much larger thing. Who knows maybe one day we'll see a revival of webrings as the ultimate anti-spam tool, monitor your neighbours in the ring and cut them out of the ring if they go spammy. It would scale.

</comment></comment><comment><USER8> This is a great project, but the idea is not as exotic as it may sound. This is more or less how an FPGA works.

Edit: Should also add that there were commercial products built like this in the 80-90's.

<comment><USER9> The lookup table part, sure, many fpgas use that. The clever part of this was turning a ROM into a flip flop.

<comment><USER8> Fpga and co have not always come with registers. Back in the days this was more or less how it was done.

<comment><USER10> Are there any particular counterexamples you have in mind? The first reprogrammable logic device, the Altera EP300 [1], included a register in each macrocell. So did the first FPGA, the Xilinx XC2064 [2]. I'd be very surprised if there were any FPGAs with no registers.

[1]: [http://www- inst.eecs.berkeley.edu/~cs294-59/fa10/resources/A...](http://www- inst.eecs.berkeley.edu/~cs294-59/fa10/resources/Altera- history/altera_ep300.pdf)

[2]: [http://www.applelogic.org/files/XILINXXC2018LCA.pdf](http://www.applelogic.org/files/XILINXXC2018LCA.pdf)

</comment></comment></comment></comment><comment><USER11> I'm in the process of writing a tutorial on this/hardware synthesis from first principles, but general purpose computers are way less magical when you think of components such the ALU as a mapping of binary inputs to binary outputs: a truth table. From there you can explicitly enumerate all possible inputs and outputs (for small word sizes at least) and go directly from said truth table to (inefficient) hardware.

Related: mov is Turing-complete [1] and X86 MMU fault handling is Turing- complete [2]

edit: formatting

[1] [https://www.cl.cam.ac.uk/~sd601/papers/mov.pdf](https://www.cl.cam.ac.uk/~sd601/papers/mov.pdf)

[2] [https://news.ycombinator.com/item?id=5261598](https://news.ycombinator.com/item?id=5261598)

<comment><USER12> I suspect it would be difficult (impossible?) without relative position addressing modes and/or other tricks. A conventional stack would need some way to access an offset away from the base, but perhaps you could use self- modifying code. Perhaps relative from current position like brainf __k.

All general computation needs either a direct or convoluted means of achieving conditions too. A tricky formula exploiting abs(x) could be used to produce {0, 1} but...

<comment><USER11> Right, my comment retarding truth tables was only in reference to combinatorial logic. That said, pure circuit models of computations exist, and are interesting from a theoretical standpoint at least. They don't translate to hardware, though. As you noted doing arbitrary computation with (finite) circuits requires some type of feedback/sequential logic.

</comment></comment></comment><comment><USER13> It's funny I was wondering recently if there is a fundamental difference between logic and memory. I mean it's obvious that memory devices can do logic, since after all Boolean tables are just data in the end, aren't they? And reciprocally transistors can be used to store memory (with a flip-flop for instance).

So is it true? Are logic and memory fundamentally equivalent?

<comment><USER14> Theoretically yes. One is stateful and the other is not, but you can always simulate one with the other. Practically no. DRAM is built with capacitors as its storage elements. Each bit has no feedback and requires refresh, which makes it slow. But it only requires one transistor per bit, so it's cheap and every bit takes up very little physical space. Static RAM, OTOH, (registers and fast cache) is built out of flip-flops. It has feedback so it doesn't need refreshing and it's fast. But every bit requires 4 or more transistors, so it's physically large per bit and expensive.

<comment><USER15> Can you elaborate on this:

>"Static RAM, OTOH, (registers and fast cache) is built out of flip-flops. It has feedback so it doesn't need refreshing and it's fast"

DRAM also has feedback no? I mean thats what differentiates is from combinatorial circuits correct?

I understand SRAM is flip flops vs single bit transistors. I guess I'm not sure why you are saying it has feedback to differentiate SRAM(registers) from DRAM.

<comment><USER14> The individual bits of DRAM don't use internal feedback; they work by storing charge on a capacitor that slowly leaks away. Their outputs don't help them maintain their state (except during a refresh cycle, and in that sense one could say they use feedback). The flip-flops in SRAM always have internal feedback that causes them to a) switch state quickly and b) maintain their state after switching. This is an example of _positive_ feedback, as opposed to the more well-known negative feedback used in control systems. I was trying to say that there's more than one way to save state, and it's state that distinguishes memory from combinational logic, not feedback. [https://en.m.wikipedia.org/wiki/Memory_cell_(binary)](https://en.m.wikipedia.org/wiki/Memory_cell_\(binary\))

<comment><USER15> I see, sure, that makes perfect sense. Thanks for clarifying.

</comment></comment></comment></comment></comment><comment><USER16> The scroll highjacking is making it hard for me to properly read the article on Google Chrome.

<comment><USER17> +1. I cannot bear it and can only close the page.

</comment></comment><comment><USER18> You can implement any logic function with a large enough lookup table. That doesn't mean it's necessarily a good idea.

It wasn't done this way then and isn't done this way now not because it's impossible (or even that hard), it's because it usually doesn't make sense compared to the alternatives. It's slow and burns a comparatively large amount of resources.

That said, the project is interesting precisely for the reasons that it renders these lessons concrete, and apparently was fun to implement.

...now back to my Minecraft RISC-V....

<comment><USER19> I'm actually gearing up for a RISC-V processor in Fallout 4. All of the design work is done, but building it with the vanilla logic gates (introduced in the Contraptions DLC) actually isn't possible. Luckily I already had a mod with "real" components when Contraptions was announced.

Assuming you weren't joking, how sophisticated is your design? (as in, "is it pipelined?")

<comment><USER18> I was joking :-)

I've made a few simple microcontrollers and microsequencers in Minecraft (as 'micro' as you can get with a 1M feature-size) and I've often fantasized about bigger projects, but it's only been fantasy.

Still, it lends itself reasonably well to old NMOS-style layouts...

</comment></comment></comment><comment><USER20> An interesting project. I did see one error in his description of how a register works. What he describes is the operation of a latch. A register does not change outputs when the clock goes low. It only changes on the rising edge of the clock. See the datasheets for 74373 vs 74374 to see the difference.

</comment><comment><USER15> This is really neat. I have question, how hard and what is the procedure for writing the ISA and subsequent microcode to implement your ISA? Do you do this in Verilog or an equivalent? And how about writing the assembler? I would love to hear more about what that entails in a homebrew set up like this.

<comment><USER19> Actually designing an ISA is voodoo, but doing the mircrocode is fairly easy. On a tiny little 4-bit design with only a few control lines you can do it by hand. Past a certain point, though, it is essentially impossible to do manually without going insane.

It isn't very sophisticated, but the project I mentioned above might satisfy some of your curiosity: [https://hackaday.io/project/18859-risk- vee](https://hackaday.io/project/18859-risk-vee) / [https://github.com/cadpnq/risk-vee](https://github.com/cadpnq/risk-vee)

My methodology was to define each microinstruction that made up every instruction (such as "move pc to memory address register"). I then took all of the control lines in my design and assigned them a bit position in the control store. From there it was just a matter of defining each control line symbolically and "assembling" each microinstruction by ORing the appropriate things together.

<comment><USER15> This looks great! I am going to follow your project. What emulator are you using to design this? I see all the assembler files are js?

Can you elaborate on this? From there it was just a matter of defining each control line symbolically and "assembling" each microinstruction by ORing the appropriate things together"?

This is basically your decoder then?

<comment><USER19> So far everything is in Logisim[1], but I'm actually thinking of writing an emulator so I can debug programs quicker. I started a lisp interpreter in risc-v assembly, but testing my code at 3khz (the max logisim will run my design at) was too painful. The assembler and microcode generator are js+node.

risk-vee is a really basic (read: dumb) "common bus" design. All of the internal components of the CPU share the same 32-bit data bus. The heart of the control unit is a ROM that is 32-bits wide. Each bit is connected to one of the various things in the CPU. To make generating the microcode easier I defined a mask for each line.

Taking "move the PC to the memory address register" as an example: The register that holds the PC has an output enable line, and the memory address register has select/enable and both would need to be high. Pulling a few lines from microcode.js:

      pc_read =           parseBin("0000-0000 0000-0000 0000-0000 0100-0000");
      mem_address_write = parseBin("0000-0000 0000-1000 0000-0000 0000-0000");

So the microinstruction we want ends up being

      pc_read | mem_address_write

I feel like I'm simultaneously under- and over-explaining it.

[1] [http://www.cburch.com/logisim/](http://www.cburch.com/logisim/)

<comment><USER15> I understand. Thanks for the explanation. I didn't know about Logism, this is cool. I can't wait to play with this. Thanks for the link!

</comment></comment></comment></comment></comment><comment><USER21> I've heard rumours that HP was developing something known as memgisters; where instead of moving data from memory to a CPU register - operations could be preformed directly on memory.

</comment><comment><USER22> Would be interesting to see how this would translate into a memristor-based CPU in the future.</comment>


Ask HN: How is "offshore" banking affected by blockchain? - <AUTHOR> <comment><AUTHOR> If people have money in "tax havens" couldn't it make sense to move some of that to the blockchain to become "extra jurisdictional"</comment>