Keep Your Stuff, for Life - <AUTHOR> https://perkeep.org/ <comment><USER2> Wow, it was really hard to find out what this software does. I finally found a demo at the end of an hour-long talk from 2 years ago: [https://youtu.be/PlAU_da_U4s?t=2687](https://youtu.be/PlAU_da_U4s?t=2687)

So it seems to have a bunch of scripts to import data into its database from a variety of sources (including cloud services), and provides a search interface to navigate that historical data. And it has a lot of under-the-hood stuff about replication, and it's entirely self-hosted.

<comment><USER3> Does the first sentence not cover that fairly well?

> Perkeep (née Camlistore) is a set of open source formats, protocols, and > software for modeling, storing, searching, sharing and synchronizing data in > the post-PC era. Data may be files or objects, tweets or 5TB videos, and you > can access it via a phone, browser or FUSE filesystem.

I checked archive.org and that text has been there for a couple of months at least. Looks interesting, I've been in the market for this kind of self hosted backup/replication/tagging/search thing.

<comment><USER4> > Does the first sentence not cover that fairly well?

It does not.

The first sentence on the site ("Perkeep [...] is a set of open source formats...") describes literally what the thing is, but not at all what it _does_.

Not to slam on these cats, because marketing copy is _hard_. For project collaborators, or open-source dorks who live in this kind of world anyway, the sentence on the homepage is probably perfectly descriptive.

But I agree with my GP post. Reading the homepage I had no idea what Perkeep actually did.

<comment><USER3> What it _does_ is:

> modeling, storing, searching, sharing and synchronizing [...] files or > objects, tweets or 5TB videos, and you can access it via a phone, browser or > FUSE filesystem

I mean maybe it could have been more explicit or they could have added more detail, but having this as the first sentence is WAY better than most of the 'professional' landing pages for startups that get posted here. 'Harmonizes synergy and increases your ability to wow your target space with your aspirations', now _that 's_ meaningless.

<comment><USER5> Them: "The project description isn't clear to me."

You: "Well I'm sorry it wasn't clear to you but it was clear to me and better than these other things and here's why it should have been clear to you."

If someone tells you something is unclear to them, arguing about it doesn't change the fact that it wasn't clear to them.

</comment></comment></comment></comment></comment><comment><USER6> Perkeep is super interesting. I have been looking at it as a media database (photos, movies and such, powerful tag-based search, all downloaded on demand), but some things regarding that really hurt:

\- The data store messes with my files (yes, there's a FUSE mount, but eh, having to adopt a special data store always makes me feel weird since it usually comes with performance and compatiiblity implications, there are also many other block-chopping data stores, for example in IPFS).

\- The last time I checked there was no way to delete something. This is okay for Tweets I guess, but if I commit a 3h video I later realize to just be too large or a photo I end up not really wanting around - well, oops.

I have huge respect for Brad Fitzpatrick in general, of course, and especially for creating this. Recent velocity has seemilgly been relatively low, however: [https://news.ycombinator.com/item?id=22161812](https://news.ycombinator.com/item?id=22161812)

<comment><USER7> > ... my time is tempered by 2.5 and 0.5 year old kids. ... I'll pick up my > involvement again as kids get a bit older.

> We have no plans to abandon it.

</comment></comment><comment><USER8> It sounds like this preserves your data from lots of different services. This is something I need! But I couldn't figure out what it actually supports. Suggestion: the very first paragraph should describe the specific inputs it can handle.

<comment><USER9> For the most part, it's just an object storage (think Amazon S3). Content addressable (think Git): you put an object (file bytes) in, and you can get it out by its hash, that's it.

There are some bits (permanodes and claims) for adding metadata to objects (filename, timestamp, geo location and other attributes, I think even arbitrary jsons) and for authentication/sharing. A few really cool bits around modularity: blob servers can be composed over network - you can transparently split your blob storage over multiple machines, databases, cloud services, set up replication, maybe encryption (unclear to me if it works or not).

Importing data from different services is not really its core competency, at least not yet. It can ingest anything you can put on your file system and there are importers for a few third-party services (see [https://github.com/perkeep/perkeep/tree/master/pkg/importer](https://github.com/perkeep/perkeep/tree/master/pkg/importer) ), but that's about it

<comment><USER10> Thank you so much for a description of what it _actually_ does which the website seems to struggle so much to convey.

One thing that I'm still trying to figure out is, if you do happen to know: how does it handle data deduplication (if at all)? How about redundancy and backups? I've been glancing over the docs and I do see mention of replication to another Perkeep instance but that's not quite what I'm looking for.

<comment><USER9> Deduplication is naturally handled by content-addressable property of this object store: the address of each object is its cryptographic hash, SHA224 in Perkeep. So if you try to put a duplicate copy, you'll find that the address at which you try to put it at is already occupied by the first copy. Perkeep assumes that you never delete anything (deletion is even simply not implemented, not even for garbage collection/compaction purposes), so if you see that one copy of an object was already put, you can discard any further puts as no-ops.

Then there is also some logic to chunk large objects into small pieces or "blobs". These small chunks are actually what the storage layer works with, rather than with the original unlimited-length blobs that the user uploaded. Chunking helps to space-efficiently store multiple versions of same large file (say, a large VM image) - the system only needs to store the set of unique chunks, which can be much smaller than N full but slightly-different copies of the same file. But I personally I find that it deteriorates its performance to the point of making it unusable for my use case of multi-TB multi-million- files storage of immutable media files. If chunking/snapshotting/versioning is important for your use case, I'd look more towards backup-flavored tools like restic, which share many of these storage ideas with Perkeep.

Redundancy and backup is handled by configuring storage layer ("blobserver") to do it. Perkeep's blobservers are composable - you can have leaf servers storing your blobs, say, directly in a local filesystem directory, remote server over sftp, or an S3 bucket, and you can compose them using special virtual blobserver implementations into bigger and more powerful systems. One such virtual blobserver is [https://github.com/perkeep/perkeep/blob/master/pkg/blobserve...](https://github.com/perkeep/perkeep/blob/master/pkg/blobserver/replica/replica.go) \- which takes addresses of 2+ other blobservers and replicates your reads and writes to them.

<comment><USER9> Backup as in backing up one perkeep instance to another is the "pk sync" command ([https://github.com/perkeep/perkeep/blob/master/cmd/pk/sync.g...](https://github.com/perkeep/perkeep/blob/master/cmd/pk/sync.go)).

You give it the addresses of source and destination blobservers, it enumerates blobs in both, and copies the source blobs missing from destination into the destination server.

</comment></comment></comment></comment></comment><comment><USER11> I admire the goal of reliable long term data storage. I don't need to store tweets or 5TB videos, but my current solution is duplicate DVDs with a bunch of par2 files to hopefully ward off bit rot.

I feel like there's some room for improvement.

<comment><USER12> You can buy a 2tb drive for $100. $200 and you've got 430 DVD's worth of data redundantly stored. $300 and you've got local redundancy AND offsite backup.

<comment><USER13> Even if magnetically they don't have 'bit rot', they use bearings where the lubrication can dry up and wear out when they're not turning for long periods of time.

You need to keep them spinning on a regular basis, and replace them as they begin to fail.

<comment><USER14> HDDs are also prone to silent bitrot, where it will simply return incorrect bytes for a sector, even without any smart errors. (Optical disks also bitrot; but so does HDDs).

This is usually a precursor to SMART errors happening in the near future, but unfortunately, it can still result in corrupted replication and corrupted backups; as your backups would be backing up the rotten (corrupt) data.

I've witnessed this happen on both Seagate and WD drives, on systems with ECC memory. I can only suspect this is due to HDD manufacturers wanting to reduce their error rates, and RMA rates: it may happen with the ECC bits in a sector is corrupt, making bitrot undetectable. Instead of giving an error (and being grounds for a RMA replacement), the HDD firmware may choose to return non- integrity-checked data; which would usually be correct but also could be corrupt.

It's why filesystems like ZFS and btrfs are so important.

My rough estimation of this, based on my own experiences and those on r/DataHoarder, suggests 1 hardware sector (4KB for most drives post 2011) will _silently_ corrupt per 10TB year. Such corruption can be detected via checksumming disc systems like ZFS.

Usually, the whole sector is garbage, which is not indicative of cosmic ray bitflips.

External flash memory storage like USB sticks and SD cards fare far worse. In my own experience, silent corruption occurs more like 1 file per device, per 2-3 years; irrespective of the size of memory. I've had USB sticks and SD cards return bogus data without errors, so often. I only know because I checksum everything, otherwise I would have thought the artefacts in my videos or photos came with the source.

If, in 2020, you are not using ZFS or btrfs for long term archival, you are doing something wrong.

ext4, NTFS, APFS, etc may be tried and tested, but they have no checksumming, and that is a problem.

<comment><USER15> Interestingly, on my home ZFS raidz with 3 4TB hard drives, I have had to replace a drive a couple of times because ZFS scrub was reporting silent corruption. They were consumer-grade SATA drives.

However, at work, I have backed up ~200TB of data to a large server with RAID-6 and ext4, storing the backups as large .tar files with par2 checksums and recovery data, and regularly scrubbing the par2 data. I have yet to see any corruption whatsoever. These are enterprise-grade hard drives. This is the strongest evidence I have yet seen that the enterprise-grade drives are actually better than the consumer-grade ones, rather than just being re- badged.

<comment><USER16> Thanks. What are the drives at your workplace?

<comment><USER15> I actually have no idea. I didn't have any part in purchasing that particular system, I don't have root, and all the drives are hidden behind a RAID controller. Sorry.

<comment><USER17> How do you know they are enterprise drives then?

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER18> How is this better than a plain old filesystem with bitrot compensation? What does this have that my btrfs or ZFS filesystem (with parity) doesn't?

</comment><comment><USER19> Does anyone know how mature Perkeep is? Is anyone using it regularly? Would love to hear if there is anyone who has experience with it.

<comment><USER9> Very immature, just have a look at its extensive absent documentation. The best bit that describes the state of things: "If you're a programmer or fairly technical, you can probably get it up and running and get some utility out of it".

So much to show off for 7 years of development, I'm pretty skeptical of its future. But some of the ideas are pretty cool, like composable blob servers.

</comment></comment><comment><USER20> past discussion, [https://news.ycombinator.com/item?id=18008240](https://news.ycombinator.com/item?id=18008240)

</comment><comment><USER21> There's a blockchain startup called arweave which is trying to do exactly this.

</comment><comment><USER9> I tried perkeep a while ago. While the ideas are cool, the implementation is meh:

I added a single 2.7G ubuntu iso - it took 5 minutes to ingest it (on a tmpfs!), and turned it into 45k(!) little chunks, wtf is up with that? At this rate indexing my multiple terabytes of data is going to take it days and I don't even want to think how much seek time it's going to need to store its repo on a spinning HDD.

<comment><USER22> > I added a single 2.7G ubuntu iso - it took 5 minutes to ingest it (on a > tmpfs!), and turned it into 45k(!) little chunks, wtf is up with that?

Ingest times correlate linearly with file sizes because it needs to compute the blobref (which is a configurable hash) for all the blobs (chunks as you call them). Splitting in blobs/chunks is necessary because a stated goal of the project is to have snapshots by default when modifications are done. Doing snapshots/versioning without chunking would be very inefficient.

<comment><USER9> Reading the docs, snapshotting/versioning doesn't strike me as a major feature of perkeep. It's more important and appropriate in the domain of backup software (e.g. restic/attic/borg) and where you'd want it together with delete functionality to reclaim space.

But perkeep's focus, as I understand it, is more on managing an unstructured collection of immutable things (e.g. photo archive), rather than being a tool to back up your mutable filesystem. So I'm not sure they made a good design decision to chunk the sh*t out of my files, which really kills the performance on large files and especially on spinning disks.

</comment></comment></comment><comment><USER23> Ah, nostalgia :).

It seems like with the recent wave of news about social media migrations (reddit, facebook, twitter, twitch, tiktok), people are hopefully starting to get more and more warmed up to the idea of protocolization of their social data.

But most of the projects doing it are still just too immature. Solid, Perkeep, Blockstack, etc. just seem like vaporware.

Seems like the only serious projects in use are Matrix, Urbit, and ActivityPub/Mastodon. But I haven't checked in with the decentralization scene in a while.

<comment><USER24> I want that protocolization too, although I don't hold out much hope that the monopolies in place can be broken, outside of fairly radical regulation.

To add to your list, there is also Secure Scuttlebutt [1] which has had a decent userbase over the past few years, and Planetary [2] which is a funded iOS client for it.

I think in general they all suffer from the chicken-egg problem and will need some reason for enough people to switch to be able to build a userbase. There isn't really any "novel hook" like tiktok, twitter, whatsapp, instagram, snapchat, etc have had in the past.

[1] [https://scuttlebutt.nz/](https://scuttlebutt.nz/)

[2] [https://planetary.social/](https://planetary.social/)

<comment><USER25> Man, I love the idea of Scuttlebutt but I hate the developer UX. I'm writing some apps that I wanted to put on SSB but have all but given up on the idea. Something about SSB, as a dev, leaves me with a lot of questions and no idea where to even get answers from.

So I'll write my app outside of SSB, hopefully in a way that's mostly compatible, and possibly with future integration.

I may also toy with an SSB-like protocol myself, as the fundamentals of SSB is a work of art imo. I really enjoy what Gossip brings to the table, and how SSB focuses on human->human relationship to bring P2P to the table.

</comment></comment></comment><comment><USER26> Last released version 0.10 is from May 2018. Is this project still alive? Last commit is from March 11 2020 so maybe they are just "slow" at releasing.

<comment><USER27> Last commit was just a simple bugfix in documentation. The last juicy commits were in december 2019 and [https://github.com/perkeep/perkeep/graphs/code- frequency](https://github.com/perkeep/perkeep/graphs/code-frequency) makes the impression that activity has slowed down in 2019 significantly. Maybe they reached the point of good enough, but the high number of open issues and mergerequests is still problematic.

<comment><USER7> They reached the point of having kids :)

[https://news.ycombinator.com/item?id=22161812](https://news.ycombinator.com/item?id=22161812)

</comment></comment></comment><comment><USER21> There's a blockchain startup called arweave which is trying to do exactly this

</comment><comment><USER28> For those curious "what does this do" and "how does this work" the presentation on the home page (Video + Slides) helps a lot.

</comment><comment> crooked-v It looks like this hasn't had any updates since May 2018.

<comment><USER29> The GitHub page has lots of fairly recent updates: [https://github.com/perkeep/perkeep](https://github.com/perkeep/perkeep)

<comment><USER30> Last commit was in March, mind you. It looks like an interesting project though

<comment><USER31> I imagine the author is pretty busy with Tailscale right now.

<comment><USER32> And kids ;-)

</comment></comment></comment></comment></comment><comment><USER33> That would be nicer to be able to "delete all your stuff, for life" in our era than to keep it

</comment><comment><USER34> Watching the LinuxFest talk now, this looks really neat!

</comment><comment><USER35> exporting/mirror content from clearnet social network to zeronet is also interesting.

For popular content, you'll see it has many seeds, instead of likes

</comment><comment><USER36> How is this different from Upspin?

</comment><comment><USER37> It's like a poor man's version of AWS Glacier</comment>


A personal jet that will fit inside your garage - <AUTHOR> http://www.golfhotelwhiskey.com/a-personal-jet-that-will-fit-inside-your-garage/

<comment><USER2> I've always had a soft spot for the Cri-Cri, billed as the smallest twin- engined 'plane in the world. It's usually powered by chainsaw engines, but someone fitted jet engines to it.

It's fully aerobatic

<http://en.wikipedia.org/wiki/Colomban_Cri-cri>

<http://www.cricri.zoomshare.com/>

[http://www.flixxy.com/worlds-smallest-twin-engine- airplane-c...](http://www.flixxy.com/worlds-smallest-twin-engine-airplane-cri- cri.htm)

<http://www.google.co.uk/#q=%2B%22Cri-Cri%22></comment>


A mysterious radio burst that keeps repeating - <AUTHOR> https://www.sciencealert.com/right-on-schedule-a-repeating-fast-radio-burst-has-woken-up <comment><USER2> Here's the section on FRB 121102 on the "Fast radio burst" Wikipedia page: [https://en.wikipedia.org/wiki/Fast_radio_burst#FRB_121102](https://en.wikipedia.org/wiki/Fast_radio_burst#FRB_121102)

</comment><comment><USER3> " within those milliseconds, they can discharge as much power as hundreds of millions of Suns."

Didn't know that

<comment><USER4> But is it the full power of the entire life of millions of suns or the equivalent of what millions of suns produce in a millisecond?

<comment><USER5> Power is by definition per unit time.

<comment><USER6> Although these bursts are coming from a place where time is rather more spread out than it is here.

<comment><USER7> What does that mean!? Might a millisecond there last as long as a few of our Earth seconds? What is this concept called ( for me to read up on it)?

<comment><USER6> I think this is a good place to have a look : [https://en.wikipedia.org/wiki/Gravitational_time_dilation](https://en.wikipedia.org/wiki/Gravitational_time_dilation)

My assumption is that the place that this burst is coming from is near a very large, very dense mass.

</comment></comment></comment></comment></comment></comment><comment><USER8> If I were trying to let intelligent life elsewhere know that I exist and that I am intelligent I would have to create a patterned noise that distinguishes itself from natural noises that could be formed spontaneously. Talk about a challenge!

<comment><USER9> I believe the standard solution is to send a sequence of signals with durations being consecutive prime numbers (multiplied by some unit of time, but that's irrelevant).

The canonical answer ("Yes, I am very smart too") is to reply with a continuation of the sequence where the transmission stopped.

<comment><USER10> You could also eg convert the sequence from binary into graycode instead of continuing it.

Just any simple transformation that's obviously not background noise.

</comment></comment></comment><comment><USER11> Interstellar numbers station.

<comment><USER12> Better than an interstellar microwave weapon

</comment></comment><comment><USER13> The article was nice, but when I hit the back button , the site scrolled to a suggested article and told me "hey, check this before you go".

No, just let me use my back button.

<comment><USER14> > _Please don 't complain about website formatting, back-button breakage, and > similar annoyances. They're too common to be interesting_

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

<comment><USER15> > Please don't complain about website formatting, back-button breakage, and > similar annoyances. They're too common to be interesting

I partially disagree with this rule. Complaining about crappy websites might be annoying for old timers and experienced techies, but let's say a 22 years old fresh web programmer joins HN to read some news and discovers that the framework/library/tool his PM suggested produces a pile of crappy code that renders every site slow as molasses, and the same product is used on a site getting hard criticism about that; he could actually learn something and ultimately find a better solution for his job, ultimately contributing to make a better web. I know I'm taking it to the extreme, but you get my point. I would rather change that line to something like "keep concise and to the minimum all comments about site formatting, slowness etc. and don't reply to those comments unless you can recomment a better solution".

<comment><USER16> But it _is_ a rule and as such should be respected whether we agree with it or not, for the same reasons we don't ignore the other rules in that list. You can email dang and try to convince him to change it, he's an extremely reasonable person.

</comment></comment></comment></comment><comment><USER17> For me it looks like a source with two rotations, one for the millisecond cycle and the other for the 157-day cycle. Perhaps something is obstructing the source and is rotating around it.

</comment><comment><USER18> Magnetar orbiting a black hole or something.

<comment><USER19> And we know it's a 157 day orbit.

<comment><USER20> Could be a tidal lock of a pulsar around a black hole. We only get the milisecond because the black hole is obscuring the burst pointing to us. Otherwise the beam is not pointed at us.

(P)--->\---- ( BH ) ----> ( Solar System )

</comment></comment></comment><comment><USER21> Reminds Stanislaw Lem's His Master's Voice.

<comment><USER22> I've been reading Lem's work lately but haven't gotten to this one. Do you recommend it?

<comment><USER21> Wrote too long a review :) in short words - yes, I'd recommend it, I like it a lot - and know many people who don't.

</comment></comment></comment><comment><USER23> Check the staff roster to see if the bursts match up with anyone using the downstairs microwave.

<comment><USER24> The background to that reference is [https://en.wikipedia.org/wiki/Peryton_(astronomy)](https://en.wikipedia.org/wiki/Peryton_\(astronomy\))

<comment><USER25> My favorite part of that story IIRC is that it only registered when people opened the microwave before the alarm went off.

I am more patient now to let the microwave complete after hearing about subtleties of the microwave emissions.

<comment><USER23> Me too, I now either wait for it to time out or hit the stop button before opening the door. It's completely irrational and I feel a bit silly for it since I don't live anywhere near a radio observatory, but I still do it.

<comment><USER26> WiFi users in your neighborhood are thankful.

</comment></comment></comment></comment></comment><comment><USER27> So. Is it aliens or not.

<comment><USER28> It's never aliens

<comment><USER29> The radio bursts are alien to us. They come from another place that is not here, therefore, alien.

<comment><USER30> You're trying too hard.

<comment><USER29> Oh not at all. I can come up with lame things like this all day long. No effort was exerted what so ever. =)</comment></comment></comment></comment></comment>


Yes Silver Bullet (2019) - <AUTHOR> https://blog.ploeh.dk/2019/07/01/yes-silver-bullet/ <comment><USER2> Here's the problem, though. Brooks made a certain observation about accidental vs. essential complexity; the author of this post makes another. Brooks made a certain prediction based on his observation (diminishing returns in impact to productivity), and many believers in the power of programming languages found his prediction to be too pessimistic at the time (he lists their objections in "No Silver Bullet, Refired"). Only it turned out that his prediction was correct, except it was too _optimistic_. So if anyone wants to claim that his observation was wrong, they would need to explain why his prediction turned out to be true, while that of those who believed it was wrong turned out false.

In addition, I agree with the author that the biggest contributions to productivity we have seen were, in this order, the availability of open-source libraries and online forums, automated testing, and garbage collection -- all of which have been adopted at rates we'd expect from adaptive traits in an environment with selective pressure. What is conspicuously missing is linguistic features, also in line with Brooks's observation. And yet the author still claims that linguistic features are the silver bullet. At this point this qualifies as an extraordinary claim that requires extraordinary evidence, but I would settle for ordinary evidence, which is also lacking -- strange, considering that a silver bullet, i.e. something with an extreme bottom-line impact, should be easily observable.

<comment><USER3> In your first paragraph you say that Brooks was right in his prediction. In your second you say you agree with the author that "open-source libraries and online forums, automated testing, and garbage collection" were the biggest gains. But those first two were the counter-examples to Brooks according to the author.

So where do you actually disagree with the author? You don't think the gains from these were substantial enough?

Also the author makes a clear argument for why functional programming is, from a complexity perspective, something entirely different from mere "linguistic features". And a "silver bullet" according to Brooks is something that might take a decade to show full impact.

<comment><USER2> > But those first two were the counter-examples to Brooks according to the > author.

They are not.

> So where do you actually disagree with the author? You don't think the gains > from these were substantial enough?

I think that 1. the collective gains might have been substantial but not 10x, and 2. the very ideas proposed by the author as being a silver bullet are not even among those in the first group.

> Also the author makes a clear argument for why functional programming is, > from a complexity perspective, something entirely different from mere > "linguistic features". And a "silver bullet" according to Brooks is > something that might take a decade to show full impact.

When I was in university, Haskell was all the rage. People were saying how in some years we'll all be programming in it or in a similar language. That was in 1998 or 1999. Functional programming in general has been well known, and taught for many decades. I'm not saying it's particularly bad, and maybe it could even have a small positive effect, but saying it's a 10x silver bullet at this point sounds delusional. If Brooks was wrong and there is a silver bullet, I very much doubt it's something we've known about, and tried again and again, in various forms, for decades.

</comment></comment></comment><comment><USER4> This is a really interesting take on the Silver Bullet argument - Brooks is right that big gains in productivity are only gained by reducing accidental complexity but he was wrong about how much accidental complexity there still is to reduce.

I think most Software Engineers probably suffer from some cognitive bias in trying to estimate exactly how much accidental complexity exists that could be eventually removed. We tend to think about incremental improvements to tools and processes rather than thinking at a higher level about improving the overall process of translating business requirements into working software. Even with much better tooling and languages (i.e. #F, Git, AWS, etc) there's still a lot of fat that could be trimmed.

I'm excited to see what the wold of low-code will do to our current assumptions about how much accidental complexity there actually is. Maybe projects like Dark[1] could actually achieve the order-of-magnitude gains that Brooks was convinced wouldn't be possible. Sure, there's no panacea, but maybe we're round the corner from a genuine "Silver Bullet" in the sense that Brooks actually meant.

[1] - [https://medium.com/darklang/the-design-of- dark-59f5d38e52d2](https://medium.com/darklang/the-design-of- dark-59f5d38e52d2)

<comment><USER5> Along the lines of your "no code" reference, I'm enamored of the potential for well-architected and -implemented design systems and their component libraries to virtually eliminate the need for non-experts to write CSS in constructing good, maintainable UI. That's a profoundly impactful change vs the status quo.

</comment></comment><comment><USER6> I'm in Brooks' camp. I see mostly essential complexity. There are problems we've been pounding on for a decade that have solutions that are mostly essential complexity. Then people come in and create pointless requirements that turn that fairly simple essential complexity... extremely complex.

I think the modern problem isn't accidental complexity, it's essential complexity that isn't actually necessary for project success.

<comment><USER7> > _There are problems we 've been pounding on for a decade that have solutions > that are mostly essential complexity. Then people come in and create > pointless requirements that turn that fairly simple essential complexity... > extremely complex. I think the modern problem isn't accidental complexity, > it's essential complexity that isn't actually necessary for project > success._

That would be a third category of complexity.

Essential complexity Accidental complexity

and the a new one: Complexity-imposed-as-"essential"-by-marketing-customer- etc.

Let's call it Requirements-Complexity

<comment><USER8> In the context of any given project, that's, if anything, a subcategory of Essential. The software is a tool to do something, and the functional requirements are what define the purpose of embarking on the project. You can't escape the business logic that needs to be enabled. It's like the old service industry joke "this job would be great...if it weren't for all the customers". There would _be no software_ if there weren't a set of requirements.

<comment><USER7> > _In the context of any given project, that 's, if anything, a subcategory of > Essential. The software is a tool to do something, and the functional > requirements are what define the purpose of embarking on the project._

It's not that clear cut.

For me "Essential" is what captures the essence of the problem domain and the requirements in actual use.

On top of that, there are lots of "requirements" that are there due to bribes, idiotic "brainstorming" where everyone felt obliged to chime in, what the CEO's spouse things should certainly be there, recent fads, over-thinking it from some execs, etc.

The distinction between the "essential" is of utmost importance.

First, because these "required but not essential" area is where an engineer can look for tradeoffs (to the benefit of essential areas).

Second, because those items might hamper the essential functionality (like a requirement to build a website in C, because the CEO heard "it's the fastest language", can hurt security, maintainability and other aspects, or the requirement to make a banking webpage with small fancy grey on white fonts because they though the design is "cool" hurts readability).

Third, because engineers should not just operate on a "follow orders" basis. They also have an ethical responsibility towards the final users (society at large for public software, customers, the customer's employees, etc).

And, of course, because not being able to make that distinction, means that the engineer can't tell good from bad, useful for useless, and it's all the same to them. Whether they can convince the customer it's another thing. But engineers should absolutely be able to make the distinction.

> _There would be no software if there weren 't a set of requirements._

Which is neither here, nor there. There would also be a hell of a lot of better, faster, more reliable software, if bogus requirements would get pointed out, and removed.

</comment></comment></comment></comment><comment><USER9> Brooks defines a "silver bullet" as a single technology or tool which yields a 10 times increase in productivity across the board. In other words, a "silver bullet" have to eliminate _90%_ of all the work a developer does during a workday.

But a typical developer is not even coding half of the time! They are discussing with the product owner to clarify requirements, reading specifications, writing specifications, thinking, surfing Jira etc. A _lot_ the work of a developer is to take vague requirements and transform them into unambiguous language. These are effects of the inherent complexity of software development. And then we have stuff like researching and evaluating what framework or library is the best fit for a given task.

Haskell is cool and all, but it will not eliminate 90% of what a developer does in a day. No single programming language, however perfect, will.

</comment><comment><USER10> The section "Accidental complexity abounds" and the diagram is reminiscent of a Twitter thread last month:

[https://twitter.com/gravislizard/status/927593460642615296](https://twitter.com/gravislizard/status/927593460642615296) "almost everything on computers is perceptually slower than it was in 1983" ; [https://news.ycombinator.com/item?id=21831931](https://news.ycombinator.com/item?id=21831931)

<comment><USER11> I disliked that twitter thread. It's rife with rose-colored views of the past, and misunderstanding of what they are referring to. The two main ones are a) library computers weren't always fast, and they definitely weren't checking as much data. We're a few orders of magnitude higher in what things are searching, and Amazon is _still_ really fast, and b) Google maps is a very specific use case of mapping, not analogous the general purpose map you would get previously. There are plenty of online trip planning services that do most or all of the interesting stuff that was mentioned. I've used a few of them.

The whole thread is analogous to a person that grew up on a farm complaining that their Honda civic doesn't deal with rough terrains as well as the tractor on the farm they grew up on did, to which the only response is either _duh_ or _no shit sherlock_ , depending on how charitable you're feeling.

</comment></comment><comment><USER12> I think Brooks is still right and what author mentions is not one thing but a set of practices that became proven enough that became almost like a standard approach in software development. And they provide some productivity gains if we compare how software was built 50 years ago. Yes, the difference is big but we forget that all those things were introduced incrementally and in the course of decades. So yes, together they make a huge difference but they haven't appeared in one day.

<comment><USER13> Git and Mercurial were both originally developed over the course of a month or so. If DVCS really results in an order-of-magnitude improvement (which do I find credible), then I think it qualifies as a silver bullet by Brook's definition.

<comment><USER14> Of all Git users, only an extremely small percentage use Git as a DVCS (Linux kernel, etc). GitHub is a centralized VCS. In fact, I think you could argue that the emergence of GitHub is a demonstration that DVCS is nearly useless for the vast majority of software development.

<comment><USER15> I think you don't understand what makes centralized VCS different from distributed. GitHub is just another clone of the repo.

For example, I can commit in my Git clone, can create a new branch, merge a branch, push it to the server, all without GitHub server being available. If GitHub would be down or closed tomorrow, this would not significantly affect my work on the repository.

This wasn't possible using centralized VCSs.

<comment><USER14> You can do all of this because Git repos are self-contained complete historical records.

You could do this with a traditional VCS as long as you took frequent backups. Git is an improvement in this regard, but not a revolution.

Furthermore, you are describing a workflow that is inconsistent from many "industry best practice" recommendations. If GitHub went down, a very large number of Git users would not be able to run tests or deploy their code to production - their CI/CD pipelines don't work without GitHub. Their historical record of issues goes away when GitHub goes down, etc.

<comment><USER15> With traditional VCS, I would lose all the history and ability to work (to commit my changes, for example) if server goes down.

And I understand that issues and CI/CD will also stop working, but neither of that is part of the VCS itself. I'm not aware of distributed issues (maybe FOSSIL) and CI/CD.

Still it doesn't mean that GitHub makes Git centralized.

</comment></comment></comment></comment></comment></comment><comment><USER16> The difference between accidental and essential complexity cannot be underscored enough. Removing accidental complexity, even if it flies in the face of "best practices," can be very powerful and appropriate as long as it's understood where you are intentionally coloring outside the lines and the conditions under which you'll need to revert to best practices for scaling up.

An example: I attended a conference presentation in which the presenter discussed dissecting the implementation of Identity Server into a dozen sub- services with half a dozen backing data stores with Redis caching layers and an Event Source model for propagating inserts and updates of the underlying data. This would be a prime example of accidental complexity gone wild if you built this just to have a multi-container microservice -- unless your single- signon service as a whole needs to support 100MM+ users, in which case this is _essential_ complexity, not accidental.

Reducing accidental complexity but being mindful of how it could become essential complexity under certain conditions in the future is the mark of a wise architect, IMO.

</comment><comment><USER17> The progress in tooling and methodology is paralleled by ever increasing complexity of software and problems it is solving. It's a zero-sum game. Still no silver bullet.

<comment><USER13> If a 10x improvement in productivity is paralleled by a 10x increase in expectations for the complexity of problems we can solve, that's not a zero- sum game, that's a silver bullet.

</comment></comment><comment><USER18> I think the points about accidental vs essential complexity are well made. However...

The title/s are about Silver Bullets, and I strongly do not believe in them.

I've been programming for an embarrassing number of years, and I've seen methodologies come and go. Increasing productivity is about developing a good team culture with a small number of simple tools, not about following the latest Silver Bullet.

In fact, it's very easy to increase the level of accidental complexity by adopting an overly prescriptive or complex tool/methodology (Jira anyone?).

<comment><USER19> Right there with you! I've also been programming a while now, and the biggest improvements to productivity, code quality, and overall success are indeed building a good team and keeping things simple. That hasn't changed since the 80's/90's when I first started out.

It's also true that overly prescriptive tooling can be such a pain. This also includes automated testing in my opinion (something the author counts as a big advance). It's fantastic for certain classes of software code and terrible for others. I've seen half my team spend multiple days just fixing integration or e2e tests instead of working on features that add value. You can grind to a halt with this stuff just as much as you can grind to a halt because of so many bugs to fix...

You have to find the balance, and I think that's where senior engineers really add value -- they've been round the block a few times and have a better idea where that balance lies.

</comment></comment><comment><USER20> The only way we'll see real 10x improvements take hold is for our discipline to mature. As long as we are experiencing high rates of growth, where all it takes is 3-5 years to become "senior", while the voices of the real senior developers are swamped by hordes of newer devs, we will wallow in inefficiency and fail to improve much.

Most of the accidental complexity I see day to day comes from slow accumulation of errors in craft. Need to validate that an ID is a least plausibly valid, "I'll just match it with a regex here", and in 27 other places in the code, each one added in a different sprint. Little things like referring to data that is passed into an api with the variable name "json" when it is really a data-structure that was parsed from the body of an API request. Is this thing I am manipulating a "tab", a "filter", a "type selector" or a "group"? When methods and variables use a mix of conflicting names for the same concept.

Until we learn, as a profession, that shipping working code is not enough, that our code must clearly, concisely, and consistently communicate the core concepts embodied in the program, accidental complexity will bury us.

</comment><comment><USER21> I would argue that a good chunk programming tools don't reduce complexity at all. Just shifts it to well understood "accidental" complexity.

Your program needs a way to store and retrieve key:value pairs. It's an inherent complexity your program requires. You solve it by using an industry standard product like Redis. Now your code doesn't contain any key:value logic itself! complexity avoided right? Nope just shifted and doubled or trippled in size to boot. No KV logic but now you have to have db libraries, parameter parsers and sanitizers, sockets between services, etc, etc, so on, and so forth.

It's well trod ground, your IDE will generate 90% of _that_ type of code for you. you throw a library or template or whatever at it and go about your day.

Software isn't just _your_ code. Now your application that needed K:V storage has the entire bug and exploit surface area of your code _and_ Redis. Do this 20, 30, 100 times with other libraries and packages. Each time _your_ code gets a little smaller, a little slicker looking and the actual running software is an unholy monster spread across an entire server cluster. Oh well, _your_ code is clean and tidy.

</comment><comment><USER22> What I dislike about all Haskell examples on every blog is that the writer happily assume that I'm familiar enough with Haskell syntax to understand the code.

In the ordinary method, there's a function name, it takes one input, and returns a result; easy-peasy. Yse, what the method does isn't clear, but at least I know the methods name!

Now lets look at the Haskel fuction. tryAccept :: Int -> Reservation -> MaybeT ReservationsProgram Int

I've read enough of these Haskel enthusiast blogs to know that the name of the function is tryAccept (it helps that it's the same as the C# method), and it takes in two parameters, an int and a Reservation and returns a Mayb of a ReservationProgram. And it's somehow an int, so it's like an enum? That's weird, clearly I don't understand Hasket syntax.

And then later, I'm told that it's simple to figure out what a ReservationsProgram is because there's some enum that doesn't include the word "ReservationProgram" in it.

No offence, but this isn't enticing me to Haskel.

</comment><comment><USER23> So, the author here is arguing that between ready reference/help on the web and TDD, that programmers today can be _100x_ more productive than they were in the 80s? Add a dash of strongly-typed functional programming and we can be 1000x more productive?

I agree that these tools can be productivity enhancers, but if that factor is even 10x that would a huge claim, much less 1000x.

<comment><USER24> Especially if we break this down to working days. There are just about 200-250 per year. Could you really build a software that used to take 4 years to develop in a single day now? Even 100x would mean a years work in three days.

</comment></comment><comment><USER25> It's a shame that the author blows his credibility asserting that garbage collection reduces accidental complexity, because he is right that the amount of the latter really has been much larger than Brooks estimated.

If you have ever watched the hoops somebody jumped through to plug a memory leak in a Java program, you will be forced to agree.

A fundamentally very similar process appears as a functional programmer unravels a program to fix a performance bottleneck. The reason it is similar is that, just as ballooning memory usage indicates a memory leak, a ballooning runtime indicates a time leak.

And, just as GC languages provide inadequate facilities to manage resource use, functional languages provide minimal resources to manage time use.

You could argue that memory and time management are on the accidental side of the ledger, and while the scientist in us would be tempted to agree, the engineer knows that managing limited resources is the essence of engineering, and taking away the tools needed to manage resources makes resource problems balloon out of control, breeding extremes of accidental complexity.

<comment><USER26> > If you have ever watched the hoops somebody jumped through to plug a memory > leak in a Java program, you will be forced to agree.

In my observation, a lot of "improvements" in software development are of this kind: they make it quicker and easier to get started, but make troubleshooting a lot more complex specifically because they've hidden what's actually going on in an attempt to simplify things. Not that managed memory is a bad thing! But ideally it would be wielded by people who'd spent a fair amount of time manually managing memory so that they know what to look for.

<comment><USER25> Managed memory is more of a superfluous thing. When you have resource management, memory is among the easier to manage, and GC looks like a solution without a problem.

</comment></comment></comment><comment><USER27> This might be just me, but Brooks explicitly put a decade time tag on his 1986 prediction, complicating the authors's comparisons from ~three decades later. I guess the way to tell would be to have people or teams build examples of 1986 level-of-complexity software and see how long it takes them.

<comment><USER13> That's fine for Brooks, but the OP is entirely correct that people still routinely cite the "no silver bullet" doctrine 3 decades later. So it's still worth revisiting even if it's technically 2 decades past its expiration date.

<comment><USER28> Brooks revisits NSB 20 years after the fact and makes pretty strong arguments that specific intervening technologies have not disproved his original conclusions. I think we could do the same today.

The author of this post conflates better with a 10x improvement from a few anecdotal experiences that are themselves extremely accidental to the act of creating software. He may have a point about the large volume of accidental improvement areas remaining but I think this is because like the JS ecosystem they are growing faster than we solve them.

I think the reality is likely the author is themselves a 10x improvement over their decade-past self.

<comment><USER13> > Brooks revisits NSB 20 years after the fact

Do you have a link to this? I'd like to read it.

<comment><USER27> Closest thing I could find was a report [0] on a 2007 panel discussion with Brooks at OOPSLA, titled "No Silver Bullet" Reloaded – A Retrospective on "Essence and Accidents of Software Engineering"

Here's [1] a link to his 1995 update, an excerpt from an anniversary edition of 'The Mythical Man Month'

[0] [https://dl.acm.org/doi/pdf/10.1145/1297846.1297973](https://dl.acm.org/doi/pdf/10.1145/1297846.1297973)

[1] [http://worrydream.com/refs/Brooks- NoSilverBullet.pdf](http://worrydream.com/refs/Brooks-NoSilverBullet.pdf)

</comment></comment></comment></comment></comment><comment><USER29> > Ostensibly in the tradition of Aristotle, Brooks distinguishes between > essential and accidental complexity. This distinction is central to his > argument, so it's worth discussing for a minute.

> Software development problems are complex, i.e. made up of many interacting > sub-problems. Some of that complexity is accidental. This doesn't imply > randomness or sloppiness, but only that the complexity isn't inherent to the > problem; that it's only the result of our (human) failure to achieve > perfection.

> If you imagine that you could whittle away all the accidental complexity, > you'd ultimately reach a point where, in the words of Saint Exupéry, there > is nothing more to remove. What's left is the essential complexity.

Okay, that might be a useful way of thinking of things...

But the author then goes on to talk about how he thinks that Fred Brooks underestimated the percentage of accidental complexity in the average project.

However, he then starts to go into things he thinks are "silver bullets", but most of them are tools that, in my opinion, address _essential complexity_ :

* The World Wide Web: in his description, this solves a problem of the complexity of finding how to do things. Can we imagine a solution where finding out how to do things isn't part of the system? No? Under the definition, that's essential complexity.

* Git: in his description this solves the problem of tracking changing source. Can we imagine programming where we don't need to keep track of what source changes happened? I can, maybe, in a distant future where programming is done an entirely different way, but I'd argue that at that point it's not really the same problem. If we're writing code to solve a problem, then tracking changes to that source is essential complexity within that solution.

* Garbage collection: again, maybe there will be a computer that works completely differently (quantum computers?) but under current architecture, the physical machinery of a computer has limited space, so we need to reallocate memory that is no longer being used. That's not accidental complexity, that's essential complexity. There are other solutions to this, but every program solves it in some way, whether it's by reference counting, manual allocation, borrow checking, preallocating everything, or just allocating everything on the stack and copying everything, and each of these has complexity: that complexity is essential.

I think what this all points to is that _essential complexity isn 't intractable: you can make it someone else's problem._ In these cases, we're taking some part of the essential complexity of solving a problem, and letting our ISP and websites solve it, or letting code that we don't maintain solve it. Some of _the most powerful tools are ones that solve a form of complexity that is essential to a wide variety of problems._</comment>


21 Rules of Thumb - How Microsoft develops its Software - <AUTHOR> http://blogs.msdn.com/David_Gristwood/archive/2004/06/24/164849.aspx

<comment><USER2> These rules actually favor being "on time", which to me is a terrible goal for a software project. Those who think software can be "on time", tend to also consider the software "done".

In reality, any project can be "on time" and still fail miserably if it's hard to debug, hard to modify, hard to support, or implodes when the lead developer quits.

You don't want software that's "on time", you want customers who are well- informed so that their dependencies are more specific and flexible than a "release date". You want them to be thinking long-term, so that they understand the true risks of insisting on early delivery (e.g. less testing, review and documentation may have been done, hence more bugs may appear in the future).</comment>


The Little Secret of Web Startups - <AUTHOR> http://www.techcrunch.com/2009/07/30/the-little-secret-of-web-startups/ <comment><USER2> This post offers some informative insights. I am reminded to at least consider proactively filtering content, etc.

I thank the author for his contributions toward helping other startups.

With that said, this tastes a little sour to me. He is saying "other businesses should be doing what we did". With all do respect, what you did got you to where you are. He is saying his business was honest by not skewing user numbers and engagement. Which is also implying that if they did, they might have secured more investment, or raised their ad rates.

The problem with Sampa is that their target market has fundamentally low market value. Share family photos online?

Aren't there about 44340902342034 services offering something of the sort?

Lastly, I'm surprised to read that Sampa had an advertising based model. I guess that shines some light on why they chose the market that they did. I mistakenly assumed any website creation system would be SAAS. Perhaps if Sampa would have stuck with saas, they would have switched to a new, more lucrative target market.

</comment><comment><USER3> If I was a investor in this startup (1.31mm raised so far)I would be pissed that my failed investments's founder is airing out laundry for money..

<comment><USER4> The start-up is shutting down, so it isn't 1.31 million so far, it's 1.31 million. And it isn't as if he is saying anything about the investors, or any secret information being released. He's trying to explain why the start-up failed in order to help others.

At the same time, the title 'the little secret' seems very misleading, unless I'm missing something.

Is there one statement in particular that struck you as offensive or 'dirty laundry'ish?

</comment></comment><comment><USER5> According to <http://www.statbrain.com/www.sampa.com/> they get 10,203 visits per day. This is an estimate, but it doesn't sound like a dead website to me.

<comment><USER6> The estimate isn't very good, my old site that gets about 300 visits a week is estimated to have about 3800 visits per day. Another one that gets about 100k visits daily is estimated to have about 50k visits. So I wouldn't trust their estimates that much.</comment></comment>


#shirtstorm leads to crowdfunding for Dr. Taylor - <AUTHOR> https://www.indiegogo.com/projects/matt-taylor-rosetta-project-scientist/gpls

<comment><USER2> A just outcome from such an absurd attack on him and by extension on the woman who designed the awesome shirt. He and his team accomplished something amazing, despite the shirt and despite the news today: [http://news.sciencemag.org/europe/2014/11/updated-main- missi...](http://news.sciencemag.org/europe/2014/11/updated-main-mission- philae-comet-lander-comes-end)</comment>