The Death of Files - <AUTHOR> http://dustincurtis.com/files.html <comment><USER3> I don't think files are the problem. It's the folder hierarchy that's the problem.

The thing iPhoneOS is most intent on eliminating are deep trees of files. It does this by training the user to think of their files/content as being managed by an individual app that's responsible for manipulating it. This creates a single top-level "folder" for the user to use as a mental key to remember how to get at something.

Contrast that much shorter path to content to the standard desktop model of having a folder in your home folder, perhaps, to store documents. The ability to make sub-folders, sub-sub-folders, etc. and have all the contents of those folders utterly disconnected from the applications you use to manipulate and create that content. It makes sense to those of us who grew up/invented it, but for people just looking to use a computer as a tool in their non-computer- driven life suffer under the cognitive load of the foreign metaphors.

In my experience, people "get" that individual files represent their content. They do not "get" folder hierarchies, though, and find themselves easily lost within them. I think many non-experts consider an app as a _destination_ and not as a program to run. It makes more sense to them to think of "going" to the app and then finding their content there than it does to "run" an app and then later point it to their content.

<comment><USER4> Why did categories instead of folders never take off? Just implementation complexity?

<comment><USER5> Originally they were trying to do as Curtis mentioned. Files were to match documents, photos etc that might go in a file folder in a file cabinet. I guess the file cabinet was the computer.

Categories would work. But they are abstract and folders/files mapped to physical objects that were part of offices at the time. Recycle Bin/Trash, same thing, trying to map to real-world at the time.

I think the death of files is overrated though. I mean even on the touch os I can go to search and search for... files. No file browser like Finder or Windows Explorer work when you have a killer Spotlight like search.

</comment></comment></comment><comment><USER6> This app-centric model works well until you find that you want different apps to operate on the same file - which will eventually happen on devices that people spend a lot of time in

The file-less model may work well for a consumption-centric system like the iPhone where you're not expected to spend much time on it trying to produce content (and the places where you do - like email and sms- the mechanism is simple and is handled by a single app). Even there, photos are exposed in a special way because it's understood that people will want to do various interesting things with them (like edit them or upload them using different apps).

I think the real problem is in how the file system is presented to users.

When a user saves a file, they have no clue where they saved it. The concept of file shortcuts is another killer - I've seen many cases where someone copied a shortcut onto a pen drive - or worse- burned it on a CD.

Instead of presenting a 'Save' dialog, have the user drag some file icon next to the top of the app into the desktop or some folder.

When browsing a list of files on the system - instead of showing folders in a hierarchy, show a list of apps - in each one, show the list of files whose last modification was done by that app.

Many users already follow a model like this- they just fire up the app and click the Open button which shows the most recently saved location.

<comment><USER7> > _This app-centric model works well until you find that you want different > apps to operate on the same file - which will eventually happen on devices > that people spend a lot of time in_

This is solvable with standards. If all photos are in a standard place, in a standard way, with a standard db schema, then any app can access it. The standards can be supplemented with app-specific stuff, but as long as they all adhere to a base level of standards, then we are ok.

Obviously there is some data that is more free-form that others, which would require apps that are more open.

<comment><USER8> What do you do when files contain other files in a different 'standard'?

<comment><USER7> Huh? I was talking about things like having a 'images' or 'photos' store where all photos can live. If you want to create a program that uses the photos, there can be a standard 'photo search' dialog rather than a 'file' dialog. This way all programs can access this store, and we can have dialogs that are tailored to the type of data that is being presented to the user. We could even make it possible for the user to search content if you had a 'docs' dialog that encompassed things like PDFs, text files, Word documents, etc.

This would make it a lot easier if say, you had a 'project management' app (not in the 'supervisor' usage of the term). Say you were creating a collection of documents for a particular project and you wanted various types of documents like photos, PDFs, etc (maybe you're doing research for something). If you were trying to pull together some of these documents from existing sources (i.e. documents you already have) you could use the lightweight search facilities of a 'docs dialog' to search the content of the PDF that you're looking for rather than trying to put all sorts of context into the filename and/or directory structure.

<comment><USER9> > Huh? I was talking about things like having a 'images' or 'photos' store > where all photos can live.

A single directory is a disaster as soon as you have a reasonable number of photos. And, if you have subdirectories of photos, you're back where you started, except that you don't know where the "root" of the photo directory is, so you can't use standard file tools to manipulate them.

And, where do you put videos? Some people organize their photos and videos together (in part because they came from the same device) while others keep them separate.

And if I take notes about my photos, where does that file go?

I really want to be able to keep related things together so I can easily provide them to someone else.

There are lot of easy answers, but getting one that covers a large fraction of the use cases is hard.

<comment><USER7> I think that the issue here is that you're still thinking of "How will this fit into a directory structure?" or "How does this map to the current paradigm?" I'm suggesting a completely different paradigm that is coming more from the perspective of a database-ish approach (either 100% DB or a mix of DB and filesystem).

<comment><USER8> What I was getting at is what happens when you get things that don't fit neatly into a database slot. For example, if you have a database that can store photos and text files, what to do with a document that has text and photos in it (like a newspaper article) -- store the text in the text DB and the photos in the photo DB? How does the DB associate the text with the proper photos?

The point being, electronic media is by its nature something that tends to defy whatever categorical schemes we come up with for organizing it, whether they be flat, tree-like, database-oriented, tags, or whatever.

<comment><USER7> NoSQL document-based rather than relational-based databases could deal with this. I admit that there are caveats to such an approach. From a high-level look at the idea, it seems that all of these things could be ironed out.

For example, you don't need to separate the text and the images of a document. For the most part, you could just 'store a MS Word document in a DB BLOB.' The idea would be that the DB (or some sort of external indexing) is aware of how to parse the MS Word document so that it can be searched insofar as the text is concerned. It might get more complex if you wanted to have the images tagged as well, (thought there may be a way to have the indexes extract tags from the image metadata, or from the original location that the image was imported into the document from).

Remember that the entire purpose of this is just to do a couple of things:

      1. Classify data by type and/or format ('image','PNG','PDF','document')
      2. Provide a way of easily accessing that data.

For specific types of data like Photos, you could have a more specific (optimized?) store. For more general types of information you could have a more generalized store. For information where the content has no practical value as far as indexing goes (e.g. SolidWorks files), you could just have a generic metadata attached to the file information (name,description,tagging).

You could have a 'general docs dialog' that does the generic searching by metadata (document name, type, tags, content if index-able).

Then when you open apps you are displayed with the content right in front of you (ala iPhoto). Rather than an empty window that wants you to open content. For example, you open SolidWorks, and it shows you all of your SolidWorks documents with thumbnailed previews.

There could also be defined 'meta-documents' which are just collections of pointers to other documents. You alluded to this in your post, but you were using the idea in the wrong fashion. If you were creating something like an article with embedded images, that would be a document in it's own right. A 'meta-document' would be a collection of disparate type of multimedia in a 'collection' (e.g. taking a bunch of documents -- pdfs, images, etc -- related to a project and stuffing them in a folder or a zip file). [ Though this would depend on the implementation, b/c they could just be implemented as cross- content tags unless you wanted to be able to attach specific meta-data to the meta-document, but not the individual documents ]

As wonderfully complex as all this seems, it could be presented to the user in a simple fashion. The user would have no idea what a meta-document is. They would just know that the open up their scrap-book program and they can create scrapbooks, for example. The underlying implementation of the scrapbook could be a multi-page document, or a meta-document.

<comment><USER8> I think you're on the right track, and a 'document' oriented interface is what I think the next step will be, as opposed to Apple's dead-end app-oriented interface. Your meta-document is similar to something I have been envisioning... the current way things work is that you can either:

a) put some photos in a folder, and to view one you open it which launches the default application to view it.

b) open up your photo application, which either knows where the photos are, or asks you to point it to where the photos are.

I think the future is where the folder _becomes_ the 'application' by the mere act of putting a bunch of photos in it. But it wouldn't be an application in the current sense, but rather a collection of separate and customizable ways of interacting with photos. The average user might just want some buttons to print photos or email them to friends, but for a web designer you could extend the meta-document by adding commands to resize images, change formats, crop, etc -- not by buying something massive like photoshop, but simply by downloading (or creating) mini-extensions that do exactly what you need to do.

So yeah, the way I see it, we'll move away from some shell which organizes files and applications that interact with the files, towards a system where the 'shell' itself evolves into whatever 'application' you need it to be.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> "When you see a photo on your desk, do you think "a file!" or do you think "a photo of my family!"?"

The same applies to a .JPG file in my documents folder. I think "a photo". That last line ruined the article for me, because it made me realize that files _are_ how we do things in the real world, just with other names. We don't deal with fridges, we deal with food. And you know what? We even have to know where to store the food! Some food, like milk, goes in the fridge. Other food, like pasta, goes in the cupboard. Same with spices. Yet other food, like frozen peas, go in the freezer, Still more food might be found on plants in the garden. There is no single app, just a lot of directories for me to search in.

<comment><USER11> I strongly agree. The concept of file is extremely popular because it _is_ intuitive. Sometimes designers get carried away in trying to simplify an interface that doesn't need any simplifying. For some simplistic devices like a DVD player or a game console, it might make sense to hide the filesystem because what the user will be doing is predetermined by the designer (console = play game. DVD = watch movie). But for general purpose computers, it makes no sense to hide such a powerful feature.

<comment><USER12> The file model is only intuitive to you because you've been using it for so long that it has become a part of your existence.

Do some user testing with casual computer users (not even pure novices), and watch as you ask them to complete a two step task with a file (like, say, resize and then email). They can usually figure it out, but it takes them a long time and they become frustrated. It's not a natural experience to them.

You know when you first start programming, and you hack together snippets of code to make your first program work? That's kind of what using a computer is like to the majority of people.

<comment><USER13> I work with novice users every day. Every part of the experience is unnatural to them. Using a mouse and typing is just as frightening as any software quirk. Scroll bars, application windows, the web browser as internet, even the very concept of saving something in any form.

Teacher: "This program saves your changes automatically"

User: "But how do I know for sure?"

<comment><USER14> As a programmer, I would be frightened of something that saved my changes automatically, without a way to force a save or verify its saved-ness. What if the auto-save stopped working and didn't tell me?

<comment><USER15> What if the manual-save stopped working and didn't tell you?

You made a change - that's it. You don't save your paper after drawing on it with a pencil, do you? Death of filesystem, rise of tags, automatic saving and versioning go all hand-in-hand.

<comment><USER14> _What if the manual-save stopped working and didn't tell you?_

I suppose it's equally as likely. The example I'm thinking of is losing your network connection while typing an e-mail in GMail. It shows a warning message, but out of the way so you might not notice. If you intentionally take an action to save, you are more likely to notice the result (and a good user interface would make a failed intentional save obvious by a popup message or something).

 _You made a change - that's it. You don't save your paper after drawing on it with a pencil, do you?_

Depends. If I was drawing something remotely important, there would probably be many revisions, some of which I'd want to keep, some of which I'd want to throw away. Some of which I'd want to make a separate copy of to work in a different direction. So yeah, versioning. How do you version things without files?

</comment></comment></comment></comment></comment></comment></comment><comment><USER16> I strongly dislike this trend. Imagine if in the real world, the only place you could put food is your refrigerator, or if a bookshelf only allowed you to place books on it. While it sounds nice to say that a person only looks through their photo album for their photos, that's not, strictly speaking, true. There's also those unflattering photos that you leave in shoeboxes, as well as the photos of loved ones that you keep in your wallet. The idea of a generic file is an extremely useful abstraction to developers and users alike. Does it really make sense to have photos separate from videos? Or photos of your loved ones alongside photos you take for your job? I think that the classification system that folders and files have given us is much more useful to users than the vague lumping of items by what amounts to a filetype.

However, I do think that there are better abstractions for a user than the bare file system. I don't have much experience with either, but I'd guess that Win 7 Libraries and something like Leap[1] on the Mac are improvements on this.

[1] <http://www.ironicsoftware.com/leap/index.html>

<comment><USER17> _Imagine if in the real world, the only place you could put food is your refrigerator..._

The machine isn't the real world. In particular: People often like to have software do things for them. But software has no intelligence. So we have to design a system that both the human and the software can understand.

If you had a personal robot that would automatically throw out, and reorder, any food that's more than six months past the expiration date -- but only if the food is in the refrigerator -- you might find that you would rather store 100% of your food in the refrigerator than have to deal with your own shopping.

Then, a few years later, when you smell the six-month-old food that you accidentally forgot to put in the fridge, you may find yourself wishing for a system that actively _constrains_ you from storing food outside the fridge. And then _hides_ the fridge from you, so that all you see is a list of foods and you don't even have to care about where those foods are being kept.

This is the age-old art of database design. On the one hand, databases often feel constrained and clunky. On the other hand, if you learn to deal with the constraints, magical machines can come along and save you thousands of hours.

</comment></comment><comment><USER18> I've mentioned this a couple of times, but coming from my deeply non-technical niche, I think files are dying and good riddance to them. They're a constant barrier in front of my customers getting their work done and going back to their lives. They're an abstraction which is -- mercifully -- more user friendly than disk sectors or IP addresses, but not by very much: my users may lack the ability to find saved files on the computer, to understand the difference between the filesystems on two computers, to be able to copy a file to a floppy or CD without active moving-the-mouse-for-her assistance, to differentiate between files and the applications that created them (expecting, e.g., that copying bingo cards onto their sister's computer will let them open them automatically -- hey, works for photos and letters, right?), etc, etc. We won't even get started with navigating a tree structure.

I've had a sustained decrease in the amount of time I spend supporting customers with these issues since releasing my web application, because it comports to how many of them think their files actually work: they live in the application, they always exist in the application no matter what computer you're on, they always show up on the top screen no matter what, they never get lost, and even if they get lost Patrick can find them for you if you send him an email.

<comment><USER19> You are right that users _think_ they are better off without files. And I completely agree with you that web applications provide a nice avenue to a life with no files (No access to the local filesystem from within the browser? Great! We'll just store everything in a database on the server).

But then we start to have these situations with data lock in that are definitely bad for users. Take Facebook since it's oh-so-popular to bash them these days. If my Facebook data were stored in an encapsulated and portable format I'd just go to a new social network that respected my privacy and say, here's my data, ready to go. Import it.

Now it may be the case that most users are willing to turn over their data to the application that created it in exchange for simplicity and ease of use. But I don't think that's a debate that many users are having with themselves right now.

<comment><USER18> "Facebook would be better if Facebook used files" is a bit of a strawman. [Edit: No, it isn't. Crikey, failing at English: what is the word for an argument premised on a proposition that is contrary to fact? It can't just be "wrong", that's what 3rd graders say -- needs more Latin.]

Do you know a single application that abstracts away as much data as Facebook does _as files_? Anything on my computer _nearly_ that complex, like an email inbox, a photo collection, or my iTunes stuff exists as an opaque blob with a proprietary reader. (In the case of my inbox, it is an opaque blob with widespread inport/export capability into opaque blobs of different colors.)

It has a file on disk like MySQL does: OK, technically accurate, but I'm not capable of actually doing any of my usual file activities with that file.

That is aside from the "whose data are we talking about, anyhow" conundrum. Probably 80% of my presence on Facebook "belongs" to other people. Tagged in a photo taken by my mother of my brother's wedding: think quick, is that in my export or not? (For that matter, is my brother in my export? I'd kind of like to import him into my new social network. You can do that, right? Oh and Farmville.)

<comment><USER20> > what is the word for an argument premised on a proposition that is contrary > to fact?

Fallacy? Counterfactual? Hypothetical?

<comment><USER21> Why did these guesses get downmodded? Is it an HN customs thing or did none of these come close enough to the mark?

</comment></comment></comment></comment></comment><comment><USER22> I don't know that files _are_ broken.

Our brains are trained to "select" things: from the correct answer on a test, to the right phrase to say on your significant other's birthday.

Limiting user exposure from little bundles of portable information to hidden databases is exactly what MS did with the registery. It's all well and good until you realize that sometimes it's nice for data to be a little more accessible and visible

"Apps" that lock in data, like all sorts of old-school applications that stored your files in hidden folders, fail. They're closed. They're a black box that you can't easily leave when you'd like to move on to another application.

Hiding data in closed databases is not only counterintuitive (it's like hiding the mustard in the back of the fridge in your analogy, and chaining it to the door), it's harmful should you ever want to buy a new fridge.

</comment><comment><USER23> Has been discussed before: <http://news.ycombinator.com/item?id=1230099>

I still think it's an awful idea to group content by type ("all photos are in the iPhoto app") rather then let the user choose how to group it.

Users usually put things together based on content, not type. They want to place that funny video in the same folder as the funny joke saved in .txt and that image of the cute cat. The document with a list of things to buy for the party along with the spreadsheet with the total costs.

And IF they one day decide they want to group by type, they do it. Abstracting the notion of files away is just removing options from the user, options that weren't even bugging them in the first place. I have yet to see someone having trouble understanding the concept of files and folders.

And don't even get me started on the fact that all applications will have to implement their own content manager...

</comment><comment><USER24> This crap drives me crazy with my touch and is what's holding me back from buying an iPad.

Download file with browser, can't. Get app to download file. Want to view it with PDF app, can't. Cause there are no files and apps can't share data. Want to move it into my photo album or music or video or ebook folders, can't cause there are no "folders". (some newer API's might allow specific exceptions I've quit paying attention). Want to transfer it to removeable media, can't. Cause I guess that's too confusing. Want to transfer it to another device, can't unless task centric app supports that cause again no files, no sharing app data.

Every app is a silo. It's the diametrically opposite of *nix philosophy of small tools connected in myriad of ways. It maybe great for consumers who can't or refuse to learn. But for computer users it sucks balls.

[edit: this is the point of iPhone and iPad. They aren't devices for computer users. They are the mass consumer device the generic/multipurpose/computer has and can not ever become.]

<comment><USER25> this is roughly the problem that KDE's Nepomuk (and its ontology) is tryin to solve. Though their implementation is a resource hog.

</comment></comment><comment><USER26> General purpose computers are system-integration machines.

My mother searches the web for content and drags/drops snippets into presentations; emails photos to friends; sends letters via fax. Those are all inter-app activities that she couldn't do on an iPad.

A friend with an iPad sync'd his Dropbox folder to it; clicked a file and chose "Send to Pages", edited it, and then ... Well, he couldn't save it back to his Dropbox because Pages doesn't support it.

I thought trapping content in apps was a _bad_ thing.

</comment><comment><USER27> I just ditched my iPhone 3G and got an Android phone just to have files back. Seriously.

And I've never been happier with a phone (it's an HTC Desire, FWIW).

The death of files is the death of freedom.

</comment><comment><USER28> The problem with files is that there are so many of them! An average computer contains millions of files yet only a small fraction are actually created by users. Those files are images, or spreadsheets, or documents and people can understand and work with them pretty easily. But these files are often stored in deep folder hierarchies (C:\Documents and Settings\UserName\My Documents really?) on a hard drive filled with other hierarchies that nobody cares about.

I don't think we need to get rid of files the way the iPad does, but we do need to rethink what a unit of a data to a user really is. For a user, a photo is a single unit of data (one file) but so is an application (many many files).

</comment><comment><USER29> "Because most computer operating systems don't organize things this way, accomplishing simple tasks can be extremely confusing for casual computer users;"

You can't compare an operating system with an application. I can use iTunes on any OS to play my music (the iPhone is nothing special in this regard). iTunes only knows about music, so all the files are "songs". I can consume any song this way on my PC. And as long as all you do is consume, this interface works beautifully.

But try creating a song, and suddenly you need to know how things work in a little more detail.

Files are not going away, but if all you want to do is consume content, then there are some pretty good apps out there that present your data in intuitive and meaningful ways.

</comment><comment><USER30> This mentality has lead OS designers to come up with those "My Documents", "My Videos", "My Photos" folders that are auto-created on installation. It's been there in Windows, recently Ubuntu has them, probably Macs have something similar. It's always the first thing I delete after a fresh install.

<comment><USER31> Macs have a home folder with the following sub folders

        Applications - User version of /Applications
        Desktop - <USER2>
        Documents
        Downloads - Safari downloads files here
        Library - Application configuration and data storage.
        Movies
        Music - iTunes library defaults to here
        Pictures - iPhoto stores it's library file in this folder
        Public - Documents visible to people on the network, not writable
        Public/Drop Box - Publicly writable but not publicly visible
        Sites - Web sharing uses this folder to share your website from here.

</comment></comment><comment><USER32> If you're a producer of content ... you care about files.

If you're a consumer of content ... you don't.

<comment><USER33> When I'm producing content, I don't _want_ to care about the files. I do care, because the technical reasons behind my choice of formats, compression schemes or packaging methods are important. But I'd really rather not have to be involved in those things.

I'm looking forward to the day when the steady march of abstraction provides tools that effectively take "files" away from _me_.

(Edit: grammar)

<comment><USER28> I think you're confusing the format of a file with the abstraction of a file as unit of data. Another way to look at the iPad is that it has files but it limits how you're able to work with them.

As a content producer, you want to able to take your unit of data (photo, image, document, etc) and use many different applications to work on it. You might want to make a copy of it, rename it, or move it to other machines too.

No abstraction will effectively take "files" away from you without also taking away the functionality behind them. I'm honestly not sure you've thought through the consequences of that.

<comment><USER33> I'm not saying the physical file as fundamental computing construct is going anywhere.

I'm saying software is getting to the point where the details of file storage won't be relevant, even to producers of content. Whether your resource is one file, five files or five parts of five larger files will be as relevant 'tomorrow' as where the disk controller has put physical bits on a platter or memory cell 'today'.

The details still matters and always will; there are certainly good and bad ways to store those bits depending on the varying contexts. But no-one's _thinking about it_ when they save a file; no-one's hand-tweaking each decision for individual performance or organizing it so you'll remember physically where you put it.

</comment></comment></comment></comment><comment><USER34> while the notion that files are "dying" may be true, the way iphone os handles them is still not usable.

 _When you see a photo on your desk, do you think "a file!" or do you think "a photo of my family!"?_

i think, "a photo of my family", but then i think, "i want to make a copy of this and mail it to someone" so i take it to walgreens and get a copy, then i go to the post office and mail it.

to send a copy of the photo on my ipad, can i carry the photo with me as i launch the e-mail application? or can i carry it with me as i launch a flickr uploading application? or carry it into beejive and send it in an instant message?

on a mac you can. you just drag the photo out of iphoto and drop it on some application. on android, you click the share button while in the photos application and up pops a list of every application that can do something with your photo.

on the ipad, it's backwards. you launch the flickr application and then have to find your photo again. and this is restricted to photos, since it's an internal api. any other 3rd party app wishing to share data with another app can't do it. right now if you want to print an e-mail from an ipad, you install a printing application and it has to have an embedded e-mail client where you have to duplicate your settings and e-mail storage. apple could make an api to access your e-mails (not likely) but it's still backwards. the e-mail client should let you print or do anything else with an e-mail from the e-mail client, not the other way around.

<comment><USER35> iPhone OS 3.2 (ships on iPad) and 4.0 (presumably will run on iPhone and iPad, not out yet) have support for apps discovering other installed apps' declared support for filetypes and for apps to hand documents off to other apps.

</comment></comment><comment><USER36> Excellent piece. I think the main challenge for UX architects is breaking the hiearchical model that users are used to.

Tags are the ideal solution for organizing like items but few have mastered the UI. Sure, it's easy to to build the refrigerator model but users get freaked out when everything is lumped together (that's why websites still have nav bars.)

If someone can master the art of the tag UI, then we're set.

<comment><USER12> I don't think tags are the solution. Human beings organize stuff by lumping similar things together, by spatial relationship. It's the way the brain is designed to make sense of things on a neurological level.

Tags create a meta layer of information on top of the actual spatial organization of objects, which is hard to comprehend naturally.

<comment><USER37> The problem is we have too many virtual things to organize them spatially. It worked with the finder when each user have a few or few dozens of or maybe even a few hundred files they cared about. Now computers have millions of files, and I don't see how we can sensibly organize them spatially and still find things.

I would say that now the dominant way of finding things on a computer is not organization at all, but search. Type some words describing the thing you want to find. We've been trained by Google that we should always be able to find anything we want this way.

<comment><USER38> True, but right now search fails miserably for images. It might not, if people tagged their images better (facebook does a good job of getting people to do this), but most people don't.

</comment></comment></comment></comment><comment><USER39> "If you think your users are idiots, only idiots will use it." --Linus Torvalds

Others have basically spoken my thoughts that this is a wrong direction to take, files actually are pretty intuitive and the trouble is often "which app can I use on this file?". I'm against trying to simplify everything to such a degree, since it's treating users like idiots. Down with single-button mice.

</comment><comment><USER40> When a human first saw a photo did he say 'an etched memory of me' or did he say 'it's a new cool thing called a photo and it's of me'

In the same way I don't think there is anything wrong with the concept of file as long as the abstraction is fairly simple to understand.

I say don't assume humans are stupid. Let them choose if they like files or task based os's. Don't indoctrinate designers to belong to only one school.

I say why only two schools - there could be better abstractions out there.

Think. Imagine. Make a new abstraction. Don't take one of the two existing sides.

</comment><comment><USER41> Suddenly it's Smalltalk 80 all over again...

Even if we were to credit Apple for such innovation, IIRC, the Newton did away with files a long time ago. If the Newton did not, then the original PalmOS did.

</comment><comment><USER42> Reminds me of that concept for a code editor UI with bubbles[1]. If the user interfaces let me visually organize the code in a way that makes sense, then the way the data is saved on disk doesn't really matter anymore.

[1] [http://geekswithblogs.net/andrewbrust/archive/2010/03/12/cod...](http://geekswithblogs.net/andrewbrust/archive/2010/03/12/code- bubbles-disruption-comes-to-the-ide.aspx)

On the flip side, I guess there's the problem of being locked to the applications

<comment><USER38> I took a look at it. It seems like a Smalltalk IDE for java... does it offer innovations besides that?

<comment><USER42> I don't know what is or is not innovative about this particular editor concept, I just pointed out that it's more useful to be able to see things like procedural flow without having other things cluttering the screen just because they happen to be in the same files as what you're interested in at a given point in time.

Another nice example of the file-centric approach failing is Photoshop comps. It's a lot easier to just have an interface where I can quickly toggle between various comps in a single project file than it is to have two dozen slightly different files, each representing one comp.

</comment></comment></comment><comment><USER43> Omg, this industry is retarded:

"This is a new model for organizing things on computers"

No it isn't. Hey, did you know that you can still make computers run programers _without_ an operating system and _without_ a hierarchical file system?

My god, we really need to add History Of Computing and History Of Technology courses to the highschool and uni curriculums. It's disgusting how this industry is full of fashion.

<comment><USER38> You're getting downvoted (not by me), probably because your post was a bit over the top, but in reality you're right. Whatever the merits of this idea, it certainly isn't new, and Apple wasn't the first to try it. But as always, when Apple implements an idea from somewhere else, they get all the credit. But like you said, it goes far beyond Apple. No doubt people discussed this very issue years and years ago, and had solid reasons for choosing a file- system. Things may have changed, so it's fine to reconsider the decision, but rather than everyone pulling an opinion out of nowhere, perhaps we should review the past rationale first, and once again weigh the pros and cons.

</comment></comment><comment><USER41> The absence of files suits a media consumption device. I don't think a media creation device would be able to do away with files in the user interface.

</comment><comment><USER44> The file-less model is not new. Anyone who coded for PalmOS is familiar with the idea of storing all of the data for your application in internal data structures.

What is also not new is that developers inevitably come along and create file managers to sit atop these file-less devices, so that you can download and store PDF's, spreadsheets, photos, etc. on your device ...

</comment><comment><USER45> This reads like the guy just discovered the phrase "object oriented". I know he's a smarty, but come on, haven't we had the idea of data as objects for years now? Is it that the iphone is the first device to make it usable by restricting the available pool of data types instead of trying to boil the ocean by making "everything an object"?

</comment><comment><USER46> _When you see a photo on your desk, do you think "a file!" or do you think "a photo of my family!"?_

This is all well and good except it comes right at the end of an article explaining why I can't have a single photo on my desk and why I shouldn't want to, all I should ever need is a photo album with all my photos in it.

Hrm.

</comment><comment><USER47> Am I the only one who thinks Dustin Curtis is kinda a retard with good design skills?</comment>


$2.50 a Year in Interest? That's What $5k in Savings Gets - <AUTHOR> https://www.nytimes.com/2020/09/18/your-money/savings-interest-rates.html <comment><USER2> I wonder why the article doesn't mention just refinancing your debts, like hose mortgage or credit card debt: you make a new mortgage to pay off your current mortgage, but the new mortgage has lower interest rate than the current one. And you might save a lot on interest alone.</comment>


Yay we are doomed - Agile lessons from the Trenches(ebook signup) - <AUTHOR> http://yaywearedoomed.com/

<comment><USER2> I had the benefit of working with rpwilcox in one of the trenches where I know I learned a lot. Very interested to read about his other experiences and learn more.

</comment><comment><USER3> Worked with Ryan in my first real job out of college. Looking forward to reading!</comment>


GitHub connectivity issues  - <AUTHOR> https://status.github.com/

<comment><USER2> wow I knew it. Everything _is_ absolutely crawling. I am trying to push up my repo and get some packages. I could tell something was definitely up. Damn it. Thanks for the update.

</comment><comment><USER3> Couldn't push, so came here to find out it wasn't only me. 'Server aborted the SSL handshake'

Edit: seems back to normal now

</comment><comment><USER4> What I actually find more interesting is the spike in hook delivery time that correlated with the spike in exception rate.

My guess would be that some hooks were timing out which possibly causes an internal exception? Maybe? It is interesting that they would include timeouts of external services in their exception count. Purely speculation though.

<comment><USER5> Ruby with resque/sidekiq will definitely cause an exception if an http call fails or times out. This is what causes it to end up back in the job queue.

</comment></comment><comment><USER6> 00:38 Eastern Standard Time All systems reporting at 100%.

[https://status.github.com/messages](https://status.github.com/messages)

</comment><comment><USER7> I'm working on GMT+3, and our work-day starts at 7am. Sort of sucks to have my core work hours be the typical SFO downtime slot...

</comment><comment><USER8> Looks like I wasn't the only one! Wonder who's behind this one.

</comment><comment><USER9> It's completely unusable for me :(

</comment><comment><USER10> works for me now.

</comment><comment><USER11> How do we know it's a DDOS and not some sort of operational failure? Mods should change the title until it is confirmed.

<comment><USER12> Ok, we changed the title from "GitHub under DDOS."

<comment><USER13> Look at the traceroute. This is a DDoS mitigated via Prolexic.

<comment><USER14> Wouldn't the traffic run through to Prolexic regardless if it was under a DDOS or not?

<comment><USER13> Typically, DDoS mitigation services offer a limited number of 48-72 hour mitigations per year (can be a large number, depending on the contract). Outside of these periods the traffic is not routed via them.

<comment><USER14> How does that work? Does the DDOS mitigation software change DNS and if so how can it change the DNS in the middle of an attack assuming the attackers?

<comment><USER13> Only very basic DNS mitigations services have you change DNS and proxy through them. Predictably GitHub is not using one of those.

If you really want to protect your service (your own DNS or say SMTP, WebSockets, TCP), then you need to change the internet routing. This is done via BGP announcements of your IP subnets, such that you announce your _inbound_ routes via the mitigation providers. The providers scrub the traffic and deliver clean traffic via a GRE tunnel to your routers. The outbound traffic is routed directly via your upstream providers.</comment></comment></comment></comment></comment></comment></comment>


Reverse engineering the Notability file format - <AUTHOR> https://jvns.ca/blog/2018/03/31/reverse-engineering-notability-format/ <comment><USER2> To summarise, it's a .zip file containing a binary serialisation of XML, which then contains the base64'd raw coordinates...

Reminds me of this (it gets even better in the comments): [https://thedailywtf.com/articles/XXL- XML](https://thedailywtf.com/articles/XXL-XML)

 _Nothing in here was really complicated – it was just some existing standard formats (zip! apple plist! an array of floats!) combined together in a pretty simple way._

It's rather telling that wrapping points in 3 layers of encodings is considered "pretty simple", almost the norm today, when years ago doing such a thing would probably elicit reactions more towards _WTF_.

<comment><USER3> • Very likely the files are only .zip files for network portability, and when in their native environment (iOS?) they're unzipped document bundles (see e.g. the RTFd format)

• Likely this file isn't literally a PList, but the output of a data storage framework (e.g. CoreData) that uses PLists as one possible backend. In the code, it just looks like adding annotations to native data structures to make them "storable" within a database-like object.

• Only the text-formatted PLists contain base64'ed binary data. The binary- formatted PLists just contain length-prefixed binary data. There isn't the "binary in base64 in XML in binary" that you're imagining.

<comment><USER4> thanks for this!! I've never worked with iOS so I was definitely fumbling in the dark. the CoreData hypothesis explains why the PList file looks like a huge weird unstructured array! So in the code there's just some object with all the data in it and then CoreData serializes that object?

This explains the feeling throughout of "this doesn't feel like a format the app developer _designed_ ".

</comment></comment></comment><comment><USER5> Julia Evans is the best. I've been following her on Twitter for years because she brings so much enthusiasm about learning technology that she makes me see familiar things anew. By the end of an exploration, even it it's something I know well, I end up saying, "Holy shit, strace is cool."

To get a feel for her work, a bunch of her posts are here: [https://jvns.ca/categories/favorite/](https://jvns.ca/categories/favorite/)

</comment><comment><USER6> Reminds me of when I had to reverse-engineer the configuration and language format for an audiobook reader - I think it was the Milestone 312 [1]. We bought it for our granpa who is hard of seeing; but installing the polish language pack would stop the device from working correctly.

In the end I had to compare a working english pack and the disfunctional one; the actual issue was something pretty simple - the names of the properties in their ini file were bad for some languages, and some checksums were wrong, or something like that. But it made for interesting black-box experience; there's of course no feedback about what went wrong in the device, nor any accessible logs :P

[https://bones.ch/milestone312.php](https://bones.ch/milestone312.php)

</comment><comment><USER7> Totally randomly I came across a paper that directly addresses this, _The Next 700 Data Description Languages_ [0].

      Today, many programmers tackle the challenge of ad hoc data by writing scripts in a language
      like PERL. Unfortunately, this process is slow, tedious, and unreliable. Error checking
      and recovery in these scripts is often minimal or nonexistent because when present,
      such error code swamps the main-line computation. The program itself is often unreadable
      by anyone other than the original authors (and usually not even them in a month or two)
      and consequently cannot stand as documentation for the format. Processing code often
      ends up intertwined with parsing code, making it difficult to reuse the parsing code for different
      analyses. Hence, in general, software produced in this way is not the high-quality,
      reliable, efficient and maintainable code one should demand.

[0] [https://www.cs.princeton.edu/~dpw/papers/ddc- jacm.pdf](https://www.cs.princeton.edu/~dpw/papers/ddc-jacm.pdf) (pdf)

</comment><comment><USER8> _Way_ back in the day, I built a PocketPC HTML editor. It used a plain textbox control. Meanwhile, a lot of people were building apps and had expressed interest in having some kind of a rich text edit control, similar to the RichEdit control available as part of the desktop Win32 API. Microsoft's PocketWord app used a rich text control, but it wasn't actually part of the mobile Win32 API, or documented at all.

After finding a couple hints on what WndProc messages the control would respond to, I spent a few weeks reverse-engineering it, and wrote up my findings here: [https://groups.google.com/d/msg/microsoft.public.dotnet.fram...](https://groups.google.com/d/msg/microsoft.public.dotnet.framework.compactframework/MOk776gh5Fs/9VUDZhFJTokJ) .

I never did wind up building anything useful with that information, but it was a pretty fun exercise.

</comment><comment><USER9> Great fun :) Good insight into the process of reverse-engineering and re- generating. I'm working (on and off) on something similar, to convert Kdenlive project files to Final Cut Pro, so that I can collaborate with people who don't use Linux. Both formats are XML, but figuring out the semantics can be tricky!

</comment><comment><USER10> Does anyone here know of a native linux drawing toolkit? Something like the apple pencil? I've wanted a portable not taking device and all of the native Linux not taking software is not up to spec with OSX/Windows counterparts.

It's kind of a tangent but it's somewhat related.

<comment><USER11> There's a developer at SUSE who has done a lot of work on tablet mode, fwiw.

</comment></comment><comment><USER12> I kind of want to see that ACID comic now.

<comment><USER4> [https://drawings.jvns.ca/acid/](https://drawings.jvns.ca/acid/)

</comment></comment><comment><USER13> Nice write-up.</comment>


Ask HN: Interesting topics for an intro InfoSec / cryptography lecture? - <AUTHOR> I'm going to hold a half hour presentation about information security and cryptography for a mostly non-programmer audience in a couple of weeks. They are however all master students in engineering, just not in CS. The purpose of the presentation is mostly to provide some insight in how stuff actually works, but still in a comprehensible and hopefully interesting and even fun fashion.
I remember when I first learnt about Diffie–Hellman key exchange and RSA, and how cool I thought it was. So I'm thinking about going trough an example using color mixing as the secure one way function (something like this video https://www.youtube.com/watch?v=YEBfamv-_do).
Other topics I'm thinking of including is how the CA-system works, in contrast to a web of trust. Maybe also proof-of-work based blockchains or Samir's Secret Sharing algorithm.
Do you know of something cool I could share? <comment><USER2> Yes, that video is one of my favorites.

I'd recommend thinking of teaching encryption like teaching knots.

You start out by listing the most basic knots[1] and what they are used for.

Then, you can dive into teaching how to tie a specific knot.

Then, you can dive into how and why a specific knot works and has the features it does.

In a half hour, you will only be able to go through the list of encryption fundamental tools (digests, symmetric encryption and signatures, asymmetric encryption signatures, and shared secret key derivations). But learning just the names and purposes of those basic tools will be damn interesting to engineers.

Also, encryption is like knots in that you should never try to invent your own. If you invent your own, it will probably fall apart at the most critical moment. Use a standard knot. Use a standard, boring, established knot.

[1] - [https://gizmodo.com/how-to-tie-the-only-five-knots-youll- eve...](https://gizmodo.com/how-to-tie-the-only-five-knots-youll-ever- need-1696628878)

</comment><comment><USER3> While 30 minutes is not a lot by any means, I hope you will take the opportunity to get across a few simple points to them:

1\. Patch your systems

2\. Use a non-trivial, _long_ password. 13+ characters at least. They cannot be easily cracked. Use a password manager and do not reuse passwords.

3\. Look out for phishing emails. Just previewing a message without clicking already gives us the password hash

4\. Chrome is more secure than IE, iOS is more secure than Android

5\. An overview of "smart" crypto choices, MD5 is simply not good enough to save passwords with

6\. Assume breach and set up decent monitoring alerts

Following these is already going to give security people, and by extension attackers, a harder time getting into networks and systems.

Yes, this is an extremely simplified view and without even the slightest nuance. But better this in 2 minutes at the end of a 30 minute talk than nothing.

</comment><comment><USER4> Bring a wifi pineapple and explain how insecure wifi really can be.

<comment><USER5> fear mongering

</comment></comment><comment><USER6> Setup a MITM proxy for an iPhone and show them how much private data is going over the wire.</comment>


The Dark Ages of Austin Startup Capital - <AUTHOR> http://techcrunch.com/2016/03/05/the-dark-ages-of-austin-startup-capital/ <comment><USER2> To me, the article shows that "startup" means something different outside the Silicon Valley bubble.

Taking 40% for $400k is probably counter productive for an investor shooting for a $1 billion plus exit. Consuming so much equity [the primary asset of an early stage company] so early for so little capital means that equity is not available for bigger rounds later.

On the other hand, for normal capital, if the business grows to $500k a year in profit over the next five years, it provides a handsome internal rate of return for a local group of investors at the Chamber of Commerce scale.

The difference I sense, is that the "Austin" investors are hardening against future rounds of investment rather than encouraging it. With a 40% stake, such investors are likely to be the biggest single shareholder and hence have a high degree of control over future deal flow.

</comment><comment><USER3> Hmm, I don't see this as a problem at all. Austin was a great place to be a software developer before that 2013 investment record they list. The cost of living is such that bootstrapping is the way things are done.

I don't see anyone hurting and it isn't raining money. So give it a little time, let those funds find a few exits and we can do it all again.

Deep Eddy Vodka and Sweet Leaf Tea aren't tech startups. If anyone wants to know why Deep Eddy Vodka was successful look just as far as the name. It tapped into some deep Austin history that everyone was familiar with. Made it instantly familiar. I'm still sad that tree got sick and they had to take it down.

</comment><comment><USER4> I can't say that I'm surprised that Austin based VC's are more risk averse than SF counterparts. Austin based startups are higher risk for several reasons. This is the city with a start ups desperately searching for veteran engineers when their CTO is 3 years out of school...

The "startup" of Austin is typically one that adds less value to their vertical than their SF counterparts, has less talented technical staff, and has very poor non-technical talent. ...But people are complaining that Austin based VC's are giving Austin based startups less love. I think some of these startups need to re-evaluate what they're doing and stop blaming VC's for not instantly falling in love with their flavor-of-the-week startup.

<comment><USER5> Not sure what experience is influencing your comment, but it's simply not true. I know many Bay Area based companies that are hiring more and more technical talent in Austin because they can not find it where they are.

Is there more overall technical talent in the Bay Area? Absolutely, but that doesn't mean the talent elsewhere is "very poor" and "3 years out of school".

<comment><USER4> You simply didn't read what I wrote. I didn't say there isn't talent in Austin, I'm saying these startups don't have it, and can't seem to attract and keep talent.

</comment></comment></comment><comment><USER6> _Look to the Deep Eddy Vodka acquisition as one. This was a quick exit for Clayton Christopher after his win for Sweet Leaf Tea_

Neither of which are tech startups.

<comment><USER7> I'd argue they're both not even startups. This is not meant degrade both companies in any way - just to clearly differentiate between two approaches suitable for different business models

The classic definition by Steve Blank is: "[A] [...] startup is an organization formed to search for a repeatable and scalable business model." [0].

Selling iced tea and vodka is a straightforward, traditional business model. Nothing to search for.

[0]: [http://steveblank.com/2010/01/25/whats-a-startup-first- princ...](http://steveblank.com/2010/01/25/whats-a-startup-first-principles/)

<comment><USER8> Selling advertising is also a straightforward traditional business model.

</comment></comment></comment><comment><USER9> Seems like a small sample size. His startup got different valuations from different set of VC's, which is a normal occurrence.

</comment><comment><USER10> I've heard people say that Austin no longer has a better startup scene than Dallas. It's probably not true, but it reaffirms that Austin's startup scene is hurting.

<comment><USER11> I'm not familiar with the scene in Dallas. Is that a high bar to beat?

<comment><USER12> Short answer: no

Long answer: Dallas has a lot of investment money, however most of that investment money goes towards cash flow businesses like commercial real estate, oil speculation, and banking. However, there has been a pretty sizable uptick in investment dollars willing to put towards tech companies within the last 4-5 years. It's relatively straightforward to find capital in Dallas, but it's insanely hard to find capital willing to stomach high burn early stage tech companies.

</comment></comment></comment><comment><USER13> While the amount of seed and early investors nearby I'm sure has a major role in startup culture building I still think Austin is a great city for startups.

Not all startups need to raise money... but most startups need smart people and a fun place to live. I think Austin provides both.

</comment><comment><USER14> _Austin's funds have less money available for early-stage investments because later-stage deals have eaten up a lot of the fund._

No, Austin has less money available for early-stage investments because Austin has an order of magnitude less VC money, period. Most of the effects that the author observes (not many big exits, lots of smaller investors, etc) can be readily explained by that one simple fact. Wealth concentration isn't just about the 1% vs the 99%, it's also about the 0.001% vs the 0.01%. And Silicon Valley will continue to widen the gap because they can place more and bigger bets, and get bigger and more profitable exits as a result.

</comment><comment><USER15> Am I wrong to think part of the problem is that the best developers in Austin don't work for Austin companies?

<comment><USER16> No you're not. The tech companies in Austin are divided in two groups: Bay Area/Boston companies that pay Bay Area/Boston salaries, and Bay Area/Boston companies that pay Texas salaries. One attracts talent that sticks around, the other attracts fresh grads who leave once they realize how much more they're worth.

</comment></comment><comment><USER17> Is the trend of falling seed rounds confined to Austin or does it extend to other US startup hubs? A report by Compass (earlier Startup Genome) last year showed Boston and Bangalore were among the most funded hubs, but Boston saw a fall in early stage deals while it was the reverse in Bangalore... [https://www.techinasia.com/bangalore-worlds-startup- ecosyste...](https://www.techinasia.com/bangalore-worlds-startup-ecosystems)

</comment><comment><USER18> Spot on. I know a founder who had to go through 15 meetings just to close one VC in Austin.

<comment><USER19> That's not weird, even on sand hill road. I met with about 75 firms before getting a seed investment, and did probably 10-15 meetings with that one before we closed. First time founder, and they made it clear how serious their interest was at each stage (meetings after the fifth were with every intention of investing, which they did).

<comment><USER20> Can you describe the progression of what happens in those 15 meetings? What do they need at meeting #14 they couldn't get from meeting #5?

<comment><USER21> It should be clear if they're serious by the second meeting, and in many cases, the first. The throwaway comment "Spend 88 percent of your time building your business and 12 percent of your time raising money" sounded right. If an investor doesn't understand that, they're broadcasting that they'll damage your business and don't belong on a board. Thankfully, interviews are two-sided :)

</comment></comment></comment></comment><comment><USER11> What does something like this imply for an employee that likes to work for startups? Is it much better to live in the bay where there is a much deeper market of open positions to keep developer demand/salary higher?

<comment><USER12> As someone who lives outside of the valley, what this article says to me is that I should be tougher in vetting the potential sources of revenue that the startup has and not just a startup that I think should exist in the market!

<comment><USER4> ^ this. Austin is still a great place for startups, as a talented professional you just have to be careful about which startup you're jumping in to.</comment></comment></comment>


Don't use Linksys routers - <AUTHOR> https://superevr.com/blog/2013/dont-use-linksys-routers/ <comment><USER3> The modern equivalent of a Linksys WRT54GL is the ASUS RT-N16. It runs OpenWrt, DD-WRT, and Tomato variants really well, does 802.11n (only one frequency) and has plenty of memory and flash storage for extra hacking. The ASUS RT-N66U is frequently advised if you want 5GHz 802.11n as well.

The other router mentioned in this article, the Linksys EA2700, doesn't seem compatible with third party firmware. And apparently the Cisco firmware is buggy, no surprise there. It is an awfully cheap Dual-Band 802.11n router, but if you can't put working software on it it's useless.

I don't understand why some major router manufacturer doesn't just sell routers pre-installed with Tomato. It's easy to use, stable, and works way better than any crap the router companies cobble together. Flashing new firmware on a stock ASUS router is too complex for ordinary consumers.

<comment><USER4> Tomato works fine until you need QoS. Then it hangs sporadically. Verified with two different router types.

<comment><USER5> Interestingly I found the N16 stock firmware to have the best QoS of any router I've used. It requires no configuration and always seemed to work great. DD-WRT's QoS, on the other hand, can require significant effort to configure correctly.

</comment></comment></comment><comment><USER6> Or just put OpenWRT[1] on it. It's a real Linux distribution with a package manager and everything. You can even disable the webinterface, if you don't trust it, and use SSH.

EDIT: WRT54GL is pretty old and it won't run the default build of OpenWRT Attitude Adjustment (the newest release). It also probably won't have enough memory to operate the package manager or the webinterface.

But I do have one running a custom build. The only downside is that you need to decide which software to include upfront. Their build tool is rather friendly[2].

EDIT2: You can have a VPN server and any routing you like on OpenWRT, same with Samba, radvd, vnstat... There are even webUI pluings (luci-app-whatever) so you can control those from the webinterface for ease of access. It is a real Linux distro that just happens to run on routers.

[1] <https://openwrt.org/>

[2] <http://wiki.openwrt.org/doc/howto/build>

<comment><USER7> The trouble with the WRT54G series (and most of these little routers) is that they have ~200MHz CPUs and ~16MB of RAM. This is, incidentally, why they often crash when you open a lot of simultaneous connections -- memory exhaustion.

I find that if you're interested in experimenting with a Linux router, old PCs are a much better choice. You can get a PowerPC G3 or G4 or a late model Pentium III for practically zero money (if not literally zero money out of a trash heap) and PCI NICs for secondary interfaces are similarly inexpensive. For only slightly more money the G4 Mac Mini is an excellent choice for a wireless router. Then you have a processor that is several times faster and can put arbitrarily much memory and storage in it to suit your needs and then put your favorite Linux router distribution (or Debian) on it and have at it.

<comment><USER8> What is it about these class of devices that makes them so expensive?

They tend to have 500Mhz CPUs, approximately no RAM, unresponsive very basic web interfaces, and fall over at the touch of a light breeze.

An el-cheapo Android tablet with 1.6Ghz dual core ARM processor, 1Gb RAM, WiFi, and a bunch of other technically hard stuff on top (IPS screen, battery) costs less than one of these style of WAP/switch/routers.

Are they really so different?

<comment><USER9> I was always wondering that too. Seems like they should be $25-50 max. Decent hardware with Tomato seems like the way to go.

</comment></comment></comment></comment><comment><USER10> Linksys went from being the "iPhone of home networking" to being something I won't recommend. In Cisco's care the company has gone from being a market leader to a dud.

Now a lot of people might say "I doesn't matter who makes it, I'll just flash OpenWRT or DD-WRT onto it!" But I say to that, "then why buy a Linksys?"

Asus for one example are cheaper, they often have external antenna giving you more power and flexibility (both literally and figuratively) plus and most importantly they can be flashed with OpenWRT or DD-WRT at your pleasure.

Even without the security issues there is no good reason to buy a Linksys.

Right now I am using my ISP supplied "router" in cable-modem "mode" (i.e. just dumb pass-through to ethernet) and have a cheap MikroTik/RouterOS device sitting behind it which was cheaper than most retail grade routers but with the functionality of commercial grade equipment.

RouterOS might not be as easy to use as DD-WRT, but if you can use it then it is far more powerful as a web-based environment. Just for one example, want a VPN server? RouterOS supports IPSec/L2TP, PPtP/GRE, SSTP, and OpenVPN. Basically everything. The list of its network functionality is almost endless...

<comment><USER11> "being the "iPhone of home networking""

Howso? All I remember from their heyday was that they were good enough, cheap enough, and flashable. I don't remember them commanding a premium for any particular reason.

<comment><USER12> I think they meant it not in terms of a premium price, but rather as the default product to buy. They were never that much more expensive (if at all), but back in the day it was the one most people bought.

<comment><USER13> Should have said iPod then as I don't think iPhone ever attained "default" status outside of SF.

</comment></comment></comment></comment><comment><USER14> I highly recommend Mikrotik to anyone fed up with traditional consumer wifi routers/APs. I dont know how they compare to other vendor hw eg Asus + OpenWRT, but this little guy has been rock solid and a joy to use: <http://routerboard.com/RB2011UAS-2HnD-IN>

<comment><USER10> I have a Mikrotik/RouterOS device.

Positives: Cheap. Powerful. Stable.

Negatives: Harder to use than OpenWRT/DD-WRT or similar. It is still a web- interface, but doesn't "baby" the user. If you aren't comfortable manually setting up interfaces and then setting up tunnels through those interfaces for example then skip this.

I love it. But I won't kid myself, it isn't for everyone. The documentation isn't comprehensive and the software is very powerful but not very intuitive (or at least it isn't if you don't have a good background configuring network equipment).

<comment><USER15> Anyone who has ever set up a Cisco router to do anything even slightly complex should be able to pick up Mikrotik in a snap. The windows based GUI configuration tool (winbox) is simply incredible. (Runs fine in wine on OSX as well)

<comment><USER16> they rewrote it into a browser based app, looks identically, just a bit more shiny.

</comment></comment></comment></comment><comment><USER17> My Linksys router has had Tomato on it from the first day, it's the only sane thing to do (OpenWRT or DD-WRT would work too) when closed-source software is regularly exploited and not patched in a timely manner - and when noone knows what kind of government-friendly backdoors exist in such products (made by companies that earn significant revenue from government contracts).

Also, there's plenty of very cheap router hardware coming from China nowdays, from TP-Link you can get OpenWRT-capable routers for less than $15, so there's not much point in paying a lot more for Linksys products.

</comment><comment><USER18> Having looked at the post, doesn't he really mean don't buy "these models" of Linksys? Or are all models open to certain vulnerabilities?

<comment><USER19> This indicates that they're not doing enough testing of _any_ of their routers. Not conclusive, of course...

</comment></comment><comment><USER20> I use Linksys routers, but not with default software (which we know is "limited"). I would recommend alternatives from here <http://tomatousb.org/mods>

<comment><USER21> Which makes it a bit odd that he mentions the "uber-popular Linksys WRT54GL router" specifically. The "L" stands for linux, and it was brought back because people specifically wanted to install third party software on it.

And the reason for it becoming popular in the first place was probably a security issue that allowed third party software to be installed.

The "L" version was introduced because newer routers didn't have this capability/vulnerability and people wanted something they could install third party software on. So when the "L" version (which I use) came it was just an older model, with even less memory than the original and a much heftier price. Unless you wanted to run third party software on it it was really bad value for your money.

Anyway, all of this truly sucks. But really, I don't expect more of any consumer router.

EDIT: Oh wow, the WRT54GL was introduced in 2005, nothing too fancy at the time and you can still buy it today - lots of stores have it in stock even.

<comment><USER22> That's inexact. The WRT54GL was brought in as the (at the time) retail version of the WRT54G had much less RAM and flash because they switched from Linux to VXWORKS.

It's not that they didn't have the "security vulnerability" but they just weren't able to run Linux in a useful manner due to low hardware resources.

I also don't remember if the WRT54G became popular in the first place because of a security issue, I think it just enabled you to upload any firmware to it and that the original firmware eventually became open-source after they received GPL violation complaints. But my memory might be fuzzy, it was a while ago.

<comment><USER10> Your memory is how I remember it too.

Basically the WRT54G with base firmware was better than anything else on the market at the time of release (within the same market segment - <USER2> routers).

Just to put that into some perspective before the WRT54G, some of the functionality in the base firmware was being sold to small-medium businesses by companies like Cisco for thousands of dollars.

Word spread quickly and instead of your local coffee shop paying Cisco $20,000 to install their WiFi, they could spend $100 on a Linksys router. This meant companies could afford to give away WiFi for free because it cost them little or nothing to install the WiFi initially.

But what happened next is what turned the Linksys WRT54G from a "great" to a "legendary" product - people found out it ran on Linux. Now Linux is open source but more specifically it is under the GPL license.

What that meant is that legally Linksys were required to share the source code that made the WRT54G run. Which after some not-so-gentle prodding and legal threats they did.

People then made distro's (in the Linux sense) which updated the Linksys firmware to add new functionality, fix issues, and similar. This made the thing even more powerful than perhaps even Linksys wanted, and ate into Cisco's small-medium business space even more.

Word spread like wildfire and soon everyone and their brother owned a Linksys WRT54G. Linksys improved the base firmware only mildly while the third party firmware was steamrolling ahead.

Cisco eventually purchased Linksys and started cutting corners on all of their retail products. Using less powerful CPUs, less RAM, and stripping out functionality while not altering the cost. Linksys stagnated.

This was likely in no small part to try and get some of their small-medium customers back onto Cisco's books, but by then it was too late. The market that Linksys had created had spread to Linksys's competitors and soon everyone was "letting" their routers get firmware updates that turned a cheap little home router into something able to fend off medium-business level commercial equipment.

<comment><USER23> AFAIK Cisco bought Linksys back in March 2003.

</comment></comment></comment></comment></comment><comment><USER24> I'm a fan of pfsense [1] on an alix board [2]. The alix boards a little pricey for a router, but has a real amount of memory (256MB). The only downside is that pfsense, since it's based on FreeBSD, doesn't support any 802.11n cards, so you're either stuck with 802.11g, or using a separate access point like I do.

Add in a managed switched and you have the start of a real network at home.

[1] <http://pfsense.org/> [2] <http://pcengines.ch/alix.htm>

<comment><USER25> FreeBSD supports some 802.11n cards, doesn't it? <https://wiki.freebsd.org/WiFi80211n>

<comment><USER24> There are no drivers for 802.11n cards in pfsense 2.0 (current stable version), which is based on FreeBSD 8.1. Some b/g/n cards will work, but only in b/g modes:

[http://doc.pfsense.org/index.php/Is_802.11n_wireless_support...](http://doc.pfsense.org/index.php/Is_802.11n_wireless_supported)

Drivers for 802.11n are in FreeBSD 9.0 and later, but that won't be the base for pfsense until 2.2:

[http://doc.pfsense.org/index.php/PfSense_and_FreeBSD_Version...](http://doc.pfsense.org/index.php/PfSense_and_FreeBSD_Versions)

</comment></comment></comment><comment><USER26> Let me just get this in before the cries of JUST INSTALL OPENWRT come raining down.

Your mother / father / grandmother / etc are not installing openWRT on their routers. Installing one of these CISCO home routers is pretty much hacking yourself. And, just update the firmware is not gonna work.

Try it one day, go up like 10 people and ask them what's a firmware. If the user isn't technical, you're going to get a 0/10 correct responses.

<comment><USER27> Maybe they should just ship with the alternative firmware installed if the open source is doing a better job?

<comment><USER19> I've read (but haven't personally confirmed) that Netgear used skinned OpenWRT on some of their routers (like the wndr3700v1)

<comment><USER28> It is true <http://www.myopenrouter.com/> is their info page. They make it really easy to flash, and provide specs as well so they are well supported so you can use a different version, but it does ship with it too.

</comment></comment></comment></comment><comment><USER29> It should be pretty easy to upgrade vulnerable WRT54GL routers. Any volunteers to setup a page that POSTs a newer firmware like OpenWrt or Tomato?

<comment><USER30> What use is that? If people don't know enough to upload a file to their router, they definitely won't know enough to configure it after it's been done.

<comment><USER31> Embed the link in a joke/lolcat/puppy/political forward-email and spam it to all of our relatives/acquaintances who regularly do the same.

</comment></comment></comment><comment><USER27> Is this a problem if I have DD-WRT?

<comment><USER32> No.

</comment></comment><comment><USER33> Just buy Draytek. Playtime is over. Pricy but mine lasted 7 years before I replaced it with another Draytek (for dual WAN support). Bomb proof and great VPN support out of the box. I bridge my parent's network to mine over VPN, and the Linux servers at either end provide failover DNS, mail etc. So useful, especially for remote support.

<comment><USER34> > Just buy Draytek. Playtime is over.

I went to their website (draytek.us) and got this:

> Database connection error (2): Could not connect to MySQL.

<comment><USER35> I guess playtime really _is_ over.

</comment></comment></comment><comment><USER36> What recommendations do people here have for an entry level commercial router instead of a high consumer level router?

<comment><USER37> Juniper SRX100 ($500), with Ubiquiti APs (~$100).

<comment><USER38> That's a bit overkill don't you think? Do you really think a home user would have a need to run BGP?

<comment><USER37> Overkill for a random home, but maybe OK if you WFH and need to vpn/IT wants manage a bunch of devices centrally/etc. $1k in equipment vs. $300 in equipment isn't _that_ big a difference for a few important home users if you're already using all the same equipment in your other offices.

(also a lot of startups have like 20 people working out of a home or condo, and they bump up against memory/crapiness/etc. limits of consumer routers and APs pretty hard.)

OP asked for "low end commercial router", though.

</comment></comment></comment></comment><comment><USER39> Google Cache of the site since it's having issues. [0]

I enjoy my Asus RT-AC66U. [1] Best commercial router I've seen, and Asus Merlin [2] firmware makes it better.

[0] [http://webcache.googleusercontent.com/search?q=cache:JNu4Z9X...](http://webcache.googleusercontent.com/search?q=cache:JNu4Z9XbAz0J:https://superevr.com/blog/2013/dont- use-linksys-routers/&hl=en&gl=us&strip=1)

[1] [http://www.newegg.com/Product/Product.aspx?Item=N82E16833320...](http://www.newegg.com/Product/Product.aspx?Item=N82E16833320115)

[2] <https://github.com/RMerl/asuswrt-merlin>

<comment><USER38> The only problem with Asus Merlin is that it uses the older "stable" wifi drivers. I've been running the unreleased Russian build 22.214.171.124.321 for months now and it's stable as a rock.

</comment></comment><comment><USER40> My ISP makes me use a "gateway"[1][2] with a wireless router built into it. In the name of reducing electricity usage, I forego running my own router and surrender to using theirs. I would be willing to bet many others do the same. I wonder how secure it is?

[1]<http://www.att.com/u-verse/explore/residential-gateway.jsp> [2][http://verrytechnical.com/wp- content/uploads/2011/10/ATTUver...](http://verrytechnical.com/wp- content/uploads/2011/10/ATTUverse2WireRG.jpg)

<comment><USER24> My router uses about 3 watts. That adds about $5 per year to my electric bill, which I'm happy to pay to avoid the painful ISP-provided router.

</comment></comment><comment><USER41> Half-OT: does anyone know a DD-WRT/OpenWRT compatible WiFi router with support for 2.4/5 GHz WiFi, as well as VLAN on the ports? Bonus points for individual VLAN assignment to the individual ports.

</comment><comment><USER42> What? Linksys routers are a great deal - you can find them at goodwill for $5, flash the firmware & configure it in 15 minutes and they work great. My one beef is why don't they put a cheap fan on them when they cost upwards of $100 now that they come with a cisco logo slapped on them.

</comment><comment><USER43> Shouldn't 'hardware' firewalls be secure? And everyone knows that software firewalls are crap. Isn't this common knowledge with professionals. ;)

<comment><USER19> ;) In the end, all routers are software. A properly set up Linux or BSD router/firewall on a regular PC can be very, very good.

A higher end hardware router actually has tested and (mostly) secured software, these don't...

</comment></comment><comment><USER44> I mostly-bricked a Linksys doing a security analysis on it. It still works, but the UI is completely locked up; I can change nothing on it.

<comment><USER45> I also bricked my Linksys by opening /upgrade.cgi (mentioned in the article) on a browser.

I was able to finally fix it by downloading a firmware from Linksys, doing a 30/30/30 reset (push reset button for 30 s, turn power off for 30 s and keep on pushing reset for another 30 s after turning it on again) and flashing it with tftp as explained in [http://community.linksys.com/t5/Wireless- Routers/E4200-Firmw...](http://community.linksys.com/t5/Wireless- Routers/E4200-Firmware-Upgrade-failed-Cannot-access- Linksys/m-p/552862/highlight/true#M236523)

After that I was able to login using the web interface again.

For Mac OS X the command to flash is just tftp, and then in the console that opens type:

        connect 192.168.1.1
        binary
        rexmt 1
        timeout 60
        put firmware_filename.bin

</comment></comment><comment><USER46> So this researcher went from notifying Linksys to open disclosure to the internet after only a month? That hardly seems responsible.

</comment><comment><USER47> Mitigating factor, the attacker would need to have been granted access to your network in the first place?

<comment><USER48> Not at all. That's what CSRF is all about. All the attacker needs to do is get you to visit a page on the internet that they control. Then the code on the page does its magic and runs on the browser and because _you_ and the browser are on your network it can work.

As a simple example, imagine that you had a test server behind a firewall in your own home network, totally inaccessible from the internet. Now let's say you have it set up so that it will, oh, let's say turn on the oven if you hit a specific URL without any authentication (like testserver/actions/oven/on, or some such). If someone knows of this then they could contrive to have you visit a web page with some embedded resource such as an inline image that causes you to hit that url from your browser. Boom, now your oven is on and you didn't even know it. Even if you switch to using logins and cookies on your test server to ensure that only authorized users on your network can use it then you'll still have the same problem, because when _your_ browser hits that URL it will be in your name, and all of the right cookies will be there. That's the nature of CSRF.

</comment></comment><comment><USER49> I think this is just what topic meant:

Error 102 (net::ERR_CONNECTION_REFUSED): The server refused the connection.

LOL'd

</comment><comment><USER50> OpenSource is a vulnerability not an Asset according to #5.

For small projects which few contributors I would agree but, for projects as large as OpenWRT and DDWRT and such, I don't agree.

<comment><USER51> In an ideal world, whether code is open source is neutral. The extra ease of finding bugs is balanced by the fact that people can and do find then fix them. Theory says that these two are, to first order, equivalent. So end user security is the same. (But code quality tends to be higher with open source software.)

However whenever code moves towards being more open, you've got all of the vulnerabilities of closed source software, and all of the bug-finding ease of open source software. This is the worst of all possible worlds.

Therefore #5 is true. The fact that you have easy access to known-to-be-crappy code increases the vulnerability of that code.

<comment><USER52> Just curious, but wouldn't this indicate that open source code which allow iterators to close off their improvements would produce less vulnerable code overall?

<comment><USER51> If a vulnerability is found in open source code, people will try it on yours. So they won't be finding it directly in yours, but that is not protecting you.

The real consequence is that how secure a product is depends more on the project than on whether it is open source. Apache and OpenBSD are two examples of very good open source code. Java and Rails are two examples of not so good open source code.

Google's website is an example of good closed source code. The software shipped by Linksys is an example of bad closed source code.

<comment><USER52> I get that there are different levels of quality regardless of the type of code. I was more interested in the security effects of hiding code after the open source community has had a chance to deal with vulnerabilities. None of the examples you gave were specific to code which has transitioned between open to closed.

What I've gotten from your answer so far is that it isn't an effect which is general, and it'll depend on the project in question. Am I on the mark?

<comment><USER51> Yes. Opening closed code is always going to be problematic. But closing opened code can go either way.</comment></comment></comment></comment></comment></comment>


How to build your own neural network from scratch in Python - <AUTHOR> https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6 <comment><USER2> Too often you see articles like this and they start with

$ import a_whole_bunch_of_stuff

Good to see that this is not the case here :)

The fast.ai course has a similar exercise in the beginning, but you'll still import the weights from somewhere else.

Their fast.ai v1 library has a very short implementation too (loading the MINIST example dataset and then using Resnet18):

        from fastai import *
        from fastai.data import *
        untar_data(MNIST_PATH)
        data = image_data_from_folder(MNIST_PATH)
        learn = ConvLearner(data, tvm.resnet18, metrics=accuracy)
        learn.fit(1)

Done!

Source: [http://docs.fast.ai/](http://docs.fast.ai/)

<comment><USER3> How is your example not "$ import a_whole_bunch_of_stuff"?

<comment><USER4> This notebook from fast.ai is more in line with what op mentioned. It starts from nothing but Python and slowly swaps out chunks of code with functions/classes from Torch/fastai:

[https://github.com/fastai/fastai/tree/master/courses/dl1](https://github.com/fastai/fastai/tree/master/courses/dl1)

</comment></comment></comment><comment><USER5> Almost obligatory, but Stanford cs231n [1] lets you implement a complete (deep) neural network from scratch [2] before venturing into pytorch/tf. Its super fun.

[1] [http://cs231n.stanford.edu/](http://cs231n.stanford.edu/)

[2] [http://cs231n.github.io/assignments2017/assignment1/](http://cs231n.github.io/assignments2017/assignment1/)

</comment><comment><USER6> I can highly recommend Joel Grus' live coding video. He creates a deep learning library only using numpy in an hour and it's really fun to watch it all come together. [https://youtu.be/o64FV-ez6Gw](https://youtu.be/o64FV- ez6Gw)

<comment><USER7> That was really good! Thanks for the link.

</comment></comment><comment><USER8> Towards Data Science is definitely one of the best Medium Publications. I hope they don't begin to monetise with Medium's member-only content. It's such a drawback to Medium.

</comment><comment><USER9> Nah, let's make that 10 lines: [https://gist.github.com/vesche/72dafed33d614710f03f1b75cff1c...](https://gist.github.com/vesche/72dafed33d614710f03f1b75cff1c807)

I got a beer for anyone who can golf it under 5 lines

<comment><USER10> Naive solution (1 line):

[https://gist.github.com/applecrazy/deda2fac6e83c07b93e001731...](https://gist.github.com/applecrazy/deda2fac6e83c07b93e0017314de6cce)

Edit: I literally took newlines, converted to \n in a string, and then exec()ed the whole thing. Here's a repl of it working: [https://repl.it/@applecrazy/Code-Golfing-a-Neural- Net](https://repl.it/@applecrazy/Code-Golfing-a-Neural-Net)

<comment><USER9> Oh dear, I got played.

PM address if you want the beer, you sly dog. Obligatory, I'm a normal dude: [http://vesche.github.io/](http://vesche.github.io/)

Edit: Wait, you can't PM on here... PM on reddit /u/vesche

<comment><USER10> Well, I don't drink (see my bio), but thanks for the offer. :P

</comment></comment></comment></comment><comment><USER11> Honest question: what are the reasons to code this up using OOP, creating a neural network object plus methods, instead of data structures and functions that operate on them?

If it's just personal preference, I'm fine with that, I'm not trying to start a flame war.

<comment><USER12> Python, both the language and community, are _very_ strong proponents of OOP. While you _can_ do a lot of more functional stuff, esp. w/ functools, the community at large tends to discourage that. "Never use map/filter" is a weirdly common phrase among pythonistas. So this, like most python-driven examples, is doing things in a _pythonic_ way.

Coming from the R side, I tend to prefer structures & functions as well, but if I tried to write Python that way I'd be wary about showing that code to anyone more entrenched in the Pythonic way of thinking.

<comment><USER13> The never use map/filter is in favor of list comprehensions, which is not really a OOP construct.

But sure, it's a conventional thing. And for a long time the built-in alternatives to a class for such a datastructure tuples and dicts, neither which are very nice for functions to operate on (dict values have to be addressed with d['key'] instead of d.key). With a class and method there is also no doubt as to what the function operates on, which is convenient when there type hints and IDE support is missing. This is changing since Python 3.5 and type checking tools like MyPy.

Since Python 3.7 there are also data classes, a primitive for classes which just hold values. [https://www.linuxjournal.com/content/introducing- python-37s-...](https://www.linuxjournal.com/content/introducing- python-37s-dataclasses) But it will take a while before programming conventions change.

</comment></comment></comment><comment><USER14> I found this very useful:

[https://makeyourownneuralnetwork.blogspot.com/](https://makeyourownneuralnetwork.blogspot.com/) [https://www.amazon.com/dp/1530826608/ref=cm_sw_r_tw_dp_U_x_x...](https://www.amazon.com/dp/1530826608/ref=cm_sw_r_tw_dp_U_x_xmXTBbB5R5A2R)

First part goes into what a NN is and how backprop works, second part is an implementation in Python.

</comment><comment><USER15> Thanks for submitting, this is going to come in handy for my education! I have an assignment next week which requires creating a neural network, then substituting various optimizers in place of backpropagation and comparing performance over iterations. Have found that simple numpy based NN's are easier to examine and connect the changes with the theory, this guide looks to help further with this understanding!

<comment><USER16> That sounds really fun! Do you get to pick optimizers? Obviously direct gradient is a thing (calculate the partial for every parameter) but also particle swarm optimization might be cool to profile.

</comment></comment><comment><USER17> I suggest a remarkably useful device: Four circular disks when attached to a cart make it easy to move the cart from location to another. Furthermore you could put goods in that cart and move them too. A lot easier than carrying the goods on you back or an animal. A brief search of the internet finds this is a new idea.

<comment><USER18> What do the disks do? A cart already moves pretty easily on it's wheels, so how much improvement have you made by attaching these disks to it?

</comment></comment><comment><USER19> Very cool! Here's the full version if you want fo run it/play with it

[https://repl.it/@turbio/neural-network](https://repl.it/@turbio/neural- network)

</comment><comment><USER20> Another thing to try is calculating backpropagation by hand, on paper, with a small NN. This is what my mentor said his final exam on NN in college involved.

<comment><USER21> I would have thought calculus 1 was a prerequisite for that course.

</comment></comment><comment><USER22> If you're looking for a similar from scratch tutorial in Java, check this one out: [https://smalldata.tech/blog/2016/05/03/building-a-simple- neu...](https://smalldata.tech/blog/2016/05/03/building-a-simple-neural-net- in-java)

</comment><comment><USER23> This is not as glorified as it sound. 100 of thousands of students around the world build neural net from scratch as their homework. People please move on, spend your time somewhere else which is more productive. I coded neural network using c++ in 2009 as homework. But I never though about showing this to people.

<comment><USER24> > People please move on

Strange reaction to a "beginner's guide", how can someone move on before they learn the basics?

</comment></comment><comment><USER25> Programming != math</comment>