Mistrust and Verify
Web Advent 2010 / Mistrust and Verify


When working on a web app with coworkers or
fellow volunteers, few things are more important within the group than trust.
Trust among developers enables them to work confidently, eases communication
between team members, and empowers each of them to make good decisions.

With the actual app, however, that trust should never come easily.
With today’s web apps, you can’t trust anything. You can’t trust
input from the user, you can’t trust your code, you can’t trust your
processes, and you can’t trust your systems. So, while the Russian proverb
says, “Доверяй, но проверяй,” meaning, “trust and verify,” these days you need to
mistrust and verify.

The user

A user doesn’t need to be malicious to break your app, but
malicious users do exist, so no input from any user can trusted. Any
data received is immediately suspect, needs to be validated that it is the expected
format (numbers are numbers, dates are dates, postal codes are valid, email
addresses match
RFC 822,
and required data is the right length) and
filtered to remove dangerous input.
Plenty of articles exist on how to avoid
SQL injection,
but preventing
cross-site scripting attacks
and
session hijacking
is also important.

You should mistrust data coming into your app enough to read the
articles in the
PHP Security Consortium library,
follow a post a day for
a month of PHP security,
or spend an hour reviewing the
basics of PHP security.

A great resource (and must-read) for beginners — that is full of great
reminders for experienced developers — is
Essential PHP Security,
which succinctly lists problem areas where all PHP developers should be
cautious, along with solutions.

Fortunately for developers who prefer using frameworks, most PHP
frameworks will handle the data checking and filtering, preventing common
security holes. If you use a framework, embrace its way of handling data so
as not to thwart its processes. Mistrust it just enough to verify it does
what you expect, then sign up for its security mailing list.  When a fix is
reported, download the patch and review the changes; have you made the same
error in your code?

The code

Input from users is only one part of the app; your code is
another  How do you trust your code, which, while brilliant at two in the
morning, is perhaps less brilliant the next morning? You don’t. You write
unit tests.

Unit tests let you automate the tedious job of verifying that your code
works correctly and is returning the right data formats. They let you know
when changes adversely affect the rest of the system.

SimpleTest and
PHPUnit
are two popular PHP testing frameworks. Creation of unit tests have been made
relatively
painless,
but not error-free nor effortless.

After your unit tests are written, testing all of your functions and
libraries, can you really trust them? Not really. Unit tests are
white box testing;
you know what your code is (supposed to be) doing, you know
what input is going into the test, and you know what you’re expecting to be
returned. With all of this knowledge, no, you can’t even trust your unit
tests, as each step is known, but input from users is an unknown.

To be effective, unit tests need to be complete and need to be updated to
handle the new cases you see during development and in production. Each
time a bug is found, a problem discovered, or an exploit exposed, a unit
test should be created for the issue. Brainstorm on edge cases; you might
not put control characters in your form input fields, but people who cut
and paste, or maybe use Emacs, just might. For multilingual applications,
have you tried to break your tests with other languages?

A review of your unit tests from time to time is also a good idea. Ever
look at a completed part of your project and find this?

/**
* Test validName function
* @param string input name
* @return boolean
*/
function testValidName($name) {
    // TODO: write validName test
    return true;
}

The deployment

So, user input is mistrusted, checked, and filtered. Code is mistrusted,
reviewed, and tested.  You’re ready to launch a new feature or an entire new
project. Now is the time to mistrust your deployment processes if they are
not documented, repeatable, and (preferably) non-destructive.  Manual
deployments are error-prone (“Was I on step 3 or 4?”), with undocumented
processes being the worst (“What did you do?” “I don’t recall.”).

Documenting a launch process forces you to think through what steps
need to happen, what requirements need to be met, and what problems you may
encounter. Each step has assumptions that should be questioned. Are
the development versions the same as the production versions for your OS,
web server, PHP, and the database? Is any step resource limited?  What
happens if a step fails? Can you repeat a step without concern?  What could
possibly go wrong here?

Are there steps that require manual intervention? Can these steps be
automated? Automating a deployment process makes it repeatable, and at that
point, debuggable.
Phing,
Capistrano
(yes, you can
use it with PHP),
shell scripts, your own PHP scripts, or even
Ant can be used to build a deployment
infrastructure tailored to your projects’ needs.

What error checks are necessary in that automation? How do you verify
that the deployment is successful? For single-page web sites, answering
“Hey, does the page display?” is sufficient. For larger apps,
you can run unit tests against the live
installation; log into an account, perform a series of common tasks, and verify
the output.

The view

Once your app is launched, you can breathe a sigh of relief,
then mistrust your view of your app. You may be close to your
server, on a fast connection, using a
browser with JavaScript enabled.
Your users may be on the other side of the planet, with a slow connection,
and JavaScript disabled.

You can mistrust your rosy view of your app and understand what
your users are really seeing by installing
Charles or another web debugging
proxy. By increasing network latency and slowing down your connection,
you can see your app from a different user’s point of view.

The server

Your app could be rock solid, perfectly secure, and blazingly
fast, but your server could become a source of mischief. It’s best to mistrust
that, too. Most web hosting these days is still shared hosting, with
hundreds of users on one server. The wrong permissions or an outdated OS
could lead to compromises that none of your code or unit tests could have
prevented.

So, attack your application as a malicious user would. Use
skipfish to attack your
app, and Nmap to scan your
server.

See what ports you have open, or what processes you have running. Figure
out what you can shut down to remove potential threats. Having database
access ports open on a server without limiting by IP address, for
example, is a recipe for disaster.

Download
rainbow tables,
and try them against your system. Attempt to install a rootkit — both for the
joy of trying to break into your system, and because, really, at this point,
you mistrust everything about your server.

The backup

Of course, in the end, the fundamental worth of your app may be
the data. Losing it could be catastrophic to the project or even your
company. At this point, though, you know to mistrust your backups.

Your databases are backed up, but where are those backups? Are they on
the database server? What happens when that hardware fails? Backed up to
another server? What happens when your hosting provider goes out? Do you
have a master-slave database server set up, so that you always have a hot
backup? When is the last time you ran
SHOW SLAVE STATUS
to determine if the slave is actually replicating the master’s data?
Do you have a
heartbeat
process going? Is someone listening to its responses? Monitoring the
database status will work only if someone is paying attention.

The biggest question, and the ultimate test of any backup system,
is have you successfully restored a system from one of your
backups? Horror stories of empty backup files abound, so restore from one
of those backups, and deploy a staging server — even if it’s only on a virtual
machine. Restore from that backup before you really need it.

Fundamentally, programming defensively, mistrusting your assumptions, and
verifying each part of your app, from the shortest line of code, to
the latest deployment, creates a better, more secure application.

There are many ways for a project to fail; it’s best to eliminate the
ones you can.

Mistrust and verify.
