Logging into Drupal 7 site with headless browser: why would it fail? NOTE: I've heavily revised the wording of my question to focus more clearly on the problem.  There's a ton of really really useful information in the answers below, but not all of it applies to solving the problem of logging in with a headless browser.  (Feel free to view the edit history for more details, as well as a more verbose version of the same info.)

I'm using a headless WebKit browser (from CasperJS to log into my Drupal 7 site, and it is failing for unknown reasons.  I first used the Ruby version of the Mechanize library and got identical results: access denied even after filling out and submitting the log in form.  I'm pretty sure the problem has to do with Drupal, and some failure on my part to jump through some hoop (that is probably there to prevent spambots and such from submitting to Drupal-based sites).

Since my script loads the log-in form before submitting it, AFAIK it should work just like a normal user logging in.  I've looked through the methods mentioned in the answers that details Drupal's form-submission process, but I'm not seeing what would cause my problem. 

This question seems to provide clues, though it is D6 and using curl.  That person is passing field_source, form_token, form_build_id, and form_id as fields.  In my form I only see form_build_id and form_id. Should I be passing all those fields, even though they don't seem to be present in the login form?  Drupal seems to look for form_token, but it's not present on my login form.  What's up with that?

NOTE: Someone will undoubtedly ask why I'm using these tools when there must be better ways with such-and-such a Drupal module, and the short answer is just that I chose to use tools I'm most familiar with.   If you have good module suggestions please leave comments (with implementation details if possible) as I'd love to hear the suggestions, but I'd still appreciate an answer to this question as it is asked.  Thanks.
Is there some security feature built into Drupal to prevent
  submissions by scripts?


Yes, there is.

In order to prevent automated form submission by bots etc.. During form build Drupal generates a token, based on session id, and a unique id, for form cache. These are used during form validation.

See:


drupal_prepare_form
drupal_get_token
drupal_validate_form
form_get_cache


In order to successfully submit the form, you would need to retrieve it and then submit it using the same token and form_build_id values.

However, there are other ways to submit forms programmatically, such as

drupal_form_submit

As well as purely code based methods to log a user in.

e.g, in a very simple form:

// Load a user account and assign to global $user
global $user;
$account = user_load(1, TRUE); // Root user
$user = $account; // This "logs" the user in
drupal_session_regenerate();
// Invoke login event
$edit = NULL;
user_module_invoke('login', $edit, $user);