Can I use drupal_get_form and drupal_render within a custom Drupal 7 input filter?
I believe you can do this, but not by default. You'll need to disable caching. This is because the filtered data is cached, so your form will only work the first time. To disable caching, you need to set cache to FALSE in your hook_filter_info() definition.

If it's timing out, then it may be a problem in your form definition itself - does the for work in non-filter context (ie embedded directly into a page)?