Relative date filter possible in api?
Not sure what you mean by "custom" relative date filter, but if you're looking for filtering the results just like we have on the "Find Participant" form, then yes, looks like it is possible. 

Eg - To find all participants registered in this fiscal year can be retrieved via -



i.e, select Other in the Parameter dropdown and enter participant_register_date_relative in the textbox as shown above.

The API that will be formed is -

$result = civicrm_api3('Participant', 'get', [
  'sequential' =&gt; 1,
  'participant_register_date_relative' =&gt; "this.fiscal_year",
]);


Similarly, for Previous Fiscal Year, enter previous.fiscal_year in the value field.