In a forced-choice task, what proportion of responses is above chance level? If I give a task in which there are only two response-options, and I want to categorise the participant according to their responses, what is the 'cutoff' or required proportion of one type of response?

E.g.

50-50 would be chance level performance: 50% Type A responses; 50% Type B responses.

100-0 would be a perfect Type A performance: 100% Type A responses.

Presumably a significant number of Type A responses (say, probably around 65%) would qualify as 'Type A' as opposed to 'chance performance'. But how should I decide on what that number is - what is a justifiable criterion and how do I justify using it?
This is a binomial distribution and the answer depends on the total number of Type A and B questions as well as what confidence level you want to use for the cutoff. There is probably a formula for determining what you want, but I don't know what it is. You can compute it yourself using a permutation test.

Generate a null hypothesis distribution of random guesses and then use this to find the cutoff for whatever p-value (probability that randomly guessing will give a score greater than a certain value) you are comfortable with. The easiest way to get that value is to sort all of the null performance scores and get one that is 95% higher than all of the others (say Score #950 if you did 1000 random guesses and sorted the scores).

Here is some Matlab code to estimate this:

n_trials = 50;
p_cutoff = 0.05;
n_permutations = 100000;
% Create a vector of n_trials 'correct' responses and replicate it across n_permutations
expected_values = repmat(randi([0 1],[1 n_trials]),[n_permutations 1]);
% Create an array of n_permutations x n_trials random guesses
random_responses = randi([0 1],[n_permutations n_trials]);
% Compute the percent correct for each permutation
percent_correct = sum(expected_values==random_responses,2)/n_trials;
% Compute and print the cutoff percentage for desired p value
fprintf('\np=%1.3f Cutoff for %d trials: %3.1f%%\n\n', p_cutoff, n_trials, quantile(percent_correct,1-p_cutoff)*100);


Here are some cutoffs for p=0.05 assuming the same number of Type A and Type B questions:

n_trials cutoff
10       80.0%
20       70.0%
30       63.3%
40       62.5%
50       62.0%
75       60.0%
100      58.0%
250      55.2%
500      53.6%


More total questions (trials) and the cutoff gets closer to 50%. 

Note this is all assuming that there is no kind of bias, say a participant choosing randomly will be more likely to choose A (like the first answer or something like that).

Here is an online calculator for computing some probabilities with the binomial distribution.