Postgresql how to return nested type in a function
To create an instance of a type, you need a row constructor
So to create a value of nestedtype you need row('test1')::nestedtype. To create a value of maintype you essentially do the same, row(...)::maintype but for the second value you need again pass a proper row type:
row('test2 value', row('test1 value')::nestedtype)::maintype

This expression can be used in your function:
CREATE OR REPLACE FUNCTION mytest() 
  returns maintype 
as 
$$
  select row('test2 value', row('test1 value')::nestedtype)::maintype;
$$ 
LANGUAGE sql STABLE;