install in subdomain with Joomla
This is more of a web server configuration question than CiviCRM-specific.  Having a subdomain (that is, civi.mysite.org, not mysite.org/civi) doesn't even require the sites be on the same server, let alone in a directory of another.
I recommend a complete separation of their folders (assuming non-crappy hosting that doesn't make this difficult/impossible).  When placing a CMS inside another CMS's folder, you can end up in scenarios where files that shouldn't be web-readable become web-readable.