How to use composer for installation profile?
You need to use the Composer Manager, which search the composer.json file in any module add the dependency on the core composer.json

How to handle the dependency ? By writing a composer.json file that contains all the php libraries that are been load it. 

Here is an example of a good composer.json

{
  "name": "drupal/mobile_detect",
  "description": "Mobile_Detect is a lightweight PHP class for detecting mobile devices.",
  "type": "drupal-module",
  "homepage": "http://drupal.org/project/mobile_detect",
  "authors": [
    {
      "name": "Matthew Donadio (mpdonadio)",
      "homepage": "https://www.drupal.org/u/mpdonadio",
      "role": "Maintainer"
    },
    {
      "name": "Darryl Norris (darol100)",
      "email": "admin@darrylnorris.com",
      "homepage": "https://www.drupal.org/u/darol100",
      "role": "Co-maintainer"
    }
  ],
  "support": {
    "issues": "http://drupal.org/project/mobile_detect",
    "irc": "irc://irc.freenode.org/drupal-contribute",
    "source": "http://cgit.drupalcode.org/mobile_detect"
  },
  "license": "GPL-2.0+",
  "minimum-stability": "dev",
  "require": {
    "mobiledetect/mobiledetectlib": "~2.8"
  }
}


This composer.json is for the mobile_detect drupal module. Once you run the core composer.json this will download the mobile_detect library into the vendor directory and support autoload. And you can do the same thing with your libraries.

For more information about D8 and composer I will recommend reading this article - https://bojanz.wordpress.com/2015/09/18/d8-composer-definitive-intro/