How to use composer for installation profile? I have a Drupal 8 installation profile with custom modules and themes and I am wondering how should I use composer so that I can run it and drupal core and other dependencies(like contrib modules) would get properly downloaded(and also Drupal's cusom composer stuff would get properly applied as well)?

Is this possible at this time or the composer approach is still not complete?
You need to use the Composer Manager, which search the composer.json file in any module add the dependency on the core composer.json

How to handle the dependency ? By writing a composer.json file that contains all the php libraries that are been load it. 

Here is an example of a good composer.json

{
  "name": "drupal/mobile_detect",
  "description": "Mobile_Detect is a lightweight PHP class for detecting mobile devices.",
  "type": "drupal-module",
  "homepage": "http://drupal.org/project/mobile_detect",
  "authors": [
    {
      "name": "Matthew Donadio (mpdonadio)",
      "homepage": "https://www.drupal.org/u/mpdonadio",
      "role": "Maintainer"
    },
    {
      "name": "Darryl Norris (darol100)",
      "email": "admin@darrylnorris.com",
      "homepage": "https://www.drupal.org/u/darol100",
      "role": "Co-maintainer"
    }
  ],
  "support": {
    "issues": "http://drupal.org/project/mobile_detect",
    "irc": "irc://irc.freenode.org/drupal-contribute",
    "source": "http://cgit.drupalcode.org/mobile_detect"
  },
  "license": "GPL-2.0+",
  "minimum-stability": "dev",
  "require": {
    "mobiledetect/mobiledetectlib": "~2.8"
  }
}


This composer.json is for the mobile_detect drupal module. Once you run the core composer.json this will download the mobile_detect library into the vendor directory and support autoload. And you can do the same thing with your libraries.

For more information about D8 and composer I will recommend reading this article - https://bojanz.wordpress.com/2015/09/18/d8-composer-definitive-intro/