How to automate TeamCity installation using docker?
Automation of installations like this can often be done with config management tools.  GitHub and the public repos for these tools is a good place to start.

There is a Puppet installer for TeamCity here: https://github.com/haf/puppet-teamcity

Here's a Chef cookbook:
https://supermarket.chef.io/cookbooks/chef-teamcity

You can also use tools like Packer to generate your image(s).  Packer can substitute for a Dockerfile in generating a Docker image. Packer will also run Provisioners (including Chef/Puppet/Ansible) to configure your image.