anchor.diff:  
   image:https://github.com/killbill/killbill-docs/raw/v3/userguide/assets/aws/ELB-check-listeners.png[align=center]
   
  -=== Creating a Certificate
  +== Creating a Certificate
   
   We recommend for security that you choose the HTTPS protocol to access KAUI. This protocol uses SSL/TLS encryption and authentication to verify and protect all communication with the KAUI server. To use this protocol you must create or provide an X.509 certificate. This section describes how to create a certificate using the *Amazon Certificate Manager (ACM).*
   
  
positive.code: 
  === Testing
  
  When your ELB is complete you can proceed to testing. If you are using HTTPS you should be able to login to KAUI from your browser using the URL https://kaui.DOMAIN, where DOMAIN is *your* domain that you have used for your certificate. If you are using HTTP you will need to use the URL http://DNS_NAME where DNS_NAME is the *DNS name* for the KAUI load balancer as given on the Load Balancer dashboard. This should display the KAUI login screen. The default credentials are: *admin/password*. The first few requests might be a bit slow as Kill Bill initializes itself.
  
  For an introduction to KAUI, see our http://docs.killbill.io/latest/getting_started.html#_using_kill_bill_with_kaui[Getting Started] guide.
  
  Similarly, you should be able to login directly to the Kill Bill server using the URL http://DNS_NAME:8443. Creating a CNAME for this login is optional.
   
  
  
  == Configuration
  
  === SSL Certificates
  
  `nginx` is configured to listen on port 443 for messages to be forwarded to KAUI. Port 443 is the standard port for HTTPS. By default, accessing the service from a web browser will show a `Not Secure` site. In order to make the site secure, you will need to add a valid certificate. The easiest option to add the certificate is to make use of the tool `certbot`, which relies on the free Certificate Authority (CA) `Let’s Encrypt`  This method provides an easy way to obtain and install free TLS/SSL certificates.
  
  `Certbot` is pre-installed in the latest Kill Bill AMI, but if you need a different version of `certbot`, this is also available through the new package manager `snap`. This package manager is also preinstalled in the Kill Bill AMI.
  
  To validate your certificate, you must first setup a CNAME, or Canonical Name, based on a verified domain that you own. If you do not have a domain, you can obtain one for free, or very low cost, from an online source such as GoDaddy. The CNAME is part of your domain's DNS profile, and your domain provider should have instructions for creating it.
  
  The CNAME should link your domain to your AWS instance. If your domain is `domain.com` your CNAME might look like `killbill.domain.com`.   
  
  
  To setup your certificate, perform the following steps:
  
  [1]. Create a CNAME linking your domain to your AWS instance.
  
  [2]. Add your CNAME to the server block in the `nginx` configuration file.
  
  Using a text editor of your choosing, edit the file `/etc/nginx/sites-enabled/killbill.conf`. You will need to use sudo to edit this file.
  
  This file contains two server blocks. The second block contains the lines:
  
  ```
  server {
      listen 443;
      server_name _;
  ```
  Replace the underscore after `server_name` with your CNAME:
  
  ```
  server {
      listen 443;
      server_name killbill.domain.com;
  ```
  
  Save the modified file.
  
  [3]. Reload the configuration file with the command:
  
  ```
  sudo nginx -s reload
  ```
  
  [4]. Enable HTTP traffic on port 80. This is just to allow `certbot` to create your certificate. To do this, go to your EC2 dashboard and add a new inbound rule as follows: Type: HTTP, Protocol: TCP, Port Range: 80, Source: 0.0.0.0/0. Your inbound rules should now look like this:
  
  image:https://github.com/killbill/killbill-docs/raw/v3/userguide/assets/aws/single-ami-inbound-port-80.png[align=center]
  
  
  [5]. Remove the existing `certbot` package:
  
  Type the command
  
  ```
  sudo apt-get remove certbot
  ```
  When asked if you want to proceed, type `Y` for yes.
  
  [6]. Install `certbot` using `snap`:
  
  The following commands will install `certbot` and link it to the `/usr/bin` directory:
  
  ```
  sudo snap install --classic certbot
  sudo ln -s /snap/bin/certbot /usr/bin/certbot
  ```
  [7].  Run `certbot` to create and install the certificate:
  
  ```
  sudo certbot --nginx
  ```
  If you are asked, confirm the server name to be used. If all goes well, you will see a message like:
  
  ```
  IMPORTANT NOTES:
   - Congratulations! Your certificate and chain have been saved at:
     /etc/letsencrypt/live/deployment.killbill.io/fullchain.pem
     Your key file has been saved at:
     /etc/letsencrypt/live/deployment.killbill.io/privkey.pem
     Your cert will expire on 2020-05-11. To obtain a new or tweaked
     version of this certificate in the future, simply run certbot again
     with the "certonly" option. To non-interactively renew *all* of
     your certificates, run "certbot renew"
   - If you like Certbot, please consider supporting our work by:
  
     Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
     Donating to EFF:                    https://eff.org/donate-le
  ```
  
  Of course, there are a few things that could go wrong:
  
  [1]. `certbot` will attempt to validate your CNAME through your domain provider. If the CNAME does not link correctly to the public DNS name for your AWS instance, the certificate will not be created.
  
  [2]. You could get a message like:
  
  `Error creating new order : Cannot issue for ec2-18-234-168-57.compute-1.amazonaws.com: The ACME server refuses to issue a certificate for this domain name, because it is forbidden by policy`
  
  This could happen if you have tried to use your public DNS name directly as your CNAME, which is not allowed.
  
  
  [3]. You could get a message like `certificate cannot be installed`.
  
  For help with these and other problems, try the `certbot` https://certbot.eff.org/help/[help page].
  
  
  When your certificate is installed, you should be able to access KAUI from you browser using `https://` followed by your CNAME as the address. The browser should indicate that the site is secure.
  
  When your certificate is successfully installed, you can edit your security groups again to remove the access through port 80.
  
  The `Let's Encrypt` certifcates are only valid 90 days and will therefore neeed to be renewed. `certbot` will create a cron entry under `/etc/cron.d/certbot` to make this process automatic.
  
  === Kill Bill Properties
  
  Kill Bill global properties are defined in `/var/lib/killbill/config/killbill.properties`. For example, this is where you can change the MySQL credentials. Take a look at our https://docs.killbill.io/latest/userguide_configuration.html[configuration guide] for more details.
  
  This is also where you can change the default Kill Bill admin credentials: specify `org.killbill.security.shiroResourcePath=/var/lib/killbill/config/shiro.ini` and create the `shiro.ini` file accordingly (see our https://docs.killbill.io/latest/user_management.html[RBAC guide] for more details).
  
  == TroubleShooting
  
  
  After launching the EC2 instance, the full stack should come up, with all services enabled and running, that is:
  
  * An nginx instance receiving traffic on port `443` and `8443`
  * A instance of Kill Bill server listening on `127.0.0.1:8080` (and receiving external traffic through nginx on port `8443`)
  * A instance of Kaui listening on `127.0.0.1:3000` (and receiving external traffic through nginx on port `443`)
  * A local `mysql` server running on port `3306`
  
  
  In this section, we will provide some tips to verify the operation of the system, and what to do when things are not working.
  
  === System Health
  
  
  To verify the overall health of the Kill Bill server instance, you can issue the following command:
  
  ```
  curl http://127.0.0.1:8080/1.0/healthcheck
  ```
  
  This will return a series of messages giving the health of various system components.
  
  A detailed system information report is available using the following command:
  
  ```
  curl -u <ADMIN>:<PASSWORD> http://127.0.0.1:8080/1.0/kb/nodesInfo
  ```
  Here <ADMIN> and <PASSWORD> are your administrator access credentials.
  
  The Kill Bill single-tier option is composed of four services. The status of each service can be checked by the following commands:
  
  * Kill Bill service: `sudo service killbill status`
  * KAUI service: `sudo service kaui status`
  * Nginx service: `sudo service nginx status`
  * Mysql service: `sudo service mysql status`
  
  For each report there should be a line near the top with the following form:
  
  ```
  Active: active (running) since Sat 2020-10-24 20:13:43 UTC; 1 day 1h ago
  ```
  
  
  Similarly you can `start` or `stop` the services using similar commands, such as `sudo service kaui stop` to stop KAUI.
  
  
  === Log Files
  
  The system maintains a series of logfiles that should be helpful when troubleshooting is needed.
  
  Tomcat logs are under `/var/lib/tomcat/logs/`:
  
  * KAUI logs: `/var/lib/tomcat/logs/kaui.out`
  * Kill Bill server logs: `/var/lib/tomcat/logs/catalina.out`
  
  Nginx logs can be found under `/var/log/nginx/`
  
  * Access logs: `/var/log/nginx/access.log`
  * Error logs: `/var/log/nginx/error.log`
  
  === System Diagnostics
  
  If support is needed, we may require some additional detailed information about your deployment. We have created a `diagnostic` command for this purpose. to run this command:
  
  ```
  # Login as 'tomcat'
  > sudo su - tomcat
  #
  # Set your access credentials. This example
  #   assumes the default credentials are used:
  #
  > kpm  diagnostic \
    --killbill-credentials=admin password \
    --bundles-dir=/var/lib/killbill/bundles \
    --database-name=killbill \
    --database-credentials=root root \
    --killbill-api-credentials=bob lazar \
    --kaui-web-path=/var/lib/tomcat/webapps2 \
    --killbill-url=http://127.0.0.1:8080 \
    --database-host=127.0.0.1:3306
  
  ...
  The diagnostic data is exported under /tmp/killbill-diagnostics-20200213-23204-u93ah5/killbill-diagnostics-02-13-20.zip 
  ```
  
  === Databases
  
  To access the mysql (MariaDB) databases, you can use the following command:
  
  ```
  mysql -u root -proot
  ```
  
  This enables interactive access to the database manager. There is one `killbill` and one `kaui` database created and used by the respective applications. To verify the tables in each database, you can type:
  
  ```
  use killbill;
  show tables;
  ```
  or
  
  ```
  use kaui;
  show tables;
  ```
  
  Standard SQL commands can be used to explore or manipulate the tables. Be sure you know what you are doing, or the databases may become corrupted!
  
  To exit the mysql interactive mode, type `exit`.
  
  
  === Nginx
  
  The load balancer `nginx` should normally require little attention. The configuration files are located under `/etc/nginx/. The configuration file for `nginx itself is `/etc/nginx/nginx.conf`. Additional configuration files are located under `/etc/nginx/sites-enabled/`. The only file normally present in this directory is `/etc/nginx/sites-enabled/killbill.conf`. This file may need to be edited to enable SSL, as explained above.
  
  == Upgrades
  
  From time to time new versions of Kill Bill and KAUI may be released. This section explains how to install these new versions.
  
  Note: you must switch to the `tomcat` user first in order to upgrade Kill Bill or Kaui (`sudo su - tomcat`).
  
  The configuration file `/var/lib/killbill/kpm.yml` specifies the Kill Bill version (and its plugins) running on the instance. Once you edit this file to specify the new version number, it will be used automatically. After editing this file, simply run the command `$KPM_INSTALL_KB_CMD`, delete the cached directory `/var/lib/tomcat/webapps/ROOT`, and restart the instance.
  
  A similar process can be used for KAUI: update `/var/lib/kaui/kpm.yml`, run `$KPM_INSTALL_KAUI_CMD`, delete the cached directory `/var/lib/tomcat/webapps2/ROOT` and restart the instance.
  
negative.code: 
  === Setup the Database
  
  Once you are logged in, the first step is to setup the RDS instance. This process begins with the RDS dashboard, which should be available from the Services menu. When the dashboard appears, select *Databases* from the left menu, and click the red button at the top right that reads *Create Database*:
  
  image:https://github.com/killbill/killbill-docs/raw/v3/userguide/assets/aws/multitier-create-database.png[align=center]
  
  You will be taken to the *Create Database* page. The first choice you will have is between *Standard Create*, which allows you to set a full range of configuration parameters, or *Easy Create*, which sets most of these parameters to defaults. We recommend *Easy Create* in most cases.
  
  image:https://github.com/killbill/killbill-docs/raw/v3/userguide/assets/aws/multitier-standard-or-easy.png[align=center]
  
  