files.new_contents: Autostarting Security Monkey
  ===============================================
  
  This document outlines how to configure Security Monkey to:
  
  1. Automatically run the API
  1. Automatically scan for changes in your environment.
  1. Configure Security Monkey to send scanning performance metrics
  
  Each section is important, please read them thoroughly.
  
  Celery and The Supervisor
  ------------
  Security Monkey makes use of the excellent [Celery Distributed Task Queue for Python](http://docs.celeryproject.org/en/latest/index.html),
  and [Supervisord](http://supervisord.org/) (hence referred to as the Supervisor) for automatically starting the required components
  to make Security Monkey function.
  
  In general:
  1. The Supervisor ensures that there is always a Security Monkey related process running.
  1. Celery is used to manage the scheduling of worker instances to fetch and audit details
     of your accounts.
  
  For each, a sample Supervisor configuration is included. More details is provided in each section below.
  
  üö®‚ö†Ô∏èü•Åüé∫ IMPORTANT PLEASE READ üé∫ü•Å‚ö†Ô∏èüö®
  -------------
  The recommended deployment strategy of Security Monkey is to have 3 types of instances:
  1. 1 or many UI instances behind a load balancer
  1. Exactly ONE "scheduler" instance (more on that below)
  1. 1 or many "worker" instances that fetch details about the environment.
  1. A Redis cache (recommended -- it's easy and works great) to act as a message broker for the scheduler and workers to orchestrate task executions.
  
  *DO NOT* mix the "scheduler" and "worker" instances together -- unless you only want 1 instance doing everything
  (not recommended -- it doesn't scale).
  
  Each function should exist on separate instances. If you followed the quickstart guide, now is a good time
  to make an image of your instance/container and clone new ones to provide the separate configurations.
  
  For AWS, you can make an AMI off of your Security Monkey instance, and even create separate launch
  configurations and auto-scaling groups.
  
  Below is a diagram of the components required by Security Monkey (these are not AWS specific):
  ![diagram](images/sm_instance_diagram.png)
  
  Autostarting the API and UI
  --------------------
  
  The UI and API go hand-in-hand. The UI is started by NGINX, and the API is autostarted by the Supervisor.
  
  All sample supervisor configurations are stored in the [`supervisor`](https://github.com/Netflix/security_monkey/tree/develop/supervisor)
  location. For the UI, you will care about `security_monkey_ui.conf`(https://github.com/Netflix/security_monkey/tree/develop/supervisor/security_monkey_ui.conf).
  
  This file needs to be copied over to the supervisor configuration directory at `/etc/supervisor/conf.d/`.
  
  To enable NGINX and the supervisor, run the following commands (these also set the proper permissions):
  
      sudo chgrp -R www-data /var/log/security_monkey
      sudo cp /usr/local/src/security_monkey/supervisor/security_monkey_ui.conf /etc/supervisor/conf.d/security_monkey_ui.conf
      sudo systemctl enable nginx
      sudo systemctl enable supervisor
      sudo systemctl start nginx
      sudo systemctl start supervisor
      sudo supervisorctl status
  
  The `supervisorctl status` should output details about the loaded supervisor job.
  
  *Note:* If you make a change to a supervisor configuration, you will need to reload the supervisor service:
  `sudo systemctl reload supervisor`
  
  The supervisor will run the Security Monkey's Python [gunicorn](http://gunicorn.org/) command that launches the API (which runs the `monkey run_api_server` command.).
  
  You can track progress by tailing `/var/log/security_monkey/securitymonkey.log`.
  
  The Scheduler
  --------------
  The scheduler is used to configure the Celery workers. This schedules jobs onto a queue (message broker) which the Celery workers
  then use to fetch the tasks to work on in the proper intervals.
  
  ### The Message Broker (queue)
  Celery has several options for configuring a message broker, which is [documented in detail here](http://docs.celeryproject.org/en/latest/getting-started/brokers/).
  
  We recommend [Redis](https://redis.io/) as it's simple to configure. For AWS, recommend ElastiCache with Redis. We have instructions for setting up
  [ElastiCache instructions here](elasticache_directions.md).
  
  Security Monkey by default will install the `redis`, and `celery[redis
  
  [... the rest is omitted ...]
files.new_path: docs/autostarting.md
files.old_contents: Autostarting Security Monkey
  ===============================================
  
  This document outlines how to configure Security Monkey to:
  
  1. Automatically run the API
  1. Automatically scan for changes in your environment.
  
  Each section is important, please read them thoroughly.
  
  Celery and The Supervisor
  ------------
  Security Monkey makes use of the excellent [Celery Distributed Task Queue for Python](http://docs.celeryproject.org/en/latest/index.html),
  and [Supervisord](http://supervisord.org/) (hence referred to as the Supervisor) for automatically starting the required components
  to make Security Monkey function.
  
  In general:
  1. The Supervisor ensures that there is always a Security Monkey related process running.
  1. Celery is used to manage the scheduling of worker instances to fetch and audit details
     of your accounts.
  
  For each, a sample Supervisor configuration is included. More details is provided in each section below.
  
  üö®‚ö†Ô∏èü•Åüé∫ IMPORTANT PLEASE READ üé∫ü•Å‚ö†Ô∏èüö®
  -------------
  The recommended deployment strategy of Security Monkey is to have 3 types of instances:
  1. 1 or many UI instances behind a load balancer
  1. Exactly ONE "scheduler" instance (more on that below)
  1. 1 or many "worker" instances that fetch details about the environment.
  1. A Redis cache (recommended -- it's easy and works great) to act as a message broker for the scheduler and workers to orchestrate task executions.
  
  *DO NOT* mix the "scheduler" and "worker" instances together -- unless you only want 1 instance doing everything
  (not recommended -- it doesn't scale).
  
  Each function should exist on separate instances. If you followed the quickstart guide, now is a good time
  to make an image of your instance/container and clone new ones to provide the separate configurations.
  
  For AWS, you can make an AMI off of your Security Monkey instance, and even create separate launch
  configurations and auto-scaling groups.
  
  Below is a diagram of the components required by Security Monkey (these are not AWS specific):
  ![diagram](images/sm_instance_diagram.png)
  
  Autostarting the API and UI
  --------------------
  
  The UI and API go hand-in-hand. The UI is started by NGINX, and the API is autostarted by the Supervisor.
  
  All sample supervisor configurations are stored in the [`supervisor`](https://github.com/Netflix/security_monkey/tree/develop/supervisor)
  location. For the UI, you will care about `security_monkey_ui.conf`(https://github.com/Netflix/security_monkey/tree/develop/supervisor/security_monkey_ui.conf).
  
  This file needs to be copied over to the supervisor configuration directory at `/etc/supervisor/conf.d/`.
  
  To enable NGINX and the supervisor, run the following commands (these also set the proper permissions):
  
      sudo chgrp -R www-data /var/log/security_monkey
      sudo cp /usr/local/src/security_monkey/supervisor/security_monkey_ui.conf /etc/supervisor/conf.d/security_monkey_ui.conf
      sudo systemctl enable nginx
      sudo systemctl enable supervisor
      sudo systemctl start nginx
      sudo systemctl start supervisor
      sudo supervisorctl status
  
  The `supervisorctl status` should output details about the loaded supervisor job.
  
  *Note:* If you make a change to a supervisor configuration, you will need to reload the supervisor service:
  `sudo systemctl reload supervisor`
  
  The supervisor will run the Security Monkey's Python [gunicorn](http://gunicorn.org/) command that launches the API (which runs the `monkey run_api_server` command.).
  
  You can track progress by tailing `/var/log/security_monkey/securitymonkey.log`.
  
  The Scheduler
  --------------
  The scheduler is used to configure the Celery workers. This schedules jobs onto a queue (message broker) which the Celery workers
  then use to fetch the tasks to work on in the proper intervals.
  
  ### The Message Broker (queue)
  Celery has several options for configuring a message broker, which is [documented in detail here](http://docs.celeryproject.org/en/latest/getting-started/brokers/).
  
  We recommend [Redis](https://redis.io/) as it's simple to configure. For AWS, recommend ElastiCache with Redis. We have instructions for setting up
  [ElastiCache instructions here](elasticache_directions.md).
  
  Security Monkey by default will install the `redis`, and `celery[redis]` packages. If you wish to install
  a
  
  [... the rest is omitted ...]
files.old_path: docs/autostarting.md
files.new_contents: # encoding=utf8
  """
  .. module: security_monkey.task_scheduler.tasks
      :platform: Unix
      :synopsis: Sets up the Celery task scheduling for watching, auditing, and reporting changes in the environment.
  
  .. version:: $$VERSION$$
  .. moduleauthor:: Mike Grima <mgrima@netflix.com>
  
  """
  import sys
  
  try:
      reload(sys)  # Python 2
      sys.setdefaultencoding('utf8')
      # ^^ Fixes UTF-8 issues
  except NameError:
      pass  # Python 3
  
  import time
  import traceback
  
  from security_monkey import app, db, jirasync, sentry
  from security_monkey.alerter import Alerter
  from security_monkey.datastore import store_exception, clear_old_exceptions, Technology, Account, Item, ItemRevision
  from security_monkey.monitors import get_monitors, get_monitors_and_dependencies
  from security_monkey.reporter import Reporter
  from security_monkey.task_scheduler.util import CELERY, setup
  import boto3
  from sqlalchemy.exc import OperationalError, InvalidRequestError, StatementError
  
  
  @CELERY.task(bind=True, max_retries=3)
  def task_account_tech(self, account_name, technology_name):
      setup()
      app.logger.info("[ ] Executing Celery task for account: {}, technology: {}".format(account_name, technology_name))
      time1 = time.time()
  
      # Verify that the account exists (was it deleted? was it renamed?):
      if not Account.query.filter(Account.name == account_name).first():
          app.logger.error("[X] Account has been removed or renamed: {}. Please restart the scheduler to fix.".format(
              account_name
          ))
          return
  
      try:
          reporter_logic(account_name, technology_name)
  
          time2 = time.time()
          app.logger.info('[@] Run Account for Technology (%s/%s) took %0.1f s' % (account_name,
                                                                                   technology_name, (time2 - time1)))
          app.logger.info(
              "[+] Completed Celery task for account: {}, technology: {}".format(account_name, technology_name))
      except Exception as e:
          if sentry:
              sentry.captureException()
          app.logger.error("[X] Task Account Scheduler Exception ({}/{}): {}".format(account_name, technology_name, e))
          app.logger.error(traceback.format_exc())
          store_exception("scheduler-exception-on-watch", None, e)
          raise self.retry(exc=e)
  
  
  @CELERY.task(bind=True, max_retries=3)
  def task_audit(self, account_name, technology_name):
      setup()
  
      app.logger.info("[ ] Executing Celery task to audit changes for Account: {} Technology: {}".format(account_name,
                                                                                                         technology_name))
      # Verify that the account exists (was it deleted? was it renamed?):
      if not Account.query.filter(Account.name == account_name).first():
          app.logger.error("[X] Account has been removed or renamed: {}. Please restart the scheduler to fix.".format(
              account_name
          ))
          return
  
      try:
          audit_changes([account_name], [technology_name], True)
  
          app.logger.info("[+] Completed Celery task for account: {}, technology: {}".format(account_name,
                                                                                             technology_name))
  
      except Exception as e:
          if sentry:
              sentry.captureException()
          app.logger.error("[X] Task Audit Scheduler Exception ({}/{}): {}".format(account_name, technology_name, e))
          app.logger.error(traceback.format_exc())
          store_exception("scheduler-exception-on-audit", None, e)
          self.retry(exc=e)
  
  
  @CELERY.task()
  def clear_expired_exceptions():
      app.logger.info("[ ] Clearing out exceptions that have an expired TTL...")
      clear_old_exceptions()
      app.logger.info("[-] Completed clearing out exceptions that have an expired TTL.")
  
  
  def fix_orphaned_deletions(account_name, technology_name):
      """
      Possible issue with orphaned items. This will check if there are any, and will assume that the item
      was deleted. This will create a deletion change record to it.
  
      :param account_name:
      :param technology_name:
      :return:
      """
      # If technology doesn't exist, then create it:
      technology = Technology.query.filter(Technology.name == technology_name).first()
      if not technology:
          technology = Technology(name=technology_name)
          db.session.add(technology)
          db.session.commit()
          app.logger.info
  
  [... the rest is omitted ...]
files.new_path: security_monkey/task_scheduler/tasks.py
files.old_contents: # encoding=utf8
  """
  .. module: security_monkey.task_scheduler.tasks
      :platform: Unix
      :synopsis: Sets up the Celery task scheduling for watching, auditing, and reporting changes in the environment.
  
  .. version:: $$VERSION$$
  .. moduleauthor:: Mike Grima <mgrima@netflix.com>
  
  """
  import sys
  
  try:
      reload(sys)  # Python 2
      sys.setdefaultencoding('utf8')
      # ^^ Fixes UTF-8 issues
  except NameError:
      pass  # Python 3
  
  import time
  import traceback
  
  from security_monkey import app, db, jirasync, sentry
  from security_monkey.alerter import Alerter
  from security_monkey.datastore import store_exception, clear_old_exceptions, Technology, Account, Item, ItemRevision
  from security_monkey.monitors import get_monitors, get_monitors_and_dependencies
  from security_monkey.reporter import Reporter
  from security_monkey.task_scheduler.util import CELERY, setup
  from sqlalchemy.exc import OperationalError, InvalidRequestError, StatementError
  
  
  @CELERY.task(bind=True, max_retries=3)
  def task_account_tech(self, account_name, technology_name):
      setup()
      app.logger.info("[ ] Executing Celery task for account: {}, technology: {}".format(account_name, technology_name))
      time1 = time.time()
  
      # Verify that the account exists (was it deleted? was it renamed?):
      if not Account.query.filter(Account.name == account_name).first():
          app.logger.error("[X] Account has been removed or renamed: {}. Please restart the scheduler to fix.".format(
              account_name
          ))
          return
  
      try:
          reporter_logic(account_name, technology_name)
  
          time2 = time.time()
          app.logger.info('[@] Run Account for Technology (%s/%s) took %0.1f s' % (account_name,
                                                                                   technology_name, (time2 - time1)))
          app.logger.info(
              "[+] Completed Celery task for account: {}, technology: {}".format(account_name, technology_name))
      except Exception as e:
          if sentry:
              sentry.captureException()
          app.logger.error("[X] Task Account Scheduler Exception ({}/{}): {}".format(account_name, technology_name, e))
          app.logger.error(traceback.format_exc())
          store_exception("scheduler-exception-on-watch", None, e)
          raise self.retry(exc=e)
  
  
  @CELERY.task(bind=True, max_retries=3)
  def task_audit(self, account_name, technology_name):
      setup()
  
      app.logger.info("[ ] Executing Celery task to audit changes for Account: {} Technology: {}".format(account_name,
                                                                                                         technology_name))
      # Verify that the account exists (was it deleted? was it renamed?):
      if not Account.query.filter(Account.name == account_name).first():
          app.logger.error("[X] Account has been removed or renamed: {}. Please restart the scheduler to fix.".format(
              account_name
          ))
          return
  
      try:
          audit_changes([account_name], [technology_name], True)
  
          app.logger.info("[+] Completed Celery task for account: {}, technology: {}".format(account_name,
                                                                                             technology_name))
  
      except Exception as e:
          if sentry:
              sentry.captureException()
          app.logger.error("[X] Task Audit Scheduler Exception ({}/{}): {}".format(account_name, technology_name, e))
          app.logger.error(traceback.format_exc())
          store_exception("scheduler-exception-on-audit", None, e)
          self.retry(exc=e)
  
  
  @CELERY.task()
  def clear_expired_exceptions():
      app.logger.info("[ ] Clearing out exceptions that have an expired TTL...")
      clear_old_exceptions()
      app.logger.info("[-] Completed clearing out exceptions that have an expired TTL.")
  
  
  def fix_orphaned_deletions(account_name, technology_name):
      """
      Possible issue with orphaned items. This will check if there are any, and will assume that the item
      was deleted. This will create a deletion change record to it.
  
      :param account_name:
      :param technology_name:
      :return:
      """
      # If technology doesn't exist, then create it:
      technology = Technology.query.filter(Technology.name == technology_name).first()
      if not technology:
          technology = Technology(name=technology_name)
          db.session.add(technology)
          db.session.commit()
          app.logger.info("Technology: {} did
  
  [... the rest is omitted ...]
files.old_path: security_monkey/task_scheduler/tasks.py