title: How should you write tests for system-to-system applications?
body: <p>I do a lot of integration work between systems such as Salesforce, Jira, etc...  I am fairly comfortable with <code>unit-testing</code> my code, as it is a simple, self contained block of code.  I have a much more difficult time writing comprehensive tests for the applications that are touching many systems, etc...</p>
  
  <p>Here is an example of a job I am re-factoring where it scrapes code-coverage data from an internal company site:</p>
  
  <pre><code>def scrape_projects(
          driver: webdriver,
          project_names: list,
          col_cnt: int,
          project_tup: namedtuple) -&gt; (list, dict):
      """
      Take a list of project names (must be the names as they appear on the
      page itself), follows the link to teh project and then scrapes the data.
      Once the data has been scraped it is dumped into 'namedtuples' for easy
      conversion to dictionaries.
  
      :param driver: A selenium.webdriver object.
      :param project_names: A list of project names.
      :param col_cnt: The number of expected data columns on the page for a
       specific project type.  For example, Ruby projects have 7 columns of data.
      :param project_tup: A namedtuple suitable for the expected columns of a
       project type.
  
      :return: A list of namedtuples containing the project data.
      """
      alerts = {}
      project_tups = []
      for name in project_names:
          logger.info('On project {name}'.format(name=name))
          link = driver.find_element_by_link_text(name)
          link.click()
          data = [item.text.replace('%', '') for item in
                  driver.find_elements_by_tag_name('td') if item.text]
          logger.debug('Page data found: {!r}'.format(data))
  
          # create offsets to split up the data
          offsets = tuple(range(0, len(data), col_cnt))
          data_len = len(data)
          logger.debug(f'Aggregate total for data columns: {data_len}')
  
          try:
              if data_len % col_cnt != 0:
                  raise TypeError(
                      'The column count is not compatible with the given '
                      'project type.')
              for offset in offsets:
                  tup = project_tup(name, *data[offset:offset + col_cnt])
                  logger.debug('Project tuple: {!r}'.format(project_tup))
                  project_tups.append(tup)
          except TypeError as e:
              logger.info(
                  f'Error occurred during processing of {name}.',
                  exc_info=True
              )
              alerts[name] = e
          finally:
              driver.back()
      return project_tups, alerts
  </code></pre>
  
  <p>Would I just write a test to target a single, known project and make sure that it returns expected output?  That seems brittle to me for some reason.  In any case, I'm just looking to improve my knowledge of adequate testing methods.</p>