title: How should I print runtime information to stdout when assigning python output to a variable in bash?
body: <p>I am calling a Python3 program from a bash script. The program takes command line arguments that include a <code>datetime</code> (eg '20151211203000') which is used to return the source URLs of images as a python list, then print them to a terminal in a for-loop using <code>print()</code>. The bash script reads a text file with rows of <code>datetime</code> strings, gets the list of URLs from the python program, then downloads each of the images in the list in turn. </p>
  
  <p>The python program also contains other print statements that are used for error exceptions. I'm relatively new to scripting in bash and working between it and python scripts - I've set the bash script to <code>grep</code> the python output for a URL as validation for the output. This worked fine until I improved the program so that it navigates to the webpage containing the images for the following day, if they're within the specified timespan, and used further <code>print()</code> statements for debugging. The output of the informative debugging strings that aren't caught up in exceptions is then appended to the list of strings that are used with <code>wget</code> and returns unwanted files from random URLs.</p>
  
  <p>This now works, as I've used the keyword argument <code>file</code> in the python3 print statements to specify <code>sys.stderr</code> as the output. Below are examples from the python program and bash script that should highlight my previous efforts and any gaps in my knowledge with regards to the right way of going about this.</p>
  
  <p>From my bash script:</p>
  
  <pre class="lang-sh prettyprint-override"><code>...
  urls=(`python3 get_keos_url.py $date -dt $2 -i $3`)
  ...
  # Check output to ensure python3 program didn't fall into exception, have URL as first output otherwise
  if echo "${urls[0]}" | grep -q "http"; then
  
      ...
  
  #   Download URLs, now iterating over each array element
      for u in ${urls[@]}; do
          wget --no-verbose "${u}" -P $dir$d &amp;&amp;
          echo "Download complete."
      done
  else
  #   Echo excepted error message to stdout
      echo ${urls[@]}
  fi
  </code></pre>
  
  <p>and from the python3 program:</p>
  
  <pre class="lang-py prettyprint-override"><code>import mechanicalsoup as ms
  import sys
  import argparse
  
  def find_keos(obs_datetime, dt=1.5,
                site="SIF"):
      url="http://www.exampleurl-"+site+".com"
      browser = ms.StatefulBrowser()
      browser.open(url)
  
      ...
  
      try:
          browser.follow_link(browser.find_link(link_text=day_text))
      except ms.LinkNotFoundError:
          # Example of print from exception - to stdout
          print("No keograms are available for {}.\n\n".\
                format(obs_datetime.strftime(r"%d-%m-%Y")))
  
          # Used this to ensure message printed to terminal
          sys.stdout.flush()
  
          return None
  
      ...
  
      if (t_end.day &gt; obs_datetime.day) or\
             (t_end.month &gt; obs_datetime.month) or\
                 (t_end.year &gt; obs_datetime.year):
  
          # Example of "informative" print, included for debugging but kept to provide runtime info.
          # This causes a bug - grep catches the URL that's appended to the string below in the first entry
          print("Looking in next day-URL:{}\n".format(browser.get_url()))
  
          # Included file=sys.stderr as fix
          print("Looking in next day-URL:{}\n".format(browser.get_url()),
                file=sys.stderr)
  
      ...
      return image_urls
  
  if __name__ == "__main__":
      ...
      keos = find_keos(date, dt=args.dt, site=args.instrument)
      ...
      if keos:
          for k in keos:
              print(k)
  </code></pre>
  
  <p>Note that the exception printouts were written simultaneously with the bash script, while the example of the debugging print was written sometime after to improve the program.</p>
  
  <p>Previously, for cases where the debugging printout is not executed, my bash script implementation worked fine. After editing the code and now seeing cases where the debugging printout is executed, the implementation fails as the stdout is somehow merged with the returned URLs. I used <code>file=sys.stderr</code> to get around this, and it now works.</p>
  
  <p>This may be a roundabout way to ask, but my primary question is:</p>
  
  <p><em>How should print statements be sent to standard output if they contain strings that are not wanted in output to bash?</em></p>
  
  <p>I know the adage "no news is good news" is often followed for computer programs, and sending output that isn't used but that also isn't an error to <code>stderr</code> seems odd.</p>
  
  <p>Is there another version of the <code>stdout</code> stream that print statements can be sent to from python? Or is there a better way to assign program output to variables in bash scripts?</p>