title: GCP message stays in the Pub/Sub after acknowledge
body: <p>I have Pub/Sub subscribe logic wrapped inside a subscribe method that is being called once during service initialization for every subscription:</p>
  
  <pre class="lang-py prettyprint-override"><code>    def subscribe(self,
                    callback: typing.Callable,
                    subscription_name: str,
                    topic_name: str,
                    project_name: str = None) -&gt; typing.Optional[SubscriberClient]:
          """Subscribes to Pub/Sub topic and return subscriber client
  
          :param callback: subscription callback method
          :param subscription_name: name of the subscription
          :param topic_name: name of the topic
          :param project_name: optional project name. Uses default project if not set
          :return: subscriber client or None if testing
          """
          project = project_name if project_name else self.pubsub_project_id
          self.logger.info('Subscribing to project `{}`, topic `{}`'.format(project, topic_name))
  
          project_path = self.pubsub_subscriber.project_path(project)
          topic_path = self.pubsub_subscriber.topic_path(project, topic_name)
          subscription_path = self.pubsub_subscriber.subscription_path(project, subscription_name)
  
          # check if there is an existing subscription, if not, create it
          if subscription_path not in [s.name for s in self.pubsub_subscriber.list_subscriptions(project_path)]:
              self.logger.info('Creating new subscription `{}`, topic `{}`'.format(subscription_name, topic_name))
              self.pubsub_subscriber.create_subscription(subscription_path, topic_path)
  
          # subscribe to the topic
          self.pubsub_subscriber.subscribe(
              subscription_path, callback=callback,
              scheduler=self.thread_scheduler
          )
          return self.pubsub_subscriber
  </code></pre>
  
  <p>This method is called like this:</p>
  
  <pre class="lang-py prettyprint-override"><code>        self.subscribe_client = self.subscribe(
              callback=self.pubsub_callback,
              subscription_name='subscription_topic',
              topic_name='topic'
          )
  </code></pre>
  
  <p>The callback method does a bunch of stuff, sends 2 emails then acknowledges the message</p>
  
  <pre class="lang-py prettyprint-override"><code>    def pubsub_callback(self, data: gcloud_pubsub_subscriber.Message):
          self.logger.debug('Processing pub sub message')
  
          try:
              self.do_something_with_message(data)
  
              self.logger.debug('Acknowledging the message')
              data.ack()
              self.logger.debug('Acknowledged')
              return
  
          except:
              self.logger.warning({
                  "message": "Failed to process Pub/Sub message",
                  "request_size": data.size,
                  "data": data.data
              }, exc_info=True)
  
          self.logger.debug('Acknowledging the message 2')
          data.ack()
  </code></pre>
  
  <p>When I run push something to the subscription, callback runs, prints all the debug messages including <code>Acknowledged</code>. The message, however, stays in the Pub/Sub, the callback gets called again and it takes exponential time after each retry. The question is what could cause the message to stay in the pub/sub even after the <code>ack</code> is called? </p>
  
  <p>I have several such subscriptions, all of them work as expected. Deadline is not an option, the callback finishes almost immediately and I played with the ack deadline anyways, nothing helped.</p>
  
  <p>When I try to process these messages from locally running app connected to that pub-sub, it completes just fine and acknowledge takes the message out of the queue as expected.</p>
  
  <ul>
  <li>So the problem manifests only in deployed service (running inside a kubernetes pod)</li>
  <li>Callback executes buck ack does seemingly nothing</li>
  <li>Acking messages from a script running locally (...and doing the exact same stuff) or through the GCP UI works as expected.</li>
  </ul>
  
  <p>Any ideas?</p>