<p dir="auto">Why are eager task results not published to the task backend? </p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/celery/celery/blob/816ff0310296aad7d44cd6e07004f770c007e226/celery/app/trace.py#L275">celery/celery/app/trace.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 275
      in
      <a data-pjax="true" class="commit-tease-sha" href="/celery/celery/commit/816ff0310296aad7d44cd6e07004f770c007e226">816ff03</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L275" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="275"></td>
          <td id="LC275" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">publish_result</span> <span class="pl-c1">=</span> <span class="pl-c1">not</span> <span class="pl-s1">eager</span> <span class="pl-c1">and</span> <span class="pl-c1">not</span> <span class="pl-s1">ignore_result</span> </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<p dir="auto">I don't think this is a good design choice. It disrupts the code flow, requiring to add more logic to handle eager results. For example, my code will do something like this:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="async_result = my_task.delay()
handle_task_result(async_result.id)

def handle_task_result(id):
  # do stuff with the result, either synchronously or asynchronously
  # the result can only be accessed through `AsyncResult(id)`"><pre class="notranslate"><span class="pl-s1">async_result</span> <span class="pl-c1">=</span> <span class="pl-s1">my_task</span>.<span class="pl-en">delay</span>()
<span class="pl-en">handle_task_result</span>(<span class="pl-s1">async_result</span>.<span class="pl-s1">id</span>)

<span class="pl-k">def</span> <span class="pl-en">handle_task_result</span>(<span class="pl-s1">id</span>):
  <span class="pl-c"># do stuff with the result, either synchronously or asynchronously</span>
  <span class="pl-c"># the result can only be accessed through `AsyncResult(id)`</span></pre></div>
<p dir="auto">This won't work at all when eager is turned on; trying <code class="notranslate">AsyncResult(id)</code> within <code class="notranslate">handle_task_result</code> will return non existing 'PENDING' garbage. Instead, I have to add a workaround in the calling code</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="async_result = my_task.delay()
if async_result.is_ready():
  # this is entered only when celery eager is on
  # forces to deal with the result synchronously
else:
  handle_task_result(async_result.id)"><pre class="notranslate"><span class="pl-s1">async_result</span> <span class="pl-c1">=</span> <span class="pl-s1">my_task</span>.<span class="pl-en">delay</span>()
<span class="pl-k">if</span> <span class="pl-s1">async_result</span>.<span class="pl-en">is_ready</span>():
  <span class="pl-c"># this is entered only when celery eager is on</span>
  <span class="pl-c"># forces to deal with the result synchronously</span>
<span class="pl-k">else</span>:
  <span class="pl-en">handle_task_result</span>(<span class="pl-s1">async_result</span>.<span class="pl-s1">id</span>)</pre></div>
<p dir="auto">This additional body of code is 1. totally useless under regular circumstances, it only has value when debugging with eager on and 2. ugly, since the two ways to get results use a totally different API. If eager task results were published instead, the application code would not need to be modified at all.</p>
<p dir="auto">Thoughts?</p>
<h2 dir="auto">Short description</h2>
<p dir="auto">Getting the result of a task according to <a href="http://docs.celeryproject.org/en/v2.2.5/faq.html#how-do-i-get-the-result-of-a-task-if-i-have-the-id-that-points-there" rel="nofollow">the FAQ instructions</a> fails in a non-obvious way when <code class="notranslate">CELERY_ALWAYS_EAGER</code> is <code class="notranslate">True</code>.</p>
<h2 dir="auto">Full description</h2>
<p dir="auto">In <code class="notranslate">celery.app.task.Task.apply_async</code>, if <code class="notranslate">CELERY_ALWAYS_EAGER</code> is <code class="notranslate">True</code>, <code class="notranslate">Task.apply</code> is called instead of <code class="notranslate">app.base.Celery.send_task</code>.  <code class="notranslate">send_task</code> hits the backend store (the queue) but <code class="notranslate">apply</code> just runs the task locally.</p>
<p dir="auto"><em>This is fine</em> (but could be improved upon - see "Solution Option 3", below).  It makes sense that retrieving the status of a task from the backend would not be possible if the backend is never touched.</p>
<p dir="auto">Unfortunately, functions for checking state fail in a non-obvious way that can be confusing to users.</p>
<p dir="auto">The base function (see <code class="notranslate">celery.backends.database.DatabaseBackend._get_task_meta_for</code>) defaults to returning a new <code class="notranslate">Task</code> object with <code class="notranslate">status = states.PENDING</code> and <code class="notranslate">result = None</code> if nothing is found in the backend. This function is called by <code class="notranslate">BaseBackend.get_task_meta</code>, which is called by <code class="notranslate">BaseBackend.get_status</code>, which is the function that is used in the <code class="notranslate">AsyncResult.state</code> property to get task state, and also in <a href="http://celery.readthedocs.org/en/latest/reference/celery.result.html#celery.result.AsyncResult.ready" rel="nofollow"><code class="notranslate">AsyncResult.ready()</code></a>.</p>
<p dir="auto">Thus, for tasks which the system does not know about, the result of a query for their state is not an <code class="notranslate">Exception</code>, but a new object with a valid state and empty results field.</p>
<h2 dir="auto">Possible Solutions</h2>
<h3 dir="auto">Solution Option 1</h3>
<p dir="auto">I think the most straightforward solution is to be explicit and raise an error if the user attempt to check the state of a task via the backend when <code class="notranslate">CELERY_ALWAYS_EAGER</code> is <code class="notranslate">True</code>.</p>
<p dir="auto">This could be fixed by raising an exception (possibly <code class="notranslate">QueueNotFound</code>, maybe a new exception) in <code class="notranslate">celery.backends.base.BaseBackend.get_task_meta</code> if <code class="notranslate">CELERY_ALWAYS_EAGER</code> is <code class="notranslate">True</code>.  For example:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="if self.conf.CELERY_ALWAYS_EAGER:
    raise QueueNotFound(&quot;Cannot access backend when CELERY_ALWAYS_EAGER is True&quot;)"><pre class="notranslate"><code class="notranslate">if self.conf.CELERY_ALWAYS_EAGER:
    raise QueueNotFound("Cannot access backend when CELERY_ALWAYS_EAGER is True")
</code></pre></div>
<p dir="auto">Without this change, <a href="http://docs.celeryproject.org/en/v2.2.5/faq.html#how-do-i-get-the-result-of-a-task-if-i-have-the-id-that-points-there" rel="nofollow">this FAQ response</a> can be a bit confusing:</p>
<blockquote>
<p dir="auto">How do I get the result of a task if I have the ID that points there?</p>
<p dir="auto">Answer: Use Task.AsyncResult:</p>
</blockquote>
<p dir="auto">When <code class="notranslate">CELERY_ALWAYS_EAGER</code> is <code class="notranslate">True</code>, it looks like the task is hanging because the state is always <code class="notranslate">PENDING</code>, even though the execution may have succeeded.</p>
<h3 dir="auto">Solution Option 2</h3>
<p dir="auto">Another path forward is to update the docs to explicitly mention that a task <strong>cannot be retrieved</strong> from celery when running with <code class="notranslate">CELERY_ALWAYS_EAGER</code>.  The user must store the result of running <code class="notranslate">apply_async</code> on the task.</p>
<p dir="auto">I don't consider this a real solution, but more of a stopgap if more discussion is required.</p>
<h3 dir="auto">Solution Option 3</h3>
<p dir="auto">Store <code class="notranslate">EagerResult</code> objects in celery's cache and allow the user to retrieve results from there.  Since celery is already looking in the cache in <code class="notranslate">BaseBackend.get_task_meta</code> this should be reasonable to implement.</p>
<h2 dir="auto">Conclusion</h2>
<p dir="auto">I hope this format of issue is acceptable.  I didn't see where this specific scenario fit in the <a href="https://github.com/celery/celery/blob/master/CONTRIBUTING.rst">Contribution Guidelines</a>.</p>
<p dir="auto">Please let me know if more documentation is required here, or if an example application demonstrating the issue would be helpful.</p>