<p dir="auto">if all historicals (and peon) respond within the query timeout specified and it is broker spending time doing the merge (usually in the case of merging lots of sketches) then timeout wouldn't be honored irrespective of however long broker takes to merge things.</p>
<p dir="auto">on the client side, clients typically use the same timeout to close the connection... so broker doing all the work is eventually wasted.</p>
<p dir="auto">one possibly simple (maybe crude) fix is that broker puts query processing <code class="notranslate">startTime</code> in query context and all things taking time to process the query continuously watch the time spent and fail if it exceeds. I'm imagining probably in QueryToolChest.mergeResults(..) impl and may be in the QueryResource where it is streaming the results to response.</p>
<p dir="auto">another alternative to <code class="notranslate">startTime</code> would be to do query processing in a thread pool and cancel the future on query timeout .... druid code that takes time in processing the query would continuously check if the thread is interrupted.</p>
<p dir="auto">Currently If I specify a <code class="notranslate">timeout</code> query context when querying a broker, that timeout is not honored. It is entirely possible to set a timeout of 90s and have the broker hog resources for the query for 15m while it tries to complete the request.</p>
<p dir="auto">The expected behavior is that the broker terminates the returning data stream once <code class="notranslate">timeout</code> has expired, and also that it cancels any outstanding query efforts on the historicals related to the query.</p>
<p dir="auto">The current state of timeouts seems to be enforced on a per-segment basis, and not a per-query basis. Meaning that if historical nodes take an abnormally long time on queries, the overall query can take a very long time compared to the timeout.</p>
<p dir="auto">This stems from 2 things:</p>
<ol dir="auto">
<li>Lack of checking in QueryResource for commands taking too long. (see <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="84279140" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/1416" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/1416/hovercard" href="https://github.com/apache/druid/pull/1416">#1416</a> )</li>
<li>Lack of ability to interrupt streaming results properly.</li>
</ol>
<p dir="auto">Part of this comes from the assumption stated in <code class="notranslate">io.druid.query.AsyncQueryRunner::run</code></p>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="            //Note: this is assumed that baseRunner does most of the work eagerly on call to the
            //run() method and resulting sequence accumulate/yield is fast."><pre class="notranslate">            <span class="pl-c">//Note: this is assumed that baseRunner does most of the work eagerly on call to the</span>
            <span class="pl-c">//run() method and resulting sequence accumulate/yield is fast.</span></pre></div>
<p dir="auto">because the a lot of the query workload is actually lazily done in yield.</p>