<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/himanshug/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/himanshug">@himanshug</a> <a href="https://github.com/druid-io/druid/issues/4312#issuecomment-356127914" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/4312/hovercard">noted</a> that there are some classes that are duplicated between <code class="notranslate">indexing-service</code> and <code class="notranslate">server</code> modules (which classes?)</p>
<p dir="auto">We don't want to resolve this by just merging this into one larger module, because it would bring hadoop dependencies to historical, broker, and router nodes.</p>
<p dir="auto">The proposal is to</p>
<ol dir="auto">
<li>Identify all duplicated classes</li>
<li>Compute a transitive dependency closure (Java source - based) of them and extract this as <code class="notranslate">server-common</code> module</li>
<li>Leave the remaining classes in <code class="notranslate">indexing-service</code> and <code class="notranslate">server</code> modules, but keep them separate.</li>
</ol>
<p dir="auto">My data events have duplicates.  Data ingestion stream like below:<br>
log -- &gt;logstash  --&gt; kafka --&gt; tranquility --&gt; druid</p>
<p dir="auto">Can druid supports to remove duplicates?  or tranquility supports to remove duplicates?<br>
or I should do it by myself . Can you give me some suggestions about how to remove duplicates?</p>