<h2 dir="auto">Add a command-line flag to optionally ignore headers passed in subsequent CSV-import files.</h2>
<p dir="auto">As per discussion <a href="https://neo4j-users.slack.com/archives/C08J7ELUC/p1532041270000195" rel="nofollow">in Slack</a>, it would be very helpful to allow CSV headers in every <code class="notranslate">neo4j-admin import</code> file. For example, in the (truncated) command:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="./bin/neo4j-admin import --id-type STRING --nodes:Nodes &quot;mynodes-\d+.csv&quot; ..."><pre class="notranslate">./bin/neo4j-admin import --id-type STRING --nodes:Nodes <span class="pl-s"><span class="pl-pds">"</span>mynodes-\d+.csv<span class="pl-pds">"</span></span> ...</pre></div>
<p dir="auto">If I have only one file named <code class="notranslate">mynodes-0.csv</code>, then all works as planned. But if I have <code class="notranslate">mynodes-0.csv</code> and <code class="notranslate">mynodes-1.csv</code>, then <code class="notranslate">mynodes-1.csv</code> <strong>must not have headers</strong>. This seems a bit silly, since it's easy enough to ignore — and it can be a huge hassle to drop that line from every incoming CSV.</p>
<p dir="auto">As an example user-story: <a href="https://dask.pydata.org/en/latest/dataframe-api.html#dask.dataframe.to_csv" rel="nofollow">Dask's <code class="notranslate">to_csv</code></a> function, for example, is a prime candidate source for Neo4j data-import; for data consistency, this API adds a header to <em>all</em> (or, optionally, none) of the output CSVs.</p>
<h2 dir="auto">Potential Short Term Fix</h2>
<p dir="auto">If you found this because you're running into this problem: You can simply remove headers from ALL files, and then pass <em>ANOTHER</em> CSV with just headers; but, of course, this is pretty frustrating :)</p>
<h2 dir="auto">And one more thing:</h2>
<p dir="auto">And one more thing! The documentation currently states:</p>
<blockquote>
<p dir="auto"><strong>Multiple files will be logically seen as one big file from the perspective of the importer.</strong> The first line must contain the header. Multiple data sources like these can be specified in one import, <strong>where each data source has its own header.</strong> [Emphasis mine.]</p>
</blockquote>
<p dir="auto">This "first line must contain the header" and "each data source has its own header" is, I think, misleading: Each "data source" isn't clearly different (at least to a beginner) than "data file". That suggests that each file may have its own header, but that's not the case!</p>
<hr>
<h1 dir="auto">An end to our sorrows?</h1>
<p dir="auto">Perhaps a <code class="notranslate">--ignore-firstline</code> flag (or something more elegant) could be passed to the importer to indicate that every file has a header? <strong>It's probably even possible to do this automatically, since the first line of every file will be the same. That should be easy to detect!</strong> But I appreciate that there might be some corner-case where existing users rely on that, and would suffer if the native behavior were changed.</p>
<h3 dir="auto">Feature request</h3>
<p dir="auto">Add an option in neo4j-import to skip the header line in very large CSV files, for use when separate header files are used.</p>
<h3 dir="auto">Scenario</h3>
<p dir="auto">In enterprise data work, it's not unusual to have a data pipeline where one team is doing data extraction and shipping very large CSVs to another team tasked with building the Neo4j graph.  Neo4j-import lets the graph team design compact header files so that the very large CSVs do not have to be opened and edited (which is great!).  Currently, neo4j-import requires that no header line is present in the very large CSVs to be loaded using separate header files.</p>
<p dir="auto">The problem is that it's difficult to diagnose extract issues on deep and wide files missing headers when imports fail.  So for very large files, it's a better practice to keep the headers on so that they can be easily validated against the designed header file CSVs that carry the key assignments, data types etc.  Under the current set of options, either the large CSV has to be generated without a header line (adding IT mystery) or the graph team has to go in and remove the header line once the extract is validated (adding Dev hassle).</p>
<p dir="auto">Currently, if the large CSV has a header row, neo4j-import will attempt to create a node using the header data as property values - which may not always work if conflicting data types are set in the header file.</p>
<h3 dir="auto">Possible implementation</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="C:/neo4j/bin/neo4j-import --skip-header-line TRUE \
--into graph.db \
--nodes:MyLabel: &quot;My_Header.csv, My_Very_Large_File.csv&quot; "><pre class="notranslate"><code class="notranslate">C:/neo4j/bin/neo4j-import --skip-header-line TRUE \
--into graph.db \
--nodes:MyLabel: "My_Header.csv, My_Very_Large_File.csv" 
</code></pre></div>
<p dir="auto">default value would be <code class="notranslate">--skip-header-line FALSE</code> to be consistent with basic usage without header CSV files.</p>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">CSV import would start at first data line of My_Very_Large_File.csv, skipping the header line.<br>
The header values are supplied by My_Header.csv, as per usual.</p>
<h3 dir="auto">Note</h3>
<p dir="auto">A more general case would be:<br>
<code class="notranslate">--start-import-at-line n</code><br>
where n is any arbitrary line number in the file...</p>
<p dir="auto">Thanks,</p>
<p dir="auto">Michael</p>