title: What is the fastest way to load a large amount of Neo4j data from a remote machine?
body: <p>I am getting a set of fairly large files, with a max of about 1 Million lines of Neo4j code. The input files look like this,</p>
  <pre><code>MERGE (:ProgNode:compilationUnit {nodeSeq:0,name:'Too Long to Display',quViolations:0,quVioDensity:0.0,quChange:False,location:0,level:'0',szlocs:1,eieo:False});
  </code></pre>
  <p>And then eventually, the connections</p>
  <pre><code>MATCH (a:ProgNode),(b:ProgNode) WITH a,b WHERE a.nodeSeq = 20502 AND b.nodeSeq = 20504 MERGE ...
  </code></pre>
  <p>Average file size is around 200,000 lines but some bump up to the million mark.</p>
  <p>I am using the python interface right now with a delay in processing when it &quot;hiccups&quot;</p>
  <pre><code>        for line in FI:
          counter +=1
          if counter &gt;= startrow:
              if (counter % 500) == 0:
                  print(&quot;     line %s was reached&quot; % counter)
              firstword = line.split()[0]    
              if firstword == &quot;MATCH&quot; and matchflag == False:
                  print(&quot;  Created %s nodes\n&quot; % counter)
                  print(&quot;  Beginning links @ %s\n&quot; % str(time.asctime()))
                  matchflag = True
              elif firstword == &quot;CREATE&quot; and createflag == False:
                  print(&quot;  Beginning Node Creation\n&quot;)
                  createflag = True
              elif firstword == &quot;//&quot; and postflag == False:
                  print(&quot;  %s  @ %s\n&quot; % (line[:-2], str(time.asctime())))
                  postflag = True    
              try:
                  if firstword != &quot;//&quot;:
                      graphDB_Session.run(line)
              except:
                  print(&quot;      Line %s was an issue. wait, then retry ....&quot; % counter,end=&quot;&quot;)
                  time.sleep(8)
                  graphDB_Session.run(line)
                  FO.write(line)
                  print(&quot;success!&quot;)
                  continue
  </code></pre>
  <p>its fairly reliable if slow. I'd rather not unformat and go to CSV but if its faster, so be it. But given the situation, any suggestions on speed ? Should I FTP it to the server and run from there ? Is there a faster API ?</p>
  <p>Based on your comments, is this how you thought it should look ? It does throw an error for  invalid literal for int() with base 10: &quot;MERGE (:ProgNode:compilationUnit so I might still not be understanding.</p>
  <pre><code>    print(&quot;Starting Node load @ %s\n&quot; % time.asctime())
  # Create nodes
  store_embedding = &quot;&quot;&quot;
  UNWIND $data as row
  MATCH (n)
  WHERE id(n) = row.id
  SET n.embedding = row.embedding
  &quot;&quot;&quot;
  embeddings = []
  with graphDB_Driver.session() as graphDB_Session:
      for record in FI:
          id = record
          # Prepare data
          embeddings.append({'id':int(id), 'embedding': [float(x) for x in list(model.wv[id])]})
      # Store embeddings to Neo4j    
      session.run(store_embedding, {'data': embeddings}) 
  </code></pre>