title: Python, PyTables, Java - tying all together
body: <h1>Question in nutshell</h1>
  What is the best way to get Python and Java to play nice with each other?
  <h1>More detailed explanation</h1>
  <p>I have a somewhat complicated situation.  I'll try my best to explain both in pictures and words.  Here's the current system architecture:</p>
  <p><a href="https://i.stack.imgur.com/N4P37.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/N4P37.png" alt="Current system architecture" /></a></p>
  
  <p>We have an agent-based modeling simulation written in Java.  It has options of either writing locally to CSV files, or remotely via a connection to a Java server to an <a href="https://www.hdfgroup.org/solutions/hdf5/" rel="nofollow noreferrer">HDF5</a> file.  Each simulation run spits out over a gigabyte of data, and we run the simulation dozens of times.  We need to be able to aggregate over multiple runs of the same scenario (with different random seeds) in order to see some trends (e.g. min, max, median, mean).  As you can imagine, trying to move around all these CSV files is a nightmare; there are multiple files produced per run, and like I said some of them are enormous.  That's the reason we've been trying to move towards an HDF5 solution, where all the data for a study is stored in one place, rather than scattered across dozens of plain text files.  Furthermore, since it is a binary file format, it should be able to get significant space savings as compared to uncompressed CSVS.</p>
  <p>As the diagram shows, the current post-processing we do of the raw output data from simulation also takes place in Java, and reads in the CSV files produced by local output.  This post-processing module uses JFreeChart to create some charts and graphs related to the simulation.</p>
  <h2> The Problem </h2>
  As I alluded to earlier, the CSVs are really untenable and are not scaling well as we generate more and more data from simulation.  Furthermore, the post-processing code is doing more than it should have to do, essentially performing the work of a very, very poor man's relational database (making joins across 'tables' (csv files) based on foreign keys (the unique agent IDs).  It is also difficult in this system to visualize the data in other ways (e.g. Prefuse, Processing, JMonkeyEngine getting some subset of the raw data to play with in MatLab or SPSS).
  <h2> Solution? </h2>
  My group decided we really need a way of filtering and querying the data we have, as well as performing cross table joins.  Given this is a write-once, read-many situation, we really don't need the overhead of a real relational database; instead we just need some way to put a nicer front end on the HDF5 files.  I found a few papers about this, such as one describing how to use [XQuery as the query language on HDF5 files][3], but the paper describes having to write a compiler to convert from XQuery/XPath into the native HDF5 calls, way beyond our needs.
  Enter [PyTables][4].  It seems to do exactly what we need (provides two different ways of querying data, either through Python list comprehension or through [in-kernel (C level) searches][5].
  <p>The proposed architecture I envision is this:
  <a href="https://i.stack.imgur.com/Q8QzN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Q8QzN.png" alt="Envisioned architecture" /></a></p>
  
  <p>What I'm not really sure how to do is to link together the python code that will be written for querying, with the Java code that serves up the HDF5 files, and the Java code that does the post processing of the data.  Obviously I will want to rewrite much of the post-processing code that is implicitly doing queries and instead let the excellent PyTables do this much more elegantly.</p>
  <h2> Java/Python options</h2>
  A simple google search turns up a few options for [communicating between Java and Python][7], but I am so new to the topic that I'm looking for some actual expertise and criticism of the proposed architecture.  It seems like the Python process should be running on same machine as the Datahose so that the large .h5 files do not have to be transferred over the network, but rather the much smaller, filtered views of it would be transmitted to the clients.  [Pyro][8] seems to be an interesting choice - does anyone have experience with that?