title: Manage almost 3.5 PB/3500 TB of data
body: <p>I am looking at an opportunity wherein we will have to manage almost 3.5 PB/3500 TB of data. From what I have read so far, GreenPlum seems to be a good choice. Having said that I am struggling to find a good resource to give me an idea around the Hardware requirements. Following are the base inputs on which we are building this:</p>
  
  <p>1). Data would be coming in at rate of 2 GBps (Giga BYTE)
  2). The data is fairly simple with only one large table with 15 odd columns. Each column/record would be close to 2 KB
  3). I would need indexing on 6 columns. Each of these columns is a varchar/string column 
  4). The use case is more write intensive and less of read intensive. The idea would be to process a set of 15-20 batch jobs per day. Real/Near real time analytics is not a necessity. This is more from a reporting purpose.
  5). Data is time series data and would be required for a month. So data older than a month would be purged.</p>
  
  <p>What I have udnerstood so far is that GreenPlum recommends 2x8 cores(Threads) and 256 GB RAM per host. Also each host should typically looks at 24 slots of hard disk. If I look at ESAS with 4TB each I should be able to host 96 TB / host.If I assume a simple linear extrapolation I would be looking at (3500 / 96 ) 37 nodes. </p>
  
  <p>Now I know its not that simple/linear a calculation. Hence, I wanted to udnerstand if there is any calculator/resource/guidelines to size the database cluster. Also I wanted to know if it is ok to not give dedicated disks to the server and rather use a single SAN storage. Each server can have 2x10G links to make sure easy data transfers between nodes and SAN.</p>
  
  <p>Many Thanks.</p>
  
  <p>Abhi</p>