1. A data manipulation system executed on a computer, the system being for combining near-real-time data and batch data to be processed by a data processing layer executed in the computer's processing environment and using the computer's memory system, the data manipulation system comprising: an event collector module that, when executed on the computer, collects incoming event data in real-time; a hierarchical processing layer executed in the computer's processing environment, the processing layer comprising a plurality of processing sub-layers, each processing sub-layer processing one or more data micro-batches stored at respective corresponding levels of temporal granularity; a processing daemon executed in the computer's processing environment, the processing daemon applying a single processing definition to the data micro-batches at the plurality of processing sub-layers, generating and storing resulting processed data micro-batches at the respective corresponding levels of temporal granularity; and a data buffering control module sending commands to the computer's memory system, the data buffering control module shifting processed data micro-batches together to a next coarser level of temporal granularity, wherein aggregation of the processed data micro-batches constitute batch data.