pub fn optimize_record_batch(batch: &RecordBatch, schema: SchemaRef) -> Result<RecordBatch> {
    let max_buf_len = batch
        .columns()
        .iter()
        .map(|a| a.get_array_memory_size())
        .max()
        .unwrap_or_default();

    let columns: Result<Vec<_>> = batch
        .columns()
        .iter()
        .map(|column| {
            if matches!(column.data_type(), DataType::Dictionary(_, _)) {
                hydrate_dictionary(column)
            } else if max_buf_len > batch.num_rows() * 100 {
                Ok(deep_clone_array(column))
            } else {
                Ok(Arc::clone(column))
            }
        })
        .collect();

    RecordBatch::try_new(schema, columns?)
}
/// Some batches are small slices of the underlying arrays.
/// At this stage we only know the number of rows in the record batch
/// and the sizes in bytes of the backing buffers of the column arrays.
/// There is no straight-forward relationship between these two quantities,
/// since some columns can host variable length data such as strings.
///
/// However we can apply a quick&dirty heuristic:
/// if the backing buffer is two orders of magnitudes bigger
/// than the number of rows in the result set, we assume
/// that deep-copying the record batch is cheaper than the and transfer costs.
///
/// Possible improvements: take the type of the columns into consideration
/// and perhaps sample a few element sizes (taking care of not doing more work
/// than to always copying the results in the first place).
///
/// Or we just fix this upstream in
/// arrow_flight::utils::flight_data_from_arrow_batch and re-encode the array
/// into a smaller buffer while we have to copy stuff around anyway.
///
/// See rationale and discussions about future improvements on
/// <https://github.com/influxdata/influxdb_iox/issues/1133>
Some batches are small slices of the underlying arrays.
At this stage we only know the number of rows in the record batch
and the sizes in bytes of the backing buffers of the column arrays.
There is no straight-forward relationship between these two quantities,
since some columns can host variable length data such as strings.

However we can apply a quick&dirty heuristic:
if the backing buffer is two orders of magnitudes bigger
than the number of rows in the result set, we assume
that deep-copying the record batch is cheaper than the and transfer costs.

Possible improvements: take the type of the columns into consideration
and perhaps sample a few element sizes (taking care of not doing more work
than to always copying the results in the first place).

Or we just fix this upstream in
arrow_flight::utils::flight_data_from_arrow_batch and re-encode the array
into a smaller buffer while we have to copy stuff around anyway.

See rationale and discussions about future improvements on
Some batches are small slices of the underlying arrays . At this stage we only know the number of rows in the record batch and the sizes in bytes of the backing buffers of the column arrays . There is no straight - forward relationship between these two quantities since some columns can host variable length data such as strings . However we can apply a quick&dirty heuristic : if the backing buffer is two orders of magnitudes bigger than the number of rows in the result set we assume that deep - copying the record batch is cheaper than the and transfer costs . Possible improvements : take the type of the columns into consideration and perhaps sample a few element sizes ( taking care of not doing more work than to always copying the results in the first place ) . Or we just fix this upstream in arrow_flight :: utils :: flight_data_from_arrow_batch and re - encode the array into a smaller buffer while we have to copy stuff around anyway . See rationale and discussions about future improvements on
pub fn optimize_record_batch(batch: &RecordBatch, schema: SchemaRef) -> Result<RecordBatch> {
    let max_buf_len = batch
        .columns()
        .iter()
        .map(|a| a.get_array_memory_size())
        .max()
        .unwrap_or_default();
    let columns: Result<Vec<_>> = batch
        .columns()
        .iter()
        .map(|column| {
            if matches!(column.data_type(), DataType::Dictionary(_, _)) {
                hydrate_dictionary(column)
            } else if max_buf_len > batch.num_rows() * 100 {
                Ok(deep_clone_array(column))
            } else {
                Ok(Arc::clone(column))
            }
        })
        .collect();
    RecordBatch::try_new(schema, columns?)
}
pub fn optimize_record_batch ( batch : & RecordBatch , schema : SchemaRef ) -> Result < RecordBatch > { let max_buf_len = batch . columns ( ) . iter ( ) . map ( | a | a . get_array_memory_size ( ) ) . max ( ) . unwrap_or_default ( ) ; let columns : Result < Vec < _ > > = batch . columns ( ) . iter ( ) . map ( | column | { if matches ! ( column data_type ( ) DataType Dictionary ( ) ) { hydrate_dictionary ( column ) } else if max_buf_len > batch . num_rows ( ) * 100 { Ok ( deep_clone_array ( column ) ) } else { Ok ( Arc :: clone ( column ) ) } } ) . collect ( ) ; RecordBatch :: try_new ( schema , columns ? ) }
Some batches are small slices of the underlying arrays.
Some batches are small slices of the underlying arrays .
batch
&RecordBatch
schema
SchemaRef
batch
&RecordBatch
schema
SchemaRef