assigning elements to buffer and specific position in buffer - how to speed up / why are the different implementations so slow?
<p>I'm trying to speed up the following code, where given a list of strings <code>str_list</code> I'm trying to convert the string into a number (<code>unpack</code>) and assign this number into the correct position of the nested list <code>data</code>. The dimensions of <code>data</code> are roughly <code>data[4][20][1024]</code>. Unfortunately, this function runs very slowly. Here's the code:</p>

<pre><code>for abs_idx in range(nbr_elements):

    # get string
    this_element = str_list[abs_idx]

    # convert into number
    this_element = unpack('d', this_element)[0]

    # calculate the buffer number
    buffer_nbr = abs_idx / NBR_DATA_POINTS_PER_BUFFER_INT

    # calculate the position inside the buffer
    index_in_buffer = abs_idx % NBR_DATA_POINTS_PER_BUFFER_INT

    # write data into correct position
    data[file_idx][buffer_nbr][index_in_buffer] = this_element
</code></pre>

<p>I also tried the following alternative solution, which is even slower:</p>

<pre><code># convert each string into a number
unpacked_values = [unpack('d', str_list[j])[0] for j in range(nbr_elements)]
for abs_idx in range(nbr_elements):

    # calculate the buffer number
    buffer_nbr = abs_idx / NBR_DATA_POINTS_PER_BUFFER_INT

    # calculate the position inside the buffer
    index_in_buffer = abs_idx % NBR_DATA_POINTS_PER_BUFFER_INT

    # write data into correct position
    data[file_idx][buffer_nbr][index_in_buffer] = unpacked_values[abs_idx]
</code></pre>

<p>To my surprise, the next implementation is the slowest (I expected it to be the fastest):</p>

<pre><code># convert each string into a number
unpacked_values = [unpack('d', str_list[j])[0] for j in range(nbr_elements)]

# calculate all buffer numbers at once
buffer_ids = np.arange(nbr_elements) / NBR_DATA_POINTS_PER_BUFFER_INT

# calculate all positions inside the buffer at once
index_in_buffer_id = np.arange(nbr_elements) % NBR_DATA_POINTS_PER_BUFFER_INT

for abs_idx in range(nbr_elements):
    data[file_idx][buffer_ids[abs_idx]][index_in_buffer_id[abs_idx]] = unpacked_values[abs_idx]
</code></pre>

<p>Why are the successive implementations performing worse? Where are the individual bottlenecks? And how can I speed up my initial code? </p>

<p><strong>EDIT:</strong> from my profiling tests, the following two steps are the bottleneck: running <code>unpack</code> and assigning the value to <code>data</code>. I don't know though how to speed-up these steps.</p>

<p><strong>EDIT2:</strong> I need to use <code>unpack</code> because my strings are in hex.</p>

<p><strong>EDIT3:</strong> <code>values = unpack("d" * n, "".join(str_list))</code> solves the problem with unpack being slow. Still, the assignment to data with the triple (original) or double (modified) nested loop eats up 50% of the time. Is there a way to reduce this time?</p>
