

[以前回答していただいた手法](https://ja.stackoverflow.com/questions/71712/csv%E3%83%95%E3%82%A1%E3%82%A4%E3%83%AB%E3%81%AE%E5%86%85%E5%AE%B9%E3%82%92%E5%88%97%E3%81%94%E3%81%A8%E3%81%AB%E5%87%A6%E7%90%86%E3%81%99%E3%82%8B%E3%81%AB%E3%81%AF)を参考にcsvに保存された17列のデータを各列において1行目に2以上の数値からスタートするプログラミングを書きましたが実行にものすごく時間がかかり1日たっても終わる気配がありません．どこが間違っていますか．

[code]

    import numpy as np
    import pandas as pd
    from pandas import Series, DataFrame
    import matplotlib.pyplot as plt
    from itertools import dropwhile
    %matplotlib inline
    
    data = '1214.csv'
    
    data = pd.read_csv(data)
    data = np.array(data)
    
    from itertools import dropwhile, zip_longest
    
    d0 = [x for x in dropwhile(lambda y: y < 0.2, data[:,0])] 
    d1 = [x for x in dropwhile(lambda y: y < 0.2, data[:,1])]
    d2 = [x for x in dropwhile(lambda y: y < 0.2, data[:,2])] 
    d3 = [x for x in dropwhile(lambda y: y < 0.2, data[:,3])]
    d4 = [x for x in dropwhile(lambda y: y < 0.2, data[:,4])] 
    d5 = [x for x in dropwhile(lambda y: y < 0.2, data[:,5])]
    d6 = [x for x in dropwhile(lambda y: y < 0.2, data[:,6])] 
    d7 = [x for x in dropwhile(lambda y: y < 0.2, data[:,7])]
    d8 = [x for x in dropwhile(lambda y: y < 0.2, data[:,8])] 
    d9 = [x for x in dropwhile(lambda y: y < 0.2, data[:,9])]
    d10= [x for x in dropwhile(lambda y: y < 0.2, data[:,10])] 
    d11= [x for x in dropwhile(lambda y: y < 0.2, data[:,11])]
    d12= [x for x in dropwhile(lambda y: y < 0.2, data[:,12])] 
    d13= [x for x in dropwhile(lambda y: y < 0.2, data[:,13])]
    d14= [x for x in dropwhile(lambda y: y < 0.2, data[:,14])] 
    d15= [x for x in dropwhile(lambda y: y < 0.2, data[:,15])]
    NewData = np.array([d0, d1,d2, d3,d4, d5,d6, d7,d8, d9,d10, d11,d12, d13,d14, d15]） 
    print(NewData)
    np.savetxt('1214-945.csv', NewData, fmt='%s', delimiter=',')
    
[/code]


CC BY-SA 4.0
2020-12-20T12:40:44.963
72771
2020-12-22T15:40:56.680
2020-12-22T05:15:38.037
42568
42568
question
python python3 jupyter-notebook
2列ではスムーズにいっていたが17列にしたとたん実行完了しなくなった


列のインデックスが`0`～`15`までと、16列分指定されています。

データがきっちり15列しか無い場合は、以下2つの行がエラーになって`NewData`が出来ないので、その後の2行もエラーになって処理されていないのでは？

[code]

    d15= [x for x in dropwhile(lambda y: y < 0.2, data[:,15])]
    NewData = np.array([d0, d1,d2, d3,d4, d5,d6, d7,d8, d9,d10, d11,d12, d13,d14, d15])
    
[/code]

* * *

**追記:**

例えば以下のCSVファイルを読み取る部分を、

[code]

    data = '1214.csv'
    
    data = pd.read_csv(data)
    data = np.array(data)
    
[/code]

こちらのように乱数でシミュレートして17列で10～1000行のデータにしても、処理そのものは完了します。

[code]

    data = []
    for _ in range(1000):
        data.append(np.random.uniform(0, 1, 17))
    
    data = np.array(data)
    
[/code]

ただし、こんな`Warning`が出るのと、出来上がった`NewData`の表示が変なので、望んだ結果とは言えないと思われますが。

表示された`Warning`  
`VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences
(which is a list-or-tuple of lists-or-tuples-or ndarrays with different
lengths or shapes) is deprecated. If you meant to do this, you must specify
'dtype=object' when creating the ndarray`

結果表示例(データ行数を10にした場合)  
[以前のQ&A](https://ja.stackoverflow.com/q/71712/26370)を考えると、縦横が転換されているのと、CSVファイルの各行先頭最後に`[`,`]`が入っているのでCSVデータとは言えない状態です。

[code]

    [list([0.8964406887196086, 0.1289855921390296, 0.9821250613116147, 0.7277624380256356, 0.597179945506056, 0.5883628593677697, 0.46642181807325156, 0.046165752886358935, 0.9831355110804288, 0.8059872223235665])
     list([0.81459734967449, 0.07064141016207148, 0.5683665463277392, 0.5798712084908151, 0.04482624113860545, 0.14102892907620534, 0.39305425436742103, 0.36942030150742966, 0.512619106492389, 0.6764118175476419])
     list([0.7591268875733456, 0.11453918216754855, 0.4647680491351407, 0.040091866778552476, 0.1429846985183626, 0.6627834295485381, 0.09036694991356253, 0.810514260026902, 0.9931642800156654])
     list([0.8625025870468651, 0.3509634752035785, 0.039771391720511695, 0.141294678559875, 0.9785141897412436, 0.7871799464751338, 0.4392150886107685, 0.8797371371672483, 0.8847149683486721])
     list([0.497940921882306, 0.9082257949953394, 0.5833805332031865, 0.47877232055889385, 0.08331201995212023, 0.4006076893255164, 0.47959612630114945, 0.5886402830771224, 0.9980262909358327, 0.5224210756830902])
     list([0.27197735039392024, 0.5755502064358436, 0.6911546654769439, 0.6995193500503479, 0.11448204495653014, 0.28069236013054844, 0.27781231267874995, 0.8300432017199777, 0.5017304070162142, 0.11823577351924519])
     list([0.5923816654094234, 0.42428317216701694, 0.21513107003802912, 0.3246311643857014, 0.8250821738073256, 0.10344631269777493, 0.038894047272288956, 0.467108730492769, 0.9424929679078838])
     list([0.9304845568823439, 0.0949142996713167, 0.197681927245144, 0.6532485659646121, 0.07650063989252631, 0.4641428636069759, 0.4930807358621857, 0.8777035315326964, 0.8079133125265369, 0.27151654275407255])
     list([0.5686227354480835, 0.20979325360116918, 0.41795367528452854, 0.6860058585789381, 0.25969874636294243, 0.3285464977855428, 0.5999579688394171, 0.4151960837685723, 0.9951317222604745, 0.46859183299562845])
     list([0.3783579341561787, 0.6599424089628657, 0.6756485785361779, 0.6579640744721366, 0.48120694336419967, 0.6699040599082838, 0.47013107179808233, 0.7897054241420943, 0.2579579777685834, 0.6984153759229584])
     list([0.2119275007054927, 0.9073692839679351, 0.7542551775514874, 0.7304899190222118, 0.8934351608341778, 0.3759023101892295, 0.06554942790036467, 0.3782961369793859, 0.24940028099595324])
     list([0.4792099212751727, 0.19767858701604624, 0.5162174625431372, 0.019386808204984396, 0.8516704517647707, 0.669500238721486, 0.10275441692328746, 0.7262108016346217, 0.27466330949461903, 0.5114110217107879])
     list([0.4736295383577239, 0.5594249849728473, 0.13118135938803988, 0.6614701297724721, 0.31594547319531097, 0.25538415218610466, 0.38813802091478633, 0.9779182451444076, 0.1292926086118813, 0.7164415691309892])
     list([0.3356971055614466, 0.9960279356408579, 0.9018106146850836, 0.7056349705349919, 0.8677843649824027, 0.713199273910345, 0.4500223204462691, 0.16791339066028255, 0.8083384665780848, 0.06577552973825185])
     list([0.8653470673875016, 0.7158305241304749, 0.9241323423093925, 0.1598556330050731, 0.4655566657028062, 0.3714871548975628, 0.30538053450909164, 0.14440821986341756, 0.9057775689030844, 0.5559513690214988])
     list([0.29634069084155035, 0.7219891196591361, 0.4799806341569959, 0.43549102434868503, 0.675185724281284, 0.5623786004405398, 0.24797232810209235, 0.755702079954496, 0.6220137098402305, 0.31865744944572993])]
    
[/code]

CSVファイル

[code]

    [0.8964406887196086, 0.1289855921390296, 0.9821250613116147, 0.7277624380256356, 0.597179945506056, 0.5883628593677697, 0.46642181807325156, 0.046165752886358935, 0.9831355110804288, 0.8059872223235665]
    [0.81459734967449, 0.07064141016207148, 0.5683665463277392, 0.5798712084908151, 0.04482624113860545, 0.14102892907620534, 0.39305425436742103, 0.36942030150742966, 0.512619106492389, 0.6764118175476419]
    [0.7591268875733456, 0.11453918216754855, 0.4647680491351407, 0.040091866778552476, 0.1429846985183626, 0.6627834295485381, 0.09036694991356253, 0.810514260026902, 0.9931642800156654]
    [0.8625025870468651, 0.3509634752035785, 0.039771391720511695, 0.141294678559875, 0.9785141897412436, 0.7871799464751338, 0.4392150886107685, 0.8797371371672483, 0.8847149683486721]
    [0.497940921882306, 0.9082257949953394, 0.5833805332031865, 0.47877232055889385, 0.08331201995212023, 0.4006076893255164, 0.47959612630114945, 0.5886402830771224, 0.9980262909358327, 0.5224210756830902]
    [0.27197735039392024, 0.5755502064358436, 0.6911546654769439, 0.6995193500503479, 0.11448204495653014, 0.28069236013054844, 0.27781231267874995, 0.8300432017199777, 0.5017304070162142, 0.11823577351924519]
    [0.5923816654094234, 0.42428317216701694, 0.21513107003802912, 0.3246311643857014, 0.8250821738073256, 0.10344631269777493, 0.038894047272288956, 0.467108730492769, 0.9424929679078838]
    [0.9304845568823439, 0.0949142996713167, 0.197681927245144, 0.6532485659646121, 0.07650063989252631, 0.4641428636069759, 0.4930807358621857, 0.8777035315326964, 0.8079133125265369, 0.27151654275407255]
    [0.5686227354480835, 0.20979325360116918, 0.41795367528452854, 0.6860058585789381, 0.25969874636294243, 0.3285464977855428, 0.5999579688394171, 0.4151960837685723, 0.9951317222604745, 0.46859183299562845]
    [0.3783579341561787, 0.6599424089628657, 0.6756485785361779, 0.6579640744721366, 0.48120694336419967, 0.6699040599082838, 0.47013107179808233, 0.7897054241420943, 0.2579579777685834, 0.6984153759229584]
    [0.2119275007054927, 0.9073692839679351, 0.7542551775514874, 0.7304899190222118, 0.8934351608341778, 0.3759023101892295, 0.06554942790036467, 0.3782961369793859, 0.24940028099595324]
    [0.4792099212751727, 0.19767858701604624, 0.5162174625431372, 0.019386808204984396, 0.8516704517647707, 0.669500238721486, 0.10275441692328746, 0.7262108016346217, 0.27466330949461903, 0.5114110217107879]
    [0.4736295383577239, 0.5594249849728473, 0.13118135938803988, 0.6614701297724721, 0.31594547319531097, 0.25538415218610466, 0.38813802091478633, 0.9779182451444076, 0.1292926086118813, 0.7164415691309892]
    [0.3356971055614466, 0.9960279356408579, 0.9018106146850836, 0.7056349705349919, 0.8677843649824027, 0.713199273910345, 0.4500223204462691, 0.16791339066028255, 0.8083384665780848, 0.06577552973825185]
    [0.8653470673875016, 0.7158305241304749, 0.9241323423093925, 0.1598556330050731, 0.4655566657028062, 0.3714871548975628, 0.30538053450909164, 0.14440821986341756, 0.9057775689030844, 0.5559513690214988]
    [0.29634069084155035, 0.7219891196591361, 0.4799806341569959, 0.43549102434868503, 0.675185724281284, 0.5623786004405398, 0.24797232810209235, 0.755702079954496, 0.6220137098402305, 0.31865744944572993]
    
[/code]

* * *

**対処案:**

上記のように色々問題があるようなので、0列目から15列目までを連続で行うのなら、以下のように`for in
range()`ループと`pandas`で行うのが良さそうです。

[code]

    work = []
    for i in range(16):
        work.append(list(dropwhile(lambda y: y < 0.2, data[:,i])))
    
    df = pd.DataFrame(work).T.fillna('')
    
    print(df)
    df.to_csv('1214-945.csv', header=False, index=False)
    
[/code]


CC BY-SA 4.0
2020-12-20T14:08:50.787
72772
2020-12-22T08:24:51.737
2020-12-22T08:24:51.737
26370
26370
72771
answer


[itertools.dropwhile()](https://docs.python.org/3/library/itertools.html#itertools.dropwhile)の場合、実行に時間が掛かる場合(状況)があり得るとのことで、

> itertools.dropwhile(predicate, iterable)
>
> Make an iterator that drops elements from the iterable as long as the
> predicate is true; afterwards, returns every element. Note, the iterator
> does not produce any output until the predicate first becomes false, so **it
> may have a lengthy start-up time**.

例えば、 `numpy.ndarray` 型の配列で要素数が10^8(1億)、全ての要素が `0.1`
であるとすると配列要素の全てが走査される事になります。

[code]

    z = np.array([0.1]*int(1e8))
    
    print(timeit.timeit(
      'list(dropwhile(lambda y: y < 0.2, z))',
      globals=globals(), number=1))
    =>
    17.093322979053482 seconds
    
[/code]

質問者さんの計算機環境(CPUの性能やメモリサイズなど)や実際のデータサイズが判らないので、「1日たっても終わる気配がない」ことの原因が
`itertools.dropwhile()` にあるのかどうかは不明です。

ところで、`itertools.dropwhile()` は単純な for loop で実装する事も可能です。また、[Numba: A High
Performance Python Compiler](https://numba.pydata.org/) を利用することで高速化する事が可能です。

[code]

    from numba import njit
    import numpy as np
    import timeit
    
    @njit
    def dropwhile_njit(lst):
      for i in range(len(lst)):
        if not (lst[i] < 0.2):
          break
      return lst[i:]
    
    z = np.array([0.1]*int(1e8))
    
    print(timeit.timeit(
      'dropwhile_njit(z)',
      globals=globals(), number=1))
    
    =>
    0.20109811995644122 seconds
    
[/code]

Numba 版の方がおおよそ 85 倍高速になります(巨大配列をフルスキャンする場合)。

[code]

    17.093322979053482/0.20109811995644122 = 84.999914
    
[/code]


CC BY-SA 4.0
2020-12-22T10:43:23.177
72824
2020-12-22T15:40:56.680
2020-12-22T15:40:56.680
72771
answer