When I apply a function to a DF to create multiple new columns, I get different results depending how many rows are in the DF I submit
Turns out the issue was having nans in the data. I got it down to the point where the function worked with 6 rows of data but not with 7, but then I added a more explicit nan check (I was using 'try except' before, changed to math.isnan() and that solved the problem.
Apologies for wordy question but it's impossible to recreate a sanitised simple example without significant effort, but in short I have a function that does some maths on a row, and then adds the output of the calculations to several new columns, and when I use df[new_columns] = df.apply(function_name,axis=1) with df having only two rows, I get the right answer, however if df has 20 rows, it generates the wrong answer.
Furthermore, I have added a print statement to the function, in both examples (2 rows and 20 rows) to tell me what the function is going to add into the new columns, and the answer is correct every single time according to the print statement, but then somehow when the answer is being added to the final df, it seems to be going wrong somewhere.
I simply do not understand this issue - is it a memory thing with apply() or something?? Any help very much appreciated!!!!
Edit: it's giving me this warning 'DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.'