问:
How to improve for-loop on DataFrame with row-wise calculation depending on previous row?
I have a working code with a for-loop that I would like to optimise in terms of speed (the project has been rejected because it's slow).

This is my situation: I have an Excel file from which I import a table with thousands of rows and a few columns into a Pandas DataFrame. The first column is a monotonic increasing series of timestamps, with 15 minutes frequency.

I have to use this table to calculate few extra columns, append them to the original table and save the resulting DataFrame into a new Excel file.

What is slowing the code is the core calculation of the extra columns. Here is a snippet of the working code:

```<code>import pandas as pd
from datetime import timedelta as td

cons_prod = pd.read_csv("sample.csv", index_col=0, parse_dates=True)

soc_dct = {}  # State of charge (kW)
charge_dct = {}  # Charge (kW)
discharge_dct = {}  # Discharge (kW)
acc_dct = {}  # Auto-consumption NEW (kW)
lst_dct = {}  # Lost injection due to battery efficiency (kW)
inj_dct = {}  # Injection NEW (kW)
gridcons_dct = {}  # Grid Consumption NEW (kW)
agg_dct = {}  # Additional Auto-consumption through battery (kW)

battery_parameters = {
    "power": 50,
    "energy": 130,
    "efficiency": 0.9,
    "minsoc": 0.1,
    "soct0": 65.0,
}

bp_energy = battery_parameters["energy"]
bp_power = battery_parameters["power"]
soct0 = 0.5 * bp_energy

for t in cons_prod.index:
    L = cons_prod.loc[t, "Injection (kW)"]
    m = cons_prod.loc[t, "Grid Consumption (kW)"]
    k = cons_prod.loc[t, "Auto-consumption (kW)"]
    f = cons_prod.loc[t, "Consumption (kW)"]
    if t == cons_prod.index[0]:
        # State of charge (kW)
        soc_dct.setdefault(t, soct0)

        # Charge (kW)
        charge_dct.setdefault(
            t,
            min(
                L,
                (bp_energy - soc_dct[t]) * 4,
                bp_power,
            )
            * battery_parameters["efficiency"]
            if L >= 0
            else 0,
        )

        # Discharge (kW)
        discharge_dct.setdefault(
            t,
            -min(
                m,
                4 * soc_dct[t] - 4 * (battery_parameters["minsoc"] * bp_energy),
                bp_power,
            )
            if m >= 0
            else 0,
        )

        # Auto-consumption NEW (kW)
        acc_dct.setdefault(t, k - discharge_dct[t])

        # Lost injection due to battery efficiency (kW)
        lst_dct.setdefault(
            t,
            (charge_dct[t] / battery_parameters["efficiency"]) - charge_dct[t],
        )

        # Injection NEW (kW)
        inj_dct.setdefault(t, L - charge_dct[t] - lst_dct[t])

        # Grid Consumption NEW (kW)
        gridcons_dct.setdefault(t, f - acc_dct[t])

        # Additional Auto-consumption through battery (kW)
        agg_dct.setdefault(t, acc_dct[t] - k)
    else:
        # State of charge (kW)
        soc_dct.setdefault(
            t,
            soc_dct[t - td(minutes=15)]
            + (charge_dct[t - td(minutes=15)] + discharge_dct[t - td(minutes=15)]) / 4,
        )

        # Charge (kW)
        charge_dct.setdefault(
            t,
            min(
                L,
                (bp_energy - soc_dct[t]) * 4,
                bp_power,
            )
            * battery_parameters["efficiency"]
            if L >= 0
            else 0,
        )

        # Discharge (kW)
        discharge_dct.setdefault(
            t,
            -min(
                m,
                4 * soc_dct[t] - 4 * (battery_parameters["minsoc"] * bp_energy),
                bp_power,
            )
            if m >= 0
            else 0,
        )

        # Auto-consumption NEW (kW)
        acc_dct.setdefault(t, k - discharge_dct[t])

        # Lost injection due to battery efficiency (kW)
        lst_dct.setdefault(
            t, charge_dct[t] / battery_parameters["efficiency"] - charge_dct[t]
        )

        # Injection NEW (kW)
        inj_dct.setdefault(t, L - charge_dct[t] - lst_dct[t])

        # Grid Consumption NEW (kW)
        gridcons_dct.setdefault(t, f - acc_dct[t])

        # Additional Auto-consumption through battery (kW)
        agg_dct.setdefault(t, acc_dct[t] - k)
# Creating a DataFrame with all the values
output_df = pd.DataFrame(
    data=[
        soc_dct,
        charge_dct,
        discharge_dct,
        acc_dct,
        lst_dct,
        inj_dct,
        gridcons_dct,
        agg_dct,
    ]
).T
output_df.columns = [
    "State of charge (kW)",
    "Charge (kW)",
    "Discharge (kW)",
    "Auto-consumption NEW (kW)",
    "Lost injection due to battery efficiency (kW)",
    "Injection NEW (kW)",
    "Grid Consumption NEW (kW)",
    "Additional Auto-consumption through battery (kW)",
]
charge_dct = {}  # Charge (kW)
discharge_dct = {}  # Discharge (kW)
acc_dct = {}  # Auto-consumption NEW (kW)
lst_dct = {}  # Lost injection due to battery efficiency (kW)
inj_dct = {}  # Injection NEW (kW)
gridcons_dct = {}  # Grid Consumption NEW (kW)
agg_dct = {}  # Additional Auto-consumption through battery (kW)

for t in cons_prod.index:
    L = cons_prod.loc[t, "Injection (kW)"]
    m = cons_prod.loc[t, "Grid Consumption (kW)"]
    k = cons_prod.loc[t, "Auto-consumption (kW)"]
    f = cons_prod.loc[t, "Consumption (kW)"]
    if t == cons_prod.index[0]:
        # State of charge (kW)
        soc_dct.setdefault(t, soct0)

        # Charge (kW)
        charge_dct.setdefault(
            t,
            min(
                L,
                (bp_energy - soc_dct[t]) * 4,
                bp_power,
            )
            * battery_parameters["efficiency"]
            if L >= 0
            else 0,
        )

        # Discharge (kW)
        discharge_dct.setdefault(
            t,
            -min(
                m,
                4 * soc_dct[t] - 4 * (battery_parameters["minsoc"] * bp_energy),
                bp_power,
            )
            if m >= 0
            else 0,
        )

        # Auto-consumption NEW (kW)
        acc_dct.setdefault(t, k - discharge_dct[t])

        # Lost injection due to battery efficiency (kW)
        lst_dct.setdefault(
            t,
            (charge_dct[t] / battery_parameters["efficiency"]) - charge_dct[t],
        )

        # Injection NEW (kW)
        inj_dct.setdefault(t, L - charge_dct[t] - lst_dct[t])

        # Grid Consumption NEW (kW)
        gridcons_dct.setdefault(t, f - acc_dct[t])

        # Additional Auto-consumption through battery (kW)
        agg_dct.setdefault(t, acc_dct[t] - k)
    else:
        # State of charge (kW)
        soc_dct.setdefault(
            t,
            soc_dct[t - td(minutes=15)]
            + (charge_dct[t - td(minutes=15)] + discharge_dct[t - td(minutes=15)]) / 4,
        )

        # Charge (kW)
        charge_dct.setdefault(
            t,
            min(
                L,
                (bp_energy - soc_dct[t]) * 4,
                bp_power,
            )
            * battery_parameters["efficiency"]
            if L >= 0
            else 0,
        )

        # Discharge (kW)
        discharge_dct.setdefault(
            t,
            -min(
                m,
                4 * soc_dct[t] - 4 * (battery_parameters["minsoc"] * bp_energy),
                bp_power,
            )
            if m >= 0
            else 0,
        )

        # Auto-consumption NEW (kW)
        acc_dct.setdefault(t, k - discharge_dct[t])

        # Lost injection due to battery efficiency (kW)
        lst_dct.setdefault(
            t, charge_dct[t] / battery_parameters["efficiency"] - charge_dct[t]
        )

        # Injection NEW (kW)
        inj_dct.setdefault(t, L - charge_dct[t] - lst_dct[t])

        # Grid Consumption NEW (kW)
        gridcons_dct.setdefault(t, f - acc_dct[t])

        # Additional Auto-consumption through battery (kW)
        agg_dct.setdefault(t, acc_dct[t] - k)
# Creating a DataFrame with all the values
output_df = pd.DataFrame(
    data=[
        soc_dct,
        charge_dct,
        discharge_dct,
        acc_dct,
        lst_dct,
        inj_dct,
        gridcons_dct,
        agg_dct,
    ]
).T
output_df.columns = [
    "State of charge (kW)",
    "Charge (kW)",
    "Discharge (kW)",
    "Auto-consumption NEW (kW)",
    "Lost injection due to battery efficiency (kW)",
    "Injection NEW (kW)",
    "Grid Consumption NEW (kW)",
    "Additional Auto-consumption through battery (kW)",
]

</code>```

cons_prod is the table imported into a DataFrame.

As you can see, we have two situations: when t == cons_prod.index[0] (i.e., the first item of the timestamps), the calculations use the values at the same t. However, from the second timestamp on, some of the calculations refer to the previous value (in here referred to the 15 minutes before using the index t - td(minutes=15)).

These are the reasons why I am struggling to move away from the for-loop.

A few explanations to possible questions

Q: Why are you using dictionaries? A: Because I figured that they fill faster than other data types, and I can use them to create a DataFrame later on.

Q: Are the timestamps coherent, e.g. do they have missing values? A: No missing values, as there is a previous function I wrote to make sure that the timestamps are filled completely.

Q: The calculations do not currently refer to a previous row in a DataFrame, but in a dictionary! Why the misleading title? A: This is the best solution I could come up with so far, but I am wondering whether my not-so-complete knowledge of Pandas is hiding a simpler, faster solution.

Hope the framework is clear.

Thank you in advance!

EDIT: as per request, added a 100-lines sample of cons_prod and modified the previous code to meet the requirements of a MRE.

EDIT 2: I have tried moving from dictionaries to a Pandas lookup, trying to optimise as much as possible. This is the code I came up with:

```<code>from time import time as tt

cp = cons_prod.copy(deep=True)

# Initialise the columns filling them with zeroes
cp["State of charge (kW)"] = 0
cp["Charge (kW)"] = 0
cp["Discharge (kW)"] = 0

# Storing the position of the columns in variables
cp_soc = cp.columns.get_loc("State of charge (kW)")
cp_charge = cp.columns.get_loc("Charge (kW)")
cp_discharge = cp.columns.get_loc("Discharge (kW)")
cp_inj = cp.columns.get_loc("Injection (kW)")
cp_gridcons = cp.columns.get_loc("Grid Consumption (kW)")

# Storing the values of the battery dictionary lookups in variables
bp_energy = dct_bp["energy"]
bp_power = dct_bp["power"]
bp_efficiency = dct_bp["efficiency"]
bp_soct0 = dct_bp["soct0"]
bp_minsoc = dct_bp["minsoc"]

start1 = tt()  # Measuring time
for row in cp.itertuples(name=None):  # Using itertuples to gain some speed
    L = cp.loc[row[0], "Injection (kW)"]
    m = cp.loc[row[0], "Grid Consumption (kW)"]
    k = cp.loc[row[0], "Auto-consumption (kW)"]
    f = cp.loc[row[0], "Consumption (kW)"]
    if row[0] == cp.index[0]:
        cp.iloc[0, cp_soc] = bp_soct0
        cp.iloc[0, cp_charge] = float(
            min(L, (bp_energy - bp_soct0) * 4, bp_power) * bp_efficiency
            if L >= 0
            else 0,
        )
        cp.iloc[0, cp_discharge] = float(
            -min(
                m,
                4 * bp_soct0 - 4 * (bp_minsoc * bp_energy),
                bp_power,
            )
            if m >= 0
            else 0
        )
    else:
        t = pd.Index(cp.index).get_loc(row[0])
        cp.iloc[t, cp_soc] = float(
            cp.iloc[t - 1, cp_soc]
            + (cp.iloc[t - 1, cp_charge] + cp.iloc[t - 1, cp_discharge]) / 4
        )
        cp.iloc[t, cp_charge] = float(
            min(L, (bp_energy - cp.iloc[t, cp_soc]) * 4, bp_power) * bp_efficiency
            if L >= 0
            else 0,
        )
        cp.iloc[t, cp_discharge] = float(
            -min(
                m,
                4 * cp.iloc[t, cp_soc] - 4 * (dct_bp["minsoc"] * bp_energy),
                bp_power,
            )
            if m >= 0
            else 0
        )
end1 = tt() - start1
print(f"Pandas lookup took {end1:.2f} seconds")

</code>```

With this code, I got an average of 42 seconds per complete task, while I used to have <20 seconds with dictionaries.

答:
Cython can solve slow looping issue and it is perfect with any type of calculation you need. Cython allows you to code a c-style program in Python: https://cython.readthedocs.io/en/latest/src/quickstart/overview.html
In order to execute a cython function, you need to create a file, write cython functions and compile the file. The documentation helps you to better understand the syntax.
Another solution is to try the code in a Jupyter notebook, following this steps:

Execute in a console/terminal pip install Cython

In Jupyter notebook, write and execute %load_ext Cython

Create your function inside a cell where the first row is a magic-keyword %%cython

I also noticed some replicated code; I tried to simplify as much as I understood

That is my solution, run in a Jupyter notebook:
I suppose the type for each column is float32/int32, instead of float64/int64; It should be a reasonable choice

```<code>cons_prod['Consumption (kW)'] = cons_prod['Consumption (kW)'].astype(np.int32)
cons_prod['Renewable Production (kW)'] = cons_prod['Renewable Production (kW)'].astype(np.float32)
cons_prod['Auto-consumption (kW)']= cons_prod['Auto-consumption (kW)'].astype(np.float32)
cons_prod['Injection (kW)']= cons_prod['Injection (kW)'].astype(np.float32)
cons_prod['Grid Consumption (kW)']= cons_prod['Grid Consumption (kW)'].astype(np.float32)

cons_prod.dtypes
</code>```

To increase the performance of cython, an explicit typing is crucial

%%cython
import cython
cimport cython
cimport numpy as np
import numpy as np

ctypedef np.float32_t float32_t
ctypedef np.int32_t int32_t

def func(
    float32_t[:] injection,
    float32_t[:] grid_consumption,
    float32_t[:] auto_consumption,
    int32_t[:] consumption,
    battery_parameters
    ):

    cdef int n_rows = len(injection)
    cdef int bp_energy = battery_parameters["energy"]
    cdef int bp_power = battery_parameters["power"]
    cdef float soct0 = 0.5 * bp_energy

    soc_dct = np.zeros(n_rows, dtype=np.float32)
    charge_dct = np.zeros(n_rows, dtype=np.float32)
    discharge_dct = np.zeros(n_rows, dtype=np.float32)
    acc_dct = np.zeros(n_rows, dtype=np.float32)
    lst_dct = np.zeros(n_rows, dtype=np.float32)
    inj_dct = np.zeros(n_rows, dtype=np.float32)
    gridcons_dct = np.zeros(n_rows, dtype=np.float32)
    agg_dct = np.zeros(n_rows, dtype=np.float32)

    cdef float32_t[:] soc_view = soc_dct
    cdef float32_t[:] charge_view = charge_dct
    cdef float32_t[:] discharge_view = discharge_dct
    cdef float32_t[:] acc_view = acc_dct
    cdef float32_t[:] lst_view = lst_dct
    cdef float32_t[:] inj_view = inj_dct
    cdef float32_t[:] gridcons_view = gridcons_dct
    cdef float32_t[:] agg_view = agg_dct

    cdef int i

    # L = injection[i]
    # m = grid_consumption[i]
    # k = auto_consumption[i]
    # f = consumption[i]

    for i in range(0, n_rows):
        if i == 0:
            # State of charge (kW)
            soc_view[i] = soct0
        else:
            # State of charge (kW)
            soc_view[i] = soc_view[i-1] + (charge_view[i-1] + discharge_view[i-1]) / 4

        # Charge (kW)
        charge_view[i] = min(
            injection[i], (bp_energy - soc_view[i]) * 4,
            bp_power
        ) * battery_parameters['efficiency'] if injection[i]>=0 else 0

        # Discharge (kW)
        discharge_view[i] = -min(
            grid_consumption[i],
            4*soc_view[i] - 4*(battery_parameters['minsoc']*bp_energy),
            bp_power
        ) if grid_consumption[i] >= 0 else 0

        # Auto-consumption NEW (kW)
        auto_consumption[i] - discharge_view[i]

        # Lost injection due to battery efficiency (kW)
        lst_view[i] = (charge_view[i] / battery_parameters["efficiency"]) - charge_view[i]

        # Injection NEW (kW)
        inj_view[i] = injection[i] - charge_view[i] - lst_view[i]

        # Grid Consumption NEW (kW)
        gridcons_view[i] = consumption[i] - acc_view[i]

        # Additional Auto-consumption through battery (kW)
        agg_view[i] = acc_view[i] - auto_consumption[i]

    return soc_dct, charge_dct, discharge_dct, acc_dct, lst_dct, inj_dct, gridcons_dct,agg_dct
```

Benchmarks

Initial code: 15.6ms

Cython code: 349 µs

Body:
I have a working code with a for-loop that I would like to optimise in terms of speed (the project has been rejected because it's slow).

This is my situation: I have an Excel file from which I import a table with thousands of rows and a few columns into a Pandas DataFrame. The first column is a monotonic increasing series of timestamps, with 15 minutes frequency.

I have to use this table to calculate few extra columns, append them to the original table and save the resulting DataFrame into a new Excel file.

What is slowing the code is the core calculation of the extra columns. Here is a snippet of the working code:

import pandas as pd
from datetime import timedelta as td

cons_prod = pd.read_csv("sample.csv", index_col=0, parse_dates=True)

soc_dct = {}  # State of charge (kW)
charge_dct = {}  # Charge (kW)
discharge_dct = {}  # Discharge (kW)
acc_dct = {}  # Auto-consumption NEW (kW)
lst_dct = {}  # Lost injection due to battery efficiency (kW)
inj_dct = {}  # Injection NEW (kW)
gridcons_dct = {}  # Grid Consumption NEW (kW)
agg_dct = {}  # Additional Auto-consumption through battery (kW)

battery_parameters = {
    "power": 50,
    "energy": 130,
    "efficiency": 0.9,
    "minsoc": 0.1,
    "soct0": 65.0,
}

bp_energy = battery_parameters["energy"]
bp_power = battery_parameters["power"]
soct0 = 0.5 * bp_energy

for t in cons_prod.index:
    L = cons_prod.loc[t, "Injection (kW)"]
    m = cons_prod.loc[t, "Grid Consumption (kW)"]
    k = cons_prod.loc[t, "Auto-consumption (kW)"]
    f = cons_prod.loc[t, "Consumption (kW)"]
    if t == cons_prod.index[0]:
        # State of charge (kW)
        soc_dct.setdefault(t, soct0)

        # Charge (kW)
        charge_dct.setdefault(
            t,
            min(
                L,
                (bp_energy - soc_dct[t]) * 4,
                bp_power,
            )
            * battery_parameters["efficiency"]
            if L >= 0
            else 0,
        )

        # Discharge (kW)
        discharge_dct.setdefault(
            t,
            -min(
                m,
                4 * soc_dct[t] - 4 * (battery_parameters["minsoc"] * bp_energy),
                bp_power,
            )
            if m >= 0
            else 0,
        )

        # Auto-consumption NEW (kW)
        acc_dct.setdefault(t, k - discharge_dct[t])

        # Lost injection due to battery efficiency (kW)
        lst_dct.setdefault(
            t,
            (charge_dct[t] / battery_parameters["efficiency"]) - charge_dct[t],
        )

        # Injection NEW (kW)
        inj_dct.setdefault(t, L - charge_dct[t] - lst_dct[t])

        # Grid Consumption NEW (kW)
        gridcons_dct.setdefault(t, f - acc_dct[t])

        # Additional Auto-consumption through battery (kW)
        agg_dct.setdefault(t, acc_dct[t] - k)
    else:
        # State of charge (kW)
        soc_dct.setdefault(
            t,
            soc_dct[t - td(minutes=15)]
            + (charge_dct[t - td(minutes=15)] + discharge_dct[t - td(minutes=15)]) / 4,
        )

        # Charge (kW)
        charge_dct.setdefault(
            t,
            min(
                L,
                (bp_energy - soc_dct[t]) * 4,
                bp_power,
            )
            * battery_parameters["efficiency"]
            if L >= 0
            else 0,
        )

        # Discharge (kW)
        discharge_dct.setdefault(
            t,
            -min(
                m,
                4 * soc_dct[t] - 4 * (battery_parameters["minsoc"] * bp_energy),
                bp_power,
            )
            if m >= 0
            else 0,
        )

        # Auto-consumption NEW (kW)
        acc_dct.setdefault(t, k - discharge_dct[t])

        # Lost injection due to battery efficiency (kW)
        lst_dct.setdefault(
            t, charge_dct[t] / battery_parameters["efficiency"] - charge_dct[t]
        )

        # Injection NEW (kW)
        inj_dct.setdefault(t, L - charge_dct[t] - lst_dct[t])

        # Grid Consumption NEW (kW)
        gridcons_dct.setdefault(t, f - acc_dct[t])

        # Additional Auto-consumption through battery (kW)
        agg_dct.setdefault(t, acc_dct[t] - k)
# Creating a DataFrame with all the values
output_df = pd.DataFrame(
    data=[
        soc_dct,
        charge_dct,
        discharge_dct,
        acc_dct,
        lst_dct,
        inj_dct,
        gridcons_dct,
        agg_dct,
    ]
).T
output_df.columns = [
    "State of charge (kW)",
    "Charge (kW)",
    "Discharge (kW)",
    "Auto-consumption NEW (kW)",
    "Lost injection due to battery efficiency (kW)",
    "Injection NEW (kW)",
    "Grid Consumption NEW (kW)",
    "Additional Auto-consumption through battery (kW)",
]
charge_dct = {}  # Charge (kW)
discharge_dct = {}  # Discharge (kW)
acc_dct = {}  # Auto-consumption NEW (kW)
lst_dct = {}  # Lost injection due to battery efficiency (kW)
inj_dct = {}  # Injection NEW (kW)
gridcons_dct = {}  # Grid Consumption NEW (kW)
agg_dct = {}  # Additional Auto-consumption through battery (kW)

for t in cons_prod.index:
    L = cons_prod.loc[t, "Injection (kW)"]
    m = cons_prod.loc[t, "Grid Consumption (kW)"]
    k = cons_prod.loc[t, "Auto-consumption (kW)"]
    f = cons_prod.loc[t, "Consumption (kW)"]
    if t == cons_prod.index[0]:
        # State of charge (kW)
        soc_dct.setdefault(t, soct0)

        # Charge (kW)
        charge_dct.setdefault(
            t,
            min(
                L,
                (bp_energy - soc_dct[t]) * 4,
                bp_power,
            )
            * battery_parameters["efficiency"]
            if L >= 0
            else 0,
        )

        # Discharge (kW)
        discharge_dct.setdefault(
            t,
            -min(
                m,
                4 * soc_dct[t] - 4 * (battery_parameters["minsoc"] * bp_energy),
                bp_power,
            )
            if m >= 0
            else 0,
        )

        # Auto-consumption NEW (kW)
        acc_dct.setdefault(t, k - discharge_dct[t])

        # Lost injection due to battery efficiency (kW)
        lst_dct.setdefault(
            t,
            (charge_dct[t] / battery_parameters["efficiency"]) - charge_dct[t],
        )

        # Injection NEW (kW)
        inj_dct.setdefault(t, L - charge_dct[t] - lst_dct[t])

        # Grid Consumption NEW (kW)
        gridcons_dct.setdefault(t, f - acc_dct[t])

        # Additional Auto-consumption through battery (kW)
        agg_dct.setdefault(t, acc_dct[t] - k)
    else:
        # State of charge (kW)
        soc_dct.setdefault(
            t,
            soc_dct[t - td(minutes=15)]
            + (charge_dct[t - td(minutes=15)] + discharge_dct[t - td(minutes=15)]) / 4,
        )

        # Charge (kW)
        charge_dct.setdefault(
            t,
            min(
                L,
                (bp_energy - soc_dct[t]) * 4,
                bp_power,
            )
            * battery_parameters["efficiency"]
            if L >= 0
            else 0,
        )

        # Discharge (kW)
        discharge_dct.setdefault(
            t,
            -min(
                m,
                4 * soc_dct[t] - 4 * (battery_parameters["minsoc"] * bp_energy),
                bp_power,
            )
            if m >= 0
            else 0,
        )

        # Auto-consumption NEW (kW)
        acc_dct.setdefault(t, k - discharge_dct[t])

        # Lost injection due to battery efficiency (kW)
        lst_dct.setdefault(
            t, charge_dct[t] / battery_parameters["efficiency"] - charge_dct[t]
        )

        # Injection NEW (kW)
        inj_dct.setdefault(t, L - charge_dct[t] - lst_dct[t])

        # Grid Consumption NEW (kW)
        gridcons_dct.setdefault(t, f - acc_dct[t])

        # Additional Auto-consumption through battery (kW)
        agg_dct.setdefault(t, acc_dct[t] - k)
# Creating a DataFrame with all the values
output_df = pd.DataFrame(
    data=[
        soc_dct,
        charge_dct,
        discharge_dct,
        acc_dct,
        lst_dct,
        inj_dct,
        gridcons_dct,
        agg_dct,
    ]
).T
output_df.columns = [
    "State of charge (kW)",
    "Charge (kW)",
    "Discharge (kW)",
    "Auto-consumption NEW (kW)",
    "Lost injection due to battery efficiency (kW)",
    "Injection NEW (kW)",
    "Grid Consumption NEW (kW)",
    "Additional Auto-consumption through battery (kW)",
]

cons_prod is the table imported into a DataFrame.

As you can see, we have two situations: when t == cons_prod.index[0] (i.e., the first item of the timestamps), the calculations use the values at the same t. However, from the second timestamp on, some of the calculations refer to the previous value (in here referred to the 15 minutes before using the index t - td(minutes=15)).

These are the reasons why I am struggling to move away from the for-loop.

A few explanations to possible questions

Q: Why are you using dictionaries? A: Because I figured that they fill faster than other data types, and I can use them to create a DataFrame later on.

Q: Are the timestamps coherent, e.g. do they have missing values? A: No missing values, as there is a previous function I wrote to make sure that the timestamps are filled completely.

Q: The calculations do not currently refer to a previous row in a DataFrame, but in a dictionary! Why the misleading title? A: This is the best solution I could come up with so far, but I am wondering whether my not-so-complete knowledge of Pandas is hiding a simpler, faster solution.

Hope the framework is clear.

Thank you in advance!

EDIT: as per request, added a 100-lines sample of cons_prod and modified the previous code to meet the requirements of a MRE.

EDIT 2: I have tried moving from dictionaries to a Pandas lookup, trying to optimise as much as possible. This is the code I came up with:

from time import time as tt

cp = cons_prod.copy(deep=True)

# Initialise the columns filling them with zeroes
cp["State of charge (kW)"] = 0
cp["Charge (kW)"] = 0
cp["Discharge (kW)"] = 0

# Storing the position of the columns in variables
cp_soc = cp.columns.get_loc("State of charge (kW)")
cp_charge = cp.columns.get_loc("Charge (kW)")
cp_discharge = cp.columns.get_loc("Discharge (kW)")
cp_inj = cp.columns.get_loc("Injection (kW)")
cp_gridcons = cp.columns.get_loc("Grid Consumption (kW)")

# Storing the values of the battery dictionary lookups in variables
bp_energy = dct_bp["energy"]
bp_power = dct_bp["power"]
bp_efficiency = dct_bp["efficiency"]
bp_soct0 = dct_bp["soct0"]
bp_minsoc = dct_bp["minsoc"]

start1 = tt()  # Measuring time
for row in cp.itertuples(name=None):  # Using itertuples to gain some speed
    L = cp.loc[row[0], "Injection (kW)"]
    m = cp.loc[row[0], "Grid Consumption (kW)"]
    k = cp.loc[row[0], "Auto-consumption (kW)"]
    f = cp.loc[row[0], "Consumption (kW)"]
    if row[0] == cp.index[0]:
        cp.iloc[0, cp_soc] = bp_soct0
        cp.iloc[0, cp_charge] = float(
            min(L, (bp_energy - bp_soct0) * 4, bp_power) * bp_efficiency
            if L >= 0
            else 0,
        )
        cp.iloc[0, cp_discharge] = float(
            -min(
                m,
                4 * bp_soct0 - 4 * (bp_minsoc * bp_energy),
                bp_power,
            )
            if m >= 0
            else 0
        )
    else:
        t = pd.Index(cp.index).get_loc(row[0])
        cp.iloc[t, cp_soc] = float(
            cp.iloc[t - 1, cp_soc]
            + (cp.iloc[t - 1, cp_charge] + cp.iloc[t - 1, cp_discharge]) / 4
        )
        cp.iloc[t, cp_charge] = float(
            min(L, (bp_energy - cp.iloc[t, cp_soc]) * 4, bp_power) * bp_efficiency
            if L >= 0
            else 0,
        )
        cp.iloc[t, cp_discharge] = float(
            -min(
                m,
                4 * cp.iloc[t, cp_soc] - 4 * (dct_bp["minsoc"] * bp_energy),
                bp_power,
            )
            if m >= 0
            else 0
        )
end1 = tt() - start1
print(f"Pandas lookup took {end1:.2f} seconds")

With this code, I got an average of 42 seconds per complete task, while I used to have <20 seconds with dictionaries.

Body:
Cython can solve slow looping issue and it is perfect with any type of calculation you need. Cython allows you to code a c-style program in Python: https://cython.readthedocs.io/en/latest/src/quickstart/overview.html
In order to execute a cython function, you need to create a file, write cython functions and compile the file. The documentation helps you to better understand the syntax.
Another solution is to try the code in a Jupyter notebook, following this steps:

Execute in a console/terminal pip install Cython

In Jupyter notebook, write and execute %load_ext Cython

Create your function inside a cell where the first row is a magic-keyword %%cython

I also noticed some replicated code; I tried to simplify as much as I understood

That is my solution, run in a Jupyter notebook:
I suppose the type for each column is float32/int32, instead of float64/int64; It should be a reasonable choice

cons_prod['Consumption (kW)'] = cons_prod['Consumption (kW)'].astype(np.int32)
cons_prod['Renewable Production (kW)'] = cons_prod['Renewable Production (kW)'].astype(np.float32)
cons_prod['Auto-consumption (kW)']= cons_prod['Auto-consumption (kW)'].astype(np.float32)
cons_prod['Injection (kW)']= cons_prod['Injection (kW)'].astype(np.float32)
cons_prod['Grid Consumption (kW)']= cons_prod['Grid Consumption (kW)'].astype(np.float32)

cons_prod.dtypes

To increase the performance of cython, an explicit typing is crucial

%%cython
import cython
cimport cython
cimport numpy as np
import numpy as np

ctypedef np.float32_t float32_t
ctypedef np.int32_t int32_t

def func(
    float32_t[:] injection,
    float32_t[:] grid_consumption,
    float32_t[:] auto_consumption,
    int32_t[:] consumption,
    battery_parameters
    ):

    cdef int n_rows = len(injection)
    cdef int bp_energy = battery_parameters["energy"]
    cdef int bp_power = battery_parameters["power"]
    cdef float soct0 = 0.5 * bp_energy

    soc_dct = np.zeros(n_rows, dtype=np.float32)
    charge_dct = np.zeros(n_rows, dtype=np.float32)
    discharge_dct = np.zeros(n_rows, dtype=np.float32)
    acc_dct = np.zeros(n_rows, dtype=np.float32)
    lst_dct = np.zeros(n_rows, dtype=np.float32)
    inj_dct = np.zeros(n_rows, dtype=np.float32)
    gridcons_dct = np.zeros(n_rows, dtype=np.float32)
    agg_dct = np.zeros(n_rows, dtype=np.float32)

    cdef float32_t[:] soc_view = soc_dct
    cdef float32_t[:] charge_view = charge_dct
    cdef float32_t[:] discharge_view = discharge_dct
    cdef float32_t[:] acc_view = acc_dct
    cdef float32_t[:] lst_view = lst_dct
    cdef float32_t[:] inj_view = inj_dct
    cdef float32_t[:] gridcons_view = gridcons_dct
    cdef float32_t[:] agg_view = agg_dct

    cdef int i

    # L = injection[i]
    # m = grid_consumption[i]
    # k = auto_consumption[i]
    # f = consumption[i]

    for i in range(0, n_rows):
        if i == 0:
            # State of charge (kW)
            soc_view[i] = soct0
        else:
            # State of charge (kW)
            soc_view[i] = soc_view[i-1] + (charge_view[i-1] + discharge_view[i-1]) / 4

        # Charge (kW)
        charge_view[i] = min(
            injection[i], (bp_energy - soc_view[i]) * 4,
            bp_power
        ) * battery_parameters['efficiency'] if injection[i]>=0 else 0

        # Discharge (kW)
        discharge_view[i] = -min(
            grid_consumption[i],
            4*soc_view[i] - 4*(battery_parameters['minsoc']*bp_energy),
            bp_power
        ) if grid_consumption[i] >= 0 else 0

        # Auto-consumption NEW (kW)
        auto_consumption[i] - discharge_view[i]

        # Lost injection due to battery efficiency (kW)
        lst_view[i] = (charge_view[i] / battery_parameters["efficiency"]) - charge_view[i]

        # Injection NEW (kW)
        inj_view[i] = injection[i] - charge_view[i] - lst_view[i]

        # Grid Consumption NEW (kW)
        gridcons_view[i] = consumption[i] - acc_view[i]

        # Additional Auto-consumption through battery (kW)
        agg_view[i] = acc_view[i] - auto_consumption[i]

    return soc_dct, charge_dct, discharge_dct, acc_dct, lst_dct, inj_dct, gridcons_dct,agg_dct

Benchmarks

Initial code: 15.6ms

Cython code: 349 µs


来源: stackexchange

问:
Creating GGPLOT Histogram
I have the following function, 'n' rolls of 'k' dice, in which the result is the sum of 'k' rolls.

```<code>rolldie = function(n, k){
  # Your code here
  die=sample(1:6, size = k, replace = TRUE)
  replicate(n, sum(sample(c(1:6),k, replace=TRUE)))
}
rolldie(5, 4)
</code>```

I then have to create a ggplot histogram to show the results of the function, using n=1000 with whatever values of k. That is where I am stuck, I am not sure how to create this histogram for the results of rolldie.

答:
Here is one solution:</p>
```<code>library(janitor)
data <- tabyl(rolldie(1000, 1)) %>% rename(Number = 1)
ggplot(data = data, aes(x= as.factor(Number), y=n, label = Number))+
  geom_col() +
  geom_text(aes(label = n), vjust = -0.5)

</code>```

ContentLicense: CC BY-SA 4.0

Title: Creating GGPLOT Histogram

Body:
I have the following function, 'n' rolls of 'k' dice, in which the result is the sum of 'k' rolls.

rolldie = function(n, k){
  # Your code here
  die=sample(1:6, size = k, replace = TRUE)
  replicate(n, sum(sample(c(1:6),k, replace=TRUE)))
}
rolldie(5, 4)

I then have to create a ggplot histogram to show the results of the function, using n=1000 with whatever values of k. That is where I am stuck, I am not sure how to create this histogram for the results of rolldie.

Body:
Here is one solution:

library(janitor)
data <- tabyl(rolldie(1000, 1)) %>% rename(Number = 1)
ggplot(data = data, aes(x= as.factor(Number), y=n, label = Number))+
  geom_col() +
  geom_text(aes(label = n), vjust = -0.5)


问:
Is there any way I can print multiple elements each of an array twice on separate lines?
My code so far prints the whole array twice but I want to print each array element twice such that "Daudu" appears twice side by side etc and others like that on separate lines.</p>
```<code>let Smart = ["Daudu", "Emma", "Skodo", "John", "Jones", "Mike", "Sierra", "Oscar", "Alpha", "Bravo"];
console.log(Smart.join('\r\n'));
console.log(Smart.join('\r\n'));

</code>```

答:
You can use a for loop

const smart = ['Daudu', 'Emma', 'Skodo', 'John', 'Jones', 'Mike', 'Sierra', 'Oscar', 'Alpha', 'Bravo'];

for (const item of smart) {
    console.log(`${item}, ${item}`);
}```
</div>
</div>
</p>
<p>Or you can create a new array then do the same thing as above</p>
<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-js lang-js prettyprint-override"><code>const smart = ['Daudu', 'Emma', 'Skodo', 'John', 'Jones', 'Mike', 'Sierra', 'Oscar', 'Alpha', 'Bravo'];

const newSmart = smart.map((item) => `${item}, ${item}`);

for (const item of newSmart) {
    console.log(item);
}</code>```

Body:
My code so far prints the whole array twice but I want to print each array element twice such that "Daudu" appears twice side by side etc and others like that on separate lines.

let Smart = ["Daudu", "Emma", "Skodo", "John", "Jones", "Mike", "Sierra", "Oscar", "Alpha", "Bravo"];
console.log(Smart.join('\r\n'));
console.log(Smart.join('\r\n'));

Body:
You can use a for loop

const smart = ['Daudu', 'Emma', 'Skodo', 'John', 'Jones', 'Mike', 'Sierra', 'Oscar', 'Alpha', 'Bravo'];

for (const item of smart) {
    console.log(`${item}, ${item}`);
}

Or you can create a new array then do the same thing as above

const smart = ['Daudu', 'Emma', 'Skodo', 'John', 'Jones', 'Mike', 'Sierra', 'Oscar', 'Alpha', 'Bravo'];

const newSmart = smart.map((item) => `${item}, ${item}`);

for (const item of newSmart) {
    console.log(item);
}


问:
How to get username in Next-Auth for Github Provider?
I'm using https://next-auth.js.org/ for my login system, but i cannot get the username from it. It's only return name, email, image, and accesstoken. But what i want is username from the GitHub account so how do i achieve that?</p>

答:
I had to do a little workaround

Use this endpoint to get your user info by id:
https://api.github.com/user/[userId]

You can get your userId from your data.user.image, which next-auth provides:
https://avatars.githubusercontent.com/u/[userId]?v=4


问:
Safe method to get value of nested dictionary
I have a nested dictionary. Is there only one way to get values out safely?

```<code>try:
    example_dict['key1']['key2']
except KeyError:
    pass
</code>```

Or maybe python has a method like get() for nested dictionary ?

答:
Here is a solution based on the unutbu's function answer plus:

python naming guidelines

default value as a parameter

not using try but just checking if key is on object

def safe_get(dictionary, *keys, default=None):
    for key in keys:
        if key not in dictionary:
            return default
        dictionary = dictionary[key]
    return dictionary
```

Body:
I have a nested dictionary. Is there only one way to get values out safely?

try:
    example_dict['key1']['key2']
except KeyError:
    pass

Or maybe python has a method like get() for nested dictionary ?


问:
python libxml2 serialization problem
I'm having some problems while attempting to serialize a libxml2 object in python.

In particular, if I run :

```<code>xmlstring = cdoc.serialize('UTF-8', 1)
</code>```

where cdoc is a libxml2 document created before,

I have something like this : ( on a mac - port python 2.5.6)

```<code><?xml version="1.0" encoding="UTF-8"?>
<body>
    <journal>
      <journal_metadata language="en">
        <full_title>Contributions of Lick Observatory</full_title>
        <issn media_type="print">1750-0095</issn>
        <coden>CoLic</coden>
      </journal_metadata>
    </journal>
</body>
</code>```

while on Linux (Centos 5.6 with python 2.4 or 2.6) I have :

```<code><?xml version="1.0" encoding="UTF-8"?>
<body>
    <journal>
      <journal_metadata language="en">
        <full_title>
Contributions of Lick Observatory
        </full_title>
        <issn media_type="print">
1750-0095
        </issn>
        <coden>
CoLic
        </coden>
      </journal_metadata>
    </journal>
</body>
</code>```

There is a problem in the indentation of the string node.

Now I could solve this by doing :

```<code>xmlstring = cdoc.serialize('UTF-8')
</code>```

without the option for the indentation, but the output format is not very readable.

Is there a way to fix the awkward indentation of the output?

答:
I just noticed that I was running:

```<code>xmlstring = cdoc.serialize('UTF-8', 2)
</code>```

and not

```<code>xmlstring = cdoc.serialize('UTF-8', 1)
</code>```

This, for some reasons, on libxml2 2.6 indents also the text node and on libxml2 2.7 doesn't.

ContentLicense: CC BY-SA 3.0

Title: python libxml2 serialization problem

Body:
I'm having some problems while attempting to serialize a libxml2 object in python.

In particular, if I run :

xmlstring = cdoc.serialize('UTF-8', 1)

where cdoc is a libxml2 document created before,

I have something like this : ( on a mac - port python 2.5.6)

<?xml version="1.0" encoding="UTF-8"?>

    <journal>
      <journal_metadata language="en">
        <full_title>Contributions of Lick Observatory</full_title>
        <issn media_type="print">1750-0095</issn>
        <coden>CoLic</coden>
      </journal_metadata>
    </journal>

while on Linux (Centos 5.6 with python 2.4 or 2.6) I have :

<?xml version="1.0" encoding="UTF-8"?>

    <journal>
      <journal_metadata language="en">
        <full_title>
Contributions of Lick Observatory
        </full_title>
        <issn media_type="print">
1750-0095
        </issn>
        <coden>
CoLic
        </coden>
      </journal_metadata>
    </journal>

There is a problem in the indentation of the string node.

Now I could solve this by doing :

xmlstring = cdoc.serialize('UTF-8')

without the option for the indentation, but the output format is not very readable.

Is there a way to fix the awkward indentation of the output?

Body:
I just noticed that I was running:

xmlstring = cdoc.serialize('UTF-8', 2)

and not

xmlstring = cdoc.serialize('UTF-8', 1)

This, for some reasons, on libxml2 2.6 indents also the text node and on libxml2 2.7 doesn't.


问:
write bash-array to json with jq: one element per object
I'm trying to save some values from a bash-array into an existing json. I tried some different approaches, but I didn't manage to solve the issue. What am I missing here?

input.json

```<code>{
"type": "FeatureCollection",
"name": "test",
"features": [
{ "type": "Feature", "properties": { "OBJECTID": 3213, "USE": "dem", "tile": "4045644"}},
{ "type": "Feature", "properties": { "OBJECTID": 3214, "USE": "dem", "tile": "4045646"}},
{ "type": "Feature", "properties": { "OBJECTID": 3215, "USE": "dem", "tile": "4045648"}}
]
}
</code>```

Output of some calculations in Bash - 100% derived from input.json. So this array length == length of the "features[]"-array in the input.json

```<code>printf '%s\n' "${pushintojson[@]}"

4045646 NV NV NV
4045648 NV 4045644 NV
4045650 NV 4045646 NV
</code>```
```<code>jq '.features |= map(.properties +=  {NOSW: $ARGS.positional})' --args "${pushintojson[@]}"  <<< "${input.json}"
</code>```

So this produces this (newlines added by me for readability):

```<code>{
"type":"FeatureCollection",
"name":"test",
"features": [
{"type":"Feature","properties":{"OBJECTID":3213,"USE":"dem","tile":"4045644",
"NOSW":[
"4045646 NV NV NV",
"4045648 NV 4045644 NV",
"4045650 NV 4045646 NV"
]}},

{"type":"Feature","properties":{"OBJECTID":3214,"USE":"dem","tile":"4045646",
"NOSW":[
"4045646 NV NV NV",
"4045648 NV 4045644 NV",
"4045650 NV 4045646 NV"
]}},

{"type":"Feature","properties":{"OBJECTID":3215,"USE":"dem","tile":"4045648",
"NOSW":[
"4045646 NV NV NV",
"4045648 NV 4045644 NV",
"4045650 NV 4045646 NV"
]}}
]
}
</code>```

All values of the bash-array are saved to EVERY element. It's supposed to be just one value per json-object - actually as an array itself.
So in pseudo: "pushintojson[0]" needs to be under "features[0]", pushintojson[1]" to "features[1]"
pushintojson[2]" to "features[2]" etc.

DESIRED OUTPUT

```<code>{
"type":"FeatureCollection",
"name":"test",
"features": [
{"type":"Feature","properties":{"OBJECTID":3213,"USE":"dem","tile":"4045644",
"NOSW":"4045646 NV NV NV"
}},
{"type":"Feature","properties":{"OBJECTID":3214,"USE":"dem","tile":"4045646",
"NOSW":"4045648 NV 4045644 NV"
}},
{"type":"Feature","properties":{"OBJECTID":3215,"USE":"dem","tile":"4045648",
"NOSW":"4045650 NV 4045646 NV"
}}
]
}
</code>```

答:
If you only use map then for each element you end up iterating over all arguments again. Instead, you have to align the .features array to the $ARGS.positional array.

One way, which is closer to your attempt, is to expand the .features array using to_entries to fit the array's elements  with their keys, which in turn can be used within map to reference by index the corresponding item of the $ARGS.positional array.

jq '
  .features |= (to_entries | map(
    .value * {properties: {NOSW: $ARGS.positional[.key]}}
  ))
' --args "${pushintojson[@]}" < input.json
```
<p>Another approach would be to use <code>transpose</code> to generate an array of aligned items from both initial arrays, and then again just <code>map</code> their contents together.</p>
<pre class="lang-sh prettyprint-override"><code>jq '
  .features |= ([., $ARGS.positional] | transpose | map(
    .[0] * {properties: {NOSW: .[1]}}
  ))
' --args "${pushintojson[@]}" < input.json
</code>```

Both produce

{
  "type": "FeatureCollection",
  "name": "test",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "OBJECTID": 3213,
        "USE": "dem",
        "tile": "4045644",
        "NOSW": "4045646 NV NV NV"
      }
    },
    {
      "type": "Feature",
      "properties": {
        "OBJECTID": 3214,
        "USE": "dem",
        "tile": "4045646",
        "NOSW": "4045648 NV 4045644 NV"
      }
    },
    {
      "type": "Feature",
      "properties": {
        "OBJECTID": 3215,
        "USE": "dem",
        "tile": "4045648",
        "NOSW": "4045650 NV 4045646 NV"
      }
    }
  ]
}
```

Body:
I'm trying to save some values from a bash-array into an existing json. I tried some different approaches, but I didn't manage to solve the issue. What am I missing here?

input.json

{
"type": "FeatureCollection",
"name": "test",
"features": [
{ "type": "Feature", "properties": { "OBJECTID": 3213, "USE": "dem", "tile": "4045644"}},
{ "type": "Feature", "properties": { "OBJECTID": 3214, "USE": "dem", "tile": "4045646"}},
{ "type": "Feature", "properties": { "OBJECTID": 3215, "USE": "dem", "tile": "4045648"}}
]
}

Output of some calculations in Bash - 100% derived from input.json. So this array length == length of the "features[]"-array in the input.json

printf '%s\n' "${pushintojson[@]}"

4045646 NV NV NV
4045648 NV 4045644 NV
4045650 NV 4045646 NV

jq '.features |= map(.properties +=  {NOSW: $ARGS.positional})' --args "${pushintojson[@]}"  <<< "${input.json}"

So this produces this (newlines added by me for readability):

{
"type":"FeatureCollection",
"name":"test",
"features": [
{"type":"Feature","properties":{"OBJECTID":3213,"USE":"dem","tile":"4045644",
"NOSW":[
"4045646 NV NV NV",
"4045648 NV 4045644 NV",
"4045650 NV 4045646 NV"
]}},

{"type":"Feature","properties":{"OBJECTID":3214,"USE":"dem","tile":"4045646",
"NOSW":[
"4045646 NV NV NV",
"4045648 NV 4045644 NV",
"4045650 NV 4045646 NV"
]}},

{"type":"Feature","properties":{"OBJECTID":3215,"USE":"dem","tile":"4045648",
"NOSW":[
"4045646 NV NV NV",
"4045648 NV 4045644 NV",
"4045650 NV 4045646 NV"
]}}
]
}

All values of the bash-array are saved to EVERY element. It's supposed to be just one value per json-object - actually as an array itself.
So in pseudo: "pushintojson[0]" needs to be under "features[0]", pushintojson[1]" to "features[1]"
pushintojson[2]" to "features[2]" etc.

DESIRED OUTPUT

{
"type":"FeatureCollection",
"name":"test",
"features": [
{"type":"Feature","properties":{"OBJECTID":3213,"USE":"dem","tile":"4045644",
"NOSW":"4045646 NV NV NV"
}},
{"type":"Feature","properties":{"OBJECTID":3214,"USE":"dem","tile":"4045646",
"NOSW":"4045648 NV 4045644 NV"
}},
{"type":"Feature","properties":{"OBJECTID":3215,"USE":"dem","tile":"4045648",
"NOSW":"4045650 NV 4045646 NV"
}}
]
}

Body:
If you only use map then for each element you end up iterating over all arguments again. Instead, you have to align the .features array to the $ARGS.positional array.

One way, which is closer to your attempt, is to expand the .features array using to_entries to fit the array's elements  with their keys, which in turn can be used within map to reference by index the corresponding item of the $ARGS.positional array.

jq '
  .features |= (to_entries | map(
    .value * {properties: {NOSW: $ARGS.positional[.key]}}
  ))
' --args "${pushintojson[@]}" < input.json

Another approach would be to use transpose to generate an array of aligned items from both initial arrays, and then again just map their contents together.

jq '
  .features |= ([., $ARGS.positional] | transpose | map(
    .[0] * {properties: {NOSW: .[1]}}
  ))
' --args "${pushintojson[@]}" < input.json

Both produce

{
  "type": "FeatureCollection",
  "name": "test",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "OBJECTID": 3213,
        "USE": "dem",
        "tile": "4045644",
        "NOSW": "4045646 NV NV NV"
      }
    },
    {
      "type": "Feature",
      "properties": {
        "OBJECTID": 3214,
        "USE": "dem",
        "tile": "4045646",
        "NOSW": "4045648 NV 4045644 NV"
      }
    },
    {
      "type": "Feature",
      "properties": {
        "OBJECTID": 3215,
        "USE": "dem",
        "tile": "4045648",
        "NOSW": "4045650 NV 4045646 NV"
      }
    }
  ]
}


问:
Is it possible to customize DryIOC type matching to workaround embedded Interop COM assemblies type mismatch?
We are currently working on a Prism project using DryIoc as the backing container and we are running into problems with the embedding of Interop assemblies for our COM interop.
In the past, when we had issues like this, for instance, with RhinoMocks, we were able to tell RhinoMocks to relax its type matching policy.

Our situation is like this:

```<code>//Main App assembly
IContainer.Register<MyComInterface>();
</code>```
```<code>//Library assembly
IContainer.Resolve<MyComInterface>();
</code>```

This exception is being thrown when trying to create a view model in the Library assembly that uses MyComInterface:

```<code>code: Error.UnableToResolveUnknownService;
message: Unable to resolve resolution root Interop.MyComInterface
  from container without scope
 with Rules with {TrackingDisposableTransients, UseDynamicRegistrationsAsFallbackOnly, FuncAndLazyWithoutRegistration, SelectLastRegisteredFactory} and without {ThrowOnRegisteringDisposableTransient, UseFastExpressionCompilerIfPlatformSupported}
 with FactorySelector=SelectLastRegisteredFactory
 with Made={FactoryMethod=ConstructorWithResolvableArguments}
Where no service registrations found
  and no dynamic registrations found in 1 of Rules.DynamicServiceProviders
  and nothing found in 0 of Rules.UnknownServiceResolvers
</code>```

So the question would be: Is there a way to customize the type matching policy at registration time?

We don't have access to the resolution point since that's inside Prism.
I combed the DryIoc source codes and the core of the issue here seems to be that type matching is done with Type.GetHashCode() which lead to these two types not matching:

```<code>Type 1
{
    Assembly: App
    Name: Interop.MyComInterface
}

Type 2
{
    Assembly: Library
    Name: Interop.MyComInterface
}
</code>```

Thank you

答:
DryIoc uses RuntimeHelpers.GetHashCode(type) for the type lookup. But even if it will provide you the way to override this behaviour, what will you gain?

Because .NET still treats the types from the different assemblies as different and won't allow you to substitute one for another.

Update after comment:

If you see the pros of such addition, please file the issue or start the discussion in the DryIoc github repo.

Body:
We are currently working on a Prism project using DryIoc as the backing container and we are running into problems with the embedding of Interop assemblies for our COM interop.
In the past, when we had issues like this, for instance, with RhinoMocks, we were able to tell RhinoMocks to relax its type matching policy.

Our situation is like this:

//Main App assembly
IContainer.Register<MyComInterface>();

//Library assembly
IContainer.Resolve<MyComInterface>();

This exception is being thrown when trying to create a view model in the Library assembly that uses MyComInterface:

code: Error.UnableToResolveUnknownService;
message: Unable to resolve resolution root Interop.MyComInterface
  from container without scope
 with Rules with {TrackingDisposableTransients, UseDynamicRegistrationsAsFallbackOnly, FuncAndLazyWithoutRegistration, SelectLastRegisteredFactory} and without {ThrowOnRegisteringDisposableTransient, UseFastExpressionCompilerIfPlatformSupported}
 with FactorySelector=SelectLastRegisteredFactory
 with Made={FactoryMethod=ConstructorWithResolvableArguments}
Where no service registrations found
  and no dynamic registrations found in 1 of Rules.DynamicServiceProviders
  and nothing found in 0 of Rules.UnknownServiceResolvers

So the question would be: Is there a way to customize the type matching policy at registration time?

We don't have access to the resolution point since that's inside Prism.
I combed the DryIoc source codes and the core of the issue here seems to be that type matching is done with Type.GetHashCode() which lead to these two types not matching:

Type 1
{
    Assembly: App
    Name: Interop.MyComInterface
}

Type 2
{
    Assembly: Library
    Name: Interop.MyComInterface
}

Thank you


问:
Cost of 3 million Invocations per month on google cloud functions, with 1 minimum instance and 2 as maximum
I am trying to understand cost of google cloud. Suppose I allocate 256 MB for the function and there is 1 minimum instance running all times and maximum instance is set to 2, then what will be monthly cost? I am wondering also if setting minimum instances to 0 will reduce bill significantly or ok to set it as 1 without affecting cost too much. I Could not understand the idle pricing, so If someone has real life example on cost and can share.</p>

答:
Your question lacks important details such as the execution time of your code or the amount of data moving out from your function but you can use the price calculator to get an estimate.

Keep in mind that since the charge for each million of invocations is 0.40 usd the price for the amount of invocations might not be as important as the price for your connectivity.

For example I did an emulation for a function with minimum 1 instance, 3 million of executions a month, 256mb, 100 ms of execution time and 1 mb of bandwidth per execution and for Uscentral1 you will be charged 354.10

In any case it would be better if you discuss this kind of questions with [Google Cloud Billing Support](Contact Cloud Billing Support) as they will be able to give you the most accurate response to this kind of questions.

Keep in mind that Stack Overflow is more focused towards code and development support


问:
Stripping non printable characters from a string in python
I use to run

```<code>$s =~ s/[^[:print:]]//g;
</code>```

on Perl to get rid of non printable characters.

In Python there's no POSIX regex classes, and I can't write [:print:] having it mean what I want. I know of no way in Python to detect if a character is printable or not.

What would you do?

EDIT: It has to support Unicode characters as well. The string.printable way will happily strip them out of the output.
curses.ascii.isprint will return false for any unicode character.

答:
An elegant pythonic solution to stripping 'non printable' characters from a string in python is to use the isprintable() string method together with a generator expression or list comprehension depending on the use case ie. size of the string:

```<code>    ''.join(c for c in my_string if c.isprintable())
</code>```

str.isprintable()
Return True if all characters in the string are printable or the string is empty, False otherwise. Nonprintable characters are those characters defined in the Unicode character database as “Other” or “Separator”, excepting the ASCII space (0x20) which is considered printable. (Note that printable characters in this context are those which should not be escaped when repr() is invoked on a string. It has no bearing on the handling of strings written to sys.stdout or sys.stderr.)

OwnerDisplayName: Vinko Vrsalovic

ContentLicense: CC BY-SA 2.5

Body:
I use to run

$s =~ s/[^[:print:]]//g;

on Perl to get rid of non printable characters.

In Python there's no POSIX regex classes, and I can't write [:print:] having it mean what I want. I know of no way in Python to detect if a character is printable or not.

What would you do?

EDIT: It has to support Unicode characters as well. The string.printable way will happily strip them out of the output.
curses.ascii.isprint will return false for any unicode character.

Body:
An elegant pythonic solution to stripping 'non printable' characters from a string in python is to use the isprintable() string method together with a generator expression or list comprehension depending on the use case ie. size of the string:

    ''.join(c for c in my_string if c.isprintable())

str.isprintable()
Return True if all characters in the string are printable or the string is empty, False otherwise. Nonprintable characters are those characters defined in the Unicode character database as “Other” or “Separator”, excepting the ASCII space (0x20) which is considered printable. (Note that printable characters in this context are those which should not be escaped when repr() is invoked on a string. It has no bearing on the handling of strings written to sys.stdout or sys.stderr.)


问:
How to find pages with tag?

JSONArray pagesArrayFromMethod = new JSONArray();

    private void getListOfChildPagesWithTagAsJSON(Page page) {
        try {
            Iterator<Page> childPages = page.listChildren();
            while (childPages.hasNext()) {
                Page childPage = childPages.next();
                JSONObject pageObject = new JSONObject();
                if (Arrays.asList(childPage.getTags()).contains("customtag")) {   // here is problem
                    pageObject.put(childPage.getTitle(), childPage.getPath());
                    this.pagesArrayFromMethod.put(pageObject);
                }
                Iterator<Page> childPagesOfChildpage = childPage.listChildren();
                while (childPagesOfChildpage.hasNext()) {
                    getListOfChildPagesWithTagAsJSON(childPagesOfChildpage.next());
                }
            }
        } catch (JSONException e) {
            LOG.error(e.getMessage(), e);
        }
    }
```

problem = 'List' may not contain objects of type 'String'!

I have a tag and if the page has this tag it should be put in an array. But when I try to determine if there is a tag on the page, I get an error, I cannot use the method contain with String. I need to convert String into Tag or what method should I use?

答:
The Page APIs getTags() method returns a Tag[] and not a String[] and hence you can't use contains() method directly.

There are a few options though.

Find whether the tag is present by looping through your Tag[]

```<code>Tag[] tags = childPage.getTags();
for (Tag tag : tags) {
    if ("customtag".equals(tag.getTitle())) { //Replace getTitle with getName or getTagID depending on whatever customtag refers to
        pageObject.put(childPage.getTitle(), childPage.getPath());
        this.pagesArrayFromMethod.put(pageObject);
        break;
    }
}
</code>```

Alernatively, you can resolve the actual tag using the TagManager API if you have the tag ID or tag title and then use your existing logic. For e.g.

```<code>Tag tag = tagManager.resolve("customtag"); //or resolveByTitle() if you only know the title of the tag.

if (tag !=null && Arrays.asList(childPage.getTags()).contains(tag)) {
    pageObject.put(childPage.getTitle(), childPage.getPath());
    this.pagesArrayFromMethod.put(pageObject);
}
</code>```

Please note that you need to inject the TagManager to your method to do this. You can get the TagManager from the resource resolver as shown below

```<code>TagManager tagManager = resourceResolver.adaptTo(TagManager.class);
</code>```

You can also find all content tagged with a particular tag or a set of tags using the TagManager API. However, please note that this would return all the nodes which contains a cq:tags property and not just the jcr:content node. Hence, you may need to further filter from the results.

Title: How to find pages with tag?

Body:
JSONArray pagesArrayFromMethod = new JSONArray();

    private void getListOfChildPagesWithTagAsJSON(Page page) {
        try {
            Iterator<Page> childPages = page.listChildren();
            while (childPages.hasNext()) {
                Page childPage = childPages.next();
                JSONObject pageObject = new JSONObject();
                if (Arrays.asList(childPage.getTags()).contains("customtag")) {   // here is problem
                    pageObject.put(childPage.getTitle(), childPage.getPath());
                    this.pagesArrayFromMethod.put(pageObject);
                }
                Iterator<Page> childPagesOfChildpage = childPage.listChildren();
                while (childPagesOfChildpage.hasNext()) {
                    getListOfChildPagesWithTagAsJSON(childPagesOfChildpage.next());
                }
            }
        } catch (JSONException e) {
            LOG.error(e.getMessage(), e);
        }
    }

problem = 'List' may not contain objects of type 'String'!

I have a tag and if the page has this tag it should be put in an array. But when I try to determine if there is a tag on the page, I get an error, I cannot use the method contain with String. I need to convert String into Tag or what method should I use?

Body:
The Page APIs getTags() method returns a Tag[] and not a String[] and hence you can't use contains() method directly.

There are a few options though.

Find whether the tag is present by looping through your Tag[]

Tag[] tags = childPage.getTags();
for (Tag tag : tags) {
    if ("customtag".equals(tag.getTitle())) { //Replace getTitle with getName or getTagID depending on whatever customtag refers to
        pageObject.put(childPage.getTitle(), childPage.getPath());
        this.pagesArrayFromMethod.put(pageObject);
        break;
    }
}

Alernatively, you can resolve the actual tag using the TagManager API if you have the tag ID or tag title and then use your existing logic. For e.g.

Tag tag = tagManager.resolve("customtag"); //or resolveByTitle() if you only know the title of the tag.

if (tag !=null && Arrays.asList(childPage.getTags()).contains(tag)) {
    pageObject.put(childPage.getTitle(), childPage.getPath());
    this.pagesArrayFromMethod.put(pageObject);
}

Please note that you need to inject the TagManager to your method to do this. You can get the TagManager from the resource resolver as shown below

TagManager tagManager = resourceResolver.adaptTo(TagManager.class);

You can also find all content tagged with a particular tag or a set of tags using the TagManager API. However, please note that this would return all the nodes which contains a cq:tags property and not just the jcr:content node. Hence, you may need to further filter from the results.


问:
How to make one method return different values for user with role ADMIN and USER?
I need to make the @Get method, and for users with ROLE_USER it should return other values than for users with ROLE_ADMIN.

The URL have to be the same.

How to do it with Spring Security?

答: Question is not 100% clear, but what I would do is make an Enum field in the user class (assuming that class exists) for the role and return the value of that field when the URL is called.</p>


问:
Cannot reach quarkus application on host from inside docker container
This is not a duplicate of From inside of a Docker container, how do I connect to the localhost of the machine?. I have already attempted the suggestions in the aforementioned thread, and I know how to fix this problem in general. This problem seems to be specific to quarkus applications. I manage to make everything work except for the quarkus server.

Context

I have obtained a quarkus application template from https://code.quarkus.io/, of which includes a shell file called mvnw, which I can use to start a development server by executing ./mvnw quarkus:dev -Dquarkus.http.host=0.0.0.0 -Dquarkus.http.port=9000 -Ddebug=9001 in the host terminal.

I can now go to http://localhost:9000 in the browser. I can also do curl localhost:9000 and see that I get the HTML/CSS/JS code.

Issue

I cannot reach the quarkus server from within a docker container. I have tried:

Run a base curl container: docker run --rm -it --name curl --entrypoint sh curlimages/curl:latest

From within the container I attempt the following:

curl host.docker.internal:9000 -> Connection refused

curl localhost:9000 -> Connection refused

curl (the machine host name here):9000 -> Operation timed out (this happens for all ports)

Run the container with --network=host, and use curl localhost:9000 -> Connection refused

Various other suggestions from the post mentioned at the top, but all get connection refused

However, I can connect to the JVM debug port using curl host.docker.internal:9001 (Empty reply from server), and I can also connect to a Python http fileserver (hosted from host) using the host.docker.internal hostname. It seems to be that it is only the quarkus server that I cannot connect to.

What I want: A reliable way to connect to the host's quarkus container from within a docker container

System info

OS: Windows 11 21H2 22000.493 , but I do everything within WSL2 (Ubuntu 20.04.3 LTS).

CPU: Intel Core i7-10850H

答:
Could you try to disable the IPV6 on your wsl2 and repeat your test?

```<code>$ sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
$ sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
</code>```

I had similar issues and this solved the problem for me.

Body:
Could you try to disable the IPV6 on your wsl2 and repeat your test?

$ sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
$ sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1

I had similar issues and this solved the problem for me.


问:
selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: invalid locator error using find_element('username') Selenium Python
I am running python 3.9 with the following code. When I run the script I get the error message.  Not sure what I am missing.  The element is called username.

```<code>  File "/Users/user/Documents/PycharmProjects/webscrapping/app/webscraping.py", line 19, in <module>
    login = driver.find_element("username")   File "/Users/user/Documents/VENV/webscrapping/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 1244, in find_element
    return self.execute(Command.FIND_ELEMENT, {   File "/Users/user/Documents/VENV/webscrapping/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 424, in execute
    self.error_handler.check_response(response)
    File "/Users/user/Documents/VENV/webscrapping/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
    raise exception_class(message, screen, stacktrace) selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: invalid locator   (Session info: chrome=98.0.4758.80) Stacktrace: 0   chromedriver                        0x0000000102a5c3c9 chromedriver + 5018569 1   chromedriver                        0x00000001029e7333 chromedriver + 4539187 2   chromedriver             0x00000001025bca88 chromedriver + 170632 3   chromedriver              0x00000001025f0d81 chromedriver + 384385 4   chromedriver              0x00000001025f14f1 chromedriver + 386289 5   chromedriver              0x00000001026238b4 chromedriver + 592052 6   chromedriver              0x000000010260e80d chromedriver + 505869 7   chromedriver              0x0000000102621604 chromedriver + 583172 8   chromedriver              0x000000010260e6d3 chromedriver + 505555 9   chromedriver              0x00000001025e495e chromedriver + 334174 10  chromedriver              0x00000001025e5935 chromedriver + 338229 11  chromedriver              0x0000000102a181ee chromedriver + 4739566 12  chromedriver             0x0000000102a31f51 chromedriver + 4845393 13  chromedriver             0x0000000102a37928 chromedriver + 4868392 14  chromedriver             0x0000000102a32a7a chromedriver + 4848250 15  chromedriver             0x0000000102a0cc31 chromedriver + 4693041 16  chromedriver             0x0000000102a4d978 chromedriver + 4958584 17  chromedriver             0x0000000102a4db01 chromedriver + 4958977 18  chromedriver             0x0000000102a63795 chromedriver + 5048213 19  libsystem_pthread.dylib  0x00007ff80f7a24f4 _pthread_start + 125 20  libsystem_pthread.dylib    0x00007ff80f79e00f thread_start + 15
</code>```

Code trials:

```<code>import os
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

CURRENT_DIR = os.getcwd()
print(f"Current Dir is {CURRENT_DIR}")

driver = webdriver.Chrome(f"{CURRENT_DIR}/chromedriver")

url = 'https://somewebsite.com/login2/'
driver.get(url)

time.sleep(2)
login = driver.find_element('username')

login.send_keys('myusername@somewebsite.com')
login.send_keys(Keys.RETURN)

time.sleep(5)
driver.quit()
</code>```

HTML im trying to get:

```<code><div class="sc-dnqmqq jomEGJ" data-reactid=".0.1.0.2.1.0.0.0">
  <label class="sc-iwsKbI hOobUj" for="username" data-reactid=".0.1.0.2.1.0.0.0.0">
    <span data-reactid=".0.1.0.2.1.0.0.0.0.0">Username</span>
  </label>
</div>
</code>```

答:
find_element()

find_element() finds an element given a By strategy and a locator where both the arguments are mandatory.

So you have to pass the By class which is a set of supported locator strategies along with the locator.

Solution

Effectively your line of code will be:

If username is the value of class attribute:

```<code>login = driver.find_element(By.CLASS_NAME, "username")
</code>```

If username is the value of id attribute:

```<code>login = driver.find_element(By.ID, "username")
</code>```

If username is the value of name attribute:

```<code>login = driver.find_element(By.NAME, "username")
</code>```

If username is the value of linktext attribute:

```<code>login = driver.find_element(By.LINK_TEXT, "username")
</code>```

Body:
I am running python 3.9 with the following code. When I run the script I get the error message.  Not sure what I am missing.  The element is called username.

  File "/Users/user/Documents/PycharmProjects/webscrapping/app/webscraping.py", line 19, in <module>
    login = driver.find_element("username")   File "/Users/user/Documents/VENV/webscrapping/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 1244, in find_element
    return self.execute(Command.FIND_ELEMENT, {   File "/Users/user/Documents/VENV/webscrapping/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 424, in execute
    self.error_handler.check_response(response)
    File "/Users/user/Documents/VENV/webscrapping/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
    raise exception_class(message, screen, stacktrace) selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: invalid locator   (Session info: chrome=98.0.4758.80) Stacktrace: 0   chromedriver                        0x0000000102a5c3c9 chromedriver + 5018569 1   chromedriver                        0x00000001029e7333 chromedriver + 4539187 2   chromedriver             0x00000001025bca88 chromedriver + 170632 3   chromedriver              0x00000001025f0d81 chromedriver + 384385 4   chromedriver              0x00000001025f14f1 chromedriver + 386289 5   chromedriver              0x00000001026238b4 chromedriver + 592052 6   chromedriver              0x000000010260e80d chromedriver + 505869 7   chromedriver              0x0000000102621604 chromedriver + 583172 8   chromedriver              0x000000010260e6d3 chromedriver + 505555 9   chromedriver              0x00000001025e495e chromedriver + 334174 10  chromedriver              0x00000001025e5935 chromedriver + 338229 11  chromedriver              0x0000000102a181ee chromedriver + 4739566 12  chromedriver             0x0000000102a31f51 chromedriver + 4845393 13  chromedriver             0x0000000102a37928 chromedriver + 4868392 14  chromedriver             0x0000000102a32a7a chromedriver + 4848250 15  chromedriver             0x0000000102a0cc31 chromedriver + 4693041 16  chromedriver             0x0000000102a4d978 chromedriver + 4958584 17  chromedriver             0x0000000102a4db01 chromedriver + 4958977 18  chromedriver             0x0000000102a63795 chromedriver + 5048213 19  libsystem_pthread.dylib  0x00007ff80f7a24f4 _pthread_start + 125 20  libsystem_pthread.dylib    0x00007ff80f79e00f thread_start + 15

Code trials:

import os
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

CURRENT_DIR = os.getcwd()
print(f"Current Dir is {CURRENT_DIR}")

driver = webdriver.Chrome(f"{CURRENT_DIR}/chromedriver")

url = 'https://somewebsite.com/login2/'
driver.get(url)

time.sleep(2)
login = driver.find_element('username')

login.send_keys('myusername@somewebsite.com')
login.send_keys(Keys.RETURN)

time.sleep(5)
driver.quit()

HTML im trying to get:

    Username

Body:
find_element()

find_element() finds an element given a By strategy and a locator where both the arguments are mandatory.

So you have to pass the By class which is a set of supported locator strategies along with the locator.

Solution

Effectively your line of code will be:

If username is the value of class attribute:

login = driver.find_element(By.CLASS_NAME, "username")

If username is the value of id attribute:

login = driver.find_element(By.ID, "username")

If username is the value of name attribute:

login = driver.find_element(By.NAME, "username")

If username is the value of linktext attribute:

login = driver.find_element(By.LINK_TEXT, "username")


答:
ggplot expects a data frame, so you could try this ...

library(ggplot2)

data.frame(die_sums = rolldie(1000, 10))|>
  ggplot()+
  geom_histogram(aes(die_sums), binwidth = 1)
```

Created on 2022-02-13 by the reprex package (v2.0.1)

Title: Creating GGPLOT Histogram

Body:
I have the following function, 'n' rolls of 'k' dice, in which the result is the sum of 'k' rolls.

rolldie = function(n, k){
  # Your code here
  die=sample(1:6, size = k, replace = TRUE)
  replicate(n, sum(sample(c(1:6),k, replace=TRUE)))
}
rolldie(5, 4)

I then have to create a ggplot histogram to show the results of the function, using n=1000 with whatever values of k. That is where I am stuck, I am not sure how to create this histogram for the results of rolldie.

Body:
ggplot expects a data frame, so you could try this ...

library(ggplot2)

data.frame(die_sums = rolldie(1000, 10))|>
  ggplot()+
  geom_histogram(aes(die_sums), binwidth = 1)

Created on 2022-02-13 by the reprex package (v2.0.1)


答:
Here is a solution with a simplified rolldie function.

Roll the dies, table the results, coerce the table to data.frame and plot.

library(ggplot2)

rolldie <- function(n, k){
  replicate(n, sum(sample(6, k, replace=TRUE)))
}

set.seed(2022)
n <- 1000
Points <- rolldie(n, 4)

Points <- as.data.frame(table(Points))
ggplot(Points, aes(Points, Freq)) +
  geom_col()
```

Created on 2022-02-13 by the reprex package (v2.0.1)

Title: Creating GGPLOT Histogram

Body:
Here is a solution with a simplified rolldie function.

Roll the dies, table the results, coerce the table to data.frame and plot.

library(ggplot2)

rolldie <- function(n, k){
  replicate(n, sum(sample(6, k, replace=TRUE)))
}

set.seed(2022)
n <- 1000
Points <- rolldie(n, 4)

Points <- as.data.frame(table(Points))
ggplot(Points, aes(Points, Freq)) +
  geom_col()

Created on 2022-02-13 by the reprex package (v2.0.1)


问:
"Trying to load a data file" and object of type 'closure' is not subsettable
[enter image description here][1]

I am trying to upload a file and see the statistics, but somehow cannot
Please help me

[enter image description here][2]

[enter image description here][3]

答:
Ok, here it goes. You have NA in your dataset. look at this solution:

```<code>cor(urate, urate_1, use="complete.obs")

</code>```

This will only take observation with numbers and skip NAs.

Body:
Ok, here it goes. You have NA in your dataset. look at this solution:

cor(urate, urate_1, use="complete.obs")

This will only take observation with numbers and skip NAs.


问:
Flutter update UI on button press
If I remove the awaits from my viewModel.sendMessage() it works as expected, however I can't set the state with the value returned. Is there a way to update the UI from setState?

I'm just not sure how to do both at the same time, the parent class for this widget is a StatefulWidget, Idk if that matters. My ConversationViewModel is just a changeNotifier.

```<code>@override
Widget build(BuildContext context) {
  UsViewModel viewModel = Provider.of<UsViewModel>(context, listen: false);
  viewModel.setUs();
  var us = Provider.of<UsViewModel>(context, listen: true).us;
  return Consumer<ConversationViewModel>(builder: (BuildContext context, viewModel, Widget child) {
    return Scaffold(
      key: viewModel.scaffoldKey,
      appBar: AppBar(
        leading: GestureDetector(
          onTap: () {
            Navigator.pop(context);
          },
          child: Icon(
            Icons.keyboard_backspace,
          ),
        ),
        elevation: 0.0,
        titleSpacing: 0,
        title: buildName(),
      ),
      body: Container(
        height: MediaQuery.of(context).size.height,
        child: Column(
          children: [
            Flexible(
              child: StreamBuilder(
                stream: messageListStream(widget.chatId),
                builder: (context, snapshot) {
                  if (snapshot.hasData) {
                    List messages = snapshot.data.docs;
                    return ListView.builder(
                      controller: scrollController,
                      padding: EdgeInsets.symmetric(horizontal: 10.0),
                      itemCount: messages.length,
                      reverse: true,
                      itemBuilder: (BuildContext context, int index) {
                        Message message = Message.fromJson(
                            messages.reversed.toList()[index].data());
                        return ChatBubble(
                            message: '${message.content}',
                            time: message?.time,
                            isMe: message?.senderUid == us?.uid,
                            type: message?.type);
                      },
                    );
                  } else {
                    return Center(child: circularProgress(context));
                  }
                },
              ),
            ),
            Align(
              alignment: Alignment.bottomCenter,
              child: BottomAppBar(
                elevation: 10.0,
                child: Container(
                  constraints: BoxConstraints(maxHeight: 100.0),
                  child: Row(
                    crossAxisAlignment: CrossAxisAlignment.end,
                    children: [
                      Flexible(
                        child: TextField(
                          controller: messageController,
                          focusNode: focusNode,
                          style: TextStyle(
                            fontSize: 15.0,
                            color:
                                Theme.of(context).textTheme.headline6.color,
                          ),
                          decoration: InputDecoration(
                            contentPadding: EdgeInsets.all(10.0),
                            enabledBorder: InputBorder.none,
                            border: InputBorder.none,
                            hintText: "Type your message",
                            hintStyle: TextStyle(
                              color:
                                  Theme.of(context).textTheme.headline6.color,
                            ),
                          ),
                          maxLines: null,
                        ),
                      ),
                      IconButton(
                        icon: Icon(
                          Feather.send,
                          color: Theme.of(context).colorScheme.secondary,
                        ),
                        onPressed: () {
                          if (messageController.text.trim().isNotEmpty) {
                            sendMessage(viewModel, us);
                          }
                        },
                      ),
                    ],
                  ),
                ),
              ),
            )
          ],
        ),
      ),
    );
  });
}

sendMessage(ConversationViewModel viewModel, var us,
    {bool isImage = false, int imageType}) async {
  String msg;

  msg = messageController.text.trim();
  messageController.clear();

  Message message = Message(
    content: '$msg',
    senderUid: us?.uid,
    time: Timestamp.now(),
  );

  if (msg.isNotEmpty) {
    String id = await viewModel.sendMessage(widget.usId, message);
    setState(() {
      chatId = id;
    });
  }
}
</code>```

答:
If you using ChangeNotifier with Provider package you don't need setState method. You should read Provider documentation: https://pub.dev/packages/provider

ft: That you create listeners for each item in this line.

isMe: message?.senderUid == context.watch<UsViewModel>.us ?.uid,
```
<p>How to use a provider according to your question, added an example;</p>
<pre class="lang-dart prettyprint-override"><code>@override
Widget build(BuildContext context) {
  context.read<UsViewModel>().setUs();
  return Scaffold(
      key: context.read<UsViewModel>().scaffoldKey,
      appBar: AppBar(
        leading: GestureDetector(
          onTap: () {
            Navigator.pop(context);
          },
          child: Icon(
            Icons.keyboard_backspace,
          ),
        ),
        elevation: 0.0,
        titleSpacing: 0,
        title: buildName(),
      ),
      body: Container(
        height: MediaQuery.of(context).size.height,
        child: Column(
          children: [
            Flexible(
              child: StreamBuilder(
                stream: messageListStream(widget.chatId),
                builder: (context, snapshot) {
                  if (snapshot.hasData) {
                    List messages = snapshot.data.docs;
                    return ListView.builder(
                      controller: scrollController,
                      padding: EdgeInsets.symmetric(horizontal: 10.0),
                      itemCount: messages.length,
                      reverse: true,
                      itemBuilder: (BuildContext context, int index) {
                        Message message = Message.fromJson(
                            messages.reversed.toList()[index].data());
                        return ChatBubble(
                            message: '${message.content}',
                            time: message?.time,
                            isMe: message?.senderUid == context.watch<UsViewModel>.us ?.uid,
                            type: message?.type);
                      },
                    );
                  } else {
                    return Center(child: circularProgress(context));
                  }
                },
              ),
            ),
            Align(
              alignment: Alignment.bottomCenter,
              child: BottomAppBar(
                elevation: 10.0,
                child: Container(
                  constraints: BoxConstraints(maxHeight: 100.0),
                  child: Row(
                    crossAxisAlignment: CrossAxisAlignment.end,
                    children: [
                      Flexible(
                        child: TextField(
                          controller: messageController,
                          focusNode: focusNode,
                          style: TextStyle(
                            fontSize: 15.0,
                            color:
                                Theme.of(context).textTheme.headline6.color,
                          ),
                          decoration: InputDecoration(
                            contentPadding: EdgeInsets.all(10.0),
                            enabledBorder: InputBorder.none,
                            border: InputBorder.none,
                            hintText: "Type your message",
                            hintStyle: TextStyle(
                              color:
                                  Theme.of(context).textTheme.headline6.color,
                            ),
                          ),
                          maxLines: null,
                        ),
                      ),
                      IconButton(
                        icon: Icon(
                          Feather.send,
                          color: Theme.of(context).colorScheme.secondary,
                        ),
                        onPressed: () {
                          if (messageController.text.trim().isNotEmpty) {
                            sendMessage(context);
                          }
                        },
                      ),
                    ],
                  ),
                ),
              ),
            )
          ],
        ),
      ),
    );
}

sendMessage(BuildContext context) async {
  String msg;

  msg = messageController.text.trim();
  messageController.clear();

  Message message = Message(
    content: '$msg',
    senderUid: context.read<UsViewModel>().us?.uid,
    time: Timestamp.now(),
  );

  if (msg.isNotEmpty) {
    String id = await context.read<UsViewModel>().sendMessage(widget.usId, message);
setState((){
chatId = id;
});
  }
}
</code>```

Title: Flutter update UI on button press

Body:
If I remove the awaits from my viewModel.sendMessage() it works as expected, however I can't set the state with the value returned. Is there a way to update the UI from setState?

I'm just not sure how to do both at the same time, the parent class for this widget is a StatefulWidget, Idk if that matters. My ConversationViewModel is just a changeNotifier.

@override
Widget build(BuildContext context) {
  UsViewModel viewModel = Provider.of<UsViewModel>(context, listen: false);
  viewModel.setUs();
  var us = Provider.of<UsViewModel>(context, listen: true).us;
  return Consumer<ConversationViewModel>(builder: (BuildContext context, viewModel, Widget child) {
    return Scaffold(
      key: viewModel.scaffoldKey,
      appBar: AppBar(
        leading: GestureDetector(
          onTap: () {
            Navigator.pop(context);
          },
          child: Icon(
            Icons.keyboard_backspace,
          ),
        ),
        elevation: 0.0,
        titleSpacing: 0,
        title: buildName(),
      ),
      body: Container(
        height: MediaQuery.of(context).size.height,
        child: Column(
          children: [
            Flexible(
              child: StreamBuilder(
                stream: messageListStream(widget.chatId),
                builder: (context, snapshot) {
                  if (snapshot.hasData) {
                    List messages = snapshot.data.docs;
                    return ListView.builder(
                      controller: scrollController,
                      padding: EdgeInsets.symmetric(horizontal: 10.0),
                      itemCount: messages.length,
                      reverse: true,
                      itemBuilder: (BuildContext context, int index) {
                        Message message = Message.fromJson(
                            messages.reversed.toList()[index].data());
                        return ChatBubble(
                            message: '${message.content}',
                            time: message?.time,
                            isMe: message?.senderUid == us?.uid,
                            type: message?.type);
                      },
                    );
                  } else {
                    return Center(child: circularProgress(context));
                  }
                },
              ),
            ),
            Align(
              alignment: Alignment.bottomCenter,
              child: BottomAppBar(
                elevation: 10.0,
                child: Container(
                  constraints: BoxConstraints(maxHeight: 100.0),
                  child: Row(
                    crossAxisAlignment: CrossAxisAlignment.end,
                    children: [
                      Flexible(
                        child: TextField(
                          controller: messageController,
                          focusNode: focusNode,
                          style: TextStyle(
                            fontSize: 15.0,
                            color:
                                Theme.of(context).textTheme.headline6.color,
                          ),
                          decoration: InputDecoration(
                            contentPadding: EdgeInsets.all(10.0),
                            enabledBorder: InputBorder.none,
                            border: InputBorder.none,
                            hintText: "Type your message",
                            hintStyle: TextStyle(
                              color:
                                  Theme.of(context).textTheme.headline6.color,
                            ),
                          ),
                          maxLines: null,
                        ),
                      ),
                      IconButton(
                        icon: Icon(
                          Feather.send,
                          color: Theme.of(context).colorScheme.secondary,
                        ),
                        onPressed: () {
                          if (messageController.text.trim().isNotEmpty) {
                            sendMessage(viewModel, us);
                          }
                        },
                      ),
                    ],
                  ),
                ),
              ),
            )
          ],
        ),
      ),
    );
  });
}

sendMessage(ConversationViewModel viewModel, var us,
    {bool isImage = false, int imageType}) async {
  String msg;

  msg = messageController.text.trim();
  messageController.clear();

  Message message = Message(
    content: '$msg',
    senderUid: us?.uid,
    time: Timestamp.now(),
  );

  if (msg.isNotEmpty) {
    String id = await viewModel.sendMessage(widget.usId, message);
    setState(() {
      chatId = id;
    });
  }
}

Body:
If you using ChangeNotifier with Provider package you don't need setState method. You should read Provider documentation: https://pub.dev/packages/provider

ft: That you create listeners for each item in this line.

isMe: message?.senderUid == context.watch<UsViewModel>.us ?.uid,

How to use a provider according to your question, added an example;

@override
Widget build(BuildContext context) {
  context.read<UsViewModel>().setUs();
  return Scaffold(
      key: context.read<UsViewModel>().scaffoldKey,
      appBar: AppBar(
        leading: GestureDetector(
          onTap: () {
            Navigator.pop(context);
          },
          child: Icon(
            Icons.keyboard_backspace,
          ),
        ),
        elevation: 0.0,
        titleSpacing: 0,
        title: buildName(),
      ),
      body: Container(
        height: MediaQuery.of(context).size.height,
        child: Column(
          children: [
            Flexible(
              child: StreamBuilder(
                stream: messageListStream(widget.chatId),
                builder: (context, snapshot) {
                  if (snapshot.hasData) {
                    List messages = snapshot.data.docs;
                    return ListView.builder(
                      controller: scrollController,
                      padding: EdgeInsets.symmetric(horizontal: 10.0),
                      itemCount: messages.length,
                      reverse: true,
                      itemBuilder: (BuildContext context, int index) {
                        Message message = Message.fromJson(
                            messages.reversed.toList()[index].data());
                        return ChatBubble(
                            message: '${message.content}',
                            time: message?.time,
                            isMe: message?.senderUid == context.watch<UsViewModel>.us ?.uid,
                            type: message?.type);
                      },
                    );
                  } else {
                    return Center(child: circularProgress(context));
                  }
                },
              ),
            ),
            Align(
              alignment: Alignment.bottomCenter,
              child: BottomAppBar(
                elevation: 10.0,
                child: Container(
                  constraints: BoxConstraints(maxHeight: 100.0),
                  child: Row(
                    crossAxisAlignment: CrossAxisAlignment.end,
                    children: [
                      Flexible(
                        child: TextField(
                          controller: messageController,
                          focusNode: focusNode,
                          style: TextStyle(
                            fontSize: 15.0,
                            color:
                                Theme.of(context).textTheme.headline6.color,
                          ),
                          decoration: InputDecoration(
                            contentPadding: EdgeInsets.all(10.0),
                            enabledBorder: InputBorder.none,
                            border: InputBorder.none,
                            hintText: "Type your message",
                            hintStyle: TextStyle(
                              color:
                                  Theme.of(context).textTheme.headline6.color,
                            ),
                          ),
                          maxLines: null,
                        ),
                      ),
                      IconButton(
                        icon: Icon(
                          Feather.send,
                          color: Theme.of(context).colorScheme.secondary,
                        ),
                        onPressed: () {
                          if (messageController.text.trim().isNotEmpty) {
                            sendMessage(context);
                          }
                        },
                      ),
                    ],
                  ),
                ),
              ),
            )
          ],
        ),
      ),
    );
}

sendMessage(BuildContext context) async {
  String msg;

  msg = messageController.text.trim();
  messageController.clear();

  Message message = Message(
    content: '$msg',
    senderUid: context.read<UsViewModel>().us?.uid,
    time: Timestamp.now(),
  );

  if (msg.isNotEmpty) {
    String id = await context.read<UsViewModel>().sendMessage(widget.usId, message);
setState((){
chatId = id;
});
  }
}


问:
Are there good reasons to wrap a single property in a #region in c#?
I recently inherited some C# code where nearly every item in a file was wrapped in an individual #region/#endregion block -- every class, every function, every property, every enum, but not the fields.  Each of these was in turn wrapped in a "grouping" #region (e.g. "Properties", "Constructor", "Methods", etc.).  There are multiple overloads of a single function with different argument lists, but they're each wrapped in individual regions with the same name, rather than a single region for all three overloads.  The person who wrote this code is no longer with the company, and from the history in source control, it appears that these were present in the initial submission, and the practice continues through successive versions of the code as new properties and methods are added.

Any idea why this might be done?  Some thoughts:

An overzealous VS feature (or code-cleanup tool) automatically inserted the #region/#endregion blocks, and the author didn't remove them.

There's an IDE that folds regions but not functions, so this was necessary to get syntax folding.

This is a method of stubbing out the structure of your code prior to implementing it.

EDIT: I chose Jonathan's answer because it contributed a new reason why someone might have chosen to do this.  Thanks for the discussion!

答: Sounds like a horrible usage to me.  My only thought is someone thought this was a great way to document code.  I would look to remove this over utilization of regions.  </p>


问:
How do I clear an Image's EXIF data with ImageSharp?
I found a GitHub issue showing how to remove an Image's exif data by setting its ExifProfile to null:

```<code> SixLabors.ImageSharp.Image image = Image.Load(imagePath);

 //remove exif
 image.Metadata.ExifProfile = null;

 //resize
 image.Mutate(x => x.Resize(width, height));

 //save
 SixLabors.ImageSharp.Formats.Jpeg.JpegEncoder encoder = new SixLabors.ImageSharp.Formats.Jpeg.JpegEncoder();

 encoder.Quality = 30;

 image.Save(thumbnailPath, encoder);
</code>```

...but it doesn't seem to work for me -- the saved jpeg's are the same size, and my when inspected by my OS they show me all the camera's EXIF settings. When I do this same inspection on images created from another utility the OS does not show me all the EXIF settings...so I'm inclined to say this ImageSharp technique is not scrubbing them correctly.

Any idea?

https://github.com/SixLabors/ImageSharp/issues/400

答:
Turns out there are two different types of metadata - EXIF and XMP. It is necessary to set both objects to null to remove them all:</p>
```<code>image.Metadata.ExifProfile = null;
image.Metadata.XmpProfile = null;
</code>```

Body:
I found a GitHub issue showing how to remove an Image's exif data by setting its ExifProfile to null:

 SixLabors.ImageSharp.Image image = Image.Load(imagePath);

 //remove exif
 image.Metadata.ExifProfile = null;

 //resize
 image.Mutate(x => x.Resize(width, height));

 //save
 SixLabors.ImageSharp.Formats.Jpeg.JpegEncoder encoder = new SixLabors.ImageSharp.Formats.Jpeg.JpegEncoder();

 encoder.Quality = 30;

 image.Save(thumbnailPath, encoder);

...but it doesn't seem to work for me -- the saved jpeg's are the same size, and my when inspected by my OS they show me all the camera's EXIF settings. When I do this same inspection on images created from another utility the OS does not show me all the EXIF settings...so I'm inclined to say this ImageSharp technique is not scrubbing them correctly.

Any idea?

https://github.com/SixLabors/ImageSharp/issues/400

Body:
Turns out there are two different types of metadata - EXIF and XMP. It is necessary to set both objects to null to remove them all:

image.Metadata.ExifProfile = null;
image.Metadata.XmpProfile = null;