问:
Get UID and token obtain from password reset in react form
I am working on password reset and I have used django rest-auth, I have successfully got the token and uid from email link by hitting rest-auth/password/reset/,

but for  to confirm I want the token and uid in react form so I can change it how can I get the uid and token which the rest auth return in email link in react js form

```<code>           axios.post('http://127.0.0.1:8000/rest-auth/password/reset/',payload)
        .then(res=>{
            console.log(res)
        })
        .catch(err=>{
            console.log(err)
        })
</code>```

its working perfect and it returns me:

```<code>    http://127.0.0.1:8000/rest-auth/password/reset/confirm/MQ/594-5faaa46be4277e6a1879/
</code>```

how can I get the uid and token from url in react form?

答:
You should configure your react router to get the params from url. </p>

```<code>  e.preventDefault();
        fetch("http://127.0.0.1:8000/rest-auth/password/reset/confirm/", {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json'
            }, body: JSON.stringify({
                "new_password1": this.state.new_password1,
                "new_password2": this.state.new_password2,
                "uid": this.props.match.params.uid,
                "token": this.props.match.params.token,
            })
        })
            // .then(response => response.json())
            .then(response => {
                console.log(response)
            })
            .catch(error => {
                console.log(error)
            })
</code>```

Body:
I am working on password reset and I have used django rest-auth, I have successfully got the token and uid from email link by hitting rest-auth/password/reset/,

but for  to confirm I want the token and uid in react form so I can change it how can I get the uid and token which the rest auth return in email link in react js form

           axios.post('http://127.0.0.1:8000/rest-auth/password/reset/',payload)
        .then(res=>{
            console.log(res)
        })
        .catch(err=>{
            console.log(err)
        })

its working perfect and it returns me:

    http://127.0.0.1:8000/rest-auth/password/reset/confirm/MQ/594-5faaa46be4277e6a1879/

how can I get the uid and token from url in react form?

Body:
You should configure your react router to get the params from url.

  e.preventDefault();
        fetch("http://127.0.0.1:8000/rest-auth/password/reset/confirm/", {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json'
            }, body: JSON.stringify({
                "new_password1": this.state.new_password1,
                "new_password2": this.state.new_password2,
                "uid": this.props.match.params.uid,
                "token": this.props.match.params.token,
            })
        })
            // .then(response => response.json())
            .then(response => {
                console.log(response)
            })
            .catch(error => {
                console.log(error)
            })


来源: stackexchange

问:
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: Service Unavailable. IN DOCKER , MAC
I am having this issue

```<code>system3:postgres saurabh-gupta2$ docker build -t postgres .
Sending build context to Docker daemon  38.91kB
Step 1/51 : FROM registry.access.redhat.com/rhel7/rhel
Get https://registry.access.redhat.com/v2/: Service Unavailable
</code>```

```<code>docker run  -t apline
Unable to find image 'apline:latest' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: Service Unavailable.
See 'docker run --help'.
</code>```

I have looked for a solution that says to set proxy, but I have set the proxy for the wifi.

https://docs.docker.com/docker-for-mac/networking/#httphttps-proxy-support

Still, it is not working.

I have set proxy for docker too. It is not working.

in Preference -> proxies

Docker version 17.12 ce

I also want to know if the proxy is the issue then how can I check it is set, what is work around for this?

答: In my case, stopping Proxifier fixed it. I added a rule to route any connections from vpnkit.exe as Direct and it now works.

ContentLicense: CC BY-SA 4.0

Body:
I am having this issue

system3:postgres saurabh-gupta2$ docker build -t postgres .
Sending build context to Docker daemon  38.91kB
Step 1/51 : FROM registry.access.redhat.com/rhel7/rhel
Get https://registry.access.redhat.com/v2/: Service Unavailable

docker run  -t apline
Unable to find image 'apline:latest' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: Service Unavailable.
See 'docker run --help'.

I have looked for a solution that says to set proxy, but I have set the proxy for the wifi.

https://docs.docker.com/docker-for-mac/networking/#httphttps-proxy-support

Still, it is not working.

I have set proxy for docker too. It is not working.

in Preference -> proxies

Docker version 17.12 ce

I also want to know if the proxy is the issue then how can I check it is set, what is work around for this?


问:
How to solve date format problem when using Window XP in C?
I am using Window 7, when I run my code in local everything is alright, but when upload to Window server 2003 the code will coming error:

"String was not recognized as a valid DateTime"

My computer setting (date format) all same with my window server. Still come out the same problem.

below is my code:

```<code>dateFrom = Convert.ToDateTime("12-6-2011");

dateTo = Convert.ToDateTime("18-6-2011");

//or
dateFrom = DateTime.ParseExact("12-6-2011", "MM/dd/yyyy", CultureInfo.InvariantCulture);

dateTo = DateTime.ParseExact("18-6-2011", "MM/dd/yyyy", CultureInfo.InvariantCulture);

//i using parse also din come out.
</code>```

答: MM stands for month. You shoud use dd/MM/yyyy to fit your data.</p>

Body:
I am using Window 7, when I run my code in local everything is alright, but when upload to Window server 2003 the code will coming error:

"String was not recognized as a valid DateTime"

My computer setting (date format) all same with my window server. Still come out the same problem.

below is my code:

dateFrom = Convert.ToDateTime("12-6-2011");

dateTo = Convert.ToDateTime("18-6-2011");

//or
dateFrom = DateTime.ParseExact("12-6-2011", "MM/dd/yyyy", CultureInfo.InvariantCulture);

dateTo = DateTime.ParseExact("18-6-2011", "MM/dd/yyyy", CultureInfo.InvariantCulture);

//i using parse also din come out.

ContentLicense: CC BY-SA 3.0


问:
Change page source without using document.write()
I want to change/replace the full page source of a page using JS (this is what happens when you use document.write), but document.write() is not recommended. I've tried document.innerHTML and some others, but I can't find one that works. Is there a way to 'rewrite' the page source using JS that respects best practices? Thanks in advance! (This is my first question, so tell me if I'm doing something wrong)

Edit: I mean change all of the page source (including head), not just the body.

Example:
Before running JS, the page source is:
Hi

After running JS: <title>ttl</title>Hi!!M

答:
Try:

```<code><script>
    document.querySelector('body').innerHTML="Hello World!";
</script>
</code>```

This will overwrite all the things of your html document and write Hello World!

Body:
Try:

<script>
    document.querySelector('body').innerHTML="Hello World!";
</script>

This will overwrite all the things of your html document and write Hello World!


问:
Count the number of vowels and print them in a manner that string having more vowels come first
In,This code I've to "n" string. Then count the number of vowels in a string and print "n" number of string in a manner that a string having more vowels print first.

```<code>Sample Input:

3
My name
is
Paul Jonas

Sample Output:

Paul Jonas 4
My name 2
is 1
</code>```

This is my attempt but I'cant process the output that a string having more vowel comes first

```<code>import java.util.*;
public class CountVowel {
    public static void  main(String[] args){
        Scanner sc = new Scanner(System.in);
        ArrayList<String> word = new ArrayList<String>();

        int n;
        n = sc.nextInt();

        String str;

        sc.nextLine();
        for(int i=0;i<n;i++){
            str = sc.nextLine();
            word.add(str);
        }

        int arr[] = new int[3];
        int j = 0;
        for(String s:word){
            int count = 0;
            for(int i=0;i<s.length();i++){
                if(s.charAt(i)=='A' || s.charAt(i)=='E' || s.charAt(i)=='I' || s.charAt(i)=='O'||s.charAt(i)=='U'
                || s.charAt(i)=='a' || s.charAt(i)=='e' || s.charAt(i)=='i' || s.charAt(i)=='o' || s.charAt(i)=='u'){

                    count++;

                }
            }
            arr[j] = count;
            j++;

        }

        int i = 0;
        for(String s: word){

            System.out.println(s+" "+arr[i]);
            i++;
        }

    }
}
</code>```

So, I'm not getting my desired Output.

```<code>Input:

3
My name
is
Paul Jonas

Output I'm getting:

My name 2
is 1
Paul Jonas 4
</code>```

答:
Store it in a Map.

Map<String,Integer> map = new HashMap<String,Integer>();

Then you can sort it by value as mentioned in this answer : https://stackoverflow.com/a/61744606/11537839

Body:
In,This code I've to "n" string. Then count the number of vowels in a string and print "n" number of string in a manner that a string having more vowels print first.

Sample Input:

3
My name
is
Paul Jonas

Sample Output:

Paul Jonas 4
My name 2
is 1

This is my attempt but I'cant process the output that a string having more vowel comes first

import java.util.*;
public class CountVowel {
    public static void  main(String[] args){
        Scanner sc = new Scanner(System.in);
        ArrayList<String> word = new ArrayList<String>();

        int n;
        n = sc.nextInt();

        String str;

        sc.nextLine();
        for(int i=0;i<n;i++){
            str = sc.nextLine();
            word.add(str);
        }

        int arr[] = new int[3];
        int j = 0;
        for(String s:word){
            int count = 0;
            for(int i=0;i<s.length();i++){
                if(s.charAt(i)=='A' || s.charAt(i)=='E' || s.charAt(i)=='I' || s.charAt(i)=='O'||s.charAt(i)=='U'
                || s.charAt(i)=='a' || s.charAt(i)=='e' || s.charAt(i)=='i' || s.charAt(i)=='o' || s.charAt(i)=='u'){

                    count++;

                }
            }
            arr[j] = count;
            j++;

        }

        int i = 0;
        for(String s: word){

            System.out.println(s+" "+arr[i]);
            i++;
        }

    }
}

So, I'm not getting my desired Output.

Input:

3
My name
is
Paul Jonas

Output I'm getting:

My name 2
is 1
Paul Jonas 4


问:
AttributeError: 'GeoDataFrame' object has no attribute 'str'
So I have tried this:

```<code>import GeoDataFrame
</code>```

But it shows the following error:

```<code>ModuleNotFoundError
Traceback (most recent call last)
<ipython-input-93-3125c1efb589> in <module>
----> 1 import GeoDataFrame

ModuleNotFoundError: No module named 'GeoDataFrame'
</code>```

when I install on cmd using the command:

```<code>pip install GeoDataFrame

ERROR: Could not find a version that satisfies the requirement GeoDataFrame (from versions: none)
ERROR: No matching distribution found for GeoDataFrame
</code>```

答: I think you need to install it via pip install geopandas. Go go through their documentation for using geodataframe.

Body:
So I have tried this:

import GeoDataFrame

But it shows the following error:

ModuleNotFoundError
Traceback (most recent call last)
<ipython-input-93-3125c1efb589> in <module>
----> 1 import GeoDataFrame

ModuleNotFoundError: No module named 'GeoDataFrame'

when I install on cmd using the command:

pip install GeoDataFrame

ERROR: Could not find a version that satisfies the requirement GeoDataFrame (from versions: none)
ERROR: No matching distribution found for GeoDataFrame


问:
How do i print a for i in range loop to a PySimpleGUI window?
I am trying to make a programm that tells you a input amount of the words rhymig with the word you choose. i am doing it in PySimpleGUI and i dont know how to print the loop in to a PySimpleGUI window. Could someone help. Thanks in advance -Josh</p>

```<code>import PySimpleGUI as sg
import requests

layout = [[sg.Text('What would you like to hear rhymes about: '), sg.InputText()],
      [sg.Submit(), sg.Cancel()]]

window = sg.Window('Rhyme.', layout)
event, values = window.read()

rhyme = values[0]

words = requests.get('https://api.datamuse.com/words?rel_rhy=' + rhyme)

words_json = words.json()
window.close()

layout = [[sg.Text(f'How many rhymes about {rhyme} would you like to hear: '), sg.InputText()],
          [sg.Submit(), sg.Cancel()]]

window = sg.Window('Rhyme.', layout)
event, values = window.read()

num = values[0]

window.close()

layout = [[sg.Text(#I dont Know how to print the list here)
</code>```

答:
If you want to show a list in Text(), you can simple use join(). I think the list you want to show is words_json, but, when I ran your code this list is empty.

However use this code to shoe a list:

```<code>layout = [[sg.Text(','.join(words_json))]]
</code>```

Body:
I am trying to make a programm that tells you a input amount of the words rhymig with the word you choose. i am doing it in PySimpleGUI and i dont know how to print the loop in to a PySimpleGUI window. Could someone help. Thanks in advance -Josh

import PySimpleGUI as sg
import requests

layout = [[sg.Text('What would you like to hear rhymes about: '), sg.InputText()],
      [sg.Submit(), sg.Cancel()]]

window = sg.Window('Rhyme.', layout)
event, values = window.read()

rhyme = values[0]

words = requests.get('https://api.datamuse.com/words?rel_rhy=' + rhyme)

words_json = words.json()
window.close()

layout = [[sg.Text(f'How many rhymes about {rhyme} would you like to hear: '), sg.InputText()],
          [sg.Submit(), sg.Cancel()]]

window = sg.Window('Rhyme.', layout)
event, values = window.read()

num = values[0]

window.close()

layout = [[sg.Text(#I dont Know how to print the list here)

Body:
If you want to show a list in Text(), you can simple use join(). I think the list you want to show is words_json, but, when I ran your code this list is empty.

However use this code to shoe a list:

layout = [[sg.Text(','.join(words_json))]]


问:
(Why) are standard errors from nnet::multinom scale dependent?
I have noticed that when I increase the scale of a predictor in a multinom() model (function is from nnet R package) the standard errors of other predictors and the intercept approach 0. Can someone tell my why that is? Feel like I'm overlooking something trivial here.

EDIT:
To be clear, it is obvious to my why the coefficient and SE for the rescaled predictor (speed) would change. What I don't get is why the coefficient for the other predictor (colorRed) would remain the same while its SE changes to 0.

```<code>library(nnet)

# Make data
data("cars")
cars$color = sample(c("Blue", "Red"), nrow(cars), replace = T)
cars$used = sample(c(T, F), nrow(cars), replace = T)

# Normal model
car_model <- multinom(used ~ color + speed, data = cars, maxit=500)
round(summary(car_model)$coefficients,3)
round(summary(car_model)$standard.errors,3)
</code>```

```<code>round(summary(car_model)$coefficients,3)
(Intercept)    colorRed       speed
     -2.216       0.054       0.136
round(summary(car_model)$standard.errors,3)
(Intercept)    colorRed       speed
      1.060       0.607       0.062
</code>```

```<code># Rescale speed model
cars$speed = cars$speed*1000
car_model <- multinom(used ~ color + speed, data = cars, maxit=500)
round(summary(car_model)$coefficients,3)
round(summary(car_model)$standard.errors,3)
</code>```

```<code>(Intercept)    colorRed       speed
     -2.216       0.054       0.000
(Intercept)    colorRed       speed
          0           0           0

</code>```

答:
In a multinomial model, the coefficient associated with a continuous independent variable (like speed) represents the change in the dependent associated with a unit change in the predictor.  So scale changes in the predictor are reflected inversely in the value of the associated coefficient (and its standard error).  So when you multiply speed by 1000, I would expect the coefficient to be reduced by 1000.  So you initial coefficient of 0.136 would be come 0.000136.  Your summary is rounded to 3dp, so the new coefficient appears as 0.000.

Factors are unaffected by this because they are not "numeric" in the sense that continuous variables are.

Body:
I have noticed that when I increase the scale of a predictor in a multinom() model (function is from nnet R package) the standard errors of other predictors and the intercept approach 0. Can someone tell my why that is? Feel like I'm overlooking something trivial here.

EDIT:
To be clear, it is obvious to my why the coefficient and SE for the rescaled predictor (speed) would change. What I don't get is why the coefficient for the other predictor (colorRed) would remain the same while its SE changes to 0.

library(nnet)

# Make data
data("cars")
cars$color = sample(c("Blue", "Red"), nrow(cars), replace = T)
cars$used = sample(c(T, F), nrow(cars), replace = T)

# Normal model
car_model <- multinom(used ~ color + speed, data = cars, maxit=500)
round(summary(car_model)$coefficients,3)
round(summary(car_model)$standard.errors,3)

round(summary(car_model)$coefficients,3)
(Intercept)    colorRed       speed
     -2.216       0.054       0.136
round(summary(car_model)$standard.errors,3)
(Intercept)    colorRed       speed
      1.060       0.607       0.062

# Rescale speed model
cars$speed = cars$speed*1000
car_model <- multinom(used ~ color + speed, data = cars, maxit=500)
round(summary(car_model)$coefficients,3)
round(summary(car_model)$standard.errors,3)

(Intercept)    colorRed       speed
     -2.216       0.054       0.000
(Intercept)    colorRed       speed
          0           0           0


问:
sort the column date/time data from csv file
hope to get your expert help, I did check on forums first. i am a beginner using repl.it (Python 3.8.2), and the first time use pandas. thanks in advance.

Start_Date_Time column could not be sorted correctly by Python from old to new (even though it can be manually correctly sorted in csv file). The Python output shows it is sorted by the first number of the cell (Australian format: dd/mm/yyyy)

```<code>import pandas as pd
csv1 = pd.read_csv("scheduleList.csv")
sort_by_date = csv1.sort_values("Start_Date_Time")
print (sort_by_date)
</code>```

Below is simplified csv data. I don't want to split the column of date and time, because want to later compare its value with the today's date and time (calculate the difference)

```<code>Event   Start_Date_Time       Duration (mins)
A         1/06/2020 7:00            60
B         1/06/2020 12:30           60
C         2/06/2020 18:30           120
D         2/06/2020 16:00           45
E         5/06/2020 17:30           60
F        12/06/2020 12:30           60
G         7/06/2020 13:00           60
H         4/06/2020 18:00           60
I         6/06/2020 11:30           60
J         6/06/2020 8:00            180
K        11/06/2020 12:30           60
L        28/06/2020 11:00           300
M        18/07/2020 19:30           120
N        20/06/2020 9:00            60
O        31/05/2020 10:00           60
P         7/06/2020 10:00           60
Q        14/06/2020 10:00           60
</code>```

答:
Try the following code. I have added a line to parse Start_Date_Time as a DateTime column and also set ascending parameter of the sort_values method to False.

import pandas as pd
csv1 = pd.read_csv("scheduleList.csv")
csv1['Start_Date_Time'] = pd.to_datetime(csv1['Start_Date_Time']) # change over here
sort_by_date = csv1.sort_values("Start_Date_Time",ascending=False) # change over here
print (sort_by_date)

```

Body:
hope to get your expert help, I did check on forums first. i am a beginner using repl.it (Python 3.8.2), and the first time use pandas. thanks in advance.

Start_Date_Time column could not be sorted correctly by Python from old to new (even though it can be manually correctly sorted in csv file). The Python output shows it is sorted by the first number of the cell (Australian format: dd/mm/yyyy)

import pandas as pd
csv1 = pd.read_csv("scheduleList.csv")
sort_by_date = csv1.sort_values("Start_Date_Time")
print (sort_by_date)

Below is simplified csv data. I don't want to split the column of date and time, because want to later compare its value with the today's date and time (calculate the difference)

Event   Start_Date_Time       Duration (mins)
A         1/06/2020 7:00            60
B         1/06/2020 12:30           60
C         2/06/2020 18:30           120
D         2/06/2020 16:00           45
E         5/06/2020 17:30           60
F        12/06/2020 12:30           60
G         7/06/2020 13:00           60
H         4/06/2020 18:00           60
I         6/06/2020 11:30           60
J         6/06/2020 8:00            180
K        11/06/2020 12:30           60
L        28/06/2020 11:00           300
M        18/07/2020 19:30           120
N        20/06/2020 9:00            60
O        31/05/2020 10:00           60
P         7/06/2020 10:00           60
Q        14/06/2020 10:00           60


问:
Do I need Vue.set() in order to change value of a data object?
I understand that Vue.set() is used to add reactive properties to an object.

But how would I change the value of an object property. Is this the correct way?

Note: This data change is intended to be sent to the database

Template

```<code> <div class="buttonList" v-for="(expense, index) in expenseButton" :key="index">
    <button type="button" @click="expenseButtonClick(expense)">{{expense.expensesKey}} </button>
    <input class="textInput" v-model.number="expense.subExpense" type="number" />
</div>
</code>```

```<code>data(){
   expenseButton[{
expenseKey: rent,
expenseValue: 10,
subExpenses: ''
}]

}

methods: {
    expenseButtonClick(expense) {
      expense.expensesValue = expense.expensesValue - expense.subExpense;
      console.log(expense);
    }
}
</code>```

答:
You need to use Vue.set only when you need to add a new property to an object and want it to be reactive. You don't need it when changing an existing property's value.

FYI, you also need to use Vue.delete only when deleting a property from an object to ensure the view gets updated.

Body:
I understand that Vue.set() is used to add reactive properties to an object.

But how would I change the value of an object property. Is this the correct way?

Note: This data change is intended to be sent to the database

Template

    <button type="button" @click="expenseButtonClick(expense)">{{expense.expensesKey}} </button>
    <input class="textInput" v-model.number="expense.subExpense" type="number" />

data(){
   expenseButton[{
expenseKey: rent,
expenseValue: 10,
subExpenses: ''
}]

}

methods: {
    expenseButtonClick(expense) {
      expense.expensesValue = expense.expensesValue - expense.subExpense;
      console.log(expense);
    }
}


问:
Generating large DataFrame in a distributed way in pyspark efficiently (without pyspark.sql.Row)
The problem boils down to the following: I want to generate a DataFrame in pyspark using existing parallelized collection of inputs and a function which given one input can generate a relatively large batch of rows. In the example below I want to generate 10^12 rows dataframe using e.g. 1000 executors:

```<code>def generate_data(one_integer):
  import numpy as np
  from pyspark.sql import Row
  M = 10000000 # number of values to generate per seed, e.g. 10M
  np.random.seed(one_integer)
  np_array = np.random.random_sample(M) # generates an array of M random values
  row_type = Row("seed", "n", "x")
  return [row_type(one_integer, i, float(np_array[i])) for i in range(M)]

N = 100000 # number of seeds to try, e.g. 100K
list_of_integers = [i for i in range(N)]
list_of_integers_rdd = spark.sparkContext.parallelize(list_of_integers)
row_rdd = list_of_integers_rdd.flatMap(list_of_integers_rdd)
from pyspark.sql.types import StructType, StructField, FloatType, IntegerType
my_schema = StructType([
       StructField("seed", IntegerType()),
       StructField("n", IntegerType()),
       StructField("x", FloatType())])
df = spark.createDataFrame(row_rdd, schema=my_schema)
</code>```

(I don't really want to study distribution of random numbers given seed - this is just an example I was able to come up with to illustrate the situation when large dataframe is not loaded from warehouse, but generated by the code)

The code above does pretty much exactly what I want. Problem is that it does it in a very inefficient way - at expense of creating a python Row object for each row, then converting the python Row objects into internal Spark columnar representation.

Is there a way I can convert batch of rows already in columnar representation (e.g. one or a few numpy arrays as above np_array) just by letting spark know that these are the columns of a batch of values?

E.g. I can write the code to generate python collection RDD where each element is an pyarrow.RecordBatch or a pandas.DataFrame, but I can't find a way to convert any of these into Spark DataFrame without creating an RDD of pyspark Row objects in the process.

There is at least a dozen of articles with examples how I can use pyarrow + pandas to convert a local (to driver) pandas dataframe to Spark dataframe efficiently, but that is not an option for me because I need data to be actually generated in a distributed way on executors rather than generating one pandas dataframe on driver and sending that to executors.

UPD.
I've found one way to avoid creation of Row objects - using RDD of python tuples. As expected it is still way too slow, but still a bit faster than using Row objects. Still, this is not really what I'm looking for (which is a really efficient way of passing columnar data to Spark from python).

Also measured time to do certain operations on a machine (crude way with quite a bit of variation in measured time, but still it representative in my opinion):
The dataset in question is 10M rows, 3 cols (one column is constant integer, other is integer range from 0 to 10M-1, third is floating point value generated using np.random.random_sample:

Locally generate pandas dataframe (10M rows): ~440-450ms

Locally generate python list of spark.sql.Row objects (10M rows): ~12-15s

Locally generate python list of tuples representing rows (10M rows): ~3.4-3.5s

Generate Spark dataframe using just 1 executor and 1 initial seed value:

using spark.createDataFrame(row_rdd, schema=my_schema): ~70-80s

using spark.createDataFrame(tuple_rdd, schema=my_schema): ~40-45s

(non-distributed creation) using spark.createDataFrame(pandas_df, schema=my_schema): ~0.4-0.5s (without pandas df generation itself which takes roughly same time) - with spark.sql.execution.arrow.enabled set to true.

The example with local-to-driver pandas dataframe converted to Spark dataframe in ~1s for 10M rows gives me a reason to believe same should be possible with dataframes generated in executors. However fastest I can achieve now is ~40s for 10M rows using RDD of python tuples.

So the question still stays - is there a way to generate a large Spark dataframe in a distributed way efficiently in pyspark?

答:
It sounds like the bottleneck is conversion from RDD -> Dataframes, and that the function at hand is fairly fast and that pandas DF conversions to spark DF via pyarrow are pretty fast. Here are two potential solutions:

Since it's easy to create pandas df's in parallel, instead of returning it from the executor, write the resulting df using df.to_parquet, ie:

```<code>def generate_data(seed):
    M = 10
    np.random.seed(seed)
    np_array = np.random.random_sample(M) # generates an array of M random values
    df = pd.DataFrame(np_array, columns=["x"])
    df["seed"] = seed
    df.reset_index().to_parquet(f"s3://bucket/part-{str(seed).zfill(5)}.parquet"
</code>```

Spark reading in the resulting parquet files should be trivial afterwards. Then your bottleneck becomes IO limits, which should be faster than spark converting tuples/Row types.

If you're not allowed to save anything to file, pandas_udf and GROUPED_MAP might help you out, assuming your spark version is recent enough. It uses pyarrow as well to convert between spark DFs and pandas DFs so it should be faster than using tuples and allows you to create and return pandas DFs from your UDF in a distributed fashion.

```<code>import numpy as np
import pandas as pd
from pyspark.sql.functions import pandas_udf, PandasUDFType

N = 10

df = spark.createDataFrame(
    [(i,) for i in range(N)], ["seed"]
)

def generate_data(seed):
    M = 10
    np.random.seed(seed)
    np_array = np.random.random_sample(M) # generates an array of M random values
    df = pd.DataFrame(np_array, columns=["x"])
    df["seed"] = seed
    return df.reset_index()

@pandas_udf("index long, x double, seed long", PandasUDFType.GROUPED_MAP)
def generate_data_udf(pdf):
    output = []
    for idx, row in pdf.iterrows():
        output.append(generate_data(row["seed"]))
    return pd.concat(output)

df.groupby("seed").apply(generate_data_udf).show()
</code>```

The slower part will be the groupby which you might be able to speed up depending on how you batch the seed going into generate_data_udf, ie:

```<code>@udf(returnType=IntegerType())
def batch_seed(seed):
    return seed // 10

df.withColumn("batch_seed", batch_seed(col("seed"))). \
groupBy("batch_seed").apply(generate_data_udf).show()
</code>```

Body:
The problem boils down to the following: I want to generate a DataFrame in pyspark using existing parallelized collection of inputs and a function which given one input can generate a relatively large batch of rows. In the example below I want to generate 10^12 rows dataframe using e.g. 1000 executors:

def generate_data(one_integer):
  import numpy as np
  from pyspark.sql import Row
  M = 10000000 # number of values to generate per seed, e.g. 10M
  np.random.seed(one_integer)
  np_array = np.random.random_sample(M) # generates an array of M random values
  row_type = Row("seed", "n", "x")
  return [row_type(one_integer, i, float(np_array[i])) for i in range(M)]

N = 100000 # number of seeds to try, e.g. 100K
list_of_integers = [i for i in range(N)]
list_of_integers_rdd = spark.sparkContext.parallelize(list_of_integers)
row_rdd = list_of_integers_rdd.flatMap(list_of_integers_rdd)
from pyspark.sql.types import StructType, StructField, FloatType, IntegerType
my_schema = StructType([
       StructField("seed", IntegerType()),
       StructField("n", IntegerType()),
       StructField("x", FloatType())])
df = spark.createDataFrame(row_rdd, schema=my_schema)

(I don't really want to study distribution of random numbers given seed - this is just an example I was able to come up with to illustrate the situation when large dataframe is not loaded from warehouse, but generated by the code)

The code above does pretty much exactly what I want. Problem is that it does it in a very inefficient way - at expense of creating a python Row object for each row, then converting the python Row objects into internal Spark columnar representation.

Is there a way I can convert batch of rows already in columnar representation (e.g. one or a few numpy arrays as above np_array) just by letting spark know that these are the columns of a batch of values?

E.g. I can write the code to generate python collection RDD where each element is an pyarrow.RecordBatch or a pandas.DataFrame, but I can't find a way to convert any of these into Spark DataFrame without creating an RDD of pyspark Row objects in the process.

There is at least a dozen of articles with examples how I can use pyarrow + pandas to convert a local (to driver) pandas dataframe to Spark dataframe efficiently, but that is not an option for me because I need data to be actually generated in a distributed way on executors rather than generating one pandas dataframe on driver and sending that to executors.

UPD.
I've found one way to avoid creation of Row objects - using RDD of python tuples. As expected it is still way too slow, but still a bit faster than using Row objects. Still, this is not really what I'm looking for (which is a really efficient way of passing columnar data to Spark from python).

Also measured time to do certain operations on a machine (crude way with quite a bit of variation in measured time, but still it representative in my opinion):
The dataset in question is 10M rows, 3 cols (one column is constant integer, other is integer range from 0 to 10M-1, third is floating point value generated using np.random.random_sample:

Locally generate pandas dataframe (10M rows): ~440-450ms

Locally generate python list of spark.sql.Row objects (10M rows): ~12-15s

Locally generate python list of tuples representing rows (10M rows): ~3.4-3.5s

Generate Spark dataframe using just 1 executor and 1 initial seed value:

using spark.createDataFrame(row_rdd, schema=my_schema): ~70-80s

using spark.createDataFrame(tuple_rdd, schema=my_schema): ~40-45s

(non-distributed creation) using spark.createDataFrame(pandas_df, schema=my_schema): ~0.4-0.5s (without pandas df generation itself which takes roughly same time) - with spark.sql.execution.arrow.enabled set to true.

The example with local-to-driver pandas dataframe converted to Spark dataframe in ~1s for 10M rows gives me a reason to believe same should be possible with dataframes generated in executors. However fastest I can achieve now is ~40s for 10M rows using RDD of python tuples.

So the question still stays - is there a way to generate a large Spark dataframe in a distributed way efficiently in pyspark?

Body:
It sounds like the bottleneck is conversion from RDD -> Dataframes, and that the function at hand is fairly fast and that pandas DF conversions to spark DF via pyarrow are pretty fast. Here are two potential solutions:

Since it's easy to create pandas df's in parallel, instead of returning it from the executor, write the resulting df using df.to_parquet, ie:

def generate_data(seed):
    M = 10
    np.random.seed(seed)
    np_array = np.random.random_sample(M) # generates an array of M random values
    df = pd.DataFrame(np_array, columns=["x"])
    df["seed"] = seed
    df.reset_index().to_parquet(f"s3://bucket/part-{str(seed).zfill(5)}.parquet"

Spark reading in the resulting parquet files should be trivial afterwards. Then your bottleneck becomes IO limits, which should be faster than spark converting tuples/Row types.

If you're not allowed to save anything to file, pandas_udf and GROUPED_MAP might help you out, assuming your spark version is recent enough. It uses pyarrow as well to convert between spark DFs and pandas DFs so it should be faster than using tuples and allows you to create and return pandas DFs from your UDF in a distributed fashion.

import numpy as np
import pandas as pd
from pyspark.sql.functions import pandas_udf, PandasUDFType

N = 10

df = spark.createDataFrame(
    [(i,) for i in range(N)], ["seed"]
)

def generate_data(seed):
    M = 10
    np.random.seed(seed)
    np_array = np.random.random_sample(M) # generates an array of M random values
    df = pd.DataFrame(np_array, columns=["x"])
    df["seed"] = seed
    return df.reset_index()

@pandas_udf("index long, x double, seed long", PandasUDFType.GROUPED_MAP)
def generate_data_udf(pdf):
    output = []
    for idx, row in pdf.iterrows():
        output.append(generate_data(row["seed"]))
    return pd.concat(output)

df.groupby("seed").apply(generate_data_udf).show()

The slower part will be the groupby which you might be able to speed up depending on how you batch the seed going into generate_data_udf, ie:

@udf(returnType=IntegerType())
def batch_seed(seed):
    return seed // 10

df.withColumn("batch_seed", batch_seed(col("seed"))). \
groupBy("batch_seed").apply(generate_data_udf).show()


问:
unable to store multiple sed operations using pipe in a variable
I am not able to execute a multiple sed statement using pipe filters in a variable.
i am trying to extract a path from a file and then working on that path to change a few letters within the path by going through another sed statement.

the code looks like this:

```<code>for i in 1 2 3 4 5 6 7
do
  for j in F E I D
  do
  cpath="`sed -n "11,11p" <path.cfg | sed -n 's/voicex/voice'$i'/g; s/voicecolx/voicecol'$i'/g; s/" F "/" '$j' "/g;P'`"
  echo $cpath
   more $cpath 2>error.log | grep " '&j' " | wc -l | echo "The count of '$j' in file is: " `wc -l` 1>countdir/count$j.txt
  done
done
</code>```

i saved this file in stest.sh. When i run the file, nothing happens. I ahve to press Ctrl+C to exit back to the command prompt.

Let me also provide the path.cfg file:

```<code>MPAY
/var/xacct_data/xxxx/log_flattener/xxxx/logfile_current

FAFF
/var/xacct_data/faff/faff1/log_flattener/faffsnp1/ logfile_current
/var/xacct_data/faff/faff1/log_flattener/faffdbt1 /logfile_current
/var/xacct_data/faff/faff1/log_flattener/fafftxn1 /logfile_current
/var/xacct_data/faff/faff2/log_flattener/faffdbt2/ logfile_current

VOICE
/var/xacct_data/voice/voicex/log_flattener/ voicecolx/logfile_current
/var/xacct_data/voice/voicex/log_flattener/ voiceprox/logfile_current
/var/xacct_data/voice/voicex/log_flattener/ voicedisx/logfile_current

GPRS
/var/xacct_data/gprs/gprsx/log_flattener/gprscolx/logfile_current
/var/xacct_data/gprs/gprsx/log_flattener/gprsprox/logfile_current
/var/xacct_data/gprs/gprsx/log_flattener/gprsdisx/logfile_current

IPC
/var/xacct_data/ipc/ipcx/log_flattener/ipccolx/logfile_current
/var/xacct_data/ipc/ipcx/log_flattener/ipcprox/logfile_current
/var/xacct_data/ipc/ipcx/log_flattener/ipcdisx/logfile_current

SMS
/var/xacct_data/smsc/smscx/log_flattener/smsccolx/logfile_current
/var/xacct_data/smsc/smscx/log_flattener/smscprox/logfile_current
/var/xacct_data/smsc/smscx/log_flattener/smscdisx/logfile_current
</code>```

答:
In order to get specific advice, you need to provide a sample path.cfg, explain what's wrong with the script current's behavior and what the expected behavior is. That being said, there are a few red flags in your code.

Use $(…) instead of `…` for command substitutions. The parsing of backquotes is rather fiddly, and you may run into issues with nested quotes.

sed -n "11,11p" means print the 11th line only. Is this what you really wanted? I guess not, but knowing neither the format of path.cfg nor what you intended I can't tell you what that command should be.

You should always use double quotes around variable substitutions, e.g. echo "$cpath". Only leave them out if you know it would be wrong to have them, do put them if you have no good reason to omit them. I don't think this is causing any problem here, but it's a good habit to get into when writing shell scripts.

Why are you calling more? Piping more into another program is useless. Just write <"$cpath" grep " '&j' " | …, assuming you did want to display all the lines containing the string '&F' (and so on for other values of $j) in the file called $cpath.

Still on this line, grep PATTERN | wc -l can just be written grep -c PATTERN.

Still on this line, you're running wc -l on the output of wc -l, which isn't useful. I supposed you meant to run it only once. Here are two ways to write this line; the first is closed to your structure, the second is rather clearer:

```<code><"$cpath" grep -c " '&j' " | echo "The count of '$j' in file is:  $(cat)" >"countdir/count$j.txt"
echo "The count of '$j' in file is:  $(<"$cpath" grep -c " '&j' ")" >"countdir/count$j.txt"
</code>```

Body:
I am not able to execute a multiple sed statement using pipe filters in a variable.
i am trying to extract a path from a file and then working on that path to change a few letters within the path by going through another sed statement.

the code looks like this:

for i in 1 2 3 4 5 6 7
do
  for j in F E I D
  do
  cpath="`sed -n "11,11p" <path.cfg | sed -n 's/voicex/voice'$i'/g; s/voicecolx/voicecol'$i'/g; s/" F "/" '$j' "/g;P'`"
  echo $cpath
   more $cpath 2>error.log | grep " '&j' " | wc -l | echo "The count of '$j' in file is: " `wc -l` 1>countdir/count$j.txt
  done
done

i saved this file in stest.sh. When i run the file, nothing happens. I ahve to press Ctrl+C to exit back to the command prompt.

Let me also provide the path.cfg file:

MPAY
/var/xacct_data/xxxx/log_flattener/xxxx/logfile_current

FAFF
/var/xacct_data/faff/faff1/log_flattener/faffsnp1/ logfile_current
/var/xacct_data/faff/faff1/log_flattener/faffdbt1 /logfile_current
/var/xacct_data/faff/faff1/log_flattener/fafftxn1 /logfile_current
/var/xacct_data/faff/faff2/log_flattener/faffdbt2/ logfile_current

VOICE
/var/xacct_data/voice/voicex/log_flattener/ voicecolx/logfile_current
/var/xacct_data/voice/voicex/log_flattener/ voiceprox/logfile_current
/var/xacct_data/voice/voicex/log_flattener/ voicedisx/logfile_current

GPRS
/var/xacct_data/gprs/gprsx/log_flattener/gprscolx/logfile_current
/var/xacct_data/gprs/gprsx/log_flattener/gprsprox/logfile_current
/var/xacct_data/gprs/gprsx/log_flattener/gprsdisx/logfile_current

IPC
/var/xacct_data/ipc/ipcx/log_flattener/ipccolx/logfile_current
/var/xacct_data/ipc/ipcx/log_flattener/ipcprox/logfile_current
/var/xacct_data/ipc/ipcx/log_flattener/ipcdisx/logfile_current

SMS
/var/xacct_data/smsc/smscx/log_flattener/smsccolx/logfile_current
/var/xacct_data/smsc/smscx/log_flattener/smscprox/logfile_current
/var/xacct_data/smsc/smscx/log_flattener/smscdisx/logfile_current

Body:
In order to get specific advice, you need to provide a sample path.cfg, explain what's wrong with the script current's behavior and what the expected behavior is. That being said, there are a few red flags in your code.

Use $(…) instead of `…` for command substitutions. The parsing of backquotes is rather fiddly, and you may run into issues with nested quotes.

sed -n "11,11p" means print the 11th line only. Is this what you really wanted? I guess not, but knowing neither the format of path.cfg nor what you intended I can't tell you what that command should be.

You should always use double quotes around variable substitutions, e.g. echo "$cpath". Only leave them out if you know it would be wrong to have them, do put them if you have no good reason to omit them. I don't think this is causing any problem here, but it's a good habit to get into when writing shell scripts.

Why are you calling more? Piping more into another program is useless. Just write <"$cpath" grep " '&j' " | …, assuming you did want to display all the lines containing the string '&F' (and so on for other values of $j) in the file called $cpath.

Still on this line, grep PATTERN | wc -l can just be written grep -c PATTERN.

Still on this line, you're running wc -l on the output of wc -l, which isn't useful. I supposed you meant to run it only once. Here are two ways to write this line; the first is closed to your structure, the second is rather clearer:

<"$cpath" grep -c " '&j' " | echo "The count of '$j' in file is:  $(cat)" >"countdir/count$j.txt"
echo "The count of '$j' in file is:  $(<"$cpath" grep -c " '&j' ")" >"countdir/count$j.txt"


问:
How to use an outer query variable in a nested sub query using update
I am using an update query in which I use an outer variable in a nested sub query, but I find an error:

SQL Error: ORA-00904: "FS"."GR_NUMBER": invalid identifier

```<code>update table fs set fs.branch_id=
(select branch_id
from
    (select  branch_id,row_number() over(PARTITION by gr_number order by updated_ts desc) as Sno
from admission_log
where gr_number=fs.gr_number )
where sno=1) ;
</code>```

答:
Since it's Oracle, you can use a merge clause and use the min() function to get the value instead of the row_number() function.

```<code>merge into table fs using
(select gr_number, min(branch_id) as branch_id from admission_log) qry
on
(fx.gr_number = qry.gr_number)
when matched then
update
set
fs.branch_id = qry.branch_id;
</code>```

If you still wanted to use the roe_number() function, below is the query. Your WHERE clause to join the table should be outside the subquery.

```<code>update table fs set fs.branch_id=
(select branch_id
from
(select  branch_id,row_number() over(PARTITION by gr_number order by updated_ts desc)
as Sno
from admission_log)
where sno=1
and gr_number=fs.gr_number) ;
</code>```

Body:
I am using an update query in which I use an outer variable in a nested sub query, but I find an error:

SQL Error: ORA-00904: "FS"."GR_NUMBER": invalid identifier

update table fs set fs.branch_id=
(select branch_id
from
    (select  branch_id,row_number() over(PARTITION by gr_number order by updated_ts desc) as Sno
from admission_log
where gr_number=fs.gr_number )
where sno=1) ;

Body:
Since it's Oracle, you can use a merge clause and use the min() function to get the value instead of the row_number() function.

merge into table fs using
(select gr_number, min(branch_id) as branch_id from admission_log) qry
on
(fx.gr_number = qry.gr_number)
when matched then
update
set
fs.branch_id = qry.branch_id;

If you still wanted to use the roe_number() function, below is the query. Your WHERE clause to join the table should be outside the subquery.

update table fs set fs.branch_id=
(select branch_id
from
(select  branch_id,row_number() over(PARTITION by gr_number order by updated_ts desc)
as Sno
from admission_log)
where sno=1
and gr_number=fs.gr_number) ;


问:
Call to MultiByteToWideChar() in 64-bit office gives wrong result
I have an Excel VBA project I'm in the process of adapting for 64-bit Office. In one part, I make calls to MultiByteToWideChar() using any of 20 or so different code pages. (So StrConv is not an alternative.)

This has been working for me for years in 32-bit Office using the following declare:

Declare Function MultiByteToWideChar Lib "kernel32" ( _
  ByVal codepage As Long, _
  ByVal dwFlags As Long, _
  lpMultiByteStr As Any, _
  ByVal cbMultiByte As Long, _
  ByVal lpWideCharStr As Long, _
  ByVal cchWideChar As Long _
  ) As Long
    'params: UINT, DWORD, LPCSTR, int, LPWSTR, int
    'return: int
```

<p>But my adaptation to 64-bit is not: I get wrong results (e.g. an empty string where a non-empty string is expected), and frequent crashes. I'm using a <code>declare</code> that I got from the Microsoft-provided Win32API_PtrSafe.TXT file. (Of course, it could have bugs.)</p>

<p>So, I'm guessing something is not right in the declare statement or in how I'm making the call.</p>

<p>Here's a minimal sample that repro's:</p>

<pre class="lang-vb prettyprint-override"><code>'Windows API declarations

Public Const MB_PRECOMPOSED = &H1       'use precomposed chars

Declare PtrSafe Function MultiByteToWideChar Lib "kernel32" ( _
  ByVal CodePage As Long, _
  ByVal dwFlags As Long, _
  ByVal lpMultiByteStr As String, _
  ByVal cchMultiByte As Long, _
  ByVal lpWideCharStr As String, _
  ByVal cchWideChar As Long _
  ) As Long
    'params: UINT, DWORD, LPCSTR, int, LPWSTR, int
    'return: int

' My function that calls MultiByteToWideChar

Private Function EncodedStringByteArrayToString(abStringData() As Byte, lngArrLen As Long, CodePage As Long) As String
    Dim lngStrLen As Long, str As String

    lngStrLen = MultiByteToWideChar(CodePage, MB_PRECOMPOSED, ByVal VarPtr(abStringData(1)), lngArrLen, 0&, 0)
    str = String(lngStrLen, " ")
    lngStrLen = MultiByteToWideChar(CodePage, MB_PRECOMPOSED, ByVal VarPtr(abStringData(1)), lngArrLen, StrPtr(str), lngStrLen)
    EncodedStringByteArrayToString = str
End Function

' Sample routine to produce repro

Private Sub TestMB2WCBug()
    Dim abStringData(1 To 9) As Byte
    Dim resultString As String

    abStringData(1) = 67
    abStringData(2) = 111
    abStringData(3) = 112
    abStringData(4) = 121
    abStringData(5) = 114
    abStringData(6) = 105
    abStringData(7) = 103
    abStringData(8) = 104
    abStringData(9) = 116

    resultString = EncodedStringByteArrayToString(abStringData(), 9, 10000)
End Sub
</code>```

答:
This has been working for me for years in 32-bit Office

It could not possibly work with the Declare that you have shown.

MultiByteToWideChar expects an LPWSTR as the output buffer. VB performs automatic conversion from Unicode to ANSI when passing strings into Declared functions, so there is no way that the function would receive a pointer to a wide string buffer when lpWideCharStr is declared As String. At best, it would receive a buffer that is large enough so no buffer overflow would occur, and then VB would perform conversion back to Unicode when returning from the function, so you will end up with a double-unicode string.

lpMultiByteStr is not a string either, it's an array of bytes in some encoding.

The code inside EncodedStringByteArrayToString seems to know all that, because it correctly passes a byte array for lpMultiByteStr and an StrPtr for lpWideCharStr. This could have not happened with the current declaration of MultiByteToWideChar.

The declaration that is assumed by the code in EncodedStringByteArrayToString is:

Declare PtrSafe Function MultiByteToWideChar Lib "kernel32" ( _
  ByVal CodePage As Long, _
  ByVal dwFlags As Long, _
  ByVal lpMultiByteStr As LongPtr, _
  ByVal cchMultiByte As Long, _
  ByVal lpWideCharStr As LongPtr, _
  ByVal cchWideChar As Long _
  ) As Long
```

Apparently you had that before, so just put it back.

Body:
I have an Excel VBA project I'm in the process of adapting for 64-bit Office. In one part, I make calls to MultiByteToWideChar() using any of 20 or so different code pages. (So StrConv is not an alternative.)

This has been working for me for years in 32-bit Office using the following declare:

Declare Function MultiByteToWideChar Lib "kernel32" ( _
  ByVal codepage As Long, _
  ByVal dwFlags As Long, _
  lpMultiByteStr As Any, _
  ByVal cbMultiByte As Long, _
  ByVal lpWideCharStr As Long, _
  ByVal cchWideChar As Long _
  ) As Long
    'params: UINT, DWORD, LPCSTR, int, LPWSTR, int
    'return: int

But my adaptation to 64-bit is not: I get wrong results (e.g. an empty string where a non-empty string is expected), and frequent crashes. I'm using a declare that I got from the Microsoft-provided Win32API_PtrSafe.TXT file. (Of course, it could have bugs.)

So, I'm guessing something is not right in the declare statement or in how I'm making the call.

Here's a minimal sample that repro's:

'Windows API declarations

Public Const MB_PRECOMPOSED = &H1       'use precomposed chars

Declare PtrSafe Function MultiByteToWideChar Lib "kernel32" ( _
  ByVal CodePage As Long, _
  ByVal dwFlags As Long, _
  ByVal lpMultiByteStr As String, _
  ByVal cchMultiByte As Long, _
  ByVal lpWideCharStr As String, _
  ByVal cchWideChar As Long _
  ) As Long
    'params: UINT, DWORD, LPCSTR, int, LPWSTR, int
    'return: int

' My function that calls MultiByteToWideChar

Private Function EncodedStringByteArrayToString(abStringData() As Byte, lngArrLen As Long, CodePage As Long) As String
    Dim lngStrLen As Long, str As String

    lngStrLen = MultiByteToWideChar(CodePage, MB_PRECOMPOSED, ByVal VarPtr(abStringData(1)), lngArrLen, 0&, 0)
    str = String(lngStrLen, " ")
    lngStrLen = MultiByteToWideChar(CodePage, MB_PRECOMPOSED, ByVal VarPtr(abStringData(1)), lngArrLen, StrPtr(str), lngStrLen)
    EncodedStringByteArrayToString = str
End Function

' Sample routine to produce repro

Private Sub TestMB2WCBug()
    Dim abStringData(1 To 9) As Byte
    Dim resultString As String

    abStringData(1) = 67
    abStringData(2) = 111
    abStringData(3) = 112
    abStringData(4) = 121
    abStringData(5) = 114
    abStringData(6) = 105
    abStringData(7) = 103
    abStringData(8) = 104
    abStringData(9) = 116

    resultString = EncodedStringByteArrayToString(abStringData(), 9, 10000)
End Sub

Body:
This has been working for me for years in 32-bit Office

It could not possibly work with the Declare that you have shown.

MultiByteToWideChar expects an LPWSTR as the output buffer. VB performs automatic conversion from Unicode to ANSI when passing strings into Declared functions, so there is no way that the function would receive a pointer to a wide string buffer when lpWideCharStr is declared As String. At best, it would receive a buffer that is large enough so no buffer overflow would occur, and then VB would perform conversion back to Unicode when returning from the function, so you will end up with a double-unicode string.

lpMultiByteStr is not a string either, it's an array of bytes in some encoding.

The code inside EncodedStringByteArrayToString seems to know all that, because it correctly passes a byte array for lpMultiByteStr and an StrPtr for lpWideCharStr. This could have not happened with the current declaration of MultiByteToWideChar.

The declaration that is assumed by the code in EncodedStringByteArrayToString is:

Declare PtrSafe Function MultiByteToWideChar Lib "kernel32" ( _
  ByVal CodePage As Long, _
  ByVal dwFlags As Long, _
  ByVal lpMultiByteStr As LongPtr, _
  ByVal cchMultiByte As Long, _
  ByVal lpWideCharStr As LongPtr, _
  ByVal cchWideChar As Long _
  ) As Long

Apparently you had that before, so just put it back.


问:
TLS version used by Outlook add in
Not an expert in .NET and in Outlook add in development. However have to look for an issue in an existing outlook add in :).

Our outlook add in makes communication with a Java based server over HTTPS. My outlook add in is on 4.5.1 .NET version.

Right now our server supports TLS 1.0 , 1.1 and 1.2 .

However i see this code in our outlook add in

```<code> ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12 | SecurityProtocolType.Ssl3;
</code>```

As i understand this code ensures that communication can happen depending on the TLS version supported by server as per this
https://learn.microsoft.com/en-us/dotnet/api/system.net.servicepointmanager.securityprotocol?view=netcore-3.1

I don't know if the communication should  have happened over the highest secure platform in my case if both client and server both support 1.2.

But still i see the communication is happening over less than < 1.2. Not sure why this is happening.  As per the documentation i understand that OS setting does not influence the TLS version for .NET versions less than 4.7.

And also is the above code correct and ensures that the communication should happen over the highest secure version. ?

Best Regards,

Saurav

答:
My outlook add in is on 4.5.1 .NET version.

The default System.Net.ServicePointManager.SecurityProtocol in both .NET 4.0/4.5 is SecurityProtocolType.Tls|SecurityProtocolType.Ssl3. But if you use the following code at startup early then everything should go smoothly:

```<code>ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12 | SecurityProtocolType.Ssl3;
</code>```

But still i see the communication is happening over less than < 1.2. Not sure why this is happening.

You need to check out settings on the server-side. I'd suggest using Fiddler to see what is going on at the low-level and why a lower TLS version is picked up.

Title: TLS version used by Outlook add in

Body:
Not an expert in .NET and in Outlook add in development. However have to look for an issue in an existing outlook add in :).

Our outlook add in makes communication with a Java based server over HTTPS. My outlook add in is on 4.5.1 .NET version.

Right now our server supports TLS 1.0 , 1.1 and 1.2 .

However i see this code in our outlook add in

 ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12 | SecurityProtocolType.Ssl3;

As i understand this code ensures that communication can happen depending on the TLS version supported by server as per this
https://learn.microsoft.com/en-us/dotnet/api/system.net.servicepointmanager.securityprotocol?view=netcore-3.1

I don't know if the communication should  have happened over the highest secure platform in my case if both client and server both support 1.2.

But still i see the communication is happening over less than < 1.2. Not sure why this is happening.  As per the documentation i understand that OS setting does not influence the TLS version for .NET versions less than 4.7.

And also is the above code correct and ensures that the communication should happen over the highest secure version. ?

Best Regards,

Saurav

Body:
My outlook add in is on 4.5.1 .NET version.

The default System.Net.ServicePointManager.SecurityProtocol in both .NET 4.0/4.5 is SecurityProtocolType.Tls|SecurityProtocolType.Ssl3. But if you use the following code at startup early then everything should go smoothly:

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12 | SecurityProtocolType.Ssl3;

But still i see the communication is happening over less than < 1.2. Not sure why this is happening.

You need to check out settings on the server-side. I'd suggest using Fiddler to see what is going on at the low-level and why a lower TLS version is picked up.


问:
Understanding @input - from parent to child
I try to learn Angular and now I'm at the @Input stage.

I have a main app and a child component. In app.component.ts I have a test variable. I want to pass this variable from app.component.ts to child.component.ts.

```<code>// app.component.ts:
export class AppComponent {
    test = 10;
}

// child.component.ts:
export class ChildComponent {
    @Input() fromParent: number;
    childVar = fromParent + 5;

    show() {
        console.log(this.childVar);
    } //this should to show 15 in console...
}
</code>```

Now, how can I do this?

答:
```<code>*So what if the apples component is almost the same for each instance, but we need it to display a different description for each instance? We use Input .*

  // app.component.html
<app-apple [description]="appleDescription"></app-apple>
</code>```

appleDescription can be any class property inside app.component.html.

```<code>// apple.component.ts
import { Component, OnInit, Input } from '@angular/core';
...
export class AppleComponent implements OnInit {
   @Input() description: string;
...
</code>```

Now, description is passed in by whomever the parent component is. In this case it’s app.component.html. Any value can be passed into [description] and description can be used anywhere inside of apple.component.html or apple.component.ts as if it were a regular class property that follows normal change detection.

```<code>// apple.component.html
{{ description || 'Nothing has been typed yet...' }}
</code>```

Body:
I try to learn Angular and now I'm at the @Input stage.

I have a main app and a child component. In app.component.ts I have a test variable. I want to pass this variable from app.component.ts to child.component.ts.

// app.component.ts:
export class AppComponent {
    test = 10;
}

// child.component.ts:
export class ChildComponent {
    @Input() fromParent: number;
    childVar = fromParent + 5;

    show() {
        console.log(this.childVar);
    } //this should to show 15 in console...
}

Now, how can I do this?

Body:
*So what if the apples component is almost the same for each instance, but we need it to display a different description for each instance? We use Input .*

  // app.component.html
<app-apple [description]="appleDescription"></app-apple>

appleDescription can be any class property inside app.component.html.

// apple.component.ts
import { Component, OnInit, Input } from '@angular/core';
...
export class AppleComponent implements OnInit {
   @Input() description: string;
...

Now, description is passed in by whomever the parent component is. In this case it’s app.component.html. Any value can be passed into [description] and description can be used anywhere inside of apple.component.html or apple.component.ts as if it were a regular class property that follows normal change detection.

// apple.component.html
{{ description || 'Nothing has been typed yet...' }}


问:
Is there a simple way to check a function/expression for linearity in Java?
i try to find a way or a library to check if a function/expression is linear or not.

I am aware, that i can write a syntax check fpr the standart form of a linear function.
The challenge is, that it could be that the input is not standartisized.

For example:
2x+1

could be

(x+1)^2-x^2

Would be glad if somebody knows an easy way to get that working.

Update 1:
The solution should work for functions/expressions with 1..n variables.

答:
If approximation is fine, the easiest, generic way would be to evaluate the function at random points, and then check for linear changes in the output.
i.e,

Generate n random points x_0, x_1, ... x_n

For each distinct pair (x_i, x_j) from it, check that (f(x_i) - f(x_j))/(x_j - x_i) is (almost) constant.


问:
Paypal Integration with reactjs -node.js -Guidance on implementation
Usage of smart payment buttons by PayPal:

how to use the smart payment button in Paypal ? when I am rendering the button I am getting errors , this is the script which I want to use here :

https://developer.paypal.com/demo/checkout/#/pattern/client

I am using the useEffect to load the script :

```<code>useEffect(() => {
    const script = document.createElement('script');

    script.src = "https://www.paypal.com/sdk/js?client-id=AZOEGr_kwtZWZ2_Hy46cHBs1pYR_Ly68zMWymBQX88AlUM70HifIuAbwaUW_92BXhCxAozxqFDBLU5wj&currency=USD";
    script.async = true;

    document.body.appendChild(script);

    return () => {
      document.body.removeChild(script);
    }
  }, []);
</code>```

after that , I am rendering the summary of the registration in card where I want to render this Paypal smart button

```<code><Card className={classes.card}>
        <CardContent style={{textAlign:"center"}}>
            .....
            //registration details
            .....
        </CardContent>
        <CardActions style={{justifyContent:"center"}} >
            {renderPaypal}
         // renderPaypal invokes the function to render the button
        </CardActions>

    </Card>
</code>```

Here this is the function where I render the button :

```<code>   const renderPaypal=()=>{
    return (
        <script>
    // Render the PayPal button into #paypal-button-container
    paypal.Buttons({

        // Set up the transaction
        createOrder: function(data, actions) {
            return fetch('/demo/checkout/api/paypal/order/create/', {
                method: 'post'
            }).then(function(res) {
                return res.json();
            }).then(function(data) {
                return data.orderID;
            });
        },

        // Finalize the transaction
        onApprove: function(data, actions) {
            return fetch('/demo/checkout/api/paypal/order/' + data.orderID + '/capture/', {
                method: 'post'
            }).then(function(res) {
                return res.json();
            }).then(function(details) {
                // Show a success message to the buyer
                alert('Transaction completed by ' + details.payer.name.given_name + '!');
            });
        }

    }).render('#paypal-button-container');
</script>

)
}
</code>```

答:
There is a guide here: https://developer.paypal.com/docs/checkout/reference/server-integration/#

But if you want to use the code in the samples directory, there is this one for the capture intent: https://github.com/paypal/Checkout-NodeJS-SDK/tree/develop/samples/CaptureIntentExamples

For your front-end web code, this is best: https://developer.paypal.com/demo/checkout/#/pattern/server

For refunding the payment, in the capture intent examples there is an example invocation of the refund method using a captureId : https://github.com/paypal/Checkout-NodeJS-SDK/blob/2476cd9c0ed7fdda5402bc7a2094522e6d719f5d/samples/CaptureIntentExamples/runAll.js

So that looks to be enough to figure it out how to use it

Body:
Usage of smart payment buttons by PayPal:

how to use the smart payment button in Paypal ? when I am rendering the button I am getting errors , this is the script which I want to use here :

https://developer.paypal.com/demo/checkout/#/pattern/client

I am using the useEffect to load the script :

useEffect(() => {
    const script = document.createElement('script');

    script.src = "https://www.paypal.com/sdk/js?client-id=AZOEGr_kwtZWZ2_Hy46cHBs1pYR_Ly68zMWymBQX88AlUM70HifIuAbwaUW_92BXhCxAozxqFDBLU5wj&currency=USD";
    script.async = true;

    document.body.appendChild(script);

    return () => {
      document.body.removeChild(script);
    }
  }, []);

after that , I am rendering the summary of the registration in card where I want to render this Paypal smart button

<Card className={classes.card}>
        <CardContent style={{textAlign:"center"}}>
            .....
            //registration details
            .....
        </CardContent>
        <CardActions style={{justifyContent:"center"}} >
            {renderPaypal}
         // renderPaypal invokes the function to render the button
        </CardActions>

    </Card>

Here this is the function where I render the button :

   const renderPaypal=()=>{
    return (

)
}


问:
Delete an array element based on a value that contains a date in PHP 7
I'm trying to delete array elements older than 1 month.

All array elements contains a date value which I'm trying to compare in PHP using strtotime +1 month.

The array looks like this:

```<code>[
  {
    "userid": "f2eca630-f89a-4b7f-b463-d834a2b50a24",
    "name": "4q9fenYDD3Czy2fyTkQmIlvj-1590978189-9BBNrQE5.webp",
    "gender": "female",
    "rating": 61231,
    "date": "2020-04-08 02:23:09",
    "description": ""
  },
  {
    "userid": "f2eca630-f89a-4b7f-b463-d834a2b50a24",
    "name": "o0SuDVEytVBIMJkveNWamamF-1590990218-27UmvlNd.webp",
    "gender": "male",
    "rating": 0,
    "date": "2020-06-01 03:41:38",
    "description": ""
  }
]
</code>```

I'm checking the array with the following PHP code:

```<code>$datenow = date("Y-m-d H:i:s");
$expireDate = date("Y-m-d H:i:s",strtotime('+1 month',strtotime($datenow)));

foreach ($monthlyArray as $key => $item) {
    $gotDate = $item['date'];
    echo "Comparing Array Date: ($gotDate) with Current Date +1 month: ($expireDate) <br>";
    if ($gotDate > $expireDate) {
        echo $gotDate . " will be removed because of: " . $expireDate ."<br>";
        //unset ($monthlyArray[$key]);
    }
}

</code>```

Note that $monthlyArray contains the array above.

The results I get when I run this code:

```<code>Comparing Array Date: (2020-04-08 02:23:09) with Current Date +1 month: (2020-06-01 09:19:09)
Comparing Array Date: (2020-06-01 03:41:38) with Current Date +1 month: (2020-06-01 09:19:09)
</code>```

As you can see, the if ($gotDate > $expireDate) didn't get fired which is very odd because the first element in the array has a date set to 2020-04-08 02:23:09 which is indeed more than 1 month old.

What am I doing wrong?

答:
Try comparing timestamps instead of date objects :

```<code>$datenow = date("Y-m-d H:i:s");
$expireDateTimestamp = strtotime('+1 month',strtotime($datenow));
$expireDate = date("Y-m-d H:i:s", $expireDateTimestamp);

foreach ($monthlyArray as $key => $item) {
    $gotDate = $item['date'];
    $gotDateTimestamp = strotime($gotDate);
    echo "Comparing Array Date: ($gotDate) with Current Date +1 month: ($expireDate) <br>";
    if ($gotDateTimestamp > $expireDateTimestamp) {
        echo $gotDate . " will be removed because of: " . $expireDate ."<br>";
        //unset ($monthlyArray[$key]);
    }
}
</code>```

Body:
I'm trying to delete array elements older than 1 month.

All array elements contains a date value which I'm trying to compare in PHP using strtotime +1 month.

The array looks like this:

[
  {
    "userid": "f2eca630-f89a-4b7f-b463-d834a2b50a24",
    "name": "4q9fenYDD3Czy2fyTkQmIlvj-1590978189-9BBNrQE5.webp",
    "gender": "female",
    "rating": 61231,
    "date": "2020-04-08 02:23:09",
    "description": ""
  },
  {
    "userid": "f2eca630-f89a-4b7f-b463-d834a2b50a24",
    "name": "o0SuDVEytVBIMJkveNWamamF-1590990218-27UmvlNd.webp",
    "gender": "male",
    "rating": 0,
    "date": "2020-06-01 03:41:38",
    "description": ""
  }
]

I'm checking the array with the following PHP code:

$datenow = date("Y-m-d H:i:s");
$expireDate = date("Y-m-d H:i:s",strtotime('+1 month',strtotime($datenow)));

foreach ($monthlyArray as $key => $item) {
    $gotDate = $item['date'];
    echo "Comparing Array Date: ($gotDate) with Current Date +1 month: ($expireDate)
";
    if ($gotDate > $expireDate) {
        echo $gotDate . " will be removed because of: " . $expireDate ."
";
        //unset ($monthlyArray[$key]);
    }
}

Note that $monthlyArray contains the array above.

The results I get when I run this code:

Comparing Array Date: (2020-04-08 02:23:09) with Current Date +1 month: (2020-06-01 09:19:09)
Comparing Array Date: (2020-06-01 03:41:38) with Current Date +1 month: (2020-06-01 09:19:09)

As you can see, the if ($gotDate > $expireDate) didn't get fired which is very odd because the first element in the array has a date set to 2020-04-08 02:23:09 which is indeed more than 1 month old.

What am I doing wrong?

Body:
Try comparing timestamps instead of date objects :

$datenow = date("Y-m-d H:i:s");
$expireDateTimestamp = strtotime('+1 month',strtotime($datenow));
$expireDate = date("Y-m-d H:i:s", $expireDateTimestamp);

foreach ($monthlyArray as $key => $item) {
    $gotDate = $item['date'];
    $gotDateTimestamp = strotime($gotDate);
    echo "Comparing Array Date: ($gotDate) with Current Date +1 month: ($expireDate) <br>";
    if ($gotDateTimestamp > $expireDateTimestamp) {
        echo $gotDate . " will be removed because of: " . $expireDate ."<br>";
        //unset ($monthlyArray[$key]);
    }
}


问:
Vehicle gear prediction using clustering algorithm (machine learning)
I am trying to predict which gear vehicle is driven.
I have Engine_Speed and vehicle_Speed column in the data set:

I have tried the k-means clustering algorithm, but it didn't succeed.

Which algorithm do I have to use? And how do I implement it using Python?

答:
Looking at the vehicle speed in relation to the engine speed, the different slopes should give the different gears.

My initial reaction would be to say that this is a linear regression problem. You don't have enough data for anything else. Looking at the data, though, we can see that it is actually two linear regression problems:

[![Engine speed vs. vehicle speed][2]][2]

There is an inflection point at about 700 revs, so you should design a cutoff that selects one of two regression lines, depending on whether you are above or below the cutoff.

To determine the regression in Python, you can use any number of packages. In scikit-learn it looks like this:
https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html

The example given there, using the Python console, is

```<code>>>> import numpy as np
>>> from sklearn.linear_model import LinearRegression
>>> X = np.array([[1, 1], [1, 2], [2, 2], [2, 3]])
>>> # y = 1 * x_0 + 2 * x_1 + 3
>>> y = np.dot(X, np.array([1, 2])) + 3
>>> reg = LinearRegression().fit(X, y)
>>> reg.score(X, y)
1.0
>>> reg.coef_
array([1., 2.])
>>> reg.intercept_
3.0000...
>>> reg.predict(np.array([[3, 5]]))
array([16.])
</code>```

Obviously you need to put your own data in X and y and in fact you would want two arrays for the two sections of your graph. You would also have two reg = LinearRegression().fit(X, y) expressions, and an if statement deciding which reg to use, depending on the input. The inflection point is at the intersection of your two regression lines.

The two regression lines have the form y = m1 x + c1 and y = m2 x + c2, where m1, m2 are the gradients of the lines and c1, c2 the intercepts. At the point of intersection m1x + c1 = m2x + c2. If you don't want to do the maths, then you can use Shapely:

```<code>import shapely
from shapely.geometry import LineString, Point

line1 = LineString([A, B])
line2 = LineString([C, D])

int_pt = line1.intersection(line2)
point_of_intersection = int_pt.x, int_pt.y

print(point_of_intersection)
</code>```

(taken from this answer on Stack Overflow: How do I compute the intersection point of two lines?)

After discussion with Sanjiv, here is the updated code (adapted from here: https://machinelearningmastery.com/clustering-algorithms-with-python/)

```<code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib
from sklearn.cluster import KMeans

matplotlib.use('TkAgg')

df = pd.read_excel("GearPredictionSanjiv.xlsx", sheet_name='FullData')
x = []
y = []
x = round(df['Engine_speed'])
y = df['Vehicle_speed']
if 'Ratio' not in df.columns or not os.path.exists('dataset.xlsx'):
    df['Ratio'] = round(x/y)

model = KMeans(n_clusters=5)

# Fit the model
model.fit(X)

# Assign a cluster to each example
yhat = model.predict(X)

# Plot
plt.scatter(yhat, X['Ratio'], c=yhat, cmap=plt.cm.coolwarm)

# Show the plot
plt.show()

</code>```

Body:
Looking at the vehicle speed in relation to the engine speed, the different slopes should give the different gears.

My initial reaction would be to say that this is a linear regression problem. You don't have enough data for anything else. Looking at the data, though, we can see that it is actually two linear regression problems:

[![Engine speed vs. vehicle speed][2]][2]

There is an inflection point at about 700 revs, so you should design a cutoff that selects one of two regression lines, depending on whether you are above or below the cutoff.

To determine the regression in Python, you can use any number of packages. In scikit-learn it looks like this:
https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html

The example given there, using the Python console, is

>>> import numpy as np
>>> from sklearn.linear_model import LinearRegression
>>> X = np.array([[1, 1], [1, 2], [2, 2], [2, 3]])
>>> # y = 1 * x_0 + 2 * x_1 + 3
>>> y = np.dot(X, np.array([1, 2])) + 3
>>> reg = LinearRegression().fit(X, y)
>>> reg.score(X, y)
1.0
>>> reg.coef_
array([1., 2.])
>>> reg.intercept_
3.0000...
>>> reg.predict(np.array([[3, 5]]))
array([16.])

Obviously you need to put your own data in X and y and in fact you would want two arrays for the two sections of your graph. You would also have two reg = LinearRegression().fit(X, y) expressions, and an if statement deciding which reg to use, depending on the input. The inflection point is at the intersection of your two regression lines.

The two regression lines have the form y = m1 x + c1 and y = m2 x + c2, where m1, m2 are the gradients of the lines and c1, c2 the intercepts. At the point of intersection m1x + c1 = m2x + c2. If you don't want to do the maths, then you can use Shapely:

import shapely
from shapely.geometry import LineString, Point

line1 = LineString([A, B])
line2 = LineString([C, D])

int_pt = line1.intersection(line2)
point_of_intersection = int_pt.x, int_pt.y

print(point_of_intersection)

(taken from this answer on Stack Overflow: How do I compute the intersection point of two lines?)

After discussion with Sanjiv, here is the updated code (adapted from here: https://machinelearningmastery.com/clustering-algorithms-with-python/)

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib
from sklearn.cluster import KMeans

matplotlib.use('TkAgg')

df = pd.read_excel("GearPredictionSanjiv.xlsx", sheet_name='FullData')
x = []
y = []
x = round(df['Engine_speed'])
y = df['Vehicle_speed']
if 'Ratio' not in df.columns or not os.path.exists('dataset.xlsx'):
    df['Ratio'] = round(x/y)

model = KMeans(n_clusters=5)

# Fit the model
model.fit(X)

# Assign a cluster to each example
yhat = model.predict(X)

# Plot
plt.scatter(yhat, X['Ratio'], c=yhat, cmap=plt.cm.coolwarm)

# Show the plot
plt.show()


问:
How to convert List to numpy Array
Here's the link to collab https://colab.research.google.com/drive/1wftAvDu_Wu2Y9ahgI1Z1FLciUH5MnSJ9

train_labels = ['GovernmentSchemes', 'GovernmentSchemes', 'GovernmentSchemes', 'GovernmentSchemes', 'CropInsurance']

```<code>training_label_seq = np.array(label_tokenizer.texts_to_sequences(train_labels))
</code>```

output coming :

```<code>[list([3]) list([3]) list([3]) ... list([2]) list([5]) list([1])]
</code>```

expected output :

```<code>[[3] [3] [3] .. [2] [5]...]
</code>```

```<code>num_epochs = 30
history = model.fit(train_padded, training_label_seq, epochs=num_epochs, validation_data=(validation_padded, validation_label_seq))
</code>```

Error =>  ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type list)

答:
I was able to recreate your issue using the below code -

Code to recreate the issue -

```<code>import numpy as np
import tensorflow as tf
print(tf.__version__)
from tensorflow.keras.preprocessing.text import Tokenizer

label_tokenizer = Tokenizer()

# Fit on a text
fit_text = "Tensorflow warriors are awesome people"
label_tokenizer.fit_on_texts(fit_text)

# Training Labels
train_labels = "Tensorflow warriors are great people"
training_label_list = np.array(label_tokenizer.texts_to_sequences(train_labels))

# Print the
print(training_label_list)
print(type(training_label_list))
print(type(training_label_list[0]))
</code>```

Output -

```<code>2.2.0
[list([9]) list([1]) list([10]) list([5]) list([3]) list([2]) list([11])
 list([7]) list([3]) list([6]) list([]) list([6]) list([4]) list([2])
 list([2]) list([12]) list([3]) list([2]) list([5]) list([]) list([4])
 list([2]) list([1]) list([]) list([4]) list([2]) list([1]) list([])
 list([]) list([2]) list([1]) list([4]) list([9]) list([]) list([8])
 list([1]) list([3]) list([8]) list([7]) list([1])]
<class 'numpy.ndarray'>
<class 'list'>
</code>```

Solution -

Replacing np.array with np.hstack will fix your problem. Your model.fit() should work fine now.

Else if you are looking for the expected output as in your question,  training_label_list = label_tokenizer.texts_to_sequences(train_labels) will give you a list of list. You can use np.array([np.array(i) for i in training_label_list]) to convert to array of array. This works only if your list of lists contains lists with same number of elements.

np.hstack Code - Code for Point number 1 in solution.

```<code>import numpy as np
import tensorflow as tf
print(tf.__version__)
from tensorflow.keras.preprocessing.text import Tokenizer

label_tokenizer = Tokenizer()

# Fit on a text
fit_text = "Tensorflow warriors are awesome people"
label_tokenizer.fit_on_texts(fit_text)

# Training Labels
train_labels = "Tensorflow warriors are great people"
training_label_list = np.hstack(label_tokenizer.texts_to_sequences(train_labels))

# Print the
print(training_label_list)
print(type(training_label_list))
print(type(training_label_list[0]))
</code>```

Output -

```<code>2.2.0
[ 9.  1. 10.  4.  2.  3. 11.  7.  2.  5.  5.  6.  3.  3. 12.  2.  3.  4.
  6.  3.  1.  3.  1.  6.  9.  8.  1.  2.  8.  7.  1.]
<class 'numpy.ndarray'>
<class 'numpy.float64'>
</code>```

Expected output as in question - Code for Point number 2 in solution.

```<code>import numpy as np
import tensorflow as tf
print(tf.__version__)
from tensorflow.keras.preprocessing.text import Tokenizer

label_tokenizer = Tokenizer()

# Fit on a text
fit_text = "Tensorflow warriors are awesome people"
label_tokenizer.fit_on_texts(fit_text)

# Training Labels
train_labels = "Tensorflow warriors are great people"
training_label_list = label_tokenizer.texts_to_sequences(train_labels)

# Print
print(training_label_list)
print(type(training_label_list))
print(type(training_label_list[0]))

# To convert elements to array
training_label_list = np.array([np.array(i) for i in training_label_list])

# Print
print(training_label_list)
print(type(training_label_list))
print(type(training_label_list[0]))
</code>```

Output -

```<code>2.2.0
[[9], [1], [10], [4], [2], [3], [11], [7], [2], [5], [], [5], [6], [3], [3], [12], [2], [3], [4], [], [6], [3], [1], [], [], [3], [1], [6], [9], [], [8], [1], [2], [8], [7], [1]]
<class 'list'>
<class 'list'>
[array([9]) array([1]) array([10]) array([4]) array([2]) array([3])
 array([11]) array([7]) array([2]) array([5]) array([], dtype=float64)
 array([5]) array([6]) array([3]) array([3]) array([12]) array([2])
 array([3]) array([4]) array([], dtype=float64) array([6]) array([3])
 array([1]) array([], dtype=float64) array([], dtype=float64) array([3])
 array([1]) array([6]) array([9]) array([], dtype=float64) array([8])
 array([1]) array([2]) array([8]) array([7]) array([1])]
<class 'numpy.ndarray'>
<class 'numpy.ndarray'>
</code>```

Hope this answers your question. Happy Learning.

Update 2/6/2020 - Anirudh_k07, As per our discussion, I had a look into your program and you are getting below error in model.fit() after using np.hstack for labels.

```<code>ValueError: Data cardinality is ambiguous:
  x sizes: 41063
  y sizes: 41429
Please provide data which shares the same first dimension.
</code>```

This error you are getting is because few of the labels have special characters like - and /. Thus on performing np.hstack(label_tokenizer.texts_to_sequences(train_labels), they are creating additional rows. You can print list of unique train_labels by using print(set(train_labels)).

Here is gist of what I am trying to say -

```<code># These Labels have special character
train_labels = ['Bio-PesticidesandBio-Fertilizers','Old/SenileOrchardRejuvenation']
training_label_seq = np.hstack(label_tokenizer.texts_to_sequences(train_labels))
print("Two labels are converted to Five :",training_label_seq)

# These Labels are fine
train_labels = ['SoilHealthCard', 'PostHarvestPreservation', 'FertilizerUseandAvailability']
training_label_seq = np.hstack(label_tokenizer.texts_to_sequences(train_labels))
print("Three labels are remain three :",training_label_seq)
</code>```

Output -

```<code>Two labels are converted to Five : [17 18 19 51 52]
Three labels are remain three : [20 36  5]
</code>```

So kindly do the proper preprocessing and eliminate these special characters in train_labels and then use np.hstack(label_tokenizer.texts_to_sequences(train_labels)) on labels. Your model.fit() should work fine after that.

Hope this answers your question. Happy Learning.

Title: How to convert List to numpy Array

Body:
Here's the link to collab https://colab.research.google.com/drive/1wftAvDu_Wu2Y9ahgI1Z1FLciUH5MnSJ9

train_labels = ['GovernmentSchemes', 'GovernmentSchemes', 'GovernmentSchemes', 'GovernmentSchemes', 'CropInsurance']

training_label_seq = np.array(label_tokenizer.texts_to_sequences(train_labels))

output coming :

[list([3]) list([3]) list([3]) ... list([2]) list([5]) list([1])]

expected output :

[[3] [3] [3] .. [2] [5]...]

num_epochs = 30
history = model.fit(train_padded, training_label_seq, epochs=num_epochs, validation_data=(validation_padded, validation_label_seq))

Error =>  ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type list)

Body:
I was able to recreate your issue using the below code -

Code to recreate the issue -

import numpy as np
import tensorflow as tf
print(tf.__version__)
from tensorflow.keras.preprocessing.text import Tokenizer

label_tokenizer = Tokenizer()

# Fit on a text
fit_text = "Tensorflow warriors are awesome people"
label_tokenizer.fit_on_texts(fit_text)

# Training Labels
train_labels = "Tensorflow warriors are great people"
training_label_list = np.array(label_tokenizer.texts_to_sequences(train_labels))

# Print the
print(training_label_list)
print(type(training_label_list))
print(type(training_label_list[0]))

Output -

2.2.0
[list([9]) list([1]) list([10]) list([5]) list([3]) list([2]) list([11])
 list([7]) list([3]) list([6]) list([]) list([6]) list([4]) list([2])
 list([2]) list([12]) list([3]) list([2]) list([5]) list([]) list([4])
 list([2]) list([1]) list([]) list([4]) list([2]) list([1]) list([])
 list([]) list([2]) list([1]) list([4]) list([9]) list([]) list([8])
 list([1]) list([3]) list([8]) list([7]) list([1])]
<class 'numpy.ndarray'>
<class 'list'>

Solution -

Replacing np.array with np.hstack will fix your problem. Your model.fit() should work fine now.

Else if you are looking for the expected output as in your question,  training_label_list = label_tokenizer.texts_to_sequences(train_labels) will give you a list of list. You can use np.array([np.array(i) for i in training_label_list]) to convert to array of array. This works only if your list of lists contains lists with same number of elements.

np.hstack Code - Code for Point number 1 in solution.

import numpy as np
import tensorflow as tf
print(tf.__version__)
from tensorflow.keras.preprocessing.text import Tokenizer

label_tokenizer = Tokenizer()

# Fit on a text
fit_text = "Tensorflow warriors are awesome people"
label_tokenizer.fit_on_texts(fit_text)

# Training Labels
train_labels = "Tensorflow warriors are great people"
training_label_list = np.hstack(label_tokenizer.texts_to_sequences(train_labels))

# Print the
print(training_label_list)
print(type(training_label_list))
print(type(training_label_list[0]))

Output -

2.2.0
[ 9.  1. 10.  4.  2.  3. 11.  7.  2.  5.  5.  6.  3.  3. 12.  2.  3.  4.
  6.  3.  1.  3.  1.  6.  9.  8.  1.  2.  8.  7.  1.]
<class 'numpy.ndarray'>
<class 'numpy.float64'>

Expected output as in question - Code for Point number 2 in solution.

import numpy as np
import tensorflow as tf
print(tf.__version__)
from tensorflow.keras.preprocessing.text import Tokenizer

label_tokenizer = Tokenizer()

# Fit on a text
fit_text = "Tensorflow warriors are awesome people"
label_tokenizer.fit_on_texts(fit_text)

# Training Labels
train_labels = "Tensorflow warriors are great people"
training_label_list = label_tokenizer.texts_to_sequences(train_labels)

# Print
print(training_label_list)
print(type(training_label_list))
print(type(training_label_list[0]))

# To convert elements to array
training_label_list = np.array([np.array(i) for i in training_label_list])

# Print
print(training_label_list)
print(type(training_label_list))
print(type(training_label_list[0]))

Output -

2.2.0
[[9], [1], [10], [4], [2], [3], [11], [7], [2], [5], [], [5], [6], [3], [3], [12], [2], [3], [4], [], [6], [3], [1], [], [], [3], [1], [6], [9], [], [8], [1], [2], [8], [7], [1]]
<class 'list'>
<class 'list'>
[array([9]) array([1]) array([10]) array([4]) array([2]) array([3])
 array([11]) array([7]) array([2]) array([5]) array([], dtype=float64)
 array([5]) array([6]) array([3]) array([3]) array([12]) array([2])
 array([3]) array([4]) array([], dtype=float64) array([6]) array([3])
 array([1]) array([], dtype=float64) array([], dtype=float64) array([3])
 array([1]) array([6]) array([9]) array([], dtype=float64) array([8])
 array([1]) array([2]) array([8]) array([7]) array([1])]
<class 'numpy.ndarray'>
<class 'numpy.ndarray'>

Hope this answers your question. Happy Learning.

Update 2/6/2020 - Anirudh_k07, As per our discussion, I had a look into your program and you are getting below error in model.fit() after using np.hstack for labels.

ValueError: Data cardinality is ambiguous:
  x sizes: 41063
  y sizes: 41429
Please provide data which shares the same first dimension.

This error you are getting is because few of the labels have special characters like - and /. Thus on performing np.hstack(label_tokenizer.texts_to_sequences(train_labels), they are creating additional rows. You can print list of unique train_labels by using print(set(train_labels)).

Here is gist of what I am trying to say -

# These Labels have special character
train_labels = ['Bio-PesticidesandBio-Fertilizers','Old/SenileOrchardRejuvenation']
training_label_seq = np.hstack(label_tokenizer.texts_to_sequences(train_labels))
print("Two labels are converted to Five :",training_label_seq)

# These Labels are fine
train_labels = ['SoilHealthCard', 'PostHarvestPreservation', 'FertilizerUseandAvailability']
training_label_seq = np.hstack(label_tokenizer.texts_to_sequences(train_labels))
print("Three labels are remain three :",training_label_seq)

Output -

Two labels are converted to Five : [17 18 19 51 52]
Three labels are remain three : [20 36  5]

So kindly do the proper preprocessing and eliminate these special characters in train_labels and then use np.hstack(label_tokenizer.texts_to_sequences(train_labels)) on labels. Your model.fit() should work fine after that.

Hope this answers your question. Happy Learning.


问:
How to get the total number of rows of a GROUP BY query?
From the PDO manual:

PDOStatement::rowCount() returns the
  number of rows affected by the last
  DELETE, INSERT, or UPDATE statement
  executed by the corresponding
  PDOStatement object.

If the last SQL statement executed by
  the associated PDOStatement was a
  SELECT statement, some databases may
  return the number of rows returned by
  that statement. However, this
  behaviour is not guaranteed for all
  databases and should not be relied on
  for portable applications.

I found that out only very recently. I had just changed my db abstraction layer to not use SELECT COUNT(1) ... anymore, because just quering the actual rows and then counting the result would be much more efficient. And now PDO doesn't support that!?

I don't use PDO for MySQL and PgSQL, but I do for SQLite. Is there a way (without completely changing the dbal back) to count rows like this in PDO? In MySQL, this would be something like this:

```<code>$q = $db->query('SELECT a, b, c FROM tbl WHERE oele = 2 GROUP BY boele');
$rows = $q->num_rows;
// and now use $q to get actual data
</code>```

With the MySQLi and PgSQL drivers, this is possible. With all PDO it isn't!?

PS. My initial solution was to extend the SQLResult->count method (my own) to replace SELECT ... FROM by SELECT COUNT(1) FROM and just return that number (very inefficient, but only for SQLite PDO). That's not good enough though, because in the example query above is a GROUP BY, which would change the meaning/function of the COUNT(1).

答:
I don't use PDO for MySQL and PgSQL, but I do for SQLite. Is there a way (without completely changing the dbal back) to count rows like this in PDO?

Accordingly to this comment, the SQLite issue was introduced by an API change in 3.x.

That said, you might want to inspect how PDO actually implements the functionality before using it.

I'm not familiar with its internals but I'd be suspicious at the idea that PDO parses your SQL (since an SQL syntax error would appear in the DB's logs) let alone tries to make the slightest sense of it in order to count rows using an optimal strategy.

Assuming it doesn't indeed, realistic strategies for it to return a count of all applicable rows in a select statement include string-manipulating the limit clause out of your SQL statement, and either of:

Running a select count() on it as a subquery (thus avoiding the issue you described in your PS);

Opening a cursor, running fetch all and counting the rows; or

Having opened such a cursor in the first place, and similarly counting the remaining rows.

A much better way to count, however, would be to execute the fully optimized query that will do so. More often than not, this means rewriting meaningful chunks of the initial query you're trying to paginate -- stripping unneeded fields and order by operations, etc.

Lastly, if your data sets are large enough that counts any kind of lag, you might also want to investigate returning the estimate derived from the statistics instead, and/or periodically caching the result in Memcache. At some point, having precisely correct counts is no longer useful...

Body:
From the PDO manual:

PDOStatement::rowCount() returns the
  number of rows affected by the last
  DELETE, INSERT, or UPDATE statement
  executed by the corresponding
  PDOStatement object.

If the last SQL statement executed by
  the associated PDOStatement was a
  SELECT statement, some databases may
  return the number of rows returned by
  that statement. However, this
  behaviour is not guaranteed for all
  databases and should not be relied on
  for portable applications.

I found that out only very recently. I had just changed my db abstraction layer to not use SELECT COUNT(1) ... anymore, because just quering the actual rows and then counting the result would be much more efficient. And now PDO doesn't support that!?

I don't use PDO for MySQL and PgSQL, but I do for SQLite. Is there a way (without completely changing the dbal back) to count rows like this in PDO? In MySQL, this would be something like this:

$q = $db->query('SELECT a, b, c FROM tbl WHERE oele = 2 GROUP BY boele');
$rows = $q->num_rows;
// and now use $q to get actual data

With the MySQLi and PgSQL drivers, this is possible. With all PDO it isn't!?

PS. My initial solution was to extend the SQLResult->count method (my own) to replace SELECT ... FROM by SELECT COUNT(1) FROM and just return that number (very inefficient, but only for SQLite PDO). That's not good enough though, because in the example query above is a GROUP BY, which would change the meaning/function of the COUNT(1).


问:
Add SSD storage to AWS EC2 reserved instance
I am new to AWS EC2 and I have just purchased a t3 medium reserved instance. I would like to add 100 GB of SSD storage to my instance and use it as the instance's primary hard disk. How can I do this? I did not see any option of adding and configuring the SSD disk when I purchased the instance.

I have purchased a linux/unix instance without any AMI. I intend to install Ubuntu 18.04 as the OS. Please advise.

答:
In AWS your SSD disk is known as an EBS Volume.

To update the volume size you would from the EC2 console want to find your current volume and right click on it. Then click modify volume and select your new size.

If you haven't created your instance yet you can specify the size during the wizard.


问:
Change default SQL Server timezone on Linux?
I have SQL Server installed on a Centos. I set the server date and time to my local date time and it is OK. But when I run getdate() in SQL Server it gives date and time based in the US time zone. I know I can get the date and with offset options but I need get the local date and time just with getdate().

How can I force SQL Server to get date and time from the operating system (Centos)?

答: I reinstall the sqlserver after setting the operating timezone and solved the problem.</p>