问:
How to perform union between sets from different rows at same column at a Dataframe
Which is the best way (fastest) to perform union between sets from different rows at same column of a Dataframe.

For example for the following dataframe:

```<code>df_input=pd.DataFrame([[1,{1,2,3}],[1,{11,12}],[2,{1111,2222}],[2,{0,99}]], columns=['name', 'set'])

    name          set
0      1     {1, 2, 3}
1      1      {11, 12}
2      2  {2222, 1111}
3      2       {0, 99}
</code>```

I would like to get:

```<code>    name                  set
0      1    {1, 2, 3, 11, 12}
1      2  {0, 99, 2222, 1111}
</code>```

And in case I have two columns wiht different sets, how can I join both columns?

For example, for this dataframe:

```<code>df_input=pd.DataFrame([[1,{1,2,3},{'a','b'}],[1,{11,12},{'j'}],[2,{1111,2222},{'m','n'}],[2,{0,99},{'p'}]], columns=['name', 'set1', 'set2'])
   name          set1    set2
0     1     {1, 2, 3}  {b, a}
1     1      {11, 12}     {j}
2     2  {2222, 1111}  {m, n}
3     2       {0, 99}     {p}
</code>```

I am looking for the way to get this as ouput:

```<code>   name                 set1       set2
0     1    {1, 2, 3, 11, 12}  {b, j, a}
1     2  {0, 99, 2222, 1111}  {m, p, n}
</code>```

Thank you.

答:
I am really not very knowleadgable in Pandas, and I'm sure there's a better way and (if you have time) you should probably wait for a better answer, but something like this seems to do the trick?

```<code>import pandas as pd
df_input=pd.DataFrame([[1,{1,2,3},{'a','b'}],[1,{11,12},{'j'}],[2,{1111,2222},{'m','n'}],[2,{0,99},{'p'}]], columns=['name', 'set1', 'set2'])

new = pd.DataFrame()
for name, agg_df in df_input.groupby('name'):
    data = {
        'name': name,
        'set1': set(),
        'set2': set(),
    }
    agg_df['set1'].apply(lambda c: data['set1'].update(c))
    agg_df['set2'].apply(lambda c: data['set2'].update(c))
    new = new.append(data, ignore_index=True)

print(new.head())
</code>```

prints:

```<code>   name                 set1       set2
0   1.0    {1, 2, 3, 11, 12}  {b, j, a}
1   2.0  {0, 99, 2222, 1111}  {p, n, m}
</code>```

There is more Python syntactic sugar that you sure can use, but that's not really pandas...

```<code>import pandas as pd
df_input=pd.DataFrame([[1,{1,2,3},{'a','b'}],[1,{11,12},{'j'}],[2,{1111,2222},{'m','n'}],[2,{0,99},{'p'}]], columns=['name', 'set1', 'set2'])

SET_COLUMNS = ('set1', 'set2')
new = pd.DataFrame()
for name, agg_df in df_input.groupby('name'):
    data = {**{'name': name}, **{set_col: set() for set_col in SET_COLUMNS}}
    for set_col in SET_COLUMNS:
        agg_df[set_col].apply(lambda c: data[set_col].update(c))
    new = new.append(data, ignore_index=True)

print(new.head())
</code>```

Body:
Which is the best way (fastest) to perform union between sets from different rows at same column of a Dataframe.

For example for the following dataframe:

df_input=pd.DataFrame([[1,{1,2,3}],[1,{11,12}],[2,{1111,2222}],[2,{0,99}]], columns=['name', 'set'])

    name          set
0      1     {1, 2, 3}
1      1      {11, 12}
2      2  {2222, 1111}
3      2       {0, 99}

I would like to get:

    name                  set
0      1    {1, 2, 3, 11, 12}
1      2  {0, 99, 2222, 1111}

And in case I have two columns wiht different sets, how can I join both columns?

For example, for this dataframe:

df_input=pd.DataFrame([[1,{1,2,3},{'a','b'}],[1,{11,12},{'j'}],[2,{1111,2222},{'m','n'}],[2,{0,99},{'p'}]], columns=['name', 'set1', 'set2'])
   name          set1    set2
0     1     {1, 2, 3}  {b, a}
1     1      {11, 12}     {j}
2     2  {2222, 1111}  {m, n}
3     2       {0, 99}     {p}

I am looking for the way to get this as ouput:

   name                 set1       set2
0     1    {1, 2, 3, 11, 12}  {b, j, a}
1     2  {0, 99, 2222, 1111}  {m, p, n}

Thank you.

Body:
I am really not very knowleadgable in Pandas, and I'm sure there's a better way and (if you have time) you should probably wait for a better answer, but something like this seems to do the trick?

import pandas as pd
df_input=pd.DataFrame([[1,{1,2,3},{'a','b'}],[1,{11,12},{'j'}],[2,{1111,2222},{'m','n'}],[2,{0,99},{'p'}]], columns=['name', 'set1', 'set2'])

new = pd.DataFrame()
for name, agg_df in df_input.groupby('name'):
    data = {
        'name': name,
        'set1': set(),
        'set2': set(),
    }
    agg_df['set1'].apply(lambda c: data['set1'].update(c))
    agg_df['set2'].apply(lambda c: data['set2'].update(c))
    new = new.append(data, ignore_index=True)

print(new.head())

prints:

   name                 set1       set2
0   1.0    {1, 2, 3, 11, 12}  {b, j, a}
1   2.0  {0, 99, 2222, 1111}  {p, n, m}

There is more Python syntactic sugar that you sure can use, but that's not really pandas...

import pandas as pd
df_input=pd.DataFrame([[1,{1,2,3},{'a','b'}],[1,{11,12},{'j'}],[2,{1111,2222},{'m','n'}],[2,{0,99},{'p'}]], columns=['name', 'set1', 'set2'])

SET_COLUMNS = ('set1', 'set2')
new = pd.DataFrame()
for name, agg_df in df_input.groupby('name'):
    data = {**{'name': name}, **{set_col: set() for set_col in SET_COLUMNS}}
    for set_col in SET_COLUMNS:
        agg_df[set_col].apply(lambda c: data[set_col].update(c))
    new = new.append(data, ignore_index=True)

print(new.head())


来源: stackexchange

问:
Problem deploying "express-gateway" to Google Kubernetes Engine with Ingress service type
I'm deploying a microservices nodejs express application to GKE, I successfully deployed the services (one as Ingress service with static IP @ and one as NodePort service which will not be exposed to Internet).

Now I'm trying to deploy the api gateway which is an express-gateway application, I deployed it in first time as a LoadBalancer service and it's working , here is my service code :

```<code>apiVersion: v1
kind: Service
metadata:
  name: express-gateway-balancer
spec:
  selector:
    app: express-gateway
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
</code>```

Now I must transform the trafic to HTTPS to be able to call the API endpoints front my react frontend application which is running with HTTPS, so I did a research and the way is to expose the api gateway application as an Ingress not LoadBalancer.

So I did the changes (the same I did with the rest of my deployments)and created a static IP @ "gateway-ip" to expose it via Ingress :

Here is my service.yaml file :

```<code>apiVersion: v1
kind: Service
metadata:
  name: express-gateway-service
spec:
  selector:
    app: express-gateway
  type: NodePort
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
</code>```

My Ingress.yaml file :

```<code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gateway-service-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "gateway-ip"
spec:
  backend:
    serviceName: express-gateway-service
    servicePort: 80
</code>```

Here is my deployment file :

```<code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: express-gateway
  labels:
    app: express-gateway
spec:
  selector:
    matchLabels:
      app: express-gateway
  replicas: 3
  template:
    metadata:
      labels:
        app: express-gateway
    spec:
      containers:
        - name:  express-gateway
          image: gcr.io/PROJECT_ID/apigatewayservice:v1
          ports:
            - name: gateway
              containerPort: 8080
            - name: admin
              containerPort: 9876
</code>```

Here are screenshot of the error I got :

Any solutions ? Thanks

答:
Do you have a default endpoint of all your scheduled pods? I mean "/" ? Ingress does the health check readiness prob inbuilt that checks if "/" is reachable or not. If not, it will declare it as unhealthy.
Hope this helps.</p>

ContentLicense: CC BY-SA 4.0

Body:
I'm deploying a microservices nodejs express application to GKE, I successfully deployed the services (one as Ingress service with static IP @ and one as NodePort service which will not be exposed to Internet).

Now I'm trying to deploy the api gateway which is an express-gateway application, I deployed it in first time as a LoadBalancer service and it's working , here is my service code :

apiVersion: v1
kind: Service
metadata:
  name: express-gateway-balancer
spec:
  selector:
    app: express-gateway
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

Now I must transform the trafic to HTTPS to be able to call the API endpoints front my react frontend application which is running with HTTPS, so I did a research and the way is to expose the api gateway application as an Ingress not LoadBalancer.

So I did the changes (the same I did with the rest of my deployments)and created a static IP @ "gateway-ip" to expose it via Ingress :

Here is my service.yaml file :

apiVersion: v1
kind: Service
metadata:
  name: express-gateway-service
spec:
  selector:
    app: express-gateway
  type: NodePort
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

My Ingress.yaml file :

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gateway-service-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "gateway-ip"
spec:
  backend:
    serviceName: express-gateway-service
    servicePort: 80

Here is my deployment file :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: express-gateway
  labels:
    app: express-gateway
spec:
  selector:
    matchLabels:
      app: express-gateway
  replicas: 3
  template:
    metadata:
      labels:
        app: express-gateway
    spec:
      containers:
        - name:  express-gateway
          image: gcr.io/PROJECT_ID/apigatewayservice:v1
          ports:
            - name: gateway
              containerPort: 8080
            - name: admin
              containerPort: 9876

Here are screenshot of the error I got :

Any solutions ? Thanks


问:
Using AWS DynamoDB DAX from local development machine
I'm trying out the DAX client for my application currently using DynamoDB and found out it's only supported within a VPC. The DAX endpoint is in the 172.31.x.y range.

Does this mean it's impossible to test DAX features from my local development machine, or is it possible to configure DAX to be reachable from the Internet for testing purposes in some way? Or am I stuck with SSH-ing into an EC2 in the VPC?

答:
It is possible to access DAX from outside your cluster if you create a VPC endpoint and nlb

```<code>import com.amazon.dax.client.dynamodbv2.DaxClient;
AmazonDynamoDB amazonDynamoDb = new DaxClient(
                "vpce-XXX-YYY.vpce-svc-ZZZ.us-west-2.vpce.amazonaws.com",
                8111, region, credentials);
</code>```

more details are here : https://stackoverflow.com/a/62094056/278266

ContentLicense: CC BY-SA 3.0

Body:
It is possible to access DAX from outside your cluster if you create a VPC endpoint and nlb

import com.amazon.dax.client.dynamodbv2.DaxClient;
AmazonDynamoDB amazonDynamoDb = new DaxClient(
                "vpce-XXX-YYY.vpce-svc-ZZZ.us-west-2.vpce.amazonaws.com",
                8111, region, credentials);

more details are here : https://stackoverflow.com/a/62094056/278266


问:
How to run Artisan command without a console
Someone know how to start command without console

Let me explain
After the file is downloaded successfully, you need to run the command php artisan db:seed
I tried this option but I got an error as a result
Maybe someone knows another solution to this problem
I will be very grateful
I tried this option but I got an error as a result

```<code>$file = file($request->file->getRealPath());
$fileName = resource_path('upload/csv/accounts/' . date('y-m-d-H-i-s') . '.csv');
$path = file_put_contents($fileName, $file);
return redirect()->route('admin.accounts');
</code>```

I tried this option

```<code>$file = file($request->file->getRealPath());
$fileName = resource_path('upload/csv/accounts/' . date('y-m-d-H-i-s') . '.csv');
$path = file_put_contents($fileName, $file);
return redirect()->route('admin.accounts');
Artisan::call('db:seed');
</code>```

And this

```<code>$file = file($request->file->getRealPath());
$fileName = resource_path('upload/csv/accounts/' . date('y-m-d-H-i-s') . '.csv');
$path = file_put_contents($fileName, $file);
return redirect()->route('admin.accounts')->with(Artisan::call('db:seed'));
</code>```

答:
I think this option might work. but put artisan call above return. any lines of code after return will be ignored.</p>

```<code>$file = file($request->file->getRealPath());
$fileName = resource_path('upload/csv/accounts/' . date('y-m-d-H-i-s') . '.csv');
$path = file_put_contents($fileName, $file);
Artisan::call('db:seed');// <---- this line
return redirect()->route('admin.accounts');
</code>```

Body:
Someone know how to start command without console

Let me explain
After the file is downloaded successfully, you need to run the command php artisan db:seed
I tried this option but I got an error as a result
Maybe someone knows another solution to this problem
I will be very grateful
I tried this option but I got an error as a result

$file = file($request->file->getRealPath());
$fileName = resource_path('upload/csv/accounts/' . date('y-m-d-H-i-s') . '.csv');
$path = file_put_contents($fileName, $file);
return redirect()->route('admin.accounts');

I tried this option

$file = file($request->file->getRealPath());
$fileName = resource_path('upload/csv/accounts/' . date('y-m-d-H-i-s') . '.csv');
$path = file_put_contents($fileName, $file);
return redirect()->route('admin.accounts');
Artisan::call('db:seed');

And this

$file = file($request->file->getRealPath());
$fileName = resource_path('upload/csv/accounts/' . date('y-m-d-H-i-s') . '.csv');
$path = file_put_contents($fileName, $file);
return redirect()->route('admin.accounts')->with(Artisan::call('db:seed'));

Body:
I think this option might work. but put artisan call above return. any lines of code after return will be ignored.

$file = file($request->file->getRealPath());
$fileName = resource_path('upload/csv/accounts/' . date('y-m-d-H-i-s') . '.csv');
$path = file_put_contents($fileName, $file);
Artisan::call('db:seed');// <---- this line
return redirect()->route('admin.accounts');


问:
"smile" classifiers in Google Earth Engine
What are the "smile" type classifiers in Google Earth Engine API? I see for example ee.classifier.NaiveBayes and ee.classifier.SmileNaiveBayes with exactly the same description, and I couldn't find anything else on the net about the "smile" thing.</p>

答:
Apparently they are to improve the 'performance and stability'. The old classifiers are going to be removed eventually. This is all according to the GEE Dev team.

https://groups.google.com/d/msg/google-earth-engine-developers/jXSgqCerhbs/sZOT5T2eAwAJ

The actual GEE site has no explanation about this that I could find. This is what the GEE Google Group announcement says.

Greetings Earth Engine developers!

As part of our ongoing efforts to improve Earth Engine, we will be
  migrating classifiers to a new system.

The new versions of classifiers are:

ee.Classifier.smileCart replaces ee.Classifier.cart.

ee.Classifer.smileRandomForest replaces ee.Classifier.randomForest.

ee.Classifier.smileNaiveBayes replaces ee.Classifer.naiveBayes.

ee.Classifer.libsvm replaces ee.Classifier.svm.

In many cases, the new versions have the same parameter sets as the
  old versions, though for some there are differences. We expect that
  performance and stability of the new versions should be better than
  the old versions.

The new versions of the CART and RandomForest classifiers allow access
  (via explain()) to more data than the old versions, including variable
  importance. Note that smileNaiveBayes truncates its inputs to integers
  (the fractional part is lost). If this is an issue, you may find that
  scaling the input values (e.g. by 10000) gives good results.

You should begin to migrate all code that uses the old classifiers to
  their new equivalents. In the future, the old classifiers will be
  removed. To avoid any break in service when those classifiers are
  eventually removed, please update to the new classifiers. If you have
  trouble with the smile transition, please post questions here.

The Google Earth Engine Team


问:
Strict Regex Validation on an input string
I have a quick regex verification for an input string.

This is an example string format I am trying to achieve: ABC=10:1;

What I currently have: var reg = new RegExp('(([AWT]\\w\\S=)?(\\d{0,9}:)?(\\d{0,9});)');

With the above regex I am able to get the string, however all these strings are also getting accepted...

ABC=

ABC=a:a;

ABC=#:#;

What I need: Three letter string = Any number(0-9) : Any number(0-9) ;

Unacceptable criteria (like the examples above)

Anything non-numerical after the equals sign and after the colon should be considered unacceptable.

Anything after the Semi-colon.

Anything before the Three letter string.

Any help would be appreciated!

Thank you.

Updates

According to the answers below, "^[a-zA-Z]{3}=\d+:\d+;$" works perfectly fine, however when I test it with my code, it's invalid.

Here is the code:

```<code>var reg = new RegExp("^[a-zA-Z]{3}=\d+:\d+;$");
var x = $('td').find('input')[7].value;
console.log(x); // AWT=10:15;
        if (!x.match(reg)) {
            return [false, "Stop"];
            console.log("Invalid");
        } else {
            return [true, 'Success'];
            console.log("Valid");
       }
</code>```

The above code always spits out Invalid even though I tested the regex.

I don't know whats wrong here.

答:
Looks like your regexp could be simplified to this:

```<code>"^[a-zA-Z]{3}=\d+:\d+;$"
</code>```

Where:

^ and $ - the beginning and the end of the line

[a-zA-Z] - any characters from the range (latin characters in this case)

{3} - exact number of the previous symbols (3 any latin characters in this case)

\d - any digit

+ - repeat last symbol one or more times

symbols =:; don't have to be escaped in this case

UPD:
If you need only one digit before the colon and one after the colon, then you shouldn't use + signs, just like in Cary Swoveland comment. But, according the example, there could be arbitrary length numbers here

Body:
I have a quick regex verification for an input string.

This is an example string format I am trying to achieve: ABC=10:1;

What I currently have: var reg = new RegExp('(([AWT]\\w\\S=)?(\\d{0,9}:)?(\\d{0,9});)');

With the above regex I am able to get the string, however all these strings are also getting accepted...

ABC=

ABC=a:a;

ABC=#:#;

What I need: Three letter string = Any number(0-9) : Any number(0-9) ;

Unacceptable criteria (like the examples above)

Anything non-numerical after the equals sign and after the colon should be considered unacceptable.

Anything after the Semi-colon.

Anything before the Three letter string.

Any help would be appreciated!

Thank you.

Updates

According to the answers below, "^[a-zA-Z]{3}=\d+:\d+;$" works perfectly fine, however when I test it with my code, it's invalid.

Here is the code:

var reg = new RegExp("^[a-zA-Z]{3}=\d+:\d+;$");
var x = $('td').find('input')[7].value;
console.log(x); // AWT=10:15;
        if (!x.match(reg)) {
            return [false, "Stop"];
            console.log("Invalid");
        } else {
            return [true, 'Success'];
            console.log("Valid");
       }

The above code always spits out Invalid even though I tested the regex.

I don't know whats wrong here.

Body:
Looks like your regexp could be simplified to this:

"^[a-zA-Z]{3}=\d+:\d+;$"

Where:

^ and $ - the beginning and the end of the line

[a-zA-Z] - any characters from the range (latin characters in this case)

{3} - exact number of the previous symbols (3 any latin characters in this case)

\d - any digit

+ - repeat last symbol one or more times

symbols =:; don't have to be escaped in this case

UPD:
If you need only one digit before the colon and one after the colon, then you shouldn't use + signs, just like in Cary Swoveland comment. But, according the example, there could be arbitrary length numbers here


问:
How to set the minimum length to 3 in the Lucene FrenchAnalyzer?
I can't figure out how to set the minimum character length to 3 in the Lucene French analyzer I create as below in Java:

```<code>private final FrenchAnalyzer analyzer = new FrenchAnalyzer();
</code>```

Does anybody know how to do this?

Thanks,
Christophe

答:
The FrenchAnalyzer is declared as final, so it can't be extended. I would work around this by copying the source and adding a LengthFilter to the copy.

The steps are as follows (assuming you are using the latest version of Lucene):

Get a copy of the source class from GitHub here.

Rename the class as appropriate (e.g. CustomFrenchAnalyzer) and change all the class's constructors accordingly.

Add the following import:

```<code>import org.apache.lucene.analysis.miscellaneous.LengthFilter;
</code>```

In the createComponents and normalize methods, add in the new length filter. For example, assuming you want to remove words which are less than 3 characters, as per your question:

```<code>@Override
protected TokenStream normalize(String fieldName, TokenStream in) {
    TokenStream result = new ElisionFilter(in, DEFAULT_ARTICLES);
    result = new LowerCaseFilter(result);
    // the following line is added:
    result = new LengthFilter(result, 3, 9999);
    return result;
}
</code>```

Do the same for the createComponents method, also.

This way, your custom class will automatically take advantage of the other features built into the French Analyzer:

the French elision filter

the default set of French stop-words (bundled with the Lucene JAR).

and so on.

And it can be constructed using any of the analyzer's non-empty constructors, if needed.

If you are not using a compatible version of Lucene, you will have to track down the relevant Git branch, or download the source JAR from the Lucene download page and grab the source from there.

Body:
I can't figure out how to set the minimum character length to 3 in the Lucene French analyzer I create as below in Java:

private final FrenchAnalyzer analyzer = new FrenchAnalyzer();

Does anybody know how to do this?

Thanks,
Christophe

Body:
The FrenchAnalyzer is declared as final, so it can't be extended. I would work around this by copying the source and adding a LengthFilter to the copy.

The steps are as follows (assuming you are using the latest version of Lucene):

Get a copy of the source class from GitHub here.

Rename the class as appropriate (e.g. CustomFrenchAnalyzer) and change all the class's constructors accordingly.

Add the following import:

import org.apache.lucene.analysis.miscellaneous.LengthFilter;

In the createComponents and normalize methods, add in the new length filter. For example, assuming you want to remove words which are less than 3 characters, as per your question:

@Override
protected TokenStream normalize(String fieldName, TokenStream in) {
    TokenStream result = new ElisionFilter(in, DEFAULT_ARTICLES);
    result = new LowerCaseFilter(result);
    // the following line is added:
    result = new LengthFilter(result, 3, 9999);
    return result;
}

Do the same for the createComponents method, also.

This way, your custom class will automatically take advantage of the other features built into the French Analyzer:

the French elision filter

the default set of French stop-words (bundled with the Lucene JAR).

and so on.

And it can be constructed using any of the analyzer's non-empty constructors, if needed.

If you are not using a compatible version of Lucene, you will have to track down the relevant Git branch, or download the source JAR from the Lucene download page and grab the source from there.


问:
It is possible to push a docker image inside a docker container?
I have a docker-in-docker container, which we'll call "dind". And a large image, which we'll call "LI".

Dind is responsible for make some docker runs calls. I need to run LI inside Dind. Normally, I push LI to a registry and pull it inside Dind. But I have a bad internet network. it takes a whole life to do this...

So, I was thinking if it is possible to push a docker image inside a docker container.

答: You can use docker registry container in your localhost and pull image from it in dind. Here is the link: https://hub.docker.com/_/registry</p>


问:
MongoError: Unrecognized pipeline stage name: '$expr' on using $divide and $expr in aggregation
New to Mongodb.

What I'm attemping to do is create a formula that returns documents if the rent value is $gte to 1% of the price value.

Right now I have both price and rent values stored in the database but not the 1% value.

I tried the code below but I get this error
MongoError: Unrecognized pipeline stage name: '$expr'

Here's my code

```<code>agg = [
        {
            "$set": {
                "onePercent": { '$divide': ["$price", 100] }
            }
        },
        {
            "$expr": {
                $gte: ["$rent", "$onePercent",]
            }
        },
    ]
    let res = client.db('myDb').collection('myCollection').aggregate(agg)
</code>```

How can I make a query that would return the documents I'm attempting to select?

答:
You're using $expr to compare two fields inside a document. Where $expr is an operator but not an aggregation stage. So operators needs to be used inside stages, can't stand on their own in an aggregation pipeline [{ stage1: {operators...} }, { stage2: {operators...} },...,{ stageN: {operators...} }].

MongoDB's aggregation stage $match helps to filter documents based on condition.

```<code>{ $match: { $expr: { $gte: ["$rent", "$onePercent"] } } } // your filter condition { $gte: ["$rent", "$onePercent"] }
</code>```

Body:
New to Mongodb.

What I'm attemping to do is create a formula that returns documents if the rent value is $gte to 1% of the price value.

Right now I have both price and rent values stored in the database but not the 1% value.

I tried the code below but I get this error
MongoError: Unrecognized pipeline stage name: '$expr'

Here's my code

agg = [
        {
            "$set": {
                "onePercent": { '$divide': ["$price", 100] }
            }
        },
        {
            "$expr": {
                $gte: ["$rent", "$onePercent",]
            }
        },
    ]
    let res = client.db('myDb').collection('myCollection').aggregate(agg)

How can I make a query that would return the documents I'm attempting to select?

Body:
You're using $expr to compare two fields inside a document. Where $expr is an operator but not an aggregation stage. So operators needs to be used inside stages, can't stand on their own in an aggregation pipeline [{ stage1: {operators...} }, { stage2: {operators...} },...,{ stageN: {operators...} }].

MongoDB's aggregation stage $match helps to filter documents based on condition.

{ $match: { $expr: { $gte: ["$rent", "$onePercent"] } } } // your filter condition { $gte: ["$rent", "$onePercent"] }


问:
TabIndex vs. KeyboardNavigation.TabIndex in WPF
What is the difference between TabIndex and KeyboardNavigation.TabIndex in WPF? When to use each?

答:
Some controls like CheckBox have TabIndex property but not all controls have this property, but you may want them to have focus and participate in focus navigation, attached property KeyboardNavigation.TabIndex can be used on in such cases. An example of such control is Hyperlink

Apart from this KeyboardNavigation class provides a lot other functionality to set focus navigation, like tab navigation behavior, KeyboardNavigationMode etc.

The navigation behavior of a
  navigation container can be changed by
  setting the attached
  KeyboardNavigation properties
  TabNavigation, ControlTabNavigation,
  and DirectionalNavigation. These
  properties are of type
  KeyboardNavigationMode and the
  possible values are Continue, Local,
  Contained, Cycle, Once, and None. The
  default value is Continue, which means
  the element is not a navigation
  container.

http://msdn.microsoft.com/en-us/library/aa969768.aspx#Keyboard_Navigation


问:
Snowflake .net connector certificate error
I am having trouble using .net connector (https://github.com/snowflakedb/snowflake-connector-net). It works from browser/Tableau/Excel. For tableau/excel I have to pass CaBundleFile.

Is there a way to pass a certificate file (CaBundleFile) using the .net connector?

I am getting following error.

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: Could not
 establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

答:
Unlike the ODBC driver, Snowflake doesn’t perform OCSP checking for .NET driver which uses the underlying .NET framework for checking validity of the HTTPS certificate.

One thing that might help is looking through similar GitHub issues for ideas (they discuss upgrading and network traces):
https://github.com/snowflakedb/snowflake-connector-net/issues/6

You might also try looking at some of the Microsoft docs and forums for the error on some troubleshooting ideas:
https://learn.microsoft.com/en-us/archive/blogs/jpsanders/troubleshooting-asp-net-the-remote-certificate-is-invalid-according-to-the-validation-procedure


问:
Angular @Input() value is not changed when the value is same
When I run this program, initially I get the output as false, 5, 500 as I have initialized them in the child component like that, but when I try to click on update button, I am not able to revert to the previous values. I am expecting the output to be true, 10, 1000, but I am getting it as false, 5, 1000. Only the number which was different got changed.

How can I solve this issue so that I can get the values whatever I set in the parent component?

Link to stackblitz.

app-component.html

```<code><span (click)="clickme()">Update data</span>
</code>```

app-component.ts

```<code>export class AppComponent  {
  public parentBool: boolean;
  public parentNum: number;
  public p2: number;

  public ngOnInit(): void {
    this.parentBool = true;
    this.parentNum = 10;
    this.p2 = 100;
  }

  public clickme(): void {
    this.parentBool = true;
    this.parentNum = 10;
    this.p2 = 1000;
  }
}
</code>```

hello.component.ts

```<code>@Component({
  selector: 'hello-comp',
  template: `
    <div>
     Boolean value is - {{boolChild}} <br/>
     Number value is - {{numChild}} <br />
     Second number is - {{numChild2}}
    </div>
  `,
  changeDetection: ChangeDetectionStrategy.OnPush
})
export class MyComponent {
  @Input() boolChild: boolean;
  @Input() numChild: number;
  @Input() numChild2: number;

  public ngOnInit(): void {
    this.boolChild = false;
    this.numChild = 5;
    this.numChild2 = 500;
  }
  constructor(private _cd: ChangeDetectorRef){}

    public ngOnChanges(changes: {}): void {
      // this._cd.detectChanges();
      console.log(changes);
    }
}
</code>```

答:
Input bindings occur right before ngOnInit, you’re overriding the values in ngOnInit.

Put your defaults in the declaration:

```<code>@Input() numChild: number = 5;
</code>```

Like that

Body:
When I run this program, initially I get the output as false, 5, 500 as I have initialized them in the child component like that, but when I try to click on update button, I am not able to revert to the previous values. I am expecting the output to be true, 10, 1000, but I am getting it as false, 5, 1000. Only the number which was different got changed.

How can I solve this issue so that I can get the values whatever I set in the parent component?

Link to stackblitz.

app-component.html

Update data

app-component.ts

export class AppComponent  {
  public parentBool: boolean;
  public parentNum: number;
  public p2: number;

  public ngOnInit(): void {
    this.parentBool = true;
    this.parentNum = 10;
    this.p2 = 100;
  }

  public clickme(): void {
    this.parentBool = true;
    this.parentNum = 10;
    this.p2 = 1000;
  }
}

hello.component.ts

@Component({
  selector: 'hello-comp',
  template: `

     Boolean value is - {{boolChild}}

     Number value is - {{numChild}}

     Second number is - {{numChild2}}

  `,
  changeDetection: ChangeDetectionStrategy.OnPush
})
export class MyComponent {
  @Input() boolChild: boolean;
  @Input() numChild: number;
  @Input() numChild2: number;

  public ngOnInit(): void {
    this.boolChild = false;
    this.numChild = 5;
    this.numChild2 = 500;
  }
  constructor(private _cd: ChangeDetectorRef){}

    public ngOnChanges(changes: {}): void {
      // this._cd.detectChanges();
      console.log(changes);
    }
}

Body:
Input bindings occur right before ngOnInit, you’re overriding the values in ngOnInit.

Put your defaults in the declaration:

@Input() numChild: number = 5;

Like that


问:
Ansible GCP IAP tunnel
I’m trying to connect to a GCP compute instance through IAP. I have a service account with permissions.

I have tried the following

Basic ansible ping,ansible -vvvv GCP -m ping, which errors because the host name is not found bc I do not have an external ip

I have set ssh_executeable=wrapper.sh like here

Number 2 is almost working but regexing commands are hacky.

Is there a native ansible solution?

Edit: The gcp_compute dynamic inventory does work for pinging instances but it does not work for managing the instances.

Ansible does NOT support package or system management while tunneling through IAP.

答: gce dynamic inventory does not work unless all the inventory are publicly accessible. For private ip, the tunnel is not invoked when ansible commands are executed. The gce dynamic inventory will return inventory, but you can't actually send commands if behind a tunnel and private IP only. The only work around i could find is to have the ssh binary point at a custom script which calls the gcloud wrapper.</p>

Title: Ansible GCP IAP tunnel


问:
How to parallelize list-comprehension calculations in Python?
Both list comprehensions and map-calculations should -- at least in theory -- be relatively easy to parallelize: each calculation inside a list-comprehension could be done independent of the calculation of all the other elements. For example in the expression

```<code>[ x*x for x in range(1000) ]
</code>```

each x*x-Calculation could (at least in theory) be done in parallel.

My question is: Is there any Python-Module / Python-Implementation / Python Programming-Trick to parallelize a list-comprehension calculation (in order to use all 16 / 32 / ... cores or distribute the calculation over a Computer-Grid or over a Cloud)?

答:
For shared-memory parallelism, I recommend joblib:</p>

```<code>from joblib import delayed, Parallel

def square(x): return x*x
values = Parallel(n_jobs=NUM_CPUS)(delayed(square)(x) for x in range(1000))
</code>```

ContentLicense: CC BY-SA 2.5

Body:
Both list comprehensions and map-calculations should -- at least in theory -- be relatively easy to parallelize: each calculation inside a list-comprehension could be done independent of the calculation of all the other elements. For example in the expression

[ x*x for x in range(1000) ]

each x*x-Calculation could (at least in theory) be done in parallel.

My question is: Is there any Python-Module / Python-Implementation / Python Programming-Trick to parallelize a list-comprehension calculation (in order to use all 16 / 32 / ... cores or distribute the calculation over a Computer-Grid or over a Cloud)?

Body:
For shared-memory parallelism, I recommend joblib:

from joblib import delayed, Parallel

def square(x): return x*x
values = Parallel(n_jobs=NUM_CPUS)(delayed(square)(x) for x in range(1000))


问:
How to integrate Mailchimp into wordpress using Elementor
I've a website built in wordpress with Elementor.
And I've created a mailchimp account and I've updated the API key from Mailchimp to wordpress in the settings.

So, now how can I integrate the the subscribe form in elementor in wordpress

答:
Open the page that has you form widget and select the form widget.

On the left of the page, on the form option, open ACTIONS AFTER SUBMIT.

Select MAILCHIMP to add it to the list of actions.

A new option for MAILCHIMP will show just bellow. Open it.

On API Key, select CUSTOM

Paste on the CUSTOM API FIELD the api key you copied from Mailchimp site.

Select the final details of how you want it to be saved on the list.

Define the key mapping, defining the email from the form field to the email field of mailchimp.

Test it.


问:
(ε, δ)-differential privacy accuracy/privacy trade-off
I'm using Renyi (ε, δ)-differential privacy as implemented in PyTorch DNNs. I do this to guarantee the privacy of the training set to the degree ε. In order to implement this, I use PyTorch-DP. As the network fits more and more on the trainingset and the accuracy goes up, the amount of privacy decreases. However, at some turning point, the privacy decreases and the accuracy decreases too; I assume that the accuracy eventually goes down because too much (privacy) noise is added to the mean of the clipped gradients over a batch of training data and/or because it overfits on the training set.

Now what I want to achieve is the following: I want to train a model to reach (ε, δ)-differential privacy with a differing value of ε (i.e. 0.5, 1, 1.5, 2). Mind that the bigger the ε-value, the less privacy the model guarantees. In an ideal situation, the accuracy would improve as ε increases. Yet this does not happen because after that turning point, the accuracy decreases. My question is, is there any way to guarantee/influence this so that the accuracy is still at a near-optimum at the point that the algorithm reaches the specified ε. Or is this not really as controllable as I'd like and is the best way to achieve this through empirically changing the learning rate, or changing the amount of additive clipping noise as well as the number of epochs so that more/less privacy is added each epoch but for a shorter/longer time.

I'm asking for ideas really.

答: I don't know of any magical way to guarantee that the privacy is at a near-optimum. What I have done (and heard other people doing) that has worked fairly well is to do some hyper parameter tuning with different values of epsilon to get the best accuracy for that epsilon, so I would recommend you go for that. Also, as an interesting side note, at a small level sometimes the noise added in the process of making an algorithm DP can help improve the accuracy a bit by helping prevent overfitting (not sure if that is what is happening in your case though).</p>


问:
How to distinguish different responses from useFetch custom hook
Basically in my App.js, I need to call useFetch twice to display spinner/data/error for two tables.

How do I go about distinguish which spinner/data/error is for which table? Because in useEffect I'm returning
{ data, loading, error }, and in App.js, I'm getting the value like so const { data, loading, error } = useFetch(url_order, date). But I want const { data_table1, loading_table1, error_table1 } = useFetch(url_order, date) instead.

This is my useFetch custom hook

```<code>import { useState, useEffect } from "react";

export default function useFetch(url, date) {
  const [data, setData] = useState([]);
  const [loading, setLoading] = useState(true);
  const [error, setError] = useState(false);

  useEffect(() => {
    const doFetch = async () => {
      setLoading(true);
      try {
        const res = await fetch(url);
        const json = await res.json;
        setData(json.result.result);
      } catch (error) {
        setError(true);
      }
      setLoading(false);
    };
    doFetch();
  }, [date]);

  return { data, loading, error };
}
</code>```

This is my App.js

```<code>import React from "react";
import useFetch from "./hooks/useFetch";
import OrderTable from "./OrderTable";
import IngredientTable from "./IngredientTable";

const App = () => {
  const { data, loading, error } = useFetch(url_order, date);
  const { data, loading, error } = useFetch(url_ingredient, date);

  return (
    <div>
      {loading ? (
        <BeatLoader css={override} color={"#36D7B7"} loading={loading} />
      ) : error ? (
        <h3>Failed to fetch data for Order's table</h3>
      ) : (
        <OrderTable data={data} />
      )}

      {loading ? (
        <BeatLoader css={override} color={"#36D7B7"} loading={loading} />
      ) : error ? (
        <h3>Failed to fetch data for Ingredient's table</h3>
      ) : (
        <IngredientTable data={data} />
      )}
    </div>
  );
};

export default App;
</code>```

答:
Your useFetch is nothing but a function at the end of the day. You can take advantage of this - Inside your app component, set the initial state with different names by calling the useFetch like so:

```<code>const [orderData,setOrderData] = useState(useFetch(url_order, date));
const [ingredientData,setIngredientData] = useState(useFetch(url_ingredient, date));
</code>```

Now orderData and ingredientData are objects with orderData.data, orderData.loading and orderData.error and so on.

Although this is not very clean. Another way is to just have different variable names for each data,loading,error by suffixing them with numbers:

```<code> const { data1, loading1, error1 } = useFetch(url_order, date);
  const { data2, loading2, error2 } = useFetch(url_ingredient, date);
</code>```

Body:
Basically in my App.js, I need to call useFetch twice to display spinner/data/error for two tables.

How do I go about distinguish which spinner/data/error is for which table? Because in useEffect I'm returning
{ data, loading, error }, and in App.js, I'm getting the value like so const { data, loading, error } = useFetch(url_order, date). But I want const { data_table1, loading_table1, error_table1 } = useFetch(url_order, date) instead.

This is my useFetch custom hook

import { useState, useEffect } from "react";

export default function useFetch(url, date) {
  const [data, setData] = useState([]);
  const [loading, setLoading] = useState(true);
  const [error, setError] = useState(false);

  useEffect(() => {
    const doFetch = async () => {
      setLoading(true);
      try {
        const res = await fetch(url);
        const json = await res.json;
        setData(json.result.result);
      } catch (error) {
        setError(true);
      }
      setLoading(false);
    };
    doFetch();
  }, [date]);

  return { data, loading, error };
}

This is my App.js

import React from "react";
import useFetch from "./hooks/useFetch";
import OrderTable from "./OrderTable";
import IngredientTable from "./IngredientTable";

const App = () => {
  const { data, loading, error } = useFetch(url_order, date);
  const { data, loading, error } = useFetch(url_ingredient, date);

  return (

      {loading ? (
        <BeatLoader css={override} color={"#36D7B7"} loading={loading} />
      ) : error ? (

Failed to fetch data for Order's table

      ) : (
        <OrderTable data={data} />
      )}

      {loading ? (
        <BeatLoader css={override} color={"#36D7B7"} loading={loading} />
      ) : error ? (

Failed to fetch data for Ingredient's table

      ) : (
        <IngredientTable data={data} />
      )}

  );
};

export default App;

Body:
Your useFetch is nothing but a function at the end of the day. You can take advantage of this - Inside your app component, set the initial state with different names by calling the useFetch like so:

const [orderData,setOrderData] = useState(useFetch(url_order, date));
const [ingredientData,setIngredientData] = useState(useFetch(url_ingredient, date));

Now orderData and ingredientData are objects with orderData.data, orderData.loading and orderData.error and so on.

Although this is not very clean. Another way is to just have different variable names for each data,loading,error by suffixing them with numbers:

 const { data1, loading1, error1 } = useFetch(url_order, date);
  const { data2, loading2, error2 } = useFetch(url_ingredient, date);


问:
xslt to transform the xml nodes dynamically
i  am working to get the nodes of my xml changed dynamically based on the key value, below is my original xml, following that is the xml i wanted to be after the xsl conversion.

original xml:

```<code><Refd>
  <randr>

  <KeyPair>
  <Key>CM_001</Key>
  <Value>sometext</Value>
  </KeyPair>

  <KeyPair>
  <Key>CM_002</Key>
  <Value>sometext/Value>
  </KeyPair>

  <KeyPair>
  <Key>CM_003</Key>
  <Value>sometext/Value>
  </KeyPair>

  </randr>
  </Refd>
</code>```

trying to get this as below:

```<code>  <Refd>
  <randr>

  <KeyPair_001>
  <Key>CM_001</Key>
  <Value>sometext</Value>
  </KeyPair_001>

  <KeyPair_002>
  <Key>CM_002</Key>
  <Value>sometext/Value>
  </KeyPair_002>

  <KeyPair_003>
  <Key>CM_003</Key>
  <Value>sometext/Value>
  </KeyPair_003>

  </randr>
  </Refd>
</code>```

trying to get the xslt working by the below , but it only gets me for just 1 tag and the others remain unchanged, can i get it to work for multiple tags. Thank you in advance

```<code>  <xsl:template match="Refd/randr/KeyPair">
    <xsl:variable name="ename">
      <xsl:text>Key</xsl:text>
      <xsl:if test="Key ='CM_001'">
        <xsl:text>001</xsl:text>
      </xsl:if>
    </xsl:variable>
    <xsl:element name="{$ename}">
      <xsl:copy-of select="@*"/>
      <xsl:apply-templates select="node()"/>
    </xsl:element>
  </xsl:template>
</code>```

答:
How about:

XSLT 1.0

```<code><xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>
<xsl:strip-space elements="*"/>

<!-- identity transform -->
<xsl:template match="@*|node()">
    <xsl:copy>
        <xsl:apply-templates select="@*|node()"/>
    </xsl:copy>
</xsl:template>

<xsl:template match="KeyPair">
    <xsl:element name="KeyPair{substring-after(Key, 'CM')}">
        <xsl:apply-templates/>
    </xsl:element>
</xsl:template>

</xsl:stylesheet>
</code>```

Note that numbering element names is considered bad practice. If your target application demands it, then it's unavoidable; but in general it is best to use an attribute for this purpose.

Body:
i  am working to get the nodes of my xml changed dynamically based on the key value, below is my original xml, following that is the xml i wanted to be after the xsl conversion.

original xml:

<Refd>
  <randr>

  <KeyPair>
  <Key>CM_001</Key>
  <Value>sometext</Value>
  </KeyPair>

  <KeyPair>
  <Key>CM_002</Key>
  <Value>sometext/Value>
  </KeyPair>

  <KeyPair>
  <Key>CM_003</Key>
  <Value>sometext/Value>
  </KeyPair>

  </randr>
  </Refd>

trying to get this as below:

  <Refd>
  <randr>

  <KeyPair_001>
  <Key>CM_001</Key>
  <Value>sometext</Value>
  </KeyPair_001>

  <KeyPair_002>
  <Key>CM_002</Key>
  <Value>sometext/Value>
  </KeyPair_002>

  <KeyPair_003>
  <Key>CM_003</Key>
  <Value>sometext/Value>
  </KeyPair_003>

  </randr>
  </Refd>

trying to get the xslt working by the below , but it only gets me for just 1 tag and the others remain unchanged, can i get it to work for multiple tags. Thank you in advance

  <xsl:template match="Refd/randr/KeyPair">
    <xsl:variable name="ename">
      <xsl:text>Key</xsl:text>
      <xsl:if test="Key ='CM_001'">
        <xsl:text>001</xsl:text>
      </xsl:if>
    </xsl:variable>
    <xsl:element name="{$ename}">
      <xsl:copy-of select="@*"/>
      <xsl:apply-templates select="node()"/>
    </xsl:element>
  </xsl:template>

Body:
How about:

XSLT 1.0

<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>
<xsl:strip-space elements="*"/>

<!-- identity transform -->
<xsl:template match="@*|node()">
    <xsl:copy>
        <xsl:apply-templates select="@*|node()"/>
    </xsl:copy>
</xsl:template>

<xsl:template match="KeyPair">
    <xsl:element name="KeyPair{substring-after(Key, 'CM')}">
        <xsl:apply-templates/>
    </xsl:element>
</xsl:template>

</xsl:stylesheet>

Note that numbering element names is considered bad practice. If your target application demands it, then it's unavoidable; but in general it is best to use an attribute for this purpose.


问:
How to select a set of fields from input data as an array of repeated fields in beam SQL
Problem Statement:

I  have  an  input  PCollection  with  following  fields:

```<code>{
   firstname_1,
   lastname_1,
   dob,
   firstname_2,
   lastname_2,
   firstname_3,
   lastname_3,
}
</code>```

then I execute a Beam SQL  operation such that output of resultant  PCollection should be like

```<code> ----------------------------------------------
   name.firstname |  name.lastname | dob
 ----------------------------------------------
      firstname_1 |  lastname_1    | 202009
      firstname_2 |  lastname_2    |
      firstname_3 |  lastname_3    |
-----------------------------------------------
</code>```

To be precise:

```<code>array[
    (firstname_1,lastname_1,dob),
    (firstname_2,lastname_2,dob),
    (firstname_3,lastname_3,dob)
]
</code>```

Here is the code snippet where I execute Beam SQL:

PCollectionTuple tuple=
    PCollectionTuple.of(new TupleTag<>("testPcollection"), testPcollection);

PCollection<Row> result = tuple
    .apply(SqlTransform.query(
        "SELECT array[(firstname_1,lastname_1,dob), (firstname_2,lastname_2,dob), (firstname_3,lastname_3,dob)]"));

```

I am not getting proper results.

Can someone guide  me  how to query an array of repeated field in Beam  SQL?

答:
Your SQL query has a few errors.

You have named the input to the SQL query testPcollection. Your SQL query does not select FROM testPcollection. Let us assume you meant it to be FROM testPcollection.

You use the syntax (firstname_1, lastname_1, doc) in both your expected output and your query. This is not any valid SQL expression.

Body:
Problem Statement:

I  have  an  input  PCollection  with  following  fields:

{
   firstname_1,
   lastname_1,
   dob,
   firstname_2,
   lastname_2,
   firstname_3,
   lastname_3,
}

then I execute a Beam SQL  operation such that output of resultant  PCollection should be like

 ----------------------------------------------
   name.firstname |  name.lastname | dob
 ----------------------------------------------
      firstname_1 |  lastname_1    | 202009
      firstname_2 |  lastname_2    |
      firstname_3 |  lastname_3    |
-----------------------------------------------

To be precise:

array[
    (firstname_1,lastname_1,dob),
    (firstname_2,lastname_2,dob),
    (firstname_3,lastname_3,dob)
]

Here is the code snippet where I execute Beam SQL:

PCollectionTuple tuple=
    PCollectionTuple.of(new TupleTag<>("testPcollection"), testPcollection);

PCollection<Row> result = tuple
    .apply(SqlTransform.query(
        "SELECT array[(firstname_1,lastname_1,dob), (firstname_2,lastname_2,dob), (firstname_3,lastname_3,dob)]"));

I am not getting proper results.

Can someone guide  me  how to query an array of repeated field in Beam  SQL?


问:
I cannot tell what I am doing wrong with this java recursion question! I posted the excercise and my code down bellow
Write a recursive method indexOf that accepts two Strings as parameters and that returns the starting index of the first occurrence of the second String inside the first String (or -1 if not found). The table below lists several calls to your method and their expected return values. Notice that case matters, as in the last example that returns -1.

```<code>Call    Value Returned
indexOf("Barack Obama", "Bar")  0
indexOf("Barack Obama", "ck")   4
indexOf("Barack Obama", "a")    1
indexOf("Barack Obama", "McCain")   -1
indexOf("Barack Obama", "BAR")  -1
</code>```

Strings have an indexOf method, but you are not allowed to call it. You are limited to these methods:

Method  Description

```<code>equals(String other)     //returns true if the two Strings contain the same characters
length()                 //returns the int number of characters in the String
substring(int fromIndex, int toIndex)
substring(int fromIndex) //returns a new String containing the characters from this
                         //String from fromIndex (inclusive) to toIndex (exclusive),
                         //or to the end of the String if toIndex is omitted
</code>```

You are not allowed to construct any structured objects other than Strings (no array, List, Scanner, etc.) and you may not use any loops to solve this problem; you must use recursion.

```<code>public int indexOf(String s, String b){
    if(s.length() <= b.length() && !s.equals(b)){return -1;}
    if(s.substring(0, b.length()).equals(b)){return 0;}
    return 1 + indexOf(s.substring(b.length()), b);
}
</code>```

答:
Two slight changes to get the answer of FreddyDs working:</p>

```<code>public static int indexOf(String s, String b){
        if(s.length() <= b.length() && !s.equals(b)){return -1;}
        if(s.substring(0, b.length()).equals(b)){return 0;}
        int result = indexOf(s.substring(1, s.length()), b);
        return result == -1 ? -1 : 1 + result;
    }
</code>```

Body:
Write a recursive method indexOf that accepts two Strings as parameters and that returns the starting index of the first occurrence of the second String inside the first String (or -1 if not found). The table below lists several calls to your method and their expected return values. Notice that case matters, as in the last example that returns -1.

Call    Value Returned
indexOf("Barack Obama", "Bar")  0
indexOf("Barack Obama", "ck")   4
indexOf("Barack Obama", "a")    1
indexOf("Barack Obama", "McCain")   -1
indexOf("Barack Obama", "BAR")  -1

Strings have an indexOf method, but you are not allowed to call it. You are limited to these methods:

Method  Description

equals(String other)     //returns true if the two Strings contain the same characters
length()                 //returns the int number of characters in the String
substring(int fromIndex, int toIndex)
substring(int fromIndex) //returns a new String containing the characters from this
                         //String from fromIndex (inclusive) to toIndex (exclusive),
                         //or to the end of the String if toIndex is omitted

You are not allowed to construct any structured objects other than Strings (no array, List, Scanner, etc.) and you may not use any loops to solve this problem; you must use recursion.

public int indexOf(String s, String b){
    if(s.length() <= b.length() && !s.equals(b)){return -1;}
    if(s.substring(0, b.length()).equals(b)){return 0;}
    return 1 + indexOf(s.substring(b.length()), b);
}

Body:
Two slight changes to get the answer of FreddyDs working:

public static int indexOf(String s, String b){
        if(s.length() <= b.length() && !s.equals(b)){return -1;}
        if(s.substring(0, b.length()).equals(b)){return 0;}
        int result = indexOf(s.substring(1, s.length()), b);
        return result == -1 ? -1 : 1 + result;
    }


问:
How to structure a feed and follow system?
I was using Firebase realtime database for my social network app were you can follow and receive posts of people you follow.

My database :

```<code>Users
--USER_ID_1
----name
----email
--USER_ID_2
----name
----email

Posts
--POST_ID_1
----image
----userid
----date
--POST_ID_2
----image
----userid
----date

Timeline
--User_ID_1
----POST_ID_2
------date
----POST_ID_1
------date
</code>```

Another node "Content" contained id of the all the user posts. If "A" followed "B" then all post id's of B where added to A's timeline. And if B posted something it's also added to all of its follower's timelines.

It has scalability issues :

If someone has 10,000 followers a new post was added to all 10,000 follower's timelines.

If someone has a large amount of posts then every new follower receives all of those in his timeline.

I want to change to Firestore as it's been claimed scalable. How should I structure my database so these problems in realtime database are eliminated in Firestore?

答: I think one possibility is to make another top-level collection named "users_following" which contains a document named "user_id" and a field for an array that contains all the users that the user is following. Within that "users_following" document one can have sub-collection of that particular user all posts or a top-level collection will also do the job. The next important thing that comes is that one has to store a recent one post inside "users-following" document as an array or map. Basically this normalized data is going to be used to populate the feed of the person who is following you. But its drawback is that you will only see one post per person even if the person has added two posts recently or even if you store your two to three posts in the normalized way than your all three posts will be shown at once (like three posts of the same user in a row). But it's something still good if you just need to show one post per user.

Body:
I was using Firebase realtime database for my social network app were you can follow and receive posts of people you follow.

My database :

Users
--USER_ID_1
----name
----email
--USER_ID_2
----name
----email

Posts
--POST_ID_1
----image
----userid
----date
--POST_ID_2
----image
----userid
----date

Timeline
--User_ID_1
----POST_ID_2
------date
----POST_ID_1
------date

Another node "Content" contained id of the all the user posts. If "A" followed "B" then all post id's of B where added to A's timeline. And if B posted something it's also added to all of its follower's timelines.

It has scalability issues :

If someone has 10,000 followers a new post was added to all 10,000 follower's timelines.

If someone has a large amount of posts then every new follower receives all of those in his timeline.

I want to change to Firestore as it's been claimed scalable. How should I structure my database so these problems in realtime database are eliminated in Firestore?


答:
Use pandas to load the data, then use groupby to group by the shared index:

```<code>import pandas as pd

dates = ['01-12-2019']*4 + ['03-12-2019']*5 + ['07-12-2019']*2
data = [142,20,20,120,30,20,45,215,15,25,5]

df = pd.DataFrame(data,)
df.index = pd.DatetimeIndex(dates)

grouped = df.groupby(df.index).apply(lambda x: x.max()-x.min())
</code>```

Output:

```<code>            product mrp
2019-01-12          122
2019-03-12          200
2019-07-12           20
</code>```

Title: Calculating the range of column value datewise through Python

Body:
I want to calculate the maximum difference of product_mrp according to the dates.
For that I was trying to group by date but not able to get after that.

INPUT:

+-------------+--------------------+
| product_mrp |     order_date     |
+-------------+--------------------+
|         142 |         01-12-2019 |
|          20 |         01-12-2019 |
|          20 |         01-12-2019 |
|         120 |         01-12-2019 |
|          30 |         03-12-2019 |
|          20 |         03-12-2019 |
|          45 |         03-12-2019 |
|         215 |         03-12-2019 |
|          15 |         03-12-2019 |
|          25 |         07-12-2019 |
|           5 |         07-12-2019 |
+-------------+--------------------+

EXPECTED OUTPUT:

 +-------------+--------------------+
| product_mrp |     order_date     |
+-------------+--------------------+
|         122 |         01-12-2019 |
|         200 |         03-12-2019 |
|          20 |         07-12-2019 |
+-------------+--------------------+

Body:
Use pandas to load the data, then use groupby to group by the shared index:

import pandas as pd

dates = ['01-12-2019']*4 + ['03-12-2019']*5 + ['07-12-2019']*2
data = [142,20,20,120,30,20,45,215,15,25,5]

df = pd.DataFrame(data,)
df.index = pd.DatetimeIndex(dates)

grouped = df.groupby(df.index).apply(lambda x: x.max()-x.min())

Output:

            product mrp
2019-01-12          122
2019-03-12          200
2019-07-12           20


问:
I Need to Call delete[] on a Dynamic Array within a Class but It Doesn't Work
I wrote code very similar to this (although not identical) for a game I'm making. This has the same issue, so I'll provide it.

```<code>#include <iostream>

using namespace std;

class test {
public:
    int* arr;
    test(int n) {
        arr = new int[n];
        for(int i=0; i<=9; i++) {
            arr[i] = i;
        }
    }
    ~test() {
        delete[] arr;
    }
};

int main() {

    test obj = test(10);
    for(int i=0; i<=8; i++) {
        cout << obj.arr[i] << endl;
    }
    cout << obj.arr[9];
    obj.~test();
    return 0;

}
</code>```

I know that I could rework my code and use the vector class to avoid this problem, but I want to actually understand why it's crashing. Removing the deconstructor call fixes the error, but that also introduces a memory leak, so I would prefer to just fix the deconstructor.

答:
This line is causing the error:

```<code>obj.~test(); // removing this will solve your problem
</code>```

You shouldn't call the destructor. Once the object goes out of scope the destructor will get called.

Body:
I wrote code very similar to this (although not identical) for a game I'm making. This has the same issue, so I'll provide it.

#include <iostream>

using namespace std;

class test {
public:
    int* arr;
    test(int n) {
        arr = new int[n];
        for(int i=0; i<=9; i++) {
            arr[i] = i;
        }
    }
    ~test() {
        delete[] arr;
    }
};

int main() {

    test obj = test(10);
    for(int i=0; i<=8; i++) {
        cout << obj.arr[i] << endl;
    }
    cout << obj.arr[9];
    obj.~test();
    return 0;

}

I know that I could rework my code and use the vector class to avoid this problem, but I want to actually understand why it's crashing. Removing the deconstructor call fixes the error, but that also introduces a memory leak, so I would prefer to just fix the deconstructor.

Body:
This line is causing the error:

obj.~test(); // removing this will solve your problem

You shouldn't call the destructor. Once the object goes out of scope the destructor will get called.


问:
Are methods that modify reference type parameters bad?
I've seen methods like this:

```<code>public void Foo(List<string> list)
{
    list.Add("Bar");
}
</code>```

Is this good practice to modify parameters in a method?

Wouldn't this be better?

```<code>public List<string> Foo(List<string> list)
{
    // Edit
    List<string> newlist = new List<string>(list);
    newlist.Add("Bar");
    return newlist;
}
</code>```

It just feels like the first example has unexpected side effects.

答:
Frankly, in this case, both methods do more or less the same thing. Both will modify the List that was passed in.

If the objective is to have lists immutable by such a method, the second example should make a copy of the List that was sent in, and then perform the Add operation on the new List and then return that.

I'm not familiar with C# nor .NET, so my guess would be something along the line of:

```<code>public List<string> Foo(List<string> list)
{
    List<string> newList = (List<string>)list.Clone();
    newList.Add("Bar");
    return newList;
}
</code>```

This way, the method which calls the Foo method will get the newly created List returned, and the original List that was passed in would not be touched.

This really is up to the "contract" of your specifications or API, so in cases where Lists can just be modified, I don't see a problem with going with the first approach.

OwnerDisplayName: Jonathan Parker

Body:
I've seen methods like this:

public void Foo(List<string> list)
{
    list.Add("Bar");
}

Is this good practice to modify parameters in a method?

Wouldn't this be better?

public List<string> Foo(List<string> list)
{
    // Edit
    List<string> newlist = new List<string>(list);
    newlist.Add("Bar");
    return newlist;
}

It just feels like the first example has unexpected side effects.

LastEditorDisplayName: Jonathan Parker

Body:
Frankly, in this case, both methods do more or less the same thing. Both will modify the List that was passed in.

If the objective is to have lists immutable by such a method, the second example should make a copy of the List that was sent in, and then perform the Add operation on the new List and then return that.

I'm not familiar with C# nor .NET, so my guess would be something along the line of:

public List<string> Foo(List<string> list)
{
    List<string> newList = (List<string>)list.Clone();
    newList.Add("Bar");
    return newList;
}

This way, the method which calls the Foo method will get the newly created List returned, and the original List that was passed in would not be touched.

This really is up to the "contract" of your specifications or API, so in cases where Lists can just be modified, I don't see a problem with going with the first approach.

OwnerDisplayName: coobird

LastEditorDisplayName: Patrick


问:
While executing mysql insert select query- can we do condition check?
I am trying to execute mysql select insert query as below.

```<code>INSERT INTO ACCOUNTS.TABLE1 (NAME,ID,STARTTIME,ENDTIME) SELECT NAME,ID,CREATIONTIME,EXPIRYTIME FROM ACCOUNTS.TABLE2 WHERE ...;
</code>```

Now, if EXPIRYTIME in table2 is null, then i want to update the ENDTIME in table1 to current timestamp. I have overriden the default behavior of this timestamp field by the description

```<code>ENDTIME TIMESTAMP NULL DEFAULT NULL
</code>```

for some reason.

Is it possible in the above select insert query to include some something like an if condition saying if (expirytime == null) then update endtime to current timestamp?

答:
```<code>INSERT INTO ACCOUNTS.TABLE1 (NAME,ID,STARTTIME,ENDTIME)
SELECT NAME,ID,CREATIONTIME,if(EXPIRYTIME is null,now(),expirytime)
FROM ACCOUNTS.TABLE2 WHERE ...;
</code>```

Body:
I am trying to execute mysql select insert query as below.

INSERT INTO ACCOUNTS.TABLE1 (NAME,ID,STARTTIME,ENDTIME) SELECT NAME,ID,CREATIONTIME,EXPIRYTIME FROM ACCOUNTS.TABLE2 WHERE ...;

Now, if EXPIRYTIME in table2 is null, then i want to update the ENDTIME in table1 to current timestamp. I have overriden the default behavior of this timestamp field by the description

ENDTIME TIMESTAMP NULL DEFAULT NULL

for some reason.

Is it possible in the above select insert query to include some something like an if condition saying if (expirytime == null) then update endtime to current timestamp?


问:
Get id_cart_rules prestashop in order
I need to get the id of the cart_Rule in the command, until now and tried with

```<code>$this->context->cart->getCartRules();
</code>```

But I don't get a result

答:
I managed to do it as follows:

```<code>  $regla =  $this->context->cart->getCartRules();//returns an array
  $codigo = $regla[0]["code"];
</code>```

Working with version 1.7

Title: Get id_cart_rules prestashop in order

Body:
I need to get the id of the cart_Rule in the command, until now and tried with

$this->context->cart->getCartRules();

But I don't get a result

Body:
I managed to do it as follows:

  $regla =  $this->context->cart->getCartRules();//returns an array
  $codigo = $regla[0]["code"];

Working with version 1.7


答:
you can use groupby as you said and max, min and reset_index like:

```<code>gr = df.groupby('order_date')['product_mrp']
df_ = (gr.max()-gr.min()).reset_index()

print (df_)
   order_date  product_mrp
0  01-12-2019          122
1  03-12-2019          200
2  07-12-2019           20
</code>```

Body:
you can use groupby as you said and max, min and reset_index like:

gr = df.groupby('order_date')['product_mrp']
df_ = (gr.max()-gr.min()).reset_index()

print (df_)
   order_date  product_mrp
0  01-12-2019          122
1  03-12-2019          200
2  07-12-2019           20


问:
POST request to `/users` for an existing user causes activation email to be re-sent
I am looking for a way to re-send the activation email for users. I noticed that making a POST request to /users for an existing user does precisely this (and in addition, if the user has userState='Active', it gets changed to 'ActivationSent').

For instance, making a POST request with the following data twice, will result in two activation emails being sent to joe@email.com:

```<code>{'newUsers': [{'email': 'joe@email.com',
               'userName': 'joe@email.com',
               'firstName': 'Joe',
               'lastName': 'Smith'}]}
</code>```

I would like to confirm that this is indeed the correct behavior which I can rely upon, as I did not find this described in the documentation.

答:
What you are doing is, in fact, adding the same user again. When a user is added, an activation email is sent and the status is ActivationSent.
So while this is accomplishing what you wanted, technically, it's still just adding a user and emailing them to activate but you are correct that it's a way to "resend" the activation email to a user.</p>

Body:
I am looking for a way to re-send the activation email for users. I noticed that making a POST request to /users for an existing user does precisely this (and in addition, if the user has userState='Active', it gets changed to 'ActivationSent').

For instance, making a POST request with the following data twice, will result in two activation emails being sent to joe@email.com:

{'newUsers': [{'email': 'joe@email.com',
               'userName': 'joe@email.com',
               'firstName': 'Joe',
               'lastName': 'Smith'}]}

I would like to confirm that this is indeed the correct behavior which I can rely upon, as I did not find this described in the documentation.


问:
Firefox Mobile and height 100vh
In Firefox Mobile, I see twitching when I scroll the page up and down, if the block has a height of 100vh. Is it possible to fix this effect?

```<code>html, body
{
padding: 0; margin: 0;
}

.block
{
height: 100vh;
background: blue;
}
</code>```

My html

```<code><!DOCTYPE html>
<html lang="en">
<head>
<title>Тест</title>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
</head>

<body>
<div class="block"></div>
<p>text</p>
<p>text</p>
<p>text</p>
<p>text</p>
<p>text</p>
<p>text</p>
<p>text</p>
<p>text</p>
<p>text</p>
<p>text</p>
<p>text</p>
<p>text</p>
<p>text</p>
<p>text</p>
</body>
</html>
</code>```

答:
I tried to replicate this through Codepen and didn't saw any twitches. Nice & smooth. But then I deployed it on GitHub Pages — it indeed twitching. My initial thoughts were that they (Codepen) probably adding something to the head section, but I didn't find anything there.

But, what is the difference between the two?
First doesn't affect the address bar while another does.
The reason behind this is that Codepen launches your page inside an iframe, therefore it doesn't trigger the scroll.

So now our objective is to find a way to "freeze" the address bar.

We can do this by wrapping everything inside the body in a div:

```<code><body>
  <div id="wrapper">
    ...
  </div>
</body>
</html>
</code>```

Then adding this to our CSS:

```<code>@-moz-document url-prefix() {
  #wrapper {
    height: 100vh;
    overflow-y: scroll;
  }
}
</code>```

Since other browsers don't have problems with animation of the address bar we are using this firefox-only property for our css.

Here you go. Nice & smooth.

Title: Firefox Mobile and height 100vh

Body:
In Firefox Mobile, I see twitching when I scroll the page up and down, if the block has a height of 100vh. Is it possible to fix this effect?

html, body
{
padding: 0; margin: 0;
}

.block
{
height: 100vh;
background: blue;
}

My html

<!DOCTYPE html>

<title>Тест</title>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />

text

text

text

text

text

text

text

text

text

text

text

text

text

text

Body:
I tried to replicate this through Codepen and didn't saw any twitches. Nice & smooth. But then I deployed it on GitHub Pages — it indeed twitching. My initial thoughts were that they (Codepen) probably adding something to the head section, but I didn't find anything there.

But, what is the difference between the two?
First doesn't affect the address bar while another does.
The reason behind this is that Codepen launches your page inside an iframe, therefore it doesn't trigger the scroll.

So now our objective is to find a way to "freeze" the address bar.

We can do this by wrapping everything inside the body in a div:

<body>
  <div id="wrapper">
    ...
  </div>
</body>
</html>

Then adding this to our CSS:

@-moz-document url-prefix() {
  #wrapper {
    height: 100vh;
    overflow-y: scroll;
  }
}

Since other browsers don't have problems with animation of the address bar we are using this firefox-only property for our css.

Here you go. Nice & smooth.


问:
Is there a way to pass an open file to pandas read_csv
I have a csv file that is too large to read into memory all at once (~46 GB), and I want a subset of that data which can fit into memory.  I am able to read in part of the file, filter out what I do not want by doing an inner join with a previously created filtering index, and then continue on to the next part of the file like below:

```<code>    with open('Filtered_By_Month/all_data.csv','r') as f:
        cols=np.array(f.readline().strip('\n').split(','))#column names
        data=[]
        df=None
        for i in f:
            data.append(i.strip('\n').split(',')
            if len(data)==1000000:
                df_sub=pd.DataFrame(data,columns=cols).set_index('KEY_')
                df_sub=df_sub.join(filtered_keys,how='inner')
                data=[]
                if df is None:
                    df=df_sub
                else:
                    df=pd.concat([df,df_sub])
</code>```

This seems to do the trick except it is extremely slow.  An alternative would be to use pandas read_csv function along with the nrows and skip_rows arguments, however this becomes really slow as well since each time I call the read_csv method, the file is reopened and the cursor is at the first line, and I have to iterate through the lines until I have gone past skip_rows.  Is there a way to combine both of these methods so that I pass a file to read_csv that is already open and thus I do not need to have the cursor start all over again.

答:
Narrow answer: yes

You can provide the engine='python' and nrows=N arguments to pick up where pandas's reader leaves off in a text file, or to use pd.read_csv multiple times for a single open file object.

Pandas searches the file for separators such as commas newlines. By default (based on your provided arguments), pandas uses an optimized C-compiled reader that is faster, but has fewer features. Using this reader, the cursor object is not always left at the end of the last-read line, and sometimes exhausts the file reader, other times it is left in the middle of the next line.

Using a simple ~200MB text file with two columns and 1,000,000 rows:

In [10]: # create some data
    ...: pd.DataFrame(
    ...:     {'A': np.arange(0, 1e6), 'B': np.arange(1e6, 0, -1)}
    ...: ).to_csv('sample.csv', index=False)
```

<p>We can see that reading all but the last two lines with the <code>engine='C'</code> parser leaves the cursor at the end of the file (nothing gets printed):</p>

<pre class="lang-py prettyprint-override"><code>In [11]: %%time
    ...: with open('sample.csv', 'r') as f:
    ...:     top = pd.read_csv(f, nrows=999998, header=0, engine='c')
    ...:     print(f.read())

CPU times: user 130 ms, sys: 2.99 ms, total: 133 ms
Wall time: 132 ms

</code>```

With engine='python', the file object is left at the end of the line, and we can read the remainder of the file, or make another pd.read_csv call:

In [12]: %%time
    ...: with open('sample.csv', 'r') as f:
    ...:     top = pd.read_csv(f, nrows=999998, header=0, engine='python')
    ...:     print(f.read())
999998,2
999999,1

CPU times: user 2.86 s, sys: 150 ms, total: 3.01 s
Wall time: 2.96 s
```

<p>However, the python reader is significantly slower.</p>

<h1>Better answer: don't</h1>

<p>Providing <code>chunksize</code> as an argument to <code>pd.read_csv</code> will return an iterator.</p>

<p>From the pandas docs on IO tools: Iterating through files chunk by chunk:</p>

<blockquote>
  <p>By specifying a <code>chunksize</code> to <code>read_csv</code>, the return value will be an iterable object of type <code>TextFileReader</code></p>
</blockquote>

<p>Using this chunked read mode allows us to return to using the C engine. With the example from above:</p>

<pre class="lang-py prettyprint-override"><code>In [13]: %%time
    ...: reader = pd.read_csv('sample.csv', chunksize=100000, header=0, engine='c')
    ...: for chunk in reader:
    ...:     tail = chunk.iloc[-2:]
    ...: print(tail)
             A  B
999998  999998  2
999999  999999  1
CPU times: user 136 ms, sys: 9.86 ms, total: 146 ms
Wall time: 146 ms
</code>```

We're now barely slower than the C parser without chunking, even for 10 iterations on a pretty small file. As this file becomes larger than memory, using chunked reads will significantly outperform any pure-python file handling.

You can use this chunked read method to filter your data:

reader = pd.read_csv('Filtered_By_Month/all_data.csv', chunksize=1000000)

chunks = []

for chunk in reader:
    # filter data
    filtered = do_something(chunk)
    chunks.append(filtered)

df = pd.concat(chunks)
```

Alternatives: parallel frameworks

If you're working with a file that is significantly larger than memory, you may want to look at a parallel computing library such as Dask.

Dask has tools for reading in CSV data using multiple threads, processors, or even servers. There is a performance overhead associated with spreading tasks across processors or through a network, but for sufficiently large tasks the gains will be worth it. There's also a learning curve, but some great work in aligning the dask.Dataframe API with pandas has brought the learning curve way down, so following the tutorials on the dask website should get you up and running pretty quickly.

Body:
I have a csv file that is too large to read into memory all at once (~46 GB), and I want a subset of that data which can fit into memory.  I am able to read in part of the file, filter out what I do not want by doing an inner join with a previously created filtering index, and then continue on to the next part of the file like below:

    with open('Filtered_By_Month/all_data.csv','r') as f:
        cols=np.array(f.readline().strip('\n').split(','))#column names
        data=[]
        df=None
        for i in f:
            data.append(i.strip('\n').split(',')
            if len(data)==1000000:
                df_sub=pd.DataFrame(data,columns=cols).set_index('KEY_')
                df_sub=df_sub.join(filtered_keys,how='inner')
                data=[]
                if df is None:
                    df=df_sub
                else:
                    df=pd.concat([df,df_sub])

This seems to do the trick except it is extremely slow.  An alternative would be to use pandas read_csv function along with the nrows and skip_rows arguments, however this becomes really slow as well since each time I call the read_csv method, the file is reopened and the cursor is at the first line, and I have to iterate through the lines until I have gone past skip_rows.  Is there a way to combine both of these methods so that I pass a file to read_csv that is already open and thus I do not need to have the cursor start all over again.

Body:
Narrow answer: yes

You can provide the engine='python' and nrows=N arguments to pick up where pandas's reader leaves off in a text file, or to use pd.read_csv multiple times for a single open file object.

Pandas searches the file for separators such as commas newlines. By default (based on your provided arguments), pandas uses an optimized C-compiled reader that is faster, but has fewer features. Using this reader, the cursor object is not always left at the end of the last-read line, and sometimes exhausts the file reader, other times it is left in the middle of the next line.

Using a simple ~200MB text file with two columns and 1,000,000 rows:

In [10]: # create some data
    ...: pd.DataFrame(
    ...:     {'A': np.arange(0, 1e6), 'B': np.arange(1e6, 0, -1)}
    ...: ).to_csv('sample.csv', index=False)

We can see that reading all but the last two lines with the engine='C' parser leaves the cursor at the end of the file (nothing gets printed):

In [11]: %%time
    ...: with open('sample.csv', 'r') as f:
    ...:     top = pd.read_csv(f, nrows=999998, header=0, engine='c')
    ...:     print(f.read())

CPU times: user 130 ms, sys: 2.99 ms, total: 133 ms
Wall time: 132 ms

With engine='python', the file object is left at the end of the line, and we can read the remainder of the file, or make another pd.read_csv call:

In [12]: %%time
    ...: with open('sample.csv', 'r') as f:
    ...:     top = pd.read_csv(f, nrows=999998, header=0, engine='python')
    ...:     print(f.read())
999998,2
999999,1

CPU times: user 2.86 s, sys: 150 ms, total: 3.01 s
Wall time: 2.96 s

However, the python reader is significantly slower.

Better answer: don't

Providing chunksize as an argument to pd.read_csv will return an iterator.

From the pandas docs on IO tools: Iterating through files chunk by chunk:

By specifying a chunksize to read_csv, the return value will be an iterable object of type TextFileReader

Using this chunked read mode allows us to return to using the C engine. With the example from above:

In [13]: %%time
    ...: reader = pd.read_csv('sample.csv', chunksize=100000, header=0, engine='c')
    ...: for chunk in reader:
    ...:     tail = chunk.iloc[-2:]
    ...: print(tail)
             A  B
999998  999998  2
999999  999999  1
CPU times: user 136 ms, sys: 9.86 ms, total: 146 ms
Wall time: 146 ms

We're now barely slower than the C parser without chunking, even for 10 iterations on a pretty small file. As this file becomes larger than memory, using chunked reads will significantly outperform any pure-python file handling.

You can use this chunked read method to filter your data:

reader = pd.read_csv('Filtered_By_Month/all_data.csv', chunksize=1000000)

chunks = []

for chunk in reader:
    # filter data
    filtered = do_something(chunk)
    chunks.append(filtered)

df = pd.concat(chunks)

Alternatives: parallel frameworks

If you're working with a file that is significantly larger than memory, you may want to look at a parallel computing library such as Dask.

Dask has tools for reading in CSV data using multiple threads, processors, or even servers. There is a performance overhead associated with spreading tasks across processors or through a network, but for sufficiently large tasks the gains will be worth it. There's also a learning curve, but some great work in aligning the dask.Dataframe API with pandas has brought the learning curve way down, so following the tutorials on the dask website should get you up and running pretty quickly.


问:
How does sudo and become in ansible work together?
```<code>- name: "read env"
  shell: "sudo env"
  become: true
</code>```

I have above snippet in Ansible, I want to know the behaviour of using become and sudo together. I am aware become's default action is to turn the action as sudo. In that case will my sudo in the beginning of shell command get nullified ?
I am getting a different results with shell: "sudo env" and shell: "env" when become is set to true/yes

答:
You do not need to add sudo to your commands when you are using become: true. You can checkout become_method from documentation. It will append sudo for you when you use become: true.

I highly recommend to read documentation for privilege escalation: https://docs.ansible.com/ansible/latest/user_guide/become.html

UPDATE

I did misunderstood your question sorry. The default become_method is sudo in ansible.cfg. When you set become: true without specifying become_method it will basically add a sudo prefix to your cmd. Here i created a example:

# privilege_escalation.yaml
---
- name: privilege escalation
  hosts: localhost
  tasks:
    - name: command without any escalation
      shell: env

    - name: command with sudo
      shell: sudo env

    - name: command with become and sudo
      shell: sudo env
      become: yes
```

<p>You can run example with this command:</p>

<pre class="lang-sh prettyprint-override"><code>ansible-playbook -vvv --ask-become-pass  privilege_escalation.yaml
</code>```

The first task will run env. In the results you can see USER=your_user line that represents current user.

When you use sudo in command, second task will run sudo env. In the results you can see USER=root and
SUDO_USER=your_user. This means you escalated your privileges to become root when running env command. SUDO_USER environment variable represents the user who invoked sudo.

The last task will run sudo sudo env. In the results you can see USER=root and
SUDO_USER=root. This means the first you become root user, after that root user executed sudo env command.

I hope this helps.

Body:
- name: "read env"
  shell: "sudo env"
  become: true

I have above snippet in Ansible, I want to know the behaviour of using become and sudo together. I am aware become's default action is to turn the action as sudo. In that case will my sudo in the beginning of shell command get nullified ?
I am getting a different results with shell: "sudo env" and shell: "env" when become is set to true/yes

Body:
You do not need to add sudo to your commands when you are using become: true. You can checkout become_method from documentation. It will append sudo for you when you use become: true.

I highly recommend to read documentation for privilege escalation: https://docs.ansible.com/ansible/latest/user_guide/become.html

UPDATE

I did misunderstood your question sorry. The default become_method is sudo in ansible.cfg. When you set become: true without specifying become_method it will basically add a sudo prefix to your cmd. Here i created a example:

# privilege_escalation.yaml
---
- name: privilege escalation
  hosts: localhost
  tasks:
    - name: command without any escalation
      shell: env

    - name: command with sudo
      shell: sudo env

    - name: command with become and sudo
      shell: sudo env
      become: yes

You can run example with this command:

ansible-playbook -vvv --ask-become-pass  privilege_escalation.yaml

The first task will run env. In the results you can see USER=your_user line that represents current user.

When you use sudo in command, second task will run sudo env. In the results you can see USER=root and
SUDO_USER=your_user. This means you escalated your privileges to become root when running env command. SUDO_USER environment variable represents the user who invoked sudo.

The last task will run sudo sudo env. In the results you can see USER=root and
SUDO_USER=root. This means the first you become root user, after that root user executed sudo env command.

I hope this helps.


问:
How to compile torch 1.5.0 without GPU support?
I want to install pytorch 1.5.0 on AWS lambda. Since the torch library is very large, I need to make it as small as possible to fit within the size limits. My script looks like this so far:

    mkdir python
    docker run \
        --rm \
        -v $(pwd):/build \
        python:3.8 \
        sh -c "
            cd /build;
            pip3 install torch==1.5.0 -t python/torch --no-cache-dir;
            find . -type d -name '__pycache__' | xargs rm -rf;
            find . -type d -name 'tests' | xargs rm -rf;
            find . -type f -name '*.py[co]' | xargs rm -rf;
        ";

    zip -r9 torch.zip python;

```

But the resulting zip file is very large (500+ MB). However, one of the largest files in the installation package is libtorch_cuda.so. Removing that file makes the zip file less than half the size. I know that cuda is a library for GPUs, and since AWS lambda doesn't have a GPU, i don't need this support. But when I remove that file torch will not import correctly.

torch 1.4.0, by comparison, is much smaller because it did not by default include the cuda libraries.

I want torch 1.5.0 without the GPU support.

Is there a way to pip install torch==1.5.0 without gpu support?

答:
As mentioned, there's a CPU only install of Pytorch. You can also make the library smaller by removing files for testing and other non-essential applications (libtorch, caffe2, etc).

For AWS Lambda, the best way to deploy Pytorch is to create a special Lambda layer with your dependencies. Lambda layers have a size limit of 500 MB, compared to 250 MB for your deployment. Your model size also contributes to the limit. You should have your actual model file in your Lambda deployment, while your dependencies (Pytorch) should be in a Lambda layer.

Also remember you don't "install" anything on Lambda. You create a package that is loaded every time your function is invoked. For this reason, large deployments will have cold start issues. It's not unreasonable to expect 10-20 seconds of latency for a cold start. You can get around this by creating another deployment that pings your lambda function every few minutes to keep it hot. This can get tricky though if you need multiple functions available concurrently.

The best way to deploy your model on Lambda would be to convert your model to a protobuf file using ONNX or Torchscript so you can run the model without needing the full Pytorch library.

Body:
I want to install pytorch 1.5.0 on AWS lambda. Since the torch library is very large, I need to make it as small as possible to fit within the size limits. My script looks like this so far:

    mkdir python
    docker run \
        --rm \
        -v $(pwd):/build \
        python:3.8 \
        sh -c "
            cd /build;
            pip3 install torch==1.5.0 -t python/torch --no-cache-dir;
            find . -type d -name '__pycache__' | xargs rm -rf;
            find . -type d -name 'tests' | xargs rm -rf;
            find . -type f -name '*.py[co]' | xargs rm -rf;
        ";

    zip -r9 torch.zip python;

But the resulting zip file is very large (500+ MB). However, one of the largest files in the installation package is libtorch_cuda.so. Removing that file makes the zip file less than half the size. I know that cuda is a library for GPUs, and since AWS lambda doesn't have a GPU, i don't need this support. But when I remove that file torch will not import correctly.

torch 1.4.0, by comparison, is much smaller because it did not by default include the cuda libraries.

I want torch 1.5.0 without the GPU support.

Is there a way to pip install torch==1.5.0 without gpu support?