问:
Admob w/Spritekit
I am trying to get a banner ad showing up on a certain scene every time a user loads it. I know I have to position it programmatically here.

So far I:

```<code>import GoogleMobileAds
</code>```

and then I:

```<code>let BannerAd = GADBannerView(adSize: kGADAdSizeBanner)
BannerAd.frame = CGRect(x: 0, y: 0, width: 320, height: 50)
BannerAd.delegate = self
BannerAd.adUnitID = "ca-app-pub-##/##"
BannerAd.rootViewController = self

let request = GADRequest()
request.testDevices = [kGADSimulatorID, "*************************"]

BannerAd.load(request)
view?.addSubview(BannerAd)
</code>```

But I get the following errors:

Cannot assign value of type 'GameOverScene' to type 'UIViewController?'

and I'm putting this code inside of my init for this scene (it's a gameOver scene that I transition to when the player wins/losses).

Is there anyway to get this banner ad to show up or do I need to use UIKit here?

答:
I believe you just need to add the GADBannerViewDelegate protocol to your class.</p>

```<code>    class GameOverScene: SKScene, GADBannerViewDelegate {
       ...
    }
</code>```

Title: Admob w/Spritekit

Body:
I am trying to get a banner ad showing up on a certain scene every time a user loads it. I know I have to position it programmatically here.

So far I:

import GoogleMobileAds

and then I:

let BannerAd = GADBannerView(adSize: kGADAdSizeBanner)
BannerAd.frame = CGRect(x: 0, y: 0, width: 320, height: 50)
BannerAd.delegate = self
BannerAd.adUnitID = "ca-app-pub-##/##"
BannerAd.rootViewController = self

let request = GADRequest()
request.testDevices = [kGADSimulatorID, "*************************"]

BannerAd.load(request)
view?.addSubview(BannerAd)

But I get the following errors:

Cannot assign value of type 'GameOverScene' to type 'UIViewController?'

and I'm putting this code inside of my init for this scene (it's a gameOver scene that I transition to when the player wins/losses).

Is there anyway to get this banner ad to show up or do I need to use UIKit here?

Body:
I believe you just need to add the GADBannerViewDelegate protocol to your class.

    class GameOverScene: SKScene, GADBannerViewDelegate {
       ...
    }


来源: stackexchange

问:
Cannot access Inputs from my controller/constructor
I have a simple Angular 2 component with @Input, which I bind to the template. The template shows input data, but I cannot access it from the constructor:

```<code>import {Component, View, bootstrap, Input} from 'angular2/angular2';
import DataService from './data-service';

@Component({
    selector: 'app-cmp'
})
@View({
    template: `{{data.firstName}} {{data.lastName}}` //-> shows the correct 'data'
})
export default class NamesComponent {
  @Input() data: any;
  constructor(dataService: DataService) {
    console.log(this.data);//undefined
  }
}
</code>```

Here is a plunker with an example (see "names-component.ts").

What am I doing wrong?

答:
The way to access your data per documentation here is to write a setter and a getter.  Inside those methods you can manipulate your object/data.  Look here:

  private _data = '';

  @Input()
  set data(data: any) {
    this._data = data;
  }

  get data(): string {
    console.log(_data) // <-------- you will see your data here
    return this._data;
  }
}```

ContentLicense: CC BY-SA 3.0

Body:
I have a simple Angular 2 component with @Input, which I bind to the template. The template shows input data, but I cannot access it from the constructor:

import {Component, View, bootstrap, Input} from 'angular2/angular2';
import DataService from './data-service';

@Component({
    selector: 'app-cmp'
})
@View({
    template: `{{data.firstName}} {{data.lastName}}` //-> shows the correct 'data'
})
export default class NamesComponent {
  @Input() data: any;
  constructor(dataService: DataService) {
    console.log(this.data);//undefined
  }
}

Here is a plunker with an example (see "names-component.ts").

What am I doing wrong?

ContentLicense: CC BY-SA 4.0


问:
Update nested array value in yaml with yq
Given a yaml file (helmfile) like the following

```<code>releases:
  - chart: ../charts/foo
    name: foo
    namespace: '{{ .Values.stack }}'
    values:
      - ../config/templates/foo-values.yaml.gotmpl
    set:
      - name: image.tag
        value: 22
      - name: replicas
        value: 1
  - chart: ../charts/bar
    name: bar
    namespace: '{{ .Values.stack }}'
    values:
      - ../config/templates/bar-values.yaml.gotmpl
    set:
      - name: image.bar_proxy.tag
        value: 46
      - name: image.bar.tag
        value: 29
      - name: replicas
        value: 1
</code>```

I'm trying to figure out a clean way to update a specific image tag. For example, I'd like to update image.bar_proxy.tag from 46 to 51.

I have the following, which does the job, but it requires that you know the exact index of the array item:

```<code>yq -y '.releases[] |= if .name=="bar" then .set[0].value |= 51 else . end' helmfile-example.yaml
</code>```

So if the array order were to change at some point this would break.

A preferred solution would be: "update image.bar_proxy.tag value from 46 to 51 where set[].name==image.bar_proxy.tag". Any ideas on how to achieve a more specific conditional selection like this?

FYI our yq version:

```<code>$ yq --version
yq 2.10.0
</code>```

答:
You can use the following filter to make it work. It works by dynamically selecting the index of the object where your tag exists. On the selected object .value=51 will update the value as you wanted. You can also use the -i flag to do in-place modification of the original file.

```<code>yq -y '.releases[].set |= map(select(.name == "image.bar_proxy.tag").value=51)' yaml
</code>```

See the underlying jq filter acting on the JSON object at jq-playground

Body:
Given a yaml file (helmfile) like the following

releases:
  - chart: ../charts/foo
    name: foo
    namespace: '{{ .Values.stack }}'
    values:
      - ../config/templates/foo-values.yaml.gotmpl
    set:
      - name: image.tag
        value: 22
      - name: replicas
        value: 1
  - chart: ../charts/bar
    name: bar
    namespace: '{{ .Values.stack }}'
    values:
      - ../config/templates/bar-values.yaml.gotmpl
    set:
      - name: image.bar_proxy.tag
        value: 46
      - name: image.bar.tag
        value: 29
      - name: replicas
        value: 1

I'm trying to figure out a clean way to update a specific image tag. For example, I'd like to update image.bar_proxy.tag from 46 to 51.

I have the following, which does the job, but it requires that you know the exact index of the array item:

yq -y '.releases[] |= if .name=="bar" then .set[0].value |= 51 else . end' helmfile-example.yaml

So if the array order were to change at some point this would break.

A preferred solution would be: "update image.bar_proxy.tag value from 46 to 51 where set[].name==image.bar_proxy.tag". Any ideas on how to achieve a more specific conditional selection like this?

FYI our yq version:

$ yq --version
yq 2.10.0

Body:
You can use the following filter to make it work. It works by dynamically selecting the index of the object where your tag exists. On the selected object .value=51 will update the value as you wanted. You can also use the -i flag to do in-place modification of the original file.

yq -y '.releases[].set |= map(select(.name == "image.bar_proxy.tag").value=51)' yaml

See the underlying jq filter acting on the JSON object at jq-playground


问:
How to add bytes together and validate them against the checksum in python
How do you calculate the checksum of incoming bytes to see if it is a valid packet?
Currently I am reading the bytes and decoding them and receiving the information but before I do that I would like to be able to validate it against the checksum to make sure I do not get any invalid/corrupt packets.

Here is what I currently have

```<code>def batteryConnect(port, baudrate):
    # establishing a serial connection
    ser = serial.Serial(port=port, baudrate= baudrate, bytesize=serial.EIGHTBITS, parity=serial.PARITY_NONE,timeout=3)
    return ser

class VictronBmv700:

    def __init__(self,port,baudrate):
        self.port = port
        self.baudrate = baudrate
        self.parameterDict = dict.fromkeys(["PID","V","I","P","CE","SOC","TTG","Alarm","Relay","AR","BMV","FW","H1","H2","H3",
                                  "H4","H5","H6","H7","H8","H9","H10","H11","H12","H17","H18"])

    def getBMVInfo(self):

        bmvdata = batteryConnect(self.port,self.baudrate)

        #getting the data
        bmvdata.flushInput()

        #getting the message then splitting the key, value pairs
        while True:
            print(bmvdata.readline())

            message = bmvdata.readline().decode(encoding='utf-8',errors='ignore')

            #splitting on tabs
            message_parts = message.split('\t')

            if len(message_parts) > 1:
                key = message_parts[0]
                value = message_parts[1].rstrip()  #stripping \r\n after the value

                #updating deictionary based on keys and their values.
                self.parameterDict[key] = value

if __name__ == "__main__":
    print("BATTERY MONITOR")
    bmv700 = VictronBmv700("COM17", 19200)
    bmv700.getBMVInfo()

</code>```

Which starts outputting the following (from the print(bmvdata.readline()) )

```<code>b'\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd7\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd6\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
</code>```

How would I check the incoming bytes against the checksum then continue to decode after validation?

EDIT
Sometimes I get different checksum values as below.
Here are two additional times I ran the code.

```<code>b'\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd7\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
</code>```

Second run

```<code>b'\r\n'
b'V\t25548\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
b'H1\t-6254\r\n'
b'H3\t0\r\n'
b'H5\t0\r\n'
b'H7\t-10\r\n'
b'H9\t0\r\n'
b'H11\t0\r\n'
b'H17\t20\r\n'
b'Checksum\tT\r\n'
b'V\t25549\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
b'H1\t-6254\r\n'
b'H3\t0\r\n'
b'H5\t0\r\n'
b'H7\t-10\r\n'
b'H9\t0\r\n'
b'H11\t0\r\n'
b'H17\t20\r\n'
b'Checksum\tT\r\n'
b'V\t25548\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
b'H1\t-6254\r\n'
b'H3\t0\r\n'
b'H5\t0\r\n'
b'H7\t-10\r\n'
b'H9\t0\r\n'
b'H11\t0\r\n'
b'H17\t20\r\n'
b'Checksum\tT\r\n'
b'V\t25548\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
b'H1\t-6254\r\n'
b'H3\t0\r\n'
b'H5\t0\r\n'
b'H7\t-10\r\n'
b'H9\t0\r\n'
b'H11\t0\r\n'
b'H17\t20\r\n'
b'Checksum\tT\r\n'
b'V\t25549\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
b'H1\t-6254\r\n'
b'H3\t0\r\n'
b'H5\t0\r\n'
b'H7\t-10\r\n'
b'H9\t0\r\n'
b'H11\t0\r\n'
b'H17\t20\r\n'
b'Checksum\tT\r\n'
b'V\t25548\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
b'H1\t-6254\r\n'
b'H3\t0\r\n'
b'H5\t0\r\n'
b'H7\t-10\r\n'
b'H9\t0\r\n'
b'H11\t0\r\n'
b'H17\t20\r\n'
b'Checksum\tT\r\n'
b'V\t25548\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
b'H1\t-6254\r\n'
b'H3\t0\r\n'
b'H5\t0\r\n'
b'H7\t-10\r\n'
b'H9\t0\r\n'
b'H11\t0\r\n'
b'H17\t20\r\n'
b'Checksum\tT\r\n'
b'V\t25548\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
b'H1\t-6254\r\n'
b'H3\t0\r\n'
b'H5\t0\r\n'
b'H7\t-10\r\n'
b'H9\t0\r\n'
b'H11\t0\r\n'
b'H17\t20\r\n'
b'Checksum\tT\r\n'
b'V\t25548\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
</code>```

EDIT
Ran the following code

```<code>current_block = []
# it's unfortunate that lines start with `\r\n` rather than end with it
# we will read the first empty line separatey, and then include these
# 2 characters in the last line that is read
message = bmvdata.readline()
while True:
    # Note that we cannot decode yet, since the Checksum value may not be a valid utf8 character
    message = bmvdata.readline()

    # we save the message in the current block before any preprocessing
    current_block.append(message)

    #splitting on tabs (as bytes !!)
    message_parts = message.split(b'\t')

    if len(message_parts) > 1:
        key = message_parts[0].decode('utf8')  # here it is safe to decode the key
        value = message_parts[1].rstrip()  #stripping \r\n after the value

    if key == 'Checksum':
        block_chars = b''.join(current_block)
        checksum = sum(block_chars) % 256
        print('Retrieved  checksum', value[-1])
        print('Calculated checksum', checksum)
        print('----')

        # reset the block
        current_block = []
</code>```

output

```<code>Retrieved checksum 84
Calculated checksum 0
-------
Retrieved checksum 219
Calculated checksum 0
-------
Retrieved checksum 84
Calculated checksum 0
-------
Retrieved checksum 220
Calculated checksum 0
-------
Retrieved checksum 84
Calculated checksum 0
-------
Retrieved checksum 219
Calculated checksum 0
-------
Retrieved checksum 84
Calculated checksum 0
-------
Retrieved checksum 219
Calculated checksum 0
-------
Retrieved checksum 84
Calculated checksum 0
-------
Retrieved checksum 220
Calculated checksum 0
-------
Retrieved checksum 84
Calculated checksum 0
-------
Retrieved checksum 220
Calculated checksum 0
-------
Retrieved checksum 84
Calculated checksum 0
-------
Retrieved checksum 220
Calculated checksum 0
-------
Retrieved checksum 84
Calculated checksum 0
-------
Retrieved checksum 219
Calculated checksum 0
-------
Retrieved checksum 84
Calculated checksum 0
-------
</code>```

答:
Below code compare two files in this case checksum as asker mentioned:

```<code> bmvdata = batteryConnect(self.port,self.baudrate)
 bmvdata.flushInput()
 print(bmvdata.readline())
</code>```

It has been done by the aid of hashlib

```<code>import hashlib
#File 1 = checksum
hasher1 = hashlib.md5()
#afile1 = open('checksum', 'rb')
buf1 = bmvdata.read()
#buf1 = afile1.read()
a = hasher1.update(buf1)
md5_a=(str(hasher1.hexdigest()))
#File 2
hasher2 = hashlib.md5()
afile2 = open('incoming-byte', 'rb')
buf2 = afile2.read()
b = hasher2.update(buf2)
md5_b=(str(hasher2.hexdigest()))
#Compare md5
if(md5_a==md5_b):
    print("Yes")
else:
    print("No")
</code>```

Body:
How do you calculate the checksum of incoming bytes to see if it is a valid packet?
Currently I am reading the bytes and decoding them and receiving the information but before I do that I would like to be able to validate it against the checksum to make sure I do not get any invalid/corrupt packets.

Here is what I currently have

def batteryConnect(port, baudrate):
    # establishing a serial connection
    ser = serial.Serial(port=port, baudrate= baudrate, bytesize=serial.EIGHTBITS, parity=serial.PARITY_NONE,timeout=3)
    return ser

class VictronBmv700:

    def __init__(self,port,baudrate):
        self.port = port
        self.baudrate = baudrate
        self.parameterDict = dict.fromkeys(["PID","V","I","P","CE","SOC","TTG","Alarm","Relay","AR","BMV","FW","H1","H2","H3",
                                  "H4","H5","H6","H7","H8","H9","H10","H11","H12","H17","H18"])

    def getBMVInfo(self):

        bmvdata = batteryConnect(self.port,self.baudrate)

        #getting the data
        bmvdata.flushInput()

        #getting the message then splitting the key, value pairs
        while True:
            print(bmvdata.readline())

            message = bmvdata.readline().decode(encoding='utf-8',errors='ignore')

            #splitting on tabs
            message_parts = message.split('\t')

            if len(message_parts) > 1:
                key = message_parts[0]
                value = message_parts[1].rstrip()  #stripping \r\n after the value

                #updating deictionary based on keys and their values.
                self.parameterDict[key] = value

if __name__ == "__main__":
    print("BATTERY MONITOR")
    bmv700 = VictronBmv700("COM17", 19200)
    bmv700.getBMVInfo()

Which starts outputting the following (from the print(bmvdata.readline()) )

b'\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd7\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd6\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'

How would I check the incoming bytes against the checksum then continue to decode after validation?

EDIT
Sometimes I get different checksum values as below.
Here are two additional times I ran the code.

b'\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd7\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd8\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'
b'PID\t0x203\r\n'
b'I\t0\r\n'
b'CE\t0\r\n'
b'TTG\t-1\r\n'
b'Relay\tOFF\r\n'
b'BMV\t700\r\n'
b'Checksum\t\xd0\r\n'
b'H2\t0\r\n'
b'H4\t0\r\n'
b'H6\t-9001\r\n'
b'H8\t28403\r\n'
b'H10\t0\r\n'
b'H12\t0\r\n'
b'H18\t87\r\n'

Second run

b'\r\n'
b'V\t25548\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
b'H1\t-6254\r\n'
b'H3\t0\r\n'
b'H5\t0\r\n'
b'H7\t-10\r\n'
b'H9\t0\r\n'
b'H11\t0\r\n'
b'H17\t20\r\n'
b'Checksum\tT\r\n'
b'V\t25549\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
b'H1\t-6254\r\n'
b'H3\t0\r\n'
b'H5\t0\r\n'
b'H7\t-10\r\n'
b'H9\t0\r\n'
b'H11\t0\r\n'
b'H17\t20\r\n'
b'Checksum\tT\r\n'
b'V\t25548\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
b'H1\t-6254\r\n'
b'H3\t0\r\n'
b'H5\t0\r\n'
b'H7\t-10\r\n'
b'H9\t0\r\n'
b'H11\t0\r\n'
b'H17\t20\r\n'
b'Checksum\tT\r\n'
b'V\t25548\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
b'H1\t-6254\r\n'
b'H3\t0\r\n'
b'H5\t0\r\n'
b'H7\t-10\r\n'
b'H9\t0\r\n'
b'H11\t0\r\n'
b'H17\t20\r\n'
b'Checksum\tT\r\n'
b'V\t25549\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
b'H1\t-6254\r\n'
b'H3\t0\r\n'
b'H5\t0\r\n'
b'H7\t-10\r\n'
b'H9\t0\r\n'
b'H11\t0\r\n'
b'H17\t20\r\n'
b'Checksum\tT\r\n'
b'V\t25548\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
b'H1\t-6254\r\n'
b'H3\t0\r\n'
b'H5\t0\r\n'
b'H7\t-10\r\n'
b'H9\t0\r\n'
b'H11\t0\r\n'
b'H17\t20\r\n'
b'Checksum\tT\r\n'
b'V\t25548\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
b'H1\t-6254\r\n'
b'H3\t0\r\n'
b'H5\t0\r\n'
b'H7\t-10\r\n'
b'H9\t0\r\n'
b'H11\t0\r\n'
b'H17\t20\r\n'
b'Checksum\tT\r\n'
b'V\t25548\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'
b'H1\t-6254\r\n'
b'H3\t0\r\n'
b'H5\t0\r\n'
b'H7\t-10\r\n'
b'H9\t0\r\n'
b'H11\t0\r\n'
b'H17\t20\r\n'
b'Checksum\tT\r\n'
b'V\t25548\r\n'
b'P\t0\r\n'
b'SOC\t1000\r\n'
b'Alarm\tOFF\r\n'
b'AR\t0\r\n'
b'FW\t0310\r\n'

EDIT
Ran the following code

current_block = []
# it's unfortunate that lines start with `\r\n` rather than end with it
# we will read the first empty line separatey, and then include these
# 2 characters in the last line that is read
message = bmvdata.readline()
while True:
    # Note that we cannot decode yet, since the Checksum value may not be a valid utf8 character
    message = bmvdata.readline()

    # we save the message in the current block before any preprocessing
    current_block.append(message)

    #splitting on tabs (as bytes !!)
    message_parts = message.split(b'\t')

    if len(message_parts) > 1:
        key = message_parts[0].decode('utf8')  # here it is safe to decode the key
        value = message_parts[1].rstrip()  #stripping \r\n after the value

    if key == 'Checksum':
        block_chars = b''.join(current_block)
        checksum = sum(block_chars) % 256
        print('Retrieved  checksum', value[-1])
        print('Calculated checksum', checksum)
        print('----')

        # reset the block
        current_block = []

output

Retrieved checksum 84
Calculated checksum 0
-------
Retrieved checksum 219
Calculated checksum 0
-------
Retrieved checksum 84
Calculated checksum 0
-------
Retrieved checksum 220
Calculated checksum 0
-------
Retrieved checksum 84
Calculated checksum 0
-------
Retrieved checksum 219
Calculated checksum 0
-------
Retrieved checksum 84
Calculated checksum 0
-------
Retrieved checksum 219
Calculated checksum 0
-------
Retrieved checksum 84
Calculated checksum 0
-------
Retrieved checksum 220
Calculated checksum 0
-------
Retrieved checksum 84
Calculated checksum 0
-------
Retrieved checksum 220
Calculated checksum 0
-------
Retrieved checksum 84
Calculated checksum 0
-------
Retrieved checksum 220
Calculated checksum 0
-------
Retrieved checksum 84
Calculated checksum 0
-------
Retrieved checksum 219
Calculated checksum 0
-------
Retrieved checksum 84
Calculated checksum 0
-------

Body:
Below code compare two files in this case checksum as asker mentioned:

 bmvdata = batteryConnect(self.port,self.baudrate)
 bmvdata.flushInput()
 print(bmvdata.readline())

It has been done by the aid of hashlib

import hashlib
#File 1 = checksum
hasher1 = hashlib.md5()
#afile1 = open('checksum', 'rb')
buf1 = bmvdata.read()
#buf1 = afile1.read()
a = hasher1.update(buf1)
md5_a=(str(hasher1.hexdigest()))
#File 2
hasher2 = hashlib.md5()
afile2 = open('incoming-byte', 'rb')
buf2 = afile2.read()
b = hasher2.update(buf2)
md5_b=(str(hasher2.hexdigest()))
#Compare md5
if(md5_a==md5_b):
    print("Yes")
else:
    print("No")


问:
How do I troubleshoot "simplexml_load_file() parser error: Entity 'nbsp' not defined"?
I use PHP to generate XML files. I have use some code below to avoid error.

```<code>$str = str_ireplace(array('<','>','&','\'','"'),array('<','>','&',''','"'),$str);
</code>```

but still cause fault.

```<code>simplexml_load_file() [function.simplexml-load-file] *[file name]* parser error : Entity 'nbsp' not defined in *[file name] [line]*
</code>```

The error text here:

```<code>Dallas  Dallas () is the third-largest city in Texas and the ninth-largest in the United States.
</code>```

In IE8, it seems to fault in (). So how many symbols should I notice?

Body:
I use PHP to generate XML files. I have use some code below to avoid error.

$str = str_ireplace(array('<','>','&','\'','"'),array('<','>','&',''','"'),$str);

but still cause fault.

simplexml_load_file() [function.simplexml-load-file] *[file name]* parser error : Entity 'nbsp' not defined in *[file name] [line]*

The error text here:

Dallas  Dallas () is the third-largest city in Texas and the ninth-largest in the United States.

In IE8, it seems to fault in (). So how many symbols should I notice?

Body: &nbsp; is no-breaking space. You have to replace it. http://en.wikipedia.org/wiki/Non-breaking_space


问:
Can we automate ETL in Azure?
I am currently working on a very interesting ETL project using Azure to transform my data manually. However, transforming data manually can be exhausting and lengthy when I start having several source files to process. My pipeline is working fine for now because I have only a few files to transform but what if I have thousands of excel files?

So what I want to achieve is that I want to extend the project and extract the excel files that are coming from Email using the logic app then apply ETL directly on top of them. Is there any way I can automate ETL in Azure. Can I do ETL without modifying the pipeline for a different type of data manually? How can I make my pipeline flexible to be able to handle data transformation for various types of source data?

Thank you in advance for your help.

答:
Can I do ETL without modifying the pipeline for a different type of
  data manually?

According to your description, i suppose that you already knew the ADF connector is supported in the Logic App. You could execute ADF pipeline in the Logic App flow and even pass parameters into ADF pipeline.

Normally, the source and sink service should be fixed in one copy activity, but you could define dynamic file path in the datasets. So you don't need to create multiple copy activities.

If the data types are different, you could try to pass the parameter from Logic App into ADF. Then before the data transmission, you could use Switch activity to route the transmission into different branches.

Title: Can we automate ETL in Azure?


问:
How to convert JSON string column row into a queryable table
I have exported to BigQuery from Firestore a whole collection to perform certain queries on it.

After the data was populated in my BigQuery console, now I can query the whole set like this

```<code>SELECT *
FROM `myapp-1a602.firestore_orders.orders_raw_changelog`
LIMIT 1000
</code>```

Now, this statement throws my different columns, but the one I'm looking for is the data column, in my data column is each document JSON, but is in json format and I need to query all this values.

Now, this is the data from one row

```<code>{
    "cart": [{
        "qty": 1,
        "description": "Sprite 1 L",
        "productName": "Sprite 1 Liter",
        "price": 1.99,
        "productId": 9
    }],
    "storeName": "My awesome shop",
    "status": 5,
    "timestamp": {
        "_seconds": 1590713204,
        "_nanoseconds": 916000000
    }
}
</code>```

This data is inside the data column, so if I do this

```<code>SELECT data
FROM `myapp-1a602.firestore_orders.orders_raw_changelog`
LIMIT 1000
</code>```

I will get all the json values for each document, but I don't know how to query that values, lets say I want to know all orders with status 5 and shopName My awesome shop , now, I need to do something with this json to convert it into a table ? does I need to perform the query in the json itself ?

How can I query this json output ?

Thanks

答:
You canwork with the json functiosn like the

```<code>CrEATE Table products (id Integer,attribs_json JSON );
</code>```

```<code>INSERT INTO products VALUES (1,'{
    "cart": [{
        "qty": 1,
        "description": "Sprite 1 L",
        "productName": "Sprite 1 Liter",
        "price": 1.99,
        "productId": 9
    }],
    "storeName": "My awesome shop",
    "status": 5,
    "timestamp": {
        "_seconds": 1590713204,
        "_nanoseconds": 916000000
    }
}');
</code>```

```<code>select * from products where attribs_json->"$.status"
= 5 AND attribs_json->"$.storeName"
= 'My awesome shop';
</code>```
```
id | attribs_json
-: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 1 | {"cart": [{"qty": 1, "price": 1.99, "productId": 9, "description": "Sprite 1 L", "productName": "Sprite 1 Liter"}], "status": 5, "storeName": "My awesome shop", "timestamp": {"_seconds": 1590713204, "_nanoseconds": 916000000}}
```

db<>fiddle here

```<code>select attribs_json->"$.storeName",attribs_json->"$.status",attribs_json->"$.cart[0].qty" from products where attribs_json->"$.status"
= 5 AND attribs_json->"$.storeName"
= 'My awesome shop';
</code>```
```
attribs_json->"$.storeName" | attribs_json->"$.status" | attribs_json->"$.cart[0].qty"
:-------------------------- | :----------------------- | :----------------------------
"My awesome shop"           | 5                        | 1
```

db<>fiddle here

And there is JSON_EXTRACT  for mysql 5.7 and above.

Finally that is in the end only text, so you could use also REGEXP or RLIKE

To transfer the jaso again to rows, you can use JSON_TABLE

Body:
I have exported to BigQuery from Firestore a whole collection to perform certain queries on it.

After the data was populated in my BigQuery console, now I can query the whole set like this

SELECT *
FROM `myapp-1a602.firestore_orders.orders_raw_changelog`
LIMIT 1000

Now, this statement throws my different columns, but the one I'm looking for is the data column, in my data column is each document JSON, but is in json format and I need to query all this values.

Now, this is the data from one row

{
    "cart": [{
        "qty": 1,
        "description": "Sprite 1 L",
        "productName": "Sprite 1 Liter",
        "price": 1.99,
        "productId": 9
    }],
    "storeName": "My awesome shop",
    "status": 5,
    "timestamp": {
        "_seconds": 1590713204,
        "_nanoseconds": 916000000
    }
}

This data is inside the data column, so if I do this

SELECT data
FROM `myapp-1a602.firestore_orders.orders_raw_changelog`
LIMIT 1000

I will get all the json values for each document, but I don't know how to query that values, lets say I want to know all orders with status 5 and shopName My awesome shop , now, I need to do something with this json to convert it into a table ? does I need to perform the query in the json itself ?

How can I query this json output ?

Thanks

Body:
You canwork with the json functiosn like the

CrEATE Table products (id Integer,attribs_json JSON );

INSERT INTO products VALUES (1,'{
    "cart": [{
        "qty": 1,
        "description": "Sprite 1 L",
        "productName": "Sprite 1 Liter",
        "price": 1.99,
        "productId": 9
    }],
    "storeName": "My awesome shop",
    "status": 5,
    "timestamp": {
        "_seconds": 1590713204,
        "_nanoseconds": 916000000
    }
}');

select * from products where attribs_json->"$.status"
= 5 AND attribs_json->"$.storeName"
= 'My awesome shop';

id | attribs_json
-: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 1 | {"cart": [{"qty": 1, "price": 1.99, "productId": 9, "description": "Sprite 1 L", "productName": "Sprite 1 Liter"}], "status": 5, "storeName": "My awesome shop", "timestamp": {"_seconds": 1590713204, "_nanoseconds": 916000000}}

db<>fiddle here

select attribs_json->"$.storeName",attribs_json->"$.status",attribs_json->"$.cart[0].qty" from products where attribs_json->"$.status"
= 5 AND attribs_json->"$.storeName"
= 'My awesome shop';

attribs_json->"$.storeName" | attribs_json->"$.status" | attribs_json->"$.cart[0].qty"
:-------------------------- | :----------------------- | :----------------------------
"My awesome shop"           | 5                        | 1

db<>fiddle here

And there is JSON_EXTRACT  for mysql 5.7 and above.

Finally that is in the end only text, so you could use also REGEXP or RLIKE

To transfer the jaso again to rows, you can use JSON_TABLE


问:
Azure Queue Trigger Functions Not picking the queued message
The queue function is working fine in the developer machine. But once published to Azure, after a day or two, the function is not picking the queue message. the queue message is displayed in the Storage Account queue and not dequeued.

When we are using the Run command in monitor then it is picking the queue message and processing successfully.

For your Information: Not access issues, all application and id and secrets are correctly configured, all hosted in the same tenant, No exceptions generated.

答:
It's been a long time since I have a similar problem, but what worked for me when some function went to sleep was:

Regenerate the connection string.

Remove the resource (function) and create a new one.

I don't really know why sometimes some functions goes idle and then it no longer trigger.


问:
Python method lookup, static vs. instance
Until like one hour ago, I was convinced that in python Foo ().bar () was nothing more than a short hand for Foo.bar (Foo () ) which passes the instance as first parameter. In this example the last two lines do (apparently) the same thing:

```<code>class Foo (object):
    def bar (self): print "baz"

qux = Foo ()
qux.bar ()
Foo.bar (qux)
</code>```

But now I have a class Animal that has a static method populate() that returns a list of all animals known to man. Also each instance of Animal has a method populate() that fills the properties of the instance with random values.

```<code>#! /usr/bin/env python
# -*- coding: utf-8 -*-

import random

animals = [ ("Bella", "cow"), ("Spike", "dog"), ("José", "iguana"), ("Tux", "penguin") ]

class Animal (object):
    @staticmethod
    def populate (*args): return map (lambda x: Animal (*x), animals)

    def __init__ (self, name = None, species = None):
        def bar (): self.name, self.species = random.choice (animals)
        self.name = name
        self.species = species
        self.populate = bar

    def __repr__ (self): return "%s of species %s" % (self.name, self.species)

print Animal.populate ()
print Animal ("Pinky", "mouse")
qux = Animal ()
qux.populate ()
print qux
</code>```

The code works fine, but what made me suspicious was the fact that print Animal.populate (qux) called the static populate method (and hence returned a list and did not populate poor qux). So apparently my conviction that Foo ().bar () was nothing more than a short hand for Foo.bar (Foo () ) is wrong.

This raises various questions for me:

What happens when I call Foo ().bar ()?

What happens when I call Foo.bar (Foo () )?

Is there an internal difference between the both?

Am I missing some fundamental concept of python?

If you had to write a class whose static populate method does something else than the populate method invoked over an instance of this class, which would be the way to go?

(Yes it must be the same name.)

答:
On the difference between Foo().bar(), Foo.bar(Foo()) and Foo.bar() (as an answer because I signed up yesterday and can't post comments yet) - this is because of Python(<3.0)'s concept of 'bound' and 'unbound' methods - it strictly requires that, except with @staticmethod or @classmethod, method calls have an instance associated with them. There's not really any easier way to explain it than just something you have to remember.
Thankfully, this has changed in Python 3 - the concept of 'bound' and 'unbound' methods as separate things has gone, and Foo.bar() works just fine for your example.</p>

Body:
Until like one hour ago, I was convinced that in python Foo ().bar () was nothing more than a short hand for Foo.bar (Foo () ) which passes the instance as first parameter. In this example the last two lines do (apparently) the same thing:

class Foo (object):
    def bar (self): print "baz"

qux = Foo ()
qux.bar ()
Foo.bar (qux)

But now I have a class Animal that has a static method populate() that returns a list of all animals known to man. Also each instance of Animal has a method populate() that fills the properties of the instance with random values.

#! /usr/bin/env python
# -*- coding: utf-8 -*-

import random

animals = [ ("Bella", "cow"), ("Spike", "dog"), ("José", "iguana"), ("Tux", "penguin") ]

class Animal (object):
    @staticmethod
    def populate (*args): return map (lambda x: Animal (*x), animals)

    def __init__ (self, name = None, species = None):
        def bar (): self.name, self.species = random.choice (animals)
        self.name = name
        self.species = species
        self.populate = bar

    def __repr__ (self): return "%s of species %s" % (self.name, self.species)

print Animal.populate ()
print Animal ("Pinky", "mouse")
qux = Animal ()
qux.populate ()
print qux

The code works fine, but what made me suspicious was the fact that print Animal.populate (qux) called the static populate method (and hence returned a list and did not populate poor qux). So apparently my conviction that Foo ().bar () was nothing more than a short hand for Foo.bar (Foo () ) is wrong.

This raises various questions for me:

What happens when I call Foo ().bar ()?

What happens when I call Foo.bar (Foo () )?

Is there an internal difference between the both?

Am I missing some fundamental concept of python?

If you had to write a class whose static populate method does something else than the populate method invoked over an instance of this class, which would be the way to go?

(Yes it must be the same name.)


问:
ansible check lineinfile for string fails
Really hoping someone is able to help, I've got this task, that checks if string XYZ exists in a log

if it doesn't exists, it should fail (ignore_errors: yes) then kick another task, however, even when it fails, it skips the other task

```<code> - name: check if file is empty
       lineinfile:
          path: /path/to/log.log
          line: "XYZ"
          state: present
       check_mode: yes
       register: exists
       failed_when: exists is changed
       ignore_errors: yes

     - name: send mail
       when: exists is not changed
       mail:
          from: mail@123.com
          subject: xyz
          body: No xyz found
          to:
           - John Doe <john.doe@123.com>
          cc: Marie Smith <marie.smit@123.com>

     - name: read files
       when: exists is changed
       do abc...
</code>```

the issue is that when the file is empty and doesn't have XYZ, it still fails, but skips the mail send and carries on with the read files task

```<code>
TASK [check if file is empty] ***********************************************************************************************************************************************************************************

fatal: [server123]: FAILED! => {"backup": "", "changed": true, "failed_when_result": true, "msg": "line added"}

...ignoring

TASK [send mail] ************************************************************************************************************************************************************************************************

skipping: [server123]

TASK [read files] ***********************************************************************************************************************************************************************************************

changed: [server123]
</code>```

答:
If I understood your expectation correctly then you can achieve the result using grep and command module also. Here is a sample:

```<code>- name: check if file is empty
  command: grep -xq 'XYZ' /path/to/log.log
  register: exists
  ignore_errors: yes

- name: send mail
  mail:
    from: mail@123.com
    subject: xyz
    body: No xyz found
    to:
      - John Doe <john.doe@123.com>
    cc: Marie Smith <marie.smit@123.com>
  when: exists.failed

- name: read files
  debug:
    msg: do abc...
  when: not exists.failed
</code>```

You can adjust the grep logic that suits.

Body:
Really hoping someone is able to help, I've got this task, that checks if string XYZ exists in a log

if it doesn't exists, it should fail (ignore_errors: yes) then kick another task, however, even when it fails, it skips the other task

 - name: check if file is empty
       lineinfile:
          path: /path/to/log.log
          line: "XYZ"
          state: present
       check_mode: yes
       register: exists
       failed_when: exists is changed
       ignore_errors: yes

     - name: send mail
       when: exists is not changed
       mail:
          from: mail@123.com
          subject: xyz
          body: No xyz found
          to:
           - John Doe <john.doe@123.com>
          cc: Marie Smith <marie.smit@123.com>

     - name: read files
       when: exists is changed
       do abc...

the issue is that when the file is empty and doesn't have XYZ, it still fails, but skips the mail send and carries on with the read files task

TASK [check if file is empty] ***********************************************************************************************************************************************************************************

fatal: [server123]: FAILED! => {"backup": "", "changed": true, "failed_when_result": true, "msg": "line added"}

...ignoring

TASK [send mail] ************************************************************************************************************************************************************************************************

skipping: [server123]

TASK [read files] ***********************************************************************************************************************************************************************************************

changed: [server123]

Body:
If I understood your expectation correctly then you can achieve the result using grep and command module also. Here is a sample:

- name: check if file is empty
  command: grep -xq 'XYZ' /path/to/log.log
  register: exists
  ignore_errors: yes

- name: send mail
  mail:
    from: mail@123.com
    subject: xyz
    body: No xyz found
    to:
      - John Doe <john.doe@123.com>
    cc: Marie Smith <marie.smit@123.com>
  when: exists.failed

- name: read files
  debug:
    msg: do abc...
  when: not exists.failed

You can adjust the grep logic that suits.


问:
Getting UnboundLocalError in data exploration coding
```<code>mt=0
ft=0
def amount_total(g):
    if g=="male":
        mt=mt+amount
    else:
        ft=ft+amount

df['gender'].apply(amount_total)
</code>```

Error:

```<code>UnboundLocalError                         Traceback (most recent call last)
<ipython-input-19-5a85616e6d1c> in <module>
      7         ft=ft+amount
      8
----> 9 df['gender'].apply(amount_total)

~\Anaconda3\lib\site-packages\pandas\core\series.py in apply(self, func, convert_dtype, args, **kwds)
   4040             else:
   4041                 values = self.astype(object).values
-> 4042                 mapped = lib.map_infer(values, f, convert=convert_dtype)
   4043
   4044         if len(mapped) and isinstance(mapped[0], Series):

pandas\_libs\lib.pyx in pandas._libs.lib.map_infer()

<ipython-input-19-5a85616e6d1c> in amount_total(g)
      3 def amount_total(g):
      4     if g=="male":
----> 5         mt=mt+amount
      6     else:
      7         ft=ft+amount

UnboundLocalError: local variable 'mt' referenced before assignment
</code>```

答:
mt and ft are global variables, maybe you need to state that:

```<code>mt=0
ft=0
def amount_total(g):
    global mt
    global ft
    if g=="male":
        mt=mt+amount   // what's amount anyway?
    else:
        ft=ft+amount

df['gender'].apply(amount_total)
</code>```

However, you should do this instead of apply:

```<code>mt, ft = df['gender'].value_counts() * amount
</code>```

Body:
mt=0
ft=0
def amount_total(g):
    if g=="male":
        mt=mt+amount
    else:
        ft=ft+amount

df['gender'].apply(amount_total)

Error:

UnboundLocalError                         Traceback (most recent call last)
<ipython-input-19-5a85616e6d1c> in <module>
      7         ft=ft+amount
      8
----> 9 df['gender'].apply(amount_total)

~\Anaconda3\lib\site-packages\pandas\core\series.py in apply(self, func, convert_dtype, args, **kwds)
   4040             else:
   4041                 values = self.astype(object).values
-> 4042                 mapped = lib.map_infer(values, f, convert=convert_dtype)
   4043
   4044         if len(mapped) and isinstance(mapped[0], Series):

pandas\_libs\lib.pyx in pandas._libs.lib.map_infer()

<ipython-input-19-5a85616e6d1c> in amount_total(g)
      3 def amount_total(g):
      4     if g=="male":
----> 5         mt=mt+amount
      6     else:
      7         ft=ft+amount

UnboundLocalError: local variable 'mt' referenced before assignment

Body:
mt and ft are global variables, maybe you need to state that:

mt=0
ft=0
def amount_total(g):
    global mt
    global ft
    if g=="male":
        mt=mt+amount   // what's amount anyway?
    else:
        ft=ft+amount

df['gender'].apply(amount_total)

However, you should do this instead of apply:

mt, ft = df['gender'].value_counts() * amount


问:
PHP mySQL JOIN and display (3 tables)
Hi I am new in php and for my study project trying to resolve some challenges

I have 3 tables "products", "categories", "product_to_categories"

products table:

```<code>product_id    product_name    product_price    product_img    product_link    date
1             item1_name      item1_price      item1_img      item1_link      item1_date
2             item2_name      item2_price      item2_img      item2_link      item2_date
</code>```

categories table:

```<code>category_id    category_name
1              category1_name
2              category2_name
3              category3_name
</code>```

and product_to_categories table:

```<code>relation_id    product_id    category_id
1              product1_id   category1_id
2              product1_id   category2_id
3              product2_id   category1_id
4              product2_id   category3_id
</code>```

I am trying to get 2 things:

write a list of all products together with the all categories names (no ids) they belong to.

list only those products that belong to the given categories

Unfortunately, I don't know how to approach this.

show_products.php:

<?php
$per_page=100;

    if(isset($_GET['page'])){
        $page = $_GET['page'];
    }else{
        $page=1;
    }
    $start_from = ($page-1) * $per_page;

    $get_products = "SELECT * FROM products ORDER BY 1 DESC LIMIT $start_from,$per_page";

    $run_products = mysqli_query($conn,$get_products);

    while($row_products=mysqli_fetch_array($run_products)){

        $pro_id = $row_products['product_id'];
        $pro_name = $row_products['product_name'];
        $pro_price = $row_products['product_price'];
        $pro_img = $row_products['product_img'];
        $pro_link = $row_products['product_link'];
        $pro_date = $row_products['date'];

        echo "

            $pro_id

            $pro_name
            ok. $pro_price zł
            $pro_link

            $pro_date

        ";
    }
?>```

答:
For the first question: You should use the JOIN clause to be able to relate products to categories through their id:

```<code>SELECT
    p.product_name,
    c.category_name,
FROM
    products p
JOIN
    products_categories pc ON p.product_id = pc.product_id
JOIN
    categories c ON pc.category_id = c.category_id
</code>```

For the second question: When you relate two tables, you can filter the results of the table you are joining by adding a where clause in the join:

```<code>SELECT
    p.product_name
FROM
    products p
JOIN
    products_categories pc ON p.product_id = pc.product_id AND pc.category_id = 1
</code>```

Title: PHP mySQL JOIN and display (3 tables)

Body:
Hi I am new in php and for my study project trying to resolve some challenges

I have 3 tables "products", "categories", "product_to_categories"

products table:

product_id    product_name    product_price    product_img    product_link    date
1             item1_name      item1_price      item1_img      item1_link      item1_date
2             item2_name      item2_price      item2_img      item2_link      item2_date

categories table:

category_id    category_name
1              category1_name
2              category2_name
3              category3_name

and product_to_categories table:

relation_id    product_id    category_id
1              product1_id   category1_id
2              product1_id   category2_id
3              product2_id   category1_id
4              product2_id   category3_id

I am trying to get 2 things:

write a list of all products together with the all categories names (no ids) they belong to.

list only those products that belong to the given categories

Unfortunately, I don't know how to approach this.

show_products.php:

<?php
$per_page=100;

    if(isset($_GET['page'])){
        $page = $_GET['page'];
    }else{
        $page=1;
    }
    $start_from = ($page-1) * $per_page;

    $get_products = "SELECT * FROM products ORDER BY 1 DESC LIMIT $start_from,$per_page";

    $run_products = mysqli_query($conn,$get_products);

    while($row_products=mysqli_fetch_array($run_products)){

        $pro_id = $row_products['product_id'];
        $pro_name = $row_products['product_name'];
        $pro_price = $row_products['product_price'];
        $pro_img = $row_products['product_img'];
        $pro_link = $row_products['product_link'];
        $pro_date = $row_products['date'];

        echo "

            $pro_id

            $pro_name
            ok. $pro_price zł
            $pro_link

            $pro_date

        ";
    }
?>

Body:
For the first question: You should use the JOIN clause to be able to relate products to categories through their id:

SELECT
    p.product_name,
    c.category_name,
FROM
    products p
JOIN
    products_categories pc ON p.product_id = pc.product_id
JOIN
    categories c ON pc.category_id = c.category_id

For the second question: When you relate two tables, you can filter the results of the table you are joining by adding a where clause in the join:

SELECT
    p.product_name
FROM
    products p
JOIN
    products_categories pc ON p.product_id = pc.product_id AND pc.category_id = 1


Title: How do I generate a pandas dataframe consisting of the adjusted close prices of a few stocks from yahoo finance?

Body:
The df example end result I am looking for is the below, with the adjusted close prices as the data:

#               GE    JPM    MSFT     PG
#date
#2015-01-02  25.06  62.49  46.760  90.44
#2015-01-05  24.60  60.55  46.325  90.01
#2015-01-06  24.07  58.98  45.650  89.60
#2015-01-07  24.08  59.07  46.230  90.07
#2015-01-08  24.37  60.39  47.590  91.10

import yfinance as yf
import datetime
import pandas as pd
import pandas_datareader as pdr

def get(tickers, startdate, enddate):
  def data(ticker):
    return (pdr.get_data_yahoo(ticker, start=startdate, end=enddate))
  datas = map (data, tickers)
  return(pd.concat(datas, keys=tickers, names=['Ticker', 'Date']))

ticker = ['GE' ,   'JPM' ,   'MSFT' ,    'PG']
all_data = get(ticker, datetime.datetime(2018, 10, 1), datetime.datetime(2020, 1, 1))

print (all_data)

                          High          Low  ...      Volume    Adj Close
Ticker Date                                  ...
GE     2018-10-01   229.419998   226.350006  ...  23600800.0   221.988266
       2018-10-02   230.000000   226.630005  ...  24788200.0   223.961426
       2018-10-03   233.470001   229.779999  ...  28654800.0   226.686707
       2018-10-04   232.350006   226.729996  ...  32042000.0   222.701340
       2018-10-05   228.410004   220.580002  ...  33580500.0   219.087158
...                        ...          ...  ...         ...          ...
JPM    2019-12-24  1350.260010  1342.780029  ...    347500.0  1343.560059
       2019-12-26  1361.327026  1344.469971  ...    667500.0  1360.400024
       2019-12-27  1364.530029  1349.310059  ...   1038400.0  1351.890015
       2019-12-30  1353.000000  1334.020020  ...   1050900.0  1336.140015
       2019-12-31  1338.000000  1329.084961  ...    961800.0  1337.020020

The above code stores the tickers in rows instead of columns.

I only need the adjusted close column for each ticker. I know how to extract the "adjusted close" column into another pandas series, but I get stuck on how to proceed further.

Body:
You're looking for pivot_table() on the ungrouped data.

df

  Ticker        Date         High          Low  Volume       Adj Close
0     GE   10/1/2018   229.419998   226.350006    23600800   221.988266
1     GE   10/2/2018   230.000000   226.630005    24788200   223.961426
2     GE   10/3/2018   233.470001   229.779999    28654800   226.686707
3     GE   10/4/2018   232.350006   226.729996    32042000   222.701340
4     GE   10/5/2018   228.410004   220.580002    33580500   219.087158
5    JPM  12/24/2019  1350.260010  1342.780029      347500  1343.560059
6    JPM  12/26/2019  1361.327026  1344.469971      667500  1360.400024
7    JPM  12/27/2019  1364.530029  1349.310059     1038400  1351.890015
8    JPM  12/30/2019  1353.000000  1334.020020     1050900  1336.140015
9    JPM  12/31/2019  1338.000000  1329.084961      961800  1337.020020

df.pivot_table(index='Date', columns='Ticker', values='Close')

Output

Ticker              GE          JPM
Date
10/1/2018   221.988266          NaN
10/2/2018   223.961426          NaN
10/3/2018   226.686707          NaN
10/4/2018   222.701340          NaN
10/5/2018   219.087158          NaN
12/24/2019         NaN  1343.560059
12/26/2019         NaN  1360.400024
12/27/2019         NaN  1351.890015
12/30/2019         NaN  1336.140015
12/31/2019         NaN  1337.020020


问:
pod update will not update to latest version of pod no matter what I do
I have looked at the related Stack Overflow questions and tried the recommendations there to no avail as well as cocoapods's github issues.

I manage the TensorIOTensorFlow pod and recently released two updates to that pod, 1.15 and 1.15.1. Both releases were tagged on GitHub and pushed to trunk after linting using pod trunk push TensorIOTensorFlow.podspec from the project's root directory.

See https://github.com/doc-ai/tensorio-tensorflow-ios

It has now been more than twelve hours since I released the 1.15.1 update and no matter what I do I cannot get pod update to update a local project from the 1.15 version to the 1.15.1 version or pod install to install anything other than the 1.15 version.

I have tried clearing the cocoapods cache:

```<code>$ rm -rf "${HOME}/Library/Caches/CocoaPods"
$ rm -rf ~/.cocoapods/repos
$ pod cache clean --all
</code>```

I have tried updating my local spec repo:

```<code>$ pod install --repo-update
$ pod update --repo-update
$ pod repo update
</code>```

I have tried both pod update and pod install after removing the project's Pods directory and Podfile.lock. I have tried both of these in an existing project and a new project and on a separate machine with a first time install of cocoapods.

I have tried targeting the version specifically in the pod file:

```<code>pod 'TensorIOTensorFlow', '1.15.1'
</code>```

Only to get the following error:

```<code>[!] CocoaPods could not find compatible versions for pod "TensorIOTensorFlow":
  In Podfile:
    TensorIOTensorFlow (= 1.15.1)

None of your spec sources contain a spec satisfying the dependency: `TensorIOTensorFlow (= 1.15.1)`.

You have either:
 * mistyped the name or version.
 * not added the source repo that hosts the Podspec to your Podfile.
</code>```

In spite of that error, and particularly frustrating, trunk seems to be aware of the 1.15.1 version:

```<code>$ pod trunk info TensorIOTensorFlow

TensorIOTensorFlow
    - Versions:
      - 0.1.0 (2019-04-09 19:29:30 UTC)
      - 0.1.1 (2019-05-09 17:43:06 UTC)
      - 0.3.0 (2019-12-18 18:44:18 UTC)
      - 0.3.1 (2020-01-23 23:36:35 UTC)
      - 1.13.4 (2020-05-01 21:16:32 UTC)
      - 1.13.5 (2020-05-08 21:13:09 UTC)
      - 1.13.6 (2020-05-26 20:05:09 UTC)
      - 1.15 (2020-05-27 17:06:58 UTC)
      - 1.15.1 (2020-05-27 23:44:11 UTC)
</code>```

But pod search does not see it:

```<code>$ pod search TensorIOTensorFlow

-> TensorIOTensorFlow (1.15)
   The TensorFlow (unofficial) build used by TensorIO for iOS.
   pod 'TensorIOTensorFlow', '~> 1.15'
   - Homepage: https://github.com/doc-ai/tensorio-tensorflow-ios
   - Source:   https://storage.googleapis.com/tensorio-build/r1.15/TensorIO-TensorFlow-1.15_0.tar.gz
   - Versions: 1.15, 1.13.6, 1.13.5, 1.13.4, 0.3.1, 0.3.0, 0.1.1, 0.1.0 [trunk repo]
</code>```

To be sure, this is after I deleted the 1.15 and 1.15.1 versions with pod delete and tried pushing them to trunk again. Before that, pod search did show the 1.15.1 version but I still couldn't get a project to upgrade to it.

Finally, I have tried uninstalling and re-installing cocoapods, and as I mentioned, I have also tried all of this on a new machine with a first time install of the latest version of cocoapods, and I still cannot get the 1.15.1 version of the pod.

答:
Here is the issue I have opened:
https://github.com/CocoaPods/CocoaPods/issues/9827

And, I have followed @paul-beusterien answer. And here is a sample

```<code>source 'https://github.com/CocoaPods/Specs.git'
platform :ios, '10.0'

use_frameworks!

target 'YourTarget' do
  inherit! :search_paths
  # Pods for YourTarget
  pod 'YourPod', '10.3.0'
end
</code>```

So I added

```<code>source 'https://github.com/CocoaPods/Specs.git'
</code>```

on the top of my Podfile

Thanks :)

Body:
I have looked at the related Stack Overflow questions and tried the recommendations there to no avail as well as cocoapods's github issues.

I manage the TensorIOTensorFlow pod and recently released two updates to that pod, 1.15 and 1.15.1. Both releases were tagged on GitHub and pushed to trunk after linting using pod trunk push TensorIOTensorFlow.podspec from the project's root directory.

See https://github.com/doc-ai/tensorio-tensorflow-ios

It has now been more than twelve hours since I released the 1.15.1 update and no matter what I do I cannot get pod update to update a local project from the 1.15 version to the 1.15.1 version or pod install to install anything other than the 1.15 version.

I have tried clearing the cocoapods cache:

$ rm -rf "${HOME}/Library/Caches/CocoaPods"
$ rm -rf ~/.cocoapods/repos
$ pod cache clean --all

I have tried updating my local spec repo:

$ pod install --repo-update
$ pod update --repo-update
$ pod repo update

I have tried both pod update and pod install after removing the project's Pods directory and Podfile.lock. I have tried both of these in an existing project and a new project and on a separate machine with a first time install of cocoapods.

I have tried targeting the version specifically in the pod file:

pod 'TensorIOTensorFlow', '1.15.1'

Only to get the following error:

[!] CocoaPods could not find compatible versions for pod "TensorIOTensorFlow":
  In Podfile:
    TensorIOTensorFlow (= 1.15.1)

None of your spec sources contain a spec satisfying the dependency: `TensorIOTensorFlow (= 1.15.1)`.

You have either:
 * mistyped the name or version.
 * not added the source repo that hosts the Podspec to your Podfile.

In spite of that error, and particularly frustrating, trunk seems to be aware of the 1.15.1 version:

$ pod trunk info TensorIOTensorFlow

TensorIOTensorFlow
    - Versions:
      - 0.1.0 (2019-04-09 19:29:30 UTC)
      - 0.1.1 (2019-05-09 17:43:06 UTC)
      - 0.3.0 (2019-12-18 18:44:18 UTC)
      - 0.3.1 (2020-01-23 23:36:35 UTC)
      - 1.13.4 (2020-05-01 21:16:32 UTC)
      - 1.13.5 (2020-05-08 21:13:09 UTC)
      - 1.13.6 (2020-05-26 20:05:09 UTC)
      - 1.15 (2020-05-27 17:06:58 UTC)
      - 1.15.1 (2020-05-27 23:44:11 UTC)

But pod search does not see it:

$ pod search TensorIOTensorFlow

-> TensorIOTensorFlow (1.15)
   The TensorFlow (unofficial) build used by TensorIO for iOS.
   pod 'TensorIOTensorFlow', '~> 1.15'
   - Homepage: https://github.com/doc-ai/tensorio-tensorflow-ios
   - Source:   https://storage.googleapis.com/tensorio-build/r1.15/TensorIO-TensorFlow-1.15_0.tar.gz
   - Versions: 1.15, 1.13.6, 1.13.5, 1.13.4, 0.3.1, 0.3.0, 0.1.1, 0.1.0 [trunk repo]

To be sure, this is after I deleted the 1.15 and 1.15.1 versions with pod delete and tried pushing them to trunk again. Before that, pod search did show the 1.15.1 version but I still couldn't get a project to upgrade to it.

Finally, I have tried uninstalling and re-installing cocoapods, and as I mentioned, I have also tried all of this on a new machine with a first time install of the latest version of cocoapods, and I still cannot get the 1.15.1 version of the pod.

Body:
Here is the issue I have opened:
https://github.com/CocoaPods/CocoaPods/issues/9827

And, I have followed @paul-beusterien answer. And here is a sample

source 'https://github.com/CocoaPods/Specs.git'
platform :ios, '10.0'

use_frameworks!

target 'YourTarget' do
  inherit! :search_paths
  # Pods for YourTarget
  pod 'YourPod', '10.3.0'
end

So I added

source 'https://github.com/CocoaPods/Specs.git'

on the top of my Podfile

Thanks :)


问:
How to express this concept without dynamic dispatch?
I am building a simulation (the coding equivalent of a model train set). It is a simulated economy with various economic agents interacting with each other. The main mode of interaction between economic agents is a transaction. At each "tic", each agent generates a list of zero or more proposed transactions (such as buying food). At each "toc" all the counter-parties process the proposed transactions that have been targeted at them in random order so that no biases are introduced. In these snippets a proposed transaction is represented as a u32.

My goal is to simulate as many of these economic agents as possible so performance is key. I am new to rust (or any kind of low level language for that matter) and my understanding from reading the rust book is if I want maximum performance then use "zero cost abstractions" and avoid dynamic dispatch.

So with that out the way I have come up with the following 3 approaches.

Option 1

```<code>trait EconomicAgent {
    fn proposed_transactions(&self) -> Vec<u32>;
}

struct Person {
    health:f64,
    energy:f64,
    nutrition:f64,
    money:f64,
    food:f64
}

impl EconomicAgent for Person {
    fn proposed_transactions(&self) -> Vec<u32> {
        vec![1, 2, 3]
    }
}

struct FoodStore {
    money:f64,
    food:f64
}

impl EconomicAgent for FoodStore {
    fn proposed_transactions(&self) -> Vec<u32> {
        vec![4, 5, 6]
    }
}
</code>```

A person and a food store are different types that implement the EconomicAgent trait. I can then iterate over a vector of trait objects to get a list of proposed transactions. Each call is dynamically dispatched, I believe.

Option 2

```<code>enum EconomicAgent2 {
    Person(Person),
    FoodStore(FoodStore)
}

impl EconomicAgent2 {
    fn proposed_transactions(&self) -> Vec<u32> {
        match self{
            EconomicAgent2::Person(person) => person.proposed_transactions(),
            EconomicAgent2::FoodStore(food_store) => food_store.proposed_transactions()
        }
    }
}
</code>```

Here, an EconomicAgent is not a trait, but rather an enum and, well you can see how it works.

Option 3

```<code>const HEALTH_INDEX : u8 = 0;
const ENERGY_INDEX : u8 = 1;
const NUTRITION_INDEX : u8 = 2;
const MONEY_INDEX : u8 = 3;
const FOOD_INDEX : u8 = 4;

enum EconomicAgentTag {
    Person,
    FoodStore
}
struct EconomicAgent3 {
    tag: EconomicAgentTag,
    resources:[f64; 5],
    proposed_transactions: Box<fn(&EconomicAgent3) -> Vec<u32>>
}

fn new_person() -> EconomicAgent3 {
    EconomicAgent3 {
        tag: EconomicAgentTag::Person,
        resources: [0.0,0.0,0.0,0.0,0.0],
        proposed_transactions: Box::new(|_| vec![1, 2, 3])
    }
}

fn new_food_Store() -> EconomicAgent3 {
    EconomicAgent3 {
        tag: EconomicAgentTag::FoodStore,
        resources: [0.0,0.0,0.0,0.0,0.0],
        proposed_transactions: Box::new(|_| vec![4, 5, 6])
    }
}
</code>```

Here an economic agent is more abstract representation.

Now imagine that there a many different types of economic agents (banks, mines, farms, clothing stores etc). They all interact by proposing and accepting transactions. Option 1 seems to suffer from dynamic dispatch. Option 2 seems to be my own version of dynamic dispatch via a match expression so is probably no better, right? Option 3 seems like it should be the most performant but does not really allow much cognitive ease on the part of the programmer.

So finally the questions:

Clearly dynamic dispatch is involved in option 1. What about options 2 and 3?

Which is expected to be most performant? Note I am not really in a position to do testing as the full idea (only on paper right now) is obviously more complex than these snippets and the choice now will affect the entire structure for the whole project.

What would be an idiomatic choice here?

答:
What is Dynamic Dispatch?

Dynamic Dispatch is usually reserved to indirect function calls, ie function calls which occurs via a function pointer.

In your case, both Option 1 and Option 3 are cases of dynamic dispatch:

Traits use a virtual table, which is a table of function pointers.

fn(...) -> ... is a function pointer.

What is the performance penalty of Dynamic Dispatch?

At run-time, there is little to no difference between a regular function call and a so-called virtual call:

Indirect function calls can be predicted, there's a special predictor for them in your CPU.

The cost of the function call is mostly saving/restoring registers, which happen in both cases.

The performance penalty is more insidious, it happens at compile-time.

The mother of optimizations is inlining, which essentially copy/paste the body of the function being called right at the call-site. Once a function is inlined, many other optimization passes can go to town on the (combined) code. This is especially lucrative on very small functions (a getter), but can also be quite beneficial on larger functions.

An indirect function call, however, is opaque. There are many candidate functions, and thus the optimizer cannot perform inlining... nipping many potential optimizations in the bud. Devirtualization is sometimes available -- the compiler deducing which function(s) can be called -- but should not be relied on.

Which Option to choose?

Among those presented: Option 2!

The main advantage of Option 2 is that there is no indirect function calls. In both branches of your match, the compiler has a known type for the receiver of the method and can therefore inline the method if suitable, enabling all the optimizations.

Is there better?

With an open-design, an Array of Structs is a better way to structure the system, mostly avoiding branch misprediction:

```<code>EconomicAgents {
    Person(Vec<Person>),
    FoodStore(Vec<FoodStore>),
}
</code>```

This is the core design of the ECS solution proposed by @Acorn.

Note: as noted by @Acorn in comments, Array of Structs also is close to optimal cache wise -- no indirection, very little padding between elements.

Going with a full ECS is a trickier proposition. Unless you have dynamic entities -- Person/FoodStores are added/removed during the runs -- I would not bother. ECS are helpful for dynamism, but have to choose a trade-off between various characteristics: do you want faster add/remove, or faster iteration? Unless you need all their features, they will likely add their own overhead due to trade-offs that do not match your needs.

Body:
I am building a simulation (the coding equivalent of a model train set). It is a simulated economy with various economic agents interacting with each other. The main mode of interaction between economic agents is a transaction. At each "tic", each agent generates a list of zero or more proposed transactions (such as buying food). At each "toc" all the counter-parties process the proposed transactions that have been targeted at them in random order so that no biases are introduced. In these snippets a proposed transaction is represented as a u32.

My goal is to simulate as many of these economic agents as possible so performance is key. I am new to rust (or any kind of low level language for that matter) and my understanding from reading the rust book is if I want maximum performance then use "zero cost abstractions" and avoid dynamic dispatch.

So with that out the way I have come up with the following 3 approaches.

Option 1

trait EconomicAgent {
    fn proposed_transactions(&self) -> Vec<u32>;
}

struct Person {
    health:f64,
    energy:f64,
    nutrition:f64,
    money:f64,
    food:f64
}

impl EconomicAgent for Person {
    fn proposed_transactions(&self) -> Vec<u32> {
        vec![1, 2, 3]
    }
}

struct FoodStore {
    money:f64,
    food:f64
}

impl EconomicAgent for FoodStore {
    fn proposed_transactions(&self) -> Vec<u32> {
        vec![4, 5, 6]
    }
}

A person and a food store are different types that implement the EconomicAgent trait. I can then iterate over a vector of trait objects to get a list of proposed transactions. Each call is dynamically dispatched, I believe.

Option 2

enum EconomicAgent2 {
    Person(Person),
    FoodStore(FoodStore)
}

impl EconomicAgent2 {
    fn proposed_transactions(&self) -> Vec<u32> {
        match self{
            EconomicAgent2::Person(person) => person.proposed_transactions(),
            EconomicAgent2::FoodStore(food_store) => food_store.proposed_transactions()
        }
    }
}

Here, an EconomicAgent is not a trait, but rather an enum and, well you can see how it works.

Option 3

const HEALTH_INDEX : u8 = 0;
const ENERGY_INDEX : u8 = 1;
const NUTRITION_INDEX : u8 = 2;
const MONEY_INDEX : u8 = 3;
const FOOD_INDEX : u8 = 4;

enum EconomicAgentTag {
    Person,
    FoodStore
}
struct EconomicAgent3 {
    tag: EconomicAgentTag,
    resources:[f64; 5],
    proposed_transactions: Box<fn(&EconomicAgent3) -> Vec<u32>>
}

fn new_person() -> EconomicAgent3 {
    EconomicAgent3 {
        tag: EconomicAgentTag::Person,
        resources: [0.0,0.0,0.0,0.0,0.0],
        proposed_transactions: Box::new(|_| vec![1, 2, 3])
    }
}

fn new_food_Store() -> EconomicAgent3 {
    EconomicAgent3 {
        tag: EconomicAgentTag::FoodStore,
        resources: [0.0,0.0,0.0,0.0,0.0],
        proposed_transactions: Box::new(|_| vec![4, 5, 6])
    }
}

Here an economic agent is more abstract representation.

Now imagine that there a many different types of economic agents (banks, mines, farms, clothing stores etc). They all interact by proposing and accepting transactions. Option 1 seems to suffer from dynamic dispatch. Option 2 seems to be my own version of dynamic dispatch via a match expression so is probably no better, right? Option 3 seems like it should be the most performant but does not really allow much cognitive ease on the part of the programmer.

So finally the questions:

Clearly dynamic dispatch is involved in option 1. What about options 2 and 3?

Which is expected to be most performant? Note I am not really in a position to do testing as the full idea (only on paper right now) is obviously more complex than these snippets and the choice now will affect the entire structure for the whole project.

What would be an idiomatic choice here?

Body:
What is Dynamic Dispatch?

Dynamic Dispatch is usually reserved to indirect function calls, ie function calls which occurs via a function pointer.

In your case, both Option 1 and Option 3 are cases of dynamic dispatch:

Traits use a virtual table, which is a table of function pointers.

fn(...) -> ... is a function pointer.

What is the performance penalty of Dynamic Dispatch?

At run-time, there is little to no difference between a regular function call and a so-called virtual call:

Indirect function calls can be predicted, there's a special predictor for them in your CPU.

The cost of the function call is mostly saving/restoring registers, which happen in both cases.

The performance penalty is more insidious, it happens at compile-time.

The mother of optimizations is inlining, which essentially copy/paste the body of the function being called right at the call-site. Once a function is inlined, many other optimization passes can go to town on the (combined) code. This is especially lucrative on very small functions (a getter), but can also be quite beneficial on larger functions.

An indirect function call, however, is opaque. There are many candidate functions, and thus the optimizer cannot perform inlining... nipping many potential optimizations in the bud. Devirtualization is sometimes available -- the compiler deducing which function(s) can be called -- but should not be relied on.

Which Option to choose?

Among those presented: Option 2!

The main advantage of Option 2 is that there is no indirect function calls. In both branches of your match, the compiler has a known type for the receiver of the method and can therefore inline the method if suitable, enabling all the optimizations.

Is there better?

With an open-design, an Array of Structs is a better way to structure the system, mostly avoiding branch misprediction:

EconomicAgents {
    Person(Vec<Person>),
    FoodStore(Vec<FoodStore>),
}

This is the core design of the ECS solution proposed by @Acorn.

Note: as noted by @Acorn in comments, Array of Structs also is close to optimal cache wise -- no indirection, very little padding between elements.

Going with a full ECS is a trickier proposition. Unless you have dynamic entities -- Person/FoodStores are added/removed during the runs -- I would not bother. ECS are helpful for dynamism, but have to choose a trade-off between various characteristics: do you want faster add/remove, or faster iteration? Unless you need all their features, they will likely add their own overhead due to trade-offs that do not match your needs.


问:
php replace specific words in a description with uppercase words and loop through it
Need to replace ALL present words within a string that tested against an array with the uppercase search words.

I am stuck with the right loop now.

My code:

```<code>   $array = [
        'weizen',
        'dinkel',
        'gerste',
        'khorasan',
        'eier',
        'haselnuss',
        'walnuss'];

   $description = "Weizenbrötchen auf Hartweizenbasis und Eiern mit Walnusscreme und Haselnusspaste sind leckerer mit Dinkelflocken."

foreach($array as $item) {
if (strpos($description, $item) !== false) {
   // $description;
 echo
 str_replace($item, mb_strtoupper($item), $description);
}
}
</code>```

Actual outcome:
"WEIZENbrötchen auf Hartweizenbasis und Eiern mit Walnusscreme und Haselnusspaste sind leckerer mit Dinkelflocken.Weizenbrötchen auf Hartweizenbasis und EIERn mit Walnusscreme und Haselnusspaste sind leckerer mit Dinkelflocken.Weizenbrötchen auf Hartweizenbasis und Eiern mit WALNUSScreme und Haselnusspaste sind leckerer mit Dinkelflocken.Weizenbrötchen auf Hartweizenbasis und Eiern mit Walnusscreme und HASELNUSSpaste sind leckerer mit Dinkelflocken.Weizenbrötchen auf Hartweizenbasis und Eiern mit Walnusscreme und Haselnusspaste sind leckerer mit DINKELflocken.

Desired outcome:

WEIZENbrötchen auf HartWEIZENbasis und EIERn mit WALNUSScreme und HASELNUSSpaste sind leckerer mit DINKELflocken.

答:
One way to do this is to make a regex out of the strings to be replaced, and then use preg_replace_callback to convert any matched values to upper case:

```<code>$array = [
        'weizen',
        'dinkel',
        'gerste',
        'khorasan',
        'eier',
        'haselnuss',
        'walnuss'];

$description = "Weizenbrötchen auf Hartweizenbasis und Eiern mit Walnusscreme und Haselnusspaste sind leckerer mit Dinkelflocken.";

$pattern = '/' . implode('|', $array) . '/iu';
echo preg_replace_callback($pattern, function ($m) { return mb_strtoupper($m[0]); }, $description);
</code>```

Output:

```<code>WEIZENbrötchen auf HartWEIZENbasis und EIERn mit WALNUSScreme und HASELNUSSpaste sind leckerer mit DINKELflocken.
</code>```

Demo on 3v4l.org

Body:
Need to replace ALL present words within a string that tested against an array with the uppercase search words.

I am stuck with the right loop now.

My code:

   $array = [
        'weizen',
        'dinkel',
        'gerste',
        'khorasan',
        'eier',
        'haselnuss',
        'walnuss'];

   $description = "Weizenbrötchen auf Hartweizenbasis und Eiern mit Walnusscreme und Haselnusspaste sind leckerer mit Dinkelflocken."

foreach($array as $item) {
if (strpos($description, $item) !== false) {
   // $description;
 echo
 str_replace($item, mb_strtoupper($item), $description);
}
}

Actual outcome:
"WEIZENbrötchen auf Hartweizenbasis und Eiern mit Walnusscreme und Haselnusspaste sind leckerer mit Dinkelflocken.Weizenbrötchen auf Hartweizenbasis und EIERn mit Walnusscreme und Haselnusspaste sind leckerer mit Dinkelflocken.Weizenbrötchen auf Hartweizenbasis und Eiern mit WALNUSScreme und Haselnusspaste sind leckerer mit Dinkelflocken.Weizenbrötchen auf Hartweizenbasis und Eiern mit Walnusscreme und HASELNUSSpaste sind leckerer mit Dinkelflocken.Weizenbrötchen auf Hartweizenbasis und Eiern mit Walnusscreme und Haselnusspaste sind leckerer mit DINKELflocken.

Desired outcome:

WEIZENbrötchen auf HartWEIZENbasis und EIERn mit WALNUSScreme und HASELNUSSpaste sind leckerer mit DINKELflocken.

Body:
One way to do this is to make a regex out of the strings to be replaced, and then use preg_replace_callback to convert any matched values to upper case:

$array = [
        'weizen',
        'dinkel',
        'gerste',
        'khorasan',
        'eier',
        'haselnuss',
        'walnuss'];

$description = "Weizenbrötchen auf Hartweizenbasis und Eiern mit Walnusscreme und Haselnusspaste sind leckerer mit Dinkelflocken.";

$pattern = '/' . implode('|', $array) . '/iu';
echo preg_replace_callback($pattern, function ($m) { return mb_strtoupper($m[0]); }, $description);

Output:

WEIZENbrötchen auf HartWEIZENbasis und EIERn mit WALNUSScreme und HASELNUSSpaste sind leckerer mit DINKELflocken.

Demo on 3v4l.org


问:
SQL query to find duplicate records and discount from COUNT
I am using below SQL query to count the records.
I would like to discount the duplicate testID after I got the count.

example:

```<code>select COUNT(*)  from PRODUCT where CLASS_TYPE = 'test'
</code>```

above query give 5 records

```<code>testID
1234-00-01
1234-00-01
1234-00-02
1234-00-02
1111222233
</code>```

I want to get the count = 5 - duplicate 2 = 3

Is there SQL query I can use to find out the duplicate records, then calculate the final count?

答:
Please use below query,</p>

```<code>select COUNT(distinct testID)  from PRODUCT where CLASS_TYPE = 'test'
</code>```

Body:
I am using below SQL query to count the records.
I would like to discount the duplicate testID after I got the count.

example:

select COUNT(*)  from PRODUCT where CLASS_TYPE = 'test'

above query give 5 records

testID
1234-00-01
1234-00-01
1234-00-02
1234-00-02
1111222233

I want to get the count = 5 - duplicate 2 = 3

Is there SQL query I can use to find out the duplicate records, then calculate the final count?

Body:
Please use below query,

select COUNT(distinct testID)  from PRODUCT where CLASS_TYPE = 'test'


问:
Proper use of Task.Run & Task.WaitAll for EF calls in ASP.NET
SHORT VERSION:

Can anyone confirm, does Task.Run/Task.WaitAll on non-async sub-functions with EF calls gain anything for ASP.NET apps?

LONG VERSION:

We have an Asp.Net WebApi service with a method which makes multiple DB calls (all using EntityFramework) to collect a set of data that all get returned into a single response object.  We broke this up into a series of sub-functions, and wanted these to run in parallel (NOTE: None of these use any async calls).

To achieve some form of parallel coding, we invoke each function with Task.Run, followed by Task.WaitAll:

```<code>public ResponseObject PopulateResponseObject(int id)
{
    var response = new ResponseObject();

    Task<DataSetA> dataSetATask = Task.Run(() => getDataSetA(id));
    Task<DataSetB> dataSetBTask = Task.Run(() => getDataSetB(id));
    Task.WaitAll(dataSetATask, dataSetBTask);

    response.SetA = dataSetATask.Result;
    response.SetB = dataSetBTask.Result;

    return response;
}
</code>```

From what I've been reading, Task.Run may not gain much in an Asp.Net application, and what we have here may just result in unnecessary thread-pool overhead.

Are we wasting time writing our code this way?

UPDATE:
We are familiar with the fact that EF has async versions, but there is a significant amount of code that would need to be modified.  We'd like to leave these functions as-is.

答:
After doing some performance testing against Sync, Async (using the method above), and Parallel.Invoke, I found some interesting results.  The parallel version was similar to:

```<code>public ResponseObject PopulateResponseObject(int id)
{
    var response = new ResponseObject();

    Parallel.Invoke(
        () => response.SetA = getDataSetA(id),
        () => response.SetB = getDataSetB(id),
        );

    return response;
}
</code>```

And the Async appears above, in my original post.

It turns out that on average, compared to basic Sync calls, Async calls (Task.Run/Task.WaitAll) ran in 57% of the time, and Parallel.Invoke ran in 65% of the time.

So there was an advantage of using Task.Run/WaitAll in an ASP.NET app.

UPDATE: BTW this testing was done by pushing our code to our test environment where our QA folks were performing tests.  It certainly  doesn't mimic our prod environment, but does show it's slightly better.

Body:
SHORT VERSION:

Can anyone confirm, does Task.Run/Task.WaitAll on non-async sub-functions with EF calls gain anything for ASP.NET apps?

LONG VERSION:

We have an Asp.Net WebApi service with a method which makes multiple DB calls (all using EntityFramework) to collect a set of data that all get returned into a single response object.  We broke this up into a series of sub-functions, and wanted these to run in parallel (NOTE: None of these use any async calls).

To achieve some form of parallel coding, we invoke each function with Task.Run, followed by Task.WaitAll:

public ResponseObject PopulateResponseObject(int id)
{
    var response = new ResponseObject();

    Task<DataSetA> dataSetATask = Task.Run(() => getDataSetA(id));
    Task<DataSetB> dataSetBTask = Task.Run(() => getDataSetB(id));
    Task.WaitAll(dataSetATask, dataSetBTask);

    response.SetA = dataSetATask.Result;
    response.SetB = dataSetBTask.Result;

    return response;
}

From what I've been reading, Task.Run may not gain much in an Asp.Net application, and what we have here may just result in unnecessary thread-pool overhead.

Are we wasting time writing our code this way?

UPDATE:
We are familiar with the fact that EF has async versions, but there is a significant amount of code that would need to be modified.  We'd like to leave these functions as-is.

Body:
After doing some performance testing against Sync, Async (using the method above), and Parallel.Invoke, I found some interesting results.  The parallel version was similar to:

public ResponseObject PopulateResponseObject(int id)
{
    var response = new ResponseObject();

    Parallel.Invoke(
        () => response.SetA = getDataSetA(id),
        () => response.SetB = getDataSetB(id),
        );

    return response;
}

And the Async appears above, in my original post.

It turns out that on average, compared to basic Sync calls, Async calls (Task.Run/Task.WaitAll) ran in 57% of the time, and Parallel.Invoke ran in 65% of the time.

So there was an advantage of using Task.Run/WaitAll in an ASP.NET app.

UPDATE: BTW this testing was done by pushing our code to our test environment where our QA folks were performing tests.  It certainly  doesn't mimic our prod environment, but does show it's slightly better.


问:
How to find convergence in vector field?
I am trying to find areas or points where there is convergence in a vector field.

I have used the code below to generate the following plot:

```<code>import matplotlib.pyplot as plt
import numpy as np

def generate_fake_data():
    return -(np.sin(X) * np.cos(Y) + np.cos(X)), -(-np.cos(X) * np.sin(Y) + np.sin(Y))

x = np.arange(0, 2 * np.pi + 2 * np.pi / 20, 2 * np.pi / 20)
y = np.arange(0, 2 * np.pi + 2 * np.pi / 20, 2 * np.pi / 20)

X, Y = np.meshgrid(x, y)

u, v = generate_fake_data()

fig, ax = plt.subplots(figsize=(7, 7))

# quiveropts = dict(headlength=0, headaxislength=0, pivot='middle', units='xy')
# ax.quiver(X, Y, u, v, **quiveropts)
ax.quiver(X, Y, u, v)

ax.xaxis.set_ticks([])
ax.yaxis.set_ticks([])
ax.axis([0, 2 * np.pi, 0, 2 * np.pi])
ax.set_aspect('equal')
ax.axis("off")
plt.gca().set_axis_off()
plt.subplots_adjust(top=1, bottom=0, right=1, left=0,
                    hspace=0, wspace=0)
plt.margins(0, 0)
plt.gca().xaxis.set_major_locator(plt.NullLocator())
plt.gca().yaxis.set_major_locator(plt.NullLocator())
plt.savefig("mock_data.png", bbox_inches='tight', pad_inches=0)

</code>```

Ideally, what I am after is to find where there is convergence in this vector field in the top right and bottom right of the image.

I hope that the curl values could be used to achieve this, but any method is good to use.

Also, this is just a proof of concept, generate_fake_data will be replaced with a function reading in data from elsewhere that's changeable.

答:
For converging points the divergence of the vector field is < 0.

```<code>from functools import reduce
conv = reduce(np.add,np.gradient(u)) + reduce(np.add,np.gradient(v))
</code>```

(see Compute divergence of vector field using python)

We only need negativ divergence:

```<code>conv[conv>=0] = np.nan
</code>```

plt.imshow(conv): the darker the more converging is the field:

Finding the absolute minimum (in the top right) is easy:

```<code>absmin = np.unravel_index(np.nanargmin(conv), conv.shape)
print(absmin, conv[absmin])
#(0, 15) -0.6669774724547413
</code>```

Finding relative minima is harder, it should be possible with argrelmin, but frankly speaking I couldn't get it work correctly to return the second local minimum at (19,15).
Using this answer I get

```<code>lm = detect_local_minima(conv)
list(zip(lm))
[(0, 15), (1, 0), (19, 15), (20, 0)]
</code>```

which also includes both minima in the left corners (which is mathematically correct, but not desired in our case, so maybe we could just exclude corners).

Body:
I am trying to find areas or points where there is convergence in a vector field.

I have used the code below to generate the following plot:

import matplotlib.pyplot as plt
import numpy as np

def generate_fake_data():
    return -(np.sin(X) * np.cos(Y) + np.cos(X)), -(-np.cos(X) * np.sin(Y) + np.sin(Y))

x = np.arange(0, 2 * np.pi + 2 * np.pi / 20, 2 * np.pi / 20)
y = np.arange(0, 2 * np.pi + 2 * np.pi / 20, 2 * np.pi / 20)

X, Y = np.meshgrid(x, y)

u, v = generate_fake_data()

fig, ax = plt.subplots(figsize=(7, 7))

# quiveropts = dict(headlength=0, headaxislength=0, pivot='middle', units='xy')
# ax.quiver(X, Y, u, v, **quiveropts)
ax.quiver(X, Y, u, v)

ax.xaxis.set_ticks([])
ax.yaxis.set_ticks([])
ax.axis([0, 2 * np.pi, 0, 2 * np.pi])
ax.set_aspect('equal')
ax.axis("off")
plt.gca().set_axis_off()
plt.subplots_adjust(top=1, bottom=0, right=1, left=0,
                    hspace=0, wspace=0)
plt.margins(0, 0)
plt.gca().xaxis.set_major_locator(plt.NullLocator())
plt.gca().yaxis.set_major_locator(plt.NullLocator())
plt.savefig("mock_data.png", bbox_inches='tight', pad_inches=0)

Ideally, what I am after is to find where there is convergence in this vector field in the top right and bottom right of the image.

I hope that the curl values could be used to achieve this, but any method is good to use.

Also, this is just a proof of concept, generate_fake_data will be replaced with a function reading in data from elsewhere that's changeable.

Body:
For converging points the divergence of the vector field is < 0.

from functools import reduce
conv = reduce(np.add,np.gradient(u)) + reduce(np.add,np.gradient(v))

(see Compute divergence of vector field using python)

We only need negativ divergence:

conv[conv>=0] = np.nan

plt.imshow(conv): the darker the more converging is the field:

Finding the absolute minimum (in the top right) is easy:

absmin = np.unravel_index(np.nanargmin(conv), conv.shape)
print(absmin, conv[absmin])
#(0, 15) -0.6669774724547413

Finding relative minima is harder, it should be possible with argrelmin, but frankly speaking I couldn't get it work correctly to return the second local minimum at (19,15).
Using this answer I get

lm = detect_local_minima(conv)
list(zip(lm))
[(0, 15), (1, 0), (19, 15), (20, 0)]

which also includes both minima in the left corners (which is mathematically correct, but not desired in our case, so maybe we could just exclude corners).


问:
Regex - Match only two digitsafter substring
I'm trying to replace all elements like \u00XY in a string that can contain multiple entries like that.

```<code>'"\u00bfIdade del titular?"'
</code>```

This can be a short string or a string containing json objects inside... (I know... but... old code)
I tried normalize after the string but it didn't work, so I got instructed to replace all of those elements in unicode with a '?' char.

Any ideas on a simple way for this purpose? I'm not being able to find the right regex for this.

Body:
I'm trying to replace all elements like \u00XY in a string that can contain multiple entries like that.

'"\u00bfIdade del titular?"'

This can be a short string or a string containing json objects inside... (I know... but... old code)
I tried normalize after the string but it didn't work, so I got instructed to replace all of those elements in unicode with a '?' char.

Any ideas on a simple way for this purpose? I'm not being able to find the right regex for this.

Body:
[0-9]{1,2}[\w]{2}?
Try this or modify it.