问:
How to create kubernetes secret as json object and load the same in kubernetes environment as json
I need to pass a JWK as kubernetes environment variable to my app.

I created a file to store my key like so:

```<code>cat deploy/keys/access-signature-public-jwk
{
  algorithm = "RS256"
  jwk = {"kty":"RSA","e":"AQAB","n":"ghhDZxuUo6TaSvAlD23mLP6n_T9pQuJsFY4JWdBYTjtcp_8Q3QeR477jou4cScPGczWw2JMGnx-Ao_b7ewagSl7VHpECBFHgcnlAgs5j6jfnd3M9ADKD2Yc756iXlIMT9xKDblIcXQQYlXalqxGvnLRLv1KAgVVVpVWzQd6Iz8WdTnexVrh7L9N87QQbOWcAVWGHCWCLCBsVE7JbC-XDt9h9P1g1sMqMV-qp7HjSXUKWuF2NwOnL2VeFSED7gdefs2Za1UYqhfwxdGl7aaPDXhjib0cfg4NvbcXMzxDEVkeJqhdDfD82wHOs4qFvnFMVxq9n6VVExSxsJq8gBJ7Z2AmfoXpmZC1L1ZwULB2KKpFXDCzgBELPLrfyIf8mNnk2nuuLT-aaMsqy2uB-ea3du4lyWo9MLk6x-L5g-n1oADKFKBY9aP2QQwruCG92XSd7jA9yLtbgr9OGVCYezxIxFp4vW6KcmPwJQjozWtwkZjeo4hv-zhRac73WDox2hDkif7WPTuEvC21fRy3GvyPIUPKPJA8pJjb2TXT7DXknR97CTnOWicuh3HMoRlVIwUzM5SVLGSXex0VjHZKgLYwQYukg5O2rab_4NxpD6LqLHx1bbPssC7BedCIfWX1Vcae40tlfvJAM09MiwQPZjWRahW_fK_9X5F5_rtUhCznm32M"}
}
</code>```

Which is then used to create a kubernetes secret like so:

```<code>kubectl create secret generic intimations-signature-public-secret --from-file=./deploy/keys/access-signature-public-jwk
</code>```

Which is then retrived in the kubernetes environment variable as:

```<code>- name: ACCESS_SIGNATURE_PUBLIC_JWK
  valueFrom:
    secretKeyRef:
      name: intimations-signature-public-secret
      key: access-signature-public-jwk
</code>```

And passed to the application.conf of the application likeso:

```<code>pac4j.lagom.jwt.authenticator {
  signatures = [
    ${ACCESS_SIGNATURE_PUBLIC_JWK}
  ]
}
</code>```

The pac4j library expects the config pac4j.lagom.jwt.authenticator as a json object. But get the following exception when I run this app:

```<code>com.typesafe.config.ConfigException$WrongType: env variables: signatures has type list of STRING rather than list of OBJECT
    at com.typesafe.config.impl.SimpleConfig.getHomogeneousWrappedList(SimpleConfig.java:452)
    at com.typesafe.config.impl.SimpleConfig.getObjectList(SimpleConfig.java:460)
    at com.typesafe.config.impl.SimpleConfig.getConfigList(SimpleConfig.java:465)
    at org.pac4j.lagom.jwt.JwtAuthenticatorHelper.parse(JwtAuthenticatorHelper.java:84)
    at com.codingkapoor.holiday.impl.core.HolidayApplication.jwtClient$lzycompute(HolidayApplication.scala
</code>```

POD Description

```<code>Name:         holiday-deployment-55b86f955d-9klk2
Namespace:    default
Priority:     0
Node:         minikube/192.168.99.103
Start Time:   Thu, 28 May 2020 12:42:50 +0530
Labels:       app=holiday
              pod-template-hash=55b86f955d
Annotations:  <none>
Status:       Running
IP:           172.17.0.5
IPs:
  IP:           172.17.0.5
Controlled By:  ReplicaSet/holiday-deployment-55b86f955d
Containers:
  holiday:
    Container ID:   docker://18443cfedc7fd39440f5fa6f038f36c58cec1660a2974e6432500e8c7d51f5e6
    Image:          codingkapoor/holiday-impl:latest
    Image ID:       docker://sha256:6e0ddcf41e0257755b7e865424671970091d555c4bad88b5d896708ded139eb7
    Port:           8558/TCP
    Host Port:      0/TCP
    State:          Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Thu, 28 May 2020 22:49:24 +0530
      Finished:     Thu, 28 May 2020 22:49:29 +0530
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Thu, 28 May 2020 22:44:15 +0530
      Finished:     Thu, 28 May 2020 22:44:21 +0530
    Ready:          False
    Restart Count:  55
    Liveness:       http-get http://:management/alive delay=20s timeout=1s period=10s #success=1 #failure=10
    Readiness:      http-get http://:management/ready delay=20s timeout=1s period=10s #success=1 #failure=10
    Environment:
      JAVA_OPTS:                     -Xms256m -Xmx256m -Dconfig.resource=prod-application.conf
      APPLICATION_SECRET:            <set to the key 'secret' in secret 'intimations-application-secret'>  Optional: false
      MYSQL_URL:                     jdbc:mysql://mysql/intimations_holiday_schema
      MYSQL_USERNAME:                <set to the key 'username' in secret 'intimations-mysql-secret'>                                 Optional: false
      MYSQL_PASSWORD:                <set to the key 'password' in secret 'intimations-mysql-secret'>                                 Optional: false
      ACCESS_SIGNATURE_PUBLIC_JWK:   <set to the key 'access-signature-public-jwk' in secret 'intimations-signature-public-secret'>   Optional: false
      REFRESH_SIGNATURE_PUBLIC_JWK:  <set to the key 'refresh-signature-public-jwk' in secret 'intimations-signature-public-secret'>  Optional: false
      REQUIRED_CONTACT_POINT_NR:     1
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kqmmv (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-kqmmv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-kqmmv
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason   Age                    From               Message
  ----     ------   ----                   ----               -------
  Normal   Pulled   5m21s (x23 over 100m)  kubelet, minikube  Container image "codingkapoor/holiday-impl:latest" already present on machine
  Warning  BackOff  27s (x466 over 100m)   kubelet, minikube  Back-off restarting failed container
</code>```

I was wondering if there is any way to pass the environment variable as a json object instead of string. Please suggest. TIA.

答:
Although not a direct answer but an alternate solution to this problem.

As @hariK pointed out environment variables are always strings and in order to consume them as json we would need to convert the env var read as string into json.

However, in my case, this was not a viable solution because I was using a lib that was expecting a Config object and not a json object directly which would have meant a lot of work. Converting string -> json -> Config. Plus this approach is inconsistent with how Config object was being built in the developement scenarios i.e., json -> Config. See here.

The framework I am using to build this app is based on Play Framework which allows to modularize application configs in separate files and then club the required pieces together in a desired config file, as shown below. You can read it more in detail here.

application.conf

```<code>include "/opt/conf/app1.conf"
include "/opt/conf/app2.conf"
</code>```

This allowed me to make use of Using Secrets as files from a Pod
 feature from kubernetes.

Basically, I created a small config file that contains a part of my main application configuration file, as shown below:

```<code>cat deploy/keys/signature-public-jwk
pac4j.lagom.jwt.authenticator {
  signatures = [
    {
      algorithm = "RS256"
      jwk = {"kty":"RSA","e":"AQAB","n":"ghhDZxuUo6TaSvAlD23mLP6n_T9pQuJsFY4JWdBYTjtcp_8Q3QeR477jou4cScPGczWw2JMGnx-Ao_b7ewagSl7VHpECBFHgcnlAgs5j6jfnd3M9ADKD2Yc756iXlIMT9xKDblIcXQQYlXalqxGvnLRLv1KAgVVVpVWzQd6Iz8WdTnexVrh7L9N87QQbOWcAVWGHCWCLCBsVE7JbC-XDt9h9P1g1sMqMV-qp7HjSXUKWuF2NwOnL2VeFSED7gdefs2Za1UYqhfwxdGl7aaPDXhjib0cfg4NvbcXMzxDEVkeJqhdDfD82wHOs4qFvnFMVxq9n6VVExSxsJq8gBJ7Z2AmfoXpmZC1L1ZwULB2KKpFXDCzgBELPLrfyIf8mNnk2nuuLT-aaMsqy2uB-ea3du4lyWo9MLk6x-L5g-n1oADKFKBY9aP2QQwruCG92XSd7jA9yLtbgr9OGVCYezxIxFp4vW6KcmPwJQjozWtwkZjeo4hv-zhRac73WDox2hDkif7WPTuEvC21fRy3GvyPIUPKPJA8pJjb2TXT7DXknR97CTnOWicuh3HMoRlVIwUzM5SVLGSXex0VjHZKgLYwQYukg5O2rab_4NxpD6LqLHx1bbPssC7BedCIfWX1Vcae40tlfvJAM09MiwQPZjWRahW_fK_9X5F5_rtUhCznm32M"}
    }
  ]
}
</code>```

Then created a kubernetes secret and mounted volumes in deployment to appear in the pod as file

```<code>kubectl create secret generic signature-public-secret --from-file=./deploy/secrets/signature-public-jwks.conf

// deployment yaml
    spec:
      containers:
        - name: employee
          image: "codingkapoor/employee-impl:latest"
          volumeMounts:
            - name: signature-public-secret-conf
              mountPath: /opt/conf/signature-public-jwks.conf
              subPath: signature-public-jwks.conf
              readOnly: true
      volumes:
        - name: signature-public-secret-conf
          secret:
            secretName: signature-public-secret
</code>```

Use this mounted file location in the application.conf to include the same

```<code>include file("/opt/conf/signature-public-jwks.conf")
</code>```

Notice that the mountPath and the file location in the application.conf are same.

Advantages of this approach:

The solution is consistent with both the development and test, production environments as we could return json instead of string to the lib, as explained above

Secrets shouldn't be passed as environment variables anyway! You can read more about it here.

Body:
I need to pass a JWK as kubernetes environment variable to my app.

I created a file to store my key like so:

cat deploy/keys/access-signature-public-jwk
{
  algorithm = "RS256"
  jwk = {"kty":"RSA","e":"AQAB","n":"ghhDZxuUo6TaSvAlD23mLP6n_T9pQuJsFY4JWdBYTjtcp_8Q3QeR477jou4cScPGczWw2JMGnx-Ao_b7ewagSl7VHpECBFHgcnlAgs5j6jfnd3M9ADKD2Yc756iXlIMT9xKDblIcXQQYlXalqxGvnLRLv1KAgVVVpVWzQd6Iz8WdTnexVrh7L9N87QQbOWcAVWGHCWCLCBsVE7JbC-XDt9h9P1g1sMqMV-qp7HjSXUKWuF2NwOnL2VeFSED7gdefs2Za1UYqhfwxdGl7aaPDXhjib0cfg4NvbcXMzxDEVkeJqhdDfD82wHOs4qFvnFMVxq9n6VVExSxsJq8gBJ7Z2AmfoXpmZC1L1ZwULB2KKpFXDCzgBELPLrfyIf8mNnk2nuuLT-aaMsqy2uB-ea3du4lyWo9MLk6x-L5g-n1oADKFKBY9aP2QQwruCG92XSd7jA9yLtbgr9OGVCYezxIxFp4vW6KcmPwJQjozWtwkZjeo4hv-zhRac73WDox2hDkif7WPTuEvC21fRy3GvyPIUPKPJA8pJjb2TXT7DXknR97CTnOWicuh3HMoRlVIwUzM5SVLGSXex0VjHZKgLYwQYukg5O2rab_4NxpD6LqLHx1bbPssC7BedCIfWX1Vcae40tlfvJAM09MiwQPZjWRahW_fK_9X5F5_rtUhCznm32M"}
}

Which is then used to create a kubernetes secret like so:

kubectl create secret generic intimations-signature-public-secret --from-file=./deploy/keys/access-signature-public-jwk

Which is then retrived in the kubernetes environment variable as:

- name: ACCESS_SIGNATURE_PUBLIC_JWK
  valueFrom:
    secretKeyRef:
      name: intimations-signature-public-secret
      key: access-signature-public-jwk

And passed to the application.conf of the application likeso:

pac4j.lagom.jwt.authenticator {
  signatures = [
    ${ACCESS_SIGNATURE_PUBLIC_JWK}
  ]
}

The pac4j library expects the config pac4j.lagom.jwt.authenticator as a json object. But get the following exception when I run this app:

com.typesafe.config.ConfigException$WrongType: env variables: signatures has type list of STRING rather than list of OBJECT
    at com.typesafe.config.impl.SimpleConfig.getHomogeneousWrappedList(SimpleConfig.java:452)
    at com.typesafe.config.impl.SimpleConfig.getObjectList(SimpleConfig.java:460)
    at com.typesafe.config.impl.SimpleConfig.getConfigList(SimpleConfig.java:465)
    at org.pac4j.lagom.jwt.JwtAuthenticatorHelper.parse(JwtAuthenticatorHelper.java:84)
    at com.codingkapoor.holiday.impl.core.HolidayApplication.jwtClient$lzycompute(HolidayApplication.scala

POD Description

Name:         holiday-deployment-55b86f955d-9klk2
Namespace:    default
Priority:     0
Node:         minikube/192.168.99.103
Start Time:   Thu, 28 May 2020 12:42:50 +0530
Labels:       app=holiday
              pod-template-hash=55b86f955d
Annotations:  <none>
Status:       Running
IP:           172.17.0.5
IPs:
  IP:           172.17.0.5
Controlled By:  ReplicaSet/holiday-deployment-55b86f955d
Containers:
  holiday:
    Container ID:   docker://18443cfedc7fd39440f5fa6f038f36c58cec1660a2974e6432500e8c7d51f5e6
    Image:          codingkapoor/holiday-impl:latest
    Image ID:       docker://sha256:6e0ddcf41e0257755b7e865424671970091d555c4bad88b5d896708ded139eb7
    Port:           8558/TCP
    Host Port:      0/TCP
    State:          Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Thu, 28 May 2020 22:49:24 +0530
      Finished:     Thu, 28 May 2020 22:49:29 +0530
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Thu, 28 May 2020 22:44:15 +0530
      Finished:     Thu, 28 May 2020 22:44:21 +0530
    Ready:          False
    Restart Count:  55
    Liveness:       http-get http://:management/alive delay=20s timeout=1s period=10s #success=1 #failure=10
    Readiness:      http-get http://:management/ready delay=20s timeout=1s period=10s #success=1 #failure=10
    Environment:
      JAVA_OPTS:                     -Xms256m -Xmx256m -Dconfig.resource=prod-application.conf
      APPLICATION_SECRET:            <set to the key 'secret' in secret 'intimations-application-secret'>  Optional: false
      MYSQL_URL:                     jdbc:mysql://mysql/intimations_holiday_schema
      MYSQL_USERNAME:                <set to the key 'username' in secret 'intimations-mysql-secret'>                                 Optional: false
      MYSQL_PASSWORD:                <set to the key 'password' in secret 'intimations-mysql-secret'>                                 Optional: false
      ACCESS_SIGNATURE_PUBLIC_JWK:   <set to the key 'access-signature-public-jwk' in secret 'intimations-signature-public-secret'>   Optional: false
      REFRESH_SIGNATURE_PUBLIC_JWK:  <set to the key 'refresh-signature-public-jwk' in secret 'intimations-signature-public-secret'>  Optional: false
      REQUIRED_CONTACT_POINT_NR:     1
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kqmmv (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-kqmmv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-kqmmv
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason   Age                    From               Message
  ----     ------   ----                   ----               -------
  Normal   Pulled   5m21s (x23 over 100m)  kubelet, minikube  Container image "codingkapoor/holiday-impl:latest" already present on machine
  Warning  BackOff  27s (x466 over 100m)   kubelet, minikube  Back-off restarting failed container

I was wondering if there is any way to pass the environment variable as a json object instead of string. Please suggest. TIA.

Body:
Although not a direct answer but an alternate solution to this problem.

As @hariK pointed out environment variables are always strings and in order to consume them as json we would need to convert the env var read as string into json.

However, in my case, this was not a viable solution because I was using a lib that was expecting a Config object and not a json object directly which would have meant a lot of work. Converting string -> json -> Config. Plus this approach is inconsistent with how Config object was being built in the developement scenarios i.e., json -> Config. See here.

The framework I am using to build this app is based on Play Framework which allows to modularize application configs in separate files and then club the required pieces together in a desired config file, as shown below. You can read it more in detail here.

application.conf

include "/opt/conf/app1.conf"
include "/opt/conf/app2.conf"

This allowed me to make use of Using Secrets as files from a Pod
 feature from kubernetes.

Basically, I created a small config file that contains a part of my main application configuration file, as shown below:

cat deploy/keys/signature-public-jwk
pac4j.lagom.jwt.authenticator {
  signatures = [
    {
      algorithm = "RS256"
      jwk = {"kty":"RSA","e":"AQAB","n":"ghhDZxuUo6TaSvAlD23mLP6n_T9pQuJsFY4JWdBYTjtcp_8Q3QeR477jou4cScPGczWw2JMGnx-Ao_b7ewagSl7VHpECBFHgcnlAgs5j6jfnd3M9ADKD2Yc756iXlIMT9xKDblIcXQQYlXalqxGvnLRLv1KAgVVVpVWzQd6Iz8WdTnexVrh7L9N87QQbOWcAVWGHCWCLCBsVE7JbC-XDt9h9P1g1sMqMV-qp7HjSXUKWuF2NwOnL2VeFSED7gdefs2Za1UYqhfwxdGl7aaPDXhjib0cfg4NvbcXMzxDEVkeJqhdDfD82wHOs4qFvnFMVxq9n6VVExSxsJq8gBJ7Z2AmfoXpmZC1L1ZwULB2KKpFXDCzgBELPLrfyIf8mNnk2nuuLT-aaMsqy2uB-ea3du4lyWo9MLk6x-L5g-n1oADKFKBY9aP2QQwruCG92XSd7jA9yLtbgr9OGVCYezxIxFp4vW6KcmPwJQjozWtwkZjeo4hv-zhRac73WDox2hDkif7WPTuEvC21fRy3GvyPIUPKPJA8pJjb2TXT7DXknR97CTnOWicuh3HMoRlVIwUzM5SVLGSXex0VjHZKgLYwQYukg5O2rab_4NxpD6LqLHx1bbPssC7BedCIfWX1Vcae40tlfvJAM09MiwQPZjWRahW_fK_9X5F5_rtUhCznm32M"}
    }
  ]
}

Then created a kubernetes secret and mounted volumes in deployment to appear in the pod as file

kubectl create secret generic signature-public-secret --from-file=./deploy/secrets/signature-public-jwks.conf

// deployment yaml
    spec:
      containers:
        - name: employee
          image: "codingkapoor/employee-impl:latest"
          volumeMounts:
            - name: signature-public-secret-conf
              mountPath: /opt/conf/signature-public-jwks.conf
              subPath: signature-public-jwks.conf
              readOnly: true
      volumes:
        - name: signature-public-secret-conf
          secret:
            secretName: signature-public-secret

Use this mounted file location in the application.conf to include the same

include file("/opt/conf/signature-public-jwks.conf")

Notice that the mountPath and the file location in the application.conf are same.

Advantages of this approach:

The solution is consistent with both the development and test, production environments as we could return json instead of string to the lib, as explained above

Secrets shouldn't be passed as environment variables anyway! You can read more about it here.


来源: stackexchange

问:
IRC function dilemma
[EDIT] Sorry about the title being off. I was thinking of a different problem I was having, and accidentally crossed the two.

Hi everyone, I'm writing my IRC bot skeleton in Python currently.

To explain a couple things before I ask my question, I am writing a IRC bot complete with a plug-in system that will receive signals (i.e. when a message is received, a person was seen joining a channel, ...), and to call these functions contained in the plug-in, I will be using a helper method, i.e. if we saw a message, call sawMessage. sawMessage will locate which plug-ins want to handle messages, using a PluginManager class, etc. I have the whole system thought out.

I also have a function for each IRC action, i.e. "doJoin", "doPart", etc. so that plug-ins can call these without me exposing my socket connection to the plug-ins directly. -- Should these functions automatically call the appropriate sawMessage, sawJoin, etc. ? Or should it be up to the plug-in to call sawMessage after calling doMessage? The reason I am asking is because if doMessage calls sawMessage, and sawMessage processes something that calls doMessage, it can end up recursing a lot before unwinding the stack.

The drawbacks of making the plug-in responsible for calling sawMessage to notify the other plug-ins is that the plug-in would have a choice of NOT notifying the other plug-ins, and also that if the plug-in DOES call the sawMessage function, the recursion issue could pop right back up.

I'm not sure how to resolve this. It seems like good practice to make doMessage automatically call sawMessage to notify other plug-ins, but there is always the possibility of a stack overflow due to recursion if the plug-in is not designed properly (i.e. not to respond to it's own messages), but if the plug-in has to call sawMessage anyway, the same issue is there.

Got any thoughts/opinions on this? I know it's not a straight-out question/answer, but I can't decide what to do with this. Ordinarily, if I was just writing a small bot for my own usage, I wouldn't care about a possible recursion problem. However, I am writing this for the purpose of distribution (I want this to be the next HUGE bot, just like every other IRC bot developer :D ) and I want it made properly.

Thanks!

答: It sounds like you need to decide between or read up more on Publish/subscribe and Chain of Responsibility. Unfortunately I don't have much experience in this area. Just happened to read a related article recently.</p>

ContentLicense: CC BY-SA 3.0

Title: IRC function dilemma


问:
Tkinter freezes when looping through OHLC data
I programmed an app in Tkinter and when I run it the ohlc plot is working properly but the tkinter window is freezing. I am a beginner of multiprocessing in Tkinter and any help is appreciated.

https://github.com/DataScientist2807/Tkinter/blob/master/SetupFrameWindowStockkApp_031_stackoverflow.py

edit: I attached the code below. The GUI is freezing. I do not know how to best solve this issue. Whether I use Multiprocessing or using the animate function in Matplotlib or any other solution.

```<code>import tkinter as tk
from tkinter import Tk, Frame, Menu, ttk
from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg, NavigationToolbar2Tk
from matplotlib.figure import Figure
from matplotlib import pyplot as plt
import os
import pandas as pd
import json
import matplotlib.dates as mdates
from mplfinance.original_flavor import candlestick_ohlc
from datetime import datetime
import time
import matplotlib.animation as animation
import matplotlib.ticker as mticker
import numpy as np
import threading
from multiprocessing import Process
import multiprocessing

def convert_ticks_to_ohlc(df, df_column, timeframe):
    data_frame = df[df_column].resample(timeframe).ohlc()
    return data_frame

valInterval = "5Min"
valCandles = 30
valValueSim = 0

class StockApp(Frame):

    def __init__(self):
        super().__init__()

        Process(target=self.initiate_mastergrid()).start()
        Process(target=self.init_menuBar()).start()
        Process(target=self.initFrames()).start()
        Process(target=self.main_frame()).start()

    def initiate_mastergrid(self):
        # Configure the grid
        self.master.grid_rowconfigure(1, weight=4)
        self.master.grid_columnconfigure(1, weight=1)
        self.master.grid_columnconfigure(2, weight=1)
        self.master.grid_columnconfigure(3, weight=1)

    def initialize_mainchart(self):
        self.figmain = plt.figure(figsize=(20,5), dpi=80)
        dataPlot2 = FigureCanvasTkAgg(self.figmain, master=self.mainFrame)
        self.am2 = plt.subplot2grid((4,5), (1,0), rowspan=2, colspan=5)
        self.am1 = plt.subplot2grid((4,5), (0,0), rowspan=1, colspan=5, facecolor="#F0F0F0", sharex=self.am2)
        self.am3 = plt.subplot2grid((4,5), (3,0), rowspan=1, colspan=5, sharex=self.am2)
        self.am2.yaxis.tick_right() # Price, Indicator and Volume Labels on the right
        self.am1.yaxis.tick_right() # Price, Indicator and Volume Labels on the right
        self.am3.yaxis.tick_right() # Price, Indicator and Volume Labels on the right

        plt.setp(self.am2.get_xticklabels(), visible=False)
        plt.setp(self.am1.get_xticklabels(), visible=False)
        plt.tight_layout(pad=0, w_pad=-1, h_pad=-1)

        plt.ion() # Necessary to animate plot
        dataPlot2.draw()
        dataPlot2.get_tk_widget().pack(side=tk.TOP, fill=tk.BOTH, expand=1)

    def load_data(self):
        global tick_data
        os.chdir("C:\\Trainings\\Python\\Tkinter\\StockApp\\SampleData")
        tick_data = pd.read_csv("EURUSD-2019_01_01.csv",
                        index_col=["time"],
                        usecols=["time", "ask", "bid"],
                        parse_dates=["time"])

        print("Data has been loaded")

    def clear_axes(self, axes):
        axes.clear() # - Clear the chart

    def update_chart(self, candle_data):

        candlestick_ohlc(self.am2, candle_data, width=(15*0.5)/(24*60),
                         colorup='#075105',
                         colordown='#AF141A')

        for label in self.am2.xaxis.get_ticklabels():
            label.set_rotation(45)
        self.am2.xaxis.set_major_locator(mticker.MaxNLocator(10))
        self.am2.grid(True)
        plt.grid(False)
        plt.xlabel('Candle count')
        plt.ylabel('Price')
        plt.title('Candlestick chart simulation')
        plt.subplots_adjust(left=0.09, bottom=0.20, right=0.94,
                            top=0.90, wspace=0.2, hspace=0)
        self.figmain.canvas.draw() # - Draw on the chart

    def candlestick_simulation(self, tick_data, time_frame, candle_window):

        candlestick_data = convert_ticks_to_ohlc(tick_data,
                                                 "ask",
                                                 "15Min").dropna()

        print("Converts tick to ohlc")

        candlestick_data = candlestick_data.reset_index()

        dvalues = candlestick_data[['open', 'high', 'low', 'close']].values.tolist()
        pdates = mdates.date2num(candlestick_data.time)
        ohlc = [[pdates[i]] + dvalues[i] for i in range(len(pdates))]

        print("Converted ohlc")

        self.ohlc_animate(ohlc)

        print("Start animation")

    def ohlc_animate(self, ohlclist):

        emptyOHLC = []

        for i in range(0, 20):

            valuesOHLC = ohlclist[i]

            emptyOHLC = emptyOHLC + [valuesOHLC]

            p_update = Process(target=self.update_chart(emptyOHLC))
            p_update.start()
            p_update.join()

            #self.update_chart(emptyOHLC)

            print("Sleep 3 seconds")

            time.sleep(3.0)

            self.clear_axes(self.am2)

    def start_simulation(self):

        Process(target=self.candlestick_simulation(tick_data, valInterval, valCandles)).start()
        print("Start Simulation")

    def main_frame(self):
        global dataB

        Process(target=self.initialize_mainchart()).start()
        print("Main Frame initialised")

        Process(target=self.load_data()).start()
        print(len(tick_data))

    def initFrames(self):

        self.mainFrame = Frame(self.master,bd=1, relief="sunken") #whatever
        self.mainFrame.grid(row=1, column=1, columnspan=2, sticky="nsew", padx=1, pady=1)

    def init_menuBar(self):

        menubar = Menu(self.master)
        self.master.config(menu=menubar)

        # Add menu items
        file_menu = Menu(menubar, tearoff=0)
        menubar.add_cascade(label="File", menu=file_menu)
        indicator_menu = Menu(menubar, tearoff=0)
        indicator_menu.add_command(label="EURUSD", command=lambda: self.start_simulation())
        file_menu.add_cascade(label="Interval", menu=indicator_menu)

def main():
    app = StockApp()
    app.master.title("Stock App 2020 by datascience2807@gmail.com")
    width_value = app.master.winfo_screenwidth()
    height_value = app.master.winfo_screenheight()
    app.master.geometry("%dx%d+0+0" % (width_value,height_value))
    app.mainloop()

if __name__ == '__main__':
    pool = multiprocessing.Pool()
    threading.Thread(target=main()).start()
</code>```

答:
The target parameter in the Thread constructor should be a callable. Instead are calling main() and passing the return value of main (which does not return of course because its running the tkinter event loop).

```<code>threading.Thread(target=main()).start()
</code>```

I think you meant to do this:

```<code>threading.Thread(target=main).start()
</code>```

ContentLicense: CC BY-SA 4.0

Body:
I programmed an app in Tkinter and when I run it the ohlc plot is working properly but the tkinter window is freezing. I am a beginner of multiprocessing in Tkinter and any help is appreciated.

https://github.com/DataScientist2807/Tkinter/blob/master/SetupFrameWindowStockkApp_031_stackoverflow.py

edit: I attached the code below. The GUI is freezing. I do not know how to best solve this issue. Whether I use Multiprocessing or using the animate function in Matplotlib or any other solution.

import tkinter as tk
from tkinter import Tk, Frame, Menu, ttk
from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg, NavigationToolbar2Tk
from matplotlib.figure import Figure
from matplotlib import pyplot as plt
import os
import pandas as pd
import json
import matplotlib.dates as mdates
from mplfinance.original_flavor import candlestick_ohlc
from datetime import datetime
import time
import matplotlib.animation as animation
import matplotlib.ticker as mticker
import numpy as np
import threading
from multiprocessing import Process
import multiprocessing

def convert_ticks_to_ohlc(df, df_column, timeframe):
    data_frame = df[df_column].resample(timeframe).ohlc()
    return data_frame

valInterval = "5Min"
valCandles = 30
valValueSim = 0

class StockApp(Frame):

    def __init__(self):
        super().__init__()

        Process(target=self.initiate_mastergrid()).start()
        Process(target=self.init_menuBar()).start()
        Process(target=self.initFrames()).start()
        Process(target=self.main_frame()).start()

    def initiate_mastergrid(self):
        # Configure the grid
        self.master.grid_rowconfigure(1, weight=4)
        self.master.grid_columnconfigure(1, weight=1)
        self.master.grid_columnconfigure(2, weight=1)
        self.master.grid_columnconfigure(3, weight=1)

    def initialize_mainchart(self):
        self.figmain = plt.figure(figsize=(20,5), dpi=80)
        dataPlot2 = FigureCanvasTkAgg(self.figmain, master=self.mainFrame)
        self.am2 = plt.subplot2grid((4,5), (1,0), rowspan=2, colspan=5)
        self.am1 = plt.subplot2grid((4,5), (0,0), rowspan=1, colspan=5, facecolor="#F0F0F0", sharex=self.am2)
        self.am3 = plt.subplot2grid((4,5), (3,0), rowspan=1, colspan=5, sharex=self.am2)
        self.am2.yaxis.tick_right() # Price, Indicator and Volume Labels on the right
        self.am1.yaxis.tick_right() # Price, Indicator and Volume Labels on the right
        self.am3.yaxis.tick_right() # Price, Indicator and Volume Labels on the right

        plt.setp(self.am2.get_xticklabels(), visible=False)
        plt.setp(self.am1.get_xticklabels(), visible=False)
        plt.tight_layout(pad=0, w_pad=-1, h_pad=-1)

        plt.ion() # Necessary to animate plot
        dataPlot2.draw()
        dataPlot2.get_tk_widget().pack(side=tk.TOP, fill=tk.BOTH, expand=1)

    def load_data(self):
        global tick_data
        os.chdir("C:\\Trainings\\Python\\Tkinter\\StockApp\\SampleData")
        tick_data = pd.read_csv("EURUSD-2019_01_01.csv",
                        index_col=["time"],
                        usecols=["time", "ask", "bid"],
                        parse_dates=["time"])

        print("Data has been loaded")

    def clear_axes(self, axes):
        axes.clear() # - Clear the chart

    def update_chart(self, candle_data):

        candlestick_ohlc(self.am2, candle_data, width=(15*0.5)/(24*60),
                         colorup='#075105',
                         colordown='#AF141A')

        for label in self.am2.xaxis.get_ticklabels():
            label.set_rotation(45)
        self.am2.xaxis.set_major_locator(mticker.MaxNLocator(10))
        self.am2.grid(True)
        plt.grid(False)
        plt.xlabel('Candle count')
        plt.ylabel('Price')
        plt.title('Candlestick chart simulation')
        plt.subplots_adjust(left=0.09, bottom=0.20, right=0.94,
                            top=0.90, wspace=0.2, hspace=0)
        self.figmain.canvas.draw() # - Draw on the chart

    def candlestick_simulation(self, tick_data, time_frame, candle_window):

        candlestick_data = convert_ticks_to_ohlc(tick_data,
                                                 "ask",
                                                 "15Min").dropna()

        print("Converts tick to ohlc")

        candlestick_data = candlestick_data.reset_index()

        dvalues = candlestick_data[['open', 'high', 'low', 'close']].values.tolist()
        pdates = mdates.date2num(candlestick_data.time)
        ohlc = [[pdates[i]] + dvalues[i] for i in range(len(pdates))]

        print("Converted ohlc")

        self.ohlc_animate(ohlc)

        print("Start animation")

    def ohlc_animate(self, ohlclist):

        emptyOHLC = []

        for i in range(0, 20):

            valuesOHLC = ohlclist[i]

            emptyOHLC = emptyOHLC + [valuesOHLC]

            p_update = Process(target=self.update_chart(emptyOHLC))
            p_update.start()
            p_update.join()

            #self.update_chart(emptyOHLC)

            print("Sleep 3 seconds")

            time.sleep(3.0)

            self.clear_axes(self.am2)

    def start_simulation(self):

        Process(target=self.candlestick_simulation(tick_data, valInterval, valCandles)).start()
        print("Start Simulation")

    def main_frame(self):
        global dataB

        Process(target=self.initialize_mainchart()).start()
        print("Main Frame initialised")

        Process(target=self.load_data()).start()
        print(len(tick_data))

    def initFrames(self):

        self.mainFrame = Frame(self.master,bd=1, relief="sunken") #whatever
        self.mainFrame.grid(row=1, column=1, columnspan=2, sticky="nsew", padx=1, pady=1)

    def init_menuBar(self):

        menubar = Menu(self.master)
        self.master.config(menu=menubar)

        # Add menu items
        file_menu = Menu(menubar, tearoff=0)
        menubar.add_cascade(label="File", menu=file_menu)
        indicator_menu = Menu(menubar, tearoff=0)
        indicator_menu.add_command(label="EURUSD", command=lambda: self.start_simulation())
        file_menu.add_cascade(label="Interval", menu=indicator_menu)

def main():
    app = StockApp()
    app.master.title("Stock App 2020 by datascience2807@gmail.com")
    width_value = app.master.winfo_screenwidth()
    height_value = app.master.winfo_screenheight()
    app.master.geometry("%dx%d+0+0" % (width_value,height_value))
    app.mainloop()

if __name__ == '__main__':
    pool = multiprocessing.Pool()
    threading.Thread(target=main()).start()

Body:
The target parameter in the Thread constructor should be a callable. Instead are calling main() and passing the return value of main (which does not return of course because its running the tkinter event loop).

threading.Thread(target=main()).start()

I think you meant to do this:

threading.Thread(target=main).start()


问:
Can you integrate SSIS project with website?
According to my project requirements, I need to develop a few pages on the company website where the user would upload an excel sheet to transfer the data to MS SQL Server. I am getting started with SSIS but I want to know how to make this part of a web page.</p>

答:
You cannot use SSIS package in web applications. Also processing Excel file inside your application might be complecated so, one common approach for this problem is:

Store uploaded document in a specific folder.

Name the files using a timestamp like (F2020_05_29_10:330:24_3283) or any other methods to make them unique

Deploy your SSIS package to SQL Server and schedule a job to run the package

In your package look into that specific folder to see if there is any new incoming file.

If there is one or more files, process them inside your SSIS package

Another way is processing your Excel file using COM Interop or using OpenXML which might be complicated.


答:
Yes you can, but you probably don't want to.

Excel and SSIS are finicky bedfellows at best. SSIS believes in a firm set of metadata being established a design time and adhered to at runtime. Excel interprets metadata based on sampling the first 8 rows of data. If the Excel file is machine generated, then SSIS and it can work together fine enough but since this will be users uploading files... you're going to spend a lot of time troubleshooting why two seemingly identical spreadsheets won't parse the same way in SSIS.

How do you get SSIS to run? Either you use the object model or use dtexec.exe to make it go. Both of those approaches are only valid if you have licensed the server for SQL Server. If you install the object library or the Integration Services Service on your web server, you will need to buy another SQL Server license for your enterprise. That's expensive.

What I, probably the person most in love with SSIS on Stack Overflow do? I'd use an OpenXml library, like EPPlus. I blogged about an approach in Reading Excel files without Excel

```<code>using OfficeOpenXml;
// ...
/// Get data from Excel worksheet, assuming columns A-G are desired
public Dictionary<string, List<string>> GetExcelDriverData(string sourceFile)
{
    Dictionary<string, List<string>> d = new Dictionary<string, List<string>>();
    System.IO.FileInfo fi = new System.IO.FileInfo(sourceFile);
    using (ExcelPackage ep = new ExcelPackage(fi))
    {
        ExcelWorkbook wb = ep.Workbook;
        ExcelWorksheet ws = wb.Worksheets.First();
        if (ws != null)
        {
            // 1 based array to 7, inclusive
            for (int i = ws.Dimension.Start.Row+1; i < ws.Dimension.End.Row+1; i++)
            {
                List<string> row = new List<string>() { ws.Cells[i, 1].Value.ToString()
                ,   ws.Cells[i, 2].Value.ToString()
                ,   ws.Cells[i, 3].Value.ToString()
                ,   ws.Cells[i, 4].Value.ToString()
                ,   ws.Cells[i, 5].Value.ToString()
                ,   ws.Cells[i, 6].Value.ToString()
                ,   ws.Cells[i, 7].Value.ToString()
                };

                d[ws.Cells[i, 7].Value.ToString()] = row;
            }
        }
    }

    return d;
}
</code>```

Note that this is the last LGPL version of EPPlus if you're nuget'ing things. Version 5+ requires licensing but it's far cheaper than a SQL Server license.

Title: Can you integrate SSIS project with website?

Body: According to my project requirements, I need to develop a few pages on the company website where the user would upload an excel sheet to transfer the data to MS SQL Server. I am getting started with SSIS but I want to know how to make this part of a web page.

Body:
Yes you can, but you probably don't want to.

Excel and SSIS are finicky bedfellows at best. SSIS believes in a firm set of metadata being established a design time and adhered to at runtime. Excel interprets metadata based on sampling the first 8 rows of data. If the Excel file is machine generated, then SSIS and it can work together fine enough but since this will be users uploading files... you're going to spend a lot of time troubleshooting why two seemingly identical spreadsheets won't parse the same way in SSIS.

How do you get SSIS to run? Either you use the object model or use dtexec.exe to make it go. Both of those approaches are only valid if you have licensed the server for SQL Server. If you install the object library or the Integration Services Service on your web server, you will need to buy another SQL Server license for your enterprise. That's expensive.

What I, probably the person most in love with SSIS on Stack Overflow do? I'd use an OpenXml library, like EPPlus. I blogged about an approach in Reading Excel files without Excel

using OfficeOpenXml;
// ...
/// Get data from Excel worksheet, assuming columns A-G are desired
public Dictionary<string, List<string>> GetExcelDriverData(string sourceFile)
{
    Dictionary<string, List<string>> d = new Dictionary<string, List<string>>();
    System.IO.FileInfo fi = new System.IO.FileInfo(sourceFile);
    using (ExcelPackage ep = new ExcelPackage(fi))
    {
        ExcelWorkbook wb = ep.Workbook;
        ExcelWorksheet ws = wb.Worksheets.First();
        if (ws != null)
        {
            // 1 based array to 7, inclusive
            for (int i = ws.Dimension.Start.Row+1; i < ws.Dimension.End.Row+1; i++)
            {
                List<string> row = new List<string>() { ws.Cells[i, 1].Value.ToString()
                ,   ws.Cells[i, 2].Value.ToString()
                ,   ws.Cells[i, 3].Value.ToString()
                ,   ws.Cells[i, 4].Value.ToString()
                ,   ws.Cells[i, 5].Value.ToString()
                ,   ws.Cells[i, 6].Value.ToString()
                ,   ws.Cells[i, 7].Value.ToString()
                };

                d[ws.Cells[i, 7].Value.ToString()] = row;
            }
        }
    }

    return d;
}

Note that this is the last LGPL version of EPPlus if you're nuget'ing things. Version 5+ requires licensing but it's far cheaper than a SQL Server license.


问:
Node.js Http.request slows down under load testing. Am I doing something wrong?
Here is my sample code:

```<code>var http = require('http');

var options1 = {
          host: 'www.google.com',
          port: 80,
          path: '/',
          method: 'GET'
        };

http.createServer(function (req, res) {

        var start = new Date();
        var myCounter = req.query['myCounter'] || 0;

        var isSent = false;
        http.request(options1, function(response) {
            response.setEncoding('utf8');
            response.on('data', function (chunk) {
                var end = new Date();
                console.log(myCounter + ' BODY: ' + chunk  + " time: " + (end-start) + " Request start time: " + start.getTime());

                if (! isSent) {
                    isSent = true;
                    res.writeHead(200, {'Content-Type': 'application/xml'});
                    res.end(chunk);
                }
            });
        }).end();

}).listen(3013);

console.log('Server running at port 3013');
</code>```

What i found out is that, if I connect to other server, (google or any other), the response will get slower and slower to few seconds.   It doesn't happen if I connect to another node.js server within the same network.

I use JMeter to test.  50 concurrent per sec with 1000 loop.

I have no idea what the problem is...

=========================

Further investigation:

I run the same script on Rackspace, and also on EC2 for testing.   And the script will use http.request to connect to: Google, Facebook, and also my another script that simply output data (like hello world) which is hosted by another EC2 instance.

The test tool I just is jMeter on my desktop.

Pre-node.js test:
  jMeter -> Google    Result:  fast and consistent.
  jMeter -> Facebook   result: fast and consistent.
  jMeter -> My Simple Output Script     Result:  fast and consistent.

Then I make a 50 Concurrent Threads /sec , with 100 loops, testing to my Rackspace nodejs, and then EC2 node.js which has the same king of performance issue
  jMeter -> node.js -> Google    Result:  from 50 ms  goes to 2000 ms in 200 requsets.

  jMeter -> node.js -> Facebook Result:  from 200 ms goes  to 3000 ms after 200 requsets.

  jMeter -> node.js -> My Simple Output Script   Result:  from 100 ms goes  to 1000 ms after 200 requsets.

  The first 10-20 requests are fast, then start slowing down.

Then, when I change to 10 Concurrent Threads, things start changing..   The response is very consistent, no slow down.

Something to do with # of concurrent threads that Node.js (http.request) can handle.

------------ More --------------

I did more test today and here it is:
I used http.Agent and increase the max socket.  However, the interesting this is that, on one testing server (EC2), it improves a lot and no more slow down.  HOwever, the other server (rackspace) only improves a bit.  It still shows slow down.  I even set "Connection: close" in the request header, it only improves 100ms.

if http.request is using connection pooling, how to increase it?

in both server, if I do "ulimit -a", the # of file open is 1024.

------------- ** MORE AND MORE ** -------------------

It seems that even I set maxSockets to higher number, it only works at some limit.  There seems to be a internal or OS dependent socket limitation.  However to bump it up?

------------- ** AFTER EXTENSIVE TESTING ** ---------------

After reading lots of post, I find out:

quoted from: https://github.com/joyent/node/issues/877

1) If I set the headers with connection = 'keep-alive', the performance is good and can go up to maxSocket = 1024 (which is my linux setting).

```<code>var options1 = {
                  host: 'www.google.com',
                  port: 80,
                  path: '/',
                  method: 'GET',
    **headers: {
            'Connection':'keep-alive'
    }**
                };
</code>```

If I set it to "Connection":"close", the response time would be 100 times slower.

Funny things happened here:

1) in EC2, when I first test with Connection:keep-alive, it will take about 20-30 ms.  Then if I change to Connection:Close  OR set Agent:false, the response time will slow down to 300 ms. WIHTOUT restarting the server, if I change to Connection:keep-alive again, the response time will slow down even further to 4000ms.   Either I have to restart server or wait for a while to get back my 20-30ms lighting speed response.

2) If I run it with agent:false, at first, the response time will slow down to 300ms.  But then, it will get faster again and back to "normal".

My guess is connection pooling still in effect even you set agent:false.   However, if you keep connection:keep-alive, then it will be fast for sure.  just don't switch it.

Update on July 25, 2011

I tried the latest node.js V0.4.9 with the http.js & https.js fix from https://github.com/mikeal/node/tree/http2

the performance is much better and stable.

答:
This will not neccesarily fix your problem, but it cleans up your code a bit and makes use of the various events in the way you should:

var http = require('http');

var options1 = {
      host: 'www.google.com',
      port: 80,
      path: '/',
      method: 'GET'
};

http.createServer(function (req, res) {
    var start = new Date();
    var myCounter = req.query['myCounter'] || 0;

    http.request(options1, function(response) {
        res.on('drain', function () { // when output stream's buffer drained
            response.resume(); // continue to receive from input stream
        });
        response.setEncoding('utf8');
        res.writeHead(response.statusCode, {'Content-Type': 'application/xml'});
        response.on('data', function (chunk) {
            if (!res.write(chunk)) { // if write failed, the stream is choking
                response.pause(); // tell the incoming stream to wait until output stream drained
            }
        }).on('end', function () {
            var end = new Date();
            console.log(myCounter + ' time: ' + (end-start) + " Request start time: " + start.getTime());
            res.end();
        });
    }).end();
}).listen(3013);

console.log('Server running at port 3013');
```

I removed the output of the body. Since we are streaming from one socket to another we cannot be sure to see the entire body at any time without buffering it.

EDIT: I believe node is using connection pooling for http.request. If you have 50 concurrent connections (and thus 50 concurrent http.request attempts) you might be running into the connection pool limit. I currently have no time to look this up for you, but you should have a look at the node documentation regarding http, especially http agent.

EDIT 2: There's a thread regarding a very similar problem on the node.js mailing list. You should have a look at it, especially Mikael's post should be of interest. He suggests turning off connection pooling entirely for the requests by passing the option agent: false to the http.request invocation. I do not have any further clues, so if this does not help maybe you should try getting help on the node.js mailing list.

Body:
Here is my sample code:

var http = require('http');

var options1 = {
          host: 'www.google.com',
          port: 80,
          path: '/',
          method: 'GET'
        };

http.createServer(function (req, res) {

        var start = new Date();
        var myCounter = req.query['myCounter'] || 0;

        var isSent = false;
        http.request(options1, function(response) {
            response.setEncoding('utf8');
            response.on('data', function (chunk) {
                var end = new Date();
                console.log(myCounter + ' BODY: ' + chunk  + " time: " + (end-start) + " Request start time: " + start.getTime());

                if (! isSent) {
                    isSent = true;
                    res.writeHead(200, {'Content-Type': 'application/xml'});
                    res.end(chunk);
                }
            });
        }).end();

}).listen(3013);

console.log('Server running at port 3013');

What i found out is that, if I connect to other server, (google or any other), the response will get slower and slower to few seconds.   It doesn't happen if I connect to another node.js server within the same network.

I use JMeter to test.  50 concurrent per sec with 1000 loop.

I have no idea what the problem is...

=========================

Further investigation:

I run the same script on Rackspace, and also on EC2 for testing.   And the script will use http.request to connect to: Google, Facebook, and also my another script that simply output data (like hello world) which is hosted by another EC2 instance.

The test tool I just is jMeter on my desktop.

Pre-node.js test:
  jMeter -> Google    Result:  fast and consistent.
  jMeter -> Facebook   result: fast and consistent.
  jMeter -> My Simple Output Script     Result:  fast and consistent.

Then I make a 50 Concurrent Threads /sec , with 100 loops, testing to my Rackspace nodejs, and then EC2 node.js which has the same king of performance issue
  jMeter -> node.js -> Google    Result:  from 50 ms  goes to 2000 ms in 200 requsets.

  jMeter -> node.js -> Facebook Result:  from 200 ms goes  to 3000 ms after 200 requsets.

  jMeter -> node.js -> My Simple Output Script   Result:  from 100 ms goes  to 1000 ms after 200 requsets.

  The first 10-20 requests are fast, then start slowing down.

Then, when I change to 10 Concurrent Threads, things start changing..   The response is very consistent, no slow down.

Something to do with # of concurrent threads that Node.js (http.request) can handle.

------------ More --------------

I did more test today and here it is:
I used http.Agent and increase the max socket.  However, the interesting this is that, on one testing server (EC2), it improves a lot and no more slow down.  HOwever, the other server (rackspace) only improves a bit.  It still shows slow down.  I even set "Connection: close" in the request header, it only improves 100ms.

if http.request is using connection pooling, how to increase it?

in both server, if I do "ulimit -a", the # of file open is 1024.

------------- ** MORE AND MORE ** -------------------

It seems that even I set maxSockets to higher number, it only works at some limit.  There seems to be a internal or OS dependent socket limitation.  However to bump it up?

------------- ** AFTER EXTENSIVE TESTING ** ---------------

After reading lots of post, I find out:

quoted from: https://github.com/joyent/node/issues/877

1) If I set the headers with connection = 'keep-alive', the performance is good and can go up to maxSocket = 1024 (which is my linux setting).

var options1 = {
                  host: 'www.google.com',
                  port: 80,
                  path: '/',
                  method: 'GET',
    **headers: {
            'Connection':'keep-alive'
    }**
                };

If I set it to "Connection":"close", the response time would be 100 times slower.

Funny things happened here:

1) in EC2, when I first test with Connection:keep-alive, it will take about 20-30 ms.  Then if I change to Connection:Close  OR set Agent:false, the response time will slow down to 300 ms. WIHTOUT restarting the server, if I change to Connection:keep-alive again, the response time will slow down even further to 4000ms.   Either I have to restart server or wait for a while to get back my 20-30ms lighting speed response.

2) If I run it with agent:false, at first, the response time will slow down to 300ms.  But then, it will get faster again and back to "normal".

My guess is connection pooling still in effect even you set agent:false.   However, if you keep connection:keep-alive, then it will be fast for sure.  just don't switch it.

Update on July 25, 2011

I tried the latest node.js V0.4.9 with the http.js & https.js fix from https://github.com/mikeal/node/tree/http2

the performance is much better and stable.

Body:
This will not neccesarily fix your problem, but it cleans up your code a bit and makes use of the various events in the way you should:

var http = require('http');

var options1 = {
      host: 'www.google.com',
      port: 80,
      path: '/',
      method: 'GET'
};

http.createServer(function (req, res) {
    var start = new Date();
    var myCounter = req.query['myCounter'] || 0;

    http.request(options1, function(response) {
        res.on('drain', function () { // when output stream's buffer drained
            response.resume(); // continue to receive from input stream
        });
        response.setEncoding('utf8');
        res.writeHead(response.statusCode, {'Content-Type': 'application/xml'});
        response.on('data', function (chunk) {
            if (!res.write(chunk)) { // if write failed, the stream is choking
                response.pause(); // tell the incoming stream to wait until output stream drained
            }
        }).on('end', function () {
            var end = new Date();
            console.log(myCounter + ' time: ' + (end-start) + " Request start time: " + start.getTime());
            res.end();
        });
    }).end();
}).listen(3013);

console.log('Server running at port 3013');

I removed the output of the body. Since we are streaming from one socket to another we cannot be sure to see the entire body at any time without buffering it.

EDIT: I believe node is using connection pooling for http.request. If you have 50 concurrent connections (and thus 50 concurrent http.request attempts) you might be running into the connection pool limit. I currently have no time to look this up for you, but you should have a look at the node documentation regarding http, especially http agent.

EDIT 2: There's a thread regarding a very similar problem on the node.js mailing list. You should have a look at it, especially Mikael's post should be of interest. He suggests turning off connection pooling entirely for the requests by passing the option agent: false to the http.request invocation. I do not have any further clues, so if this does not help maybe you should try getting help on the node.js mailing list.


问:
Change varchar column to datetime?
I have column in mysql table called "dtm"

Data in it is like this "20200525/0704Z"

It has date and time in Zulu format.

Currently column type is varchar 255.

How do I convert this to more useful column type like datetime?

Im getting "too big precision error max is 6" when trying to just change column type.

Thanks

答: You are better off creating a second column with a datetime datatype. Then use php to convert the old dtm format and update it into its new datetime column, before finally deleting the old dtm column.</p>

Title: Change varchar column to datetime?


问:
Access Firestore ID generator on the front end
I want to set my document ids on the front end, at the same time I set the doc, so I was wondering if there is a way to generate Firestore IDs, which could look like this:

```<code>const theID = firebase.firestore().generateID() // something like this

firebase.firestore().collection('posts').doc(theID).set({
    id: theID,
    ...otherData
})

</code>```

I could use uuid or some other id generator package, but I'm looking for a Firestore id generator. This SO answer points to some newId method, but I can't find it in the JS SDK... (https://www.npmjs.com/package/firebase)

答:
```<code>import {randomBytes} from 'crypto';

export function autoId(): string {
  const chars =
    'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
  let autoId = '';
  while (autoId.length < 20) {
    const bytes = randomBytes(40);
    bytes.forEach(b => {
      // Length of `chars` is 62. We only take bytes between 0 and 62*4-1
      // (both inclusive). The value is then evenly mapped to indices of `char`
      // via a modulo operation.
      const maxValue = 62 * 4 - 1;
      if (autoId.length < 20 && b <= maxValue) {
        autoId += chars.charAt(b % 62);
      }
    });
  }
  return autoId;
}
</code>```

Taken from the Firestore Node.js SDK:
https://github.com/googleapis/nodejs-firestore/blob/4f4574afaa8cf817d06b5965492791c2eff01ed5/dev/src/util.ts#L52

Body:
I want to set my document ids on the front end, at the same time I set the doc, so I was wondering if there is a way to generate Firestore IDs, which could look like this:

const theID = firebase.firestore().generateID() // something like this

firebase.firestore().collection('posts').doc(theID).set({
    id: theID,
    ...otherData
})

I could use uuid or some other id generator package, but I'm looking for a Firestore id generator. This SO answer points to some newId method, but I can't find it in the JS SDK... (https://www.npmjs.com/package/firebase)

Body:
import {randomBytes} from 'crypto';

export function autoId(): string {
  const chars =
    'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
  let autoId = '';
  while (autoId.length < 20) {
    const bytes = randomBytes(40);
    bytes.forEach(b => {
      // Length of `chars` is 62. We only take bytes between 0 and 62*4-1
      // (both inclusive). The value is then evenly mapped to indices of `char`
      // via a modulo operation.
      const maxValue = 62 * 4 - 1;
      if (autoId.length < 20 && b <= maxValue) {
        autoId += chars.charAt(b % 62);
      }
    });
  }
  return autoId;
}

Taken from the Firestore Node.js SDK:
https://github.com/googleapis/nodejs-firestore/blob/4f4574afaa8cf817d06b5965492791c2eff01ed5/dev/src/util.ts#L52


问:
My app stops when I open it with 4.2 "api 16"?
When I open my app with samsung s3 its not opening.  It works with other versions.

This is the error:

My app works well with higher versions and my database is back4app.

My minsdk is 15.

答:
the error now is :</p>

```<code>E/AndroidRuntime: FATAL EXCEPTION: main
java.lang.NoClassDefFoundError: com.parse.ParsePlugins
    at com.parse.Parse.isInitialized(Parse.java:273)
    at com.parse.Parse.initialize(Parse.java:130)
    at com.parse.Parse.initialize(Parse.java:126)
    at com.pluspath.App.onCreate(App.java:14)
    at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1003)
    at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4228)
    at android.app.ActivityThread.access$1300(ActivityThread.java:140)
    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1287)
    at android.os.Handler.dispatchMessage(Handler.java:99)
    at android.os.Looper.loop(Looper.java:137)
    at android.app.ActivityThread.main(ActivityThread.java:4898)
    at java.lang.reflect.Method.invokeNative(Native Method)
    at java.lang.reflect.Method.invoke(Method.java:511)
    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1006)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:773)
    at dalvik.system.NativeStart.main(Native Method)
</code>```

Body:
the error now is :

E/AndroidRuntime: FATAL EXCEPTION: main
java.lang.NoClassDefFoundError: com.parse.ParsePlugins
    at com.parse.Parse.isInitialized(Parse.java:273)
    at com.parse.Parse.initialize(Parse.java:130)
    at com.parse.Parse.initialize(Parse.java:126)
    at com.pluspath.App.onCreate(App.java:14)
    at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1003)
    at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4228)
    at android.app.ActivityThread.access$1300(ActivityThread.java:140)
    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1287)
    at android.os.Handler.dispatchMessage(Handler.java:99)
    at android.os.Looper.loop(Looper.java:137)
    at android.app.ActivityThread.main(ActivityThread.java:4898)
    at java.lang.reflect.Method.invokeNative(Native Method)
    at java.lang.reflect.Method.invoke(Method.java:511)
    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1006)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:773)
    at dalvik.system.NativeStart.main(Native Method)


问:
Azure service bus can u configure topic to only accept xml according to xsd?
Can an Azure service bus Topic (or Queue) be configured to only accept Xml messages according to a specific Xsd? </p>

答: No, that is not possible. Azure Service Bus does not touch the payload. Subscription filters also can only operate on the custom headers, not the values. You would need a process outside of Azure Service Bus to validate the payload.</p>


问:
How i can get Token from headers?
I did authorization via http. post, send a JSON Body with username and password, in the response I get a Header, the Header has a token, it is stored in 'set_cookie: Authorization=token', how do I get it and write it to the storage?</p>

答:
You can get the cookie from the response of login request using the following code

HttpClient _httpClient = new HttpClient();
HttpClientRequest request = await _httpClient.postUrl(Uri.parse(url));
request.headers.set('content-type', 'application/json');
request.add(utf8.encode(json.encode(jsonMap)));
HttpClientResponse response = await request.close();
print(response.cookies); // this is a List<Cookie>, you can iterate and find the required cookie
```

<p>Now you can store the cookie using shared_preference plugin, and use it in your all the future requests.</p>

<pre class="lang-dart prettyprint-override"><code>HttpClient client = new HttpClient();
HttpClientRequest clientRequest =
      await client.getUrl(Uri.parse("http: //www.example.com/"));
clientRequest.cookies.add(Cookie("sessionid", "asdasdasqqwd"));
</code>```

You can also explore dio library and use AuthInterceptor to add the token for all the requests for you.

Title: How i can get Token from headers?

Body:
You can get the cookie from the response of login request using the following code

HttpClient _httpClient = new HttpClient();
HttpClientRequest request = await _httpClient.postUrl(Uri.parse(url));
request.headers.set('content-type', 'application/json');
request.add(utf8.encode(json.encode(jsonMap)));
HttpClientResponse response = await request.close();
print(response.cookies); // this is a List<Cookie>, you can iterate and find the required cookie

Now you can store the cookie using shared_preference plugin, and use it in your all the future requests.

HttpClient client = new HttpClient();
HttpClientRequest clientRequest =
      await client.getUrl(Uri.parse("http: //www.example.com/"));
clientRequest.cookies.add(Cookie("sessionid", "asdasdasqqwd"));

You can also explore dio library and use AuthInterceptor to add the token for all the requests for you.


问:
How to get the data the server sent back from a reqwest http call?
I have a simple http server, that expects a POST of json data. It will send back the exact same json data, but with an extra 'received' field.

I can demonstrate this using curl:

```<code>$ curl -X POST -H "Content-Type: application/json" -d '{"id":1,"jsonrpc":"2.0","method":"getNext","params":[{"flavour":"apples"}]}' http://127.0.0.1:8008
{"params": [{"flavour": "apples"}], "received": "ok", "jsonrpc": "2.0", "id": 1, "method": "getNext"}
</code>```

The http server was send some json, and it replied the same json but with the extra 'received' field.

I am trying to do the exact same in rust with reqwest.

```<code>use serde_json::{Value};

#[tokio::main]
async fn main() {

    let url = "http://127.0.0.1:8008";

    let data = r#"
        {
            "id": 1,
            "jsonrpc": "2.0",
            "method": "getNext",
            "params": [ {
                "flavour": "apple"
            } ]
        }"#;

    let v: Value = match serde_json::from_str(data) {
        Err(_) => panic!("this is a terrible mistake!"),
        Ok(y) => y,
    };

    let client = reqwest::Client::new();

    let res = client.post(url)
        .json(&v)
        .send()
        .await;

    println!("{:?}",res);
}
</code>```

While this does make the call, and I can see the sent json in the server log, and i can see the server sends back the json, just like with curl... I don't know who to actually get at the data in Rust. All there seems to be is an Ok that the call worked ... but how to get the json the server sent back?

```<code>$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.06s
     Running `/Users/x/RUST/projects/monitor/target/debug/monitor`
Ok(Response { url: "http://127.0.0.1:8008/", status: 200, headers: {"server": "BaseHTTP/0.3 Python/2.7.16", "date": "Fri, 29 May 2020 13:54:39 GMT", "content-type": "application/json"} })
</code>```

答:
This works.</p>

```<code>use serde_json::{Value};

#[tokio::main]
async fn main() {

    let url = "http://127.0.0.1:8008";

    let data = r#"
        {
            "id": 1,
            "jsonrpc": "2.0",
            "method": "getNext",
            "params": [ {
                "flavour": "apple"
            } ]
        }"#;

    let v: Value = match serde_json::from_str(data) {
        Err(_) => panic!("this is a terrible mistake!"),
        Ok(y) => y,
    };

    let client = reqwest::Client::new();

    let result = client.post(url)
        .json(&v)
        .send()
        .await;

    if let Ok(response) = result {
        if let Ok(text) = response.text().await {
            let result: serde_json::Result<Value> = serde_json::from_str(&text);
            if let Ok(j) = result {
                println!("received: {}, flavour still: {}", j["received"], j["params"][0]["flavour"]);
            } else {
                panic!("Could not convert text to json");
            }
        } else {
            panic!("Could not extract text from response");
        }
    } else {
        panic!("No response from server");
    }
}
</code>```

```<code>    Finished dev [unoptimized + debuginfo] target(s) in 1.93s
     Running `target/debug/monitor`
received: "ok", flavour still: "apple"
</code>```

Body:
I have a simple http server, that expects a POST of json data. It will send back the exact same json data, but with an extra 'received' field.

I can demonstrate this using curl:

$ curl -X POST -H "Content-Type: application/json" -d '{"id":1,"jsonrpc":"2.0","method":"getNext","params":[{"flavour":"apples"}]}' http://127.0.0.1:8008
{"params": [{"flavour": "apples"}], "received": "ok", "jsonrpc": "2.0", "id": 1, "method": "getNext"}

The http server was send some json, and it replied the same json but with the extra 'received' field.

I am trying to do the exact same in rust with reqwest.

use serde_json::{Value};

#[tokio::main]
async fn main() {

    let url = "http://127.0.0.1:8008";

    let data = r#"
        {
            "id": 1,
            "jsonrpc": "2.0",
            "method": "getNext",
            "params": [ {
                "flavour": "apple"
            } ]
        }"#;

    let v: Value = match serde_json::from_str(data) {
        Err(_) => panic!("this is a terrible mistake!"),
        Ok(y) => y,
    };

    let client = reqwest::Client::new();

    let res = client.post(url)
        .json(&v)
        .send()
        .await;

    println!("{:?}",res);
}

While this does make the call, and I can see the sent json in the server log, and i can see the server sends back the json, just like with curl... I don't know who to actually get at the data in Rust. All there seems to be is an Ok that the call worked ... but how to get the json the server sent back?

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.06s
     Running `/Users/x/RUST/projects/monitor/target/debug/monitor`
Ok(Response { url: "http://127.0.0.1:8008/", status: 200, headers: {"server": "BaseHTTP/0.3 Python/2.7.16", "date": "Fri, 29 May 2020 13:54:39 GMT", "content-type": "application/json"} })

Body:
This works.

use serde_json::{Value};

#[tokio::main]
async fn main() {

    let url = "http://127.0.0.1:8008";

    let data = r#"
        {
            "id": 1,
            "jsonrpc": "2.0",
            "method": "getNext",
            "params": [ {
                "flavour": "apple"
            } ]
        }"#;

    let v: Value = match serde_json::from_str(data) {
        Err(_) => panic!("this is a terrible mistake!"),
        Ok(y) => y,
    };

    let client = reqwest::Client::new();

    let result = client.post(url)
        .json(&v)
        .send()
        .await;

    if let Ok(response) = result {
        if let Ok(text) = response.text().await {
            let result: serde_json::Result<Value> = serde_json::from_str(&text);
            if let Ok(j) = result {
                println!("received: {}, flavour still: {}", j["received"], j["params"][0]["flavour"]);
            } else {
                panic!("Could not convert text to json");
            }
        } else {
            panic!("Could not extract text from response");
        }
    } else {
        panic!("No response from server");
    }
}

    Finished dev [unoptimized + debuginfo] target(s) in 1.93s
     Running `target/debug/monitor`
received: "ok", flavour still: "apple"


问:
Pass MKMapType variable from UISegmentedControl in FlipsideViewController to MainViewController
I'm having trouble passing the index of a UISegmentedControl from FlipSideViewController.m to my MainViewController.m which holds my MKMapView.  Here is my code:

MainViewController.h

```<code> @interface MainViewController : UIViewController <FlipsideViewControllerDelegate, MKMapViewDelegate> {
   MKMapView *mapView;
 }

 @property (nonatomic, retain) UISegmentedControl *mapType;

 @property (nonatomic, retain) IBOutlet MKMapView *mapView;
 - (IBAction)showInfo:(id)sender;
 @end
</code>```

FlipSideViewController.h

```<code>@protocol FlipsideViewControllerDelegate;

@interface FlipsideViewController : UIViewController {
  id <FlipsideViewControllerDelegate> delegate;
  IBOutlet UISegmentedControl *mapType_;
}

@property (nonatomic, assign) id <FlipsideViewControllerDelegate> delegate;
  - (IBAction)done:(id)sender;

@property (nonatomic, retain) UISegmentedControl *mapType_;
@end

@protocol FlipsideViewControllerDelegate
    - (void)flipsideViewControllerDidFinish:(FlipsideViewController *)controller;
    - (void)flipsideViewControllerSelectionChangedToIndex:(int)index;
@end
</code>```

FlipSideViewController.m

```<code> - (IBAction)changeView {
  [delegate flipsideViewControllerSelectionChangesToIndex: mapType_.selectedSegmentIndex];
  }
</code>```

MainViewController.m

```<code>- (void)flipsideViewControllerSelectionChangedToIndex:(int)index {
  if ( index == 0 )mapView.mapType = MKMapTypeStandard;
  if ( index == 1 )mapView.mapType = MKMapTypeHybrid;
  if ( index == 2 )mapView.mapType = MKMapTypeSatellite;
 }
</code>```

My problem is that in my simulator when I click any of the segments in the UISegmentedControl the application shuts down.  The console shows this error thrown:

```<code>-[FlipsideViewController changeView:]: unrecognized
 selector sent to instance 0x6d43f30
2011-06-01 19:50:31.055 GermTracker2[893:207] ***
Terminating app due to uncaught exception 'NSInvalidArgumentException',
reason: '-[FlipsideViewController changeView:]:
unrecognized selector sent to instance 0x6d43f30'
</code>```

答: you have a double @end in your FlipSideViewController.h. Try removing @end after - (IBAction)done:(id)sender;

Body:
I'm having trouble passing the index of a UISegmentedControl from FlipSideViewController.m to my MainViewController.m which holds my MKMapView.  Here is my code:

MainViewController.h

 @interface MainViewController : UIViewController <FlipsideViewControllerDelegate, MKMapViewDelegate> {
   MKMapView *mapView;
 }

 @property (nonatomic, retain) UISegmentedControl *mapType;

 @property (nonatomic, retain) IBOutlet MKMapView *mapView;
 - (IBAction)showInfo:(id)sender;
 @end

FlipSideViewController.h

@protocol FlipsideViewControllerDelegate;

@interface FlipsideViewController : UIViewController {
  id <FlipsideViewControllerDelegate> delegate;
  IBOutlet UISegmentedControl *mapType_;
}

@property (nonatomic, assign) id <FlipsideViewControllerDelegate> delegate;
  - (IBAction)done:(id)sender;

@property (nonatomic, retain) UISegmentedControl *mapType_;
@end

@protocol FlipsideViewControllerDelegate
    - (void)flipsideViewControllerDidFinish:(FlipsideViewController *)controller;
    - (void)flipsideViewControllerSelectionChangedToIndex:(int)index;
@end

FlipSideViewController.m

 - (IBAction)changeView {
  [delegate flipsideViewControllerSelectionChangesToIndex: mapType_.selectedSegmentIndex];
  }

MainViewController.m

- (void)flipsideViewControllerSelectionChangedToIndex:(int)index {
  if ( index == 0 )mapView.mapType = MKMapTypeStandard;
  if ( index == 1 )mapView.mapType = MKMapTypeHybrid;
  if ( index == 2 )mapView.mapType = MKMapTypeSatellite;
 }

My problem is that in my simulator when I click any of the segments in the UISegmentedControl the application shuts down.  The console shows this error thrown:

-[FlipsideViewController changeView:]: unrecognized
 selector sent to instance 0x6d43f30
2011-06-01 19:50:31.055 GermTracker2[893:207] ***
Terminating app due to uncaught exception 'NSInvalidArgumentException',
reason: '-[FlipsideViewController changeView:]:
unrecognized selector sent to instance 0x6d43f30'


问:
Refresh MySQL views?
At work I'm constantly being told that when changes are made to a MySQL db that the views need to be 'refreshed'. The accepted manual solution seems to be going into Workbench, right clicking, and hitting 'Refresh All'

Is this just meant to be clearing the cache? Or does that rebuild the views from scratch, or is this totally bogus? They seem to be able to tell when views have not been 'refreshed', and I'm not sure they understand it any more than, "Because things need to be refreshed when they are changed."

If it is just clearing the cache, would 'FLUSH TABLES WITH READ LOCK' be enough?

答: I was working on a project and my view didn't show new data because I was trying to make a join on a NULL field to another table. So I just updated the new data with a correct value for that field and it worked.

ContentLicense: CC BY-SA 2.5

Title: Refresh MySQL views?


问:
Solve a linear equation system with non fixed coefficients
I'm searching for a library that can give a closed form solution to a quadratic linear equation system where the coefficients are not fixed. I.E.:

```<code>x1 +  1*x2 = a1
x1 - a2*x1 = 0
</code>```

Where x1 and x2 are the variables and a1 and a2 are the coefficients. The solution to this system would be

```<code>x1 = (a1*a2)/(1+a2)
x2 = a1/(1+a2)
</code>```

Is there a library in any language that can solve this automatically? All the libraries I found only worked for fixed coefficients.
Any tips are appreciated.

答:
You don't need symbolic algebra. Use the determinant formulas for the solutions, one determinant in the numerator, another in the denominator. Each of them is a quadratic form in the a's and b's. This works just as well for n linear equations in n unknowns, but then each of them is an n-form.

I understand that you only need the solution for n=2. Here it is:

```<code>x1 = (b1 a22 - b2 a12) / (a11 a22 - a21 a12)
x2 = (a11 b2 - a21 b1) / (a11 a22 - a21 a12)
</code>```

Body:
I'm searching for a library that can give a closed form solution to a quadratic linear equation system where the coefficients are not fixed. I.E.:

x1 +  1*x2 = a1
x1 - a2*x1 = 0

Where x1 and x2 are the variables and a1 and a2 are the coefficients. The solution to this system would be

x1 = (a1*a2)/(1+a2)
x2 = a1/(1+a2)

Is there a library in any language that can solve this automatically? All the libraries I found only worked for fixed coefficients.
Any tips are appreciated.

Body:
You don't need symbolic algebra. Use the determinant formulas for the solutions, one determinant in the numerator, another in the denominator. Each of them is a quadratic form in the a's and b's. This works just as well for n linear equations in n unknowns, but then each of them is an n-form.

I understand that you only need the solution for n=2. Here it is:

x1 = (b1 a22 - b2 a12) / (a11 a22 - a21 a12)
x2 = (a11 b2 - a21 b1) / (a11 a22 - a21 a12)


问:
How do I remove duplicates in mongodb based on multiple fields?
Here's an example of my docs:

```<code>[{name:"duplicate", value:true, id:2910921},{name:"duplicate", value:true, id:32838293},{name:"duplicate", value:false, id:3283232},{name:"notDuplicate", value:true, id:382932}]
</code>```

I want to remove if there are multiple documents that contain the same name and the same value. In the example above it would remove one document, either {name:"duplicate", value:true, id:2910921} or {name:"duplicate", value:true, id:32838293}, it does't matter to me which one.

So far, I've considered just creating a new field for each of these which would be something like newField: "duplicatetrue" and then I could just use distinct on these to remove dupes, but I am having trouble figuring out how to concat two different fields with different types into a new field. I'm definitely open to better suggestions as well. Here's what I have so far:

```<code>db.collection(collectionName).updateMany({}, {$set: {"newField": ["$name","$value"]  }})
</code>```

However, the above line doesn't output the values, rather it outputs exactly newField: ["$name","$value"]

Removing the quotes from $name and $value does not work either.

I'm using the Node mongodb driver: 3.5.8

答:
i'm not sure about mongo but using the node you can remove duplicates. i tried this method for one of requirement that worked fine. please try this by chagning as you need var names.   </p>

```<code>function arrUnique(arr) {
    var cleaned = [];
    arr.forEach(function(itm) {
        var unique = true;
        cleaned.forEach(function(itm2) {
            if (_.isEqual(itm, itm2)) unique = false;
        });
        if (unique)  cleaned.push(itm);
    });
    return cleaned;
}

var newField = arrUnique(newField);
</code>```

Body:
Here's an example of my docs:

[{name:"duplicate", value:true, id:2910921},{name:"duplicate", value:true, id:32838293},{name:"duplicate", value:false, id:3283232},{name:"notDuplicate", value:true, id:382932}]

I want to remove if there are multiple documents that contain the same name and the same value. In the example above it would remove one document, either {name:"duplicate", value:true, id:2910921} or {name:"duplicate", value:true, id:32838293}, it does't matter to me which one.

So far, I've considered just creating a new field for each of these which would be something like newField: "duplicatetrue" and then I could just use distinct on these to remove dupes, but I am having trouble figuring out how to concat two different fields with different types into a new field. I'm definitely open to better suggestions as well. Here's what I have so far:

db.collection(collectionName).updateMany({}, {$set: {"newField": ["$name","$value"]  }})

However, the above line doesn't output the values, rather it outputs exactly newField: ["$name","$value"]

Removing the quotes from $name and $value does not work either.

I'm using the Node mongodb driver: 3.5.8

Body:
i'm not sure about mongo but using the node you can remove duplicates. i tried this method for one of requirement that worked fine. please try this by chagning as you need var names.

function arrUnique(arr) {
    var cleaned = [];
    arr.forEach(function(itm) {
        var unique = true;
        cleaned.forEach(function(itm2) {
            if (_.isEqual(itm, itm2)) unique = false;
        });
        if (unique)  cleaned.push(itm);
    });
    return cleaned;
}

var newField = arrUnique(newField);


问:
Python checking condition and safety in same if statement
I've been writing some python code where i have a condition if object is None return if its not get object property and check it.Here is an example code:

```<code>b = None
if b is None or b.property == "something":
    print("yay")
</code>```

when runned code above prints AttributeError: 'NoneType' object has no attribute 'property' code below prints the same result.

```<code>b = None
if b.property == "something" or b is None:
    print("yay")
</code>```

Question:
   Asume there is many or checks in if clause after evaluating one of them true other condisions can be bypassed becouse the result is gonna be true so its waste of #compute power.
Same can be applied to and operator too if any condition is 0 result is zero.So why the example above fires an exception?

答:
I was writing my question than it hit me.Anyway i will leave solution to my problem aswell.

```<code>if (b is None) or b.property == "something":
    print("yay")
</code>```

There is a natural order to this world! =)

Body:
I've been writing some python code where i have a condition if object is None return if its not get object property and check it.Here is an example code:

b = None
if b is None or b.property == "something":
    print("yay")

when runned code above prints AttributeError: 'NoneType' object has no attribute 'property' code below prints the same result.

b = None
if b.property == "something" or b is None:
    print("yay")

Question:
   Asume there is many or checks in if clause after evaluating one of them true other condisions can be bypassed becouse the result is gonna be true so its waste of #compute power.
Same can be applied to and operator too if any condition is 0 result is zero.So why the example above fires an exception?

Body:
I was writing my question than it hit me.Anyway i will leave solution to my problem aswell.

if (b is None) or b.property == "something":
    print("yay")

There is a natural order to this world! =)


问:
How to override JColorChooser Reset Button?
In a previous post, I discussed overriding the Preview panel and all is working well, except a new question has come forth and hoping the fix for it is as easy.

This new question pertains to the Reset button. Based on the documentation, the Reset button will reset the Colors back to the original color that was passed in. This works great, but what if there is a secondary field?

Here is the code...

```<code>import java.awt.BorderLayout;
import java.awt.Color;
import java.awt.Component;
import java.awt.Dimension;
import java.awt.HeadlessException;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;

import javax.swing.JButton;
import javax.swing.JColorChooser;
import javax.swing.JDialog;
import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JPanel;
import javax.swing.JTextField;
import javax.swing.SwingUtilities;
import javax.swing.event.ChangeEvent;
import javax.swing.event.ChangeListener;

public class ColorChooserSample implements Runnable{

    public static void main(String args[]) {
        SwingUtilities.invokeLater(
                new ColorChooserSample());
    }

    private JPanel panel;
    JTextField counter;
    int vCounter = 1;

    @Override
    public void run() {
        JFrame frame = new JFrame(
                "JColorChooser Sample");
        frame.setDefaultCloseOperation(
                JFrame.EXIT_ON_CLOSE);

        panel = new JPanel();
        panel.setPreferredSize(new Dimension(300, 200));
        JButton button = new JButton(
                "Pick to Change JPanel Background");
        button.addActionListener(new ColorListener());
        panel.add(button);

        frame.add(panel, BorderLayout.CENTER);
        frame.pack();
        frame.setLocationByPlatform(true);
        frame.setVisible(true);
    }

    public void setJPanelBackground(Color color) {
        panel.setBackground(color);
        panel.repaint();
    }

    public class ColorListener implements
            ActionListener, ChangeListener {

        private JColorChooser chooser;

        private JPanel previewPanel;

        @Override
        public void actionPerformed(
                ActionEvent actionEvent) {
            Color backgroundColor = showDialog(panel,
                    "Set JPanel Background",
                    panel.getBackground());
            setJPanelBackground(backgroundColor);
        }

        private Color showDialog(Component component,
                String title, Color initialColor)
                        throws HeadlessException {
            chooser = new JColorChooser(initialColor);
            chooser.getSelectionModel()
                .addChangeListener(this);

            // configuring color chooser panel
            previewPanel = new JPanel();
            previewPanel.setBackground(initialColor);
            JLabel label = new JLabel("Hello World!");
            counter = new JTextField("0");
            previewPanel.add(label, BorderLayout.WEST);
            previewPanel.add(counter, BorderLayout.EAST);
            chooser.setPreviewPanel(previewPanel);

            // creating dialog
            ColorTracker ok = new ColorTracker(chooser);
            JDialog dialog = JColorChooser.createDialog(
                    component, title, true, chooser,
                    ok, null);
            dialog.setVisible(true);
            return ok.getColor();
        }

        @Override
        public void stateChanged(ChangeEvent event) {
            Color newColor = chooser.getColor();
            previewPanel.setBackground(newColor);
            counter.setText(Integer.toString(vCounter++));
        }
    }

    private class ColorTracker implements ActionListener {

        private Color color;

        private JColorChooser chooser;

        public ColorTracker(JColorChooser chooser) {
            this.chooser = chooser;
        }

        @Override
        public void actionPerformed(ActionEvent event) {
            color = chooser.getColor();
        }

        public Color getColor() {
            return color;
        }

    }

}
</code>```

The above code will first open this dialog.

Once the "Pick to Change JPanelBackground" button is clicked, it opens the JColorDialog with a modified Preview section. This section includes "Hello World" that will change background colors and a counter to count how many times the color has changed.

The following shows the last color selected but also shows 5 in the Textbox to signify that there were 5 color options clicked.

Click the reset button and the color background is set to the original, but the count was NOT reset back to 0.

You can pass in a Listener for both the OK and Cancel button but not the Reset button.

This is only an example, as there could be other items in the Preview section. The objective is how can the values other than Color be reset?

答:
The only thing done by reset button is calling chooserPane.setColor(initialColor). So if what you really need is listening to color changes, you might do just that, by something like:

```<code>        chooser.getSelectionModel().addChangeListener(new ChangeListener() {
            @Override
            public void stateChanged(ChangeEvent e) {
                Color newColor = chooser.getColor();
                previewPanel.setBackground(newColor);
                counter.setText(Integer.toString(vCounter++));
            }
        });
</code>```

And your secondary panel gets updated.

Having said that, if you really want to add a listener to the button, there's no exposed way of accessing it - you need to iterate the components all the way down until you find the button, and then add a listener to it.

EDIT
Alright, so it has to be the reset button. Since that's a local variable of some content creating method, it has to be removed surgically. Just change new JButton("abc") to your own button, with your own listener(s).

```<code>        Locale locale = dialog.getLocale();
        String resetString = UIManager.getString("ColorChooser.resetText", locale);

        Container contentPane = dialog.getContentPane();
        JPanel buttonPanel = null;
        for (Component c : contentPane.getComponents()) {
            if (c instanceof JPanel) {
                buttonPanel = (JPanel) c;
            }
        }

        JButton resetButton = null;
        if (buttonPanel != null) {
            for (Component b : buttonPanel.getComponents()) {
                if (b instanceof JButton) {
                    JButton button = (JButton) b;
                    if (resetString.equals(button.getText())) {
                        resetButton = button;
                        break;
                    }
                }
            }
            if (resetButton != null) {
                buttonPanel.remove(resetButton);
                buttonPanel.add(new JButton("abc"));
            }
        }
</code>```

Body:
In a previous post, I discussed overriding the Preview panel and all is working well, except a new question has come forth and hoping the fix for it is as easy.

This new question pertains to the Reset button. Based on the documentation, the Reset button will reset the Colors back to the original color that was passed in. This works great, but what if there is a secondary field?

Here is the code...

import java.awt.BorderLayout;
import java.awt.Color;
import java.awt.Component;
import java.awt.Dimension;
import java.awt.HeadlessException;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;

import javax.swing.JButton;
import javax.swing.JColorChooser;
import javax.swing.JDialog;
import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JPanel;
import javax.swing.JTextField;
import javax.swing.SwingUtilities;
import javax.swing.event.ChangeEvent;
import javax.swing.event.ChangeListener;

public class ColorChooserSample implements Runnable{

    public static void main(String args[]) {
        SwingUtilities.invokeLater(
                new ColorChooserSample());
    }

    private JPanel panel;
    JTextField counter;
    int vCounter = 1;

    @Override
    public void run() {
        JFrame frame = new JFrame(
                "JColorChooser Sample");
        frame.setDefaultCloseOperation(
                JFrame.EXIT_ON_CLOSE);

        panel = new JPanel();
        panel.setPreferredSize(new Dimension(300, 200));
        JButton button = new JButton(
                "Pick to Change JPanel Background");
        button.addActionListener(new ColorListener());
        panel.add(button);

        frame.add(panel, BorderLayout.CENTER);
        frame.pack();
        frame.setLocationByPlatform(true);
        frame.setVisible(true);
    }

    public void setJPanelBackground(Color color) {
        panel.setBackground(color);
        panel.repaint();
    }

    public class ColorListener implements
            ActionListener, ChangeListener {

        private JColorChooser chooser;

        private JPanel previewPanel;

        @Override
        public void actionPerformed(
                ActionEvent actionEvent) {
            Color backgroundColor = showDialog(panel,
                    "Set JPanel Background",
                    panel.getBackground());
            setJPanelBackground(backgroundColor);
        }

        private Color showDialog(Component component,
                String title, Color initialColor)
                        throws HeadlessException {
            chooser = new JColorChooser(initialColor);
            chooser.getSelectionModel()
                .addChangeListener(this);

            // configuring color chooser panel
            previewPanel = new JPanel();
            previewPanel.setBackground(initialColor);
            JLabel label = new JLabel("Hello World!");
            counter = new JTextField("0");
            previewPanel.add(label, BorderLayout.WEST);
            previewPanel.add(counter, BorderLayout.EAST);
            chooser.setPreviewPanel(previewPanel);

            // creating dialog
            ColorTracker ok = new ColorTracker(chooser);
            JDialog dialog = JColorChooser.createDialog(
                    component, title, true, chooser,
                    ok, null);
            dialog.setVisible(true);
            return ok.getColor();
        }

        @Override
        public void stateChanged(ChangeEvent event) {
            Color newColor = chooser.getColor();
            previewPanel.setBackground(newColor);
            counter.setText(Integer.toString(vCounter++));
        }
    }

    private class ColorTracker implements ActionListener {

        private Color color;

        private JColorChooser chooser;

        public ColorTracker(JColorChooser chooser) {
            this.chooser = chooser;
        }

        @Override
        public void actionPerformed(ActionEvent event) {
            color = chooser.getColor();
        }

        public Color getColor() {
            return color;
        }

    }

}

The above code will first open this dialog.

Once the "Pick to Change JPanelBackground" button is clicked, it opens the JColorDialog with a modified Preview section. This section includes "Hello World" that will change background colors and a counter to count how many times the color has changed.

The following shows the last color selected but also shows 5 in the Textbox to signify that there were 5 color options clicked.

Click the reset button and the color background is set to the original, but the count was NOT reset back to 0.

You can pass in a Listener for both the OK and Cancel button but not the Reset button.

This is only an example, as there could be other items in the Preview section. The objective is how can the values other than Color be reset?

Body:
The only thing done by reset button is calling chooserPane.setColor(initialColor). So if what you really need is listening to color changes, you might do just that, by something like:

        chooser.getSelectionModel().addChangeListener(new ChangeListener() {
            @Override
            public void stateChanged(ChangeEvent e) {
                Color newColor = chooser.getColor();
                previewPanel.setBackground(newColor);
                counter.setText(Integer.toString(vCounter++));
            }
        });

And your secondary panel gets updated.

Having said that, if you really want to add a listener to the button, there's no exposed way of accessing it - you need to iterate the components all the way down until you find the button, and then add a listener to it.

EDIT
Alright, so it has to be the reset button. Since that's a local variable of some content creating method, it has to be removed surgically. Just change new JButton("abc") to your own button, with your own listener(s).

        Locale locale = dialog.getLocale();
        String resetString = UIManager.getString("ColorChooser.resetText", locale);

        Container contentPane = dialog.getContentPane();
        JPanel buttonPanel = null;
        for (Component c : contentPane.getComponents()) {
            if (c instanceof JPanel) {
                buttonPanel = (JPanel) c;
            }
        }

        JButton resetButton = null;
        if (buttonPanel != null) {
            for (Component b : buttonPanel.getComponents()) {
                if (b instanceof JButton) {
                    JButton button = (JButton) b;
                    if (resetString.equals(button.getText())) {
                        resetButton = button;
                        break;
                    }
                }
            }
            if (resetButton != null) {
                buttonPanel.remove(resetButton);
                buttonPanel.add(new JButton("abc"));
            }
        }


问:
File with CSS and JS files not serving on server and prints ��U�v�8��+
When I run my hapijs app locally it is working but when I deploy it in the server it doesn't work. The page can serve html files but if the HTML file has css or bootsrap then it prints weird
symbols like ��U�v�8��+

The link that shows the weird response : https://us-central1-fir-app-85853.cloudfunctions.net/v1/teacher/list

Here is my index.js file.

How can I fix this issue?

```<code>'use strict';
const functions = require('firebase-functions');
const api = require('./server');

exports.v1 = functions.https.onRequest(async (event, resp) => {
  let server = await api.startServer();
  const options = {
    method: event.httpMethod,
    headers: event.headers,
    url: event.path,
    payload: event.body
  };

  return server
    .inject(options)
    .then(response => {
        delete response.headers['content-encoding']
        delete response.headers['transfer-encoding']
        response.headers['x-powered-by'] = 'hapijs'
        resp.set(response.headers);
        return resp.status(response.statusCode).send(response.result);
    })
    .catch(error => resp.status(500).send(error.message || "unknown error"));
});
</code>```

Here is my full source code on github https://github.com/kartikgreen/hapijs-firebase

答:
This is very likely a character encoding issue.

There are a few causes, one is a different character encoding being outputted than UTF-8.

so say your document outputs in ISO-8859-1 (known as Latin-1), if you do not set the character set the browser will default to UTF-8. Think of it like outputting chinese and then asking a language parser to treat it as japanese, it will never understand!

Not sure how to do it in hapjs but in PHP you would just add a header as follows:-

header("Content-Type: text/html; charset=ISO-8859-1");

The other possible cause is a Byte Order Mark (BOM).

If your script is accidentally outputting some blank data before it should this can add these invisible characters.

There are loads of answers on BOMs on Stack Overflow so I won't repeat those.

I would normally say run the page through the W3C Internationalisation Checker but your page seems to kill it (which could be down to firebase). Perhaps you could capture the raw output to a file and run it through that way?

Body:
When I run my hapijs app locally it is working but when I deploy it in the server it doesn't work. The page can serve html files but if the HTML file has css or bootsrap then it prints weird
symbols like ��U�v�8��+

The link that shows the weird response : https://us-central1-fir-app-85853.cloudfunctions.net/v1/teacher/list

Here is my index.js file.

How can I fix this issue?

'use strict';
const functions = require('firebase-functions');
const api = require('./server');

exports.v1 = functions.https.onRequest(async (event, resp) => {
  let server = await api.startServer();
  const options = {
    method: event.httpMethod,
    headers: event.headers,
    url: event.path,
    payload: event.body
  };

  return server
    .inject(options)
    .then(response => {
        delete response.headers['content-encoding']
        delete response.headers['transfer-encoding']
        response.headers['x-powered-by'] = 'hapijs'
        resp.set(response.headers);
        return resp.status(response.statusCode).send(response.result);
    })
    .catch(error => resp.status(500).send(error.message || "unknown error"));
});

Here is my full source code on github https://github.com/kartikgreen/hapijs-firebase