问:
Getting Invalid Key Store format error when running MUnit Tests in Jenkins
I am enabling SSL in my mulesoft application.
I have added a keyStore in my application in http request connector.
Everything works fine. Deployed to cloudhub by skipping Munits and tested. Everything works fine. Munit are running fine in local machine.
But when I run MUnit tests in jenkins. It throws below error.

Caused by:
org.mule.runtime.api.lifecycle.InitialisationException:Unable to
initialise TLS configuration

Caused by:org.mule.runtime.api.lifecycle.CreateException: Failed to
load KeyStore:
/mnt/jenkins/workspace/mule-jobs/application-prc-nonprod/target/munitworkingdir-5223661232046316/container/apps/application-prc/server-keystore-nonprod.jks

Caused by: java.io.IOException: Invalid keystore format

I have enabled this TLS for mulesoft system API and I am mocking this request connector. Not sure why it wants this jks file from jenkins location as shown in the error.

Any lead would be really helpful.

Http connection configuration

```<code><http:request-config name="Mule_AWS_Request_configuration" doc:name="HTTP Request configuration" doc:id="067166d7-313b-432a-b653-88617b728c5a" responseTimeout="${muleApi.aws.response.timeout}">
        <http:request-connection protocol="HTTPS" host="${muleApi.aws.host} " connectionIdleTimeout="${muleApi.aws.idletTimeout}">
            <reconnection >
                <reconnect frequency="${muleApi.aws.retryFrequency}" count="${muleApi.aws.reconnectionAttempts}" />
            </reconnection>
            <tls:context>
                <tls:trust-store insecure="true"/>
                <tls:key-store type="jks" path="${tls.keyStore.path}" keyPassword="${secure::tls.keyStore.keyPassword}" password="${secure::tls.keyStore.password}" />
            </tls:context>
        </http:request-connection>
    </http:request-config>
</code>```

Http request

```<code><http:request method="POST" doc:name="POST ZIP / aws system api call" doc:id="965033c6-4623-45cd-9326-920534684ace" config-ref="Mule_AWS_Request_configuration" path="${muleApi.aws.path}" responseTimeout="600000" requestStreamingMode="ALWAYS">
            <non-repeatable-stream />
                <http:headers><![CDATA[#[output application/java
---
{
    "client_secret" : p('secure::muleApi.aws.client_secret'),
    "Content-Type" : "application/zip",
    "client_id" : p('secure::muleApi.aws.client_id')
}]]]></http:headers>
            <http:query-params><![CDATA[#[output application/java
---
{
    "bucketName" : p('muleApi.aws.params.bucketName'),
    "key" : vars.key,
    "extractZipContent": vars.extractZipContent default "true",
    "extractContentType": vars.extractContentType default "csv"
}]]]></http:query-params>
            <http:response-validator>
                <http:success-status-code-validator values="200..499" />
            </http:response-validator>
        </http:request>
</code>```

Location of keyStore.

path: src/main/resource/server-keystore-nonprod.jks

I am just mocking the request connector as shown below.

[![enter image description here][3]][3]

Mule Version is 4.4.0

MUnit version is 2.3.5

Providing the logs till i see this error below.

```<code>INFO  2022-12-07 09:27:52,417 [munit.01] org.mule.munit.runner.remote.api.server.RunnerServer: Waiting for client connection
INFO  2022-12-07 09:27:52,418 [munit.01] org.mule.munit.runner.remote.api.server.RunnerServer: Client connection received from localhost - true
WARN  2022-12-07 09:27:52,859 [munit.01] org.mule.runtime.config.internal.dsl.spring.ComponentConfigurationBuilder: Parameter ignore from extension munit has a defaultValue configured in the componentBuildingDefinition but not in the extensionModel.
INFO  2022-12-07 09:27:53,029 [munit.01] org.mule.runtime.api.message.AbstractMuleMessageBuilderFactory: Loaded MuleMessageBuilderFactory implementation 'org.mule.runtime.core.internal.message.DefaultMessageBuilderFactory' from classloader 'java.net.URLClassLoader@3bbf841e'
WARN  2022-12-07 09:27:53,056 [munit.01] org.mule.runtime.core.internal.security.tls.TlsProperties: File tls-default.conf not found, using default configuration.
INFO  2022-12-07 09:27:53,057 [munit.01] org.mule.runtime.api.tls.AbstractTlsContextFactoryBuilderFactory: Loaded TlsContextFactoryBuilderFactory implementation 'org.mule.runtime.module.tls.api.DefaultTlsContextFactoryBuilderFactory' from classloader 'java.net.URLClassLoader@3bbf841e'
WARN  2022-12-07 09:27:53,097 [munit.01] org.mule.runtime.module.tls.internal.DefaultTlsContextFactory: TLS context  trust store set as insecure. No certificate validations will be performed, rendering connections vulnerable to attacks. Use at own risk.
INFO  2022-12-07 09:27:57,038 [munit.01] org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager: Initialising Bean: org.mule.runtime.module.extension.internal.runtime.config.ConfigurationProviderToolingAdapter-HTTP-requestConfig-implicit
INFO  2022-12-07 09:27:57,054 [munit.01] org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager: Starting Bean: org.mule.runtime.module.extension.internal.runtime.config.ConfigurationProviderToolingAdapter-HTTP-requestConfig-implicit
INFO  2022-12-07 09:27:58,454 [munit.01] org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager: Initialising Bean: org.mule.runtime.module.extension.internal.runtime.config.DynamicConfigurationProvider-JSON_Logger_Config
INFO  2022-12-07 09:27:58,458 [munit.01] org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager: Initialising Bean: org.mule.runtime.module.extension.internal.runtime.config.ConfigurationProviderToolingAdapter-Mule_AWS_Request_configuration
org.mule.runtime.api.exception.MuleRuntimeException: org.mule.runtime.api.lifecycle.InitialisationException: Unable to initialise TLS configuration
Caused by: org.mule.runtime.api.lifecycle.InitialisationException: Unable to initialise TLS configuration
Caused by: org.mule.runtime.api.lifecycle.CreateException: Failed to load KeyStore: /mnt/jenkins/workspace/mule-jobs-nonprod/biz-prc-marketing-bulk-api-nonprod/target/munitworkingdir-5223661232046316/container/apps/biz-prc-marketing-bulk-api/server-keystore-nonprod.jks
Caused by: java.io.IOException: Invalid keystore format
</code>```

Body:
I am enabling SSL in my mulesoft application.
I have added a keyStore in my application in http request connector.
Everything works fine. Deployed to cloudhub by skipping Munits and tested. Everything works fine. Munit are running fine in local machine.
But when I run MUnit tests in jenkins. It throws below error.

Caused by:
org.mule.runtime.api.lifecycle.InitialisationException:Unable to
initialise TLS configuration

Caused by:org.mule.runtime.api.lifecycle.CreateException: Failed to
load KeyStore:
/mnt/jenkins/workspace/mule-jobs/application-prc-nonprod/target/munitworkingdir-5223661232046316/container/apps/application-prc/server-keystore-nonprod.jks

Caused by: java.io.IOException: Invalid keystore format

I have enabled this TLS for mulesoft system API and I am mocking this request connector. Not sure why it wants this jks file from jenkins location as shown in the error.

Any lead would be really helpful.

Http connection configuration

<http:request-config name="Mule_AWS_Request_configuration" doc:name="HTTP Request configuration" doc:id="067166d7-313b-432a-b653-88617b728c5a" responseTimeout="${muleApi.aws.response.timeout}">
        <http:request-connection protocol="HTTPS" host="${muleApi.aws.host} " connectionIdleTimeout="${muleApi.aws.idletTimeout}">
            <reconnection >
                <reconnect frequency="${muleApi.aws.retryFrequency}" count="${muleApi.aws.reconnectionAttempts}" />
            </reconnection>
            <tls:context>
                <tls:trust-store insecure="true"/>
                <tls:key-store type="jks" path="${tls.keyStore.path}" keyPassword="${secure::tls.keyStore.keyPassword}" password="${secure::tls.keyStore.password}" />
            </tls:context>
        </http:request-connection>
    </http:request-config>

Http request

<http:request method="POST" doc:name="POST ZIP / aws system api call" doc:id="965033c6-4623-45cd-9326-920534684ace" config-ref="Mule_AWS_Request_configuration" path="${muleApi.aws.path}" responseTimeout="600000" requestStreamingMode="ALWAYS">
            <non-repeatable-stream />
                <http:headers><![CDATA[#[output application/java
---
{
    "client_secret" : p('secure::muleApi.aws.client_secret'),
    "Content-Type" : "application/zip",
    "client_id" : p('secure::muleApi.aws.client_id')
}]]]></http:headers>
            <http:query-params><![CDATA[#[output application/java
---
{
    "bucketName" : p('muleApi.aws.params.bucketName'),
    "key" : vars.key,
    "extractZipContent": vars.extractZipContent default "true",
    "extractContentType": vars.extractContentType default "csv"
}]]]></http:query-params>
            <http:response-validator>
                <http:success-status-code-validator values="200..499" />
            </http:response-validator>
        </http:request>

Location of keyStore.

path: src/main/resource/server-keystore-nonprod.jks

I am just mocking the request connector as shown below.

[![enter image description here][3]][3]

Mule Version is 4.4.0

MUnit version is 2.3.5

Providing the logs till i see this error below.

INFO  2022-12-07 09:27:52,417 [munit.01] org.mule.munit.runner.remote.api.server.RunnerServer: Waiting for client connection
INFO  2022-12-07 09:27:52,418 [munit.01] org.mule.munit.runner.remote.api.server.RunnerServer: Client connection received from localhost - true
WARN  2022-12-07 09:27:52,859 [munit.01] org.mule.runtime.config.internal.dsl.spring.ComponentConfigurationBuilder: Parameter ignore from extension munit has a defaultValue configured in the componentBuildingDefinition but not in the extensionModel.
INFO  2022-12-07 09:27:53,029 [munit.01] org.mule.runtime.api.message.AbstractMuleMessageBuilderFactory: Loaded MuleMessageBuilderFactory implementation 'org.mule.runtime.core.internal.message.DefaultMessageBuilderFactory' from classloader 'java.net.URLClassLoader@3bbf841e'
WARN  2022-12-07 09:27:53,056 [munit.01] org.mule.runtime.core.internal.security.tls.TlsProperties: File tls-default.conf not found, using default configuration.
INFO  2022-12-07 09:27:53,057 [munit.01] org.mule.runtime.api.tls.AbstractTlsContextFactoryBuilderFactory: Loaded TlsContextFactoryBuilderFactory implementation 'org.mule.runtime.module.tls.api.DefaultTlsContextFactoryBuilderFactory' from classloader 'java.net.URLClassLoader@3bbf841e'
WARN  2022-12-07 09:27:53,097 [munit.01] org.mule.runtime.module.tls.internal.DefaultTlsContextFactory: TLS context  trust store set as insecure. No certificate validations will be performed, rendering connections vulnerable to attacks. Use at own risk.
INFO  2022-12-07 09:27:57,038 [munit.01] org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager: Initialising Bean: org.mule.runtime.module.extension.internal.runtime.config.ConfigurationProviderToolingAdapter-HTTP-requestConfig-implicit
INFO  2022-12-07 09:27:57,054 [munit.01] org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager: Starting Bean: org.mule.runtime.module.extension.internal.runtime.config.ConfigurationProviderToolingAdapter-HTTP-requestConfig-implicit
INFO  2022-12-07 09:27:58,454 [munit.01] org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager: Initialising Bean: org.mule.runtime.module.extension.internal.runtime.config.DynamicConfigurationProvider-JSON_Logger_Config
INFO  2022-12-07 09:27:58,458 [munit.01] org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager: Initialising Bean: org.mule.runtime.module.extension.internal.runtime.config.ConfigurationProviderToolingAdapter-Mule_AWS_Request_configuration
org.mule.runtime.api.exception.MuleRuntimeException: org.mule.runtime.api.lifecycle.InitialisationException: Unable to initialise TLS configuration
Caused by: org.mule.runtime.api.lifecycle.InitialisationException: Unable to initialise TLS configuration
Caused by: org.mule.runtime.api.lifecycle.CreateException: Failed to load KeyStore: /mnt/jenkins/workspace/mule-jobs-nonprod/biz-prc-marketing-bulk-api-nonprod/target/munitworkingdir-5223661232046316/container/apps/biz-prc-marketing-bulk-api/server-keystore-nonprod.jks
Caused by: java.io.IOException: Invalid keystore format


来源: stackexchange

问:
AbstractAPI GeoLocation IP | codeigniter Call to a member function getIPAddress() on null
I try integrate  geo location ip this library Abstractapi  with codeigniter 3:

```<code>https://github.com/abstractapi/php-ip-geolocation
</code>```

I install this via composer, then I copy  files from root vendor to application/libraries/abstractipgeolocation/vendor/

Then I create controller and call:

```<code>require_once(APPPATH.'libraries/abstractipgeolocation/vendor/autoload.php');

use Abstractapi\IpGeolocation\AbstractIpGeolocation;
use Abstractapi\IpGeolocation\Common\Exception\AbstractHttpErrorException;
</code>```

now in controller I add function and load to view coupons:

```<code>public function coupons()
    {
        $this->check_vendor_permission();
        $this->load->model('coupon_model');
        $data['title'] = trans("coupons");
        $data['description'] = trans("coupons") . " - " . $this->app_name;
        $data['keywords'] = trans("coupons") . "," . $this->app_name;

        $data['num_rows'] = $this->coupon_model->get_coupons_count($this->auth_user->id);
        $pagination = $this->paginate(generate_dash_url("coupons"), $data['num_rows'], $this->per_page);
        $data['coupons'] = $this->coupon_model->get_coupons_paginated($this->auth_user->id, $pagination['per_page'], $pagination['offset']);

        try {
            // build required data
            $formData = http_build_query([
                'api_key' => 'xxxxxxxxxxxxxxxxxx',
                'ip_address' => $this->request->getIPAddress()
            ]);
            // make curl request to abstractapi.com to get ip information
            $curl = curl_init();
            curl_setopt_array($curl, array(
                CURLOPT_URL => "https://ipgeolocation.abstractapi.com/v1/?" . $formData,
                CURLOPT_RETURNTRANSFER => true,
                CURLOPT_ENCODING => "",
                CURLOPT_MAXREDIRS => 10,
                CURLOPT_TIMEOUT => 30,
                CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
                CURLOPT_CUSTOMREQUEST => "GET",
                CURLOPT_HTTPHEADER => array(
                    "accept: application/json",
                    "cache-control: no-cache",
                    "content-type: application/json",
                ),
            ));
            $response = curl_exec($curl);
            $err = curl_error($curl);
            curl_close($curl);
            if ($err) {
                $data = [
                    'status' => false,
                    'message' => $err
                ];
            } else {
                $response = json_decode($response, true);
                $data = [
                    'status' => true,
                    'ipv4' => $response['ip_address'] ?? '',
                    'ipv6' => $this->request->getIPAddress() ?? '',
                    'isp_name' => $response['connection']['isp_name'] ?? 'Unknown',
                    'isp_city' => $response['city'] ?? 'Unknown',
                    'isp_region' => $response['region'] ?? 'Unknown',
                    'isp_country' => $response['country'] ?? 'Unknown',
                    'isp_lat' => $response['latitude'] ?? 0,
                    'isp_lng' => $response['longitude'] ?? 0,
                ];
            }
            return view('welcome_message', $data);
        } catch (\Exception $e) {
            exit($e->getMessage());
        }

        $this->load->view('dashboard/includes/_header', $data);
        $this->load->view('dashboard/coupon/coupons', $data);
        $this->load->view('dashboard/includes/_footer');
    }
</code>```

Now in view coupons.php I try get country:

```<code><section>
        <div class="container">
            <div class="row info mt-5 p-5 rounded">
                <?=
                $status ?
                '<div class="col-md-8">
                    <p id="p-text">My IP Address is:</p>
                    <p class="ip-address">
                        IPv4:
                        <span class="fa fa-question" data-toggle="tooltip" data-html="true" title="The most common IP version assigned by ISPs."></span>
                        <span id="ipv4">' . $ipv4 . '</span>
                    </p>

                    <p class="ip-address">
                        IPv6:
                        <span class="fa fa-question" data-toggle="tooltip" data-html="true" title="The IP version assigned by routers and other devices that do not support IPv4."></span>
                        <span id="ipv6"> '.$ipv6. '</span>
                    </p>

                    <p class="mt-6">
                    <p id="p-text">My IP Information:</p>
                    <p class="isp-address">
                        ISP:
                        <span id="ipv4"> ' . $isp_name . ' </span> <br><br>

                        City:
                        <span id="ipv4">' . $isp_city . ' </span> <br><br>

                        Region:
                        <span id="ipv4">' . $isp_region . '</span> <br><br>

                        Country:
                        <span id="ipv4">' . $isp_country . ' </span>

                    </p>

                    </p>

                </div>
                <div class="col-md-4">
                    <div style="height: 253px;" id="map">
                    </div>
                </div>'
                :
                '<p> Could not find your IP Address. </p>'
                ?>
            </div>
        </div>
    </section>
</code>```

But after this I get:

```<code>Message: Undefined property: Dashboard_controller::$request

Filename: controllers/Dashboard_controller.php

Type: Error

Message: Call to a member function getIPAddress() on null

Filename: public_html/application/controllers/Dashboard_controller.php

Line Number: 920
</code>```

line 920:

```<code>'ip_address' => $this->request->getIPAddress()
</code>```

Can anyone help me from where this error come and how to resolve?

ContentLicense: CC BY-SA 4.0

Body:
I try integrate  geo location ip this library Abstractapi  with codeigniter 3:

https://github.com/abstractapi/php-ip-geolocation

I install this via composer, then I copy  files from root vendor to application/libraries/abstractipgeolocation/vendor/

Then I create controller and call:

require_once(APPPATH.'libraries/abstractipgeolocation/vendor/autoload.php');

use Abstractapi\IpGeolocation\AbstractIpGeolocation;
use Abstractapi\IpGeolocation\Common\Exception\AbstractHttpErrorException;

now in controller I add function and load to view coupons:

public function coupons()
    {
        $this->check_vendor_permission();
        $this->load->model('coupon_model');
        $data['title'] = trans("coupons");
        $data['description'] = trans("coupons") . " - " . $this->app_name;
        $data['keywords'] = trans("coupons") . "," . $this->app_name;

        $data['num_rows'] = $this->coupon_model->get_coupons_count($this->auth_user->id);
        $pagination = $this->paginate(generate_dash_url("coupons"), $data['num_rows'], $this->per_page);
        $data['coupons'] = $this->coupon_model->get_coupons_paginated($this->auth_user->id, $pagination['per_page'], $pagination['offset']);

        try {
            // build required data
            $formData = http_build_query([
                'api_key' => 'xxxxxxxxxxxxxxxxxx',
                'ip_address' => $this->request->getIPAddress()
            ]);
            // make curl request to abstractapi.com to get ip information
            $curl = curl_init();
            curl_setopt_array($curl, array(
                CURLOPT_URL => "https://ipgeolocation.abstractapi.com/v1/?" . $formData,
                CURLOPT_RETURNTRANSFER => true,
                CURLOPT_ENCODING => "",
                CURLOPT_MAXREDIRS => 10,
                CURLOPT_TIMEOUT => 30,
                CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
                CURLOPT_CUSTOMREQUEST => "GET",
                CURLOPT_HTTPHEADER => array(
                    "accept: application/json",
                    "cache-control: no-cache",
                    "content-type: application/json",
                ),
            ));
            $response = curl_exec($curl);
            $err = curl_error($curl);
            curl_close($curl);
            if ($err) {
                $data = [
                    'status' => false,
                    'message' => $err
                ];
            } else {
                $response = json_decode($response, true);
                $data = [
                    'status' => true,
                    'ipv4' => $response['ip_address'] ?? '',
                    'ipv6' => $this->request->getIPAddress() ?? '',
                    'isp_name' => $response['connection']['isp_name'] ?? 'Unknown',
                    'isp_city' => $response['city'] ?? 'Unknown',
                    'isp_region' => $response['region'] ?? 'Unknown',
                    'isp_country' => $response['country'] ?? 'Unknown',
                    'isp_lat' => $response['latitude'] ?? 0,
                    'isp_lng' => $response['longitude'] ?? 0,
                ];
            }
            return view('welcome_message', $data);
        } catch (\Exception $e) {
            exit($e->getMessage());
        }

        $this->load->view('dashboard/includes/_header', $data);
        $this->load->view('dashboard/coupon/coupons', $data);
        $this->load->view('dashboard/includes/_footer');
    }

Now in view coupons.php I try get country:

                <?=
                $status ?
                '

My IP Address is:

                        IPv4:

                        ' . $ipv4 . '

                        IPv6:

                         '.$ipv6. '

My IP Information:

                        ISP:
                         ' . $isp_name . '

                        City:
                        ' . $isp_city . '

                        Region:
                        ' . $isp_region . '

                        Country:
                        ' . $isp_country . '

'
                :
                '
 Could not find your IP Address.
'
                ?>

But after this I get:

Message: Undefined property: Dashboard_controller::$request

Filename: controllers/Dashboard_controller.php

Type: Error

Message: Call to a member function getIPAddress() on null

Filename: public_html/application/controllers/Dashboard_controller.php

Line Number: 920

line 920:

'ip_address' => $this->request->getIPAddress()

Can anyone help me from where this error come and how to resolve?


问:
Spark evicting storage memory beyond the limit defined by spark.memory.storageFraction
I am running a small pyspark experiment on a dataproc cluster. I am not able to put together the dynamic memory allocation with what I am observing in the sparkUI

cluster configuration (directly from the env tab of the sparkUI):

```<code>spark.executor.instances = 4
spark.executor.memory = 6G
spark.executor.memoryOverHead = 2G
spark.executor.pyspark.memory = 2G
spark.memory.fraction = 0.6
spark.memory.offHeap.size = 2G
spark.memory.storageFraction = 0.5
</code>```

So, my immune to eviction part of the heap is (accounting for 300 MB reserved memory)

```<code>(6*(1024)-300)*0.6*0.5
1753.2 MBs per executor
</code>```

Since I have 4 executors, my overall heap immune to eviction should be

```<code>1753.2*4
7012.8 MBs or 6.85 GBs
</code>```

Now, I read in a dataset (It is around 9.5 GB) and put it in the cache

```<code>df_sales = spark.read.option("format","parquet").option("header",True).option("inferSchema",True).load("gs://monsoon-credittech.appspot.com/spark_datasets/sales_parquet")
from pyspark import StorageLevel
df_sales = df_sales.persist(StorageLevel.MEMORY_AND_DISK)
df_sales.count()
</code>```

My storage tab in the spark UI shows that I have been able to put all of the data in the memory and no disk spill occurred. It has just one row (expected) for the df_sales

```<code>Storage Level: Disk Memory Serialized 1x Replicated
Cached Partitions 83
Fraction Cached 100%
Size in Memory 9.5 GiB
Size on Disk 0.0B
</code>```

Now, the part of the storage that is immune to eviction (6.5 GBs) should never be overtaken by execution unless I unpersist some data and make that memory available.

To test that I ran an operation that uses heavy execution memory (join) and then persisted the output and checked how the persisted output got divided between the disk and the memory. I used StorageLevel.MEMORY_AND_DISK

This is how I am doing it:

```<code>df_sales_copy = spark.read.option("format","parquet").option("header",True).option("inferSchema",True).load("gs://monsoon-credittech.appspot.com/spark_datasets/sales_parquet")
df_merged = df_sales.join(df_sales_copy,df_sales.order_id==df_sales_copy.order_id,'inner')
df_merged = df_merged.persist(StorageLevel.MEMORY_AND_DISK)
df_merged.count()
</code>```

Now, my storage tab shows me numbers that I cannot make sense of. The eviction should never have reduced the storage memory below 6.85 GBs. However, the storage shows that it is reduced to 3.5 GBs. The storage tab now has two rows (expected), one for df_sales and the other for the df_merged

```<code>df_sales:
Storage Level: Disk Memory Serialized 1x Replicated
Cached Partitions 83
Fraction Cached 100%
Size in Memory 0.0 B #evicted as earlier it was 9.5 GiB
Size on Disk 9.5 GiB

df_merged:
Storage Level: Disk Memory Serialized 1x Replicated
Cached Partitions 200
Fraction Cached 100%
Size in Memory 3.4 GiB
Size on Disk 15.8 GiB
</code>```

I have not called unpersist anywhere then why is spark allowed to evict memory below the hard threshold set by spark.memory.storageFraction

Body:
I am running a small pyspark experiment on a dataproc cluster. I am not able to put together the dynamic memory allocation with what I am observing in the sparkUI

cluster configuration (directly from the env tab of the sparkUI):

spark.executor.instances = 4
spark.executor.memory = 6G
spark.executor.memoryOverHead = 2G
spark.executor.pyspark.memory = 2G
spark.memory.fraction = 0.6
spark.memory.offHeap.size = 2G
spark.memory.storageFraction = 0.5

So, my immune to eviction part of the heap is (accounting for 300 MB reserved memory)

(6*(1024)-300)*0.6*0.5
1753.2 MBs per executor

Since I have 4 executors, my overall heap immune to eviction should be

1753.2*4
7012.8 MBs or 6.85 GBs

Now, I read in a dataset (It is around 9.5 GB) and put it in the cache

df_sales = spark.read.option("format","parquet").option("header",True).option("inferSchema",True).load("gs://monsoon-credittech.appspot.com/spark_datasets/sales_parquet")
from pyspark import StorageLevel
df_sales = df_sales.persist(StorageLevel.MEMORY_AND_DISK)
df_sales.count()

My storage tab in the spark UI shows that I have been able to put all of the data in the memory and no disk spill occurred. It has just one row (expected) for the df_sales

Storage Level: Disk Memory Serialized 1x Replicated
Cached Partitions 83
Fraction Cached 100%
Size in Memory 9.5 GiB
Size on Disk 0.0B

Now, the part of the storage that is immune to eviction (6.5 GBs) should never be overtaken by execution unless I unpersist some data and make that memory available.

To test that I ran an operation that uses heavy execution memory (join) and then persisted the output and checked how the persisted output got divided between the disk and the memory. I used StorageLevel.MEMORY_AND_DISK

This is how I am doing it:

df_sales_copy = spark.read.option("format","parquet").option("header",True).option("inferSchema",True).load("gs://monsoon-credittech.appspot.com/spark_datasets/sales_parquet")
df_merged = df_sales.join(df_sales_copy,df_sales.order_id==df_sales_copy.order_id,'inner')
df_merged = df_merged.persist(StorageLevel.MEMORY_AND_DISK)
df_merged.count()

Now, my storage tab shows me numbers that I cannot make sense of. The eviction should never have reduced the storage memory below 6.85 GBs. However, the storage shows that it is reduced to 3.5 GBs. The storage tab now has two rows (expected), one for df_sales and the other for the df_merged

df_sales:
Storage Level: Disk Memory Serialized 1x Replicated
Cached Partitions 83
Fraction Cached 100%
Size in Memory 0.0 B #evicted as earlier it was 9.5 GiB
Size on Disk 9.5 GiB

df_merged:
Storage Level: Disk Memory Serialized 1x Replicated
Cached Partitions 200
Fraction Cached 100%
Size in Memory 3.4 GiB
Size on Disk 15.8 GiB

I have not called unpersist anywhere then why is spark allowed to evict memory below the hard threshold set by spark.memory.storageFraction


问:
the login was successful but when the next step always appears the message '401' Unauthorized on Vue.js
when the account login endpoint localhost:5000/login a successful message appears, but when I call localhost:5000/status an error message '401' Unauthorized appears as if the login session was not saved

Here My Backend controllers/Auth.js

```<code>export const Status = async(req, res) =>{
    if(!req.session.userId){
        return res.status(401).json({msg: "Mohon login terlebih dahulu"});
    }
    const user = await User.findOne({
        attributes: ['uuid', 'name', 'email', 'number','role'],
        where: {
            uuid: req.session.userId
        }
    });
    if (!user) return res.status(400).json({msg: "User Tidak Ada"});
    res.status(200).json(user);
}
</code>```

Here My Frontend Auth.vue

```<code>import axios from 'axios';

export default{
    namespace: true,
    state: {
        uuid: null,
        user: null,
    },

    mutations: {
        SET_UUID(state, uuid){
            state.uuid = uuid
        },
        SET_USER(state, data){
            state.user = data
        }
    },

    actions: {
        async signIn({ dispatch }, credentials){
            let response = await axios.post('login', credentials)
            dispatch('attempt', response.data.uuid)
          },
          async attempt({ commit }, uuid){
            commit('SET_UUID', uuid)

            try {
                let response = await axios.get('status',{
                    headers: {
                        'Authorization': 'Bearer ' + uuid
                    }
                })
                commit('SET_USER', response.data)
            } catch (error) {
                 commit('SET_UUID', null);
                 commit('SET_USER', null);
            }
        }
    },
}
</code>```

My Route route/AuthRoute.js

```<code>import express from "express";
import {
    Login,
    Status,
    logOut
} from "../controllers/Atuh.js";

const router = express.Router();

router.get("/status", Status);
router.post("/login", Login);
router.delete("/logout", logOut);

export default router;
</code>```

Body:
when the account login endpoint localhost:5000/login a successful message appears, but when I call localhost:5000/status an error message '401' Unauthorized appears as if the login session was not saved

Here My Backend controllers/Auth.js

export const Status = async(req, res) =>{
    if(!req.session.userId){
        return res.status(401).json({msg: "Mohon login terlebih dahulu"});
    }
    const user = await User.findOne({
        attributes: ['uuid', 'name', 'email', 'number','role'],
        where: {
            uuid: req.session.userId
        }
    });
    if (!user) return res.status(400).json({msg: "User Tidak Ada"});
    res.status(200).json(user);
}

Here My Frontend Auth.vue

import axios from 'axios';

export default{
    namespace: true,
    state: {
        uuid: null,
        user: null,
    },

    mutations: {
        SET_UUID(state, uuid){
            state.uuid = uuid
        },
        SET_USER(state, data){
            state.user = data
        }
    },

    actions: {
        async signIn({ dispatch }, credentials){
            let response = await axios.post('login', credentials)
            dispatch('attempt', response.data.uuid)
          },
          async attempt({ commit }, uuid){
            commit('SET_UUID', uuid)

            try {
                let response = await axios.get('status',{
                    headers: {
                        'Authorization': 'Bearer ' + uuid
                    }
                })
                commit('SET_USER', response.data)
            } catch (error) {
                 commit('SET_UUID', null);
                 commit('SET_USER', null);
            }
        }
    },
}

My Route route/AuthRoute.js

import express from "express";
import {
    Login,
    Status,
    logOut
} from "../controllers/Atuh.js";

const router = express.Router();

router.get("/status", Status);
router.post("/login", Login);
router.delete("/logout", logOut);

export default router;


问:
how to establish a udp connection between two different devices in java using client and server archiecture
I am able to establish server and client connection in same device using InetAddress.getByName("localhost")
how can i connect to other device when i run server program in one and client in other device

how to establish a udp connection between two different devices in java using client and server archiecture

SERVER SIDE

```<code>import java.net.*;
import java.net.InetAddress;
class UDPServer
{
public static void main(String args[])throws Exception
{
DatagramSocket serverSocket = new DatagramSocket(9876);
byte[] receiveData=new byte[1024];
byte[] sendData=new byte[1024];
while(true)
{
System.out.println("Server is Up");

DatagramPacket receivePacket=new DatagramPacket(receiveData,receiveData.length);

serverSocket.receive(receivePacket);

String sentence=new String(receivePacket.getData());

System.out.println("RECEIVED:"+sentence);

InetAddress IPAddress=receivePacket.getAddress();

int port=receivePacket.getPort();

String capitalizedSentence=sentence.toUpperCase();

sendData=capitalizedSentence.getBytes();

DatagramPacket sendPacket=new
DatagramPacket(sendData,sendData.length,IPAddress,port);
serverSocket.send(sendPacket);
}
}
}

</code>```

CLIENT SIDE

```<code>
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.*;
import java.net.InetAddress;
public class UDPclient {
    public static void main(String[] args)throws Exception
    {
    BufferedReader inFromUser=new BufferedReader(new InputStreamReader(System.in));

    DatagramSocket clientSocket=new DatagramSocket();

    InetAddress IPAddress=InetAddress.getByName("localhost");

    byte[] sendData=new byte[1024];
    byte[] receiveData=new byte[1024];

    System.out.println("Enter the sting to be converted in to Upper case");
    String sentence=inFromUser.readLine();

    sendData=sentence.getBytes();

    DatagramPacket sendPacket=new
    DatagramPacket(sendData,sendData.length,InetAddress.getLocalHost(),9876);

    clientSocket.send(sendPacket);

    DatagramPacket receivePacket=new DatagramPacket(receiveData,receiveData.length);

    clientSocket.receive(receivePacket);

    String modifiedSentence=new String(receivePacket.getData());

    System.out.println("FROM SERVER:"+modifiedSentence);

    clientSocket.close();
    }
    }
</code>```

Body:
I am able to establish server and client connection in same device using InetAddress.getByName("localhost")
how can i connect to other device when i run server program in one and client in other device

how to establish a udp connection between two different devices in java using client and server archiecture

SERVER SIDE

import java.net.*;
import java.net.InetAddress;
class UDPServer
{
public static void main(String args[])throws Exception
{
DatagramSocket serverSocket = new DatagramSocket(9876);
byte[] receiveData=new byte[1024];
byte[] sendData=new byte[1024];
while(true)
{
System.out.println("Server is Up");

DatagramPacket receivePacket=new DatagramPacket(receiveData,receiveData.length);

serverSocket.receive(receivePacket);

String sentence=new String(receivePacket.getData());

System.out.println("RECEIVED:"+sentence);

InetAddress IPAddress=receivePacket.getAddress();

int port=receivePacket.getPort();

String capitalizedSentence=sentence.toUpperCase();

sendData=capitalizedSentence.getBytes();

DatagramPacket sendPacket=new
DatagramPacket(sendData,sendData.length,IPAddress,port);
serverSocket.send(sendPacket);
}
}
}

CLIENT SIDE

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.*;
import java.net.InetAddress;
public class UDPclient {
    public static void main(String[] args)throws Exception
    {
    BufferedReader inFromUser=new BufferedReader(new InputStreamReader(System.in));

    DatagramSocket clientSocket=new DatagramSocket();

    InetAddress IPAddress=InetAddress.getByName("localhost");

    byte[] sendData=new byte[1024];
    byte[] receiveData=new byte[1024];

    System.out.println("Enter the sting to be converted in to Upper case");
    String sentence=inFromUser.readLine();

    sendData=sentence.getBytes();

    DatagramPacket sendPacket=new
    DatagramPacket(sendData,sendData.length,InetAddress.getLocalHost(),9876);

    clientSocket.send(sendPacket);

    DatagramPacket receivePacket=new DatagramPacket(receiveData,receiveData.length);

    clientSocket.receive(receivePacket);

    String modifiedSentence=new String(receivePacket.getData());

    System.out.println("FROM SERVER:"+modifiedSentence);

    clientSocket.close();
    }
    }


问:
How to create Infinispan ClusteredLockManager in Wildfly?
I have Wildfly 26.1.2 and Infinispan 13.0.10.Final.
We already have a cluster and Wildfly Infinispan subsystem with cache containers, transport, jgroups, and local caches.

Now we want to use clustered-locks, and I need to create ClusteredLockManager

If I read the manuals here
https://infinispan.org/docs/stable/titles/embedding/embedding.html

They say

```<code>    // Set up a clustered Cache Manager.
    GlobalConfigurationBuilder global = GlobalConfigurationBuilder.defaultClusteredBuilder();

    // Configure the cache mode, in this case it is distributed and synchronous.
    ConfigurationBuilder builder = new ConfigurationBuilder();
    builder.clustering().cacheMode(CacheMode.DIST_SYNC);

    // Initialize a new default Cache Manager.
    DefaultCacheManager cm = new DefaultCacheManager(global.build(), builder.build());

    // Initialize a Clustered Lock Manager.
    ClusteredLockManager clm1 =

EmbeddedClusteredLockManagerFactory.from(cm);
</code>```

In the logs, I see it creates a NEW cluster with a default name (ISPN), and I need to put here all the configurations (at least with jgroups and transport), and duplicate them from the inifinispan-wildfly subsystems

But I don't want to do it because I already have configured cluster in my standalone-ec2-full-ha.xml file with Infinispan caches containers and managers and all that stuff

Can I somehow create my ClusteredLockManager using an already-defined configuration and infinispan-cluster from a standalone-ec2-full-ha.xml file?
Or can I define a config for ClusteredLockManager in my Wildfly subsystem?
And just inject it as a resource in Java Code

If I do something like

```<code>    @Resource(name = "java:jboss/infinispan/container/cacheManager")
    private EmbeddedCacheManager cacheManager;

...

  ClusteredLockManager lockManager = EmbeddedClusteredLockManagerFactory.from(cacheManager);
</code>```

It returns null. And the factory can create ClusteredLockManager only from cacheManager.

Are there any options for creating ClusteredLockManager using the existing cluster and wildfly-infinispan-subsystems configuration?

Body:
I have Wildfly 26.1.2 and Infinispan 13.0.10.Final.
We already have a cluster and Wildfly Infinispan subsystem with cache containers, transport, jgroups, and local caches.

Now we want to use clustered-locks, and I need to create ClusteredLockManager

If I read the manuals here
https://infinispan.org/docs/stable/titles/embedding/embedding.html

They say

    // Set up a clustered Cache Manager.
    GlobalConfigurationBuilder global = GlobalConfigurationBuilder.defaultClusteredBuilder();

    // Configure the cache mode, in this case it is distributed and synchronous.
    ConfigurationBuilder builder = new ConfigurationBuilder();
    builder.clustering().cacheMode(CacheMode.DIST_SYNC);

    // Initialize a new default Cache Manager.
    DefaultCacheManager cm = new DefaultCacheManager(global.build(), builder.build());

    // Initialize a Clustered Lock Manager.
    ClusteredLockManager clm1 =

EmbeddedClusteredLockManagerFactory.from(cm);

In the logs, I see it creates a NEW cluster with a default name (ISPN), and I need to put here all the configurations (at least with jgroups and transport), and duplicate them from the inifinispan-wildfly subsystems

But I don't want to do it because I already have configured cluster in my standalone-ec2-full-ha.xml file with Infinispan caches containers and managers and all that stuff

Can I somehow create my ClusteredLockManager using an already-defined configuration and infinispan-cluster from a standalone-ec2-full-ha.xml file?
Or can I define a config for ClusteredLockManager in my Wildfly subsystem?
And just inject it as a resource in Java Code

If I do something like

    @Resource(name = "java:jboss/infinispan/container/cacheManager")
    private EmbeddedCacheManager cacheManager;

...

  ClusteredLockManager lockManager = EmbeddedClusteredLockManagerFactory.from(cacheManager);

It returns null. And the factory can create ClusteredLockManager only from cacheManager.

Are there any options for creating ClusteredLockManager using the existing cluster and wildfly-infinispan-subsystems configuration?


问:
Selenium tests not running on Docker using C#
I am trying to run my selenium tests in Docker but I don't see any containers running and don't see anything in the docker logs.
I have created 3 different versions of DockerFiles which can be found in https://github.com/devanasri/googlesearchauto

The commands I am using to create image and start container is

```<code>docker build -t googlesearchauto .
docker run -it automation
</code>```

When run above commands to start the container and run selenium tests, nothing happens and sometimes I get below error when trying to start the container.

```<code>The command could not be loaded, possibly because:
  * You intended to execute a .NET application:
      The application 'test' does not exist.
  * You intended to execute a .NET SDK command:
      No .NET SDKs were found.

Download a .NET SDK:
https://aka.ms/dotnet-download

Learn about SDK resolution:
https://aka.ms/dotnet/sdk-not-found
</code>```

I tried to run each version of the DockerFile but all of them not working. Any help is appreciated.

Body:
I am trying to run my selenium tests in Docker but I don't see any containers running and don't see anything in the docker logs.
I have created 3 different versions of DockerFiles which can be found in https://github.com/devanasri/googlesearchauto

The commands I am using to create image and start container is

docker build -t googlesearchauto .
docker run -it automation

When run above commands to start the container and run selenium tests, nothing happens and sometimes I get below error when trying to start the container.

The command could not be loaded, possibly because:
  * You intended to execute a .NET application:
      The application 'test' does not exist.
  * You intended to execute a .NET SDK command:
      No .NET SDKs were found.

Download a .NET SDK:
https://aka.ms/dotnet-download

Learn about SDK resolution:
https://aka.ms/dotnet/sdk-not-found

I tried to run each version of the DockerFile but all of them not working. Any help is appreciated.


问:
How can I use my VBA when Excel's hyperlink-following doesn't work, without getting two browser pages when it does?

Unable to open yourURL. Cannot download the information you requested.

The above error message upon clicking a hyperlink in Excel is reported commonly online, for multiple versions of Excel (I have 2010, with Windows 7), with several stock answers that many users find don't work for them.  I am among the users who have tried several registry hacks and other solutions, finding none of them work.

I only get the error with some URLs, and not with others (so far, all of the ones that are broken appear to be https, but some https links work fine).  Of course, all of the URLs work fine if I simply paste them into my browser.  Firefox is my preferred browser, I only use Chrome if Firefox isn't rendering a particular web site correctly, and I don't use IE or Edge at all.

Upon coming across a couple of VBA examples for linking to web pages, I decided to try something myself.  Here is what I came up with (still needs to be pared down, I tend to be verbose during initial explorations):

```<code>Private Sub Worksheet_FollowHyperlink(ByVal Target As Hyperlink)
    Dim targetAddress As String
    Dim targetSubAddress As String
    Dim targetUrl As String

    targetAddress = Target.Address
    targetSubAddress = Target.SubAddress

    targetUrl = targetAddress
    'If Not targetSubAddress Is Nothing Then
    targetUrl = targetUrl & "#" & targetSubAddress
    'End If

    result = MsgBox("View Web URL " & targetUrl & "?", vbYesNo)

    If result = vbYes Then
    MsgBox ("C:\Program Files\Mozilla Firefox\firefox.exe " & targetUrl)
    Shell ("C:\Program Files\Mozilla Firefox\firefox.exe " & targetUrl)
    End If
End Sub
</code>```

The good news is, it works for my most recent problem URL!  The annoying news (but less annoying than the links not working) is that the links that already worked are still working with the original mechanism, plus now my VBA is working, too, so for them, I get two Firefox tabs popping up instead of one.

Is there a way to suppress Excel's built-in hyperlink-following functionality for links to external web sites (it would be OK to use it for links to other spreadsheets, for example)?  Conversely, is there a way to limit my VBA to only apply to links to external web sites -- or better still, only to the ones that didn't succeed with the built-in mechanism?

Is this something I should submit to SuperUser instead?

Body:
Unable to open yourURL. Cannot download the information you requested.

The above error message upon clicking a hyperlink in Excel is reported commonly online, for multiple versions of Excel (I have 2010, with Windows 7), with several stock answers that many users find don't work for them.  I am among the users who have tried several registry hacks and other solutions, finding none of them work.

I only get the error with some URLs, and not with others (so far, all of the ones that are broken appear to be https, but some https links work fine).  Of course, all of the URLs work fine if I simply paste them into my browser.  Firefox is my preferred browser, I only use Chrome if Firefox isn't rendering a particular web site correctly, and I don't use IE or Edge at all.

Upon coming across a couple of VBA examples for linking to web pages, I decided to try something myself.  Here is what I came up with (still needs to be pared down, I tend to be verbose during initial explorations):

Private Sub Worksheet_FollowHyperlink(ByVal Target As Hyperlink)
    Dim targetAddress As String
    Dim targetSubAddress As String
    Dim targetUrl As String

    targetAddress = Target.Address
    targetSubAddress = Target.SubAddress

    targetUrl = targetAddress
    'If Not targetSubAddress Is Nothing Then
    targetUrl = targetUrl & "#" & targetSubAddress
    'End If

    result = MsgBox("View Web URL " & targetUrl & "?", vbYesNo)

    If result = vbYes Then
    MsgBox ("C:\Program Files\Mozilla Firefox\firefox.exe " & targetUrl)
    Shell ("C:\Program Files\Mozilla Firefox\firefox.exe " & targetUrl)
    End If
End Sub

The good news is, it works for my most recent problem URL!  The annoying news (but less annoying than the links not working) is that the links that already worked are still working with the original mechanism, plus now my VBA is working, too, so for them, I get two Firefox tabs popping up instead of one.

Is there a way to suppress Excel's built-in hyperlink-following functionality for links to external web sites (it would be OK to use it for links to other spreadsheets, for example)?  Conversely, is there a way to limit my VBA to only apply to links to external web sites -- or better still, only to the ones that didn't succeed with the built-in mechanism?

Is this something I should submit to SuperUser instead?


问:
Python Selenium webdriver, Click on a image link without name/id
```<code> <span><a href="javascript:;" onclick="refreshOCPage('equity')"><img class="refreshIcon" src="/assets/images/refresh-icon.svg" alt="Refresh" title="Refresh" /></a></span>
</code>```

This is the part of HTML source I want to webscrap / download..
I want to click on the link image on the page..Or trigger the javascript call refreshOCPage('equity')

There are actually four instances of the image, with four different parameter on Onclick="refreshOCPage('gold')",
Onclick="refreshOCPage('commodity')"
, Onclick="refreshOCPage('currency')"
....

I want to specifically find one with Onclick="refreshOCPage('equity')

I used

```<code>refreshlink = driver.find_element(By.XPATH("//a[@onclick='refreshOCPage(\'equity\')']"))
refreshlink.click()
</code>```

Does not work as argument is not acceptable syntax...

I also tried to run the script by

```<code>driver.execute("refreshOCPage()",'equity')
</code>```

Assuming it would call the refreshOCPage('equity') as in the onclick of the link of interest.

But it gives

```<code>TypeError: 'str' object does not support item assignment
</code>```

Body:
This is the part of HTML source I want to webscrap / download..
I want to click on the link image on the page..Or trigger the javascript call refreshOCPage('equity')

There are actually four instances of the image, with four different parameter on Onclick="refreshOCPage('gold')",
Onclick="refreshOCPage('commodity')"
, Onclick="refreshOCPage('currency')"
....

I want to specifically find one with Onclick="refreshOCPage('equity')

I used

refreshlink = driver.find_element(By.XPATH("//a[@onclick='refreshOCPage(\'equity\')']"))
refreshlink.click()

Does not work as argument is not acceptable syntax...

I also tried to run the script by

driver.execute("refreshOCPage()",'equity')

Assuming it would call the refreshOCPage('equity') as in the onclick of the link of interest.

But it gives

TypeError: 'str' object does not support item assignment


问:
How to make my code rebuild when a dependency is updated by the system
I have a C++ CMake project using the ninja generator (but this should also apply to GNU Make). My project depends on some system libraries, with headers and shared libraries.

Whenever I update those dependency on my system (with APT), the headers and the .so are given the modification time of there build (which may be a few days ago). The result is that my compiled files may have a more recent modification time that the headers, causing them to not be re-built and sometimes have link issue because the .so is not ABI compatible.

Also, if I modify a source file, it is compiled using the new version of the headers while the other source file are not.

Is there a solution to this, besides cleaning my project at each update?


问:
how to trigger chrome extension function from puppeteer Cluster
I need to trigger a function in background.js in extension from puppeteer Cluster
here is my code :

```<code>const wait = (ms) => new Promise(resolve => setTimeout(resolve, ms));
(async () => {
    const puppeteer = addExtra(puppeteerStream);
    const pathToExtension = "C:/Users/Proj/test-extension";
    const extensionId = 'lpnlgnlkloegindjk443sfdbjipo';

    const cluster = await Cluster.launch({
        concurrency: Cluster.CONCURRENCY_PAGE,
        maxConcurrency: 50,
        timeout: 86400000,
        retryLimit: 10,
        retryDelay: 1000,
        puppeteer: puppeteer,
        executablePath: executablePath(),
        puppeteerOptions: {
            executablePath: "C:/Program Files/Google/Chrome/Application/chrome.exe",
            timeout: 120000,
            headless: false,
            pipe: true,
            defaultViewport:null,
            ignoreDefaultArgs: [
            ],
            args: ["--no-sandbox", "--disable-setuid-sandbox", "--disable-notifications", "--allow-http-screen-capture", "--autoplay-policy=no-user-gesture-required",
                '--disable-gpu',
                '--disable-dev-shm-usage',
                '--no-first-run',
                '--enable-usermedia-screen-capturing',
                '--auto-select-desktop-capture-source=pickme',
                '--no-zygote',
                `--whitelisted-extension-id=${extensionId}`,
                '--enable-remote-extensions',
                '--start-maximized',
                `--disable-extensions-except=${pathToExtension}`,
                `--load-extension=${pathToExtension}`,
            ]
        }
    });

    // setup the function to be executed for each request
    cluster.task(async ({ page, data: url }) => {
        return start(page, url);
    });

    app.get('/', async function (req, res) {
        res.send("index page");
    });
    app.get('/start', async function (req, res) { // expects URL to be given by ?url=...
        try {
            // run the task function for the URL
            const resp = cluster.queue(req.query.url);
            res.send(resp);
        } catch (err) {
            res.end('Error: ' + err.message);
        }
    });

    async function start(page, Url) {
        const context = page.browser().defaultBrowserContext();
        await page.goto(Url);
    }
})();
</code>```

now I can run mt puppeteer and click a button in my extension manually but what I need is call the function (same as button click in extension) from puppeteer cluster automatically after  await page.goto(Url);

I can do that when I'm using puppeteer without cluster like :

```<code>const browser = await puppeteer.launch({
        headless: false,
        args: [
            `--whitelisted-extension-id=${extensionId}`,
            '--no-sandbox',
            '--disable-setuid-sandbox',
            '--disable-dev-shm-usage',
            `--disable-extensions-except=${pathToExtension}`,
            `--load-extension=${pathToExtension}`,
        ],
    });
    const page = await browser.newPage();
    await page.goto('https://www.youtube.com/', { waitUntil: 'networkidle2' });

    const targets = await browser.targets();
    const backgroundPageTarget = targets.find(target => target.type() === 'background_page' && target.url().startsWith(`chrome-extension://${extensionId}/`));
    const backgroundPage = await backgroundPageTarget.page();

    backgroundPage.on('console', msg => {
        for (let i = 0; i < msg.args().length; i++) {
            console.log(`${i}: ${msg.args()[i]}`);
        }
    });

    await backgroundPage.evaluate(() => {
        startFunction();
        return Promise.resolve(42);
    });
</code>```

But I cannot get the targets in cluster any suggestion

Body:
I need to trigger a function in background.js in extension from puppeteer Cluster
here is my code :

const wait = (ms) => new Promise(resolve => setTimeout(resolve, ms));
(async () => {
    const puppeteer = addExtra(puppeteerStream);
    const pathToExtension = "C:/Users/Proj/test-extension";
    const extensionId = 'lpnlgnlkloegindjk443sfdbjipo';

    const cluster = await Cluster.launch({
        concurrency: Cluster.CONCURRENCY_PAGE,
        maxConcurrency: 50,
        timeout: 86400000,
        retryLimit: 10,
        retryDelay: 1000,
        puppeteer: puppeteer,
        executablePath: executablePath(),
        puppeteerOptions: {
            executablePath: "C:/Program Files/Google/Chrome/Application/chrome.exe",
            timeout: 120000,
            headless: false,
            pipe: true,
            defaultViewport:null,
            ignoreDefaultArgs: [
            ],
            args: ["--no-sandbox", "--disable-setuid-sandbox", "--disable-notifications", "--allow-http-screen-capture", "--autoplay-policy=no-user-gesture-required",
                '--disable-gpu',
                '--disable-dev-shm-usage',
                '--no-first-run',
                '--enable-usermedia-screen-capturing',
                '--auto-select-desktop-capture-source=pickme',
                '--no-zygote',
                `--whitelisted-extension-id=${extensionId}`,
                '--enable-remote-extensions',
                '--start-maximized',
                `--disable-extensions-except=${pathToExtension}`,
                `--load-extension=${pathToExtension}`,
            ]
        }
    });

    // setup the function to be executed for each request
    cluster.task(async ({ page, data: url }) => {
        return start(page, url);
    });

    app.get('/', async function (req, res) {
        res.send("index page");
    });
    app.get('/start', async function (req, res) { // expects URL to be given by ?url=...
        try {
            // run the task function for the URL
            const resp = cluster.queue(req.query.url);
            res.send(resp);
        } catch (err) {
            res.end('Error: ' + err.message);
        }
    });

    async function start(page, Url) {
        const context = page.browser().defaultBrowserContext();
        await page.goto(Url);
    }
})();

now I can run mt puppeteer and click a button in my extension manually but what I need is call the function (same as button click in extension) from puppeteer cluster automatically after  await page.goto(Url);

I can do that when I'm using puppeteer without cluster like :

const browser = await puppeteer.launch({
        headless: false,
        args: [
            `--whitelisted-extension-id=${extensionId}`,
            '--no-sandbox',
            '--disable-setuid-sandbox',
            '--disable-dev-shm-usage',
            `--disable-extensions-except=${pathToExtension}`,
            `--load-extension=${pathToExtension}`,
        ],
    });
    const page = await browser.newPage();
    await page.goto('https://www.youtube.com/', { waitUntil: 'networkidle2' });

    const targets = await browser.targets();
    const backgroundPageTarget = targets.find(target => target.type() === 'background_page' && target.url().startsWith(`chrome-extension://${extensionId}/`));
    const backgroundPage = await backgroundPageTarget.page();

    backgroundPage.on('console', msg => {
        for (let i = 0; i < msg.args().length; i++) {
            console.log(`${i}: ${msg.args()[i]}`);
        }
    });

    await backgroundPage.evaluate(() => {
        startFunction();
        return Promise.resolve(42);
    });

But I cannot get the targets in cluster any suggestion


问:
How can I fix pip doesn't work via proxy with new venv
I have a python project working in venv with python3.7.
When I create new venv with python 3.10 for this project pip can not install packages via proxy. However it work in old venv.

cli:

```<code>C:\WINDOWS\system32>C:\Users\user\PycharmProjects\PytestHabr\ch2\venv\Scripts\activate.bat

(venv) C:\WINDOWS\system32>pip install --proxy http://user:pass@bproxy:port pytest
Collecting pytest
  Downloading pytest-7.2.0-py3-none-any.whl (316 kB)
     |████████████████████████████████| 316 kB 939 kB/s

(venv)C:\WINDOWS\system32>C:\Users\user\PycharmProjects\PytestHabr\ch2\venv_py310\Scripts\activate.bat
(venv_py310) C:\WINDOWS\system32>pip install --proxy http://user:pass@bproxy:port pytest

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', FileNotFoundError(2, 'No such file or directory'))': /simple/pytest/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', FileNotFoundError(2, 'No such file or directory'))': /simple/pytest/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', FileNotFoundError(2, 'No such file or directory'))': /simple/pytest/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', FileNotFoundError(2, 'No such file or directory'))': /simple/pytest/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', FileNotFoundError(2, 'No such file or directory'))': /simple/pytest/

ERROR: Could not find a version that satisfies the requirement pytest (from versions: none)
ERROR: No matching distribution found for pytest

(venv_py310) C:\WINDOWS\system32>pip --version
pip 21.3.1 from
C:\Users\user\PycharmProjects\PytestHabr\ch2\venv_py310\lib\site-packages\pip (python 3.10)
</code>```

I tried use PyCharm with proxy setting, but it works only with venv Python 3.7.

Body:
I have a python project working in venv with python3.7.
When I create new venv with python 3.10 for this project pip can not install packages via proxy. However it work in old venv.

cli:

C:\WINDOWS\system32>C:\Users\user\PycharmProjects\PytestHabr\ch2\venv\Scripts\activate.bat

(venv) C:\WINDOWS\system32>pip install --proxy http://user:pass@bproxy:port pytest
Collecting pytest
  Downloading pytest-7.2.0-py3-none-any.whl (316 kB)
     |████████████████████████████████| 316 kB 939 kB/s

(venv)C:\WINDOWS\system32>C:\Users\user\PycharmProjects\PytestHabr\ch2\venv_py310\Scripts\activate.bat
(venv_py310) C:\WINDOWS\system32>pip install --proxy http://user:pass@bproxy:port pytest

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', FileNotFoundError(2, 'No such file or directory'))': /simple/pytest/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', FileNotFoundError(2, 'No such file or directory'))': /simple/pytest/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', FileNotFoundError(2, 'No such file or directory'))': /simple/pytest/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', FileNotFoundError(2, 'No such file or directory'))': /simple/pytest/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', FileNotFoundError(2, 'No such file or directory'))': /simple/pytest/

ERROR: Could not find a version that satisfies the requirement pytest (from versions: none)
ERROR: No matching distribution found for pytest

(venv_py310) C:\WINDOWS\system32>pip --version
pip 21.3.1 from
C:\Users\user\PycharmProjects\PytestHabr\ch2\venv_py310\lib\site-packages\pip (python 3.10)

I tried use PyCharm with proxy setting, but it works only with venv Python 3.7.


问:
add values to csv column based on if condition
I'm trying to fill a column with values based on an if condition.

I have something like this at the moment..

review
action

poor
null

ok
null

excellent
null

poor
null

and I need for example:

review = poor,      action = terminate

review = ok,        action = n/a

review = excellent, action = prize

below is the code I wrote.

```<code>file = performance_review.csv
headers = ['review', 'action']

with open(file, 'a+', newline="") as csvfile:
    writer = csv.DictWriter(csvfile, fieldnames=headers)
    writer.writeheader()
    writer.writerows(performance_review)
    for row in csvfile:
        if row["review"] == "poor":
            row["action"] = "terminate"
        elif row["review"] == "excellent":
            row["action"] = "prize"
        else:
            row["action"] = "NA"
        writer.writerows(row)
        file.close()

with open(file, 'r') as file:
    reader = csv.reader(file)
    for row in reader:
        print(row)

</code>```

when I printed out the csv, action column remained blank and nothing was filled in.
can someone shed a light please?

below is what i expected and wanted.

review
action

poor
terminate

ok
n/a

excellent
prize

poor
terminate

much appreciated!

Body:
I'm trying to fill a column with values based on an if condition.

I have something like this at the moment..

review
action

poor
null

ok
null

excellent
null

poor
null

and I need for example:

review = poor,      action = terminate

review = ok,        action = n/a

review = excellent, action = prize

below is the code I wrote.

file = performance_review.csv
headers = ['review', 'action']

with open(file, 'a+', newline="") as csvfile:
    writer = csv.DictWriter(csvfile, fieldnames=headers)
    writer.writeheader()
    writer.writerows(performance_review)
    for row in csvfile:
        if row["review"] == "poor":
            row["action"] = "terminate"
        elif row["review"] == "excellent":
            row["action"] = "prize"
        else:
            row["action"] = "NA"
        writer.writerows(row)
        file.close()

with open(file, 'r') as file:
    reader = csv.reader(file)
    for row in reader:
        print(row)

when I printed out the csv, action column remained blank and nothing was filled in.
can someone shed a light please?

below is what i expected and wanted.

review
action

poor
terminate

ok
n/a

excellent
prize

poor
terminate

much appreciated!


问:
Angular Material v13 Table not Sorting
I made an Angular Material Table and now I'm trying to implement Sort. I build the table following the documentation for version 13, but the sorting doesn't work (even though the header is clickable and the sorting arrow changes direction on click.
Here is my component:

```<code>import { Component, VERSION, ViewChild } from '@angular/core';
import { MatSort } from '@angular/material/sort';

@Component({
  selector: 'my-app',
  templateUrl: './app.component.html',
  styleUrls: ['./app.component.css'],
})
export class AppComponent {
  @ViewChild(MatSort, { static: true }) sort!: MatSort;

  dataSource = [
    { value: 'One %value% none', type: 'Type is %type%', time: 1 },
    { value: 'Another %value%', type: 'This type is %type%', time: 3 },
    { value: '%value% none', type: 'The type is %type%', time: 2 },
  ];
  displayedColumns: string[] = Object.keys(this.dataSource[0]);

  name = 'Angular ' + VERSION.major;
}
</code>```

And here is the html:

```<code><p>Table works</p>
<table mat-table matSort [dataSource]="dataSource" class="mat-elevation-z8">
  <!-- Value Column -->
  <ng-container matColumnDef="value">
    <th mat-sort-header mat-header-cell *matHeaderCellDef>Vaule</th>
    <td mat-cell *matCellDef="let element">{{ element.value }}</td>
  </ng-container>

  <!-- Type Column -->
  <ng-container matColumnDef="type">
    <th mat-sort-header mat-header-cell *matHeaderCellDef>Type</th>
    <td mat-cell *matCellDef="let element">{{ element.type }}</td>
  </ng-container>

  <!-- Time Column -->
  <ng-container matColumnDef="time">
    <th mat-sort-header mat-header-cell *matHeaderCellDef>Time</th>
    <td mat-cell *matCellDef="let element">{{ element.time }}</td>
  </ng-container>

  <tr mat-header-row *matHeaderRowDef="displayedColumns"></tr>
  <tr mat-row *matRowDef="let row; columns: displayedColumns"></tr>
</table>
</code>```

The full reproducible example can be found here:
stackblitz

ps: there are a few similar questions in SO, but I couldn't find one with this version and this specifications.

Title: Angular Material v13 Table not Sorting

Body:
I made an Angular Material Table and now I'm trying to implement Sort. I build the table following the documentation for version 13, but the sorting doesn't work (even though the header is clickable and the sorting arrow changes direction on click.
Here is my component:

import { Component, VERSION, ViewChild } from '@angular/core';
import { MatSort } from '@angular/material/sort';

@Component({
  selector: 'my-app',
  templateUrl: './app.component.html',
  styleUrls: ['./app.component.css'],
})
export class AppComponent {
  @ViewChild(MatSort, { static: true }) sort!: MatSort;

  dataSource = [
    { value: 'One %value% none', type: 'Type is %type%', time: 1 },
    { value: 'Another %value%', type: 'This type is %type%', time: 3 },
    { value: '%value% none', type: 'The type is %type%', time: 2 },
  ];
  displayedColumns: string[] = Object.keys(this.dataSource[0]);

  name = 'Angular ' + VERSION.major;
}

And here is the html:

Table works

  <ng-container matColumnDef="value">
    Vaule
    {{ element.value }}
  </ng-container>

  <ng-container matColumnDef="type">
    Type
    {{ element.type }}
  </ng-container>

  <ng-container matColumnDef="time">
    Time
    {{ element.time }}
  </ng-container>

The full reproducible example can be found here:
stackblitz

ps: there are a few similar questions in SO, but I couldn't find one with this version and this specifications.


问:
Python OpenCV with Imaging device USB3 Vision compliant camera - VideoCapture() index of camera not found
I have a problem with using external camera in Python OpenCV.
I am using Windows.
In device manager, camera is not under "Cameras", but under "Imaging devices" and is called USB3 Vision compliant camera.
The camera is working fine on app given by company (UcamViewer).

I want to connect to this camera using cv2.VideoCapture().

I have of course tried:

```<code>cv2.VideoCapture(k)
</code>```

for k = [0..100]
Only k = 0 is occupied and works - that is my webcam.

I have also tried adding multiple possible values to the second parameter of the VideoCapture(), which did not work.

how to get video from a "USB3 Vision" device?
This question suggested to use cv.CAP_XIMEA as the second parameter of VideoCapture(), but CAP_XIMEA does not seem to be a possible value.

If I can not fix this issue and use VideoCapture(), what would you suggest to use to capture the video and convert it to format that can be used by openCV methods?

Any ideas how could I solve this issue will be appreciated.
Thank you for your help.

Body:
I have a problem with using external camera in Python OpenCV.
I am using Windows.
In device manager, camera is not under "Cameras", but under "Imaging devices" and is called USB3 Vision compliant camera.
The camera is working fine on app given by company (UcamViewer).

I want to connect to this camera using cv2.VideoCapture().

I have of course tried:

cv2.VideoCapture(k)

for k = [0..100]
Only k = 0 is occupied and works - that is my webcam.

I have also tried adding multiple possible values to the second parameter of the VideoCapture(), which did not work.

how to get video from a "USB3 Vision" device?
This question suggested to use cv.CAP_XIMEA as the second parameter of VideoCapture(), but CAP_XIMEA does not seem to be a possible value.

If I can not fix this issue and use VideoCapture(), what would you suggest to use to capture the video and convert it to format that can be used by openCV methods?

Any ideas how could I solve this issue will be appreciated.
Thank you for your help.


问:
Local storage object value doesn't change
I want my local storage value changed, after i click the edit button, change the value of an item and then after blur it should change and be saved to the local storage. However, in the following example value in the local storage changes only if I put a string in the value, like "text", however, not the input value itself (in the local storage).
I think something must be wrong with my condition (see in the blur section). Because once I do the condition inside blur's editItem, it doesn't see the e.target.value anymore (changed value on the screen), but e.target.value works perfectly outside that const editItem, and shows the one which was changed for.
Please help. A part of my code

I do expect the value of a local storage object changed for e.target.value after blur event listener

```<code>todoList.addEventListener('click', (e) => {
    if (e.target.dataset.button === 'delete-button') {
        const arrayWithoutRemovedItem = todoLocalStorage.filter((item) => {
            return item.text !== e.target.closest('.todo-item-wrapper').firstElementChild.value;
        });

        todoLocalStorage = arrayWithoutRemovedItem;

        localStorage.setItem('todo', JSON.stringify(todoLocalStorage));

        const itemWrapper = e.target.closest('.todo-item-wrapper');

        itemWrapper.replaceChildren();

    } else if (e.target.dataset.button === 'edit-button') {
        const itemInput = e.target.closest('.todo-item-wrapper').firstElementChild;

        itemInput.removeAttribute("readonly");

        itemInput.focus();

        itemInput.addEventListener('blur', (e) => {
            const editItem = todoLocalStorage.map(object => {
                if (object.text === itemInput.value) {
                    return {...object, text:itemInput.value};
                }

                return object;
            });

            todoLocalStorage = editItem;

            localStorage.setItem('todo', JSON.stringify(todoLocalStorage));

            itemInput.setAttribute("readonly", true);
        });
    }
});
</code>```

Body:
I want my local storage value changed, after i click the edit button, change the value of an item and then after blur it should change and be saved to the local storage. However, in the following example value in the local storage changes only if I put a string in the value, like "text", however, not the input value itself (in the local storage).
I think something must be wrong with my condition (see in the blur section). Because once I do the condition inside blur's editItem, it doesn't see the e.target.value anymore (changed value on the screen), but e.target.value works perfectly outside that const editItem, and shows the one which was changed for.
Please help. A part of my code

I do expect the value of a local storage object changed for e.target.value after blur event listener

todoList.addEventListener('click', (e) => {
    if (e.target.dataset.button === 'delete-button') {
        const arrayWithoutRemovedItem = todoLocalStorage.filter((item) => {
            return item.text !== e.target.closest('.todo-item-wrapper').firstElementChild.value;
        });

        todoLocalStorage = arrayWithoutRemovedItem;

        localStorage.setItem('todo', JSON.stringify(todoLocalStorage));

        const itemWrapper = e.target.closest('.todo-item-wrapper');

        itemWrapper.replaceChildren();

    } else if (e.target.dataset.button === 'edit-button') {
        const itemInput = e.target.closest('.todo-item-wrapper').firstElementChild;

        itemInput.removeAttribute("readonly");

        itemInput.focus();

        itemInput.addEventListener('blur', (e) => {
            const editItem = todoLocalStorage.map(object => {
                if (object.text === itemInput.value) {
                    return {...object, text:itemInput.value};
                }

                return object;
            });

            todoLocalStorage = editItem;

            localStorage.setItem('todo', JSON.stringify(todoLocalStorage));

            itemInput.setAttribute("readonly", true);
        });
    }
});


问:
net6/7: Build x86 and x64 in Github script and deploy to nuget
Currently I'm building a .NET 4.5.2 library in x86 and x64 and use the microsoft/setup-msbuild@v1 script on Github to build and NuGet/setup-nuget@v1.0.7 to deploy to Nuget, see pushPackageToNugetOrg.yml

For this I have my own .nuspec (ToolBox.nuspec) and .props file (DoenaSoft.ToolBox.props)

In a test balloon, I'm trying to migrate to a net6.0 windows project which in itself isn't hard. But I'm struggling with the Nuget configuration. By default it can be done in the .csproj file.

But if I configure it like this:

```<code><Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup Label="Globals">
        <Configurations>Debug;Release</Configurations>
        <Platforms>x86;x64</Platforms>
    </PropertyGroup>

    <PropertyGroup>
        <TargetFrameworks>net452;net6.0-windows;net7.0-windows;</TargetFrameworks>
        <RootNamespace>DoenaSoft.ToolBox</RootNamespace>
        <AssemblyName>DoenaSoft.ToolBox</AssemblyName>
        <Version>1.0.0</Version>
        <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
    </PropertyGroup>

</Project>
</code>```

(reduced for brevity) and then trigger the build then I end up with two different .nupkg files, one in bin\x64\Release and one in bin\x86\Release

Do I need my specific .nuspec still and if so, is there at least a way to not have to specify nuget packages that I depend on? Because the latter one is always a source for misconfiguration when you update a nuget package but forget to update the .nuspec as well. From  a net452 nuspec:

```<code><dependencies>
  <dependency id="NReco.VideoInfo" version="1.1.1" />
  <dependency id="System.ValueTuple" version="4.5.0" />
</dependencies>
</code>```

Body:
Currently I'm building a .NET 4.5.2 library in x86 and x64 and use the microsoft/setup-msbuild@v1 script on Github to build and NuGet/setup-nuget@v1.0.7 to deploy to Nuget, see pushPackageToNugetOrg.yml

For this I have my own .nuspec (ToolBox.nuspec) and .props file (DoenaSoft.ToolBox.props)

In a test balloon, I'm trying to migrate to a net6.0 windows project which in itself isn't hard. But I'm struggling with the Nuget configuration. By default it can be done in the .csproj file.

But if I configure it like this:

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup Label="Globals">
        <Configurations>Debug;Release</Configurations>
        <Platforms>x86;x64</Platforms>
    </PropertyGroup>

    <PropertyGroup>
        <TargetFrameworks>net452;net6.0-windows;net7.0-windows;</TargetFrameworks>
        <RootNamespace>DoenaSoft.ToolBox</RootNamespace>
        <AssemblyName>DoenaSoft.ToolBox</AssemblyName>
        <Version>1.0.0</Version>
        <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
    </PropertyGroup>

</Project>

(reduced for brevity) and then trigger the build then I end up with two different .nupkg files, one in bin\x64\Release and one in bin\x86\Release

Do I need my specific .nuspec still and if so, is there at least a way to not have to specify nuget packages that I depend on? Because the latter one is always a source for misconfiguration when you update a nuget package but forget to update the .nuspec as well. From  a net452 nuspec:

<dependencies>
  <dependency id="NReco.VideoInfo" version="1.1.1" />
  <dependency id="System.ValueTuple" version="4.5.0" />
</dependencies>


问:
Drag and drop Beautiful dnd
I'm building a web app with the drag and drop context but the issue is whenever the last element leaves one of the columns the particular column disappears

I used beautiful dnd, I tried using separate card components, I tried using different functions but all to no avail.. I want the column to remain even after removing the last draggable element out of it

Title: Drag and drop Beautiful dnd


问:
Random string in prefix and/or suffix of QR/Barcode data input from Barcode Reader using node-hid
I am trying to read qr code for my electron app using node-hid. But the decoded data shows additional random characters appended to the string.

```<code>  var HID = require('node-hid')
  var devices = HID.devices(); // Fetches list all the connected devices
  let device_path = '';
  for(const x of devices){
    if (x.product === 'HID POS'){ // To capture the device path of that device having product as 'HID POS'
      device_path = x.path;
    }
  }

  const qr_scanner = new HID.HID(device_path)
  qr_scanner.on('data', function(d) { // When QR is scanned with the Barcode reader
    console.log(d.toString());
  },
</code>```

With this code if I scan this QR:

Then the result is s1]Q1https://www.stackoverflow.com instead of https://www.stackoverflow.com.
Why is there a random set of characters in the decoded data?

Device Used : Honeywell Voyager 1450g

I have programmed the device for USB HID bar code scanners using this documentation

Node Version: 12.18.2

Electron Version: ^13.0.0

NOTE: I have also tried clearing prefix and suffix from the barcode reader using this documentation

Body:
I am trying to read qr code for my electron app using node-hid. But the decoded data shows additional random characters appended to the string.

  var HID = require('node-hid')
  var devices = HID.devices(); // Fetches list all the connected devices
  let device_path = '';
  for(const x of devices){
    if (x.product === 'HID POS'){ // To capture the device path of that device having product as 'HID POS'
      device_path = x.path;
    }
  }

  const qr_scanner = new HID.HID(device_path)
  qr_scanner.on('data', function(d) { // When QR is scanned with the Barcode reader
    console.log(d.toString());
  },

With this code if I scan this QR:

Then the result is s1]Q1https://www.stackoverflow.com instead of https://www.stackoverflow.com.
Why is there a random set of characters in the decoded data?

Device Used : Honeywell Voyager 1450g

I have programmed the device for USB HID bar code scanners using this documentation

Node Version: 12.18.2

Electron Version: ^13.0.0

NOTE: I have also tried clearing prefix and suffix from the barcode reader using this documentation


问:
Mybatis - Spring batch operations Not Working
im trying to batch insert a number of records using mybatis.

this is in my applicationContext

```<code> <bean id="refDataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
   <property name="driverClassName">
     <value>com.mysql.cj.jdbc.Driver</value>
   </property>
   <property name="url">
     <value>jdbc:mysql://localhost:3306/somedatabase?rewriteBatchedStatements=true</value>
   </property>
   <property name="username"><value>root</value></property>
   <property name="password"><value>root</value></property>
 </bean>
 <bean id="refSqlMapClientIbatis" class="org.mybatis.spring.SqlSessionFactoryBean">
   <property name="configLocation" value="/WEB-INF/jsp/ref/sql-map-config.xml" />
   <property name="dataSource" ref="refDataSource" />
 </bean>

 <bean id="refSqlSession" class="org.mybatis.spring.SqlSessionTemplate">
   <constructor-arg index="0" ref="refSqlMapClientIbatis" />
 </bean>
 <bean id="refSqlMapClient" class="mypackage.base.IbatisSqlMapClient">
   <property name="sqlSession" ref="refSqlSession" />
 </bean>

 <bean id="refBatchSqlSession" class="org.mybatis.spring.SqlSessionTemplate">
   <constructor-arg index="0" ref="refSqlMapClientIbatis" />
   <constructor-arg index="1" value="BATCH" />
 </bean>
 <bean id="refBatchSqlMapClient" class="mypackage.base.MybatisSqlSession">
   <property name="batchSqlSession" ref="refBatchSqlSession" />
 </bean>

 <bean id="RefBaseDAO" abstract="true">
   <property name="sqlMap" ref="refSqlMapClient"/>
   <property name="batchMap" ref="refBatchSqlMapClient"/>
 </bean>
 `
</code>```

when i use batchMap to insert inside of a loop all commands are executed on the spot without me calling flush .

this is the log file which shows that they're getting executed without me calling any execute or commit methods.

```<code> java.sql.PreparedStatement.addBatch: INSERT INTO  fz_ref_um_rates
        (um_rate_value, um_id, to_um_id, relation_between_prod_spec, um_rate_product_id, um_rate_product_assigned_spec_id, brch_id, created_by, create_date)
        VALUES
        (2.0, '952', '391', N, null, null, '284', 2, NOW());

[09/12/2022 11:48:52] org.slf4j.jul.JDK14LoggerAdapter.innerNormalizedLoggingCallHandler(JDK14LoggerAdapter.java:156)
 [INFO]: java.sql.Statement.executeBatch:

[09/12/2022 11:48:52] org.slf4j.jul.JDK14LoggerAdapter.innerNormalizedLoggingCallHandler(JDK14LoggerAdapter.java:156)
 [INFO]: java.sql.PreparedStatement.addBatch: INSERT INTO  fz_ref_um_rates
        (um_rate_value, um_id, to_um_id, relation_between_prod_spec, um_rate_product_id, um_rate_product_assigned_spec_id, brch_id, created_by, create_date)
        VALUES
        (3.0, '952', '339', N, null, null, '284', 2, NOW());

[09/12/2022 11:48:52] org.slf4j.jul.JDK14LoggerAdapter.innerNormalizedLoggingCallHandler(JDK14LoggerAdapter.java:156)
 [INFO]: java.sql.Statement.executeBatch:

[09/12/2022 11:48:52] org.slf4j.jul.JDK14LoggerAdapter.innerNormalizedLoggingCallHandler(JDK14LoggerAdapter.java:156)
 [INFO]: java.sql.PreparedStatement.addBatch: INSERT INTO  fz_ref_um_rates
        (um_rate_value, um_id, to_um_id, relation_between_prod_spec, um_rate_product_id, um_rate_product_assigned_spec_id, brch_id, created_by, create_date)
        VALUES
        (45.0, '952', '338', N, null, null, '284', 2, NOW());

[09/12/2022 11:48:52] org.slf4j.jul.JDK14LoggerAdapter.innerNormalizedLoggingCallHandler(JDK14LoggerAdapter.java:156)
 [INFO]: java.sql.Statement.executeBatch:

```
`

> this is my batchmap class

    public class MybatisSqlSession {
        private SqlSession batchSqlSession = null;

    public Object insert(String string, Object object) throws FZDAOException {
            try {
                System.out.println("insert called");
                return batchSqlSession.insert(string, object);

            } catch (SqlSessionException ex) {
                throw new MyException(ex.getMessage(), ex);
            }
        }
    }

> this is where im calling and getting the above logs :

    public void insertUmRates(List<UmRatesVO> addList) throws FZDAOException
        {
            for(UmRatesVO umRatesVO : addList)
            {
                batchMap.insert("UmRatesQueries.insertUmRates", umRatesVO);
            }
        }

if any further code is needed let me know

```

Body:
im trying to batch insert a number of records using mybatis.

this is in my applicationContext

 <bean id="refDataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
   <property name="driverClassName">
     <value>com.mysql.cj.jdbc.Driver</value>
   </property>
   <property name="url">
     <value>jdbc:mysql://localhost:3306/somedatabase?rewriteBatchedStatements=true</value>
   </property>
   <property name="username"><value>root</value></property>
   <property name="password"><value>root</value></property>
 </bean>
 <bean id="refSqlMapClientIbatis" class="org.mybatis.spring.SqlSessionFactoryBean">
   <property name="configLocation" value="/WEB-INF/jsp/ref/sql-map-config.xml" />
   <property name="dataSource" ref="refDataSource" />
 </bean>

 <bean id="refSqlSession" class="org.mybatis.spring.SqlSessionTemplate">
   <constructor-arg index="0" ref="refSqlMapClientIbatis" />
 </bean>
 <bean id="refSqlMapClient" class="mypackage.base.IbatisSqlMapClient">
   <property name="sqlSession" ref="refSqlSession" />
 </bean>

 <bean id="refBatchSqlSession" class="org.mybatis.spring.SqlSessionTemplate">
   <constructor-arg index="0" ref="refSqlMapClientIbatis" />
   <constructor-arg index="1" value="BATCH" />
 </bean>
 <bean id="refBatchSqlMapClient" class="mypackage.base.MybatisSqlSession">
   <property name="batchSqlSession" ref="refBatchSqlSession" />
 </bean>

 <bean id="RefBaseDAO" abstract="true">
   <property name="sqlMap" ref="refSqlMapClient"/>
   <property name="batchMap" ref="refBatchSqlMapClient"/>
 </bean>
 `

when i use batchMap to insert inside of a loop all commands are executed on the spot without me calling flush .

this is the log file which shows that they're getting executed without me calling any execute or commit methods.

 java.sql.PreparedStatement.addBatch: INSERT INTO  fz_ref_um_rates
        (um_rate_value, um_id, to_um_id, relation_between_prod_spec, um_rate_product_id, um_rate_product_assigned_spec_id, brch_id, created_by, create_date)
        VALUES
        (2.0, '952', '391', N, null, null, '284', 2, NOW());

[09/12/2022 11:48:52] org.slf4j.jul.JDK14LoggerAdapter.innerNormalizedLoggingCallHandler(JDK14LoggerAdapter.java:156)
 [INFO]: java.sql.Statement.executeBatch:

[09/12/2022 11:48:52] org.slf4j.jul.JDK14LoggerAdapter.innerNormalizedLoggingCallHandler(JDK14LoggerAdapter.java:156)
 [INFO]: java.sql.PreparedStatement.addBatch: INSERT INTO  fz_ref_um_rates
        (um_rate_value, um_id, to_um_id, relation_between_prod_spec, um_rate_product_id, um_rate_product_assigned_spec_id, brch_id, created_by, create_date)
        VALUES
        (3.0, '952', '339', N, null, null, '284', 2, NOW());

[09/12/2022 11:48:52] org.slf4j.jul.JDK14LoggerAdapter.innerNormalizedLoggingCallHandler(JDK14LoggerAdapter.java:156)
 [INFO]: java.sql.Statement.executeBatch:

[09/12/2022 11:48:52] org.slf4j.jul.JDK14LoggerAdapter.innerNormalizedLoggingCallHandler(JDK14LoggerAdapter.java:156)
 [INFO]: java.sql.PreparedStatement.addBatch: INSERT INTO  fz_ref_um_rates
        (um_rate_value, um_id, to_um_id, relation_between_prod_spec, um_rate_product_id, um_rate_product_assigned_spec_id, brch_id, created_by, create_date)
        VALUES
        (45.0, '952', '338', N, null, null, '284', 2, NOW());

[09/12/2022 11:48:52] org.slf4j.jul.JDK14LoggerAdapter.innerNormalizedLoggingCallHandler(JDK14LoggerAdapter.java:156)
 [INFO]: java.sql.Statement.executeBatch:

```
`

> this is my batchmap class

    public class MybatisSqlSession {
        private SqlSession batchSqlSession = null;

    public Object insert(String string, Object object) throws FZDAOException {
            try {
                System.out.println("insert called");
                return batchSqlSession.insert(string, object);

            } catch (SqlSessionException ex) {
                throw new MyException(ex.getMessage(), ex);
            }
        }
    }

> this is where im calling and getting the above logs :

    public void insertUmRates(List<UmRatesVO> addList) throws FZDAOException
        {
            for(UmRatesVO umRatesVO : addList)
            {
                batchMap.insert("UmRatesQueries.insertUmRates", umRatesVO);
            }
        }

if any further code is needed let me know


问:
Spark GroupBy on nested struct in Java
I have a DataSet in spark that has the following schema:

```<code>root
 |-- _1: struct (nullable = true)
 |    |-- A: map (nullable = true)
 |    |    |-- key: string
 |    |    |-- value: array (valueContainsNull = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |-- B: struct (nullable = true)
 |    |    |-- BA: struct (nullable = true)
 |    |    |    |-- affluence: string (nullable = true)
 |    |-- C: string (nullable = true)
 |    |-- D: struct (nullable = true)
 |    |    |-- DA: integer (nullable = true)
 |    |    |-- DB: integer (nullable = true)
 |    |    |-- DC: integer (nullable = true)
 |    |    |-- DD: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DE: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DF: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DG: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DH: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DI: integer (nullable = true)
 |    |-- E: string (nullable = true)
 |-- _2: struct (nullable = true)
 |    |-- A: map (nullable = true)
 |    |    |-- key: string
 |    |    |-- value: array (valueContainsNull = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |-- B: struct (nullable = true)
 |    |    |-- BA: struct (nullable = true)
 |    |    |    |-- affluence: string (nullable = true)
 |    |-- C: string (nullable = true)
 |    |-- D: struct (nullable = true)
 |    |    |-- DA: integer (nullable = true)
 |    |    |-- DB: integer (nullable = true)
 |    |    |-- DC: integer (nullable = true)
 |    |    |-- DD: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DE: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DF: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DG: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DH: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DI: integer (nullable = true)
 |    |-- E: string (nullable = true)
</code>```

I want to run a group by on this dataset on the entire struct _1, except the column _1.D.

What is the simplest and most elegant solution to this? When I try to do it, it says:

```<code>... cannot be used as a grouping expression because its data type is not an orderable data type.;
</code>```

P.S. I know I explode each column individually and then group by on all of them, but that is cumbersome, error-prone and will not result in clean code. I want a clean solution to this.

Title: Spark GroupBy on nested struct in Java

Body:
I have a DataSet in spark that has the following schema:

root
 |-- _1: struct (nullable = true)
 |    |-- A: map (nullable = true)
 |    |    |-- key: string
 |    |    |-- value: array (valueContainsNull = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |-- B: struct (nullable = true)
 |    |    |-- BA: struct (nullable = true)
 |    |    |    |-- affluence: string (nullable = true)
 |    |-- C: string (nullable = true)
 |    |-- D: struct (nullable = true)
 |    |    |-- DA: integer (nullable = true)
 |    |    |-- DB: integer (nullable = true)
 |    |    |-- DC: integer (nullable = true)
 |    |    |-- DD: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DE: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DF: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DG: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DH: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DI: integer (nullable = true)
 |    |-- E: string (nullable = true)
 |-- _2: struct (nullable = true)
 |    |-- A: map (nullable = true)
 |    |    |-- key: string
 |    |    |-- value: array (valueContainsNull = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |-- B: struct (nullable = true)
 |    |    |-- BA: struct (nullable = true)
 |    |    |    |-- affluence: string (nullable = true)
 |    |-- C: string (nullable = true)
 |    |-- D: struct (nullable = true)
 |    |    |-- DA: integer (nullable = true)
 |    |    |-- DB: integer (nullable = true)
 |    |    |-- DC: integer (nullable = true)
 |    |    |-- DD: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DE: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DF: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DG: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DH: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- DI: integer (nullable = true)
 |    |-- E: string (nullable = true)

I want to run a group by on this dataset on the entire struct _1, except the column _1.D.

What is the simplest and most elegant solution to this? When I try to do it, it says:

... cannot be used as a grouping expression because its data type is not an orderable data type.;

P.S. I know I explode each column individually and then group by on all of them, but that is cumbersome, error-prone and will not result in clean code. I want a clean solution to this.


问:
How can I get recursion data with pagination
I have a table category

id
name
parent_id

1
blogs
0

2
activity news
0

3
recruitment
0

4
technology
1

5
sport
1

6
company culture
2

7
entertainment
1

8
IT
3

9
art
1

10
soccer
5

11
table tennis
5

12
accountant
3

How can I query recursion data with pagination.
Ex: limit = 5, I want the output to look like this

I get all the data and then paginated them, but that's not ok with a lot of data


问:
Getting checked box checked/unchecked only when attachment gets added/deleted from Workorder app only
I have a requirement to get a checkbox checked/unchecked only and only when any attachment gets added/ deleted from the wo application.

I am new to automation script and in learning phase, thus, will be very grateful if you reply to my message

I have written the automation script but it fails when the WO has more than 1 attachment (attached from WO APP itself) , then removing any 1 attachment, unchecks the checkbox, but this should not be the case. The checkbox should only be zero, if there are no attachments left from WO app(which were uploaded from wo app). I have created a relationship between doclinks and workorder (child object).

I understand that a i need to take count of no of attachment from wo ,but how do i do that?


问:
Mongodb partitioning collection divided into 12 collection within a database
I'm using mongoDB as a backend. The collection which I have 1B documents and more, I want to Partitioning the collection to 12month collection. JanCollection , FebCollection .... based on Month_of_sale as a key. How can we create a 12 collection as a chunks within a database.

In this case, WE can use 12 different collection on db. and Insert and update the document on that collection based on MonthOfSale. For example. If the MonthOfSale is 2, insert/update into the FebCollection. If we search data, There also we find the document on particular collection by MonthOfSale. My question is Any other efficient way to handle this situation in php and MongoDB.


问:
7.0.100/NuGet.targets(396,5): error MSB3202: The project file "/src/WebApplication1/WebApplication1.csproj" was not found. [/src/MyMicroservice.sln]
#11 0.734 /usr/share/dotnet/sdk/7.0.100/NuGet.targets(396,5): error MSB3202: The project file "/src/WebApplication1/WebApplication1.csproj" was not found. [/src/MyMicroservice.sln]

tried changing its location but it was no good