问:
Checkbox classList Toggle
I'm trying to use a custom slider with an input type=checkbox to toggle between dark-mode and light-mode. It is not working with the javascript code I have.

var checkbox = document.getElementById('checkbox');
    checkbox.addEventListener('change', themeChange);

    function themeChange(){
      if (checkbox.checked == true){
        document.getElementById('body').classList.toggle('dark-mode');
      } else {
        document.getElementById('body').classList.toggle('light-mode');
      }
    }```
<pre class="snippet-code-css lang-css prettyprint-override"><code>.dark-mode {
      background-color: black;
      color: white;
    }

    .light-mode {
      background-color: white;
      color: black;
    }</code>```

       <input id="checkbox" type="checkbox" >

    ```

答:
I noticed some problem:

You must wrap all elements in the  element.

You don't need to give the id attribute to select the  element.

Before you define a function, you should not bind it to an event. First define the function then bind the predefined function to an event.

For the toggle() method to work properly, you must apply the .light-mode class style to the  element by default.

You don't need to check the checkbox as it is a toggle check.

var checkbox = document.getElementById('checkbox');

function themeChange() {
  document.body.classList.toggle("light-mode");
  document.body.classList.toggle("dark-mode");
}

checkbox.addEventListener('change', themeChange);```
<pre class="snippet-code-css lang-css prettyprint-override"><code>.dark-mode {
  background-color: black;
  color: white;
}
.light-mode {
  background-color: white;
  color: black;
}</code>```

    <input id="checkbox" type="checkbox" >

```
</div>
</div>
</p>
<p>To apply the <code>toggle()</code> method to multiple class styles simultaneously, you can update the <code>themeChange()</code> method as follows:</p>
<pre class="lang-js prettyprint-override"><code>function themeChange() {
  ["light-mode", "dark-mode"].map(cn => document.body.classList.toggle(cn));
}
</code>```

Title: Checkbox classList Toggle

Body:
I'm trying to use a custom slider with an input type=checkbox to toggle between dark-mode and light-mode. It is not working with the javascript code I have.

var checkbox = document.getElementById('checkbox');
    checkbox.addEventListener('change', themeChange);

    function themeChange(){
      if (checkbox.checked == true){
        document.getElementById('body').classList.toggle('dark-mode');
      } else {
        document.getElementById('body').classList.toggle('light-mode');
      }
    }

.dark-mode {
      background-color: black;
      color: white;
    }

    .light-mode {
      background-color: white;
      color: black;
    }

       <input id="checkbox" type="checkbox" >

Body:
I noticed some problem:

You must wrap all elements in the <body> element.

You don't need to give the id attribute to select the <body> element.

Before you define a function, you should not bind it to an event. First define the function then bind the predefined function to an event.

For the toggle() method to work properly, you must apply the .light-mode class style to the <body> element by default.

You don't need to check the checkbox as it is a toggle check.

var checkbox = document.getElementById('checkbox');

function themeChange() {
  document.body.classList.toggle("light-mode");
  document.body.classList.toggle("dark-mode");
}

checkbox.addEventListener('change', themeChange);

.dark-mode {
  background-color: black;
  color: white;
}
.light-mode {
  background-color: white;
  color: black;
}

<!-- By default, the "light-mode" style was applied to the <body> element. -->
<body class="light-mode">
  <label class="switch">
    <input id="checkbox" type="checkbox" >
    <span class="slider round"></span>
  </label>
</body>

To apply the toggle() method to multiple class styles simultaneously, you can update the themeChange() method as follows:

function themeChange() {
  ["light-mode", "dark-mode"].map(cn => document.body.classList.toggle(cn));
}


来源: stackexchange

问:
How does asking an procedure to run in ask patches produce a different output than running the patches as ask patches?
I was working on a NetLogo project for school, when I ran into this issue with my NetLogo code. When I first made the code, I put the procedure death-check in the ask patches command with this being my program:

```<code>patches-own [num-live]

to setup ;;this sets up all of the code
  clear-all
  ask patches [
    if random 100 < percent-alive [
      cell-birth
    ]
  ]
  reset-ticks
end

to go
  ask patches [
    set num-live count neighbors with [
      pcolor = yellow
    ]
    death-check
  ]
  tick
end

to cell-birth
  set pcolor yellow
end

to cell-death
  set pcolor black
end

to death-check
  if num-live < 2 or num-live > 3 [
    cell-death
  ]
  if num-live = 3 [
  cell-birth
  ]
end
</code>```

And I got this output

Now if I put the ask patches inside the procedure, it comes out with a different output. Here is what I changed inside the code:

```<code>to go
  ask patches [
    set num-live count neighbors with [
      pcolor = yellow
    ]
  ]
  death-check
  tick
end

to death-check
  ask patches [
    if num-live < 2 or num-live > 3 [
      cell-death
    ]
    if num-live = 3 [
      cell-birth
    ]
  ]
end
</code>```

With this being the output

What I was wondering was why did this small change make such a big difference for my program?

答:
The difference here is that in one instance, you are calling ask twice, whereas in the other you are only calling ask once. Consider  two examples, which I'm calling go-1-ask and go-2-ask:

go-1-ask:

```<code>to go-1-ask
  ask patches [
    set num-live count neighbors with [
      pcolor = yellow
    ]
    if num-live < 2 or num-live > 3 [
      cell-death
    ]
    if num-live = 3 [
      cell-birth
    ]
  ]
  tick
end
</code>```

go-2-ask:

```<code>to go-2-ask
  ask patches [
    set num-live count neighbors with [
      pcolor = yellow
    ]
  ]
  ask patches [
    if num-live < 2 or num-live > 3 [
      cell-death
    ]
    if num-live = 3 [
      cell-birth
    ]
  ]
  tick
end
</code>```

All I've done relative to your code is nest the code from each different death-check examples within the go procedure.

In go-1-ask, you are asking all patches, in a random order, to set num-live then to evaluate it. So, patch 43 will run the code, then maybe patch 71, and so on. In contrast, in go-2-ask, you are first asking all patches to assess their num-live. Only then, once all patches have performed that first check, are you performing the death-check. In short- the order of operations is different between the two methods, which accounts for the difference in behaviour.

ContentLicense: CC BY-SA 4.0

Body:
I was working on a NetLogo project for school, when I ran into this issue with my NetLogo code. When I first made the code, I put the procedure death-check in the ask patches command with this being my program:

patches-own [num-live]

to setup ;;this sets up all of the code
  clear-all
  ask patches [
    if random 100 < percent-alive [
      cell-birth
    ]
  ]
  reset-ticks
end

to go
  ask patches [
    set num-live count neighbors with [
      pcolor = yellow
    ]
    death-check
  ]
  tick
end

to cell-birth
  set pcolor yellow
end

to cell-death
  set pcolor black
end

to death-check
  if num-live < 2 or num-live > 3 [
    cell-death
  ]
  if num-live = 3 [
  cell-birth
  ]
end

And I got this output

Now if I put the ask patches inside the procedure, it comes out with a different output. Here is what I changed inside the code:

to go
  ask patches [
    set num-live count neighbors with [
      pcolor = yellow
    ]
  ]
  death-check
  tick
end

to death-check
  ask patches [
    if num-live < 2 or num-live > 3 [
      cell-death
    ]
    if num-live = 3 [
      cell-birth
    ]
  ]
end

With this being the output

What I was wondering was why did this small change make such a big difference for my program?

Body:
The difference here is that in one instance, you are calling ask twice, whereas in the other you are only calling ask once. Consider  two examples, which I'm calling go-1-ask and go-2-ask:

go-1-ask:

to go-1-ask
  ask patches [
    set num-live count neighbors with [
      pcolor = yellow
    ]
    if num-live < 2 or num-live > 3 [
      cell-death
    ]
    if num-live = 3 [
      cell-birth
    ]
  ]
  tick
end

go-2-ask:

to go-2-ask
  ask patches [
    set num-live count neighbors with [
      pcolor = yellow
    ]
  ]
  ask patches [
    if num-live < 2 or num-live > 3 [
      cell-death
    ]
    if num-live = 3 [
      cell-birth
    ]
  ]
  tick
end

All I've done relative to your code is nest the code from each different death-check examples within the go procedure.

In go-1-ask, you are asking all patches, in a random order, to set num-live then to evaluate it. So, patch 43 will run the code, then maybe patch 71, and so on. In contrast, in go-2-ask, you are first asking all patches to assess their num-live. Only then, once all patches have performed that first check, are you performing the death-check. In short- the order of operations is different between the two methods, which accounts for the difference in behaviour.


问:
How to force the android system to open only my app based on the registered intent filter
Issue:
I've been working on an app that listens to a certain kind of intents depending on a given link format. My intent filter works as expected, I'm just wondering whether it's possible to force the android system to bypass the kind of dialog below and open only my app.

答:
I'm just wondering whether it's possible to force the android system to bypass the kind of dialog below and open only my app.

Do you own the domain of the URI that is being invoked? If so, than yes, with app links.

If not, then no.


问:
C# serialize only values into a byte stream
Is there any possibility to automatically serialize properties of a class into a byte[] array or stream.

```<code>Stream stream = File.Open(@"C:/traiBin.bin", FileMode.Create);
BinaryFormatter bFormatter = new BinaryFormatter();
bFormatter.Serialize(stream, this.rcpt);
stream.Close();
</code>```

This example above serialized the whole class including namespaces, class name, variable names etc.
If there is no automated solution it will do it by hand.

Regards,
Mark

答: Marc Gravell's protobuf implementation supports the attributed model, as well as DataContracts - protobuf is also pretty lean when it comes the final size of the data.</p>

ContentLicense: CC BY-SA 3.0

Body:
Is there any possibility to automatically serialize properties of a class into a byte[] array or stream.

Stream stream = File.Open(@"C:/traiBin.bin", FileMode.Create);
BinaryFormatter bFormatter = new BinaryFormatter();
bFormatter.Serialize(stream, this.rcpt);
stream.Close();

This example above serialized the whole class including namespaces, class name, variable names etc.
If there is no automated solution it will do it by hand.

Regards,
Mark


问:
updating nodejs on ubuntu 16.04
I was recently going through the version of node in my ubuntu 16.04 when node -v command was used it shows me version 6.9.1 but when nodejs -v it shows 6.9.2 previously before using this commands npm update command was used.

Now what's these difference in node -v and nodejs -v? and how to update to the latest LTS version of node/nodejs?

答:
I am also facing problem while going to install react app, so i found the solution,

```<code>npx create-react-app shodkk
</code>```

First install the npm latest version using

```<code>sudo npm install -g npm@8.4.1
</code>```

So to install the  node 16.x you need to go terminal and type

```<code>curl -fsSL https://deb.nodesource.com/setup_16.x | sudo -E bash -
</code>```

Doing this you install the node LTS that is 16.14.o at the time of writing this post.

Try this 2-3 times, also do the

```<code>sudo apt-get update
</code>```

Then Now install the package using

```<code>sudo apt-get install -y nodejs
</code>```

At last this helps you remove any unwanted package, that remains after updating that is depreciated and need not be there, So use autoremove command.
sudo apt autoremove

So if like the post, Upvote and motivate me to write more, thanks, givingBack to the Community.

Title: updating nodejs on ubuntu 16.04

Body:
I am also facing problem while going to install react app, so i found the solution,

npx create-react-app shodkk

First install the npm latest version using

sudo npm install -g npm@8.4.1

So to install the  node 16.x you need to go terminal and type

curl -fsSL https://deb.nodesource.com/setup_16.x | sudo -E bash -

Doing this you install the node LTS that is 16.14.o at the time of writing this post.

Try this 2-3 times, also do the

sudo apt-get update

Then Now install the package using

sudo apt-get install -y nodejs

At last this helps you remove any unwanted package, that remains after updating that is depreciated and need not be there, So use autoremove command.
sudo apt autoremove

So if like the post, Upvote and motivate me to write more, thanks, givingBack to the Community.


问:
How can I get the pixel color value of an image in javascript?
How can I get the pixel color value of an image in javascript?

I have an image like this:

Click on the image to create a red circle.

so that a red circle is created in the forest,
I don't want the red circle to be created outside the forest.

For this, I want to compare the backgroundcolor and the background value of the red circle created by clicking.

I am wondering how can I get the pixel color value of the clicked image.

答:
There is a whole section on HTML spec for image manipulation.
https://html.spec.whatwg.org/multipage/canvas.html#pixel-manipulation

But concretely, one possibility is to use the getImageData function : https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial/Pixel_manipulation_with_canvas#getting_the_pixel_data_for_a_context

This article helped me a lot for this kind of thing : https://hacks.mozilla.org/2011/12/faster-canvas-pixel-manipulation-with-typed-arrays/

Here my example of a basic implementation (I'm using Typescript for make it simpler):

```<code>function GetColorPixel(
  context: CanvasRenderingContext2D,
  x: number,
  y: number
): Array<number> {
  const imageData: Uint8ClampedArray = context.getImageData(x, y, 1, 1).data
  const rgbColor: Array<number> = [imageData[0], imageData[1], imageData[2]]
  return rgbColor
}
</code>```

In this function, you need to pass the context of canvas for example

Body:
There is a whole section on HTML spec for image manipulation.
https://html.spec.whatwg.org/multipage/canvas.html#pixel-manipulation

But concretely, one possibility is to use the getImageData function : https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial/Pixel_manipulation_with_canvas#getting_the_pixel_data_for_a_context

This article helped me a lot for this kind of thing : https://hacks.mozilla.org/2011/12/faster-canvas-pixel-manipulation-with-typed-arrays/

Here my example of a basic implementation (I'm using Typescript for make it simpler):

function GetColorPixel(
  context: CanvasRenderingContext2D,
  x: number,
  y: number
): Array<number> {
  const imageData: Uint8ClampedArray = context.getImageData(x, y, 1, 1).data
  const rgbColor: Array<number> = [imageData[0], imageData[1], imageData[2]]
  return rgbColor
}

In this function, you need to pass the context of canvas for example


问:
Jekyll Serve command not working - Errors include MissingDependencyException and MissingSpecError
As the title above says, I'm trying to build a site on localhost with jekyll serve but I keep getting several errors.

I've done all the steps described here to build my site, but nothing seems to work. I've tried several troubleshooting sources I found online, as well.

Also, the directory I'm trying to build with is a local GitHub repository.

Any suggestions on what I should do?

答:
Nvm, I fixed it.

I ended up deleting my local repository and cloning it remotely from GitHub. That didn't solve it immediately, but the Gemfile on my local repository was from before I started having issues.

After that, I then kept getting an error where the version of Jekyll was incompatible with building GitHub pages. This page had the answer I needed - which was just changing the version number for gem "jekyll" in the Gemfile for the directory.

Finally, I ran a bundle exec jekyll serve and I got my localhost to run!


问:
How to read Meta data in gRPC using Java at client side
I am using Java and Protoc 3.0 compiler and my proto file is mention below.
    https://github.com/openconfig/public/blob/master/release/models/rpc/openconfig-rpc-api.yang

```<code>syntax = "proto3";

package Telemetry;

// Interface exported by Agent
service OpenConfigTelemetry {
    // Request an inline subscription for data at the specified path.
    // The device should send telemetry data back on the same
    // connection as the subscription request.
    rpc telemetrySubscribe(SubscriptionRequest)                     returns (stream OpenConfigData) {}

    // Terminates and removes an exisiting telemetry subscription
    rpc cancelTelemetrySubscription(CancelSubscriptionRequest)      returns (CancelSubscriptionReply) {}

    // Get the list of current telemetry subscriptions from the
    // target. This command returns a list of existing subscriptions
    // not including those that are established via configuration.
    rpc getTelemetrySubscriptions(GetSubscriptionsRequest)          returns (GetSubscriptionsReply) {}

    // Get Telemetry Agent Operational States
    rpc getTelemetryOperationalState(GetOperationalStateRequest)    returns (GetOperationalStateReply) {}

    // Return the set of data encodings supported by the device for
    // telemetry data
    rpc getDataEncodings(DataEncodingRequest)                       returns (DataEncodingReply) {}
}

// Message sent for a telemetry subscription request
message SubscriptionRequest {
    // Data associated with a telemetry subscription
    SubscriptionInput input                                 = 1;

    // List of data models paths and filters
    // which are used in a telemetry operation.
    repeated Path path_list                                 = 2;

    // The below configuration is not defined in Openconfig RPC.
    // It is a proposed extension to configure additional
    // subscription request features.
    SubscriptionAdditionalConfig additional_config          = 3;
}

// Data associated with a telemetry subscription
message SubscriptionInput {
    // List of optional collector endpoints to send data for
    // this subscription.
    // If no collector destinations are specified, the collector
    // destination is assumed to be the requester on the rpc channel.
    repeated Collector  collector_list                      = 1;
}

// Collector endpoints to send data specified as an ip+port combination.
message Collector {
    // IP address of collector endpoint
    string address                                          = 1;

    // Transport protocol port number for the collector destination.
    uint32 port                                             = 2;
}

// Data model path
message Path {
    // Data model path of interest
    // Path specification for elements of OpenConfig data models
    string path                                             = 1;

    // Regular expression to be used in filtering state leaves
    string filter                                           = 2;

    // If this is set to true, the target device will only send
    // updates to the collector upon a change in data value
    bool suppress_unchanged                                 = 3;

    // Maximum time in ms the target device may go without sending
    // a message to the collector. If this time expires with
    // suppress-unchanged set, the target device must send an update
    // message regardless if the data values have changed.
    uint32 max_silent_interval                              = 4;

    // Time in ms between collection and transmission of the
    // specified data to the collector platform. The target device
    // will sample the corresponding data (e.g,. a counter) and
    // immediately send to the collector destination.
    //
    // If sample-frequency is set to 0, then the network device
    // must emit an update upon every datum change.
    uint32 sample_frequency                                 = 5;
}

// Configure subscription request additional features.
message SubscriptionAdditionalConfig {
    // limit the number of records sent in the stream
    int32 limit_records                                     = 1;

    // limit the time the stream remains open
    int32 limit_time_seconds                                = 2;
}

// Reply to inline subscription for data at the specified path is done in
// two-folds.
// 1. Reply data message sent out using out-of-band channel.
// 2. Telemetry data send back on the same connection as the
//    subscription request.

// 1. Reply data message sent out using out-of-band channel.
message SubscriptionReply {
    // Response message to a telemetry subscription creation or
    // get request.
    SubscriptionResponse response                           = 1;

    // List of data models paths and filters
    // which are used in a telemetry operation.
    repeated Path path_list                                 = 2;
}

// Response message to a telemetry subscription creation or get request.
message SubscriptionResponse {
    // Unique id for the subscription on the device. This is
    // generated by the device and returned in a subscription
    // request or when listing existing subscriptions
    uint32 subscription_id = 1;
}

// 2. Telemetry data send back on the same connection as the
//    subscription request.
message OpenConfigData {
    // router name:export IP address
    string system_id                                        = 1;

    // line card / RE (slot number)
    uint32 component_id                                     = 2;

    // PFE (if applicable)
    uint32 sub_component_id                                 = 3;

    // Path specification for elements of OpenConfig data models
    string path                                             = 4;

    // Sequence number, monotonically increasing for each
    // system_id, component_id, sub_component_id + path.
    uint64 sequence_number                                  = 5;

    // timestamp (milliseconds since epoch)
    uint64 timestamp                                        = 6;

    // List of key-value pairs
    repeated KeyValue kv                                    = 7;
}

// Simple Key-value, where value could be one of scalar types
message KeyValue {
    // Key
    string key                                              =  1;

    // One of possible values
    oneof value {
        double double_value                                 =  5;
        int64  int_value                                    =  6;
        uint64 uint_value                                   =  7;
        sint64 sint_value                                   =  8;
        bool   bool_value                                   =  9;
        string str_value                                    = 10;
        bytes  bytes_value                                  = 11;
    }
}

// Message sent for a telemetry subscription cancellation request
message CancelSubscriptionRequest {
    // Subscription identifier as returned by the device when
    // subscription was requested
    uint32 subscription_id                                  = 1;
}

// Reply to telemetry subscription cancellation request
message CancelSubscriptionReply {
    // Return code
    ReturnCode code                                         = 1;

    // Return code string
    string     code_str                                     = 2;
};

// Result of the operation
enum ReturnCode {
    SUCCESS                                                 = 0;
    NO_SUBSCRIPTION_ENTRY                                   = 1;
    UNKNOWN_ERROR                                           = 2;
}

// Message sent for a telemetry get request
message GetSubscriptionsRequest {
    // Subscription identifier as returned by the device when
    // subscription was requested
    // --- or ---
    // 0xFFFFFFFF for all subscription identifiers
    uint32 subscription_id                                  = 1;
}

// Reply to telemetry subscription get request
message GetSubscriptionsReply {
    // List of current telemetry subscriptions
    repeated SubscriptionReply subscription_list            = 1;
}

// Message sent for telemetry agent operational states request
message GetOperationalStateRequest {
    // Per-subscription_id level operational state can be requested.
    //
    // Subscription identifier as returned by the device when
    // subscription was requested
    // --- or ---
    // 0xFFFFFFFF for all subscription identifiers including agent-level
    // operational stats
    // --- or ---
    // If subscription_id is not present then sent only agent-level
    // operational stats
    uint32 subscription_id                                  = 1;

    // Control verbosity of the output
    VerbosityLevel verbosity                                = 2;
}

// Verbosity Level
enum VerbosityLevel {
    DETAIL                                                  = 0;
    TERSE                                                   = 1;
    BRIEF                                                   = 2;
}

// Reply to telemetry agent operational states request
message GetOperationalStateReply {
    // List of key-value pairs where
    //     key      = operational state definition
    //     value    = operational state value
    repeated KeyValue kv                                    = 1;
}

// Message sent for a data encoding request
message DataEncodingRequest {
}

// Reply to data encodings supported request
message DataEncodingReply {
    repeated EncodingType  encoding_list                    = 1;
}

// Encoding Type Supported
enum EncodingType {
    UNDEFINED                                               = 0;
    XML                                                     = 1;
    JSON_IETF                                               = 2;
    PROTO3                                                  = 3;
}
</code>```

In order to do the service call (rpc TelemetrySubscribe) first i need to read header which have subscription id and then start reading messages. Now, using Java i am able to connect with the service, i did introduce the interceptor but when i print/retrieve header it is null. My code of calling interceptor is below,

```<code> ClientInterceptor interceptor = new HeaderClientInterceptor();
      originChannel = OkHttpChannelBuilder.forAddress(host, port)
        .usePlaintext(true)
        .build();
     Channel channel =  ClientInterceptors.intercept(originChannel, interceptor);
      telemetryStub = OpenConfigTelemetryGrpc.newStub(channel);
</code>```

This is interceptor code to read meta Data.

```<code>  @Override
  public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(MethodDescriptor<ReqT, RespT> method,
      CallOptions callOptions, Channel next) {
    return new SimpleForwardingClientCall<ReqT, RespT>(next.newCall(method, callOptions)) {

      @Override
      public void start(Listener<RespT> responseListener, Metadata headers) {

        super.start(new SimpleForwardingClientCallListener<RespT>(responseListener) {
          @Override
          public void onHeaders(Metadata headers) {

             Key<String> CUSTOM_HEADER_KEY = Metadata.Key.of("responseKEY", Metadata.ASCII_STRING_MARSHALLER);

            System.out.println("Contains Key?? "+headers.containsKey(CUSTOM_HEADER_KEY));
</code>```

Wondering is there any other way to read meta data or first message which have subscription ID in it? All i need to read first message which have subscription Id, and return the same subscription id to server so that streaming can start I have equivalent Python code using same proto file and it is communicating with server by code mention below for reference only:

```<code>     sub_req = SubscribeRequestMsg("host",port)
     data_itr = stub.telemetrySubscribe(sub_req, _TIMEOUT_SECONDS)
     metadata = data_itr.initial_metadata()

                   if metadata[0][0] == "responseKey":
                    metainfo = metadata[0][1]
                    print metainfo

                    subreply = agent_pb2.SubscriptionReply()
                    subreply.SetInParent()
                    google.protobuf.text_format.Merge(metainfo, subreply)

                    if subreply.response.subscription_id:
                    SUB_ID = subreply.response.subscription_id
</code>```

From the python code above i can easily retrieve meta data object, not sure how to retrieve same using Java?

After reading metaData all i am getting is: Metadata({content-type=[application/grpc], grpc-encoding=[identity], grpc-accept-encoding=[identity,deflate,gzip]})

But i know there is one more line from meta data to it, which is

```<code>response {
  subscription_id: 2
}
</code>```

How can i extract last response from Header which have subscription id in it. I did try many options and i am lost here.

答:
In case this helps anyone else out:

I needed to write a specific response header in my gRPC-java server based upon the request/response.

What I ended up doing was storing the response header value in a Context using the Context::withValue (which doesn't modify the existing context but instead creates a new Context actually), and then calling the request service handler method's StreamObserver::onNext inside of the Context::run callback. StremableObserver::onNext calls the ServerCall::sendHeaders I have in my set from my ServerInterceptor. There in the sendHeaders, it can read the value in the Context I stored and set the response header value.

I think this is similar to @carl-mastrangelo's approach, just may be spelled out a little bit more.

public enum MyServerInterceptor implements ServerInterceptor {
    INSTANCE;

    public static final Metadata.Key<String> METADATA_KEY =
            Metadata.Key.of("fish", ASCII_STRING_MARSHALLER);

    public static final Context.Key<String> CONTEXT_KEY = Context.key("dog");

    @Override
    public <ReqT, RespT> ServerCall.Listener<ReqT> interceptCall(ServerCall<ReqT, RespT> call,
                                                                 Metadata requestHeaders,
                                                                 ServerCallHandler<ReqT, RespT> next) {

        ServerCall<ReqT, RespT> myServerCall = new ForwardingServerCall.SimpleForwardingServerCall<ReqT, RespT>(call) {
            @Override
            public void sendHeaders(Metadata responseHeaders) {
                String value = CONTEXT_KEY.get(Context.current());
                responseHeaders.put(METADATA_KEY, value);
                super.sendHeaders(responseHeaders);
            }
        };

        return next.startCall(myServerCall, requestHeaders);
    }
}

public class MyService extends MyServiceGrpc.MyServiceyImplBase {
    @Override
    public void serviceMethod(MyRequest request, StreamObserver<MyResponse> responseObserver) {
        MyResponse response = new MyResponse();

        Context.current().withValue(Context.key("dog"), "cat").run(() -> {
            responseObserver.onNext(response);
        });

        responseObserver.onCompleted();
    }
}
```

The again the critical piece here is that responseObserver::onNext  calls into ForwardingServerCall.SimpleForwardingServerCall::sendHeaders.

Please let me know if there is a better way. This all seems more complex than I'd like.

Body:
I am using Java and Protoc 3.0 compiler and my proto file is mention below.
    https://github.com/openconfig/public/blob/master/release/models/rpc/openconfig-rpc-api.yang

syntax = "proto3";

package Telemetry;

// Interface exported by Agent
service OpenConfigTelemetry {
    // Request an inline subscription for data at the specified path.
    // The device should send telemetry data back on the same
    // connection as the subscription request.
    rpc telemetrySubscribe(SubscriptionRequest)                     returns (stream OpenConfigData) {}

    // Terminates and removes an exisiting telemetry subscription
    rpc cancelTelemetrySubscription(CancelSubscriptionRequest)      returns (CancelSubscriptionReply) {}

    // Get the list of current telemetry subscriptions from the
    // target. This command returns a list of existing subscriptions
    // not including those that are established via configuration.
    rpc getTelemetrySubscriptions(GetSubscriptionsRequest)          returns (GetSubscriptionsReply) {}

    // Get Telemetry Agent Operational States
    rpc getTelemetryOperationalState(GetOperationalStateRequest)    returns (GetOperationalStateReply) {}

    // Return the set of data encodings supported by the device for
    // telemetry data
    rpc getDataEncodings(DataEncodingRequest)                       returns (DataEncodingReply) {}
}

// Message sent for a telemetry subscription request
message SubscriptionRequest {
    // Data associated with a telemetry subscription
    SubscriptionInput input                                 = 1;

    // List of data models paths and filters
    // which are used in a telemetry operation.
    repeated Path path_list                                 = 2;

    // The below configuration is not defined in Openconfig RPC.
    // It is a proposed extension to configure additional
    // subscription request features.
    SubscriptionAdditionalConfig additional_config          = 3;
}

// Data associated with a telemetry subscription
message SubscriptionInput {
    // List of optional collector endpoints to send data for
    // this subscription.
    // If no collector destinations are specified, the collector
    // destination is assumed to be the requester on the rpc channel.
    repeated Collector  collector_list                      = 1;
}

// Collector endpoints to send data specified as an ip+port combination.
message Collector {
    // IP address of collector endpoint
    string address                                          = 1;

    // Transport protocol port number for the collector destination.
    uint32 port                                             = 2;
}

// Data model path
message Path {
    // Data model path of interest
    // Path specification for elements of OpenConfig data models
    string path                                             = 1;

    // Regular expression to be used in filtering state leaves
    string filter                                           = 2;

    // If this is set to true, the target device will only send
    // updates to the collector upon a change in data value
    bool suppress_unchanged                                 = 3;

    // Maximum time in ms the target device may go without sending
    // a message to the collector. If this time expires with
    // suppress-unchanged set, the target device must send an update
    // message regardless if the data values have changed.
    uint32 max_silent_interval                              = 4;

    // Time in ms between collection and transmission of the
    // specified data to the collector platform. The target device
    // will sample the corresponding data (e.g,. a counter) and
    // immediately send to the collector destination.
    //
    // If sample-frequency is set to 0, then the network device
    // must emit an update upon every datum change.
    uint32 sample_frequency                                 = 5;
}

// Configure subscription request additional features.
message SubscriptionAdditionalConfig {
    // limit the number of records sent in the stream
    int32 limit_records                                     = 1;

    // limit the time the stream remains open
    int32 limit_time_seconds                                = 2;
}

// Reply to inline subscription for data at the specified path is done in
// two-folds.
// 1. Reply data message sent out using out-of-band channel.
// 2. Telemetry data send back on the same connection as the
//    subscription request.

// 1. Reply data message sent out using out-of-band channel.
message SubscriptionReply {
    // Response message to a telemetry subscription creation or
    // get request.
    SubscriptionResponse response                           = 1;

    // List of data models paths and filters
    // which are used in a telemetry operation.
    repeated Path path_list                                 = 2;
}

// Response message to a telemetry subscription creation or get request.
message SubscriptionResponse {
    // Unique id for the subscription on the device. This is
    // generated by the device and returned in a subscription
    // request or when listing existing subscriptions
    uint32 subscription_id = 1;
}

// 2. Telemetry data send back on the same connection as the
//    subscription request.
message OpenConfigData {
    // router name:export IP address
    string system_id                                        = 1;

    // line card / RE (slot number)
    uint32 component_id                                     = 2;

    // PFE (if applicable)
    uint32 sub_component_id                                 = 3;

    // Path specification for elements of OpenConfig data models
    string path                                             = 4;

    // Sequence number, monotonically increasing for each
    // system_id, component_id, sub_component_id + path.
    uint64 sequence_number                                  = 5;

    // timestamp (milliseconds since epoch)
    uint64 timestamp                                        = 6;

    // List of key-value pairs
    repeated KeyValue kv                                    = 7;
}

// Simple Key-value, where value could be one of scalar types
message KeyValue {
    // Key
    string key                                              =  1;

    // One of possible values
    oneof value {
        double double_value                                 =  5;
        int64  int_value                                    =  6;
        uint64 uint_value                                   =  7;
        sint64 sint_value                                   =  8;
        bool   bool_value                                   =  9;
        string str_value                                    = 10;
        bytes  bytes_value                                  = 11;
    }
}

// Message sent for a telemetry subscription cancellation request
message CancelSubscriptionRequest {
    // Subscription identifier as returned by the device when
    // subscription was requested
    uint32 subscription_id                                  = 1;
}

// Reply to telemetry subscription cancellation request
message CancelSubscriptionReply {
    // Return code
    ReturnCode code                                         = 1;

    // Return code string
    string     code_str                                     = 2;
};

// Result of the operation
enum ReturnCode {
    SUCCESS                                                 = 0;
    NO_SUBSCRIPTION_ENTRY                                   = 1;
    UNKNOWN_ERROR                                           = 2;
}

// Message sent for a telemetry get request
message GetSubscriptionsRequest {
    // Subscription identifier as returned by the device when
    // subscription was requested
    // --- or ---
    // 0xFFFFFFFF for all subscription identifiers
    uint32 subscription_id                                  = 1;
}

// Reply to telemetry subscription get request
message GetSubscriptionsReply {
    // List of current telemetry subscriptions
    repeated SubscriptionReply subscription_list            = 1;
}

// Message sent for telemetry agent operational states request
message GetOperationalStateRequest {
    // Per-subscription_id level operational state can be requested.
    //
    // Subscription identifier as returned by the device when
    // subscription was requested
    // --- or ---
    // 0xFFFFFFFF for all subscription identifiers including agent-level
    // operational stats
    // --- or ---
    // If subscription_id is not present then sent only agent-level
    // operational stats
    uint32 subscription_id                                  = 1;

    // Control verbosity of the output
    VerbosityLevel verbosity                                = 2;
}

// Verbosity Level
enum VerbosityLevel {
    DETAIL                                                  = 0;
    TERSE                                                   = 1;
    BRIEF                                                   = 2;
}

// Reply to telemetry agent operational states request
message GetOperationalStateReply {
    // List of key-value pairs where
    //     key      = operational state definition
    //     value    = operational state value
    repeated KeyValue kv                                    = 1;
}

// Message sent for a data encoding request
message DataEncodingRequest {
}

// Reply to data encodings supported request
message DataEncodingReply {
    repeated EncodingType  encoding_list                    = 1;
}

// Encoding Type Supported
enum EncodingType {
    UNDEFINED                                               = 0;
    XML                                                     = 1;
    JSON_IETF                                               = 2;
    PROTO3                                                  = 3;
}

In order to do the service call (rpc TelemetrySubscribe) first i need to read header which have subscription id and then start reading messages. Now, using Java i am able to connect with the service, i did introduce the interceptor but when i print/retrieve header it is null. My code of calling interceptor is below,

 ClientInterceptor interceptor = new HeaderClientInterceptor();
      originChannel = OkHttpChannelBuilder.forAddress(host, port)
        .usePlaintext(true)
        .build();
     Channel channel =  ClientInterceptors.intercept(originChannel, interceptor);
      telemetryStub = OpenConfigTelemetryGrpc.newStub(channel);

This is interceptor code to read meta Data.

  @Override
  public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(MethodDescriptor<ReqT, RespT> method,
      CallOptions callOptions, Channel next) {
    return new SimpleForwardingClientCall<ReqT, RespT>(next.newCall(method, callOptions)) {

      @Override
      public void start(Listener<RespT> responseListener, Metadata headers) {

        super.start(new SimpleForwardingClientCallListener<RespT>(responseListener) {
          @Override
          public void onHeaders(Metadata headers) {

             Key<String> CUSTOM_HEADER_KEY = Metadata.Key.of("responseKEY", Metadata.ASCII_STRING_MARSHALLER);

            System.out.println("Contains Key?? "+headers.containsKey(CUSTOM_HEADER_KEY));

Wondering is there any other way to read meta data or first message which have subscription ID in it? All i need to read first message which have subscription Id, and return the same subscription id to server so that streaming can start I have equivalent Python code using same proto file and it is communicating with server by code mention below for reference only:

     sub_req = SubscribeRequestMsg("host",port)
     data_itr = stub.telemetrySubscribe(sub_req, _TIMEOUT_SECONDS)
     metadata = data_itr.initial_metadata()

                   if metadata[0][0] == "responseKey":
                    metainfo = metadata[0][1]
                    print metainfo

                    subreply = agent_pb2.SubscriptionReply()
                    subreply.SetInParent()
                    google.protobuf.text_format.Merge(metainfo, subreply)

                    if subreply.response.subscription_id:
                    SUB_ID = subreply.response.subscription_id

From the python code above i can easily retrieve meta data object, not sure how to retrieve same using Java?

After reading metaData all i am getting is: Metadata({content-type=[application/grpc], grpc-encoding=[identity], grpc-accept-encoding=[identity,deflate,gzip]})

But i know there is one more line from meta data to it, which is

response {
  subscription_id: 2
}

How can i extract last response from Header which have subscription id in it. I did try many options and i am lost here.

Body:
In case this helps anyone else out:

I needed to write a specific response header in my gRPC-java server based upon the request/response.

What I ended up doing was storing the response header value in a Context using the Context::withValue (which doesn't modify the existing context but instead creates a new Context actually), and then calling the request service handler method's StreamObserver::onNext inside of the Context::run callback. StremableObserver::onNext calls the ServerCall::sendHeaders I have in my set from my ServerInterceptor. There in the sendHeaders, it can read the value in the Context I stored and set the response header value.

I think this is similar to @carl-mastrangelo's approach, just may be spelled out a little bit more.

public enum MyServerInterceptor implements ServerInterceptor {
    INSTANCE;

    public static final Metadata.Key<String> METADATA_KEY =
            Metadata.Key.of("fish", ASCII_STRING_MARSHALLER);

    public static final Context.Key<String> CONTEXT_KEY = Context.key("dog");

    @Override
    public <ReqT, RespT> ServerCall.Listener<ReqT> interceptCall(ServerCall<ReqT, RespT> call,
                                                                 Metadata requestHeaders,
                                                                 ServerCallHandler<ReqT, RespT> next) {

        ServerCall<ReqT, RespT> myServerCall = new ForwardingServerCall.SimpleForwardingServerCall<ReqT, RespT>(call) {
            @Override
            public void sendHeaders(Metadata responseHeaders) {
                String value = CONTEXT_KEY.get(Context.current());
                responseHeaders.put(METADATA_KEY, value);
                super.sendHeaders(responseHeaders);
            }
        };

        return next.startCall(myServerCall, requestHeaders);
    }
}

public class MyService extends MyServiceGrpc.MyServiceyImplBase {
    @Override
    public void serviceMethod(MyRequest request, StreamObserver<MyResponse> responseObserver) {
        MyResponse response = new MyResponse();

        Context.current().withValue(Context.key("dog"), "cat").run(() -> {
            responseObserver.onNext(response);
        });

        responseObserver.onCompleted();
    }
}

The again the critical piece here is that responseObserver::onNext  calls into ForwardingServerCall.SimpleForwardingServerCall::sendHeaders.

Please let me know if there is a better way. This all seems more complex than I'd like.


问:
Picture source with srcset
I want to use  tag and 3 image formats. avif \ webp \ png. At the same time, but not to forget about retina displays.

At the moment I came up with something like this format:

```<code><picture>
    <source srcset="components/header/img/logo.avif, components/header/img/logo@2x.avif 2x" type="image/avif">
    <source srcset="components/header/img/logo.webp, components/header/img/logo@2x.webp 2x" type="image/webp">
    <img class="someClass" src="components/header/img/logo.png" srcset="components/header/img/logo@2x.png 2x" alt="Logo">
</picture>
</code>```

But how much of this is correct ? Maybe it should be implemented in some other way ?

答:
You can combine both concepts via an art-direction rule.

The art direction concept is basically intended to switch between different image versions with different aspect ratios depending on your viewport size.

Example from german web studio "kulturbanause":

```<code><picture>
  <source media="(min-width: 56.25em)" srcset="large.webp" type="image/webp">
  <source media="(min-width: 56.25em)" srcset="large.jpg">

  <source media="(min-width: 37.5em)" srcset="medium.webp" type="image/webp">
  <source media="(min-width: 37.5em)" srcset="medium.jpg">

  <source srcset="small.webp" type="image/webp">
  <source srcset="small.jpg">
  <img src="fallback.jpg" alt="">
</picture>
</code>```

Use svg if feasible (logos, info graphics etc)

Referring to your code example referencing a logo img:

Most likely, your logo could be used as a svg.

If it's created (or optimized) well, you don't need any HiRes candidates.
(and most likely svg will also wipe the floor with raster images regarding filesize)

See also: MDN Docs: Responsive images

Title: Picture source with srcset

Body:
I want to use  tag and 3 image formats. avif \ webp \ png. At the same time, but not to forget about retina displays.

At the moment I came up with something like this format:

<picture>
    <source srcset="components/header/img/logo.avif, components/header/img/logo@2x.avif 2x" type="image/avif">
    <source srcset="components/header/img/logo.webp, components/header/img/logo@2x.webp 2x" type="image/webp">

</picture>

But how much of this is correct ? Maybe it should be implemented in some other way ?

Body:
You can combine both concepts via an art-direction rule.

The art direction concept is basically intended to switch between different image versions with different aspect ratios depending on your viewport size.

Example from german web studio "kulturbanause":

<picture>
  <source media="(min-width: 56.25em)" srcset="large.webp" type="image/webp">
  <source media="(min-width: 56.25em)" srcset="large.jpg">

  <source media="(min-width: 37.5em)" srcset="medium.webp" type="image/webp">
  <source media="(min-width: 37.5em)" srcset="medium.jpg">

  <source srcset="small.webp" type="image/webp">
  <source srcset="small.jpg">
  <img src="fallback.jpg" alt="">
</picture>

Use svg if feasible (logos, info graphics etc)

Referring to your code example referencing a logo img:

Most likely, your logo could be used as a svg.

If it's created (or optimized) well, you don't need any HiRes candidates.
(and most likely svg will also wipe the floor with raster images regarding filesize)

See also: MDN Docs: Responsive images


问:
None Type Error Given to me when accessing LinkedList
I am currently working on a LeetCode problem called Remove Nth Node From End of List. In it I am getting a 'NoneType' Error has no object next error when trying to access a variable equal to the head. The logic may be wrong but I am unsure why I would get an error on this specific variable(seeing as I already checked to see if the head or head.next is None). The " while runner.next is not None:" line is giving me an error. I don;t necessarily need help with the problem as much as why this error is coming about. Thanks!
Small Edit. The code runs fine for most sample cases but the one where this error is prevalent is where the list looks like 1 -> 2 - >None(1 is the head).</p>
```<code>        if not head:
            return
        if head.next == None:
            return

        count = 0

        walker = head
        runner = head
        if not walker or not runner:
            return

        while runner.next is not None:
            while(count < n):
                runner = runner.next
                count = count + 1

            walker = walker.next
            runner = runner.next

        walker.next = walker.next.next

        return head
</code>```

答: On the sample case which you provided, the runner.next property cannot be accessed because runner is None. Your inner while loop iterates at least once if n > 0, progressing runner down the linked list, and then runner is progressed once more at the end of the outer loop. This leaves runner equal to None. Thus when the while loop attempts to check the condition, it attempts to access runner.next and gives you your error. You may wish to update your condition to check that runner is also not None.

Body:
I am currently working on a LeetCode problem called Remove Nth Node From End of List. In it I am getting a 'NoneType' Error has no object next error when trying to access a variable equal to the head. The logic may be wrong but I am unsure why I would get an error on this specific variable(seeing as I already checked to see if the head or head.next is None). The " while runner.next is not None:" line is giving me an error. I don;t necessarily need help with the problem as much as why this error is coming about. Thanks!
Small Edit. The code runs fine for most sample cases but the one where this error is prevalent is where the list looks like 1 -> 2 - >None(1 is the head).

        if not head:
            return
        if head.next == None:
            return

        count = 0

        walker = head
        runner = head
        if not walker or not runner:
            return

        while runner.next is not None:
            while(count < n):
                runner = runner.next
                count = count + 1

            walker = walker.next
            runner = runner.next

        walker.next = walker.next.next

        return head


问:
PHP Regular expressions problem
I'm using regex to pull info from a html table.

But I'm messing up some how, and have no idea why.

PHP CODE:

```<code>$printable = file_get_contents('./testplaylist.php', true);

if(preg_match_all('/<TR[^>]*>(.*?)<\/TR>/si', $printable, $matches, PREG_SET_ORDER)); {
foreach($matches as $match) {
$data = "$match[1]";

echo("$data <br />");

}
}
</code>```

HTML DATA:

```<code><TR class=" light ">
Stuff in here
</TR>
</code>```

Any help would be appreciated,

Thanks!

答:
Works fine here. It should work unless you have nested tables.

The problem must be in your data source. Do some tracing with var_dump.

Title: PHP Regular expressions problem

Body:
I'm using regex to pull info from a html table.

But I'm messing up some how, and have no idea why.

PHP CODE:

$printable = file_get_contents('./testplaylist.php', true);

if(preg_match_all('/
]*>(.*?)<\/TR>/si', $printable, $matches, PREG_SET_ORDER)); {
foreach($matches as $match) {
$data = "$match[1]";

echo("$data
");

}
}

HTML DATA:

Stuff in here

Any help would be appreciated,

Thanks!


问:
How to query by multiple conditions in faunadb?
I try to improve my understanding of FaunaDB.

I have a collection that contains records like:

```<code>{
  "ref": Ref(Collection("regions"), "261442015390073344"),
  "ts": 1587576285055000,
  "data": {
    "name": "italy",
    "attributes": {
      "amenities": {
        "camping": 1,
        "swimming": 7,
        "hiking": 3,
        "culture": 7,
        "nightlife": 10,
        "budget": 6
      }
    }
  }
}
</code>```

I would like to query in a flexible way by different attributes like:

data.attributes.amenities.camping > 5

data.attributes.amenities.camping > 5 AND data.attributes.amenities.hiking > 6

data.attributes.amenities.camping < 6 AND data.attributes.amenities.culture > 6 AND hiking > 5 AND ...

I created an index containing all attributes, but I don't know how to do greater equals filtering in an index that contains multiple terms.

My fallback would be to create an index for each attribute and use Intersection to get the records that are in all subqueries that I want to check, but this feels somehow wrong:

The query: budget >= 6 AND camping >=8 would be:

```<code>Index:
{
  name: "all_regions_by_all_attributes",
  unique: false,
  serialized: true,
  source: "regions",
  terms: [],
  values: [
    {
      field: ["data", "attributes", "amenities", "culture"]
    },
    {
      field: ["data", "attributes", "amenities", "hiking"]
    },
    {
      field: ["data", "attributes", "amenities", "swimming"]
    },
    {
      field: ["data", "attributes", "amenities", "budget"]
    },
    {
      field: ["data", "attributes", "amenities", "nightlife"]
    },
    {
      field: ["data", "attributes", "amenities", "camping"]
    },
    {
      field: ["ref"]
    }
  ]
}
</code>```

Query:

```<code>Map(
  Paginate(
    Intersection(
      Range(Match(Index("all_regions_by_all_attributes")), [0, 0, 0, 6, 0, 8], [10, 10, 10, 10, 10, 10]),
    )

  ),
  Lambda(
    ["culture", "hiking", "swimming", "budget", "nightlife", "camping", "ref"],
    Get(Var("ref"))
  )
)
</code>```

This approach has the following disadvantages:

It does not work like expected, if for example the first (culture) attribute is in this range, but the second (hiking) not, then I would still get a return values

It causes a lot of reads due to the reference that I need to follow for each result.

Is it possible to store all values in this kind of index that would contain all the data? I know I can just add more values to the index and access them. But this would mean I have to create a new index as soon as we add more fields to the entity. But maybe this is a common thing.

thanks in advance

答:
Easy way to query with the multiple conditions I think with the query it with documents differences, In my solutions it is like:

    const response = await client.query(
      q.Let(
        {
          activeUsers: q.Difference(
            q.Match(q.Index("allUsers")),
            q.Match(q.Index("usersByStatus"), "ARCHIVE")
          ),
          paginatedDocuments: q.Map(
            q.Paginate(q.Var("activeUsers"), {
              size,
              before: reqBefore,
              after: reqAfter
            }),
            q.Lambda("x", q.Get(q.Var("x")))
          ),
          total: q.Count(q.Var("activeUsers"))
        },
        {
          documents: q.Var("paginatedDocuments"),
          total: q.Var("total")
        }
      )
    );

    const {
      documents: {
        data: dbData = [],
        before: dbBefore = [],
        after: dbAfter = []
      } = {},
      total = 0
    } = response || {};

    const respBefore = dbBefore[0]?.value?.id || null;
    const respAfter = dbAfter[0]?.value?.id || null;

    const data = await dbData.map((userData) => {
      const {
        ref: { id = null } = {},
        data: { firstName = "", lastName = "" }
      } = userData;

      return {
        id,
        firstName,
        lastName
      };
    });```

So in the query builder you can filter each nested document in variable in Let section by the index that you want.

Body:
I try to improve my understanding of FaunaDB.

I have a collection that contains records like:

{
  "ref": Ref(Collection("regions"), "261442015390073344"),
  "ts": 1587576285055000,
  "data": {
    "name": "italy",
    "attributes": {
      "amenities": {
        "camping": 1,
        "swimming": 7,
        "hiking": 3,
        "culture": 7,
        "nightlife": 10,
        "budget": 6
      }
    }
  }
}

I would like to query in a flexible way by different attributes like:

data.attributes.amenities.camping > 5

data.attributes.amenities.camping > 5 AND data.attributes.amenities.hiking > 6

data.attributes.amenities.camping < 6 AND data.attributes.amenities.culture > 6 AND hiking > 5 AND ...

I created an index containing all attributes, but I don't know how to do greater equals filtering in an index that contains multiple terms.

My fallback would be to create an index for each attribute and use Intersection to get the records that are in all subqueries that I want to check, but this feels somehow wrong:

The query: budget >= 6 AND camping >=8 would be:

Index:
{
  name: "all_regions_by_all_attributes",
  unique: false,
  serialized: true,
  source: "regions",
  terms: [],
  values: [
    {
      field: ["data", "attributes", "amenities", "culture"]
    },
    {
      field: ["data", "attributes", "amenities", "hiking"]
    },
    {
      field: ["data", "attributes", "amenities", "swimming"]
    },
    {
      field: ["data", "attributes", "amenities", "budget"]
    },
    {
      field: ["data", "attributes", "amenities", "nightlife"]
    },
    {
      field: ["data", "attributes", "amenities", "camping"]
    },
    {
      field: ["ref"]
    }
  ]
}

Query:

Map(
  Paginate(
    Intersection(
      Range(Match(Index("all_regions_by_all_attributes")), [0, 0, 0, 6, 0, 8], [10, 10, 10, 10, 10, 10]),
    )

  ),
  Lambda(
    ["culture", "hiking", "swimming", "budget", "nightlife", "camping", "ref"],
    Get(Var("ref"))
  )
)

This approach has the following disadvantages:

It does not work like expected, if for example the first (culture) attribute is in this range, but the second (hiking) not, then I would still get a return values

It causes a lot of reads due to the reference that I need to follow for each result.

Is it possible to store all values in this kind of index that would contain all the data? I know I can just add more values to the index and access them. But this would mean I have to create a new index as soon as we add more fields to the entity. But maybe this is a common thing.

thanks in advance

Body:
Easy way to query with the multiple conditions I think with the query it with documents differences, In my solutions it is like:

    const response = await client.query(
      q.Let(
        {
          activeUsers: q.Difference(
            q.Match(q.Index("allUsers")),
            q.Match(q.Index("usersByStatus"), "ARCHIVE")
          ),
          paginatedDocuments: q.Map(
            q.Paginate(q.Var("activeUsers"), {
              size,
              before: reqBefore,
              after: reqAfter
            }),
            q.Lambda("x", q.Get(q.Var("x")))
          ),
          total: q.Count(q.Var("activeUsers"))
        },
        {
          documents: q.Var("paginatedDocuments"),
          total: q.Var("total")
        }
      )
    );

    const {
      documents: {
        data: dbData = [],
        before: dbBefore = [],
        after: dbAfter = []
      } = {},
      total = 0
    } = response || {};

    const respBefore = dbBefore[0]?.value?.id || null;
    const respAfter = dbAfter[0]?.value?.id || null;

    const data = await dbData.map((userData) => {
      const {
        ref: { id = null } = {},
        data: { firstName = "", lastName = "" }
      } = userData;

      return {
        id,
        firstName,
        lastName
      };
    });

So in the query builder you can filter each nested document in variable in Let section by the index that you want.


答:
Try it this way instead</p>
```<code>var checkBox = document.getElementById('checkbox');
let body =  document.getElementById('body');
let i =2;

checkbox.addEventListener('change', ()=>{
if (i%2==0){
   div.classList.toggle('dark-mode');
  } else {
    div.classList.toggle('light-mode');

  }
});
</code>```

Title: Checkbox classList Toggle

Body:
Try it this way instead

var checkBox = document.getElementById('checkbox');
let body =  document.getElementById('body');
let i =2;

checkbox.addEventListener('change', ()=>{
if (i%2==0){
   div.classList.toggle('dark-mode');
  } else {
    div.classList.toggle('light-mode');

  }
});


问:
Testing whether a large list of alpha characters is contained in a string (without using RegEx)
Without using RegEx, I'd like to be able to test whether a given string contains certain lowercase characters (e.g. 22 of the 26 lower-case characters in the alphabet).

For example:

```<code>my_str = 'foobar'
if 'c' in str or 'd' in my_str or 'e' in my_str:   # I don't want to enumerate the entire alphabet
    print('fails the test')
else:
    print('passes the test')
</code>```

This approach works, but I would have to enumerate most of the lowercase alphabet.  This is a hacky approach!

Is there another way to do this?  (yes, I know that RegEx would be a good choice, but not for my use case)

Thanks!

答:
You could use a set with the isdisjoint method:

```<code>my_str = 'foobar'
if not set('cde').isdisjoint(my_str):
    print('fails the test')
else:
    print('passes the test')
</code>```

If you want to exclude most letters, you can setup your set with all letters and remove the ones that are acceptable:

```<code>import string

excludeSet = set(string.ascii_lowercase)
excludeSet -= set('aeiou')               # e.g. allow vowels

my_str = 'foobar'
if not excludeSet.isdisjoint(my_str):
    print('fails the test')
else:
    print('passes the test')
</code>```

Body:
Without using RegEx, I'd like to be able to test whether a given string contains certain lowercase characters (e.g. 22 of the 26 lower-case characters in the alphabet).

For example:

my_str = 'foobar'
if 'c' in str or 'd' in my_str or 'e' in my_str:   # I don't want to enumerate the entire alphabet
    print('fails the test')
else:
    print('passes the test')

This approach works, but I would have to enumerate most of the lowercase alphabet.  This is a hacky approach!

Is there another way to do this?  (yes, I know that RegEx would be a good choice, but not for my use case)

Thanks!

Body:
You could use a set with the isdisjoint method:

my_str = 'foobar'
if not set('cde').isdisjoint(my_str):
    print('fails the test')
else:
    print('passes the test')

If you want to exclude most letters, you can setup your set with all letters and remove the ones that are acceptable:

import string

excludeSet = set(string.ascii_lowercase)
excludeSet -= set('aeiou')               # e.g. allow vowels

my_str = 'foobar'
if not excludeSet.isdisjoint(my_str):
    print('fails the test')
else:
    print('passes the test')


问:
how to retrieve data that contains special characters including alphabet
I'm using oracle and hive db engine and supposed that in a varchar column (phone number), I only want to retrieve record with digit and hyphen '-'
i.e. 03-1234 5678

But how if I want to retrieve if column has special chars (except hyphen) and alphabet using like or rlike.

i.e 03-ABC123$#45XYZ or 03-AB123 Y123#& (with space)

Thanks in advance!

答:
Using RLIKE we can try:

SELECT * FROM yourTable WHERE phone RLIKE '^[0-9 -]+$';
```

This will match only phone numbers consisting of digits, space, or hyphen.

Body:
Using RLIKE we can try:

SELECT * FROM yourTable WHERE phone RLIKE '^[0-9 -]+$';

This will match only phone numbers consisting of digits, space, or hyphen.


问:
How to speed up BeautifulSoup for loop
I have a for loop that parses through 6 urls to get the text of the first class with "GARawf". The loop works however I've noticed that it now takes the page about 9 seconds to load compared to 1 second before. As I am new to Django and BeautifulSoup I was wondering if there was a way I could refactor the code so it loads the view faster.

views.py

```<code>def results(request, result1, result2, result3, result4, result5, result6, broad_variable1, broad_variable2, broad_variable3, specific_variable_dictionary, user_city):
    result1 = City.objects.filter(city=result1).first()
    result2 = City.objects.filter(city=result2).first()
    result3 = City.objects.filter(city=result3).first()
    result4 = City.objects.filter(city=result4).first()
    result5 = City.objects.filter(city=result5).first()
    result6 = City.objects.filter(city=result6).first()

    # get the first user selected specific variable value for each result
    result1_value1 = City.objects.filter(city=result1.city).values(broad_variable1)[0][broad_variable1]
    result2_value1 = City.objects.filter(city=result2.city).values(broad_variable1)[0][broad_variable1]
    result3_value1 = City.objects.filter(city=result3.city).values(broad_variable1)[0][broad_variable1]
    result4_value1 = City.objects.filter(city=result4.city).values(broad_variable1)[0][broad_variable1]
    result5_value1 = City.objects.filter(city=result5.city).values(broad_variable1)[0][broad_variable1]
    result6_value1 = City.objects.filter(city=result6.city).values(broad_variable1)[0][broad_variable1]

    # assign variables before referencing them
    result1_value2 = None
    result2_value2 = None
    result3_value2 = None
    result4_value2 = None
    result5_value2 = None
    result6_value2 = None

    # check if the user chose a second variable
    # get the second user selected specific variable value for each result
    if broad_variable2 != "Nothing":
        result1_value2 = City.objects.filter(city=result1.city).values(broad_variable2)[0][broad_variable2]
        result2_value2 = City.objects.filter(city=result2.city).values(broad_variable2)[0][broad_variable2]
        result3_value2 = City.objects.filter(city=result3.city).values(broad_variable2)[0][broad_variable2]
        result4_value2 = City.objects.filter(city=result4.city).values(broad_variable2)[0][broad_variable2]
        result5_value2 = City.objects.filter(city=result5.city).values(broad_variable2)[0][broad_variable2]
        result6_value2 = City.objects.filter(city=result6.city).values(broad_variable2)[0][broad_variable2]

    # assign variables before referencing them
    result1_value3 = None
    result2_value3 = None
    result3_value3 = None
    result4_value3 = None
    result5_value3 = None
    result6_value3 = None

    # check if the user chose a third variable
    # get the third user selected specific variable value for each result
    if broad_variable3 != "Nothing":
        result1_value3 = City.objects.filter(city=result1.city).values(broad_variable3)[0][broad_variable3]
        result2_value3 = City.objects.filter(city=result2.city).values(broad_variable3)[0][broad_variable3]
        result3_value3 = City.objects.filter(city=result3.city).values(broad_variable3)[0][broad_variable3]
        result4_value3 = City.objects.filter(city=result4.city).values(broad_variable3)[0][broad_variable3]
        result5_value3 = City.objects.filter(city=result5.city).values(broad_variable3)[0][broad_variable3]
        result6_value3 = City.objects.filter(city=result6.city).values(broad_variable3)[0][broad_variable3]

    # create list of cities
    city_list = [result1.city, result2.city, result3.city, result4.city, result5.city, result6.city]

    # create price list
    prices_list = []

    # set origin for flight
    origin = user_city

    # set headers
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36"
    }

    for city in city_list:

        # set search query
        url = "https://google.com/search?q=" + origin + " to " + city + " Google Flights"

        response = requests.get(url, headers=headers)

        soup = BeautifulSoup(response.text, 'lxml')

        # get price element
        prices = soup.find("span", attrs={"class": "GARawf"})
        if prices != None:
            prices_list.append(prices.text.strip())
        else:
            prices_list.append("Not Available")

    # change string dictionary into actual dictionary
    specific_variable_dictionary = ast.literal_eval(specific_variable_dictionary)

    context = {
        "result1": result1, "result2": result2, "result3": result3, "result4": result4, "result5": result5,
        "result6": result6, "result1_value1": result1_value1, "result2_value1": result2_value1, "result3_value1": result3_value1,
        "result4_value1": result4_value1, "result5_value1": result5_value1, "result6_value1": result6_value1, "result1_value2": result1_value2,
        "result2_value2": result2_value2, "result3_value2": result3_value2, "result4_value2": result4_value2,
        "result5_value2": result5_value2, "result6_value2": result6_value2, "result1_value3": result1_value3, "result2_value3": result2_value3,
        "result3_value3": result3_value3, "result4_value3": result4_value3, "result5_value3": result5_value3, "result6_value3": result6_value3,
        "broad_variable1": broad_variable1, "broad_variable2": broad_variable2, "broad_variable3": broad_variable3,
        "specific_variable_dictionary": specific_variable_dictionary, "prices_list": prices_list
    }
    return render(request, 'Discovery_App/results.html', context)
</code>```

答:
I would use threading to call the function doing the requests, so that the requests will run concurrently. You will want to import concurrent.futures

Then move citylist list to threading function like below. Read more about using Thread Pools

Update 2/11:
Due to your recent comments, I have changed the code below. In your original post you stated that the code works, but its very slow. What you were doing is using a for loop to go one by one through each city. Doing this is slow because the program will wait for the previous request to finish and return before making a new request.

What I have changed in the code below is seperated your code into 2 functions. The first function ends right after you define city_list, because that is the last information collected before you open a new request.

After city_list is defined as list, we then define the threadpoolexecutor, and pass whatever values you need into the next function, in this case it's items from city_list, and user_city.

This will concurrently call the function get_price for each item in the city_list. No need for the for loop. But speaking of for loops, your first function could definitely use some to cut down the amount of lines.

import concurrent.futures

def results(request, result1, result2, result3, result4, result5, result6, broad_variable1, broad_variable2, broad_variable3, specific_variable_dictionary, user_city):
    result1 = City.objects.filter(city=result1).first()
    result2 = City.objects.filter(city=result2).first()
    result3 = City.objects.filter(city=result3).first()
    result4 = City.objects.filter(city=result4).first()
    result5 = City.objects.filter(city=result5).first()
    result6 = City.objects.filter(city=result6).first()

    # get the first user selected specific variable value for each result
    result1_value1 = City.objects.filter(city=result1.city).values(broad_variable1)[0][broad_variable1]
    result2_value1 = City.objects.filter(city=result2.city).values(broad_variable1)[0][broad_variable1]
    result3_value1 = City.objects.filter(city=result3.city).values(broad_variable1)[0][broad_variable1]
    result4_value1 = City.objects.filter(city=result4.city).values(broad_variable1)[0][broad_variable1]
    result5_value1 = City.objects.filter(city=result5.city).values(broad_variable1)[0][broad_variable1]
    result6_value1 = City.objects.filter(city=result6.city).values(broad_variable1)[0][broad_variable1]

    # assign variables before referencing them
    result1_value2 = None
    result2_value2 = None
    result3_value2 = None
    result4_value2 = None
    result5_value2 = None
    result6_value2 = None

    # check if the user chose a second variable
    # get the second user selected specific variable value for each result
    if broad_variable2 != "Nothing":
        result1_value2 = City.objects.filter(city=result1.city).values(broad_variable2)[0][broad_variable2]
        result2_value2 = City.objects.filter(city=result2.city).values(broad_variable2)[0][broad_variable2]
        result3_value2 = City.objects.filter(city=result3.city).values(broad_variable2)[0][broad_variable2]
        result4_value2 = City.objects.filter(city=result4.city).values(broad_variable2)[0][broad_variable2]
        result5_value2 = City.objects.filter(city=result5.city).values(broad_variable2)[0][broad_variable2]
        result6_value2 = City.objects.filter(city=result6.city).values(broad_variable2)[0][broad_variable2]

    # assign variables before referencing them
    result1_value3 = None
    result2_value3 = None
    result3_value3 = None
    result4_value3 = None
    result5_value3 = None
    result6_value3 = None

    # check if the user chose a third variable
    # get the third user selected specific variable value for each result
    if broad_variable3 != "Nothing":
        result1_value3 = City.objects.filter(city=result1.city).values(broad_variable3)[0][broad_variable3]
        result2_value3 = City.objects.filter(city=result2.city).values(broad_variable3)[0][broad_variable3]
        result3_value3 = City.objects.filter(city=result3.city).values(broad_variable3)[0][broad_variable3]
        result4_value3 = City.objects.filter(city=result4.city).values(broad_variable3)[0][broad_variable3]
        result5_value3 = City.objects.filter(city=result5.city).values(broad_variable3)[0][broad_variable3]
        result6_value3 = City.objects.filter(city=result6.city).values(broad_variable3)[0][broad_variable3]

    # create list of cities
    city_list = [result1.city, result2.city, result3.city, result4.city, result5.city, result6.city]

     #define threadpool executor
    with concurrent.futures.ThreadPoolExecutor() as executor:

        results = executor.map(get_price, city_list, user_city)
        for result in results:
            print(result)

def get_price(cities, user_city):

    # create price list
    prices_list = []

    # set origin for flight
    origin = user_city

    # set headers
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36"
    }

    url = "https://google.com/search?q=" + origin + " to " + city + " Google Flights"

    response = requests.get(url, headers=headers)

    soup = BeautifulSoup(response.text, 'lxml')

    # get price element
    prices = soup.find("span", attrs={"class": "GARawf"})
    if prices != None:
        prices_list.append(prices.text.strip())
    else:
        prices_list.append("Not Available")

# change string dictionary into actual dictionary
specific_variable_dictionary = ast.literal_eval(specific_variable_dictionary)

context = {
    "result1": result1, "result2": result2, "result3": result3, "result4": result4, "result5": result5,
    "result6": result6, "result1_value1": result1_value1, "result2_value1": result2_value1, "result3_value1": result3_value1,
    "result4_value1": result4_value1, "result5_value1": result5_value1, "result6_value1": result6_value1, "result1_value2": result1_value2,
    "result2_value2": result2_value2, "result3_value2": result3_value2, "result4_value2": result4_value2,
    "result5_value2": result5_value2, "result6_value2": result6_value2, "result1_value3": result1_value3, "result2_value3": result2_value3,
    "result3_value3": result3_value3, "result4_value3": result4_value3, "result5_value3": result5_value3, "result6_value3": result6_value3,
    "broad_variable1": broad_variable1, "broad_variable2": broad_variable2, "broad_variable3": broad_variable3,
    "specific_variable_dictionary": specific_variable_dictionary, "prices_list": prices_list
}
return render(request, 'Discovery_App/results.html', context)
```

Title: How to speed up BeautifulSoup for loop

Body:
I have a for loop that parses through 6 urls to get the text of the first class with "GARawf". The loop works however I've noticed that it now takes the page about 9 seconds to load compared to 1 second before. As I am new to Django and BeautifulSoup I was wondering if there was a way I could refactor the code so it loads the view faster.

views.py

def results(request, result1, result2, result3, result4, result5, result6, broad_variable1, broad_variable2, broad_variable3, specific_variable_dictionary, user_city):
    result1 = City.objects.filter(city=result1).first()
    result2 = City.objects.filter(city=result2).first()
    result3 = City.objects.filter(city=result3).first()
    result4 = City.objects.filter(city=result4).first()
    result5 = City.objects.filter(city=result5).first()
    result6 = City.objects.filter(city=result6).first()

    # get the first user selected specific variable value for each result
    result1_value1 = City.objects.filter(city=result1.city).values(broad_variable1)[0][broad_variable1]
    result2_value1 = City.objects.filter(city=result2.city).values(broad_variable1)[0][broad_variable1]
    result3_value1 = City.objects.filter(city=result3.city).values(broad_variable1)[0][broad_variable1]
    result4_value1 = City.objects.filter(city=result4.city).values(broad_variable1)[0][broad_variable1]
    result5_value1 = City.objects.filter(city=result5.city).values(broad_variable1)[0][broad_variable1]
    result6_value1 = City.objects.filter(city=result6.city).values(broad_variable1)[0][broad_variable1]

    # assign variables before referencing them
    result1_value2 = None
    result2_value2 = None
    result3_value2 = None
    result4_value2 = None
    result5_value2 = None
    result6_value2 = None

    # check if the user chose a second variable
    # get the second user selected specific variable value for each result
    if broad_variable2 != "Nothing":
        result1_value2 = City.objects.filter(city=result1.city).values(broad_variable2)[0][broad_variable2]
        result2_value2 = City.objects.filter(city=result2.city).values(broad_variable2)[0][broad_variable2]
        result3_value2 = City.objects.filter(city=result3.city).values(broad_variable2)[0][broad_variable2]
        result4_value2 = City.objects.filter(city=result4.city).values(broad_variable2)[0][broad_variable2]
        result5_value2 = City.objects.filter(city=result5.city).values(broad_variable2)[0][broad_variable2]
        result6_value2 = City.objects.filter(city=result6.city).values(broad_variable2)[0][broad_variable2]

    # assign variables before referencing them
    result1_value3 = None
    result2_value3 = None
    result3_value3 = None
    result4_value3 = None
    result5_value3 = None
    result6_value3 = None

    # check if the user chose a third variable
    # get the third user selected specific variable value for each result
    if broad_variable3 != "Nothing":
        result1_value3 = City.objects.filter(city=result1.city).values(broad_variable3)[0][broad_variable3]
        result2_value3 = City.objects.filter(city=result2.city).values(broad_variable3)[0][broad_variable3]
        result3_value3 = City.objects.filter(city=result3.city).values(broad_variable3)[0][broad_variable3]
        result4_value3 = City.objects.filter(city=result4.city).values(broad_variable3)[0][broad_variable3]
        result5_value3 = City.objects.filter(city=result5.city).values(broad_variable3)[0][broad_variable3]
        result6_value3 = City.objects.filter(city=result6.city).values(broad_variable3)[0][broad_variable3]

    # create list of cities
    city_list = [result1.city, result2.city, result3.city, result4.city, result5.city, result6.city]

    # create price list
    prices_list = []

    # set origin for flight
    origin = user_city

    # set headers
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36"
    }

    for city in city_list:

        # set search query
        url = "https://google.com/search?q=" + origin + " to " + city + " Google Flights"

        response = requests.get(url, headers=headers)

        soup = BeautifulSoup(response.text, 'lxml')

        # get price element
        prices = soup.find("span", attrs={"class": "GARawf"})
        if prices != None:
            prices_list.append(prices.text.strip())
        else:
            prices_list.append("Not Available")

    # change string dictionary into actual dictionary
    specific_variable_dictionary = ast.literal_eval(specific_variable_dictionary)

    context = {
        "result1": result1, "result2": result2, "result3": result3, "result4": result4, "result5": result5,
        "result6": result6, "result1_value1": result1_value1, "result2_value1": result2_value1, "result3_value1": result3_value1,
        "result4_value1": result4_value1, "result5_value1": result5_value1, "result6_value1": result6_value1, "result1_value2": result1_value2,
        "result2_value2": result2_value2, "result3_value2": result3_value2, "result4_value2": result4_value2,
        "result5_value2": result5_value2, "result6_value2": result6_value2, "result1_value3": result1_value3, "result2_value3": result2_value3,
        "result3_value3": result3_value3, "result4_value3": result4_value3, "result5_value3": result5_value3, "result6_value3": result6_value3,
        "broad_variable1": broad_variable1, "broad_variable2": broad_variable2, "broad_variable3": broad_variable3,
        "specific_variable_dictionary": specific_variable_dictionary, "prices_list": prices_list
    }
    return render(request, 'Discovery_App/results.html', context)


问:
GCP dataproc - java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/ByteArraySerializer
i'm trying to run a StructuredStreaming job on GCP DataProc, which reads from Kafka nd prints out the values. The code is giving error -> java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/ByteArraySerializer

Here is the code:

```<code>import sys, datetime, time, os
from pyspark.sql.functions import col, rank, dense_rank, to_date, to_timestamp, format_number, row_number, lead, lag,monotonically_increasing_id
from pyspark.sql import SparkSession, Window
from confluent_kafka import Producer
from google.cloud import storage

spark = SparkSession.builder.appName('StructuredStreaming_VersaSase').getOrCreate()

spark.sparkContext.setLogLevel("ERROR")

kafkaBrokers='34.75.148.41:9094'
topic = "versa-sase"
# bootstrap.servers=my-cluster-lb-ssl-cert-kafka-bootstrap:9093
security_protocol="SSL"

client = storage.Client()
print(" client ", client)

bucket = client.get_bucket('ssl-certs-karan')
print(" bucket ", bucket)

allblobs = bucket.list_blobs()
print(" allblobs -> ", allblobs)

for b in allblobs:
    print(" b -> ", b)

blob_ssl_truststore_location = bucket.get_blob('ca.p12')
print(" blob_ssl_truststore_location.name ", blob_ssl_truststore_location.name)
blob_ssl_truststore_location.download_to_filename(blob_ssl_truststore_location.name)

ssl_truststore_location=blob_ssl_truststore_location.name
print(" type - blob_ssl_truststore_location ", type(blob_ssl_truststore_location))
ssl_truststore_password="NAvqbh5c9fB4"

blob_ssl_keystore_location = bucket.get_blob('dataproc-versa-sase.p12')
print(" blob_ssl_keystore_location.name ", blob_ssl_keystore_location.name)
blob_ssl_keystore_location.download_to_filename(blob_ssl_keystore_location.name)
ssl_keystore_location=blob_ssl_keystore_location.name
ssl_keystore_password="jBGsWrBv7258"
consumerGroupId = "versa-sase-grp"
checkpoint = "gs://ss-checkpoint/"

print(" SPARK.SPARKCONTEXT -> ", spark.sparkContext)

df = spark.read.format('kafka')\
    .option("kafka.bootstrap.servers",kafkaBrokers)\
    .option("kafka.security.protocol","SSL") \
    .option("kafka.ssl.truststore.location",ssl_truststore_location) \
    .option("kafka.ssl.truststore.password",ssl_truststore_password) \
    .option("kafka.ssl.keystore.location", ssl_keystore_location)\
    .option("kafka.ssl.keystore.password", ssl_keystore_password)\
    .option("subscribe", topic) \
    .option("kafka.group.id", consumerGroupId)\
    .option("startingOffsets", "earliest") \
    .load()
#

query = df.selectExpr("CAST(value AS STRING)") \
    .write \
    .format("console") \
    .option("numRows",100)\
    .option("checkpointLocation", checkpoint) \
    .option("outputMode", "complete")\
    .save("output")

# query.awaitTermination()
</code>```

Command to launch the job on Dataproc cluster :

```<code>gcloud dataproc jobs submit pyspark \
StructuredStreaming_Kafka_GCP-Batch-feb1.py --cluster=dataproc-ss-poc
--jars=gs://spark-jars-karan/spark-sql-kafka-0-10_2.12-3.1.2.jar
--region=us-central1

</code>```

Error :

```<code> SPARK.SPARKCONTEXT ->  <SparkContext master=yarn appName=StructuredStreaming_VersaSase>
Traceback (most recent call last):
  File "/tmp/b87ff69307344e2db5b43f4a73c377cf/StructuredStreaming_Kafka_GCP-Batch-feb1.py", line 49, in <module>
    df = spark.read.format('kafka')\
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 210, in load
  File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1304, in __call__
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, in deco
  File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o69.load.
: java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/ByteArraySerializer
    at org.apache.spark.sql.kafka010.KafkaSourceProvider$.<init>(KafkaSourceProvider.scala:556)
    at org.apache.spark.sql.kafka010.KafkaSourceProvider$.<clinit>(KafkaSourceProvider.scala)
    at org.apache.spark.sql.kafka010.KafkaSourceProvider.org$apache$spark$sql$kafka010$KafkaSourceProvider$$validateBatchOptions(KafkaSourceProvider.scala:336)
    at org.apache.spark.sql.kafka010.KafkaSourceProvider.createRelation(KafkaSourceProvider.scala:127)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:355)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:325)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:307)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:307)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:225)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.serialization.ByteArraySerializer
    at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)

</code>```

I've checked the Spark version on the Dataproc cluster, and spark version - 3.1.2,and scala version - 2.12 .. so the version of the spark-sql jar being passed seems to be correct.
Are there any other jars to be passed ?

What needs to be done to fix/debug this issue ?

tia !

答:
The missing class org/apache/kafka/common/serialization/ByteArraySerializer is in the kafka-clients package 1 which is a dependency of the spark-sql-kafka-0-10_2.12 package 2.

You can use either --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.2.1 which automatically pulls the transitive dependencies, or --jars=gs://my-bucket/spark-sql-kafka-0-10_2.12-3.1.2.jar,gs://my-bucket/kafka-clients-0.10.2.2.jar to add all the dependencies.

Body:
i'm trying to run a StructuredStreaming job on GCP DataProc, which reads from Kafka nd prints out the values. The code is giving error -> java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/ByteArraySerializer

Here is the code:

import sys, datetime, time, os
from pyspark.sql.functions import col, rank, dense_rank, to_date, to_timestamp, format_number, row_number, lead, lag,monotonically_increasing_id
from pyspark.sql import SparkSession, Window
from confluent_kafka import Producer
from google.cloud import storage

spark = SparkSession.builder.appName('StructuredStreaming_VersaSase').getOrCreate()

spark.sparkContext.setLogLevel("ERROR")

kafkaBrokers='34.75.148.41:9094'
topic = "versa-sase"
# bootstrap.servers=my-cluster-lb-ssl-cert-kafka-bootstrap:9093
security_protocol="SSL"

client = storage.Client()
print(" client ", client)

bucket = client.get_bucket('ssl-certs-karan')
print(" bucket ", bucket)

allblobs = bucket.list_blobs()
print(" allblobs -> ", allblobs)

for b in allblobs:
    print(" b -> ", b)

blob_ssl_truststore_location = bucket.get_blob('ca.p12')
print(" blob_ssl_truststore_location.name ", blob_ssl_truststore_location.name)
blob_ssl_truststore_location.download_to_filename(blob_ssl_truststore_location.name)

ssl_truststore_location=blob_ssl_truststore_location.name
print(" type - blob_ssl_truststore_location ", type(blob_ssl_truststore_location))
ssl_truststore_password="NAvqbh5c9fB4"

blob_ssl_keystore_location = bucket.get_blob('dataproc-versa-sase.p12')
print(" blob_ssl_keystore_location.name ", blob_ssl_keystore_location.name)
blob_ssl_keystore_location.download_to_filename(blob_ssl_keystore_location.name)
ssl_keystore_location=blob_ssl_keystore_location.name
ssl_keystore_password="jBGsWrBv7258"
consumerGroupId = "versa-sase-grp"
checkpoint = "gs://ss-checkpoint/"

print(" SPARK.SPARKCONTEXT -> ", spark.sparkContext)

df = spark.read.format('kafka')\
    .option("kafka.bootstrap.servers",kafkaBrokers)\
    .option("kafka.security.protocol","SSL") \
    .option("kafka.ssl.truststore.location",ssl_truststore_location) \
    .option("kafka.ssl.truststore.password",ssl_truststore_password) \
    .option("kafka.ssl.keystore.location", ssl_keystore_location)\
    .option("kafka.ssl.keystore.password", ssl_keystore_password)\
    .option("subscribe", topic) \
    .option("kafka.group.id", consumerGroupId)\
    .option("startingOffsets", "earliest") \
    .load()
#

query = df.selectExpr("CAST(value AS STRING)") \
    .write \
    .format("console") \
    .option("numRows",100)\
    .option("checkpointLocation", checkpoint) \
    .option("outputMode", "complete")\
    .save("output")

# query.awaitTermination()

Command to launch the job on Dataproc cluster :

gcloud dataproc jobs submit pyspark \
StructuredStreaming_Kafka_GCP-Batch-feb1.py --cluster=dataproc-ss-poc
--jars=gs://spark-jars-karan/spark-sql-kafka-0-10_2.12-3.1.2.jar
--region=us-central1

Error :

 SPARK.SPARKCONTEXT ->  <SparkContext master=yarn appName=StructuredStreaming_VersaSase>
Traceback (most recent call last):
  File "/tmp/b87ff69307344e2db5b43f4a73c377cf/StructuredStreaming_Kafka_GCP-Batch-feb1.py", line 49, in <module>
    df = spark.read.format('kafka')\
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 210, in load
  File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1304, in __call__
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, in deco
  File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o69.load.
: java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/ByteArraySerializer
    at org.apache.spark.sql.kafka010.KafkaSourceProvider$.<init>(KafkaSourceProvider.scala:556)
    at org.apache.spark.sql.kafka010.KafkaSourceProvider$.<clinit>(KafkaSourceProvider.scala)
    at org.apache.spark.sql.kafka010.KafkaSourceProvider.org$apache$spark$sql$kafka010$KafkaSourceProvider$$validateBatchOptions(KafkaSourceProvider.scala:336)
    at org.apache.spark.sql.kafka010.KafkaSourceProvider.createRelation(KafkaSourceProvider.scala:127)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:355)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:325)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:307)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:307)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:225)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.serialization.ByteArraySerializer
    at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)

I've checked the Spark version on the Dataproc cluster, and spark version - 3.1.2,and scala version - 2.12 .. so the version of the spark-sql jar being passed seems to be correct.
Are there any other jars to be passed ?

What needs to be done to fix/debug this issue ?

tia !