问:
Jboss 5, classloader and multiple class instances
I had a problem with my application. To resume the problem, I had to do migrate an application from jboss 4 to jboss 5.

During the war deployement, I had this error:

```<code>java.lang.LinkageError: loader constraint violation: when resolving field "DATETIME"
the class loader (instance of org/jboss/classloader/spi/base/BaseClassLoader) of the referring class,
javax/xml/datatype/DatatypeConstants, and the class loader (instance of <bootloader>)
for the field's resolved type, javax/xml/namespace/QName,
have different Class objects for that type
</code>```

After many searchs, I found this error was here because I had several time the same class in differents packages. Once in a dependency package (from my pom.xml) and once provided by jboss.

So, to resolve this problem, I have give a scope "provided" for my dependency.

But I don't understand why this solution works. I thought it works to have several time the same class in an application. I know it's not a good thing, but with jboss 4, it's work.

Someone can explain me why it works with jboss 4 and not with jboss 5.

Thanks for your explanation :)

答:
What you are seeing is the effect of an application server loading  JBoss' libraries and EAR libraries in separate classloaders

You can think of an EAR's class loader hierarchy similar (but not necessarily) to :

Bootstrap ClassLoader -> System Class Loader -> JBoss System Class Loader -> Ear Class Loader -> War Class Loader.

Where war class loader's parent is the ear class loader and so forth.

Now if Bootstrap ClasssLoader has a jar A loaded and the ear is also being deployed with jar A, Bootstrap Class Lodaer and Ear Class Loader will have the same class created twice in separate class loaders.

I would assume (not 100% sure of this) that JBoss 4 didnt come bundled with javax/xml/namespace/QName. If that is true JBoss 5 is more then likely a different, upgraded,  version of Java (4 -> 5 or 5 -> 6).  As a result (with the new JBoss 5), when you try to pass javax/xml/namespace/QName into one of your classes, it is expecting the class from the ear.  However you are giving it the QName class from the Bootstrap classloader because of the classloader preferences (parent first and so forth).

Since the class types are equal but the class instances are not equal you get a LinkageError

Edit:

Just two address the two comments -

The class loading behavior as  jtahlborn noted is deffinitely different.  In normal applications, the system classes like QName will be consistently looked for in the bootstrap classloader.  In your error it looks as if javax/xml/datatype/DatatypeConstants is being loaded in org/jboss/classloader/spi/base/BaseClassLoader.  Lets assume thats the EAR classloader (or WAR).  A quick google shows that is part of the xml-apis' family  and possibly jaxp-api.

So somewhere in your code (or another libraries code located in the EAR's class loader) needed DatatypeConstants - which forced the lookup of the class in the EAR's classloader.  The creation of the QName object though loaded the class from the bootstrap classloader (instead of the EAR).  This would happen if the QName class has already been initialized by the system, which it probably has.

This as you can imagine isn't suppose to happen.  It actually looks like you have parent-last.  Because when loading a class off the JBoss class loading mechanism, had parent-first been enabled, the initial DatatypeConstants would have returned the parent's (bootstrap) DatatypeConstants and not the childs.  So as jtahlborn noted you would want the children's classloader here to be ignored.

As far as the resolution goes, unless you need the dependencies for a specific reason (like a slightly newer version is better then the current) I would delegate to the jboss's implementation.  If thats not the case, you can take a look at the class-loading java2ClassLoadingCompliance element that the jboss configuration has.

Body:
I had a problem with my application. To resume the problem, I had to do migrate an application from jboss 4 to jboss 5.

During the war deployement, I had this error:

java.lang.LinkageError: loader constraint violation: when resolving field "DATETIME"
the class loader (instance of org/jboss/classloader/spi/base/BaseClassLoader) of the referring class,
javax/xml/datatype/DatatypeConstants, and the class loader (instance of <bootloader>)
for the field's resolved type, javax/xml/namespace/QName,
have different Class objects for that type

After many searchs, I found this error was here because I had several time the same class in differents packages. Once in a dependency package (from my pom.xml) and once provided by jboss.

So, to resolve this problem, I have give a scope "provided" for my dependency.

But I don't understand why this solution works. I thought it works to have several time the same class in an application. I know it's not a good thing, but with jboss 4, it's work.

Someone can explain me why it works with jboss 4 and not with jboss 5.

Thanks for your explanation :)


来源: stackexchange

问:
Submit a form in a popup, and then close the popup
This seemed so trivial when I started off with it! My objective is this:

When user clicks on a button, open up a form in a new window.

Once the form is submitted, close the popup and return to the original page. The reverse order is fine too - i.e., I am okay if the popup is closed and then the form submits.

Here's how I'm doing this:

```<code><form action="/system/wpacert" method="post" enctype="multipart/form-data" onsubmit="return closeSelf()" name="certform">
                <div>Certificate 1: <input type="file" name="cert1"/></div>
                <div>Certificate 2: <input type="file" name="cert2"/></div>
                <div>Certificate 3: <input type="file" name="cert3"/></div>

                <div><input type="submit" value="Upload"/></div>
            </form>
</code>```

and the Javascript looks like this:

```<code><script type="text/javascript">
function closeSelf(){
    self.close();
    return true;
}
</script>
</code>```

This seems to work fine on IE8 and Chrome; but Firefox refuses to submit the form; it just closes the popup. What might I be doing wrong?

EDIT:

Had forgotten to post the code the opens the popup. Here it is:

```<code><div><input type="submit" value="Upload Certificates" onclick="popupUploadForm()"/>
</code>```

And the corresponding javascript:

```<code>function popupUploadForm(){
        var newWindow = window.open('/cert.html', 'name', 'height=500,width=600');
    }
</code>```

答:
Try like this as well</p>

```<code>covertPostSub("/xyz/test.jsp","?param1=param1&param2=param2","_self","true");
covertPostSub("/xyz/test.jsp","?param1=param1&param2=param2","_blank","true");

var convPop = null;
function covertPostSub(action,paramsTosend,targetIframe,isWindow){
    var Popup = null;
    var form = document.createElement("form");
    form.setAttribute("method", "POST");
    form.setAttribute("id","TheForm");
    form.setAttribute("action", action);
    form.setAttribute("target", targetIframe);
    var params = paramsTosend;
    params = params.substring(1, params.length);
    params = params.split("&");
    for(var key=0; key<params.length; key++) {
        var sa = params[key];
        sa = sa.split("=");
        var xs = (sa[1]);

        if(params.hasOwnProperty(key)) {
            var hiddenField = document.createElement("input");
            hiddenField.setAttribute("type", "hidden");
            hiddenField.setAttribute("name", sa[0]);
            hiddenField.setAttribute("value",xs);

            form.appendChild(hiddenField);
         }
    }
    document.body.appendChild(form);
    form.style.display = "none";
    if(isWindow){
        window.open('', "formpopup","width=900,height=590,toolbar=no,scrollbars=yes,resizable=no,location=0,directories=0,status=1,menubar=0,left=60,top=60");
        form.target = 'formpopup';
        form.submit();
    }else{
        form.submit();
    }

}
</code>```

ContentLicense: CC BY-SA 3.0

Body:
This seemed so trivial when I started off with it! My objective is this:

When user clicks on a button, open up a form in a new window.

Once the form is submitted, close the popup and return to the original page. The reverse order is fine too - i.e., I am okay if the popup is closed and then the form submits.

Here's how I'm doing this:

<form action="/system/wpacert" method="post" enctype="multipart/form-data" onsubmit="return closeSelf()" name="certform">

Certificate 1: <input type="file" name="cert1"/>

Certificate 2: <input type="file" name="cert2"/>

Certificate 3: <input type="file" name="cert3"/>

<input type="submit" value="Upload"/>

            </form>

and the Javascript looks like this:

This seems to work fine on IE8 and Chrome; but Firefox refuses to submit the form; it just closes the popup. What might I be doing wrong?

EDIT:

Had forgotten to post the code the opens the popup. Here it is:

<input type="submit" value="Upload Certificates" onclick="popupUploadForm()"/>

And the corresponding javascript:

function popupUploadForm(){
        var newWindow = window.open('/cert.html', 'name', 'height=500,width=600');
    }

Body:
Try like this as well

covertPostSub("/xyz/test.jsp","?param1=param1&param2=param2","_self","true");
covertPostSub("/xyz/test.jsp","?param1=param1&param2=param2","_blank","true");

var convPop = null;
function covertPostSub(action,paramsTosend,targetIframe,isWindow){
    var Popup = null;
    var form = document.createElement("form");
    form.setAttribute("method", "POST");
    form.setAttribute("id","TheForm");
    form.setAttribute("action", action);
    form.setAttribute("target", targetIframe);
    var params = paramsTosend;
    params = params.substring(1, params.length);
    params = params.split("&");
    for(var key=0; key<params.length; key++) {
        var sa = params[key];
        sa = sa.split("=");
        var xs = (sa[1]);

        if(params.hasOwnProperty(key)) {
            var hiddenField = document.createElement("input");
            hiddenField.setAttribute("type", "hidden");
            hiddenField.setAttribute("name", sa[0]);
            hiddenField.setAttribute("value",xs);

            form.appendChild(hiddenField);
         }
    }
    document.body.appendChild(form);
    form.style.display = "none";
    if(isWindow){
        window.open('', "formpopup","width=900,height=590,toolbar=no,scrollbars=yes,resizable=no,location=0,directories=0,status=1,menubar=0,left=60,top=60");
        form.target = 'formpopup';
        form.submit();
    }else{
        form.submit();
    }

}

ContentLicense: CC BY-SA 4.0


问:
How do I check if a given user ID has authenticated my app?
I'm using the Facebook Graph API and want to check if a user has authenticated my Facebook app by user ID. How do I do this? </p>

答:
Expanding on ifaour's answer, in PHP this query would look something like this:</p>

```<code><?php
    $facebook = new Facebook(
        'appID' => YOUR_APP_ID,
        'secret' => YOUR_APP_SECRET
    );

    $result = $facebook->api(array(
        'method' => 'fql.query',
        'query' => "SELECT is_app_user FROM user WHERE uid=$user_id"
    ));

    $is_installed = $result[0]['is_app_user'];
</code>```

Body:
Expanding on ifaour's answer, in PHP this query would look something like this:

<?php
    $facebook = new Facebook(
        'appID' => YOUR_APP_ID,
        'secret' => YOUR_APP_SECRET
    );

    $result = $facebook->api(array(
        'method' => 'fql.query',
        'query' => "SELECT is_app_user FROM user WHERE uid=$user_id"
    ));

    $is_installed = $result[0]['is_app_user'];


问:
Watson Knowledge Catalog Data Source Connectivity
Is there the ability to profile data from cloud repositories such as Azure storage, Google Storage, etc ?</p>

答:
Yes, data profiling is supported for data from cloud repositories such as Azure, AWS, Google in addition to IBM Cloud.

Those connectors are supported, and here's the full list of supported connectors in Watson Knowledge Catalog:
https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/conn_types.html?audience=wdp&context=wdp


问:
lz4 exception when reading data from kafka using spark streaming
I am trying to read json data from kafka using spark streaming api, when I do that it throws java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.init exception. Stack trace is -

```<code>java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
at org.apache.spark.serializer.SerializerManager.wrapStream(SerializerManager.scala:124)
at org.apache.spark.shuffle.BlockStoreShuffleReader$$anonfun$3.apply(BlockStoreShuffleReader.scala:50)
at org.apache.spark.shuffle.BlockStoreShuffleReader$$anonfun$3.apply(BlockStoreShuffleReader.scala:50)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:421)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:61)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:30)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.agg_doAggregateWithKeys_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at org.apache.spark.sql.execution.streaming.StateStoreRestoreExec$$anonfun$doExecute$1.apply(statefulOperators.scala:217)
at org.apache.spark.sql.execution.streaming.StateStoreRestoreExec$$anonfun$doExecute$1.apply(statefulOperators.scala:215)
at org.apache.spark.sql.execution.streaming.state.package$StateStoreOps$$anonfun$1.apply(package.scala:67)
at org.apache.spark.sql.execution.streaming.state.package$StateStoreOps$$anonfun$1.apply(package.scala:62)
at org.apache.spark.sql.execution.streaming.state.StateStoreRDD.compute(StateStoreRDD.scala:78)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.sql.execution.streaming.state.StateStoreRDD.compute(StateStoreRDD.scala:77)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
</code>```

my pom.xml file is having following depedencies-

```<code>    <!-- https://mvnrepository.com/artifact/net.jpountz.lz4/lz4 -->
    <dependency>
        <groupId>net.jpountz.lz4</groupId>
        <artifactId>lz4</artifactId>
        <version>1.3.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.3.1</version>
        <exclusions>
            <exclusion>
                <artifactId>lz4-java</artifactId>
                <groupId>org.lz4</groupId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.3.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql-kafka-0-10_2.11</artifactId>
        <version>2.3.1</version>
        <scope>provided</scope>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka -->
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.11</artifactId>
        <version>1.1.0</version>
    </dependency>
</code>```

And the spark streaming class to show how I am trying the read kafka value as string and then parsing the same into Person class using custom parser-

```<code>  public static void main( String[] args ) throws Exception
{
    if( args.length < 3 )
    {
        System.err
                .println("Usage: JavaStructuredKafkaWordCount <bootstrap-servers> " + "<subscribe-type> <topics>");
        System.exit(1);
    }

    String bootstrapServers = args[0];
    String subscribeType = args[1];
    String topics = args[2];

    SparkSession spark = SparkSession.builder().appName("JavaStructuredKafkaWordCount")
            .config("spark.master", "local").getOrCreate();

    // Create DataSet representing the stream of input lines from kafka
    Dataset<String> df = spark.readStream().format("kafka").option("kafka.bootstrap.servers", bootstrapServers)
            .option(subscribeType, topics).load().selectExpr("CAST(value AS STRING)").as(Encoders.STRING());

    Dataset<Person> stringMein = df.map(
            (MapFunction<String, Person>) row -> JSONToPerson.parseJsonToPerson(row),
            Encoders.bean(Person.class));

    //stringMein.printSchema();
    // Generate running word count
    Dataset<Row> cardDF = stringMein.groupBy("age").count();
    // Start running the query that prints the running counts to the console
    StreamingQuery query = cardDF.writeStream().outputMode("update").format("console").start();

    query.awaitTermination();
}
</code>```

}

答:
Adding the next dependency , works for me: </p>

```<code><dependency>
<groupId>net.jpountz.lz4</groupId>
<artifactId>lz4</artifactId>
<version>1.3.0</version>
</dependency>enter code here
</code>```

Body:
I am trying to read json data from kafka using spark streaming api, when I do that it throws java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.init exception. Stack trace is -

java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
at org.apache.spark.serializer.SerializerManager.wrapStream(SerializerManager.scala:124)
at org.apache.spark.shuffle.BlockStoreShuffleReader$$anonfun$3.apply(BlockStoreShuffleReader.scala:50)
at org.apache.spark.shuffle.BlockStoreShuffleReader$$anonfun$3.apply(BlockStoreShuffleReader.scala:50)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:421)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:61)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:30)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.agg_doAggregateWithKeys_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at org.apache.spark.sql.execution.streaming.StateStoreRestoreExec$$anonfun$doExecute$1.apply(statefulOperators.scala:217)
at org.apache.spark.sql.execution.streaming.StateStoreRestoreExec$$anonfun$doExecute$1.apply(statefulOperators.scala:215)
at org.apache.spark.sql.execution.streaming.state.package$StateStoreOps$$anonfun$1.apply(package.scala:67)
at org.apache.spark.sql.execution.streaming.state.package$StateStoreOps$$anonfun$1.apply(package.scala:62)
at org.apache.spark.sql.execution.streaming.state.StateStoreRDD.compute(StateStoreRDD.scala:78)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.sql.execution.streaming.state.StateStoreRDD.compute(StateStoreRDD.scala:77)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

my pom.xml file is having following depedencies-

    <dependency>
        <groupId>net.jpountz.lz4</groupId>
        <artifactId>lz4</artifactId>
        <version>1.3.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.3.1</version>
        <exclusions>
            <exclusion>
                <artifactId>lz4-java</artifactId>
                <groupId>org.lz4</groupId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.3.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql-kafka-0-10_2.11</artifactId>
        <version>2.3.1</version>
        <scope>provided</scope>
    </dependency>

    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.11</artifactId>
        <version>1.1.0</version>
    </dependency>

And the spark streaming class to show how I am trying the read kafka value as string and then parsing the same into Person class using custom parser-

  public static void main( String[] args ) throws Exception
{
    if( args.length < 3 )
    {
        System.err
                .println("Usage: JavaStructuredKafkaWordCount <bootstrap-servers> " + "<subscribe-type> <topics>");
        System.exit(1);
    }

    String bootstrapServers = args[0];
    String subscribeType = args[1];
    String topics = args[2];

    SparkSession spark = SparkSession.builder().appName("JavaStructuredKafkaWordCount")
            .config("spark.master", "local").getOrCreate();

    // Create DataSet representing the stream of input lines from kafka
    Dataset<String> df = spark.readStream().format("kafka").option("kafka.bootstrap.servers", bootstrapServers)
            .option(subscribeType, topics).load().selectExpr("CAST(value AS STRING)").as(Encoders.STRING());

    Dataset<Person> stringMein = df.map(
            (MapFunction<String, Person>) row -> JSONToPerson.parseJsonToPerson(row),
            Encoders.bean(Person.class));

    //stringMein.printSchema();
    // Generate running word count
    Dataset<Row> cardDF = stringMein.groupBy("age").count();
    // Start running the query that prints the running counts to the console
    StreamingQuery query = cardDF.writeStream().outputMode("update").format("console").start();

    query.awaitTermination();
}

}

Body:
Adding the next dependency , works for me:

<dependency>
<groupId>net.jpountz.lz4</groupId>
<artifactId>lz4</artifactId>
<version>1.3.0</version>
</dependency>enter code here


问:
how to set up RGB leds on arduino ethernet with udp instruction?
I'm working on a kind of traffic light with RGB LEDs. I try to take a udp value sent from a server to the arduino mega and ethernet shield, then the arduino should change the color of the LED.

Sadly until now, it does not work. In the serial monitor, I find out that the udp packet was received, but then the LED does not work. I was hoping that you guys can help me to figure out why my code isn't working. Thank you in advance! Here is my code:

```<code>/*
  Web Server
 */

#include <SPI.h>
#include <Ethernet.h>
#include <EthernetUdp.h>

    void Color(int R, int G, int B) //set up for the RGB led
    {
          analogWrite(3, R) ;   // Rojo
          analogWrite(5, G) ;   // Green - Verde
          analogWrite(6, B) ;   // Blue - Azul
    }

// Enter a MAC address and IP address for your controller below.
// The IP address will be dependent on your local network:
byte mac[] = {
  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED};
IPAddress ip(10, 90, 111, 150);

unsigned int localPort = 8888;      // local port to listen on

// buffers for receiving and sending data
char packetBuffer[UDP_TX_PACKET_MAX_SIZE];  // buffer to hold incoming packet,
char ReplyBuffer[] = "acknowledged";        // a string to send back

// An EthernetUDP instance to let us send and receive packets over UDP
EthernetUDP Udp;

// Initialize the Ethernet server library
// with the IP address and port you want to use
// (port 80 is default for HTTP):
EthernetServer server(80);

void setup() {  //seting up the outputs for the rgb
       pinMode(3, OUTPUT);
       pinMode(5, OUTPUT);
       pinMode(6, OUTPUT);

  // You can use Ethernet.init(pin) to configure the CS pin
  //Ethernet.init(10);  // Most Arduino shields
  //Ethernet.init(5);   // MKR ETH shield
  //Ethernet.init(0);   // Teensy 2.0
  //Ethernet.init(20);  // Teensy++ 2.0
  //Ethernet.init(15);  // ESP8266 with Adafruit Featherwing Ethernet
  //Ethernet.init(33);  // ESP32 with Adafruit Featherwing Ethernet

    // start the Ethernet connection and the server:
  Ethernet.begin(mac, ip);

  // Open serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }
  Serial.println("Ethernet WebServer Example");

  // Check for Ethernet hardware present
  if (Ethernet.hardwareStatus() == EthernetNoHardware) {
    Serial.println("Ethernet shield was not found.  Sorry, can't run without hardware. :(");
    while (true) {
      delay(1); // do nothing, no point running without Ethernet hardware
    }
  }
  if (Ethernet.linkStatus() == LinkOFF) {
    Serial.println("Ethernet cable is not connected.");
  }

  // start the server
  Udp.begin(localPort);
  server.begin();
  Serial.print("server is at ");
  Serial.println(Ethernet.localIP());
}

void loop() {

// if there's data available, read a packet
  int packetSize = Udp.parsePacket();
  if (packetSize) {
    Serial.print("Received packet of size ");
    Serial.println(packetSize);
    Serial.print("From ");
    IPAddress remote = Udp.remoteIP();
    for (int i=0; i < 4; i++) {
      Serial.print(remote[i], DEC);
      if (i < 3) {
        Serial.print(".");
      }
    }
    Serial.print(", port ");
    Serial.println(Udp.remotePort());

    // read the packet into packetBufffer
    Udp.read(packetBuffer, UDP_TX_PACKET_MAX_SIZE);
    Serial.println("Contents:");
    Serial.println(packetBuffer);

    // send a reply to the IP address and port that sent us the packet we received
    Udp.beginPacket(Udp.remoteIP(), Udp.remotePort());
    Udp.write(ReplyBuffer);
    Udp.endPacket();

  }
       char numero = packetBuffer;
         if (numero = "1") Color(250, 0, 0) ;
         if (numero = "2") Color(100, 110, 0);
         if (numero = "3") Color(0, 255, 0);
  delay(10);

}
</code>```

From now, it just receives the udp, and while compiling I get:

```<code>warning: invalid conversion from 'const char*' to 'char' [-fpermissive]
</code>```

but in the end, it compiles.

edited:

i changed the code in this way (just in the final part), and now when i send the udp packet, no matter what i send, i get the green light on, but only the green one, here is the code

```<code>  void loop() {

// if there's data available, read a packet
  int packetSize = Udp.parsePacket();
  if (packetSize) {
    Serial.print("Received packet of size ");
    Serial.println(packetSize);
    Serial.print("From ");
    IPAddress remote = Udp.remoteIP();
    for (int i=0; i < 4; i++) {
      Serial.print(remote[i], DEC);
      if (i < 3) {
        Serial.print(".");
      }
    }
    Serial.print(", port ");
    Serial.println(Udp.remotePort());

    // read the packet into packetBufffer
    Udp.read(packetBuffer, UDP_TX_PACKET_MAX_SIZE);
    Serial.println("Contents:");
    Serial.println(packetBuffer);
    char numero =  packetBuffer;
         if (numero = 'R') Color(250, 0, 0) ;
         if (numero = 'A') Color(100, 100, 0);
         if (numero = 'V') Color(0, 255, 0);
    // send a reply to the IP address and port that sent us the packet we received
    Udp.beginPacket(Udp.remoteIP(), Udp.remotePort());
    Udp.write(ReplyBuffer);
    Udp.endPacket();

  }
</code>```

答:
I'd recommend to find some C++ tutorial for starters.

Anyway:

```<code>char numero =  packetBuffer; // wrong, you are assigning part of address to char, not character at that address...
if (numero = 'R') Color(250, 0, 0) ; // better but still wrong - numero is set to 'R' and it's also always true
if (numero = 'A') Color(100, 100, 0); // similar to previous
if (numero = 'V') Color(0, 255, 0);   // and again
</code>```

Correct version:

```<code>char numero =  packetBuffer[0]; // get the first character in buffer
if (numero == 'R') Color(250, 0, 0) ; // compare numero with 'R'
if (numero == 'A') Color(100, 100, 0); // compare numero with 'A'
if (numero == 'V') Color(0, 255, 0);   // ...
</code>```

If there is nothing else wrong, that should fix it

Body:
I'm working on a kind of traffic light with RGB LEDs. I try to take a udp value sent from a server to the arduino mega and ethernet shield, then the arduino should change the color of the LED.

Sadly until now, it does not work. In the serial monitor, I find out that the udp packet was received, but then the LED does not work. I was hoping that you guys can help me to figure out why my code isn't working. Thank you in advance! Here is my code:

/*
  Web Server
 */

#include <SPI.h>
#include <Ethernet.h>
#include <EthernetUdp.h>

    void Color(int R, int G, int B) //set up for the RGB led
    {
          analogWrite(3, R) ;   // Rojo
          analogWrite(5, G) ;   // Green - Verde
          analogWrite(6, B) ;   // Blue - Azul
    }

// Enter a MAC address and IP address for your controller below.
// The IP address will be dependent on your local network:
byte mac[] = {
  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED};
IPAddress ip(10, 90, 111, 150);

unsigned int localPort = 8888;      // local port to listen on

// buffers for receiving and sending data
char packetBuffer[UDP_TX_PACKET_MAX_SIZE];  // buffer to hold incoming packet,
char ReplyBuffer[] = "acknowledged";        // a string to send back

// An EthernetUDP instance to let us send and receive packets over UDP
EthernetUDP Udp;

// Initialize the Ethernet server library
// with the IP address and port you want to use
// (port 80 is default for HTTP):
EthernetServer server(80);

void setup() {  //seting up the outputs for the rgb
       pinMode(3, OUTPUT);
       pinMode(5, OUTPUT);
       pinMode(6, OUTPUT);

  // You can use Ethernet.init(pin) to configure the CS pin
  //Ethernet.init(10);  // Most Arduino shields
  //Ethernet.init(5);   // MKR ETH shield
  //Ethernet.init(0);   // Teensy 2.0
  //Ethernet.init(20);  // Teensy++ 2.0
  //Ethernet.init(15);  // ESP8266 with Adafruit Featherwing Ethernet
  //Ethernet.init(33);  // ESP32 with Adafruit Featherwing Ethernet

    // start the Ethernet connection and the server:
  Ethernet.begin(mac, ip);

  // Open serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }
  Serial.println("Ethernet WebServer Example");

  // Check for Ethernet hardware present
  if (Ethernet.hardwareStatus() == EthernetNoHardware) {
    Serial.println("Ethernet shield was not found.  Sorry, can't run without hardware. :(");
    while (true) {
      delay(1); // do nothing, no point running without Ethernet hardware
    }
  }
  if (Ethernet.linkStatus() == LinkOFF) {
    Serial.println("Ethernet cable is not connected.");
  }

  // start the server
  Udp.begin(localPort);
  server.begin();
  Serial.print("server is at ");
  Serial.println(Ethernet.localIP());
}

void loop() {

// if there's data available, read a packet
  int packetSize = Udp.parsePacket();
  if (packetSize) {
    Serial.print("Received packet of size ");
    Serial.println(packetSize);
    Serial.print("From ");
    IPAddress remote = Udp.remoteIP();
    for (int i=0; i < 4; i++) {
      Serial.print(remote[i], DEC);
      if (i < 3) {
        Serial.print(".");
      }
    }
    Serial.print(", port ");
    Serial.println(Udp.remotePort());

    // read the packet into packetBufffer
    Udp.read(packetBuffer, UDP_TX_PACKET_MAX_SIZE);
    Serial.println("Contents:");
    Serial.println(packetBuffer);

    // send a reply to the IP address and port that sent us the packet we received
    Udp.beginPacket(Udp.remoteIP(), Udp.remotePort());
    Udp.write(ReplyBuffer);
    Udp.endPacket();

  }
       char numero = packetBuffer;
         if (numero = "1") Color(250, 0, 0) ;
         if (numero = "2") Color(100, 110, 0);
         if (numero = "3") Color(0, 255, 0);
  delay(10);

}

From now, it just receives the udp, and while compiling I get:

warning: invalid conversion from 'const char*' to 'char' [-fpermissive]

but in the end, it compiles.

edited:

i changed the code in this way (just in the final part), and now when i send the udp packet, no matter what i send, i get the green light on, but only the green one, here is the code

  void loop() {

// if there's data available, read a packet
  int packetSize = Udp.parsePacket();
  if (packetSize) {
    Serial.print("Received packet of size ");
    Serial.println(packetSize);
    Serial.print("From ");
    IPAddress remote = Udp.remoteIP();
    for (int i=0; i < 4; i++) {
      Serial.print(remote[i], DEC);
      if (i < 3) {
        Serial.print(".");
      }
    }
    Serial.print(", port ");
    Serial.println(Udp.remotePort());

    // read the packet into packetBufffer
    Udp.read(packetBuffer, UDP_TX_PACKET_MAX_SIZE);
    Serial.println("Contents:");
    Serial.println(packetBuffer);
    char numero =  packetBuffer;
         if (numero = 'R') Color(250, 0, 0) ;
         if (numero = 'A') Color(100, 100, 0);
         if (numero = 'V') Color(0, 255, 0);
    // send a reply to the IP address and port that sent us the packet we received
    Udp.beginPacket(Udp.remoteIP(), Udp.remotePort());
    Udp.write(ReplyBuffer);
    Udp.endPacket();

  }

Body:
I'd recommend to find some C++ tutorial for starters.

Anyway:

char numero =  packetBuffer; // wrong, you are assigning part of address to char, not character at that address...
if (numero = 'R') Color(250, 0, 0) ; // better but still wrong - numero is set to 'R' and it's also always true
if (numero = 'A') Color(100, 100, 0); // similar to previous
if (numero = 'V') Color(0, 255, 0);   // and again

Correct version:

char numero =  packetBuffer[0]; // get the first character in buffer
if (numero == 'R') Color(250, 0, 0) ; // compare numero with 'R'
if (numero == 'A') Color(100, 100, 0); // compare numero with 'A'
if (numero == 'V') Color(0, 255, 0);   // ...

If there is nothing else wrong, that should fix it


问:
Why do I get System.String [ ] in my code while debugging?
I am on my final project in c# for beginners, but I can't seem to get my list to work properly. It also closes automatically after being 'filled' with orders.

```<code>class Burgare
{
    private string[] hamburger = new string[24];
    private int no_burger = 0;

    public void add_burger()//Ordering of burgers
    {
        Console.Clear(); //Clears console to make it look cleaner
        while (true)
        {
            int Burger_option = 0;
            do
            {
                Console.WriteLine("");// The options user can choose from
                Console.WriteLine("Please choose burgers from our menu:");
                Console.WriteLine("-------Burgers--------");
                Console.WriteLine("1 Original One     109");
                Console.WriteLine("2 Pig & Cow        109");
                Console.WriteLine("3 Spice & Nice     109");
                Console.WriteLine("4 Green One        109");
                Console.WriteLine("0 Go back to main menu");
                Console.WriteLine("----------------------");
                Console.WriteLine("");

                try //Making sure the user only picks what is on the menu
                {
                    Burger_option = int.Parse(Console.ReadLine());
                }

                catch
                {
                    Console.WriteLine("You can only choose what is on the menu!");

                }

                switch (Burger_option) //Depending on the number user presses on keyboard different burgers will be added to order
                {
                    case 1:
                        Console.WriteLine("You've added the Original One to your order");
                        Console.WriteLine("If you're done press 0 to go back to the main menu");
                        hamburger[no_burger] = "Original One";
                        break;

                    case 2:
                        Console.WriteLine("You've added the Pig & Cow to your order");
                        Console.WriteLine("If you're done press 0 to go back to the main menu");
                        hamburger[no_burger] = "Pig & Cow";
                        break;

                    case 3:
                        Console.WriteLine("You've added the Spice & Nice to your order");
                        Console.WriteLine("If you're done press 0 to go back to the main menu");
                        hamburger[no_burger] = "Spice & Nice";
                        break;

                    case 4:
                        Console.WriteLine("You've added the Green One to your order");
                        Console.WriteLine("If you're done press 0 to go back to the main menu");
                        hamburger[no_burger] = "Green One";
                        break;

                    case 0: //Sends user back to main menu
                        Run();
                        break;
                }
                no_burger++; //Adds burger

            } while (no_burger != 0);

            if (no_burger <= 25) //Maximum number of orders
            {
                Console.WriteLine("The restaurant can't take more than 24 orders of burgers at the time");
                Run(); //Sends user back to main menu when the order is over 24
            }
        }
    }

    public void print_order()
    {

        Console.WriteLine("-Your current order-");

    foreach (var burger in hamburger) //Showing a list of what is in the order
        {
         Console.WriteLine(hamburger);

         if (burger != null)

                Console.WriteLine(burger);
         else
                Console.WriteLine("Empty space"); //Making empty space to show where you can add more burgers
        }

    }
</code>```

After entering 24 orders I get the error "System.IndexOutOfRangeException: 'Index was outside the bounds of the array.'" I've looked around a bit but still don't really understand how to fix my code. I'd like for it to return to the main menu after telling the user the order is full.

For my second problem the System.String [] problem, it appears when I enter the 'print_order'. It shows itself as
System.String []
Empty space
System.String []
Empty space
And so on. If possible I'd like to either remove it completely or at least replace it with 1,2,3...etc.

答:
You never actually check to see how many items you've added inside your do...while loop - you only check this after the loop is complete. That's why it crashes when you try to add the 25th item.

Move the logic to handle this inside your do...while loop and you won't get the exception anymore.

Also, your array only has room for 24 items, but I assume that you meant for it to be 25.

Also, since you're new to programming, you may want to look at @Eric Lppert's helpful article on debugging techniques - it'll save you a lot of time. You should also read about how to use a step debugger to diagnose problems.

Body:
I am on my final project in c# for beginners, but I can't seem to get my list to work properly. It also closes automatically after being 'filled' with orders.

class Burgare
{
    private string[] hamburger = new string[24];
    private int no_burger = 0;

    public void add_burger()//Ordering of burgers
    {
        Console.Clear(); //Clears console to make it look cleaner
        while (true)
        {
            int Burger_option = 0;
            do
            {
                Console.WriteLine("");// The options user can choose from
                Console.WriteLine("Please choose burgers from our menu:");
                Console.WriteLine("-------Burgers--------");
                Console.WriteLine("1 Original One     109");
                Console.WriteLine("2 Pig & Cow        109");
                Console.WriteLine("3 Spice & Nice     109");
                Console.WriteLine("4 Green One        109");
                Console.WriteLine("0 Go back to main menu");
                Console.WriteLine("----------------------");
                Console.WriteLine("");

                try //Making sure the user only picks what is on the menu
                {
                    Burger_option = int.Parse(Console.ReadLine());
                }

                catch
                {
                    Console.WriteLine("You can only choose what is on the menu!");

                }

                switch (Burger_option) //Depending on the number user presses on keyboard different burgers will be added to order
                {
                    case 1:
                        Console.WriteLine("You've added the Original One to your order");
                        Console.WriteLine("If you're done press 0 to go back to the main menu");
                        hamburger[no_burger] = "Original One";
                        break;

                    case 2:
                        Console.WriteLine("You've added the Pig & Cow to your order");
                        Console.WriteLine("If you're done press 0 to go back to the main menu");
                        hamburger[no_burger] = "Pig & Cow";
                        break;

                    case 3:
                        Console.WriteLine("You've added the Spice & Nice to your order");
                        Console.WriteLine("If you're done press 0 to go back to the main menu");
                        hamburger[no_burger] = "Spice & Nice";
                        break;

                    case 4:
                        Console.WriteLine("You've added the Green One to your order");
                        Console.WriteLine("If you're done press 0 to go back to the main menu");
                        hamburger[no_burger] = "Green One";
                        break;

                    case 0: //Sends user back to main menu
                        Run();
                        break;
                }
                no_burger++; //Adds burger

            } while (no_burger != 0);

            if (no_burger <= 25) //Maximum number of orders
            {
                Console.WriteLine("The restaurant can't take more than 24 orders of burgers at the time");
                Run(); //Sends user back to main menu when the order is over 24
            }
        }
    }

    public void print_order()
    {

        Console.WriteLine("-Your current order-");

    foreach (var burger in hamburger) //Showing a list of what is in the order
        {
         Console.WriteLine(hamburger);

         if (burger != null)

                Console.WriteLine(burger);
         else
                Console.WriteLine("Empty space"); //Making empty space to show where you can add more burgers
        }

    }

After entering 24 orders I get the error "System.IndexOutOfRangeException: 'Index was outside the bounds of the array.'" I've looked around a bit but still don't really understand how to fix my code. I'd like for it to return to the main menu after telling the user the order is full.

For my second problem the System.String [] problem, it appears when I enter the 'print_order'. It shows itself as
System.String []
Empty space
System.String []
Empty space
And so on. If possible I'd like to either remove it completely or at least replace it with 1,2,3...etc.


问:
Oracle SQL INNER JOIN and Count(*)=1 doesn't get rid of duplicates
My original Oracle SQL Query (at end of post) outputs

but I want it to output this

I was advised to use  INNER JOIN to remove any duplicates.

This didn't work.

Then I was advised to use HAVING COUNT(*)=1

This partially works

Instead of there being 3 or 4 duplicates, there is at most 2 duplicates.

Any ideas?

Original SQL Query

```<code>SELECT TO_CHAR(MIN(I.INCIDENTID))                  AS "Incident ID",
          MIN(I.CREATIONDATE)                      AS "Creation Date",
          TO_CHAR(I.CREATIONDATE,'MM-DD-YYYY')     AS "Date",
          TRIM(MO.DOMAINUSERNAME)                  AS "Login ID",
          TRIM(M.MESSAGESUBJECT)                   AS "Email Subject"
   FROM  INCIDENT I
   JOIN  MESSAGE M
   ON M.MESSAGEID = I.MESSAGEID
   JOIN   MESSAGEORIGINATOR MO
   ON     M.MESSAGEORIGINATORID = MO.MESSAGEORIGINATORID
   GROUP BY TO_CHAR(I.CREATIONDATE,'MM-DD-YYYY'),
            TRIM(MO.DOMAINUSERNAME),
            TRIM(M.MESSAGESUBJECT)
</code>```

Identical query, except I added INNER JOIN and HAVING COUNT(*)=1

```<code>SELECT TO_CHAR(MIN(I.INCIDENTID))                  AS "Incident ID",
          MIN(I.CREATIONDATE)                      AS "Creation Date",
          TO_CHAR(I.CREATIONDATE,'MM-DD-YYYY')     AS "Date",
          TRIM(MO.DOMAINUSERNAME)                  AS "Login ID",
          TRIM(M.MESSAGESUBJECT)                   AS "Email Subject"
   FROM  INCIDENT I
   INNER JOIN  MESSAGE M
   ON M.MESSAGEID = I.MESSAGEID
   INNER JOIN   MESSAGEORIGINATOR MO
   ON     M.MESSAGEORIGINATORID = MO.MESSAGEORIGINATORID
   GROUP BY TO_CHAR(I.CREATIONDATE,'MM-DD-YYYY'),
            TRIM(MO.DOMAINUSERNAME),
            TRIM(M.MESSAGESUBJECT)
HAVING COUNT(*)=1
</code>```

答:
use row_number()</p>

```<code>  with cte as (  SELECT TO_CHAR(I.INCIDENTID))                  AS "Incident ID",
              (I.CREATIONDATE)                      AS "Creation Date",
              TO_CHAR(I.CREATIONDATE,'MM-DD-YYYY')     AS "Date",
              TRIM(MO.DOMAINUSERNAME)                  AS "Login ID",
              TRIM(M.MESSAGESUBJECT)                   AS "Email Subject",
              row_number() over(partition by TO_CHAR(I.CREATIONDATE,'MM-DD-YYYY'),
            TRIM(MO.DOMAINUSERNAME), TRIM(M.MESSAGESUBJECT) order by TO_CHAR(I.INCIDENTID)),I.CREATIONDATE) rn
       FROM  INCIDENT I
       JOIN  MESSAGE M
       ON M.MESSAGEID = I.MESSAGEID
       JOIN   MESSAGEORIGINATOR MO
       ON     M.MESSAGEORIGINATORID = MO.MESSAGEORIGINATORID
) select * from cte where rn=1
</code>```

Body:
My original Oracle SQL Query (at end of post) outputs

but I want it to output this

I was advised to use  INNER JOIN to remove any duplicates.

This didn't work.

Then I was advised to use HAVING COUNT(*)=1

This partially works

Instead of there being 3 or 4 duplicates, there is at most 2 duplicates.

Any ideas?

Original SQL Query

SELECT TO_CHAR(MIN(I.INCIDENTID))                  AS "Incident ID",
          MIN(I.CREATIONDATE)                      AS "Creation Date",
          TO_CHAR(I.CREATIONDATE,'MM-DD-YYYY')     AS "Date",
          TRIM(MO.DOMAINUSERNAME)                  AS "Login ID",
          TRIM(M.MESSAGESUBJECT)                   AS "Email Subject"
   FROM  INCIDENT I
   JOIN  MESSAGE M
   ON M.MESSAGEID = I.MESSAGEID
   JOIN   MESSAGEORIGINATOR MO
   ON     M.MESSAGEORIGINATORID = MO.MESSAGEORIGINATORID
   GROUP BY TO_CHAR(I.CREATIONDATE,'MM-DD-YYYY'),
            TRIM(MO.DOMAINUSERNAME),
            TRIM(M.MESSAGESUBJECT)

Identical query, except I added INNER JOIN and HAVING COUNT(*)=1

SELECT TO_CHAR(MIN(I.INCIDENTID))                  AS "Incident ID",
          MIN(I.CREATIONDATE)                      AS "Creation Date",
          TO_CHAR(I.CREATIONDATE,'MM-DD-YYYY')     AS "Date",
          TRIM(MO.DOMAINUSERNAME)                  AS "Login ID",
          TRIM(M.MESSAGESUBJECT)                   AS "Email Subject"
   FROM  INCIDENT I
   INNER JOIN  MESSAGE M
   ON M.MESSAGEID = I.MESSAGEID
   INNER JOIN   MESSAGEORIGINATOR MO
   ON     M.MESSAGEORIGINATORID = MO.MESSAGEORIGINATORID
   GROUP BY TO_CHAR(I.CREATIONDATE,'MM-DD-YYYY'),
            TRIM(MO.DOMAINUSERNAME),
            TRIM(M.MESSAGESUBJECT)
HAVING COUNT(*)=1

Body:
use row_number()

  with cte as (  SELECT TO_CHAR(I.INCIDENTID))                  AS "Incident ID",
              (I.CREATIONDATE)                      AS "Creation Date",
              TO_CHAR(I.CREATIONDATE,'MM-DD-YYYY')     AS "Date",
              TRIM(MO.DOMAINUSERNAME)                  AS "Login ID",
              TRIM(M.MESSAGESUBJECT)                   AS "Email Subject",
              row_number() over(partition by TO_CHAR(I.CREATIONDATE,'MM-DD-YYYY'),
            TRIM(MO.DOMAINUSERNAME), TRIM(M.MESSAGESUBJECT) order by TO_CHAR(I.INCIDENTID)),I.CREATIONDATE) rn
       FROM  INCIDENT I
       JOIN  MESSAGE M
       ON M.MESSAGEID = I.MESSAGEID
       JOIN   MESSAGEORIGINATOR MO
       ON     M.MESSAGEORIGINATORID = MO.MESSAGEORIGINATORID
) select * from cte where rn=1


问:
Slow populating of Java JFrame Application with NetBeans 11.0
I'm learning Java so I started off by creating a Digital Clock JFrame Form.  It all works, but upon load of the application, it takes about 2/3 second to display my border and my label on the background panel.

File: DigitalClock.java

```<code>package digitalclock;

import JForms.DigitalClockJForm;

public class DigitalClock {

    public static void main(String[] args) {
        new DigitalClockJForm().setVisible(true);

    }

}
</code>```

File: DigitalClockJForm.java

```<code>package JForms;

import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Date;
import java.util.GregorianCalendar;

public class DigitalClockJForm extends javax.swing.JFrame
{
    // Member Variables
    private int second, minute, hour, day, month, year;
    String timeDate;

    // Constructors
    public DigitalClockJForm() {
        initComponents();
        clock();
    }

    public void clock()
    {
        Thread t = new Thread(){

            @Override
            public void run(){

                try {
                    while (true){
                        Calendar cal = new GregorianCalendar();

                        day = cal.get(Calendar.DAY_OF_MONTH);
                        month = cal.get(Calendar.MONTH);
                        year = cal.get(Calendar.YEAR);

                        second = cal.get(Calendar.SECOND);
                        minute = cal.get(Calendar.MINUTE);
                        hour = cal.get(Calendar.HOUR);

                        SimpleDateFormat sdf = new SimpleDateFormat("hh:mm:ss a dd/MM/yyyy");

                        Date date = cal.getTime();

                        timeDate = sdf.format(date);

                        mainLabel.setText(timeDate);
                    }
                }
                catch (Exception e) {
                    e.printStackTrace();
                }
            }
        };

        t.start();
    }

 << Net Beans generated code >>

// Variables declaration - do not modify
private javax.swing.JLabel mainLabel;
private javax.swing.JPanel mainPanel;
// End of variables declaration
</code>```

The Label and Panel take 2/3 seconds to display on the form.

The form launches like this:

Then after 2/3 seconds, everything, including border is displayed correctly:

I would have expected the label and panel to display as soon as the application is launched.

答:
There is no need to subclass JFrame, or even JPanel in your case, since you are not overriding any of their methods.
Just create a JPanel and add your components to it, then add that JPanel to a new JFrame, pack the frame, and display it.

Create a Swing.Timer with a 1000 millisecond delay, and in the actionPerformed method update the displayed time.

It is probably overkill to create a new GregorianCalendar every time the actionPerformed method is called. Updating the display by one second each time should be sufficient, perhaps with a new Calendar every several minutes to re-synch, unless your CPU is extremely busy executing other applications.

Body:
I'm learning Java so I started off by creating a Digital Clock JFrame Form.  It all works, but upon load of the application, it takes about 2/3 second to display my border and my label on the background panel.

File: DigitalClock.java

package digitalclock;

import JForms.DigitalClockJForm;

public class DigitalClock {

    public static void main(String[] args) {
        new DigitalClockJForm().setVisible(true);

    }

}

File: DigitalClockJForm.java

package JForms;

import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Date;
import java.util.GregorianCalendar;

public class DigitalClockJForm extends javax.swing.JFrame
{
    // Member Variables
    private int second, minute, hour, day, month, year;
    String timeDate;

    // Constructors
    public DigitalClockJForm() {
        initComponents();
        clock();
    }

    public void clock()
    {
        Thread t = new Thread(){

            @Override
            public void run(){

                try {
                    while (true){
                        Calendar cal = new GregorianCalendar();

                        day = cal.get(Calendar.DAY_OF_MONTH);
                        month = cal.get(Calendar.MONTH);
                        year = cal.get(Calendar.YEAR);

                        second = cal.get(Calendar.SECOND);
                        minute = cal.get(Calendar.MINUTE);
                        hour = cal.get(Calendar.HOUR);

                        SimpleDateFormat sdf = new SimpleDateFormat("hh:mm:ss a dd/MM/yyyy");

                        Date date = cal.getTime();

                        timeDate = sdf.format(date);

                        mainLabel.setText(timeDate);
                    }
                }
                catch (Exception e) {
                    e.printStackTrace();
                }
            }
        };

        t.start();
    }

 << Net Beans generated code >>

// Variables declaration - do not modify
private javax.swing.JLabel mainLabel;
private javax.swing.JPanel mainPanel;
// End of variables declaration

The Label and Panel take 2/3 seconds to display on the form.

The form launches like this:

Then after 2/3 seconds, everything, including border is displayed correctly:

I would have expected the label and panel to display as soon as the application is launched.


问:
Selenium webdriver: How do I find ALL of an element's attributes?
In the Python Selenium module, once I have a WebElement object I can get the value of any of its attributes with get_attribute():

```<code>foo = elem.get_attribute('href')
</code>```

If the attribute named 'href' doesn't exist, None is returned.

My question is, how can I get a list of all of the attributes that an element has? There doesn't seem to be a get_attributes() or get_attribute_names() method.

I'm using version 2.44.0 of the Selenium module for Python.

答:
You can find using element.get_property() method.

```<code>from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://www.ultimateqa.com/complicated-page/")

logo = driver.find_element(By.XPATH, "//img[@id='logo']")
attrs=[]
for attr in logo.get_property('attributes'):
    attrs.append([attr['name'], attr['value']])
print(attrs)
</code>```

Output:

```<code>[['src', 'https://www.ultimateqa.com/wp-content/uploads/2019/01/horizontal_on_transparent_by_logaster-2.png'], ['alt', 'Ultimate QA'], ['id', 'logo'], ['data-height-percentage', '100'], ['data-actual-width', '912'], ['data-actual-height', '410']]
</code>```

Body:
In the Python Selenium module, once I have a WebElement object I can get the value of any of its attributes with get_attribute():

foo = elem.get_attribute('href')

If the attribute named 'href' doesn't exist, None is returned.

My question is, how can I get a list of all of the attributes that an element has? There doesn't seem to be a get_attributes() or get_attribute_names() method.

I'm using version 2.44.0 of the Selenium module for Python.

Body:
You can find using element.get_property() method.

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://www.ultimateqa.com/complicated-page/")

logo = driver.find_element(By.XPATH, "//img[@id='logo']")
attrs=[]
for attr in logo.get_property('attributes'):
    attrs.append([attr['name'], attr['value']])
print(attrs)

Output:

[['src', 'https://www.ultimateqa.com/wp-content/uploads/2019/01/horizontal_on_transparent_by_logaster-2.png'], ['alt', 'Ultimate QA'], ['id', 'logo'], ['data-height-percentage', '100'], ['data-actual-width', '912'], ['data-actual-height', '410']]


问:
Matching values from multiple columns in 1 data frame to key in second data frame and creating columns
I have 2 data frames. One (df1) looks like this:

```<code> var.1 var.2 var.3 var.4
1     7     9     1     2
2     4     6     9     7
3     2     NA    NA    NA
</code>```

And the other (df2) looks like this:

```<code> var.a var.b var.c var.d
1     1     b     c     d
2     2     f     g     h
3     4     j     k     l
3     7     j     k     z
...
</code>```

with all of the values listed out in var.1-var.4 in df1 in var.a of df2.

I want to match var.a from df2 across all of the columns listed in df1 and then add these columns to df1 with new/combined column names. So for instance it'll look like this:

```<code> var.1 var1.b var1.c var1.d ... var.4 var4.b var4.c var4.d
1     7     j     k     z         2     f     g     h
2     4     j     k     l         7     j     k     z
3     2     f     g     h         NA    NA    NA    NA
</code>```

Thanks in advance!

答:
Here's a tidyverse solution. First, I define the data frames.

df1 <- read.table(text = " var.1 var.2 var.3 var.4
                  1     7     9     1     2
                  2     4     6     9     7
                  3     2     NA    NA    NA", header = TRUE)

df2 <- read.table(text = "  var.a var.b var.c var.d
1     1     b     c     d
2     2     f     g     h
3     4     j     k     l
4     7     j     k     z", header=TRUE)
```

<p>Then, I load the libraries.</p>

<pre class="lang-r prettyprint-override"><code># Load libraries
library(tidyr)
library(dplyr)
library(tibble)
</code>```

Finally, I restructure the data.

# Manipulate data
df1 %>%
  rownames_to_column() %>%
  gather(variable, value, -rowname) %>%
  left_join(df2, by = c("value" = "var.a")) %>%
  gather(foo, bar, -variable, -rowname) %>%
  unite(goop, variable, foo) %>%
  spread(goop, bar) %>%
  select(-rowname)
#> Warning: attributes are not identical across measure variables;
#> they will be dropped
```

<p>which gives,</p>

<pre class="lang-r prettyprint-override"><code>#>   var.1_value var.1_var.b var.1_var.c var.1_var.d var.2_value var.2_var.b
#> 1           7           j           k           z           9        <NA>
#> 2           4           j           k           l           6        <NA>
#> 3           2           f           g           h        <NA>        <NA>
#>   var.2_var.c var.2_var.d var.3_value var.3_var.b var.3_var.c var.3_var.d
#> 1        <NA>        <NA>           1           b           c           d
#> 2        <NA>        <NA>           9        <NA>        <NA>        <NA>
#> 3        <NA>        <NA>        <NA>        <NA>        <NA>        <NA>
#>   var.4_value var.4_var.b var.4_var.c var.4_var.d
#> 1           2           f           g           h
#> 2           7           j           k           z
#> 3        <NA>        <NA>        <NA>        <NA>
</code>```

Created on 2019-05-30 by the reprex package (v0.3.0)

This is a little bit convoluted, but I'll try to explain.

I turn row numbers into a column at first, as this will help me put the data back together at the very end.

I go from wide to long format for df1.

I join df2 to df1 based on var.a and var.1 (now called value), respectively.

I go from wide to long again.

I combine the variable names from each data frame into one variable.

Finally, I go from long to wide format (this is where the row numbers come in handy) and drop the row numbers.

Body:
I have 2 data frames. One (df1) looks like this:

 var.1 var.2 var.3 var.4
1     7     9     1     2
2     4     6     9     7
3     2     NA    NA    NA

And the other (df2) looks like this:

 var.a var.b var.c var.d
1     1     b     c     d
2     2     f     g     h
3     4     j     k     l
3     7     j     k     z
...

with all of the values listed out in var.1-var.4 in df1 in var.a of df2.

I want to match var.a from df2 across all of the columns listed in df1 and then add these columns to df1 with new/combined column names. So for instance it'll look like this:

 var.1 var1.b var1.c var1.d ... var.4 var4.b var4.c var4.d
1     7     j     k     z         2     f     g     h
2     4     j     k     l         7     j     k     z
3     2     f     g     h         NA    NA    NA    NA

Thanks in advance!

Body:
Here's a tidyverse solution. First, I define the data frames.

df1 <- read.table(text = " var.1 var.2 var.3 var.4
                  1     7     9     1     2
                  2     4     6     9     7
                  3     2     NA    NA    NA", header = TRUE)

df2 <- read.table(text = "  var.a var.b var.c var.d
1     1     b     c     d
2     2     f     g     h
3     4     j     k     l
4     7     j     k     z", header=TRUE)

Then, I load the libraries.

# Load libraries
library(tidyr)
library(dplyr)
library(tibble)

Finally, I restructure the data.

# Manipulate data
df1 %>%
  rownames_to_column() %>%
  gather(variable, value, -rowname) %>%
  left_join(df2, by = c("value" = "var.a")) %>%
  gather(foo, bar, -variable, -rowname) %>%
  unite(goop, variable, foo) %>%
  spread(goop, bar) %>%
  select(-rowname)
#> Warning: attributes are not identical across measure variables;
#> they will be dropped

which gives,

#>   var.1_value var.1_var.b var.1_var.c var.1_var.d var.2_value var.2_var.b
#> 1           7           j           k           z           9        <NA>
#> 2           4           j           k           l           6        <NA>
#> 3           2           f           g           h        <NA>        <NA>
#>   var.2_var.c var.2_var.d var.3_value var.3_var.b var.3_var.c var.3_var.d
#> 1        <NA>        <NA>           1           b           c           d
#> 2        <NA>        <NA>           9        <NA>        <NA>        <NA>
#> 3        <NA>        <NA>        <NA>        <NA>        <NA>        <NA>
#>   var.4_value var.4_var.b var.4_var.c var.4_var.d
#> 1           2           f           g           h
#> 2           7           j           k           z
#> 3        <NA>        <NA>        <NA>        <NA>

Created on 2019-05-30 by the reprex package (v0.3.0)

This is a little bit convoluted, but I'll try to explain.

I turn row numbers into a column at first, as this will help me put the data back together at the very end.

I go from wide to long format for df1.

I join df2 to df1 based on var.a and var.1 (now called value), respectively.

I go from wide to long again.

I combine the variable names from each data frame into one variable.

Finally, I go from long to wide format (this is where the row numbers come in handy) and drop the row numbers.


答:
This should work.

import pandas as pd

# Sample data
data = pd.DataFrame({
               "ID": ["ID1", "ID2"],
               "time": ["2017-09-17 15:30:00", "2017-09-24 18:00:00"]
               })

data['time'] = pd.to_datetime(data['time'])
print(data.head())
    ID                time
0  ID1 2017-09-17 15:30:00
1  ID2 2017-09-24 18:00:00
```

<pre class="lang-py prettyprint-override"><code># Logic
ID = []
time = []
for idx, row in data.iterrows():
    tm = row['time']
    split = pd.date_range(
                tm - pd.DateOffset(hours=1),
                tm + pd.DateOffset(hours=2),
                freq="1min"
                )

    val = [row['ID']] * len(split)
    ID.extend(val)
    time.extend(split)
</code>```

# Result
df = pd.DataFrame({"ID": ID, "time": time})
print(df.head(20))
     ID                time
0   ID1 2017-09-17 14:30:00
1   ID1 2017-09-17 14:31:00
2   ID1 2017-09-17 14:32:00
3   ID1 2017-09-17 14:33:00
4   ID1 2017-09-17 14:34:00
5   ID1 2017-09-17 14:35:00
.
.
.
```

Title: Generate offsets for each row in a dataframe before and after datetime value

Body:
I have the following dataframe:

    ID  time
0   12  2017-09-17 15:30:00
1   13  2017-09-24 18:00:00

I would like to add 1h before each time value and 2 hours after with a minute frequency:

    ID  time
0   12  2017-09-17 14:30:00
1   12  2017-09-17 14:31:00
2   12  2017-09-17 14:32:00
3   12  2017-09-17 14:33:00
...
59  12  2017-09-17 15:29:00
60  12  2017-09-17 15:30:00
...
179 12  2017-09-17 17:29:00
180 12  2017-09-17 17:30:00
181 13  2017-09-24 17:00:00
...

Does anyone know how to generate these kinds of offsets?

Body:
This should work.

import pandas as pd

# Sample data
data = pd.DataFrame({
               "ID": ["ID1", "ID2"],
               "time": ["2017-09-17 15:30:00", "2017-09-24 18:00:00"]
               })

data['time'] = pd.to_datetime(data['time'])
print(data.head())
    ID                time
0  ID1 2017-09-17 15:30:00
1  ID2 2017-09-24 18:00:00

# Logic
ID = []
time = []
for idx, row in data.iterrows():
    tm = row['time']
    split = pd.date_range(
                tm - pd.DateOffset(hours=1),
                tm + pd.DateOffset(hours=2),
                freq="1min"
                )

    val = [row['ID']] * len(split)
    ID.extend(val)
    time.extend(split)

# Result
df = pd.DataFrame({"ID": ID, "time": time})
print(df.head(20))
     ID                time
0   ID1 2017-09-17 14:30:00
1   ID1 2017-09-17 14:31:00
2   ID1 2017-09-17 14:32:00
3   ID1 2017-09-17 14:33:00
4   ID1 2017-09-17 14:34:00
5   ID1 2017-09-17 14:35:00
.
.
.


问:
How do I get a vocabulary from columns in a dataframe?
I have a rather large dataset stored in a dataframe. So large , in fact, that sorting through the dataset to generate an example dataset has caused my text editor to crash. Because of this, I am providing links to the datasets I'm using:

https://github.com/moonman239/Capstone-project/blob/master/data.zip

For planning purposes, I need to retrieve the vocabulary of words from the question,article_title and paragraph_context columns.

However, it appears that in the process of splitting and combining the columns, I have inadvertently created some words by mashing together two words end-to-end (e.g: "raised" and "in" become "raisedin" catalans")

```<code>### Loading JSON datasets

import json
import re
regex = re.compile(r'\W+')
def readFile(filename):
  with open(filename) as file:
    fields = []
    JSON = json.loads(file.read())
    for article in JSON["data"]:
      articleTitle = article["title"]
      for paragraph in article["paragraphs"]:
        paragraphContext = paragraph["context"]
        for qas in paragraph["qas"]:
          question = qas["question"]
          for answer in qas["answers"]:
            fields.append({"question":question,"answer_text":answer["text"],"answer_start":answer["answer_start"],"paragraph_context":paragraphContext,"article_title":articleTitle})
  fields = pd.DataFrame(fields)
  fields["question"] = fields["question"].str.replace(regex," ")
  assert not (fields["question"].str.contains("catalanswhat").any())
  fields["paragraph_context"] = fields["paragraph_context"].str.replace(regex," ")
  fields["answer_text"] = fields["answer_text"].str.replace(regex," ")
  assert not (fields["answer_text"].str.contains("catalanswhat").any())
  fields["article_title"] = fields["article_title"].str.replace("_"," ")
  assert not (fields["article_title"].str.contains("catalanswhat").any())
  return fields
# Load training dataset.
trainingData = readFile("train-v1.1.json")

# Vocabulary functions
def vocabulary():
  data_frame = trainingData
  data_frame = data_frame.astype("str")
  text_split = pd.concat((data_frame["question"],data_frame["paragraph_context"],data_frame["article_title"]),ignore_index=True)
  text_split = text_split.str.split()
  words = set()
  text_split.apply(words.update)
  return words
def vocabularySize():
  return len(vocabulary())
</code>```

Alternate code that also fails:

```<code>def vocabulary():
  data_frame = trainingData
  data_frame = data_frame.astype("str")
  concat = data_frame["question"].str.cat(sep=" ",others=[data_frame["paragraph_context"],data_frame["article_title"]])
  concat = concat.str.split(" ")
  words = set()
  concat.apply(words.update)
  print(words)
  assert "raisedin" not in words
  return words
</code>```

答:
This is my approach to the problem:

```<code>from sklearn.feature_extraction.text import CountVectorizer

df = pd.read_json('train-v1.1.json')

words = []
for idx, row in df.iterrows():
    #title
    words.append(json_normalize(df['data'][idx])['title'].str.replace("_"," ").to_string(index = False))
    #paragraph context
    words.append(json_normalize(df['data'][idx], record_path = 'paragraphs')['context'].to_string(index = False))
    #question
    words.append(json_normalize(df['data'][idx], record_path = ['paragraphs', 'qas'])['question'].to_string(index = False))

vectorizer = CountVectorizer()
count = vectorizer.fit_transform(words)
vectorizer.get_feature_names()
</code>```

sklearn has a function that does literally what you want to do, get all the individual words for a set of data. In order to use this we need to put all the data into 1 list or series.

How we accomplish creation of the list is by first reading in the file. I noticed that there are many json files embedded within, so next we are going to loop over all the different jsons and pull the data we want, then add it to a list called words.

How we pull the info we need is from the following:

```<code>json_normalize(df['data'][idx], record_path = ['paragraphs', 'qas'])['question'].to_string(index = False)
</code>```

We look at the data column of our df which contains the individual jsons. The we navigate through the json until we are at the point we want to be through record_path. Next we take the column we want, convert it all to strings, then append the new list to our master word list. We do that for all the different json files.

If you want to perform any more string manipulation (such as removing "_" for " ") you can do it in the for loop or with the master words list. I only did it for the titles in my case.

Finally we are going to add up the words. We create a CountVectorizer called vectorizer, we fit and transform our list. Finally we can investigate our CountVectorizer through the get_feature_names() function to see every word. Note, if there are spelling mistakes in the text they will also be there.

EDIT:

You can use the code below to search for words and see where they are. Change the values in checking to whatever you want.

```<code>df = pd.read_json('train-v1.1.json')

vectorizer = CountVectorizer()
checking = ['raisedin']

for idx, row in df.iterrows():
    title = []
    para = []
    quest = []

    getTitle = json_normalize(df['data'][idx])['title'].str.replace("_"," ")
    getPara = json_normalize(df['data'][idx], record_path = 'paragraphs')['context']
    getQuest = json_normalize(df['data'][idx], record_path = ['paragraphs', 'qas'])['question']

    title.append(getTitle.str.replace("_"," ").to_string(index = False))
    para.append(getPara.to_string(index = False))
    quest.append(getQuest.to_string(index = False))

    for word in checking:
        for allwords in [getTitle, getPara, getQuest]:
            count = vectorizer.fit_transform(allwords)
            test = vectorizer.get_feature_names()
            if word in test:
                print(getTitle)
                print(f"{word} is in: " +  allwords.loc[allwords.str.contains(word)])

0    Poultry
Name: title, dtype: object
93    raisedin is in: How long does it take for an broiler raisedin...
Name: question, dtype: object

</code>```

Body:
I have a rather large dataset stored in a dataframe. So large , in fact, that sorting through the dataset to generate an example dataset has caused my text editor to crash. Because of this, I am providing links to the datasets I'm using:

https://github.com/moonman239/Capstone-project/blob/master/data.zip

For planning purposes, I need to retrieve the vocabulary of words from the question,article_title and paragraph_context columns.

However, it appears that in the process of splitting and combining the columns, I have inadvertently created some words by mashing together two words end-to-end (e.g: "raised" and "in" become "raisedin" catalans")

### Loading JSON datasets

import json
import re
regex = re.compile(r'\W+')
def readFile(filename):
  with open(filename) as file:
    fields = []
    JSON = json.loads(file.read())
    for article in JSON["data"]:
      articleTitle = article["title"]
      for paragraph in article["paragraphs"]:
        paragraphContext = paragraph["context"]
        for qas in paragraph["qas"]:
          question = qas["question"]
          for answer in qas["answers"]:
            fields.append({"question":question,"answer_text":answer["text"],"answer_start":answer["answer_start"],"paragraph_context":paragraphContext,"article_title":articleTitle})
  fields = pd.DataFrame(fields)
  fields["question"] = fields["question"].str.replace(regex," ")
  assert not (fields["question"].str.contains("catalanswhat").any())
  fields["paragraph_context"] = fields["paragraph_context"].str.replace(regex," ")
  fields["answer_text"] = fields["answer_text"].str.replace(regex," ")
  assert not (fields["answer_text"].str.contains("catalanswhat").any())
  fields["article_title"] = fields["article_title"].str.replace("_"," ")
  assert not (fields["article_title"].str.contains("catalanswhat").any())
  return fields
# Load training dataset.
trainingData = readFile("train-v1.1.json")

# Vocabulary functions
def vocabulary():
  data_frame = trainingData
  data_frame = data_frame.astype("str")
  text_split = pd.concat((data_frame["question"],data_frame["paragraph_context"],data_frame["article_title"]),ignore_index=True)
  text_split = text_split.str.split()
  words = set()
  text_split.apply(words.update)
  return words
def vocabularySize():
  return len(vocabulary())

Alternate code that also fails:

def vocabulary():
  data_frame = trainingData
  data_frame = data_frame.astype("str")
  concat = data_frame["question"].str.cat(sep=" ",others=[data_frame["paragraph_context"],data_frame["article_title"]])
  concat = concat.str.split(" ")
  words = set()
  concat.apply(words.update)
  print(words)
  assert "raisedin" not in words
  return words

Body:
This is my approach to the problem:

from sklearn.feature_extraction.text import CountVectorizer

df = pd.read_json('train-v1.1.json')

words = []
for idx, row in df.iterrows():
    #title
    words.append(json_normalize(df['data'][idx])['title'].str.replace("_"," ").to_string(index = False))
    #paragraph context
    words.append(json_normalize(df['data'][idx], record_path = 'paragraphs')['context'].to_string(index = False))
    #question
    words.append(json_normalize(df['data'][idx], record_path = ['paragraphs', 'qas'])['question'].to_string(index = False))

vectorizer = CountVectorizer()
count = vectorizer.fit_transform(words)
vectorizer.get_feature_names()

sklearn has a function that does literally what you want to do, get all the individual words for a set of data. In order to use this we need to put all the data into 1 list or series.

How we accomplish creation of the list is by first reading in the file. I noticed that there are many json files embedded within, so next we are going to loop over all the different jsons and pull the data we want, then add it to a list called words.

How we pull the info we need is from the following:

json_normalize(df['data'][idx], record_path = ['paragraphs', 'qas'])['question'].to_string(index = False)

We look at the data column of our df which contains the individual jsons. The we navigate through the json until we are at the point we want to be through record_path. Next we take the column we want, convert it all to strings, then append the new list to our master word list. We do that for all the different json files.

If you want to perform any more string manipulation (such as removing "_" for " ") you can do it in the for loop or with the master words list. I only did it for the titles in my case.

Finally we are going to add up the words. We create a CountVectorizer called vectorizer, we fit and transform our list. Finally we can investigate our CountVectorizer through the get_feature_names() function to see every word. Note, if there are spelling mistakes in the text they will also be there.

EDIT:

You can use the code below to search for words and see where they are. Change the values in checking to whatever you want.

df = pd.read_json('train-v1.1.json')

vectorizer = CountVectorizer()
checking = ['raisedin']

for idx, row in df.iterrows():
    title = []
    para = []
    quest = []

    getTitle = json_normalize(df['data'][idx])['title'].str.replace("_"," ")
    getPara = json_normalize(df['data'][idx], record_path = 'paragraphs')['context']
    getQuest = json_normalize(df['data'][idx], record_path = ['paragraphs', 'qas'])['question']

    title.append(getTitle.str.replace("_"," ").to_string(index = False))
    para.append(getPara.to_string(index = False))
    quest.append(getQuest.to_string(index = False))

    for word in checking:
        for allwords in [getTitle, getPara, getQuest]:
            count = vectorizer.fit_transform(allwords)
            test = vectorizer.get_feature_names()
            if word in test:
                print(getTitle)
                print(f"{word} is in: " +  allwords.loc[allwords.str.contains(word)])

0    Poultry
Name: title, dtype: object
93    raisedin is in: How long does it take for an broiler raisedin...
Name: question, dtype: object


答:
You can set time to index and reindex:

```<code>df.time = pd.to_datetime(df.time)
df.set_index('time', inplace=True)

new_index = pd.concat(pd.date_range(end=a,freq='T', periods=61)
                        .to_series() for a in df.index)

new_df = df.reindex(new_index).bfill().reset_index()
</code>```

Output (new_df.head()):

```<code>                  index       ID
0   2017-09-17 14:30:00     12.0
1   2017-09-17 14:31:00     12.0
2   2017-09-17 14:32:00     12.0
3   2017-09-17 14:33:00     12.0
4   2017-09-17 14:34:00     12.0
</code>```

Body:
You can set time to index and reindex:

df.time = pd.to_datetime(df.time)
df.set_index('time', inplace=True)

new_index = pd.concat(pd.date_range(end=a,freq='T', periods=61)
                        .to_series() for a in df.index)

new_df = df.reindex(new_index).bfill().reset_index()

Output (new_df.head()):

                  index       ID
0   2017-09-17 14:30:00     12.0
1   2017-09-17 14:31:00     12.0
2   2017-09-17 14:32:00     12.0
3   2017-09-17 14:33:00     12.0
4   2017-09-17 14:34:00     12.0


问:
Is "slicing" a video attached to iOS app assets possible?
With PNG images, one can attach img.png, img@2x.png, and img@3x.png via assets and then load the correct one at runtime based on the device screen dimensions. In addition, the iTunes upload/download process will "slice" asset resources so that a large app that contains 1x, 2x, and 3x assets can avoid having to download 1x and 3x assets when running on a 2x device. What I am wondering is this, can videos attached to an app also take advantage of slicing so that the app need not download 3 sets of video resources in those cases where videos created for specific screen sizes are included in the app? I see that assets do contain a generic "data" type, but it seems to only be able to mark data files as differing in terms of the Metal API version.</p>

答: So, I looked all around the internets but was unable to find anything on how to solve this tricky issue. In the end, I rolled my own custom solution that basically wraps the m4v video file and treats the binary file as a PNG image, so that it can be included in an iOS asset catalog. This makes it possible to load @2x or @3x assets on iPhone and use iPad specific video asset dimensions on an iPad while also supporting slicing. For a working example, take a peek at my AlphaOverVideo Framework on github, the Bloom demo shows the client side logic to decode from PNG and then load the decoded .m4v video into a looping player. This demo is cool because it shows off the slicing idea, but it also contains a full screen 1 to 1 pixel aspect video of a flower blooming that shows how amazing perfectly rendered video can look. If anyone is interested in the command line encoder, I uploaded it to github at PNGRewrap.</p>


问:
What are the pros and cons of using webworks versus J2ME and vice versa in blackberry development?
I have written some basic applications using J2Me and felt that getting things done in terms of UI/UX were a night mare especially with blackberry OS 5. I then started playing around with Webworks and found it to be better and faster however i keep getting told that it is better to stick with J2ME. Which one is a better choice? Is one limited in features than the other? What will RIM support for the long term future?</p>

答:
This is a decision to be made for each application.

How well do you know HTML, Javascript, and CSS? How much device integration do you need? How graphics intensive will the application be?

HTML et al have a much lower learning curve than J2ME and the RIM APIs. You are missing some of the more functionality that integrates deep into the phones abilities. But you have a distinct advantage in that anything that you find missing in the Webworks libraries, you can write as Javascript extension in Java.

Webworks itself is limited to the blackberry platform, but HTML, Javascript, and CSS are practically universal. It's simple to turn the application into something that will work in a web browser. Just take out Javascript calls to any blackberry.* apis (or wrap their use in blocks like if (blackberry) {), and redirect anything in an XMLHTTPRequest to be proxied through something on the local server. I'm doing it this way for a Playbook app I'm working on. It's nice that I can show others that don't use a BB the functionality of my app.

It's probably pretty easy to take your Webworks app and alter it for another framework like Adobe AIR or PhoneGap (I haven't tried, but intend to at some point).


问:
iOS 11 UISearchBar background color
I understand that this question has been asked many, many times on SO. However, as Apple does best, with the release of iOS 11, they seem to have made a seemingly unnecessary change to the UISearchBar, specifically it's view hierarchy.

In further, the "text field" of a search bar is no longer accessible in the search bar's subviews, causing all of the previous solutions to "access" and change the background color of the text field, or any property of the text field for that matter.

Does anyone know how to actually adjust the background color of a search bar in iOS 11?

FYI:
I am specifically talking about the color behind the text... which now as of 11 defaults to white unless you specify the search bar style to be minimal.

UPDATE 1:

Since my posting of this question, I still have not found a valid or really any real solution to this issue. The closest I have seem to come is to dive deep into the appearance for instance properties

```<code>[[UISearchBar class] appearanceWhenContainedInInstancesOfClasses:(nonnull NSArray<Class<UIAppearanceContainer>> *)]
</code>```

of the UISearchBar. Playing around with the found UITextField via methods such as the following:

```<code>if ([view isKindOfClass:[UITextField class]]) {
    return (UITextField*)view;
}
UITextField *searchTextField;
for (UIView *subview in view.subviews) {
    searchTextField = [self searchViewForTextFieldBg:subview];
    if (searchTextField) {
        break;
    }
}
return searchTextField;
</code>```

you can begin drawing a new background view to be placed behind the view. However, the issues I had found too tedious to pursue further were drawing the a view with the correct frame / bounds to mimic exactly the original background.

Hopefully someone can find the actual solution to this problem. Nice miss apple...

答:
Swift 4-5

searchController.searchBar.barTintColor = .white

Title: iOS 11 UISearchBar background color

Body:
I understand that this question has been asked many, many times on SO. However, as Apple does best, with the release of iOS 11, they seem to have made a seemingly unnecessary change to the UISearchBar, specifically it's view hierarchy.

In further, the "text field" of a search bar is no longer accessible in the search bar's subviews, causing all of the previous solutions to "access" and change the background color of the text field, or any property of the text field for that matter.

Does anyone know how to actually adjust the background color of a search bar in iOS 11?

FYI:
I am specifically talking about the color behind the text... which now as of 11 defaults to white unless you specify the search bar style to be minimal.

UPDATE 1:

Since my posting of this question, I still have not found a valid or really any real solution to this issue. The closest I have seem to come is to dive deep into the appearance for instance properties

[[UISearchBar class] appearanceWhenContainedInInstancesOfClasses:(nonnull NSArray<Class<UIAppearanceContainer>> *)]

of the UISearchBar. Playing around with the found UITextField via methods such as the following:

if ([view isKindOfClass:[UITextField class]]) {
    return (UITextField*)view;
}
UITextField *searchTextField;
for (UIView *subview in view.subviews) {
    searchTextField = [self searchViewForTextFieldBg:subview];
    if (searchTextField) {
        break;
    }
}
return searchTextField;

you can begin drawing a new background view to be placed behind the view. However, the issues I had found too tedious to pursue further were drawing the a view with the correct frame / bounds to mimic exactly the original background.

Hopefully someone can find the actual solution to this problem. Nice miss apple...


问:
Problems with win32com
I have the following error in my code, I have tried everything and I can not solve:

1° Not Successful

excel = win32com.client.Dispatch("Excel.Application")
wb = excel.Workbooks.Open(os.path.join(os.getcwd(),diretorio_BKP + filename +
                          '  '+str(nome_arq_parcial)+' -.xlsx'))
ws = wb.Worksheets(1).Select()
ws = wb.Worksheets(1)
print('abrir o arquivo de novo')
```
<h1>2° Not Successful</h1>
<pre class="lang-py prettyprint-override"><code>excel = win32com.client.Dispatch("Excel.Application")
wb = excel.Workbooks.Open(diretorio_BKP + filename +'  '+
                          str(nome_arq_parcial)+' -.xlsx')
ws = wb.Worksheets(1).Select()
ws = wb.Worksheets(1)
print('abrir o arquivo de novo')
</code>```

The two attempts return the following problem to me:

ERRO:    (-2147352567, 'Exception.', (0, 'Microsoft Excel', "Microsoft Excel does not you can access the file '\\\\ IEMEN \\ file location -.xlsx'. There are several reasons possible: \ r \ n \ r \ n • The file name or path does not exist. \ r \ n • The file is being used by another program. \ r \ n • The workbook you are trying to save has the same name as another currently open \ r \ nfolder. ", 'xlmain11.chm', 0, -2146827284), None)
```

As you can see already tried to leave in absolute path, already tried to leave the full path, I already put excel.Application.Quit()
before that stretch as well and nothing.

I need the code to go through this step to start formatting the whole file.

答:
SOLUTION! AMAZINGLY:

I could not believe how this error was obscure and we found the solution looking for similar problems with the equivalent .NET code:

To fix it, create a folder named 'Desktop' under 'C: \ Windows \ SysWOW64 \ config \ systemprofile \' on 64-bit architecture or 'C: \ Windows \ System32 \ config \ systemprofile \' on 32-bit servers.

WORKED WITH PERFECTION.

(Source: StackOverflow answers in English: excel access denied with win32 python pywin32)

Title: Problems with win32com

Body:
I have the following error in my code, I have tried everything and I can not solve:

1° Not Successful

excel = win32com.client.Dispatch("Excel.Application")
wb = excel.Workbooks.Open(os.path.join(os.getcwd(),diretorio_BKP + filename +
                          '  '+str(nome_arq_parcial)+' -.xlsx'))
ws = wb.Worksheets(1).Select()
ws = wb.Worksheets(1)
print('abrir o arquivo de novo')

2° Not Successful

excel = win32com.client.Dispatch("Excel.Application")
wb = excel.Workbooks.Open(diretorio_BKP + filename +'  '+
                          str(nome_arq_parcial)+' -.xlsx')
ws = wb.Worksheets(1).Select()
ws = wb.Worksheets(1)
print('abrir o arquivo de novo')

The two attempts return the following problem to me:

ERRO:    (-2147352567, 'Exception.', (0, 'Microsoft Excel', "Microsoft Excel does not you can access the file '\\\\ IEMEN \\ file location -.xlsx'. There are several reasons possible: \ r \ n \ r \ n • The file name or path does not exist. \ r \ n • The file is being used by another program. \ r \ n • The workbook you are trying to save has the same name as another currently open \ r \ nfolder. ", 'xlmain11.chm', 0, -2146827284), None)

As you can see already tried to leave in absolute path, already tried to leave the full path, I already put excel.Application.Quit()
before that stretch as well and nothing.

I need the code to go through this step to start formatting the whole file.


问:
Plot true color Sentinel-2A imagery in Matlab
Through a combination of non-matlab/non-native tools (GDAL) as well as native tools (geoimread) I can ingest Sentinel-2A data either a indiviual bands or as an RGB image having employed gdal merge.  I'm stuck at a point where using

```<code>imshow(I, [])
</code>```

Produces a black image, with apparently no signal.  The range of intensity values in the image are 271 - 4349.  I know that there is a good signal in the image because when I do:

```<code>bit_depth = 2^15;
I = swapbytes(I);
[I_indexed, color_map] = rgb2ind(I, bit_depth);
I_double = im2double(I_indexed, 'indexed');
ax1 = figure;
colormap(ax1, color_map);
image(I_double)
</code>```

i.e. index the image, collect a colormap, set the colormap and then call the image function, I get a likeness of the region I'm exploring (albeit very strangely colored)

I'm currently considering whether I should try:

Find a low-level description of Sentinel-2A data, implement the scaling/correction

Use a toolbox, possibly this one.

Possibly adjust ouput settings in one of the earlier steps involving GDAL
Comments or suggestions are greatly appreciated.

A basic scaling scheme is:

```<code>% convert image to double
I_double = im2double(I);

% scaling
max_intensity = max(I_double(:));
min_intensity = min(I_double(:));
range_intensity = max_intensity - min_intensity;
I_scaled = 2^16.*((I_double - min_intensity) ./ range_intensity);

% display
imshow(uint16(I_scaled))
</code>```

noting the importance of casting to uint16 from double for imshow.

答:
A couple points...

You mention that I is an RGB image (i.e. N-by-M-by-3 data). If this is the case, the [] argument to imshow will have no effect. That only applies automatic scaling of the display for grayscale images.

Given the range of intensity values you list (271 to 4349), I'm guessing you are dealing with a uint16 data type. Since this data type has a maximum value of 65535, your image data only covers about the lower 16th of this range. This is why your image looks practically black. It also explains why you can see the signal with your given code: you apply swapbytes to I before displaying it with image, which in this case will shift values into the higher intensity ranges (e.g. swapbytes(uint16(4349)) gives a value of 64784).

In order to better visualize your data, you'll need to scale it. As a simple test, you'll probably be able to see something appear by just scaling it by 8 (to cover a little more than half of your dynamic range):

```<code>imshow(8.*I);
</code>```

Body:
Through a combination of non-matlab/non-native tools (GDAL) as well as native tools (geoimread) I can ingest Sentinel-2A data either a indiviual bands or as an RGB image having employed gdal merge.  I'm stuck at a point where using

imshow(I, [])

Produces a black image, with apparently no signal.  The range of intensity values in the image are 271 - 4349.  I know that there is a good signal in the image because when I do:

bit_depth = 2^15;
I = swapbytes(I);
[I_indexed, color_map] = rgb2ind(I, bit_depth);
I_double = im2double(I_indexed, 'indexed');
ax1 = figure;
colormap(ax1, color_map);
image(I_double)

i.e. index the image, collect a colormap, set the colormap and then call the image function, I get a likeness of the region I'm exploring (albeit very strangely colored)

I'm currently considering whether I should try:

Find a low-level description of Sentinel-2A data, implement the scaling/correction

Use a toolbox, possibly this one.

Possibly adjust ouput settings in one of the earlier steps involving GDAL
Comments or suggestions are greatly appreciated.

A basic scaling scheme is:

% convert image to double
I_double = im2double(I);

% scaling
max_intensity = max(I_double(:));
min_intensity = min(I_double(:));
range_intensity = max_intensity - min_intensity;
I_scaled = 2^16.*((I_double - min_intensity) ./ range_intensity);

% display
imshow(uint16(I_scaled))

noting the importance of casting to uint16 from double for imshow.

Body:
A couple points...

You mention that I is an RGB image (i.e. N-by-M-by-3 data). If this is the case, the [] argument to imshow will have no effect. That only applies automatic scaling of the display for grayscale images.

Given the range of intensity values you list (271 to 4349), I'm guessing you are dealing with a uint16 data type. Since this data type has a maximum value of 65535, your image data only covers about the lower 16th of this range. This is why your image looks practically black. It also explains why you can see the signal with your given code: you apply swapbytes to I before displaying it with image, which in this case will shift values into the higher intensity ranges (e.g. swapbytes(uint16(4349)) gives a value of 64784).

In order to better visualize your data, you'll need to scale it. As a simple test, you'll probably be able to see something appear by just scaling it by 8 (to cover a little more than half of your dynamic range):

imshow(8.*I);