问:
java.lang.NoClassDefFoundError: com/amazonaws/services/s3/AmazonS3Client in Spark
I have a java spark process which I call using -
spark-submit --class MyClass target/MyJar.jar

The last part of this process is to write it locally and then copy it to s3 as it need to have specific name (can also be written to s3 and mv'ed there, for the sake of the question the error stays the same).

The code compiles and runs but when it get to the code snippet below I get the following error -

java.lang.NoClassDefFoundError: com/amazonaws/services/s3/AmazonS3    at
  java.lang.Class.forName0(Native Method)   at
  java.lang.Class.forName(Class.java:348)   at
  org.apache.spark.util.Utils$.classForName(Utils.scala:229)    at
  org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:700)
    at
  org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused
  by: java.lang.ClassNotFoundException:
  com.amazonaws.services.s3.AmazonS3    at
  java.net.URLClassLoader.findClass(URLClassLoader.java:381)    at
  java.lang.ClassLoader.loadClass(ClassLoader.java:424)     at
  java.lang.ClassLoader.loadClass(ClassLoader.java:357)

public static void saveToS3(Dataset<Row> df, String outputBucket, String outputPath) throws IOException {
    String tmpFile = "temp" + Long.toString(System.nanoTime());
    df.coalesce(1).write().option("header", true).csv(tmpFile);
    File directory = new File(tmpFile);
    AmazonS3 s3client = new AmazonS3Client();
    for (File file : directory.listFiles()) {
        if (file.getName().startsWith("part_") || file.getName().endsWith("csv")) {
            s3client.putObject(new PutObjectRequest(outputBucket, getS3path(outputPath), file));
            }
        file.delete();
    }
    directory.delete();
}
```

<p>I use the following dependencies which might be relevant </p>

```        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk</artifactId>
            <version>1.7.4</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-aws</artifactId>
            <version>2.7.2</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>servlet-api</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.mortbay.jetty</groupId>
                    <artifactId>servlet-api-2.5</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
```

<p>Any idea?</p>

<p>Dependency tree - </p>

```[INFO] +- org.apache.spark:spark-core_2.11:jar:2.1.0:compile
[INFO] |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:1.7.7:compile
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:tests:1.7.7:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  |  \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  +- com.twitter:chill_2.11:jar:0.8.0:compile
[INFO] |  |  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile
[INFO] |  |     +- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] |  |     \- org.objenesis:objenesis:jar:2.1:compile
[INFO] |  +- com.twitter:chill-java:jar:0.8.0:compile
[INFO] |  +- org.apache.xbean:xbean-asm5-shaded:jar:4.4:compile
[INFO] |  +- org.apache.hadoop:hadoop-client:jar:2.2.0:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:compile
[INFO] |  |  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
[INFO] |  |  |  |  |  \- com.google.inject:guice:jar:3.0:compile
[INFO] |  |  |  |  |     +- javax.inject:javax.inject:jar:1:compile
[INFO] |  |  |  |  |     \- aopalliance:aopalliance:jar:1.0:compile
[INFO] |  |  |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:compile
[INFO] |  |  |  \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:compile
[INFO] |  |  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:compile
[INFO] |  |  \- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile
[INFO] |  +- org.apache.spark:spark-launcher_2.11:jar:2.1.0:compile
[INFO] |  +- org.apache.spark:spark-network-common_2.11:jar:2.1.0:compile
[INFO] |  |  \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO] |  +- org.apache.spark:spark-network-shuffle_2.11:jar:2.1.0:compile
[INFO] |  +- org.apache.spark:spark-unsafe_2.11:jar:2.1.0:compile
[INFO] |  +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
[INFO] |  |  \- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] |  +- org.apache.curator:curator-recipes:jar:2.4.0:compile
[INFO] |  |  +- org.apache.curator:curator-framework:jar:2.4.0:compile
[INFO] |  |  +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
[INFO] |  |  \- com.google.guava:guava:jar:14.0.1:compile
[INFO] |  +- javax.servlet:javax.servlet-api:jar:3.1.0:compile
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.5:compile
[INFO] |  +- org.apache.commons:commons-math3:jar:3.4.1:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.16:compile
[INFO] |  +- org.slf4j:jul-to-slf4j:jar:1.7.16:compile
[INFO] |  +- org.slf4j:jcl-over-slf4j:jar:1.7.16:compile
[INFO] |  +- com.ning:compress-lzf:jar:1.0.3:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
[INFO] |  +- net.jpountz.lz4:lz4:jar:1.3.0:compile
[INFO] |  +- org.roaringbitmap:RoaringBitmap:jar:0.5.11:compile
[INFO] |  +- commons-net:commons-net:jar:2.2:compile
[INFO] |  +- org.scala-lang:scala-library:jar:2.11.8:compile
[INFO] |  +- org.json4s:json4s-jackson_2.11:jar:3.2.11:compile
[INFO] |  |  \- org.json4s:json4s-core_2.11:jar:3.2.11:compile
[INFO] |  |     +- org.json4s:json4s-ast_2.11:jar:3.2.11:compile
[INFO] |  |     +- com.thoughtworks.paranamer:paranamer:jar:2.6:compile
[INFO] |  |     \- org.scala-lang:scalap:jar:2.11.0:compile
[INFO] |  |        \- org.scala-lang:scala-compiler:jar:2.11.0:compile
[INFO] |  |           \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.1:compile
[INFO] |  +- org.glassfish.jersey.core:jersey-client:jar:2.22.2:compile
[INFO] |  |  +- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:compile
[INFO] |  |  +- org.glassfish.hk2:hk2-api:jar:2.4.0-b34:compile
[INFO] |  |  |  +- org.glassfish.hk2:hk2-utils:jar:2.4.0-b34:compile
[INFO] |  |  |  \- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.4.0-b34:compile
[INFO] |  |  +- org.glassfish.hk2.external:javax.inject:jar:2.4.0-b34:compile
[INFO] |  |  \- org.glassfish.hk2:hk2-locator:jar:2.4.0-b34:compile
[INFO] |  |     \- org.javassist:javassist:jar:3.18.1-GA:compile
[INFO] |  +- org.glassfish.jersey.core:jersey-common:jar:2.22.2:compile
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.2:compile
[INFO] |  |  +- org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.22.2:compile
[INFO] |  |  \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile
[INFO] |  +- org.glassfish.jersey.core:jersey-server:jar:2.22.2:compile
[INFO] |  |  +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.22.2:compile
[INFO] |  |  \- javax.validation:validation-api:jar:1.1.0.Final:compile
[INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.22.2:compile
[INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.22.2:compile
[INFO] |  +- io.netty:netty-all:jar:4.0.42.Final:compile
[INFO] |  +- io.netty:netty:jar:3.8.0.Final:compile
[INFO] |  +- com.clearspring.analytics:stream:jar:2.7.0:compile
[INFO] |  +- io.dropwizard.metrics:metrics-core:jar:3.1.2:compile
[INFO] |  +- io.dropwizard.metrics:metrics-jvm:jar:3.1.2:compile
[INFO] |  +- io.dropwizard.metrics:metrics-json:jar:3.1.2:compile
[INFO] |  +- io.dropwizard.metrics:metrics-graphite:jar:3.1.2:compile
[INFO] |  +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.6.5:compile
[INFO] |  |  +- org.scala-lang:scala-reflect:jar:2.11.7:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.6.5:compile
[INFO] |  +- org.apache.ivy:ivy:jar:2.4.0:compile
[INFO] |  +- oro:oro:jar:2.0.8:compile
[INFO] |  +- net.razorvine:pyrolite:jar:4.13:compile
[INFO] |  +- net.sf.py4j:py4j:jar:0.10.4:compile
[INFO] |  +- org.apache.spark:spark-tags_2.11:jar:2.1.0:compile
[INFO] |  |  \- org.scalatest:scalatest_2.11:jar:2.2.6:compile
[INFO] |  |     \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.2:compile
[INFO] |  +- org.apache.commons:commons-crypto:jar:1.0.0:compile
[INFO] |  \- org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] +- org.apache.hadoop:hadoop-aws:jar:2.7.2:compile
[INFO] |  \- org.apache.hadoop:hadoop-common:jar:2.7.2:compile
[INFO] |     +- commons-cli:commons-cli:jar:1.2:compile
[INFO] |     +- xmlenc:xmlenc:jar:0.52:compile
[INFO] |     +- commons-io:commons-io:jar:2.4:compile
[INFO] |     +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] |     +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] |     +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] |     +- javax.servlet.jsp:jsp-api:jar:2.1:runtime
[INFO] |     +- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO] |     +- com.sun.jersey:jersey-json:jar:1.9:compile
[INFO] |     |  +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] |     |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] |     |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO] |     |  |     +- javax.xml.stream:stax-api:jar:1.0-2:compile
[INFO] |     |  |     \- javax.activation:activation:jar:1.1:compile
[INFO] |     |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
[INFO] |     |  \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
[INFO] |     +- com.sun.jersey:jersey-server:jar:1.9:compile
[INFO] |     |  \- asm:asm:jar:3.1:compile
[INFO] |     +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |     +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] |     |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO] |     |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] |     |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] |     +- org.apache.avro:avro:jar:1.7.4:compile
[INFO] |     +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] |     +- com.google.code.gson:gson:jar:2.2.4:compile
[INFO] |     +- org.apache.hadoop:hadoop-auth:jar:2.7.2:compile
[INFO] |     |  \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] |     |     +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] |     |     +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] |     |     \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] |     +- com.jcraft:jsch:jar:0.1.42:compile
[INFO] |     +- org.apache.curator:curator-client:jar:2.7.1:compile
[INFO] |     +- org.apache.htrace:htrace-core:jar:3.1.0-incubating:compile
[INFO] |     \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] |        \- org.tukaani:xz:jar:1.0:compile
[INFO] +- org.apache.spark:spark-sql_2.10:jar:2.1.0:compile
[INFO] |  +- com.univocity:univocity-parsers:jar:2.2.1:compile
[INFO] |  +- org.apache.spark:spark-sketch_2.10:jar:2.1.0:compile
[INFO] |  +- org.apache.spark:spark-core_2.10:jar:2.1.0:compile
[INFO] |  |  +- com.twitter:chill_2.10:jar:0.8.0:compile
[INFO] |  |  +- org.apache.spark:spark-launcher_2.10:jar:2.1.0:compile
[INFO] |  |  +- org.apache.spark:spark-network-common_2.10:jar:2.1.0:compile
[INFO] |  |  +- org.apache.spark:spark-network-shuffle_2.10:jar:2.1.0:compile
[INFO] |  |  +- org.apache.spark:spark-unsafe_2.10:jar:2.1.0:compile
[INFO] |  |  +- org.json4s:json4s-jackson_2.10:jar:3.2.11:compile
[INFO] |  |  |  \- org.json4s:json4s-core_2.10:jar:3.2.11:compile
[INFO] |  |  |     \- org.json4s:json4s-ast_2.10:jar:3.2.11:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-scala_2.10:jar:2.6.5:compile
[INFO] |  +- org.apache.spark:spark-catalyst_2.10:jar:2.1.0:compile
[INFO] |  |  +- org.codehaus.janino:janino:jar:3.0.0:compile
[INFO] |  |  +- org.codehaus.janino:commons-compiler:jar:3.0.0:compile
[INFO] |  |  \- org.antlr:antlr4-runtime:jar:4.5.3:compile
[INFO] |  +- org.apache.spark:spark-tags_2.10:jar:2.1.0:compile
[INFO] |  |  \- org.scalatest:scalatest_2.10:jar:2.2.6:compile
[INFO] |  +- org.apache.parquet:parquet-column:jar:1.8.1:compile
[INFO] |  |  +- org.apache.parquet:parquet-common:jar:1.8.1:compile
[INFO] |  |  \- org.apache.parquet:parquet-encoding:jar:1.8.1:compile
[INFO] |  \- org.apache.parquet:parquet-hadoop:jar:1.8.1:compile
[INFO] |     +- org.apache.parquet:parquet-format:jar:2.3.0-incubating:compile
[INFO] |     \- org.apache.parquet:parquet-jackson:jar:1.8.1:compile
[INFO] +- com.amazonaws:aws-java-sdk:jar:1.7.4:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.2:compile
[INFO] |  |  \- org.apache.httpcomponents:httpcore:jar:4.2:compile
[INFO] |  +- commons-codec:commons-codec:jar:1.3:compile
[INFO] |  \- joda-time:joda-time:jar:2.9.9:compile (version selected from constraint [2.2,))
[INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.7.3:compile
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.7.3:compile
[INFO] \- com.fasterxml.jackson.core:jackson-annotations:jar:2.7.3:compile
```

答:
You appear to be using a really old version of AWS SDK (1.7.4) from 2014. Your apache spark is 2.1, which is from 2016.

Commit history for the AmazonS3Client only goes back to 1.9, so it very well may not exist in your AWS SDK.

The reason your code compiles is that your code itself isn't the problem, but one of your dependencies is trying to use a more recent version of S3 client that your old SDK doesn't have. In other words, this isn't a code problem, it's a dependency management problem. If you're not directly using the SDK at all, it's safe to use a more appropriate AWS SDK 1.11.X, if you are, you'll have to coax your code to that point.

Typically, the spark-core module would point out its dependency to a version of aws, but I assume it was specifically excluded from its own so you could provide your own version of the SDK for your use without any sort of conflict, with the caveat that if you use too old of a dependency, it just breaks.

Body:
I have a java spark process which I call using -
spark-submit --class MyClass target/MyJar.jar

The last part of this process is to write it locally and then copy it to s3 as it need to have specific name (can also be written to s3 and mv'ed there, for the sake of the question the error stays the same).

The code compiles and runs but when it get to the code snippet below I get the following error -

java.lang.NoClassDefFoundError: com/amazonaws/services/s3/AmazonS3    at
  java.lang.Class.forName0(Native Method)   at
  java.lang.Class.forName(Class.java:348)   at
  org.apache.spark.util.Utils$.classForName(Utils.scala:229)    at
  org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:700)
    at
  org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused
  by: java.lang.ClassNotFoundException:
  com.amazonaws.services.s3.AmazonS3    at
  java.net.URLClassLoader.findClass(URLClassLoader.java:381)    at
  java.lang.ClassLoader.loadClass(ClassLoader.java:424)     at
  java.lang.ClassLoader.loadClass(ClassLoader.java:357)

public static void saveToS3(Dataset<Row> df, String outputBucket, String outputPath) throws IOException {
    String tmpFile = "temp" + Long.toString(System.nanoTime());
    df.coalesce(1).write().option("header", true).csv(tmpFile);
    File directory = new File(tmpFile);
    AmazonS3 s3client = new AmazonS3Client();
    for (File file : directory.listFiles()) {
        if (file.getName().startsWith("part_") || file.getName().endsWith("csv")) {
            s3client.putObject(new PutObjectRequest(outputBucket, getS3path(outputPath), file));
            }
        file.delete();
    }
    directory.delete();
}

I use the following dependencies which might be relevant

        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk</artifactId>
            <version>1.7.4</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-aws</artifactId>
            <version>2.7.2</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>servlet-api</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.mortbay.jetty</groupId>
                    <artifactId>servlet-api-2.5</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

Any idea?

Dependency tree -

[INFO] +- org.apache.spark:spark-core_2.11:jar:2.1.0:compile
[INFO] |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:1.7.7:compile
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:tests:1.7.7:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  |  \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  +- com.twitter:chill_2.11:jar:0.8.0:compile
[INFO] |  |  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile
[INFO] |  |     +- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] |  |     \- org.objenesis:objenesis:jar:2.1:compile
[INFO] |  +- com.twitter:chill-java:jar:0.8.0:compile
[INFO] |  +- org.apache.xbean:xbean-asm5-shaded:jar:4.4:compile
[INFO] |  +- org.apache.hadoop:hadoop-client:jar:2.2.0:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:compile
[INFO] |  |  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
[INFO] |  |  |  |  |  \- com.google.inject:guice:jar:3.0:compile
[INFO] |  |  |  |  |     +- javax.inject:javax.inject:jar:1:compile
[INFO] |  |  |  |  |     \- aopalliance:aopalliance:jar:1.0:compile
[INFO] |  |  |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:compile
[INFO] |  |  |  \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:compile
[INFO] |  |  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:compile
[INFO] |  |  \- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile
[INFO] |  +- org.apache.spark:spark-launcher_2.11:jar:2.1.0:compile
[INFO] |  +- org.apache.spark:spark-network-common_2.11:jar:2.1.0:compile
[INFO] |  |  \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO] |  +- org.apache.spark:spark-network-shuffle_2.11:jar:2.1.0:compile
[INFO] |  +- org.apache.spark:spark-unsafe_2.11:jar:2.1.0:compile
[INFO] |  +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
[INFO] |  |  \- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] |  +- org.apache.curator:curator-recipes:jar:2.4.0:compile
[INFO] |  |  +- org.apache.curator:curator-framework:jar:2.4.0:compile
[INFO] |  |  +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
[INFO] |  |  \- com.google.guava:guava:jar:14.0.1:compile
[INFO] |  +- javax.servlet:javax.servlet-api:jar:3.1.0:compile
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.5:compile
[INFO] |  +- org.apache.commons:commons-math3:jar:3.4.1:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.16:compile
[INFO] |  +- org.slf4j:jul-to-slf4j:jar:1.7.16:compile
[INFO] |  +- org.slf4j:jcl-over-slf4j:jar:1.7.16:compile
[INFO] |  +- com.ning:compress-lzf:jar:1.0.3:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
[INFO] |  +- net.jpountz.lz4:lz4:jar:1.3.0:compile
[INFO] |  +- org.roaringbitmap:RoaringBitmap:jar:0.5.11:compile
[INFO] |  +- commons-net:commons-net:jar:2.2:compile
[INFO] |  +- org.scala-lang:scala-library:jar:2.11.8:compile
[INFO] |  +- org.json4s:json4s-jackson_2.11:jar:3.2.11:compile
[INFO] |  |  \- org.json4s:json4s-core_2.11:jar:3.2.11:compile
[INFO] |  |     +- org.json4s:json4s-ast_2.11:jar:3.2.11:compile
[INFO] |  |     +- com.thoughtworks.paranamer:paranamer:jar:2.6:compile
[INFO] |  |     \- org.scala-lang:scalap:jar:2.11.0:compile
[INFO] |  |        \- org.scala-lang:scala-compiler:jar:2.11.0:compile
[INFO] |  |           \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.1:compile
[INFO] |  +- org.glassfish.jersey.core:jersey-client:jar:2.22.2:compile
[INFO] |  |  +- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:compile
[INFO] |  |  +- org.glassfish.hk2:hk2-api:jar:2.4.0-b34:compile
[INFO] |  |  |  +- org.glassfish.hk2:hk2-utils:jar:2.4.0-b34:compile
[INFO] |  |  |  \- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.4.0-b34:compile
[INFO] |  |  +- org.glassfish.hk2.external:javax.inject:jar:2.4.0-b34:compile
[INFO] |  |  \- org.glassfish.hk2:hk2-locator:jar:2.4.0-b34:compile
[INFO] |  |     \- org.javassist:javassist:jar:3.18.1-GA:compile
[INFO] |  +- org.glassfish.jersey.core:jersey-common:jar:2.22.2:compile
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.2:compile
[INFO] |  |  +- org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.22.2:compile
[INFO] |  |  \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile
[INFO] |  +- org.glassfish.jersey.core:jersey-server:jar:2.22.2:compile
[INFO] |  |  +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.22.2:compile
[INFO] |  |  \- javax.validation:validation-api:jar:1.1.0.Final:compile
[INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.22.2:compile
[INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.22.2:compile
[INFO] |  +- io.netty:netty-all:jar:4.0.42.Final:compile
[INFO] |  +- io.netty:netty:jar:3.8.0.Final:compile
[INFO] |  +- com.clearspring.analytics:stream:jar:2.7.0:compile
[INFO] |  +- io.dropwizard.metrics:metrics-core:jar:3.1.2:compile
[INFO] |  +- io.dropwizard.metrics:metrics-jvm:jar:3.1.2:compile
[INFO] |  +- io.dropwizard.metrics:metrics-json:jar:3.1.2:compile
[INFO] |  +- io.dropwizard.metrics:metrics-graphite:jar:3.1.2:compile
[INFO] |  +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.6.5:compile
[INFO] |  |  +- org.scala-lang:scala-reflect:jar:2.11.7:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.6.5:compile
[INFO] |  +- org.apache.ivy:ivy:jar:2.4.0:compile
[INFO] |  +- oro:oro:jar:2.0.8:compile
[INFO] |  +- net.razorvine:pyrolite:jar:4.13:compile
[INFO] |  +- net.sf.py4j:py4j:jar:0.10.4:compile
[INFO] |  +- org.apache.spark:spark-tags_2.11:jar:2.1.0:compile
[INFO] |  |  \- org.scalatest:scalatest_2.11:jar:2.2.6:compile
[INFO] |  |     \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.2:compile
[INFO] |  +- org.apache.commons:commons-crypto:jar:1.0.0:compile
[INFO] |  \- org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] +- org.apache.hadoop:hadoop-aws:jar:2.7.2:compile
[INFO] |  \- org.apache.hadoop:hadoop-common:jar:2.7.2:compile
[INFO] |     +- commons-cli:commons-cli:jar:1.2:compile
[INFO] |     +- xmlenc:xmlenc:jar:0.52:compile
[INFO] |     +- commons-io:commons-io:jar:2.4:compile
[INFO] |     +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] |     +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] |     +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] |     +- javax.servlet.jsp:jsp-api:jar:2.1:runtime
[INFO] |     +- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO] |     +- com.sun.jersey:jersey-json:jar:1.9:compile
[INFO] |     |  +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] |     |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] |     |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO] |     |  |     +- javax.xml.stream:stax-api:jar:1.0-2:compile
[INFO] |     |  |     \- javax.activation:activation:jar:1.1:compile
[INFO] |     |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
[INFO] |     |  \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
[INFO] |     +- com.sun.jersey:jersey-server:jar:1.9:compile
[INFO] |     |  \- asm:asm:jar:3.1:compile
[INFO] |     +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |     +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] |     |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO] |     |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] |     |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] |     +- org.apache.avro:avro:jar:1.7.4:compile
[INFO] |     +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] |     +- com.google.code.gson:gson:jar:2.2.4:compile
[INFO] |     +- org.apache.hadoop:hadoop-auth:jar:2.7.2:compile
[INFO] |     |  \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] |     |     +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] |     |     +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] |     |     \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] |     +- com.jcraft:jsch:jar:0.1.42:compile
[INFO] |     +- org.apache.curator:curator-client:jar:2.7.1:compile
[INFO] |     +- org.apache.htrace:htrace-core:jar:3.1.0-incubating:compile
[INFO] |     \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] |        \- org.tukaani:xz:jar:1.0:compile
[INFO] +- org.apache.spark:spark-sql_2.10:jar:2.1.0:compile
[INFO] |  +- com.univocity:univocity-parsers:jar:2.2.1:compile
[INFO] |  +- org.apache.spark:spark-sketch_2.10:jar:2.1.0:compile
[INFO] |  +- org.apache.spark:spark-core_2.10:jar:2.1.0:compile
[INFO] |  |  +- com.twitter:chill_2.10:jar:0.8.0:compile
[INFO] |  |  +- org.apache.spark:spark-launcher_2.10:jar:2.1.0:compile
[INFO] |  |  +- org.apache.spark:spark-network-common_2.10:jar:2.1.0:compile
[INFO] |  |  +- org.apache.spark:spark-network-shuffle_2.10:jar:2.1.0:compile
[INFO] |  |  +- org.apache.spark:spark-unsafe_2.10:jar:2.1.0:compile
[INFO] |  |  +- org.json4s:json4s-jackson_2.10:jar:3.2.11:compile
[INFO] |  |  |  \- org.json4s:json4s-core_2.10:jar:3.2.11:compile
[INFO] |  |  |     \- org.json4s:json4s-ast_2.10:jar:3.2.11:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-scala_2.10:jar:2.6.5:compile
[INFO] |  +- org.apache.spark:spark-catalyst_2.10:jar:2.1.0:compile
[INFO] |  |  +- org.codehaus.janino:janino:jar:3.0.0:compile
[INFO] |  |  +- org.codehaus.janino:commons-compiler:jar:3.0.0:compile
[INFO] |  |  \- org.antlr:antlr4-runtime:jar:4.5.3:compile
[INFO] |  +- org.apache.spark:spark-tags_2.10:jar:2.1.0:compile
[INFO] |  |  \- org.scalatest:scalatest_2.10:jar:2.2.6:compile
[INFO] |  +- org.apache.parquet:parquet-column:jar:1.8.1:compile
[INFO] |  |  +- org.apache.parquet:parquet-common:jar:1.8.1:compile
[INFO] |  |  \- org.apache.parquet:parquet-encoding:jar:1.8.1:compile
[INFO] |  \- org.apache.parquet:parquet-hadoop:jar:1.8.1:compile
[INFO] |     +- org.apache.parquet:parquet-format:jar:2.3.0-incubating:compile
[INFO] |     \- org.apache.parquet:parquet-jackson:jar:1.8.1:compile
[INFO] +- com.amazonaws:aws-java-sdk:jar:1.7.4:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.2:compile
[INFO] |  |  \- org.apache.httpcomponents:httpcore:jar:4.2:compile
[INFO] |  +- commons-codec:commons-codec:jar:1.3:compile
[INFO] |  \- joda-time:joda-time:jar:2.9.9:compile (version selected from constraint [2.2,))
[INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.7.3:compile
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.7.3:compile
[INFO] \- com.fasterxml.jackson.core:jackson-annotations:jar:2.7.3:compile


来源: stackexchange

问:
Call Powershell from Command Prompt Relative Path
I'm calling a PS script from Command Prompt using the following line

```<code>powershell "& 'c:/script.ps1'"
</code>```

However I need the script to be relative to the command prompt window. So if command prompt was looking at C: then the script would effectively be

```<code>powershell "& 'script.ps1'"
</code>```

Is there a way to inject the relative path?

答:
To inject the relative path, where you start your batch file you have to write </p>

```<code>powershell "& '%~dp0\script.ps1'"
</code>```

ContentLicense: CC BY-SA 3.0

Body:
I'm calling a PS script from Command Prompt using the following line

powershell "& 'c:/script.ps1'"

However I need the script to be relative to the command prompt window. So if command prompt was looking at C: then the script would effectively be

powershell "& 'script.ps1'"

Is there a way to inject the relative path?

Body:
To inject the relative path, where you start your batch file you have to write

powershell "& '%~dp0\script.ps1'"


问:
Write reliably to same file from different processes
I did create a small C++ Tracing solution which works very well. Within one process all is well but when I open the output file from diferent processes the data gets not correctly written. I did open the file with FILE_SHARE_WRITE to be able to write to the file when it is already open. Then I did create a named mutex to ensure proper synchronisation between processes. But it appears that this is not enough. According to the MSDN this does work within one process but not between different processes.
Next I tried to call FlushFileBuffers after every write while the mutex was still held but data was still distorted like this

The format is time process id/Thread id method enter/leave/severity namespace+method and then the message text.

```<code>10:29:42.994 7448/2236       }} Dll2.Test.fndll2 L1 -> Duration: 0.094s
10:29:43.040 7448/2236 {{       Dll2.DllMain L1
10:29:43.134 7448/2236 Info     Dll2.DllMain L1 Process detach
10:29:43.181 7448/2236       }} Dll2.DllMain L1 -> Duration: 0.141s
     }} Dll2.DllMain L1 -10:29:42.681 7448/2236 Info     Dll1.DllMain L1 Process attach
10:29:42.728 7448/2236       }} Dll1.DllMain L1 -10:29:42.744 2216/5510:29:42.775 7448/2236 {{       Dll1.Test.fndll1 10:210:29:42.822 7448/2236 Info     Dll1.Test.fndll1 10:29:42.837 2216/557610:29:42.853 7448/2236       }} Dll1.Test.fndll1 L110:29:42.884 2216/557610:29:43.306 7448/2236 {{       Dll1.DllMain L1
10:29:43.353 7448/2236 Info     Dll1.DllMain L1 Process detach
10:29:43.400 7448/2236       }} Dll1.DllMain L1 -> Duration: 0.094s
</code>```

I have looked at FILE_FLAG_NO_BUFFERING but it has severe limitations and it seems not easy to use.

Does anybody know the right way to write synchronized to the same file without distoriting the output?

Yours,

Alois Kraus

答: As a starting point, I'd suggest ditching the shared write mode and making use of your named mutex to manage exclusive access to the file. If everything's working as it should, this should give you correct output. Then do some performance testing to see if you can afford to operate like this - you might find it's sufficient.</p>

ContentLicense: CC BY-SA 2.5

Body:
I did create a small C++ Tracing solution which works very well. Within one process all is well but when I open the output file from diferent processes the data gets not correctly written. I did open the file with FILE_SHARE_WRITE to be able to write to the file when it is already open. Then I did create a named mutex to ensure proper synchronisation between processes. But it appears that this is not enough. According to the MSDN this does work within one process but not between different processes.
Next I tried to call FlushFileBuffers after every write while the mutex was still held but data was still distorted like this

The format is time process id/Thread id method enter/leave/severity namespace+method and then the message text.

10:29:42.994 7448/2236       }} Dll2.Test.fndll2 L1 -> Duration: 0.094s
10:29:43.040 7448/2236 {{       Dll2.DllMain L1
10:29:43.134 7448/2236 Info     Dll2.DllMain L1 Process detach
10:29:43.181 7448/2236       }} Dll2.DllMain L1 -> Duration: 0.141s
     }} Dll2.DllMain L1 -10:29:42.681 7448/2236 Info     Dll1.DllMain L1 Process attach
10:29:42.728 7448/2236       }} Dll1.DllMain L1 -10:29:42.744 2216/5510:29:42.775 7448/2236 {{       Dll1.Test.fndll1 10:210:29:42.822 7448/2236 Info     Dll1.Test.fndll1 10:29:42.837 2216/557610:29:42.853 7448/2236       }} Dll1.Test.fndll1 L110:29:42.884 2216/557610:29:43.306 7448/2236 {{       Dll1.DllMain L1
10:29:43.353 7448/2236 Info     Dll1.DllMain L1 Process detach
10:29:43.400 7448/2236       }} Dll1.DllMain L1 -> Duration: 0.094s

I have looked at FILE_FLAG_NO_BUFFERING but it has severe limitations and it seems not easy to use.

Does anybody know the right way to write synchronized to the same file without distoriting the output?

Yours,

Alois Kraus


问:
R - Correlation Matrix - How to plot with this data?
I have the data like below:

```<code>A B 100
A C 100
B D 80
A D 50
B C 5
B D 60
</code>```

basically three columns column 1 and Column 2 are character columns and column 3 is an integer which represents percentage of match between Col1 and Col2. Now I would like to represent this data as a correlation matrix. How can I do the same?

答:
There is already one answer, but here is an idea with only base functions

```<code>heatmap(
  xtabs(
    data = aggregate(V3 ~ V1 + V2, data, 'mean'),
    formula = V3 ~ V1 + V2
  ),
  Rowv = NA,
  Colv = NA,
  revC = T)
)
</code>```

Body:
I have the data like below:

A B 100
A C 100
B D 80
A D 50
B C 5
B D 60

basically three columns column 1 and Column 2 are character columns and column 3 is an integer which represents percentage of match between Col1 and Col2. Now I would like to represent this data as a correlation matrix. How can I do the same?

Body:
There is already one answer, but here is an idea with only base functions

heatmap(
  xtabs(
    data = aggregate(V3 ~ V1 + V2, data, 'mean'),
    formula = V3 ~ V1 + V2
  ),
  Rowv = NA,
  Colv = NA,
  revC = T)
)


问:
Can't edit text on Google Web Designer after I've played with keyframes
Normally I can select the layer I want then select the text tool and edit it but for some reason, after putting in some key frames so the texts fades in and out I can't seem to select the text to edit.

I've locked all the other layers so only the text layer I want to edit is editable but I can't change it. All I can do is move it about.

答: I have encounter this problem when editing a template. I could only edit the text of first layer... In order to edit the second layer text and so on you must hide the fist layer.</p>


问:
Replace two buttons with a toggle switch
I have the following code:

```<code><img id="v1" src="pic1.jpg"><br>
<button onclick="document.getElementById('v1').src='pic1.jpg'">Before</button>
<button onclick="document.getElementById('v1').src='pic2.jpg'">After</button>
<br>
<img id="v2" src="pic3.jpg"><br>
<button onclick="document.getElementById('v2').src='pic3.jpg'">Before</button>
<button onclick="document.getElementById('v2').src='pic4.jpg'">After</button>
<br>
</code>```

However, I would like to replace these 'Before' and 'After' buttons with a toggle switch (already made) in the form of a checkbox:

```<code><label class="switchBA">
<input type="checkbox" checked>
<span class="slider"></span>
</label>
</code>```

In a way that each time it's clicked it switches between the two functions. I guess this needs to be done inline since these are just two of many comparisons.

Thanks in advance.

P.S: I would like to do with only with JS. No need for jQuery or other frameworks.

答:
You can use onchange event to get the event on state change as below

function oncheckchange(e)
{
console.log(event.currentTarget.checked)
if(event.currentTarget.checked)
  document.getElementById('v2').src='pic4.jpg'
else
  document.getElementById('v2').src='pic3.jpg'
}```
<pre class="snippet-code-html lang-html prettyprint-override"><code><img id="v1" src="pic1.jpg"><br>
<button onclick="document.getElementById('v1').src='pic1.jpg'">Before</button>
<button onclick="document.getElementById('v1').src='pic2.jpg'">After</button>
<br>
<img id="v2" src="pic3.jpg"><br>
<button onclick="document.getElementById('v2').src='pic3.jpg'">Before</button>
<button onclick="document.getElementById('v2').src='pic4.jpg'">After</button>
<br>

<label class="switchBA">
<input type="checkbox" checked onchange="oncheckchange()">
<span class="slider"></span>
</label></code>```

Body:
I have the following code:

<button onclick="document.getElementById('v1').src='pic1.jpg'">Before</button>
<button onclick="document.getElementById('v1').src='pic2.jpg'">After</button>

<button onclick="document.getElementById('v2').src='pic3.jpg'">Before</button>
<button onclick="document.getElementById('v2').src='pic4.jpg'">After</button>

However, I would like to replace these 'Before' and 'After' buttons with a toggle switch (already made) in the form of a checkbox:

<input type="checkbox" checked>

In a way that each time it's clicked it switches between the two functions. I guess this needs to be done inline since these are just two of many comparisons.

Thanks in advance.

P.S: I would like to do with only with JS. No need for jQuery or other frameworks.

Body:
You can use onchange event to get the event on state change as below

function oncheckchange(e)
{
console.log(event.currentTarget.checked)
if(event.currentTarget.checked)
  document.getElementById('v2').src='pic4.jpg'
else
  document.getElementById('v2').src='pic3.jpg'
}

<img id="v1" src="pic1.jpg"><br>
<button onclick="document.getElementById('v1').src='pic1.jpg'">Before</button>
<button onclick="document.getElementById('v1').src='pic2.jpg'">After</button>
<br>
<img id="v2" src="pic3.jpg"><br>
<button onclick="document.getElementById('v2').src='pic3.jpg'">Before</button>
<button onclick="document.getElementById('v2').src='pic4.jpg'">After</button>
<br>

<label class="switchBA">
<input type="checkbox" checked onchange="oncheckchange()">
<span class="slider"></span>
</label>


问:
LINQ query to retrieve pivoted data taking too long
I am working on a LINQ query which includes some pivot data as below

```<code>var q = data.GroupBy(x => new
            {
                x.Med.Name,
                x.Med.GenericName,
            }).ToList().Select(g =>
                new SummaryDto
                {
                    Name= g.Key.Name,
                    GenericName = g.Key.GenericName,
                    Data2012 = g.Where(z => z.ProcessDate.Year == 2012).Count(),
                    Data2013 = g.Where(z => z.ProcessDate.Year == 2013).Count(),
                    Data2014 = g.Where(z => z.ProcessDate.Year == 2014).Count(),
                    Data2015 = g.Where(z => z.ProcessDate.Year == 2015).Count(),
                    Data2016 = g.Where(z => z.ProcessDate.Year == 2016).Count(),
                    Data2017 = g.Where(z => z.ProcessDate.Year == 2017).Count(),
                    TotalCount = g.Count(),
                }).AsQueryable();

            return q;
</code>```

The above LINQ takes too long as it queries grp q.Count()*6 times. If there are 10000  records, then it queries 60000 times
Is there a better way to make this faster?

答:
Add year to the group key, then group again, and harvest per-group counts:

```<code>return data.GroupBy(x => new {
    x.Med.Name
,   x.Med.GenericName
,   x.ProcessDate.Year
}).Select(g => new {
    g.Key.Name
,   g.Key.GenericName
,   g.Key.Year
,   Count = g.Count()
}).GroupBy(g => new {
    g.Name
,   g.GenericName
}).Select(g => new SummaryDto {
    Name = g.Key.Name
,   GenericName = g.Key.GenericName
,   Data2012 = g.SingleOrDefault(x => x.Year == 2012)?.Count ?? 0
,   Data2013 = g.SingleOrDefault(x => x.Year == 2013)?.Count ?? 0
,   Data2014 = g.SingleOrDefault(x => x.Year == 2014)?.Count ?? 0
,   Data2015 = g.SingleOrDefault(x => x.Year == 2015)?.Count ?? 0
,   Data2016 = g.SingleOrDefault(x => x.Year == 2016)?.Count ?? 0
,   Data2017 = g.SingleOrDefault(x => x.Year == 2017)?.Count ?? 0
,   TotalCount = g.Sum(x => x.Count)
}).AsQueryable();
</code>```

Note: This approach is problematic, because year is hard-coded in the SummaryDto class. You would be better off passing your DTO constructor an IDictionary<int,int> with counts for each year. If you make this change, the final Select(...) would look like this:

```<code>.Select(g => new SummaryDto {
    Name = g.Key.Name
,   GenericName = g.Key.GenericName
,   TotalCount = g.Sum(x => x.Count)
,   DataByYear = g.ToDictionary(i => i.Year, i => i.Count)
}).AsQueryable();
</code>```

Body:
I am working on a LINQ query which includes some pivot data as below

var q = data.GroupBy(x => new
            {
                x.Med.Name,
                x.Med.GenericName,
            }).ToList().Select(g =>
                new SummaryDto
                {
                    Name= g.Key.Name,
                    GenericName = g.Key.GenericName,
                    Data2012 = g.Where(z => z.ProcessDate.Year == 2012).Count(),
                    Data2013 = g.Where(z => z.ProcessDate.Year == 2013).Count(),
                    Data2014 = g.Where(z => z.ProcessDate.Year == 2014).Count(),
                    Data2015 = g.Where(z => z.ProcessDate.Year == 2015).Count(),
                    Data2016 = g.Where(z => z.ProcessDate.Year == 2016).Count(),
                    Data2017 = g.Where(z => z.ProcessDate.Year == 2017).Count(),
                    TotalCount = g.Count(),
                }).AsQueryable();

            return q;

The above LINQ takes too long as it queries grp q.Count()*6 times. If there are 10000  records, then it queries 60000 times
Is there a better way to make this faster?

Body:
Add year to the group key, then group again, and harvest per-group counts:

return data.GroupBy(x => new {
    x.Med.Name
,   x.Med.GenericName
,   x.ProcessDate.Year
}).Select(g => new {
    g.Key.Name
,   g.Key.GenericName
,   g.Key.Year
,   Count = g.Count()
}).GroupBy(g => new {
    g.Name
,   g.GenericName
}).Select(g => new SummaryDto {
    Name = g.Key.Name
,   GenericName = g.Key.GenericName
,   Data2012 = g.SingleOrDefault(x => x.Year == 2012)?.Count ?? 0
,   Data2013 = g.SingleOrDefault(x => x.Year == 2013)?.Count ?? 0
,   Data2014 = g.SingleOrDefault(x => x.Year == 2014)?.Count ?? 0
,   Data2015 = g.SingleOrDefault(x => x.Year == 2015)?.Count ?? 0
,   Data2016 = g.SingleOrDefault(x => x.Year == 2016)?.Count ?? 0
,   Data2017 = g.SingleOrDefault(x => x.Year == 2017)?.Count ?? 0
,   TotalCount = g.Sum(x => x.Count)
}).AsQueryable();

Note: This approach is problematic, because year is hard-coded in the SummaryDto class. You would be better off passing your DTO constructor an IDictionary<int,int> with counts for each year. If you make this change, the final Select(...) would look like this:

.Select(g => new SummaryDto {
    Name = g.Key.Name
,   GenericName = g.Key.GenericName
,   TotalCount = g.Sum(x => x.Count)
,   DataByYear = g.ToDictionary(i => i.Year, i => i.Count)
}).AsQueryable();


问:
How do I import configuration classes in a @DataJpaTest in a SpringBootTest?
I have a SpringBoot Application and I a config package with

```<code>@Configuration
@EnableJpaAuditing
public class PersistenceConfig {
}
</code>```

But the PersistenceConfig does not get picked up in a PersonRepositoryTest

```<code>@RunWith( SpringRunner.class )
@DataJpaTest
public class PersonRepositoryTest {

    // Tests ...
}
</code>```

However, if I change from @DataJpaTest to @SpringBootTest, PersonRepositoryTest will pick up the config.

My package structure is

```<code>- main
    - java
        - config
              PersistenceConfig.java
        - domain
              Person.java
        - persistence
              PersonRepository.java
          Application.java // @SpringBootApplication

- test
    - java
        - persistence
              PersonRepositoryTest.java
</code>```

The Testing improvements in Spring Boot 1.4 suggest to test the persistence layer with @DataJpaTest

Observation:
Doing both annotations on the Test class still do not import the config
@SpringBootTest
@DataJpaTest

Question 1:
When testing the Persistence Layer with @DataJpaTest
how do I properly (best practise way in Spring Boot) import the config package into my Tests?

Question 2:
Can it be an acceptable work around using @SpringBootTest? I am aware that @DataJpaTest is also a meta annotation with sensible auto configuration for my database including transaction management. But what If I do not need it?

答:
A solution is to use @Import to import your configuration to the configuration done by @DataJpaTest. This is my understanding of @Import.

```<code>@RunWith(SpringRunner.class)
@DataJpaTest
@Import(AuditConfiguration.class)
public class AuditTest {
}
</code>```

with AuditConfiguration that enables auditing

```<code>@Configuration
@EnableJpaAuditing
public class AuditConfiguration {
}
</code>```

Body:
I have a SpringBoot Application and I a config package with

@Configuration
@EnableJpaAuditing
public class PersistenceConfig {
}

But the PersistenceConfig does not get picked up in a PersonRepositoryTest

@RunWith( SpringRunner.class )
@DataJpaTest
public class PersonRepositoryTest {

    // Tests ...
}

However, if I change from @DataJpaTest to @SpringBootTest, PersonRepositoryTest will pick up the config.

My package structure is

- main
    - java
        - config
              PersistenceConfig.java
        - domain
              Person.java
        - persistence
              PersonRepository.java
          Application.java // @SpringBootApplication

- test
    - java
        - persistence
              PersonRepositoryTest.java

The Testing improvements in Spring Boot 1.4 suggest to test the persistence layer with @DataJpaTest

Observation:
Doing both annotations on the Test class still do not import the config
@SpringBootTest
@DataJpaTest

Question 1:
When testing the Persistence Layer with @DataJpaTest
how do I properly (best practise way in Spring Boot) import the config package into my Tests?

Question 2:
Can it be an acceptable work around using @SpringBootTest? I am aware that @DataJpaTest is also a meta annotation with sensible auto configuration for my database including transaction management. But what If I do not need it?

Body:
A solution is to use @Import to import your configuration to the configuration done by @DataJpaTest. This is my understanding of @Import.

@RunWith(SpringRunner.class)
@DataJpaTest
@Import(AuditConfiguration.class)
public class AuditTest {
}

with AuditConfiguration that enables auditing

@Configuration
@EnableJpaAuditing
public class AuditConfiguration {
}


问:
Varied Constructor Dependency Injection based on Interface
Say I have this:

```<code>public class Human : SentientLifeForm, IHuman {
    IBreathingService breathingService = null;

    public Human(IBreathingService breathingService) {
        this.breathingService = breathingService
    }
}

public class Martian : SentientLifeForm, IMartian {
    IBreathingService breathingService = null;

    public Martian(IBreathingService breathingService) {
        this.breathingService = breathingService
    }
}
</code>```

Then I have two different implementations of the BreathingService...

```<code>public class HumanBreathingService { ... }
public class MartianBreathingService { ... }
</code>```

How can I use DI to pass the correct service implementation to the associated object? Or do I have to specify an IHumanBreathingService and an IMartianBreathingService?

答:
Looks like Autofac - How to create a generated factory with parameters already answers this.

Autofac enables the ability to create a factory for creating the correct service.

Body:
Say I have this:

public class Human : SentientLifeForm, IHuman {
    IBreathingService breathingService = null;

    public Human(IBreathingService breathingService) {
        this.breathingService = breathingService
    }
}

public class Martian : SentientLifeForm, IMartian {
    IBreathingService breathingService = null;

    public Martian(IBreathingService breathingService) {
        this.breathingService = breathingService
    }
}

Then I have two different implementations of the BreathingService...

public class HumanBreathingService { ... }
public class MartianBreathingService { ... }

How can I use DI to pass the correct service implementation to the associated object? Or do I have to specify an IHumanBreathingService and an IMartianBreathingService?


问:
Enabling Internet Connection Through Proxy in Android in Windows
How to Enable Internet Connection Through Proxy in Android in Windows?</p>

答:
http://www.coderanch.com/t/462906/Android/Mobile/connect-internet-behind-proxy

In the earlier versions of Android emulators (up to version 1.1r2), you were required to make an entry in the system table of“com.android.provider.setting.db” database and/or start the emulator with –http-proxy switch supplying it the IP address and port number of your proxy.

These methods however, have become obsolete and don’t work with SDKv1.5.
Step 1: On emulator, go to:
Home->Menu->Settings->WirelessControls->MobileNetworks->Access Point Names->T-mobile US->set Proxy IP and Port#

Step 2: Now the next time you go to a web address, the browser will prompt for your user-id and password. Enter your credentials and you should be good to go.

Reference: http://groups.google.com/group/android-beginners/b…d4f948c4ed622#5ced4f948c4ed622” target=”_new” rel=”nofollow”>

Steps for Android 2.2

1) Create an emulator with name say android_2.2_emulator using AVD manager in eclipse.

2) Create a batch file (.bat) with following command and a keep a handy shortcut.

emulator -avd “android_2.2_emulator” -http-proxy “your_proxy_url”:”port”

3) Run the batch file before starting the eclipse IDE, so that the emulator session will have internet connection.

That’s it.


答:
Your question lacks specificity therefore I will explicitly explain the assumption I am making.

Assumption 1:  SentientLifeForm implements IBreathingService.

Assumption 2:  HumanBreathingService and MartianBreathingService implement/derive-from IBreathingService.

Assumption 3:  HumanBreathingService will be passed to the constructor for Human as a parameter and MartianBreathingService will be passed to the constructor for Martian as a parameter.  THIS IS THE DI IN THIS PROJECT.

Thereafter, say you have a method SolarSystemBeing(..) that takes as an argument an instance of SentientLifeForm.  Then depending on which instance it gets a Human or a martian the breathingService will be the appropriate one...

Title: Varied Constructor Dependency Injection based on Interface


问:
PostgreSQL connection WITH OpenS(QL)hift from a Local OS X Python Script
I'm trying to connect to OpenShift from a OS X Python Script.  OpenShift for sure works.  However, I'm looking for an "easy" way with python without downloading additional PGSQL libs to OS X.  Thanks!

Question: What is the correct domain name to use for this??  And can we keep it simple without doing an import to python?

I don't want to use any additional libraries.

the main problem I'm having is with pgsql.connect(dbname = 'orion', host = '127.7.74.2:5432', #'localhost', port = 5432, user = 'earthling', password = 'extraterrestrial')

Here is what the code file looks like "pgsql.py"...

```<code># PostgreSQL Client:
# http://www.postgresql.org/docs/current/static/reference-client.html

# PostgreSQL Server:
# http://www.postgresql.org/docs/current/static/runtime.html

# PostgreSQL Commands:
# http://www.postgresql.org/docs/current/static/reference.html

# PostgreSQL Data Types:
# http://www.postgresql.org/docs/current/static/datatype.html

conn = psycopg2.connect(dbname = 'orion',
                host = '127.7.74.2:5432', #'localhost',
                port = 5432,
                user = 'earthling',
                password = 'extraterrestrial')

sql = conn.cursor()

def CREATE_TABLE(table, text):
    sql.execute("CREATE TABLE " + table + " (" + text + ");")

def INSERT(table, url, text):
    #date = datetime.date.today()
    date = datetime.datetime.fromtimestamp(time.time()).strftime('%Y-%m-%d %H:%M:%S')
    print date
    query = "INSERT INTO " + table + " (url, text, date) VALUES (%s, %s, %s);"
    params = (url, text, date,)
    sql.execute(query, params)

def SELECT(table, url):
    query = "SELECT * FROM " + table + " WHERE url = %s;"
    sql.execute(query, (url,))

def QUERY(text):
    sql.execute(text)

INSERT("main", "www.whoi.edu", "test")
print SELECT("main", "www.whoi.edu")
</code>```

Thanks for your help!

Question: This should fairly simple and shouldn't require psycopg2?

答:
Can you share what problem you're having when you attempt to connect?

As far as working with Python, we've had to use cx_oracle to connect to our oracle tables, so I'd expect you'd have to use psycopg2 to connect with PostGresQL as there's probably not a native functionality without a library.

However I just read you're on OS X, so it may be entirely different. Tag your question with OS X, see if that helps fetch an answer better.

Body:
I'm trying to connect to OpenShift from a OS X Python Script.  OpenShift for sure works.  However, I'm looking for an "easy" way with python without downloading additional PGSQL libs to OS X.  Thanks!

Question: What is the correct domain name to use for this??  And can we keep it simple without doing an import to python?

I don't want to use any additional libraries.

the main problem I'm having is with pgsql.connect(dbname = 'orion', host = '127.7.74.2:5432', #'localhost', port = 5432, user = 'earthling', password = 'extraterrestrial')

Here is what the code file looks like "pgsql.py"...

# PostgreSQL Client:
# http://www.postgresql.org/docs/current/static/reference-client.html

# PostgreSQL Server:
# http://www.postgresql.org/docs/current/static/runtime.html

# PostgreSQL Commands:
# http://www.postgresql.org/docs/current/static/reference.html

# PostgreSQL Data Types:
# http://www.postgresql.org/docs/current/static/datatype.html

conn = psycopg2.connect(dbname = 'orion',
                host = '127.7.74.2:5432', #'localhost',
                port = 5432,
                user = 'earthling',
                password = 'extraterrestrial')

sql = conn.cursor()

def CREATE_TABLE(table, text):
    sql.execute("CREATE TABLE " + table + " (" + text + ");")

def INSERT(table, url, text):
    #date = datetime.date.today()
    date = datetime.datetime.fromtimestamp(time.time()).strftime('%Y-%m-%d %H:%M:%S')
    print date
    query = "INSERT INTO " + table + " (url, text, date) VALUES (%s, %s, %s);"
    params = (url, text, date,)
    sql.execute(query, params)

def SELECT(table, url):
    query = "SELECT * FROM " + table + " WHERE url = %s;"
    sql.execute(query, (url,))

def QUERY(text):
    sql.execute(text)

INSERT("main", "www.whoi.edu", "test")
print SELECT("main", "www.whoi.edu")

Thanks for your help!

Question: This should fairly simple and shouldn't require psycopg2?


问:
Showing 12 months in 4 rows using jQuery UI
I can't for the life of me figure out how to show 12 months over 4 rows with this calendar. I've tried a bunch of different combos for the numberOfMonths and nothing. Using jQuery 1.12.1. here's my code:

```<code>$('#datepicker').datepicker({
    numberOfMonths: [1,4]
});
</code>```

答:
Just replace [1,4] with [4,3]

kindly refer to this fiddle

```<code>$('#datepicker').datepicker({
              numberOfMonths: [4,3]
            });
</code>```

Body:
I can't for the life of me figure out how to show 12 months over 4 rows with this calendar. I've tried a bunch of different combos for the numberOfMonths and nothing. Using jQuery 1.12.1. here's my code:

$('#datepicker').datepicker({
    numberOfMonths: [1,4]
});

Body:
Just replace [1,4] with [4,3]

kindly refer to this fiddle

$('#datepicker').datepicker({
              numberOfMonths: [4,3]
            });


问:
How to open another app from ionic app?
I want to open an app in my phone from my ionic app. Along with that I want to send some data(as given below) too. I want to do this using uri scheme. URI for

Android:
someapp://com.example.example/?someField=someData

iOS:
someapp://?someField=someData

I have triend this but it's not working:

```<code>var browserRef = this.inAppBrowser.create(
        "twitter://com.twitter.android",
        "_system",
        "location=no,hidden=no,zoom=no"
);
</code>```

however when i use url as https://twitter.com like below, it's working just fine.

```<code>var browserRef = this.inAppBrowser.create(
        'https://twitter.com',
        "_system",
        "location=no,hidden=no,zoom=no"
      );
</code>```

but I can't use the http link as I want to send data in above specified format. Please help!!

My system configuration is:

```<code>    @ionic/cli-utils  : 1.12.0
    ionic (Ionic CLI) : 3.12.0

global packages:

    cordova (Cordova CLI) : 7.0.1

local packages:

    @ionic/app-scripts : 1.3.12
    Cordova Platforms  : android 6.2.3 ios 4.4.0
    Ionic Framework    : ionic-angular 3.7.1

System:

    Android SDK Tools : 26.1.1
    Node              : v8.6.0
    npm               : 4.6.1
    OS                : macOS Sierra
    Xcode             : Xcode 8.3.3
</code>```

答:
Try to add this in your config.xml.

```<code><access origin="someapp:*" launch-external="yes" />
</code>```

Add this in your Whitelist, is needed to launch external application.

Title: How to open another app from ionic app?

Body:
I want to open an app in my phone from my ionic app. Along with that I want to send some data(as given below) too. I want to do this using uri scheme. URI for

Android:
someapp://com.example.example/?someField=someData

iOS:
someapp://?someField=someData

I have triend this but it's not working:

var browserRef = this.inAppBrowser.create(
        "twitter://com.twitter.android",
        "_system",
        "location=no,hidden=no,zoom=no"
);

however when i use url as https://twitter.com like below, it's working just fine.

var browserRef = this.inAppBrowser.create(
        'https://twitter.com',
        "_system",
        "location=no,hidden=no,zoom=no"
      );

but I can't use the http link as I want to send data in above specified format. Please help!!

My system configuration is:

    @ionic/cli-utils  : 1.12.0
    ionic (Ionic CLI) : 3.12.0

global packages:

    cordova (Cordova CLI) : 7.0.1

local packages:

    @ionic/app-scripts : 1.3.12
    Cordova Platforms  : android 6.2.3 ios 4.4.0
    Ionic Framework    : ionic-angular 3.7.1

System:

    Android SDK Tools : 26.1.1
    Node              : v8.6.0
    npm               : 4.6.1
    OS                : macOS Sierra
    Xcode             : Xcode 8.3.3

Body:
Try to add this in your config.xml.

<access origin="someapp:*" launch-external="yes" />

Add this in your Whitelist, is needed to launch external application.


问:
Have different initial_data fixtures for different stages (testing v. production)
I have an initial_data fixture that I want to load everytime except for production. I already have different settings file for production and non-production deployments.

Any suggestions on how to accomplish this?

Clarification: I do not want test fixtures. Basically, I just need the fixture to be loaded based on a setting change of some sort. I'll be digging into the Django code to see if I could figure out an elegant way to accomplish this.

答:
You can actually setup different test fixtures for each test if you want:
http://docs.djangoproject.com/en/dev/topics/testing/#topics-testing-fixtures

If you only want to load the fixtures in one time, you can also write a custom TestRunner that will allow you to do that setup at the beginning:

docs.djangoproject.com/en/dev/topics/testing/#using-different-testing-frameworks

Both of those will still load the data from the production fixtures as that is done with syncdb, but you can override the data, or even delete it all.  This may not be optimal if you are loading large amounts of data into your production product.  If this is the case, I would recommend you adding a custom command like load_production_data that allows you to do it quickly and easily from the command line.


问:
Select non empty elements of list in Python
Given:

```<code>a = 'aaa'
b = ''
c = 'ccc'
d = ''
e = 'eee'

list = (a, b, c, d, e)
</code>```

How can I get a string using all the non empty elements of the list ?

Desired output:

```<code>'aaa,ccc,eee'
</code>```

答:
You can use a generator-expression:

```<code>','.join(s for s in list if s)
</code>```

which outputs:

```<code>'aaa,ccc,eee'
</code>```

Why?

This takes advantage of the fact that an empty string evaluates to False.

This can be seen clearer through some examples:

```<code>>>> if "a":
...     print("yes")
...
yes
>>> if " ":
...     print("yes")
...
yes
>>> if "":
...     print("yes")
...
>>>
</code>```

So the generator says: for each string in list, keep 'if string' - i.e. if the string is not empty.

We then finally use the str.join method that will take each string in the iterator that is passed into it (here a generator) and concatenate them together with whatever the str is. So here we use a ',' as the string to get the desired result.

A little example of this:

```<code>>>> ','.join(['abc', 'def', 'ghi'])
'abc,def,ghi'
</code>```

**As a side note, you shouldn't name your variable list as it overrides the built-in list() function:

```<code>>>> list((1, 2, 3))
[1, 2, 3]
>>> list = 1
>>> list((1, 2, 3))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not callable
</code>```

Body:
Given:

a = 'aaa'
b = ''
c = 'ccc'
d = ''
e = 'eee'

list = (a, b, c, d, e)

How can I get a string using all the non empty elements of the list ?

Desired output:

'aaa,ccc,eee'

Body:
You can use a generator-expression:

','.join(s for s in list if s)

which outputs:

'aaa,ccc,eee'

Why?

This takes advantage of the fact that an empty string evaluates to False.

This can be seen clearer through some examples:

>>> if "a":
...     print("yes")
...
yes
>>> if " ":
...     print("yes")
...
yes
>>> if "":
...     print("yes")
...
>>>

So the generator says: for each string in list, keep 'if string' - i.e. if the string is not empty.

We then finally use the str.join method that will take each string in the iterator that is passed into it (here a generator) and concatenate them together with whatever the str is. So here we use a ',' as the string to get the desired result.

A little example of this:

>>> ','.join(['abc', 'def', 'ghi'])
'abc,def,ghi'

**As a side note, you shouldn't name your variable list as it overrides the built-in list() function:

>>> list((1, 2, 3))
[1, 2, 3]
>>> list = 1
>>> list((1, 2, 3))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not callable


问:
How to get exact position in pixels of my TextView?
I have a smartphone of 2560 x 1440 px. Now I am using this function for my TextView:

```<code>int[] locationOnScreen = new int[2];
txtAp.GetLocationInWindow(locationOnScreen);
</code>```

It is supposed to give me the total x and y coordinates in pixels.
My TextView is pretty much in the middle of the screen, so supposedly at
(1280,770). But the function returns [69, 1111].

How can that be? If that is not the way of doing that, then what is?

答:
I have a smartphone of 2560 x 1440 px... My TextView is pretty much in the middle of the screen, so supposedly at (1280,770).

Not correct, unless you talk about the center of the TextView. Any view is rendered on screen within its rectangle, so its coordinates are considered to be of [left, top, right, bottom].

The GetLocationInWindow() method returns (left, top) coordinates of a view. So (x = 69, y = 1111) looks to be meaningful for the left top corner of your TextView located in the middle of screen.

Important note: GetLocationInWindow() returns the coordinates w.r.t the root view, not actual window. You should take the status bar height into consideration. And here is how you can calculate Height of status bar in Android.

Body:
I have a smartphone of 2560 x 1440 px. Now I am using this function for my TextView:

int[] locationOnScreen = new int[2];
txtAp.GetLocationInWindow(locationOnScreen);

It is supposed to give me the total x and y coordinates in pixels.
My TextView is pretty much in the middle of the screen, so supposedly at
(1280,770). But the function returns [69, 1111].

How can that be? If that is not the way of doing that, then what is?


问:
Unable to delete Charts while deleting specific columns in VBA for Excel
I am trying to remove some columns based on different conditions. These column sets are having tables along with their respective charts below them. When i run the code line by line, it works perfect but when i run the complete code then it works well on the tables but the charts below them doesn't delete at all.

source file

```<code>If oper(j) = "ABC" Then

   wtemp.Sheets("Graphs").Range("A:H, Q:AF").Delete

         ElseIf oper(j) = "XYZ" Then

               wtemp.Sheets("Graphs").Range("A:P, Y:AF").Delete

                    ElseIf oper(j) = "KHG" Then

                          wtemp.Sheets("Graphs").Range("A:X").Delete

                    Else

                    wtemp.Sheets("Graphs").Range("I:AF").Delete

End If
</code>```

答:
A ChartObject's Placement property controls the way it is attached to its underlying cells; see XlPlacement Enumeration. The chart objects that remain (after columns completely covering them are deleted) most probably have Placement = xlFreeFloating.

Here's a sub to handle this situation. It first buffers the Placement values, then performs the column deletion, and re-establishes the original Placement values.

```<code>Public Sub DeleteColumnsAndCharts(ByVal pColumnsToDelete As Excel.Range)
    On Error GoTo errHandler

    Dim PlacementByChartName As Object 'Scripting.Dictionary
    Dim chartsWorksheet As Excel.Worksheet
    Dim chart As Excel.ChartObject

    Set PlacementByChartName = CreateObject("Scripting.Dictionary")
    Set chartsWorksheet = pColumnsToDelete.Worksheet

    'Keep the original Placement values before changing them.
    For Each chart In chartsWorksheet.ChartObjects
        PlacementByChartName(chart.Name) = chart.Placement
        chart.Placement = xlMoveAndSize
    Next

    'Should the Delete method below fail, the Recover section will leave the worksheet intact.
    pColumnsToDelete.EntireColumn.Delete

Recover:
    On Error Resume Next
    'Restore the original Placement values on the remaining chart objects.
    For Each chart In chartsWorksheet.ChartObjects
        chart.Placement = PlacementByChartName(chart.Name)
    Next
    Set chart = Nothing
    Set chartsWorksheet = Nothing
    Set PlacementByChartName = Nothing
    Exit Sub

errHandler:
    MsgBox Err.Description, vbExclamation + vbOKOnly, "Error"
    Resume Recover
End Sub
</code>```

You can call it like this:

```<code>DeleteColumnsAndCharts wtemp.Sheets("Graphs").Range("A:X")
</code>```

Body:
I am trying to remove some columns based on different conditions. These column sets are having tables along with their respective charts below them. When i run the code line by line, it works perfect but when i run the complete code then it works well on the tables but the charts below them doesn't delete at all.

source file

If oper(j) = "ABC" Then

   wtemp.Sheets("Graphs").Range("A:H, Q:AF").Delete

         ElseIf oper(j) = "XYZ" Then

               wtemp.Sheets("Graphs").Range("A:P, Y:AF").Delete

                    ElseIf oper(j) = "KHG" Then

                          wtemp.Sheets("Graphs").Range("A:X").Delete

                    Else

                    wtemp.Sheets("Graphs").Range("I:AF").Delete

End If

Body:
A ChartObject's Placement property controls the way it is attached to its underlying cells; see XlPlacement Enumeration. The chart objects that remain (after columns completely covering them are deleted) most probably have Placement = xlFreeFloating.

Here's a sub to handle this situation. It first buffers the Placement values, then performs the column deletion, and re-establishes the original Placement values.

Public Sub DeleteColumnsAndCharts(ByVal pColumnsToDelete As Excel.Range)
    On Error GoTo errHandler

    Dim PlacementByChartName As Object 'Scripting.Dictionary
    Dim chartsWorksheet As Excel.Worksheet
    Dim chart As Excel.ChartObject

    Set PlacementByChartName = CreateObject("Scripting.Dictionary")
    Set chartsWorksheet = pColumnsToDelete.Worksheet

    'Keep the original Placement values before changing them.
    For Each chart In chartsWorksheet.ChartObjects
        PlacementByChartName(chart.Name) = chart.Placement
        chart.Placement = xlMoveAndSize
    Next

    'Should the Delete method below fail, the Recover section will leave the worksheet intact.
    pColumnsToDelete.EntireColumn.Delete

Recover:
    On Error Resume Next
    'Restore the original Placement values on the remaining chart objects.
    For Each chart In chartsWorksheet.ChartObjects
        chart.Placement = PlacementByChartName(chart.Name)
    Next
    Set chart = Nothing
    Set chartsWorksheet = Nothing
    Set PlacementByChartName = Nothing
    Exit Sub

errHandler:
    MsgBox Err.Description, vbExclamation + vbOKOnly, "Error"
    Resume Recover
End Sub

You can call it like this:

DeleteColumnsAndCharts wtemp.Sheets("Graphs").Range("A:X")


问:
PL/pgSQL Syntax Error at end of Input
I'm new to PL/pgSQL so this is kind of trial and error for me. I'm trying to create a blank function that when called will populate a table. Here is what I have so far:

```<code>create or replace function counties()
returns table(code varchar(16), county varchar(50), count int) as $$
DECLARE
    new_version varchar(50) := concat('sometext', max(REGEXP_REPLACE(COALESCE(schema_name, '0'), '[^0-9]*' ,'0')::integer), 'sometext') from information_schema.schemata where schema_name like 'gcversa%';
    old_version varchar(50) := concat('sometext', max(REGEXP_REPLACE(COALESCE(schema_name, '0'), '[^0-9]*' ,'0')::integer)-2, 'sometext') from information_schema.schemata where schema_name like 'gcversa%';
BEGIN RETURN QUERY
    with cte_current as (select distinct a.code as code, b.countyname as county from old_version a, public.counties b
                 where st_intersects(a.geom,b.geom) = True group by a.code, b.countyname),

         cte_new     as (select distinct a.code as code, b.countyname as county from new_version a, public.counties b
                 where st_intersects(a.geom,b.geom) = True group by a.code, b.countyname),

        cte_union as (select code, county from cte_current
        union all
        select code, county from cte_new)

    select code,county, count(*) as count
    from cte_union
    group by code, county
    Having count (*) <> 2;
END;
$$
LANGUAGE plpgsql;
</code>```

The two variables that I have declared are referencing and concatenating two PostgreSQL schema's, and then I'm trying to insert those variables into the CTE's, in order to then union them all together. I got the function the execute properly, but when I call select counties(), I get this error:

```<code>ERROR:  relation "old_version" does not exist
LINE 1: ...tinct a.code as code, b.countyname as county from old_versio...
</code>```

So, it's not picking up the variable that I declared, any suggestions?

UPDATE
Per Vao Tsun's suggestion I utilized dynamic SQL and the format() function. Here is what the final working model looks like:

```<code>drop function counties();

create or replace function counties()
returns table(code varchar(16), county varchar(50), count bigint) as $$
DECLARE
    new_version_number varchar(50) := concat('gcversa00', MAX("versionnumber")) from "gcdefault"."versionhistory";
    old_version_number varchar(50) := concat('gcversa00', MAX("versionnumber"-2)) from "gcdefault"."versionhistory";
BEGIN RETURN QUERY EXECUTE
format(
    'with cte_current as (select distinct a.code as code, b.countyname as county from %I.servicearea a, public.counties b
                 where st_intersects(a.geom,b.geom) = True group by a.code, b.countyname),

         cte_new     as (select distinct a.code as code, b.countyname as county from %I.servicearea a, public.counties b
                 where st_intersects(a.geom,b.geom) = True group by a.code, b.countyname),

        cte_union as (select code, county from cte_current
        union all
        select code, county from cte_new)

    select code,county, count(*) as count
    from cte_union
    group by code, county
    Having count (*) <> 2', new_version_number, old_version_number
    );
END;
$$
LANGUAGE plpgsql;
</code>```

答:
you would need dynamic sql here, eg:</p>

```<code>t=# create or replace function ds() returns table (i int) as
$$
declare
tn text := 'pg_database';
begin
return query execute format('select oid::int from %I',tn);
end;
$$language plpgsql
;
CREATE FUNCTION
t=# select * from ds();
     i
-----------
         1
     12945
     12950
  12963038
 111274822
  32059515
  26947889
 173381559
  32061155
  32061156
  82287221
 203004236
 214018521
(13 rows)
</code>```

Title: PL/pgSQL Syntax Error at end of Input

Body:
I'm new to PL/pgSQL so this is kind of trial and error for me. I'm trying to create a blank function that when called will populate a table. Here is what I have so far:

create or replace function counties()
returns table(code varchar(16), county varchar(50), count int) as $$
DECLARE
    new_version varchar(50) := concat('sometext', max(REGEXP_REPLACE(COALESCE(schema_name, '0'), '[^0-9]*' ,'0')::integer), 'sometext') from information_schema.schemata where schema_name like 'gcversa%';
    old_version varchar(50) := concat('sometext', max(REGEXP_REPLACE(COALESCE(schema_name, '0'), '[^0-9]*' ,'0')::integer)-2, 'sometext') from information_schema.schemata where schema_name like 'gcversa%';
BEGIN RETURN QUERY
    with cte_current as (select distinct a.code as code, b.countyname as county from old_version a, public.counties b
                 where st_intersects(a.geom,b.geom) = True group by a.code, b.countyname),

         cte_new     as (select distinct a.code as code, b.countyname as county from new_version a, public.counties b
                 where st_intersects(a.geom,b.geom) = True group by a.code, b.countyname),

        cte_union as (select code, county from cte_current
        union all
        select code, county from cte_new)

    select code,county, count(*) as count
    from cte_union
    group by code, county
    Having count (*) <> 2;
END;
$$
LANGUAGE plpgsql;

The two variables that I have declared are referencing and concatenating two PostgreSQL schema's, and then I'm trying to insert those variables into the CTE's, in order to then union them all together. I got the function the execute properly, but when I call select counties(), I get this error:

ERROR:  relation "old_version" does not exist
LINE 1: ...tinct a.code as code, b.countyname as county from old_versio...

So, it's not picking up the variable that I declared, any suggestions?

UPDATE
Per Vao Tsun's suggestion I utilized dynamic SQL and the format() function. Here is what the final working model looks like:

drop function counties();

create or replace function counties()
returns table(code varchar(16), county varchar(50), count bigint) as $$
DECLARE
    new_version_number varchar(50) := concat('gcversa00', MAX("versionnumber")) from "gcdefault"."versionhistory";
    old_version_number varchar(50) := concat('gcversa00', MAX("versionnumber"-2)) from "gcdefault"."versionhistory";
BEGIN RETURN QUERY EXECUTE
format(
    'with cte_current as (select distinct a.code as code, b.countyname as county from %I.servicearea a, public.counties b
                 where st_intersects(a.geom,b.geom) = True group by a.code, b.countyname),

         cte_new     as (select distinct a.code as code, b.countyname as county from %I.servicearea a, public.counties b
                 where st_intersects(a.geom,b.geom) = True group by a.code, b.countyname),

        cte_union as (select code, county from cte_current
        union all
        select code, county from cte_new)

    select code,county, count(*) as count
    from cte_union
    group by code, county
    Having count (*) <> 2', new_version_number, old_version_number
    );
END;
$$
LANGUAGE plpgsql;

Body:
you would need dynamic sql here, eg:

t=# create or replace function ds() returns table (i int) as
$$
declare
tn text := 'pg_database';
begin
return query execute format('select oid::int from %I',tn);
end;
$$language plpgsql
;
CREATE FUNCTION
t=# select * from ds();
     i
-----------
         1
     12945
     12950
  12963038
 111274822
  32059515
  26947889
 173381559
  32061155
  32061156
  82287221
 203004236
 214018521
(13 rows)


问:
Mapping nested properties in same object
I've a question and I hope this is possible.
I want to map properties from an nested class into his parent class (without creating a new object, this should be the same object)
Below I've attached an example.

I want to map the properties from the Bar class (which is in de Foo class) to the properties on the Bar class. So I've an instance of Foo and SomeIntProperty and SomeString property has their default value, the Bar property instead has some real values. I want to map those values from the Bar property from Foo to Foo. Is this possible, and is it possible without specifying a mapper for the properties? My classes contains a lot of properties..

I find it really hard to explain, sorry if it is still confusing.

```<code>    public class Foo
{
    public int SomeIntProperty { get; set; }
    public string SomeStringProperty { get; set; }

    // the properties from this Bar instance should be mapped to the properties above.
    public Bar Bar { get; set; }
}

public class Bar
{
    public int SomeIntProperty { get; set; }
    public string SomeStringProperty { get; set; }
    public bool ThisPropertyDoesntExistsOnFooClass { get; set; }
}
</code>```

答: You can use Automapper as described here: Automapper: Update property values without creating a new object</p>

Body:
I've a question and I hope this is possible.
I want to map properties from an nested class into his parent class (without creating a new object, this should be the same object)
Below I've attached an example.

I want to map the properties from the Bar class (which is in de Foo class) to the properties on the Bar class. So I've an instance of Foo and SomeIntProperty and SomeString property has their default value, the Bar property instead has some real values. I want to map those values from the Bar property from Foo to Foo. Is this possible, and is it possible without specifying a mapper for the properties? My classes contains a lot of properties..

I find it really hard to explain, sorry if it is still confusing.

    public class Foo
{
    public int SomeIntProperty { get; set; }
    public string SomeStringProperty { get; set; }

    // the properties from this Bar instance should be mapped to the properties above.
    public Bar Bar { get; set; }
}

public class Bar
{
    public int SomeIntProperty { get; set; }
    public string SomeStringProperty { get; set; }
    public bool ThisPropertyDoesntExistsOnFooClass { get; set; }
}


问:
++someVariable vs. someVariable++ in JavaScript
In JavaScript you can use ++ operator before (pre-increment) or after the variable name (post-increment).  What, if any, are the differences between these ways of incrementing a variable?

ContentLicense: CC BY-SA 4.0

Body:
var a = 1;
var b = ++a;
alert('a:' + a + ';b:' + b); //a:2;b:2

var c = 1;
var d = c++;
alert('c:' + c + ';d:' + d); //c:2;d:1

jsfiddle


问:
Browser not launching with BrowserSync or live-server
I've been using live-server with VS Code to debug some web pages. Everything was fine, I ran live-server from my command line in the page directory and Chrome opened a new tab with the page.

Then suddenly it stopped working! I though it was a bit weird so I installed Nodejs again and an issue appeared afterwards. The server is working but it isn't able to open Chrome (nor any other Browser)

I tried using BrowserSync instead of live-server and again, the server works but it isn't able to automatically open Chrome

I can always manually type http://localhost:3000 but I shouldn't (since it worked before)

I've tried uninstalling and installing both live-server and browserSync, Chrome.
Flushing my dns, checking for malware, checking if Windows Firewall is blocking anything related, and some other things that hadn't done anything for me, I still get the:

Couldn't open browser (if you are using BrowserSync in a headless environment, you might want to set
the open option to false)

as seen here

Can someone please tell me what could be causing this issue or how should I attempt to fix it?

答:
The marked answer didn't actually work for me but manually setting the browser property in the init function did.

```<code>gulp.task('browserSync', () => {
        browserSync.init({
            server: {
                baseDir: task.dir.base,
                middleware: [
                    webpackDevMiddleware(bundler, {
                        publicPath: webpackConfig.output.publicPath,
                        stats: 'errors-only'
                    })
                  ]
            },
            browser: 'chrome'
        });
    });
</code>```

I'm running windows 10 x64

Body:
The marked answer didn't actually work for me but manually setting the browser property in the init function did.

gulp.task('browserSync', () => {
        browserSync.init({
            server: {
                baseDir: task.dir.base,
                middleware: [
                    webpackDevMiddleware(bundler, {
                        publicPath: webpackConfig.output.publicPath,
                        stats: 'errors-only'
                    })
                  ]
            },
            browser: 'chrome'
        });
    });

I'm running windows 10 x64


问:
Scrapy: USER_AGENT and ROBOTSTXT_OBEY are properly set, but I still get error 403
Hello and thanks in advance for the help or direction you can bring. This is my scraper:

```<code>import scrapy
class RakutenSpider(scrapy.Spider):
    name = "rak"
    allowed_domains = ["rakuten.com"]
    start_urls = ['https://www.rakuten.com/deals?omadtrack=hp_deals_viewmore']
    def parse(self, response):
        for sel in response.xpath('//div[@class="page-bottom"]/div'):
            yield {
                'titles': sel.xpath("//div[@class='slider-prod-title']").extract_first(),
                'prices': sel.xpath("//span[@class='price-bold']").extract_first(),
                'images': sel.xpath("//div[@class='deal-img']/img").extract_first()
            }
</code>```

And this is part of my settings.py

```<code>USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'
CONCURRENT_REQUESTS = 1
DOWNLOAD_DELAY = 5
# Obey robots.txt rules
ROBOTSTXT_OBEY = 'False'
</code>```

and this is part of the log:

```<code>DEBUG: Crawled (403) <GET https://www.rakuten.com/deals?omadtrack=hp_deals_viewmore> (referer: None)
</code>```

I have tried almost all solutions I found in s/o

log file: This is a new log after installing Firefox driver. Now I get an ERROR: Error downloading https://www.rakuten.com/deals?omadtrack=hp_deals_viewmore>

```<code>2017-11-17 00:38:45 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: scrapybot)
2017-11-17 00:38:45 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'deals.spiders', 'CONCURRENT_REQUESTS': 1, 'SPIDER_MODULES': ['deals.spiders'], 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36', 'TELNETCONSOLE_ENABLED': False, 'DOWNLOAD_DELAY': 5}
2017-11-17 00:38:45 [py.warnings] WARNING: :0: UserWarning: You do not have a working installation of the service_identity module: 'No module named cryptography.x509'.  Please install it from <https://pypi.python.org/pypi/service_identity> and make sure all of its dependencies are satisfied.  Without the service_identity module and a recent enough pyOpenSSL to support it, Twisted can perform only rudimentary TLS client hostname verification.  Many valid certificate/hostname mappings may be rejected.

2017-11-17 00:38:45 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.corestats.CoreStats']
2017-11-17 00:38:45 [scrapy.middleware] INFO: Enabled downloader middlewares:
['deals.middlewares.JSMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-11-17 00:38:45 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-11-17 00:38:45 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-11-17 00:38:45 [scrapy.core.engine] INFO: Spider opened
2017-11-17 00:38:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-11-17 00:38:45 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.rakuten.com/deals?omadtrack=hp_deals_viewmore>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/core/downloader/middleware.py", line 37, in process_request
    response = yield method(request=request, spider=spider)
  File "/home/seealldeals/tmp/scrapy/deals/deals/middlewares.py", line 63, in process_request
    driver = webdriver.Firefox()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 144, in __init__
    self.service.start()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/service.py", line 74, in start
    stdout=self.log_file, stderr=self.log_file)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 8] Exec format error
2017-11-17 00:38:45 [scrapy.core.engine] INFO: Closing spider (finished)
2017-11-17 00:38:45 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 1,
 'downloader/exception_type_count/exceptions.OSError': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 11, 17, 5, 38, 45, 328366),
 'log_count/ERROR': 1,
 'log_count/INFO': 7,
 'log_count/WARNING': 1,
 'memusage/max': 33509376,
 'memusage/startup': 33509376,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2017, 11, 17, 5, 38, 45, 112667)}
2017-11-17 00:38:45 [scrapy.core.engine] INFO: Spider closed (finished)
</code>```

答:
There is a problem with your settings, it should be:

```<code>ROBOTSTXT_OBEY = False
</code>```

the ROBOTSTXT_OBEY variable needs a boolean, you were setting it with string. You can check your logs, that it was visiting the robots.txt request first.

Body:
Hello and thanks in advance for the help or direction you can bring. This is my scraper:

import scrapy
class RakutenSpider(scrapy.Spider):
    name = "rak"
    allowed_domains = ["rakuten.com"]
    start_urls = ['https://www.rakuten.com/deals?omadtrack=hp_deals_viewmore']
    def parse(self, response):
        for sel in response.xpath('//div[@class="page-bottom"]/div'):
            yield {
                'titles': sel.xpath("//div[@class='slider-prod-title']").extract_first(),
                'prices': sel.xpath("//span[@class='price-bold']").extract_first(),
                'images': sel.xpath("//div[@class='deal-img']/img").extract_first()
            }

And this is part of my settings.py

USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'
CONCURRENT_REQUESTS = 1
DOWNLOAD_DELAY = 5
# Obey robots.txt rules
ROBOTSTXT_OBEY = 'False'

and this is part of the log:

DEBUG: Crawled (403) <GET https://www.rakuten.com/deals?omadtrack=hp_deals_viewmore> (referer: None)

I have tried almost all solutions I found in s/o

log file: This is a new log after installing Firefox driver. Now I get an ERROR: Error downloading https://www.rakuten.com/deals?omadtrack=hp_deals_viewmore>

2017-11-17 00:38:45 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: scrapybot)
2017-11-17 00:38:45 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'deals.spiders', 'CONCURRENT_REQUESTS': 1, 'SPIDER_MODULES': ['deals.spiders'], 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36', 'TELNETCONSOLE_ENABLED': False, 'DOWNLOAD_DELAY': 5}
2017-11-17 00:38:45 [py.warnings] WARNING: :0: UserWarning: You do not have a working installation of the service_identity module: 'No module named cryptography.x509'.  Please install it from <https://pypi.python.org/pypi/service_identity> and make sure all of its dependencies are satisfied.  Without the service_identity module and a recent enough pyOpenSSL to support it, Twisted can perform only rudimentary TLS client hostname verification.  Many valid certificate/hostname mappings may be rejected.

2017-11-17 00:38:45 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.corestats.CoreStats']
2017-11-17 00:38:45 [scrapy.middleware] INFO: Enabled downloader middlewares:
['deals.middlewares.JSMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-11-17 00:38:45 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-11-17 00:38:45 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-11-17 00:38:45 [scrapy.core.engine] INFO: Spider opened
2017-11-17 00:38:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-11-17 00:38:45 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.rakuten.com/deals?omadtrack=hp_deals_viewmore>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/core/downloader/middleware.py", line 37, in process_request
    response = yield method(request=request, spider=spider)
  File "/home/seealldeals/tmp/scrapy/deals/deals/middlewares.py", line 63, in process_request
    driver = webdriver.Firefox()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 144, in __init__
    self.service.start()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/service.py", line 74, in start
    stdout=self.log_file, stderr=self.log_file)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 8] Exec format error
2017-11-17 00:38:45 [scrapy.core.engine] INFO: Closing spider (finished)
2017-11-17 00:38:45 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 1,
 'downloader/exception_type_count/exceptions.OSError': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 11, 17, 5, 38, 45, 328366),
 'log_count/ERROR': 1,
 'log_count/INFO': 7,
 'log_count/WARNING': 1,
 'memusage/max': 33509376,
 'memusage/startup': 33509376,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2017, 11, 17, 5, 38, 45, 112667)}
2017-11-17 00:38:45 [scrapy.core.engine] INFO: Spider closed (finished)

Body:
There is a problem with your settings, it should be:

ROBOTSTXT_OBEY = False

the ROBOTSTXT_OBEY variable needs a boolean, you were setting it with string. You can check your logs, that it was visiting the robots.txt request first.


问:
R: using gsub in a for loop
Low level R user here.

I have 3 population data frames (low.proj, med.proj, high.proj) with the exact same number of rows and columns I'm trying to clean and reshape.

I want to eliminate some extra commas in the country column all three of the frames so I'm trying this loop with gsub:

```<code>for(i in c("low.proj", "med.proj", "high.proj")){
    i$Country <- gsub(",","",i[,"Country"])
    }
</code>```

When I run this I get the error "Error in i[, "Country"] : incorrect number of dimensions"

When I run the code without the loop:

```<code>low.proj$Country <- gsub(",","",low.proj[,"Country"])
</code>```

It works. What causes this error and how do I fix it?

答:
You're indexing the wrong variable:

```<code>i$Country <- gsub(",","",i[,"Country"])
</code>```

i is a string, so i$Country doesn't have any meaning.

Title: R: using gsub in a for loop

Body:
Low level R user here.

I have 3 population data frames (low.proj, med.proj, high.proj) with the exact same number of rows and columns I'm trying to clean and reshape.

I want to eliminate some extra commas in the country column all three of the frames so I'm trying this loop with gsub:

for(i in c("low.proj", "med.proj", "high.proj")){
    i$Country <- gsub(",","",i[,"Country"])
    }

When I run this I get the error "Error in i[, "Country"] : incorrect number of dimensions"

When I run the code without the loop:

low.proj$Country <- gsub(",","",low.proj[,"Country"])

It works. What causes this error and how do I fix it?

Body:
You're indexing the wrong variable:

i$Country <- gsub(",","",i[,"Country"])

i is a string, so i$Country doesn't have any meaning.