答:
The two most obvious tips are:

Create an index on the column.

Don't convert every single row to hexadecimal on every search:

```<code>select hex(pwndpass)
from pwnd
where hex(pwndpass) = '0000000A1D4B746FAA3FD526FF6D5BC8052FDB38';
--    ^^^ This is forcing MySQL to convert every hash stored from binary to hexadecimal
--        so it can determine whether there's a match
</code>```

In fact, you don't even need hexadecimal at all, save for display purposes:

```<code>select id, hex(pwndpass) -- This is fine, will just convert matching rows
from pwnd
where pwndpass = ?
</code>```

... where ? is a placeholder that, in your client language, corresponds to a binary string.

If you need to run the query right in command-line, you can also use an hexadecimal literal:

```<code>select id, hex(pwndpass) -- This is fine, will just convert matching rows
from pwnd
where pwndpass = 0x0000000A1D4B746FAA3FD526FF6D5BC8052FDB38
</code>```

Title: How should I construct a database to store a lot of SHA1 data

Body:
I'm having trouble constructing a database to store a lot of SHA1 data and efficiently return results.

I will admit SQL is not my strongest skill but as an exercise I am trying to use the data from https://haveibeenpwned.com/Passwords which returns results pretty quickly

This is my data:

mysql> describe pwnd;
+----------+------------------+------+-----+---------+----------------+
| Field    | Type             | Null | Key | Default | Extra          |
+----------+------------------+------+-----+---------+----------------+
| id       | int(10) unsigned | NO   | PRI | NULL    | auto_increment |
| pwndpass | binary(20)       | NO   |     | NULL    |                |
+----------+------------------+------+-----+---------+----------------+

mysql> select id, hex(pwndpass) from pwnd order by id desc limit 10;
+-----------+------------------------------------------+
| id        | hex(pwndpass)                            |
+-----------+------------------------------------------+
| 306259512 | FFFFFFFEE791CBAC0F6305CAF0CEE06BBE131160 |
| 306259511 | FFFFFFF8A0382AA9C8D9536EFBA77F261815334D |
| 306259510 | FFFFFFF1A63ACC70BEA924C5DBABEE4B9B18C82D |
| 306259509 | FFFFFFE3C3C05FCB0B211FD0C23404F75E397E8F |
| 306259508 | FFFFFFD691D669D3364161E05538A6E81E80B7A3 |
| 306259507 | FFFFFFCC6BD39537AB7398B59CEC917C66A496EB |
| 306259506 | FFFFFFBFAD0B653BDAC698485C6D105F3C3682B2 |
| 306259505 | FFFFFFBBFC923A29A3B4931B63684CAAE48EAC4F |
| 306259504 | FFFFFFB58E389A0FB9A27D153798956187B1B786 |
| 306259503 | FFFFFFB54953F45EA030FF13619B930C96A9C0E3 |
+-----------+------------------------------------------+
10 rows in set (0.01 sec)

My question relates to quickly finding entries as it currently takes over 6 minutes

mysql> select hex(pwndpass) from pwnd where hex(pwndpass) = '0000000A1D4B746FAA3FD526FF6D5BC8052FDB38';
+------------------------------------------+
| hex(pwndpass)                            |
+------------------------------------------+
| 0000000A1D4B746FAA3FD526FF6D5BC8052FDB38 |
+------------------------------------------+
1 row in set (6 min 31.82 sec)

Do I have the correct data types? I search for storing sha1 data and a Binary(20) field is advised but not sure how to optimising it for searching the data.

My MySQL install is a clean turnkey VM https://www.turnkeylinux.org/mysql I have not adjusted any settings other than giving the VM more disk space

Body:
The two most obvious tips are:

Create an index on the column.

Don't convert every single row to hexadecimal on every search:

select hex(pwndpass)
from pwnd
where hex(pwndpass) = '0000000A1D4B746FAA3FD526FF6D5BC8052FDB38';
--    ^^^ This is forcing MySQL to convert every hash stored from binary to hexadecimal
--        so it can determine whether there's a match

In fact, you don't even need hexadecimal at all, save for display purposes:

select id, hex(pwndpass) -- This is fine, will just convert matching rows
from pwnd
where pwndpass = ?

... where ? is a placeholder that, in your client language, corresponds to a binary string.

If you need to run the query right in command-line, you can also use an hexadecimal literal:

select id, hex(pwndpass) -- This is fine, will just convert matching rows
from pwnd
where pwndpass = 0x0000000A1D4B746FAA3FD526FF6D5BC8052FDB38


来源: stackexchange

问:
Generating URL for Controller action in Spring
How can I dynamically create an URL for a Controller action?

Consider the following case:

```<code>@Controller
@RequestMapping("controller")
public class Controller {
    @RequestMapping("url")
    public String method() {
        return "Whatever"
    }
}
</code>```

What I'd like to do is get the base URL and concat controller/url to it.
For this behavior, Laravel for example provides the URL helper (action() method). Is there something similar in Spring Boot?

答:
if you want to map only Controller , using a property file

application.properties

```<code>server.context-path=/rest
</code>```

If your Controllers are serving Data from a Repository, then Spring Data REST can take out much of the boilerplate & solve your initial problem.

Spring Data REST

pom.xml

```<code><dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-rest</artifactId>
</dependency>
</code>```

You can control the base URL by using a property file.

application.properties

```<code>spring.data.rest.basePath=/rest
</code>```

That's you want /rest concat controller/url

ContentLicense: CC BY-SA 3.0

Body:
How can I dynamically create an URL for a Controller action?

Consider the following case:

@Controller
@RequestMapping("controller")
public class Controller {
    @RequestMapping("url")
    public String method() {
        return "Whatever"
    }
}

What I'd like to do is get the base URL and concat controller/url to it.
For this behavior, Laravel for example provides the URL helper (action() method). Is there something similar in Spring Boot?

Body:
if you want to map only Controller , using a property file

application.properties

server.context-path=/rest

If your Controllers are serving Data from a Repository, then Spring Data REST can take out much of the boilerplate & solve your initial problem.

Spring Data REST

pom.xml

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-rest</artifactId>
</dependency>

You can control the base URL by using a property file.

application.properties

spring.data.rest.basePath=/rest

That's you want /rest concat controller/url


问:
Use of agents to complete side-effects in STM transactions
I'm aware that it is generally bad practice to put functions with side-effects within STM transactions, as they can potentially be retried and called multiple times.

It occurs to me however that you could use agents to ensure that the side effects get executed only after the transaction successfully completes.

e.g.

```<code>(dosync
  // transactional stuff
  (send some-agent #(function-with-side-effects params))
  // more transactional stuff
  )
</code>```

Is this good practice?

What are the pros/cons/pitfalls?

答:
Original:

Seems like that should work to me.  Depending on what your side effects are, you might want to use send-off (for IO-bound ops) instead of send (for cpu-bound ops).  The send/send-off will enqueue the task into one of the internal agent executor pools (there is a fixed size pool for cpu and unbounded size pool for io ops).  Once the task is enqueued, the work is off the dosync's thread so you're disconnected at that point.

You'll need to capture any values you need from within the transaction into the sent function of course.  And you need to deal with that send possibly occurring multiple times due to retries.

Update (see comments):

Agent sends within the ref's transaction are held until the ref transaction successfully completes and are executed once.  So in my answer above, the send will NOT occur multiple times, however it won't occur during the ref transaction which may not be what you want (if you expect to log or do side-effecty stuff).

ContentLicense: CC BY-SA 2.5

Body:
I'm aware that it is generally bad practice to put functions with side-effects within STM transactions, as they can potentially be retried and called multiple times.

It occurs to me however that you could use agents to ensure that the side effects get executed only after the transaction successfully completes.

e.g.

(dosync
  // transactional stuff
  (send some-agent #(function-with-side-effects params))
  // more transactional stuff
  )

Is this good practice?

What are the pros/cons/pitfalls?


问:
Hive 2.1.1 MetaException(message:Version information not found in metastore. )
I'm running Hadoop 2.7.3, MySQL 5.7.17 and Hive 2.1.1 on Ubuntu 16.04.

When I run ./hive, I keep getting the following warning and exception:

```<code>SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/server/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/server/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/home/server/hive/lib/hive-common-2.1.1.jar!/hive-log4j2.properties Async: true
Mon Feb 13 12:01:41 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Mon Feb 13 12:01:41 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Mon Feb 13 12:01:41 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Mon Feb 13 12:01:41 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Mon Feb 13 12:01:42 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Mon Feb 13 12:01:42 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Mon Feb 13 12:01:42 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Mon Feb 13 12:01:42 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:591)
    at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226)
    at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:366)
    at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:310)
    at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:290)
    at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:266)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:558)
    ... 9 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3367)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3406)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3386)
    at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3640)
    at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:236)
    at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:221)
    ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1652)
    ... 23 more
Caused by: MetaException(message:Version information not found in metastore. )
    at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7753)
    at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7731)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
    at com.sun.proxy.$Proxy21.verifySchema(Unknown Source)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:565)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:626)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:416)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6490)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:238)
    at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
    ... 28 more
</code>```

Here is my hive-site.xml

```<code><?xml version="1.0" encoding="UTF-8" standalone="no"?>
 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 <configuration>

     <property>
         <name>javax.jdo.option.ConnectionURL</name>
         <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
     </property>

     <property>
         <name>javax.jdo.option.ConnectionDriverName</name>
         <value>com.mysql.jdbc.Driver</value>
     </property>

     <property>
         <name>javax.jdo.option.ConnectionUserName</name>
         <value>hive</value>
     </property>

     <property>
         <name>javax.jdo.option.ConnectionPassword</name>
         <value>password</value>
     </property>

 </configuration>
</code>```

In order to fix the error, I've tried Hive Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient and Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient. But I am still getting the same exceptions.

I'm newbie to Hadoop and Hive, how could I fix the exception?
Thanks!

答:
I faced similar issue when i was setting up HIVE 2.3.1 for the first time.
I am using postgres database as a metastore.

Then i had to set the below property to false.

```<code>  <property>
    <name>hive.metastore.schema.verification</name>
    <value>false</value>
    <description>
      Enforce metastore schema version consistency.
      True: Verify that version information stored in is compatible with one from Hive jars.  Also disable automatic
            schema migration attempt. Users are required to manually migrate schema after Hive upgrade which ensures
            proper metastore schema migration. (Default)
      False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.
    </description>
  </property>
</code>```

Body:
I'm running Hadoop 2.7.3, MySQL 5.7.17 and Hive 2.1.1 on Ubuntu 16.04.

When I run ./hive, I keep getting the following warning and exception:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/server/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/server/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/home/server/hive/lib/hive-common-2.1.1.jar!/hive-log4j2.properties Async: true
Mon Feb 13 12:01:41 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Mon Feb 13 12:01:41 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Mon Feb 13 12:01:41 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Mon Feb 13 12:01:41 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Mon Feb 13 12:01:42 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Mon Feb 13 12:01:42 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Mon Feb 13 12:01:42 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Mon Feb 13 12:01:42 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:591)
    at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226)
    at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:366)
    at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:310)
    at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:290)
    at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:266)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:558)
    ... 9 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3367)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3406)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3386)
    at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3640)
    at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:236)
    at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:221)
    ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1652)
    ... 23 more
Caused by: MetaException(message:Version information not found in metastore. )
    at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7753)
    at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7731)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
    at com.sun.proxy.$Proxy21.verifySchema(Unknown Source)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:565)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:626)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:416)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6490)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:238)
    at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
    ... 28 more

Here is my hive-site.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 <configuration>

     <property>
         <name>javax.jdo.option.ConnectionURL</name>
         <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
     </property>

     <property>
         <name>javax.jdo.option.ConnectionDriverName</name>
         <value>com.mysql.jdbc.Driver</value>
     </property>

     <property>
         <name>javax.jdo.option.ConnectionUserName</name>
         <value>hive</value>
     </property>

     <property>
         <name>javax.jdo.option.ConnectionPassword</name>
         <value>password</value>
     </property>

 </configuration>

In order to fix the error, I've tried Hive Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient and Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient. But I am still getting the same exceptions.

I'm newbie to Hadoop and Hive, how could I fix the exception?
Thanks!

Body:
I faced similar issue when i was setting up HIVE 2.3.1 for the first time.
I am using postgres database as a metastore.

Then i had to set the below property to false.

  <property>
    <name>hive.metastore.schema.verification</name>
    <value>false</value>
    <description>
      Enforce metastore schema version consistency.
      True: Verify that version information stored in is compatible with one from Hive jars.  Also disable automatic
            schema migration attempt. Users are required to manually migrate schema after Hive upgrade which ensures
            proper metastore schema migration. (Default)
      False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.
    </description>
  </property>

ContentLicense: CC BY-SA 4.0


问:
Adding jar files in each jvm run dont seem to work
I hava a java program which requires me to add an additional jar file when I run it.

What I first do is:

```<code>$ javac -classpath postgresql-42.1.4.jar myFile.java
</code>```

It executes with no errors. But when I then try to do:

```<code>$ java -classpath postgresql-42.1.4.jar myFile
</code>```

It says:

```<code>Error: Could not find or load main class myFile
</code>```

The current folder im in looks like this:

postgresql-42.1.4.jar  myFile.class  myFile.java

There are no errors in myFile.java file (it is setup correctly).

答:
You've overridden the classpath, so it only contains the driver. You need to include your current directory there as well.

java -cp postgresql-42.1.4.jar:. myFile

Also please adhere to Java naming conventions. It should be MyFile, not myFile.

Body:
I hava a java program which requires me to add an additional jar file when I run it.

What I first do is:

$ javac -classpath postgresql-42.1.4.jar myFile.java

It executes with no errors. But when I then try to do:

$ java -classpath postgresql-42.1.4.jar myFile

It says:

Error: Could not find or load main class myFile

The current folder im in looks like this:

postgresql-42.1.4.jar  myFile.class  myFile.java

There are no errors in myFile.java file (it is setup correctly).


问:
CSS Center recaptcha object
Any idea how I can center this recaptcha object at the bottom of the page? I have tried to text-align: center; margin: 0 auto; to all parent objects and it is still stuck on the left.

答:
css:

```<code>#recaptcha-box {
    width: 304px;
    margin: 0 auto;
}
</code>```

html:

```<code><div id="recaptcha-box">
<div id="getdata" class="g-recaptcha" data-sitekey="xxx"></div>
</div>
</code>```

Title: CSS Center recaptcha object

Body:
css:

#recaptcha-box {
    width: 304px;
    margin: 0 auto;
}

html:

<div id="recaptcha-box">
<div id="getdata" class="g-recaptcha" data-sitekey="xxx"></div>
</div>


问:
Securely encrypting / decrypting appsettings.json on server
How do I protect the encryption key used to encrypt sensitive data in my web application's appsettings.json?

I would like to protect sensitive data in my web application's config.

In ASP.NET MVC4 applications we did it like this:

Sensitive data (e.g. password in a connection string) is not added directly to web.config (or web.prod.config etc.), instead a placeholder variable is written.

At deployment time, our deployment service (Octopus) would retrieve sensitive data from its secure storage and overwrite the variable in the web.config.

The deployment process would then encrypt sensitive sections of the web.config using aspnet_regiis.exe.

Now we are using ASP.NET Core the process we are following is a bit different.

appsettings.json holds the placeholder variables for sensitive data,
similar to how web.config worked before.

Deployment process substitutes placeholders with sensitive data from it's secure store as before.

Instead of using aspnet_regiis, I believe I need to:

a) make my own custom tool to encrypt parts of the appsettings.json file.

b) make a custom configuration provider that can decrypt (all / parts of) the appsettings.json

What I don't understand is how to protect the encryption key used for (a) and (b). The old way leveraged the server's machine key to encrypt the file.

The threat I am trying to mitigate is someone gaining access to the appsettings.json on the server and reading sensitive data out of it (e.g. database password etc.)

I am also intrested in alternative ways to mitigate this threat and/or other problems with this approach in general.

答:
If you do not "trust" server - you can't protect "secrets" you store inside this server.

Explanation:

No matter what "secured" (=unsecured) scheme you invent - when application starts is must be able do "decrypt" secrets into some "usable" form.

This means that all "keys" (certs, etc) required to "decrypt" must exist on this server and be accessible to app (otherwise app can't start).

This means that some bad guy who access the server and app also can access all "keys" on it, and "decrypt" your secrets. May be by copying files, may be by decompiling you app, may be dumping you app memory - but it can.

There is no absolute protection.


问:
How can I install lxml 3.5.0 in python 3.6 on windows 7?
How can I install lxml 3.5.0 in python 3.6 on windows 7?
I want to fetch old tweets from twitter using twitter streaming API, and for that I went through https://github.com/Jefferson-Henrique/GetOldTweets-python.
In that it has suggested to install lxml 3.5.0 and pyquery 1.2.0. Though pyquery was installed successfully, installation of lxml showed error.So,from where can I download lxml 3.5.0 for python 3.6?</p>

答:
lxml 3.5.0 is available till python version 3.4 - check this https://pypi.python.org/pypi/lxml/3.5.0

If you want to install lxml 3.8.0 it's available for python 3.6 https://pypi.python.org/pypi/lxml/3.5.0

Download the whl for the OS that you are working on and use pip install.


问:
Node8/Expresss 4 How to get the value of a specific response header property?
I currently can access the response header property 'X-Total-Count' set by json-server, but I don't see how to get the value ...

In chrome-web-tools , is stated as

Response Headers

```<code>HTTP/1.1 304 Not Modified
X-Powered-By: Express
X-Total-Count: 14
Access-Control-Expose-Headers: X-Total-Count, Link
Link: <http://localhost:8080/api/users?_sort=lastName%7Casc&_page=1&_limit=5>; rel="first", <http://localhost:8080/api/users?_sort=lastName%7Casc&_page=2&_limit=5>; rel="next", <http://localhost:8080/api/users?_sort=lastName%7Casc&_page=3&_limit=5>; rel="last"
X-Content-Type-Options: nosniff
ETag: W/"4a2-UH/ZrGuCnz4JcDB1ZNEu7sRXZro"
Date: Thu, 07 Dec 2017 07:42:32 GMT
Connection: keep-alive
</code>```

I can access it in my server.js :

```<code>apiRouter.render = (req, res) => {
  console.log('RES HEADERS: ', res.getHeaders())
  console.log('RES TOTAL: ', res.getHeader('X-Total-Count'))
  console.log('RES LINK: ', res.getHeader('Link'))
  ....
</code>```

And get the log display:

```<code>RES HEADERS:  { 'x-powered-by': 'Express',
  'x-total-count':
   { [Number: 14]
     __wrapped__: { shoppinglists: [Array], users: [Array] },
     __actions__: [ [Object], [Object], [Object] ],
     __chain__: true,
     __index__: 0,
     __values__: undefined },
  'access-control-expose-headers': 'X-Total-Count, Link',
  link: '<http://localhost:8080/api/users?_sort=lastName%7Casc&_page=1&_limit=5>; rel="first", <http://localhost:8080/api/users?_sort=lastName%7Casc&_page=2&_limit=5>; rel="next", <http://localhost:8080/api/users?_sort=lastName%7Casc&_page=3&_limit=5>; rel="last"' }

RES TOTAL:  { [Number: 14]
  __wrapped__:
   { shoppinglists:
      [ [Object],
        ...
        [Object] ],
     users:
      [ [Object],
        ...
        [Object] ] },
  __actions__:
   [ { func: [Function: get], args: [Object], thisArg: [Object] },
     { func: [Function: orderBy], args: [Object], thisArg: [Object] },
     { func: [Function: size], args: {}, thisArg: [Object] } ],
  __chain__: true,
  __index__: 0,
  __values__: undefined }
</code>```

But how can I get the value ? :  14....

thanks for feedback

In json-server , this response header property is set as following :

```<code>// Slice result
if (_end || _limit || _page) {
  res.setHeader('X-Total-Count', chain.size())
  res.setHeader(
    'Access-Control-Expose-Headers',
    `X-Total-Count${_page ? ', Link' : ''}`
  )
}
</code>```

答:
res.getHeader('X-Total-Count') is a FUNCTION...  so need to write :</p>

```<code>     const total = res.getHeader('X-Total-Count')
     console.log('TOTAL: ', total). // => displaying 14
</code>```

Body:
I currently can access the response header property 'X-Total-Count' set by json-server, but I don't see how to get the value ...

In chrome-web-tools , is stated as

Response Headers

HTTP/1.1 304 Not Modified
X-Powered-By: Express
X-Total-Count: 14
Access-Control-Expose-Headers: X-Total-Count, Link
Link: <http://localhost:8080/api/users?_sort=lastName%7Casc&_page=1&_limit=5>; rel="first", <http://localhost:8080/api/users?_sort=lastName%7Casc&_page=2&_limit=5>; rel="next", <http://localhost:8080/api/users?_sort=lastName%7Casc&_page=3&_limit=5>; rel="last"
X-Content-Type-Options: nosniff
ETag: W/"4a2-UH/ZrGuCnz4JcDB1ZNEu7sRXZro"
Date: Thu, 07 Dec 2017 07:42:32 GMT
Connection: keep-alive

I can access it in my server.js :

apiRouter.render = (req, res) => {
  console.log('RES HEADERS: ', res.getHeaders())
  console.log('RES TOTAL: ', res.getHeader('X-Total-Count'))
  console.log('RES LINK: ', res.getHeader('Link'))
  ....

And get the log display:

RES HEADERS:  { 'x-powered-by': 'Express',
  'x-total-count':
   { [Number: 14]
     __wrapped__: { shoppinglists: [Array], users: [Array] },
     __actions__: [ [Object], [Object], [Object] ],
     __chain__: true,
     __index__: 0,
     __values__: undefined },
  'access-control-expose-headers': 'X-Total-Count, Link',
  link: '<http://localhost:8080/api/users?_sort=lastName%7Casc&_page=1&_limit=5>; rel="first", <http://localhost:8080/api/users?_sort=lastName%7Casc&_page=2&_limit=5>; rel="next", <http://localhost:8080/api/users?_sort=lastName%7Casc&_page=3&_limit=5>; rel="last"' }

RES TOTAL:  { [Number: 14]
  __wrapped__:
   { shoppinglists:
      [ [Object],
        ...
        [Object] ],
     users:
      [ [Object],
        ...
        [Object] ] },
  __actions__:
   [ { func: [Function: get], args: [Object], thisArg: [Object] },
     { func: [Function: orderBy], args: [Object], thisArg: [Object] },
     { func: [Function: size], args: {}, thisArg: [Object] } ],
  __chain__: true,
  __index__: 0,
  __values__: undefined }

But how can I get the value ? :  14....

thanks for feedback

In json-server , this response header property is set as following :

// Slice result
if (_end || _limit || _page) {
  res.setHeader('X-Total-Count', chain.size())
  res.setHeader(
    'Access-Control-Expose-Headers',
    `X-Total-Count${_page ? ', Link' : ''}`
  )
}

Body:
res.getHeader('X-Total-Count') is a FUNCTION...  so need to write :

     const total = res.getHeader('X-Total-Count')
     console.log('TOTAL: ', total). // => displaying 14


问:
Shared objects in shared objects with multiprocessing
I face a very persistent problem: I'd like to share complex objects between processes using the process-shareable types proposed in multiprocessing(such as dict, list, etc. which have proxies handled by SyncManager). I even implemented a few other common types such as deque and set. Everything works as long as I store simple values in those objects (floats, integers, etc.).

For instance, if I use the following, this works perfectly as expected:

```<code>import multiprocessing, time

manager = multiprocessing.Manager()
d       = manager.list()
lock    = manager.Lock()

def reader(d, lock):
    for i in range(5):
        with lock:
            print(d)
            sys.stdout.flush()
        time.sleep(0.5)

def writer(d, lock):
    for i in range(5):
        with lock:
            d.append(i)
        time.sleep(0.5)

# Try to read and write the deque:
r = multiprocessing.Process(target=reader, args=(d, lock))
w = multiprocessing.Process(target=writer, args=(d, lock))
r.start()
w.start()
r.join()
w.join()
</code>```

As expected, this output the updated list at each iteration of the reader.

A problem arises when I want to store objects in those objects. Let's say a dictionary of lists. Obviously, if I store simple objects in a process-shareable dict, that won't do it. So I tried the following:

```<code>import multiprocessing, time

manager = multiprocessing.Manager()
d       = manager.dict()
lock    = manager.Lock()

# Add lists:
for i in range(5):
    d[i] = manager.list()

def reader(d, lock):
    for i in range(10):
        with lock:
            print(d)
            sys.stdout.flush()
        time.sleep(1)

def writer(d, lock):
    for i in range(10):
        with lock:
            for j in range(5):
                d[j].append(i)
        time.sleep(1)

# Try to read and write the dict:
r = multiprocessing.Process(target=reader, args=(d, lock))
w = multiprocessing.Process(target=writer, args=(d, lock))
r.start()
w.start()
r.join()
w.join()
</code>```

Unfortunately, with this last piece of code, the updated lists do not get shared between the processes. I thought that using proxies would do it but it doesn't.

The only solution I've found is to commit the change made to a list by reassigning the key in the dictionary explicitly. Obviously, if I do so, I don't need to use proxies on lists anymore as it has a unnecessary cost. So a solution I've found so far to replace

```<code>d[j].append(i)
</code>```

in the writer by:

```<code>l = d[j]     # Somehow creates a local copy
l.append(i)  # Modify the copy
d[j] = l     # Reassign to commit the change
</code>```

While this does the job, this isn't as trivial to write/understand and might be error prone (forgetting to commit stuff, etc.).

Am I missing something here? Is there a way to get shared objects within shared objects?

Just for the sake of completeness, here is the code I have in the end and that does what I what but not in the way I'd like to do it obviously:

```<code>import multiprocessing, time

manager = multiprocessing.Manager()
d       = manager.dict()
lock    = manager.Lock()
# Add lists:
for i in range(5):
    d[i]    = [] # manager.list() doesn't change anything

def reader(d, lock):
    for i in range(10):
        with lock:
            print(d)
            sys.stdout.flush()
        time.sleep(1)

def writer(d, lock):
    for i in range(10):
        with lock:
            for j in range(5):
                l = d[j]
                l.append(i)
                d[j] = l
        time.sleep(1)

# Try to read and write the dict:
r = multiprocessing.Process(target=reader, args=(d, lock))
w = multiprocessing.Process(target=writer, args=(d, lock))
r.start()
w.start()
r.join()
w.join()
</code>```

答:
It is indeed not possible to do this directly, since the proxy object has no way that an internal variable has changed, since it is just pointing to a memory location, and storing the reference to that memory location in stead of the actual values when using custom or more advanced objects.

The documentation has the following to say about this:

Note

Modifications to mutable values or items in dict and list proxies will not be propagated through the manager, because the proxy has no way of knowing when its values or items are modified. To modify such an item, you can re-assign the modified object to the container proxy:

```<code># create a list proxy and append a mutable object (a dictionary)
lproxy = manager.list()
lproxy.append({})
# now mutate the dictionary
d = lproxy[0]
d['a'] = 1
d['b'] = 2
# at this point, the changes to d are not yet synced, but by
# reassigning the dictionary, the proxy is notified of the change
lproxy[0] = d
</code>```

Cfr:
http://www.cmi.ac.in/~madhavan/courses/prog2-2015/docs/python-3.4.2-docs-html/library/multiprocessing.html#multiprocessing.managers.SyncManager.list

Body:
I face a very persistent problem: I'd like to share complex objects between processes using the process-shareable types proposed in multiprocessing(such as dict, list, etc. which have proxies handled by SyncManager). I even implemented a few other common types such as deque and set. Everything works as long as I store simple values in those objects (floats, integers, etc.).

For instance, if I use the following, this works perfectly as expected:

import multiprocessing, time

manager = multiprocessing.Manager()
d       = manager.list()
lock    = manager.Lock()

def reader(d, lock):
    for i in range(5):
        with lock:
            print(d)
            sys.stdout.flush()
        time.sleep(0.5)

def writer(d, lock):
    for i in range(5):
        with lock:
            d.append(i)
        time.sleep(0.5)

# Try to read and write the deque:
r = multiprocessing.Process(target=reader, args=(d, lock))
w = multiprocessing.Process(target=writer, args=(d, lock))
r.start()
w.start()
r.join()
w.join()

As expected, this output the updated list at each iteration of the reader.

A problem arises when I want to store objects in those objects. Let's say a dictionary of lists. Obviously, if I store simple objects in a process-shareable dict, that won't do it. So I tried the following:

import multiprocessing, time

manager = multiprocessing.Manager()
d       = manager.dict()
lock    = manager.Lock()

# Add lists:
for i in range(5):
    d[i] = manager.list()

def reader(d, lock):
    for i in range(10):
        with lock:
            print(d)
            sys.stdout.flush()
        time.sleep(1)

def writer(d, lock):
    for i in range(10):
        with lock:
            for j in range(5):
                d[j].append(i)
        time.sleep(1)

# Try to read and write the dict:
r = multiprocessing.Process(target=reader, args=(d, lock))
w = multiprocessing.Process(target=writer, args=(d, lock))
r.start()
w.start()
r.join()
w.join()

Unfortunately, with this last piece of code, the updated lists do not get shared between the processes. I thought that using proxies would do it but it doesn't.

The only solution I've found is to commit the change made to a list by reassigning the key in the dictionary explicitly. Obviously, if I do so, I don't need to use proxies on lists anymore as it has a unnecessary cost. So a solution I've found so far to replace

d[j].append(i)

in the writer by:

l = d[j]     # Somehow creates a local copy
l.append(i)  # Modify the copy
d[j] = l     # Reassign to commit the change

While this does the job, this isn't as trivial to write/understand and might be error prone (forgetting to commit stuff, etc.).

Am I missing something here? Is there a way to get shared objects within shared objects?

Just for the sake of completeness, here is the code I have in the end and that does what I what but not in the way I'd like to do it obviously:

import multiprocessing, time

manager = multiprocessing.Manager()
d       = manager.dict()
lock    = manager.Lock()
# Add lists:
for i in range(5):
    d[i]    = [] # manager.list() doesn't change anything

def reader(d, lock):
    for i in range(10):
        with lock:
            print(d)
            sys.stdout.flush()
        time.sleep(1)

def writer(d, lock):
    for i in range(10):
        with lock:
            for j in range(5):
                l = d[j]
                l.append(i)
                d[j] = l
        time.sleep(1)

# Try to read and write the dict:
r = multiprocessing.Process(target=reader, args=(d, lock))
w = multiprocessing.Process(target=writer, args=(d, lock))
r.start()
w.start()
r.join()
w.join()

Body:
It is indeed not possible to do this directly, since the proxy object has no way that an internal variable has changed, since it is just pointing to a memory location, and storing the reference to that memory location in stead of the actual values when using custom or more advanced objects.

The documentation has the following to say about this:

Note

Modifications to mutable values or items in dict and list proxies will not be propagated through the manager, because the proxy has no way of knowing when its values or items are modified. To modify such an item, you can re-assign the modified object to the container proxy:

# create a list proxy and append a mutable object (a dictionary)
lproxy = manager.list()
lproxy.append({})
# now mutate the dictionary
d = lproxy[0]
d['a'] = 1
d['b'] = 2
# at this point, the changes to d are not yet synced, but by
# reassigning the dictionary, the proxy is notified of the change
lproxy[0] = d

Cfr:
http://www.cmi.ac.in/~madhavan/courses/prog2-2015/docs/python-3.4.2-docs-html/library/multiprocessing.html#multiprocessing.managers.SyncManager.list


问:
Comparing passwords with crypt() in PHP
I need to get the basics of this function. The php.net documentation states, for the blowfish algorithm, that:

Blowfish hashing with a salt as follows: "$2a$", a two digit cost parameter, "$", and 22 base 64 digits from the alphabet "./0-9A-Za-z". Using characters outside of this range in the salt will cause crypt() to return a zero-length string

So this, by definition, should not work:

```<code>echo crypt('rasmuslerdorf', '$2a$07$usesomadasdsadsadsadasdasdasdsadesillystringforsalt$');
</code>```

However, it spits out:

```<code>$2a$07$usesomadasdsadsadsadaeMTUHlZEItvtV00u0.kb7qhDlC0Kou9e
</code>```

Where it seems that crypt() has cut the salt itself to a length of 22. Could somebody please explain this?

Another aspect of this function I can't get my head around is when they use crypt() to compare passwords. http://php.net/manual/en/function.crypt.php (look at ex. #1). Does this mean that if I use the same salt for all encrypting all my passwords, I have to crypt it first? ie:

```<code>$salt = "usesomadasdsadsadsadae";
$salt_crypt = crypt($salt);

if (crypt($user_input, $salt) == $password) {
   // FAIL WONT WORK
}

if (crypt($user_input, $salt_crypt) == $password) {
   // I HAVE TO DO THIS?
}
</code>```

Thanks for your time

答:
BCrypt uses 128 bits for salt, so 22 bytes Base64, with only two bits of the last byte being used.

The hash is computed using the salt and the password. When you pass the crypted password, the algorithm reads the strength, the salt (ignoring everything beyond it), and the password you gave, and computes the hash, appending it. If you have PostgreSQL and pg_crypto handy, SELECT gen_salt('bf'); will show you what of $salt is being read.

Here's a code sample for salt generation, from my .NET implementation's test-vector-gen.php, alternatively:

```<code>$salt = sprintf('$2a$%02d$%s', [strength goes here],
    strtr(str_replace(
    '=', '', base64_encode(openssl_random_pseudo_bytes(16))
    ),
    'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/',
    './ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789'));
</code>```

There is no reason to use the same salt for all of your passwords. The salt is part of the output anyway so you gain nothing in convenience... though I grant PHP ought to have a built-in gen_salt function.

Title: Comparing passwords with crypt() in PHP

Body:
I need to get the basics of this function. The php.net documentation states, for the blowfish algorithm, that:

Blowfish hashing with a salt as follows: "$2a$", a two digit cost parameter, "$", and 22 base 64 digits from the alphabet "./0-9A-Za-z". Using characters outside of this range in the salt will cause crypt() to return a zero-length string

So this, by definition, should not work:

echo crypt('rasmuslerdorf', '$2a$07$usesomadasdsadsadsadasdasdasdsadesillystringforsalt$');

However, it spits out:

$2a$07$usesomadasdsadsadsadaeMTUHlZEItvtV00u0.kb7qhDlC0Kou9e

Where it seems that crypt() has cut the salt itself to a length of 22. Could somebody please explain this?

Another aspect of this function I can't get my head around is when they use crypt() to compare passwords. http://php.net/manual/en/function.crypt.php (look at ex. #1). Does this mean that if I use the same salt for all encrypting all my passwords, I have to crypt it first? ie:

$salt = "usesomadasdsadsadsadae";
$salt_crypt = crypt($salt);

if (crypt($user_input, $salt) == $password) {
   // FAIL WONT WORK
}

if (crypt($user_input, $salt_crypt) == $password) {
   // I HAVE TO DO THIS?
}

Thanks for your time

Body:
BCrypt uses 128 bits for salt, so 22 bytes Base64, with only two bits of the last byte being used.

The hash is computed using the salt and the password. When you pass the crypted password, the algorithm reads the strength, the salt (ignoring everything beyond it), and the password you gave, and computes the hash, appending it. If you have PostgreSQL and pg_crypto handy, SELECT gen_salt('bf'); will show you what of $salt is being read.

Here's a code sample for salt generation, from my .NET implementation's test-vector-gen.php, alternatively:

$salt = sprintf('$2a$%02d$%s', [strength goes here],
    strtr(str_replace(
    '=', '', base64_encode(openssl_random_pseudo_bytes(16))
    ),
    'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/',
    './ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789'));

There is no reason to use the same salt for all of your passwords. The salt is part of the output anyway so you gain nothing in convenience... though I grant PHP ought to have a built-in gen_salt function.


问:
Modify legend symbol of highchart on export not working
I have a line chart and I customized the legend symbol. In chart, the customization by using useHTML and style of the div as the legend symbol is working but I am struggled on exporting customization.

I tried something like this :

```<code>exporting:{
    allowHTML : true,
    sourceWidth:1024,
    chartOptions: {
        title: {
            style: {
                fontSize: '14px'
            }
        },
        legend : {
            symbolPadding: 0,
            symbolWidth: 0,
            symbolHeight : 0,
            symbolRadius: 0,
            useHTML : true,
            labelFormatter : function () {
                return '<div>' +
                        '<div class="legend-symbol-bar" style="background-color: ' + this.color +';"> </div>' +
                        "<span> " + this.name + " </span>" +
                        '</div>'
            }
        }
    }
}
</code>```

But the legend of exporting is kinda not effective.

Here is the js fiddle.

答:
It seems that Highcharts exporting omits the styles defined in CSS file. They work if you assign them via JS though (load event):

```<code>    chartOptions: {
      chart: {
        events: {
          load: function() {
            this.legend.update({
              symbolWidth: 0,
              labelFormatter: function() {
                this.legendSymbol.css({
                  display: 'none'
                });
                return '<div>' +
                  '<div style="width: 18px; height: 12px; display: inline-block; background-color: ' + this.color + ';"> </div>' +
                  "<span> " + this.name + " </span>" +
                  '</div>'
              }
            });
          }
        }
      },
    (...)
</code>```

Legend symbols can be hidden via JS code too (css function).

Live demo: http://jsfiddle.net/kkulig/efhyabzs/

API references:

https://api.highcharts.com/class-reference/Highcharts.SVGElement#css

https://api.highcharts.com/highcharts/chart.events.load

Body:
I have a line chart and I customized the legend symbol. In chart, the customization by using useHTML and style of the div as the legend symbol is working but I am struggled on exporting customization.

I tried something like this :

exporting:{
    allowHTML : true,
    sourceWidth:1024,
    chartOptions: {
        title: {
            style: {
                fontSize: '14px'
            }
        },
        legend : {
            symbolPadding: 0,
            symbolWidth: 0,
            symbolHeight : 0,
            symbolRadius: 0,
            useHTML : true,
            labelFormatter : function () {
                return '
' +
                        '

' +
                        " " + this.name + " " +
                        '
'
            }
        }
    }
}

But the legend of exporting is kinda not effective.

Here is the js fiddle.

Body:
It seems that Highcharts exporting omits the styles defined in CSS file. They work if you assign them via JS though (load event):

    chartOptions: {
      chart: {
        events: {
          load: function() {
            this.legend.update({
              symbolWidth: 0,
              labelFormatter: function() {
                this.legendSymbol.css({
                  display: 'none'
                });
                return '<div>' +
                  '<div style="width: 18px; height: 12px; display: inline-block; background-color: ' + this.color + ';"> </div>' +
                  "<span> " + this.name + " </span>" +
                  '</div>'
              }
            });
          }
        }
      },
    (...)

Legend symbols can be hidden via JS code too (css function).

Live demo: http://jsfiddle.net/kkulig/efhyabzs/

API references:

https://api.highcharts.com/class-reference/Highcharts.SVGElement#css

https://api.highcharts.com/highcharts/chart.events.load


问:
How to step inside/debugging the code source of web.api 2 and System.Web dll
I wanted to step inside/debugging the web-api 2 (version 5.2.3) code source to understand it.

I've created a simple web api application, and in the WebApi.config, I've set a break point at this line:

```<code> config.Routes.MapHttpRoute(
           name: "ExcludedRoute",
           routeTemplate: "api/{controller}/{action}",
           defaults: new { id = RouteParameter.Optional },
           constraints: new { Controller = "healthcheck", action = "check" }
       );
</code>```

I could step inside the MapHttpRoute method and the other classes, but the debugger couldn't step inside the Route class from the System.Web dll, below the image:

I tried to use this symbol file locations:

I've also used the dotpeek tool for generating the pdb files from the bin folder, by setting to true the "copy local" feature  of the referenced assemblies, but with no success. The path of system.web dll is

C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework.NETFramework\v4.5\System.Web.dll

I'm wondering Why the debugger couldn't step inside the Route class which belongs to the system.Web dll, its version is 4.0 and the version of web.api is the 5.2.3.

In my searching, I've downloaded the code source of the asp.net and i found that the project System.Web.Http.WebHost which contains the HttpWebRoute class.

Any idea?

Thanks!

答:
Assembly System.Web.pdb was in the .NET Framework source code. You need to configure Visual Studio for debugging .NET framework in the Tools -> Options -> Debugging -> General menu.

Reference:

How do I debug .NET 4.6 framework source code in Visual Studio 2017?

Body:
I wanted to step inside/debugging the web-api 2 (version 5.2.3) code source to understand it.

I've created a simple web api application, and in the WebApi.config, I've set a break point at this line:

 config.Routes.MapHttpRoute(
           name: "ExcludedRoute",
           routeTemplate: "api/{controller}/{action}",
           defaults: new { id = RouteParameter.Optional },
           constraints: new { Controller = "healthcheck", action = "check" }
       );

I could step inside the MapHttpRoute method and the other classes, but the debugger couldn't step inside the Route class from the System.Web dll, below the image:

I tried to use this symbol file locations:

I've also used the dotpeek tool for generating the pdb files from the bin folder, by setting to true the "copy local" feature  of the referenced assemblies, but with no success. The path of system.web dll is

C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework.NETFramework\v4.5\System.Web.dll

I'm wondering Why the debugger couldn't step inside the Route class which belongs to the system.Web dll, its version is 4.0 and the version of web.api is the 5.2.3.

In my searching, I've downloaded the code source of the asp.net and i found that the project System.Web.Http.WebHost which contains the HttpWebRoute class.

Any idea?

Thanks!


问:
Evidence that neural network has converged?
I've been training a convolutional network with 7 learnable layers for about 20 hours. What are some general ways to tell if the network has converged or still needs training?

Here's a histogram of the parameters of the first convolutional layer:

Here's a graph of the loss and accuracy of the training and test set:

答:
Obviously while you are getting score rising (train and test) it's meaning you on the right way and you are going to local/global minimum. When you'll see changing of direction score moving (traing still going down, test going up) or both scores go in stagnation, then it's time to stop.

BUT

While you are using accuracy as metric of evaluating you can just get anomal behavior of model. for ex: all result of network output would be number of most valuable classes. Explanation. This problem can be solved by using another metric of evaluating like f1, logloss and so on and you'll see any problems on learning period.

Also for imbalanced data you can use any strategies for avoiding negative effects of imbalance. Like weights in softmax_cross_entropy in tensorflow. Implementation you can find there.


问:
How can I able to pass the value of field Rating in vue js?
My form is

<form id="enquiryBox" method="POST" onSubmit="return false;" data-parsley-validate="true" v-on:submit="handelSubmit($event);">

          <fieldset class="rating">
            <input v-model="rating" type="radio" id="rating" name="rating" v-bind:value="5" >
               <input v-model="rating" type="radio" id="rating" name="rating" v-bind:value="4" >
                <input v-model="rating" type="radio" id="rating" name="rating" v-bind:value="3" >              </fieldset>

          Email Address
          <input type="text" placeholder="Enter Your Email" id="enquiryEmail" name="enquiryEmail" class="form-control required" title="Email" v-model="enquiryEmail" required="required">

          Phone Number
          <input type="text" placeholder="Enter Your Phone Number" id="enquiryPhone" name="enquiryPhone" class="form-control required" title="Phone" v-model="enquiryPhone" required="required">

          Enquiry
          <textarea placeholder="Write your enquiry here" rows="7" id="enquiryDesc" name="enquiryDesc" class="form-control required" title="Desc" v-model="enquiryDesc" required="required"></textarea>

    <button id="btn-submit-enquiry" class="btn whiteButton" type="submit">Post Enquiry</button>
    <button data-dismiss="modal" class="btn darkGrayButton" type="button">Cancel</button>

</form>
```

<p>I am able to get data other than Rating from the above form. How can I able to pass the rating values.. Now I am getting empty values for rating. For, all other fields I am able to pass the data.</p>

<p>My vue js code</p>

<pre class="lang-js prettyprint-override"><code>enquiryBox = new Vue({
  el: "#enquiryBox",
  data: {
    rating: '',
    enquiryPhone: '',
    enquiryEmail: '',
    enquiryDesc: '',
  },
  methods: {
    handelSubmit: function(e) {
      var vm = this;
      data = {};
      data['rating'] = this.rating;
      data['enquiryEmail'] = this.enquiryEmail;
      data['enquiryPhone'] = this.enquiryPhone;
      data['enquiryDesc'] = this.enquiryDesc;
      $.ajax({
        url: 'https://n2s.herokuapp.com/api/post/add_review/',
        data: data,
        type: "POST",
        dataType: 'json',
        success: function(e) {
          if (e.status) {
            alert("Review Success")

          } else {
            alert(" Failed")
          }
        }
      });
      return false;
    }
  },
});
</code>```

So, how can I able to pass the value of rating. If I select first, I need to pass value 5 otherwise value 4 .. But currently I am not able to pass the data of rating.. I am a beginner.. Please help me to achieve the same??

答:
You forgot to add v-model="rating" to your radio inputs. Try it like this:

new Vue({
  el: "#enquiryBox",
  data: {
    rating: '',
    enquiryPhone: '',
    enquiryEmail: '',
    enquiryDesc: '',
  },
  methods: {
    handelSubmit: function(e) {
      var vm = this;
      data = {};
      data['rating'] = this.rating;
      data['enquiryEmail'] = this.enquiryEmail;
      data['enquiryPhone'] = this.enquiryPhone;
      data['enquiryDesc'] = this.enquiryDesc;

      // TODO add your AJAX instead
      console.log(data);

      return false;
    }
  },
});```
<pre class="snippet-code-html lang-html prettyprint-override"><code><script src="https://cdnjs.cloudflare.com/ajax/libs/vue/2.4.2/vue.min.js"></script>
<form id="enquiryBox" method="POST" onSubmit="return false;" data-parsley-validate="true" v-on:submit="handelSubmit($event);">
  <div class="modal-body brbottom-20">
    <div class="clearfix">
      <div class="col-lg-6">
        <div class="form-group required">
          <fieldset class="rating">
            <input type="radio" id="rating" name="rating" v-bind:value="5" v-model="rating" />
            <label class="full" for="star5" title="Awesome"></label>
            <input type="radio" id="rating" name="rating" v-bind:value="4" v-model="rating" />
            <label class="half" for="star4half" title="Pretty good"></label>
            <input type="radio" id="rating" name="rating" v-bind:value="3" v-model="rating" />
            <label class="full" for="star4" title="Pretty good"></label>
          </fieldset>
        </div>
        <div class="form-group required">
          <label>Email Address</label>
          <input type="text" placeholder="Enter Your Email" id="enquiryEmail" name="enquiryEmail" class="form-control required" title="Email" v-model="enquiryEmail" required="required">
        </div>
        <div class="form-group required">
          <label>Phone Number</label>
          <input type="text" placeholder="Enter Your Phone Number" id="enquiryPhone" name="enquiryPhone" class="form-control required" title="Phone" v-model="enquiryPhone" required="required">
        </div>
      </div>
      <div class="col-lg-6">
        <div class="form-group required">
          <label>Enquiry</label>
          <textarea placeholder="Write your enquiry here" rows="7" id="enquiryDesc" name="enquiryDesc" class="form-control required" title="Desc" v-model="enquiryDesc" required="required"></textarea>
        </div>
      </div>
    </div>
  </div>
  <div class="modal-footer center-med-res center-sm-res center-xs-res">
    <button id="btn-submit-enquiry" class="btn whiteButton" type="submit">Post Enquiry</button>
    <button data-dismiss="modal" class="btn darkGrayButton" type="button">Cancel</button>
  </div>
</form></code>```

Body:
My form is

<form id="enquiryBox" method="POST" onSubmit="return false;" data-parsley-validate="true" v-on:submit="handelSubmit($event);">

          <fieldset class="rating">
            <input v-model="rating" type="radio" id="rating" name="rating" v-bind:value="5" >
               <input v-model="rating" type="radio" id="rating" name="rating" v-bind:value="4" >
                <input v-model="rating" type="radio" id="rating" name="rating" v-bind:value="3" >              </fieldset>

          Email Address
          <input type="text" placeholder="Enter Your Email" id="enquiryEmail" name="enquiryEmail" class="form-control required" title="Email" v-model="enquiryEmail" required="required">

          Phone Number
          <input type="text" placeholder="Enter Your Phone Number" id="enquiryPhone" name="enquiryPhone" class="form-control required" title="Phone" v-model="enquiryPhone" required="required">

          Enquiry
          <textarea placeholder="Write your enquiry here" rows="7" id="enquiryDesc" name="enquiryDesc" class="form-control required" title="Desc" v-model="enquiryDesc" required="required"></textarea>

    <button id="btn-submit-enquiry" class="btn whiteButton" type="submit">Post Enquiry</button>
    <button data-dismiss="modal" class="btn darkGrayButton" type="button">Cancel</button>

</form>

I am able to get data other than Rating from the above form. How can I able to pass the rating values.. Now I am getting empty values for rating. For, all other fields I am able to pass the data.

My vue js code

enquiryBox = new Vue({
  el: "#enquiryBox",
  data: {
    rating: '',
    enquiryPhone: '',
    enquiryEmail: '',
    enquiryDesc: '',
  },
  methods: {
    handelSubmit: function(e) {
      var vm = this;
      data = {};
      data['rating'] = this.rating;
      data['enquiryEmail'] = this.enquiryEmail;
      data['enquiryPhone'] = this.enquiryPhone;
      data['enquiryDesc'] = this.enquiryDesc;
      $.ajax({
        url: 'https://n2s.herokuapp.com/api/post/add_review/',
        data: data,
        type: "POST",
        dataType: 'json',
        success: function(e) {
          if (e.status) {
            alert("Review Success")

          } else {
            alert(" Failed")
          }
        }
      });
      return false;
    }
  },
});

So, how can I able to pass the value of rating. If I select first, I need to pass value 5 otherwise value 4 .. But currently I am not able to pass the data of rating.. I am a beginner.. Please help me to achieve the same??

Body:
You forgot to add v-model="rating" to your radio inputs. Try it like this:

new Vue({
  el: "#enquiryBox",
  data: {
    rating: '',
    enquiryPhone: '',
    enquiryEmail: '',
    enquiryDesc: '',
  },
  methods: {
    handelSubmit: function(e) {
      var vm = this;
      data = {};
      data['rating'] = this.rating;
      data['enquiryEmail'] = this.enquiryEmail;
      data['enquiryPhone'] = this.enquiryPhone;
      data['enquiryDesc'] = this.enquiryDesc;

      // TODO add your AJAX instead
      console.log(data);

      return false;
    }
  },
});

<script src="https://cdnjs.cloudflare.com/ajax/libs/vue/2.4.2/vue.min.js"></script>
<form id="enquiryBox" method="POST" onSubmit="return false;" data-parsley-validate="true" v-on:submit="handelSubmit($event);">
  <div class="modal-body brbottom-20">
    <div class="clearfix">
      <div class="col-lg-6">
        <div class="form-group required">
          <fieldset class="rating">
            <input type="radio" id="rating" name="rating" v-bind:value="5" v-model="rating" />
            <label class="full" for="star5" title="Awesome"></label>
            <input type="radio" id="rating" name="rating" v-bind:value="4" v-model="rating" />
            <label class="half" for="star4half" title="Pretty good"></label>
            <input type="radio" id="rating" name="rating" v-bind:value="3" v-model="rating" />
            <label class="full" for="star4" title="Pretty good"></label>
          </fieldset>
        </div>
        <div class="form-group required">
          <label>Email Address</label>
          <input type="text" placeholder="Enter Your Email" id="enquiryEmail" name="enquiryEmail" class="form-control required" title="Email" v-model="enquiryEmail" required="required">
        </div>
        <div class="form-group required">
          <label>Phone Number</label>
          <input type="text" placeholder="Enter Your Phone Number" id="enquiryPhone" name="enquiryPhone" class="form-control required" title="Phone" v-model="enquiryPhone" required="required">
        </div>
      </div>
      <div class="col-lg-6">
        <div class="form-group required">
          <label>Enquiry</label>
          <textarea placeholder="Write your enquiry here" rows="7" id="enquiryDesc" name="enquiryDesc" class="form-control required" title="Desc" v-model="enquiryDesc" required="required"></textarea>
        </div>
      </div>
    </div>
  </div>
  <div class="modal-footer center-med-res center-sm-res center-xs-res">
    <button id="btn-submit-enquiry" class="btn whiteButton" type="submit">Post Enquiry</button>
    <button data-dismiss="modal" class="btn darkGrayButton" type="button">Cancel</button>
  </div>
</form>


问:
How can i convert a DBQuery<T> to an ObjectQuery<T>?
I've got a DBQuery<T> which converts to an IQueryable<T> (this bit works fine). But then I'm trying to convert the IQueryable to an ObjectQuery .. which fails :-

```<code>public void Foo(this IQueryable<T> source)
{
    // ... snip ...

    ObjectQuery<T> objectQuery = source as ObjectQuery<T>;
    if (objectQuery != null)
    {
        // ... do stuff ...
    }
}
</code>```

This used to work before I changed over to Entity-Framework 4 CTP5 Magic Unicorn blah blah blah. Now, it's not working - ie. objectQuery is null.

Now, DBQuery<T> inherits IQueryable<T> .. so I thought this should work.

If i change the code to ..

```<code>var x = (ObjectQuery<T>) source;
</code>```

then the following exception is thrown :-

System.InvalidCastException: Unable to
  cast object of type
  'System.Data.Entity.Infrastructure.DbQuery1[Tests.Models.Order]'
  to type
  'System.Data.Objects.ObjectQuery1[Tests.Models.Order]'.

Any suggestions?

答:
DbQuery<T> contains Include method so you don't need to convert to ObjectQuery. ObjectQuery is not accessible from DbQuery instance - it is wrapped in internal type InternalQuery and conversion operator is not defined.

Btw. when you add using System.Data.Entity and refrence CTP5 you will be able to call Include on IQueryable<T>!

Body:
I've got a DBQuery<T> which converts to an IQueryable<T> (this bit works fine). But then I'm trying to convert the IQueryable to an ObjectQuery .. which fails :-

public void Foo(this IQueryable<T> source)
{
    // ... snip ...

    ObjectQuery<T> objectQuery = source as ObjectQuery<T>;
    if (objectQuery != null)
    {
        // ... do stuff ...
    }
}

This used to work before I changed over to Entity-Framework 4 CTP5 Magic Unicorn blah blah blah. Now, it's not working - ie. objectQuery is null.

Now, DBQuery<T> inherits IQueryable<T> .. so I thought this should work.

If i change the code to ..

var x = (ObjectQuery<T>) source;

then the following exception is thrown :-

System.InvalidCastException: Unable to
  cast object of type
  'System.Data.Entity.Infrastructure.DbQuery1[Tests.Models.Order]'
  to type
  'System.Data.Objects.ObjectQuery1[Tests.Models.Order]'.

Any suggestions?


问:
SQL query flatten data of 1-many relationship table
User table

```<code>id    | emailAddress
1       user@test.com
2       user2@test.com
</code>```

Device table

```<code> uuid   | user_id
 xyz     1
 abc     1
 zzz     1
 a11     2
 g222    2
</code>```

I want to flatten the result of related device table in a column. Here's my attempt using a subquery.

```<code>SELECT id, emailAddress, (
    SELECT uuid FROM device
    WHERE user_id = user.id
) AS devices
FROM user;
</code>```

I want my result output to be like this.

```<code>1, user@test.com  xyz, abc, zzz
2, user2@test.com a112, g222
</code>```

But I also want to clear my objective perhaps for a lateral approach to achieve what I want, Ultimately I want to populate this UserBean in my Java Code. i.e. Get all Users and each user with a list of devices.

```<code>public class UserBean implements Serializable {
    private int id;
    private String emailAddress;
    private ArrayList<DeviceBean> devices = new ArrayList<>();
    // ... some getter setters etc.
}

public class DeviceBean {
    private String uuid;
}
</code>```

答:
If you want to flatten the result you so use MySQL GROUP_CONCAT Function. You should join your tables as shown below:

```<code>SELECT A.id, A.emailAddress, GROUP_CONCAT(DISTINCT B.uuid ORDER BY B.uuid SEPARATOR ',') AS devices
FROM `user` A LEFT JOIN device B
ON A.id=B.user_id
GROUP BY A.id, A.emailAddress;
</code>```

See DEMO on SQL Fiddle.

Body:
User table

id    | emailAddress
1       user@test.com
2       user2@test.com

Device table

 uuid   | user_id
 xyz     1
 abc     1
 zzz     1
 a11     2
 g222    2

I want to flatten the result of related device table in a column. Here's my attempt using a subquery.

SELECT id, emailAddress, (
    SELECT uuid FROM device
    WHERE user_id = user.id
) AS devices
FROM user;

I want my result output to be like this.

1, user@test.com  xyz, abc, zzz
2, user2@test.com a112, g222

But I also want to clear my objective perhaps for a lateral approach to achieve what I want, Ultimately I want to populate this UserBean in my Java Code. i.e. Get all Users and each user with a list of devices.

public class UserBean implements Serializable {
    private int id;
    private String emailAddress;
    private ArrayList<DeviceBean> devices = new ArrayList<>();
    // ... some getter setters etc.
}

public class DeviceBean {
    private String uuid;
}

Body:
If you want to flatten the result you so use MySQL GROUP_CONCAT Function. You should join your tables as shown below:

SELECT A.id, A.emailAddress, GROUP_CONCAT(DISTINCT B.uuid ORDER BY B.uuid SEPARATOR ',') AS devices
FROM `user` A LEFT JOIN device B
ON A.id=B.user_id
GROUP BY A.id, A.emailAddress;

See DEMO on SQL Fiddle.


问:
returning error message to front end Scala Play
I need to send an error status to the front end using Scala play. Ok is to say all is ok but which one is for error? Can this be done from any Scala method I created?
thanks</p>

答:
It can be done from any method, you just need to use correspondent Result. For the general server error it is InternalServerErrorlike

```<code>def index = Action {
  InternalServerError("Some error happens")
}
</code>```

You can find more results here: https://playframework.com/documentation/2.6.7/api/scala/index.html#play.api.mvc.Results

The examples are:

```<code>InsufficientStorage  //507
Locked  //423
</code>```

You can also send your own state as simple as

```<code>new Status(500) //aka InternalServerError
</code>```

Body:
It can be done from any method, you just need to use correspondent Result. For the general server error it is InternalServerErrorlike

def index = Action {
  InternalServerError("Some error happens")
}

You can find more results here: https://playframework.com/documentation/2.6.7/api/scala/index.html#play.api.mvc.Results

The examples are:

InsufficientStorage  //507
Locked  //423

You can also send your own state as simple as

new Status(500) //aka InternalServerError


问:
Give the classes a specific id
I have a question about the class, and give the class a specific id.

For example I've got a html code like this:

```<code><div class="test">text1</div>
<div class="test">text2</div>
<div class="test">text3</div>
<div class="test">text4</div>
<div class="test">text5</div>
</code>```

My question is how can I add an id by each .test class.

I thought something like this in Jquery:

```<code>var i = 1;
$('.test').each(function(){
   $('.test').attr('id','id_'+i+'');
   i++
});
</code>```

This doesn't work I think it's something with .closest() or .next()
to solve this problem.

Regards,

Frank

EDIT:

I solved the problem by myself the answer is:

```<code>    $('.test').each(function(){
        $(this).attr('class','test').attr('id','id_'+i+'');
        i++;
    });
</code>```

答:
Use this jQuery instead:

```<code>var i, $test = $('.test'), tl = $test.length;
for (i = 0; i < tl; i++) {
   $test.eq(i).attr('id','id_'+i+'');
}
</code>```

As i is incremented in the for loop, you select the specific element you want to give the i based id with .eq(i).

Title: Give the classes a specific id

Body:
I have a question about the class, and give the class a specific id.

For example I've got a html code like this:

text1

text2

text3

text4

text5

My question is how can I add an id by each .test class.

I thought something like this in Jquery:

var i = 1;
$('.test').each(function(){
   $('.test').attr('id','id_'+i+'');
   i++
});

This doesn't work I think it's something with .closest() or .next()
to solve this problem.

Regards,

Frank

EDIT:

I solved the problem by myself the answer is:

    $('.test').each(function(){
        $(this).attr('class','test').attr('id','id_'+i+'');
        i++;
    });

Body:
Use this jQuery instead:

var i, $test = $('.test'), tl = $test.length;
for (i = 0; i < tl; i++) {
   $test.eq(i).attr('id','id_'+i+'');
}

As i is incremented in the for loop, you select the specific element you want to give the i based id with .eq(i).


答:
You can use GROUP_CONCAT to concatenate the uuid column values group by id and emailAddress.

Query

```<code>select t1.`id`, t1.`emailAddress`,
group_concat(t2.`uuid`)
from `users` t1
join `devices` t2
on t1.`id` = t2.`user_id`
group by t1.`id`, t1.`emailAddress`;
</code>```

Find a demo here

Title: SQL query flatten data of 1-many relationship table

Body:
You can use GROUP_CONCAT to concatenate the uuid column values group by id and emailAddress.

Query

select t1.`id`, t1.`emailAddress`,
group_concat(t2.`uuid`)
from `users` t1
join `devices` t2
on t1.`id` = t2.`user_id`
group by t1.`id`, t1.`emailAddress`;

Find a demo here


问:
I have a CSV file with header. Want to remove first 5 row of csv but not header? In Python
I have a below CSV file with the header(A,B,C,D):

```<code>A,B,C,D
1,2,3,4
2,1,3,5
6,8,0,9
4,7,9,2
2,5,4,9
1,1,7,3
2,9,5,6
</code>```

I want output after removing top 5 lines but not header:

```<code>A,B,C,D
1,1,7,3
2,9,5,6
</code>```

Below is my Python code snippet, but am not able to add any header retaining code:

with open(filename.csv , 'rb') as infile:
              data_in = infile.readlines()

with open ('temp.csv', 'wb') as outfile:
              outfile.writelines(data_in[5:])

Please help me in this.In my case header is also deleting, but I want to retain header every time.

答:
I'd advise to use csv.DictReader and csv.DictWriter:</p>

```<code>filename = os.path.join(datapath, "input.csv")
with open(filename, 'rb') as infile:
    reader = csv.DictReader(infile)
    data_in = [row for row in reader]
    fieldnames = reader.fieldnames

filename = os.path.join(datapath, "temp.csv")
with open(filename, 'wb') as outfile:
    writer = csv.DictWriter(outfile, fieldnames=fieldnames)
    writer.writeheader()
    writer.writerows(data_in[5:])
</code>```

Body:
I have a below CSV file with the header(A,B,C,D):

A,B,C,D
1,2,3,4
2,1,3,5
6,8,0,9
4,7,9,2
2,5,4,9
1,1,7,3
2,9,5,6

I want output after removing top 5 lines but not header:

A,B,C,D
1,1,7,3
2,9,5,6

Below is my Python code snippet, but am not able to add any header retaining code:

with open(filename.csv , 'rb') as infile:
              data_in = infile.readlines()

with open ('temp.csv', 'wb') as outfile:
              outfile.writelines(data_in[5:])

Please help me in this.In my case header is also deleting, but I want to retain header every time.

Body:
I'd advise to use csv.DictReader and csv.DictWriter:

filename = os.path.join(datapath, "input.csv")
with open(filename, 'rb') as infile:
    reader = csv.DictReader(infile)
    data_in = [row for row in reader]
    fieldnames = reader.fieldnames

filename = os.path.join(datapath, "temp.csv")
with open(filename, 'wb') as outfile:
    writer = csv.DictWriter(outfile, fieldnames=fieldnames)
    writer.writeheader()
    writer.writerows(data_in[5:])


问:
How to fetch data from table as per user input date with formatting using MySQL and PHP
I need one solution. I need to fetch data from table as per date using PHP and MySQL but here the user input is string type. I am explaining my table below.

```<code>id        name             date

 1         aaa         2017-10-23 16:56:34

 2         bbb         2017-10-23 16:56:34

 3         ccc         2017-10-12 13:56:34

 4         ddd         2017-10-23 16:56:34
</code>```

Here My date column datatype is datetime and my user input is string like 2017-10-23. Here I need query to fetch data from table after formatting the user input date.

答:
You can use the mysql's DATE() function to compare only the date and not the time.

eg.:
select * from table where date(date) = '2017-10-23'

Body:
I need one solution. I need to fetch data from table as per date using PHP and MySQL but here the user input is string type. I am explaining my table below.

id        name             date

 1         aaa         2017-10-23 16:56:34

 2         bbb         2017-10-23 16:56:34

 3         ccc         2017-10-12 13:56:34

 4         ddd         2017-10-23 16:56:34

Here My date column datatype is datetime and my user input is string like 2017-10-23. Here I need query to fetch data from table after formatting the user input date.


问:
What are the new testing criteria for the Excel add-in for Office 365?
Seems like new host and platform requirements have changed according to doc published on Dec. 1, 2017 here
https://learn.microsoft.com/en-us/office/dev/add-ins/overview/office-add-in-availability

We were told in a report from the MS Seller Dashboard to use add-in commands and make it compatible with Excel 2013, iOS Mac and the iPad.

However, according to the doc above by Microsoft, add-in commands are not listed for Excel 2013 and iOS.

The add-in availability Chart is gone
https://dev.office.com/add-in-availablity

Therefore, what are the new testing criteria for the Excel add-in for Office 365?

答:
All Excel add-ins submitted to the Office Store must be compatible with Excel 2016, Excel 2013, Excel for Mac and Excel Online, unless they are using APIs unsupported by one of these clients. See Specify Office hosts and API requirements.

Your add-in is only required to support iPad, should you opt in during the submission process. See Develop Office Add-ins for the iPad.

Since February 2017, all Excel add-ins must use add-in commands. In Office 2013, your add-in won't display add-in commands, and uses the value of SourceLocation to run your add-in as a single task pane add-in. See Step 2 of Create add-in commands in your manifest for Excel, Word, and PowerPoint.


问:
How do I conform to a protocol with associatedtype in Swift?
It's such a shame that I really have no idea with this. I have this project that was developed by someone else and it's using TRON library.

My question is pretty straightforward. How do I conform to a protocol with an associatedtype. I know how to write a protocol and conform to it just like with UITableViewDataSource and UITableViewDelegate.

Here's the code of TRON's protocol:

Serializer.swift

```<code>import Foundation
import Alamofire

/// The type in which all data and upload response serializers must conform to in order to serialize a response.
public protocol ErrorHandlingDataResponseSerializerProtocol : DataResponseSerializerProtocol {
    /// The type of serialized object to be created by this `ErrorHandlingDataResponseSerializerProtocol`.
    associatedtype SerializedError

    /// A closure used by response handlers that takes a parsed result, request, response, data and error and returns a serialized error.
    var serializeError: (Alamofire.Result<SerializedObject>?,URLRequest?, HTTPURLResponse?, Data?, Error?) -> APIError<SerializedError> { get }
}
</code>```

And I'm trying to conform to that protocol like so:

```<code> class CustomErrorHandlingSerializer: ErrorHandlingDataResponseSerializerProtocol {

 }
</code>```

I need to conform to that because there is a function here that needs an ErrorHandlingDataResponseSerializerProtocol,

I have tried reading the library's documentation and even their migration guide. It's either they really do not have a straightforward example on their usage OR I just don't understand their documentation.

https://github.com/MLSDev/TRON/blob/master/Docs/4.0%20Migration%20Guide.md

https://github.com/MLSDev/TRON

答:
TRON author here.

Most of the time you don't need to implement this protocol, because responseSerializer is provided by default, if your models are JSONDecodable or Codable. I assume you were using older version of TRON with SwiftyJSON, in that case you would probably need to call request in this way:

```<code>let request : APIRequest<...,...> = tron.swiftyJSON.request("path")
</code>```

If you really want to implement custom response serializer, you would basically need to implement two protocols - ErrorHandlingDataResponseSerializerProtocol and DataResponseSerializerProtocol, which both contain one associatedtype and a variable.

You can use typealias, but it's better to let Swift infer associatedtype from variable signature. You can take a look at Response Serializers doc for reference, i updated it to latest syntax.

Body:
It's such a shame that I really have no idea with this. I have this project that was developed by someone else and it's using TRON library.

My question is pretty straightforward. How do I conform to a protocol with an associatedtype. I know how to write a protocol and conform to it just like with UITableViewDataSource and UITableViewDelegate.

Here's the code of TRON's protocol:

Serializer.swift

import Foundation
import Alamofire

/// The type in which all data and upload response serializers must conform to in order to serialize a response.
public protocol ErrorHandlingDataResponseSerializerProtocol : DataResponseSerializerProtocol {
    /// The type of serialized object to be created by this `ErrorHandlingDataResponseSerializerProtocol`.
    associatedtype SerializedError

    /// A closure used by response handlers that takes a parsed result, request, response, data and error and returns a serialized error.
    var serializeError: (Alamofire.Result<SerializedObject>?,URLRequest?, HTTPURLResponse?, Data?, Error?) -> APIError<SerializedError> { get }
}

And I'm trying to conform to that protocol like so:

 class CustomErrorHandlingSerializer: ErrorHandlingDataResponseSerializerProtocol {

 }

I need to conform to that because there is a function here that needs an ErrorHandlingDataResponseSerializerProtocol,

I have tried reading the library's documentation and even their migration guide. It's either they really do not have a straightforward example on their usage OR I just don't understand their documentation.

https://github.com/MLSDev/TRON/blob/master/Docs/4.0%20Migration%20Guide.md

https://github.com/MLSDev/TRON

Body:
TRON author here.

Most of the time you don't need to implement this protocol, because responseSerializer is provided by default, if your models are JSONDecodable or Codable. I assume you were using older version of TRON with SwiftyJSON, in that case you would probably need to call request in this way:

let request : APIRequest<...,...> = tron.swiftyJSON.request("path")

If you really want to implement custom response serializer, you would basically need to implement two protocols - ErrorHandlingDataResponseSerializerProtocol and DataResponseSerializerProtocol, which both contain one associatedtype and a variable.

You can use typealias, but it's better to let Swift infer associatedtype from variable signature. You can take a look at Response Serializers doc for reference, i updated it to latest syntax.


问:
Sum values grouped by key in list of dict
I have a list of dicts and now I am trying to find the total jobs for each remote identifier.

In this case I am expecting  for the id 64 -> 11 jobs and 68 -> 0 jobs

```<code>[{
    'jobs': {
        'count': 4
    },
    'remote_identifier': {
        'id': '64'
    }
}, {
    'jobs': {
        'count': 0
    },
    'remote_identifier': {
        'id': '68'
    }
}, {
    'jobs': {
        'count': 7
    },
    'remote_identifier': {
        'id': '64'
    }
}]
</code>```

I already tried something like this, but I don't know how to adapt it to my needs, since that only counts the number of occurrences.

```<code>from collections import Counter
print Counter(item['remote_identifier']['id'] for item in items )
</code>```

答:
Pretty straight forward with a defaultdict. (data is your original list.)

```<code>>>> from collections import defaultdict
>>> d = defaultdict(int)
>>>
>>> for d_inner in data:
...     id_ = d_inner['remote_identifier']['id']
...     d[int(id_)] += d_inner['jobs']['count']
...
>>> d
defaultdict(<type 'int'>, {64: 11, 68: 0})
</code>```

Body:
I have a list of dicts and now I am trying to find the total jobs for each remote identifier.

In this case I am expecting  for the id 64 -> 11 jobs and 68 -> 0 jobs

[{
    'jobs': {
        'count': 4
    },
    'remote_identifier': {
        'id': '64'
    }
}, {
    'jobs': {
        'count': 0
    },
    'remote_identifier': {
        'id': '68'
    }
}, {
    'jobs': {
        'count': 7
    },
    'remote_identifier': {
        'id': '64'
    }
}]

I already tried something like this, but I don't know how to adapt it to my needs, since that only counts the number of occurrences.

from collections import Counter
print Counter(item['remote_identifier']['id'] for item in items )

Body:
Pretty straight forward with a defaultdict. (data is your original list.)

>>> from collections import defaultdict
>>> d = defaultdict(int)
>>>
>>> for d_inner in data:
...     id_ = d_inner['remote_identifier']['id']
...     d[int(id_)] += d_inner['jobs']['count']
...
>>> d
defaultdict(<type 'int'>, {64: 11, 68: 0})


问:
Iterate through two dimensional array while changing each element
I want to fill a two dimensional array with certain values. I want to get:

```<code>[
  ['Num00', 'Num01', 'Num02'],
  ['Num10', 'Num11', 'Num12'],
  ['Num20', 'Num21', 'Num22']
]
</code>```

How can I do it?

This is my code:

```<code>game_field = Array.new(3, Array.new(3))
3.times do |i|
  3.times do |j|
    game_field[i][j] = 'Num' + i.to_s + j.to_s
  end
end
game_field
</code>```

答:
Well using Eric and Sawa's answer along with repeated_permutation we have

```<code>[0,1,2].repeated_permutation(2)
       .map { |a, b| "Num#{a}#{b}" }
       .each_slice(3).to_a
</code>```

Body:
I want to fill a two dimensional array with certain values. I want to get:

[
  ['Num00', 'Num01', 'Num02'],
  ['Num10', 'Num11', 'Num12'],
  ['Num20', 'Num21', 'Num22']
]

How can I do it?

This is my code:

game_field = Array.new(3, Array.new(3))
3.times do |i|
  3.times do |j|
    game_field[i][j] = 'Num' + i.to_s + j.to_s
  end
end
game_field

Body:
Well using Eric and Sawa's answer along with repeated_permutation we have

[0,1,2].repeated_permutation(2)
       .map { |a, b| "Num#{a}#{b}" }
       .each_slice(3).to_a


问:
PHP redirect without header() or meta
I'm trying to design a page which does some database actions, then redirects to user back to the page they came from. The problem is that I use a require() function to get the connection to the database, so the headers are already sent. A meta tag is out of the question since I want it to look like all the processes are done from the page they came from. Any tips? Is there a way I can use the require() and the header() or do I have to drop one? Is there an alternative to header()?

答:
```<code>ob_start(); // start output buffering
echo "<html......"; // You can even output some content, it will still work.
.
.
.
.
.
header("Location: mypage.php");
ob_flush(); //flush the buffer
</code>```

In this case, all output is buffered. This means, the headers are processed first, then the output comes to play...

Title: PHP redirect without header() or meta

Body:
ob_start(); // start output buffering
echo "<html......"; // You can even output some content, it will still work.
.
.
.
.
.
header("Location: mypage.php");
ob_flush(); //flush the buffer

In this case, all output is buffered. This means, the headers are processed first, then the output comes to play...