问:
Poor performance with Spark streaming, Kafka and multiple topics
Spark 2.1 + Kafka 0.10 + Spark streaming.

Batch duration is 30s.

I have 13 nodes, 2 brokers and I use 1 core per executor per topic/partition.

LocationStrategy is PreferConsistent.

When consuming 1 topic, no problem executors always process the same topic/partition (tested until 24 partitions).

When I add another topic, some executors used to process topic/partitions change from one batch to another.

When an executor processes the same topic/partition again (for example 3 batches after, so 1:30 after the previous processing) I get a deconnection of my KafkaConsumer because of request timeout (request.timeout.ms parameter) from the broker and then my new fetch query to the Kafka is blocked during 40s (request.timeout.ms parameter again).

```<code>2017-10-09 16:51:30.336 DEBUG    [Executor task launch worker for task 315]:org.apache.spark.internal.Logging$class - Seeking to topic2-7 136136613
2017-10-09 16:51:30.336 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.KafkaConsumer - Seeking to offset 136136613 for partition topic2-7
2017-10-09 16:51:30.337 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.NetworkClient - Disconnecting from node 1005 due to request timeout.
2017-10-09 16:51:30.337 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler - Cancelled FETCH request ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@30ea3352, request=RequestSend(header={api_key=1,api_version=2,correlation_id=25,client_id=consumer-1}, body={replica_id=-1,max_wait_time=500,min_bytes=1,topics=[{topic=topic2,partitions=[{partition=7,fetch_offset=136125064,max_bytes=1048576}]}]}), createdTimeMs=1507557031875, sendTimeMs=1507557031875) with correlation id 25 due to node 1005 being disconnected
2017-10-09 16:51:30.338 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.Fetcher$1 - Fetch failed org.apache.kafka.common.errors.DisconnectException
2017-10-09 16:51:30.341 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater - Initialize connection to node 1006 for sending metadata request
2017-10-09 16:51:30.341 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.NetworkClient - Initiating connection to node 1006 at broker001.domain.loc:9092.
2017-10-09 16:51:30.342 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.common.metrics.Metrics - Added sensor with name node-1006.bytes-sent
2017-10-09 16:51:30.342 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.common.metrics.Metrics - Added sensor with name node-1006.bytes-received
2017-10-09 16:51:30.342 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.common.metrics.Metrics - Added sensor with name node-1006.latency
2017-10-09 16:51:30.343 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.NetworkClient - Completed connection to node 1006
2017-10-09 16:51:30.343 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler - Cancelled FETCH request ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@7d9e82c8, request=RequestSend(header={api_key=1,api_version=2,correlation_id=26,client_id=consumer-1}, body={replica_id=-1,max_wait_time=500,min_bytes=1,topics=[{topic=topic2,partitions=[{partition=7,fetch_offset=136136613,max_bytes=1048576}]}]}), createdTimeMs=1507557090341, sendTimeMs=0) with correlation id 26 due to node 1005 being disconnected
2017-10-09 16:51:30.343 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.Fetcher$1 - Fetch failed org.apache.kafka.common.errors.DisconnectException
2017-10-09 16:51:30.343 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater - Sending metadata request {topics=[topic2]} to node 1006
2017-10-09 16:51:30.344 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler - Cancelled FETCH request ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@4512b012, request=RequestSend(header={api_key=1,api_version=2,correlation_id=27,client_id=consumer-1}, body={replica_id=-1,max_wait_time=500,min_bytes=1,topics=[{topic=topic2,partitions=[{partition=7,fetch_offset=136136613,max_bytes=1048576}]}]}), createdTimeMs=1507557090343, sendTimeMs=0) with correlation id 27 due to node 1005 being disconnected
2017-10-09 16:51:30.344 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.Fetcher$1 - Fetch failed org.apache.kafka.common.errors.DisconnectException
2017-10-09 16:51:30.344 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.Metadata - Updated cluster metadata version 3 to Cluster(nodes = [broker002.domain.loc:9092 (id: 1005 rack: null), broker001.domain.loc:9092 (id: 1006 rack: null)], partitions = [Partition(topic = topic2, partition = 14, leader = 1006, replicas = [1005,1006,], isr = [1006,1005,], Partition(topic = topic2, partition = 13, leader = 1005, replicas = [1005,1006,], isr = [1005,1006,], Partition(topic = topic2, partition = 12, leader = 1006, replicas = [1005,1006,], isr = [1006,1005,], Partition(topic = topic2, partition = 11, leader = 1005, replicas = [1005,1006,], isr = [1005,1006,], Partition(topic = topic2, partition = 10, leader = 1006, replicas = [1005,1006,], isr = [1006,1005,], Partition(topic = topic2, partition = 9, leader = 1005, replicas = [1005,1006,], isr = [1005,1006,], Partition(topic = topic2, partition = 8, leader = 1006, replicas = [1005,1006,], isr = [1006,1005,], Partition(topic = topic2, partition = 7, leader = 1005, replicas = [1005,1006,], isr = [1005,1006,], Partition(topic = topic2, partition = 6, leader = 1006, replicas = [1005,1006,], isr = [1006,1005,], Partition(topic = topic2, partition = 5, leader = 1005, replicas = [1005,1006,], isr = [1005,1006,], Partition(topic = topic2, partition = 4, leader = 1006, replicas = [1005,1006,], isr = [1006,1005,], Partition(topic = topic2, partition = 3, leader = 1005, replicas = [1005,1006,], isr = [1005,1006,], Partition(topic = topic2, partition = 2, leader = 1006, replicas = [1005,1006,], isr = [1006,1005,], Partition(topic = topic2, partition = 1, leader = 1005, replicas = [1005,1006,], isr = [1005,1006,], Partition(topic = topic2, partition = 0, leader = 1006, replicas = [1005,1006,], isr = [1006,1005,]])
2017-10-09 16:51:30.345 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler - Cancelled FETCH request ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@4214186f, request=RequestSend(header={api_key=1,api_version=2,correlation_id=29,client_id=consumer-1}, body={replica_id=-1,max_wait_time=500,min_bytes=1,topics=[{topic=topic2,partitions=[{partition=7,fetch_offset=136136613,max_bytes=1048576}]}]}), createdTimeMs=1507557090344, sendTimeMs=0) with correlation id 29 due to node 1005 being disconnected
2017-10-09 16:51:30.345 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.Fetcher$1 - Fetch failed org.apache.kafka.common.errors.DisconnectException
2017-10-09 16:51:42.942 DEBUG    [LeaseRenewer:hdfs_user@master001.domain.loc:8020]:org.apache.hadoop.hdfs.LeaseRenewer - Lease renewer daemon for [] with renew id 1 executed
2017-10-09 16:52:00.293 DEBUG    [IPC Client (1926664485) connection to master001.domain.loc/10.0.10.1:8020 from hdfs_user]:org.apache.hadoop.ipc.Client$Connection - IPC Client (1926664485) connection to master001.domain.loc/10.0.10.1:8020 from hdfs_user: closed
2017-10-09 16:52:00.293 DEBUG    [IPC Client (1926664485) connection to master001.domain.loc/10.0.10.1:8020 from hdfs_user]:org.apache.hadoop.ipc.Client$Connection - IPC Client (1926664485) connection to master001.domain.loc/10.0.10.1:8020 from hdfs_user: stopped, remaining connections 0
2017-10-09 16:52:10.388 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler - Cancelled FETCH request ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@4b954a27, request=RequestSend(header={api_key=1,api_version=2,correlation_id=30,client_id=consumer-1}, body={replica_id=-1,max_wait_time=500,min_bytes=1,topics=[{topic=topic2,partitions=[{partition=7,fetch_offset=136136613,max_bytes=1048576}]}]}), createdTimeMs=1507557090345, sendTimeMs=0) with correlation id 30 due to node 1005 being disconnected
2017-10-09 16:52:10.389 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.Fetcher$1 - Fetch failed org.apache.kafka.common.errors.DisconnectException
2017-10-09 16:52:10.389 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.NetworkClient - Initiating connection to node 1005 at broker002.domain.loc:9092.
2017-10-09 16:52:10.390 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.NetworkClient - Completed connection to node 1005
2017-10-09 16:52:10.397 DEBUG    [Executor task launch worker for task 315]:org.apache.spark.internal.Logging$class - Polled [topic2-7]  2603
2017-10-09 16:52:10.398 DEBUG    [Executor task launch worker for task 315]:org.apache.spark.internal.Logging$class - Getting local block broadcast_13
2017-10-09 16:52:10.398 DEBUG    [Executor task launch worker for task 315]:org.apache.spark.internal.Logging$class - Level for block broadcast_13 is StorageLevel(disk, memory, deserialized, 1 replicas)
</code>```

What can I do to overcome this kind of problem?
Increase the request.timeout.ms parameter does not seem a good solution to me.

I have seen a parameter to disable cache for Kafka Consumers that could possibly solve this problem but it is available with Spark 2.2 and I can't go to Spark 2.2.

Only solution I can see for now should be to go back to a mono topic processing...

Thank you for your help!

2017/10/18 : Update about this problem

Switches of executors to process a topic/partition were due to data locality problems. For some topic/partition the executor needed to locally process data (locality level PROCESS_LOCAL) was not available so another executor was scheduled to process (locality level RACK_LOCAL), and this executor can be different from a batch to another.

My configuration was 1 core per executor.

I changed my configuration to allow 2 cores per executor and it is OK, all tasks are processed locally.

If a want to process 3 topics I have to change my configuration to 3 cores per executor (topics are uneven, 15 partitions for topic1, 3 for topic2 and 6 for topic 3 for example with 3 topics).

1 topic, 24 topic/partitions, 24 executors, 1 core per executors : OK

2 topics, 24 topic/partitions, 12 executors, 2 cores per executors : OK

3 topics, 24 topic/partitions, 8 executors, 3 cores per executors : OK

4 topics, 24 topic/partitions, 6 executors, 4 cores per executors : OK

6 topics, 24 topic/partitions, 4 executors, 6 cores per executors : KO

With 6 topics I run again into data locality problems.
What can I do to scale my Spark process with the number of topics?

答:
Perform a repartition on your RDD, it will trigger a shuffle and make sure every executor has almost equally local data (in-memory) to process.

For your 6 topics example, try to go with 12 executors, 2 cores per executor and .repartition(48).

Call repartition before any transformation/action you make on the given RDD from the Kafka Consumer.

Note that repartition might have performance implications.

Body:
Spark 2.1 + Kafka 0.10 + Spark streaming.

Batch duration is 30s.

I have 13 nodes, 2 brokers and I use 1 core per executor per topic/partition.

LocationStrategy is PreferConsistent.

When consuming 1 topic, no problem executors always process the same topic/partition (tested until 24 partitions).

When I add another topic, some executors used to process topic/partitions change from one batch to another.

When an executor processes the same topic/partition again (for example 3 batches after, so 1:30 after the previous processing) I get a deconnection of my KafkaConsumer because of request timeout (request.timeout.ms parameter) from the broker and then my new fetch query to the Kafka is blocked during 40s (request.timeout.ms parameter again).

2017-10-09 16:51:30.336 DEBUG    [Executor task launch worker for task 315]:org.apache.spark.internal.Logging$class - Seeking to topic2-7 136136613
2017-10-09 16:51:30.336 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.KafkaConsumer - Seeking to offset 136136613 for partition topic2-7
2017-10-09 16:51:30.337 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.NetworkClient - Disconnecting from node 1005 due to request timeout.
2017-10-09 16:51:30.337 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler - Cancelled FETCH request ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@30ea3352, request=RequestSend(header={api_key=1,api_version=2,correlation_id=25,client_id=consumer-1}, body={replica_id=-1,max_wait_time=500,min_bytes=1,topics=[{topic=topic2,partitions=[{partition=7,fetch_offset=136125064,max_bytes=1048576}]}]}), createdTimeMs=1507557031875, sendTimeMs=1507557031875) with correlation id 25 due to node 1005 being disconnected
2017-10-09 16:51:30.338 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.Fetcher$1 - Fetch failed org.apache.kafka.common.errors.DisconnectException
2017-10-09 16:51:30.341 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater - Initialize connection to node 1006 for sending metadata request
2017-10-09 16:51:30.341 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.NetworkClient - Initiating connection to node 1006 at broker001.domain.loc:9092.
2017-10-09 16:51:30.342 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.common.metrics.Metrics - Added sensor with name node-1006.bytes-sent
2017-10-09 16:51:30.342 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.common.metrics.Metrics - Added sensor with name node-1006.bytes-received
2017-10-09 16:51:30.342 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.common.metrics.Metrics - Added sensor with name node-1006.latency
2017-10-09 16:51:30.343 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.NetworkClient - Completed connection to node 1006
2017-10-09 16:51:30.343 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler - Cancelled FETCH request ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@7d9e82c8, request=RequestSend(header={api_key=1,api_version=2,correlation_id=26,client_id=consumer-1}, body={replica_id=-1,max_wait_time=500,min_bytes=1,topics=[{topic=topic2,partitions=[{partition=7,fetch_offset=136136613,max_bytes=1048576}]}]}), createdTimeMs=1507557090341, sendTimeMs=0) with correlation id 26 due to node 1005 being disconnected
2017-10-09 16:51:30.343 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.Fetcher$1 - Fetch failed org.apache.kafka.common.errors.DisconnectException
2017-10-09 16:51:30.343 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater - Sending metadata request {topics=[topic2]} to node 1006
2017-10-09 16:51:30.344 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler - Cancelled FETCH request ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@4512b012, request=RequestSend(header={api_key=1,api_version=2,correlation_id=27,client_id=consumer-1}, body={replica_id=-1,max_wait_time=500,min_bytes=1,topics=[{topic=topic2,partitions=[{partition=7,fetch_offset=136136613,max_bytes=1048576}]}]}), createdTimeMs=1507557090343, sendTimeMs=0) with correlation id 27 due to node 1005 being disconnected
2017-10-09 16:51:30.344 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.Fetcher$1 - Fetch failed org.apache.kafka.common.errors.DisconnectException
2017-10-09 16:51:30.344 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.Metadata - Updated cluster metadata version 3 to Cluster(nodes = [broker002.domain.loc:9092 (id: 1005 rack: null), broker001.domain.loc:9092 (id: 1006 rack: null)], partitions = [Partition(topic = topic2, partition = 14, leader = 1006, replicas = [1005,1006,], isr = [1006,1005,], Partition(topic = topic2, partition = 13, leader = 1005, replicas = [1005,1006,], isr = [1005,1006,], Partition(topic = topic2, partition = 12, leader = 1006, replicas = [1005,1006,], isr = [1006,1005,], Partition(topic = topic2, partition = 11, leader = 1005, replicas = [1005,1006,], isr = [1005,1006,], Partition(topic = topic2, partition = 10, leader = 1006, replicas = [1005,1006,], isr = [1006,1005,], Partition(topic = topic2, partition = 9, leader = 1005, replicas = [1005,1006,], isr = [1005,1006,], Partition(topic = topic2, partition = 8, leader = 1006, replicas = [1005,1006,], isr = [1006,1005,], Partition(topic = topic2, partition = 7, leader = 1005, replicas = [1005,1006,], isr = [1005,1006,], Partition(topic = topic2, partition = 6, leader = 1006, replicas = [1005,1006,], isr = [1006,1005,], Partition(topic = topic2, partition = 5, leader = 1005, replicas = [1005,1006,], isr = [1005,1006,], Partition(topic = topic2, partition = 4, leader = 1006, replicas = [1005,1006,], isr = [1006,1005,], Partition(topic = topic2, partition = 3, leader = 1005, replicas = [1005,1006,], isr = [1005,1006,], Partition(topic = topic2, partition = 2, leader = 1006, replicas = [1005,1006,], isr = [1006,1005,], Partition(topic = topic2, partition = 1, leader = 1005, replicas = [1005,1006,], isr = [1005,1006,], Partition(topic = topic2, partition = 0, leader = 1006, replicas = [1005,1006,], isr = [1006,1005,]])
2017-10-09 16:51:30.345 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler - Cancelled FETCH request ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@4214186f, request=RequestSend(header={api_key=1,api_version=2,correlation_id=29,client_id=consumer-1}, body={replica_id=-1,max_wait_time=500,min_bytes=1,topics=[{topic=topic2,partitions=[{partition=7,fetch_offset=136136613,max_bytes=1048576}]}]}), createdTimeMs=1507557090344, sendTimeMs=0) with correlation id 29 due to node 1005 being disconnected
2017-10-09 16:51:30.345 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.Fetcher$1 - Fetch failed org.apache.kafka.common.errors.DisconnectException
2017-10-09 16:51:42.942 DEBUG    [LeaseRenewer:hdfs_user@master001.domain.loc:8020]:org.apache.hadoop.hdfs.LeaseRenewer - Lease renewer daemon for [] with renew id 1 executed
2017-10-09 16:52:00.293 DEBUG    [IPC Client (1926664485) connection to master001.domain.loc/10.0.10.1:8020 from hdfs_user]:org.apache.hadoop.ipc.Client$Connection - IPC Client (1926664485) connection to master001.domain.loc/10.0.10.1:8020 from hdfs_user: closed
2017-10-09 16:52:00.293 DEBUG    [IPC Client (1926664485) connection to master001.domain.loc/10.0.10.1:8020 from hdfs_user]:org.apache.hadoop.ipc.Client$Connection - IPC Client (1926664485) connection to master001.domain.loc/10.0.10.1:8020 from hdfs_user: stopped, remaining connections 0
2017-10-09 16:52:10.388 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler - Cancelled FETCH request ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@4b954a27, request=RequestSend(header={api_key=1,api_version=2,correlation_id=30,client_id=consumer-1}, body={replica_id=-1,max_wait_time=500,min_bytes=1,topics=[{topic=topic2,partitions=[{partition=7,fetch_offset=136136613,max_bytes=1048576}]}]}), createdTimeMs=1507557090345, sendTimeMs=0) with correlation id 30 due to node 1005 being disconnected
2017-10-09 16:52:10.389 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.consumer.internals.Fetcher$1 - Fetch failed org.apache.kafka.common.errors.DisconnectException
2017-10-09 16:52:10.389 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.NetworkClient - Initiating connection to node 1005 at broker002.domain.loc:9092.
2017-10-09 16:52:10.390 DEBUG    [Executor task launch worker for task 315]:org.apache.kafka.clients.NetworkClient - Completed connection to node 1005
2017-10-09 16:52:10.397 DEBUG    [Executor task launch worker for task 315]:org.apache.spark.internal.Logging$class - Polled [topic2-7]  2603
2017-10-09 16:52:10.398 DEBUG    [Executor task launch worker for task 315]:org.apache.spark.internal.Logging$class - Getting local block broadcast_13
2017-10-09 16:52:10.398 DEBUG    [Executor task launch worker for task 315]:org.apache.spark.internal.Logging$class - Level for block broadcast_13 is StorageLevel(disk, memory, deserialized, 1 replicas)

What can I do to overcome this kind of problem?
Increase the request.timeout.ms parameter does not seem a good solution to me.

I have seen a parameter to disable cache for Kafka Consumers that could possibly solve this problem but it is available with Spark 2.2 and I can't go to Spark 2.2.

Only solution I can see for now should be to go back to a mono topic processing...

Thank you for your help!

2017/10/18 : Update about this problem

Switches of executors to process a topic/partition were due to data locality problems. For some topic/partition the executor needed to locally process data (locality level PROCESS_LOCAL) was not available so another executor was scheduled to process (locality level RACK_LOCAL), and this executor can be different from a batch to another.

My configuration was 1 core per executor.

I changed my configuration to allow 2 cores per executor and it is OK, all tasks are processed locally.

If a want to process 3 topics I have to change my configuration to 3 cores per executor (topics are uneven, 15 partitions for topic1, 3 for topic2 and 6 for topic 3 for example with 3 topics).

1 topic, 24 topic/partitions, 24 executors, 1 core per executors : OK

2 topics, 24 topic/partitions, 12 executors, 2 cores per executors : OK

3 topics, 24 topic/partitions, 8 executors, 3 cores per executors : OK

4 topics, 24 topic/partitions, 6 executors, 4 cores per executors : OK

6 topics, 24 topic/partitions, 4 executors, 6 cores per executors : KO

With 6 topics I run again into data locality problems.
What can I do to scale my Spark process with the number of topics?


来源: stackexchange

问:
Android Retrofit - Models are GSON arrays from API
When receiving data from an open weather API I am receiving nulls. I am using retroFit2 within android to get the data. Here is the message from the log:

```<code> onResponse: server response Response{protocol=http/1.1, code=200, message=OK,
 onResponse: WeatherResponse{weather=com.weatherapp.models.Weather@197d237}
</code>```

Here is the GSON:

```<code> {
"coord": {
    "lon": -5.93,
    "lat": 54.58
},
"weather": [
    {
        "id": 804,
        "main": "Clouds",
        "description": "overcast clouds",
        "icon": "04d"
    }
],
"base": "stations",
"main": {
    "temp": 293.15,
    "feels_like": 288.01,
    "temp_min": 293.15,
    "temp_max": 293.15,
    "pressure": 1024,
    "humidity": 37
},
"visibility": 10000,
"wind": {
    "speed": 5.7,
    "deg": 160
},
"clouds": {
    "all": 96
},
"dt": 1590770221,
"sys": {
    "type": 1,
    "id": 1376,
    "country": "GB",
    "sunrise": 1590724667,
    "sunset": 1590785095
},
</code>```

MODEL

```<code>@SerializedName("coord")
@Expose
private CoOrdinateModel coOrdinateModel;

@SerializedName("weather")
@Expose
private WeatherModel[] weatherModel;

@SerializedName("main")
@Expose
private MainModel main;

@SerializedName("visibility")
@Expose
private String visibility;

@SerializedName("wind")
@Expose
private WindModel wind;
</code>```

RESPONSE

```<code> public class WeatherResponse {

@SerializedName("coord")
@Expose()
private Weather weather;

public Weather getWeather() {
    return weather;
}

@Override
public String toString() {
    return "WeatherResponse{" +
            "weather=" + weather +
            '}';
}
}
</code>```

API

```<code>  @GET("/data/2.5/weather")
Call<WeatherResponse> getWeather(
        @Query("id") String ID,
        @Query("APPID") String API_KEY
);
</code>```

I ideally need almost all the data from the GSON to parse through into java objects within the android app. I've done this before with simpler GSON, but this one has arrays inside and I'm not sure if my MODEL class is done correctly to accommodate this.

Any help in identifying the root cause of the nulls would be greatly appreciated

答:
Your response model is incorrect -

```<code>@SerializedName("main")
@Expose
private float main[];
</code>```

it should be -

```<code>@SerializedName("main")
@Expose
private MainModel main;
</code>```

and your MainModel would be the POJO class which you are trying to parse -

```<code>public class MainModel {
   @SerializedName("temp")
   public Double temp;
   @SerializedName("feels_like")
   public Double feelsLike;
}
</code>```

Refer same for others as well like wind as they can't be parsed with String class.

ContentLicense: CC BY-SA 4.0

Body:
When receiving data from an open weather API I am receiving nulls. I am using retroFit2 within android to get the data. Here is the message from the log:

 onResponse: server response Response{protocol=http/1.1, code=200, message=OK,
 onResponse: WeatherResponse{weather=com.weatherapp.models.Weather@197d237}

Here is the GSON:

 {
"coord": {
    "lon": -5.93,
    "lat": 54.58
},
"weather": [
    {
        "id": 804,
        "main": "Clouds",
        "description": "overcast clouds",
        "icon": "04d"
    }
],
"base": "stations",
"main": {
    "temp": 293.15,
    "feels_like": 288.01,
    "temp_min": 293.15,
    "temp_max": 293.15,
    "pressure": 1024,
    "humidity": 37
},
"visibility": 10000,
"wind": {
    "speed": 5.7,
    "deg": 160
},
"clouds": {
    "all": 96
},
"dt": 1590770221,
"sys": {
    "type": 1,
    "id": 1376,
    "country": "GB",
    "sunrise": 1590724667,
    "sunset": 1590785095
},

MODEL

@SerializedName("coord")
@Expose
private CoOrdinateModel coOrdinateModel;

@SerializedName("weather")
@Expose
private WeatherModel[] weatherModel;

@SerializedName("main")
@Expose
private MainModel main;

@SerializedName("visibility")
@Expose
private String visibility;

@SerializedName("wind")
@Expose
private WindModel wind;

RESPONSE

 public class WeatherResponse {

@SerializedName("coord")
@Expose()
private Weather weather;

public Weather getWeather() {
    return weather;
}

@Override
public String toString() {
    return "WeatherResponse{" +
            "weather=" + weather +
            '}';
}
}

API

  @GET("/data/2.5/weather")
Call<WeatherResponse> getWeather(
        @Query("id") String ID,
        @Query("APPID") String API_KEY
);

I ideally need almost all the data from the GSON to parse through into java objects within the android app. I've done this before with simpler GSON, but this one has arrays inside and I'm not sure if my MODEL class is done correctly to accommodate this.

Any help in identifying the root cause of the nulls would be greatly appreciated

Body:
Your response model is incorrect -

@SerializedName("main")
@Expose
private float main[];

it should be -

@SerializedName("main")
@Expose
private MainModel main;

and your MainModel would be the POJO class which you are trying to parse -

public class MainModel {
   @SerializedName("temp")
   public Double temp;
   @SerializedName("feels_like")
   public Double feelsLike;
}

Refer same for others as well like wind as they can't be parsed with String class.


问:
JSF JavaScript problem - "jsf" is undefined
my webapplication works very fine with: Opera, FF, Chrome, Safari.
Now i tested it on IE and there are some weird errors.
I debugged my website with IE and it said:

```<code><a id="navi:searchButton" href="#" onclick="jsf.util.chain(this,event,'activateSearchDiv();return false;','mojarra.jsfcljs(document.getElementById(\'navi\'),{\'navi:searchButton\':\'navi:searchButton\'},\'\')');return false">Search</a>
</code>```

>> "jsf" is not defined  ( in jsf.util.chain ).

But I do load these scripts:

```<code><h:body>
    <f:view contentType="text/html">
        <h:outputScript library="js" name="chooseDevice.js" target="head" />
        <h:outputScript library="js" name="navigationScript.js" target="head" />
        <h:outputScript library="js" name="jquery.js" target="head" />
        <h:outputScript name="jsf.js" library="javax.faces" target="head" />
...
</code>```

I think the identified "jsf" is available in jsf.js. Why does IE return this error?

These are just simple commandbuttons/links created with JSF tags:

Did I miss anything? Why does it work on all browsers except IE?
Unfortunately I didnt find any hints on google.

Regards
Johnny

答:
do you have a DOM element with the id set to "jsf"? IE has a strange bug that happens when we try to use global variables which occurs to have the same name as a element's id.

if that's the case, you can predeclare the variable (put a "var varName;" in a script block in the top of the page) or just rename the element's id.

More on that in the link below:
http://www.west-wind.com/weblog/posts/2009/Mar/22/Internet-Explorer-Global-Variable-Blow-ups

ContentLicense: CC BY-SA 2.5

Body:
my webapplication works very fine with: Opera, FF, Chrome, Safari.
Now i tested it on IE and there are some weird errors.
I debugged my website with IE and it said:

Search

>> "jsf" is not defined  ( in jsf.util.chain ).

But I do load these scripts:

<h:body>
    <f:view contentType="text/html">
        <h:outputScript library="js" name="chooseDevice.js" target="head" />
        <h:outputScript library="js" name="navigationScript.js" target="head" />
        <h:outputScript library="js" name="jquery.js" target="head" />
        <h:outputScript name="jsf.js" library="javax.faces" target="head" />
...

I think the identified "jsf" is available in jsf.js. Why does IE return this error?

These are just simple commandbuttons/links created with JSF tags:

Did I miss anything? Why does it work on all browsers except IE?
Unfortunately I didnt find any hints on google.

Regards
Johnny

ContentLicense: CC BY-SA 3.0


问:
JSON Array to UITableView insert
Hello I have one question, I have one json url and this json url is return bellow json content;

```<code>{"ROW”:2,”COLUMN”:[“Name”,”Surname”],"DATA”:{“Name”:[“your_name”,”your_name1”],“Surname”:[“your_surname”,”your_surname1”]}}
</code>```

But for me is needed this json format to convert  Swift UITableView,  im tested SwiftyJson but is not worked correctly for my issue. Im created one array and UITableView and im inserted correctly this array to UITableView, but I don’t know how can i add  this json content to UITableView.
Thank you very much.

My Swift Code:

```<code>let animalArray = ["Firsname","Firsname","Firsname"]

func numberOfSections(in tableView: UITableView) -> Int {
    return 1
}

func tableView(_ tableView:UITableView, numberOfRowsInSection section: Int)-> Int{
    return animalArray.count

}

func tableView(_ tableView:UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell{
    let cell = tableView.dequeueReusableCell(withIdentifier: "MyCell", for:indexPath)
    cell.textLabel!.text = animalArray[indexPath.row]
    return cell
}
</code>```

答:
You should first fetch JSON for example in viewDidLoad(), assign its values to some var or let declared in your view controller and then use it just the same way you just did.
Guide to fetch and decode JSON

Note your struct must be the exact copy of the JSON. Since it is

```<code>{
  "ROW”: 2,
  ”COLUMN”:[“Name”,”Surname”],
  "DATA”: { “Name”: [“your_name”,”your_name1”],
            “Surname”: [“your_surname”,”your_surname1”] }
}
</code>```

your struct must be

```<code>struct JSONStruct: Codable {
    var ROW: Int
    var COLUMN: [String]
    var DATA: Dictionary<String: [String]>
}
</code>```

Title: JSON Array to UITableView insert

Body:
Hello I have one question, I have one json url and this json url is return bellow json content;

{"ROW”:2,”COLUMN”:[“Name”,”Surname”],"DATA”:{“Name”:[“your_name”,”your_name1”],“Surname”:[“your_surname”,”your_surname1”]}}

But for me is needed this json format to convert  Swift UITableView,  im tested SwiftyJson but is not worked correctly for my issue. Im created one array and UITableView and im inserted correctly this array to UITableView, but I don’t know how can i add  this json content to UITableView.
Thank you very much.

My Swift Code:

let animalArray = ["Firsname","Firsname","Firsname"]

func numberOfSections(in tableView: UITableView) -> Int {
    return 1
}

func tableView(_ tableView:UITableView, numberOfRowsInSection section: Int)-> Int{
    return animalArray.count

}

func tableView(_ tableView:UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell{
    let cell = tableView.dequeueReusableCell(withIdentifier: "MyCell", for:indexPath)
    cell.textLabel!.text = animalArray[indexPath.row]
    return cell
}

Body:
You should first fetch JSON for example in viewDidLoad(), assign its values to some var or let declared in your view controller and then use it just the same way you just did.
Guide to fetch and decode JSON

Note your struct must be the exact copy of the JSON. Since it is

{
  "ROW”: 2,
  ”COLUMN”:[“Name”,”Surname”],
  "DATA”: { “Name”: [“your_name”,”your_name1”],
            “Surname”: [“your_surname”,”your_surname1”] }
}

your struct must be

struct JSONStruct: Codable {
    var ROW: Int
    var COLUMN: [String]
    var DATA: Dictionary<String: [String]>
}


问:
How to find the calling viewController in my stack?
I have an application with a two storyboards, one tab bar controller and 6+ viewControllers embedded in their own navigationControllers.

I present CartViewController modally from a barButtonItem in all VC's, and from the cart I present CustomersVC from a tableViewCell. My problem is that when I present Customers from the Cart, I can then again go to the Cart from the CustomersVC and then just keep going in loop..

Assumption

My assumption is that if i present Cart modally from anyVC, then present a navigationController modally from cart, which then presents CustomersVC, an array would like this would exists:

Any viewController

CartViewController

CustomersNavigationController

CustomersViewController

To fix this I'm attemting to remove the "ViewCart" barButtonItem from customersVC IF it is called from the cart. My problem is that I can't manage to traverse the stack to check it out. I've tried out several methods found here at SO, but none works.

I know I can use a delegate and set a variable to show/hide the button, but I'm eager to learn and would also like it to be more robust instead of having to remember to set and reset the var manually.

Any suggestions?

Code

extension UIViewController {
    @objc func viewCart() {
        if Cart.sharedInstance.order.items.count != 0 {

            let storyboard = UIStoryboard(name: "Cart", bundle: nil)
            if let cartVC = storyboard.instantiateViewController(withIdentifier: "Cart") as? CartViewController {
                let cartNavigationController = UINavigationController(rootViewController: cartVC)
                self.present(cartNavigationController, animated: true)
            }
        }
    }
}

Class CartViewController {

  let storyboard = UIStoryboard(name: "Main", bundle: nil)
   if let customersNC = storyboard.instantiateViewController(withIdentifier: "CustomersNC") as? UINavigationController {
   present(customersNC, animated: true)

}

class CustomersTableViewController {

override func viewDidLoad() {

// Tried presentingViewController:

    if let CartVC = presentingViewController as? CartViewController {
        myButton.isEnabled = false
    }

// Tried navigationController?.viewControllers:

        guard let controllersInStack = navigationController?.viewControllers else { return }
        if let CartVC controllersInStack.first(where: { $0 is CartViewController }) as? CartViewController  {
            myButton.isEnabled = false
        }

// Tried an extension I found here https://stackoverflow.com/questions/38583941/get-all-list-of-uiviewcontrollers-in-ios-swift, but it returns nil

extension UIViewController {
   /**
    * Traverses the entire VC hirearchy downwards and collects VC'w that are of speccific PARAM: type
    * - Fixme: ⚠ this can be written more elegantly with flatMap
    * ## Example:
    * let vc: CustomVC? = UIApplication.shared.delegate?.window?.rootViewController?.descendants().first
    */
   func descendants<T>(type: T.Type? = nil) -> [T] {
       var childrenOfType: [T] = []
       self.children.forEach {
           if let child: T = ($0 as? T) {
               childrenOfType.append(child)
           }
           if !$0.children.isEmpty {
               childrenOfType += $0.descendants(type: type)
           }
       }
       return childrenOfType
   }
}

```

答:
Every view controller has either a parent or a presentingViewController (or both), so by asking for these, you can figure out "where you are".

That will usually be sufficient to tell you the situation, especially if you use class types judiciously (for example, you can make your navigation controllers different UINavigationController subclasses just for the sake of knowing where you are later).

If you want a complete conspectus of the view controller chain to where you are, you can trace your way up through the chain recursively, like this:

```<code>func trace(_ vc: UIViewController) {
    print(vc)
    if let parent = vc.parent {
        print("parent:")
        trace(parent)
        return
    }
    if let presenter = vc.presentingViewController {
        print("presenter:")
        trace(presenter)
        return
    }
    print("done")
}
</code>```

That example prints rather than accumulating a list of view controllers along with the nature of the connection between each of them (which is what you really need), but by calling it from the "last" view controller in the chain, you can get a mental picture of what the chain must look like at the point where you want to say "go no deeper".

Here's a more complete example that shows how to accumulate a backward trace into an array:

```<code>class MyVC: UIViewController {
    func makeTrace() {
        super.viewDidAppear(animated)
        let result = self.trace([(.start, self)])
        print(result)
    }
    enum Link: String {
        case parent
        case presenter
        case start
    }
    typealias Chain = [(Link, UIViewController)]
    func trace(_ chain: Chain) -> Chain {
        if let parent = chain.last!.1.parent {
            return trace(chain + [(.parent, parent)])
        }
        if let presenter = chain.last!.1.presentingViewController {
            return trace(chain + [(.presenter, presenter)])
        }
        return chain
    }
}
</code>```

So result will tell you enough to know "where you are".

Body:
Every view controller has either a parent or a presentingViewController (or both), so by asking for these, you can figure out "where you are".

That will usually be sufficient to tell you the situation, especially if you use class types judiciously (for example, you can make your navigation controllers different UINavigationController subclasses just for the sake of knowing where you are later).

If you want a complete conspectus of the view controller chain to where you are, you can trace your way up through the chain recursively, like this:

func trace(_ vc: UIViewController) {
    print(vc)
    if let parent = vc.parent {
        print("parent:")
        trace(parent)
        return
    }
    if let presenter = vc.presentingViewController {
        print("presenter:")
        trace(presenter)
        return
    }
    print("done")
}

That example prints rather than accumulating a list of view controllers along with the nature of the connection between each of them (which is what you really need), but by calling it from the "last" view controller in the chain, you can get a mental picture of what the chain must look like at the point where you want to say "go no deeper".

Here's a more complete example that shows how to accumulate a backward trace into an array:

class MyVC: UIViewController {
    func makeTrace() {
        super.viewDidAppear(animated)
        let result = self.trace([(.start, self)])
        print(result)
    }
    enum Link: String {
        case parent
        case presenter
        case start
    }
    typealias Chain = [(Link, UIViewController)]
    func trace(_ chain: Chain) -> Chain {
        if let parent = chain.last!.1.parent {
            return trace(chain + [(.parent, parent)])
        }
        if let presenter = chain.last!.1.presentingViewController {
            return trace(chain + [(.presenter, presenter)])
        }
        return chain
    }
}

So result will tell you enough to know "where you are".


问:
iPhone: Why does screen go blank after simulate memory warning?
When I run my app on the simulator, then I go Hardware->Simulate Memory Warning it causes my view to go blank (I just see white). Does anyone know how I can make it NOT do that? -- I don't want an actual memory warning to make my screen blank.

Thanks

答: bah, it was mainly because I set my navigation controller incorrectly</p>


问:
Saving Files with SaveFileDialog
I am trying to save Files with SaveFileDialog but they are not appearing on the Directory i have given.
This is what i have tried:

private void button1_Click(object sender, EventArgs e)
{
    SaveFileDialog saveFileDialog1 = new SaveFileDialog();
    saveFileDialog1.InitialDirectory = Application.StartupPath + "\\Scripts\\";
    saveFileDialog1.Title = "Save text Files";
    saveFileDialog1.CheckFileExists = true;
    saveFileDialog1.CheckPathExists = true;
    saveFileDialog1.DefaultExt = "txt";
    saveFileDialog1.Filter = "Text files (*.txt)|*.txt|All files (*.*)|*.*";
    saveFileDialog1.FilterIndex = 2;
    saveFileDialog1.FileName = textBox1.Text;
    saveFileDialog1.RestoreDirectory = true;
    Executor executor = new Executor();
    this.Hide();
}
```

What could be the Problem?

答:
```<code>SaveFileDialog sfd = new SaveFileDialog()
{
    InitialDirectory = Application.StartupPath + "\\Scripts\\",
    Title = "Save Text Files",
    CheckPathExists = true,
    DefaultExt = "txt",
    Filter = "Text files (*.txt)|*.txt|All files (*.*)|*.*",
    FilterIndex = 1,
    RestoreDirectory = true
};

if (sfd.ShowDialog() == DialogResult.OK)
{
    File.WriteAllText(sfd.FileName, "your data here...");
}
</code>```

You don't need to assign filename, if you know filename then no use of savefiledialog just directly use File.WriteAllText method with filename.

No need to check if file exists cause user might have to create new one

If you want to append text then use AppendAllText Method of File class.

Title: Saving Files with SaveFileDialog

Body:
I am trying to save Files with SaveFileDialog but they are not appearing on the Directory i have given.
This is what i have tried:

private void button1_Click(object sender, EventArgs e)
{
    SaveFileDialog saveFileDialog1 = new SaveFileDialog();
    saveFileDialog1.InitialDirectory = Application.StartupPath + "\\Scripts\\";
    saveFileDialog1.Title = "Save text Files";
    saveFileDialog1.CheckFileExists = true;
    saveFileDialog1.CheckPathExists = true;
    saveFileDialog1.DefaultExt = "txt";
    saveFileDialog1.Filter = "Text files (*.txt)|*.txt|All files (*.*)|*.*";
    saveFileDialog1.FilterIndex = 2;
    saveFileDialog1.FileName = textBox1.Text;
    saveFileDialog1.RestoreDirectory = true;
    Executor executor = new Executor();
    this.Hide();
}

What could be the Problem?

Body:
SaveFileDialog sfd = new SaveFileDialog()
{
    InitialDirectory = Application.StartupPath + "\\Scripts\\",
    Title = "Save Text Files",
    CheckPathExists = true,
    DefaultExt = "txt",
    Filter = "Text files (*.txt)|*.txt|All files (*.*)|*.*",
    FilterIndex = 1,
    RestoreDirectory = true
};

if (sfd.ShowDialog() == DialogResult.OK)
{
    File.WriteAllText(sfd.FileName, "your data here...");
}

You don't need to assign filename, if you know filename then no use of savefiledialog just directly use File.WriteAllText method with filename.

No need to check if file exists cause user might have to create new one

If you want to append text then use AppendAllText Method of File class.


问:
Error trying to send image in wordpress api
I'm trying to use wordpress API to send an image to wordpress (I aim to then use it in a post, also created with the API). This is my code:

```<code>data = open(os.path.join(settings.STATIC_ROOT, "test.png"), "rb").read()
    filename="test"
    extension = "png"
    headers = {
        'authorization': 'Basic ' + token.decode('utf-8'),
        'cache-control': 'no-cache',
        'content-disposition': 'attachment; filename=%s' % filename,
        'content-type': 'image/%s' % extension
    }
    r = requests.post(url + '/media', headers=headers, data=data)
</code>```

I get the following error in response :

```<code>{"code":"rest_upload_sideload_error","message":"D\u00e9sol\u00e9, ce type de fichier n\u2019est pas autoris\u00e9 pour des raisons de s\u00e9curit\u00e9.","data":{"status":500}}
</code>```

For those who don't understand french at all, it says that for security reasons, this type of files is not allowed.I don't understand why, png is not a very special format, and my image is a random simple image taken on internet for my tests. I also tried an ico file, same result.
I know that the credential part is working since I used it to test the API for posting a post, which worked.

Thanks for the help

答:
I just figured it out!

I had two things to change in my initial code posted above.
First, as S.D suggested, I had to include the extension in my filenam and second, send the file in the data, not in files.
Corrected working code:

```<code>token = base64.b64encode(creds.encode())
    headers = {
        'Authorization': 'Basic ' + token.decode('utf-8'),
        'Content-disposition': 'attachment; filename=test.png',
        'Content-type': 'image/png'
    }
    data = open(os.path.join(settings.STATIC_ROOT, "test.png"), "rb").read();
    r = requests.post(url + '/media', headers=headers, data=data)
</code>```

Hope this may help others, I really struggled finding the last part of the problem.

Body:
I'm trying to use wordpress API to send an image to wordpress (I aim to then use it in a post, also created with the API). This is my code:

data = open(os.path.join(settings.STATIC_ROOT, "test.png"), "rb").read()
    filename="test"
    extension = "png"
    headers = {
        'authorization': 'Basic ' + token.decode('utf-8'),
        'cache-control': 'no-cache',
        'content-disposition': 'attachment; filename=%s' % filename,
        'content-type': 'image/%s' % extension
    }
    r = requests.post(url + '/media', headers=headers, data=data)

I get the following error in response :

{"code":"rest_upload_sideload_error","message":"D\u00e9sol\u00e9, ce type de fichier n\u2019est pas autoris\u00e9 pour des raisons de s\u00e9curit\u00e9.","data":{"status":500}}

For those who don't understand french at all, it says that for security reasons, this type of files is not allowed.I don't understand why, png is not a very special format, and my image is a random simple image taken on internet for my tests. I also tried an ico file, same result.
I know that the credential part is working since I used it to test the API for posting a post, which worked.

Thanks for the help

Body:
I just figured it out!

I had two things to change in my initial code posted above.
First, as S.D suggested, I had to include the extension in my filenam and second, send the file in the data, not in files.
Corrected working code:

token = base64.b64encode(creds.encode())
    headers = {
        'Authorization': 'Basic ' + token.decode('utf-8'),
        'Content-disposition': 'attachment; filename=test.png',
        'Content-type': 'image/png'
    }
    data = open(os.path.join(settings.STATIC_ROOT, "test.png"), "rb").read();
    r = requests.post(url + '/media', headers=headers, data=data)

Hope this may help others, I really struggled finding the last part of the problem.


问:
Gitkraken doesn't show branches and commits
When I try to open a repository in a xampp server with Gitkraken I can't view branches or commits, just the message "Displaying 2000 commits. Adjust this setting in Preferences". I tried to reinstall Gitkraken, reopen the repository and reboot xampp/my computer but it didn't work. My computer is a MacBook Pro (Retina, 13-inch, Early 2015) with macOS Mojave 10.14.6.

答: I got the same issue, I already paid a one-year membership. So frustrated. I have to find another application instead of the gitkraken. I find Git Tower and Git Ahead. </p>


问:
How to fix cut of UILabel in Swift
I have a UILabel inside a UIStackView in a xib file. the label is cut off. What I am trying to accomplish is have a small UIView circle that will be Green when a user is online and red when they are offline. I embedded the UIView and UILabel in a stack view and set the UIView height and width to be 5. However it is also changing the label, but I want the label to be sized to its text. Any suggestions on how I can fix this.

答:
Tip: it helps to change the names of your elements... Tough to refer to "Stack View" and "View" and "Label" when you have multiple of each.

What you need to do is...

For your Horizontal Stack view that is holding your 5 x 5 sized view, set the Alignment to Center.

This will allow the stack view to grow tall enough to fit the label, without stretching the height of the 5 x 5 view.

Title: How to fix cut of UILabel in Swift


问:
How to split strings separated by many semicolons in R?
My desire is to know the length of a certain text separated by ; which comes after any number. In the text named txt below, I don't want to consider the first two semicolons. To get the length, the ; comes after 6, 5 should be considered. I mean the code should lookbehind some number(s) to consider the appropriate ;.

```<code>library(stringr)
txt <- "A;B; dd (2020) text  pp. 805-806; Mining; exercise (1999), ee, p-123-125; F;G;H text, (2017) kk"
</code>```

lenghths(strsplit(txt,";")) gives me 8. In my case, however, it should be 3. Any help is highly appreciated.

答:
We can use a regex lookaround to match a ; that succeeds a digit ((?<=[0-9])) and get the lengths

```<code>lengths(strsplit(txt, "(?<=[5-6]);", perl = TRUE))
#[1] 3
</code>```

Or using str_count

```<code>library(stringr)
str_count(txt, '[5-6];') + 1
#[1] 3
</code>```

Body:
My desire is to know the length of a certain text separated by ; which comes after any number. In the text named txt below, I don't want to consider the first two semicolons. To get the length, the ; comes after 6, 5 should be considered. I mean the code should lookbehind some number(s) to consider the appropriate ;.

library(stringr)
txt <- "A;B; dd (2020) text  pp. 805-806; Mining; exercise (1999), ee, p-123-125; F;G;H text, (2017) kk"

lenghths(strsplit(txt,";")) gives me 8. In my case, however, it should be 3. Any help is highly appreciated.

Body:
We can use a regex lookaround to match a ; that succeeds a digit ((?<=[0-9])) and get the lengths

lengths(strsplit(txt, "(?<=[5-6]);", perl = TRUE))
#[1] 3

Or using str_count

library(stringr)
str_count(txt, '[5-6];') + 1
#[1] 3


问:
Loopback 4 use existing express server with loopback
I have two server Loopback4 and express. Loopback's controllers are protected with loopback's jwt authentication and authorization.

Now I want to mount express server on loopback's server and use the same loopback's authentication for express server routes.

How to achieve this?
Thank you in advance

答: Loopback has a function mountExpressRouter that can mount the existing express server on loopback. here are docs https://loopback.io/doc/en/lb4/Routes.html#mounting-an-express-router


问:
Run a function and Post request Laravel
Scource Code
https://github.com/jslack2537/PHP-Calculator

Im having issues running my business logic for my calculator app at the same time as saving the results into my database they work independently just not at the same time. Im not sure if I should be combining my two controllers or what here. Any advice on the best way to combine them or make the process work as is.

```<code>    <?php

namespace App\Http\Controllers;

use App\Result;
use Illuminate\Http\Request;

class ResultController extends Controller
{
    public function postResultCreate(Request $request)
    {   $result = 2;
         $post = new Result([
             'first_number' => $request->input('first'),
             'second_number' => $request->input('second'),
             'operator' => $request->input('operator'),
             'sum' => $result
         ]);
        $post->save();
        return redirect('/')->with('info', 'Answer: ' .$result);
    }
}
</code>```

My Second controller has the basic math logic

```<code><?php

namespace App\Http\Controllers;

use Illuminate\Http\Request;

class CalculatorController extends Controller
{
    public function calculatefunc(Request $request)
    {
        $operator = $request->input('operator');
        $first_number = $request->input('first');
        $second_number = $request->input('second');
        $result = 0;

        if ($operator == "plus")
        {
            $result = $first_number + $second_number;
        }
        elseif($operator == "minus")
        {
            $result = $first_number - $second_number;
        }
        elseif($operator == "multiply")
        {
            $result = $first_number * $second_number;
        }
        elseif($operator == "divide")
        {
            $result = $first_number / $second_number;
        }
        else{
            $result= 0;
        }
        return redirect('/')->with('info', 'Answer: ' .$result);
    }
}
</code>```

答:
I think you should combine this two classes.

Try this:

```<code>class CalculatorController extends Controller
{
    public function calculatefunc(Request $request)
    {
        $operator = $request->input('operator');
        $first_number = $request->input('first');
        $second_number = $request->input('second');
        $result = $this->calculateByOperator($operator, $first_number, $second_number);

        return redirect('/')->with('info', 'Answer: ' . $result);
    }

    private function calculateByOperator($operator, $first_number, $second_number)
    {
        if ($operator == "plus") {
            $result = $first_number + $second_number;
        } elseif ($operator == "minus") {
            $result = $first_number - $second_number;
        } elseif ($operator == "multiply") {
            $result = $first_number * $second_number;
        } elseif ($operator == "divide") {
            $result = $first_number / $second_number;
        } else {
            $result = 0;
        }

        return $result;
    }

    public function postResultCreate(Request $request)
    {
        $first_number = $request->input('first');
        $second_number = $request->input('second');
        $operator = $request->input('operator');
        $result = $this->calculateByOperator($operator, $first_number, $second_number);
        $post = new Result([
            'first_number' => $first_number,
            'second_number' => $second_number,
            'operator' => $operator,
            'result' => $result
            // it should be result not sum because it will depend on your operator. you can also use operator here
        ]);
        $post->save();
        return redirect('/')->with('info', 'Answer: ' . $result);
    }
}
</code>```

Title: Run a function and Post request Laravel

Body:
Scource Code
https://github.com/jslack2537/PHP-Calculator

Im having issues running my business logic for my calculator app at the same time as saving the results into my database they work independently just not at the same time. Im not sure if I should be combining my two controllers or what here. Any advice on the best way to combine them or make the process work as is.

    <?php

namespace App\Http\Controllers;

use App\Result;
use Illuminate\Http\Request;

class ResultController extends Controller
{
    public function postResultCreate(Request $request)
    {   $result = 2;
         $post = new Result([
             'first_number' => $request->input('first'),
             'second_number' => $request->input('second'),
             'operator' => $request->input('operator'),
             'sum' => $result
         ]);
        $post->save();
        return redirect('/')->with('info', 'Answer: ' .$result);
    }
}

My Second controller has the basic math logic

<?php

namespace App\Http\Controllers;

use Illuminate\Http\Request;

class CalculatorController extends Controller
{
    public function calculatefunc(Request $request)
    {
        $operator = $request->input('operator');
        $first_number = $request->input('first');
        $second_number = $request->input('second');
        $result = 0;

        if ($operator == "plus")
        {
            $result = $first_number + $second_number;
        }
        elseif($operator == "minus")
        {
            $result = $first_number - $second_number;
        }
        elseif($operator == "multiply")
        {
            $result = $first_number * $second_number;
        }
        elseif($operator == "divide")
        {
            $result = $first_number / $second_number;
        }
        else{
            $result= 0;
        }
        return redirect('/')->with('info', 'Answer: ' .$result);
    }
}

Body:
I think you should combine this two classes.

Try this:

class CalculatorController extends Controller
{
    public function calculatefunc(Request $request)
    {
        $operator = $request->input('operator');
        $first_number = $request->input('first');
        $second_number = $request->input('second');
        $result = $this->calculateByOperator($operator, $first_number, $second_number);

        return redirect('/')->with('info', 'Answer: ' . $result);
    }

    private function calculateByOperator($operator, $first_number, $second_number)
    {
        if ($operator == "plus") {
            $result = $first_number + $second_number;
        } elseif ($operator == "minus") {
            $result = $first_number - $second_number;
        } elseif ($operator == "multiply") {
            $result = $first_number * $second_number;
        } elseif ($operator == "divide") {
            $result = $first_number / $second_number;
        } else {
            $result = 0;
        }

        return $result;
    }

    public function postResultCreate(Request $request)
    {
        $first_number = $request->input('first');
        $second_number = $request->input('second');
        $operator = $request->input('operator');
        $result = $this->calculateByOperator($operator, $first_number, $second_number);
        $post = new Result([
            'first_number' => $first_number,
            'second_number' => $second_number,
            'operator' => $operator,
            'result' => $result
            // it should be result not sum because it will depend on your operator. you can also use operator here
        ]);
        $post->save();
        return redirect('/')->with('info', 'Answer: ' . $result);
    }
}


问:
What TypeScript identifiers and constructs (apart from enums) 'survive' the compilation and can be accessed in JavaScript at runtime?
TypeScript compiles to JavaScript stripping nearly all of the TypeScript syntax and identifiers from the resulting JavaScript code. If given piece of TypeScript syntax is not a valid JavaScript code then you won't see a trace of it in the compiled output.

One exception that I know of is enum.

TypeScript code (invalid in JS)

```<code>enum Alph {
    a, b, c = "www"
}
</code>```

actually shows in compiled JavaScript output as

```<code>var Alph;
(function (Alph) {
    Alph[Alph["a"] = 0] = "a";
    Alph[Alph["b"] = 1] = "b";
    Alph["c"] = "www";
})(Alph || (Alph = {}));
</code>```

Question

What TypeScript syntax constructs, other than enum, that are not JS constructs, compile to some JavaScript code and not just disappear? I'm especially interested in constructs that expose TypeScript identifiers in JS, like enum does.

Clarification

Here's an example of what I have in mind when I say about TypeScript specific constructs and identifiers 'disappearing' from the output.

TypeScript

```<code>interface Inter {
    a: number;
}

class C implements Inter {
    a: number = 1;
}

var b: number = 1;
</code>```

JavaScript compiler output

```<code>class C {
    constructor() {
        this.a = 1;
    }
}

var b = 1;
</code>```

You can see that there's no trace of interface, implement, Inter or number. But the variable declaration and assignment is unchanged and class is passed to JS barely changed.

Default value was moved to the constructor so properties default values might be a minor example of what I'm looking for. Constructs that can't be just copied or removed but have to undergo some transformation to make sense in JS.

答:
"What are TS constructs other than enum that have to be transformed into something at runtime?"

The short answer is all of them, it's just a matter of how much. Keep in mind that Typescript is both a type system and a compiler. You code is being reconstructed into an abstract source tree, then re-emitted according the the logic of the compiler. So everything is always getting transformed, it's just that most of the time the output and the original code are pretty similar.

So I'm going to answer a more defined question that I think you're asking:

What Typescript keywords do not exist in JS, yet emit JS code when compiled.

There are a few things (like private instance variables #foo, decorators @f(), or JSX). These are items that are in proposal stages of JS but not yet widely adopted, or are defined by other standards or transpilers, that have become first class citizen within the language of typescript. For those cases typescript outputs an equivalent implementation that should work in all modern JS interpreters that generates a lot of code the programmer isn't meant to utilize directly.

But I'm assuming you don't care about those cases too much either since the generated code could be removed if all interpreters supported these constructs the same way.

So that leaves:

enum

namespace (Though I think usage of namespace is discouraged for new typescript code these days, and the supporting code is pretty minimal)

And yeah. Pretty sure that's all of them.

Body:
TypeScript compiles to JavaScript stripping nearly all of the TypeScript syntax and identifiers from the resulting JavaScript code. If given piece of TypeScript syntax is not a valid JavaScript code then you won't see a trace of it in the compiled output.

One exception that I know of is enum.

TypeScript code (invalid in JS)

enum Alph {
    a, b, c = "www"
}

actually shows in compiled JavaScript output as

var Alph;
(function (Alph) {
    Alph[Alph["a"] = 0] = "a";
    Alph[Alph["b"] = 1] = "b";
    Alph["c"] = "www";
})(Alph || (Alph = {}));

Question

What TypeScript syntax constructs, other than enum, that are not JS constructs, compile to some JavaScript code and not just disappear? I'm especially interested in constructs that expose TypeScript identifiers in JS, like enum does.

Clarification

Here's an example of what I have in mind when I say about TypeScript specific constructs and identifiers 'disappearing' from the output.

TypeScript

interface Inter {
    a: number;
}

class C implements Inter {
    a: number = 1;
}

var b: number = 1;

JavaScript compiler output

class C {
    constructor() {
        this.a = 1;
    }
}

var b = 1;

You can see that there's no trace of interface, implement, Inter or number. But the variable declaration and assignment is unchanged and class is passed to JS barely changed.

Default value was moved to the constructor so properties default values might be a minor example of what I'm looking for. Constructs that can't be just copied or removed but have to undergo some transformation to make sense in JS.


问:
Generate UniqueIdentifier GUID in SSIS
I am using a GUID for a batch identifier in SSIS.  My final output goes to SQL Server.

I know how I can generate one using Select NewId() MyUniqueIdentifier in Sql Server - I can generate one using a query and an Execute SQL task.

I am however looking to do this within a SSIS package if possible without SQL Server available.

Can I generate a GUID within SSIS?

答:
I had a similar problem. To fix it, I created an SSIS "Composant Script" in which I created a "guid" output. The script VS C# code was the following :

```<code>Row.guid = Guid.NewGuid();
</code>```

Finally, I routed the output as a derived column into my database "OLE DB Destination" to generate a guid for every new entry.

Title: Generate UniqueIdentifier GUID in SSIS

Body:
I had a similar problem. To fix it, I created an SSIS "Composant Script" in which I created a "guid" output. The script VS C# code was the following :

Row.guid = Guid.NewGuid();

Finally, I routed the output as a derived column into my database "OLE DB Destination" to generate a guid for every new entry.


问:
Fix "This file is not commonly downloaded" warning - Firefox only
I run a website (www.pixelscrapper.com) that serves file downloads of images, and zipped collections of images (which are zip files containing multiple imags, psds, vectors, etc.) These files are hosted on Amazon S3, and served via download urls generated by aws sdk for php (v1).

Just recently, users trying to download our zip files using Firefox have started getting "This file is not commonly downloaded" warnings (after the download finishes), which forces them to override the warning before accessing the file via the Firefox download manager. Naturally, this kind of warning causes concern for our users.

This warning shows up IN FIREFOX ONLY--Chrome, Edge, and Internet Explorer show no warnings when downloading and opening our zip files. The warning also only seems to show up for (surprise, surprise) files that have been more recently added to the site, and have relatively few total downloads--but many of our files never receive large numbers of downloads, so this warning has the potential to plague many of our files indefinitely.

My question is: is there anything I can do to prevent this warning? By adjusting headers, signing files in some way, etc.? (From what I understand, Chrome and Edge also have "uncommon file" warning, but they don't seem to be concerned with our files--why is this warning only firing in Firefox?) I've searched around on Stack Overflow and elsewhere, but most of the questions I've seen about "uncommon download" warnings are targeted at Chrome or Internet Explorer, and I can't seem to find any Firefox-specific information about this warning.

Here's a sample file download url (generated by aws sdk) that is causing warnings:

```<code>https://pixelscrapper-user-content.s3.amazonaws.com/template-attachment/user-2/node-13574/paper-037-template-polka-dots.zip?response-content-disposition=attachment%3B%20filename%3D%22ps_marisa-lerin_13574_paper-037-template-polka-dots_cu.zip%22&AWSAccessKeyId=AKIAIWM7MZMHRPA6FHEA&Expires=1495386939&Signature=HDmwRFPX81CIVrQgu1BkEyR9iRQ%3D
</code>```

Here's an inspection of headers in Firefox:

UPDATE:

The issue here is not the nasty-looking url generated by the aws sdk: I checked downloading the same zip file (containing one jpg, one psd) from the following "clean" url, and it still gives the warning: http://pixelscrapper-misc-files.s3.amazonaws.com/ps_marisa-lerin_13574_paper-037-template-polka-dots_cu.zip

答:
Check the settings under menu path Tools->Options->Security. You may need to uncheck the options under the "General" section of that dialogue box. Simply unchecking "Warn me about unwanted and uncommon software" resolves this.

No need to change other settings. I hope it helps.

Also see: https://support.mozilla.org/en-US/kb/how-does-phishing-and-malware-protection-work

Body:
I run a website (www.pixelscrapper.com) that serves file downloads of images, and zipped collections of images (which are zip files containing multiple imags, psds, vectors, etc.) These files are hosted on Amazon S3, and served via download urls generated by aws sdk for php (v1).

Just recently, users trying to download our zip files using Firefox have started getting "This file is not commonly downloaded" warnings (after the download finishes), which forces them to override the warning before accessing the file via the Firefox download manager. Naturally, this kind of warning causes concern for our users.

This warning shows up IN FIREFOX ONLY--Chrome, Edge, and Internet Explorer show no warnings when downloading and opening our zip files. The warning also only seems to show up for (surprise, surprise) files that have been more recently added to the site, and have relatively few total downloads--but many of our files never receive large numbers of downloads, so this warning has the potential to plague many of our files indefinitely.

My question is: is there anything I can do to prevent this warning? By adjusting headers, signing files in some way, etc.? (From what I understand, Chrome and Edge also have "uncommon file" warning, but they don't seem to be concerned with our files--why is this warning only firing in Firefox?) I've searched around on Stack Overflow and elsewhere, but most of the questions I've seen about "uncommon download" warnings are targeted at Chrome or Internet Explorer, and I can't seem to find any Firefox-specific information about this warning.

Here's a sample file download url (generated by aws sdk) that is causing warnings:

https://pixelscrapper-user-content.s3.amazonaws.com/template-attachment/user-2/node-13574/paper-037-template-polka-dots.zip?response-content-disposition=attachment%3B%20filename%3D%22ps_marisa-lerin_13574_paper-037-template-polka-dots_cu.zip%22&AWSAccessKeyId=AKIAIWM7MZMHRPA6FHEA&Expires=1495386939&Signature=HDmwRFPX81CIVrQgu1BkEyR9iRQ%3D

Here's an inspection of headers in Firefox:

UPDATE:

The issue here is not the nasty-looking url generated by the aws sdk: I checked downloading the same zip file (containing one jpg, one psd) from the following "clean" url, and it still gives the warning: http://pixelscrapper-misc-files.s3.amazonaws.com/ps_marisa-lerin_13574_paper-037-template-polka-dots_cu.zip


问:
ANGULAR - How the EventEmitter Service below, works exactly?
On my work, we've been using this service below to subscribe and emit notifications right and fourth. As a junior developer I already understand Input and Outputs of Angular and sort of how Emitters work.

But this service simplys confuse me. The dev who made this service already left the company and since this code is almost a boilerplate for all projects, I cannot ask him how exactly it works.

specifically this line: private static emitters: { [notificationName: string]: EventEmitter } = {}

```<code>import { Injectable } from '@angular/core';
import { EventEmitter } from '@angular/core';

@Injectable({ providedIn: 'root' })
export class EventEmitterService {

    private static emitters: { [notificationName: string]: EventEmitter<any> } = {}

    static get(notificationName: string): EventEmitter<any> {
        if (!this.emitters[notificationName])
            this.emitters[notificationName] = new EventEmitter<any>();
        return this.emitters[notificationName];
    }
}

/**
to SUBSCRIBE:

private subscriptions = new Subscription();
constructor(){
    this.subscriptions.add( EventEmitterService.get("notification_name").subscribe(payload => this.notificationHandler(payload)) );
}

TO EMIT:

EventEmitterService.get("notificationName").emit(payload);

to UNSUBSCRIBE:

ngOnDestroy(){
    this.subscriptions.unsubscribe();
}
*/
</code>```

Following instructions from the user Michael D, I've change the code

```<code>import { Subject } from 'rxjs';

private emitters: { [notificationName: string]: Subject<any> } = { };
</code>```

and got this error:

The error was caused because the return of static get(notificationName: string): was not changed to Observable<any>

So the final form of service, updated according to MichaelD comment :

```<code>import { Injectable } from '@angular/core';
import { EventEmitter } from '@angular/core';
import { Subject, Observable } from 'rxjs';

@Injectable({ providedIn: 'root' })
export class EventEmitterService {

    // private static emitters: { [notificationName: string]: EventEmitter<any> } = {}
    private  emitters: { [notificationName: string]: Subject<any> } = {};

    get(notificationName: string): Observable<any>{
        if (!this.emitters[notificationName])
            this.emitters[notificationName] = new EventEmitter<any>();
        return this.emitters[notificationName];
    }
}
</code>```

答:
I've been seeing this pattern of using EventEmitters in services frequently. EventEmitters are NOT meant to used in services. It is used to emit custom events from components to it's parent components.

The EventEmitter is an interface extension of RxJS Subject. If a multicast observable is required in the service, then the RxJS Subject (or BehaviorSubject) could be used.

In your case, the emitter variable is an object with a property notificationName that will point to the emitter.

Also I do not understand the meaning of static here. The service is already made a singleton with the { providedIn: 'root' } argument. There isn't going to be multiple instances of the emitters.

You could also have the same behavior with the following

import { Subject } from 'rxjs';

private emitters: { [notificationName: string]: Subject<any> } = { };
```

Body:
On my work, we've been using this service below to subscribe and emit notifications right and fourth. As a junior developer I already understand Input and Outputs of Angular and sort of how Emitters work.

But this service simplys confuse me. The dev who made this service already left the company and since this code is almost a boilerplate for all projects, I cannot ask him how exactly it works.

specifically this line: private static emitters: { [notificationName: string]: EventEmitter } = {}

import { Injectable } from '@angular/core';
import { EventEmitter } from '@angular/core';

@Injectable({ providedIn: 'root' })
export class EventEmitterService {

    private static emitters: { [notificationName: string]: EventEmitter<any> } = {}

    static get(notificationName: string): EventEmitter<any> {
        if (!this.emitters[notificationName])
            this.emitters[notificationName] = new EventEmitter<any>();
        return this.emitters[notificationName];
    }
}

/**
to SUBSCRIBE:

private subscriptions = new Subscription();
constructor(){
    this.subscriptions.add( EventEmitterService.get("notification_name").subscribe(payload => this.notificationHandler(payload)) );
}

TO EMIT:

EventEmitterService.get("notificationName").emit(payload);

to UNSUBSCRIBE:

ngOnDestroy(){
    this.subscriptions.unsubscribe();
}
*/

Following instructions from the user Michael D, I've change the code

import { Subject } from 'rxjs';

private emitters: { [notificationName: string]: Subject<any> } = { };

and got this error:

The error was caused because the return of static get(notificationName: string): was not changed to Observable<any>

So the final form of service, updated according to MichaelD comment :

import { Injectable } from '@angular/core';
import { EventEmitter } from '@angular/core';
import { Subject, Observable } from 'rxjs';

@Injectable({ providedIn: 'root' })
export class EventEmitterService {

    // private static emitters: { [notificationName: string]: EventEmitter<any> } = {}
    private  emitters: { [notificationName: string]: Subject<any> } = {};

    get(notificationName: string): Observable<any>{
        if (!this.emitters[notificationName])
            this.emitters[notificationName] = new EventEmitter<any>();
        return this.emitters[notificationName];
    }
}


问:
NGRX effect dispatched an invalid action
I am trying to create an @Effect() for my action. When I run action with type AuthenticationUserLoad I get an error.

```<code>ERROR Error: Effect "AuthenticationEffects.getUser$" dispatched an invalid action: [object Object]
</code>```

Here is my Effect code

```<code>    @Effect()
      getUser$ = this.actions$.pipe(
       ofType(AuthenticationUserActions.AuthenticationUserTypes.AuthenticationUserLoad),
       map((action) => {

          return this.authService.getUser().pipe(
            map((user: User) => new AuthenticationUserActions.AuthenticationUserLoadSuccess({user})),
            catchError(error => of(new AuthenticationUserActions.AuthenticationUserLoadFailure({error})))

          );
        })
     );
</code>```

UPDATE

I changed map to switchMap and it works.

```<code> @Effect()
  getUser$ = this.actions$.pipe(
    ofType(AuthenticationUserActions.AuthenticationUserTypes.AuthenticationUserLoad),
    switchMap((action) => {

      return this.authService.getUser().pipe(
        map((user: User) => new AuthenticationUserActions.AuthenticationUserLoadSuccess({user})),
        catchError(error => of(new AuthenticationUserActions.AuthenticationUserLoadFailure({error})))
      );
    })
  );
</code>```

Maybe I dont understand the difference between map and switchMap.

答:
map operator maps current value, it doesn't care whether it is an observable or not.
Whereas switchMap or mergeMap or concatMap expect a callback to return an observable value, subscribe to it and emits its values.

Therefore when you call map you say that current value should be transformed to some other.

map(action => return this.authService.getUser()),
// here the value is an observable stream, but not its emits.
```

<p>when you call <code>switchMap</code> you say now I want to subscribe to another stream and emit its values. Because it's <code>switchMap</code> it also says once the parent stream emits (the same action comes) unsubscribe from current child stream and subscribe again to what a new call of <code>this.authService.getUser()</code> returns.</p>

<pre class="lang-js prettyprint-override"><code>switchMap(action => return this.authService.getUser()),
// here the value is an emit from this.authService.getUser() stream.
</code>```

Body:
I am trying to create an @Effect() for my action. When I run action with type AuthenticationUserLoad I get an error.

ERROR Error: Effect "AuthenticationEffects.getUser$" dispatched an invalid action: [object Object]

Here is my Effect code

    @Effect()
      getUser$ = this.actions$.pipe(
       ofType(AuthenticationUserActions.AuthenticationUserTypes.AuthenticationUserLoad),
       map((action) => {

          return this.authService.getUser().pipe(
            map((user: User) => new AuthenticationUserActions.AuthenticationUserLoadSuccess({user})),
            catchError(error => of(new AuthenticationUserActions.AuthenticationUserLoadFailure({error})))

          );
        })
     );

UPDATE

I changed map to switchMap and it works.

 @Effect()
  getUser$ = this.actions$.pipe(
    ofType(AuthenticationUserActions.AuthenticationUserTypes.AuthenticationUserLoad),
    switchMap((action) => {

      return this.authService.getUser().pipe(
        map((user: User) => new AuthenticationUserActions.AuthenticationUserLoadSuccess({user})),
        catchError(error => of(new AuthenticationUserActions.AuthenticationUserLoadFailure({error})))
      );
    })
  );

Maybe I dont understand the difference between map and switchMap.

Body:
map operator maps current value, it doesn't care whether it is an observable or not.
Whereas switchMap or mergeMap or concatMap expect a callback to return an observable value, subscribe to it and emits its values.

Therefore when you call map you say that current value should be transformed to some other.

map(action => return this.authService.getUser()),
// here the value is an observable stream, but not its emits.

when you call switchMap you say now I want to subscribe to another stream and emit its values. Because it's switchMap it also says once the parent stream emits (the same action comes) unsubscribe from current child stream and subscribe again to what a new call of this.authService.getUser() returns.

switchMap(action => return this.authService.getUser()),
// here the value is an emit from this.authService.getUser() stream.


问:
http requests not working on Kotlin in AndroidStudio
I have a problem in Kotlin in Android Studio.
HTTP requests are not working for me and I have tried Fuel and Volley libraries for it.
I have added <uses-permission android:name="android.permission.INTERNET" /> line in AndroidManifest.xlm file.

Volley code:

```<code>val queue = Volley.newRequestQueue(this)
        val url = "http://drevo.kybernado.com/app/get_count.php?code=200528-0961"

        val stringRequest = StringRequest(
            Request.Method.GET, url,
            Response.Listener<String> { response ->
                Toast.makeText(this, "Response is: ${response.substring(0, 500)}", Toast.LENGTH_SHORT).show()
            },
            Response.ErrorListener {
                Toast.makeText(this, "That didn't work!", Toast.LENGTH_SHORT).show()
            })

        queue.add(stringRequest)
</code>```

Fuel code:

```<code>Fuel.get("http://drevo.kybernado.com/app/get_count.php?code=200528-0961")
            .response { request, response, result ->
                println(request)
                println(response)
                Toast.makeText(this, response.toString(), Toast.LENGTH_SHORT).show()
                val (bytes, error) = result
                Toast.makeText(this, result.toString(), Toast.LENGTH_SHORT).show()
                if (bytes != null) {
                    println("[response bytes] ${String(bytes)}")
                    Toast.makeText(this, String(bytes), Toast.LENGTH_SHORT).show()
                }
            }
</code>```

In volley it always gets an error and in Fuel there is no sign of activity - no toaster pops up.
Do you know why is it acting like this?
Thanks

PS: I'm new to Kotlin and Android Studio, but not in programming.

答: It's better to use this one RetrofitApi without coroutines, where your BASE_URL is "http://drevo.kybernado.com/app/" and yours GET_VALUE is "get_count.php" and add a code param in the interface fun get(@Query("code") code: String): Call<String>, you should pass this param in the get invoke Api.retrofitService.get("200528-0961").enqueue( object: Callback<String>

Body:
I have a problem in Kotlin in Android Studio.
HTTP requests are not working for me and I have tried Fuel and Volley libraries for it.
I have added <uses-permission android:name="android.permission.INTERNET" /> line in AndroidManifest.xlm file.

Volley code:

val queue = Volley.newRequestQueue(this)
        val url = "http://drevo.kybernado.com/app/get_count.php?code=200528-0961"

        val stringRequest = StringRequest(
            Request.Method.GET, url,
            Response.Listener<String> { response ->
                Toast.makeText(this, "Response is: ${response.substring(0, 500)}", Toast.LENGTH_SHORT).show()
            },
            Response.ErrorListener {
                Toast.makeText(this, "That didn't work!", Toast.LENGTH_SHORT).show()
            })

        queue.add(stringRequest)

Fuel code:

Fuel.get("http://drevo.kybernado.com/app/get_count.php?code=200528-0961")
            .response { request, response, result ->
                println(request)
                println(response)
                Toast.makeText(this, response.toString(), Toast.LENGTH_SHORT).show()
                val (bytes, error) = result
                Toast.makeText(this, result.toString(), Toast.LENGTH_SHORT).show()
                if (bytes != null) {
                    println("[response bytes] ${String(bytes)}")
                    Toast.makeText(this, String(bytes), Toast.LENGTH_SHORT).show()
                }
            }

In volley it always gets an error and in Fuel there is no sign of activity - no toaster pops up.
Do you know why is it acting like this?
Thanks

PS: I'm new to Kotlin and Android Studio, but not in programming.


问:
How to grab text values from getAllByTestId?
I have a component that lists a bunch of items. In this, they're cryptocurrency assets. When I click the header labelled Name, it sorts the items in alphabetical order. I would like to test this functionality by clicking the Name button in order to fire the event, and then asserting that the values in the name column are sorted alphabetically:

```<code>  it("Sorts by name on click", () => {
    const sortedByName = spotData
      .sort((a, b) => {
        return a.name < b.name ? -1 : a.name === b.name ? 0 : 1;
      })
      .map((ticker) => ticker.name);

    const { getByText, getAllByTestId } = renderWithProviders(
      <MarketSelectorPanel marketsList={spotData} />
    );

    fireEvent.click(getByText("Name"));
    expect(getAllByTestId("market-selector-row-name")).toEqual(
      sortedByName
    );
  });
</code>```

The above doesn't work because expect(getAllByTestId("market-selector-row-name")) grabs the entire HTML element:

● MarketSelectorPanel tables › Sorts by name on click

```<code>expect(received).toEqual(expected) // deep equality

- Expected
+ Received

  Array [
-   "BCH/USD",
-   "BTC/USD",
-   "ETH/USD",
+   <span
+     class="market-selector-row-val"
+     data-testid="market-selector-row-name"
+   >
+     BCH/USD
+   </span>,
+   <span
+     class="market-selector-row-val"
+     data-testid="market-selector-row-name"
+   >
+     BTC/USD
+   </span>,
+   <span
+     class="market-selector-row-val"
+     data-testid="market-selector-row-name"
+   >
+     ETH/USD
+   </span>
]
</code>```

答: Please try passing the XPath of the component for testing rather than the id</p>

Body:
I have a component that lists a bunch of items. In this, they're cryptocurrency assets. When I click the header labelled Name, it sorts the items in alphabetical order. I would like to test this functionality by clicking the Name button in order to fire the event, and then asserting that the values in the name column are sorted alphabetically:

  it("Sorts by name on click", () => {
    const sortedByName = spotData
      .sort((a, b) => {
        return a.name  ticker.name);

    const { getByText, getAllByTestId } = renderWithProviders(
      <MarketSelectorPanel marketsList={spotData} />
    );

    fireEvent.click(getByText("Name"));
    expect(getAllByTestId("market-selector-row-name")).toEqual(
      sortedByName
    );
  });

The above doesn't work because expect(getAllByTestId("market-selector-row-name")) grabs the entire HTML element:

● MarketSelectorPanel tables › Sorts by name on click

expect(received).toEqual(expected) // deep equality

- Expected
+ Received

  Array [
-   "BCH/USD",
-   "BTC/USD",
-   "ETH/USD",
+
+     BCH/USD
+   ,
+
+     BTC/USD
+   ,
+
+     ETH/USD
+
]


问:
Date-picker isn't captured by html2canvas
I'm using Html2canvas to take a screenshot of my page, on the page, I'm using date-picker. but when I capture the screenshot the date is not be shown in the screenshot. any idea why is that and how I solve it?

this is my code

```<code><div id='ownerDetails'>
  <input type="date" class="the-inp date-input">
</div>

<button id="sig-submitBtn">save</button >
<button id='nextbt'>capture</button>

<img id='itemImg' src=''>

</code>```

```<code>//////////////

    $(".date-input").datepicker({ dateFormat: "yy-mm-dd"}).datepicker("setDate", new Date());

/////////

var element = $("#ownerDetails"); // global variable
            var getCanvas; //global variable

    $('#sig-submitBtn').on('click', function () {

            html2canvas(element, {
                onrendered: function (canvas) {
                    getCanvas = canvas;
                }
            });

    });

            $("#nextbt").on('click', function () {
                var imgageData = getCanvas.toDataURL("image/png");
                //Now browser starts downloading it instead of just showing it
                document.getElementById("itemImg").src = imgageData;
            });

</code>```

答:
Added solution different way here is fiddle :

https://jsfiddle.net/vpysd9zk/

additional code :

.selectedDate{
 background: #fff;
    border: 1px solid #000;
    padding: 3px;
    padding-left: 18px;
    padding-right: 17px;
    font-size: 14px;
  }
```

<pre class="lang-html prettyprint-override"><code><div id='ownerDetails'>
  <input type="date" class="the-inp date-input">
  <lable class="selectedDate" style="display:none;"></lable>

</div>
</code>```

/////////

var element = $("#ownerDetails"); // global variable
            var getCanvas; //global variable

    $('#sig-submitBtn').on('click', function () {

            var selectedDate = $('.date-input').val();
            $('.date-input').hide();
            $(".selectedDate").show();
            $(".selectedDate").html(selectedDate);
            html2canvas(element, {
                onrendered: function (canvas) {
                    getCanvas = canvas;
                }
            });

    });

            $("#nextbt").on('click', function () {
                var imgageData = getCanvas.toDataURL("image/png");
                //Now browser starts downloading it instead of just showing it
                document.getElementById("itemImg").src = imgageData;
            });

```

Body:
I'm using Html2canvas to take a screenshot of my page, on the page, I'm using date-picker. but when I capture the screenshot the date is not be shown in the screenshot. any idea why is that and how I solve it?

this is my code

  <input type="date" class="the-inp date-input">

<button id="sig-submitBtn">save</button >
<button id='nextbt'>capture</button>

//////////////

    $(".date-input").datepicker({ dateFormat: "yy-mm-dd"}).datepicker("setDate", new Date());

/////////

var element = $("#ownerDetails"); // global variable
            var getCanvas; //global variable

    $('#sig-submitBtn').on('click', function () {

            html2canvas(element, {
                onrendered: function (canvas) {
                    getCanvas = canvas;
                }
            });

    });

            $("#nextbt").on('click', function () {
                var imgageData = getCanvas.toDataURL("image/png");
                //Now browser starts downloading it instead of just showing it
                document.getElementById("itemImg").src = imgageData;
            });

Body:
Added solution different way here is fiddle :

https://jsfiddle.net/vpysd9zk/

additional code :

.selectedDate{
 background: #fff;
    border: 1px solid #000;
    padding: 3px;
    padding-left: 18px;
    padding-right: 17px;
    font-size: 14px;
  }

<div id='ownerDetails'>
  <input type="date" class="the-inp date-input">
  <lable class="selectedDate" style="display:none;"></lable>

</div>

/////////

var element = $("#ownerDetails"); // global variable
            var getCanvas; //global variable

    $('#sig-submitBtn').on('click', function () {

            var selectedDate = $('.date-input').val();
            $('.date-input').hide();
            $(".selectedDate").show();
            $(".selectedDate").html(selectedDate);
            html2canvas(element, {
                onrendered: function (canvas) {
                    getCanvas = canvas;
                }
            });

    });

            $("#nextbt").on('click', function () {
                var imgageData = getCanvas.toDataURL("image/png");
                //Now browser starts downloading it instead of just showing it
                document.getElementById("itemImg").src = imgageData;
            });


问:
Read and write from/to S3 bucket using access points with boto3
I have to access S3 bucket using access points with boto3.

I have created an access point with a policy to allow reading and writing (<access_point_arn> is my access point ARN):

```<code>{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": ["s3:GetObject", "s3:PutObject"],
            "Resource": "<access_point_arn>/object/*"
    ]
}
</code>```

In the official documentation there is a mention about access points, where access point ARN has to come in place of bucket name (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html). There are no examples on the official documentation site for developers (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html).

So based on the information I assume that the right way to use it is:

```<code>import boto3
s3 = boto3.resource('s3')
s3.Bucket('<access_point_arn>').download_file('hello.txt', '/tmp/hello.txt')
</code>```

When I execute this code in Lambda with AmazonS3FullAccess managed policy attached I am getting an ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Both Lambda and S3 access point are connected to the same VPC.

答:
replace...

"Resource": "arn:aws:s3:region_name:<12-digit account_id>:bucket_name"

s3.Bucket('bucket_name').download_file('hello.txt', '/tmp/hello.txt')

Hope it helps...

Body:
I have to access S3 bucket using access points with boto3.

I have created an access point with a policy to allow reading and writing (<access_point_arn> is my access point ARN):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": ["s3:GetObject", "s3:PutObject"],
            "Resource": "<access_point_arn>/object/*"
    ]
}

In the official documentation there is a mention about access points, where access point ARN has to come in place of bucket name (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html). There are no examples on the official documentation site for developers (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html).

So based on the information I assume that the right way to use it is:

import boto3
s3 = boto3.resource('s3')
s3.Bucket('<access_point_arn>').download_file('hello.txt', '/tmp/hello.txt')

When I execute this code in Lambda with AmazonS3FullAccess managed policy attached I am getting an ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Both Lambda and S3 access point are connected to the same VPC.