问:
ERROR sparklyr: Gateway xxxxx failed calling take on xxx when running spark-apply
I am trying to run spark_apply from the sparklyr package to perform kmeans clustering on a bunch of data hosted on a spark cluster. But I am receiving a spark error that I am having difficulty understanding. The data is as follows where the features column is an aggregated vector combining the latitude and longitude columns but not used in this case.

```<code>> samplog1
# Source:   table<sparklyr_tmp_64d4941e1a2> [?? x 6]
# Database: spark_connection
                                     id           timestamp    hr latitude longitude  features
                                  <chr>               <chr> <int>    <dbl>     <dbl>    <list>
 1 fffc68e3-866e-4be5-b1bc-5d21b89622ae 2017-10-30 04:29:59     4 1.373545  104.1265 <dbl [2]>
 2 fffc7412-deb1-4587-9c22-29ca833865ed 2017-10-30 02:49:47     2 5.701320  117.4892 <dbl [2]>
 3 fffd16d5-83f1-4ea1-95de-34b1fcad392b 2017-10-30 04:25:44     4 5.334012  100.2172 <dbl [2]>
 4 fffc68e3-866e-4be5-b1bc-5d21b89622ae 2017-10-30 04:29:44     4 1.373545  104.1265 <dbl [2]>
 5 fffd16d5-83f1-4ea1-95de-34b1fcad392b 2017-10-30 02:58:30     2 5.334061  100.2173 <dbl [2]>
 6 fffd16d5-83f1-4ea1-95de-34b1fcad392b 2017-10-30 04:55:41     4 5.334012  100.2172 <dbl [2]>
 7 fffc7412-deb1-4587-9c22-29ca833865ed 2017-10-30 04:49:07     4 5.729879  117.5787 <dbl [2]>
 8 fffc68e3-866e-4be5-b1bc-5d21b89622ae 2017-10-30 05:02:08     5 1.373545  104.1265 <dbl [2]>
 9 fffc7412-deb1-4587-9c22-29ca833865ed 2017-10-30 00:53:12     0 5.701320  117.4892 <dbl [2]>
10 fffc7412-deb1-4587-9c22-29ca833865ed 2017-10-30 04:08:12     4 5.670300  117.4990 <dbl [2]>
# ... with more rows
</code>```

The R code is as follows:

```<code>kms <- function(idLogs){
    tryCatch({

    #idLogs <- sparklyr::ft_vector_assembler(idLogs, input_cols= c("latitude", "longitude"), output_col = "features")

    km  <- sparklyr::ml_kmeans(x = idLogs, centers = 3,features = c("latitude","longitude"))

    km1 <- copy_to(sc, km$centers, overwrite = T)

    return(data.frame(x=1,y=1))

    cluster <- sdf_predict(km)

    clustCounts <- cluster %>% group_by(prediction) %>%
      tally  %>%
      mutate(conf=n/sum(n),
             prediction=prediction+1)

    clustCounts <- merge(clustCounts, km$centers, by.x=3, by.y=0)

    clustCounts <- clustCounts %>% filter(., conf==max(conf)) %>% select(latitude, longitude, conf)

    #clustCounts <- cbind.data.frame(id, hr, clustCounts)

    #clustCounts1 <- copy_to(sc, clustCounts, overwrite = T)

    return(data.frame(clustCounts))
  }, error = function(e) {
    return(
      data.frame(string_categories = c(substr(e, 1, 20)))
    )
  })
}
</code>```

and called like so

```<code>likelyLocs <- spark_apply(samplog, kms)
</code>```

The error I am receiving in RStudio is:

```<code>Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 187.0 failed 4 times, most recent failure: Lost task 0.3 in stage 187.0 (TID 250, spark-1.c.halogen-order-184815.internal, executor 2): java.lang.Exception: sparklyr worker rscript failure with status 255, check worker logs for details.
    at sparklyr.Rscript.init(rscript.scala:67)
    at sparklyr.WorkerRDD$$anon$2.run(rdd.scala:92)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1457)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1445)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1444)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1444)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
    at scala.Option.foreach(Option.scala:236)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1668)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1627)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1616)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1862)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1875)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1888)
    at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1328)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
    at org.apache.spark.rdd.RDD.take(RDD.scala:1302)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sparklyr.Invoke$.invoke(invoke.scala:102)
    at sparklyr.StreamHandler$.handleMethodCall(stream.scala:97)
    at sparklyr.StreamHandler$.read(stream.scala:62)
    at sparklyr.BackendHandler.channelRead0(handler.scala:52)
    at sparklyr.BackendHandler.channelRead0(handler.scala:14)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: sparklyr worker rscript failure with status 255, check worker logs for details.
    at sparklyr.Rscript.init(rscript.scala:67)
    at sparklyr.WorkerRDD$$anon$2.run(rdd.scala:92)
</code>```

As directed in the error details, I checked the spark log and got the following.

```<code>> spark_log(sc)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 402
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_237_piece0 on 10.148.0.3:34567 in memory (size: 250.0 B, free: 530.0 MB)
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_237_piece0 on spark-1.c.halogen-order-184815.internal:35671 in memory (size: 250.0 B, free: 530.2 MB)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 401
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_236_piece0 on 10.148.0.3:34567 in memory (size: 1658.0 B, free: 530.0 MB)
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_236_piece0 on spark-1.c.halogen-order-184815.internal:35671 in memory (size: 1658.0 B, free: 530.2 MB)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 400
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_235_piece0 on 10.148.0.3:34567 in memory (size: 9.4 KB, free: 530.0 MB)
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_235_piece0 on spark-1.c.halogen-order-184815.internal:35671 in memory (size: 9.4 KB, free: 530.2 MB)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 399
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned shuffle 31
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_234_piece0 on 10.148.0.3:34567 in memory (size: 202.0 B, free: 530.0 MB)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 398
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_233_piece0 on 10.148.0.3:34567 in memory (size: 1550.0 B, free: 530.0 MB)
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_233_piece0 on spark-1.c.halogen-order-184815.internal:35671 in memory (size: 1550.0 B, free: 530.2 MB)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 397
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_232_piece0 on 10.148.0.3:34567 in memory (size: 9.3 KB, free: 530.0 MB)
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_232_piece0 on spark-1.c.halogen-order-184815.internal:35671 in memory (size: 9.3 KB, free: 530.2 MB)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 396
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned shuffle 30
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_231_piece0 on 10.148.0.3:34567 in memory (size: 421.0 B, free: 530.0 MB)
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_230_piece0 on 10.148.0.3:34567 in memory (size: 9.5 KB, free: 530.0 MB)
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_230_piece0 on spark-1.c.halogen-order-184815.internal:35671 in memory (size: 9.5 KB, free: 530.2 MB)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 395
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_229_piece0 on 10.148.0.3:34567 in memory (size: 9.4 KB, free: 530.0 MB)
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_229_piece0 on spark-1.c.halogen-order-184815.internal:35671 in memory (size: 9.4 KB, free: 530.3 MB)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 394
17/11/09 21:48:05 INFO storage.BlockManager: Removing RDD 515
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned RDD 515
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_228_piece0 on 10.148.0.3:34567 in memory (size: 175.0 B, free: 530.0 MB)
17/11/13 12:11:09 INFO spark.SparkContext: Starting job: collect at utils.scala:196
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Got job 153 (collect at utils.scala:196) with 1 output partitions
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Final stage: ResultStage 185 (collect at utils.scala:196)
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Parents of final stage: List()
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Missing parents: List()
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Submitting ResultStage 185 (MapPartitionsRDD[536] at collect at utils.scala:196), which has no missing parents
17/11/13 12:11:09 INFO storage.MemoryStore: Block broadcast_241 stored as values in memory (estimated size 1968.0 B, free 530.0 MB)
17/11/13 12:11:09 INFO storage.MemoryStore: Block broadcast_241_piece0 stored as bytes in memory (estimated size 1206.0 B, free 530.0 MB)
17/11/13 12:11:09 INFO storage.BlockManagerInfo: Added broadcast_241_piece0 in memory on 10.148.0.3:34567 (size: 1206.0 B, free: 530.0 MB)
17/11/13 12:11:09 INFO spark.SparkContext: Created broadcast 241 from broadcast at DAGScheduler.scala:1004
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 185 (MapPartitionsRDD[536] at collect at utils.scala:196) (first 15 tasks are for partitions Vector(0))
17/11/13 12:11:09 INFO cluster.YarnScheduler: Adding task set 185.0 with 1 tasks
17/11/13 12:11:09 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 185.0 (TID 245, spark-1.c.halogen-order-184815.internal, executor 2, partition 0, PROCESS_LOCAL, 3699 bytes)
17/11/13 12:11:09 INFO storage.BlockManagerInfo: Added broadcast_241_piece0 in memory on spark-1.c.halogen-order-184815.internal:35671 (size: 1206.0 B, free: 530.3 MB)
17/11/13 12:11:09 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 185.0 (TID 245) in 94 ms on spark-1.c.halogen-order-184815.internal (executor 2) (1/1)
17/11/13 12:11:09 INFO scheduler.DAGScheduler: ResultStage 185 (collect at utils.scala:196) finished in 0.096 s
17/11/13 12:11:09 INFO cluster.YarnScheduler: Removed TaskSet 185.0, whose tasks have all completed, from pool
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Job 153 finished: collect at utils.scala:196, took 0.111329 s
17/11/13 12:11:09 INFO spark.SparkContext: Starting job: collect at utils.scala:196
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Got job 154 (collect at utils.scala:196) with 1 output partitions
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Final stage: ResultStage 186 (collect at utils.scala:196)
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Parents of final stage: List()
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Missing parents: List()
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Submitting ResultStage 186 (MapPartitionsRDD[538] at collect at utils.scala:196), which has no missing parents
17/11/13 12:11:09 INFO storage.MemoryStore: Block broadcast_242 stored as values in memory (estimated size 1968.0 B, free 530.0 MB)
17/11/13 12:11:09 INFO storage.MemoryStore: Block broadcast_242_piece0 stored as bytes in memory (estimated size 1207.0 B, free 530.0 MB)
17/11/13 12:11:09 INFO storage.BlockManagerInfo: Added broadcast_242_piece0 in memory on 10.148.0.3:34567 (size: 1207.0 B, free: 530.0 MB)
17/11/13 12:11:09 INFO spark.SparkContext: Created broadcast 242 from broadcast at DAGScheduler.scala:1004
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 186 (MapPartitionsRDD[538] at collect at utils.scala:196) (first 15 tasks are for partitions Vector(0))
17/11/13 12:11:09 INFO cluster.YarnScheduler: Adding task set 186.0 with 1 tasks
17/11/13 12:11:09 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 186.0 (TID 246, spark-1.c.halogen-order-184815.internal, executor 2, partition 0, PROCESS_LOCAL, 3699 bytes)
17/11/13 12:11:09 INFO storage.BlockManagerInfo: Added broadcast_242_piece0 in memory on spark-1.c.halogen-order-184815.internal:35671 (size: 1207.0 B, free: 530.3 MB)
17/11/13 12:11:09 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 186.0 (TID 246) in 22 ms on spark-1.c.halogen-order-184815.internal (executor 2) (1/1)
17/11/13 12:11:09 INFO scheduler.DAGScheduler: ResultStage 186 (collect at utils.scala:196) finished in 0.022 s
17/11/13 12:11:09 INFO cluster.YarnScheduler: Removed TaskSet 186.0, whose tasks have all completed, from pool
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Job 154 finished: collect at utils.scala:196, took 0.031006 s
17/11/13 12:11:22 INFO spark.SparkContext: Starting job: take at NativeMethodAccessorImpl.java:-2
17/11/13 12:11:22 INFO scheduler.DAGScheduler: Got job 155 (take at NativeMethodAccessorImpl.java:-2) with 1 output partitions
17/11/13 12:11:22 INFO scheduler.DAGScheduler: Final stage: ResultStage 187 (take at NativeMethodAccessorImpl.java:-2)
17/11/13 12:11:22 INFO scheduler.DAGScheduler: Parents of final stage: List()
17/11/13 12:11:22 INFO scheduler.DAGScheduler: Missing parents: List()
17/11/13 12:11:22 INFO scheduler.DAGScheduler: Submitting ResultStage 187 (WorkerRDD[542] at RDD at rdd.scala:7), which has no missing parents
17/11/13 12:11:22 INFO storage.MemoryStore: Block broadcast_243 stored as values in memory (estimated size 35.2 KB, free 530.0 MB)
17/11/13 12:11:22 INFO storage.MemoryStore: Block broadcast_243_piece0 stored as bytes in memory (estimated size 14.4 KB, free 530.0 MB)
17/11/13 12:11:22 INFO storage.BlockManagerInfo: Added broadcast_243_piece0 in memory on 10.148.0.3:34567 (size: 14.4 KB, free: 530.0 MB)
17/11/13 12:11:22 INFO spark.SparkContext: Created broadcast 243 from broadcast at DAGScheduler.scala:1004
17/11/13 12:11:22 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 187 (WorkerRDD[542] at RDD at rdd.scala:7) (first 15 tasks are for partitions Vector(0))
17/11/13 12:11:22 INFO cluster.YarnScheduler: Adding task set 187.0 with 1 tasks
17/11/13 12:11:22 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 187.0 (TID 247, spark-1.c.halogen-order-184815.internal, executor 2, partition 0, PROCESS_LOCAL, 3488 bytes)
17/11/13 12:11:22 INFO storage.BlockManagerInfo: Added broadcast_243_piece0 in memory on spark-1.c.halogen-order-184815.internal:35671 (size: 14.4 KB, free: 530.2 MB)
17/11/13 12:11:23 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 187.0 (TID 247, spark-1.c.halogen-order-184815.internal, executor 2): java.lang.Exception: sparklyr worker rscript failure with status 255, check worker logs for details.
    at sparklyr.Rscript.init(rscript.scala:67)
    at sparklyr.WorkerRDD$$anon$2.run(rdd.scala:92)

17/11/13 12:11:23 INFO scheduler.TaskSetManager: Starting task 0.1 in stage 187.0 (TID 248, spark-1.c.halogen-order-184815.internal, executor 2, partition 0, PROCESS_LOCAL, 3488 bytes)
17/11/13 12:11:24 INFO scheduler.TaskSetManager: Lost task 0.1 in stage 187.0 (TID 248) on spark-1.c.halogen-order-184815.internal, executor 2: java.lang.Exception (sparklyr worker rscript failure with status 255, check worker logs for details.) [duplicate 1]
17/11/13 12:11:24 INFO scheduler.TaskSetManager: Starting task 0.2 in stage 187.0 (TID 249, spark-1.c.halogen-order-184815.internal, executor 2, partition 0, PROCESS_LOCAL, 3488 bytes)
17/11/13 12:11:25 INFO scheduler.TaskSetManager: Lost task 0.2 in stage 187.0 (TID 249) on spark-1.c.halogen-order-184815.internal, executor 2: java.lang.Exception (sparklyr worker rscript failure with status 255, check worker logs for details.) [duplicate 2]
17/11/13 12:11:25 INFO scheduler.TaskSetManager: Starting task 0.3 in stage 187.0 (TID 250, spark-1.c.halogen-order-184815.internal, executor 2, partition 0, PROCESS_LOCAL, 3488 bytes)
17/11/13 12:11:25 INFO scheduler.TaskSetManager: Lost task 0.3 in stage 187.0 (TID 250) on spark-1.c.halogen-order-184815.internal, executor 2: java.lang.Exception (sparklyr worker rscript failure with status 255, check worker logs for details.) [duplicate 3]
17/11/13 12:11:25 ERROR scheduler.TaskSetManager: Task 0 in stage 187.0 failed 4 times; aborting job
17/11/13 12:11:25 INFO cluster.YarnScheduler: Removed TaskSet 187.0, whose tasks have all completed, from pool
17/11/13 12:11:25 INFO cluster.YarnScheduler: Cancelling stage 187
17/11/13 12:11:25 INFO scheduler.DAGScheduler: ResultStage 187 (take at NativeMethodAccessorImpl.java:-2) failed in 3.496 s due to Job aborted due to stage failure: Task 0 in stage 187.0 failed 4 times, most recent failure: Lost task 0.3 in stage 187.0 (TID 250, spark-1.c.halogen-order-184815.internal, executor 2): java.lang.Exception: sparklyr worker rscript failure with status 255, check worker logs for details.
    at sparklyr.Rscript.init(rscript.scala:67)
    at sparklyr.WorkerRDD$$anon$2.run(rdd.scala:92)

Driver stacktrace:
17/11/13 12:11:25 INFO scheduler.DAGScheduler: Job 155 failed: take at NativeMethodAccessorImpl.java:-2, took 3.506663 s
17/11/13 12:11:25 ERROR sparklyr: Gateway (37351) failed calling take on 699
</code>```

All I can seem to fathom is that the spark job is failing somewhere in the last stage so probably merging the output from the different workers? Can anyone help find what may be the problem?

答:
Solved with assistance on the package github issues page here: https://github.com/rstudio/sparklyr/issues/1121

The relevant part:

Still not sure why but adding config=list() to spark_connect did
  it - idea came from
  Can sparklyr be used with spark deployed on yarn-managed hadoop cluster?.

Body:
I am trying to run spark_apply from the sparklyr package to perform kmeans clustering on a bunch of data hosted on a spark cluster. But I am receiving a spark error that I am having difficulty understanding. The data is as follows where the features column is an aggregated vector combining the latitude and longitude columns but not used in this case.

> samplog1
# Source:   table<sparklyr_tmp_64d4941e1a2> [?? x 6]
# Database: spark_connection
                                     id           timestamp    hr latitude longitude  features
                                  <chr>               <chr> <int>    <dbl>     <dbl>    <list>
 1 fffc68e3-866e-4be5-b1bc-5d21b89622ae 2017-10-30 04:29:59     4 1.373545  104.1265 <dbl [2]>
 2 fffc7412-deb1-4587-9c22-29ca833865ed 2017-10-30 02:49:47     2 5.701320  117.4892 <dbl [2]>
 3 fffd16d5-83f1-4ea1-95de-34b1fcad392b 2017-10-30 04:25:44     4 5.334012  100.2172 <dbl [2]>
 4 fffc68e3-866e-4be5-b1bc-5d21b89622ae 2017-10-30 04:29:44     4 1.373545  104.1265 <dbl [2]>
 5 fffd16d5-83f1-4ea1-95de-34b1fcad392b 2017-10-30 02:58:30     2 5.334061  100.2173 <dbl [2]>
 6 fffd16d5-83f1-4ea1-95de-34b1fcad392b 2017-10-30 04:55:41     4 5.334012  100.2172 <dbl [2]>
 7 fffc7412-deb1-4587-9c22-29ca833865ed 2017-10-30 04:49:07     4 5.729879  117.5787 <dbl [2]>
 8 fffc68e3-866e-4be5-b1bc-5d21b89622ae 2017-10-30 05:02:08     5 1.373545  104.1265 <dbl [2]>
 9 fffc7412-deb1-4587-9c22-29ca833865ed 2017-10-30 00:53:12     0 5.701320  117.4892 <dbl [2]>
10 fffc7412-deb1-4587-9c22-29ca833865ed 2017-10-30 04:08:12     4 5.670300  117.4990 <dbl [2]>
# ... with more rows

The R code is as follows:

kms <- function(idLogs){
    tryCatch({

    #idLogs <- sparklyr::ft_vector_assembler(idLogs, input_cols= c("latitude", "longitude"), output_col = "features")

    km  <- sparklyr::ml_kmeans(x = idLogs, centers = 3,features = c("latitude","longitude"))

    km1 <- copy_to(sc, km$centers, overwrite = T)

    return(data.frame(x=1,y=1))

    cluster <- sdf_predict(km)

    clustCounts <- cluster %>% group_by(prediction) %>%
      tally  %>%
      mutate(conf=n/sum(n),
             prediction=prediction+1)

    clustCounts <- merge(clustCounts, km$centers, by.x=3, by.y=0)

    clustCounts <- clustCounts %>% filter(., conf==max(conf)) %>% select(latitude, longitude, conf)

    #clustCounts <- cbind.data.frame(id, hr, clustCounts)

    #clustCounts1 <- copy_to(sc, clustCounts, overwrite = T)

    return(data.frame(clustCounts))
  }, error = function(e) {
    return(
      data.frame(string_categories = c(substr(e, 1, 20)))
    )
  })
}

and called like so

likelyLocs <- spark_apply(samplog, kms)

The error I am receiving in RStudio is:

Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 187.0 failed 4 times, most recent failure: Lost task 0.3 in stage 187.0 (TID 250, spark-1.c.halogen-order-184815.internal, executor 2): java.lang.Exception: sparklyr worker rscript failure with status 255, check worker logs for details.
    at sparklyr.Rscript.init(rscript.scala:67)
    at sparklyr.WorkerRDD$$anon$2.run(rdd.scala:92)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1457)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1445)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1444)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1444)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
    at scala.Option.foreach(Option.scala:236)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1668)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1627)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1616)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1862)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1875)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1888)
    at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1328)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
    at org.apache.spark.rdd.RDD.take(RDD.scala:1302)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sparklyr.Invoke$.invoke(invoke.scala:102)
    at sparklyr.StreamHandler$.handleMethodCall(stream.scala:97)
    at sparklyr.StreamHandler$.read(stream.scala:62)
    at sparklyr.BackendHandler.channelRead0(handler.scala:52)
    at sparklyr.BackendHandler.channelRead0(handler.scala:14)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: sparklyr worker rscript failure with status 255, check worker logs for details.
    at sparklyr.Rscript.init(rscript.scala:67)
    at sparklyr.WorkerRDD$$anon$2.run(rdd.scala:92)

As directed in the error details, I checked the spark log and got the following.

> spark_log(sc)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 402
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_237_piece0 on 10.148.0.3:34567 in memory (size: 250.0 B, free: 530.0 MB)
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_237_piece0 on spark-1.c.halogen-order-184815.internal:35671 in memory (size: 250.0 B, free: 530.2 MB)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 401
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_236_piece0 on 10.148.0.3:34567 in memory (size: 1658.0 B, free: 530.0 MB)
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_236_piece0 on spark-1.c.halogen-order-184815.internal:35671 in memory (size: 1658.0 B, free: 530.2 MB)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 400
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_235_piece0 on 10.148.0.3:34567 in memory (size: 9.4 KB, free: 530.0 MB)
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_235_piece0 on spark-1.c.halogen-order-184815.internal:35671 in memory (size: 9.4 KB, free: 530.2 MB)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 399
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned shuffle 31
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_234_piece0 on 10.148.0.3:34567 in memory (size: 202.0 B, free: 530.0 MB)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 398
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_233_piece0 on 10.148.0.3:34567 in memory (size: 1550.0 B, free: 530.0 MB)
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_233_piece0 on spark-1.c.halogen-order-184815.internal:35671 in memory (size: 1550.0 B, free: 530.2 MB)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 397
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_232_piece0 on 10.148.0.3:34567 in memory (size: 9.3 KB, free: 530.0 MB)
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_232_piece0 on spark-1.c.halogen-order-184815.internal:35671 in memory (size: 9.3 KB, free: 530.2 MB)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 396
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned shuffle 30
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_231_piece0 on 10.148.0.3:34567 in memory (size: 421.0 B, free: 530.0 MB)
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_230_piece0 on 10.148.0.3:34567 in memory (size: 9.5 KB, free: 530.0 MB)
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_230_piece0 on spark-1.c.halogen-order-184815.internal:35671 in memory (size: 9.5 KB, free: 530.2 MB)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 395
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_229_piece0 on 10.148.0.3:34567 in memory (size: 9.4 KB, free: 530.0 MB)
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_229_piece0 on spark-1.c.halogen-order-184815.internal:35671 in memory (size: 9.4 KB, free: 530.3 MB)
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned accumulator 394
17/11/09 21:48:05 INFO storage.BlockManager: Removing RDD 515
17/11/09 21:48:05 INFO spark.ContextCleaner: Cleaned RDD 515
17/11/09 21:48:05 INFO storage.BlockManagerInfo: Removed broadcast_228_piece0 on 10.148.0.3:34567 in memory (size: 175.0 B, free: 530.0 MB)
17/11/13 12:11:09 INFO spark.SparkContext: Starting job: collect at utils.scala:196
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Got job 153 (collect at utils.scala:196) with 1 output partitions
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Final stage: ResultStage 185 (collect at utils.scala:196)
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Parents of final stage: List()
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Missing parents: List()
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Submitting ResultStage 185 (MapPartitionsRDD[536] at collect at utils.scala:196), which has no missing parents
17/11/13 12:11:09 INFO storage.MemoryStore: Block broadcast_241 stored as values in memory (estimated size 1968.0 B, free 530.0 MB)
17/11/13 12:11:09 INFO storage.MemoryStore: Block broadcast_241_piece0 stored as bytes in memory (estimated size 1206.0 B, free 530.0 MB)
17/11/13 12:11:09 INFO storage.BlockManagerInfo: Added broadcast_241_piece0 in memory on 10.148.0.3:34567 (size: 1206.0 B, free: 530.0 MB)
17/11/13 12:11:09 INFO spark.SparkContext: Created broadcast 241 from broadcast at DAGScheduler.scala:1004
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 185 (MapPartitionsRDD[536] at collect at utils.scala:196) (first 15 tasks are for partitions Vector(0))
17/11/13 12:11:09 INFO cluster.YarnScheduler: Adding task set 185.0 with 1 tasks
17/11/13 12:11:09 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 185.0 (TID 245, spark-1.c.halogen-order-184815.internal, executor 2, partition 0, PROCESS_LOCAL, 3699 bytes)
17/11/13 12:11:09 INFO storage.BlockManagerInfo: Added broadcast_241_piece0 in memory on spark-1.c.halogen-order-184815.internal:35671 (size: 1206.0 B, free: 530.3 MB)
17/11/13 12:11:09 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 185.0 (TID 245) in 94 ms on spark-1.c.halogen-order-184815.internal (executor 2) (1/1)
17/11/13 12:11:09 INFO scheduler.DAGScheduler: ResultStage 185 (collect at utils.scala:196) finished in 0.096 s
17/11/13 12:11:09 INFO cluster.YarnScheduler: Removed TaskSet 185.0, whose tasks have all completed, from pool
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Job 153 finished: collect at utils.scala:196, took 0.111329 s
17/11/13 12:11:09 INFO spark.SparkContext: Starting job: collect at utils.scala:196
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Got job 154 (collect at utils.scala:196) with 1 output partitions
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Final stage: ResultStage 186 (collect at utils.scala:196)
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Parents of final stage: List()
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Missing parents: List()
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Submitting ResultStage 186 (MapPartitionsRDD[538] at collect at utils.scala:196), which has no missing parents
17/11/13 12:11:09 INFO storage.MemoryStore: Block broadcast_242 stored as values in memory (estimated size 1968.0 B, free 530.0 MB)
17/11/13 12:11:09 INFO storage.MemoryStore: Block broadcast_242_piece0 stored as bytes in memory (estimated size 1207.0 B, free 530.0 MB)
17/11/13 12:11:09 INFO storage.BlockManagerInfo: Added broadcast_242_piece0 in memory on 10.148.0.3:34567 (size: 1207.0 B, free: 530.0 MB)
17/11/13 12:11:09 INFO spark.SparkContext: Created broadcast 242 from broadcast at DAGScheduler.scala:1004
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 186 (MapPartitionsRDD[538] at collect at utils.scala:196) (first 15 tasks are for partitions Vector(0))
17/11/13 12:11:09 INFO cluster.YarnScheduler: Adding task set 186.0 with 1 tasks
17/11/13 12:11:09 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 186.0 (TID 246, spark-1.c.halogen-order-184815.internal, executor 2, partition 0, PROCESS_LOCAL, 3699 bytes)
17/11/13 12:11:09 INFO storage.BlockManagerInfo: Added broadcast_242_piece0 in memory on spark-1.c.halogen-order-184815.internal:35671 (size: 1207.0 B, free: 530.3 MB)
17/11/13 12:11:09 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 186.0 (TID 246) in 22 ms on spark-1.c.halogen-order-184815.internal (executor 2) (1/1)
17/11/13 12:11:09 INFO scheduler.DAGScheduler: ResultStage 186 (collect at utils.scala:196) finished in 0.022 s
17/11/13 12:11:09 INFO cluster.YarnScheduler: Removed TaskSet 186.0, whose tasks have all completed, from pool
17/11/13 12:11:09 INFO scheduler.DAGScheduler: Job 154 finished: collect at utils.scala:196, took 0.031006 s
17/11/13 12:11:22 INFO spark.SparkContext: Starting job: take at NativeMethodAccessorImpl.java:-2
17/11/13 12:11:22 INFO scheduler.DAGScheduler: Got job 155 (take at NativeMethodAccessorImpl.java:-2) with 1 output partitions
17/11/13 12:11:22 INFO scheduler.DAGScheduler: Final stage: ResultStage 187 (take at NativeMethodAccessorImpl.java:-2)
17/11/13 12:11:22 INFO scheduler.DAGScheduler: Parents of final stage: List()
17/11/13 12:11:22 INFO scheduler.DAGScheduler: Missing parents: List()
17/11/13 12:11:22 INFO scheduler.DAGScheduler: Submitting ResultStage 187 (WorkerRDD[542] at RDD at rdd.scala:7), which has no missing parents
17/11/13 12:11:22 INFO storage.MemoryStore: Block broadcast_243 stored as values in memory (estimated size 35.2 KB, free 530.0 MB)
17/11/13 12:11:22 INFO storage.MemoryStore: Block broadcast_243_piece0 stored as bytes in memory (estimated size 14.4 KB, free 530.0 MB)
17/11/13 12:11:22 INFO storage.BlockManagerInfo: Added broadcast_243_piece0 in memory on 10.148.0.3:34567 (size: 14.4 KB, free: 530.0 MB)
17/11/13 12:11:22 INFO spark.SparkContext: Created broadcast 243 from broadcast at DAGScheduler.scala:1004
17/11/13 12:11:22 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 187 (WorkerRDD[542] at RDD at rdd.scala:7) (first 15 tasks are for partitions Vector(0))
17/11/13 12:11:22 INFO cluster.YarnScheduler: Adding task set 187.0 with 1 tasks
17/11/13 12:11:22 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 187.0 (TID 247, spark-1.c.halogen-order-184815.internal, executor 2, partition 0, PROCESS_LOCAL, 3488 bytes)
17/11/13 12:11:22 INFO storage.BlockManagerInfo: Added broadcast_243_piece0 in memory on spark-1.c.halogen-order-184815.internal:35671 (size: 14.4 KB, free: 530.2 MB)
17/11/13 12:11:23 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 187.0 (TID 247, spark-1.c.halogen-order-184815.internal, executor 2): java.lang.Exception: sparklyr worker rscript failure with status 255, check worker logs for details.
    at sparklyr.Rscript.init(rscript.scala:67)
    at sparklyr.WorkerRDD$$anon$2.run(rdd.scala:92)

17/11/13 12:11:23 INFO scheduler.TaskSetManager: Starting task 0.1 in stage 187.0 (TID 248, spark-1.c.halogen-order-184815.internal, executor 2, partition 0, PROCESS_LOCAL, 3488 bytes)
17/11/13 12:11:24 INFO scheduler.TaskSetManager: Lost task 0.1 in stage 187.0 (TID 248) on spark-1.c.halogen-order-184815.internal, executor 2: java.lang.Exception (sparklyr worker rscript failure with status 255, check worker logs for details.) [duplicate 1]
17/11/13 12:11:24 INFO scheduler.TaskSetManager: Starting task 0.2 in stage 187.0 (TID 249, spark-1.c.halogen-order-184815.internal, executor 2, partition 0, PROCESS_LOCAL, 3488 bytes)
17/11/13 12:11:25 INFO scheduler.TaskSetManager: Lost task 0.2 in stage 187.0 (TID 249) on spark-1.c.halogen-order-184815.internal, executor 2: java.lang.Exception (sparklyr worker rscript failure with status 255, check worker logs for details.) [duplicate 2]
17/11/13 12:11:25 INFO scheduler.TaskSetManager: Starting task 0.3 in stage 187.0 (TID 250, spark-1.c.halogen-order-184815.internal, executor 2, partition 0, PROCESS_LOCAL, 3488 bytes)
17/11/13 12:11:25 INFO scheduler.TaskSetManager: Lost task 0.3 in stage 187.0 (TID 250) on spark-1.c.halogen-order-184815.internal, executor 2: java.lang.Exception (sparklyr worker rscript failure with status 255, check worker logs for details.) [duplicate 3]
17/11/13 12:11:25 ERROR scheduler.TaskSetManager: Task 0 in stage 187.0 failed 4 times; aborting job
17/11/13 12:11:25 INFO cluster.YarnScheduler: Removed TaskSet 187.0, whose tasks have all completed, from pool
17/11/13 12:11:25 INFO cluster.YarnScheduler: Cancelling stage 187
17/11/13 12:11:25 INFO scheduler.DAGScheduler: ResultStage 187 (take at NativeMethodAccessorImpl.java:-2) failed in 3.496 s due to Job aborted due to stage failure: Task 0 in stage 187.0 failed 4 times, most recent failure: Lost task 0.3 in stage 187.0 (TID 250, spark-1.c.halogen-order-184815.internal, executor 2): java.lang.Exception: sparklyr worker rscript failure with status 255, check worker logs for details.
    at sparklyr.Rscript.init(rscript.scala:67)
    at sparklyr.WorkerRDD$$anon$2.run(rdd.scala:92)

Driver stacktrace:
17/11/13 12:11:25 INFO scheduler.DAGScheduler: Job 155 failed: take at NativeMethodAccessorImpl.java:-2, took 3.506663 s
17/11/13 12:11:25 ERROR sparklyr: Gateway (37351) failed calling take on 699

All I can seem to fathom is that the spark job is failing somewhere in the last stage so probably merging the output from the different workers? Can anyone help find what may be the problem?


来源: stackexchange

问:
How to parse a column while referring to values from other multiple columns?
I have this sample dataframe where column a to d are reference columns and column x1-3 need to be parsed and plugged with new values.

Here is the code to re-produce the data frame:

```<code>    df1 <- data_frame(a = c(0,1,0,1), b = c(0,0,1,1), c = c(0,0,0,0), d =
      c(1,0,0,1), x1= c(NA, NA, NA, NA), x2= c(NA, NA, NA, NA), x3= c(NA, NA, NA, NA))
</code>```

I want to give new values to x1 -x3 based on different value combination from column a, b, c, d. My pseudocode is as follows:

```<code>    for df1[ , "x1"]:
        if a = 1: then return 1
        else: return 0
    for df1[ , "x2"]:
        if a = 1 & b = 1: then return 1
        else: return 0
    for df1[ , "x3"]:
        all conditions: return 1
</code>```

Ideally, all the values in x1 and x2 will be changed according to their given conditions. X3 should be filled with 1 no matter what. Can anyone suggest a efficient way to loop & parse through those columns, please?

答:
You don't need loops:

```<code>df1$x1 <- df1$a
df1$x2 <- as.integer(df1$a & df1$b)
df1$x3 <- 1
</code>```

Result:

```<code>  a b c d x1 x2 x3
1 0 0 0 1  0  0  1
2 1 0 0 0  1  0  1
3 0 1 0 0  0  0  1
4 1 1 0 1  1  1  1
</code>```

Edit:

If columns a-d are not binary values (0 or 1) you still can use the same expressions to create columns x1-3. Let's say you have this data frame:

```<code>  a b c d x1 x2 x3
1 0 0 1 5 NA NA NA
2 3 9 2 1 NA NA NA
3 4 2 3 5 NA NA NA
4 2 1 4 1 NA NA NA
</code>```

And your conditions are:

```<code>x1 = 1 if (b >= 2) and (d < 4) 0 otherwise
x2 = 1 if (a > b) and (b < d) 0 otherwise
x3 = always 1
</code>```

You can use the same methodology:

```<code>df1$x1 <- as.integer(df1$b >= 2 & df1$d < 4)
df1$x2 <- as.integer(df1$a > df1$b & df1$b < df1$d)
df1$x3 <- 1
</code>```

Result:

```<code>  a b c d x1 x2 x3
1 0 0 1 5  0  0  1
2 3 9 2 1  1  0  1
3 4 2 3 5  0  1  1
4 2 1 4 1  0  0  1
</code>```

ContentLicense: CC BY-SA 3.0

Body:
I have this sample dataframe where column a to d are reference columns and column x1-3 need to be parsed and plugged with new values.

Here is the code to re-produce the data frame:

    df1 <- data_frame(a = c(0,1,0,1), b = c(0,0,1,1), c = c(0,0,0,0), d =
      c(1,0,0,1), x1= c(NA, NA, NA, NA), x2= c(NA, NA, NA, NA), x3= c(NA, NA, NA, NA))

I want to give new values to x1 -x3 based on different value combination from column a, b, c, d. My pseudocode is as follows:

    for df1[ , "x1"]:
        if a = 1: then return 1
        else: return 0
    for df1[ , "x2"]:
        if a = 1 & b = 1: then return 1
        else: return 0
    for df1[ , "x3"]:
        all conditions: return 1

Ideally, all the values in x1 and x2 will be changed according to their given conditions. X3 should be filled with 1 no matter what. Can anyone suggest a efficient way to loop & parse through those columns, please?

Body:
You don't need loops:

df1$x1 <- df1$a
df1$x2 <- as.integer(df1$a & df1$b)
df1$x3 <- 1

Result:

  a b c d x1 x2 x3
1 0 0 0 1  0  0  1
2 1 0 0 0  1  0  1
3 0 1 0 0  0  0  1
4 1 1 0 1  1  1  1

Edit:

If columns a-d are not binary values (0 or 1) you still can use the same expressions to create columns x1-3. Let's say you have this data frame:

  a b c d x1 x2 x3
1 0 0 1 5 NA NA NA
2 3 9 2 1 NA NA NA
3 4 2 3 5 NA NA NA
4 2 1 4 1 NA NA NA

And your conditions are:

x1 = 1 if (b >= 2) and (d < 4) 0 otherwise
x2 = 1 if (a > b) and (b < d) 0 otherwise
x3 = always 1

You can use the same methodology:

df1$x1 <- as.integer(df1$b >= 2 & df1$d < 4)
df1$x2 <- as.integer(df1$a > df1$b & df1$b < df1$d)
df1$x3 <- 1

Result:

  a b c d x1 x2 x3
1 0 0 1 5  0  0  1
2 3 9 2 1  1  0  1
3 4 2 3 5  0  1  1
4 2 1 4 1  0  0  1


问:
Is there a single call method of splitting a list (iterable) into two parts?
Is there a built-in or Pythonic way to do this? Or maybe my question should be: Is there a better way to write the dictionary comprehension example at the end of this question?

```<code>def split_list(list, index):
    return list[:index], list[index:]
</code>```

I find myself needing to do this in list comprehensions where I get lists from some generator and want to split the lists into two pieces. For example, a text file like the following:

```<code>txt_file = """key1 value value value value
key2 value value value value
"""
import io
{k:v
 for line in io.StringIO(txt_file)
 for items in (line.strip().split(),)
 for k, v in ((items[0], items[1:]),)}
</code>```

Output:

```<code>{'key1': ['value', 'value', 'value', 'value'],
 'key2': ['value', 'value', 'value', 'value']}
</code>```

答:
The only comment I'd make is that there's no real reason for your last for in that comprehension.

```<code>{items[0]:items[1:]
 for line in io.StringIO(txt_file)
 for items in (line.strip().split(),)}
</code>```

And I don't know of a "better" way to split a list in two.

Body:
Is there a built-in or Pythonic way to do this? Or maybe my question should be: Is there a better way to write the dictionary comprehension example at the end of this question?

def split_list(list, index):
    return list[:index], list[index:]

I find myself needing to do this in list comprehensions where I get lists from some generator and want to split the lists into two pieces. For example, a text file like the following:

txt_file = """key1 value value value value
key2 value value value value
"""
import io
{k:v
 for line in io.StringIO(txt_file)
 for items in (line.strip().split(),)
 for k, v in ((items[0], items[1:]),)}

Output:

{'key1': ['value', 'value', 'value', 'value'],
 'key2': ['value', 'value', 'value', 'value']}

Body:
The only comment I'd make is that there's no real reason for your last for in that comprehension.

{items[0]:items[1:]
 for line in io.StringIO(txt_file)
 for items in (line.strip().split(),)}

And I don't know of a "better" way to split a list in two.


问:
Divide and conquer - k element of unsorted array
I am having a problem trying to implement an algorithm using Divide and conquer.

Given an unsorted array T v[] find de v[k] element of that array as if the array was sorted but without sorting the array v.

For example, if k = 3 and v = {2, -1, -6, 7, 4} the k element of that array is 2.

Since I can't edit the passed array I can't think another way to sort the array without saving it on another local variable or trying to divide the array like quicksort and returning the position where the last element of v should be.

And if it helps, this is my code:

```<code>public static <T extends Comparable<? super T>> T kesimoRec(T v[], int izq, int der, int k) {
    if(izq < der){
    int inf = izq-1;
    T pivote = v[der];
       for(int i = izq; i < der; ++i){
           if(pivote.compareTo(v[i]) >= 0){
               ++inf;
           }
       }
       ++inf;
       if(inf > izq + k-1){
           return (kesimoRec(v, izq, inf-1, k));
       }else if( inf < izq + k-1){
           return (kesimoRec(v, inf+1, der, k-inf+izq-1));
       }else{
           return v[inf];
       }
    }else{
        return v[der];
    }
}
</code>```

答:
With divide and conquer</p>

```<code>public static <T extends Comparable<? super T>> T kesimoRec(T v[], int izq,
            int der, int k) {
        if (izq == der) {
            return v[izq];
        }
        int pivote = partir(v, v[der], izq, der);
        if (k == pivote) {
            return v[k];
        } else if (k < pivote) {
            return kesimoRec(v, izq, pivote - 1, k);
        } else {
            return kesimoRec(v, pivote + 1, der, k);
        }
    }
</code>```

Body:
I am having a problem trying to implement an algorithm using Divide and conquer.

Given an unsorted array T v[] find de v[k] element of that array as if the array was sorted but without sorting the array v.

For example, if k = 3 and v = {2, -1, -6, 7, 4} the k element of that array is 2.

Since I can't edit the passed array I can't think another way to sort the array without saving it on another local variable or trying to divide the array like quicksort and returning the position where the last element of v should be.

And if it helps, this is my code:

public static <T extends Comparable<? super T>> T kesimoRec(T v[], int izq, int der, int k) {
    if(izq < der){
    int inf = izq-1;
    T pivote = v[der];
       for(int i = izq; i < der; ++i){
           if(pivote.compareTo(v[i]) >= 0){
               ++inf;
           }
       }
       ++inf;
       if(inf > izq + k-1){
           return (kesimoRec(v, izq, inf-1, k));
       }else if( inf < izq + k-1){
           return (kesimoRec(v, inf+1, der, k-inf+izq-1));
       }else{
           return v[inf];
       }
    }else{
        return v[der];
    }
}

Body:
With divide and conquer

public static <T extends Comparable<? super T>> T kesimoRec(T v[], int izq,
            int der, int k) {
        if (izq == der) {
            return v[izq];
        }
        int pivote = partir(v, v[der], izq, der);
        if (k == pivote) {
            return v[k];
        } else if (k < pivote) {
            return kesimoRec(v, izq, pivote - 1, k);
        } else {
            return kesimoRec(v, pivote + 1, der, k);
        }
    }


问:
Is it possible to convert byte array/stream to files such as Word,Excel, PDF in JavaScript
I have a query regarding file operations using JavaScript-

Scenario - My JS function calls a wcf service which returns the file content in the form of byte array or stream and the mime type. This byte array/stream needs to be converted to a file and which will be downloaded on  user's machine.

Reference code -

```<code>var arr = "This is test content";
var byteArray = new Uint8Array(arr);
var a = window.document.createElement('a');

a.href = window.URL.createObjectURL(new Blob([byteArray], {
    type: 'text/plain'
}));
a.download = "Test";

document.body.appendChild(a);
a.click();

document.body.removeChild(a);
</code>```

The code works for only text files. Files with mime type other than text are corrupted.
I understand that file operations are severly restricted at client side, but just to confirm - Is there anyway to convert byte array/stream to files such as Word,Excel, PDF and etc ?

答:
I accomplish a similar goal with this.  Pick up from where you have your byteArray, and try this:

```<code>    var byteArray = new Uint8Array(arrayBuffer);
    var a = window.document.createElement('a');
    a.href = window.URL.createObjectURL(new Blob([byteArray], { type:'application/octet-stream' }));

    // supply your own fileName here...
    a.download = "YourFileName.XLSX";

    document.body.appendChild(a)
    a.click();
    document.body.removeChild(a)
</code>```

Setting contentType to "application/octet-stream" will accommodate any binary file type.

Body:
I have a query regarding file operations using JavaScript-

Scenario - My JS function calls a wcf service which returns the file content in the form of byte array or stream and the mime type. This byte array/stream needs to be converted to a file and which will be downloaded on  user's machine.

Reference code -

var arr = "This is test content";
var byteArray = new Uint8Array(arr);
var a = window.document.createElement('a');

a.href = window.URL.createObjectURL(new Blob([byteArray], {
    type: 'text/plain'
}));
a.download = "Test";

document.body.appendChild(a);
a.click();

document.body.removeChild(a);

The code works for only text files. Files with mime type other than text are corrupted.
I understand that file operations are severly restricted at client side, but just to confirm - Is there anyway to convert byte array/stream to files such as Word,Excel, PDF and etc ?

Body:
I accomplish a similar goal with this.  Pick up from where you have your byteArray, and try this:

    var byteArray = new Uint8Array(arrayBuffer);
    var a = window.document.createElement('a');
    a.href = window.URL.createObjectURL(new Blob([byteArray], { type:'application/octet-stream' }));

    // supply your own fileName here...
    a.download = "YourFileName.XLSX";

    document.body.appendChild(a)
    a.click();
    document.body.removeChild(a)

Setting contentType to "application/octet-stream" will accommodate any binary file type.

ContentLicense: CC BY-SA 4.0


问:
How to Import SQL Data Directly Into Encompass
I am attempting to create a small PHP file that will have the capabilities to import data pulled from a MySQL schema directly into an SQL table used with an Encompass account.

Encompass is a program by Ellie Mae that is used for housing loans. To simplify, it pulls in application forms to be utilized with various functions. It stores these forms in an SQL server. I understand that specifying this application may narrow the number of people this question is applicable to, but because it is a widely used application in this field and there is a huge lack of presence online for developers working with this, this question can be relevant to others.

I am making this project because currently, after my team receives a form application, it is sent and stored in a cloud-based system, where they then download the application and manually import it into Encompass. This takes a ton of time and is very error prone.

With this tool, it will pull any new forms to a display, then, after being approved, will update the table that corresponds to the Encompass account.

Unfortunately, development support is just about nonexistent online. I have no idea how to target the table used by our Encompass account to retrieve data. The online SDK makes no mention of how to utilize SQL, just that it uses an SQL server. This blog post is the closest thing I found to what I'm looking for, but I don't understand how to target the database.

I am avoiding using any of the objects mentioned in the SDK since I want to make a very lightweight application many of the not-so-technically-savvy people could use, and could be easily integrated into other projects in the future.

As a small example of what I'm doing, here is the code that targets my database. I would like to make a similar connection to the Encompass one, but I have no idea how to find out what the database name is.

```<code>     $link = mysqli_connect("[server]", "[username]", "[password]", "[input_schema]");
     $query = "SELECT * FROM input_schema.input_table";
     $result = mysqli_query($link, $query);

     while ($row = mysqli_fetch_array($result)) {
         ?>
         <tr id="row">
             <td><?php echo $row['ID']; ?></td>
             <td><?php echo $row['firstName']; ?></td>
             <td><?php echo $row['lastName']; ?></td>
         </tr>
</code>```

If there is a more broadly-reaching answer to find out an applications SQL information, such as the database, I'd love to hear it as well. Thank you very much for the help.

答: Unfortunately there is no way to directly access the Ellie Mae Encompass360 application's database directly in the way you are describing. In the current .NET Winforms smart client incarnation of the product, you MUST use the Encompass SDK to be able to programmatically read/update data. And utilizing the SDK is a quite involved endeavor. Not that it is rocket science in an absolute sense, but I found it very challenging trying to find any information to get started with the basics of the SDK. You may want to do some research on what is called "Encompass NG", which is the next generation of the Ellie Mae Encompass360 application that is a cloud web application that is all angularjs/api based. It is in development now and is expected to be generally available in end of 2018 timeframe. (Encompass360 clients would have to migrate over to it as well.) In that incarnation of the application, there will be APIs to easily load new information into Encompass such as creating loans, contacts, etc. Note that as far as I know, Ellie Mae NEVER intends to allow anyone to directly access the raw application database itself.</p>

Body:
I am attempting to create a small PHP file that will have the capabilities to import data pulled from a MySQL schema directly into an SQL table used with an Encompass account.

Encompass is a program by Ellie Mae that is used for housing loans. To simplify, it pulls in application forms to be utilized with various functions. It stores these forms in an SQL server. I understand that specifying this application may narrow the number of people this question is applicable to, but because it is a widely used application in this field and there is a huge lack of presence online for developers working with this, this question can be relevant to others.

I am making this project because currently, after my team receives a form application, it is sent and stored in a cloud-based system, where they then download the application and manually import it into Encompass. This takes a ton of time and is very error prone.

With this tool, it will pull any new forms to a display, then, after being approved, will update the table that corresponds to the Encompass account.

Unfortunately, development support is just about nonexistent online. I have no idea how to target the table used by our Encompass account to retrieve data. The online SDK makes no mention of how to utilize SQL, just that it uses an SQL server. This blog post is the closest thing I found to what I'm looking for, but I don't understand how to target the database.

I am avoiding using any of the objects mentioned in the SDK since I want to make a very lightweight application many of the not-so-technically-savvy people could use, and could be easily integrated into other projects in the future.

As a small example of what I'm doing, here is the code that targets my database. I would like to make a similar connection to the Encompass one, but I have no idea how to find out what the database name is.

     $link = mysqli_connect("[server]", "[username]", "[password]", "[input_schema]");
     $query = "SELECT * FROM input_schema.input_table";
     $result = mysqli_query($link, $query);

     while ($row = mysqli_fetch_array($result)) {
         ?>

             <?php echo $row['ID']; ?>
             <?php echo $row['firstName']; ?>
             <?php echo $row['lastName']; ?>

If there is a more broadly-reaching answer to find out an applications SQL information, such as the database, I'd love to hear it as well. Thank you very much for the help.


问:
Selenium opens new tab after SendKeys
I'm trying to test my web app with selenium. For testing i'm using Selenium 3.5.2, chromeWebDriver 2.33 and chrome version 62.

The problem is that after i create action and call sendKeys, chrome opens help tab. I tried on differents web pages, even on youtube. Here's code:

```<code>Actions actions = new Actions(driver);
WebElement element = driver.findElement(By.Id("search_field"));
actions.sendKeys(element, "A").build().perform();
</code>```

After perform() chrome opens a new help tab.

Thank you for answers!

答: updating chrome driver to 2.33 worked for me.</p>

Title: Selenium opens new tab after SendKeys

Body:
I'm trying to test my web app with selenium. For testing i'm using Selenium 3.5.2, chromeWebDriver 2.33 and chrome version 62.

The problem is that after i create action and call sendKeys, chrome opens help tab. I tried on differents web pages, even on youtube. Here's code:

Actions actions = new Actions(driver);
WebElement element = driver.findElement(By.Id("search_field"));
actions.sendKeys(element, "A").build().perform();

After perform() chrome opens a new help tab.

Thank you for answers!


问:
Programmatically position ggplot labels
I am creating a lot of charts programmatically in R using ggplot2 and have everything working perfectly except the position of bar labels.

This requires inputs of the plot height, y axis scale and text size.

Example (stripped down) plot code:

```<code>testInput <- data.frame("xAxis" = c("first", "second", "third"), "yAxis" = c(20, 200, 60))

# Changeable variables
yMax <- 220
plotHeight <- 5
textSize <- 4

# Set up labels
geomTextList <- {
  textHeightRatio <- textSize / height
  maxHeightRatio <- yMax / height

  values <- testInput[["yAxis"]]

  ### THIS IS THE FORMULA NEEDING UPDATING
  testInput[["labelPositions"]] <- values + 5 # # Should instead be a formula eg. (x * height) + (y * textSize) + (z * yMax)?
  list(
    ggplot2::geom_text(data = testInput, ggplot2::aes_string(x = "xAxis", y = "labelPositions", label = "yAxis"), hjust = 0.5, size = textSize)
  )
}

# Create plot
outputPlot <- ggplot2::ggplot(testInput) +
  ggplot2::geom_bar(data = testInput, ggplot2::aes_string(x = "xAxis", y = "yAxis"), stat = "identity", position = "dodge", width = 0.5) +
  geomTextList +
  ggplot2::scale_y_continuous(breaks = seq(0, yMax, yInterval), limits = c(0, yMax))

ggplot2::ggsave(filename = "test.png", plot = outputPlot, width = 4, height = plotHeight, device = "png")
</code>```

I have tried various combinations of coefficients for the formula, but suspect that at leat one of the factors isn't linear.  If this is purely a statistical problem, I could take it to Cross-Validation, but I wondered whether anyone had already solved this?

答:
If your problem is with offsetting the text to not overlap the bar while dealing with varying text sizes, just use vjust which is already proportional to the text size. A value of 0 will make the bottom of the text touch the bar, and a small negative value will give you some space between them:

```<code>testInput <- data.frame("xAxis" = c("first", "second", "third"), "yAxis" = c(20, 200, 60))

# Changeable variables
yMax <- 220
plotHeight <- 5
textSize <- 4

# Set up labels
geomTextList <- {

  values <- testInput[["yAxis"]]

  testInput[["labelPositions"]] <- values # Use the exact value
  list(
    ggplot2::geom_text(
      data = testInput,
      # vjust provides proportional offset
      ggplot2::aes_string(x = "xAxis", y = "labelPositions", label = "yAxis"),
      hjust = 0.5, vjust = -0.15,  size = textSize
    )
  )
}

# Create plot
outputPlot <- ggplot2::ggplot(testInput) +
  ggplot2::geom_bar(data = testInput, ggplot2::aes_string(x = "xAxis", y = "yAxis"), stat = "identity", position = "dodge", width = 0.5) +
  geomTextList +
  ggplot2::scale_y_continuous(limits = c(0, yMax))
ggplot2::ggsave(filename = "test.png", plot = outputPlot, width = 4, height = plotHeight, device = "png")
</code>```

Title: Programmatically position ggplot labels

Body:
I am creating a lot of charts programmatically in R using ggplot2 and have everything working perfectly except the position of bar labels.

This requires inputs of the plot height, y axis scale and text size.

Example (stripped down) plot code:

testInput <- data.frame("xAxis" = c("first", "second", "third"), "yAxis" = c(20, 200, 60))

# Changeable variables
yMax <- 220
plotHeight <- 5
textSize <- 4

# Set up labels
geomTextList <- {
  textHeightRatio <- textSize / height
  maxHeightRatio <- yMax / height

  values <- testInput[["yAxis"]]

  ### THIS IS THE FORMULA NEEDING UPDATING
  testInput[["labelPositions"]] <- values + 5 # # Should instead be a formula eg. (x * height) + (y * textSize) + (z * yMax)?
  list(
    ggplot2::geom_text(data = testInput, ggplot2::aes_string(x = "xAxis", y = "labelPositions", label = "yAxis"), hjust = 0.5, size = textSize)
  )
}

# Create plot
outputPlot <- ggplot2::ggplot(testInput) +
  ggplot2::geom_bar(data = testInput, ggplot2::aes_string(x = "xAxis", y = "yAxis"), stat = "identity", position = "dodge", width = 0.5) +
  geomTextList +
  ggplot2::scale_y_continuous(breaks = seq(0, yMax, yInterval), limits = c(0, yMax))

ggplot2::ggsave(filename = "test.png", plot = outputPlot, width = 4, height = plotHeight, device = "png")

I have tried various combinations of coefficients for the formula, but suspect that at leat one of the factors isn't linear.  If this is purely a statistical problem, I could take it to Cross-Validation, but I wondered whether anyone had already solved this?

Body:
If your problem is with offsetting the text to not overlap the bar while dealing with varying text sizes, just use vjust which is already proportional to the text size. A value of 0 will make the bottom of the text touch the bar, and a small negative value will give you some space between them:

testInput <- data.frame("xAxis" = c("first", "second", "third"), "yAxis" = c(20, 200, 60))

# Changeable variables
yMax <- 220
plotHeight <- 5
textSize <- 4

# Set up labels
geomTextList <- {

  values <- testInput[["yAxis"]]

  testInput[["labelPositions"]] <- values # Use the exact value
  list(
    ggplot2::geom_text(
      data = testInput,
      # vjust provides proportional offset
      ggplot2::aes_string(x = "xAxis", y = "labelPositions", label = "yAxis"),
      hjust = 0.5, vjust = -0.15,  size = textSize
    )
  )
}

# Create plot
outputPlot <- ggplot2::ggplot(testInput) +
  ggplot2::geom_bar(data = testInput, ggplot2::aes_string(x = "xAxis", y = "yAxis"), stat = "identity", position = "dodge", width = 0.5) +
  geomTextList +
  ggplot2::scale_y_continuous(limits = c(0, yMax))
ggplot2::ggsave(filename = "test.png", plot = outputPlot, width = 4, height = plotHeight, device = "png")


问:
How to group ranged values using SQL Server
I have a table of values like this

```<code>978412, 400
978813, 20
978834, 50
981001, 20
</code>```

As you can see the second number when added to the first is 1 number before the next in the sequence. The last number is not in the range (doesnt follow a direct sequence, as in the next value). What I need is a CTE (yes, ideally) that will output this

```<code>978412, 472
981001, 20
</code>```

The first row contains the start number of the range then the sum of the nodes within. The next row is the next range which in this example is the same as the original data.

答:
From the article that Josh posted, here's my take (tested and working):</p>

```<code>SELECT
    MAX(t1.gapID) as gapID,
    t2.gapID-MAX(t1.gapID)+t2.gapSize as gapSize
    -- max(t1) is the specific lower bound of t2 because of the group by.
FROM
  ( -- t1 is the lower boundary of an island.
    SELECT gapID
    FROM gaps tbl1
    WHERE
      NOT EXISTS(
        SELECT *
        FROM gaps tbl2
        WHERE tbl1.gapID = tbl2.gapID + tbl2.gapSize + 1
      )
  ) t1
  INNER JOIN ( -- t2 is the upper boundary of an island.
    SELECT gapID, gapSize
    FROM gaps tbl1
    WHERE
      NOT EXISTS(
        SELECT * FROM gaps tbl2
        WHERE tbl2.gapID = tbl1.gapID + tbl1.gapSize + 1
      )
  ) t2 ON t1.gapID <= t2.gapID -- For all t1, we get all bigger t2 and opposite.
GROUP BY t2.gapID, t2.gapSize
</code>```

OwnerDisplayName: Coolcoder

ContentLicense: CC BY-SA 2.5

Body:
I have a table of values like this

978412, 400
978813, 20
978834, 50
981001, 20

As you can see the second number when added to the first is 1 number before the next in the sequence. The last number is not in the range (doesnt follow a direct sequence, as in the next value). What I need is a CTE (yes, ideally) that will output this

978412, 472
981001, 20

The first row contains the start number of the range then the sum of the nodes within. The next row is the next range which in this example is the same as the original data.

LastEditorDisplayName: Coolcoder

Body:
From the article that Josh posted, here's my take (tested and working):

SELECT
    MAX(t1.gapID) as gapID,
    t2.gapID-MAX(t1.gapID)+t2.gapSize as gapSize
    -- max(t1) is the specific lower bound of t2 because of the group by.
FROM
  ( -- t1 is the lower boundary of an island.
    SELECT gapID
    FROM gaps tbl1
    WHERE
      NOT EXISTS(
        SELECT *
        FROM gaps tbl2
        WHERE tbl1.gapID = tbl2.gapID + tbl2.gapSize + 1
      )
  ) t1
  INNER JOIN ( -- t2 is the upper boundary of an island.
    SELECT gapID, gapSize
    FROM gaps tbl1
    WHERE
      NOT EXISTS(
        SELECT * FROM gaps tbl2
        WHERE tbl2.gapID = tbl1.gapID + tbl1.gapSize + 1
      )
  ) t2 ON t1.gapID <= t2.gapID -- For all t1, we get all bigger t2 and opposite.
GROUP BY t2.gapID, t2.gapSize

OwnerDisplayName: Loki

LastEditorDisplayName: Loki


问:
Core Data (Mac) How to implement auto increment on an ID field
I was wondering if anyone can point me in the direction of how to set up auto increment on an attribute in a coredata entity. I know the indexed check box just makes the field indexed for searching and lookup speeds.

I also know that I will have to do this in code, but I'm not sure on how to get the last/highest ID to set the new objects ID. I found how to do this years ago but can't find it again.

I have a very limited knowledge of Objective-C and even more so of coredata. Also if someone could suggest how to make sure that a field is unique, such as the name field as well that would be great.

My code is here if you want to have a look at what I have done so far. The project is a basic database that contains parts and there locations and also tools. It's only partly finished, basically a small database for my person electronics kit. I'm sure there are apps that will do this but its always better when you make it your self

答:
I'm not quite sure what the id does in your app, but my first observation is that all the entities in your data model have both a name and an id, so they should probably have an abstract super-entity to contain name and id. Other than that, you can retrieve the max id in a set like this:

```<code>Category *cat = /* get a category  */;
NSSet *parts = cat.parts;
NSNumber *maxID = [parts valueForKeyPath:@"@max.id"];
</code>```

for the id's of a category you could perform a fetch request for all the categories and then use this technique on the array that is returned. I don't know if you want to keep the id's of each model object unique or you want id's to start with 0 for the first object added to each entity.

To make sure a field is unique you can use managed object validation. this is described in the Core Data Programming Guide and in more detail in the Model Object Implementation Guide. Making sure that a name is unique would require you to test it against all the names in a validateName:error: method. making an abstract super entity will make this a lot easier. To do it you get all the names of objects in the abstract super entity and test them against the value to be validated.

Edit: If you use the abstract super entity then you should get the max id of the objects in the abstract super entity if you want them to be unique. Also as someone else pointed out the objectID of a managed object is a unique identifier for it that will always be the same, but looking at the app I don't think that's really what you want, because they will be computer numbers rather than the friendlier results of incrementing the highest known id when a new object is created. Another thing you should set the objet id in awakeFromInsert so that it is only called when the object is created.

Body:
I'm not quite sure what the id does in your app, but my first observation is that all the entities in your data model have both a name and an id, so they should probably have an abstract super-entity to contain name and id. Other than that, you can retrieve the max id in a set like this:

Category *cat = /* get a category  */;
NSSet *parts = cat.parts;
NSNumber *maxID = [parts valueForKeyPath:@"@max.id"];

for the id's of a category you could perform a fetch request for all the categories and then use this technique on the array that is returned. I don't know if you want to keep the id's of each model object unique or you want id's to start with 0 for the first object added to each entity.

To make sure a field is unique you can use managed object validation. this is described in the Core Data Programming Guide and in more detail in the Model Object Implementation Guide. Making sure that a name is unique would require you to test it against all the names in a validateName:error: method. making an abstract super entity will make this a lot easier. To do it you get all the names of objects in the abstract super entity and test them against the value to be validated.

Edit: If you use the abstract super entity then you should get the max id of the objects in the abstract super entity if you want them to be unique. Also as someone else pointed out the objectID of a managed object is a unique identifier for it that will always be the same, but looking at the app I don't think that's really what you want, because they will be computer numbers rather than the friendlier results of incrementing the highest known id when a new object is created. Another thing you should set the objet id in awakeFromInsert so that it is only called when the object is created.


问:
Angular / Ionic - Handle JSON "error" property
I'm using TypeScript, Angular 5.0. My backend response is summarized in this interface:

```<code>export interface JSONResponse {
  error?: {
    code: number,
    message: string
  };
  data?: {};
}
</code>```

The function in my service to get the data from the server is:

```<code>httpGet(url: string, httpParams: HttpParams) {
  return new Promise((resolve, reject) => {
    this.http.get<LoginResponse>(url, {params: httpParams})
      .subscribe(res => {
        resolve(res.data);
      }, err => {
        // handle http get related errors here
        console.log(err);
      });
  });
}
</code>```

And then the component that consume this service to render the template:

```<code>buttonClicked(event) {
  this.myService.httpGet('myAPIurl', someRequestParams)
    .then((data) => {
      this.myData = data;
    }, (err) => {
      console.log(err);
    });
}
</code>```

Where would be the place to handle the error checking of the response from my backend?

That is, if property data is present, the response is successful and do the proper processing of my data; if property error is present, I notify the user with the code/message error.

答:
If you need to do it in a more generic way you can use IonicErrorHandler.

The IonicErrorHandler intercepts the default Console error handling
  and displays runtime errors as an overlay when using Ionic's Dev Build
  Server.

```<code>import { NgModule, ErrorHandler } from '@angular/core';
import { IonicErrorHandler } from 'ionic-angular';

@NgModule({
  providers: [{ provide: ErrorHandler, useClass: IonicErrorHandler }]
})
class AppModule {}
</code>```

If you want to customise it you can do that too.

```<code>class MyErrorHandler implements ErrorHandler {
  handleError(err: any): void {
    // do something with the error
  }
}

@NgModule({
  providers: [{ provide: ErrorHandler, useClass: MyErrorHandler }]
})
class AppModule {}
</code>```

Body:
I'm using TypeScript, Angular 5.0. My backend response is summarized in this interface:

export interface JSONResponse {
  error?: {
    code: number,
    message: string
  };
  data?: {};
}

The function in my service to get the data from the server is:

httpGet(url: string, httpParams: HttpParams) {
  return new Promise((resolve, reject) => {
    this.http.get<LoginResponse>(url, {params: httpParams})
      .subscribe(res => {
        resolve(res.data);
      }, err => {
        // handle http get related errors here
        console.log(err);
      });
  });
}

And then the component that consume this service to render the template:

buttonClicked(event) {
  this.myService.httpGet('myAPIurl', someRequestParams)
    .then((data) => {
      this.myData = data;
    }, (err) => {
      console.log(err);
    });
}

Where would be the place to handle the error checking of the response from my backend?

That is, if property data is present, the response is successful and do the proper processing of my data; if property error is present, I notify the user with the code/message error.

Body:
If you need to do it in a more generic way you can use IonicErrorHandler.

The IonicErrorHandler intercepts the default Console error handling
  and displays runtime errors as an overlay when using Ionic's Dev Build
  Server.

import { NgModule, ErrorHandler } from '@angular/core';
import { IonicErrorHandler } from 'ionic-angular';

@NgModule({
  providers: [{ provide: ErrorHandler, useClass: IonicErrorHandler }]
})
class AppModule {}

If you want to customise it you can do that too.

class MyErrorHandler implements ErrorHandler {
  handleError(err: any): void {
    // do something with the error
  }
}

@NgModule({
  providers: [{ provide: ErrorHandler, useClass: MyErrorHandler }]
})
class AppModule {}


问:
Why do scipy and numpy fft plots look different?
I am currently doing some spectrum analysis for a piece of coursework, though we haven't been explicitly taught Fourier transforms yet. I have been playing around with the various fft algorithms in scipy and numpy on some data that I know what the answer should look like

In this case its an AM signal at 8kHz carrier frequency and 1kHz modulated sine wave on top so should have 3 clear peaks on an fft

When applying scipy.fftpack.rfft and numpy.fft.rfft I get the following plots respectively:

Scipy:

Numpy:

While the shape of the 2 FFTs are roughly the same with the correct ratios between the peaks, the numpy one looks much smoother, whereas the scipy one has slightly smaller max peaks, and has much more noise.

I'm assuming this is largely down to different applications of a Discrete Fourier Transform algorithm, and have seen other articles about how the scipy implementation is faster in run time. But I was wandering what it is that specifically causes the difference, and which one is actually more accurate?

EDIT: Code used to generate plots:

```<code>data = pd.read_csv("./Waveforms/AM waveform Sine.csv", sep = ',', dtype = float)

data = data.as_matrix()
time = data[:,0]
voltage = data[:,1]/data[:,1].max() # normalise the values

#scipy plot:
plt.figure()
magnitude =  scipy.fftpack.rfft(voltage)
freq = scipy.fftpack.rfftfreq(len(time),np.diff(time)[0])
plt.figure()
plt.plot(freq, np.absolute(magnitude), lw = 1)
plt.ylim(0,2500)
plt.xlim(0,15)

#numpy plot
magnitude = np.fft.rfft(voltage)
freq = np.fft.rfftfreq(len(time),np.diff(time)[0])

plt.figure()
plt.plot(freq, np.absolute(magnitude), lw = 1)
plt.ylim(0,2500)
plt.xlim(0,15)
</code>```

答:
From NumPy's doc for rfft:

Returns:

out : complex ndarray

The truncated or zero-padded input, transformed along the axis
  indicated by axis, or the last one if axis is not specified. If n is
  even, the length of the transformed axis is (n/2)+1. If n is odd, the
  length is (n+1)/2.

It is not written explicitly but the "transformed data" is here complex.

From SciPy's doc for rfft

z : real ndarray

The returned real array contains:

```<code>[y(0),Re(y(1)),Im(y(1)),...,Re(y(n/2))]              if n is even
[y(0),Re(y(1)),Im(y(1)),...,Re(y(n/2)),Im(y(n/2))]   if n is odd
</code>```

Conclusion: the storage is different.

For a starter, look at the length of magnitude, it will be different in both cases. I give an example below for clarity:

```<code>In [33]: data = np.random.random(size=8)

In [34]: np.fft.rfft(data)
Out[34]:
array([ 3.33822983+0.j        ,  0.15879369+0.48542266j,
        0.00614876+0.03590621j, -0.67376592-0.69793372j,  1.51730861+0.j        ])

In [35]: scipy.fftpack.rfft(data)
Out[35]:
array([ 3.33822983,  0.15879369,  0.48542266,  0.00614876,  0.03590621,
       -0.67376592, -0.69793372,  1.51730861])
</code>```

The first element in both cases is the so-called "DC component" (the mean of the signal).

Then, you can recognize in the SciPy version the succession of real and imaginary parts of the NumPy version.

Body:
I am currently doing some spectrum analysis for a piece of coursework, though we haven't been explicitly taught Fourier transforms yet. I have been playing around with the various fft algorithms in scipy and numpy on some data that I know what the answer should look like

In this case its an AM signal at 8kHz carrier frequency and 1kHz modulated sine wave on top so should have 3 clear peaks on an fft

When applying scipy.fftpack.rfft and numpy.fft.rfft I get the following plots respectively:

Scipy:

Numpy:

While the shape of the 2 FFTs are roughly the same with the correct ratios between the peaks, the numpy one looks much smoother, whereas the scipy one has slightly smaller max peaks, and has much more noise.

I'm assuming this is largely down to different applications of a Discrete Fourier Transform algorithm, and have seen other articles about how the scipy implementation is faster in run time. But I was wandering what it is that specifically causes the difference, and which one is actually more accurate?

EDIT: Code used to generate plots:

data = pd.read_csv("./Waveforms/AM waveform Sine.csv", sep = ',', dtype = float)

data = data.as_matrix()
time = data[:,0]
voltage = data[:,1]/data[:,1].max() # normalise the values

#scipy plot:
plt.figure()
magnitude =  scipy.fftpack.rfft(voltage)
freq = scipy.fftpack.rfftfreq(len(time),np.diff(time)[0])
plt.figure()
plt.plot(freq, np.absolute(magnitude), lw = 1)
plt.ylim(0,2500)
plt.xlim(0,15)

#numpy plot
magnitude = np.fft.rfft(voltage)
freq = np.fft.rfftfreq(len(time),np.diff(time)[0])

plt.figure()
plt.plot(freq, np.absolute(magnitude), lw = 1)
plt.ylim(0,2500)
plt.xlim(0,15)

Body:
From NumPy's doc for rfft:

Returns:

out : complex ndarray

The truncated or zero-padded input, transformed along the axis
  indicated by axis, or the last one if axis is not specified. If n is
  even, the length of the transformed axis is (n/2)+1. If n is odd, the
  length is (n+1)/2.

It is not written explicitly but the "transformed data" is here complex.

From SciPy's doc for rfft

z : real ndarray

The returned real array contains:

[y(0),Re(y(1)),Im(y(1)),...,Re(y(n/2))]              if n is even
[y(0),Re(y(1)),Im(y(1)),...,Re(y(n/2)),Im(y(n/2))]   if n is odd

Conclusion: the storage is different.

For a starter, look at the length of magnitude, it will be different in both cases. I give an example below for clarity:

In [33]: data = np.random.random(size=8)

In [34]: np.fft.rfft(data)
Out[34]:
array([ 3.33822983+0.j        ,  0.15879369+0.48542266j,
        0.00614876+0.03590621j, -0.67376592-0.69793372j,  1.51730861+0.j        ])

In [35]: scipy.fftpack.rfft(data)
Out[35]:
array([ 3.33822983,  0.15879369,  0.48542266,  0.00614876,  0.03590621,
       -0.67376592, -0.69793372,  1.51730861])

The first element in both cases is the so-called "DC component" (the mean of the signal).

Then, you can recognize in the SciPy version the succession of real and imaginary parts of the NumPy version.


问:
How to download the content of a file from S3 bucket into memory with Ruby
I have a bucket in Amazon AWS S3 which has a file in it called users.csv.

How can I load the content of this file from the S3 bucket into memory with Ruby so I can parse it?

Here is my code:

```<code>require 'aws-sdk'
s3 = Aws::S3::Resource.new(region: 'us-west-1')
resp = s3.list_objects(bucket: 'bucket-name', max_keys: 1)
resp.contents.each do |object|
    puts #{object.value}
end
</code>```

When I tried this in IRB I got:

```<code>struct Aws::S3::Types::Object key="users.csv", last_modified=2017-11-15 19:10:28 UTC, etag="\"9a3d50c07aa4aa6976037ce774294a26\"", size=101, storage_class="STANDARD", owner=struct Aws::S3::Types::Owner display_name="owner-name", id="42093cfa4ccb23a8156cdab8500a41a10bdbf90deebb0ee8a3b340dd1e0c3622"
</code>```

How can I parse the content of users.csv?

答:
From the AWS Documentation:

Downloading Objects into Memory

For small objects, it can be useful to get an object and have it available in your Ruby processes. If you
  do not specify a :target for the download, the entire object is loaded
  into memory into a StringIO object.

```<code>def import_from_s3

  #initiate the client
  s3 = Aws::S3::Client.new({
      region:            region,
      access_key_id:     key_id,
      secret_access_key: secret
  })

  #Get the object
  resp = s3.get_object(bucket: bucket, key: key)

 resp.body
 #=> #<StringIO ...>

 resp.body.read
 #=> '...'
</code>```

Call #read or #string on the StringIO to get the body as a String object.

For more information see "Downloading Objects from Amazon S3 using the AWS SDK for Ruby".

Body:
I have a bucket in Amazon AWS S3 which has a file in it called users.csv.

How can I load the content of this file from the S3 bucket into memory with Ruby so I can parse it?

Here is my code:

require 'aws-sdk'
s3 = Aws::S3::Resource.new(region: 'us-west-1')
resp = s3.list_objects(bucket: 'bucket-name', max_keys: 1)
resp.contents.each do |object|
    puts #{object.value}
end

When I tried this in IRB I got:

struct Aws::S3::Types::Object key="users.csv", last_modified=2017-11-15 19:10:28 UTC, etag="\"9a3d50c07aa4aa6976037ce774294a26\"", size=101, storage_class="STANDARD", owner=struct Aws::S3::Types::Owner display_name="owner-name", id="42093cfa4ccb23a8156cdab8500a41a10bdbf90deebb0ee8a3b340dd1e0c3622"

How can I parse the content of users.csv?

Body:
From the AWS Documentation:

Downloading Objects into Memory

For small objects, it can be useful to get an object and have it available in your Ruby processes. If you
  do not specify a :target for the download, the entire object is loaded
  into memory into a StringIO object.

def import_from_s3

  #initiate the client
  s3 = Aws::S3::Client.new({
      region:            region,
      access_key_id:     key_id,
      secret_access_key: secret
  })

  #Get the object
  resp = s3.get_object(bucket: bucket, key: key)

 resp.body
 #=> #<StringIO ...>

 resp.body.read
 #=> '...'

Call #read or #string on the StringIO to get the body as a String object.

For more information see "Downloading Objects from Amazon S3 using the AWS SDK for Ruby".


问:
Finding equal rows between multiple arrays of different sizes by voting
Let's say I have 5 arrays of different sizes. All of the arrays have the same number of columns, in my case 2, but a different number of rows. I need to find the elements of the rows that appear in at least 3 of such arrays.

Right now, I compare two arrays using ismember, then compare that result with the third array and then save the row values which occur in all the three arrays. I do this for every possible combination of 3 arrays; basically in my case, I have 10 of such operations in total. It's like choosing three out of 5 without repetitions.

It works, but I am looking for a more efficient implementation. In particular, I was looking for any implementation that can perform this by voting. It's like having sets of different sizes and trying to find the elements that appear in the majority of sets, in my case 3 arrays out of 5.

答:
Not sure what you mean by voting but I think does what you are looking for.

It creates 1 big unique matrix of all the rows from the arrays.  The does an ismember by rows of the unique array with each individual array.  The ismember is summed together to get a count of how many times each unique row exists across your set of arrays.

You can then use that count to return a new array that has at least minNum occurrences.

You would call it like this:

```<code>>> [outRows, uRows, memCount]= getRowDuplicates(3,a,b,c,d,e)
</code>```

Where a,b,c,d,e are you arrays and 3 is the minimum number of occurrences

```<code>function [outRows, uRows, memCount]= getRowDuplicates(minNum,varargin)

uRows = unique(vertcat(varargin{:}),'rows');
memCount = false(size(uRows,1),1);
for j = 1:nargin-1
    memCount = memCount + ismember(uRows,varargin{j},'rows');
end
rowIdx = memCount >= minNum;
outRows = uRows(rowIdx,:);
</code>```

Body:
Not sure what you mean by voting but I think does what you are looking for.

It creates 1 big unique matrix of all the rows from the arrays.  The does an ismember by rows of the unique array with each individual array.  The ismember is summed together to get a count of how many times each unique row exists across your set of arrays.

You can then use that count to return a new array that has at least minNum occurrences.

You would call it like this:

>> [outRows, uRows, memCount]= getRowDuplicates(3,a,b,c,d,e)

Where a,b,c,d,e are you arrays and 3 is the minimum number of occurrences

function [outRows, uRows, memCount]= getRowDuplicates(minNum,varargin)

uRows = unique(vertcat(varargin{:}),'rows');
memCount = false(size(uRows,1),1);
for j = 1:nargin-1
    memCount = memCount + ismember(uRows,varargin{j},'rows');
end
rowIdx = memCount >= minNum;
outRows = uRows(rowIdx,:);


问:
Python 4.7: How to count the frequency of specific value in array
Basically I have a list with 250 elements, some of them are 1 some 2 and some 3. How do I get a variable that represents the number of ‘1’s ?</p>

答:
I assume this will still be the case in python 4.7

Jokes aside, this will work in Python 2.7 (and 3 etc)

Using count():

```<code>your_list = [1,1,2,3,4,5,6,7,8]
distinct_values = set(your_list)
if len(distinct_values) == len(your_list)
    print('All values have a tf count of 1')
else:
    for distinct_value in distinct_values:
        print('%s n occurences: %s' % (distinct_value, str(your_list.count(distinct_value))))
</code>```

Using in (Solution provided by @nfn neil via comments):

```<code>your_list = [1,1,2,3,4,5,6,7,8]
di = {}
for item in your_list:
    if item in di:
        di[item] += 1
    else:
        di[item] = 1
print(di)
</code>```

benchmarks running both variants 300 times with a list of 500 000 items:

The runtime for using_in took 18.3120000362 seconds to complete

The runtime for using_count took 743.9941547110 seconds to complete

Benchmark code:

https://pastebin.com/dWS8UH7c

Body:
I assume this will still be the case in python 4.7

Jokes aside, this will work in Python 2.7 (and 3 etc)

Using count():

your_list = [1,1,2,3,4,5,6,7,8]
distinct_values = set(your_list)
if len(distinct_values) == len(your_list)
    print('All values have a tf count of 1')
else:
    for distinct_value in distinct_values:
        print('%s n occurences: %s' % (distinct_value, str(your_list.count(distinct_value))))

Using in (Solution provided by @nfn neil via comments):

your_list = [1,1,2,3,4,5,6,7,8]
di = {}
for item in your_list:
    if item in di:
        di[item] += 1
    else:
        di[item] = 1
print(di)

benchmarks running both variants 300 times with a list of 500 000 items:

The runtime for using_in took 18.3120000362 seconds to complete

The runtime for using_count took 743.9941547110 seconds to complete

Benchmark code:

https://pastebin.com/dWS8UH7c


问:
Record device audio output with AudioUnit
We need to record the device audio output from cable or bluetooth headset speakers. For example if user listens to some song when Music app runs in background we want to record this song. Is it possible to implement output recording with AudioUnit? We use PalayAndRecord category for shared audioSession with AVAudioSessionCategoryOptionMixWithOthers option.</p>

答: No, in general you can not listen to the output from another app. There is inter app audio and audio unit technology that enables this, but the connection must be arranged by both apps. It is not possible to just eavesdrop on whatever the user outputs from the device.</p>


问:
Can't drop jquery ui helper on droppable
Why will this code not let me drop the helper onto the droppable region?

```<code>  $(".product").draggable({
    revert: 'invalid',
    cursorAt: { top: -12, left: -20 },
    helper: function(event) {
      return $('<div class="product_helper"></div>');
    }
  });
  $(".droppable").droppable({
    accept: '.product_helper',
    drop: function(event, ui) {
      $(this).append( ui.helper );
    }
  });
</code>```

Is it even possible to drop a helper onto a droppable?

答:
one problem I had was having a draggable element that too wide to be dropped on to a droppable element if the tolerance option is at the default value of 'intersect'.

'intersect' means the draggable can be dropped when it 50% of it is over the droppable.  And that's impossible if its more than twice as wide.

My draggables had a variable value, so their width varied and whether they could be dropped varied too.  I used 'pointer' instead and it goes by where the mouse is.

Body:
Why will this code not let me drop the helper onto the droppable region?

  $(".product").draggable({
    revert: 'invalid',
    cursorAt: { top: -12, left: -20 },
    helper: function(event) {
      return $('

');
    }
  });
  $(".droppable").droppable({
    accept: '.product_helper',
    drop: function(event, ui) {
      $(this).append( ui.helper );
    }
  });

Is it even possible to drop a helper onto a droppable?


问:
How to remove empty tags AND line breaks with xslt?
I want to remove empty tags as well as line breaks from an XML file via XSLT.

Criteria 1:
Line breaks should only be removed within text nodes, i.e. the XML should remain indented.

Criteria 2:
Should work on any OS.

Criteria 3:
If possible there wouldn't be any other changes to the XML.

An example XML would be:

```<code><?xml version="1.0" encoding="UTF-8"?>
<Container xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <tag1>22</tag1>
    <tag2>33</tag2>
    <tag3/>
    <minimalValue/>
    <maximalValue/>
    <minimalFee/>
    <maximalFee/>
    <tag4>This is text with a
        line break.
    </tag4>
</Container>
</code>```

The result should be:

```<code><Container xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
   <tag1>22</tag1>
   <tag2>33</tag2>
   <tag4>This is text with a line break.</tag4>
</Container>
</code>```

答:
To omit initial XML declaration you should use omit-xml-declaration
attribute in xsl:output.

To have indentation use indent="yes".

To remove empty tags, you need a template for elements (match="*"),
but with a predicate requiring that this element has non-whitespace-only
content ([normalize-space()]).
Since there is no template for empty elements, they will not be copied.

To remove line breaks you need a template for text() nodes,
copying the normalized content.

So to sum up, you can use the following script, slightly more concise
than the other proposal:

```<code><xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output omit-xml-declaration="yes" indent="yes"/>

  <xsl:template match="*[normalize-space()]|@*">
    <xsl:copy>
      <xsl:apply-templates select="node()|@*"/>
    </xsl:copy>
  </xsl:template>

  <xsl:template match="text()">
    <xsl:value-of select="normalize-space()"/>
  </xsl:template>
</xsl:stylesheet>
</code>```

Note that I added |@* in 2 places, to handle attribute nodes,
like in the identity template.
But since your source XML does not containt any attribute nodes,
you could do even without these additions.

Body:
I want to remove empty tags as well as line breaks from an XML file via XSLT.

Criteria 1:
Line breaks should only be removed within text nodes, i.e. the XML should remain indented.

Criteria 2:
Should work on any OS.

Criteria 3:
If possible there wouldn't be any other changes to the XML.

An example XML would be:

<?xml version="1.0" encoding="UTF-8"?>
<Container xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <tag1>22</tag1>
    <tag2>33</tag2>
    <tag3/>
    <minimalValue/>
    <maximalValue/>
    <minimalFee/>
    <maximalFee/>
    <tag4>This is text with a
        line break.
    </tag4>
</Container>

The result should be:

<Container xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
   <tag1>22</tag1>
   <tag2>33</tag2>
   <tag4>This is text with a line break.</tag4>
</Container>

Body:
To omit initial XML declaration you should use omit-xml-declaration
attribute in xsl:output.

To have indentation use indent="yes".

To remove empty tags, you need a template for elements (match="*"),
but with a predicate requiring that this element has non-whitespace-only
content ([normalize-space()]).
Since there is no template for empty elements, they will not be copied.

To remove line breaks you need a template for text() nodes,
copying the normalized content.

So to sum up, you can use the following script, slightly more concise
than the other proposal:

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output omit-xml-declaration="yes" indent="yes"/>

  <xsl:template match="*[normalize-space()]|@*">
    <xsl:copy>
      <xsl:apply-templates select="node()|@*"/>
    </xsl:copy>
  </xsl:template>

  <xsl:template match="text()">
    <xsl:value-of select="normalize-space()"/>
  </xsl:template>
</xsl:stylesheet>

Note that I added |@* in 2 places, to handle attribute nodes,
like in the identity template.
But since your source XML does not containt any attribute nodes,
you could do even without these additions.