问:
Spark executors can't see implicit value (java.lang.NullPointerException)
I have a spark application which I can not start on Yarn. I am getting null pointer exceptions. (on master local application works fine). My application is converting nested XMLs to JSONs.

I am loading XMLs into RDD with:
val xml_rdd = spark.sparkContext.textFile(path)

I am converting XMLs into JSONs with scala.xml.XML.loadString(xmlStr).toJson.toString() . toJson method required implicit writer so I am creating it:

```<code>implicit val writer: JsonWriter[Elem]  = new JsonWriter[Elem] {
    def write(elem: Elem): JsValue = {
        ...
    }
}
</code>```

Full code:

```<code>import com.typesafe.scalalogging.LazyLogging
import org.apache.spark.sql.SparkSession
import spray.json.{JsArray, JsObject, JsString, JsValue, JsonWriter, _}

import scala.xml._

object Test  extends  App with LazyLogging{

    val in_path = args(0)
    val spark: SparkSession = SparkSession.builder().appName(this.getClass.getSimpleName).getOrCreate()
    val xml_rdd = spark.sparkContext.textFile(in_path)

    implicit val writer: JsonWriter[Elem]  = new JsonWriter[Elem] {
        def write(elem: Elem): JsValue = {
            if (List("Persons").contains(elem.label)) {
                JsArray(
                    elem.child.collect {
                        case e: Elem => write(e)
                    }: _*)
            } else if (elem.child.isEmpty) {
                JsString("")
            } else if (elem.child.count(_.isInstanceOf[Text]) == 1){
                JsString(elem.text)
            } else
                JsObject(
                    elem.child.collect {
                        case e: Elem => e.label -> write(e)
                    }: _*)
        }
    }
    val json_rdd = xml_rdd.map(xmlStr => scala.xml.XML.loadString(xmlStr).toJson.toString())

    json_rdd.collect().foreach(println)
}
</code>```

My debugging showed that problem is in toJson method. It seems implicit writer exists only on driver and executors can not see it. I have tried to  broadcast writer in order to make in available on executors with spark.sparkContext.broadcast(jsonSerializer) but then I have received object not serializable exception.

Can you help me please in making implicit writer available on executors or point me on mistake which I am doing?

Full stacktrace exception:

```<code>20/05/28 12:45:25 INFO scheduler.DAGScheduler: ResultStage 0 (collect at Test.scala:36) failed in 3.992 s due to Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 8, something.bla, executor 5): java.lang.NullPointerException
        at spray.json.RichAny.toJson(package.scala:44)
        at com.something.Test$$anonfun$1.apply(Test.scala:34)
        at com.something.Test$$anonfun$1.apply(Test.scala:34)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
        at scala.collection.AbstractIterator.to(Iterator.scala:1334)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1334)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1334)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:121)
        at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
20/05/28 12:45:25 INFO scheduler.DAGScheduler: Job 0 failed: collect at Test.scala:36, took 4.031537 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 8, something.bla, executor 5): java.lang.NullPointerException
        at spray.json.RichAny.toJson(package.scala:44)
        at com.something.Test$$anonfun$1.apply(Test.scala:34)
        at com.something.Test$$anonfun$1.apply(Test.scala:34)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
        at scala.collection.AbstractIterator.to(Iterator.scala:1334)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1334)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1334)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:121)
        at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
        at org.apache.spark.rdd.RDD.collect(RDD.scala:944)
        at com.something.Test$.delayedEndpoint$com.something$Test$1(Test.scala:36)
        at com.something.Test$delayedInit$body.apply(Test.scala:10)
        at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
        at scala.App$$anonfun$main$1.apply(App.scala:76)
        at scala.App$$anonfun$main$1.apply(App.scala:76)
        at scala.collection.immutable.List.foreach(List.scala:392)
        at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
        at scala.App$class.main(App.scala:76)
        at com.something.Test$.main(Test.scala:10)
        at com.something.Test.main(Test.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NullPointerException
        at spray.json.RichAny.toJson(package.scala:44)
        at com.something.Test$$anonfun$1.apply(Test.scala:34)
        at com.something.Test$$anonfun$1.apply(Test.scala:34)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
        at scala.collection.AbstractIterator.to(Iterator.scala:1334)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1334)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1334)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:121)
        at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
</code>```

答:
Usually non-serializable stuff is created on each executor. E.g. consider these examples:</p>

```<code>// this fails when you call an "action" on myrdd2
val myNonSerializableObj = new Thread();
val myrdd2 = myrdd.map( elem => myNonSerializableObj.toString() + "-" + elem )

// this works
val myrdd3 = myrdd.mapPartitions{ partition =>
  val myNonSerializableObj = new Thread();
  partition.map{ elem =>
    myNonSerializableObj.toString() + "-" + elem
  }
}
</code>```

Body:
I have a spark application which I can not start on Yarn. I am getting null pointer exceptions. (on master local application works fine). My application is converting nested XMLs to JSONs.

I am loading XMLs into RDD with:
val xml_rdd = spark.sparkContext.textFile(path)

I am converting XMLs into JSONs with scala.xml.XML.loadString(xmlStr).toJson.toString() . toJson method required implicit writer so I am creating it:

implicit val writer: JsonWriter[Elem]  = new JsonWriter[Elem] {
    def write(elem: Elem): JsValue = {
        ...
    }
}

Full code:

import com.typesafe.scalalogging.LazyLogging
import org.apache.spark.sql.SparkSession
import spray.json.{JsArray, JsObject, JsString, JsValue, JsonWriter, _}

import scala.xml._

object Test  extends  App with LazyLogging{

    val in_path = args(0)
    val spark: SparkSession = SparkSession.builder().appName(this.getClass.getSimpleName).getOrCreate()
    val xml_rdd = spark.sparkContext.textFile(in_path)

    implicit val writer: JsonWriter[Elem]  = new JsonWriter[Elem] {
        def write(elem: Elem): JsValue = {
            if (List("Persons").contains(elem.label)) {
                JsArray(
                    elem.child.collect {
                        case e: Elem => write(e)
                    }: _*)
            } else if (elem.child.isEmpty) {
                JsString("")
            } else if (elem.child.count(_.isInstanceOf[Text]) == 1){
                JsString(elem.text)
            } else
                JsObject(
                    elem.child.collect {
                        case e: Elem => e.label -> write(e)
                    }: _*)
        }
    }
    val json_rdd = xml_rdd.map(xmlStr => scala.xml.XML.loadString(xmlStr).toJson.toString())

    json_rdd.collect().foreach(println)
}

My debugging showed that problem is in toJson method. It seems implicit writer exists only on driver and executors can not see it. I have tried to  broadcast writer in order to make in available on executors with spark.sparkContext.broadcast(jsonSerializer) but then I have received object not serializable exception.

Can you help me please in making implicit writer available on executors or point me on mistake which I am doing?

Full stacktrace exception:

20/05/28 12:45:25 INFO scheduler.DAGScheduler: ResultStage 0 (collect at Test.scala:36) failed in 3.992 s due to Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 8, something.bla, executor 5): java.lang.NullPointerException
        at spray.json.RichAny.toJson(package.scala:44)
        at com.something.Test$$anonfun$1.apply(Test.scala:34)
        at com.something.Test$$anonfun$1.apply(Test.scala:34)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
        at scala.collection.AbstractIterator.to(Iterator.scala:1334)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1334)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1334)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:121)
        at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
20/05/28 12:45:25 INFO scheduler.DAGScheduler: Job 0 failed: collect at Test.scala:36, took 4.031537 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 8, something.bla, executor 5): java.lang.NullPointerException
        at spray.json.RichAny.toJson(package.scala:44)
        at com.something.Test$$anonfun$1.apply(Test.scala:34)
        at com.something.Test$$anonfun$1.apply(Test.scala:34)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
        at scala.collection.AbstractIterator.to(Iterator.scala:1334)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1334)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1334)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:121)
        at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
        at org.apache.spark.rdd.RDD.collect(RDD.scala:944)
        at com.something.Test$.delayedEndpoint$com.something$Test$1(Test.scala:36)
        at com.something.Test$delayedInit$body.apply(Test.scala:10)
        at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
        at scala.App$$anonfun$main$1.apply(App.scala:76)
        at scala.App$$anonfun$main$1.apply(App.scala:76)
        at scala.collection.immutable.List.foreach(List.scala:392)
        at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
        at scala.App$class.main(App.scala:76)
        at com.something.Test$.main(Test.scala:10)
        at com.something.Test.main(Test.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NullPointerException
        at spray.json.RichAny.toJson(package.scala:44)
        at com.something.Test$$anonfun$1.apply(Test.scala:34)
        at com.something.Test$$anonfun$1.apply(Test.scala:34)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
        at scala.collection.AbstractIterator.to(Iterator.scala:1334)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1334)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1334)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:121)
        at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Body:
Usually non-serializable stuff is created on each executor. E.g. consider these examples:

// this fails when you call an "action" on myrdd2
val myNonSerializableObj = new Thread();
val myrdd2 = myrdd.map( elem => myNonSerializableObj.toString() + "-" + elem )

// this works
val myrdd3 = myrdd.mapPartitions{ partition =>
  val myNonSerializableObj = new Thread();
  partition.map{ elem =>
    myNonSerializableObj.toString() + "-" + elem
  }
}


来源: stackexchange

问:
Update to typescript 3.8, made array readonly
I recently update my project from Typescript 3.5 to 3.8.
Everything run smoothly expect some operation on array.

```<code>    const usedROI = this.rois.find((roi) => roi.id === roiId);
    usedROI.position.height = position.height;
</code>```

ERROR TypeError: Cannot assign to read only property 'height' of object '[object Object]'

```<code>   let rerouteWaypoint = route.waypoints.filter((wp) => wp.type === IWaypointType.REROUTE_POINT);
   rerouteWaypoint.forEach((wp) => (wp.type = IWaypointType.USER_POINT));
</code>```

ERROR TypeError: Cannot assign to read only property 'type' of object '[object Array]'

```<code>  const wps = state.entities[action.payload.routeId].waypoints;
  const wp = {
    ...wps[index],
    ...action.payload.changes,
  };
  wps[index] = { ...wp }; <= this line breaks
</code>```

core.js:4117 ERROR TypeError: Cannot assign to read only property '0' of object '[object Array]'

I do not use readonly anywhere in my app, it looks like all my array are frozen and uneditable.

Why so ? I do not see it in the breaking changes. Is there a way to find in visual code ALL the spot where this problem occur ? cause right now only the error pop at runtime and it's really unsafe to deploy such a version.

EDIT : Someone talk about a similar problem here => Error while sorting array of objects Cannot assign to read only property '2' of object '[object Array]'

but I do not see why simple array are also throwing it without beeing redux stored elements.

答:
Following @T.J.Crowder comment, I looked into my code and by default, if not typing a selector in NGRX, the returned value is NOT typed readonly.

I went trought all my selector and typed them appropriately for ex.

```<code>export const getSelectedProjectId = (state: State): Readonly<number> => state.selectedProjectId;
export const getError = (state: State): Readonly<string> => state.error;
export const getIsLoading = (state: State): Readonly<boolean> => state.isLoading;

export const selectAll: (state: object) => ReadonlyArray<IProject> = featureAdapter.getSelectors(selectProjectState).selectAll;
export const selectAllEntities: (state: object) => Dictionary<IProject> = featureAdapter.getSelectors(
  selectProjectState
).selectEntities;
</code>```

This triggered ALL the error in typescript everywhere, and I then fixed them and fixed the project.

ContentLicense: CC BY-SA 4.0

Body:
I recently update my project from Typescript 3.5 to 3.8.
Everything run smoothly expect some operation on array.

    const usedROI = this.rois.find((roi) => roi.id === roiId);
    usedROI.position.height = position.height;

ERROR TypeError: Cannot assign to read only property 'height' of object '[object Object]'

   let rerouteWaypoint = route.waypoints.filter((wp) => wp.type === IWaypointType.REROUTE_POINT);
   rerouteWaypoint.forEach((wp) => (wp.type = IWaypointType.USER_POINT));

ERROR TypeError: Cannot assign to read only property 'type' of object '[object Array]'

  const wps = state.entities[action.payload.routeId].waypoints;
  const wp = {
    ...wps[index],
    ...action.payload.changes,
  };
  wps[index] = { ...wp }; <= this line breaks

core.js:4117 ERROR TypeError: Cannot assign to read only property '0' of object '[object Array]'

I do not use readonly anywhere in my app, it looks like all my array are frozen and uneditable.

Why so ? I do not see it in the breaking changes. Is there a way to find in visual code ALL the spot where this problem occur ? cause right now only the error pop at runtime and it's really unsafe to deploy such a version.

EDIT : Someone talk about a similar problem here => Error while sorting array of objects Cannot assign to read only property '2' of object '[object Array]'

but I do not see why simple array are also throwing it without beeing redux stored elements.

Body:
Following @T.J.Crowder comment, I looked into my code and by default, if not typing a selector in NGRX, the returned value is NOT typed readonly.

I went trought all my selector and typed them appropriately for ex.

export const getSelectedProjectId = (state: State): Readonly<number> => state.selectedProjectId;
export const getError = (state: State): Readonly<string> => state.error;
export const getIsLoading = (state: State): Readonly<boolean> => state.isLoading;

export const selectAll: (state: object) => ReadonlyArray<IProject> = featureAdapter.getSelectors(selectProjectState).selectAll;
export const selectAllEntities: (state: object) => Dictionary<IProject> = featureAdapter.getSelectors(
  selectProjectState
).selectEntities;

This triggered ALL the error in typescript everywhere, and I then fixed them and fixed the project.


问:
how to list files in Unix without the modified date
I'm trying to create a zip file that will run on multiple servers to compare the contents of the servers.  For many reasons, the easiest and best piece of information for me to compare is a text file of the directory listings... but I need that without the modified date as they make the comparisons show differences that don't matter in this case.

So if I run command ls -la to create text output for comparing, I get would get something like this:

```<code>drwxr-xr-x.  6 root   root       4096 Mar 20 14:59 .
dr-xr-xr-x. 22 root   root       4096 Feb 18 03:20 ..
drwxr-xr-x.  9  web   ad         4096 Oct 30 14:35 apache-tomcat-6.0.18
drwxr-xr-x.  9  web   ad         4096 Mar 24 03:00 apache-tomcat-6.0.36
lrwxrwxrwx.  1 root   root          5 Oct 30 14:06 java -> java6
lrwxrwxrwx.  1 root   root         16 Oct 30 14:05 java6 -> jdk/jdk1.6.0_37/
lrwxrwxrwx.  1 root   root         16 Mar 20 14:59 java7 -> jdk/jdk1.7.0_17/
drwxr-xr-x.  4 root   root       4096 Mar 20 15:02 jdk
lrwxrwxrwx.  1 root   root         21 Nov  6 15:09 tomcat -> apache-tomcat-6.0.36/
</code>```

What I would like is to use is ls -la | cut -c 1-31 but that only takes one list (the characters 1-31) and I really want the data after the date.  I'm fairly new with Unix and was curious if anyone knows how to produce a list that would look something like this:

```<code>drwxr-xr-x.  6 root   root       4096 .
dr-xr-xr-x. 22 root   root       4096 ..
drwxr-xr-x.  9  web   ad         4096 apache-tomcat-6.0.18
drwxr-xr-x.  9  web   ad         4096 apache-tomcat-6.0.36
lrwxrwxrwx.  1 root   root          5 java -> java6
lrwxrwxrwx.  1 root   root         16 java6 -> jdk/jdk1.6.0_37/
lrwxrwxrwx.  1 root   root         16 java7 -> jdk/jdk1.7.0_17/
drwxr-xr-x.  4 root   root       4096 jdk
lrwxrwxrwx.  1 root   root         21 tomcat -> apache-tomcat-6.0.36/
</code>```

Thanks

答:
cut let's you specify ranges, more easily than awk.

Use it in combination with tr -s ' ' to trim repeating whitespaces.

$ ls -la /etc/ |tr -s ' ' |cut -d' ' -f1,2,3,4,5,9-
drwxr-xr-x 2 root root 4096 opt
lrwxrwxrwx 1 root root 21 os-release -> ../usr/lib/os-release
-rw-r--r-- 1 root root 6920 overlayroot.conf
drwxr-xr-x 4 root root 4096 vmware-tools
lrwxrwxrwx 1 root root 23 vtrgb -> /etc/alternatives/vtrgb
```

This is useful e.g. when using a parallel shell and you want to check files on multiple hosts, created at different times.

ContentLicense: CC BY-SA 3.0

Body:
I'm trying to create a zip file that will run on multiple servers to compare the contents of the servers.  For many reasons, the easiest and best piece of information for me to compare is a text file of the directory listings... but I need that without the modified date as they make the comparisons show differences that don't matter in this case.

So if I run command ls -la to create text output for comparing, I get would get something like this:

drwxr-xr-x.  6 root   root       4096 Mar 20 14:59 .
dr-xr-xr-x. 22 root   root       4096 Feb 18 03:20 ..
drwxr-xr-x.  9  web   ad         4096 Oct 30 14:35 apache-tomcat-6.0.18
drwxr-xr-x.  9  web   ad         4096 Mar 24 03:00 apache-tomcat-6.0.36
lrwxrwxrwx.  1 root   root          5 Oct 30 14:06 java -> java6
lrwxrwxrwx.  1 root   root         16 Oct 30 14:05 java6 -> jdk/jdk1.6.0_37/
lrwxrwxrwx.  1 root   root         16 Mar 20 14:59 java7 -> jdk/jdk1.7.0_17/
drwxr-xr-x.  4 root   root       4096 Mar 20 15:02 jdk
lrwxrwxrwx.  1 root   root         21 Nov  6 15:09 tomcat -> apache-tomcat-6.0.36/

What I would like is to use is ls -la | cut -c 1-31 but that only takes one list (the characters 1-31) and I really want the data after the date.  I'm fairly new with Unix and was curious if anyone knows how to produce a list that would look something like this:

drwxr-xr-x.  6 root   root       4096 .
dr-xr-xr-x. 22 root   root       4096 ..
drwxr-xr-x.  9  web   ad         4096 apache-tomcat-6.0.18
drwxr-xr-x.  9  web   ad         4096 apache-tomcat-6.0.36
lrwxrwxrwx.  1 root   root          5 java -> java6
lrwxrwxrwx.  1 root   root         16 java6 -> jdk/jdk1.6.0_37/
lrwxrwxrwx.  1 root   root         16 java7 -> jdk/jdk1.7.0_17/
drwxr-xr-x.  4 root   root       4096 jdk
lrwxrwxrwx.  1 root   root         21 tomcat -> apache-tomcat-6.0.36/

Thanks

Body:
cut let's you specify ranges, more easily than awk.

Use it in combination with tr -s ' ' to trim repeating whitespaces.

$ ls -la /etc/ |tr -s ' ' |cut -d' ' -f1,2,3,4,5,9-
drwxr-xr-x 2 root root 4096 opt
lrwxrwxrwx 1 root root 21 os-release -> ../usr/lib/os-release
-rw-r--r-- 1 root root 6920 overlayroot.conf
drwxr-xr-x 4 root root 4096 vmware-tools
lrwxrwxrwx 1 root root 23 vtrgb -> /etc/alternatives/vtrgb

This is useful e.g. when using a parallel shell and you want to check files on multiple hosts, created at different times.


问:
Why can't I import candlestick_ohlc from mplfinance
So I have been able to successfully install mplfinance with pip and when I import it alone I receive no error. Though when I do: from mplfinance import candlestick_ohlc I get the error ImportError: cannot import name 'candlestick_ohlc' from 'mplfinance' I have checked command prompt again, and it says it has successfully installed mplfinance. Why am I receiving this error?

答:
So from what I understand the Matplotlib for finance has changed so that:

To access the old API with the new mplfinance package installed, change statments

from:

```<code>from mpl_finance import
</code>```

to:

```<code>    from mplfinance.original_flavor import candlestick_ohlc
</code>```

and then it should work fine.

Body:
So from what I understand the Matplotlib for finance has changed so that:

To access the old API with the new mplfinance package installed, change statments

from:

from mpl_finance import

to:

    from mplfinance.original_flavor import candlestick_ohlc

and then it should work fine.


问:
What can be the reasons of connection refused errors?
I'm trying to write a server program in C,
using another client, I get this error when I try to connect through port 2080 for example.

```<code>connection refused
</code>```

What can be the reasons of this error?

答:
In my case, it happens when the site is blocked in my country and I don't use VPN.
For example when I try to access vimeo.com from Indonesia which is blocked.</p>

ContentLicense: CC BY-SA 2.5

Body:
I'm trying to write a server program in C,
using another client, I get this error when I try to connect through port 2080 for example.

connection refused

What can be the reasons of this error?


问:
How change position element in Array?
I have project in Laravel 5. Project work fine, but I have problem with sort position in my array.
I have array with wrong position ID and email:

```<code>Array
(
    [0] => Array
        (
            [0] => sda@swqdqwdwq.pll
            [1] => 957
            [2] => xxxx
            [3] => xxxx
            [4] => xxxx
            [5] => 2021-01-19
            [6] => tak
            [7] =>
<a href="http://projekt2.test/admin/client/info/957" class="btn btn-xs blue get-info ">
    <i class="fa fa-search"></i>
     Przeglądaj
</a>
<a href="http://projekt2.test/admin/client/edit/957" class="btn btn-xs blue ">
    <i class="fa fa-pencil"></i>
     Edytuj
</a>
<a href="http://projekt2.test/admin/client/delete/957" class="btn btn-xs red delete">
    <i class="fa fa-trash-o"></i>
     Usuń
</a>
<a href="http://projekt2.test/admin/client/deactive/957" class="btn btn-xs yellow">
    <i class="fa fa-ban"></i>
     Dezaktywuj
</a>

        )

    [1] => Array
        (
            [0] => xxxx@op.pl
            [1] => 958
            [2] => xxxx
            [3] => xxxx
            [4] => firmowe
            [5] => 2021-02-03
            [6] => tak
            [7] =>
<a href="http://projekt2.test/admin/client/info/958" class="btn btn-xs blue get-info ">
    <i class="fa fa-search"></i>
     Przeglądaj
</a>
<a href="http://projekt2.test/admin/client/edit/958" class="btn btn-xs blue ">
    <i class="fa fa-pencil"></i>
     Edytuj
</a>
<a href="http://projekt2.test/admin/client/delete/958" class="btn btn-xs red delete">
    <i class="fa fa-trash-o"></i>
     Usuń
</a>
<a href="http://projekt2.test/admin/client/deactive/958" class="btn btn-xs yellow">
    <i class="fa fa-ban"></i>
     Dezaktywuj
</a>

        )
...
}
</code>```

My php code:

```<code>$data = $this->getList();
print_r($data);
</code>```

I need change position ID and email (first position ID, second email).
I need result:

```<code>Array
    (
        [0] => Array
            (
                [0] => 957
                [1] => sda@swqdqwdwq.pll
                [2] => xxxx
                [3] => xxxx
                [4] => xxxx
                [5] => 2021-01-19
                [6] => tak
                [7] =>
    <a href="http://projekt2.test/admin/client/info/957" class="btn btn-xs blue get-info ">
        <i class="fa fa-search"></i>
         Przeglądaj
    </a>
    <a href="http://projekt2.test/admin/client/edit/957" class="btn btn-xs blue ">
        <i class="fa fa-pencil"></i>
         Edytuj
    </a>
    <a href="http://projekt2.test/admin/client/delete/957" class="btn btn-xs red delete">
        <i class="fa fa-trash-o"></i>
         Usuń
    </a>
    <a href="http://projekt2.test/admin/client/deactive/957" class="btn btn-xs yellow">
        <i class="fa fa-ban"></i>
         Dezaktywuj
    </a>

            )
</code>```

How can I make it?
I have project in old Laravel 5.

Please help me

答:
Try it:

<?php

$arr = ['foo', 'foobar@gmail.com', 'myName', 'myInfo', 954];
$arrMask = ['/\d+/', '/\w+@\w+.\w+/', '/*/', '/*/', '/*/'];

print_r(sortArrayByExp($arr, $arrMask));

/**
 * Sort a array by RegExp
 *
 * Sort a array by regular expression
 *
 * @param   array   $arr        Array to sort
 * @param   array   $arrRegExp  Array to postion Regex
 *
 * @return  array   Return a new array mirrored by value in $arrRegExp
 *
 * @throws  \InvalidArgumentException
 */
function sortArrayByExp($arr, $arrRegExp)
{

    if(! is_array($arr))
        throw new \InvalidArgumentException('Value in parameter $arr must be array!', 556);

    if(! is_array($arrRegExp))
        throw new \InvalidArgumentException('Value in parameter $arrRegExp must be array!', 557);

    $newArray = [];

    foreach($arrRegExp as $key => $mask)
    {
        if(! is_string($mask))
            throw new \InvalidArgumentException('Each value in $arrRegExp must be string or RegExp', 558);

        $response = null;
        if(@preg_match($mask, '') !== false)
        {
            foreach($arr as $posArray => $value)
                if(preg_match($mask, $value))
                {
                    $response = $value;
                    unset($arr[$posArray]);
                    break;
                }

        }

        if($response !== null)
           $newArray[$key] = $response;

    }

    return $newArray;
}

/* output must be:
    Array
    (
        [0] => 954
        [1] => foobar@gmail.com
        [2] => foo
        [3] => myName
        [4] => myInfo
    )
*/
```

<p>This function sort a array by each value in new array that contains a <code>regular expression</code>, try my exemple here</p>

<p><strong>Edit 1:</strong></p>

<p>You can use too <code>collect</code> of Laravel to sort your array, like this:</p>

<pre class="lang-php prettyprint-override"><code><?php
require "vendor/autoload.php";

$myCollect = collect(['foo', 'foobar@gmail.com', 'myName', 'myInfo', 954]);

print_r($myCollect->sort(function($above, $below)
{
    $belowIsId = is_numeric($below);
    $aboveIsId = is_numeric($above);
    $belowIsMail = preg_match('/\w+@\w+.[\w\.]+/', $below);
    $aboveIsMail = preg_match('/\w+@\w+.[\w\.]+/', $above);

    $pos = 0;
    if($belowIsId)
        $pos = 1;
    elseif($belowIsMail)
        $pos = 1;
    elseif($aboveIsId || $aboveIsMail)
        $pos = -1;

    return $pos;

})->toArray());

/* output must be:
    Array (
        [4] => 954
        [1] => foobar@gmail.com
        [0] => foo
        [2] => myName
        [3] => myInfo
    )
*/
</code>```

See more details how to use method sort here, but look that the keys are preserved, the id are first but your key in array still the same

Title: How change position element in Array?

Body:
I have project in Laravel 5. Project work fine, but I have problem with sort position in my array.
I have array with wrong position ID and email:

Array
(
    [0] => Array
        (
            [0] => sda@swqdqwdwq.pll
            [1] => 957
            [2] => xxxx
            [3] => xxxx
            [4] => xxxx
            [5] => 2021-01-19
            [6] => tak
            [7] =>

     Przeglądaj

     Edytuj

     Usuń

     Dezaktywuj

        )

    [1] => Array
        (
            [0] => xxxx@op.pl
            [1] => 958
            [2] => xxxx
            [3] => xxxx
            [4] => firmowe
            [5] => 2021-02-03
            [6] => tak
            [7] =>

     Przeglądaj

     Edytuj

     Usuń

     Dezaktywuj

        )
...
}

My php code:

$data = $this->getList();
print_r($data);

I need change position ID and email (first position ID, second email).
I need result:

Array
    (
        [0] => Array
            (
                [0] => 957
                [1] => sda@swqdqwdwq.pll
                [2] => xxxx
                [3] => xxxx
                [4] => xxxx
                [5] => 2021-01-19
                [6] => tak
                [7] =>

         Przeglądaj

         Edytuj

         Usuń

         Dezaktywuj

            )

How can I make it?
I have project in old Laravel 5.

Please help me

Body:
Try it:

<?php

$arr = ['foo', 'foobar@gmail.com', 'myName', 'myInfo', 954];
$arrMask = ['/\d+/', '/\w+@\w+.\w+/', '/*/', '/*/', '/*/'];

print_r(sortArrayByExp($arr, $arrMask));

/**
 * Sort a array by RegExp
 *
 * Sort a array by regular expression
 *
 * @param   array   $arr        Array to sort
 * @param   array   $arrRegExp  Array to postion Regex
 *
 * @return  array   Return a new array mirrored by value in $arrRegExp
 *
 * @throws  \InvalidArgumentException
 */
function sortArrayByExp($arr, $arrRegExp)
{

    if(! is_array($arr))
        throw new \InvalidArgumentException('Value in parameter $arr must be array!', 556);

    if(! is_array($arrRegExp))
        throw new \InvalidArgumentException('Value in parameter $arrRegExp must be array!', 557);

    $newArray = [];

    foreach($arrRegExp as $key => $mask)
    {
        if(! is_string($mask))
            throw new \InvalidArgumentException('Each value in $arrRegExp must be string or RegExp', 558);

        $response = null;
        if(@preg_match($mask, '') !== false)
        {
            foreach($arr as $posArray => $value)
                if(preg_match($mask, $value))
                {
                    $response = $value;
                    unset($arr[$posArray]);
                    break;
                }

        }

        if($response !== null)
           $newArray[$key] = $response;

    }

    return $newArray;
}

/* output must be:
    Array
    (
        [0] => 954
        [1] => foobar@gmail.com
        [2] => foo
        [3] => myName
        [4] => myInfo
    )
*/

This function sort a array by each value in new array that contains a regular expression, try my exemple here

Edit 1:

You can use too collect of Laravel to sort your array, like this:

<?php
require "vendor/autoload.php";

$myCollect = collect(['foo', 'foobar@gmail.com', 'myName', 'myInfo', 954]);

print_r($myCollect->sort(function($above, $below)
{
    $belowIsId = is_numeric($below);
    $aboveIsId = is_numeric($above);
    $belowIsMail = preg_match('/\w+@\w+.[\w\.]+/', $below);
    $aboveIsMail = preg_match('/\w+@\w+.[\w\.]+/', $above);

    $pos = 0;
    if($belowIsId)
        $pos = 1;
    elseif($belowIsMail)
        $pos = 1;
    elseif($aboveIsId || $aboveIsMail)
        $pos = -1;

    return $pos;

})->toArray());

/* output must be:
    Array (
        [4] => 954
        [1] => foobar@gmail.com
        [0] => foo
        [2] => myName
        [3] => myInfo
    )
*/

See more details how to use method sort here, but look that the keys are preserved, the id are first but your key in array still the same


问:
Can I create K6 load test for the video stream
My project has pages with videos added from youtube. Can I do simulation using k6 that virtual users see those videos? </p>

答:
I have to ask, what is your end goal here? Generally speaking there is rarely a good reason to test a third party resource. You have no way to control their performance/fix issues and it typically violates TOS to do so.  In short, if you aren't google, why would you load test YouTube? Load and performance testing should be focused on systems you control.

That said - testing of streaming video is possible with k6, if transported over HTTP(s). You would need to inspect how the content is delivered and the mechanism in place to to get the next part of the video stream. It would require some deal of scripting to get working, but is possible.  To be clear, there is no player launched, so it would just be the underlying data that is loaded.


问:
How to find & replace a string appearing at nth position in a pipe delimited csv file using powershell
I have csv files which contain data as per below sample

MACADD||TEST|Street1|CITY||Taiwan|||10000000|||FIRE||Taiwan||||||||12 days||30 Days|DDTE||812148709231890||124-Taiwan|DENE|||

I want to replace the 7th value in every csv files but problem is when I replace it the entire data gets separated by comma as the replacement value contained comma in it.

is there any way to ignore/escape commas while after the replacement has happened so that entire data appears in 1st cell of the final csv file.

```<code>$Files = Get-content -Path F:\Newfolder\*.csv

 $CountryCodeLookup = @{
    'USA'= 'United States'
    'Taiwan' = "Taiwan, Republic of China" # for this item csv file become strange
    'Delhi' = "Delhi, Capital of India, U.T" # for this item csv file become strange
    }

foreach ($File in $Files)
    {
    $DelimCount = ($File -replace '[^|]', '').Length
    $CSV_Thing = ConvertFrom-Csv -Delimiter '|' -InputObject $File -Header @(1..$DelimCount)

    If ($CountryCodeLookup.ContainsKey($CSV_Thing.7))
    {
    $CSV_Thing.7 = $CountryCodeLookup[$CSV_Thing.7]
    }

    $OutString = (($CSV_Thing |
        ConvertTo-Csv -Delimiter '|' -NoTypeInformation |
        Select-Object -Skip 1) -replace '"', '') + '|' | Set-Content $File.PSPath -Force

    }

</code>```

答:
```<code>$Files = Get-content -Path F:\Newfolder\*.txt
$Path = "F:\Newfolder"

 $CountryCodeLookup = @{
    'USA'= 'United States'
    'Taiwan' = "Taiwan, Republic of China" # for this item csv file become strange
    'Delhi' = "Delhi, Capital of India, U.T" # for this item csv file become strange
    }

foreach ($File in $Files)
    {
    $DelimCount = ($File -replace '[^|]', '').Length
    $CSV_Thing = ConvertFrom-Csv -Delimiter '|' -InputObject $File -Header @(1..$DelimCount)

    If ($CountryCodeLookup.ContainsKey($CSV_Thing.7))
    {
    $CSV_Thing.7 = $CountryCodeLookup[$CSV_Thing.7]
    }

    $OutString = (($CSV_Thing |
        ConvertTo-Csv -Delimiter '|' -NoTypeInformation |
        Select-Object -Skip 1) -replace '"', '') + '|' | Set-Content $File.PSPath -Force

    }

    $files = Get-ChildItem $Path | where { ! $_.PSIsContainer }
foreach ($file in $files){
    $newFileName = ($file.Fullname) -Replace ".txt",".csv"
    Get-Content $file.FullName | Out-File $newFileName
}
</code>```

Body:
I have csv files which contain data as per below sample

MACADD||TEST|Street1|CITY||Taiwan|||10000000|||FIRE||Taiwan||||||||12 days||30 Days|DDTE||812148709231890||124-Taiwan|DENE|||

I want to replace the 7th value in every csv files but problem is when I replace it the entire data gets separated by comma as the replacement value contained comma in it.

is there any way to ignore/escape commas while after the replacement has happened so that entire data appears in 1st cell of the final csv file.

$Files = Get-content -Path F:\Newfolder\*.csv

 $CountryCodeLookup = @{
    'USA'= 'United States'
    'Taiwan' = "Taiwan, Republic of China" # for this item csv file become strange
    'Delhi' = "Delhi, Capital of India, U.T" # for this item csv file become strange
    }

foreach ($File in $Files)
    {
    $DelimCount = ($File -replace '[^|]', '').Length
    $CSV_Thing = ConvertFrom-Csv -Delimiter '|' -InputObject $File -Header @(1..$DelimCount)

    If ($CountryCodeLookup.ContainsKey($CSV_Thing.7))
    {
    $CSV_Thing.7 = $CountryCodeLookup[$CSV_Thing.7]
    }

    $OutString = (($CSV_Thing |
        ConvertTo-Csv -Delimiter '|' -NoTypeInformation |
        Select-Object -Skip 1) -replace '"', '') + '|' | Set-Content $File.PSPath -Force

    }


问:
Classification method for unevenly spread data
I have several datasets with very unevenly distributed values: Most values are very low, but a few are very high, for example, in the histogram screenshot or even more extreme.

I am actually interested in the differences in the high values.

So what I am looking for is a classification method that sets many break values where there are few data values and large classes where there are many values. Maybe something like a reversed quantile classification.

Do you have a suggestion on which algorithm could help with this task, preferably in Python?

答:
if you are using pandas, couldn't you just select the values above your chosen threshold and analyze the difference seperately?</p>

```<code>import pandas as pd

df = pd.DataFrame(your data)

df_to_analyze_large_values = df[df.your_Column_of_interest > 100000]

</code>```

Body:
if you are using pandas, couldn't you just select the values above your chosen threshold and analyze the difference seperately?

import pandas as pd

df = pd.DataFrame(your data)

df_to_analyze_large_values = df[df.your_Column_of_interest > 100000]


问:
Is Guice's @ImplementedBy evil? Is it appropriate in some cases?
I've heard claims that "@ImplementedBy is evil", on grounds that it breaks DI concepts and makes the interface aware of its implementors.

This might be true in some cases, but often I found that it just leads to cleaner code (no long Modules to maintain), while not really hurting anything in the process.

As pragmatics, not purists, when do you think it's worthwhile to use @ImplementedBy?

答:
You should generally prefer explicit bindings over just-in-time (JIT) bindings. Explicit bindings allow the injector to crawl the dependency graph at injector-creation time. This allows Guice to fail-fast if a dependency is missing or invalid. With just-in-time bindings like @ImplementedBy, Guice can't report the problem until the binding is exercised.

JIT bindings also interact poorly with PrivateModules/child injectors. Though most apps shouldn't need these features, when you do it's less painful if every binding belongs to a specific module.


问:
How to run script with boost::process::system?
My script:

ps aux | awk '{s += $3} END {print s}'

So I'm trying to execute it

```<code>namespace bp = boost::process;

bp::ipstream out;
bp::system("ps aux | awk '{s += $3} END {print s}'", bp::std_out > out);

std::string line;
std::getline(out, line);
</code>```

Output:

```<code>error: garbage option
</code>```

What am I doing wrong?

答:
You need to make a pipe object (p in this case). Then pass the output from the first process to the next one.

```<code>bp::ipstream out;
bp::pipe p;
bp::system("ps aux", bp::std_out > p);
bp::system("awk '{s += $3} END {print s}'", bp::std_in < p, bp::std_out > out);
</code>```

Referring the boost documentation.

Body:
My script:

ps aux | awk '{s += $3} END {print s}'

So I'm trying to execute it

namespace bp = boost::process;

bp::ipstream out;
bp::system("ps aux | awk '{s += $3} END {print s}'", bp::std_out > out);

std::string line;
std::getline(out, line);

Output:

error: garbage option

What am I doing wrong?

Body:
You need to make a pipe object (p in this case). Then pass the output from the first process to the next one.

bp::ipstream out;
bp::pipe p;
bp::system("ps aux", bp::std_out > p);
bp::system("awk '{s += $3} END {print s}'", bp::std_in < p, bp::std_out > out);

Referring the boost documentation.


问:
SpringBoot openJ9 Kubernetes Crash InterpreterEmulator.cpp:208
I'm randomly getting this crash recently and I'm not sure how to debug this. Can anyone please point me in the right direction.
Tools I have used are,

Spring Boot 2 + Kubernetes + adopt openj9 jdk13

```<code>Assertion failed at /home/jenkins/workspace/build-scripts/jobs/jdk13u/jdk13u-linux-x64-openj9/workspace/build/src/build/linux-x86_64-server-release/vm/compiler/../compiler/optimizer/InterpreterEmulator.cpp:208: 0
VMState: 0x000501ff
        unexpected bytecode in thunk archetype 0x7f98f6483ee0 at bcIndex 40 JBdload2 (41)
compiling java/lang/invoke/AsTypeHandle.invokeExact_thunkArchetype_X(Ljava/lang/Object;Ljava/lang/Object;)V at level: warm
0:/opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x798015) [0x7f9962316015]
1: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x7a2dc0) [0x7f9962320dc0]
JVMDUMP039I Processing dump event “abort”, detail “” at 2020/05/12 16:35:10 - please wait.
JVMDUMP032I JVM requested System dump using ‘/home/springboot/core.20200512.163510.1.0001.dmp’ in response to an event
</code>```

答:
TLDR: This is fixed in the v.0.20.0 release (April 2020) of OpenJ9. The latest release binaries from AdoptOpenJDK for JDK 8, 11 & 14 contain the fix.

The assertion failure is an assertion JIT compiling the MethodHandle (MH) code.  A new abstract interpreter was added to give the JIT a better view of the MH "thunkArchetype" code and it missed some of the bytecodes that could occur in MH thunkArchetypes.

See https://github.com/eclipse/openj9/pull/8598 for more details of the fix.

Body:
I'm randomly getting this crash recently and I'm not sure how to debug this. Can anyone please point me in the right direction.
Tools I have used are,

Spring Boot 2 + Kubernetes + adopt openj9 jdk13

Assertion failed at /home/jenkins/workspace/build-scripts/jobs/jdk13u/jdk13u-linux-x64-openj9/workspace/build/src/build/linux-x86_64-server-release/vm/compiler/../compiler/optimizer/InterpreterEmulator.cpp:208: 0
VMState: 0x000501ff
        unexpected bytecode in thunk archetype 0x7f98f6483ee0 at bcIndex 40 JBdload2 (41)
compiling java/lang/invoke/AsTypeHandle.invokeExact_thunkArchetype_X(Ljava/lang/Object;Ljava/lang/Object;)V at level: warm
0:/opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x798015) [0x7f9962316015]
1: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x7a2dc0) [0x7f9962320dc0]
JVMDUMP039I Processing dump event “abort”, detail “” at 2020/05/12 16:35:10 - please wait.
JVMDUMP032I JVM requested System dump using ‘/home/springboot/core.20200512.163510.1.0001.dmp’ in response to an event


问:
Conditional NULL replacement
I am having a table with many columns:(previous post had just one id per one source_id, but this case has more than one id per one source_id)

```<code>id    col1             source_id
a1    765.3            a5
a2    3298.3           a4
a3    8762.1           a8
a4    3298.3           (null)
a5    (null)           a6
a6    (null)           (null)
a7    10               a5
</code>```

I want to fill null values of source _id with values from id.
For example, source_id a5 row has null which has to replaced with id a1 + id a7 values, subsequently, source_id a6 row having null to be replaced with a5 row

Output:

```<code>id    col1             source_id
a1    765.3            a5
a2    3298.3           a4
a3    8762.1           a8
a4    3298.3           (null)
a5    765.3+10=775.3   a6
a6    765.3+10=775.3   (null)
a7    10               a5
</code>```

Thanks!!

EDIT:

To make it more clear, null values in columns other than source_id and id needs to be filled. Given only col1 for simplified post, there might be many columns as well

答:
The next select will give you those rows that have a reference in the source_id column and value in the col1 column.

```<code>select id
       , case when col1 is null then
              (select sum(col1) from test tt where tt.sourceid = t.id)
         else
              col1
         end result
       , sourceid
from test t
order by id;
</code>```

here is a demo

This is an option for the complete solution:

```<code>with cte as (select t.id
       , case when t.col1 is null then
              (select sum(tt.col1) from test tt where tt.sourceid = t.id)
         else
              t.col1
         end result
       , t.sourceid
from test t
order by id)
select t.id
       ,
          case when t.result is null then
              (select sum(tt.result) from cte tt where tt.sourceid = t.id)
         else
              t.result
         end result
       , t.sourceid
from cte t
order by id
</code>```

here is a demo

Title: Conditional NULL replacement

Body:
I am having a table with many columns:(previous post had just one id per one source_id, but this case has more than one id per one source_id)

id    col1             source_id
a1    765.3            a5
a2    3298.3           a4
a3    8762.1           a8
a4    3298.3           (null)
a5    (null)           a6
a6    (null)           (null)
a7    10               a5

I want to fill null values of source _id with values from id.
For example, source_id a5 row has null which has to replaced with id a1 + id a7 values, subsequently, source_id a6 row having null to be replaced with a5 row

Output:

id    col1             source_id
a1    765.3            a5
a2    3298.3           a4
a3    8762.1           a8
a4    3298.3           (null)
a5    765.3+10=775.3   a6
a6    765.3+10=775.3   (null)
a7    10               a5

Thanks!!

EDIT:

To make it more clear, null values in columns other than source_id and id needs to be filled. Given only col1 for simplified post, there might be many columns as well

Body:
The next select will give you those rows that have a reference in the source_id column and value in the col1 column.

select id
       , case when col1 is null then
              (select sum(col1) from test tt where tt.sourceid = t.id)
         else
              col1
         end result
       , sourceid
from test t
order by id;

here is a demo

This is an option for the complete solution:

with cte as (select t.id
       , case when t.col1 is null then
              (select sum(tt.col1) from test tt where tt.sourceid = t.id)
         else
              t.col1
         end result
       , t.sourceid
from test t
order by id)
select t.id
       ,
          case when t.result is null then
              (select sum(tt.result) from cte tt where tt.sourceid = t.id)
         else
              t.result
         end result
       , t.sourceid
from cte t
order by id

here is a demo


问:
Issue with image and link alignment
So i've got this issue with my CSS where I cannot display a text-link inline with my image.
The links are on a different height relative to my image? What can the issue be? I've tried using different alignment methods such as vertical-align etc.

Maybe I should rewrite the whole code?

```<code>html,
body {
    font-family: 'Open Sans', sans-serif;
    background-color: #efefef;
}

body{
    margin: 0;
}

.left_div {
    margin: 0;
    padding: 0;
    width: 25%;
    background-color: #fff;
    position: fixed;
    height: 100%;
    overflow: auto;
    text-align: center;
}

.left_div img {
    width: 50%;
    height: auto;
    margin: 2vh 0 2vh 0;
}

.left_div a {
    display: block;
    color: black;
    background-color: rgb(217, 50, 141);
    padding: 16px;
    text-decoration: none;
    width: 75%;
    margin: 2vh auto;
    border-radius: 5px;
    color: rgb(255, 255, 255);
    transition: all .2s ease-in-out;
}

.left_div a.active {
    background-color: rgb(65, 29, 100);
    color: white;
}

.left_div a:hover:not(.active) {
    background-color: rgb(153, 22, 94);
    color: white;
}

@media screen and (min-width: 765px) and (max-width: 968px) {
    .left_div {
        width: 100%;
        height: auto;
        position: relative;
        text-align: left;
    }

    .left_div img {
        width: 12.5%;
        height: auto;
        margin: 0 0 0 2vw;
    }

    .left_div a {
        display: inline;
        color: black;
        background-color: rgb(217, 50, 141);
        padding: 16px;
        text-decoration: none;
        margin: 0 auto;
        border-radius: 5px;
        color: rgb(255, 255, 255);
        transition: all .2s ease-in-out;

    }
}

@media screen and (max-width: 765px) {
    .left_div {
        width: 100%;
        height: auto;
        position: relative;
    }
    .left_div a {
        text-align: center;
        float: none;
    }
}

<!-- HTML -->

    <div class="left_div">

        <img src="img/mlp_logo.png" alt="My Little Pony Logo">

        <a class="active" href="#">Home</a>
        <a href="#">News</a>
        <a href="#">Contact</a>
        <a href="#">About</a>
      </div>

</code>```

Here is a jsfiddle to show the issue.

https://jsfiddle.net/bcyu0f52

Thanks!

edit (screenshot added)

https://imgur.com/hwbmAKR
As you see on the screenshot the image and text are misaligned height-wise.

答:
Solved by @SaschaM78.

I think you want to have your a tags vertically aligned with the image, right? If  so, add vertical-align: middle; to .left_div img. Updated fiddle can be found here - @SaschaM78

Thanks! A little noob mistake by me.

Title: Issue with image and link alignment

Body:
So i've got this issue with my CSS where I cannot display a text-link inline with my image.
The links are on a different height relative to my image? What can the issue be? I've tried using different alignment methods such as vertical-align etc.

Maybe I should rewrite the whole code?

html,
body {
    font-family: 'Open Sans', sans-serif;
    background-color: #efefef;
}

body{
    margin: 0;
}

.left_div {
    margin: 0;
    padding: 0;
    width: 25%;
    background-color: #fff;
    position: fixed;
    height: 100%;
    overflow: auto;
    text-align: center;
}

.left_div img {
    width: 50%;
    height: auto;
    margin: 2vh 0 2vh 0;
}

.left_div a {
    display: block;
    color: black;
    background-color: rgb(217, 50, 141);
    padding: 16px;
    text-decoration: none;
    width: 75%;
    margin: 2vh auto;
    border-radius: 5px;
    color: rgb(255, 255, 255);
    transition: all .2s ease-in-out;
}

.left_div a.active {
    background-color: rgb(65, 29, 100);
    color: white;
}

.left_div a:hover:not(.active) {
    background-color: rgb(153, 22, 94);
    color: white;
}

@media screen and (min-width: 765px) and (max-width: 968px) {
    .left_div {
        width: 100%;
        height: auto;
        position: relative;
        text-align: left;
    }

    .left_div img {
        width: 12.5%;
        height: auto;
        margin: 0 0 0 2vw;
    }

    .left_div a {
        display: inline;
        color: black;
        background-color: rgb(217, 50, 141);
        padding: 16px;
        text-decoration: none;
        margin: 0 auto;
        border-radius: 5px;
        color: rgb(255, 255, 255);
        transition: all .2s ease-in-out;

    }
}

@media screen and (max-width: 765px) {
    .left_div {
        width: 100%;
        height: auto;
        position: relative;
    }
    .left_div a {
        text-align: center;
        float: none;
    }
}

        Home
        News
        Contact
        About

Here is a jsfiddle to show the issue.

https://jsfiddle.net/bcyu0f52

Thanks!

edit (screenshot added)

https://imgur.com/hwbmAKR
As you see on the screenshot the image and text are misaligned height-wise.


问:
Send alert notification with series value and rich text format

I saw this grafana test alert notification and would like to do the same for my charts. But I cannot find documentation on how to achieve this. I need to at least show metrics value and an error message if possible

答:
When creating the alert for your dashboard you can specify the alert message.

By default it will send you the value and name of the time-series that triggered the alert.

To enable Grafana to send images you have to set up External Image Storage and enable it in the Notification Channel settings.

Hope this helps


问:
How to tell teamcity agent, which aws account to run terraform to
I spun up TeamCity into a Kubernetes cluster. I have one deployment for the TeamCity server and one deployment for the TeamCity build agent. When I run Terraform on the TeamCity build agent, it creates resources in the same AWS account as the EC2 instance which is hosting TeamCity Kubernetes cluster.

I would like to run a build, which creates AWS resources in a separate AWS account. My thinking is to assign an AWS access key and secret key to the TeamCity server and pass them to the TeamCity build agent, but I don't know how the work flow is going to be.

Currently, I have declared AWS access key and secret keys as environment variables on the build but they are not getting passed to agent. My build steps only contains 3 lines

```<code>terraform init
terraform plan
terraform apply -auto-approve
</code>```

答:
I dealt with a very similar situation. When we started using AWS, we did not follow best practice around how our scripts authenticate to AWS.

We created users in each account and exported the access and secret keys into our TeamCity build automation to allow our scripts to run in the context of these accounts.

You will wind up with several users in different accounts and will wind up manually rotating the keys for these users, then updating the access and secret keys in TeamCity everytime you need to rotate the keys.

As the TeamCity build agent servers are EC2 instances, it is best practice in AWS to use roles instead of users in situations like this. From the AWS documentation:

Applications that run on an Amazon EC2 instance need credentials in
  order to access other AWS services. To provide credentials to the
  application in a secure way, use IAM roles. A role is an entity that
  has its own set of permissions, but that isn't a user or group. Roles
  also don't have their own permanent set of credentials the way IAM
  users do. In the case of Amazon EC2, IAM dynamically provides
  temporary credentials to the EC2 instance, and these credentials are
  automatically rotated for you.

Source: https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#use-roles-with-ec2

What you should do in this situation is create a role in each account that has access to create the resources that terraform will build. You will need another role for the TeamCity build agent servers, this role should be able to assume the roles created in each account.

Let's say you have two accounts:

AccountA is where you want the build agents to run build resources with terraform.

AccountB is the account your build agents are running from.

What you will need is:

A role in AccountA with access to create resources.

A role in AccountB that grants access for ec2 instances to assume the above role.

The instance profile of the role in AccountB should be assigned to the TeamCity build agent servers. The build agent servers are then able to assume the role in AccountA and run in the context of that account with the permissions they need without the need for hard-coded access and secret keys.

You can repeat this process for as many other accounts as you want to build resources in through Terraform.

You will have to add the following block to the aws provider in your terraform scripts to let terraform know what role to assume:

```<code>provider "aws" {
assume_role {
role_arn = "arn:aws:iam::<ACCOUNTA>:role/<ROLENAME>"
}
}
</code>```

Here is some AWS documentation on how to grant access to an S3 bucket in a different account from an EC2 instance:
https://aws.amazon.com/premiumsupport/knowledge-center/s3-instance-access-bucket/

Another useful documentation page on that covers IAM Roes, EC2 instance profiles & cross account access via Roles is here:
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html

One last link goes through exactly what you are trying to do:
https://blog.e-zest.com/aws-ec2-assume-role-with-terraform

Body:
I spun up TeamCity into a Kubernetes cluster. I have one deployment for the TeamCity server and one deployment for the TeamCity build agent. When I run Terraform on the TeamCity build agent, it creates resources in the same AWS account as the EC2 instance which is hosting TeamCity Kubernetes cluster.

I would like to run a build, which creates AWS resources in a separate AWS account. My thinking is to assign an AWS access key and secret key to the TeamCity server and pass them to the TeamCity build agent, but I don't know how the work flow is going to be.

Currently, I have declared AWS access key and secret keys as environment variables on the build but they are not getting passed to agent. My build steps only contains 3 lines

terraform init
terraform plan
terraform apply -auto-approve

Body:
I dealt with a very similar situation. When we started using AWS, we did not follow best practice around how our scripts authenticate to AWS.

We created users in each account and exported the access and secret keys into our TeamCity build automation to allow our scripts to run in the context of these accounts.

You will wind up with several users in different accounts and will wind up manually rotating the keys for these users, then updating the access and secret keys in TeamCity everytime you need to rotate the keys.

As the TeamCity build agent servers are EC2 instances, it is best practice in AWS to use roles instead of users in situations like this. From the AWS documentation:

Applications that run on an Amazon EC2 instance need credentials in
  order to access other AWS services. To provide credentials to the
  application in a secure way, use IAM roles. A role is an entity that
  has its own set of permissions, but that isn't a user or group. Roles
  also don't have their own permanent set of credentials the way IAM
  users do. In the case of Amazon EC2, IAM dynamically provides
  temporary credentials to the EC2 instance, and these credentials are
  automatically rotated for you.

Source: https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#use-roles-with-ec2

What you should do in this situation is create a role in each account that has access to create the resources that terraform will build. You will need another role for the TeamCity build agent servers, this role should be able to assume the roles created in each account.

Let's say you have two accounts:

AccountA is where you want the build agents to run build resources with terraform.

AccountB is the account your build agents are running from.

What you will need is:

A role in AccountA with access to create resources.

A role in AccountB that grants access for ec2 instances to assume the above role.

The instance profile of the role in AccountB should be assigned to the TeamCity build agent servers. The build agent servers are then able to assume the role in AccountA and run in the context of that account with the permissions they need without the need for hard-coded access and secret keys.

You can repeat this process for as many other accounts as you want to build resources in through Terraform.

You will have to add the following block to the aws provider in your terraform scripts to let terraform know what role to assume:

provider "aws" {
assume_role {
role_arn = "arn:aws:iam::<ACCOUNTA>:role/<ROLENAME>"
}
}

Here is some AWS documentation on how to grant access to an S3 bucket in a different account from an EC2 instance:
https://aws.amazon.com/premiumsupport/knowledge-center/s3-instance-access-bucket/

Another useful documentation page on that covers IAM Roes, EC2 instance profiles & cross account access via Roles is here:
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html

One last link goes through exactly what you are trying to do:
https://blog.e-zest.com/aws-ec2-assume-role-with-terraform


问:
Compare String with each element of a String Array using Streams
Hi i want to count how many time a String is found in a Array of Strings using Streams

What i have thought so far is this:

```<code>Stream<String> stream=Arrays.stream(array);

int counter= (int) stream.filter(c-> c.contains("something")).count();

return counter;
</code>```

The problem that i get is that most of the time i get an error of NullPointerException and i think is because of .count() if it doesn't get any much inside filter(c-> c.contains("something")).
And i came to this conclusion cause if i run it with out .count() like that stream.filter(c-> c.contains("something")); without returning nothing, it won't throw an Exception. I'm not sure about it but that's what i think.

Any ideas on how i can count the times a String appears in and Array of Strings using Streams?

答:
You have to filter for null values first. Do it either the way @pafauk. answered or by filtering sepraretly. That requires the null filter to be applied before the one you already use:

```<code>public static void main(String[] args) {
    List<String> chainedChars = new ArrayList<>();
    chainedChars.add("something new");  // match
    chainedChars.add("something else"); // match
    chainedChars.add("anything new");
    chainedChars.add("anything else");
    chainedChars.add("some things will never change");
    chainedChars.add("sometimes");
    chainedChars.add(null);
    chainedChars.add("some kind of thing");
    chainedChars.add("sumthin");
    chainedChars.add("I have something in mind");   // match
    chainedChars.add("handsome thing");

    long somethings = chainedChars.stream()
                                    .filter(java.util.Objects::nonNull)
                                    .filter(cc -> cc.contains("something"))
                                    .count();

    System.out.printf("Found %d somethings", somethings);
}
</code>```

outputs

Found 3 somethings
```

while switching the filter lines will result in a NullPointerException.

Body:
Hi i want to count how many time a String is found in a Array of Strings using Streams

What i have thought so far is this:

Stream<String> stream=Arrays.stream(array);

int counter= (int) stream.filter(c-> c.contains("something")).count();

return counter;

The problem that i get is that most of the time i get an error of NullPointerException and i think is because of .count() if it doesn't get any much inside filter(c-> c.contains("something")).
And i came to this conclusion cause if i run it with out .count() like that stream.filter(c-> c.contains("something")); without returning nothing, it won't throw an Exception. I'm not sure about it but that's what i think.

Any ideas on how i can count the times a String appears in and Array of Strings using Streams?

Body:
You have to filter for null values first. Do it either the way @pafauk. answered or by filtering sepraretly. That requires the null filter to be applied before the one you already use:

public static void main(String[] args) {
    List<String> chainedChars = new ArrayList<>();
    chainedChars.add("something new");  // match
    chainedChars.add("something else"); // match
    chainedChars.add("anything new");
    chainedChars.add("anything else");
    chainedChars.add("some things will never change");
    chainedChars.add("sometimes");
    chainedChars.add(null);
    chainedChars.add("some kind of thing");
    chainedChars.add("sumthin");
    chainedChars.add("I have something in mind");   // match
    chainedChars.add("handsome thing");

    long somethings = chainedChars.stream()
                                    .filter(java.util.Objects::nonNull)
                                    .filter(cc -> cc.contains("something"))
                                    .count();

    System.out.printf("Found %d somethings", somethings);
}

outputs

Found 3 somethings

while switching the filter lines will result in a NullPointerException.


问:
How to implement curl -u in Python?
I am trying to use http://developer.github.com/v3/ to retrieve project issues. This works:

```<code>curl -u "Littlemaple:mypassword" https://api.github.com/repos/MyClient/project/issues
</code>```

It returns all private issues of my client's project. However, I am not able to find out how to implement this in Python. Both ways I have found (e.g. Python urllib2 Basic Auth Problem) doesn't work, they return 404 or 403 errors:

```<code>def fetch(url, username, password):
    """Wonderful method found on forums which does not work.""""
    passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
    passman.add_password(None, url, username, password)
    urllib2.install_opener(urllib2.build_opener(urllib2.HTTPBasicAuthHandler(passman)))

    req = urllib2.Request(url)
    f = urllib2.urlopen(req)
    return f.read()
</code>```

...and:

```<code>def fetch(url, username, password):
    """Wonderful method found on forums which does not work neither.""""
    request = urllib2.Request(url)
    base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
    request.add_header("Authorization", "Basic %s" % base64string)
    return urllib2.urlopen(request).read()
</code>```

Any ideas? Thanks in advance!

答:
If it's 404, you probably just have the wrong URL. If it's 403, perhaps you have the realm wrong.

For starters, you're passing the URL to add_password, when in fact you should only be passing the base URL. Also, instead of install_opener, you should probably just create a new opener.

See this recipe for an example:

```<code>class NoOpHandler(urllib2.HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newUrl):
        return None

passmanager = urllib2.HTTPPasswordMgrWithDefaultRealm()
passmanager.add_password(None, baseurl, username, password)
auth_handler = urllib2.HTTPBasicAuthHandler(passmanager)
opener = urllib2.build_opener(auth_handler, NoOpHandler())
</code>```

Title: How to implement curl -u in Python?

Body:
I am trying to use http://developer.github.com/v3/ to retrieve project issues. This works:

curl -u "Littlemaple:mypassword" https://api.github.com/repos/MyClient/project/issues

It returns all private issues of my client's project. However, I am not able to find out how to implement this in Python. Both ways I have found (e.g. Python urllib2 Basic Auth Problem) doesn't work, they return 404 or 403 errors:

def fetch(url, username, password):
    """Wonderful method found on forums which does not work.""""
    passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
    passman.add_password(None, url, username, password)
    urllib2.install_opener(urllib2.build_opener(urllib2.HTTPBasicAuthHandler(passman)))

    req = urllib2.Request(url)
    f = urllib2.urlopen(req)
    return f.read()

...and:

def fetch(url, username, password):
    """Wonderful method found on forums which does not work neither.""""
    request = urllib2.Request(url)
    base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
    request.add_header("Authorization", "Basic %s" % base64string)
    return urllib2.urlopen(request).read()

Any ideas? Thanks in advance!

Body:
If it's 404, you probably just have the wrong URL. If it's 403, perhaps you have the realm wrong.

For starters, you're passing the URL to add_password, when in fact you should only be passing the base URL. Also, instead of install_opener, you should probably just create a new opener.

See this recipe for an example:

class NoOpHandler(urllib2.HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newUrl):
        return None

passmanager = urllib2.HTTPPasswordMgrWithDefaultRealm()
passmanager.add_password(None, baseurl, username, password)
auth_handler = urllib2.HTTPBasicAuthHandler(passmanager)
opener = urllib2.build_opener(auth_handler, NoOpHandler())


问:
Filtering White Space from String in powershell
I am trying to filter a file name down for example, what i want to do is filter to the underscore and leave the rest.
Info10_ComputerHostName
so i run

```<code>$FileName="Info10_ComputerHostName"
$FinalFileName = $FileName -split("*._")
</code>```

When i output the variable what i get
" ComputerHostName"

What I want
"ComputerHostName"

How do I remove the whitespace that it always returns?

答:
```<code>$FinalFileName = ([regex]::Match($FileName,'([^_]*)_(.*)'))
$FinalFileName.Groups[2].Value

#or

$FileName.Split('_')[1]

</code>```

Body:
I am trying to filter a file name down for example, what i want to do is filter to the underscore and leave the rest.
Info10_ComputerHostName
so i run

$FileName="Info10_ComputerHostName"
$FinalFileName = $FileName -split("*._")

When i output the variable what i get
" ComputerHostName"

What I want
"ComputerHostName"

How do I remove the whitespace that it always returns?


问:
Convert table's selected columns to rows in SQL server and keep the Column title as value
I am trying t convert the following table to the detailed table for reporting:

Sample:

Item Name	Item location	Item code	Item type	Item number	4/1/2020	4/2/2020	4/3/2020	4/4/2020	4/5/2020

Milk	Location 1	1024	Food-1	1	 1,548 	 1,561 	 6,584 	 1,344 	 1,485

Bread	Location 2	1524	Food-2	2	 -   	 -   	 -   	 1,524 	 1,597

Toast	Location 2	1525	Food-3	4	 1,504 	 1,526 	 5,423 	 1,434 	 1,244

WheatBread	Location 3	1600	Food-2	5	 1,587 	 1,547 	 1,545 	 8,896 	 1,236

```
</div>
</div>
</p>

<p>the final output will be like this:</p>

<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-html lang-html prettyprint-override"><code><table><tbody><tr><th>Item Name</th><th>Item location</th><th>Item code</th><th>Item type</th><th>Item number</th><th>Values</th><th>Date Posted</th></tr><tr><td>Milk</td><td>Location 1</td><td>1024</td><td>Food-1</td><td>1</td><td> 1,548 </td><td>4/1/2020</td></tr><tr><td>Bread</td><td>Location 2</td><td>1524</td><td>Food-2</td><td>2</td><td> -   </td><td>4/1/2020</td></tr><tr><td>Toast</td><td>Location 2</td><td>1525</td><td>Food-3</td><td>4</td><td> 1,504 </td><td>4/1/2020</td></tr><tr><td>WheatBread</td><td>Location 3</td><td>1600</td><td>Food-2</td><td>5</td><td> 1,587 </td><td>4/1/2020</td></tr><tr><td>Milk</td><td>Location 1</td><td>1024</td><td>Food-1</td><td>1</td><td> 1,561 </td><td>4/2/2020</td></tr><tr><td>Bread</td><td>Location 2</td><td>1524</td><td>Food-2</td><td>2</td><td> -   </td><td>4/2/2020</td></tr><tr><td>Toast</td><td>Location 2</td><td>1525</td><td>Food-3</td><td>4</td><td> 1,526 </td><td>4/2/2020</td></tr><tr><td>WheatBread</td><td>Location 3</td><td>1600</td><td>Food-2</td><td>5</td><td> 1,547 </td><td>4/2/2020</td></tr><tr><td>Milk</td><td>Location 1</td><td>1024</td><td>Food-1</td><td>1</td><td> 6,584 </td><td>4/3/2020</td></tr><tr><td>Bread</td><td>Location 2</td><td>1524</td><td>Food-2</td><td>2</td><td> -   </td><td>4/3/2020</td></tr><tr><td>Toast</td><td>Location 2</td><td>1525</td><td>Food-3</td><td>4</td><td> 5,423 </td><td>4/3/2020</td></tr><tr><td>WheatBread</td><td>Location 3</td><td>1600</td><td>Food-2</td><td>5</td><td> 1,545 </td><td>4/3/2020</td></tr><tr><td>Milk</td><td>Location 1</td><td>1024</td><td>Food-1</td><td>1</td><td> 1,344 </td><td>4/4/2020</td></tr><tr><td>Bread</td><td>Location 2</td><td>1524</td><td>Food-2</td><td>2</td><td> 1,524 </td><td>4/4/2020</td></tr><tr><td>Toast</td><td>Location 2</td><td>1525</td><td>Food-3</td><td>4</td><td> 1,434 </td><td>4/4/2020</td></tr><tr><td>WheatBread</td><td>Location 3</td><td>1600</td><td>Food-2</td><td>5</td><td> 8,896 </td><td>4/4/2020</td></tr><tr><td>Milk</td><td>Location 1</td><td>1024</td><td>Food-1</td><td>1</td><td> 1,485 </td><td>4/5/2020</td></tr><tr><td>Bread</td><td>Location 2</td><td>1524</td><td>Food-2</td><td>2</td><td> 1,597 </td><td>4/5/2020</td></tr><tr><td>Toast</td><td>Location 2</td><td>1525</td><td>Food-3</td><td>4</td><td> 1,244 </td><td>4/5/2020</td></tr><tr><td>WheatBread</td><td>Location 3</td><td>1600</td><td>Food-2</td><td>5</td><td> 1,236 </td><td>4/5/2020</td></tr></tbody></table></code>```

I have tried this already... but it doesn't work somewhere near the insert statement, I can create new table out of the input table... need help validating the code...

```<code>DECLARE @StartDate AS DATETIME
DECLARE @EndDate AS DATETIME
DECLARE @CurrentDate AS DATETIME
DECLARE @sqlText as nvarchar(max)

SET @StartDate = '2020-04-01'
SET @EndDate = '2020-04-06'
SET @CurrentDate = @StartDate

WHILE (@CurrentDate < @EndDate)
BEGIN

   SET @sqlText = N'SELECT
    distinct
        [Item Name],
        [Item location],
        [Item code],
        [Item type],
        [Item number],
        ['+(format(@CurrentDate, 'M/d/yyyy'))+'] as 'Qunatity',
        '+(format(@CurrentDate, 'M/d/yyyy'))+'  as [DatePosted]
    FROM [dbo].[BaseTable]';

    -- Insert into table ---> this where it's erroring

    INSERT INTO  [dbo].[FinalTable] ([Item Name],[Item location],[Item code],[Item type],[Item number],[Values],[Date Posted])
    EXEC sp_executesql (@sqlText)

    SET @CurrentDate = convert(varchar(30), dateadd(day,1, @CurrentDate), 101); /*increment current date*/
END
</code>```

答:
You can generate single query from your table definition:

  SELECT Query =
  'SELECT
        [Item Name],
        [Item location],
        [Item code],
        [Item type],
        [Item number],
        RotatedValues.Value,
        RotatedValues.DatePosted
   FROM dbo.MySource
   CROSS APPLY ('
     + STRING_AGG('SELECT Value = ' + QUOTENAME(columns.name) + ', DatePosted = ''' + columns.name + '''', ' UNION ALL ') +
   ') AS RotatedValues'
  FROM sys.tables
  INNER JOIN sys.columns ON columns.object_id = tables.object_id
  WHERE tables.name = 'MySource'
  AND columns.name LIKE '[0-9]%'
```

<p>It will generate:</p>

<pre class="lang-sql prettyprint-override"><code>SELECT [Item Name],
       [Item location],
       [Item code],
       [Item type],
       [Item number],
       RotatedValues.Value,
       RotatedValues.DatePosted
FROM dbo.MySource
CROSS APPLY (
    SELECT Value      = [4/1/2020],
           DatePosted = '4/1/2020'
    UNION ALL
    SELECT Value      = [4/2/2020],
           DatePosted = '4/2/2020'
    UNION ALL
    SELECT Value      = [4/3/2020],
           DatePosted = '4/3/2020'
) AS RotatedValues
</code>```

Body:
I am trying t convert the following table to the detailed table for reporting:

Sample:

Item Name	Item location	Item code	Item type	Item number	4/1/2020	4/2/2020	4/3/2020	4/4/2020	4/5/2020

Milk	Location 1	1024	Food-1	1	 1,548 	 1,561 	 6,584 	 1,344 	 1,485

Bread	Location 2	1524	Food-2	2	 -   	 -   	 -   	 1,524 	 1,597

Toast	Location 2	1525	Food-3	4	 1,504 	 1,526 	 5,423 	 1,434 	 1,244

WheatBread	Location 3	1600	Food-2	5	 1,587 	 1,547 	 1,545 	 8,896 	 1,236

the final output will be like this:

Item Name	Item location	Item code	Item type	Item number	Values	Date Posted

Milk	Location 1	1024	Food-1	1	 1,548 	4/1/2020

Bread	Location 2	1524	Food-2	2	 -   	4/1/2020

Toast	Location 2	1525	Food-3	4	 1,504 	4/1/2020

WheatBread	Location 3	1600	Food-2	5	 1,587 	4/1/2020

Milk	Location 1	1024	Food-1	1	 1,561 	4/2/2020

Bread	Location 2	1524	Food-2	2	 -   	4/2/2020

Toast	Location 2	1525	Food-3	4	 1,526 	4/2/2020

WheatBread	Location 3	1600	Food-2	5	 1,547 	4/2/2020

Milk	Location 1	1024	Food-1	1	 6,584 	4/3/2020

Bread	Location 2	1524	Food-2	2	 -   	4/3/2020

Toast	Location 2	1525	Food-3	4	 5,423 	4/3/2020

WheatBread	Location 3	1600	Food-2	5	 1,545 	4/3/2020

Milk	Location 1	1024	Food-1	1	 1,344 	4/4/2020

Bread	Location 2	1524	Food-2	2	 1,524 	4/4/2020

Toast	Location 2	1525	Food-3	4	 1,434 	4/4/2020

WheatBread	Location 3	1600	Food-2	5	 8,896 	4/4/2020

Milk	Location 1	1024	Food-1	1	 1,485 	4/5/2020

Bread	Location 2	1524	Food-2	2	 1,597 	4/5/2020

Toast	Location 2	1525	Food-3	4	 1,244 	4/5/2020

WheatBread	Location 3	1600	Food-2	5	 1,236 	4/5/2020

I have tried this already... but it doesn't work somewhere near the insert statement, I can create new table out of the input table... need help validating the code...

DECLARE @StartDate AS DATETIME
DECLARE @EndDate AS DATETIME
DECLARE @CurrentDate AS DATETIME
DECLARE @sqlText as nvarchar(max)

SET @StartDate = '2020-04-01'
SET @EndDate = '2020-04-06'
SET @CurrentDate = @StartDate

WHILE (@CurrentDate < @EndDate)
BEGIN

   SET @sqlText = N'SELECT
    distinct
        [Item Name],
        [Item location],
        [Item code],
        [Item type],
        [Item number],
        ['+(format(@CurrentDate, 'M/d/yyyy'))+'] as 'Qunatity',
        '+(format(@CurrentDate, 'M/d/yyyy'))+'  as [DatePosted]
    FROM [dbo].[BaseTable]';

    -- Insert into table ---> this where it's erroring

    INSERT INTO  [dbo].[FinalTable] ([Item Name],[Item location],[Item code],[Item type],[Item number],[Values],[Date Posted])
    EXEC sp_executesql (@sqlText)

    SET @CurrentDate = convert(varchar(30), dateadd(day,1, @CurrentDate), 101); /*increment current date*/
END

Body:
You can generate single query from your table definition:

  SELECT Query =
  'SELECT
        [Item Name],
        [Item location],
        [Item code],
        [Item type],
        [Item number],
        RotatedValues.Value,
        RotatedValues.DatePosted
   FROM dbo.MySource
   CROSS APPLY ('
     + STRING_AGG('SELECT Value = ' + QUOTENAME(columns.name) + ', DatePosted = ''' + columns.name + '''', ' UNION ALL ') +
   ') AS RotatedValues'
  FROM sys.tables
  INNER JOIN sys.columns ON columns.object_id = tables.object_id
  WHERE tables.name = 'MySource'
  AND columns.name LIKE '[0-9]%'

It will generate:

SELECT [Item Name],
       [Item location],
       [Item code],
       [Item type],
       [Item number],
       RotatedValues.Value,
       RotatedValues.DatePosted
FROM dbo.MySource
CROSS APPLY (
    SELECT Value      = [4/1/2020],
           DatePosted = '4/1/2020'
    UNION ALL
    SELECT Value      = [4/2/2020],
           DatePosted = '4/2/2020'
    UNION ALL
    SELECT Value      = [4/3/2020],
           DatePosted = '4/3/2020'
) AS RotatedValues