问:
Can't write data files using Pyspark?
I'm having touble writing my dataframes to other file formats.  From the online tutorials, it looks like this should work.

```<code>sc=sparkContext()
spark=SparkSession(sc)

df = spark.read.csv("table.csv")

df.write.orc("tests/file.orc")

</code>```

but this (write.orc) results in this long error

```20/06/01 13:54:32 ERROR Executor: Exception in task 0.0 in stage 63.0 (TID 63)

java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\t-aldouc\PycharmProjects\SE-DAT-EAIB-RecSysSpark\tests\test_data\test_orc.orc\_temporary\0\_temporary\attempt_20200601135431_0063_m_000000_63\part-00000-1b6b5b32-a036-414b-bb33-07e39b81c5d3-c000.snappy.orc

              at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)

              at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)

              at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)

              at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

              at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:95)

              at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:177)

              at org.apache.orc.OrcFile.createWriter(OrcFile.java:860)

              at org.apache.orc.mapreduce.OrcOutputFormat.getRecordWriter(OrcOutputFormat.java:50)

              at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:43)

              at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:121)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:236)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)

              at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)

              at org.apache.spark.scheduler.Task.run(Task.scala:123)

              at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)

              at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)

              at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

              at java.lang.Thread.run(Unknown Source)

20/06/01 13:54:32 WARN TaskSetManager: Lost task 0.0 in stage 63.0 (TID 63, localhost, executor driver): java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\t-aldouc\PycharmProjects\SE-DAT-EAIB-RecSysSpark\tests\test_data\test_orc.orc\_temporary\0\_temporary\attempt_20200601135431_0063_m_000000_63\part-00000-1b6b5b32-a036-414b-bb33-07e39b81c5d3-c000.snappy.orc

              at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)

              at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)

              at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)

              at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

              at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:95)

              at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:177)

              at org.apache.orc.OrcFile.createWriter(OrcFile.java:860)

              at org.apache.orc.mapreduce.OrcOutputFormat.getRecordWriter(OrcOutputFormat.java:50)

              at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:43)

              at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:121)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:236)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)

              at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)

              at org.apache.spark.scheduler.Task.run(Task.scala:123)

              at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)

              at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)

              at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

              at java.lang.Thread.run(Unknown Source)

20/06/01 13:54:32 ERROR TaskSetManager: Task 0 in stage 63.0 failed 1 times; aborting job

20/06/01 13:54:32 ERROR FileFormatWriter: Aborting job 55384644-29f4-4a2c-8a40-def2a2e2da73.

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 63.0 failed 1 times, most recent failure: Lost task 0.0 in stage 63.0 (TID 63, localhost, executor driver): java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\t-aldouc\PycharmProjects\SE-DAT-EAIB-RecSysSpark\tests\test_data\test_orc.orc\_temporary\0\_temporary\attempt_20200601135431_0063_m_000000_63\part-00000-1b6b5b32-a036-414b-bb33-07e39b81c5d3-c000.snappy.orc

              at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)

              at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)

              at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)

              at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

              at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:95)

              at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:177)

              at org.apache.orc.OrcFile.createWriter(OrcFile.java:860)

              at org.apache.orc.mapreduce.OrcOutputFormat.getRecordWriter(OrcOutputFormat.java:50)

              at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:43)

              at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:121)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:236)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)

              at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)

              at org.apache.spark.scheduler.Task.run(Task.scala:123)

              at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)

              at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)

              at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

              at java.lang.Thread.run(Unknown Source)

Driver stacktrace:

              at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1891)

              at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1879)

              at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1878)

              at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

              at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)

              at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1878)

              at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:927)

              at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:927)

              at scala.Option.foreach(Option.scala:257)

              at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:927)

              at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2112)

              at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2061)

              at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2050)

              at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)

              at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:738)

              at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:167)

              at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)

              at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)

              at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)

              at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)

              at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)

              at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)

              at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)

              at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

              at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)

              at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)

              at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)

              at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)

              at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)

              at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)

              at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)

              at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)

              at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)

              at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)

              at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)

              at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)

              at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)

              at org.apache.spark.sql.DataFrameWriter.orc(DataFrameWriter.scala:588)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

              at java.lang.reflect.Method.invoke(Unknown Source)

              at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)

              at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)

              at py4j.Gateway.invoke(Gateway.java:282)

              at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)

              at py4j.commands.CallCommand.execute(CallCommand.java:79)

              at py4j.GatewayConnection.run(GatewayConnection.java:238)

              at java.lang.Thread.run(Unknown Source)

Caused by: java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\t-aldouc\PycharmProjects\SE-DAT-EAIB-RecSysSpark\tests\test_data\test_orc.orc\_temporary\0\_temporary\attempt_20200601135431_0063_m_000000_63\part-00000-1b6b5b32-a036-414b-bb33-07e39b81c5d3-c000.snappy.orc

              at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)

              at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)

              at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)

              at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

              at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:95)

              at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:177)

              at org.apache.orc.OrcFile.createWriter(OrcFile.java:860)

              at org.apache.orc.mapreduce.OrcOutputFormat.getRecordWriter(OrcOutputFormat.java:50)

              at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:43)

              at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:121)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:236)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)

              at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)

              at org.apache.spark.scheduler.Task.run(Task.scala:123)

              at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)

              at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)

              at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

              ... 1 more

Traceback (most recent call last):

  File "<input>", line 1, in <module>

  File "C:\Users\t-aldouc\AppData\Local\Programs\Python\Python37\lib\site-packages\pyspark\sql\readwriter.py", line 960, in orc

    self._jwrite.orc(path)

  File "C:\Users\t-aldouc\AppData\Local\Programs\Python\Python37\lib\site-packages\py4j\java_gateway.py", line 1257, in __call__

    answer, self.gateway_client, self.target_id, self.name)

  File "C:\Users\t-aldouc\AppData\Local\Programs\Python\Python37\lib\site-packages\pyspark\sql\utils.py", line 63, in deco

   return f(*a, **kw)

  File "C:\Users\t-aldouc\AppData\Local\Programs\Python\Python37\lib\site-packages\py4j\protocol.py", line 328, in get_return_value

    format(target_id, ".", name), value)

py4j.protocol.Py4JJavaError: An error occurred while calling o1488.orc.

: org.apache.spark.SparkException: Job aborted.

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:198)

              at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)

              at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)

              at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)

              at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)

              at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)

              at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)

              at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)

              at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

              at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)

              at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)

              at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)

              at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)

              at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)

              at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)

              at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)

              at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)

              at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)

              at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)

              at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)

              at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)

              at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)

              at org.apache.spark.sql.DataFrameWriter.orc(DataFrameWriter.scala:588)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

              at java.lang.reflect.Method.invoke(Unknown Source)

              at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)

              at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)

              at py4j.Gateway.invoke(Gateway.java:282)

              at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)

              at py4j.commands.CallCommand.execute(CallCommand.java:79)

              at py4j.GatewayConnection.run(GatewayConnection.java:238)

              at java.lang.Thread.run(Unknown Source)

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 63.0 failed 1 times, most recent failure: Lost task 0.0 in stage 63.0 (TID 63, localhost, executor driver): java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\t-aldouc\PycharmProjects\SE-DAT-EAIB-RecSysSpark\tests\test_data\test_orc.orc\_temporary\0\_temporary\attempt_20200601135431_0063_m_000000_63\part-00000-1b6b5b32-a036-414b-bb33-07e39b81c5d3-c000.snappy.orc

              at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)

              at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)

              at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)

              at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

              at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:95)

              at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:177)

              at org.apache.orc.OrcFile.createWriter(OrcFile.java:860)

              at org.apache.orc.mapreduce.OrcOutputFormat.getRecordWriter(OrcOutputFormat.java:50)

              at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:43)

              at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:121)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:236)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)

              at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)

              at org.apache.spark.scheduler.Task.run(Task.scala:123)

              at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)

              at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)

              at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

              at java.lang.Thread.run(Unknown Source)

Driver stacktrace:

              at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1891)

.....can't enter this many characters
```

I received a similar error while trying to write a parquet file and a csv, but managed to write the parquet file by converting df to a pandas dataframe and using to_parquet(), but I can't find a similar work around for .orc files.  How do I fix this?  I already tried adding the HADOOP PATH variable, but it did nothing

答: I had to run Pycharm as administrator then it worked fine</p>

Title: Can't write data files using Pyspark?

Body:
I'm having touble writing my dataframes to other file formats.  From the online tutorials, it looks like this should work.

sc=sparkContext()
spark=SparkSession(sc)

df = spark.read.csv("table.csv")

df.write.orc("tests/file.orc")

but this (write.orc) results in this long error

20/06/01 13:54:32 ERROR Executor: Exception in task 0.0 in stage 63.0 (TID 63)

java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\t-aldouc\PycharmProjects\SE-DAT-EAIB-RecSysSpark\tests\test_data\test_orc.orc\_temporary\0\_temporary\attempt_20200601135431_0063_m_000000_63\part-00000-1b6b5b32-a036-414b-bb33-07e39b81c5d3-c000.snappy.orc

              at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)

              at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)

              at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)

              at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

              at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:95)

              at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:177)

              at org.apache.orc.OrcFile.createWriter(OrcFile.java:860)

              at org.apache.orc.mapreduce.OrcOutputFormat.getRecordWriter(OrcOutputFormat.java:50)

              at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:43)

              at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:121)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:236)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)

              at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)

              at org.apache.spark.scheduler.Task.run(Task.scala:123)

              at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)

              at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)

              at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

              at java.lang.Thread.run(Unknown Source)

20/06/01 13:54:32 WARN TaskSetManager: Lost task 0.0 in stage 63.0 (TID 63, localhost, executor driver): java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\t-aldouc\PycharmProjects\SE-DAT-EAIB-RecSysSpark\tests\test_data\test_orc.orc\_temporary\0\_temporary\attempt_20200601135431_0063_m_000000_63\part-00000-1b6b5b32-a036-414b-bb33-07e39b81c5d3-c000.snappy.orc

              at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)

              at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)

              at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)

              at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

              at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:95)

              at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:177)

              at org.apache.orc.OrcFile.createWriter(OrcFile.java:860)

              at org.apache.orc.mapreduce.OrcOutputFormat.getRecordWriter(OrcOutputFormat.java:50)

              at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:43)

              at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:121)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:236)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)

              at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)

              at org.apache.spark.scheduler.Task.run(Task.scala:123)

              at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)

              at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)

              at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

              at java.lang.Thread.run(Unknown Source)

20/06/01 13:54:32 ERROR TaskSetManager: Task 0 in stage 63.0 failed 1 times; aborting job

20/06/01 13:54:32 ERROR FileFormatWriter: Aborting job 55384644-29f4-4a2c-8a40-def2a2e2da73.

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 63.0 failed 1 times, most recent failure: Lost task 0.0 in stage 63.0 (TID 63, localhost, executor driver): java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\t-aldouc\PycharmProjects\SE-DAT-EAIB-RecSysSpark\tests\test_data\test_orc.orc\_temporary\0\_temporary\attempt_20200601135431_0063_m_000000_63\part-00000-1b6b5b32-a036-414b-bb33-07e39b81c5d3-c000.snappy.orc

              at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)

              at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)

              at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)

              at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

              at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:95)

              at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:177)

              at org.apache.orc.OrcFile.createWriter(OrcFile.java:860)

              at org.apache.orc.mapreduce.OrcOutputFormat.getRecordWriter(OrcOutputFormat.java:50)

              at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:43)

              at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:121)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:236)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)

              at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)

              at org.apache.spark.scheduler.Task.run(Task.scala:123)

              at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)

              at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)

              at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

              at java.lang.Thread.run(Unknown Source)

Driver stacktrace:

              at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1891)

              at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1879)

              at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1878)

              at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

              at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)

              at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1878)

              at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:927)

              at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:927)

              at scala.Option.foreach(Option.scala:257)

              at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:927)

              at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2112)

              at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2061)

              at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2050)

              at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)

              at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:738)

              at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:167)

              at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)

              at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)

              at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)

              at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)

              at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)

              at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)

              at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)

              at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

              at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)

              at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)

              at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)

              at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)

              at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)

              at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)

              at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)

              at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)

              at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)

              at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)

              at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)

              at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)

              at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)

              at org.apache.spark.sql.DataFrameWriter.orc(DataFrameWriter.scala:588)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

              at java.lang.reflect.Method.invoke(Unknown Source)

              at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)

              at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)

              at py4j.Gateway.invoke(Gateway.java:282)

              at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)

              at py4j.commands.CallCommand.execute(CallCommand.java:79)

              at py4j.GatewayConnection.run(GatewayConnection.java:238)

              at java.lang.Thread.run(Unknown Source)

Caused by: java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\t-aldouc\PycharmProjects\SE-DAT-EAIB-RecSysSpark\tests\test_data\test_orc.orc\_temporary\0\_temporary\attempt_20200601135431_0063_m_000000_63\part-00000-1b6b5b32-a036-414b-bb33-07e39b81c5d3-c000.snappy.orc

              at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)

              at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)

              at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)

              at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

              at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:95)

              at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:177)

              at org.apache.orc.OrcFile.createWriter(OrcFile.java:860)

              at org.apache.orc.mapreduce.OrcOutputFormat.getRecordWriter(OrcOutputFormat.java:50)

              at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:43)

              at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:121)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:236)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)

              at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)

              at org.apache.spark.scheduler.Task.run(Task.scala:123)

              at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)

              at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)

              at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

              ... 1 more

Traceback (most recent call last):

  File "<input>", line 1, in <module>

  File "C:\Users\t-aldouc\AppData\Local\Programs\Python\Python37\lib\site-packages\pyspark\sql\readwriter.py", line 960, in orc

    self._jwrite.orc(path)

  File "C:\Users\t-aldouc\AppData\Local\Programs\Python\Python37\lib\site-packages\py4j\java_gateway.py", line 1257, in __call__

    answer, self.gateway_client, self.target_id, self.name)

  File "C:\Users\t-aldouc\AppData\Local\Programs\Python\Python37\lib\site-packages\pyspark\sql\utils.py", line 63, in deco

   return f(*a, **kw)

  File "C:\Users\t-aldouc\AppData\Local\Programs\Python\Python37\lib\site-packages\py4j\protocol.py", line 328, in get_return_value

    format(target_id, ".", name), value)

py4j.protocol.Py4JJavaError: An error occurred while calling o1488.orc.

: org.apache.spark.SparkException: Job aborted.

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:198)

              at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)

              at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)

              at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)

              at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)

              at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)

              at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)

              at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)

              at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

              at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)

              at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)

              at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)

              at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)

              at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)

              at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)

              at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)

              at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)

              at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)

              at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)

              at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)

              at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)

              at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)

              at org.apache.spark.sql.DataFrameWriter.orc(DataFrameWriter.scala:588)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

              at java.lang.reflect.Method.invoke(Unknown Source)

              at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)

              at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)

              at py4j.Gateway.invoke(Gateway.java:282)

              at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)

              at py4j.commands.CallCommand.execute(CallCommand.java:79)

              at py4j.GatewayConnection.run(GatewayConnection.java:238)

              at java.lang.Thread.run(Unknown Source)

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 63.0 failed 1 times, most recent failure: Lost task 0.0 in stage 63.0 (TID 63, localhost, executor driver): java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\t-aldouc\PycharmProjects\SE-DAT-EAIB-RecSysSpark\tests\test_data\test_orc.orc\_temporary\0\_temporary\attempt_20200601135431_0063_m_000000_63\part-00000-1b6b5b32-a036-414b-bb33-07e39b81c5d3-c000.snappy.orc

              at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)

              at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)

              at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)

              at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)

              at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)

              at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)

              at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)

              at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

              at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

              at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:95)

              at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:177)

              at org.apache.orc.OrcFile.createWriter(OrcFile.java:860)

              at org.apache.orc.mapreduce.OrcOutputFormat.getRecordWriter(OrcOutputFormat.java:50)

              at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:43)

              at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:121)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)

              at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:236)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)

              at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)

              at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)

              at org.apache.spark.scheduler.Task.run(Task.scala:123)

              at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)

              at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)

              at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

              at java.lang.Thread.run(Unknown Source)

Driver stacktrace:

              at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1891)

.....can't enter this many characters

I received a similar error while trying to write a parquet file and a csv, but managed to write the parquet file by converting df to a pandas dataframe and using to_parquet(), but I can't find a similar work around for .orc files.  How do I fix this?  I already tried adding the HADOOP PATH variable, but it did nothing


来源: stackexchange

问:
How do I use a pipe in the exec parameter for a find command?
I'm trying to construct a find command to process a bunch of files in a directory using two different executables.  Unfortunately, -exec on find doesn't allow to use pipe or even \| because the shell interprets that character first.

Here is specifically what I'm trying to do (which doesn't work because pipe ends the find command):

```<code>find /path/to/jpgs -type f -exec jhead -v {} | grep 123 \; -print
</code>```

答:
A slightly different approach would be to use xargs:

```<code>find /path/to/jpgs -type f -print0 | xargs -0 jhead -v | grep 123
</code>```

which I always found a bit easier to understand and to adapt (the -print0 and -0 arguments are necessary to cope with filenames containing blanks)

This might (not tested) be more effective than using -exec because it will pipe the list of files to xargs and xargs makes sure that the jhead commandline does not get too long.

OwnerDisplayName: hoyhoy

ContentLicense: CC BY-SA 3.0

Body:
I'm trying to construct a find command to process a bunch of files in a directory using two different executables.  Unfortunately, -exec on find doesn't allow to use pipe or even \| because the shell interprets that character first.

Here is specifically what I'm trying to do (which doesn't work because pipe ends the find command):

find /path/to/jpgs -type f -exec jhead -v {} | grep 123 \; -print

LastEditorDisplayName: Mo

Body:
A slightly different approach would be to use xargs:

find /path/to/jpgs -type f -print0 | xargs -0 jhead -v | grep 123

which I always found a bit easier to understand and to adapt (the -print0 and -0 arguments are necessary to cope with filenames containing blanks)

This might (not tested) be more effective than using -exec because it will pipe the list of files to xargs and xargs makes sure that the jhead commandline does not get too long.

OwnerDisplayName: Palmin

ContentLicense: CC BY-SA 2.5


问:
redirect output of gcloud command to a file in cloud build
I am using cloudbuild.yml file.

I am trying to grab the build output from inside the cloud build and push it to a file. This is how my step looks like:

```<code>- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk:slim'
  args: ['gcloud', 'builds', 'log', '$BUILD_ID', '>buildlog.log']
  id: 'fetch-build-log'
</code>```

This throws me an error saying ERROR: (gcloud.builds.log) unrecognized arguments: >buildlog.log

If I execute that command in cloud shell, it works fine: gcloud builds log xxxxx-xxxx-xxxx-xxxx-xxxxxxx >guildlog.log

I am not sure why cloud build considers >buildlog.log an argument when it is to redirect the output to the file.

Am I missing something here or is there another way of doing it?

答:
At the shell prompt, when you run the command:

```<code>gcloud builds logs XXXX > buildlog.log
</code>```

The actual command you are running is

```<code>gcloud builds log XXXX
</code>```

with the additional instruction to the shell that you want any output from the command to be re-directed to a local file.  The gcloud binary (the application that you are actually running) isn't passed the trailing > buildlog.log and, if it were, would give you the error message you reported.  It is the shell that is interpreting the file output redirection.

What I think you want to do is simply remove the > buildlog.log from your arguments.

Since you are no longer passing in this parameter, the next question becomes "Where is the output from the command going?" ... the answer to that should be GCP Cloud Logging where you should be able to see the output from the commands.

Should you really want to create a local file of output, consider using the Cloud Build entrypoint:

https://cloud.google.com/cloud-build/docs/build-config#entrypoint

and specify a value for that of bash.  This should start a shell and then pass your parameters to the shell rather than a raw fork/exec.

Finally, it appears that your question may be a variant of:

How can I save google cloud build step text output to file

ContentLicense: CC BY-SA 4.0

Body:
I am using cloudbuild.yml file.

I am trying to grab the build output from inside the cloud build and push it to a file. This is how my step looks like:

- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk:slim'
  args: ['gcloud', 'builds', 'log', '$BUILD_ID', '>buildlog.log']
  id: 'fetch-build-log'

This throws me an error saying ERROR: (gcloud.builds.log) unrecognized arguments: >buildlog.log

If I execute that command in cloud shell, it works fine: gcloud builds log xxxxx-xxxx-xxxx-xxxx-xxxxxxx >guildlog.log

I am not sure why cloud build considers >buildlog.log an argument when it is to redirect the output to the file.

Am I missing something here or is there another way of doing it?

Body:
At the shell prompt, when you run the command:

gcloud builds logs XXXX > buildlog.log

The actual command you are running is

gcloud builds log XXXX

with the additional instruction to the shell that you want any output from the command to be re-directed to a local file.  The gcloud binary (the application that you are actually running) isn't passed the trailing > buildlog.log and, if it were, would give you the error message you reported.  It is the shell that is interpreting the file output redirection.

What I think you want to do is simply remove the > buildlog.log from your arguments.

Since you are no longer passing in this parameter, the next question becomes "Where is the output from the command going?" ... the answer to that should be GCP Cloud Logging where you should be able to see the output from the commands.

Should you really want to create a local file of output, consider using the Cloud Build entrypoint:

https://cloud.google.com/cloud-build/docs/build-config#entrypoint

and specify a value for that of bash.  This should start a shell and then pass your parameters to the shell rather than a raw fork/exec.

Finally, it appears that your question may be a variant of:

How can I save google cloud build step text output to file


问:
How to divide each column with a different number
I used the "table" function of R to summarize data and got the following result:

Summarized Data

Now, I want to divide all the numbers in the first column with 32, all the numbers in the second column with 51, and all the numbers in the third column with 54. Then these numbers will form a new 3*5 matrix, and I'll use the new matrix to continue to draw a barplot.

Does anyone here know how I can write the command in R?

答:
```<code>testMartix = dataframe(matrix(seq(1:100), nrow = 5, ncol = 20))
print(testMartix)
divisors = c(32, 51, 54)
new.matrix = data.frame(matrix(NA, nrow = 5, ncol = 3))
new.matrix

i=1
for (i in 1:3){
  print(i)
  new.matrix[,i] = testMartix[,i]/divisors[i]
  i = i+1
}
</code>```

Akrun's way is better. BUT, this is kind of intuative. it boils down to:

```<code>print(testMartix[,1]/divisors[1])
print(testMartix[,2]/divisors[2])
print(testMartix[,3]/divisors[3])
</code>```

Body:
testMartix = dataframe(matrix(seq(1:100), nrow = 5, ncol = 20))
print(testMartix)
divisors = c(32, 51, 54)
new.matrix = data.frame(matrix(NA, nrow = 5, ncol = 3))
new.matrix

i=1
for (i in 1:3){
  print(i)
  new.matrix[,i] = testMartix[,i]/divisors[i]
  i = i+1
}

Akrun's way is better. BUT, this is kind of intuative. it boils down to:

print(testMartix[,1]/divisors[1])
print(testMartix[,2]/divisors[2])
print(testMartix[,3]/divisors[3])


问:
MariaDB Trigger Calculating Time Difference as a decimal of total hours and tenth of hour always rounding up
I have a table that contains these columns:

```<code>start_time datetime
end_time datetime
billable_time decimal(3,1)
</code>```

The billable_time column is calculated with a trigger on insert and update as the total number of hours and tenth of hour as a decimal; however, I want it to always round up the decimal as opposed to rounding to the nearest digit. When creating the trigger, I cannot figure out how to make it always round up the decimal. This trigger calculates it correctly, but it rounds it down if that is the nearest digit:

```<code>SET new.billable_time = CAST(TIME_TO_SEC(TIMEDIFF(new.end_time, new.start_time))/3600 as decimal(3,1))
</code>```

For example:

```<code>start time = 2020-06-01 11:00:00
end_time = 2020-06-01 12:22:08
</code>```

Results in

```<code>billable_time = 1.4
</code>```

But it should be 1.5 if it always rounds up.

I have tried using the CEIL() and CEILING() functions, but they round it up to the nearest integer, which is 2.

I cannot seem to figure out how to make it do what I want. Is it possible to do this in a trigger? Any help would be greatly appreciated. The database is MariaDB v10.4.13.

Thanks in advance.

答:
A simple solution is to add half of a tenth of second to the value before casting it:

```<code>SET new.billable_time = CAST(
    TIME_TO_SEC(TIMEDIFF(new.end_time, new.start_time))/3600 + 0.05
    AS decimal(3,1)
)
</code>```

You can also use ceil(): the idea is to first divide by 360 instead of 3600, then round with ceil(), and finally divide by 10:

```<code>SET new.billable_time =
    CEIL(TIME_TO_SEC(TIMEDIFF(new.end_time, new.start_time))/360) / 10
</code>```

Body:
I have a table that contains these columns:

start_time datetime
end_time datetime
billable_time decimal(3,1)

The billable_time column is calculated with a trigger on insert and update as the total number of hours and tenth of hour as a decimal; however, I want it to always round up the decimal as opposed to rounding to the nearest digit. When creating the trigger, I cannot figure out how to make it always round up the decimal. This trigger calculates it correctly, but it rounds it down if that is the nearest digit:

SET new.billable_time = CAST(TIME_TO_SEC(TIMEDIFF(new.end_time, new.start_time))/3600 as decimal(3,1))

For example:

start time = 2020-06-01 11:00:00
end_time = 2020-06-01 12:22:08

Results in

billable_time = 1.4

But it should be 1.5 if it always rounds up.

I have tried using the CEIL() and CEILING() functions, but they round it up to the nearest integer, which is 2.

I cannot seem to figure out how to make it do what I want. Is it possible to do this in a trigger? Any help would be greatly appreciated. The database is MariaDB v10.4.13.

Thanks in advance.

Body:
A simple solution is to add half of a tenth of second to the value before casting it:

SET new.billable_time = CAST(
    TIME_TO_SEC(TIMEDIFF(new.end_time, new.start_time))/3600 + 0.05
    AS decimal(3,1)
)

You can also use ceil(): the idea is to first divide by 360 instead of 3600, then round with ceil(), and finally divide by 10:

SET new.billable_time =
    CEIL(TIME_TO_SEC(TIMEDIFF(new.end_time, new.start_time))/360) / 10


问:
Getting Policy sync failed with status code 503 while integrating mip sdk on Android
I am working on integrating mip sdk version 1.6 as per latest release document on Android. I am able to get access token but when provided to initialize policy engine, it throws below error.

Failed API call: profile_add_engine_async Failed with: [NetworkError: 'Policy sync failed, NetworkError.Category=Throttled, HttpRequest.SanitizedUrl=https://dataservice.protection.outlook.com/PsorWebService/v1/ClientSyncFile/MipPolicies, HttpRequest.Id={EC1D0F76-3A38-4F8A-AAC6-TB9059W1B9AM}, HttpResponse.StatusCode=503, CorrelationId=a9381643-2n1r-4q66-933j-77848ca3e15a, CorrelationId.Description=PolicyProfile']

Appreciate any help.

答:
It started working after migrating AIP labels to unified labels as mentioned in this link. The returned error code does not provide any information about failure of this type. </p>

<p>I am still trying to understand how the step is mandatory. As per the description in link, AIP should be supported till 03/31/2021 . </p>

Body:
It started working after migrating AIP labels to unified labels as mentioned in this link. The returned error code does not provide any information about failure of this type.

I am still trying to understand how the step is mandatory. As per the description in link, AIP should be supported till 03/31/2021 .


问:
How to differently write reinterpret_cast<>
I have a lot of reinterpret_casts in my code. Can I do something like this

```<code>using c = reinterpret_cast<T*>
</code>```

where T is a template arg. So it would look like this

```<code>std::uninitialized_move(c(temp)....)
</code>```

Here is an example:

```<code>char* new_buffer_ = new char[size_* sizeof(T) * 2];
std::uninitialized_move(reinterpret_cast<T*>(buffer_),
                        reinterpret_cast<T*>(buffer_) + size_,
                        reinterpret_cast<T*>(new_buffer_));
</code>```

答:
If the goal is to save typing without any change in functionality, you can write:

```<code>template<typename T>
static T* c(char *p) { return reinterpret_cast<T*>(p); }
</code>```

or to avoid scope pollution,

```<code>template<typename T>
auto c = [](char *p) { return reinterpret_cast<T *>(p); };
</code>```

But, as stressed in comments, there is likely a better way to design your code in order to not need all of these casts in the first place.

Body:
I have a lot of reinterpret_casts in my code. Can I do something like this

using c = reinterpret_cast<T*>

where T is a template arg. So it would look like this

std::uninitialized_move(c(temp)....)

Here is an example:

char* new_buffer_ = new char[size_* sizeof(T) * 2];
std::uninitialized_move(reinterpret_cast<T*>(buffer_),
                        reinterpret_cast<T*>(buffer_) + size_,
                        reinterpret_cast<T*>(new_buffer_));

Body:
If the goal is to save typing without any change in functionality, you can write:

template<typename T>
static T* c(char *p) { return reinterpret_cast<T*>(p); }

or to avoid scope pollution,

template<typename T>
auto c = [](char *p) { return reinterpret_cast<T *>(p); };

But, as stressed in comments, there is likely a better way to design your code in order to not need all of these casts in the first place.


问:
C# SetValues, EntityState.Modified not working
I have the following entries in my database:

```<code>MeetingID   AgendaItem  LegistarID  Title
48620       3           60710       Comment
48620       5           60615       Extending report date
48620       6           60714       Update on Additional meeting dates
48620       7           59909       Budget Rules & Procedures
48620       8           60703       Update Director name
</code>```

That I need to update with these values:

```<code>MeetingID   AgendaItem  LegistarID  Title
48620       3           60710       Public Comment
48620       5           60769       Briefing by Victor
48620       6           60615       Extending report dates
48620       7           60714       Update on Additional meeting dates
48620       8           60703       Update from Director on new processes
</code>```

The way I am trying doing this in C#, is as follows:

```<code>if (ModelState.IsValid)
{
    var errors = new List<string>();
    var rowCounter = 1;

    using (Entities db = new Entities())
    {
        foreach (var i in meeting)
        {
            if (i.MeetingID == 0)
            {
                // Let the user know this row is bad
                errors.Add($"Row {rowCounter}: Missing Meeting ID value. " +
                            "Verify that the data you are trying to upload meets the required criteria, " +
                            "and then try to upload your file again." );
                break;
            }
            // Check if LegistarID is missing
            if (i.LegistarID == 0)
            {
                // Check if Agenda Item is present
                if (i.AgendaItem == 0)
                {
                    errors.Add($"Row {rowCounter}: Meeting has no LegistarID and no Agenda Item. Please check data.");
                    break;
                }
                else
                {
                    i.LegistarID = i.AgendaItem;
              }
            }

            var compositeKey = db.Meeting.Find(i.MeetingID, i.AgendaItem);

            if (compositeKey == null)
            {

                // Add new
                db.Meeting.Add(i);

            }
            else
            {
                // Serves as an update, or addition of a previously imported dataset
                db.Entry(compositeKey).CurrentValues.SetValues(i.MeetingID);
                db.Entry(compositeKey).CurrentValues.SetValues(i.AgendaItem);
                db.Entry(compositeKey).CurrentValues.SetValues(i.LegistarID);
                db.Entry(compositeKey).CurrentValues.SetValues(i.Title);
                db.Entry(compositeKey).State = EntityState.Modified;
            }
            rowCounter++;
        }
        // If there are errors do not save and return error message
        if (errors.Count > 0)
        {
            return new JsonResult { Data = new { status = false, message = string.Join("\n", errors) } };
        }
        db.SaveChanges();
        status = true;
    }
}
else
{
    message = string.Format(@"Please, verify that the file you are trying to upload is correctly formatted,
                            and that the data it contains, meets the expected criteria,
                            then click the upload button again. \n Thank you!");
    return new JsonResult { Data = new { status = status, message = message } };
}
</code>```

The code for the Add part works well, but the part that updates the record if the composite key is found does not work, the update is not working.

I am not sure if I am doing this the best way, but if there is a better way I am open to change the code, or if I have an error on how I am doing the process, please let me know

Any help is appreciated.

Thank you,
Erasmo

答:
Remove all your calls to SetValues and replace them with single one:

```<code>db.Entry(compositeKey).CurrentValues.SetValues(i);
</code>```

SetValues which accepts object as parameter copies data to entity based on object properties names:

Any property on the object with a name that matches a property name in the entity type and can be read will be copied. Other properties will be ignored.

Body:
I have the following entries in my database:

MeetingID   AgendaItem  LegistarID  Title
48620       3           60710       Comment
48620       5           60615       Extending report date
48620       6           60714       Update on Additional meeting dates
48620       7           59909       Budget Rules & Procedures
48620       8           60703       Update Director name

That I need to update with these values:

MeetingID   AgendaItem  LegistarID  Title
48620       3           60710       Public Comment
48620       5           60769       Briefing by Victor
48620       6           60615       Extending report dates
48620       7           60714       Update on Additional meeting dates
48620       8           60703       Update from Director on new processes

The way I am trying doing this in C#, is as follows:

if (ModelState.IsValid)
{
    var errors = new List<string>();
    var rowCounter = 1;

    using (Entities db = new Entities())
    {
        foreach (var i in meeting)
        {
            if (i.MeetingID == 0)
            {
                // Let the user know this row is bad
                errors.Add($"Row {rowCounter}: Missing Meeting ID value. " +
                            "Verify that the data you are trying to upload meets the required criteria, " +
                            "and then try to upload your file again." );
                break;
            }
            // Check if LegistarID is missing
            if (i.LegistarID == 0)
            {
                // Check if Agenda Item is present
                if (i.AgendaItem == 0)
                {
                    errors.Add($"Row {rowCounter}: Meeting has no LegistarID and no Agenda Item. Please check data.");
                    break;
                }
                else
                {
                    i.LegistarID = i.AgendaItem;
              }
            }

            var compositeKey = db.Meeting.Find(i.MeetingID, i.AgendaItem);

            if (compositeKey == null)
            {

                // Add new
                db.Meeting.Add(i);

            }
            else
            {
                // Serves as an update, or addition of a previously imported dataset
                db.Entry(compositeKey).CurrentValues.SetValues(i.MeetingID);
                db.Entry(compositeKey).CurrentValues.SetValues(i.AgendaItem);
                db.Entry(compositeKey).CurrentValues.SetValues(i.LegistarID);
                db.Entry(compositeKey).CurrentValues.SetValues(i.Title);
                db.Entry(compositeKey).State = EntityState.Modified;
            }
            rowCounter++;
        }
        // If there are errors do not save and return error message
        if (errors.Count > 0)
        {
            return new JsonResult { Data = new { status = false, message = string.Join("\n", errors) } };
        }
        db.SaveChanges();
        status = true;
    }
}
else
{
    message = string.Format(@"Please, verify that the file you are trying to upload is correctly formatted,
                            and that the data it contains, meets the expected criteria,
                            then click the upload button again. \n Thank you!");
    return new JsonResult { Data = new { status = status, message = message } };
}

The code for the Add part works well, but the part that updates the record if the composite key is found does not work, the update is not working.

I am not sure if I am doing this the best way, but if there is a better way I am open to change the code, or if I have an error on how I am doing the process, please let me know

Any help is appreciated.

Thank you,
Erasmo

Body:
Remove all your calls to SetValues and replace them with single one:

db.Entry(compositeKey).CurrentValues.SetValues(i);

SetValues which accepts object as parameter copies data to entity based on object properties names:

Any property on the object with a name that matches a property name in the entity type and can be read will be copied. Other properties will be ignored.


问:
Recompile PHP with ZTS enabled on MAC OS X
Homebrew PHP does not support a thread-safe php binary. Is there any way to enable ZTS in brew version.

I tried phpbrew as well but, I am getting an error "Command too long". Could you please suggest me the best way to enable ZTS in MACOS Catalina.

答:
Which command is used? Complete your question by providing the entire error and return of the console.

You will need to recompile PHP with the parameters: --enable-maintainer-zts or --enable-zts on Windows

See more on PHP.net


问:
In converting List<T> to DataTable add select properties from class
I am trying to convert a List to a DataTable. I have the basic outline done, but am stuck on one part. I am trying to only get a few of the properties form the class.

```<code>public static DataTable ToDataTable<T>(this IList<T> data)
{
    PropertyDescriptorCollection properties = TypeDescriptor.GetProperties(typeof(T));
    DataTable table = new DataTable();

    table.Columns.Add("type", typeof(string));
    table.Columns.Add("id", typeof(Int32));
    table.Columns.Add("name", typeof(string));
    table.Columns.Add("city", typeof(string));

    foreach (T item in data)
    {
        DataRow row = table.NewRow();
        foreach (PropertyDescriptor prop in properties)
        {
            row[prop.Name] = prop.GetValue(item) ?? DBNull.Value;
        }
        table.Rows.Add(row);
    }
    return table;
}
</code>```

The problem is when it gets to the second property it throws an error saying column 'unincludedCol' does not belong to table.

How do I get around this?

答:
If you need just type, id, name, and city, you can add them to a list of string :

```<code>List<string> columnNames = new List<string>
{
  "type","id","name","city"
};
</code>```

Or extract them from dataTable like :

```<code>List<string> columnNames = new List<string>();

foreach(DataColumn column in dataTable.Columns)
{
    columnNames.Add(column.ColumnName);
}
</code>```

And in the loop check if columnNames contains prop.Name, like:

```<code>foreach (T item in data)
{
    DataRow row = table.NewRow();
    foreach (PropertyDescriptor prop in properties)
    {
        if(!columnNames.Contains(prop.Name))
            continue;

        row[prop.Name] = prop.GetValue(item) ?? DBNull.Value;
    }

    table.Rows.Add(row);
}
</code>```

Note : instead if(!columnNames.Contains(prop.Name)), you can use .Any() and Equals methods to ignore case :

```<code>if (!columnNames.Any(c => c.Equals(prop.Name, StringComparison.OrdinalIgnoreCase)))
    continue;
</code>```

I hope you find this helpful.

Body:
I am trying to convert a List to a DataTable. I have the basic outline done, but am stuck on one part. I am trying to only get a few of the properties form the class.

public static DataTable ToDataTable<T>(this IList<T> data)
{
    PropertyDescriptorCollection properties = TypeDescriptor.GetProperties(typeof(T));
    DataTable table = new DataTable();

    table.Columns.Add("type", typeof(string));
    table.Columns.Add("id", typeof(Int32));
    table.Columns.Add("name", typeof(string));
    table.Columns.Add("city", typeof(string));

    foreach (T item in data)
    {
        DataRow row = table.NewRow();
        foreach (PropertyDescriptor prop in properties)
        {
            row[prop.Name] = prop.GetValue(item) ?? DBNull.Value;
        }
        table.Rows.Add(row);
    }
    return table;
}

The problem is when it gets to the second property it throws an error saying column 'unincludedCol' does not belong to table.

How do I get around this?

Body:
If you need just type, id, name, and city, you can add them to a list of string :

List<string> columnNames = new List<string>
{
  "type","id","name","city"
};

Or extract them from dataTable like :

List<string> columnNames = new List<string>();

foreach(DataColumn column in dataTable.Columns)
{
    columnNames.Add(column.ColumnName);
}

And in the loop check if columnNames contains prop.Name, like:

foreach (T item in data)
{
    DataRow row = table.NewRow();
    foreach (PropertyDescriptor prop in properties)
    {
        if(!columnNames.Contains(prop.Name))
            continue;

        row[prop.Name] = prop.GetValue(item) ?? DBNull.Value;
    }

    table.Rows.Add(row);
}

Note : instead if(!columnNames.Contains(prop.Name)), you can use .Any() and Equals methods to ignore case :

if (!columnNames.Any(c => c.Equals(prop.Name, StringComparison.OrdinalIgnoreCase)))
    continue;

I hope you find this helpful.


问:
django admin - subquery inside list_filter
Given are the following models

```<code>class Report(models.Model):
    id = models.CharField(max_length=256, unique=True)
    name = models.CharField(max_length=256)
    confidential = models.BooleanField(default=False)

class Owner(models.Model):
    first_name = models.CharField(max_length=512)
    last_name = models.CharField(max_length=512)
    title = models.CharField(max_length=512)
    birth_date = models.CharField(max_length=512)
    company = models.CharField(max_length=512)
    address = models.CharField(max_length=512)
    city = models.CharField(max_length=512)
    last_login = models.DateTimeField(auto_now=True)
    access_granted = models.DateTimeField(auto_now_add=True)
    report = models.ForeignKey(Report, on_delete=models.CASCADE)
</code>```

and the following admin.py from django

```<code>class OwnerAdmin(admin.ModelAdmin):
    .
    .
    .
    list_filter = ('report__name',)
</code>```

As expected I get the option to filter based on the name of the report. Yet I would like to only get displayed to filter if the report is confidential, means that the confidential field of the given report is true. How can i achieve this?

答:
If I am understanding your question correctly, you would like to only list the confidential reports in your list_filter. To do this you will need to create a custom filter:

```<code>class ReportListFilter(admin.SimpleListFilter):

    title = 'Confidential Reports'
    parameter_name = 'confidential_reports'

    def lookups(self, request, model_admin):
        confidential_reports = Report.objects.filter(confidential=True)
        return tuple(
            (report.id, report.name) for report in confidential_reports
        )

    def queryset(self, request, queryset):
        if self.value()
            return queryset.filter(report_id=self.value())
        return queryset.all()

</code>```

To understand the above, you can read more about it in the docs, but a summary is that:

title: gives the title for your filter

parameter_name: gives the string that will be used in the URL when your filter is used

lookups returns a list of tuples that gives the values for the filter

queryset is what does the actual filtering.

Then you just need to add it into your OwnerAdmin and everything should work:

```<code>class OwnerAdmin(admin.ModelAdmin):
    .
    .
    .
    list_filter = (ReportListFilter,)
</code>```

Body:
Given are the following models

class Report(models.Model):
    id = models.CharField(max_length=256, unique=True)
    name = models.CharField(max_length=256)
    confidential = models.BooleanField(default=False)

class Owner(models.Model):
    first_name = models.CharField(max_length=512)
    last_name = models.CharField(max_length=512)
    title = models.CharField(max_length=512)
    birth_date = models.CharField(max_length=512)
    company = models.CharField(max_length=512)
    address = models.CharField(max_length=512)
    city = models.CharField(max_length=512)
    last_login = models.DateTimeField(auto_now=True)
    access_granted = models.DateTimeField(auto_now_add=True)
    report = models.ForeignKey(Report, on_delete=models.CASCADE)

and the following admin.py from django

class OwnerAdmin(admin.ModelAdmin):
    .
    .
    .
    list_filter = ('report__name',)

As expected I get the option to filter based on the name of the report. Yet I would like to only get displayed to filter if the report is confidential, means that the confidential field of the given report is true. How can i achieve this?

Body:
If I am understanding your question correctly, you would like to only list the confidential reports in your list_filter. To do this you will need to create a custom filter:

class ReportListFilter(admin.SimpleListFilter):

    title = 'Confidential Reports'
    parameter_name = 'confidential_reports'

    def lookups(self, request, model_admin):
        confidential_reports = Report.objects.filter(confidential=True)
        return tuple(
            (report.id, report.name) for report in confidential_reports
        )

    def queryset(self, request, queryset):
        if self.value()
            return queryset.filter(report_id=self.value())
        return queryset.all()

To understand the above, you can read more about it in the docs, but a summary is that:

title: gives the title for your filter

parameter_name: gives the string that will be used in the URL when your filter is used

lookups returns a list of tuples that gives the values for the filter

queryset is what does the actual filtering.

Then you just need to add it into your OwnerAdmin and everything should work:

class OwnerAdmin(admin.ModelAdmin):
    .
    .
    .
    list_filter = (ReportListFilter,)


问:
What is the -m used (or useful) for in setting up virtual environments?
I've learned that when setting up a virtual environment to use for python scripts and libraries, it's recommended to first do this in terminal:

```<code>python3 -m venv venv
</code>```

why is the '-m' needed and where else is it commonly used?

答:
It's about how python finds the module.

From the documentation:

When called with -m module-name, the given module is located on the
  Python module path and executed as a script.

Since the argument is a module name, you must not give a file
  extension (.py).

The -m flag tells python to search for the module (or package) venv in it's own path, not a script in the current directory.

The distinction is important when you try to run a package. There is a big difference between:

python foo/bar/baz.py

and

python -m foo.bar.baz

In the latter case, foo.bar is imported and relative imports will work correctly with foo.bar as the starting point.

Body:
I've learned that when setting up a virtual environment to use for python scripts and libraries, it's recommended to first do this in terminal:

python3 -m venv venv

why is the '-m' needed and where else is it commonly used?


问:
Write to stdin of running docker container
Say I run a docker container as a daemon:

```<code>docker run -d foo
</code>```

is there a way to write to the stdin of that container? Something like:

```<code>docker exec -i foo echo 'hi'
</code>```

last time I checked the -i and -d flags were mutually exclusive when used with the docker run command.

答:
According to another answer on ServerFault, you can use socat to pipe input to a docker container like this:

```<code>echo 'hi' | socat EXEC:"docker attach container0",pty STDIN
</code>```

Note that the echo command includes a newline at the end of the output, so the line above actually sends hi\n. Use echo -n  if you don't want a newline.

Let's see how this looks like with the example script from David's answer:

# Create a new empty directory
mkdir x

# Run a container, in the background, that copies its stdin
# to a file in that directory
docker run -itd --rm -v $PWD/x:/x --name cattainer busybox sh -c 'cat >/x/y'

# Send some strings in
echo 'hi' | socat EXEC:"docker attach cattainer",pty STDIN
echo 'still there?' | socat EXEC:"docker attach cattainer",pty STDIN

# Stop container (cleans up itself because of --rm)
docker stop cattainer

# See what we got out
cat x/y

# should output:
# hi
# still there?
```
<p>You could also wrap it in a shell function:</p>
<pre class="lang-sh prettyprint-override"><code>docker_send() {
    container="$1"; shift
    echo "$@" | socat EXEC:"docker attach $container",pty STDIN
}

docker_send cattainer "Hello cat!"
docker_send cattainer -n "No newline here:"    # flag -n is passed to echo
</code>```

Trivia: I'm actually using this approach to control a Terraria server running in a docker container, because TerrariaServer.exe only accepts server commands (like save or exit) on stdin.

Body:
Say I run a docker container as a daemon:

docker run -d foo

is there a way to write to the stdin of that container? Something like:

docker exec -i foo echo 'hi'

last time I checked the -i and -d flags were mutually exclusive when used with the docker run command.

Body:
According to another answer on ServerFault, you can use socat to pipe input to a docker container like this:

echo 'hi' | socat EXEC:"docker attach container0",pty STDIN

Note that the echo command includes a newline at the end of the output, so the line above actually sends hi\n. Use echo -n  if you don't want a newline.

Let's see how this looks like with the example script from David's answer:

# Create a new empty directory
mkdir x

# Run a container, in the background, that copies its stdin
# to a file in that directory
docker run -itd --rm -v $PWD/x:/x --name cattainer busybox sh -c 'cat >/x/y'

# Send some strings in
echo 'hi' | socat EXEC:"docker attach cattainer",pty STDIN
echo 'still there?' | socat EXEC:"docker attach cattainer",pty STDIN

# Stop container (cleans up itself because of --rm)
docker stop cattainer

# See what we got out
cat x/y

# should output:
# hi
# still there?

You could also wrap it in a shell function:

docker_send() {
    container="$1"; shift
    echo "$@" | socat EXEC:"docker attach $container",pty STDIN
}

docker_send cattainer "Hello cat!"
docker_send cattainer -n "No newline here:"    # flag -n is passed to echo

Trivia: I'm actually using this approach to control a Terraria server running in a docker container, because TerrariaServer.exe only accepts server commands (like save or exit) on stdin.


问:
IF - ELSE IF - ELSE Structure in Excel
Requirement :

If the string in cell A1 contains "abc" as part of the string

Then assign value "Green"

Else if the string in cell A1 contains "xyz" as part of the string

Then assign value "Yellow"

Else

Assign value "Red"

I tried this :

=IF(FIND("~",SUBSTITUTE(A1,"abc","~",1))<>"#VALUE!", "Green", IF(FIND("~",SUBSTITUTE(A1,"xyz","~",1))<>"#VALUE!", "Yellow", "Red"))

It works if first IF is True in other cases it gives #VALUE! as output.

Can't figure out whats wrong.

答:
Say P7 is a Cell then you can use the following Syntex to check the value of the cell and assign appropriate value to another cell based on this following nested if:</p>

```<code>=IF(P7=0,200,IF(P7=1,100,IF(P7=2,25,IF(P7=3,10,IF((P7=4),5,0)))))
</code>```

Title: IF - ELSE IF - ELSE Structure in Excel

Body:
Say P7 is a Cell then you can use the following Syntex to check the value of the cell and assign appropriate value to another cell based on this following nested if:

=IF(P7=0,200,IF(P7=1,100,IF(P7=2,25,IF(P7=3,10,IF((P7=4),5,0)))))


问:
Recursion error in R (Fibonacci sequence)
So I am trying to learn R on my own and am just working through the online tutorial. I am trying to code a recursive function that prints the first n terms of the Fibonacci sequence and can't get the code to run without the error:

Error in if (nterms <= 0) { : missing value where TRUE/FALSE needed

My code does ask me for input before entering the if else statement either which I think is odd as well. Below is my code any help is appreciated.

```<code>#Define the fibonacci sequence

recurse_fibonacci <- function(n) {
    # Define the initial two values of the sequence

    if (n <= 1){
        return(n)
    } else {

    # define the rest of the terms of the sequence using recursion
    return(recurse_fibonacci(n-1) + recurse_fibonacci(n-2))
    }
}

#Take input from the user
nterms = as.integer(readline(prompt="How many terms? "))

# check to see if the number of terms entered is valid
if(nterms <= 0) {
    print("please enter a positive integer")
} else {

    # This part actually calculates and displays the first n terms of the sequence
    print("Fibonacci Sequence: ")
    for(i in 0:(nterms - 1)){
        print(recurse_fibonacci(i))
    }
}
</code>```

答:
The code works fine but you shouldn't enter it into the terminal as is. My suggestion: put the code into a script file (ending .R) and source it (get help about it with ?source but it's actually pretty straightforward).

In R-Studio you can simply hit the source button.

Body:
So I am trying to learn R on my own and am just working through the online tutorial. I am trying to code a recursive function that prints the first n terms of the Fibonacci sequence and can't get the code to run without the error:

Error in if (nterms <= 0) { : missing value where TRUE/FALSE needed

My code does ask me for input before entering the if else statement either which I think is odd as well. Below is my code any help is appreciated.

#Define the fibonacci sequence

recurse_fibonacci <- function(n) {
    # Define the initial two values of the sequence

    if (n <= 1){
        return(n)
    } else {

    # define the rest of the terms of the sequence using recursion
    return(recurse_fibonacci(n-1) + recurse_fibonacci(n-2))
    }
}

#Take input from the user
nterms = as.integer(readline(prompt="How many terms? "))

# check to see if the number of terms entered is valid
if(nterms <= 0) {
    print("please enter a positive integer")
} else {

    # This part actually calculates and displays the first n terms of the sequence
    print("Fibonacci Sequence: ")
    for(i in 0:(nterms - 1)){
        print(recurse_fibonacci(i))
    }
}


问:
Run same thread in loop
I want to run the thread below multiple times and everytime it finished print the result to tkinter's Label.

```<code>    def hasher(path):

        hashed_file = self.hash256(path)
        tk.Label(self, text=hashed_file).pack()

    t = []
    for directory in files_dict:

        for file in files_dict[directory]:

            t.append(threading.Thread(target=hasher, args=(directory+"/"+file,)))

    for thr in t:
        thr.start()
        thr.join()
</code>```

Basically to be able to later update a label with "Finished proccessing : "
However if I add .join() it freezes and if I remove it will finish and then load the page.

答:
In the last for-loop you are starting a thread thr from list t with thr.start() and then waiting for it, preventing other threads from the list t to be started in the first place. You need need an extra loop for the thr.join() statement:

for thr in t:
    thr.start()

for thr in t:
    thr.join()

Title: Run same thread in loop

Body:
I want to run the thread below multiple times and everytime it finished print the result to tkinter's Label.

    def hasher(path):

        hashed_file = self.hash256(path)
        tk.Label(self, text=hashed_file).pack()

    t = []
    for directory in files_dict:

        for file in files_dict[directory]:

            t.append(threading.Thread(target=hasher, args=(directory+"/"+file,)))

    for thr in t:
        thr.start()
        thr.join()

Basically to be able to later update a label with "Finished proccessing : "
However if I add .join() it freezes and if I remove it will finish and then load the page.