问:
Azure databricks: Problem saving to csv with +100 columns and many nulls, org.apache.spark.SparkException: Job aborted
I have seen many ppl having this issue but none of those were able to resolve my issue. I have a json file that I read in as a spark dataframe and then flatten it. Lastly, I try to save the flatten file as a csv with +100 columns and >47 million rows and many nulls.

I get the error org.apache.spark.SparkException: Job aborted.
Here is part of the stack trace:

```WARN TaskSetManager: Lost task 0.2 in stage 41.0 (TID 164) (10.139.64.7 executor 22): java.lang.BootstrapMethodError: call site initialization exception
    at java.lang.invoke.CallSite.makeSite(CallSite.java:341)
    at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:307)
    at java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:297)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.recordWriteFailure$1(FileFormatWriter.scala:403)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:431)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$12(FileFormatWriter.scala:310)
    at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$3(ResultTask.scala:75)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:75)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:55)
    at org.apache.spark.scheduler.Task.doRunTask(Task.scala:169)
    at org.apache.spark.scheduler.Task.$anonfun$run$4(Task.scala:137)
    at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:104)
    at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:137)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.Task.run(Task.scala:96)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$13(Executor.scala:902)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1696)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:905)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:760)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.InternalError: BMH.reinvoke=Lambda(a0:L/SpeciesData<LLL>,a1:L,a2:L,a3:L,a4:L,a5:L,a6:L,a7:L,a8:L,a9:L,a10:L,a11:L)=>{
    t12:L=BoundMethodHandle$Species_L3.argL2(a0:L);
    t13:L=MethodHandle.invokeBasic(t12:L,a2:L);
    t14:L=BoundMethodHandle$Species_L3.argL1(a0:L);
    t15:L=MethodHandle.invokeBasic(t14:L,a1:L);
    t16:L=MethodHandleImpl.array(a4:L,a5:L,a6:L,a7:L,a8:L,a9:L,a10:L,a11:L);
    t17:L=BoundMethodHandle$Species_L3.argL0(a0:L);
    t18:L=MethodHandle.invokeBasic(t17:L,t15:L,t13:L,a3:L,t16:L);t18:L}
    at java.lang.invoke.MethodHandleStatics.newInternalError(MethodHandleStatics.java:127)
    at java.lang.invoke.LambdaForm.compileToBytecode(LambdaForm.java:660)
    at java.lang.invoke.LambdaForm.prepare(LambdaForm.java:635)
    at java.lang.invoke.MethodHandle.<init>(MethodHandle.java:461)
    at java.lang.invoke.BoundMethodHandle.<init>(BoundMethodHandle.java:58)
    at java.lang.invoke.BoundMethodHandle$Species_L3.<init>(Species_L3)
    at java.lang.invoke.BoundMethodHandle$Species_L3.make(Species_L3)
    at java.lang.invoke.BoundMethodHandle$Species_LL.copyWithExtendL(Species_LL)
    at java.lang.invoke.MethodHandleImpl.makePairwiseConvertByEditor(MethodHandleImpl.java:231)
    at java.lang.invoke.MethodHandleImpl.makePairwiseConvert(MethodHandleImpl.java:194)
    at java.lang.invoke.MethodHandleImpl.makePairwiseConvert(MethodHandleImpl.java:380)
    at java.lang.invoke.MethodHandle.asTypeUncached(MethodHandle.java:776)
    at java.lang.invoke.MethodHandle.asType(MethodHandle.java:761)
    at java.lang.invoke.MethodHandleImpl$AsVarargsCollector.asTypeUncached(MethodHandleImpl.java:508)
    at java.lang.invoke.MethodHandle.asType(MethodHandle.java:761)
    at java.lang.invoke.CallSite.makeSite(CallSite.java:323)
    ... 25 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

22/12/13 09:56:11 INFO TaskSetManager: Starting task 0.3 in stage 41.0 (TID 165) (10.139.64.5, executor 23, partition 0, PROCESS_LOCAL, taskResourceAssignments Map())
22/12/13 09:56:12 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:12 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20221213083124-0000/22 is now EXITED (Command exited with code 50)
22/12/13 09:56:12 INFO StandaloneSchedulerBackend: Executor app-20221213083124-0000/22 removed: Command exited with code 50
22/12/13 09:56:12 ERROR TaskSchedulerImpl: Lost executor 22 on 10.139.64.7: Command exited with code 50
22/12/13 09:56:12 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20221213083124-0000/25 on worker-20221213085244-10.139.64.7-41835 (10.139.64.7:41835) with 4 core(s)
22/12/13 09:56:12 INFO DAGScheduler: Executor lost: 22 (epoch 18)
22/12/13 09:56:12 INFO StandaloneSchedulerBackend: Granted executor ID app-20221213083124-0000/25 on hostPort 10.139.64.7:41835 with 4 core(s), 3.1 GiB RAM
22/12/13 09:56:12 INFO BlockManagerMasterEndpoint: Trying to remove executor 22 from BlockManagerMaster.
22/12/13 09:56:12 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(22, 10.139.64.7, 40355, None)
22/12/13 09:56:12 INFO BlockManagerMaster: Removed 22 successfully in removeExecutor
22/12/13 09:56:12 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20221213083124-0000/25 is now RUNNING
22/12/13 09:56:15 INFO StandaloneSchedulerBackend$StandaloneDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.139.64.6:55600) with ID 24,  ResourceProfileId 0
22/12/13 09:56:15 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:16 INFO BlockManagerMasterEndpoint: Registering block manager 10.139.64.6:39649 with 1504.5 MiB RAM, BlockManagerId(24, 10.139.64.6, 39649, None)
22/12/13 09:56:16 INFO BlockManagerInfo: Added broadcast_36_piece0 in memory on 10.139.64.5:45187 (size: 220.5 KiB, free: 1504.3 MiB)
22/12/13 09:56:18 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:18 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.139.64.5:45187 (size: 18.1 KiB, free: 1504.3 MiB)
22/12/13 09:56:21 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:22 INFO StandaloneSchedulerBackend$StandaloneDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.139.64.7:55476) with ID 25,  ResourceProfileId 0
22/12/13 09:56:24 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:24 INFO BlockManagerMasterEndpoint: Registering block manager 10.139.64.7:43527 with 1504.5 MiB RAM, BlockManagerId(25, 10.139.64.7, 43527, None)
22/12/13 09:56:24 INFO DataSourceFactory$: DataSource Jdbc URL: jdbc:mariadb://consolidated-westeuropec2-prod-metastore-1.mysql.database.azure.com:3306/organization8773400708728304?useSSL=true&sslMode=VERIFY_CA&disableSslHostnameVerification=true&trustServerCertificate=false&serverSslCert=/databricks/common/mysql-ssl-ca-cert.crt
22/12/13 09:56:24 INFO HikariDataSource: metastore-monitor - Starting...
22/12/13 09:56:24 INFO HikariDataSource: metastore-monitor - Start completed.
22/12/13 09:56:24 INFO HikariDataSource: metastore-monitor - Shutdown initiated...
22/12/13 09:56:24 INFO HikariDataSource: metastore-monitor - Shutdown completed.
22/12/13 09:56:24 INFO MetastoreMonitor: Metastore healthcheck successful (connection duration = 207 milliseconds)
22/12/13 09:56:27 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:27 INFO BlockManagerInfo: Added broadcast_35_piece0 in memory on 10.139.64.5:45187 (size: 13.6 KiB, free: 1504.3 MiB)
22/12/13 09:56:30 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:33 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:34 INFO DriverCorral: DBFS health check ok
22/12/13 09:56:34 INFO HiveMetaStore: 1: get_database: default
22/12/13 09:56:34 INFO audit: ugi=root  ip=unknown-ip-addr  cmd=get_database: default
22/12/13 09:56:34 INFO DriverCorral: Metastore health check ok
22/12/13 09:56:36 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:39 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:42 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:45 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:48 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:51 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:54 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:57 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:57:00 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:57:03 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:57:06 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:57:08 INFO CommChannelWebSocket: onWebSocketConnect: websocket connected with session: WebSocketSession[websocket=JettyAnnotatedEventDriver[com.databricks.backend.daemon.driver.CommChannelWebSocket@e83bc3c],behavior=SERVER,connection=WebSocketServerConnection@52d196e1::DecryptedEndPoint@685a7d89{l=/10.139.64.4:6062,r=/10.139.0.4:51020,OPEN,fill=-,flush=-,to=2/7200000},remote=WebSocketRemoteEndpoint@486edc7e[batching=true],incoming=JettyAnnotatedEventDriver[com.databricks.backend.daemon.driver.CommChannelWebSocket@e83bc3c],outgoing=ExtensionStack[queueSize=0,extensions=[],incoming=org.eclipse.jetty.websocket.common.WebSocketSession,outgoing=org.eclipse.jetty.websocket.server.WebSocketServerConnection]]
22/12/13 09:57:08 ERROR OutgoingDirectNotebookMessageBuffer: Session should be closed before interrupting.
22/12/13 09:57:08 INFO OutgoingDirectNotebookMessageBuffer: Stop MessageSendTask with session: 1914687926
22/12/13 09:57:08 INFO OutgoingDirectNotebookMessageBuffer: Start MessageSendTask with session: 349608004
22/12/13 09:57:08 INFO CommChannelWebSocket: onWebSocketClose: websocket closed with statusCode: 1006, reason: Disconnected
22/12/13 09:57:09 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:57:32 INFO BlockManagerMaster: Removed 23 successfully in removeExecutor
22/12/13 09:57:32 ERROR FileFormatWriter: Aborting job 6535aa05-18e2-43f7-83e5-bbcf56a1ee02.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 41.0 failed 4 times, most recent failure: Lost task 0.3 in stage 41.0 (TID 165) (10.139.64.5 executor 23): ExecutorLostFailure (executor 23 exited caused by one of the running tasks) Reason: Command exited with code 52
Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:3312)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:3244)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:3235)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:3235)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1424)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1424)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1424)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3524)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3462)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3450)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:51)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$runJob$1(DAGScheduler.scala:1169)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1157)
    at org.apache.spark.SparkContext.runJobInternal(SparkContext.scala:2713)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2696)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$1(FileFormatWriter.scala:299)
    at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:154)
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:207)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:126)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:124)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:138)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.$anonfun$applyOrElse$2(QueryExecution.scala:241)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$8(SQLExecution.scala:243)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:392)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$1(SQLExecution.scala:188)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:985)
    at org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:142)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:342)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.$anonfun$applyOrElse$1(QueryExecution.scala:241)
    at org.apache.spark.sql.execution.QueryExecution.org$apache$spark$sql$execution$QueryExecution$$withMVTagsIfNecessary(QueryExecution.scala:226)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.applyOrElse(QueryExecution.scala:239)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.applyOrElse(QueryExecution.scala:232)
    at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:512)
    at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:99)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:512)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:31)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:268)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:264)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:488)
    at org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$1(QueryExecution.scala:232)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:324)
    at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:232)
    at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:186)
    at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:177)
    at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:268)
    at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:965)
    at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:430)
    at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:397)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:251)
    at org.apache.spark.sql.DataFrameWriter.csv(DataFrameWriter.scala:956)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
    at py4j.Gateway.invoke(Gateway.java:306)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:115)
    at java.lang.Thread.run(Thread.java:750)
22/12/13 09:57:32 INFO ClusterLoadMonitor: Removed query with execution ID:9. Current active queries:0
22/12/13 09:57:32 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20221213083124-0000/26 is now RUNNING
22/12/13 09:57:32 WARN JupyterDriverLocal: User code returned error with traceback: [0;31m---------------------------------------------------------------------------[0m
[0;31mPy4JJavaError[0m                             Traceback (most recent call last)
[0;32m<command-3309668900398459>[0m in [0;36m<cell line: 268>[0;34m()[0m
[1;32m    266[0m [0;34m[0m[0m
[1;32m    267[0m [0m_type[0m [0;34m=[0m [0;34m"orderState"[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 268[0;31m [0mcheck_if_dupe[0m[0;34m([0m[0m_type[0m[0;34m,[0m [0myear[0m[0;34m,[0m [0mmonth[0m[0;34m,[0m [0mday[0m[0;34m,[0m [0myear_prev[0m[0;34m,[0m [0mmonth_prev[0m[0;34m,[0m [0mday_prev[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    269[0m [0;31m#display(df)[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m
[1;32m    270[0m [0;31m#check_df(df, _type)[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m

[0;32m<command-3309668900398459>[0m in [0;36mcheck_if_dupe[0;34m(_type, year, month, day, year_prev, month_prev, day_prev)[0m
[1;32m    235[0m     [0mdf_flattened[0m [0;34m=[0m [0mdf_flattened[0m[0;34m.[0m[0mna[0m[0;34m.[0m[0mfill[0m[0;34m([0m[0mvalue[0m[0;34m=[0m[0;36m0[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0;32m/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py[0m in [0;36mget_return_value[0;34m(answer, gateway_client, target_id, name)[0m
[1;32m    324[0m             [0mvalue[0m [0;34m=[0m [0mOUTPUT_CONVERTER[0m[0;34m[[0m[0mtype[0m[0;34m][0m[0;34m([0m[0manswer[0m[0;34m[[0m[0;36m2[0m[0;34m:[0m[0;34m][0m[0;34m,[0m [0mgateway_client[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m    325[0m             [0;32mif[0m [0manswer[0m[0;34m[[0m[0;36m1[0m[0;34m][0m [0;34m==[0m [0mREFERENCE_TYPE[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 326[0;31m                 raise Py4JJavaError(
[0m[1;32m    327[0m                     [0;34m"An error occurred while calling {0}{1}{2}.\n"[0m[0;34m.[0m[0;34m[0m[0;34m[0m[0m
[1;32m    328[0m                     format(target_id, ".", name), value)

[0;31mPy4JJavaError[0m: An error occurred while calling o3023.csv.
: org.apache.spark.SparkException: Job aborted.
    at org.apache.spark.sql.errors.QueryExecutionErrors$.jobAbortedError(QueryExecutionErrors.scala:882)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$1(FileFormatWriter.scala:334)
    at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:154)
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:207)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:126)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:124)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:138)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.$anonfun$applyOrElse$2(QueryExecution.scala:241)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$8(SQLExecution.scala:243)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:392)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$1(SQLExecution.scala:188)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:985)
    at org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:142)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:342)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.$anonfun$applyOrElse$1(QueryExecution.scala:241)
    at org.apache.spark.sql.execution.QueryExecution.org$apache$spark$sql$execution$QueryExecution$$withMVTagsIfNecessary(QueryExecution.scala:226)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.applyOrElse(QueryExecution.scala:239)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.applyOrElse(QueryExecution.scala:232)
    at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:512)
    at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:99)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:512)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:31)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:268)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:264)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:488)
    at org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$1(QueryExecution.scala:232)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:324)
    at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:232)
    at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:186)
    ... 49 more

22/12/13 09:57:32 INFO ProgressReporter$: Removed result fetcher for 3479767264321116845_5405572452950350648_76aea015601745f79d10ebdb188a0ee2
22/12/13 09:57:33 INFO ClusterLoadAvgHelper: Current cluster load: 0, Old Ema: 1.0, New Ema: 0.85
22/12/13 09:57:36 INFO ClusterLoadAvgHelper: Current cluster load: 0, Old Ema: 0.85, New Ema: 0.0
22/12/13 09:57:40 INFO StandaloneSchedulerBackend$StandaloneDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.139.64.5:50180) with ID 26,  ResourceProfileId 0
22/12/13 09:57:42 INFO BlockManagerMasterEndpoint: Registering block manager 10.139.64.5:44301 with 1504.5 MiB RAM, BlockManagerId(26, 10.139.64.5, 44301, None)
22/12/13 09:58:28 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20221213083124-0000/25 is now DECOMMISSIONED (worker decommissioned because of kill request from HTTP endpoint (data migration disabled))
22/12/13 09:58:28 INFO StandaloneSchedulerBackend: Asked to decommission executor app-20221213083124-0000/25
22/12/13 09:58:28 INFO StandaloneSchedulerBackend: Decommission executors: 25
22/12/13 09:58:28 INFO BlockManagerMasterEndpoint: Mark BlockManagers (BlockManagerId(25, 10.139.64.7, 43527, None)) as being decommissioning.
22/12/13 09:58:28 INFO StandaloneSchedulerBackend: Notify executor 25 to decommissioning.
22/12/13 09:58:28 INFO StandaloneSchedulerBackend: Executor app-20221213083124-0000/25 decommissioned: ExecutorDecommissionInfo(worker decommissioned because of kill request from HTTP endpoint (data migration disabled),Some(10.139.64.7),false)
22/12/13 09:58:28 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20221213083124-0000/25 is now LOST (worker lost)
22/12/13 09:58:28 INFO StandaloneSchedulerBackend: Executor app-20221213083124-0000/25 removed: worker lost
22/12/13 09:58:28 INFO StandaloneAppClient$ClientEndpoint: Master removed worker worker-20221213085244-10.139.64.7-41835: 10.139.64.7:41835 got disassociated
22/12/13 09:58:28 INFO StandaloneSchedulerBackend: Worker worker-20221213085244-10.139.64.7-41835 removed: 10.139.64.7:41835 got disassociated
22/12/13 09:58:28 ERROR TaskSchedulerImpl: Lost executor 25 on 10.139.64.7: Executor decommission: worker decommissioned because of kill request from HTTP endpoint (data migration disabled)
22/12/13 09:58:28 INFO DAGScheduler: Executor lost: 25 (epoch 18)
22/12/13 09:58:28 INFO TaskSchedulerImpl: Handle removed worker worker-20221213085244-10.139.64.7-41835: 10.139.64.7:41835 got disassociated
22/12/13 09:58:28 INFO BlockManagerMasterEndpoint: Trying to remove executor 25 from BlockManagerMaster.
22/12/13 09:58:28 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(25, 10.139.64.7, 43527, None)
22/12/13 09:58:28 INFO BlockManagerMaster: Removed 25 successfully in removeExecutor
22/12/13 09:58:28 INFO DAGScheduler: Shuffle files lost for host: 10.139.64.7 (epoch 18)
22/12/13 09:58:28 INFO DAGScheduler: Shuffle files lost for worker worker-20221213085244-10.139.64.7-41835 on host 10.139.64.7
```

I have tried to increase the driver memory without any success and switching to a higher performer cluster without any success.

Body:
I have seen many ppl having this issue but none of those were able to resolve my issue. I have a json file that I read in as a spark dataframe and then flatten it. Lastly, I try to save the flatten file as a csv with +100 columns and >47 million rows and many nulls.

I get the error org.apache.spark.SparkException: Job aborted.
Here is part of the stack trace:

WARN TaskSetManager: Lost task 0.2 in stage 41.0 (TID 164) (10.139.64.7 executor 22): java.lang.BootstrapMethodError: call site initialization exception
    at java.lang.invoke.CallSite.makeSite(CallSite.java:341)
    at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:307)
    at java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:297)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.recordWriteFailure$1(FileFormatWriter.scala:403)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:431)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$12(FileFormatWriter.scala:310)
    at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$3(ResultTask.scala:75)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:75)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:55)
    at org.apache.spark.scheduler.Task.doRunTask(Task.scala:169)
    at org.apache.spark.scheduler.Task.$anonfun$run$4(Task.scala:137)
    at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:104)
    at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:137)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.Task.run(Task.scala:96)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$13(Executor.scala:902)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1696)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:905)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:760)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.InternalError: BMH.reinvoke=Lambda(a0:L/SpeciesData<LLL>,a1:L,a2:L,a3:L,a4:L,a5:L,a6:L,a7:L,a8:L,a9:L,a10:L,a11:L)=>{
    t12:L=BoundMethodHandle$Species_L3.argL2(a0:L);
    t13:L=MethodHandle.invokeBasic(t12:L,a2:L);
    t14:L=BoundMethodHandle$Species_L3.argL1(a0:L);
    t15:L=MethodHandle.invokeBasic(t14:L,a1:L);
    t16:L=MethodHandleImpl.array(a4:L,a5:L,a6:L,a7:L,a8:L,a9:L,a10:L,a11:L);
    t17:L=BoundMethodHandle$Species_L3.argL0(a0:L);
    t18:L=MethodHandle.invokeBasic(t17:L,t15:L,t13:L,a3:L,t16:L);t18:L}
    at java.lang.invoke.MethodHandleStatics.newInternalError(MethodHandleStatics.java:127)
    at java.lang.invoke.LambdaForm.compileToBytecode(LambdaForm.java:660)
    at java.lang.invoke.LambdaForm.prepare(LambdaForm.java:635)
    at java.lang.invoke.MethodHandle.<init>(MethodHandle.java:461)
    at java.lang.invoke.BoundMethodHandle.<init>(BoundMethodHandle.java:58)
    at java.lang.invoke.BoundMethodHandle$Species_L3.<init>(Species_L3)
    at java.lang.invoke.BoundMethodHandle$Species_L3.make(Species_L3)
    at java.lang.invoke.BoundMethodHandle$Species_LL.copyWithExtendL(Species_LL)
    at java.lang.invoke.MethodHandleImpl.makePairwiseConvertByEditor(MethodHandleImpl.java:231)
    at java.lang.invoke.MethodHandleImpl.makePairwiseConvert(MethodHandleImpl.java:194)
    at java.lang.invoke.MethodHandleImpl.makePairwiseConvert(MethodHandleImpl.java:380)
    at java.lang.invoke.MethodHandle.asTypeUncached(MethodHandle.java:776)
    at java.lang.invoke.MethodHandle.asType(MethodHandle.java:761)
    at java.lang.invoke.MethodHandleImpl$AsVarargsCollector.asTypeUncached(MethodHandleImpl.java:508)
    at java.lang.invoke.MethodHandle.asType(MethodHandle.java:761)
    at java.lang.invoke.CallSite.makeSite(CallSite.java:323)
    ... 25 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

22/12/13 09:56:11 INFO TaskSetManager: Starting task 0.3 in stage 41.0 (TID 165) (10.139.64.5, executor 23, partition 0, PROCESS_LOCAL, taskResourceAssignments Map())
22/12/13 09:56:12 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:12 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20221213083124-0000/22 is now EXITED (Command exited with code 50)
22/12/13 09:56:12 INFO StandaloneSchedulerBackend: Executor app-20221213083124-0000/22 removed: Command exited with code 50
22/12/13 09:56:12 ERROR TaskSchedulerImpl: Lost executor 22 on 10.139.64.7: Command exited with code 50
22/12/13 09:56:12 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20221213083124-0000/25 on worker-20221213085244-10.139.64.7-41835 (10.139.64.7:41835) with 4 core(s)
22/12/13 09:56:12 INFO DAGScheduler: Executor lost: 22 (epoch 18)
22/12/13 09:56:12 INFO StandaloneSchedulerBackend: Granted executor ID app-20221213083124-0000/25 on hostPort 10.139.64.7:41835 with 4 core(s), 3.1 GiB RAM
22/12/13 09:56:12 INFO BlockManagerMasterEndpoint: Trying to remove executor 22 from BlockManagerMaster.
22/12/13 09:56:12 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(22, 10.139.64.7, 40355, None)
22/12/13 09:56:12 INFO BlockManagerMaster: Removed 22 successfully in removeExecutor
22/12/13 09:56:12 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20221213083124-0000/25 is now RUNNING
22/12/13 09:56:15 INFO StandaloneSchedulerBackend$StandaloneDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.139.64.6:55600) with ID 24,  ResourceProfileId 0
22/12/13 09:56:15 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:16 INFO BlockManagerMasterEndpoint: Registering block manager 10.139.64.6:39649 with 1504.5 MiB RAM, BlockManagerId(24, 10.139.64.6, 39649, None)
22/12/13 09:56:16 INFO BlockManagerInfo: Added broadcast_36_piece0 in memory on 10.139.64.5:45187 (size: 220.5 KiB, free: 1504.3 MiB)
22/12/13 09:56:18 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:18 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.139.64.5:45187 (size: 18.1 KiB, free: 1504.3 MiB)
22/12/13 09:56:21 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:22 INFO StandaloneSchedulerBackend$StandaloneDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.139.64.7:55476) with ID 25,  ResourceProfileId 0
22/12/13 09:56:24 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:24 INFO BlockManagerMasterEndpoint: Registering block manager 10.139.64.7:43527 with 1504.5 MiB RAM, BlockManagerId(25, 10.139.64.7, 43527, None)
22/12/13 09:56:24 INFO DataSourceFactory$: DataSource Jdbc URL: jdbc:mariadb://consolidated-westeuropec2-prod-metastore-1.mysql.database.azure.com:3306/organization8773400708728304?useSSL=true&sslMode=VERIFY_CA&disableSslHostnameVerification=true&trustServerCertificate=false&serverSslCert=/databricks/common/mysql-ssl-ca-cert.crt
22/12/13 09:56:24 INFO HikariDataSource: metastore-monitor - Starting...
22/12/13 09:56:24 INFO HikariDataSource: metastore-monitor - Start completed.
22/12/13 09:56:24 INFO HikariDataSource: metastore-monitor - Shutdown initiated...
22/12/13 09:56:24 INFO HikariDataSource: metastore-monitor - Shutdown completed.
22/12/13 09:56:24 INFO MetastoreMonitor: Metastore healthcheck successful (connection duration = 207 milliseconds)
22/12/13 09:56:27 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:27 INFO BlockManagerInfo: Added broadcast_35_piece0 in memory on 10.139.64.5:45187 (size: 13.6 KiB, free: 1504.3 MiB)
22/12/13 09:56:30 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:33 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:34 INFO DriverCorral: DBFS health check ok
22/12/13 09:56:34 INFO HiveMetaStore: 1: get_database: default
22/12/13 09:56:34 INFO audit: ugi=root  ip=unknown-ip-addr  cmd=get_database: default
22/12/13 09:56:34 INFO DriverCorral: Metastore health check ok
22/12/13 09:56:36 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:39 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:42 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:45 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:48 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:51 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:54 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:56:57 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:57:00 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:57:03 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:57:06 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:57:08 INFO CommChannelWebSocket: onWebSocketConnect: websocket connected with session: WebSocketSession[websocket=JettyAnnotatedEventDriver[com.databricks.backend.daemon.driver.CommChannelWebSocket@e83bc3c],behavior=SERVER,connection=WebSocketServerConnection@52d196e1::DecryptedEndPoint@685a7d89{l=/10.139.64.4:6062,r=/10.139.0.4:51020,OPEN,fill=-,flush=-,to=2/7200000},remote=WebSocketRemoteEndpoint@486edc7e[batching=true],incoming=JettyAnnotatedEventDriver[com.databricks.backend.daemon.driver.CommChannelWebSocket@e83bc3c],outgoing=ExtensionStack[queueSize=0,extensions=[],incoming=org.eclipse.jetty.websocket.common.WebSocketSession,outgoing=org.eclipse.jetty.websocket.server.WebSocketServerConnection]]
22/12/13 09:57:08 ERROR OutgoingDirectNotebookMessageBuffer: Session should be closed before interrupting.
22/12/13 09:57:08 INFO OutgoingDirectNotebookMessageBuffer: Stop MessageSendTask with session: 1914687926
22/12/13 09:57:08 INFO OutgoingDirectNotebookMessageBuffer: Start MessageSendTask with session: 349608004
22/12/13 09:57:08 INFO CommChannelWebSocket: onWebSocketClose: websocket closed with statusCode: 1006, reason: Disconnected
22/12/13 09:57:09 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0
22/12/13 09:57:32 INFO BlockManagerMaster: Removed 23 successfully in removeExecutor
22/12/13 09:57:32 ERROR FileFormatWriter: Aborting job 6535aa05-18e2-43f7-83e5-bbcf56a1ee02.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 41.0 failed 4 times, most recent failure: Lost task 0.3 in stage 41.0 (TID 165) (10.139.64.5 executor 23): ExecutorLostFailure (executor 23 exited caused by one of the running tasks) Reason: Command exited with code 52
Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:3312)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:3244)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:3235)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:3235)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1424)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1424)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1424)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3524)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3462)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3450)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:51)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$runJob$1(DAGScheduler.scala:1169)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1157)
    at org.apache.spark.SparkContext.runJobInternal(SparkContext.scala:2713)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2696)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$1(FileFormatWriter.scala:299)
    at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:154)
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:207)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:126)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:124)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:138)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.$anonfun$applyOrElse$2(QueryExecution.scala:241)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$8(SQLExecution.scala:243)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:392)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$1(SQLExecution.scala:188)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:985)
    at org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:142)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:342)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.$anonfun$applyOrElse$1(QueryExecution.scala:241)
    at org.apache.spark.sql.execution.QueryExecution.org$apache$spark$sql$execution$QueryExecution$$withMVTagsIfNecessary(QueryExecution.scala:226)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.applyOrElse(QueryExecution.scala:239)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.applyOrElse(QueryExecution.scala:232)
    at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:512)
    at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:99)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:512)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:31)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:268)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:264)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:488)
    at org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$1(QueryExecution.scala:232)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:324)
    at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:232)
    at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:186)
    at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:177)
    at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:268)
    at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:965)
    at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:430)
    at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:397)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:251)
    at org.apache.spark.sql.DataFrameWriter.csv(DataFrameWriter.scala:956)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
    at py4j.Gateway.invoke(Gateway.java:306)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:115)
    at java.lang.Thread.run(Thread.java:750)
22/12/13 09:57:32 INFO ClusterLoadMonitor: Removed query with execution ID:9. Current active queries:0
22/12/13 09:57:32 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20221213083124-0000/26 is now RUNNING
22/12/13 09:57:32 WARN JupyterDriverLocal: User code returned error with traceback: [0;31m---------------------------------------------------------------------------[0m
[0;31mPy4JJavaError[0m                             Traceback (most recent call last)
[0;32m<command-3309668900398459>[0m in [0;36m<cell line: 268>[0;34m()[0m
[1;32m    266[0m [0;34m[0m[0m
[1;32m    267[0m [0m_type[0m [0;34m=[0m [0;34m"orderState"[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 268[0;31m [0mcheck_if_dupe[0m[0;34m([0m[0m_type[0m[0;34m,[0m [0myear[0m[0;34m,[0m [0mmonth[0m[0;34m,[0m [0mday[0m[0;34m,[0m [0myear_prev[0m[0;34m,[0m [0mmonth_prev[0m[0;34m,[0m [0mday_prev[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m    269[0m [0;31m#display(df)[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m
[1;32m    270[0m [0;31m#check_df(df, _type)[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m

[0;32m<command-3309668900398459>[0m in [0;36mcheck_if_dupe[0;34m(_type, year, month, day, year_prev, month_prev, day_prev)[0m
[1;32m    235[0m     [0mdf_flattened[0m [0;34m=[0m [0mdf_flattened[0m[0;34m.[0m[0mna[0m[0;34m.[0m[0mfill[0m[0;34m([0m[0mvalue[0m[0;34m=[0m[0;36m0[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0;32m/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py[0m in [0;36mget_return_value[0;34m(answer, gateway_client, target_id, name)[0m
[1;32m    324[0m             [0mvalue[0m [0;34m=[0m [0mOUTPUT_CONVERTER[0m[0;34m[[0m[0mtype[0m[0;34m][0m[0;34m([0m[0manswer[0m[0;34m[[0m[0;36m2[0m[0;34m:[0m[0;34m][0m[0;34m,[0m [0mgateway_client[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m    325[0m             [0;32mif[0m [0manswer[0m[0;34m[[0m[0;36m1[0m[0;34m][0m [0;34m==[0m [0mREFERENCE_TYPE[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 326[0;31m                 raise Py4JJavaError(
[0m[1;32m    327[0m                     [0;34m"An error occurred while calling {0}{1}{2}.\n"[0m[0;34m.[0m[0;34m[0m[0;34m[0m[0m
[1;32m    328[0m                     format(target_id, ".", name), value)

[0;31mPy4JJavaError[0m: An error occurred while calling o3023.csv.
: org.apache.spark.SparkException: Job aborted.
    at org.apache.spark.sql.errors.QueryExecutionErrors$.jobAbortedError(QueryExecutionErrors.scala:882)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$1(FileFormatWriter.scala:334)
    at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:154)
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:207)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:126)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:124)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:138)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.$anonfun$applyOrElse$2(QueryExecution.scala:241)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$8(SQLExecution.scala:243)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:392)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$1(SQLExecution.scala:188)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:985)
    at org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:142)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:342)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.$anonfun$applyOrElse$1(QueryExecution.scala:241)
    at org.apache.spark.sql.execution.QueryExecution.org$apache$spark$sql$execution$QueryExecution$$withMVTagsIfNecessary(QueryExecution.scala:226)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.applyOrElse(QueryExecution.scala:239)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.applyOrElse(QueryExecution.scala:232)
    at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:512)
    at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:99)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:512)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:31)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:268)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:264)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:488)
    at org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$1(QueryExecution.scala:232)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:324)
    at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:232)
    at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:186)
    ... 49 more

22/12/13 09:57:32 INFO ProgressReporter$: Removed result fetcher for 3479767264321116845_5405572452950350648_76aea015601745f79d10ebdb188a0ee2
22/12/13 09:57:33 INFO ClusterLoadAvgHelper: Current cluster load: 0, Old Ema: 1.0, New Ema: 0.85
22/12/13 09:57:36 INFO ClusterLoadAvgHelper: Current cluster load: 0, Old Ema: 0.85, New Ema: 0.0
22/12/13 09:57:40 INFO StandaloneSchedulerBackend$StandaloneDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.139.64.5:50180) with ID 26,  ResourceProfileId 0
22/12/13 09:57:42 INFO BlockManagerMasterEndpoint: Registering block manager 10.139.64.5:44301 with 1504.5 MiB RAM, BlockManagerId(26, 10.139.64.5, 44301, None)
22/12/13 09:58:28 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20221213083124-0000/25 is now DECOMMISSIONED (worker decommissioned because of kill request from HTTP endpoint (data migration disabled))
22/12/13 09:58:28 INFO StandaloneSchedulerBackend: Asked to decommission executor app-20221213083124-0000/25
22/12/13 09:58:28 INFO StandaloneSchedulerBackend: Decommission executors: 25
22/12/13 09:58:28 INFO BlockManagerMasterEndpoint: Mark BlockManagers (BlockManagerId(25, 10.139.64.7, 43527, None)) as being decommissioning.
22/12/13 09:58:28 INFO StandaloneSchedulerBackend: Notify executor 25 to decommissioning.
22/12/13 09:58:28 INFO StandaloneSchedulerBackend: Executor app-20221213083124-0000/25 decommissioned: ExecutorDecommissionInfo(worker decommissioned because of kill request from HTTP endpoint (data migration disabled),Some(10.139.64.7),false)
22/12/13 09:58:28 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20221213083124-0000/25 is now LOST (worker lost)
22/12/13 09:58:28 INFO StandaloneSchedulerBackend: Executor app-20221213083124-0000/25 removed: worker lost
22/12/13 09:58:28 INFO StandaloneAppClient$ClientEndpoint: Master removed worker worker-20221213085244-10.139.64.7-41835: 10.139.64.7:41835 got disassociated
22/12/13 09:58:28 INFO StandaloneSchedulerBackend: Worker worker-20221213085244-10.139.64.7-41835 removed: 10.139.64.7:41835 got disassociated
22/12/13 09:58:28 ERROR TaskSchedulerImpl: Lost executor 25 on 10.139.64.7: Executor decommission: worker decommissioned because of kill request from HTTP endpoint (data migration disabled)
22/12/13 09:58:28 INFO DAGScheduler: Executor lost: 25 (epoch 18)
22/12/13 09:58:28 INFO TaskSchedulerImpl: Handle removed worker worker-20221213085244-10.139.64.7-41835: 10.139.64.7:41835 got disassociated
22/12/13 09:58:28 INFO BlockManagerMasterEndpoint: Trying to remove executor 25 from BlockManagerMaster.
22/12/13 09:58:28 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(25, 10.139.64.7, 43527, None)
22/12/13 09:58:28 INFO BlockManagerMaster: Removed 25 successfully in removeExecutor
22/12/13 09:58:28 INFO DAGScheduler: Shuffle files lost for host: 10.139.64.7 (epoch 18)
22/12/13 09:58:28 INFO DAGScheduler: Shuffle files lost for worker worker-20221213085244-10.139.64.7-41835 on host 10.139.64.7

I have tried to increase the driver memory without any success and switching to a higher performer cluster without any success.


来源: stackexchange

问:
Notice: Undefined index: & Fatal error: Uncaught PDOException: SQLSTATE[42S22]:
Fatal error: Uncaught PDOException: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'expirationDate' in 'field list' in C:\xampp\htdocs\webapps-project-main\model\item\insertItem.php:76 Stack trace: #0 C:\xampp\htdocs\webapps-project-main\model\item\insertItem.php(76): PDOStatement->execute(Array) #1 {main} thrown in C:\xampp\htdocs\webapps-project-main\model\item\insertItem.php on line 76

also this..

Notice: Undefined index: itemDetailsexpirationDate in C:\xampp\htdocs\webapps-project-main\model\item\insertItem.php on line 18

```<code>$insertItemStatement->execute(['itemNumber' => $itemNumber, 'itemName' => $itemName, 'discount' => $discount, 'stock' => $quantity, 'unitPrice' => $unitPrice, 'status' => $status, 'description' => $description, 'expirationDate' => $expirationDate]);
</code>```

also this..

```<code>$expirationDate = htmlentities($_POST['itemDetailsexpirationDate']);
</code>```

ContentLicense: CC BY-SA 4.0

Body:
Fatal error: Uncaught PDOException: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'expirationDate' in 'field list' in C:\xampp\htdocs\webapps-project-main\model\item\insertItem.php:76 Stack trace: #0 C:\xampp\htdocs\webapps-project-main\model\item\insertItem.php(76): PDOStatement->execute(Array) #1 {main} thrown in C:\xampp\htdocs\webapps-project-main\model\item\insertItem.php on line 76

also this..

Notice: Undefined index: itemDetailsexpirationDate in C:\xampp\htdocs\webapps-project-main\model\item\insertItem.php on line 18

$insertItemStatement->execute(['itemNumber' => $itemNumber, 'itemName' => $itemName, 'discount' => $discount, 'stock' => $quantity, 'unitPrice' => $unitPrice, 'status' => $status, 'description' => $description, 'expirationDate' => $expirationDate]);

also this..

$expirationDate = htmlentities($_POST['itemDetailsexpirationDate']);


问:
java.lang.NoClassDefFoundError: com/google/gson/JsonElement
The project runs okay on Eclipse which I'm using but when I try to run it on the terminal it has this error:

bin % java Parsing

Error: Unable to initialize main class Parsing

Caused by: java.lang.NoClassDefFoundError: com/google/gson/JsonElement

I also selected .jars on the Order and Export tab? What am I possibly missing?

Parsing.java:

```<code>import java.util.*;
import com.google.gson.Gson;
import com.google.gson.JsonObject;

public class Parsing {
    private static Scanner scanner = new Scanner(System.in);
    Gson gson = new Gson();

    private ArrayList<String> get(String[] field) {
        ArrayList<String> valueList = new ArrayList<>();
        String value;

        for(int i=0; i<field.length; i++) {
            System.out.print(field[i] + ": ");
            value = scanner.nextLine();
            valueList.add(i, value);
        }

        return valueList;
    }

    public void put() {
        JsonObject data = new JsonObject();

        String[] field = {
                "name", "category", "price", "note", "description"
        };

        ArrayList<String> valueList = new ArrayList<>();
        valueList = get(field);

        for(int i=0; i<field.length; i++) {
            data.addProperty(field[i], valueList.get(i));
        }

        String jsonField = gson.toJson(data);
        System.out.println(jsonField);
    }

    public static void main(String[] args) {
        Parsing p = new Parsing();

        p.put();
    }
}

</code>```

The command line:

/Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home/bin/java -Dfile.encoding=UTF-8 -classpath  /Users/thiscomputer/java/Challenge3_Parsing/bin:/Users/thiscomputer/java/Challenge3_Parsing/gson-2.8.6-javadoc.jar:/Users/thiscomputer/java/Challenge3_Parsing/gson-2.8.6.jar:/Users/thiscomputer/java/Challenge3_Parsing/gson-2.9.1.jar Parsing

Body:
The project runs okay on Eclipse which I'm using but when I try to run it on the terminal it has this error:

bin % java Parsing

Error: Unable to initialize main class Parsing

Caused by: java.lang.NoClassDefFoundError: com/google/gson/JsonElement

I also selected .jars on the Order and Export tab? What am I possibly missing?

Parsing.java:

import java.util.*;
import com.google.gson.Gson;
import com.google.gson.JsonObject;

public class Parsing {
    private static Scanner scanner = new Scanner(System.in);
    Gson gson = new Gson();

    private ArrayList<String> get(String[] field) {
        ArrayList<String> valueList = new ArrayList<>();
        String value;

        for(int i=0; i<field.length; i++) {
            System.out.print(field[i] + ": ");
            value = scanner.nextLine();
            valueList.add(i, value);
        }

        return valueList;
    }

    public void put() {
        JsonObject data = new JsonObject();

        String[] field = {
                "name", "category", "price", "note", "description"
        };

        ArrayList<String> valueList = new ArrayList<>();
        valueList = get(field);

        for(int i=0; i<field.length; i++) {
            data.addProperty(field[i], valueList.get(i));
        }

        String jsonField = gson.toJson(data);
        System.out.println(jsonField);
    }

    public static void main(String[] args) {
        Parsing p = new Parsing();

        p.put();
    }
}

The command line:

/Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home/bin/java -Dfile.encoding=UTF-8 -classpath  /Users/thiscomputer/java/Challenge3_Parsing/bin:/Users/thiscomputer/java/Challenge3_Parsing/gson-2.8.6-javadoc.jar:/Users/thiscomputer/java/Challenge3_Parsing/gson-2.8.6.jar:/Users/thiscomputer/java/Challenge3_Parsing/gson-2.9.1.jar Parsing


问:
Issue sending & receiving streams between two clients in LiveKit's React Native SDK
I'm trying to build on the example app provided by livekit, so far I've implemented everything like the example app and I've been successful with connecting to a room on example website, I recieve audio from website, but I don't read the video stream, and I also can't send audio or video at all.

Steps to reproduce the behavior:

add the following to index.js

```<code>import { registerRootComponent } from "expo";
import { registerGlobals } from "livekit-react-native";

import App from "./App";
registerRootComponent(App);
registerGlobals();
</code>```

Rendering the following component in App.tsx

```<code>import { Participant, Room, Track } from "livekit-client";
import {
  useRoom,
  useParticipant,
  AudioSession,
  VideoView,
} from "livekit-react-native";
import { useEffect, useState } from "react";
import { Text, ListRenderItem, StyleSheet, FlatList, View } from "react-native";
import { ParticipantView } from "./ParticipantView";
import { RoomControls } from "./RoomControls";
import type { TrackPublication } from "livekit-client";

const App = () => {
  // Create a room state
  const [, setIsConnected] = useState(false);
  const [room] = useState(
    () =>
      new Room({
        publishDefaults: { simulcast: false },
        adaptiveStream: true,
      })
  );

  // Get the participants from the room
  const { participants } = useRoom(room);
  const url = "[hard-coded-url]";
  const token =
    "[hard-coded-token";
  useEffect(() => {
    let connect = async () => {
      // If you wish to configure audio, uncomment the following:
      await AudioSession.configureAudio({
        android: {
          preferredOutputList: ["speaker"],
        },
        ios: {
          defaultOutput: "speaker",
        },
      });
      await AudioSession.startAudioSession();
      await room.connect(url, token, {});
      await room.localParticipant.setCameraEnabled(true);
      await room.localParticipant.setMicrophoneEnabled(true);
      await room.localParticipant.enableCameraAndMicrophone();
      console.log("connected to ", url);
      setIsConnected(true);
    };

    connect();
    return () => {
      room.disconnect();
      AudioSession.stopAudioSession();
    };
  }, [url, token, room]);
  // Setup views.
  const stageView = participants.length > 0 && (
    <ParticipantView participant={participants[0]} style={styles.stage} />
  );

  const renderParticipant: ListRenderItem<Participant> = ({ item }) => {
    return (
      <ParticipantView participant={item} style={styles.otherParticipantView} />
    );
  };

  const otherParticipantsView = participants.length > 0 && (
    <FlatList
      data={participants}
      renderItem={renderParticipant}
      keyExtractor={(item) => item.sid}
      horizontal={true}
      style={styles.otherParticipantsList}
    />
  );

  const { cameraPublication, microphonePublication } = useParticipant(
    room.localParticipant
  );

  return (
    <View style={styles.container}>
      {stageView}
      {otherParticipantsView}
      <RoomControls
        micEnabled={isTrackEnabled(microphonePublication)}
        setMicEnabled={(enabled: boolean) => {
          room.localParticipant.setMicrophoneEnabled(enabled);
        }}
        cameraEnabled={isTrackEnabled(cameraPublication)}
        setCameraEnabled={(enabled: boolean) => {
          room.localParticipant.setCameraEnabled(enabled);
        }}
        onDisconnectClick={() => {
          //   navigation.pop();
          console.log("disconnected");
        }}
      />
    </View>
  );
};

function isTrackEnabled(pub?: TrackPublication): boolean {
  return !(pub?.isMuted ?? true);
}
const styles = StyleSheet.create({
  container: {
    flex: 1,
    alignItems: "center",
    justifyContent: "center",
  },
  stage: {
    flex: 1,
    width: "100%",
  },
  otherParticipantsList: {
    width: "100%",
    height: 150,
    flexGrow: 0,
  },
  otherParticipantView: {
    width: 150,
    height: 150,
  },
});

export default App;
</code>```

the components used here are mostly the same as what's in the example, I've removed the screensharing logic and the messages
5. I run the app using an expo development build
6. it will log that it's connected, you'll be able to hear sound from the remote participant, but not see any video or send any sound.
7. if i try to add
await room.localParticipant.enableCameraAndMicrophone();
in the useEffect, I get the following error:

```<code>Possible Unhandled Promise Rejection (id: 0):
Error: Not implemented.
getSettings@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:103733:24
@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:120307:109
generatorResume@[native code]
asyncGeneratorStep@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:21908:26
_next@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:21927:29
@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:21932:14
tryCallTwo@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:26656:9
doResolve@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:26788:25
Promise@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:26675:14
@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:21924:25
@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:120173:52
generatorResume@[native code]
asyncGeneratorStep@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:21908:26
_next@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:21927:29
tryCallOne@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:26648:16
@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:26729:27
@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:27687:26
_callTimer@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:27602:17
_callReactNativeMicrotasksPass@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:27635:17
callReactNativeMicrotasks@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:27799:44
__callReactNativeMicrotasks@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:21006:46
@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:20806:45
__guard@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:20986:15
flushedQueue@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:20805:21
flushedQueue@[native code]
</code>```

Expected behavior
This should both receive & send video and audio streams between the two clients

Body:
I'm trying to build on the example app provided by livekit, so far I've implemented everything like the example app and I've been successful with connecting to a room on example website, I recieve audio from website, but I don't read the video stream, and I also can't send audio or video at all.

Steps to reproduce the behavior:

add the following to index.js

import { registerRootComponent } from "expo";
import { registerGlobals } from "livekit-react-native";

import App from "./App";
registerRootComponent(App);
registerGlobals();

Rendering the following component in App.tsx

import { Participant, Room, Track } from "livekit-client";
import {
  useRoom,
  useParticipant,
  AudioSession,
  VideoView,
} from "livekit-react-native";
import { useEffect, useState } from "react";
import { Text, ListRenderItem, StyleSheet, FlatList, View } from "react-native";
import { ParticipantView } from "./ParticipantView";
import { RoomControls } from "./RoomControls";
import type { TrackPublication } from "livekit-client";

const App = () => {
  // Create a room state
  const [, setIsConnected] = useState(false);
  const [room] = useState(
    () =>
      new Room({
        publishDefaults: { simulcast: false },
        adaptiveStream: true,
      })
  );

  // Get the participants from the room
  const { participants } = useRoom(room);
  const url = "[hard-coded-url]";
  const token =
    "[hard-coded-token";
  useEffect(() => {
    let connect = async () => {
      // If you wish to configure audio, uncomment the following:
      await AudioSession.configureAudio({
        android: {
          preferredOutputList: ["speaker"],
        },
        ios: {
          defaultOutput: "speaker",
        },
      });
      await AudioSession.startAudioSession();
      await room.connect(url, token, {});
      await room.localParticipant.setCameraEnabled(true);
      await room.localParticipant.setMicrophoneEnabled(true);
      await room.localParticipant.enableCameraAndMicrophone();
      console.log("connected to ", url);
      setIsConnected(true);
    };

    connect();
    return () => {
      room.disconnect();
      AudioSession.stopAudioSession();
    };
  }, [url, token, room]);
  // Setup views.
  const stageView = participants.length > 0 && (
    <ParticipantView participant={participants[0]} style={styles.stage} />
  );

  const renderParticipant: ListRenderItem<Participant> = ({ item }) => {
    return (
      <ParticipantView participant={item} style={styles.otherParticipantView} />
    );
  };

  const otherParticipantsView = participants.length > 0 && (
    <FlatList
      data={participants}
      renderItem={renderParticipant}
      keyExtractor={(item) => item.sid}
      horizontal={true}
      style={styles.otherParticipantsList}
    />
  );

  const { cameraPublication, microphonePublication } = useParticipant(
    room.localParticipant
  );

  return (
    <View style={styles.container}>
      {stageView}
      {otherParticipantsView}
      <RoomControls
        micEnabled={isTrackEnabled(microphonePublication)}
        setMicEnabled={(enabled: boolean) => {
          room.localParticipant.setMicrophoneEnabled(enabled);
        }}
        cameraEnabled={isTrackEnabled(cameraPublication)}
        setCameraEnabled={(enabled: boolean) => {
          room.localParticipant.setCameraEnabled(enabled);
        }}
        onDisconnectClick={() => {
          //   navigation.pop();
          console.log("disconnected");
        }}
      />
    </View>
  );
};

function isTrackEnabled(pub?: TrackPublication): boolean {
  return !(pub?.isMuted ?? true);
}
const styles = StyleSheet.create({
  container: {
    flex: 1,
    alignItems: "center",
    justifyContent: "center",
  },
  stage: {
    flex: 1,
    width: "100%",
  },
  otherParticipantsList: {
    width: "100%",
    height: 150,
    flexGrow: 0,
  },
  otherParticipantView: {
    width: 150,
    height: 150,
  },
});

export default App;

the components used here are mostly the same as what's in the example, I've removed the screensharing logic and the messages
5. I run the app using an expo development build
6. it will log that it's connected, you'll be able to hear sound from the remote participant, but not see any video or send any sound.
7. if i try to add
await room.localParticipant.enableCameraAndMicrophone();
in the useEffect, I get the following error:

Possible Unhandled Promise Rejection (id: 0):
Error: Not implemented.
getSettings@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:103733:24
@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:120307:109
generatorResume@[native code]
asyncGeneratorStep@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:21908:26
_next@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:21927:29
@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:21932:14
tryCallTwo@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:26656:9
doResolve@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:26788:25
Promise@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:26675:14
@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:21924:25
@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:120173:52
generatorResume@[native code]
asyncGeneratorStep@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:21908:26
_next@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:21927:29
tryCallOne@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:26648:16
@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:26729:27
@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:27687:26
_callTimer@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:27602:17
_callReactNativeMicrotasksPass@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:27635:17
callReactNativeMicrotasks@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:27799:44
__callReactNativeMicrotasks@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:21006:46
@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:20806:45
__guard@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:20986:15
flushedQueue@http://192.168.1.150:8081/index.bundle?platform=ios&dev=true&hot=false:20805:21
flushedQueue@[native code]

Expected behavior
This should both receive & send video and audio streams between the two clients


问:
Read binary data of video files & Open it from Google Drive
I tried to read a video store on Google Drive by get the file data content following the code block below, i don't want to download the file to local, just read & load directly from GD.,

```<code>...
def print_file_content(service, file_id):
  """Print a file's content.

  Args:
    service: Drive API service instance.
    file_id: ID of the file.

  Returns:
    File's content if successful, None otherwise.
  """
  try:
    # GET FILE'S DATA AND READ
    content = service.files().get_media(fileId=file_id).execute()
    < CODE - OPEN CONTENT/VIDEO >
    #
  except Exception as e:
    print('An error occurred: %s' % e)
print_file_content(service, file_id)
</code>```

i got file's data, It has this format:

```<code>b'\x1aE\xdf\xa3\x01\x00\x00\x00\x00\x00\x00#B\x86\x81\x01B\xf7\x81\x01B\xf2\x81\x04B\xf3\x81\x08B\x82\x88matroskaB\x87\x81\x04B\x85\x81\x02\x18S\x80g\x01\x00\x00\x00\x00A\xffE\x11M\x9bt\xbeM\xbb\x8cS\xab\x84f\xa9I\x15S\xac\x82\x00\xdfM\xbb\x8cS\xab\x84k\xaeT\x16S\xac\x82\x01\'M\xbb\x8cS\xab\x84g\xc3T\x12S\xac\x82\x01\xafM...'
</code>```

then tried to read content by the mean of using: binascii, open(content,'rb'), io.BytesIO but all is not working, error.

i found a way to open video file is vlc, but when apply data read to it by all of above ways, even just paste content itself, it still not working.

HOW TO READ & OPEN A VIDEO FROM BINARY BYTES DATA OF IT BUT NOT A VIDEO OBJECT?

Body:
I tried to read a video store on Google Drive by get the file data content following the code block below, i don't want to download the file to local, just read & load directly from GD.,

...
def print_file_content(service, file_id):
  """Print a file's content.

  Args:
    service: Drive API service instance.
    file_id: ID of the file.

  Returns:
    File's content if successful, None otherwise.
  """
  try:
    # GET FILE'S DATA AND READ
    content = service.files().get_media(fileId=file_id).execute()

    #
  except Exception as e:
    print('An error occurred: %s' % e)
print_file_content(service, file_id)

i got file's data, It has this format:

b'\x1aE\xdf\xa3\x01\x00\x00\x00\x00\x00\x00#B\x86\x81\x01B\xf7\x81\x01B\xf2\x81\x04B\xf3\x81\x08B\x82\x88matroskaB\x87\x81\x04B\x85\x81\x02\x18S\x80g\x01\x00\x00\x00\x00A\xffE\x11M\x9bt\xbeM\xbb\x8cS\xab\x84f\xa9I\x15S\xac\x82\x00\xdfM\xbb\x8cS\xab\x84k\xaeT\x16S\xac\x82\x01\'M\xbb\x8cS\xab\x84g\xc3T\x12S\xac\x82\x01\xafM...'

then tried to read content by the mean of using: binascii, open(content,'rb'), io.BytesIO but all is not working, error.

i found a way to open video file is vlc, but when apply data read to it by all of above ways, even just paste content itself, it still not working.

HOW TO READ & OPEN A VIDEO FROM BINARY BYTES DATA OF IT BUT NOT A VIDEO OBJECT?


问:
Python - Distributing nodes in to mutex pools - but make sure no node is orphan
I have n no of nodes ranges from 2 to 100, I want to generate mutex pool and assign each node a mutex.
Want to make sure no node is orphan.
Mutex pool size should be of range (1-12), 0 because if there is only one node then there will be no mutex as it will be orphan.

problem I am trying to solve -
reading nodes, iterating over them and generating shell command for each node, but assigning a mutex so that all commands don't run at same time and overload system, so putting them in mutex pool . But there is requirement that no command is orphan at least it should be in pool with some other node

code

for item in nodes: run.sh item  'mutex=' + random.randint(1.10).__str()__


问:
WiX Burn BA - download payloads behind OAUTH, and is there a possibility to change the UpdateLocation and re-run engine detect?
I currently spike with WiX 3.11.2 to build a burn bootstrapping application that downloads packages from a local http server currently without any authentication.

Due to a lack of documentation of WiX burn I mostly reverse engineered how to set the UpdateLocation + how to check for updates via skimming through the UpdateViewModel.cs from the Wix3 repository.

As far as I have read, the

Engine.SetDownloadSource(ea.PackageOrContainerId,ea.PayloadId, dlUrl,user,pass);

can be used to download with no AUTH (last 2 parameters null) or plain authentication. Without authentication I already have a working prototype - but I will need the possibility to authenticate against a OAUTH secured web server.

If I were to need for instance to download behind a OAUTH2 authentication would the best practice be to use the OnResolveSourdce Event and instead of setting the Result to Result.Download, just do it myself with OAUTH2 libraries and download it to the ea.LocalSource so the engine basically thinks that the source has always been already available?

My second question is with regards to the UpdateLocation.
In the OnDetectUpdateBegin event I can set the ea.Result to Result.Ok if I want to check for upates - but I would like to be able to change the UpdateLocation during runtime, similar to what the Visual Studio Installer allows - to have different ReleaseFeeds, something like PROD or TESTING or Preview.

is there any possiblity to tell the Engine to set the UpdateLocation to a different URI? It would seem that the OnDetectUpdateBegin event would be perfect to change that, but the ea.UpdateLocation only has a Get, and I did not find a Engine.SetUpdate location or anything similar.

If I am not mistaken the VStudio installer is also built upon WiX so I would presume it should be somehow possible - am I missing something here?


问:
Programmatic access to SQLModel class attributes
I am trying to plug a SQLModel table onto a dash data table, handling pagination, filtering and page count in the backend, as explained here https://dash.plotly.com/datatable/callbacks

Working code

from sqlmodel import SQLModel
from sqlmodel import select
from sqlmodel import col
from sqlmodel import Session
from typing import Dict
from typing import List
from typing import Tuple

class Foo(SQLModel, table=True):
    bara: str
    barb: int

filters = {'bara': ('contains ', 'toto'), 'barb': ('>', 1)}

def filter_foos(filters: Dict[str, Tuple[str, str]]):
    """
    try to filter foos
    """
    query = select(Foo)
    if values := filters.get('bara'):
        query = query.where(col(Foo.bara).contains(values[1]))
    if values := filters.get('barb'):
        if values[0] == '>=':
            query = query.where(col(Foo.barb) >= values[1])
        elif values[0] == '<=':
            query = query.where(col(Foo.barb) <= values[1])
        elif values[0] == '!=':
            query = query.where(col(Foo.barb) != values[1])
        elif values[0] == '=':
            query = query.where(col(Foo.barb) == values[1])
        elif values[0] == '>':
            query = query.where(col(Foo.barb) > values[1])
        elif values[0] == '<':
            query = query.where(col(Foo.barb) < values[1])
    return query

def select_relevant_db_lines(
                              session: Session,
                              limit: int,
                              offset: int,
                              filters: Dict[str, Tuple[str, str]]
                              ) -> List:
    """
    Select relevant row lines from Foo.
    """
    if limit is not None and offset is not None:
        return list(session.exec(filter_foos(filters).offset(offset*limit).limit(limit)))
```
<p>My issue is that the filter function is awefully ugly, and not modular at all. If I have an new class</p>
<pre class="lang-py prettyprint-override"><code>class Fooo(SQLModel, table=True):
    toto: str
    titi: int
    tutu: int
</code>```

I will hand up redoing the same filter_foos boiler plate code

What I would like would be to have a dictionary to access Foo class attributes, something like (pseudo-code, does not wok)

foo_attributes: Dict = {
'bara': Foo.bara
'barb': Foo.barb
}
```
<p>That way I can dissociate the generic <code>str</code> <code>int</code> <code>datetime</code> and whatnot treatment and then map them on class attributes. Something like (pseudo-code, not working)</p>
<pre class="lang-py prettyprint-override"><code>def filter_ints(query, model_field: ???, operator: str, value: int):
    """
    try to filter ints
    """
    if not operator or not value:
        return query
    if operator == '>=':
        query = query.where(col(model_field) >= value)
    elif operator == '<=':
        query = query.where(col(model_field) <= value)
    elif operator == '!=':
        query = query.where(col(model_field) != value)
    elif operator == '=':
        query = query.where(col(model_field) == value)
    elif operator == '>':
        query = query.where(col(model_field) > value)
    elif operator == '<':
        query = query.where(col(model_field) < value)
    return query

def filter_strs(query, model_field: ???, value: int):
 """
    try to filter strs
    """
    if not value:
        return query
    query = query.where(col(model_field).contains(value))

def filter_models(model: Any, filters: Dict[str, Tuple[str, str]]):
    """
    try to filter any model
    """
    query = select(model)
    if not filters:
        return query
    for key, (operator, value) in filters:
        update_query(query, model, key, operator, value)
</code>```

Is it possible to do such a thing, or will I have to implement one ugly method after another each time I add a new table to show in my dash app?

Body:
I am trying to plug a SQLModel table onto a dash data table, handling pagination, filtering and page count in the backend, as explained here https://dash.plotly.com/datatable/callbacks

Working code

from sqlmodel import SQLModel
from sqlmodel import select
from sqlmodel import col
from sqlmodel import Session
from typing import Dict
from typing import List
from typing import Tuple

class Foo(SQLModel, table=True):
    bara: str
    barb: int

filters = {'bara': ('contains ', 'toto'), 'barb': ('>', 1)}

def filter_foos(filters: Dict[str, Tuple[str, str]]):
    """
    try to filter foos
    """
    query = select(Foo)
    if values := filters.get('bara'):
        query = query.where(col(Foo.bara).contains(values[1]))
    if values := filters.get('barb'):
        if values[0] == '>=':
            query = query.where(col(Foo.barb) >= values[1])
        elif values[0] == '<=':
            query = query.where(col(Foo.barb) <= values[1])
        elif values[0] == '!=':
            query = query.where(col(Foo.barb) != values[1])
        elif values[0] == '=':
            query = query.where(col(Foo.barb) == values[1])
        elif values[0] == '>':
            query = query.where(col(Foo.barb) > values[1])
        elif values[0] == '<':
            query = query.where(col(Foo.barb) < values[1])
    return query

def select_relevant_db_lines(
                              session: Session,
                              limit: int,
                              offset: int,
                              filters: Dict[str, Tuple[str, str]]
                              ) -> List:
    """
    Select relevant row lines from Foo.
    """
    if limit is not None and offset is not None:
        return list(session.exec(filter_foos(filters).offset(offset*limit).limit(limit)))

My issue is that the filter function is awefully ugly, and not modular at all. If I have an new class

class Fooo(SQLModel, table=True):
    toto: str
    titi: int
    tutu: int

I will hand up redoing the same filter_foos boiler plate code

What I would like would be to have a dictionary to access Foo class attributes, something like (pseudo-code, does not wok)

foo_attributes: Dict = {
'bara': Foo.bara
'barb': Foo.barb
}

That way I can dissociate the generic str int datetime and whatnot treatment and then map them on class attributes. Something like (pseudo-code, not working)

def filter_ints(query, model_field: ???, operator: str, value: int):
    """
    try to filter ints
    """
    if not operator or not value:
        return query
    if operator == '>=':
        query = query.where(col(model_field) >= value)
    elif operator == '<=':
        query = query.where(col(model_field) <= value)
    elif operator == '!=':
        query = query.where(col(model_field) != value)
    elif operator == '=':
        query = query.where(col(model_field) == value)
    elif operator == '>':
        query = query.where(col(model_field) > value)
    elif operator == '<':
        query = query.where(col(model_field) < value)
    return query

def filter_strs(query, model_field: ???, value: int):
 """
    try to filter strs
    """
    if not value:
        return query
    query = query.where(col(model_field).contains(value))

def filter_models(model: Any, filters: Dict[str, Tuple[str, str]]):
    """
    try to filter any model
    """
    query = select(model)
    if not filters:
        return query
    for key, (operator, value) in filters:
        update_query(query, model, key, operator, value)

Is it possible to do such a thing, or will I have to implement one ugly method after another each time I add a new table to show in my dash app?


问:
detecting a captcha pop-up
Hello I'm trying to get multiple results from a website, depending on the value in column E, filling all cells in the column H.
I'm wondering if there is a way to pause the execution if a captcha pop-up is opened.
Please do not mention Selenium. Either I'm not interested in OCR, I only want the program stops and waits for solving the captcha puzzle.</p>
```<code>Sub getPEC()

Dim MyHTML_Element As IHTMLElement
Dim HTMLDoc As HTMLDocument
Dim MyURL As String
Dim results As IHTMLElementCollection
Dim MyBrowser As Object

    MyURL = "https://www.registroimprese.it/home"
    Set MyBrowser = New InternetExplorer
    MyBrowser.Silent = True
    MyBrowser.navigate MyURL
    MyBrowser.Visible = True

    While Selection.Column = 5 And ActiveCell.Value <> ""
    Dim r As VbMsgBoxResult

        Do
        DoEvents
        Loop Until MyBrowser.readyState = READYSTATE_COMPLETE

        Application.Wait Now + TimeValue("00:00:02")

        Set HTMLDoc = MyBrowser.document

        HTMLDoc.getElementById("inputSearchFieldMob").Value = Selection.Value
        HTMLDoc.getElementById("selProvincia-Mob").Value = Range("P" & Selection.Row).Value
        HTMLDoc.getElementById("btnCercaGratuitaMob").Click

        Do
            DoEvents
        Loop Until MyBrowser.readyState = READYSTATE_COMPLETE
        Application.Wait Now + TimeValue("00:00:03")

        If a captcha pop-up is present Then
            r = MsgBox("continue?", vbYesNo, "Captcha?")
            If r = vbNo Then Exit Sub
        End If

        Set tbody = HTMLDoc.getElementsByClassName("table tableRisultatiGratuita")(0).getElementsByTagName("tbody")(0)
        Set datarow = tbody.getElementsByTagName("tr")

        If datarow.Length = 1 Then
            Set datarowtdlist = datarow(0).getElementsByTagName("td")
            Set datarowDIVlist = datarowtdlist(0).getElementsByTagName("DIV")
            Set datarowAlist = datarowDIVlist(0).getElementsByTagName("A")
            MyBrowser.navigate datarowAlist(0).href
            Do
                DoEvents
            Loop Until MyBrowser.readyState = READYSTATE_COMPLETE
        Else
            r = MsgBox("Select one", vbOKCancel)
            If r = vbCancel Then Exit Sub
        End If
        Set HTMLDoc = MyBrowser.document
        Range("H" & Selection.Row).Value = HTMLDoc.getElementsByClassName("ddPec")(0).Value
        MyBrowser.GoBack

        Range("E" & ActiveCell.Row + 1).Select

        i = i + 1
    Wend

End Sub

</code>```
```<code></code>```

Title: detecting a captcha pop-up

Body:
Hello I'm trying to get multiple results from a website, depending on the value in column E, filling all cells in the column H.
I'm wondering if there is a way to pause the execution if a captcha pop-up is opened.
Please do not mention Selenium. Either I'm not interested in OCR, I only want the program stops and waits for solving the captcha puzzle.

Sub getPEC()

Dim MyHTML_Element As IHTMLElement
Dim HTMLDoc As HTMLDocument
Dim MyURL As String
Dim results As IHTMLElementCollection
Dim MyBrowser As Object

    MyURL = "https://www.registroimprese.it/home"
    Set MyBrowser = New InternetExplorer
    MyBrowser.Silent = True
    MyBrowser.navigate MyURL
    MyBrowser.Visible = True

    While Selection.Column = 5 And ActiveCell.Value <> ""
    Dim r As VbMsgBoxResult

        Do
        DoEvents
        Loop Until MyBrowser.readyState = READYSTATE_COMPLETE

        Application.Wait Now + TimeValue("00:00:02")

        Set HTMLDoc = MyBrowser.document

        HTMLDoc.getElementById("inputSearchFieldMob").Value = Selection.Value
        HTMLDoc.getElementById("selProvincia-Mob").Value = Range("P" & Selection.Row).Value
        HTMLDoc.getElementById("btnCercaGratuitaMob").Click

        Do
            DoEvents
        Loop Until MyBrowser.readyState = READYSTATE_COMPLETE
        Application.Wait Now + TimeValue("00:00:03")

        If a captcha pop-up is present Then
            r = MsgBox("continue?", vbYesNo, "Captcha?")
            If r = vbNo Then Exit Sub
        End If

        Set tbody = HTMLDoc.getElementsByClassName("table tableRisultatiGratuita")(0).getElementsByTagName("tbody")(0)
        Set datarow = tbody.getElementsByTagName("tr")

        If datarow.Length = 1 Then
            Set datarowtdlist = datarow(0).getElementsByTagName("td")
            Set datarowDIVlist = datarowtdlist(0).getElementsByTagName("DIV")
            Set datarowAlist = datarowDIVlist(0).getElementsByTagName("A")
            MyBrowser.navigate datarowAlist(0).href
            Do
                DoEvents
            Loop Until MyBrowser.readyState = READYSTATE_COMPLETE
        Else
            r = MsgBox("Select one", vbOKCancel)
            If r = vbCancel Then Exit Sub
        End If
        Set HTMLDoc = MyBrowser.document
        Range("H" & Selection.Row).Value = HTMLDoc.getElementsByClassName("ddPec")(0).Value
        MyBrowser.GoBack

        Range("E" & ActiveCell.Row + 1).Select

        i = i + 1
    Wend

End Sub


问:
Is there possibility to map data in one dataframe based on mapping data from another dataframe in Python?
I need to map data based on piece of text. So, I have one df with raw data , and another df with mapping and which is static data containing two columns: “key words” and category. Based on key words searching in raw data description column , it should map appropriate category.

In SQL in access  it would look like:

```<code>DoCmd.RunSQL “Update Table_A Set Category = ‘XXX’ Where Description Like ‘%string%’
</code>```

But in case on above logic , I would need to change the code every time as new category will be created. So, that’s why I would like to make that mapping as static table , and make python categorize data based on key words placed in mapping table. Once file will be updated, python will just read already updated file , and there will be no need to update the logic.

Can anyone please suggest some solution to that?

Body:
I need to map data based on piece of text. So, I have one df with raw data , and another df with mapping and which is static data containing two columns: “key words” and category. Based on key words searching in raw data description column , it should map appropriate category.

In SQL in access  it would look like:

DoCmd.RunSQL “Update Table_A Set Category = ‘XXX’ Where Description Like ‘%string%’

But in case on above logic , I would need to change the code every time as new category will be created. So, that’s why I would like to make that mapping as static table , and make python categorize data based on key words placed in mapping table. Once file will be updated, python will just read already updated file , and there will be no need to update the logic.

Can anyone please suggest some solution to that?


问:
How to use Decorate with Factory in Zenject?
Could someone explain me, how to decorate object created by factory?

I've tried like this, but decorator's code not running

Container.BindFactory<Notification, INotificationHandler, NotificationFactory>()
    .FromFactory<CustomNotificationFactory>();
Container.Decorate<INotificationHandler>()
    .With<NotificationHandlerSendTimeOffsetDecorator>();
```