问:
Need msgbox text in a cell
I want below message in a cell:

```<code>Dim Msg As String
Msg = "Files moved: " & MovedCount & "(" & NotMovedCount + MovedCount & ")"
If NotMovedCount > 0 Then
    Msg = Msg & vbLf & "Files not moved:" & NotMovedCount & "(" _
        & NotMovedCount + MovedCount & ")" & vbLf & vbLf _
        & "The following files were not moved:" & vbLf _
        & Join(dict.keys, vbLf)
End If

MsgBox Msg, IIf(NotMovedCount = 0, vbInformation, vbCritical)
</code>```

Could you please help me.

I am moving some files from one folder to another. If the files are already available in the target folder, this code gives list in message box. However i need that information in excel cell for example sheet 1, cell AF5.

Title: Need msgbox text in a cell

Body:
I want below message in a cell:

Dim Msg As String
Msg = "Files moved: " & MovedCount & "(" & NotMovedCount + MovedCount & ")"
If NotMovedCount > 0 Then
    Msg = Msg & vbLf & "Files not moved:" & NotMovedCount & "(" _
        & NotMovedCount + MovedCount & ")" & vbLf & vbLf _
        & "The following files were not moved:" & vbLf _
        & Join(dict.keys, vbLf)
End If

MsgBox Msg, IIf(NotMovedCount = 0, vbInformation, vbCritical)

Could you please help me.

I am moving some files from one folder to another. If the files are already available in the target folder, this code gives list in message box. However i need that information in excel cell for example sheet 1, cell AF5.


来源: stackexchange

问:
Adding user object to scope django-channels
I am trying to add websockets to my app. I use JWT tokens so I have to overide middleware for it.

```<code>@database_sync_to_async
def get_user(token):
    try:
        payload = jwt.decode(token, settings.SECRET_KEY, algorithms=ALGORITHM)
    except:
        return AnonymousUser()
    token_exp = datetime.fromtimestamp(payload['exp'])
    if token_exp < datetime.utcnow():
        return AnonymousUser()

    try:
        response = requests.get(settings.OAUTH_URL, headers={'Authorization': f"Bearer {token}"})
        if response.status_code == 200:
            user = response.json()
            return user
    except:
        error_logger.exception("Server is not available!")

class TokenAuthMiddleware(BaseMiddleware):

    async def __call__(self, scope, receive, send):

        close_old_connections()
        try:
            token_key = (dict((x.split('=') for x in scope['query_string'].decode().split("&")))).get('token', None)
        except ValueError:
            token_key = None

        scope['user'] = await get_user(token_key)
        return await super().__call__(scope, receive, send)

def JwtAuthMiddlewareStack(inner):
    return TokenAuthMiddleware(AuthMiddlewareStack(inner))
</code>```

My service is a microservice so the User model is located in another service. So I have to send a request to the service where the instance model is located. When I try to add "user" dict-like object to scope it returns me an error:

```<code>helpdesk_web            |   File "/usr/local/lib/python3.10/site-packages/channels/auth.py", line 176, in resolve_scope
helpdesk_web            |     scope["user"]._wrapped = await get_user(scope)
helpdesk_web            | AttributeError: 'dict' object has no attribute '_wrapped'
</code>```

If I delete this line scope['user'] = await get_user(token_key) or pass to the function model instance everything works.

Is there a way to add 'users' dict-like object instead of model instance to scope?

ContentLicense: CC BY-SA 4.0

Body:
I am trying to add websockets to my app. I use JWT tokens so I have to overide middleware for it.

@database_sync_to_async
def get_user(token):
    try:
        payload = jwt.decode(token, settings.SECRET_KEY, algorithms=ALGORITHM)
    except:
        return AnonymousUser()
    token_exp = datetime.fromtimestamp(payload['exp'])
    if token_exp < datetime.utcnow():
        return AnonymousUser()

    try:
        response = requests.get(settings.OAUTH_URL, headers={'Authorization': f"Bearer {token}"})
        if response.status_code == 200:
            user = response.json()
            return user
    except:
        error_logger.exception("Server is not available!")

class TokenAuthMiddleware(BaseMiddleware):

    async def __call__(self, scope, receive, send):

        close_old_connections()
        try:
            token_key = (dict((x.split('=') for x in scope['query_string'].decode().split("&")))).get('token', None)
        except ValueError:
            token_key = None

        scope['user'] = await get_user(token_key)
        return await super().__call__(scope, receive, send)

def JwtAuthMiddlewareStack(inner):
    return TokenAuthMiddleware(AuthMiddlewareStack(inner))

My service is a microservice so the User model is located in another service. So I have to send a request to the service where the instance model is located. When I try to add "user" dict-like object to scope it returns me an error:

helpdesk_web            |   File "/usr/local/lib/python3.10/site-packages/channels/auth.py", line 176, in resolve_scope
helpdesk_web            |     scope["user"]._wrapped = await get_user(scope)
helpdesk_web            | AttributeError: 'dict' object has no attribute '_wrapped'

If I delete this line scope['user'] = await get_user(token_key) or pass to the function model instance everything works.

Is there a way to add 'users' dict-like object instead of model instance to scope?


问:
How to add simple custom pytorch-crf layer on top of TokenClassification model using pytorch and Trainer
I followed this link, but its implemented in Keras.

Cannot add CRF layer on top of BERT in keras for NER

Model description

Is it possible to add simple custom pytorch-crf layer on top of TokenClassification model. It will make the model more robust.

```<code>from torchcrf import CRF

model_checkpoint = "dslim/bert-base-NER"
tokenizer = BertTokenizer.from_pretrained(model_checkpoint,add_prefix_space=True)
config = BertConfig.from_pretrained(model_checkpoint, output_hidden_states=True)
bert_model = BertForTokenClassification.from_pretrained(model_checkpoint,id2label=id2label,label2id=label2id,ignore_mismatched_sizes=True)

class BERT_CRF(nn.Module):

    def __init__(self, bert_model, num_labels):
        super(BERT_CRF, self).__init__()
        self.bert = bert_model
        self.dropout = nn.Dropout(0.25)

        self.classifier = nn.Linear(4*768, num_labels)

        self.crf = CRF(num_labels, batch_first = True)

    def forward(self, input_ids, attention_mask,  labels=None, token_type_ids=None):
        outputs = self.bert(input_ids, attention_mask=attention_mask)

        **sequence_output = torch.cat((outputs[1][-1], outputs[1][-2], outputs[1][-3], outputs[1][-4]),-1)**
        sequence_output = self.dropout(sequence_output)

        emission = self.classifier(sequence_output) # [32,256,17]
        labels=labels.reshape(attention_mask.size()[0],attention_mask.size()[1])

        if labels is not None:
            loss = -self.crf(log_soft(emission, 2), labels, mask=attention_mask.type(torch.uint8), reduction='mean')
            prediction = self.crf.decode(emission, mask=attention_mask.type(torch.uint8))
            return [loss, prediction]

        else:
            prediction = self.crf.decode(emission, mask=attention_mask.type(torch.uint8))
            return prediction

</code>```
```<code>args = TrainingArguments(
    "spanbert_crf_ner-pos2",
    # evaluation_strategy="epoch",
    save_strategy="epoch",
    learning_rate=2e-5,
    num_train_epochs=1,
    weight_decay=0.01,
    per_device_train_batch_size=8,
    # per_device_eval_batch_size=32
    fp16=True
    # bf16=True #Ampere GPU
)

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train_data,
    # eval_dataset=train_data,
    # data_collator=data_collator,
    # compute_metrics=compute_metrics,
    tokenizer=tokenizer)
</code>```

I get error on line    **sequence_output = torch.cat((outputs[1][-1], outputs[1][-2], outputs[1][-3], outputs[1][-4]),-1)**

As outputs = self.bert(input_ids, attention_mask=attention_mask) gives the logits for tokenclassification. How can we get hidden states so that I can concate last 4 hidden states. so that I can dooutputs[1][-1]`?

Or is their easier way to implement BERT-CRF model?

Body:
I followed this link, but its implemented in Keras.

Cannot add CRF layer on top of BERT in keras for NER

Model description

Is it possible to add simple custom pytorch-crf layer on top of TokenClassification model. It will make the model more robust.

from torchcrf import CRF

model_checkpoint = "dslim/bert-base-NER"
tokenizer = BertTokenizer.from_pretrained(model_checkpoint,add_prefix_space=True)
config = BertConfig.from_pretrained(model_checkpoint, output_hidden_states=True)
bert_model = BertForTokenClassification.from_pretrained(model_checkpoint,id2label=id2label,label2id=label2id,ignore_mismatched_sizes=True)

class BERT_CRF(nn.Module):

    def __init__(self, bert_model, num_labels):
        super(BERT_CRF, self).__init__()
        self.bert = bert_model
        self.dropout = nn.Dropout(0.25)

        self.classifier = nn.Linear(4*768, num_labels)

        self.crf = CRF(num_labels, batch_first = True)

    def forward(self, input_ids, attention_mask,  labels=None, token_type_ids=None):
        outputs = self.bert(input_ids, attention_mask=attention_mask)

        **sequence_output = torch.cat((outputs[1][-1], outputs[1][-2], outputs[1][-3], outputs[1][-4]),-1)**
        sequence_output = self.dropout(sequence_output)

        emission = self.classifier(sequence_output) # [32,256,17]
        labels=labels.reshape(attention_mask.size()[0],attention_mask.size()[1])

        if labels is not None:
            loss = -self.crf(log_soft(emission, 2), labels, mask=attention_mask.type(torch.uint8), reduction='mean')
            prediction = self.crf.decode(emission, mask=attention_mask.type(torch.uint8))
            return [loss, prediction]

        else:
            prediction = self.crf.decode(emission, mask=attention_mask.type(torch.uint8))
            return prediction

args = TrainingArguments(
    "spanbert_crf_ner-pos2",
    # evaluation_strategy="epoch",
    save_strategy="epoch",
    learning_rate=2e-5,
    num_train_epochs=1,
    weight_decay=0.01,
    per_device_train_batch_size=8,
    # per_device_eval_batch_size=32
    fp16=True
    # bf16=True #Ampere GPU
)

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train_data,
    # eval_dataset=train_data,
    # data_collator=data_collator,
    # compute_metrics=compute_metrics,
    tokenizer=tokenizer)

I get error on line    **sequence_output = torch.cat((outputs[1][-1], outputs[1][-2], outputs[1][-3], outputs[1][-4]),-1)**

As outputs = self.bert(input_ids, attention_mask=attention_mask) gives the logits for tokenclassification. How can we get hidden states so that I can concate last 4 hidden states. so that I can dooutputs[1][-1]`?

Or is their easier way to implement BERT-CRF model?


问:
I can't export any certificate to PKCS 12, Could not read private key / openssl program
I can't export domain signed certificate, with the command:
openssl pkcs12 -export -in domain.crt -inkey domain.key -out domain.pfx
It pops up for me:

```<code>Could not read private key from -inkey file from domain.key
FC210000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:..\crypto\store\store_result.c:151:
FC210000:error:1C800064:Provider routines:ossl_cipher_unpadblock:bad decrypt:..\providers\implementations\ciphers\ciphercommon_block.c:124:
FC210000:error:11800074:PKCS12 routines:PKCS12_pbe_crypt_ex:pkcs12 cipherfinal error:..\crypto\pkcs12\p12_decr.c:86:empty password
</code>```

I have my own certification authority certificate which is valid. I would like to export my regular certificate and sign it with this CA. But I can't do that because I can't export domain signed certificate.
I use OpenSSL on Windows 11 Pro for home users. I know it should be done through the server but I don't want to install it additionally
I simply wanted to export the certificate with the private key to PFX via openssl. I received the error described exactly above:
Could not read private key from -inkey file from domain.key.

I was expecting a successful export and subsequent signing of the certificate (if I had to) by an existing self-signed (it's in the CA trusted certificate store) CA certificate

Body:
I can't export domain signed certificate, with the command:
openssl pkcs12 -export -in domain.crt -inkey domain.key -out domain.pfx
It pops up for me:

Could not read private key from -inkey file from domain.key
FC210000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:..\crypto\store\store_result.c:151:
FC210000:error:1C800064:Provider routines:ossl_cipher_unpadblock:bad decrypt:..\providers\implementations\ciphers\ciphercommon_block.c:124:
FC210000:error:11800074:PKCS12 routines:PKCS12_pbe_crypt_ex:pkcs12 cipherfinal error:..\crypto\pkcs12\p12_decr.c:86:empty password

I have my own certification authority certificate which is valid. I would like to export my regular certificate and sign it with this CA. But I can't do that because I can't export domain signed certificate.
I use OpenSSL on Windows 11 Pro for home users. I know it should be done through the server but I don't want to install it additionally
I simply wanted to export the certificate with the private key to PFX via openssl. I received the error described exactly above:
Could not read private key from -inkey file from domain.key.

I was expecting a successful export and subsequent signing of the certificate (if I had to) by an existing self-signed (it's in the CA trusted certificate store) CA certificate


问:
How to resolve py4jjavaerror?
How to resolve py4jjavaerror

I got this error:

```"name": "Py4JJavaError",
    "message": "An error occurred while calling o1993.collectToPython.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 88.0 failed 1 times, most recent failure: Lost task 4.0 in stage 88.0 (TID 339) (LAPTOP-LT9BMOHO executor driver): org.apache.spark.SparkException: Python worker failed to connect back.\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:189)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)\r\n\tat org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)\r\n\tat org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:164)\r\n\tat org.apache.spark.sql.execution.python.PandasGroupUtils$.executePython(PandasGroupUtils.scala:44)\r\n\tat org.apache.spark.sql.execution.python.FlatMapGroupsInPandasExec.$anonfun$doExecute$1(FlatMapGroupsInPandasExec.scala:94)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:136)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)\r\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\tat java.lang.Thread.run(Thread.java:750)\r\nCaused by: java.net.SocketTimeoutException: Accept timed out\r\n\tat java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)\r\n\tat java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)\r\n\tat java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:535)\r\n\tat java.net.PlainSocketImpl.accept(PlainSocketImpl.java:189)\r\n\tat java.net.ServerSocket.implAccept(ServerSocket.java:545)\r\n\tat java.net.ServerSocket.accept(ServerSocket.java:513)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:176)\r\n\t... 26 more\r\n\nDriver stacktrace:\r\n\tat org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2672)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2608)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2607)\r\n\tat scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)\r\n\tat scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)\r\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)\r\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2607)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1182)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1182)\r\n\tat scala.Option.foreach(Option.scala:407)\r\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1182)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2860)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2802)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2791)\r\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)\r\nCaused by: org.apache.spark.SparkException: Python worker failed to connect back.\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:189)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)\r\n\tat org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)\r\n\tat org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:164)\r\n\tat org.apache.spark.sql.execution.python.PandasGroupUtils$.executePython(PandasGroupUtils.scala:44)\r\n\tat org.apache.spark.sql.execution.python.FlatMapGroupsInPandasExec.$anonfun$doExecute$1(FlatMapGroupsInPandasExec.scala:94)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:136)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)\r\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\tat java.lang.Thread.run(Thread.java:750)\r\nCaused by: java.net.SocketTimeoutException: Accept timed out\r\n\tat java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)\r\n\tat java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)\r\n\tat java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:535)\r\n\tat java.net.PlainSocketImpl.accept(PlainSocketImpl.java:189)\r\n\tat java.net.ServerSocket.implAccept(ServerSocket.java:545)\r\n\tat java.net.ServerSocket.accept(ServerSocket.java:513)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:176)\r\n\t... 26 more\r\n",
    "stack": "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[1;31mPy4JJavaError\u001b[0m                             Traceback (most recent call last)\nCell \u001b[1;32mIn[16], line 22\u001b[0m\n\u001b[0;32m     20\u001b[0m model_df_validation[i]\u001b[39m=\u001b[39mmodel_data\u001b[39m.\u001b[39mfilter((col(\u001b[39m\"\u001b[39m\u001b[39mWEEK_START_DATE\u001b[39m\u001b[39m\"\u001b[39m) \u001b[39m>\u001b[39m\u001b[39m=\u001b[39m data_start_dates[i]) \u001b[39m&\u001b[39m (col(\u001b[39m\"\u001b[39m\u001b[39mWEEK_START_DATE\u001b[39m\u001b[39m\"\u001b[39m) \u001b[39m<\u001b[39m testing_start_dates[i]))\n\u001b[0;32m     21\u001b[0m model_df_validation[i]\u001b[39m=\u001b[39mmodel_df_validation[i]\u001b[39m.\u001b[39mwithColumn(\u001b[39m\"\u001b[39m\u001b[39mTRAIN_VALID\u001b[39m\u001b[39m\"\u001b[39m,when(model_df_validation[i]\u001b[39m.\u001b[39mWEEK_START_DATE\u001b[39m<\u001b[39mvalidation_start_dates[i],\u001b[39m\"\u001b[39m\u001b[39mTRAIN\u001b[39m\u001b[39m\"\u001b[39m)\u001b[39m.\u001b[39motherwise(\u001b[39m\"\u001b[39m\u001b[39mVALID\u001b[39m\u001b[39m\"\u001b[39m))\u001b[39m.\u001b[39mcache()\n\u001b[1;32m---> 22\u001b[0m all_errors[i], best_error_mean[i], best_loss_init[i], best_n_estimators_init[i], best_max_depth_init[i], best_learning_rate_init[i], best_subsample_init[i], best_max_features_init[i] \u001b[39m=\u001b[39m run_validation(model_df_validation[i], model_input_cols, model_output_col, loss_values, n_estimators_values, max_depth_values, learning_rate_values, subsample_values, max_features_values)\n\nCell \u001b[1;32mIn[13], line 82\u001b[0m, in \u001b[0;36mrun_validation\u001b[1;34m(model_df, train_model_inputs, model_output_col, loss_values, n_estimators_values, max_depth_values, learning_rate_values, subsample_values, max_features_values)\u001b[0m\n\u001b[0;32m     76\u001b[0m coeff \u001b[39m=\u001b[39m coeff\u001b[39m.\u001b[39mwithColumn(\n\u001b[0;32m     77\u001b[0m   \u001b[39m\"\u001b[39m\u001b[39mE\u001b[39m\u001b[39m\"\u001b[39m, col(\u001b[39m\"\u001b[39m\u001b[39mVALID_Y_PREDICTED_UNITS\u001b[39m\u001b[39m\"\u001b[39m) \u001b[39m-\u001b[39m col(\u001b[39m\"\u001b[39m\u001b[39mVALID_Y_TRUE_UNITS\u001b[39m\u001b[39m\"\u001b[39m))\u001b[39m.\u001b[39mwithColumn(\n\u001b[0;32m     78\u001b[0m   \u001b[39m\"\u001b[39m\u001b[39mAE\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39mabs\u001b[39m(col(\u001b[39m\"\u001b[39m\u001b[39mE\u001b[39m\u001b[39m\"\u001b[39m)))\u001b[39m.\u001b[39mwithColumn(\n\u001b[0;32m     79\u001b[0m   \u001b[39m\"\u001b[39m\u001b[39mSE\u001b[39m\u001b[39m\"\u001b[39m, col(\u001b[39m\"\u001b[39m\u001b[39mE\u001b[39m\u001b[39m\"\u001b[39m) \u001b[39m*\u001b[39m col(\u001b[39m\"\u001b[39m\u001b[39mE\u001b[39m\u001b[39m\"\u001b[39m))\u001b[39m.\u001b[39mwithColumn(\n\u001b[0;32m     80\u001b[0m   \u001b[39m\"\u001b[39m\u001b[39mAPE\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39m100\u001b[39m\u001b[39m*\u001b[39m\u001b[39mabs\u001b[39m(col(\u001b[39m\"\u001b[39m\u001b[39mVALID_Y_PREDICTED_UNITS\u001b[39m\u001b[39m\"\u001b[39m) \u001b[39m-\u001b[39m col(\u001b[39m\"\u001b[39m\u001b[39mVALID_Y_TRUE_UNITS\u001b[39m\u001b[39m\"\u001b[39m))\u001b[39m/\u001b[39mcol(\u001b[39m\"\u001b[39m\u001b[39mVALID_Y_TRUE_UNITS\u001b[39m\u001b[39m\"\u001b[39m))\n\u001b[0;32m     81\u001b[0m coeff \u001b[39m=\u001b[39m coeff\u001b[39m.\u001b[39mwithColumn(\u001b[39m\"\u001b[39m\u001b[39mused_error\u001b[39m\u001b[39m\"\u001b[39m, when(col(\u001b[39m\"\u001b[39m\u001b[39mVALID_Y_TRUE_UNITS\u001b[39m\u001b[39m\"\u001b[39m)\u001b[39m<\u001b[39m\u001b[39m=\u001b[39m\u001b[39m100\u001b[39m, \u001b[39m2\u001b[39m\u001b[39m*\u001b[39mcol(\u001b[39m\"\u001b[39m\u001b[39mAE\u001b[39m\u001b[39m\"\u001b[39m))\u001b[39m.\u001b[39motherwise(col(\u001b[39m\"\u001b[39m\u001b[39mAPE\u001b[39m\u001b[39m\"\u001b[39m))) \u001b[39m# to do: pass relative importance as parameter\u001b[39;00m\n\u001b[1;32m---> 82\u001b[0m error_mean \u001b[39m=\u001b[39m coeff\u001b[39m.\u001b[39;49mfilter(coeff\u001b[39m.\u001b[39;49mused_error \u001b[39m<\u001b[39;49m \u001b[39m300\u001b[39;49m)\u001b[39m.\u001b[39;49mselect(mean(\u001b[39m'\u001b[39;49m\u001b[39mused_error\u001b[39;49m\u001b[39m'\u001b[39;49m))\u001b[39m.\u001b[39;49mcollect() \u001b[39m#to do: pass threshold as a parameter\u001b[39;00m\n\u001b[0;32m     84\u001b[0m all_errors \u001b[39m=\u001b[39m []\n\u001b[0;32m     85\u001b[0m all_errors\u001b[39m.\u001b[39mappend({\u001b[39m\"\u001b[39m\u001b[39mloss\u001b[39m\u001b[39m\"\u001b[39m:loss, \u001b[39m\"\u001b[39m\u001b[39mn_estimators\u001b[39m\u001b[39m\"\u001b[39m:n_estimators, \u001b[39m\"\u001b[39m\u001b[39mmax_depth\u001b[39m\u001b[39m\"\u001b[39m:max_depth, \u001b[39m\"\u001b[39m\u001b[39mlearning_rate\u001b[39m\u001b[39m\"\u001b[39m:learning_rate, \u001b[39m\"\u001b[39m\u001b[39msubsample\u001b[39m\u001b[39m\"\u001b[39m:subsample, \u001b[39m\"\u001b[39m\u001b[39mmax_features\u001b[39m\u001b[39m\"\u001b[39m:max_features, \u001b[39m\"\u001b[39m\u001b[39merror_mean\u001b[39m\u001b[39m\"\u001b[39m: error_mean})\n\nFile \u001b[1;32mC:\\Spark\\spark-3.3.1-bin-hadoop3\\spark-3.3.1-bin-hadoop3\\spark-3.3.1-bin-hadoop3\\python\\pyspark\\sql\\dataframe.py:817\u001b[0m, in \u001b[0;36mDataFrame.collect\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    807\u001b[0m \u001b[39m\"\"\"Returns all the records as a list of :class:`Row`.\u001b[39;00m\n\u001b[0;32m    808\u001b[0m \n\u001b[0;32m    809\u001b[0m \u001b[39m.. versionadded:: 1.3.0\u001b[39;00m\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    814\u001b[0m \u001b[39m[Row(age=2, name='Alice'), Row(age=5, name='Bob')]\u001b[39;00m\n\u001b[0;32m    815\u001b[0m \u001b[39m\"\"\"\u001b[39;00m\n\u001b[0;32m    816\u001b[0m \u001b[39mwith\u001b[39;00m SCCallSiteSync(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_sc):\n\u001b[1;32m--> 817\u001b[0m     sock_info \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_jdf\u001b[39m.\u001b[39;49mcollectToPython()\n\u001b[0;32m    818\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mlist\u001b[39m(_load_from_socket(sock_info, BatchedSerializer(CPickleSerializer())))\n\nFile \u001b[1;32mC:\\Spark\\spark-3.3.1-bin-hadoop3\\spark-3.3.1-bin-hadoop3\\spark-3.3.1-bin-hadoop3\\python\\lib\\py4j-0.10.9.5-src.zip\\py4j\\java_gateway.py:1321\u001b[0m, in \u001b[0;36mJavaMember.__call__\u001b[1;34m(self, *args)\u001b[0m\n\u001b[0;32m   1315\u001b[0m command \u001b[39m=\u001b[39m proto\u001b[39m.\u001b[39mCALL_COMMAND_NAME \u001b[39m+\u001b[39m\\\n\u001b[0;32m   1316\u001b[0m     \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mcommand_header \u001b[39m+\u001b[39m\\\n\u001b[0;32m   1317\u001b[0m     args_command \u001b[39m+\u001b[39m\\\n\u001b[0;32m   1318\u001b[0m     proto\u001b[39m.\u001b[39mEND_COMMAND_PART\n\u001b[0;32m   1320\u001b[0m answer \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mgateway_client\u001b[39m.\u001b[39msend_command(command)\n\u001b[1;32m-> 1321\u001b[0m return_value \u001b[39m=\u001b[39m get_return_value(\n\u001b[0;32m   1322\u001b[0m     answer, \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mgateway_client, \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mtarget_id, \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mname)\n\u001b[0;32m   1324\u001b[0m \u001b[39mfor\u001b[39;00m temp_arg \u001b[39min\u001b[39;00m temp_args:\n\u001b[0;32m   1325\u001b[0m     temp_arg\u001b[39m.\u001b[39m_detach()\n\nFile \u001b[1;32mC:\\Spark\\spark-3.3.1-bin-hadoop3\\spark-3.3.1-bin-hadoop3\\spark-3.3.1-bin-hadoop3\\python\\pyspark\\sql\\utils.py:190\u001b[0m, in \u001b[0;36mcapture_sql_exception.<locals>.deco\u001b[1;34m(*a, **kw)\u001b[0m\n\u001b[0;32m    188\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mdeco\u001b[39m(\u001b[39m*\u001b[39ma: Any, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkw: Any) \u001b[39m-\u001b[39m\u001b[39m>\u001b[39m Any:\n\u001b[0;32m    189\u001b[0m     \u001b[39mtry\u001b[39;00m:\n\u001b[1;32m--> 190\u001b[0m         \u001b[39mreturn\u001b[39;00m f(\u001b[39m*\u001b[39ma, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkw)\n\u001b[0;32m    191\u001b[0m     \u001b[39mexcept\u001b[39;00m Py4JJavaError \u001b[39mas\u001b[39;00m e:\n\u001b[0;32m    192\u001b[0m         converted \u001b[39m=\u001b[39m convert_exception(e\u001b[39m.\u001b[39mjava_exception)\n\nFile \u001b[1;32mC:\\Spark\\spark-3.3.1-bin-hadoop3\\spark-3.3.1-bin-hadoop3\\spark-3.3.1-bin-hadoop3\\python\\lib\\py4j-0.10.9.5-src.zip\\py4j\\protocol.py:326\u001b[0m, in \u001b[0;36mget_return_value\u001b[1;34m(answer, gateway_client, target_id, name)\u001b[0m\n\u001b[0;32m    324\u001b[0m value \u001b[39m=\u001b[39m OUTPUT_CONVERTER[\u001b[39mtype\u001b[39m](answer[\u001b[39m2\u001b[39m:], gateway_client)\n\u001b[0;32m    325\u001b[0m \u001b[39mif\u001b[39;00m answer[\u001b[39m1\u001b[39m] \u001b[39m==\u001b[39m REFERENCE_TYPE:\n\u001b[1;32m--> 326\u001b[0m     \u001b[39mraise\u001b[39;00m Py4JJavaError(\n\u001b[0;32m    327\u001b[0m         \u001b[39m\"\u001b[39m\u001b[39mAn error occurred while calling \u001b[39m\u001b[39m{0}\u001b[39;00m\u001b[39m{1}\u001b[39;00m\u001b[39m{2}\u001b[39;00m\u001b[39m.\u001b[39m\u001b[39m\\n\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39m\n\u001b[0;32m    328\u001b[0m         \u001b[39mformat\u001b[39m(target_id, \u001b[39m\"\u001b[39m\u001b[39m.\u001b[39m\u001b[39m\"\u001b[39m, name), value)\n\u001b[0;32m    329\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[0;32m    330\u001b[0m     \u001b[39mraise\u001b[39;00m Py4JError(\n\u001b[0;32m    331\u001b[0m         \u001b[39m\"\u001b[39m\u001b[39mAn error occurred while calling \u001b[39m\u001b[39m{0}\u001b[39;00m\u001b[39m{1}\u001b[39;00m\u001b[39m{2}\u001b[39;00m\u001b[39m. Trace:\u001b[39m\u001b[39m\\n\u001b[39;00m\u001b[39m{3}\u001b[39;00m\u001b[39m\\n\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39m\n\u001b[0;32m    332\u001b[0m         \u001b[39mformat\u001b[39m(target_id, \u001b[39m\"\u001b[39m\u001b[39m.\u001b[39m\u001b[39m\"\u001b[39m, name, value))\n\n\u001b[1;31mPy4JJavaError\u001b[0m: An error occurred while calling o1993.collectToPython.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 88.0 failed 1 times, most recent failure: Lost task 4.0 in stage 88.0 (TID 339) (LAPTOP-LT9BMOHO executor driver): org.apache.spark.SparkException: Python worker failed to connect back.\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:189)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)\r\n\tat org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)\r\n\tat org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:164)\r\n\tat org.apache.spark.sql.execution.python.PandasGroupUtils$.executePython(PandasGroupUtils.scala:44)\r\n\tat org.apache.spark.sql.execution.python.FlatMapGroupsInPandasExec.$anonfun$doExecute$1(FlatMapGroupsInPandasExec.scala:94)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:136)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)\r\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\tat java.lang.Thread.run(Thread.java:750)\r\nCaused by: java.net.SocketTimeoutException: Accept timed out\r\n\tat java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)\r\n\tat java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)\r\n\tat java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:535)\r\n\tat java.net.PlainSocketImpl.accept(PlainSocketImpl.java:189)\r\n\tat java.net.ServerSocket.implAccept(ServerSocket.java:545)\r\n\tat java.net.ServerSocket.accept(ServerSocket.java:513)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:176)\r\n\t... 26 more\r\n\nDriver stacktrace:\r\n\tat org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2672)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2608)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2607)\r\n\tat scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)\r\n\tat scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)\r\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)\r\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2607)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1182)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1182)\r\n\tat scala.Option.foreach(Option.scala:407)\r\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1182)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2860)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2802)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2791)\r\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)\r\nCaused by: org.apache.spark.SparkException: Python worker failed to connect back.\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:189)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)\r\n\tat org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)\r\n\tat org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:164)\r\n\tat org.apache.spark.sql.execution.python.PandasGroupUtils$.executePython(PandasGroupUtils.scala:44)\r\n\tat org.apache.spark.sql.execution.python.FlatMapGroupsInPandasExec.$anonfun$doExecute$1(FlatMapGroupsInPandasExec.scala:94)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:136)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)\r\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\tat java.lang.Thread.run(Thread.java:750)\r\nCaused by: java.net.SocketTimeoutException: Accept timed out\r\n\tat java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)\r\n\tat java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)\r\n\tat java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:535)\r\n\tat java.net.PlainSocketImpl.accept(PlainSocketImpl.java:189)\r\n\tat java.net.ServerSocket.implAccept(ServerSocket.java:545)\r\n\tat java.net.ServerSocket.accept(ServerSocket.java:513)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:176)\r\n\t... 26 more\r\n"
}
```

Title: How to resolve py4jjavaerror?

Body:
How to resolve py4jjavaerror

I got this error:

"name": "Py4JJavaError",
    "message": "An error occurred while calling o1993.collectToPython.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 88.0 failed 1 times, most recent failure: Lost task 4.0 in stage 88.0 (TID 339) (LAPTOP-LT9BMOHO executor driver): org.apache.spark.SparkException: Python worker failed to connect back.\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:189)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)\r\n\tat org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)\r\n\tat org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:164)\r\n\tat org.apache.spark.sql.execution.python.PandasGroupUtils$.executePython(PandasGroupUtils.scala:44)\r\n\tat org.apache.spark.sql.execution.python.FlatMapGroupsInPandasExec.$anonfun$doExecute$1(FlatMapGroupsInPandasExec.scala:94)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:136)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)\r\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\tat java.lang.Thread.run(Thread.java:750)\r\nCaused by: java.net.SocketTimeoutException: Accept timed out\r\n\tat java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)\r\n\tat java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)\r\n\tat java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:535)\r\n\tat java.net.PlainSocketImpl.accept(PlainSocketImpl.java:189)\r\n\tat java.net.ServerSocket.implAccept(ServerSocket.java:545)\r\n\tat java.net.ServerSocket.accept(ServerSocket.java:513)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:176)\r\n\t... 26 more\r\n\nDriver stacktrace:\r\n\tat org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2672)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2608)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2607)\r\n\tat scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)\r\n\tat scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)\r\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)\r\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2607)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1182)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1182)\r\n\tat scala.Option.foreach(Option.scala:407)\r\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1182)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2860)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2802)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2791)\r\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)\r\nCaused by: org.apache.spark.SparkException: Python worker failed to connect back.\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:189)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)\r\n\tat org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)\r\n\tat org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:164)\r\n\tat org.apache.spark.sql.execution.python.PandasGroupUtils$.executePython(PandasGroupUtils.scala:44)\r\n\tat org.apache.spark.sql.execution.python.FlatMapGroupsInPandasExec.$anonfun$doExecute$1(FlatMapGroupsInPandasExec.scala:94)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:136)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)\r\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\tat java.lang.Thread.run(Thread.java:750)\r\nCaused by: java.net.SocketTimeoutException: Accept timed out\r\n\tat java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)\r\n\tat java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)\r\n\tat java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:535)\r\n\tat java.net.PlainSocketImpl.accept(PlainSocketImpl.java:189)\r\n\tat java.net.ServerSocket.implAccept(ServerSocket.java:545)\r\n\tat java.net.ServerSocket.accept(ServerSocket.java:513)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:176)\r\n\t... 26 more\r\n",
    "stack": "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[1;31mPy4JJavaError\u001b[0m                             Traceback (most recent call last)\nCell \u001b[1;32mIn[16], line 22\u001b[0m\n\u001b[0;32m     20\u001b[0m model_df_validation[i]\u001b[39m=\u001b[39mmodel_data\u001b[39m.\u001b[39mfilter((col(\u001b[39m\"\u001b[39m\u001b[39mWEEK_START_DATE\u001b[39m\u001b[39m\"\u001b[39m) \u001b[39m>\u001b[39m\u001b[39m=\u001b[39m data_start_dates[i]) \u001b[39m&\u001b[39m (col(\u001b[39m\"\u001b[39m\u001b[39mWEEK_START_DATE\u001b[39m\u001b[39m\"\u001b[39m) \u001b[39m<\u001b[39m testing_start_dates[i]))\n\u001b[0;32m     21\u001b[0m model_df_validation[i]\u001b[39m=\u001b[39mmodel_df_validation[i]\u001b[39m.\u001b[39mwithColumn(\u001b[39m\"\u001b[39m\u001b[39mTRAIN_VALID\u001b[39m\u001b[39m\"\u001b[39m,when(model_df_validation[i]\u001b[39m.\u001b[39mWEEK_START_DATE\u001b[39m<\u001b[39mvalidation_start_dates[i],\u001b[39m\"\u001b[39m\u001b[39mTRAIN\u001b[39m\u001b[39m\"\u001b[39m)\u001b[39m.\u001b[39motherwise(\u001b[39m\"\u001b[39m\u001b[39mVALID\u001b[39m\u001b[39m\"\u001b[39m))\u001b[39m.\u001b[39mcache()\n\u001b[1;32m---> 22\u001b[0m all_errors[i], best_error_mean[i], best_loss_init[i], best_n_estimators_init[i], best_max_depth_init[i], best_learning_rate_init[i], best_subsample_init[i], best_max_features_init[i] \u001b[39m=\u001b[39m run_validation(model_df_validation[i], model_input_cols, model_output_col, loss_values, n_estimators_values, max_depth_values, learning_rate_values, subsample_values, max_features_values)\n\nCell \u001b[1;32mIn[13], line 82\u001b[0m, in \u001b[0;36mrun_validation\u001b[1;34m(model_df, train_model_inputs, model_output_col, loss_values, n_estimators_values, max_depth_values, learning_rate_values, subsample_values, max_features_values)\u001b[0m\n\u001b[0;32m     76\u001b[0m coeff \u001b[39m=\u001b[39m coeff\u001b[39m.\u001b[39mwithColumn(\n\u001b[0;32m     77\u001b[0m   \u001b[39m\"\u001b[39m\u001b[39mE\u001b[39m\u001b[39m\"\u001b[39m, col(\u001b[39m\"\u001b[39m\u001b[39mVALID_Y_PREDICTED_UNITS\u001b[39m\u001b[39m\"\u001b[39m) \u001b[39m-\u001b[39m col(\u001b[39m\"\u001b[39m\u001b[39mVALID_Y_TRUE_UNITS\u001b[39m\u001b[39m\"\u001b[39m))\u001b[39m.\u001b[39mwithColumn(\n\u001b[0;32m     78\u001b[0m   \u001b[39m\"\u001b[39m\u001b[39mAE\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39mabs\u001b[39m(col(\u001b[39m\"\u001b[39m\u001b[39mE\u001b[39m\u001b[39m\"\u001b[39m)))\u001b[39m.\u001b[39mwithColumn(\n\u001b[0;32m     79\u001b[0m   \u001b[39m\"\u001b[39m\u001b[39mSE\u001b[39m\u001b[39m\"\u001b[39m, col(\u001b[39m\"\u001b[39m\u001b[39mE\u001b[39m\u001b[39m\"\u001b[39m) \u001b[39m*\u001b[39m col(\u001b[39m\"\u001b[39m\u001b[39mE\u001b[39m\u001b[39m\"\u001b[39m))\u001b[39m.\u001b[39mwithColumn(\n\u001b[0;32m     80\u001b[0m   \u001b[39m\"\u001b[39m\u001b[39mAPE\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39m100\u001b[39m\u001b[39m*\u001b[39m\u001b[39mabs\u001b[39m(col(\u001b[39m\"\u001b[39m\u001b[39mVALID_Y_PREDICTED_UNITS\u001b[39m\u001b[39m\"\u001b[39m) \u001b[39m-\u001b[39m col(\u001b[39m\"\u001b[39m\u001b[39mVALID_Y_TRUE_UNITS\u001b[39m\u001b[39m\"\u001b[39m))\u001b[39m/\u001b[39mcol(\u001b[39m\"\u001b[39m\u001b[39mVALID_Y_TRUE_UNITS\u001b[39m\u001b[39m\"\u001b[39m))\n\u001b[0;32m     81\u001b[0m coeff \u001b[39m=\u001b[39m coeff\u001b[39m.\u001b[39mwithColumn(\u001b[39m\"\u001b[39m\u001b[39mused_error\u001b[39m\u001b[39m\"\u001b[39m, when(col(\u001b[39m\"\u001b[39m\u001b[39mVALID_Y_TRUE_UNITS\u001b[39m\u001b[39m\"\u001b[39m)\u001b[39m<\u001b[39m\u001b[39m=\u001b[39m\u001b[39m100\u001b[39m, \u001b[39m2\u001b[39m\u001b[39m*\u001b[39mcol(\u001b[39m\"\u001b[39m\u001b[39mAE\u001b[39m\u001b[39m\"\u001b[39m))\u001b[39m.\u001b[39motherwise(col(\u001b[39m\"\u001b[39m\u001b[39mAPE\u001b[39m\u001b[39m\"\u001b[39m))) \u001b[39m# to do: pass relative importance as parameter\u001b[39;00m\n\u001b[1;32m---> 82\u001b[0m error_mean \u001b[39m=\u001b[39m coeff\u001b[39m.\u001b[39;49mfilter(coeff\u001b[39m.\u001b[39;49mused_error \u001b[39m<\u001b[39;49m \u001b[39m300\u001b[39;49m)\u001b[39m.\u001b[39;49mselect(mean(\u001b[39m'\u001b[39;49m\u001b[39mused_error\u001b[39;49m\u001b[39m'\u001b[39;49m))\u001b[39m.\u001b[39;49mcollect() \u001b[39m#to do: pass threshold as a parameter\u001b[39;00m\n\u001b[0;32m     84\u001b[0m all_errors \u001b[39m=\u001b[39m []\n\u001b[0;32m     85\u001b[0m all_errors\u001b[39m.\u001b[39mappend({\u001b[39m\"\u001b[39m\u001b[39mloss\u001b[39m\u001b[39m\"\u001b[39m:loss, \u001b[39m\"\u001b[39m\u001b[39mn_estimators\u001b[39m\u001b[39m\"\u001b[39m:n_estimators, \u001b[39m\"\u001b[39m\u001b[39mmax_depth\u001b[39m\u001b[39m\"\u001b[39m:max_depth, \u001b[39m\"\u001b[39m\u001b[39mlearning_rate\u001b[39m\u001b[39m\"\u001b[39m:learning_rate, \u001b[39m\"\u001b[39m\u001b[39msubsample\u001b[39m\u001b[39m\"\u001b[39m:subsample, \u001b[39m\"\u001b[39m\u001b[39mmax_features\u001b[39m\u001b[39m\"\u001b[39m:max_features, \u001b[39m\"\u001b[39m\u001b[39merror_mean\u001b[39m\u001b[39m\"\u001b[39m: error_mean})\n\nFile \u001b[1;32mC:\\Spark\\spark-3.3.1-bin-hadoop3\\spark-3.3.1-bin-hadoop3\\spark-3.3.1-bin-hadoop3\\python\\pyspark\\sql\\dataframe.py:817\u001b[0m, in \u001b[0;36mDataFrame.collect\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    807\u001b[0m \u001b[39m\"\"\"Returns all the records as a list of :class:`Row`.\u001b[39;00m\n\u001b[0;32m    808\u001b[0m \n\u001b[0;32m    809\u001b[0m \u001b[39m.. versionadded:: 1.3.0\u001b[39;00m\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    814\u001b[0m \u001b[39m[Row(age=2, name='Alice'), Row(age=5, name='Bob')]\u001b[39;00m\n\u001b[0;32m    815\u001b[0m \u001b[39m\"\"\"\u001b[39;00m\n\u001b[0;32m    816\u001b[0m \u001b[39mwith\u001b[39;00m SCCallSiteSync(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_sc):\n\u001b[1;32m--> 817\u001b[0m     sock_info \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_jdf\u001b[39m.\u001b[39;49mcollectToPython()\n\u001b[0;32m    818\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mlist\u001b[39m(_load_from_socket(sock_info, BatchedSerializer(CPickleSerializer())))\n\nFile \u001b[1;32mC:\\Spark\\spark-3.3.1-bin-hadoop3\\spark-3.3.1-bin-hadoop3\\spark-3.3.1-bin-hadoop3\\python\\lib\\py4j-0.10.9.5-src.zip\\py4j\\java_gateway.py:1321\u001b[0m, in \u001b[0;36mJavaMember.__call__\u001b[1;34m(self, *args)\u001b[0m\n\u001b[0;32m   1315\u001b[0m command \u001b[39m=\u001b[39m proto\u001b[39m.\u001b[39mCALL_COMMAND_NAME \u001b[39m+\u001b[39m\\\n\u001b[0;32m   1316\u001b[0m     \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mcommand_header \u001b[39m+\u001b[39m\\\n\u001b[0;32m   1317\u001b[0m     args_command \u001b[39m+\u001b[39m\\\n\u001b[0;32m   1318\u001b[0m     proto\u001b[39m.\u001b[39mEND_COMMAND_PART\n\u001b[0;32m   1320\u001b[0m answer \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mgateway_client\u001b[39m.\u001b[39msend_command(command)\n\u001b[1;32m-> 1321\u001b[0m return_value \u001b[39m=\u001b[39m get_return_value(\n\u001b[0;32m   1322\u001b[0m     answer, \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mgateway_client, \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mtarget_id, \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mname)\n\u001b[0;32m   1324\u001b[0m \u001b[39mfor\u001b[39;00m temp_arg \u001b[39min\u001b[39;00m temp_args:\n\u001b[0;32m   1325\u001b[0m     temp_arg\u001b[39m.\u001b[39m_detach()\n\nFile \u001b[1;32mC:\\Spark\\spark-3.3.1-bin-hadoop3\\spark-3.3.1-bin-hadoop3\\spark-3.3.1-bin-hadoop3\\python\\pyspark\\sql\\utils.py:190\u001b[0m, in \u001b[0;36mcapture_sql_exception.<locals>.deco\u001b[1;34m(*a, **kw)\u001b[0m\n\u001b[0;32m    188\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mdeco\u001b[39m(\u001b[39m*\u001b[39ma: Any, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkw: Any) \u001b[39m-\u001b[39m\u001b[39m>\u001b[39m Any:\n\u001b[0;32m    189\u001b[0m     \u001b[39mtry\u001b[39;00m:\n\u001b[1;32m--> 190\u001b[0m         \u001b[39mreturn\u001b[39;00m f(\u001b[39m*\u001b[39ma, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkw)\n\u001b[0;32m    191\u001b[0m     \u001b[39mexcept\u001b[39;00m Py4JJavaError \u001b[39mas\u001b[39;00m e:\n\u001b[0;32m    192\u001b[0m         converted \u001b[39m=\u001b[39m convert_exception(e\u001b[39m.\u001b[39mjava_exception)\n\nFile \u001b[1;32mC:\\Spark\\spark-3.3.1-bin-hadoop3\\spark-3.3.1-bin-hadoop3\\spark-3.3.1-bin-hadoop3\\python\\lib\\py4j-0.10.9.5-src.zip\\py4j\\protocol.py:326\u001b[0m, in \u001b[0;36mget_return_value\u001b[1;34m(answer, gateway_client, target_id, name)\u001b[0m\n\u001b[0;32m    324\u001b[0m value \u001b[39m=\u001b[39m OUTPUT_CONVERTER[\u001b[39mtype\u001b[39m](answer[\u001b[39m2\u001b[39m:], gateway_client)\n\u001b[0;32m    325\u001b[0m \u001b[39mif\u001b[39;00m answer[\u001b[39m1\u001b[39m] \u001b[39m==\u001b[39m REFERENCE_TYPE:\n\u001b[1;32m--> 326\u001b[0m     \u001b[39mraise\u001b[39;00m Py4JJavaError(\n\u001b[0;32m    327\u001b[0m         \u001b[39m\"\u001b[39m\u001b[39mAn error occurred while calling \u001b[39m\u001b[39m{0}\u001b[39;00m\u001b[39m{1}\u001b[39;00m\u001b[39m{2}\u001b[39;00m\u001b[39m.\u001b[39m\u001b[39m\\n\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39m\n\u001b[0;32m    328\u001b[0m         \u001b[39mformat\u001b[39m(target_id, \u001b[39m\"\u001b[39m\u001b[39m.\u001b[39m\u001b[39m\"\u001b[39m, name), value)\n\u001b[0;32m    329\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[0;32m    330\u001b[0m     \u001b[39mraise\u001b[39;00m Py4JError(\n\u001b[0;32m    331\u001b[0m         \u001b[39m\"\u001b[39m\u001b[39mAn error occurred while calling \u001b[39m\u001b[39m{0}\u001b[39;00m\u001b[39m{1}\u001b[39;00m\u001b[39m{2}\u001b[39;00m\u001b[39m. Trace:\u001b[39m\u001b[39m\\n\u001b[39;00m\u001b[39m{3}\u001b[39;00m\u001b[39m\\n\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39m\n\u001b[0;32m    332\u001b[0m         \u001b[39mformat\u001b[39m(target_id, \u001b[39m\"\u001b[39m\u001b[39m.\u001b[39m\u001b[39m\"\u001b[39m, name, value))\n\n\u001b[1;31mPy4JJavaError\u001b[0m: An error occurred while calling o1993.collectToPython.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 88.0 failed 1 times, most recent failure: Lost task 4.0 in stage 88.0 (TID 339) (LAPTOP-LT9BMOHO executor driver): org.apache.spark.SparkException: Python worker failed to connect back.\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:189)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)\r\n\tat org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)\r\n\tat org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:164)\r\n\tat org.apache.spark.sql.execution.python.PandasGroupUtils$.executePython(PandasGroupUtils.scala:44)\r\n\tat org.apache.spark.sql.execution.python.FlatMapGroupsInPandasExec.$anonfun$doExecute$1(FlatMapGroupsInPandasExec.scala:94)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:136)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)\r\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\tat java.lang.Thread.run(Thread.java:750)\r\nCaused by: java.net.SocketTimeoutException: Accept timed out\r\n\tat java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)\r\n\tat java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)\r\n\tat java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:535)\r\n\tat java.net.PlainSocketImpl.accept(PlainSocketImpl.java:189)\r\n\tat java.net.ServerSocket.implAccept(ServerSocket.java:545)\r\n\tat java.net.ServerSocket.accept(ServerSocket.java:513)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:176)\r\n\t... 26 more\r\n\nDriver stacktrace:\r\n\tat org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2672)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2608)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2607)\r\n\tat scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)\r\n\tat scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)\r\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)\r\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2607)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1182)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1182)\r\n\tat scala.Option.foreach(Option.scala:407)\r\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1182)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2860)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2802)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2791)\r\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)\r\nCaused by: org.apache.spark.SparkException: Python worker failed to connect back.\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:189)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)\r\n\tat org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)\r\n\tat org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:164)\r\n\tat org.apache.spark.sql.execution.python.PandasGroupUtils$.executePython(PandasGroupUtils.scala:44)\r\n\tat org.apache.spark.sql.execution.python.FlatMapGroupsInPandasExec.$anonfun$doExecute$1(FlatMapGroupsInPandasExec.scala:94)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:329)\r\n\tat org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:136)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)\r\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\tat java.lang.Thread.run(Thread.java:750)\r\nCaused by: java.net.SocketTimeoutException: Accept timed out\r\n\tat java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)\r\n\tat java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)\r\n\tat java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:535)\r\n\tat java.net.PlainSocketImpl.accept(PlainSocketImpl.java:189)\r\n\tat java.net.ServerSocket.implAccept(ServerSocket.java:545)\r\n\tat java.net.ServerSocket.accept(ServerSocket.java:513)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:176)\r\n\t... 26 more\r\n"
}


问:
After installing shub in python, none of the shub commands work
I'm trying to use scrapy cloud and deploy by code via shub.  I'm using Anaconda and Python version 3.8.8 on Windows.

In the command line, I install shub by entering

```<code>pip install shub
</code>```

This seems to work and it installs (did it a few times and now it just says all the requirements are already satisfied.)

However, when I type shub, I just get a response that,

```<code>'shub' is not recognized as an internal or external command.
</code>```

Any ideas?

Body:
I'm trying to use scrapy cloud and deploy by code via shub.  I'm using Anaconda and Python version 3.8.8 on Windows.

In the command line, I install shub by entering

pip install shub

This seems to work and it installs (did it a few times and now it just says all the requirements are already satisfied.)

However, when I type shub, I just get a response that,

'shub' is not recognized as an internal or external command.

Any ideas?


问:
Export SQL Database from GCP in Java Spring Boot
I am trying to export the database from my GCP to my local drive but I don't know how to achieve this. I am using Java Spring Boot for my backend. So, what would happen is that the user will be using Angular frontend and will send the request to backend to download the sql file. Then the Spring boot will communicate with GCP to export the sql file and then send to the frontend. Is there any api endpoint or library that can help me in doing this?</p>


问:
Linux: Monitor stack size (per thread) and heap size
From: monitor stack size

I learned how get stack size of main thread.
Is there a way to monitor the stack size for each thread ?

Thank you,
Zvika


问:
How to get specific hashtagged posts from facebook and linkedin to webpage in wordpress?
I want to create socialwall.

I want to get public posts from facebook and linkedin having specific hashtag on my webpage.

For example:

If some post have caption as "#technology",

Then I want that post on my webpage.

Is someone help me for this?

I have search graph api for it.

but I am not getting graph API for Facebook and LinkedIn.

Its showing for Instagram only.


问:
@react-google-maps/api flickering issue with react 18
I'm using Next.js version 12.3.1 and React.js 18.2.0. When I select point A and point B the direction service and direction renderer gets called and the map prints the direction with the markers. But when I change points A and B the map prints a new direction but the very first direction/line stays on the map.
How do I fix the issue?
Here is the Map functions file:

```<code>import React, { useCallback, useState } from "react";
import CustomGoogleMap from "@components/core/CustomGoogleMap";
import { Box } from "@mui/material";
import { anyObjectType } from "@services/types";
import MapAPI from "@services/api/map";
import { CreateBookingActions } from "@services/constants";
import { useCreateBookingContext } from "@contexts/CreateBookingContext";

const BookingMap: React.FC<BookingMapProps> = ({
  memoizedloadingPoint,
  memoizedunloadingPoint,
  memoizedwaypointsList,
  setToast,
  setLoadingPoint,
  setUnLoadingPoint,
}) => {
  const cbCtx = useCreateBookingContext();
  const { state, dispatch } = cbCtx;
  const { waypointsList } = state;
  const [call, setCall] = useState(false);
  const handleLoadingPointDragLocation = useCallback(
    async (lat: number, lng: number) => {
      if (lat && lng) {
        const query = {
          latlng: `${lat},${lng}`,
        };
        const { response, error }: any = await MapAPI.onDrag(query);
        if (response) {
          setLoadingPoint({
            ...response.data.data,
          });
        } else {
          setToast("Somthing went wrong", "error");
        }
      }
    },
    []
  );

  const handleUnLoadingPointDragLocation = useCallback(
    async (lat: number, lng: number) => {
      if (lat && lng) {
        const query = {
          latlng: `${lat},${lng}`,
        };
        const { response, error }: any = await MapAPI.onDrag(query);
        if (response) {
          setUnLoadingPoint({
            ...response.data.data,
          });
        } else {
          setToast("Somthing went wrong", "error");
        }
      }
    },
    []
  );

  const setWaypointsList = (val: any[]) => {
    dispatch({
      type: CreateBookingActions.SET_WAYPOINT_LIST,
      payload: val,
    });
  };

  const handleStopageDragLocation = useCallback(
    async (lat: number, lng: number, index: number) => {
      if (lat && lng) {
        const query = {
          latlng: `${lat},${lng}`,
        };
        const { response, error }: any = await MapAPI.onDrag(query);
        if (response) {
          const updatedWaypoints = JSON.parse(JSON.stringify(waypointsList));
          updatedWaypoints.splice(index, 1, { ...response.data.data });
          setWaypointsList([...updatedWaypoints]);
          setCall(true);
        } else {
          setToast("Somthing went wrong", "error");
        }
      }
    },
    []
  );

  return (
    <Box sx={{ height: "420px" }}>
      <CustomGoogleMap
        call={call}
        setCall={setCall}
        loadingP={memoizedloadingPoint}
        unloadingP={memoizedunloadingPoint}
        waypointsList2={memoizedwaypointsList}
        handleLoadingPointDragLocation={handleLoadingPointDragLocation}
        handleUnLoadingPointDragLocation={handleUnLoadingPointDragLocation}
        handleStopageDragLocation={handleStopageDragLocation}
      />
    </Box>
  );
};

export default BookingMap;

interface BookingMapProps {
  memoizedloadingPoint: anyObjectType;
  memoizedunloadingPoint: anyObjectType;
  memoizedwaypointsList: any[];
  setToast: (a: string, b: string) => void;
  setLoadingPoint: (a: anyObjectType) => void;
  setUnLoadingPoint: (a: anyObjectType) => void;
}
</code>```

Here is the google map file:

```<code>import React, { useEffect, useState, memo } from "react";
import {
  GoogleMap,
  MarkerF,
  DirectionsRenderer,
  DirectionsService,
  InfoWindow,
  useLoadScript,
  LoadScriptNext,
} from "@react-google-maps/api";
import LoadingMarker from "../SvgIcons/loading-marker.svg";
import UnLoadingMarker from "../SvgIcons/unloading-marker.svg";
import StoppageIcon from "../SvgIcons/stopageIcon.svg";
import { CircularProgress } from "@mui/material";
import { Box } from "@mui/system";
import { isEmptyArray } from "@services/utils";
import { anyObjectType, Dispatcher } from "@services/types";

const containerStyle = {
  width: "100%",
  height: "100%",
};

const center = {
  lat: 23.810331,
  lng: 90.412521,
};

interface CustomLatLong {
  latitude: number;
  longitude: number;
  textEn: string;
}

type CustomDirectionsWaypoint = google.maps.DirectionsWaypoint & CustomLatLong;

interface CustomGoogleMapProps {
  call: Boolean;
  setCall: Dispatcher<boolean>;
  handleStopageDragLocation?: (a: number, b: number, c: number) => void;
  handleLoadingPointDragLocation?: (a: number, b: number) => void;
  handleUnLoadingPointDragLocation?: (a: number, b: number) => void;
  waypointsList2: CustomDirectionsWaypoint[];
  loadingP: anyObjectType;
  unloadingP: anyObjectType;
  isAddressModal?: Boolean;
}

const CustomGoogleMap: React.FC<CustomGoogleMapProps> = (props) => {
  const {
    call,
    setCall,
    loadingP,
    unloadingP,
    waypointsList2,
    handleLoadingPointDragLocation,
    handleUnLoadingPointDragLocation,
    handleStopageDragLocation,
    isAddressModal = false,
  } = props;

  // const { isLoaded, loadError } = useLoadScript({
  //   googleMapsApiKey: process.env.NEXT_PUBLIC_GOOGLE_MAPS_API_KEY!,
  // });

  const [infoBox, setInfoWindow] = useState({
    position: {
      lat: 0,
      lng: 0,
    },
    info: "",
  });

  const [directions, setDirections] = useState(null);
  console.log("point:", loadingP, unloadingP);

  useEffect(() => {
    if (loadingP.latitude && unloadingP.latitude) {
      setCall(true);
    }
  }, [loadingP.latitude, unloadingP.latitude]);

  const directionsCallback = (response: any) => {
    if (response !== null) {
      console.log("directionsCallback called");
      if (response.status === "OK") {
        setCall(false);
        setDirections(response);
      }
    }
  };

  const getDirectionRender = () => {
    return (
      <DirectionsService
        options={{
          destination: {
            lat: unloadingP.latitude,
            lng: unloadingP.longitude,
          },
          origin: {
            lat: loadingP.latitude,
            lng: loadingP.longitude,
          },
          travelMode: google.maps.TravelMode.DRIVING,
          waypoints: waypointsList2.map(({ latitude, longitude }) => {
            return {
              location: {
                lat: latitude,
                lng: longitude,
              }.toString(),
              stopover: false,
            };
          }),
        }}
        callback={directionsCallback}
      />
    );
  };

  // if (loadError) {
  //   return <div>Map cannot be loaded right now, sorry.</div>;
  // }

  const renderMap = () => {
    return (
      <LoadScriptNext
        googleMapsApiKey={process.env.NEXT_PUBLIC_GOOGLE_MAPS_API_KEY!}
      >
        <GoogleMap
          mapContainerStyle={containerStyle}
          center={
            loadingP.latitude
              ? {
                  lat: loadingP.latitude,
                  lng: loadingP.longitude,
                }
              : center
          }
          zoom={10}
          onClick={({ latLng }) => {
            isAddressModal &&
              handleLoadingPointDragLocation &&
              handleLoadingPointDragLocation(latLng!.lat(), latLng!.lng());
          }}
        >
          {loadingP.latitude &&
            unloadingP.latitude &&
            call &&
            getDirectionRender()}
          {directions && (
            <DirectionsRenderer
              options={{
                directions: directions,
                suppressMarkers: true,
              }}
            />
          )}

          {infoBox.position.lat && (
            <InfoWindow
              // options={infoBoxOptions}
              position={infoBox.position}
              onCloseClick={() =>
                setInfoWindow({
                  position: {
                    lat: 0,
                    lng: 0,
                  },
                  info: "",
                })
              }
            >
              <p> {infoBox.info}</p>
            </InfoWindow>
          )}

          <MarkerF
            key={loadingP.latitude}
            draggable
            onDragEnd={({ latLng }) => {
              handleLoadingPointDragLocation &&
                handleLoadingPointDragLocation(latLng!.lat(), latLng!.lng());
            }}
            position={{ lat: loadingP.latitude, lng: loadingP.longitude }}
            icon={LoadingMarker}
            onClick={() =>
              setInfoWindow({
                ...infoBox,
                position: {
                  lat: loadingP.latitude,
                  lng: loadingP.longitude,
                },
                info: loadingP.textEn,
              })
            }
          />
          <MarkerF
            key={unloadingP.latitude}
            draggable
            onDragEnd={({ latLng }) => {
              handleUnLoadingPointDragLocation &&
                handleUnLoadingPointDragLocation(latLng!.lat(), latLng!.lng());
            }}
            position={{
              lat: unloadingP.latitude,
              lng: unloadingP.longitude,
            }}
            icon={UnLoadingMarker}
            onClick={() =>
              setInfoWindow({
                ...infoBox,
                position: {
                  lat: unloadingP.latitude,
                  lng: unloadingP.longitude,
                },
                info: unloadingP.textEn,
              })
            }
          />
          {!isEmptyArray(waypointsList2) &&
            waypointsList2.map((item, index) => {
              return (
                <MarkerF
                  key={index}
                  draggable
                  visible
                  onDragEnd={({ latLng }) => {
                    handleStopageDragLocation &&
                      handleStopageDragLocation(
                        latLng!.lat(),
                        latLng!.lng(),
                        index
                      );
                  }}
                  position={{
                    lat: item.latitude,
                    lng: item.longitude,
                  }}
                  icon={StoppageIcon}
                  onClick={() =>
                    setInfoWindow({
                      ...infoBox,
                      position: {
                        lat: item.latitude,
                        lng: item.longitude,
                      },
                      info: item.textEn,
                    })
                  }
                />
              );
            })}
        </GoogleMap>
      </LoadScriptNext>
    );
  };
  return renderMap();
};
</code>```

export default memo(CustomGoogleMap);

Body:
I'm using Next.js version 12.3.1 and React.js 18.2.0. When I select point A and point B the direction service and direction renderer gets called and the map prints the direction with the markers. But when I change points A and B the map prints a new direction but the very first direction/line stays on the map.
How do I fix the issue?
Here is the Map functions file:

import React, { useCallback, useState } from "react";
import CustomGoogleMap from "@components/core/CustomGoogleMap";
import { Box } from "@mui/material";
import { anyObjectType } from "@services/types";
import MapAPI from "@services/api/map";
import { CreateBookingActions } from "@services/constants";
import { useCreateBookingContext } from "@contexts/CreateBookingContext";

const BookingMap: React.FC<BookingMapProps> = ({
  memoizedloadingPoint,
  memoizedunloadingPoint,
  memoizedwaypointsList,
  setToast,
  setLoadingPoint,
  setUnLoadingPoint,
}) => {
  const cbCtx = useCreateBookingContext();
  const { state, dispatch } = cbCtx;
  const { waypointsList } = state;
  const [call, setCall] = useState(false);
  const handleLoadingPointDragLocation = useCallback(
    async (lat: number, lng: number) => {
      if (lat && lng) {
        const query = {
          latlng: `${lat},${lng}`,
        };
        const { response, error }: any = await MapAPI.onDrag(query);
        if (response) {
          setLoadingPoint({
            ...response.data.data,
          });
        } else {
          setToast("Somthing went wrong", "error");
        }
      }
    },
    []
  );

  const handleUnLoadingPointDragLocation = useCallback(
    async (lat: number, lng: number) => {
      if (lat && lng) {
        const query = {
          latlng: `${lat},${lng}`,
        };
        const { response, error }: any = await MapAPI.onDrag(query);
        if (response) {
          setUnLoadingPoint({
            ...response.data.data,
          });
        } else {
          setToast("Somthing went wrong", "error");
        }
      }
    },
    []
  );

  const setWaypointsList = (val: any[]) => {
    dispatch({
      type: CreateBookingActions.SET_WAYPOINT_LIST,
      payload: val,
    });
  };

  const handleStopageDragLocation = useCallback(
    async (lat: number, lng: number, index: number) => {
      if (lat && lng) {
        const query = {
          latlng: `${lat},${lng}`,
        };
        const { response, error }: any = await MapAPI.onDrag(query);
        if (response) {
          const updatedWaypoints = JSON.parse(JSON.stringify(waypointsList));
          updatedWaypoints.splice(index, 1, { ...response.data.data });
          setWaypointsList([...updatedWaypoints]);
          setCall(true);
        } else {
          setToast("Somthing went wrong", "error");
        }
      }
    },
    []
  );

  return (
    <Box sx={{ height: "420px" }}>
      <CustomGoogleMap
        call={call}
        setCall={setCall}
        loadingP={memoizedloadingPoint}
        unloadingP={memoizedunloadingPoint}
        waypointsList2={memoizedwaypointsList}
        handleLoadingPointDragLocation={handleLoadingPointDragLocation}
        handleUnLoadingPointDragLocation={handleUnLoadingPointDragLocation}
        handleStopageDragLocation={handleStopageDragLocation}
      />
    </Box>
  );
};

export default BookingMap;

interface BookingMapProps {
  memoizedloadingPoint: anyObjectType;
  memoizedunloadingPoint: anyObjectType;
  memoizedwaypointsList: any[];
  setToast: (a: string, b: string) => void;
  setLoadingPoint: (a: anyObjectType) => void;
  setUnLoadingPoint: (a: anyObjectType) => void;
}

Here is the google map file:

import React, { useEffect, useState, memo } from "react";
import {
  GoogleMap,
  MarkerF,
  DirectionsRenderer,
  DirectionsService,
  InfoWindow,
  useLoadScript,
  LoadScriptNext,
} from "@react-google-maps/api";
import LoadingMarker from "../SvgIcons/loading-marker.svg";
import UnLoadingMarker from "../SvgIcons/unloading-marker.svg";
import StoppageIcon from "../SvgIcons/stopageIcon.svg";
import { CircularProgress } from "@mui/material";
import { Box } from "@mui/system";
import { isEmptyArray } from "@services/utils";
import { anyObjectType, Dispatcher } from "@services/types";

const containerStyle = {
  width: "100%",
  height: "100%",
};

const center = {
  lat: 23.810331,
  lng: 90.412521,
};

interface CustomLatLong {
  latitude: number;
  longitude: number;
  textEn: string;
}

type CustomDirectionsWaypoint = google.maps.DirectionsWaypoint & CustomLatLong;

interface CustomGoogleMapProps {
  call: Boolean;
  setCall: Dispatcher<boolean>;
  handleStopageDragLocation?: (a: number, b: number, c: number) => void;
  handleLoadingPointDragLocation?: (a: number, b: number) => void;
  handleUnLoadingPointDragLocation?: (a: number, b: number) => void;
  waypointsList2: CustomDirectionsWaypoint[];
  loadingP: anyObjectType;
  unloadingP: anyObjectType;
  isAddressModal?: Boolean;
}

const CustomGoogleMap: React.FC<CustomGoogleMapProps> = (props) => {
  const {
    call,
    setCall,
    loadingP,
    unloadingP,
    waypointsList2,
    handleLoadingPointDragLocation,
    handleUnLoadingPointDragLocation,
    handleStopageDragLocation,
    isAddressModal = false,
  } = props;

  // const { isLoaded, loadError } = useLoadScript({
  //   googleMapsApiKey: process.env.NEXT_PUBLIC_GOOGLE_MAPS_API_KEY!,
  // });

  const [infoBox, setInfoWindow] = useState({
    position: {
      lat: 0,
      lng: 0,
    },
    info: "",
  });

  const [directions, setDirections] = useState(null);
  console.log("point:", loadingP, unloadingP);

  useEffect(() => {
    if (loadingP.latitude && unloadingP.latitude) {
      setCall(true);
    }
  }, [loadingP.latitude, unloadingP.latitude]);

  const directionsCallback = (response: any) => {
    if (response !== null) {
      console.log("directionsCallback called");
      if (response.status === "OK") {
        setCall(false);
        setDirections(response);
      }
    }
  };

  const getDirectionRender = () => {
    return (
      <DirectionsService
        options={{
          destination: {
            lat: unloadingP.latitude,
            lng: unloadingP.longitude,
          },
          origin: {
            lat: loadingP.latitude,
            lng: loadingP.longitude,
          },
          travelMode: google.maps.TravelMode.DRIVING,
          waypoints: waypointsList2.map(({ latitude, longitude }) => {
            return {
              location: {
                lat: latitude,
                lng: longitude,
              }.toString(),
              stopover: false,
            };
          }),
        }}
        callback={directionsCallback}
      />
    );
  };

  // if (loadError) {
  //   return
Map cannot be loaded right now, sorry.
;
  // }

  const renderMap = () => {
    return (
      <LoadScriptNext
        googleMapsApiKey={process.env.NEXT_PUBLIC_GOOGLE_MAPS_API_KEY!}
      >
        <GoogleMap
          mapContainerStyle={containerStyle}
          center={
            loadingP.latitude
              ? {
                  lat: loadingP.latitude,
                  lng: loadingP.longitude,
                }
              : center
          }
          zoom={10}
          onClick={({ latLng }) => {
            isAddressModal &&
              handleLoadingPointDragLocation &&
              handleLoadingPointDragLocation(latLng!.lat(), latLng!.lng());
          }}
        >
          {loadingP.latitude &&
            unloadingP.latitude &&
            call &&
            getDirectionRender()}
          {directions && (
            <DirectionsRenderer
              options={{
                directions: directions,
                suppressMarkers: true,
              }}
            />
          )}

          {infoBox.position.lat && (
            <InfoWindow
              // options={infoBoxOptions}
              position={infoBox.position}
              onCloseClick={() =>
                setInfoWindow({
                  position: {
                    lat: 0,
                    lng: 0,
                  },
                  info: "",
                })
              }
            >

 {infoBox.info}

            </InfoWindow>
          )}

          <MarkerF
            key={loadingP.latitude}
            draggable
            onDragEnd={({ latLng }) => {
              handleLoadingPointDragLocation &&
                handleLoadingPointDragLocation(latLng!.lat(), latLng!.lng());
            }}
            position={{ lat: loadingP.latitude, lng: loadingP.longitude }}
            icon={LoadingMarker}
            onClick={() =>
              setInfoWindow({
                ...infoBox,
                position: {
                  lat: loadingP.latitude,
                  lng: loadingP.longitude,
                },
                info: loadingP.textEn,
              })
            }
          />
          <MarkerF
            key={unloadingP.latitude}
            draggable
            onDragEnd={({ latLng }) => {
              handleUnLoadingPointDragLocation &&
                handleUnLoadingPointDragLocation(latLng!.lat(), latLng!.lng());
            }}
            position={{
              lat: unloadingP.latitude,
              lng: unloadingP.longitude,
            }}
            icon={UnLoadingMarker}
            onClick={() =>
              setInfoWindow({
                ...infoBox,
                position: {
                  lat: unloadingP.latitude,
                  lng: unloadingP.longitude,
                },
                info: unloadingP.textEn,
              })
            }
          />
          {!isEmptyArray(waypointsList2) &&
            waypointsList2.map((item, index) => {
              return (
                <MarkerF
                  key={index}
                  draggable
                  visible
                  onDragEnd={({ latLng }) => {
                    handleStopageDragLocation &&
                      handleStopageDragLocation(
                        latLng!.lat(),
                        latLng!.lng(),
                        index
                      );
                  }}
                  position={{
                    lat: item.latitude,
                    lng: item.longitude,
                  }}
                  icon={StoppageIcon}
                  onClick={() =>
                    setInfoWindow({
                      ...infoBox,
                      position: {
                        lat: item.latitude,
                        lng: item.longitude,
                      },
                      info: item.textEn,
                    })
                  }
                />
              );
            })}
        </GoogleMap>
      </LoadScriptNext>
    );
  };
  return renderMap();
};

export default memo(CustomGoogleMap);


问:
Call stored procedures of postgresql using node js
How can I call the stored procedures from postgresql in nodejs using async await.

This is my pool.

```<code>const pool = new pg.Pool({
user: "postgres",
password: "password",
host: "localhost",
database: "mydatabase",
port: 5432
});
</code>```

This is  the code.

```<code>app.put("/:id", async(req, res) => {
try {
    const { id } = req.params;
    const { name, email, hobby } = req.body;
    const updateStudent = await pool.proc('updateUser', [name, email, hobby, id]);
    res.json("Student was updated")
} catch (err) {
    console.error(err.message);
}
})
</code>```

Body:
How can I call the stored procedures from postgresql in nodejs using async await.

This is my pool.

const pool = new pg.Pool({
user: "postgres",
password: "password",
host: "localhost",
database: "mydatabase",
port: 5432
});

This is  the code.

app.put("/:id", async(req, res) => {
try {
    const { id } = req.params;
    const { name, email, hobby } = req.body;
    const updateStudent = await pool.proc('updateUser', [name, email, hobby, id]);
    res.json("Student was updated")
} catch (err) {
    console.error(err.message);
}
})