问:
All executors dead MinHash LSH PySpark approxSimilarityJoin self-join on EMR cluster
I run into problems when calling Spark's MinHashLSH's approxSimilarityJoin on a dataframe of (name_id, name) combinations.

A summary of the problem I try to solve:

I  have a dataframe of around 30 million unique (name_id, name) combinations for company names. Some of those names refer to the same company, but are (i) either misspelled, and/or (ii) include additional names. Performing fuzzy string matching for every combination is not possible. To reduce the number of fuzzy string matching combinations, I use MinHashLSH in Spark. My intended approach is to use a approxSimilarityJoin (self-join) with a relatively large Jaccard threshold, such that I am able to run a fuzzy matching algorithm on the matched combinations to further improve the disambiguation.

A summary of the steps I took:

Used CountVectorizer to create a vector of character counts for every name,

Used MinHashLSH and its approxSimilarityJoin with the following settings:

numHashTables=100

threshold=0.3 (Jaccard threshold for approxSimilarityJoin)

After the approxSimilarityJoin, I remove duplicate combinations (for which holds that there exists a matched combination (i,j) and (j,i), then I remove (j,i))

After removing the duplicate combinations, I run a fuzzy string matching algorithm using the FuzzyWuzzy package to reduce the number of records and improve the disambiguation of the names.

Eventually I run a connectedComponents algorithm on the remaining edges (i,j) to match which company names belong together.

Part of code used:

```<code>    id_col = 'id'
    name_col = 'name'
    num_hastables = 100
    max_jaccard = 0.3
    fuzzy_threshold = 90
    fuzzy_method = fuzz.token_set_ratio

    # Calculate edges using minhash practices
    edges = MinHashLSH(inputCol='vectorized_char_lst', outputCol='hashes', numHashTables=num_hastables).\
        fit(data).\
        approxSimilarityJoin(data, data, max_jaccard).\
        select(col('datasetA.'+id_col).alias('src'),
               col('datasetA.clean').alias('src_name'),
               col('datasetB.'+id_col).alias('dst'),
               col('datasetB.clean').alias('dst_name')).\
        withColumn('comb', sort_array(array(*('src', 'dst')))).\
        dropDuplicates(['comb']).\
        rdd.\
        filter(lambda x: fuzzy_method(x['src_name'], x['dst_name']) >= fuzzy_threshold if x['src'] != x['dst'] else False).\
        toDF().\
        drop(*('src_name', 'dst_name', 'comb'))
</code>```

Explain plan of edges

```<code>== Physical Plan ==
*(5) HashAggregate(keys=[datasetA#232, datasetB#263], functions=[])
+- Exchange hashpartitioning(datasetA#232, datasetB#263, 200)
   +- *(4) HashAggregate(keys=[datasetA#232, datasetB#263], functions=[])
      +- *(4) Project [datasetA#232, datasetB#263]
         +- *(4) BroadcastHashJoin [entry#233, hashValue#234], [entry#264, hashValue#265], Inner, BuildRight, (UDF(datasetA#232.vectorized_char_lst, datasetB#263.vectorized_char_lst) < 0.3)
            :- *(4) Project [named_struct(id, id#10, name, name#11, clean, clean#90, char_lst, char_lst#95, vectorized_char_lst, vectorized_char_lst#107, hashes, hashes#225) AS datasetA#232, entry#233, hashValue#234]
            :  +- *(4) Filter isnotnull(hashValue#234)
            :     +- Generate posexplode(hashes#225), [id#10, name#11, clean#90, char_lst#95, vectorized_char_lst#107, hashes#225], false, [entry#233, hashValue#234]
            :        +- *(1) Project [id#10, name#11, clean#90, char_lst#95, vectorized_char_lst#107, UDF(vectorized_char_lst#107) AS hashes#225]
            :           +- InMemoryTableScan [char_lst#95, clean#90, id#10, name#11, vectorized_char_lst#107]
            :                 +- InMemoryRelation [id#10, name#11, clean#90, char_lst#95, vectorized_char_lst#107], StorageLevel(disk, memory, deserialized, 1 replicas)
            :                       +- *(4) Project [id#10, name#11, pythonUDF0#114 AS clean#90, pythonUDF2#116 AS char_lst#95, UDF(pythonUDF2#116) AS vectorized_char_lst#107]
            :                          +- BatchEvalPython [<lambda>(name#11), <lambda>(<lambda>(name#11)), <lambda>(<lambda>(name#11))], [id#10, name#11, pythonUDF0#114, pythonUDF1#115, pythonUDF2#116]
            :                             +- SortAggregate(key=[name#11], functions=[first(id#10, false)])
            :                                +- *(3) Sort [name#11 ASC NULLS FIRST], false, 0
            :                                   +- Exchange hashpartitioning(name#11, 200)
            :                                      +- SortAggregate(key=[name#11], functions=[partial_first(id#10, false)])
            :                                         +- *(2) Sort [name#11 ASC NULLS FIRST], false, 0
            :                                            +- Exchange RoundRobinPartitioning(8)
            :                                               +- *(1) Filter AtLeastNNulls(n, id#10,name#11)
            :                                                  +- *(1) FileScan csv [id#10,name#11] Batched: false, Format: CSV, Location: InMemoryFileIndex[file:<path>, PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:string,name:string>
            +- BroadcastExchange HashedRelationBroadcastMode(List(input[1, int, false], input[2, vector, true]))
               +- *(3) Project [named_struct(id, id#10, name, name#11, clean, clean#90, char_lst, char_lst#95, vectorized_char_lst, vectorized_char_lst#107, hashes, hashes#256) AS datasetB#263, entry#264, hashValue#265]
                  +- *(3) Filter isnotnull(hashValue#265)
                     +- Generate posexplode(hashes#256), [id#10, name#11, clean#90, char_lst#95, vectorized_char_lst#107, hashes#256], false, [entry#264, hashValue#265]
                        +- *(2) Project [id#10, name#11, clean#90, char_lst#95, vectorized_char_lst#107, UDF(vectorized_char_lst#107) AS hashes#256]
                           +- InMemoryTableScan [char_lst#95, clean#90, id#10, name#11, vectorized_char_lst#107]
                                 +- InMemoryRelation [id#10, name#11, clean#90, char_lst#95, vectorized_char_lst#107], StorageLevel(disk, memory, deserialized, 1 replicas)
                                       +- *(4) Project [id#10, name#11, pythonUDF0#114 AS clean#90, pythonUDF2#116 AS char_lst#95, UDF(pythonUDF2#116) AS vectorized_char_lst#107]
                                          +- BatchEvalPython [<lambda>(name#11), <lambda>(<lambda>(name#11)), <lambda>(<lambda>(name#11))], [id#10, name#11, pythonUDF0#114, pythonUDF1#115, pythonUDF2#116]
                                             +- SortAggregate(key=[name#11], functions=[first(id#10, false)])
                                                +- *(3) Sort [name#11 ASC NULLS FIRST], false, 0
                                                   +- Exchange hashpartitioning(name#11, 200)
                                                      +- SortAggregate(key=[name#11], functions=[partial_first(id#10, false)])
                                                         +- *(2) Sort [name#11 ASC NULLS FIRST], false, 0
                                                            +- Exchange RoundRobinPartitioning(8)
                                                               +- *(1) Filter AtLeastNNulls(n, id#10,name#11)
                                                                  +- *(1) FileScan csv [id#10,name#11] Batched: false, Format: CSV, Location: InMemoryFileIndex[file:<path>, PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:string,name:string>

</code>```

How data looks:

```<code>+-------+--------------------+--------------------+--------------------+--------------------+
|     id|                name|               clean|            char_lst| vectorized_char_lst|
+-------+--------------------+--------------------+--------------------+--------------------+
|3633038|MURATA MACHINERY LTD|    MURATA MACHINERY|[M, U, R, A, T, A...|(33,[0,1,2,3,4,5,...|
|3632811|SOCIETE ANONYME D...|SOCIETE ANONYME D...|[S, O, C, I, E, T...|(33,[0,1,2,3,4,5,...|
|3632655|FUJIFILM CORPORATION|            FUJIFILM|[F, U, J, I, F, I...|(33,[3,10,12,13,2...|
|3633318|HEINE OPTOTECHNIK...|HEINE OPTOTECHNIK...|[H, E, I, N, E,  ...|(33,[0,1,2,3,4,5,...|
|3633523|SUNBEAM PRODUCTS INC|    SUNBEAM PRODUCTS|[S, U, N, B, E, A...|(33,[0,1,2,4,5,6,...|
|3633300|           HIVAL LTD|               HIVAL|     [H, I, V, A, L]|(33,[2,3,10,11,21...|
|3632657|             NSK LTD|                 NSK|           [N, S, K]|(33,[5,6,16],[1.0...|
|3633240|REHABILITATION IN...|REHABILITATION IN...|[R, E, H, A, B, I...|(33,[0,1,2,3,4,5,...|
|3632732|STUDIENGESELLSCHA...|STUDIENGESELLSCHA...|[S, T, U, D, I, E...|(33,[0,1,2,3,4,5,...|
|3632866|ENERGY CONVERSION...|ENERGY CONVERSION...|[E, N, E, R, G, Y...|(33,[0,1,3,5,6,7,...|
|3632895|ERGENICS POWER SY...|ERGENICS POWER SY...|[E, R, G, E, N, I...|(33,[0,1,3,4,5,6,...|
|3632897| MOLI ENERGY LIMITED|         MOLI ENERGY|[M, O, L, I,  , E...|(33,[0,1,3,5,7,8,...|
|3633275| NORDSON CORPORATION|             NORDSON|[N, O, R, D, S, O...|(33,[5,6,7,8,14],...|
|3633256|  PEROXIDCHEMIE GMBH|       PEROXIDCHEMIE|[P, E, R, O, X, I...|(33,[0,3,7,8,9,11...|
|3632695|      POWER CELL INC|          POWER CELL|[P, O, W, E, R,  ...|(33,[0,1,7,8,9,10...|
|3633037|        ERGENICS INC|            ERGENICS|[E, R, G, E, N, I...|(33,[0,3,5,6,8,9,...|
|3632878|  FORD MOTOR COMPANY|          FORD MOTOR|[F, O, R, D,  , M...|(33,[1,4,7,8,13,1...|
|3632573|    SAFT AMERICA INC|        SAFT AMERICA|[S, A, F, T,  , A...|(33,[0,1,2,3,4,6,...|
|3632852|ALCAN INTERNATION...| ALCAN INTERNATIONAL|[A, L, C, A, N,  ...|(33,[0,1,2,3,4,5,...|
|3632698|   KRUPPKOPPERS GMBH|        KRUPPKOPPERS|[K, R, U, P, P, K...|(33,[0,6,7,8,12,1...|
|3633150|ALCAN INTERNATION...| ALCAN INTERNATIONAL|[A, L, C, A, N,  ...|(33,[0,1,2,3,4,5,...|
|3632761|AMERICAN TELEPHON...|AMERICAN TELEPHON...|[A, M, E, R, I, C...|(33,[0,1,2,3,4,5,...|
|3632757|HITACHI KOKI COMP...|        HITACHI KOKI|[H, I, T, A, C, H...|(33,[1,2,3,4,7,9,...|
|3632836|HUGHES AIRCRAFT C...|     HUGHES AIRCRAFT|[H, U, G, H, E, S...|(33,[0,1,2,3,4,6,...|
|3633152|            SOSY INC|                SOSY|        [S, O, S, Y]|(33,[6,7,18],[2.0...|
|3633052|HAMAMATSU PHOTONI...|HAMAMATSU PHOTONI...|[H, A, M, A, M, A...|(33,[1,2,3,4,5,6,...|
|3633450|       AKZO NOBEL NV|          AKZO NOBEL|[A, K, Z, O,  , N...|(33,[0,1,2,5,7,10...|
|3632713| ELTRON RESEARCH INC|     ELTRON RESEARCH|[E, L, T, R, O, N...|(33,[0,1,2,4,5,6,...|
|3632533|NEC ELECTRONICS C...|     NEC ELECTRONICS|[N, E, C,  , E, L...|(33,[0,1,3,4,5,6,...|
|3632562| TARGETTI SANKEY SPA| TARGETTI SANKEY SPA|[T, A, R, G, E, T...|(33,[0,1,2,3,4,5,...|
+-------+--------------------+--------------------+--------------------+--------------------+
only showing top 30 rows
</code>```

Hardware used:

Master node: m5.2xlarge
8 vCore, 32 GiB memory, EBS only storage
EBS Storage:128 GiB

Slave nodes (10x): m5.4xlarge
16 vCore, 64 GiB memory, EBS only storage
EBS Storage:500 GiB

Spark-submit settings used:

```<code>spark-submit --master yarn --conf "spark.executor.instances=40" --conf "spark.default.parallelism=640" --conf "spark.shuffle.partitions=2000" --conf "spark.executor.cores=4" --conf "spark.executor.memory=14g" --conf "spark.driver.memory=14g" --conf "spark.driver.maxResultSize=14g" --conf "spark.dynamicAllocation.enabled=false" --packages graphframes:graphframes:0.7.0-spark2.4-s_2.11 run_disambiguation.py
</code>```

Task errors from Web UI

```<code>ExecutorLostFailure (executor 21 exited caused by one of the running tasks) Reason: Slave lost
</code>```

```<code>ExecutorLostFailure (executor 31 exited unrelated to the running tasks) Reason: Container marked as failed: container_1590592506722_0001_02_000002 on host: ip-172-31-47-180.eu-central-1.compute.internal. Exit status: -100. Diagnostics: Container released on a *lost* node.
</code>```

(Part of) executor logs:

```<code>
20/05/27 16:29:09 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (25  times so far)
20/05/27 16:29:13 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (26  times so far)
20/05/27 16:29:15 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (28  times so far)
20/05/27 16:29:17 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (0  time so far)
20/05/27 16:29:28 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (27  times so far)
20/05/27 16:29:28 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (26  times so far)
20/05/27 16:29:33 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (29  times so far)
20/05/27 16:29:38 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (1  time so far)
20/05/27 16:29:42 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (27  times so far)
20/05/27 16:29:46 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (28  times so far)
20/05/27 16:29:53 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (30  times so far)
20/05/27 16:29:57 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (2  times so far)
20/05/27 16:30:00 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (28  times so far)
20/05/27 16:30:05 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (29  times so far)
20/05/27 16:30:10 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (31  times so far)
20/05/27 16:30:15 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (3  times so far)
20/05/27 16:30:19 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (29  times so far)
20/05/27 16:30:22 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (30  times so far)
20/05/27 16:30:29 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (32  times so far)
20/05/27 16:30:32 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (4  times so far)
20/05/27 16:30:39 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (31  times so far)
20/05/27 16:30:39 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (30  times so far)
20/05/27 16:30:46 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (33  times so far)
20/05/27 16:30:47 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (5  times so far)
20/05/27 16:30:55 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (32  times so far)
20/05/27 16:30:59 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (31  times so far)
20/05/27 16:31:03 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (34  times so far)
20/05/27 16:31:06 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (6  times so far)
20/05/27 16:31:13 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (33  times so far)
20/05/27 16:31:14 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (32  times so far)
20/05/27 16:31:22 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (35  times so far)
20/05/27 16:31:24 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (7  times so far)
20/05/27 16:31:30 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (34  times so far)
20/05/27 16:31:32 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (33  times so far)
20/05/27 16:31:41 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (36  times so far)
20/05/27 16:31:44 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (8  times so far)
20/05/27 16:31:47 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (35  times so far)
20/05/27 16:31:48 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (34  times so far)
20/05/27 16:32:02 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (37  times so far)
20/05/27 16:32:03 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (9  times so far)
20/05/27 16:32:04 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (36  times so far)
20/05/27 16:32:08 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (35  times so far)
20/05/27 16:32:19 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (38  times so far)
20/05/27 16:32:20 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (37  times so far)
20/05/27 16:32:21 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (10  times so far)
20/05/27 16:32:26 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (36  times so far)
20/05/27 16:32:37 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (39  times so far)
20/05/27 16:32:37 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (11  times so far)
20/05/27 16:32:38 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (38  times so far)
20/05/27 16:32:45 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (37  times so far)
20/05/27 16:32:51 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (40  times so far)
20/05/27 16:32:56 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (12  times so far)
20/05/27 16:32:58 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (39  times so far)
20/05/27 16:33:03 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (38  times so far)
20/05/27 16:33:08 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (41  times so far)
20/05/27 16:33:13 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (13  times so far)
20/05/27 16:33:15 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (40  times so far)
20/05/27 16:33:20 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (39  times so far)
20/05/27 16:33:26 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (42  times so far)
20/05/27 16:33:30 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (41  times so far)
20/05/27 16:33:31 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (14  times so far)
20/05/27 16:33:36 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (40  times so far)
20/05/27 16:33:46 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1992.0 MB to disk (43  times so far)
20/05/27 16:33:47 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (42  times so far)
20/05/27 16:33:51 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (15  times so far)
20/05/27 16:33:54 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (41  times so far)
20/05/27 16:34:03 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1992.0 MB to disk (43  times so far)
20/05/27 16:34:04 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1992.0 MB to disk (44  times so far)
20/05/27 16:34:08 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (16  times so far)
20/05/27 16:34:14 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (42  times so far)
20/05/27 16:34:16 INFO PythonUDFRunner: Times: total = 774701, boot = 3, init = 10, finish = 774688
20/05/27 16:34:21 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1992.0 MB to disk (44  times so far)
20/05/27 16:34:22 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (17  times so far)
20/05/27 16:34:30 INFO PythonUDFRunner: Times: total = 773372, boot = 2, init = 9, finish = 773361
20/05/27 16:34:32 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1992.0 MB to disk (43  times so far)
20/05/27 16:34:39 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (18  times so far)
20/05/27 16:34:46 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1992.0 MB to disk (44  times so far)
20/05/27 16:34:52 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (19  times so far)
20/05/27 16:35:01 INFO PythonUDFRunner: Times: total = 776905, boot = 3, init = 11, finish = 776891
20/05/27 16:35:05 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (20  times so far)
20/05/27 16:35:19 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (21  times so far)
20/05/27 16:35:35 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (22  times so far)
20/05/27 16:35:52 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (23  times so far)
20/05/27 16:36:10 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (24  times so far)
20/05/27 16:36:29 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (25  times so far)
20/05/27 16:36:47 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (26  times so far)
20/05/27 16:37:06 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (27  times so far)
20/05/27 16:37:25 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (28  times so far)
20/05/27 16:37:44 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (29  times so far)
20/05/27 16:38:03 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (30  times so far)
20/05/27 16:38:22 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (31  times so far)
20/05/27 16:38:41 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (32  times so far)
20/05/27 16:38:59 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (33  times so far)
20/05/27 16:39:19 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (34  times so far)
20/05/27 16:39:39 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (35  times so far)
20/05/27 16:39:58 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (36  times so far)
20/05/27 16:40:18 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (37  times so far)
20/05/27 16:40:38 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (38  times so far)
20/05/27 16:40:57 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (39  times so far)
20/05/27 16:41:16 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (40  times so far)
20/05/27 16:41:35 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (41  times so far)
20/05/27 16:41:55 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (42  times so far)
20/05/27 16:42:19 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1992.0 MB to disk (43  times so far)
20/05/27 16:42:41 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1992.0 MB to disk (44  times so far)
20/05/27 16:42:59 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
20/05/27 16:42:59 INFO DiskBlockManager: Shutdown hook called
20/05/27 16:42:59 INFO ShutdownHookManager: Shutdown hook called
20/05/27 16:42:59 INFO ShutdownHookManager: Deleting directory /mnt/yarn/usercache/hadoop/appcache/application_1590592506722_0001/spark-73af8e3b-f428-47d4-9e13-fed4e19cc2cd
</code>```

```<code>2020-05-27T16:41:16.336+0000: [GC (Allocation Failure) 2020-05-27T16:41:16.336+0000: [ParNew: 272234K->242K(305984K), 0.0094375 secs] 9076907K->8804915K(13188748K), 0.0094895 secs] [Times: user=0.12 sys=0.00, real=0.01 secs]
2020-05-27T16:41:34.686+0000: [GC (Allocation Failure) 2020-05-27T16:41:34.686+0000: [ParNew: 272242K->257K(305984K), 0.0084179 secs] 9076915K->8804947K(13188748K), 0.0084840 secs] [Times: user=0.09 sys=0.01, real=0.01 secs]
2020-05-27T16:41:35.145+0000: [GC (Allocation Failure) 2020-05-27T16:41:35.145+0000: [ParNew: 272257K->1382K(305984K), 0.0095541 secs] 9076947K->8806073K(13188748K), 0.0096080 secs] [Times: user=0.12 sys=0.00, real=0.01 secs]
2020-05-27T16:41:55.077+0000: [GC (Allocation Failure) 2020-05-27T16:41:55.077+0000: [ParNew: 273382K->2683K(305984K), 0.0097177 secs] 9078073K->8807392K(13188748K), 0.0097754 secs] [Times: user=0.12 sys=0.00, real=0.01 secs]
2020-05-27T16:41:55.513+0000: [GC (Allocation Failure) 2020-05-27T16:41:55.513+0000: [ParNew: 274683K->3025K(305984K), 0.0093345 secs] 9079392K->8807734K(13188748K), 0.0093892 secs] [Times: user=0.12 sys=0.00, real=0.01 secs]
2020-05-27T16:42:05.481+0000: [GC (Allocation Failure) 2020-05-27T16:42:05.481+0000: [ParNew: 275025K->4102K(305984K), 0.0092950 secs] 9079734K->8808830K(13188748K), 0.0093464 secs] [Times: user=0.12 sys=0.00, real=0.01 secs]
2020-05-27T16:42:18.711+0000: [GC (Allocation Failure) 2020-05-27T16:42:18.711+0000: [ParNew: 276102K->2972K(305984K), 0.0098928 secs] 9080830K->8807700K(13188748K), 0.0099510 secs] [Times: user=0.13 sys=0.00, real=0.01 secs]
2020-05-27T16:42:36.493+0000: [GC (Allocation Failure) 2020-05-27T16:42:36.493+0000: [ParNew: 274972K->3852K(305984K), 0.0094324 secs] 9079700K->8808598K(13188748K), 0.0094897 secs] [Times: user=0.11 sys=0.00, real=0.01 secs]
2020-05-27T16:42:40.880+0000: [GC (Allocation Failure) 2020-05-27T16:42:40.880+0000: [ParNew: 275852K->2568K(305984K), 0.0111794 secs] 9080598K->8807882K(13188748K), 0.0112352 secs] [Times: user=0.13 sys=0.00, real=0.01 secs]
Heap
 par new generation   total 305984K, used 261139K [0x0000000440000000, 0x0000000454c00000, 0x0000000483990000)
  eden space 272000K,  95% used [0x0000000440000000, 0x000000044fc82cf8, 0x00000004509a0000)
  from space 33984K,   7% used [0x00000004509a0000, 0x0000000450c220a8, 0x0000000452ad0000)
  to   space 33984K,   0% used [0x0000000452ad0000, 0x0000000452ad0000, 0x0000000454c00000)
 concurrent mark-sweep generation total 12882764K, used 8805314K [0x0000000483990000, 0x0000000795e63000, 0x00000007c0000000)
 Metaspace       used 77726K, capacity 79553K, committed 79604K, reserved 1118208K
  class space    used 10289K, capacity 10704K, committed 10740K, reserved 1048576K
</code>```

Screenshot of executors

What I tried:

Changing spark.sql.shuffle.partitions

Changing spark.default.parallelism

Repartition the dataframe

How can I solve this issue?

Thanks in advance!

Thijs

答:
approxSimilarityJoin will only parallelize well across workers if the tokens being input into MinHash are sufficiently distinct.  Since individual character tokens appear frequently across many records; include an NGram transformation on your character list to make the appearance of each token less frequent; this will greatly reduce data skew and will resolve memory strain.

MinHash simulates the process of creating a random permutation of your token population and selects the token in the sample set that appears first in the permutation.  Since you are using individual characters as tokens, let's say you select a MinHash seed that makes the character e the first in your random permutation.  In this case, every row with the letter e in it will have a matching MinHash and will be shuffled to the same worker for set comparison.  This will cause extreme data skew and out of memory errors.

Body:
I run into problems when calling Spark's MinHashLSH's approxSimilarityJoin on a dataframe of (name_id, name) combinations.

A summary of the problem I try to solve:

I  have a dataframe of around 30 million unique (name_id, name) combinations for company names. Some of those names refer to the same company, but are (i) either misspelled, and/or (ii) include additional names. Performing fuzzy string matching for every combination is not possible. To reduce the number of fuzzy string matching combinations, I use MinHashLSH in Spark. My intended approach is to use a approxSimilarityJoin (self-join) with a relatively large Jaccard threshold, such that I am able to run a fuzzy matching algorithm on the matched combinations to further improve the disambiguation.

A summary of the steps I took:

Used CountVectorizer to create a vector of character counts for every name,

Used MinHashLSH and its approxSimilarityJoin with the following settings:

numHashTables=100

threshold=0.3 (Jaccard threshold for approxSimilarityJoin)

After the approxSimilarityJoin, I remove duplicate combinations (for which holds that there exists a matched combination (i,j) and (j,i), then I remove (j,i))

After removing the duplicate combinations, I run a fuzzy string matching algorithm using the FuzzyWuzzy package to reduce the number of records and improve the disambiguation of the names.

Eventually I run a connectedComponents algorithm on the remaining edges (i,j) to match which company names belong together.

Part of code used:

    id_col = 'id'
    name_col = 'name'
    num_hastables = 100
    max_jaccard = 0.3
    fuzzy_threshold = 90
    fuzzy_method = fuzz.token_set_ratio

    # Calculate edges using minhash practices
    edges = MinHashLSH(inputCol='vectorized_char_lst', outputCol='hashes', numHashTables=num_hastables).\
        fit(data).\
        approxSimilarityJoin(data, data, max_jaccard).\
        select(col('datasetA.'+id_col).alias('src'),
               col('datasetA.clean').alias('src_name'),
               col('datasetB.'+id_col).alias('dst'),
               col('datasetB.clean').alias('dst_name')).\
        withColumn('comb', sort_array(array(*('src', 'dst')))).\
        dropDuplicates(['comb']).\
        rdd.\
        filter(lambda x: fuzzy_method(x['src_name'], x['dst_name']) >= fuzzy_threshold if x['src'] != x['dst'] else False).\
        toDF().\
        drop(*('src_name', 'dst_name', 'comb'))

Explain plan of edges

== Physical Plan ==
*(5) HashAggregate(keys=[datasetA#232, datasetB#263], functions=[])
+- Exchange hashpartitioning(datasetA#232, datasetB#263, 200)
   +- *(4) HashAggregate(keys=[datasetA#232, datasetB#263], functions=[])
      +- *(4) Project [datasetA#232, datasetB#263]
         +- *(4) BroadcastHashJoin [entry#233, hashValue#234], [entry#264, hashValue#265], Inner, BuildRight, (UDF(datasetA#232.vectorized_char_lst, datasetB#263.vectorized_char_lst) < 0.3)
            :- *(4) Project [named_struct(id, id#10, name, name#11, clean, clean#90, char_lst, char_lst#95, vectorized_char_lst, vectorized_char_lst#107, hashes, hashes#225) AS datasetA#232, entry#233, hashValue#234]
            :  +- *(4) Filter isnotnull(hashValue#234)
            :     +- Generate posexplode(hashes#225), [id#10, name#11, clean#90, char_lst#95, vectorized_char_lst#107, hashes#225], false, [entry#233, hashValue#234]
            :        +- *(1) Project [id#10, name#11, clean#90, char_lst#95, vectorized_char_lst#107, UDF(vectorized_char_lst#107) AS hashes#225]
            :           +- InMemoryTableScan [char_lst#95, clean#90, id#10, name#11, vectorized_char_lst#107]
            :                 +- InMemoryRelation [id#10, name#11, clean#90, char_lst#95, vectorized_char_lst#107], StorageLevel(disk, memory, deserialized, 1 replicas)
            :                       +- *(4) Project [id#10, name#11, pythonUDF0#114 AS clean#90, pythonUDF2#116 AS char_lst#95, UDF(pythonUDF2#116) AS vectorized_char_lst#107]
            :                          +- BatchEvalPython [<lambda>(name#11), <lambda>(<lambda>(name#11)), <lambda>(<lambda>(name#11))], [id#10, name#11, pythonUDF0#114, pythonUDF1#115, pythonUDF2#116]
            :                             +- SortAggregate(key=[name#11], functions=[first(id#10, false)])
            :                                +- *(3) Sort [name#11 ASC NULLS FIRST], false, 0
            :                                   +- Exchange hashpartitioning(name#11, 200)
            :                                      +- SortAggregate(key=[name#11], functions=[partial_first(id#10, false)])
            :                                         +- *(2) Sort [name#11 ASC NULLS FIRST], false, 0
            :                                            +- Exchange RoundRobinPartitioning(8)
            :                                               +- *(1) Filter AtLeastNNulls(n, id#10,name#11)
            :                                                  +- *(1) FileScan csv [id#10,name#11] Batched: false, Format: CSV, Location: InMemoryFileIndex[file:<path>, PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:string,name:string>
            +- BroadcastExchange HashedRelationBroadcastMode(List(input[1, int, false], input[2, vector, true]))
               +- *(3) Project [named_struct(id, id#10, name, name#11, clean, clean#90, char_lst, char_lst#95, vectorized_char_lst, vectorized_char_lst#107, hashes, hashes#256) AS datasetB#263, entry#264, hashValue#265]
                  +- *(3) Filter isnotnull(hashValue#265)
                     +- Generate posexplode(hashes#256), [id#10, name#11, clean#90, char_lst#95, vectorized_char_lst#107, hashes#256], false, [entry#264, hashValue#265]
                        +- *(2) Project [id#10, name#11, clean#90, char_lst#95, vectorized_char_lst#107, UDF(vectorized_char_lst#107) AS hashes#256]
                           +- InMemoryTableScan [char_lst#95, clean#90, id#10, name#11, vectorized_char_lst#107]
                                 +- InMemoryRelation [id#10, name#11, clean#90, char_lst#95, vectorized_char_lst#107], StorageLevel(disk, memory, deserialized, 1 replicas)
                                       +- *(4) Project [id#10, name#11, pythonUDF0#114 AS clean#90, pythonUDF2#116 AS char_lst#95, UDF(pythonUDF2#116) AS vectorized_char_lst#107]
                                          +- BatchEvalPython [<lambda>(name#11), <lambda>(<lambda>(name#11)), <lambda>(<lambda>(name#11))], [id#10, name#11, pythonUDF0#114, pythonUDF1#115, pythonUDF2#116]
                                             +- SortAggregate(key=[name#11], functions=[first(id#10, false)])
                                                +- *(3) Sort [name#11 ASC NULLS FIRST], false, 0
                                                   +- Exchange hashpartitioning(name#11, 200)
                                                      +- SortAggregate(key=[name#11], functions=[partial_first(id#10, false)])
                                                         +- *(2) Sort [name#11 ASC NULLS FIRST], false, 0
                                                            +- Exchange RoundRobinPartitioning(8)
                                                               +- *(1) Filter AtLeastNNulls(n, id#10,name#11)
                                                                  +- *(1) FileScan csv [id#10,name#11] Batched: false, Format: CSV, Location: InMemoryFileIndex[file:<path>, PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:string,name:string>

How data looks:

+-------+--------------------+--------------------+--------------------+--------------------+
|     id|                name|               clean|            char_lst| vectorized_char_lst|
+-------+--------------------+--------------------+--------------------+--------------------+
|3633038|MURATA MACHINERY LTD|    MURATA MACHINERY|[M, U, R, A, T, A...|(33,[0,1,2,3,4,5,...|
|3632811|SOCIETE ANONYME D...|SOCIETE ANONYME D...|[S, O, C, I, E, T...|(33,[0,1,2,3,4,5,...|
|3632655|FUJIFILM CORPORATION|            FUJIFILM|[F, U, J, I, F, I...|(33,[3,10,12,13,2...|
|3633318|HEINE OPTOTECHNIK...|HEINE OPTOTECHNIK...|[H, E, I, N, E,  ...|(33,[0,1,2,3,4,5,...|
|3633523|SUNBEAM PRODUCTS INC|    SUNBEAM PRODUCTS|[S, U, N, B, E, A...|(33,[0,1,2,4,5,6,...|
|3633300|           HIVAL LTD|               HIVAL|     [H, I, V, A, L]|(33,[2,3,10,11,21...|
|3632657|             NSK LTD|                 NSK|           [N, S, K]|(33,[5,6,16],[1.0...|
|3633240|REHABILITATION IN...|REHABILITATION IN...|[R, E, H, A, B, I...|(33,[0,1,2,3,4,5,...|
|3632732|STUDIENGESELLSCHA...|STUDIENGESELLSCHA...|[S, T, U, D, I, E...|(33,[0,1,2,3,4,5,...|
|3632866|ENERGY CONVERSION...|ENERGY CONVERSION...|[E, N, E, R, G, Y...|(33,[0,1,3,5,6,7,...|
|3632895|ERGENICS POWER SY...|ERGENICS POWER SY...|[E, R, G, E, N, I...|(33,[0,1,3,4,5,6,...|
|3632897| MOLI ENERGY LIMITED|         MOLI ENERGY|[M, O, L, I,  , E...|(33,[0,1,3,5,7,8,...|
|3633275| NORDSON CORPORATION|             NORDSON|[N, O, R, D, S, O...|(33,[5,6,7,8,14],...|
|3633256|  PEROXIDCHEMIE GMBH|       PEROXIDCHEMIE|[P, E, R, O, X, I...|(33,[0,3,7,8,9,11...|
|3632695|      POWER CELL INC|          POWER CELL|[P, O, W, E, R,  ...|(33,[0,1,7,8,9,10...|
|3633037|        ERGENICS INC|            ERGENICS|[E, R, G, E, N, I...|(33,[0,3,5,6,8,9,...|
|3632878|  FORD MOTOR COMPANY|          FORD MOTOR|[F, O, R, D,  , M...|(33,[1,4,7,8,13,1...|
|3632573|    SAFT AMERICA INC|        SAFT AMERICA|[S, A, F, T,  , A...|(33,[0,1,2,3,4,6,...|
|3632852|ALCAN INTERNATION...| ALCAN INTERNATIONAL|[A, L, C, A, N,  ...|(33,[0,1,2,3,4,5,...|
|3632698|   KRUPPKOPPERS GMBH|        KRUPPKOPPERS|[K, R, U, P, P, K...|(33,[0,6,7,8,12,1...|
|3633150|ALCAN INTERNATION...| ALCAN INTERNATIONAL|[A, L, C, A, N,  ...|(33,[0,1,2,3,4,5,...|
|3632761|AMERICAN TELEPHON...|AMERICAN TELEPHON...|[A, M, E, R, I, C...|(33,[0,1,2,3,4,5,...|
|3632757|HITACHI KOKI COMP...|        HITACHI KOKI|[H, I, T, A, C, H...|(33,[1,2,3,4,7,9,...|
|3632836|HUGHES AIRCRAFT C...|     HUGHES AIRCRAFT|[H, U, G, H, E, S...|(33,[0,1,2,3,4,6,...|
|3633152|            SOSY INC|                SOSY|        [S, O, S, Y]|(33,[6,7,18],[2.0...|
|3633052|HAMAMATSU PHOTONI...|HAMAMATSU PHOTONI...|[H, A, M, A, M, A...|(33,[1,2,3,4,5,6,...|
|3633450|       AKZO NOBEL NV|          AKZO NOBEL|[A, K, Z, O,  , N...|(33,[0,1,2,5,7,10...|
|3632713| ELTRON RESEARCH INC|     ELTRON RESEARCH|[E, L, T, R, O, N...|(33,[0,1,2,4,5,6,...|
|3632533|NEC ELECTRONICS C...|     NEC ELECTRONICS|[N, E, C,  , E, L...|(33,[0,1,3,4,5,6,...|
|3632562| TARGETTI SANKEY SPA| TARGETTI SANKEY SPA|[T, A, R, G, E, T...|(33,[0,1,2,3,4,5,...|
+-------+--------------------+--------------------+--------------------+--------------------+
only showing top 30 rows

Hardware used:

Master node: m5.2xlarge
8 vCore, 32 GiB memory, EBS only storage
EBS Storage:128 GiB

Slave nodes (10x): m5.4xlarge
16 vCore, 64 GiB memory, EBS only storage
EBS Storage:500 GiB

Spark-submit settings used:

spark-submit --master yarn --conf "spark.executor.instances=40" --conf "spark.default.parallelism=640" --conf "spark.shuffle.partitions=2000" --conf "spark.executor.cores=4" --conf "spark.executor.memory=14g" --conf "spark.driver.memory=14g" --conf "spark.driver.maxResultSize=14g" --conf "spark.dynamicAllocation.enabled=false" --packages graphframes:graphframes:0.7.0-spark2.4-s_2.11 run_disambiguation.py

Task errors from Web UI

ExecutorLostFailure (executor 21 exited caused by one of the running tasks) Reason: Slave lost

ExecutorLostFailure (executor 31 exited unrelated to the running tasks) Reason: Container marked as failed: container_1590592506722_0001_02_000002 on host: ip-172-31-47-180.eu-central-1.compute.internal. Exit status: -100. Diagnostics: Container released on a *lost* node.

(Part of) executor logs:

20/05/27 16:29:09 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (25  times so far)
20/05/27 16:29:13 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (26  times so far)
20/05/27 16:29:15 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (28  times so far)
20/05/27 16:29:17 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (0  time so far)
20/05/27 16:29:28 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (27  times so far)
20/05/27 16:29:28 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (26  times so far)
20/05/27 16:29:33 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (29  times so far)
20/05/27 16:29:38 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (1  time so far)
20/05/27 16:29:42 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (27  times so far)
20/05/27 16:29:46 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (28  times so far)
20/05/27 16:29:53 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (30  times so far)
20/05/27 16:29:57 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (2  times so far)
20/05/27 16:30:00 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (28  times so far)
20/05/27 16:30:05 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (29  times so far)
20/05/27 16:30:10 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (31  times so far)
20/05/27 16:30:15 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (3  times so far)
20/05/27 16:30:19 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (29  times so far)
20/05/27 16:30:22 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (30  times so far)
20/05/27 16:30:29 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (32  times so far)
20/05/27 16:30:32 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (4  times so far)
20/05/27 16:30:39 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (31  times so far)
20/05/27 16:30:39 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (30  times so far)
20/05/27 16:30:46 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (33  times so far)
20/05/27 16:30:47 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (5  times so far)
20/05/27 16:30:55 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (32  times so far)
20/05/27 16:30:59 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (31  times so far)
20/05/27 16:31:03 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (34  times so far)
20/05/27 16:31:06 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (6  times so far)
20/05/27 16:31:13 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (33  times so far)
20/05/27 16:31:14 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (32  times so far)
20/05/27 16:31:22 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (35  times so far)
20/05/27 16:31:24 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (7  times so far)
20/05/27 16:31:30 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (34  times so far)
20/05/27 16:31:32 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (33  times so far)
20/05/27 16:31:41 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (36  times so far)
20/05/27 16:31:44 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (8  times so far)
20/05/27 16:31:47 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (35  times so far)
20/05/27 16:31:48 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (34  times so far)
20/05/27 16:32:02 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (37  times so far)
20/05/27 16:32:03 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (9  times so far)
20/05/27 16:32:04 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (36  times so far)
20/05/27 16:32:08 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (35  times so far)
20/05/27 16:32:19 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (38  times so far)
20/05/27 16:32:20 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (37  times so far)
20/05/27 16:32:21 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (10  times so far)
20/05/27 16:32:26 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (36  times so far)
20/05/27 16:32:37 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (39  times so far)
20/05/27 16:32:37 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (11  times so far)
20/05/27 16:32:38 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (38  times so far)
20/05/27 16:32:45 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (37  times so far)
20/05/27 16:32:51 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (40  times so far)
20/05/27 16:32:56 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (12  times so far)
20/05/27 16:32:58 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (39  times so far)
20/05/27 16:33:03 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (38  times so far)
20/05/27 16:33:08 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (41  times so far)
20/05/27 16:33:13 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (13  times so far)
20/05/27 16:33:15 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (40  times so far)
20/05/27 16:33:20 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (39  times so far)
20/05/27 16:33:26 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1988.0 MB to disk (42  times so far)
20/05/27 16:33:30 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (41  times so far)
20/05/27 16:33:31 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (14  times so far)
20/05/27 16:33:36 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (40  times so far)
20/05/27 16:33:46 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1992.0 MB to disk (43  times so far)
20/05/27 16:33:47 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1988.0 MB to disk (42  times so far)
20/05/27 16:33:51 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (15  times so far)
20/05/27 16:33:54 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (41  times so far)
20/05/27 16:34:03 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1992.0 MB to disk (43  times so far)
20/05/27 16:34:04 INFO ShuffleExternalSorter: Thread 146 spilling sort data of 1992.0 MB to disk (44  times so far)
20/05/27 16:34:08 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (16  times so far)
20/05/27 16:34:14 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1988.0 MB to disk (42  times so far)
20/05/27 16:34:16 INFO PythonUDFRunner: Times: total = 774701, boot = 3, init = 10, finish = 774688
20/05/27 16:34:21 INFO ShuffleExternalSorter: Thread 147 spilling sort data of 1992.0 MB to disk (44  times so far)
20/05/27 16:34:22 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (17  times so far)
20/05/27 16:34:30 INFO PythonUDFRunner: Times: total = 773372, boot = 2, init = 9, finish = 773361
20/05/27 16:34:32 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1992.0 MB to disk (43  times so far)
20/05/27 16:34:39 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (18  times so far)
20/05/27 16:34:46 INFO ShuffleExternalSorter: Thread 89 spilling sort data of 1992.0 MB to disk (44  times so far)
20/05/27 16:34:52 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (19  times so far)
20/05/27 16:35:01 INFO PythonUDFRunner: Times: total = 776905, boot = 3, init = 11, finish = 776891
20/05/27 16:35:05 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (20  times so far)
20/05/27 16:35:19 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (21  times so far)
20/05/27 16:35:35 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (22  times so far)
20/05/27 16:35:52 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (23  times so far)
20/05/27 16:36:10 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (24  times so far)
20/05/27 16:36:29 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (25  times so far)
20/05/27 16:36:47 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (26  times so far)
20/05/27 16:37:06 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (27  times so far)
20/05/27 16:37:25 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (28  times so far)
20/05/27 16:37:44 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (29  times so far)
20/05/27 16:38:03 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (30  times so far)
20/05/27 16:38:22 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (31  times so far)
20/05/27 16:38:41 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (32  times so far)
20/05/27 16:38:59 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (33  times so far)
20/05/27 16:39:19 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (34  times so far)
20/05/27 16:39:39 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (35  times so far)
20/05/27 16:39:58 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (36  times so far)
20/05/27 16:40:18 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (37  times so far)
20/05/27 16:40:38 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (38  times so far)
20/05/27 16:40:57 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (39  times so far)
20/05/27 16:41:16 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (40  times so far)
20/05/27 16:41:35 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (41  times so far)
20/05/27 16:41:55 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1988.0 MB to disk (42  times so far)
20/05/27 16:42:19 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1992.0 MB to disk (43  times so far)
20/05/27 16:42:41 INFO ShuffleExternalSorter: Thread 145 spilling sort data of 1992.0 MB to disk (44  times so far)
20/05/27 16:42:59 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
20/05/27 16:42:59 INFO DiskBlockManager: Shutdown hook called
20/05/27 16:42:59 INFO ShutdownHookManager: Shutdown hook called
20/05/27 16:42:59 INFO ShutdownHookManager: Deleting directory /mnt/yarn/usercache/hadoop/appcache/application_1590592506722_0001/spark-73af8e3b-f428-47d4-9e13-fed4e19cc2cd

2020-05-27T16:41:16.336+0000: [GC (Allocation Failure) 2020-05-27T16:41:16.336+0000: [ParNew: 272234K->242K(305984K), 0.0094375 secs] 9076907K->8804915K(13188748K), 0.0094895 secs] [Times: user=0.12 sys=0.00, real=0.01 secs]
2020-05-27T16:41:34.686+0000: [GC (Allocation Failure) 2020-05-27T16:41:34.686+0000: [ParNew: 272242K->257K(305984K), 0.0084179 secs] 9076915K->8804947K(13188748K), 0.0084840 secs] [Times: user=0.09 sys=0.01, real=0.01 secs]
2020-05-27T16:41:35.145+0000: [GC (Allocation Failure) 2020-05-27T16:41:35.145+0000: [ParNew: 272257K->1382K(305984K), 0.0095541 secs] 9076947K->8806073K(13188748K), 0.0096080 secs] [Times: user=0.12 sys=0.00, real=0.01 secs]
2020-05-27T16:41:55.077+0000: [GC (Allocation Failure) 2020-05-27T16:41:55.077+0000: [ParNew: 273382K->2683K(305984K), 0.0097177 secs] 9078073K->8807392K(13188748K), 0.0097754 secs] [Times: user=0.12 sys=0.00, real=0.01 secs]
2020-05-27T16:41:55.513+0000: [GC (Allocation Failure) 2020-05-27T16:41:55.513+0000: [ParNew: 274683K->3025K(305984K), 0.0093345 secs] 9079392K->8807734K(13188748K), 0.0093892 secs] [Times: user=0.12 sys=0.00, real=0.01 secs]
2020-05-27T16:42:05.481+0000: [GC (Allocation Failure) 2020-05-27T16:42:05.481+0000: [ParNew: 275025K->4102K(305984K), 0.0092950 secs] 9079734K->8808830K(13188748K), 0.0093464 secs] [Times: user=0.12 sys=0.00, real=0.01 secs]
2020-05-27T16:42:18.711+0000: [GC (Allocation Failure) 2020-05-27T16:42:18.711+0000: [ParNew: 276102K->2972K(305984K), 0.0098928 secs] 9080830K->8807700K(13188748K), 0.0099510 secs] [Times: user=0.13 sys=0.00, real=0.01 secs]
2020-05-27T16:42:36.493+0000: [GC (Allocation Failure) 2020-05-27T16:42:36.493+0000: [ParNew: 274972K->3852K(305984K), 0.0094324 secs] 9079700K->8808598K(13188748K), 0.0094897 secs] [Times: user=0.11 sys=0.00, real=0.01 secs]
2020-05-27T16:42:40.880+0000: [GC (Allocation Failure) 2020-05-27T16:42:40.880+0000: [ParNew: 275852K->2568K(305984K), 0.0111794 secs] 9080598K->8807882K(13188748K), 0.0112352 secs] [Times: user=0.13 sys=0.00, real=0.01 secs]
Heap
 par new generation   total 305984K, used 261139K [0x0000000440000000, 0x0000000454c00000, 0x0000000483990000)
  eden space 272000K,  95% used [0x0000000440000000, 0x000000044fc82cf8, 0x00000004509a0000)
  from space 33984K,   7% used [0x00000004509a0000, 0x0000000450c220a8, 0x0000000452ad0000)
  to   space 33984K,   0% used [0x0000000452ad0000, 0x0000000452ad0000, 0x0000000454c00000)
 concurrent mark-sweep generation total 12882764K, used 8805314K [0x0000000483990000, 0x0000000795e63000, 0x00000007c0000000)
 Metaspace       used 77726K, capacity 79553K, committed 79604K, reserved 1118208K
  class space    used 10289K, capacity 10704K, committed 10740K, reserved 1048576K

Screenshot of executors

What I tried:

Changing spark.sql.shuffle.partitions

Changing spark.default.parallelism

Repartition the dataframe

How can I solve this issue?

Thanks in advance!

Thijs


来源: stackexchange

问:
Some troubles with Web scraping using R
I have some troubles to scrape the text information from this webpage: http://www.iplant.cn/info/Acer%20stachyophyllum?t=foc

What I need is the text information in the center of this webpage: "Trees to 15 m tall, dioecious. ..."
I tried to use the read_html function in R package rvest, but got nothing. Could anyone help me with that? Thanks so much.

答:
This part of the page is generated from an xhr call. You can get the specific piece of text you are looking for from any species by doing:

```<code>get_description <- function(species_name)
{
  url   <- "http://www.iplant.cn/ashx/getfoc.ashx"
  query <- paste0("?key=", gsub(" ", "+", species_name),
                  "&key_no=&m=", runif(1), 9)
  jsonlite::fromJSON(paste0(url, query))$Description
}
</code>```

So for example:

```<code>get_description("Actaea asiatica")
#> [1] "<p>Rhizome black-brown, with numerous slender fibrous roots.
#> Stems 30--80 cm tall, terete, 4--6(--9) mm in diam., unbranched,
#> basally glabrous, apically white pubescent. Leaves 2 or 3, proximal
#> cauline leaves 3 × ternately pinnate ...<truncated>

get_description("Acer stachyophyllum")
# > [1] "<p>Trees to 15 m tall, dioecious. Bark yellowish brown, smooth.
#> Branchlets glabrous. Leaves deciduous; petiole 2.5-8 cm, slightly
#> pubescent near apex; leaf blade ovate or oblong, 5-11 × 2.5-6 cm,
#> undivided or 3-lobed, papery, abaxially densely pale or white pubescent,
#>  becoming less so when mature or nearly glabrous, adaxially glabrous,
#> 3-5-veined at base abaxially, rarely with rudimentary...<truncated>
</code>```

ContentLicense: CC BY-SA 4.0

Title: Some troubles with Web scraping using R

Body:
This part of the page is generated from an xhr call. You can get the specific piece of text you are looking for from any species by doing:

get_description <- function(species_name)
{
  url   <- "http://www.iplant.cn/ashx/getfoc.ashx"
  query <- paste0("?key=", gsub(" ", "+", species_name),
                  "&key_no=&m=", runif(1), 9)
  jsonlite::fromJSON(paste0(url, query))$Description
}

So for example:

get_description("Actaea asiatica")
#> [1] "<p>Rhizome black-brown, with numerous slender fibrous roots.
#> Stems 30--80 cm tall, terete, 4--6(--9) mm in diam., unbranched,
#> basally glabrous, apically white pubescent. Leaves 2 or 3, proximal
#> cauline leaves 3 × ternately pinnate ...<truncated>

get_description("Acer stachyophyllum")
# > [1] "<p>Trees to 15 m tall, dioecious. Bark yellowish brown, smooth.
#> Branchlets glabrous. Leaves deciduous; petiole 2.5-8 cm, slightly
#> pubescent near apex; leaf blade ovate or oblong, 5-11 × 2.5-6 cm,
#> undivided or 3-lobed, papery, abaxially densely pale or white pubescent,
#>  becoming less so when mature or nearly glabrous, adaxially glabrous,
#> 3-5-veined at base abaxially, rarely with rudimentary...<truncated>


问:
The dragging and dropping of image Box
My Code of HTML file in angular9 is, I want to know where I have to make edits in my code to give preview of images in the browser at the place of broken images:

```<code><div cdkDropListGroup>
  <div class="box-container">
    <h2>TABLE A</h2>

    <div
      cdkDropList
      [cdkDropListData]="GridA"
      class="boxList"
      (cdkDropListDropped)="drop($event)"
    >
      <div class="box" *ngFor="let item of GridA" cdkDrag>
        {{ item }}
      </div>
    </div>
  </div>
  <div class="box-container">
    <h2>TABLE B</h2>
    <div
      cdkDropList
      [cdkDropListData]="GridB"
      class="boxList"
      (cdkDropListDropped)="drop($event)"
    >
      <div class="box" *ngFor="let item of GridB" cdkDrag>
        {{ item }}
      </div>
    </div>
  </div>
  <div class="box-container">
    <h2>TABLE C</h2>
    <div
      cdkDropList
      [cdkDropListData]="GridC"
      class="boxList"
      (cdkDropListDropped)="drop($event)"
    >
      <div class="box" *ngFor="let item of GridC" cdkDrag>
        {{ item }}
      </div>
    </div>
  </div>
</div>
</code>```

The code of my ts file is:

```<code>GridA = [
    'https://images.pexels.com/photos/3966779/pexels-photo-3966779.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3933273/pexels-photo-3933273.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/4450380/pexels-photo-4450380.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3932695/pexels-photo-3932695.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
  ];

  GridB = [
    'https://images.pexels.com/photos/4449319/pexels-photo-4449319.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3933042/pexels-photo-3933042.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3768998/pexels-photo-3768998.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3814272/pexels-photo-3814272.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
  ];

  GridC = [
    'https://images.pexels.com/photos/3932938/pexels-photo-3932938.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3755494/pexels-photo-3755494.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/4148847/pexels-photo-4148847.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/4148888/pexels-photo-4148888.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
  ];

drop(event: CdkDragDrop<string[]>) {
    if (event.previousContainer === event.container) {
      moveItemInArray(
        event.container.data,
        event.previousIndex,
        event.currentIndex
      );
    } else {
      transferArrayItem(
        event.previousContainer.data,
        event.container.data,
        event.previousIndex,
        event.currentIndex
      );
    }
  }
</code>```

.

I am able to move the image boxes, but I am not able to watch the preview of my images in the browser when I run the code. I have attached the image for your reference

答:
Working Stackblitz :- https://stackblitz.com/edit/angular-mghxdm

CSS :-

```<code>img {
  width: 100%;
  height: 30vh;
}

.box-container {
  display: inline-block;
  margin: 0 1em;
  width: 30%;
}

.placeHolder {
  width: 30vw;
  height: 30vh;
  background: #ccc;
  border: dotted 3px #999;
  min-height: 60px;
  transition: transform 250ms cubic-bezier(0, 0, 0.2, 1);
}

.headers {
  font-weight: bold;
}
.headers div {
  width: 30%;
  display: inline-block;
  text-align: center;
  margin-bottom: 2em;
}
</code>```

HTML :-

```<code><div class="headers">
  <div>TABLE A</div>
  <div>TABLE B</div>
  <div>TABLE C</div>
</div>
<div cdkDropListGroup>
  <div class="box-container">

    <div id="list1"
      cdkDropList
      [cdkDropListData]="GridA"
      [cdkDropListConnectedTo]="['list2','list3']"
      class="boxList"
      (cdkDropListDropped)="drop($event)"
    >
      <div class="box" *ngFor="let item of GridA">
        <span><div class="placeHolder" *cdkDragPlaceHolder></div><img [src]="item" cdkDrag></span>
      </div>
    </div>
  </div>
  <div class="box-container">
    <div id="list2"
      cdkDropList
      [cdkDropListData]="GridB"
      [cdkDropListConnectedTo]="['list1','list3']"
      class="boxList"
      (cdkDropListDropped)="drop($event)"
    >
      <div class="box" *ngFor="let item of GridB">
        <span><div class="placeHolder" *cdkDragPlaceHolder></div><img [src]="item" cdkDrag></span>
      </div>
    </div>
  </div>
  <div class="box-container">
    <div id="list3"
      cdkDropList
      [cdkDropListData]="GridC"
      [cdkDropListConnectedTo]="['list1','list2']"
      class="boxList"
      (cdkDropListDropped)="drop($event)"
    >
      <div class="box" *ngFor="let item of GridC">
        <span><div class="placeHolder" *cdkDragPlaceHolder></div><img [src]="item" cdkDrag></span>
      </div>
    </div>
  </div>
</div>
</code>```

TS :-

```<code>import { Component } from '@angular/core';
import { CdkDragDrop, moveItemInArray, transferArrayItem } from '@angular/cdk/drag-drop';

@Component({
  selector: 'my-app',
  templateUrl: './app.component.html',
  styleUrls: [ './app.component.css' ]
})
export class AppComponent  {
GridA = [
    'https://images.pexels.com/photos/3966779/pexels-photo-3966779.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3933273/pexels-photo-3933273.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/4450380/pexels-photo-4450380.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3932695/pexels-photo-3932695.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
  ];

  GridB = [
    'https://images.pexels.com/photos/4449319/pexels-photo-4449319.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3933042/pexels-photo-3933042.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3768998/pexels-photo-3768998.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3814272/pexels-photo-3814272.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
  ];

  GridC = [
    'https://images.pexels.com/photos/3932938/pexels-photo-3932938.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3755494/pexels-photo-3755494.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/4148847/pexels-photo-4148847.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/4148888/pexels-photo-4148888.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
  ];

drop(event: CdkDragDrop<string[]>) {
    console.log(event);
    if (event.previousContainer === event.container) {
      moveItemInArray(
        event.container.data,
        event.previousIndex,
        event.currentIndex
      );
    } else {
      transferArrayItem(
        event.previousContainer.data,
        event.container.data,
        event.previousIndex,
        event.currentIndex
      );
    }
  }
}
</code>```

Title: The dragging and dropping of image Box

Body:
My Code of HTML file in angular9 is, I want to know where I have to make edits in my code to give preview of images in the browser at the place of broken images:

TABLE A

        {{ item }}

TABLE B

        {{ item }}

TABLE C

        {{ item }}

The code of my ts file is:

GridA = [
    'https://images.pexels.com/photos/3966779/pexels-photo-3966779.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3933273/pexels-photo-3933273.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/4450380/pexels-photo-4450380.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3932695/pexels-photo-3932695.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
  ];

  GridB = [
    'https://images.pexels.com/photos/4449319/pexels-photo-4449319.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3933042/pexels-photo-3933042.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3768998/pexels-photo-3768998.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3814272/pexels-photo-3814272.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
  ];

  GridC = [
    'https://images.pexels.com/photos/3932938/pexels-photo-3932938.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3755494/pexels-photo-3755494.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/4148847/pexels-photo-4148847.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/4148888/pexels-photo-4148888.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
  ];

drop(event: CdkDragDrop<string[]>) {
    if (event.previousContainer === event.container) {
      moveItemInArray(
        event.container.data,
        event.previousIndex,
        event.currentIndex
      );
    } else {
      transferArrayItem(
        event.previousContainer.data,
        event.container.data,
        event.previousIndex,
        event.currentIndex
      );
    }
  }

.

I am able to move the image boxes, but I am not able to watch the preview of my images in the browser when I run the code. I have attached the image for your reference

Body:
Working Stackblitz :- https://stackblitz.com/edit/angular-mghxdm

CSS :-

img {
  width: 100%;
  height: 30vh;
}

.box-container {
  display: inline-block;
  margin: 0 1em;
  width: 30%;
}

.placeHolder {
  width: 30vw;
  height: 30vh;
  background: #ccc;
  border: dotted 3px #999;
  min-height: 60px;
  transition: transform 250ms cubic-bezier(0, 0, 0.2, 1);
}

.headers {
  font-weight: bold;
}
.headers div {
  width: 30%;
  display: inline-block;
  text-align: center;
  margin-bottom: 2em;
}

HTML :-

<div class="headers">
  <div>TABLE A</div>
  <div>TABLE B</div>
  <div>TABLE C</div>
</div>
<div cdkDropListGroup>
  <div class="box-container">

    <div id="list1"
      cdkDropList
      [cdkDropListData]="GridA"
      [cdkDropListConnectedTo]="['list2','list3']"
      class="boxList"
      (cdkDropListDropped)="drop($event)"
    >
      <div class="box" *ngFor="let item of GridA">
        <span><div class="placeHolder" *cdkDragPlaceHolder></div><img [src]="item" cdkDrag></span>
      </div>
    </div>
  </div>
  <div class="box-container">
    <div id="list2"
      cdkDropList
      [cdkDropListData]="GridB"
      [cdkDropListConnectedTo]="['list1','list3']"
      class="boxList"
      (cdkDropListDropped)="drop($event)"
    >
      <div class="box" *ngFor="let item of GridB">
        <span><div class="placeHolder" *cdkDragPlaceHolder></div><img [src]="item" cdkDrag></span>
      </div>
    </div>
  </div>
  <div class="box-container">
    <div id="list3"
      cdkDropList
      [cdkDropListData]="GridC"
      [cdkDropListConnectedTo]="['list1','list2']"
      class="boxList"
      (cdkDropListDropped)="drop($event)"
    >
      <div class="box" *ngFor="let item of GridC">
        <span><div class="placeHolder" *cdkDragPlaceHolder></div><img [src]="item" cdkDrag></span>
      </div>
    </div>
  </div>
</div>

TS :-

import { Component } from '@angular/core';
import { CdkDragDrop, moveItemInArray, transferArrayItem } from '@angular/cdk/drag-drop';

@Component({
  selector: 'my-app',
  templateUrl: './app.component.html',
  styleUrls: [ './app.component.css' ]
})
export class AppComponent  {
GridA = [
    'https://images.pexels.com/photos/3966779/pexels-photo-3966779.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3933273/pexels-photo-3933273.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/4450380/pexels-photo-4450380.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3932695/pexels-photo-3932695.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
  ];

  GridB = [
    'https://images.pexels.com/photos/4449319/pexels-photo-4449319.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3933042/pexels-photo-3933042.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3768998/pexels-photo-3768998.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3814272/pexels-photo-3814272.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
  ];

  GridC = [
    'https://images.pexels.com/photos/3932938/pexels-photo-3932938.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/3755494/pexels-photo-3755494.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/4148847/pexels-photo-4148847.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
    'https://images.pexels.com/photos/4148888/pexels-photo-4148888.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500',
  ];

drop(event: CdkDragDrop<string[]>) {
    console.log(event);
    if (event.previousContainer === event.container) {
      moveItemInArray(
        event.container.data,
        event.previousIndex,
        event.currentIndex
      );
    } else {
      transferArrayItem(
        event.previousContainer.data,
        event.container.data,
        event.previousIndex,
        event.currentIndex
      );
    }
  }
}


问:
How do I return a element using LINQ query from an ObservableCollection?
I have an ObservableCollection:

```<code>public ObservableCollection<NurDeviceWatcherInfo> KnownDevices = new ObservableCollection<NurDeviceWatcherInfo>();
</code>```

NurDeviceWatcherInfo has these string properties: Address, ConnState, Info, Name, Spec, SpecStr, Tag, TransportType.

What I am trying to use is:

```<code>NurDeviceWatcherInfo device = from NurDeviceWatcherInfo in KnownDevices
                                where NurDeviceWatcherInfo.Name.Contains("NUR")
                                select new NurDeviceWatcherInfo();
</code>```

For which I receive the following error:

Severity   Code    Description Project File    Line    Suppression State
  Error   CS0266  Cannot implicitly convert type 'System.Collections.Generic.IEnumerable' to 'NurApiDotNet.UWP.NurDeviceWatcherInfo'. An explicit conversion exists (are you missing a cast?)

What would be the correct syntax?

答:
I solved it using :</p>

```<code>device = KnownDevices.Where(x => x.Name.Contains("NUR")).FirstOrDefault();
</code>```

Body:
I have an ObservableCollection:

public ObservableCollection<NurDeviceWatcherInfo> KnownDevices = new ObservableCollection<NurDeviceWatcherInfo>();

NurDeviceWatcherInfo has these string properties: Address, ConnState, Info, Name, Spec, SpecStr, Tag, TransportType.

What I am trying to use is:

NurDeviceWatcherInfo device = from NurDeviceWatcherInfo in KnownDevices
                                where NurDeviceWatcherInfo.Name.Contains("NUR")
                                select new NurDeviceWatcherInfo();

For which I receive the following error:

Severity   Code    Description Project File    Line    Suppression State
  Error   CS0266  Cannot implicitly convert type 'System.Collections.Generic.IEnumerable' to 'NurApiDotNet.UWP.NurDeviceWatcherInfo'. An explicit conversion exists (are you missing a cast?)

What would be the correct syntax?

Body:
I solved it using :

device = KnownDevices.Where(x => x.Name.Contains("NUR")).FirstOrDefault();


问:
How to achieve instagram kind of image filter effect with Core Image
I have tried a few image filter effects in Core Image with SwiftUI. Like grouping different filters and connect relevant intensity with a Slider to change the filter's intensity.

I'm wondering is there a way to achieve the similar image filter effect like instagram. Because the single one of Core Image filter looks like not that fancy as instagram's. I guess chaining several effects in CI may achieve some of them on instagram, but I simply don't know which one to chain... No idea how to perform those filter effect according their name like "Sierra", "Willow".

So is there a way can achieve those filter effect in Core Image? Or some 3rd party library is needed? Any live example, project, framework/lib or hint is appreciated.

答:
Most of Instagram's filters are achieved by simple lookup tables, i.e. using a map (image) for replacing colors with a mapped color. You can use the CIColorCube filter to achieve something like that in Core Image. You can search the internet for example lookup tables (they usually come in the form of images). There is probably also example code on how to bring them into the right format.

Another thing Instagram does is adding a vignette effect. There is also a Core Image filter for that: CIVignetteEffect.


问:
Dismiss keyboard on drag with TableView and UITextView doesn't work
I've a setup as shown in the screenshot below. I've set Keyboard to Dismiss on drag in IB for scroll view. It doesn't dismiss keyboard on drag in my project.

However it does work on a dummy test project so it seems there is something in my project which is causing it not to work. I can't figure out what that thing is.

Any help or insights on how to find out the issue and fix?

P.S. It's lot of code so I may not be able to paste all the code here for this VC. If you need any specific section code to see then let me know.

答: Try to set Dismiss interactively

Body: Try to set Dismiss interactively


问:
Find out web traffic source by the referrer header
I've honestly tried to find libraries (in Python) which would allow me to easily do this. Or to find some precise instructions how to parse the referrer header properly. But nothing so far.
Any ideas? Need to do this myself on the backed, not by importing data from Google Analytics or something. Thank you.</p>

答:
Use Flask API. The "request" object has a bounch of important information, such as the IP.
https://flask.palletsprojects.com/en/1.1.x/reqcontext/

You can you external services to check the location based in the IP, such as https://ipstack.com/.

Another, more complete option is using Google Analytics:
https://analytics.google.com/analytics/web/
You just add a tag to you HTML and Google does the rest!


问:
Add headers to @aspnet/signalr Javascript client
I'm using @aspnet/signalr Official Javascript client from This npm package

I wondering if there is the way to add Header configuration to the client connection header

How I build the connection

let connection = new signalR.HubConnectionBuilder()
  .withUrl(
    "https://some.signalr-host.com"
    )
  .build();

答:
Basically you can provide a custom HttpClient to the hubConnecitonBuilder which allows you do whatever you want with the signalR requests, it can be very interesting for the negotiation part which was behind an azure APIManagement for me:

```<code> let options: IHttpConnectionOptions = {
      httpClient: <HttpClientSignalR>{
        getCookieString: (url) => {
          return '';
        },
        post: (url, httpOptions) => {
          return this.http.post(url, null, {
            headers: httpOptions.headers,
            withCredentials: false
          })
            .toPromise()
            .then(
              res => {
                return {
                  statusCode: 200,
                  statusText: null,
                  content: JSON.stringify(res)
                };
              },
              err => err
            );
        }
      }
    }

    this._hubConnection = new HubConnectionBuilder()
      .withUrl(environment.host_url + this.routes.notification + '?userId=' + userId, options)
      .build();
</code>```

In my case, my custom headers (invisible here) are added by my angular app http interceptors which can now intercept the  @aspnet/signalr http requests and add proper headers, as I m using my app HttpClient to emit my request instead of the library one.

Body:
Basically you can provide a custom HttpClient to the hubConnecitonBuilder which allows you do whatever you want with the signalR requests, it can be very interesting for the negotiation part which was behind an azure APIManagement for me:

 let options: IHttpConnectionOptions = {
      httpClient: <HttpClientSignalR>{
        getCookieString: (url) => {
          return '';
        },
        post: (url, httpOptions) => {
          return this.http.post(url, null, {
            headers: httpOptions.headers,
            withCredentials: false
          })
            .toPromise()
            .then(
              res => {
                return {
                  statusCode: 200,
                  statusText: null,
                  content: JSON.stringify(res)
                };
              },
              err => err
            );
        }
      }
    }

    this._hubConnection = new HubConnectionBuilder()
      .withUrl(environment.host_url + this.routes.notification + '?userId=' + userId, options)
      .build();

In my case, my custom headers (invisible here) are added by my angular app http interceptors which can now intercept the  @aspnet/signalr http requests and add proper headers, as I m using my app HttpClient to emit my request instead of the library one.


问:
Python requiring an attribute be converted to a string
```<code>from sys import exit
from random import randint

class Map(object):

 def death():
    print quips[randint (0, len(quips)-1)]
    exit(1)

 def princess_lives_here():
    print "You see a beautiful Princess with a shiny crown."
    print "She offers you some cake."

    eat_it = raw_input(">")

    if eat_it == "eat it":
        print "You explode like a pinata full of frogs."
        print "The Princess cackles and eats the frogs. Yum!"
        return 'death'

    elif eat_it == "do not eat it":
        print "She throws the cake at you and it cuts off your head."
        print "The last thing you see is her munching on your face. Yum!"
        return 'death'

    elif eat_it == "make her eat it":
        print "The Princess screams as you cram the cake in her mouth."
        print "Then she smiles and cries and thank you for saving her."
        print "She points to a tiny door and says, 'The Koi needs cake too.'"
        print "She gives you the very last bit of cake and shoves you in."
        return 'gold_koi_pond'

    else:
        print "The Princess looks at you confused and just points at the cake."
        return 'princess_lives_here'

class Engine(object):

 def __init__(self, start, quips):
    self.quips = [
        "You died. You suck at this.",
        "Your mom would be proud, if she were smarter",
        "Such a luser.",
        "I have a small puppy that's better at this."
    ]
    self.start = start

 def play(self):
    next = self.start

    while True:
        print "\n-----"
        room = getattr(self, next)
        next = room()

m = Map()
e = Engine(m, "princess_lives_here")

e.play()
</code>```

Trace back i get in terminal is:

```<code>    Traceback (most recent call last):
  File "ec42.py", line 162, in <module>
    e.play()
  File "ec42.py", line 156, in play
    room = getattr(self, next)
TypeError: getattr(): attribute name must be string
</code>```

I have been working on this far too long, and just cannot nail it down. The main issue is getting the map class to run inside the engine class as an object. thanks in advance for the help.

答:
self.start is an instance of Map, but the second argument of getattr() must be a string, the name of the attribute.

Also, instance methods should have "self" as the first argument.

ContentLicense: CC BY-SA 3.0

Body:
from sys import exit
from random import randint

class Map(object):

 def death():
    print quips[randint (0, len(quips)-1)]
    exit(1)

 def princess_lives_here():
    print "You see a beautiful Princess with a shiny crown."
    print "She offers you some cake."

    eat_it = raw_input(">")

    if eat_it == "eat it":
        print "You explode like a pinata full of frogs."
        print "The Princess cackles and eats the frogs. Yum!"
        return 'death'

    elif eat_it == "do not eat it":
        print "She throws the cake at you and it cuts off your head."
        print "The last thing you see is her munching on your face. Yum!"
        return 'death'

    elif eat_it == "make her eat it":
        print "The Princess screams as you cram the cake in her mouth."
        print "Then she smiles and cries and thank you for saving her."
        print "She points to a tiny door and says, 'The Koi needs cake too.'"
        print "She gives you the very last bit of cake and shoves you in."
        return 'gold_koi_pond'

    else:
        print "The Princess looks at you confused and just points at the cake."
        return 'princess_lives_here'

class Engine(object):

 def __init__(self, start, quips):
    self.quips = [
        "You died. You suck at this.",
        "Your mom would be proud, if she were smarter",
        "Such a luser.",
        "I have a small puppy that's better at this."
    ]
    self.start = start

 def play(self):
    next = self.start

    while True:
        print "\n-----"
        room = getattr(self, next)
        next = room()

m = Map()
e = Engine(m, "princess_lives_here")

e.play()

Trace back i get in terminal is:

    Traceback (most recent call last):
  File "ec42.py", line 162, in <module>
    e.play()
  File "ec42.py", line 156, in play
    room = getattr(self, next)
TypeError: getattr(): attribute name must be string

I have been working on this far too long, and just cannot nail it down. The main issue is getting the map class to run inside the engine class as an object. thanks in advance for the help.


问:
How to reduce Android app size by compressing images?
I am developing an Android app which has hundreds of .jpg files (over 300) each one of around 40kB. I would like to know if there is a way of reducing the size of my app. I looked at a similar question here Reducing Android App Size, but the problem still exists. Is there perhaps a way to compress the images and decompress them in real time when needed, or any other way to make my app more space efficient while not sacrificing speed?</p>

答:
If you have used tinypng for every resource you did your best with this kind of solution. In general, it's better to use vector graphics where the general icon will be <1kb. Also, a vector resource can be animated. If it's quite simply bitmaps, you can generate them in code on demand. Also, you can divide your app by dynamic features and each will be downloaded on demand with their part of the resources.

Is there perhaps a way to compress the images and decompress them in real-time when needed?

There is no standard Android solution out of the box. Probably, you can write something on your own. But this looks like too much effort.

Still, the most practical solution: use vector graphics as max as possible, generate in code what you can generate, compress with tinypng the others. That should be enough or you should have a very good reason for making some extra work.

For more info about vector graphics in android. For standard vector graphic import right in the android studio.
Web-site where you can download icons and insert them into the project.


问:
How to make API request javascript code with using api_key?
I made API request in Python.

```<code>import requests

key = 'my key'

params = {
  "file1" : 'file'
}

headers = {
  "Authorization" : "Bearer {}".format(key)
}

r = requests.get('url', params=params, headers=headers)

json_response = r.content.decode("utf-8", "ignore")
writeFile =open('samples.json', 'w')
writeFile.write(json_response)
writeFile.close()
print(r.json())
</code>```

I want to make API request in javascript using ajax,

```<code><script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
<script>
$(document).ready(function (){
    var key = 'my key';
    $.ajax({
        type : 'GET',
        url : 'url',
        headers: {'Authorization': "Bearer "+ (key)},
        data : {"file1" : 'files'}
        ,success : function(result){
            console.log(result);
        }
    })

})
</script>
</code>```

But it doesn't work with this error.

Access to XMLHttpRequest at 'url?file1=files' from origin 'http://127.0.0.1:5500' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

And the server returns this status

"OPTIONS url?file1=file HTTP/1.1" 200 -

What should I do?

答: You can do this way, take out your key and header code and see if the same message persists, if it does, you might be aware that this problem should be solved on the applications backend (server-side), by managing cors access, if you don't have backend access, try using an extesion to mozzila's browser (firefox), called 'CORS Everywhere', turn it on and try again but keep mind that is just a temporary manner to solve this. Good Luck </p>

Body:
I made API request in Python.

import requests

key = 'my key'

params = {
  "file1" : 'file'
}

headers = {
  "Authorization" : "Bearer {}".format(key)
}

r = requests.get('url', params=params, headers=headers)

json_response = r.content.decode("utf-8", "ignore")
writeFile =open('samples.json', 'w')
writeFile.write(json_response)
writeFile.close()
print(r.json())

I want to make API request in javascript using ajax,

But it doesn't work with this error.

Access to XMLHttpRequest at 'url?file1=files' from origin 'http://127.0.0.1:5500' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

And the server returns this status

"OPTIONS url?file1=file HTTP/1.1" 200 -

What should I do?


问:
How to stop BERT from breaking apart specific words into word-piece
I am using a pre-trained BERT model to tokenize a text into meaningful tokens. However, the text has many specific words and I don't want BERT model to break them into word-pieces. Is there any solution to it?
For example:

```<code>tokenizer = BertTokenizer('bert-base-uncased-vocab.txt')
tokens = tokenizer.tokenize("metastasis")
</code>```

Create tokens like this:

```<code>['meta', '##sta', '##sis']
</code>```

However, I want to keep the whole words as one token, like this:

```<code>['metastasis']
</code>```

答:
Based on the discussion here, one way to use my own additional vocabulary dictionary which is containing the specific words is to modify the first ~1000 lines of the vocab.txt file ([unused] lines) with the specific words. For example I replaced '[unused1]' with 'metastasis' in the vocab.txt and after tokenization with the modified vocab.txt I got this output:</p>

```<code>tokens = tokenizer.tokenize("metastasis")
Output: ['metastasis']
</code>```

Body:
I am using a pre-trained BERT model to tokenize a text into meaningful tokens. However, the text has many specific words and I don't want BERT model to break them into word-pieces. Is there any solution to it?
For example:

tokenizer = BertTokenizer('bert-base-uncased-vocab.txt')
tokens = tokenizer.tokenize("metastasis")

Create tokens like this:

['meta', '##sta', '##sis']

However, I want to keep the whole words as one token, like this:

['metastasis']

Body:
Based on the discussion here, one way to use my own additional vocabulary dictionary which is containing the specific words is to modify the first ~1000 lines of the vocab.txt file ([unused] lines) with the specific words. For example I replaced '[unused1]' with 'metastasis' in the vocab.txt and after tokenization with the modified vocab.txt I got this output:

tokens = tokenizer.tokenize("metastasis")
Output: ['metastasis']


问:
Ambari on Docker Cloudera repo file needs user and password now
I had an script that automatically would create an Ambari cluster using docker. In the file:
https://github.com/ortizfabio/caochong/blob/master/from-ambari/Dockerfile
There is a line to download Ambari. This used to work until Cloudera took over. I am trying to change the command that download Ambari from:

```<code>RUN wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.6.0.0/ambari.repo
</code>```

to:

```<code>RUN wget --http-user=email_logon --http-password=my_password  -O /etc/yum.repos.d/ambari.repo https://archive.cloudera.com/p/ambari/centos7/2.x/updates/2.7.5.0/ambari.repo
</code>```

However I get error:

```<code>Connecting to archive.cloudera.com (archive.cloudera.com)|151.101.208.167|:443... connected.
HTTP request sent, awaiting response... 401 Authentication required
Authentication selected: Basic realm=Secured
Connecting to archive.cloudera.com (archive.cloudera.com)|151.101.208.167|:443... connected.
HTTP request sent, awaiting response... 403 Authentication failed
2020-05-29 00:35:40 ERROR 403: Authentication failed.
</code>```

答:
@Fabio  Your options are:

Contact Cloudera, become a customer, and get an Enterprise License with user and password authorized to access their paywalled versions

Use the last free version 2.7.4

Build From Source

If you are looking to use HDP,  I suggest #2 as the easiest option:

```<code>wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.4.0/ambari.repo
</code>```

If you truly want 2.7.5 you can build from source yourself, or I have created some RPMS you can use without HDP:

```<code>wget -O /etc/yum.repos.d/mosga.repo https://makeopensourcegreatagain.com/rpms/mosga.repo
yum install ambari-server ambari-agent -y
ambari-server setup -s
</code>```

I am currently still working on the next part where my RPMS will install DDP (Demo Data Platform) with common HDP Components and some other Third Party Components we use here at DFHz.

Body:
I had an script that automatically would create an Ambari cluster using docker. In the file:
https://github.com/ortizfabio/caochong/blob/master/from-ambari/Dockerfile
There is a line to download Ambari. This used to work until Cloudera took over. I am trying to change the command that download Ambari from:

RUN wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.6.0.0/ambari.repo

to:

RUN wget --http-user=email_logon --http-password=my_password  -O /etc/yum.repos.d/ambari.repo https://archive.cloudera.com/p/ambari/centos7/2.x/updates/2.7.5.0/ambari.repo

However I get error:

Connecting to archive.cloudera.com (archive.cloudera.com)|151.101.208.167|:443... connected.
HTTP request sent, awaiting response... 401 Authentication required
Authentication selected: Basic realm=Secured
Connecting to archive.cloudera.com (archive.cloudera.com)|151.101.208.167|:443... connected.
HTTP request sent, awaiting response... 403 Authentication failed
2020-05-29 00:35:40 ERROR 403: Authentication failed.

Body:
@Fabio  Your options are:

Contact Cloudera, become a customer, and get an Enterprise License with user and password authorized to access their paywalled versions

Use the last free version 2.7.4

Build From Source

If you are looking to use HDP,  I suggest #2 as the easiest option:

wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.4.0/ambari.repo

If you truly want 2.7.5 you can build from source yourself, or I have created some RPMS you can use without HDP:

wget -O /etc/yum.repos.d/mosga.repo https://makeopensourcegreatagain.com/rpms/mosga.repo
yum install ambari-server ambari-agent -y
ambari-server setup -s

I am currently still working on the next part where my RPMS will install DDP (Demo Data Platform) with common HDP Components and some other Third Party Components we use here at DFHz.


问:
The scanf is not working after the terminal printed out the first character
I was making a code and I want to take the first character of the string and edit it out, but the code is not working like it should: </p>

```<code>#include <stdio.h>
#include <string.h>

#define MAX_lIMIT 100

int main() {
    char link[MAX_lIMIT];
    int length;
    int codex;

    scanf("%s", link);
    printf("%s, Write the new version of the script.\n", link);
    length = strlen(link);
    printf("%c", link[0]);

    scanf("%c", &link[0]);
    printf("%c", link);
}
</code>```

答:
When scanf("%c", &link[0]); reads the next byte from stdin, it gets the pending newline left by the previous call scanf("%s", link);.

You should use scanf(" %c", &link[0]); to first skip any white space and read the next character.

There are other problems in your code:

MAX_lIMIT should probably be MAX_LIMIT

scanf("%s", link); may overflow the destination array if more than 99 characters are typed without whitespace. Use scanf("%99s", link);

always test the return value of scanf()

printf("%c", link[0]);` should probably also print a newline for the output to be visible to the user.

printf("%c", link);is incorrect, it should beprintf("%s\n", link);`

Here is a modified version:

#include <stdio.h>

#define MAX_LIMIT 100

int main() {
    char link[MAX_LIMIT];

    if (scanf("%99s", link) != 1)
        return 1;
    printf("%s, Write the new version of the script.\n", link);
    printf("%c\n", link[0]);

    if (scanf(" %c", &link[0]) != 1)
        return 1;
    printf("%s\n", link);
    return 0;
}
```

Body:
I was making a code and I want to take the first character of the string and edit it out, but the code is not working like it should:

#include <stdio.h>
#include <string.h>

#define MAX_lIMIT 100

int main() {
    char link[MAX_lIMIT];
    int length;
    int codex;

    scanf("%s", link);
    printf("%s, Write the new version of the script.\n", link);
    length = strlen(link);
    printf("%c", link[0]);

    scanf("%c", &link[0]);
    printf("%c", link);
}


问:
Apollo Federation Gateway can't load when parent service goes down
I have 3 services: Org, Gatekeeper and Fleet. Gatekeeper and Fleet extend the Org schema from the Org service. then the Org server is up and running along with fleet and gatekeeper, everything works fine, but if org is down and I try run gateway with just fleet and gatekeeper, I get an error "Unknown Type: Org"

I read in the API reference for apollo that gatekeeper / fleet should run standalone, but does that mean only if I spin up a gatekeeper server? The standalone server DOES work, but not through the gateway if a parent service is down. Is there a way to get it to work?

答:
The Apollo docs mean that you should be able to run each of those services independently. For example, imagine that you had only developed gatekeeper and not fleet/org. You can test this by running the server standalone and querying/mutating it. Everything that federation adds to the schema "can be ignored".

However, once you try to run that in a federated environment it cannot be ignored. That is because the directives and other spec changes have meaning to the gateway. They will all have to be up for gateway to work.