问:
What is the mcrypt_get_block alternative and how to use it
I have an old wordpress plugin built using symfony framework and which uses mcrypt_get_block_size('tripledes', 'ecb'). However, I have just upgraded php on my server to version 7.3 and now I am getting a Fatal error: Call to undefined function HealthShieldForm\mcrypt_encrypt().

I have read few articles where they suggest using openssl_encrypt(). Unfortunately I dont know where to begin in terms of replacing the functions with eachother and insure I am getting the same results.

My current code is this:

```<code>private function encrypt($data)
    {
        //Pad for PKCS7
        $blockSize = mcrypt_get_block_size('tripledes', 'ecb');
        $len = strlen($data);
        $pad = $blockSize - ($len % $blockSize);
        $data .= str_repeat(chr($pad), $pad);

        //Encrypt data
        $encData = mcrypt_encrypt('tripledes', $this->key, $data, 'ecb');

        return base64_encode($encData);
    }

    private function decrypt($data)
    {
        $data = base64_decode($data);

        $data = mcrypt_decrypt('tripledes', $this->key, $data, 'ecb');

        $block = mcrypt_get_block_size('tripledes', 'ecb');
        $len = strlen($data);
        $pad = ord($data[$len - 1]);

        return substr($data, 0, strlen($data) - $pad);
    }
</code>```

UPDATE

I have adapted Peter's example to reflect on my code:

```<code>function encrypt($data)
    {
        $ciphertext = openssl_encrypt($data, 'des-ede3-ecb', $this->key, OPENSSL_RAW_DATA);
        return base64_encode($ciphertext);
    }

    function decrypt($data)
    {
        $ciphertext = base64_decode($data);
        $plaintext = openssl_decrypt($ciphertext, 'des-ede3-ecb', $this->key, OPENSSL_RAW_DATA);
        return $plaintext;
    }
</code>```

However, now I am getting this error: Warning: openssl_encrypt(): Unknown cipher algorithm

答:
First of all, this may be the perfect opportunity to update your crypto. Triple DES is generally fine, but pretty much only used because it can be implemented very efficiently in hardware (such as smart cards). For servers and PCs and such AES has equal if not better performance characteristics (modern CPUs have special instructions for AES due to its ubiquitous use in TLS).

Do not use ECB if you can help it, no matter which block cipher you choose. You can see penguins through it.

If you are free to choose, use libsodium and pick one of the AEAD constructions (the PHP docs are pretty much non-existant, but you can fairly easily correlate the official docs with the matching wrapper functions in PHP).

In can only recommend against using OpenSSL (it has countless sharp edges; won't even tell you if your key is too short, for instance), but in case you insist on rolling with it:

OpenSSL does not have a function that returns the block size for a given cipher. However, you also don't need it here because openssl_encrypt applies PKCS#7 by default:

```<code>function encrypt(string $plaintext, string $key): string
{
    $ciphertext = openssl_encrypt($plaintext, 'des-ede3-ecb', $key, OPENSSL_RAW_DATA);
    return base64_encode($ciphertext);
}

function decrypt(string $ciphertext64, string $key): string
{
    $ciphertext = base64_decode($ciphertext64)
    $plaintext = openssl_decrypt($ciphertext, 'des-ede3-ecb', $key, OPENSSL_RAW_DATA);
    return $plaintext;
}
</code>```

Body:
I have an old wordpress plugin built using symfony framework and which uses mcrypt_get_block_size('tripledes', 'ecb'). However, I have just upgraded php on my server to version 7.3 and now I am getting a Fatal error: Call to undefined function HealthShieldForm\mcrypt_encrypt().

I have read few articles where they suggest using openssl_encrypt(). Unfortunately I dont know where to begin in terms of replacing the functions with eachother and insure I am getting the same results.

My current code is this:

private function encrypt($data)
    {
        //Pad for PKCS7
        $blockSize = mcrypt_get_block_size('tripledes', 'ecb');
        $len = strlen($data);
        $pad = $blockSize - ($len % $blockSize);
        $data .= str_repeat(chr($pad), $pad);

        //Encrypt data
        $encData = mcrypt_encrypt('tripledes', $this->key, $data, 'ecb');

        return base64_encode($encData);
    }

    private function decrypt($data)
    {
        $data = base64_decode($data);

        $data = mcrypt_decrypt('tripledes', $this->key, $data, 'ecb');

        $block = mcrypt_get_block_size('tripledes', 'ecb');
        $len = strlen($data);
        $pad = ord($data[$len - 1]);

        return substr($data, 0, strlen($data) - $pad);
    }

UPDATE

I have adapted Peter's example to reflect on my code:

function encrypt($data)
    {
        $ciphertext = openssl_encrypt($data, 'des-ede3-ecb', $this->key, OPENSSL_RAW_DATA);
        return base64_encode($ciphertext);
    }

    function decrypt($data)
    {
        $ciphertext = base64_decode($data);
        $plaintext = openssl_decrypt($ciphertext, 'des-ede3-ecb', $this->key, OPENSSL_RAW_DATA);
        return $plaintext;
    }

However, now I am getting this error: Warning: openssl_encrypt(): Unknown cipher algorithm

Body:
First of all, this may be the perfect opportunity to update your crypto. Triple DES is generally fine, but pretty much only used because it can be implemented very efficiently in hardware (such as smart cards). For servers and PCs and such AES has equal if not better performance characteristics (modern CPUs have special instructions for AES due to its ubiquitous use in TLS).

Do not use ECB if you can help it, no matter which block cipher you choose. You can see penguins through it.

If you are free to choose, use libsodium and pick one of the AEAD constructions (the PHP docs are pretty much non-existant, but you can fairly easily correlate the official docs with the matching wrapper functions in PHP).

In can only recommend against using OpenSSL (it has countless sharp edges; won't even tell you if your key is too short, for instance), but in case you insist on rolling with it:

OpenSSL does not have a function that returns the block size for a given cipher. However, you also don't need it here because openssl_encrypt applies PKCS#7 by default:

function encrypt(string $plaintext, string $key): string
{
    $ciphertext = openssl_encrypt($plaintext, 'des-ede3-ecb', $key, OPENSSL_RAW_DATA);
    return base64_encode($ciphertext);
}

function decrypt(string $ciphertext64, string $key): string
{
    $ciphertext = base64_decode($ciphertext64)
    $plaintext = openssl_decrypt($ciphertext, 'des-ede3-ecb', $key, OPENSSL_RAW_DATA);
    return $plaintext;
}


来源: stackexchange

问:
Spark 2.2 Sort fails with huge dataset
I am facing an issue when sorting a huge dataset (1.2 T) based on 4 columns. I also need right after the sort, to partition this dataset when writing the final dataset in HDFS ,based on one of the columns used in the sort function.

Here is a stackoverflow post I posted a few days ago describing an other issue I had with the same code but with regards to joining two datasets :

previous issue

I used the answer of this post to improve my code. Now the join works fine.

I tested the code without the sort and it works fine. In order to perform the sort, I thought about partitioning the data based on the four columns.

The size of one partition is 500MB. I have then 2600=1.2T/500MB partitions.

When executing the spark job, I get an shuffle.RetryingBlockFetcher error (see the error logs below).

My questions are :

What is the best way to sort data in spark to avoid shuffles ? Or reducing it ?

Could I correct/add improvements to my code in order to perform the sort ?

Do I really have to sort this way ? Can't I use other techniques like a Group By ?

Snippet of my code :

EDITED

```<code>    val uh = uh_months
      .withColumn("UHDIN", datediff(to_date(unix_timestamp(col("UHDIN_YYYYMMDD"), "yyyyMMdd").cast(TimestampType)),
        to_date(unix_timestamp(col("january"), "yyyy-MM-dd").cast(TimestampType))))
      //      .withColumn("DVA_1", to_date((unix_timestamp(col("DVA"), "ddMMMyyyy")).cast(TimestampType)))
      .withColumn("DVA_1", date_format(col("DVA"), "dd/MM/yyyy"))
      .drop("UHDIN_YYYYMMDD")
      .drop("january")
      .drop("DVA").repartition(1300,col("MMED"),col("DEBCRED"),col("NMTGP"))//.repartition(1300,col("NO_NUM"))

    val uh_flag_comment = new TransactionType().transform(uh)
    val uh_repartitioned = uh_flag_comment.repartition(1300,col("NO_NUM"))

    val uh_joined = uh_repartitioned.join(broadcast(smallDF), "NO_NUM")
      .select(
        uh.col("*"),
        smallDF.col("PSP"),
        smallDF.col("minrel"),
        smallDF.col("Label"),
        smallDF.col("StartDate"))
      .withColumnRenamed("DVA_1", "DVA")

    val uh_final = uh_joined.repartition(1300, col("PSP")).sortWithinPartitions(col("NO_NUM"), col("UHDIN"), col("HOURMV"))

    return uh_final
</code>```

TransactionType is a class where I add a new column to my uh dataframe based on the value of 3 columns (MMED, DEBCRED, NMTGP), using regex.

Without the sort, and using the full capacity of the cluster, the code runs within 1 hour approximately.

Execution plan

```<code>== Physical Plan ==
Exchange hashpartitioning(PSP#82, 2600)
+- *Sort [PSP#82 ASC NULLS FIRST, NO_NUM#252 ASC NULLS FIRST, UHDIN#547 ASC NULLS FIRST, HOURMV#175 ASC NULLS FIRST], true, 0
   +- Exchange rangepartitioning(PSP#82 ASC NULLS FIRST, NO_NUM#252 ASC NULLS FIRST, UHDIN#547 ASC NULLS FIRST, HOURMV#175 ASC NULLS FIRST, 200)
      +- Exchange hashpartitioning(PSP#82, NO_NUM#252, UHDIN#547, HOURMV#175, 2600)
         +- *Project [NO_NUM#252, DEV#153, DEBCRED#154, BDGRORI#155, BDGREUR#156, BEWC#157, MSG30_NL#158, SCAPMV#159, USERID#160, MMED#161, TNUM#162, NMTGP#163, BKA#164, CATEXT#165, SEQETAT#166, ACCTYPE#167, BRAND#168, FAMILY#169, SUBFAMILY#170, FORCED_DVA#172, BYBANK#173, CPTE_PROTEGE#174, HOURMV#175, RDFB#176, ... 30 more fields]
            +- *BroadcastHashJoin [NO_NUM#252], [NO_NUM#13], Inner, BuildRight
               :- Exchange hashpartitioning(NO_NUM#252, 1300)
               :  +- *Project [NUM#152 AS NO_NUM#252, DEV#153, DEBCRED#154, BDGRORI#155, BDGREUR#156, BEWC#157, MSG30_NL#158, SCAPMV#159, USERID#160, MMED#161, TNUM#162, NMTGP#163, BKA#164, CATEXT#165, SEQETAT#166, ACCTYPE#167, BRAND#168, FAMILY#169, SUBFAMILY#170, FORCED_DVA#172, BYBANK#173, CPTE_PROTEGE#174, HOURMV#175, RDFB#176, ... 26 more fields]
               :     +- *Filter (BEWC#157 INSET (25003,25302,25114,20113,12017,20108,25046,12018,15379,15358,11011,20114,10118,12003,25097,20106,20133,10133,10142,15402,25026,25345,28023,15376,25019,28004,21701,25001,11008,15310,15003,2SOMEPORT,22048,15470,25300,25514,25381,25339,15099,25301,28005,28026,25098,25018,15323,25376,15804,15414,25344,25102,15458,15313,28002,25385,22051,25214,15031,12005,15425,20145,22011,15304,25027,14020,11007,25901,15343,22049,20112,12031,20127,15339,25421,15432,28025,25340,25325,20150,28011,25368,25304,22501,25369,28022,15098,12032,15375,25002,25008,10116,10101,22502,25090,15004,20105,12030,22503,15095,22007,15809,15342,15311,25216,10103,20122,11019,20142,15097,20147,20149,25005,25205,25380,15380,10120,25015,15384,11003,10110,25016,15090,25307,15001,25390,15312,10115,25219,15806,15459,12016,15359,15395,15302,12021,11701,10111,10148,25379,15807,10102,25352,25355,12010,25095,25394,20101,25413,15385,25322,28027,11026,15533,25201,25371,10128,11028,12020,15819,10143,28028,10123,10125,11020,25029,10122,25343,15015,12033,25014,12012,25024,25375,11023,25501,25402,22001,15317,12014,16114,20501,15046,12001,12022,10104,10117,12002,25499,10145,10153,12011,15350,15300,10119,25305,15345,25374,11027,25430,28021,25202,10121,28024,25101,28001,15321,11025,25358,15333,15501,25533,15372,12008,11015,10114,10113,10112,15303,15320,28006,22002,25359,10132,15497,25353,11029,25425,15374,12019,25437,11022,15357,20148,20111,26114,25099,25354,10124,25303,11010,20120,20135,15820,15331,28029) && isnotnull(NUM#152))
               :        +- *FileScan csv [UHDIN_YYYYMMDD#151,NUM#152,DEV#153,DEBCRED#154,BDGRORI#155,BDGREUR#156,BEWC#157,MSG30_NL#158,SCAPMV#159,USERID#160,MMED#161,TNUM#162,NMTGP#163,BKA#164,CATEXT#165,SEQETAT#166,ACCTYPE#167,BRAND#168,FAMILY#169,SUBFAMILY#170,DVA#171,FORCED_DVA#172,BYBANK#173,CPTE_PROTEGE#174,... 26 more fields] Batched: false, Format: CSV, Location: InMemoryFileIndex[SOMEHOST:SOMEPORT/SOMEPATH, PartitionFilters: [], PushedFilters: [In(BEWC, [25003,25302,25114,20113,12017,20108,25046,12018,15379,15358,11011,20114,10118,12003,25..., ReadSchema: struct<UHDIN_YYYYMMDD:string,NUM:string,DEV:string,DEBCRED:string,BDGRORI:string,BDGREUR:string,B...
               +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, true]))
                  +- *Project [NO_NUM#13, PSP#82, minrel#370, Label#105, StartDate#106]
                     +- *SortMergeJoin [PSP#381], [PSP#82], Inner
                        :- *Sort [PSP#381 ASC NULLS FIRST], false, 0
                        :  +- Exchange hashpartitioning(PSP#381, 200)
                        :     +- *Project [PSP#381, NO_NUM#13, minrel#370]
                        :        +- SortMergeJoin [PSP#381, C_SNUM#14, minrel#370, NO_NUM#13], [NO_PSP#47, C_SNUM_1#387, C_NRELPR#50, NO_NUM_1#400], LeftOuter
                        :           :- *Sort [PSP#381 ASC NULLS FIRST, C_SNUM#14 ASC NULLS FIRST, minrel#370 ASC NULLS FIRST, NO_NUM#13 ASC NULLS FIRST], false, 0
                        :           :  +- Exchange hashpartitioning(PSP#381, C_SNUM#14, minrel#370, NO_NUM#13, 200)
                        :           :     +- SortAggregate(key=[NO_PSP#12, C_SNUM#14, NO_NUM#13], functions=[min(C_NRELPR#15)])
                        :           :        +- *Sort [NO_PSP#12 ASC NULLS FIRST, C_SNUM#14 ASC NULLS FIRST, NO_NUM#13 ASC NULLS FIRST], false, 0
                        :           :           +- Exchange hashpartitioning(NO_PSP#12, C_SNUM#14, NO_NUM#13, 200)
                        :           :              +- SortAggregate(key=[NO_PSP#12, C_SNUM#14, NO_NUM#13], functions=[partial_min(C_NRELPR#15)])
                        :           :                 +- *Sort [NO_PSP#12 ASC NULLS FIRST, C_SNUM#14 ASC NULLS FIRST, NO_NUM#13 ASC NULLS FIRST], false, 0
                        :           :                    +- *Project [NO_PSP#12, C_SNUM#14, NO_NUM#13, C_NRELPR#15]
                        :           :                       +- *Filter (((C_NRELPR#15 IN (001,006) && C_SNUM#14 IN (030,033)) && isnotnull(NO_NUM#13)) && isnotnull(NO_PSP#12))
                        :           :                          +- *FileScan csv [NO_PSP#12,NO_NUM#13,C_SNUM#14,c_nrelpr#15] Batched: false, Format: CSV, Location: InMemoryFileIndex[SOMEHOST:SOMEPORT/SOMEPATH, PartitionFilters: [], PushedFilters: [In(c_nrelpr, [001,006]), In(C_SNUM, [030,033]), IsNotNull(NO_NUM), IsNotNull(NO_PSP)], ReadSchema: struct<NO_PSP:string,NO_NUM:string,C_SNUM:string,c_nrelpr:string>
                        :           +- *Sort [NO_PSP#47 ASC NULLS FIRST, C_SNUM_1#387 ASC NULLS FIRST, C_NRELPR#50 ASC NULLS FIRST, NO_NUM_1#400 ASC NULLS FIRST], false, 0
                        :              +- Exchange hashpartitioning(NO_PSP#47, C_SNUM_1#387, C_NRELPR#50, NO_NUM_1#400, 200)
                        :                 +- *Project [NO_PSP#47, NO_NUM#48 AS NO_NUM_1#400, C_SNUM#49 AS C_SNUM_1#387, c_nrelpr#50]
                        :                    +- *FileScan csv [NO_PSP#47,NO_NUM#48,C_SNUM#49,c_nrelpr#50] Batched: false, Format: CSV, Location: InMemoryFileIndex[SOMEHOST:SOMEPORT/SOMEPATH, PartitionFilters: [], PushedFilters: [], ReadSchema: struct<NO_PSP:string,NO_NUM:string,C_SNUM:string,c_nrelpr:string>
                        +- *Sort [PSP#82 ASC NULLS FIRST], false, 0
                           +- Exchange hashpartitioning(PSP#82, 200)
                              +- *Project [PSP#82, Label#105, StartDate#106]
                                 +- *Filter isnotnull(PSP#82)
                                    +- *FileScan csv [PSP#82,Label#105,StartDate#106] Batched: false, Format: CSV, Location: InMemoryFileIndex[SOMEHOST:SOMEPORT/SOMEPATH, PartitionFilters: [], PushedFilters: [IsNotNull(PSP)], ReadSchema: struct<PSP:string,Label:string,StartDate:string>

</code>```

Here is the main error(s) I got when launching the job with the sort :

```<code>19/05/06 18:02:25 ERROR shuffle.RetryingBlockFetcher: Exception while beginning fetch of 214 outstanding blocks
java.io.IOException: Failed to connect to SOMEHOST/SOMEADDRESS:SOMEPORT
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:232)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:182)
    at org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:98)
    at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:141)
    at org.apache.spark.network.shuffle.RetryingBlockFetcher.start(RetryingBlockFetcher.java:121)
    at org.apache.spark.network.netty.NettyBlockTransferService.fetchBlocks(NettyBlockTransferService.scala:108)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.sendRequest(ShuffleBlockFetcherIterator.scala:228)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.fetchUpToMaxBytes(ShuffleBlockFetcherIterator.scala:435)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.initialize(ShuffleBlockFetcherIterator.scala:323)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.<init>(ShuffleBlockFetcherIterator.scala:140)
    at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:45)
    at org.apache.spark.sql.execution.ShuffledRowRDD.compute(ShuffledRowRDD.scala:165)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: SOMEHOST/SOMEADDRESS:SOMEPORT
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:631)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
    ... 1 more
19/05/06 18:02:25 INFO shuffle.RetryingBlockFetcher: Retrying fetch (1/3) for 214 outstanding blocks after 5000 ms
19/05/06 18:02:25 INFO storage.ShuffleBlockFetcherIterator: Started 6 remote fetches in 13 ms
19/05/06 18:02:28 INFO executor.Executor: Finished task 408.0 in stage 14.0 (TID 6696). 1733 bytes result sent to driver
19/05/06 18:02:28 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 6816
19/05/06 18:02:28 INFO executor.Executor: Running task 466.1 in stage 14.0 (TID 6816)
19/05/06 18:02:28 INFO storage.ShuffleBlockFetcherIterator: Getting 5073 non-empty blocks out of 5089 blocks
19/05/06 18:02:28 INFO client.TransportClientFactory: Found inactive connection to SOMEHOST/SOMEADDRESS:SOMEPORT, creating a new one.
19/05/06 18:02:28 ERROR shuffle.RetryingBlockFetcher: Exception while beginning fetch of 82 outstanding blocks
java.io.IOException: Failed to connect to SOMEHOST/SOMEADDRESS:SOMEPORT
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:232)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:182)
    at org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:98)
    at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:141)
    at org.apache.spark.network.shuffle.RetryingBlockFetcher.start(RetryingBlockFetcher.java:121)
    at org.apache.spark.network.netty.NettyBlockTransferService.fetchBlocks(NettyBlockTransferService.scala:108)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.sendRequest(ShuffleBlockFetcherIterator.scala:228)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.fetchUpToMaxBytes(ShuffleBlockFetcherIterator.scala:435)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.initialize(ShuffleBlockFetcherIterator.scala:323)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.<init>(ShuffleBlockFetcherIterator.scala:140)
    at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:45)
    at org.apache.spark.sql.execution.ShuffledRowRDD.compute(ShuffledRowRDD.scala:165)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: SOMEHOST/SOMEADDRESS:SOMEPORT
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:631)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
    ... 1 more
</code>```

EDITED

An other type of error :

```<code>19/05/06 18:06:16 ERROR executor.Executor: Exception in task 309.1 in stage 13.1 (TID 7592)
java.io.FileNotFoundException: /applis/hadoop/yarn/local/usercache/MYUSER/appcache/application_1555263602441_0123/blockmgr-aa586b76-ff58-4f88-b168-288c3e1b9f61/3c/temp_shuffle_ea967624-f633-4481-9a05-249b561e3c38 (No such file or directory)
    at java.io.FileInputStream.open0(Native Method)
    at java.io.FileInputStream.open(FileInputStream.java:195)
    at java.io.FileInputStream.<init>(FileInputStream.java:138)
    at org.spark_project.guava.io.Files$FileByteSource.openStream(Files.java:124)
    at org.spark_project.guava.io.Files$FileByteSource.openStream(Files.java:114)
    at org.spark_project.guava.io.ByteSource.copyTo(ByteSource.java:202)
    at org.spark_project.guava.io.Files.copy(Files.java:436)
    at org.spark_project.guava.io.Files.move(Files.java:651)
    at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.mergeSpills(UnsafeShuffleWriter.java:277)
    at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.closeAndWriteOutput(UnsafeShuffleWriter.java:216)
    at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:169)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
19/05/06 18:06:16 ERROR executor.Executor: Exception in task 502.1 in stage 13.1 (TID 7599)
java.io.FileNotFoundException: /applis/hadoop/yarn/local/usercache/MYUSER/appcache/application_1555263602441_0123/blockmgr-aa586b76-ff58-4f88-b168-288c3e1b9f61/34/temp_shuffle_dd202cd1-ad8f-41c4-b4d1-d79621cd169e (No such file or directory)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:102)
    at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)
    at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:247)
    at org.apache.spark.shuffle.sort.ShuffleExternalSorter.writeSortedFile(ShuffleExternalSorter.java:201)
    at org.apache.spark.shuffle.sort.ShuffleExternalSorter.closeAndGetSpills(ShuffleExternalSorter.java:405)
    at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.closeAndWriteOutput(UnsafeShuffleWriter.java:209)
    at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:169)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
19/05/06 18:06:16 INFO executor.Executor: Finished task 200.2 in stage 13.1 (TID 7568). 2826 bytes result sent to driver
19/05/06 18:06:16 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
19/05/06 18:06:16 INFO util.ShutdownHookManager: Shutdown hook called
</code>```

Some information/context:

I am working on a production environment (see the cluster configuration below). I cannot upgrade my spark version. I do not have spark UI or yarn UI to monitor my jobs. All I can retrieve are the yarn logs.

Spark Version: 2.2

Cluster configuration:

21 compute nodes (workers)

8 cores each

64 GB RAM per node

Current Spark configuration:

-master: yarn

-executor-memory: 42G

-executor-cores: 5

-driver memory: 42G

-num-executors: 32

-spark.sql.broadcastTimeout=3600

-spark.kryoserializer.buffer.max=512

-spark.yarn.executor.memoryOverhead=2400

-spark.driver.maxResultSize=500m

-spark.memory.storageFraction=0.3

-spark.memory.fraction=0.9

-spark.hadoop.fs.permissions.umask-mode=007

How is the job executed:

We build an artifact (jar) with IntelliJ and then send it to a server. Then a bash script is executed. This script:

export some environment variables (SPARK_HOME, HADOOP_CONF_DIR, PATH and SPARK_LOCAL_DIRS)

launch the spark-submit command with all the parameters defined in the spark configuration above

retrieves the yarn logs of the application

答:
I managed to sort the data successfully (and then to run the entire code) using a significant part of the answer posted by @Alexandros.

I however did some changes in terms of configurations in the cluster :

I increased the Executor memory to 45 G (it was 42 G)

I changed the spark parameter --executor-memory increasing it to 45G instead of 42G

I increased the number of executors to 40

I also increased on each node the disk space of /applis/hadoop/yarn/local/usercache/MYUSER/, adding 20-25G (there was a bit less than 50 G available on each node for this folder). This is the yarn usercache where Spark writes intermediary shuffled blocks of data. Since I have a dataset of 1.2T, and I have 21 nodes, I need approximately 60-65G of disk space on each node when the data is distributed across the nodes.

I also used the sortWithinPartition function (it was working fine with this function but it fails with the classic sort function). Besides, I only need to sort each partition since I partition based on PSP (if the dataset is not sorted based on PSP, it's fine).

Below is the code :

```<code>val uh = uh_months
  .withColumn("UHDIN", datediff(to_date(unix_timestamp(col("UHDIN_YYYYMMDD"), "yyyyMMdd").cast(TimestampType)),
    to_date(unix_timestamp(col("january"), "yyyy-MM-dd").cast(TimestampType))))
  //      .withColumn("DVA_1", to_date((unix_timestamp(col("DVA"), "ddMMMyyyy")).cast(TimestampType)))
  .withColumn("DVA_1", date_format(col("DVA"), "dd/MM/yyyy"))
  .drop("UHDIN_YYYYMMDD")
  .drop("january")
  .drop("DVA")
  .repartition(3000, col("NO_NUM"))
  .persist()

val uh_flag_comment = new TransactionType().transform(uh)

val uh_joined = uh_flag_comment.join(broadcast(smallDF), "NO_NUM")
  .select(
    uh_flag_comment.col("*"),
    kl_holdmand_pruned.col("PSP"),
    kl_holdmand_pruned.col("minrel"),
    kl_holdmand_pruned.col("TerroLabel"),
    kl_holdmand_pruned.col("TerroStartDate"))
  .withColumnRenamed("DVA_1", "DVA")

smallDF.unpersist()
uh.unpersist()

val uh_to_be_sorted = uh_joined.repartition(3000, col("PSP"))
val uh_final = uh_to_be_sorted.sortWithinPartitions(col("NO_NUM"), col("UHDIN"), col("HOURMV"))

uh_final
</code>```

ContentLicense: CC BY-SA 4.0

Title: Spark 2.2 Sort fails with huge dataset

Body:
I am facing an issue when sorting a huge dataset (1.2 T) based on 4 columns. I also need right after the sort, to partition this dataset when writing the final dataset in HDFS ,based on one of the columns used in the sort function.

Here is a stackoverflow post I posted a few days ago describing an other issue I had with the same code but with regards to joining two datasets :

previous issue

I used the answer of this post to improve my code. Now the join works fine.

I tested the code without the sort and it works fine. In order to perform the sort, I thought about partitioning the data based on the four columns.

The size of one partition is 500MB. I have then 2600=1.2T/500MB partitions.

When executing the spark job, I get an shuffle.RetryingBlockFetcher error (see the error logs below).

My questions are :

What is the best way to sort data in spark to avoid shuffles ? Or reducing it ?

Could I correct/add improvements to my code in order to perform the sort ?

Do I really have to sort this way ? Can't I use other techniques like a Group By ?

Snippet of my code :

EDITED

    val uh = uh_months
      .withColumn("UHDIN", datediff(to_date(unix_timestamp(col("UHDIN_YYYYMMDD"), "yyyyMMdd").cast(TimestampType)),
        to_date(unix_timestamp(col("january"), "yyyy-MM-dd").cast(TimestampType))))
      //      .withColumn("DVA_1", to_date((unix_timestamp(col("DVA"), "ddMMMyyyy")).cast(TimestampType)))
      .withColumn("DVA_1", date_format(col("DVA"), "dd/MM/yyyy"))
      .drop("UHDIN_YYYYMMDD")
      .drop("january")
      .drop("DVA").repartition(1300,col("MMED"),col("DEBCRED"),col("NMTGP"))//.repartition(1300,col("NO_NUM"))

    val uh_flag_comment = new TransactionType().transform(uh)
    val uh_repartitioned = uh_flag_comment.repartition(1300,col("NO_NUM"))

    val uh_joined = uh_repartitioned.join(broadcast(smallDF), "NO_NUM")
      .select(
        uh.col("*"),
        smallDF.col("PSP"),
        smallDF.col("minrel"),
        smallDF.col("Label"),
        smallDF.col("StartDate"))
      .withColumnRenamed("DVA_1", "DVA")

    val uh_final = uh_joined.repartition(1300, col("PSP")).sortWithinPartitions(col("NO_NUM"), col("UHDIN"), col("HOURMV"))

    return uh_final

TransactionType is a class where I add a new column to my uh dataframe based on the value of 3 columns (MMED, DEBCRED, NMTGP), using regex.

Without the sort, and using the full capacity of the cluster, the code runs within 1 hour approximately.

Execution plan

== Physical Plan ==
Exchange hashpartitioning(PSP#82, 2600)
+- *Sort [PSP#82 ASC NULLS FIRST, NO_NUM#252 ASC NULLS FIRST, UHDIN#547 ASC NULLS FIRST, HOURMV#175 ASC NULLS FIRST], true, 0
   +- Exchange rangepartitioning(PSP#82 ASC NULLS FIRST, NO_NUM#252 ASC NULLS FIRST, UHDIN#547 ASC NULLS FIRST, HOURMV#175 ASC NULLS FIRST, 200)
      +- Exchange hashpartitioning(PSP#82, NO_NUM#252, UHDIN#547, HOURMV#175, 2600)
         +- *Project [NO_NUM#252, DEV#153, DEBCRED#154, BDGRORI#155, BDGREUR#156, BEWC#157, MSG30_NL#158, SCAPMV#159, USERID#160, MMED#161, TNUM#162, NMTGP#163, BKA#164, CATEXT#165, SEQETAT#166, ACCTYPE#167, BRAND#168, FAMILY#169, SUBFAMILY#170, FORCED_DVA#172, BYBANK#173, CPTE_PROTEGE#174, HOURMV#175, RDFB#176, ... 30 more fields]
            +- *BroadcastHashJoin [NO_NUM#252], [NO_NUM#13], Inner, BuildRight
               :- Exchange hashpartitioning(NO_NUM#252, 1300)
               :  +- *Project [NUM#152 AS NO_NUM#252, DEV#153, DEBCRED#154, BDGRORI#155, BDGREUR#156, BEWC#157, MSG30_NL#158, SCAPMV#159, USERID#160, MMED#161, TNUM#162, NMTGP#163, BKA#164, CATEXT#165, SEQETAT#166, ACCTYPE#167, BRAND#168, FAMILY#169, SUBFAMILY#170, FORCED_DVA#172, BYBANK#173, CPTE_PROTEGE#174, HOURMV#175, RDFB#176, ... 26 more fields]
               :     +- *Filter (BEWC#157 INSET (25003,25302,25114,20113,12017,20108,25046,12018,15379,15358,11011,20114,10118,12003,25097,20106,20133,10133,10142,15402,25026,25345,28023,15376,25019,28004,21701,25001,11008,15310,15003,2SOMEPORT,22048,15470,25300,25514,25381,25339,15099,25301,28005,28026,25098,25018,15323,25376,15804,15414,25344,25102,15458,15313,28002,25385,22051,25214,15031,12005,15425,20145,22011,15304,25027,14020,11007,25901,15343,22049,20112,12031,20127,15339,25421,15432,28025,25340,25325,20150,28011,25368,25304,22501,25369,28022,15098,12032,15375,25002,25008,10116,10101,22502,25090,15004,20105,12030,22503,15095,22007,15809,15342,15311,25216,10103,20122,11019,20142,15097,20147,20149,25005,25205,25380,15380,10120,25015,15384,11003,10110,25016,15090,25307,15001,25390,15312,10115,25219,15806,15459,12016,15359,15395,15302,12021,11701,10111,10148,25379,15807,10102,25352,25355,12010,25095,25394,20101,25413,15385,25322,28027,11026,15533,25201,25371,10128,11028,12020,15819,10143,28028,10123,10125,11020,25029,10122,25343,15015,12033,25014,12012,25024,25375,11023,25501,25402,22001,15317,12014,16114,20501,15046,12001,12022,10104,10117,12002,25499,10145,10153,12011,15350,15300,10119,25305,15345,25374,11027,25430,28021,25202,10121,28024,25101,28001,15321,11025,25358,15333,15501,25533,15372,12008,11015,10114,10113,10112,15303,15320,28006,22002,25359,10132,15497,25353,11029,25425,15374,12019,25437,11022,15357,20148,20111,26114,25099,25354,10124,25303,11010,20120,20135,15820,15331,28029) && isnotnull(NUM#152))
               :        +- *FileScan csv [UHDIN_YYYYMMDD#151,NUM#152,DEV#153,DEBCRED#154,BDGRORI#155,BDGREUR#156,BEWC#157,MSG30_NL#158,SCAPMV#159,USERID#160,MMED#161,TNUM#162,NMTGP#163,BKA#164,CATEXT#165,SEQETAT#166,ACCTYPE#167,BRAND#168,FAMILY#169,SUBFAMILY#170,DVA#171,FORCED_DVA#172,BYBANK#173,CPTE_PROTEGE#174,... 26 more fields] Batched: false, Format: CSV, Location: InMemoryFileIndex[SOMEHOST:SOMEPORT/SOMEPATH, PartitionFilters: [], PushedFilters: [In(BEWC, [25003,25302,25114,20113,12017,20108,25046,12018,15379,15358,11011,20114,10118,12003,25..., ReadSchema: struct<UHDIN_YYYYMMDD:string,NUM:string,DEV:string,DEBCRED:string,BDGRORI:string,BDGREUR:string,B...
               +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, true]))
                  +- *Project [NO_NUM#13, PSP#82, minrel#370, Label#105, StartDate#106]
                     +- *SortMergeJoin [PSP#381], [PSP#82], Inner
                        :- *Sort [PSP#381 ASC NULLS FIRST], false, 0
                        :  +- Exchange hashpartitioning(PSP#381, 200)
                        :     +- *Project [PSP#381, NO_NUM#13, minrel#370]
                        :        +- SortMergeJoin [PSP#381, C_SNUM#14, minrel#370, NO_NUM#13], [NO_PSP#47, C_SNUM_1#387, C_NRELPR#50, NO_NUM_1#400], LeftOuter
                        :           :- *Sort [PSP#381 ASC NULLS FIRST, C_SNUM#14 ASC NULLS FIRST, minrel#370 ASC NULLS FIRST, NO_NUM#13 ASC NULLS FIRST], false, 0
                        :           :  +- Exchange hashpartitioning(PSP#381, C_SNUM#14, minrel#370, NO_NUM#13, 200)
                        :           :     +- SortAggregate(key=[NO_PSP#12, C_SNUM#14, NO_NUM#13], functions=[min(C_NRELPR#15)])
                        :           :        +- *Sort [NO_PSP#12 ASC NULLS FIRST, C_SNUM#14 ASC NULLS FIRST, NO_NUM#13 ASC NULLS FIRST], false, 0
                        :           :           +- Exchange hashpartitioning(NO_PSP#12, C_SNUM#14, NO_NUM#13, 200)
                        :           :              +- SortAggregate(key=[NO_PSP#12, C_SNUM#14, NO_NUM#13], functions=[partial_min(C_NRELPR#15)])
                        :           :                 +- *Sort [NO_PSP#12 ASC NULLS FIRST, C_SNUM#14 ASC NULLS FIRST, NO_NUM#13 ASC NULLS FIRST], false, 0
                        :           :                    +- *Project [NO_PSP#12, C_SNUM#14, NO_NUM#13, C_NRELPR#15]
                        :           :                       +- *Filter (((C_NRELPR#15 IN (001,006) && C_SNUM#14 IN (030,033)) && isnotnull(NO_NUM#13)) && isnotnull(NO_PSP#12))
                        :           :                          +- *FileScan csv [NO_PSP#12,NO_NUM#13,C_SNUM#14,c_nrelpr#15] Batched: false, Format: CSV, Location: InMemoryFileIndex[SOMEHOST:SOMEPORT/SOMEPATH, PartitionFilters: [], PushedFilters: [In(c_nrelpr, [001,006]), In(C_SNUM, [030,033]), IsNotNull(NO_NUM), IsNotNull(NO_PSP)], ReadSchema: struct<NO_PSP:string,NO_NUM:string,C_SNUM:string,c_nrelpr:string>
                        :           +- *Sort [NO_PSP#47 ASC NULLS FIRST, C_SNUM_1#387 ASC NULLS FIRST, C_NRELPR#50 ASC NULLS FIRST, NO_NUM_1#400 ASC NULLS FIRST], false, 0
                        :              +- Exchange hashpartitioning(NO_PSP#47, C_SNUM_1#387, C_NRELPR#50, NO_NUM_1#400, 200)
                        :                 +- *Project [NO_PSP#47, NO_NUM#48 AS NO_NUM_1#400, C_SNUM#49 AS C_SNUM_1#387, c_nrelpr#50]
                        :                    +- *FileScan csv [NO_PSP#47,NO_NUM#48,C_SNUM#49,c_nrelpr#50] Batched: false, Format: CSV, Location: InMemoryFileIndex[SOMEHOST:SOMEPORT/SOMEPATH, PartitionFilters: [], PushedFilters: [], ReadSchema: struct<NO_PSP:string,NO_NUM:string,C_SNUM:string,c_nrelpr:string>
                        +- *Sort [PSP#82 ASC NULLS FIRST], false, 0
                           +- Exchange hashpartitioning(PSP#82, 200)
                              +- *Project [PSP#82, Label#105, StartDate#106]
                                 +- *Filter isnotnull(PSP#82)
                                    +- *FileScan csv [PSP#82,Label#105,StartDate#106] Batched: false, Format: CSV, Location: InMemoryFileIndex[SOMEHOST:SOMEPORT/SOMEPATH, PartitionFilters: [], PushedFilters: [IsNotNull(PSP)], ReadSchema: struct<PSP:string,Label:string,StartDate:string>

Here is the main error(s) I got when launching the job with the sort :

19/05/06 18:02:25 ERROR shuffle.RetryingBlockFetcher: Exception while beginning fetch of 214 outstanding blocks
java.io.IOException: Failed to connect to SOMEHOST/SOMEADDRESS:SOMEPORT
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:232)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:182)
    at org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:98)
    at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:141)
    at org.apache.spark.network.shuffle.RetryingBlockFetcher.start(RetryingBlockFetcher.java:121)
    at org.apache.spark.network.netty.NettyBlockTransferService.fetchBlocks(NettyBlockTransferService.scala:108)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.sendRequest(ShuffleBlockFetcherIterator.scala:228)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.fetchUpToMaxBytes(ShuffleBlockFetcherIterator.scala:435)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.initialize(ShuffleBlockFetcherIterator.scala:323)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.<init>(ShuffleBlockFetcherIterator.scala:140)
    at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:45)
    at org.apache.spark.sql.execution.ShuffledRowRDD.compute(ShuffledRowRDD.scala:165)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: SOMEHOST/SOMEADDRESS:SOMEPORT
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:631)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
    ... 1 more
19/05/06 18:02:25 INFO shuffle.RetryingBlockFetcher: Retrying fetch (1/3) for 214 outstanding blocks after 5000 ms
19/05/06 18:02:25 INFO storage.ShuffleBlockFetcherIterator: Started 6 remote fetches in 13 ms
19/05/06 18:02:28 INFO executor.Executor: Finished task 408.0 in stage 14.0 (TID 6696). 1733 bytes result sent to driver
19/05/06 18:02:28 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 6816
19/05/06 18:02:28 INFO executor.Executor: Running task 466.1 in stage 14.0 (TID 6816)
19/05/06 18:02:28 INFO storage.ShuffleBlockFetcherIterator: Getting 5073 non-empty blocks out of 5089 blocks
19/05/06 18:02:28 INFO client.TransportClientFactory: Found inactive connection to SOMEHOST/SOMEADDRESS:SOMEPORT, creating a new one.
19/05/06 18:02:28 ERROR shuffle.RetryingBlockFetcher: Exception while beginning fetch of 82 outstanding blocks
java.io.IOException: Failed to connect to SOMEHOST/SOMEADDRESS:SOMEPORT
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:232)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:182)
    at org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:98)
    at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:141)
    at org.apache.spark.network.shuffle.RetryingBlockFetcher.start(RetryingBlockFetcher.java:121)
    at org.apache.spark.network.netty.NettyBlockTransferService.fetchBlocks(NettyBlockTransferService.scala:108)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.sendRequest(ShuffleBlockFetcherIterator.scala:228)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.fetchUpToMaxBytes(ShuffleBlockFetcherIterator.scala:435)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.initialize(ShuffleBlockFetcherIterator.scala:323)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.<init>(ShuffleBlockFetcherIterator.scala:140)
    at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:45)
    at org.apache.spark.sql.execution.ShuffledRowRDD.compute(ShuffledRowRDD.scala:165)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: SOMEHOST/SOMEADDRESS:SOMEPORT
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:631)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
    ... 1 more

EDITED

An other type of error :

19/05/06 18:06:16 ERROR executor.Executor: Exception in task 309.1 in stage 13.1 (TID 7592)
java.io.FileNotFoundException: /applis/hadoop/yarn/local/usercache/MYUSER/appcache/application_1555263602441_0123/blockmgr-aa586b76-ff58-4f88-b168-288c3e1b9f61/3c/temp_shuffle_ea967624-f633-4481-9a05-249b561e3c38 (No such file or directory)
    at java.io.FileInputStream.open0(Native Method)
    at java.io.FileInputStream.open(FileInputStream.java:195)
    at java.io.FileInputStream.<init>(FileInputStream.java:138)
    at org.spark_project.guava.io.Files$FileByteSource.openStream(Files.java:124)
    at org.spark_project.guava.io.Files$FileByteSource.openStream(Files.java:114)
    at org.spark_project.guava.io.ByteSource.copyTo(ByteSource.java:202)
    at org.spark_project.guava.io.Files.copy(Files.java:436)
    at org.spark_project.guava.io.Files.move(Files.java:651)
    at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.mergeSpills(UnsafeShuffleWriter.java:277)
    at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.closeAndWriteOutput(UnsafeShuffleWriter.java:216)
    at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:169)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
19/05/06 18:06:16 ERROR executor.Executor: Exception in task 502.1 in stage 13.1 (TID 7599)
java.io.FileNotFoundException: /applis/hadoop/yarn/local/usercache/MYUSER/appcache/application_1555263602441_0123/blockmgr-aa586b76-ff58-4f88-b168-288c3e1b9f61/34/temp_shuffle_dd202cd1-ad8f-41c4-b4d1-d79621cd169e (No such file or directory)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:102)
    at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)
    at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:247)
    at org.apache.spark.shuffle.sort.ShuffleExternalSorter.writeSortedFile(ShuffleExternalSorter.java:201)
    at org.apache.spark.shuffle.sort.ShuffleExternalSorter.closeAndGetSpills(ShuffleExternalSorter.java:405)
    at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.closeAndWriteOutput(UnsafeShuffleWriter.java:209)
    at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:169)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
19/05/06 18:06:16 INFO executor.Executor: Finished task 200.2 in stage 13.1 (TID 7568). 2826 bytes result sent to driver
19/05/06 18:06:16 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
19/05/06 18:06:16 INFO util.ShutdownHookManager: Shutdown hook called

Some information/context:

I am working on a production environment (see the cluster configuration below). I cannot upgrade my spark version. I do not have spark UI or yarn UI to monitor my jobs. All I can retrieve are the yarn logs.

Spark Version: 2.2

Cluster configuration:

21 compute nodes (workers)

8 cores each

64 GB RAM per node

Current Spark configuration:

-master: yarn

-executor-memory: 42G

-executor-cores: 5

-driver memory: 42G

-num-executors: 32

-spark.sql.broadcastTimeout=3600

-spark.kryoserializer.buffer.max=512

-spark.yarn.executor.memoryOverhead=2400

-spark.driver.maxResultSize=500m

-spark.memory.storageFraction=0.3

-spark.memory.fraction=0.9

-spark.hadoop.fs.permissions.umask-mode=007

How is the job executed:

We build an artifact (jar) with IntelliJ and then send it to a server. Then a bash script is executed. This script:

export some environment variables (SPARK_HOME, HADOOP_CONF_DIR, PATH and SPARK_LOCAL_DIRS)

launch the spark-submit command with all the parameters defined in the spark configuration above

retrieves the yarn logs of the application

Body:
I managed to sort the data successfully (and then to run the entire code) using a significant part of the answer posted by @Alexandros.

I however did some changes in terms of configurations in the cluster :

I increased the Executor memory to 45 G (it was 42 G)

I changed the spark parameter --executor-memory increasing it to 45G instead of 42G

I increased the number of executors to 40

I also increased on each node the disk space of /applis/hadoop/yarn/local/usercache/MYUSER/, adding 20-25G (there was a bit less than 50 G available on each node for this folder). This is the yarn usercache where Spark writes intermediary shuffled blocks of data. Since I have a dataset of 1.2T, and I have 21 nodes, I need approximately 60-65G of disk space on each node when the data is distributed across the nodes.

I also used the sortWithinPartition function (it was working fine with this function but it fails with the classic sort function). Besides, I only need to sort each partition since I partition based on PSP (if the dataset is not sorted based on PSP, it's fine).

Below is the code :

val uh = uh_months
  .withColumn("UHDIN", datediff(to_date(unix_timestamp(col("UHDIN_YYYYMMDD"), "yyyyMMdd").cast(TimestampType)),
    to_date(unix_timestamp(col("january"), "yyyy-MM-dd").cast(TimestampType))))
  //      .withColumn("DVA_1", to_date((unix_timestamp(col("DVA"), "ddMMMyyyy")).cast(TimestampType)))
  .withColumn("DVA_1", date_format(col("DVA"), "dd/MM/yyyy"))
  .drop("UHDIN_YYYYMMDD")
  .drop("january")
  .drop("DVA")
  .repartition(3000, col("NO_NUM"))
  .persist()

val uh_flag_comment = new TransactionType().transform(uh)

val uh_joined = uh_flag_comment.join(broadcast(smallDF), "NO_NUM")
  .select(
    uh_flag_comment.col("*"),
    kl_holdmand_pruned.col("PSP"),
    kl_holdmand_pruned.col("minrel"),
    kl_holdmand_pruned.col("TerroLabel"),
    kl_holdmand_pruned.col("TerroStartDate"))
  .withColumnRenamed("DVA_1", "DVA")

smallDF.unpersist()
uh.unpersist()

val uh_to_be_sorted = uh_joined.repartition(3000, col("PSP"))
val uh_final = uh_to_be_sorted.sortWithinPartitions(col("NO_NUM"), col("UHDIN"), col("HOURMV"))

uh_final


答:
From experience i know that it isn't safe to make this assumption.

However, the HTML5 standard does describe this standard value for the language attribute being "text/javascript".

ContentLicense: CC BY-SA 3.0

Title: Can "text/javascript" be assumed a de facto default for the <script> element?

Body: Despite that the HTML4 standard states there is no default for the


问:
Using Ruby CSV header converters
Say I have the following class:

```<code>class Buyer < ActiveRecord::Base
  attr_accesible :first_name, :last_name
</code>```

and the following in a CSV file:

```<code>First Name,Last Name
John,Doe
Jane,Doe
</code>```

I want to save the contents of the CSV into the database.  I have the following in a Rake file:

```<code>namespace :migration do
  desc "Migrate CSV data"
  task :import, [:model, :file_path] => :environment do |t, args|
    require 'csv'

    model = args.model.constantize
    path = args.file_path
    CSV.foreach(path, :headers => true,
                      :converters => :all,
                      :header_converters => lambda { |h| h.downcase.gsub(' ', '_') }
                      ) do |row|
    model.create!(row.to_hash)
  end
end
</code>```

end

I am getting an undefined method 'downcase' for nil:NilClass.  If I exclude the header converters then I get unknown attribute 'First Name'.  What's the correct syntax for converting a header from, say, First Name to first_name?

答:
Passing :symbol to :header_converters will automatically convert to strings to snake case as well.</p>

```<code>options = {:headers => true,
           :header_converters => :symbol}
CSV.foreach(filepath, options) ...
#<CSV::Row first_name:"John" last_name:"Doe">
#<CSV::Row first_name:"Jane" last_name:"Doe">
</code>```

Title: Using Ruby CSV header converters

Body:
Say I have the following class:

class Buyer < ActiveRecord::Base
  attr_accesible :first_name, :last_name

and the following in a CSV file:

First Name,Last Name
John,Doe
Jane,Doe

I want to save the contents of the CSV into the database.  I have the following in a Rake file:

namespace :migration do
  desc "Migrate CSV data"
  task :import, [:model, :file_path] => :environment do |t, args|
    require 'csv'

    model = args.model.constantize
    path = args.file_path
    CSV.foreach(path, :headers => true,
                      :converters => :all,
                      :header_converters => lambda { |h| h.downcase.gsub(' ', '_') }
                      ) do |row|
    model.create!(row.to_hash)
  end
end

end

I am getting an undefined method 'downcase' for nil:NilClass.  If I exclude the header converters then I get unknown attribute 'First Name'.  What's the correct syntax for converting a header from, say, First Name to first_name?

Body:
Passing :symbol to :header_converters will automatically convert to strings to snake case as well.

options = {:headers => true,
           :header_converters => :symbol}
CSV.foreach(filepath, options) ...
#<CSV::Row first_name:"John" last_name:"Doe">
#<CSV::Row first_name:"Jane" last_name:"Doe">


问:
How do I edit PYTHONPATH?
I have edited my PYTHONPATH in the .bashrc file but now I want to edit a section of it.

I have a program which uses a particular source file, say libraryX

So I set the PYTHONPATH to

```<code>export PYTHONPATH="${PYTHONPATH}:/path1/"
</code>```

I added the above in my .bashrc file

3. source .bashrc

4. Next, I realized I had to set PYTHONPATH to path2 instead so I repeated steps 2 and 3 except I replaced path1 by path2 in the .bashrc file
5. Now, there is a conflict and since the PYTHONPATH is now

```<code>/path1/path2/
</code>```

and path1 is first, it does not consider path2 which is the right path.

How do I remove path1? i.e. edit PYTHONPATH?

答:
Reset pythonpath and re-create it from bashrc</p>

```<code>unset PYTHONPATH
source ~/.bashrc
</code>```

Title: How do I edit PYTHONPATH?

Body:
I have edited my PYTHONPATH in the .bashrc file but now I want to edit a section of it.

I have a program which uses a particular source file, say libraryX

So I set the PYTHONPATH to

export PYTHONPATH="${PYTHONPATH}:/path1/"

I added the above in my .bashrc file

3. source .bashrc

4. Next, I realized I had to set PYTHONPATH to path2 instead so I repeated steps 2 and 3 except I replaced path1 by path2 in the .bashrc file
5. Now, there is a conflict and since the PYTHONPATH is now

/path1/path2/

and path1 is first, it does not consider path2 which is the right path.

How do I remove path1? i.e. edit PYTHONPATH?

Body:
Reset pythonpath and re-create it from bashrc

unset PYTHONPATH
source ~/.bashrc


问:
Dart: cancelable post delay / future
I am new to flutter and I want to translate some text from an InputField by calling an API. However I don't want to call it on every key stroke, but instead only when the user paused typing.

On Android I would just use the Handler class with postDelay() with beforehand calling removeAllCallbacksAndMessages(null). Is there a way to do something similar on Dart?

Here is my current code:

```<code>Future<String> getTranslation(String query, Language from, Language to) async {
    // cancel here if a call to this function was less than 500 millis ago.
    return Future.delayed(const Duration(milliseconds: 500), () {
      return _translator.translate(query, from: from.code, to: to.code)
    });
  }
</code>```

Edit 1

I'm calling the code from my Bloc like so:

```<code>@override
  Stream<State> mapEventToState(Event event) async* {
    if (event is QueryChangeEvent) {
      yield TextTranslationChangeState(
          query: event.query ?? "",
          translation: await _repo.getTranslation(event.query, currentState.fromLang, currentState.toLang));
  }
</code>```

This is why I cannot call .then() on the future because I wouldn't be able to yield the new state from the block of the nested function.

Any help is appreciated!

答:
Yes there is, it is called Timer

https://api.dartlang.org/stable/2.3.1/dart-async/Timer-class.html

You can delay execution as well as cancel the trigger.

Title: Dart: cancelable post delay / future

Body:
I am new to flutter and I want to translate some text from an InputField by calling an API. However I don't want to call it on every key stroke, but instead only when the user paused typing.

On Android I would just use the Handler class with postDelay() with beforehand calling removeAllCallbacksAndMessages(null). Is there a way to do something similar on Dart?

Here is my current code:

Future<String> getTranslation(String query, Language from, Language to) async {
    // cancel here if a call to this function was less than 500 millis ago.
    return Future.delayed(const Duration(milliseconds: 500), () {
      return _translator.translate(query, from: from.code, to: to.code)
    });
  }

Edit 1

I'm calling the code from my Bloc like so:

@override
  Stream<State> mapEventToState(Event event) async* {
    if (event is QueryChangeEvent) {
      yield TextTranslationChangeState(
          query: event.query ?? "",
          translation: await _repo.getTranslation(event.query, currentState.fromLang, currentState.toLang));
  }

This is why I cannot call .then() on the future because I wouldn't be able to yield the new state from the block of the nested function.

Any help is appreciated!


问:
Why service-worker does not cache js file
SW caches an image file, but not JS (size 0 bytes)

```

    /*serviceWorker*/
    if ('serviceWorker' in navigator) {
        navigator.serviceWorker
            .register('service-worker.js', { scope: '/' })
            .then(function(registration) {
                console.log("Service Worker Registered");
            })
            .catch(function(err) {
                console.log("Service Worker Failed to Register", err);
            })

    };

    const CACHE = 'cache-only-v1';
    const timeout = 400;

    self.addEventListener('install', (event) => {
        event.waitUntil(
        caches.open(CACHE).then((cache) => {
            return cache.addAll([
                '/sys/Raycaster/3d.js',
                '/sys/js/popmotion.global.min.js',
                '/sys/js/anime.min.js',
                'sys/b/bgFeedback.jpg',
                '/sys/b/ic.svg',
                '/sys/b/ani/img_ani_1.webp',
                '/sys/b/ani/img_ani_2.webp',
                '/sys/b/ani/img_ani_3.webp',
                '/sys/printProd/p.webp',
                '/sys/printProd/p2.webp',
                '/sys/printProd/p3.webp',
                '/sys/printProd/p4.webp',
                '/sys/printProd/p5.webp',
                '/sys/printProd/p6.webp',
                '/sys/printProd/p7.webp',
                '/sys/printProd/p8.webp',
                '/sys/printProd/p9.webp',
                '/sys/printProd/p10.webp',
                '/sys/printProd/p11.webp',
                '/sys/printProd/p12.webp',
                '/sys/sliderInSlider/01_el.webp',
                '/sys/sliderInSlider/02_dv.webp',
                '/sys/sliderInSlider/05_snth.webp',
                '/sys/sliderInSlider/06_ton.webp',
                '/sys/sliderInSlider/07_pty.webp',
                '/sys/sliderInSlider/08_lyu.webp',
                '/sys/sliderInSlider/09_kar.webp'
                ]);
            })
        );
    });
```

Image here: https://f.usemind.org/img/2019-05-28_143451.jpg

答:
Information displayed in chrome DevTools are based on Response Headers and sometimes, it doesn't reflect reality. Your js files are cached but it doesn't have a Content-Lenght in it's Headers.

to test it you can do :

```<code>caches.open(CACHES)
.then( cache => cache.match('/sys/js/popmotion.global.min.js'))
.then( res =>res.text())
.then( js => console.log(js))
</code>```

This code will check in your caches for popmotion.global.min.js, and display it's content.

Body:
SW caches an image file, but not JS (size 0 bytes)

    /*serviceWorker*/
    if ('serviceWorker' in navigator) {
        navigator.serviceWorker
            .register('service-worker.js', { scope: '/' })
            .then(function(registration) {
                console.log("Service Worker Registered");
            })
            .catch(function(err) {
                console.log("Service Worker Failed to Register", err);
            })

    };

    const CACHE = 'cache-only-v1';
    const timeout = 400;

    self.addEventListener('install', (event) => {
        event.waitUntil(
        caches.open(CACHE).then((cache) => {
            return cache.addAll([
                '/sys/Raycaster/3d.js',
                '/sys/js/popmotion.global.min.js',
                '/sys/js/anime.min.js',
                'sys/b/bgFeedback.jpg',
                '/sys/b/ic.svg',
                '/sys/b/ani/img_ani_1.webp',
                '/sys/b/ani/img_ani_2.webp',
                '/sys/b/ani/img_ani_3.webp',
                '/sys/printProd/p.webp',
                '/sys/printProd/p2.webp',
                '/sys/printProd/p3.webp',
                '/sys/printProd/p4.webp',
                '/sys/printProd/p5.webp',
                '/sys/printProd/p6.webp',
                '/sys/printProd/p7.webp',
                '/sys/printProd/p8.webp',
                '/sys/printProd/p9.webp',
                '/sys/printProd/p10.webp',
                '/sys/printProd/p11.webp',
                '/sys/printProd/p12.webp',
                '/sys/sliderInSlider/01_el.webp',
                '/sys/sliderInSlider/02_dv.webp',
                '/sys/sliderInSlider/05_snth.webp',
                '/sys/sliderInSlider/06_ton.webp',
                '/sys/sliderInSlider/07_pty.webp',
                '/sys/sliderInSlider/08_lyu.webp',
                '/sys/sliderInSlider/09_kar.webp'
                ]);
            })
        );
    });

Image here: https://f.usemind.org/img/2019-05-28_143451.jpg

Body:
Information displayed in chrome DevTools are based on Response Headers and sometimes, it doesn't reflect reality. Your js files are cached but it doesn't have a Content-Lenght in it's Headers.

to test it you can do :

caches.open(CACHES)
.then( cache => cache.match('/sys/js/popmotion.global.min.js'))
.then( res =>res.text())
.then( js => console.log(js))

This code will check in your caches for popmotion.global.min.js, and display it's content.


问:
DropDownList allows setting DataSource but error at DataBind
I have a dropdownlist control.

```<code><asp:DropDownList runat="server" ID="ddlMarketerContract" CssClass="form-control" />
</code>```

I set the DataSource in the code-behind using LINQ and EF

```<code>ddlMarketerContract.DataSource = context.Table.Select(x => x.ColumnName).Distinct().ToList();
</code>```

But while stepping through the debug, when I go to bind the data

```<code>ddlMarketerContract.DataBind();
</code>```

I get an "Object reference not set to an instance of an object" error.

Not sure why it lets me set the DataSource, but doesn't find the object when binding the control.

I have a similar setup on another page, but that works as expected. I thought maybe it's because I don't have DataTextField/DataValueField set, but the other page didn't have those set, and it still worked fine.

答: Problem was with the LINQ statement. Upon creating a generic list of string values I was able to bind the control just fine. Will modify the LINQ query to resolve the issue.</p>

Body:
I have a dropdownlist control.

<asp:DropDownList runat="server" ID="ddlMarketerContract" CssClass="form-control" />

I set the DataSource in the code-behind using LINQ and EF

ddlMarketerContract.DataSource = context.Table.Select(x => x.ColumnName).Distinct().ToList();

But while stepping through the debug, when I go to bind the data

ddlMarketerContract.DataBind();

I get an "Object reference not set to an instance of an object" error.

Not sure why it lets me set the DataSource, but doesn't find the object when binding the control.

I have a similar setup on another page, but that works as expected. I thought maybe it's because I don't have DataTextField/DataValueField set, but the other page didn't have those set, and it still worked fine.


问:
How to diff two keyword lists?
I have two files of version information of the following shape:

```<code>$ cat file1
cabal 2.2.0.1
http 4000.3.12
hunit 1.6.0.0
only 0.1
...
</code>```

I'm looking for (preferably simple) ways to compare these using command line tools, allowing me to generate things such as
1. all lines in file1 where the name doesn't show up in file2 at all
2. lines of the form name <v1> <v2> for names that show up in both files

I thought I had the right ingredient when I discovered diff -y, but that sometimes puts different names on the same line.

答:
all lines in file1 where the name doesn't show up in file2 at all

Requires GNU awk for sorted results.

```<code>$ gawk 'NR == FNR { packages[$1] = $0; next }
        { delete packages[$1] }
        END { PROCINF0["sorted_in"] = "@ind_str_asc";
              for (p in packages) print packages[p] }' file1 file2
</code>```

lines of the form name <v1> <v2> for names that show up in both files

This one requires the files to be sorted on the first column.

```<code>$ join file1 file2
</code>```

Title: How to diff two keyword lists?

Body:
I have two files of version information of the following shape:

$ cat file1
cabal 2.2.0.1
http 4000.3.12
hunit 1.6.0.0
only 0.1
...

I'm looking for (preferably simple) ways to compare these using command line tools, allowing me to generate things such as
1. all lines in file1 where the name doesn't show up in file2 at all
2. lines of the form name <v1> <v2> for names that show up in both files

I thought I had the right ingredient when I discovered diff -y, but that sometimes puts different names on the same line.

Body:
all lines in file1 where the name doesn't show up in file2 at all

Requires GNU awk for sorted results.

$ gawk 'NR == FNR { packages[$1] = $0; next }
        { delete packages[$1] }
        END { PROCINF0["sorted_in"] = "@ind_str_asc";
              for (p in packages) print packages[p] }' file1 file2

lines of the form name <v1> <v2> for names that show up in both files

This one requires the files to be sorted on the first column.

$ join file1 file2


问:
Python update list of dictionaries in a loop
I am working on a python code to update list of dictionaries if input exists in list of dictionaries. If input doesn't exist in list of dictionaries it should print "value doesn't exist in entire list" or do some other operation.
Below is the code I have written

```<code>a = [{'main_color': 'red', 'second_color':'blue'},{'main_color': 'yellow', 'second_color':'green'},{'main_color': 'yellow', 'second_color':'blue'}]

print('Enter main color:')

conType=input()

for d in a:

    if d['main_color']==conType:
        print('matched')
        d['second_color']='bluetooth'
    else:
        print('no value')
print(a)
</code>```

Issue here is if input is 'red' "no value" is getting printed twice and "matched" is printed once.

My use case is if input doesn't exist in list of dictionaries it should print "no value" only once. If input exists code should update next key with new value and print "matched" once.

I have been through other questions in stack overflow. I couldn't find answer to this scenario.

Please help

答:
There's for .. else statement which you can use here:

```<code>a = [{'main_color': 'red', 'second_color': 'blue'},
     {'main_color': 'yellow', 'second_color': 'green'},
     {'main_color': 'yellow', 'second_color': 'blue'}]

conType = input('Enter main color: ')

for d in a:
    if d['main_color'] == conType:
        print('matched')
        d['second_color'] = 'bluetooth'
        break
else:
    print('no value')

print(a)
</code>```

UPD.

Sorry, I've missed that there's multiple objects with same main_color in the list (thanks, @HaleemurAli). In case if you don't need to break loop after matching you can't use for .. else statement, but you can use boolean flag. To check if string equals any of list of string you can membership test operator (in):

```<code>a = [{'main_color': 'red', 'second_color': 'blue'},
     {'main_color': 'yellow', 'second_color': 'green'},
     {'main_color': 'blue', 'second_color': 'blue1'}]

conType = input('Enter main color: ')
conType1 = input('Enter another main color: ')

b = [conType, conType1]
matched = False
for d in a:
    if d['main_color'] in b:
        print('matched')
        matched = True
        d['second_color'] = 'bluetooth'

if not matched:
    print('no value')
</code>```

Body:
I am working on a python code to update list of dictionaries if input exists in list of dictionaries. If input doesn't exist in list of dictionaries it should print "value doesn't exist in entire list" or do some other operation.
Below is the code I have written

a = [{'main_color': 'red', 'second_color':'blue'},{'main_color': 'yellow', 'second_color':'green'},{'main_color': 'yellow', 'second_color':'blue'}]

print('Enter main color:')

conType=input()

for d in a:

    if d['main_color']==conType:
        print('matched')
        d['second_color']='bluetooth'
    else:
        print('no value')
print(a)

Issue here is if input is 'red' "no value" is getting printed twice and "matched" is printed once.

My use case is if input doesn't exist in list of dictionaries it should print "no value" only once. If input exists code should update next key with new value and print "matched" once.

I have been through other questions in stack overflow. I couldn't find answer to this scenario.

Please help

Body:
There's for .. else statement which you can use here:

a = [{'main_color': 'red', 'second_color': 'blue'},
     {'main_color': 'yellow', 'second_color': 'green'},
     {'main_color': 'yellow', 'second_color': 'blue'}]

conType = input('Enter main color: ')

for d in a:
    if d['main_color'] == conType:
        print('matched')
        d['second_color'] = 'bluetooth'
        break
else:
    print('no value')

print(a)

UPD.

Sorry, I've missed that there's multiple objects with same main_color in the list (thanks, @HaleemurAli). In case if you don't need to break loop after matching you can't use for .. else statement, but you can use boolean flag. To check if string equals any of list of string you can membership test operator (in):

a = [{'main_color': 'red', 'second_color': 'blue'},
     {'main_color': 'yellow', 'second_color': 'green'},
     {'main_color': 'blue', 'second_color': 'blue1'}]

conType = input('Enter main color: ')
conType1 = input('Enter another main color: ')

b = [conType, conType1]
matched = False
for d in a:
    if d['main_color'] in b:
        print('matched')
        matched = True
        d['second_color'] = 'bluetooth'

if not matched:
    print('no value')


问:
The document.execCommand() method is not working on safari in Iphone
I'm doing copy text when click on the button to copy the text from the text field. But not working on safari in Iphone
This is my function</p>

```<code>function myFunction() {
      var copyText = document.getElementById("myInput");
      copyText.select();
      document.execCommand("copy");
     }
</code>```

答:
Copy command works from iOS 10. I think you may not have the supported version.
See here for browsers/iOS support Reference</p>

Body:
I'm doing copy text when click on the button to copy the text from the text field. But not working on safari in Iphone
This is my function

function myFunction() {
      var copyText = document.getElementById("myInput");
      copyText.select();
      document.execCommand("copy");
     }


问:
How discriminate between two arrays?
i want to get just the users which not friends

i have two list, friends and users,
if they are already friends, then the user list should no longer exist

friends = [{
  friend: {
    id: 3,
    name: "tres"
  },
  owner: {
    id: 1,
    name: "uno"
  }
}];

let users = [{
  id: 1,
  name: "uno",
}, {
  id: 2,
  name: "dos"
}]

users = users.filter(currentUser =>
  let flag = false, friends.forEach(function(current) {
      if (currentUser.id == current.friend.id) {
        flag = true;
      }
    }
    return flag;
  ));

console.log(users);```

i can to use FOR in the function FILTER? is correct?

答:
If I understand correctly, your, friends array is a list of relationships.  So

{
  friend: {id: 3, name: "tres"},
  owner: {id: 1,  name: "uno"}
}
```

<p>means that user id 1 is friends with user id 3.</p>

<p>So to check through a collection just use <code>Array.prototype.some</code> or <code>Array.prototype.every</code>, whichever one reads better:</p>

<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-js lang-js prettyprint-override"><code>const friendless = (users, friends) =>
  users.filter(user => !friends.some(relationship => relationship.friend.id === user.id))

let users = [
  {id: 1, name: "uno"},
  {id: 2, name: "dos"},
  {id: 3, name: "tres"}
]

let friends = [{
  friend: {id: 3, name: "tres"},
  owner: {id: 1,  name: "uno"}
},{
  friend: {id: 2, name: "dos"},
  owner: {id: 1, name: "uno"}
}, {
  friend: {id: 3, name: "tres"},
  owner: {id: 2, name: "dos"}
}];

console .log (
  friendless (users, friends)
)</code>```

(or like this:

const friendless = (users, friends) =>
  users.filter(user => friends.every(relationship => relationship.friend.id !== user.id))
```

<h2>Update</h2>

<p>I don't know if you're in control of the structure of <code>users</code> and <code>friends</code>, but if you are, you might want to reconsider your data structures.  There seems to be a lot of unnecessary duplication in your current approach.  </p>

<p>Any of these approaches might clean it up:</p>

<h3>A</h3>

<pre class="lang-js prettyprint-override"><code>let users = [
  {id: 1, name: "uno"},
  {id: 2, name: "dos"},
  {id: 3, name: "tres"}
]

let friends = [
  {id: 1, friend: 3},
  {id: 1, friend: 2},
  {id: 2, friend: 3},
];
</code>```

B

let users = [
  {id: 1, name: "uno"},
  {id: 2, name: "dos"},
  {id: 3, name: "tres"}
]

let friends = [
  [1, 3],
  [1, 2],
  [2, 3],
];
```

<h3>C</h3>

<pre class="lang-js prettyprint-override"><code>let users = [
  {id: 1, name: "uno", friends: [3, 2]},  // (or `friendIds`)
  {id: 2, name: "dos", friends: [3]},
  {id: 3, name: "tres", friends: []}
]
</code>```

Option C is especially useful for your current question:

const friendless = users => users.filter(u => u.friends.length === 0)
```

Title: How discriminate between two arrays?

Body:
i want to get just the users which not friends

i have two list, friends and users,
if they are already friends, then the user list should no longer exist

friends = [{
  friend: {
    id: 3,
    name: "tres"
  },
  owner: {
    id: 1,
    name: "uno"
  }
}];

let users = [{
  id: 1,
  name: "uno",
}, {
  id: 2,
  name: "dos"
}]

users = users.filter(currentUser =>
  let flag = false, friends.forEach(function(current) {
      if (currentUser.id == current.friend.id) {
        flag = true;
      }
    }
    return flag;
  ));

console.log(users);

i can to use FOR in the function FILTER? is correct?

Body:
If I understand correctly, your, friends array is a list of relationships.  So

{
  friend: {id: 3, name: "tres"},
  owner: {id: 1,  name: "uno"}
}

means that user id 1 is friends with user id 3.

So to check through a collection just use Array.prototype.some or Array.prototype.every, whichever one reads better:

const friendless = (users, friends) =>
  users.filter(user => !friends.some(relationship => relationship.friend.id === user.id))

let users = [
  {id: 1, name: "uno"},
  {id: 2, name: "dos"},
  {id: 3, name: "tres"}
]

let friends = [{
  friend: {id: 3, name: "tres"},
  owner: {id: 1,  name: "uno"}
},{
  friend: {id: 2, name: "dos"},
  owner: {id: 1, name: "uno"}
}, {
  friend: {id: 3, name: "tres"},
  owner: {id: 2, name: "dos"}
}];

console .log (
  friendless (users, friends)
)

(or like this:

const friendless = (users, friends) =>
  users.filter(user => friends.every(relationship => relationship.friend.id !== user.id))

Update

I don't know if you're in control of the structure of users and friends, but if you are, you might want to reconsider your data structures.  There seems to be a lot of unnecessary duplication in your current approach.

Any of these approaches might clean it up:

A

let users = [
  {id: 1, name: "uno"},
  {id: 2, name: "dos"},
  {id: 3, name: "tres"}
]

let friends = [
  {id: 1, friend: 3},
  {id: 1, friend: 2},
  {id: 2, friend: 3},
];

B

let users = [
  {id: 1, name: "uno"},
  {id: 2, name: "dos"},
  {id: 3, name: "tres"}
]

let friends = [
  [1, 3],
  [1, 2],
  [2, 3],
];

C

let users = [
  {id: 1, name: "uno", friends: [3, 2]},  // (or `friendIds`)
  {id: 2, name: "dos", friends: [3]},
  {id: 3, name: "tres", friends: []}
]

Option C is especially useful for your current question:

const friendless = users => users.filter(u => u.friends.length === 0)


问:
Make optional property in the formgroup
I am trying to create a formgroup with optional property. imageName should be optional.

Component

```<code>this.formbuilder.group({
      details:[''],
      date:[''],
      imageName : ['']
    })
</code>```

答:
try this with formBuilder</p>

```<code>this.formbuilder.group({
  details:['', Validators.required],
  date:['', Validators.required],
  imageName : ['']
})
</code>```

Title: Make optional property in the formgroup

Body:
I am trying to create a formgroup with optional property. imageName should be optional.

Component

this.formbuilder.group({
      details:[''],
      date:[''],
      imageName : ['']
    })

Body:
try this with formBuilder

this.formbuilder.group({
  details:['', Validators.required],
  date:['', Validators.required],
  imageName : ['']
})


问:
Selenium Timed out receiving message from renderer
After Chrome released their newest version yesterday (64.0.3282), I am now receiving this error rather sporadically:

Timed out receiving message from renderer: 600.000

I'm running about 2,000 selenium tests within a docker container and I see this failure at a rate of about 1 in 100. There are no reproducible steps as far as I can tell- the tests that fail are different with each iteration. I updated to the newest Chromedriver (2.35), but that didn't seem to have any effect. I was previously using Selenium 2.41, but have updated to the newest version (3.8.1) hoping that it might help... it did not. I'm completely at a loss as to why this might be occurring. Has anyone else noticed this? Is it possibly a bug with Chrome's newest release?

Thank you in advance for any help you may be able to provide.

答:
If your website is https, and having trouble with chromedriver on timedout issue, use

```<code>option.addArguments("enable-features=NetworkServiceInProcess")
</code>```

If the above doesn't work, use

```<code>option.addArguments("disable-features=NetworkService")
</code>```

instead

Credit goes to https://groups.google.com/forum/#!topic/chromedriver-users/yHuW_Z7tdy0

Body:
If your website is https, and having trouble with chromedriver on timedout issue, use

option.addArguments("enable-features=NetworkServiceInProcess")

If the above doesn't work, use

option.addArguments("disable-features=NetworkService")

instead

Credit goes to https://groups.google.com/forum/#!topic/chromedriver-users/yHuW_Z7tdy0


问:
Frequency of HTML5 updates
HTML5 is about to be published with a couple of interesting features.  However, I do have concerns regarding the frequency of the HTML updates in general.

Last version of HTML 4 has been published more than a decade ago and in IT, 10 years is an eternity.

What kind of update frequency should we expect after the release of the first HTML5 version?  Will it be on a regular basis (yearly?) or can it get even worst with all the people involved in the process?

(Please, try to provide a link to official answer from W3C or an article because I don’t want to start an opinion debate here)

答:
The HTML spec will be updated continuously, it is a living document:

we moved to a new development model,
  where the technology is not versioned
  and instead we just have a living
  document that defines the technology
  as it evolves

There's some further discussion of what this means in questions 2.5 to 2.21 of the FAQ.

Title: Frequency of HTML5 updates


问:
Javafx Css variable defination Error: Expected '<number>' while parsing 'font-size'
I get javaFx css parse error while defining new variables in css.

It perfectly runs ok with a given constant like: -fx-font-size: 10;

Here is the code:

```<code>.root {
    tab-label-text-size: 10;
}
.label {
    -fx-font-size: tab-label-text-size;
}
</code>```

Whatever I tried, I couldnt get rid of this exception.
I also used other variables like this and they also run ok.

Expected '' while parsing '-fx-font-size' at [9,19]

Maybe a bug in javaFx??

答:
In JavaFX CSS you can assign a color constant:

```<code>* {
    -color : #f44242;
}

.button {
    -fx-text-fill: -color;
}
</code>```

But unfortunately, font size constants are not supported by the CssParser

Body:
I get javaFx css parse error while defining new variables in css.

It perfectly runs ok with a given constant like: -fx-font-size: 10;

Here is the code:

.root {
    tab-label-text-size: 10;
}
.label {
    -fx-font-size: tab-label-text-size;
}

Whatever I tried, I couldnt get rid of this exception.
I also used other variables like this and they also run ok.

Expected '' while parsing '-fx-font-size' at [9,19]

Maybe a bug in javaFx??

Body:
In JavaFX CSS you can assign a color constant:

* {
    -color : #f44242;
}

.button {
    -fx-text-fill: -color;
}

But unfortunately, font size constants are not supported by the CssParser


问:
Jackson deserializes object from JSON in reversed order
Jackson is deserializing the JSON elements in reversed order of what I get. We have an environment where we have a frontend that sends something like the next JSON:

```<code>{
    "nameElement": "element",
    "subElementList": [
        {
            "nameSubElement": "a"
        },
        {
            "nameSubElement": "b"
        },
        {
            "nameSubElement": "c"
        }
    ]
}
</code>```

And a backend that deserializes it through Jackson to objects like this:

```<code>@Entity
public class Element {

    private String nameElement;

    @OneToMany(cascade = CascadeType.ALL, mappedBy = "element", fetch = FetchType.LAZY, orphanRemoval = true)
    @SortNatural
    private SortedSet<SubElement> subElementList;

}
</code>```

```<code>@Entity
public class SubElement {

    private String nameSubElement;

}
</code>```

The problem comes when we look at the database, where the subElements of the list are deserialized and saved in the reversed order: element c would be id 1, element b would be id 2, and element a would be id 3, but we need exactly the opposite, since the order of the frontend is the opposite. I tried to change SortedSet by List but it didn't work.

答: You need to implement Comparable interface and provide implementation for compareTo method interface as @SortNatural uses sorting with `Comparable implemented by entity.

Body:
Jackson is deserializing the JSON elements in reversed order of what I get. We have an environment where we have a frontend that sends something like the next JSON:

{
    "nameElement": "element",
    "subElementList": [
        {
            "nameSubElement": "a"
        },
        {
            "nameSubElement": "b"
        },
        {
            "nameSubElement": "c"
        }
    ]
}

And a backend that deserializes it through Jackson to objects like this:

@Entity
public class Element {

    private String nameElement;

    @OneToMany(cascade = CascadeType.ALL, mappedBy = "element", fetch = FetchType.LAZY, orphanRemoval = true)
    @SortNatural
    private SortedSet<SubElement> subElementList;

}

@Entity
public class SubElement {

    private String nameSubElement;

}

The problem comes when we look at the database, where the subElements of the list are deserialized and saved in the reversed order: element c would be id 1, element b would be id 2, and element a would be id 3, but we need exactly the opposite, since the order of the frontend is the opposite. I tried to change SortedSet by List but it didn't work.


问:
Angular 7 Http Client and Asp Core - Cors error

I'm stuck with an error in Angular 7. I'm doing a post request to Asp Core API passing an object, and that API response an Ok() with another object. My code is like that:

```<code>[HttpPost]
public async Task<IActionResult> UserRegistration(ViewModel viewModel){
    //Ignored code
    return Ok(someObject);
}
</code>```

That action is working (apparently) great, it accepts the request, resolves any necessary decision and returns the object exactly as I need.
 Now lets to Angular app, where I have a method in service.ts which call the api, and another one in component.ts which consumes that first method. They are like that:

```<code>//Service
userRegistration(model): Observable<any>{
  return this.http.post(url, model, httpOptions);
}

//Component
saveUser(model){
   this.myService.userRegistration(model).subscribe(
      data => {
         //do something
      },
      error => {
         //show message
      }
   );
}
</code>```

The point here is that Angular has success on calling Asp Core API which saves properly the data in database, returns the correct object, and then Angular gets error:

Access to XMLHttpRequest at 'endpoint' from origin 'origin' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

ERROR HttpErrorResponse {headers: HttpHeaders, status: 0, statusText: "Unknown Error", url: "URL", ok: false, …}

Why is it saying CORS error if API accepts the request and responses it properly? How can I solve that?

答:
Add CORS in startup.cs in backend

```<code>  public void ConfigureServices(IServiceCollection services)
  {
    services.AddMvc();
    services.Configure<MvcOptions>(options => {
      options.Filters.Add(new CorsAuthorizationFilterFactory("AllowMyOrigin"));
    });
  }
</code>```

For more details, follow this article

Body:
I'm stuck with an error in Angular 7. I'm doing a post request to Asp Core API passing an object, and that API response an Ok() with another object. My code is like that:

[HttpPost]
public async Task<IActionResult> UserRegistration(ViewModel viewModel){
    //Ignored code
    return Ok(someObject);
}

That action is working (apparently) great, it accepts the request, resolves any necessary decision and returns the object exactly as I need.
 Now lets to Angular app, where I have a method in service.ts which call the api, and another one in component.ts which consumes that first method. They are like that:

//Service
userRegistration(model): Observable<any>{
  return this.http.post(url, model, httpOptions);
}

//Component
saveUser(model){
   this.myService.userRegistration(model).subscribe(
      data => {
         //do something
      },
      error => {
         //show message
      }
   );
}

The point here is that Angular has success on calling Asp Core API which saves properly the data in database, returns the correct object, and then Angular gets error:

Access to XMLHttpRequest at 'endpoint' from origin 'origin' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

ERROR HttpErrorResponse {headers: HttpHeaders, status: 0, statusText: "Unknown Error", url: "URL", ok: false, …}

Why is it saying CORS error if API accepts the request and responses it properly? How can I solve that?

Body:
Add CORS in startup.cs in backend

  public void ConfigureServices(IServiceCollection services)
  {
    services.AddMvc();
    services.Configure<MvcOptions>(options => {
      options.Filters.Add(new CorsAuthorizationFilterFactory("AllowMyOrigin"));
    });
  }

For more details, follow this article


问:
How to solve this query statement?
Please, help me solve this SQL statement. I spent several days with this issue.

I use SQL Server 2012.

Level = N'First'   --I use Unicode text

```<code>SELECT ss.id,
       ss.SID,
       CONCAT(CONCAT(ss.Name, ' '), ss.Surname) AS Sname,
       ss.Level,
       pp.PID,
       CONCAT(CONCAT(pp.Name1, ' '), pp.Name2) AS Pname,
       ss.Chk
FROM tb_student1 AS ss,
     tb_parent AS pp
WHERE ss.ParentID = pp.PID
  AND ss.Level = N'First'
  AND ss.Chk = 1
  AND NOT EXISTS (SELECT st.SID
                         FROM (SELECT s.id,
                                      s.SID,
                                      CONCAT(CONCAT(s.Name, ' '), s.Surname) AS Sname,
                                      p.PID,
                                      CONCAT(CONCAT(p.Name1, ' '), p.Name2) AS Pname,
                                      s.Chk
                               FROM tb_student1 AS s,
                                    tb_parent AS p
                               WHERE s.ParentID = p.PID
                                 AND s.Level = N'First'
                                 AND s.Chk = 1) AS st ,
                              (SELECT * FROM tb_New_Reg AS r) AS rnp
                         WHERE st.SID = rnp.SID
                           AND rnp.ClassID = 'C104'
                           AND rnp.Year = '2018-2019'
                           AND rnp.Term = 'Jan'
                           AND rnp.Level = N'First'
                           AND st.Chk = 1);
</code>```

If I use (...and ss.SID not exists ...) error here. If I use (...and not exists ....) return null

The first query contains 11 rows, the subquery contain 6 rows

How to solve with it?

答:
```<code>WITH data AS (
    SELECT
        ss.id, ss.SID,
        CONCAT(CONCAT(ss.Name, ' '), ss.Surname) AS Sname,
        ss.Level, pp.PID,
        CONCAT(CONCAT(pp.Name1, ' '), pp.Name2) AS Pname,
        ss.Chk
    FROM tb_student1 AS ss INNER JOIN tb_parent AS pp ON ss.ParentID = pp.PID
    WHERE ss.Level = N'First' AND ss.Chk = 1
)
SELECT * FROM data AS s
WHERE NOT EXISTS (
    SELECT * FROM tb_New_Reg AS rnp
    WHERE s.SID = rnp.SID
        AND rnp.ClassID = 'C104'
        AND rnp.Year = '2018-2019'
        AND rnp.Term = 'Jan'
        AND rnp.Level = N'First'
);
</code>```

I think this is along the lines of what you're looking for. I did break out your main query into a CTE thinking I would be duplicating that same query inside the subquery. It turns out that probably wasn't really necessary but I've left it that way.

I presume that your attempt failed because the inner query is searching across the entire table when what you really intended was to narrow that search to a matching SID from the outer query. You'll probably find it useful to look up correlated subquery for better understanding of the concept.

Title: How to solve this query statement?

Body:
Please, help me solve this SQL statement. I spent several days with this issue.

I use SQL Server 2012.

Level = N'First'   --I use Unicode text

SELECT ss.id,
       ss.SID,
       CONCAT(CONCAT(ss.Name, ' '), ss.Surname) AS Sname,
       ss.Level,
       pp.PID,
       CONCAT(CONCAT(pp.Name1, ' '), pp.Name2) AS Pname,
       ss.Chk
FROM tb_student1 AS ss,
     tb_parent AS pp
WHERE ss.ParentID = pp.PID
  AND ss.Level = N'First'
  AND ss.Chk = 1
  AND NOT EXISTS (SELECT st.SID
                         FROM (SELECT s.id,
                                      s.SID,
                                      CONCAT(CONCAT(s.Name, ' '), s.Surname) AS Sname,
                                      p.PID,
                                      CONCAT(CONCAT(p.Name1, ' '), p.Name2) AS Pname,
                                      s.Chk
                               FROM tb_student1 AS s,
                                    tb_parent AS p
                               WHERE s.ParentID = p.PID
                                 AND s.Level = N'First'
                                 AND s.Chk = 1) AS st ,
                              (SELECT * FROM tb_New_Reg AS r) AS rnp
                         WHERE st.SID = rnp.SID
                           AND rnp.ClassID = 'C104'
                           AND rnp.Year = '2018-2019'
                           AND rnp.Term = 'Jan'
                           AND rnp.Level = N'First'
                           AND st.Chk = 1);

If I use (...and ss.SID not exists ...) error here. If I use (...and not exists ....) return null

The first query contains 11 rows, the subquery contain 6 rows

How to solve with it?

Body:
WITH data AS (
    SELECT
        ss.id, ss.SID,
        CONCAT(CONCAT(ss.Name, ' '), ss.Surname) AS Sname,
        ss.Level, pp.PID,
        CONCAT(CONCAT(pp.Name1, ' '), pp.Name2) AS Pname,
        ss.Chk
    FROM tb_student1 AS ss INNER JOIN tb_parent AS pp ON ss.ParentID = pp.PID
    WHERE ss.Level = N'First' AND ss.Chk = 1
)
SELECT * FROM data AS s
WHERE NOT EXISTS (
    SELECT * FROM tb_New_Reg AS rnp
    WHERE s.SID = rnp.SID
        AND rnp.ClassID = 'C104'
        AND rnp.Year = '2018-2019'
        AND rnp.Term = 'Jan'
        AND rnp.Level = N'First'
);

I think this is along the lines of what you're looking for. I did break out your main query into a CTE thinking I would be duplicating that same query inside the subquery. It turns out that probably wasn't really necessary but I've left it that way.

I presume that your attempt failed because the inner query is searching across the entire table when what you really intended was to narrow that search to a matching SID from the outer query. You'll probably find it useful to look up correlated subquery for better understanding of the concept.