问:
How to return specified text if true in pandas column
I am creating a dataframe that looks at the column 'Study Title' and returns a text category in a new column 'Categories' if the 'Study Title' column contains specific text in the string. For example, if the study title contains the text 'Child Care' or 'Head Start' then it will return a value if true in the new 'Categories' column with 'Child Care' for 'Child Care' and 'School Readiness' for 'Head Start', etc. How can I add multiple conditions with more values if true so that I have a list of strings to look for that returns a value I assign?

This is an example of what I would like the output to look like where the column 'Categories' returns the value 'Child Care' if the title contains the words 'Child Care'.

I have more conditions I want to include for other title considerations like 'School Readiness' for 'Head Start'. So far what I have only returns a true/false value based on the one condition for 'Child Care'.

This is the code I have so far:

```<code>df['Categories'] = df['Study Title'].str.contains(r'Child Care', na=True)
df
</code>```

Copy-Paste CSV Sample

```<code>
Study Title,URL,Funding Agency,Category
"American Indian and Alaska Native Head Start Family and Child Experiences Survey, 2015",https://www.icpsr.umich.edu/web/ICPSR/studies/36804,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
American Indian and Alaska Native Head Start Family and Child Experiences Survey 2019 (AIAN FACES 2019),https://www.icpsr.umich.edu/web/ICPSR/studies/38028,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
"Carolina Abecedarian Project (ABC) and the Carolina Approach to Responsive Education (CARE), Age 21 Follow Up Study, 1993 - 2003",https://www.icpsr.umich.edu/web/ICPSR/studies/32262,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
"Child Care and Development Fund (CCDF) Policies Database, 2009",https://www.icpsr.umich.edu/web/ICPSR/studies/32261,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
"Child Care and Development Fund (CCDF) Policies Database, 2011",https://www.icpsr.umich.edu/web/ICPSR/studies/34390,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
"Child Care and Development Fund (CCDF) Policies Database, 2012",https://www.icpsr.umich.edu/web/ICPSR/studies/34902,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
"Child Care and Development Fund (CCDF) Policies Database, 2013",https://www.icpsr.umich.edu/web/ICPSR/studies/35482,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
"Child Care and Development Fund (CCDF) Policies Database, 2014",https://www.icpsr.umich.edu/web/ICPSR/studies/36276,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
"Child Care and Development Fund (CCDF) Policies Database, 2015",https://www.icpsr.umich.edu/web/ICPSR/studies/36581,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
</code>```

答:
What you're trying to do is essentially a mapping, which a basic membership test is not suitable for. You need to define a mapping function:

In [8]: def determine_category(title):
   ...:     if "Child Care" in title:
   ...:         return "Child"
   ...:     elif "Head Start" in title:
   ...:         return "School Readiness"
   ...:     return "no matching category"
   ...:

In [9]: df["Study Title"].apply(determine_category)
Out[9]:
0        School Readiness
1        School Readiness
2    no matching category
3                   Child
4                   Child
5                   Child
6                   Child
7                   Child
8                   Child
Name: Study Title, dtype: object
```

Body:
I am creating a dataframe that looks at the column 'Study Title' and returns a text category in a new column 'Categories' if the 'Study Title' column contains specific text in the string. For example, if the study title contains the text 'Child Care' or 'Head Start' then it will return a value if true in the new 'Categories' column with 'Child Care' for 'Child Care' and 'School Readiness' for 'Head Start', etc. How can I add multiple conditions with more values if true so that I have a list of strings to look for that returns a value I assign?

This is an example of what I would like the output to look like where the column 'Categories' returns the value 'Child Care' if the title contains the words 'Child Care'.

I have more conditions I want to include for other title considerations like 'School Readiness' for 'Head Start'. So far what I have only returns a true/false value based on the one condition for 'Child Care'.

This is the code I have so far:

df['Categories'] = df['Study Title'].str.contains(r'Child Care', na=True)
df

Copy-Paste CSV Sample

Study Title,URL,Funding Agency,Category
"American Indian and Alaska Native Head Start Family and Child Experiences Survey, 2015",https://www.icpsr.umich.edu/web/ICPSR/studies/36804,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
American Indian and Alaska Native Head Start Family and Child Experiences Survey 2019 (AIAN FACES 2019),https://www.icpsr.umich.edu/web/ICPSR/studies/38028,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
"Carolina Abecedarian Project (ABC) and the Carolina Approach to Responsive Education (CARE), Age 21 Follow Up Study, 1993 - 2003",https://www.icpsr.umich.edu/web/ICPSR/studies/32262,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
"Child Care and Development Fund (CCDF) Policies Database, 2009",https://www.icpsr.umich.edu/web/ICPSR/studies/32261,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
"Child Care and Development Fund (CCDF) Policies Database, 2011",https://www.icpsr.umich.edu/web/ICPSR/studies/34390,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
"Child Care and Development Fund (CCDF) Policies Database, 2012",https://www.icpsr.umich.edu/web/ICPSR/studies/34902,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
"Child Care and Development Fund (CCDF) Policies Database, 2013",https://www.icpsr.umich.edu/web/ICPSR/studies/35482,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
"Child Care and Development Fund (CCDF) Policies Database, 2014",https://www.icpsr.umich.edu/web/ICPSR/studies/36276,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,
"Child Care and Development Fund (CCDF) Policies Database, 2015",https://www.icpsr.umich.edu/web/ICPSR/studies/36581,United States Department of Health and Human Services. Administration for Children and Families. Office of Planning Research and Evaluation,


来源: stackexchange

问:
How to run Chrome from Android Studio (in debug mode) using a custom Google user profile
I am developing a flutter web app in Anroid Studio.
When testing the app and run it from Android Studio, I would like to it to run in Chrome using my current user profile, as this user profile has all necessary passwords stored.

Whatever I try, when runnung the app from Android Studio, Anroid Studio would start a new Chrome instance with an empty Profile. This makes sense, as Android Studio starts a new virtual device which don't have any information on user profiles stored.
This makes my developement-test cycles slow because I need to login with my google account credentials each time.

I learned the Anroid Studio Flutter plugin starts Flutter applications using the flutter run command from the Flutter SDK.
What I tried so far, is change the Settings in Anroid Studio: Settings > Tools > Web Browsers > Chrome
both to use a custom user data directory:
"%LOCALAPPDATA%\Google\Chrome\User Data" which I get from typing chrome://version/ in my current chrome browser.
and/or to use command line options:
--profile-directory=Default

This didn't help, and also the JetBrain Support says,
Thanks in advance for your help.

答:
You can run the app using flutter run -d web-server. It will start a local web server and provide you with a URL that you can open in any browser.

```<code>$ flutter run -d web-server

Launching lib/main.dart on Web Server in debug mode...
Waiting for connection from debug service on Web Server...         21.3s
lib/main.dart is being served at http://localhost:56153
The web-server device requires the Dart Debug Chrome extension for debugging. Consider using the Chrome or Edge devices for an improved development
workflow.

  To hot restart changes while running, press "r" or "R".
For a more detailed help message, press "h". To quit, press "q".
</code>```

ContentLicense: CC BY-SA 4.0

Body:
You can run the app using flutter run -d web-server. It will start a local web server and provide you with a URL that you can open in any browser.

$ flutter run -d web-server

Launching lib/main.dart on Web Server in debug mode...
Waiting for connection from debug service on Web Server...         21.3s
lib/main.dart is being served at http://localhost:56153
The web-server device requires the Dart Debug Chrome extension for debugging. Consider using the Chrome or Edge devices for an improved development
workflow.

  To hot restart changes while running, press "r" or "R".
For a more detailed help message, press "h". To quit, press "q".


问:
create several lists with position
I have this super simple code to generate a list in python. I want to make several lists like this

```<code>0: [0, 2]
1: [2, 4]
2: [4, 6]
</code>```

it is possible . thanks

```<code>n = range(0,169,2)
num_list=list(n)
print (num_list)
</code>```

答:
num_list = []
for n in range(0, 169, 2):
  num_list.append([n, n+2])

print(num_list)
```
<p>Does the trick.</p>
<p>If you want them to have dictionary keys:</p>
<pre class="lang-py prettyprint-override"><code>num_list = {}
for n in range(0, 169, 2):
  num_list[int(n/2)] = [n, n+2]

print(num_list)
</code>```

Either way, num_list[48] returns [96, 98].

Title: create several lists with position

Body:
I have this super simple code to generate a list in python. I want to make several lists like this

0: [0, 2]
1: [2, 4]
2: [4, 6]

it is possible . thanks

n = range(0,169,2)
num_list=list(n)
print (num_list)

Body:
num_list = []
for n in range(0, 169, 2):
  num_list.append([n, n+2])

print(num_list)

Does the trick.

If you want them to have dictionary keys:

num_list = {}
for n in range(0, 169, 2):
  num_list[int(n/2)] = [n, n+2]

print(num_list)

Either way, num_list[48] returns [96, 98].


问:
Snowflake - How to create summary table containing unique records
I'm looking for some Snowflake syntax assistance in how to generate a summary table or view from an existing table. My summary table should have 1 row per unique id from the existing table along with boolean values indicating if the various milestones (as per the summary column names) have been hit. Any help is appreciated as I am a Snowflake novice. Thanks.

Existing Table

Desired Summary Table/View

答:
So using Himanshu's data, thank you:

```<code>WITH fake_data(id, updated, pipeline_id, stage_id) AS (
    SELECT column1, to_date(column2,'mm/dd/yyyy hh:mm:ss'), column3, column4
    FROM VALUES
        (1111, '02/01/2022 09:01:00', 'A', '1' ),
        (1111, '02/01/2022 10:01:00', 'A', '2' ),
        (1111, '02/01/2022 11:01:00', 'B', '5' ),
        (2222, '02/02/2022 13:01:00', 'A', '1' ),
        (2222, '02/03/2022 18:01:00', 'B', '5' ),
        (2222, '02/04/2022 07:01:00', 'B', '6' ),
        (3333, '02/02/2022 14:01:00', 'A', '1' ),
        (3333, '02/03/2022 18:01:00', 'A', '2' ),
        (3333, '02/03/2022 07:01:00', 'C', '7' ),
        (3333, '02/03/2022 21:01:00', 'C', '8' ),
        (3333, '02/05/2022 17:01:00', 'C', '9' )
)
</code>```

we are doing an aggregation across each id and we want to use COUNT_IF to see how many row meet out criteria, and if it is >0 we are happy

```<code>SELECT
    id,
    count_if(pipeline_id='A')>0 AS hit_stage_a,
    count_if(pipeline_id='B')>0 AS hit_stage_b,
    count_if(pipeline_id='C')>0 AS hit_stage_c,
    count_if(stage_id='4')>0 AS hit_stage_4,
    count_if(stage_id='5')>0 AS hit_stage_5,
    count_if(stage_id='6')>0 AS hit_stage_6
FROM fake_data
GROUP BY 1
ORDER BY 1;
</code>```

gives:

ID
HIT_STAGE_A
HIT_STAGE_B
HIT_STAGE_C
HIT_STAGE_4
HIT_STAGE_5
HIT_STAGE_6

1111
TRUE
TRUE
FALSE
FALSE
TRUE
FALSE

2222
TRUE
TRUE
FALSE
FALSE
TRUE
TRUE

3333
TRUE
FALSE
TRUE
FALSE
FALSE
FALSE

Body:
So using Himanshu's data, thank you:

WITH fake_data(id, updated, pipeline_id, stage_id) AS (
    SELECT column1, to_date(column2,'mm/dd/yyyy hh:mm:ss'), column3, column4
    FROM VALUES
        (1111, '02/01/2022 09:01:00', 'A', '1' ),
        (1111, '02/01/2022 10:01:00', 'A', '2' ),
        (1111, '02/01/2022 11:01:00', 'B', '5' ),
        (2222, '02/02/2022 13:01:00', 'A', '1' ),
        (2222, '02/03/2022 18:01:00', 'B', '5' ),
        (2222, '02/04/2022 07:01:00', 'B', '6' ),
        (3333, '02/02/2022 14:01:00', 'A', '1' ),
        (3333, '02/03/2022 18:01:00', 'A', '2' ),
        (3333, '02/03/2022 07:01:00', 'C', '7' ),
        (3333, '02/03/2022 21:01:00', 'C', '8' ),
        (3333, '02/05/2022 17:01:00', 'C', '9' )
)

we are doing an aggregation across each id and we want to use COUNT_IF to see how many row meet out criteria, and if it is >0 we are happy

SELECT
    id,
    count_if(pipeline_id='A')>0 AS hit_stage_a,
    count_if(pipeline_id='B')>0 AS hit_stage_b,
    count_if(pipeline_id='C')>0 AS hit_stage_c,
    count_if(stage_id='4')>0 AS hit_stage_4,
    count_if(stage_id='5')>0 AS hit_stage_5,
    count_if(stage_id='6')>0 AS hit_stage_6
FROM fake_data
GROUP BY 1
ORDER BY 1;

gives:

ID
HIT_STAGE_A
HIT_STAGE_B
HIT_STAGE_C
HIT_STAGE_4
HIT_STAGE_5
HIT_STAGE_6

1111
TRUE
TRUE
FALSE
FALSE
TRUE
FALSE

2222
TRUE
TRUE
FALSE
FALSE
TRUE
TRUE

3333
TRUE
FALSE
TRUE
FALSE
FALSE
FALSE


问:
Alert if last closed candle is x larger than the average of y previous candles
How can I implement a function which alerts me if the latest closed candle is x, say 30%, larger than the average of y, say 300, previous candles?</p>

答:
You can use the ta.sma() function to calculate the average.

```<code>//@version=5
indicator("My script", overlay=true)

avg = ta.sma(close, 300)
large = close > (1.3 * avg)
alertcondition(large, "Large", "Large")
</code>```

Body:
You can use the ta.sma() function to calculate the average.

//@version=5
indicator("My script", overlay=true)

avg = ta.sma(close, 300)
large = close > (1.3 * avg)
alertcondition(large, "Large", "Large")


问:
A generic error occurred in GDI+
I loaded an image into a Picture Box using:

```<code>picturebox1.Image = Image.FromFile()
</code>```

and I save it by using:

```<code>Bitmap bm = new Bitmap(pictureBox1.Image);
bm.Save(FileName, ImageFormat.Bmp);
</code>```

It works perfectly fine when creating a new file, but when I try to replace the existing image, I get thrown the following runtime error:

A generic error occurred in GDI+

So what can I do to solve this problem??

答:
try this.</p>

```<code>picturebox1.Image = Image.FromFile(FileName);
Bitmap bm = new Bitmat(pictureBox1.Image);
Image img = (Image)b;
img.Save(FileName, ImageFormat.Bmp);
</code>```

ContentLicense: CC BY-SA 3.0

Title: A generic error occurred in GDI+

Body:
I loaded an image into a Picture Box using:

picturebox1.Image = Image.FromFile()

and I save it by using:

Bitmap bm = new Bitmap(pictureBox1.Image);
bm.Save(FileName, ImageFormat.Bmp);

It works perfectly fine when creating a new file, but when I try to replace the existing image, I get thrown the following runtime error:

A generic error occurred in GDI+

So what can I do to solve this problem??

Body:
try this.

picturebox1.Image = Image.FromFile(FileName);
Bitmap bm = new Bitmat(pictureBox1.Image);
Image img = (Image)b;
img.Save(FileName, ImageFormat.Bmp);


问:
Wagtail. Ability to edit html within editor
My editors want to have an ability to edit raw html within editor.
For example, we can have simple list markup:

```<code><ul>
   <li>Some text<li>
</ul>
</code>```

Or with custom classes and event tags:

```<code><ul>
   <li class="my_class">Some text
       <span class="special">Some additional info</span>
   <li>
</ul>
</code>```

And so on.

I know that built-in Draftail doesnt support html editing.
But using third-party editor comes with problem:

Losing integration with system. For example, i cant just put image like with draftail. Or, preview mode cant handle all functionality of third-party editor.

So, maybe someone has working solution for this situation.
I thought about using special StreamField panels, but it seems to be overhead of blocks. Dont like this idea of having lots of blocks with little differences

答:
At the end i made my own solution
Simple rewrite some methods in wagtail`s Html DbWhiteLister and HtmlConverter.
This gives me an opportunity to allow any tags with any attributes.

https://github.com/Chenger1/wagtailtinymce.-Full-Rich-Text-Editor

Body:
My editors want to have an ability to edit raw html within editor.
For example, we can have simple list markup:

Some text

Or with custom classes and event tags:

Some text
       Some additional info

And so on.

I know that built-in Draftail doesnt support html editing.
But using third-party editor comes with problem:

Losing integration with system. For example, i cant just put image like with draftail. Or, preview mode cant handle all functionality of third-party editor.

So, maybe someone has working solution for this situation.
I thought about using special StreamField panels, but it seems to be overhead of blocks. Dont like this idea of having lots of blocks with little differences


问:
Using "tidyverse" to replace NA values in columns not working correctly nor giving an error message. [R]
I am trying to replace the NA values in columns with 'UNK' to be able to execute a logistic regression.

Here is the code and their outputs respectively. I just want to lay out each step I took for context (It is important to note I did not include every column, but the same issue happens with all of the columns):

```<code>donors <- read_csv("donors.csv", col_types = "nnffnnnnnnnnffffffffff")

glimpse(donors)
</code>```
```<code>Rows: 95,412
Columns: 22
$ age                     <dbl> 60, 46, NA, 70, 78, NA, 38, ~
$ numberChildren          <dbl> NA, 1, NA, NA, 1, NA, 1, NA,~
$ incomeRating            <fct> NA, 6, 3, 1, 3, NA, 4, 2, 3,~
</code>```

Here, I just singled out the factored features to see visualize them more clearly:

```<code>donors %>% keep( is.factor) %>% summary()
</code>```
```<code>  incomeRating    wealthRating   inHouseDonor
 NA     :21286   NA     :44732   FALSE:88709
 5      :15451   9      : 7585   TRUE : 6703
 2      :13114   8      : 6793
 4      :12732   7      : 6198
 1      : 9022   6      : 5825
 3      : 8558   5      : 5280
 (Other):15249   (Other):18999
</code>```

Now, I try to replace all of the NA values in the incomeRating column (and other columns) with 'UNK':

```<code>donors <- donors %>% mutate( incomeRating = as.character( incomeRating))
%>% mutate( incomeRating = as.factor( ifelse( is.na( incomeRating), 'UNK', incomeRating)))
</code>```

There is no error message, but when I retrieve the proportional values table like so, the NA's are not replaced:

```<code>donors%>%
  select(incomeRating) %>%
  table() %>%
  prop.table()
</code>```
```<code>         1          2          3          4          5
0.09455834 0.13744602 0.08969522 0.13344233 0.16193980
         6          7         NA
0.08152014 0.07830252 0.22309563
</code>```

Again, this happens with all columns. I believe that R reads the NA as actual values, therefore I cannot use the is.na() command to read those values. If this is the case, what is a solution for this? Thank you ahead of time.

答:
Try fct_explicit_na from forcats package: Code not tested!

```<code>library(forcats)
library(dplyr)

donors <- donors %>%
  mutate(incomeRating = fct_explicit_na(incomeRating, "UNK")
</code>```

Body:
I am trying to replace the NA values in columns with 'UNK' to be able to execute a logistic regression.

Here is the code and their outputs respectively. I just want to lay out each step I took for context (It is important to note I did not include every column, but the same issue happens with all of the columns):

donors <- read_csv("donors.csv", col_types = "nnffnnnnnnnnffffffffff")

glimpse(donors)

Rows: 95,412
Columns: 22
$ age                     <dbl> 60, 46, NA, 70, 78, NA, 38, ~
$ numberChildren          <dbl> NA, 1, NA, NA, 1, NA, 1, NA,~
$ incomeRating            <fct> NA, 6, 3, 1, 3, NA, 4, 2, 3,~

Here, I just singled out the factored features to see visualize them more clearly:

donors %>% keep( is.factor) %>% summary()

  incomeRating    wealthRating   inHouseDonor
 NA     :21286   NA     :44732   FALSE:88709
 5      :15451   9      : 7585   TRUE : 6703
 2      :13114   8      : 6793
 4      :12732   7      : 6198
 1      : 9022   6      : 5825
 3      : 8558   5      : 5280
 (Other):15249   (Other):18999

Now, I try to replace all of the NA values in the incomeRating column (and other columns) with 'UNK':

donors <- donors %>% mutate( incomeRating = as.character( incomeRating))
%>% mutate( incomeRating = as.factor( ifelse( is.na( incomeRating), 'UNK', incomeRating)))

There is no error message, but when I retrieve the proportional values table like so, the NA's are not replaced:

donors%>%
  select(incomeRating) %>%
  table() %>%
  prop.table()

         1          2          3          4          5
0.09455834 0.13744602 0.08969522 0.13344233 0.16193980
         6          7         NA
0.08152014 0.07830252 0.22309563

Again, this happens with all columns. I believe that R reads the NA as actual values, therefore I cannot use the is.na() command to read those values. If this is the case, what is a solution for this? Thank you ahead of time.

Body:
Try fct_explicit_na from forcats package: Code not tested!

library(forcats)
library(dplyr)

donors <- donors %>%
  mutate(incomeRating = fct_explicit_na(incomeRating, "UNK")


问:
Exclude Certain Year from a Date Column
I'm using Oracle Database version 12.1.0.2.0. I'm running a query that returns two columns, "Permit #" and "Date Issued."

```<code>SELECT id.B1_ALT_ID AS "Permit #", process.REC_DATE AS "Date Issued"
FROM process
LEFT JOIN id ON ( id.B1_PER_ID1 = process.B1_PER_ID1 AND id.B1_PER_ID3 = process.B1_PER_ID3 )
WHERE id.B1_PER_ID1 like 'B2021%'
AND process.SD_APP_DES = 'Issued';
</code>```

Sample result:

Question: How can I exclude records from the year 2022?

I tried adding to my original query the following clause to exclude the year:

```<code>AND TO_CHAR( process.REC_DATE ) NOT LIKE '%2022%';
</code>```

But I received the same result. How can I results that don't include 2022 in the "Date Issued" column?

答:
WHERE process.rec_date <  DATE '2022-01-01'
OR    process.rec_date >= DATE '2023-01-01'
```
<p><em>(Note: this may use an index on the <code>date_issued</code> column. The options below would require a function-based index to use an index.)</em></p>
<p>or</p>
<pre class="lang-sql prettyprint-override"><code>WHERE EXTRACT(YEAR FROM process.rec_date) != 2022
</code>```

or

WHERE TO_CHAR(process.rec_date, 'YYYY') != '2022'
```
<p>or</p>
<pre class="lang-sql prettyprint-override"><code>WHERE TRUNC(process.rec_date, 'YYYY') != DATE '2022-01-01'
</code>```

Title: Exclude Certain Year from a Date Column

Body:
I'm using Oracle Database version 12.1.0.2.0. I'm running a query that returns two columns, "Permit #" and "Date Issued."

SELECT id.B1_ALT_ID AS "Permit #", process.REC_DATE AS "Date Issued"
FROM process
LEFT JOIN id ON ( id.B1_PER_ID1 = process.B1_PER_ID1 AND id.B1_PER_ID3 = process.B1_PER_ID3 )
WHERE id.B1_PER_ID1 like 'B2021%'
AND process.SD_APP_DES = 'Issued';

Sample result:

Question: How can I exclude records from the year 2022?

I tried adding to my original query the following clause to exclude the year:

AND TO_CHAR( process.REC_DATE ) NOT LIKE '%2022%';

But I received the same result. How can I results that don't include 2022 in the "Date Issued" column?

Body:
WHERE process.rec_date <  DATE '2022-01-01'
OR    process.rec_date >= DATE '2023-01-01'

(Note: this may use an index on the date_issued column. The options below would require a function-based index to use an index.)

or

WHERE EXTRACT(YEAR FROM process.rec_date) != 2022

or

WHERE TO_CHAR(process.rec_date, 'YYYY') != '2022'

or

WHERE TRUNC(process.rec_date, 'YYYY') != DATE '2022-01-01'


问:
Tableau Angular2 Integration
Has anyone tried building an Angular2 application with tableau visualizations integrated in it using the Tableau JavaScript API?

According to the documentation, you're supposed to include the following script in your file which will create a tableau global variable:

I'm not sure how to access this global variable within an Angular2 class.

答:
For anyone who wants to embed Tableau visualizations in Angular apps, ngx-tableau provides a simple interface to the Tableau JavaScript API.

It has multiple options for configuring the connection to Tableau Server, handling authentication tickets, adding filters to reports, and managing different sites.

Disclaimer: I am the main contributor to the library.

Title: Tableau Angular2 Integration


问:
Postgresql - How to improve low cache hit ratio
If the cache hit ratio for a database is below 50%. Are there other ways than increasing the shared_buffers to improve this ratio</p>

答:
Look at your explain plans. Hit and Read are cache and disk reads respectively. There can be a number of reasons for this the most common being lack of indexes, or bloated indexes. Poorly vacuumed/analyzed tables which have a lot of dead rows misleading the optimiser.

Look at the pg_stat_statements view. This will show you the blocks hit and read for the most common statements that are executed. You can use this information to determine appropriate index usage.

Good management of your tables and queries will help lower your read over hit rate.


问:
use lambda expression to count sum of f(i)
I have function which count sum of f(i) for i in

```<code>(define (F a b f)
      (if (> a b)
          0
          (+ (f a) (F (+ a 1) b f))
        )
)
</code>```

Now I want transform it to use lambda like there: https://mitpress.mit.edu/sites/default/files/sicp/full-text/book/book-Z-H-12.html#%_sec_1.3.2

So I created:

```<code>(define (F7 a b f)
    (sum (lambda (x) (f x)) a
           (lambda (x) (+ x 1)) b
         )
)
</code>```

But got error UnhandledException: sum() takes at most 2 arguments (4 given)
After research I tried to change it to:

```<code>(define (F7 a b f)
    (sum ((lambda (x) (f x)) a)
           ((lambda (x) (+ x 1)) b)
         )
)
</code>```

I got error: UnhandledException: 'int' object is not iterable

So I tried run code from link but when I put into jupyter code

```<code>(define (pi-sum a b)
  (sum (lambda (x) (/ 1.0 (* x (+ x 2))))
       a
       (lambda (x) (+ x 4))
       b))
</code>```

I got error

```<code>Traceback (most recent call last):
  File "In [36]", line 14, col 1, in 'integral'
  File "In [35]", line 4, col 6, in 'sum'
UnhandledException: sum() takes at most 2 arguments (4 given)
</code>```

答:
```<code>(define (F7 a b f)
  (sum (lambda (x)(f x))
       a
       (lambda (x)(+ 1 x))
       b))
</code>```

Body:
I have function which count sum of f(i) for i in

(define (F a b f)
      (if (> a b)
          0
          (+ (f a) (F (+ a 1) b f))
        )
)

Now I want transform it to use lambda like there: https://mitpress.mit.edu/sites/default/files/sicp/full-text/book/book-Z-H-12.html#%_sec_1.3.2

So I created:

(define (F7 a b f)
    (sum (lambda (x) (f x)) a
           (lambda (x) (+ x 1)) b
         )
)

But got error UnhandledException: sum() takes at most 2 arguments (4 given)
After research I tried to change it to:

(define (F7 a b f)
    (sum ((lambda (x) (f x)) a)
           ((lambda (x) (+ x 1)) b)
         )
)

I got error: UnhandledException: 'int' object is not iterable

So I tried run code from link but when I put into jupyter code

(define (pi-sum a b)
  (sum (lambda (x) (/ 1.0 (* x (+ x 2))))
       a
       (lambda (x) (+ x 4))
       b))

I got error

Traceback (most recent call last):
  File "In [36]", line 14, col 1, in 'integral'
  File "In [35]", line 4, col 6, in 'sum'
UnhandledException: sum() takes at most 2 arguments (4 given)


问:
List<Integer> to List<Long> conversion in Java 11 using Streams
I have a list of Integer and would like to convert into a list of Long (or any other type) by using Streams. I tried multiple options but all failed. How do I do that?</p>
```<code>List<Integer> arr = new ArrayList<Integer>();
    arr.add(1);
    arr.add(3);
    arr.add(5);
    arr.add(7);
    arr.add(9);
</code>```

答:
```<code>List<Integer> arr = new ArrayList<>();
...

List<Long> longs = arr.stream()    // Stream<Integer>
    .mapToLong(Integer::longValue) // LongStream
    .boxed()                       // Stream<Long>
    .collect(Collectors.toList());

List<Long> longs = arr.stream()            // Stream<Integer>
    .map(n -> Long.valueOf(n.longValue())) // Stream<Long>
    .collect(Collectors.toList());
</code>```

The latter uses Integer.longValue(), though Integer.intValue() would have done also.
Relying on Integer to int unboxing and then Long.valueOf widening int to long, is too convoluted. Not sure whether there is a speed/memory difference. Should not.

Alternatively with a different data structure for arr:

```<code>IntStream arr = IntStream.of(1, 3, 5, 7, 9);
List<Long> longs = arr.asLongStream() // LongStream
    .boxed()                          // Stream<Long>
    .collect(Collectors.toList());
</code>```

Basically you have to map an Integer to a Long.

Body:
I have a list of Integer and would like to convert into a list of Long (or any other type) by using Streams. I tried multiple options but all failed. How do I do that?

List<Integer> arr = new ArrayList<Integer>();
    arr.add(1);
    arr.add(3);
    arr.add(5);
    arr.add(7);
    arr.add(9);

Body:
List<Integer> arr = new ArrayList<>();
...

List<Long> longs = arr.stream()    // Stream<Integer>
    .mapToLong(Integer::longValue) // LongStream
    .boxed()                       // Stream<Long>
    .collect(Collectors.toList());

List<Long> longs = arr.stream()            // Stream<Integer>
    .map(n -> Long.valueOf(n.longValue())) // Stream<Long>
    .collect(Collectors.toList());

The latter uses Integer.longValue(), though Integer.intValue() would have done also.
Relying on Integer to int unboxing and then Long.valueOf widening int to long, is too convoluted. Not sure whether there is a speed/memory difference. Should not.

Alternatively with a different data structure for arr:

IntStream arr = IntStream.of(1, 3, 5, 7, 9);
List<Long> longs = arr.asLongStream() // LongStream
    .boxed()                          // Stream<Long>
    .collect(Collectors.toList());

Basically you have to map an Integer to a Long.


问:
RAM-efficient set of 1 billion of short strings, and fast lookup
I want to be able to test, as fast as possible, if a given string is a member of a set S of 1 billion of strings:

each string is of length 10 on average, so the whole data is ~ 10 GB, the max length is 100

99% of the strings uses only a-z A-Z 0-9 + a few special chars (.!-;\/), in some rare cases there are other chars

if necessary, the data could be sorted:

```<code>aaaaa
...
helloworld
hellu
help
...
zzzzz
</code>```

and we could maybe use a prefix tree / trie, but I'm not sure if it's the best solution here.

Question: Is there a way to work with an in-memory compressed set with Python, possibly with the standard library, with fast lookup?

The goal is to test 'hello' in S as fast as possible.

I know how to do this with 10 GB of RAM, but out of curiosity, I'd like to know if this is possible with, say, only 1 or 2 GB of RAM:

```<code>S = set()
with open('db.txt', 'r') as f:
   for l in f:
       S.add(l)
print('hello' in S)
</code>```

答:
You should be able to compress your strings and work with the compressed version.  However, just how much memory you save will depend on how compressible your dataset is.

I would suggest a custom compression system: build a single compression dictionary based on your entire string corpus, optimized to minimize the total size of dictionary + compressed string storage.

I would use a hash table to look up your compressed strings, not a tree.  Each hash bucket can just be a bunch of compressed data, scanned sequentially from the beginning.  For buckets that happen to have a lot of long strings, provide an overflow pointer to the rest of the data; the in-table bucket size is a performance parameter -- or you can slide that size all the way to 0 for a fully indirect table.

The sequential scan sounds problematic from a performance point of view, but both the initial hash lookup and the sequential scan are fast on current hardware.  The number of hash buckets is what (statistically) limits the sequential scan length; it is another performance parameter.

To implement this in Python, I'd do the hash indexing manually, and store the hash table as an array of bytestrings.  I think this will effectively be the indirect table mentioned above, which might not be as fast as possible but it should be pretty good.

```<code># Python-like pseudocode for building the table
compression_dict = compressor.train(strings)
hash_table = [BytesIO() for i in range(table_size)]
for s in strings:
    hash_table[hash(s) % table_size].write(compression_dict.compress(s))
</code>```

Body:
I want to be able to test, as fast as possible, if a given string is a member of a set S of 1 billion of strings:

each string is of length 10 on average, so the whole data is ~ 10 GB, the max length is 100

99% of the strings uses only a-z A-Z 0-9 + a few special chars (.!-;\/), in some rare cases there are other chars

if necessary, the data could be sorted:

aaaaa
...
helloworld
hellu
help
...
zzzzz

and we could maybe use a prefix tree / trie, but I'm not sure if it's the best solution here.

Question: Is there a way to work with an in-memory compressed set with Python, possibly with the standard library, with fast lookup?

The goal is to test 'hello' in S as fast as possible.

I know how to do this with 10 GB of RAM, but out of curiosity, I'd like to know if this is possible with, say, only 1 or 2 GB of RAM:

S = set()
with open('db.txt', 'r') as f:
   for l in f:
       S.add(l)
print('hello' in S)

Body:
You should be able to compress your strings and work with the compressed version.  However, just how much memory you save will depend on how compressible your dataset is.

I would suggest a custom compression system: build a single compression dictionary based on your entire string corpus, optimized to minimize the total size of dictionary + compressed string storage.

I would use a hash table to look up your compressed strings, not a tree.  Each hash bucket can just be a bunch of compressed data, scanned sequentially from the beginning.  For buckets that happen to have a lot of long strings, provide an overflow pointer to the rest of the data; the in-table bucket size is a performance parameter -- or you can slide that size all the way to 0 for a fully indirect table.

The sequential scan sounds problematic from a performance point of view, but both the initial hash lookup and the sequential scan are fast on current hardware.  The number of hash buckets is what (statistically) limits the sequential scan length; it is another performance parameter.

To implement this in Python, I'd do the hash indexing manually, and store the hash table as an array of bytestrings.  I think this will effectively be the indirect table mentioned above, which might not be as fast as possible but it should be pretty good.

# Python-like pseudocode for building the table
compression_dict = compressor.train(strings)
hash_table = [BytesIO() for i in range(table_size)]
for s in strings:
    hash_table[hash(s) % table_size].write(compression_dict.compress(s))


问:
How to count rows with SELECT COUNT(*) with SQLAlchemy?
I'd like to know if it's possible to generate a SELECT COUNT(*) FROM TABLE statement in SQLAlchemy without explicitly asking for it with execute().
If I use:

session.query(table).count()

then it generates something like:

```<code>SELECT count(*) AS count_1 FROM
    (SELECT table.col1 as col1, table.col2 as col2, ... from table)
</code>```

which is significantly slower in MySQL with InnoDB. I am looking for a solution that doesn't require the table to have a known primary key, as suggested in Get the number of rows in table using SQLAlchemy.

答:
I'm not clear on what you mean by "without explicitly asking for it with execute()" So this might be exactly what you are not asking for.
OTOH, this might help others.

You can just run the textual SQL:

```<code>your_query="""
SELECT count(*) from table
"""
the_count = session.execute(text(your_query)).scalar()
</code>```

Body:
I'd like to know if it's possible to generate a SELECT COUNT(*) FROM TABLE statement in SQLAlchemy without explicitly asking for it with execute().
If I use:

session.query(table).count()

then it generates something like:

SELECT count(*) AS count_1 FROM
    (SELECT table.col1 as col1, table.col2 as col2, ... from table)

which is significantly slower in MySQL with InnoDB. I am looking for a solution that doesn't require the table to have a known primary key, as suggested in Get the number of rows in table using SQLAlchemy.

Body:
I'm not clear on what you mean by "without explicitly asking for it with execute()" So this might be exactly what you are not asking for.
OTOH, this might help others.

You can just run the textual SQL:

your_query="""
SELECT count(*) from table
"""
the_count = session.execute(text(your_query)).scalar()


问:
Is there any way to exclude any accounts mentioning specific terms from twitter search results
I'm trying to build some advanced twitter searches to monitor certain items.
At present, they get thrown off by people using one of the keywords in the username.

I can't just mute / block all those users, as some of them are ones I follow.

At the moment, the only way I can see to completely exclude an account from results is:

```<code>    -(@username) -from:username
</code>```

This works, but the approach rapidly gets lengthy if there are quite a few names to exclude.
It also doesn't allow me to exclude all accounts containing username - i.e. @username1 @username2

One thought would be to create a list of accounts to exclude and then filter the list. That doesn't seem to work though as it won't help in excluding replies to these people.

答:
I was confronting a similar problem and the way I ended up solving it is exactly as you suggested with a tiny addition:

Like you, I created a list for accounts with usernames similar to my keywords, then used the list: operator with my list's ID, in minus - to exclude tweets from the list members.

I also ended up getting lots of replies to accounts on my list, so i used the exclude:replies operator and it worked quite good.

[keyword(s)] -list:[list_id] exclude:replies

The only false positives I kept on getting were tweets which quoted members of my list but adding exclude:quote did more damage than help as it filtered also legitimate tweets with benign quotes.

Body:
I'm trying to build some advanced twitter searches to monitor certain items.
At present, they get thrown off by people using one of the keywords in the username.

I can't just mute / block all those users, as some of them are ones I follow.

At the moment, the only way I can see to completely exclude an account from results is:

    -(@username) -from:username

This works, but the approach rapidly gets lengthy if there are quite a few names to exclude.
It also doesn't allow me to exclude all accounts containing username - i.e. @username1 @username2

One thought would be to create a list of accounts to exclude and then filter the list. That doesn't seem to work though as it won't help in excluding replies to these people.


问:
How to render JSON data as fields in the UI
I have a json data element in a table

```<code><Table responsive>
 <thead>
  <tr>
   <th className="hand" onClick={sort('serialized')}>
    <Translate contentKey="waTpaApp.tpaAssessment.serialized">Serialized</Translate>
   </th>
  </tr>
 </thead>
<tbody>
 <tr>
  <td>{assessment.serialized}</td>
 </tr>
</tbody>
</Table>
</code>```

The JSON is displayed in a column called 'Serialized'.

Some of the data looks like this (after formatted)

```<code>{
  "app": {
    "determination": {
      "home": {
        "title": "Determinations",
        "createLabel": "Create a new Determination",
        "createOrEditLabel": "Create or edit a Determination",
        "search": "Search for Determination",
        "notFound": "No Determinations found"
      },
    }
   }
}
</code>```

I would like to display the data in an easy to read way, for example

```<code><Row>
<Col sm="4">
{title}
</Col>
<Col sm="4">
{createLabel}
</Col>
<Col sm="4">
{createOrEditLabel}
</Col>
</Row>
</code>```

Or more simply

Title  --   create label -- create or edit label

答:
I added some javascript code to parse the data

```<code>const data = JSON.parse(assessmentEntity.serialized);
</code>```

and am able to access it
e.g.

```<code><Row>
{data.app.determination.home.title}
</Row>
</code>```

However I am getting an error when I parse the data and I think it is because the browser needs more time to load the element before parsing it.

```<code>SyntaxError: JSON Parse error: Unexpected identifier "undefined"
</code>```

when I console log the data it appears just fine

Body:
I have a json data element in a table

    <Translate contentKey="waTpaApp.tpaAssessment.serialized">Serialized</Translate>

  {assessment.serialized}

The JSON is displayed in a column called 'Serialized'.

Some of the data looks like this (after formatted)

{
  "app": {
    "determination": {
      "home": {
        "title": "Determinations",
        "createLabel": "Create a new Determination",
        "createOrEditLabel": "Create or edit a Determination",
        "search": "Search for Determination",
        "notFound": "No Determinations found"
      },
    }
   }
}

I would like to display the data in an easy to read way, for example

<Row>
<Col sm="4">
{title}
</Col>
<Col sm="4">
{createLabel}
</Col>
<Col sm="4">
{createOrEditLabel}
</Col>
</Row>

Or more simply

Title  --   create label -- create or edit label

Body:
I added some javascript code to parse the data

const data = JSON.parse(assessmentEntity.serialized);

and am able to access it
e.g.

<Row>
{data.app.determination.home.title}
</Row>

However I am getting an error when I parse the data and I think it is because the browser needs more time to load the element before parsing it.

SyntaxError: JSON Parse error: Unexpected identifier "undefined"

when I console log the data it appears just fine


问:
Access JSON Array without array name
I have a JSON object:

```<code>{
   "login_name":"hello@gmail.com",
   "login_pass":"abc123",
   "created_on":"2021-01-17 19:20:07",
   "user_id":"1",
   "active":"1"
}
</code>```

I don't know how to access it because it doesn't have a name.
This is using Volley:

```<code>val jsonObjectRequest = JsonObjectRequest(Request.Method.GET, url, null,
    { response ->
        val obj = JSONObject(response.toString()) // this works and outputs the JSON object
        val test: JSONObject = obj.getJSONObject("login_name") // this doesn't work

        val intent = Intent(this, MainActivity::class.java)
        startActivity(intent)
    },
    { error ->
        Toast.makeText(this@Login, "Error!", Toast.LENGTH_LONG).show()
    }
)
</code>```

I also tried converting the Json object to an array but that didn't work either...

Looked at:
Get JSONArray without array name, How can i write Android Json parsing without array name

EDIT:

```<code>val obj = JSONObject(response)
</code>```

That didn't work for me because:
None of the following functions can be called with the arguments supplied.
(String!) defined in org.json.JSONObject
((MutableMap<Any?, Any?>..Map<*, *>?)) defined in org.json.JSONObject
(JSONTokener!) defined in org.json.JSONObject

But after 2 days of trying, this worked... Didn't think I could just do that

```<code>val test = response.getString("login_name")
</code>```

答:
You are all most doing it right. Ass i see you convert the object to a string, and thats why you cant access the data afterwards.

```<code>   val jsonObjectRequest = JsonObjectRequest(Request.Method.GET, url, null,
    { response ->
        val obj = JSONObject(response) // this works and outputs the JSON object
        val test: JSONObject = obj.getJSONObject("login_name") // this doesn't work

        val intent = Intent(this, MainActivity::class.java)
        startActivity(intent)
    },
    { error ->
        Toast.makeText(this@Login, "Error!", Toast.LENGTH_LONG).show()
    }
)
</code>```

Try this, without the toString() function on response.

Title: Access JSON Array without array name

Body:
I have a JSON object:

{
   "login_name":"hello@gmail.com",
   "login_pass":"abc123",
   "created_on":"2021-01-17 19:20:07",
   "user_id":"1",
   "active":"1"
}

I don't know how to access it because it doesn't have a name.
This is using Volley:

val jsonObjectRequest = JsonObjectRequest(Request.Method.GET, url, null,
    { response ->
        val obj = JSONObject(response.toString()) // this works and outputs the JSON object
        val test: JSONObject = obj.getJSONObject("login_name") // this doesn't work

        val intent = Intent(this, MainActivity::class.java)
        startActivity(intent)
    },
    { error ->
        Toast.makeText(this@Login, "Error!", Toast.LENGTH_LONG).show()
    }
)

I also tried converting the Json object to an array but that didn't work either...

Looked at:
Get JSONArray without array name, How can i write Android Json parsing without array name

EDIT:

val obj = JSONObject(response)

That didn't work for me because:
None of the following functions can be called with the arguments supplied.
(String!) defined in org.json.JSONObject
((MutableMap<Any?, Any?>..Map<*, *>?)) defined in org.json.JSONObject
(JSONTokener!) defined in org.json.JSONObject

But after 2 days of trying, this worked... Didn't think I could just do that

val test = response.getString("login_name")

Body:
You are all most doing it right. Ass i see you convert the object to a string, and thats why you cant access the data afterwards.

   val jsonObjectRequest = JsonObjectRequest(Request.Method.GET, url, null,
    { response ->
        val obj = JSONObject(response) // this works and outputs the JSON object
        val test: JSONObject = obj.getJSONObject("login_name") // this doesn't work

        val intent = Intent(this, MainActivity::class.java)
        startActivity(intent)
    },
    { error ->
        Toast.makeText(this@Login, "Error!", Toast.LENGTH_LONG).show()
    }
)

Try this, without the toString() function on response.


问:
SqlExceptionHelper : ERROR: function count(character varying, integer) does not exist
I have a spring application and I have a native query with this syntax:

```<code>select
    COUNT(DISTINCT person.id,(CASE WHEN salary_person.rating = 'Satisfactory' THEN 1 END)) AS totalSatisfactory,
    COUNT(DISTINCT person.id,(CASE WHEN salary_person.rating = 'Unsatisfactory' THEN 1 END)) AS totalUnsatisfactory
    from person
    join salary_person on person.id = salary_person.person_id;
</code>```

I get the error:

```<code> ERROR: function count(character varying, integer) does not exist
</code>```

As database I use PostgreSQL. I mention that in mysql, the query is working.

答:
A quick and dirty alternative to count distinct on multiple columns regardless of type is to just concat the two columns into a single column and count distinct that

```<code>SELECT
  COUNT(DISTINCT CONCAT(person.id, salary_person.rating))
FROM person
  JOIN salary_person on person.id = salary_person.person_id;
</code>```

However, based on what you want to do, can't you just count distinct by rating and filter for certain ratings?

```<code>SELECT
  salary_person.rating,
  COUNT(DISTINCT person.id)
FROM person
  JOIN salary_person on person.id = salary_person.person_id
WHERE
  salary_person.rating in ('Satisfactory', 'Unsatisfactory')
GROUP BY 1;

</code>```

Body:
I have a spring application and I have a native query with this syntax:

select
    COUNT(DISTINCT person.id,(CASE WHEN salary_person.rating = 'Satisfactory' THEN 1 END)) AS totalSatisfactory,
    COUNT(DISTINCT person.id,(CASE WHEN salary_person.rating = 'Unsatisfactory' THEN 1 END)) AS totalUnsatisfactory
    from person
    join salary_person on person.id = salary_person.person_id;

I get the error:

 ERROR: function count(character varying, integer) does not exist

As database I use PostgreSQL. I mention that in mysql, the query is working.

Body:
A quick and dirty alternative to count distinct on multiple columns regardless of type is to just concat the two columns into a single column and count distinct that

SELECT
  COUNT(DISTINCT CONCAT(person.id, salary_person.rating))
FROM person
  JOIN salary_person on person.id = salary_person.person_id;

However, based on what you want to do, can't you just count distinct by rating and filter for certain ratings?

SELECT
  salary_person.rating,
  COUNT(DISTINCT person.id)
FROM person
  JOIN salary_person on person.id = salary_person.person_id
WHERE
  salary_person.rating in ('Satisfactory', 'Unsatisfactory')
GROUP BY 1;


问:
Disable Multipart Upload to S3 On Spark
I'm trying to write on a bucket that access is granted anonymously (policy allows our VPC).
For a small workload, it works fine, but for a big one, I get the following exception:

```<code>22/02/08 19:25:40 WARN TaskSetManager: Lost task 7.0 in stage 2.0 (TID 227) (172.20.64.7 executor 1): org.apache.spark.SparkException: Task failed while writing rows.
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:396)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$15(FileFormatWriter.scala:284)
    at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$3(ResultTask.scala:75)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:75)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:55)
    at org.apache.spark.scheduler.Task.doRunTask(Task.scala:150)
    at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:119)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.Task.run(Task.scala:91)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$13(Executor.scala:813)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1620)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:816)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:672)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.AccessDeniedException: cdc_streaming/sitemercado/data/groceries-sm.dbo.produto_integracao/part-00007-43d2b092-ea37-43c3-908a-a987df7a9a88-c000.snappy.parquet: initiate MultiPartUpload on cdc_streaming/sitemercado/data/groceries-sm.dbo.produto_integracao/part-00007-43d2b092-ea37-43c3-908a-a987df7a9a88-c000.snappy.parquet: com.amazonaws.services.s3.model.AmazonS3Exception: Anonymous users cannot initiate multipart uploads.  Please authenticate.; request: POST https://prd-ifood-data-lake-transient-groceries.bucket.vpce-08b663a29475cd9f4-wcex1383.s3.us-east-1.vpce.amazonaws.com cdc_streaming/sitemercado/data/groceries-sm.dbo.produto_integracao/part-00007-43d2b092-ea37-43c3-908a-a987df7a9a88-c000.snappy.parquet {key=[null]} Hadoop 2.7.4, aws-sdk-java/1.11.655 Linux/5.4.0-1063-azure OpenJDK_64-Bit_Server_VM/25.302-b08 java/1.8.0_302 scala/2.12.10 vendor/Azul_Systems,_Inc. com.amazonaws.services.s3.model.InitiateMultipartUploadRequest; Request ID: WFX4E5H30RYT9SXJ, Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=, Cloud Provider: Azure, Instance ID: da1fea5b08ef43f1b09adb89a162772c (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: WFX4E5H30RYT9SXJ; S3 Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=), S3 Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=:AccessDenied
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:248)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:236)
    at shaded.databricks.org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:132)
    at shaded.databricks.org.apache.hadoop.fs.s3a.WriteOperationHelper.initiateMultiPartUpload(WriteOperationHelper.java:215)
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.<init>(S3ABlockOutputStream.java:579)
    at shaded.databricks.org.apache.hadoop.fs.s3a.PageCRCVerifyingS3ABlockOutputStream$PageCRCVerifyingMultiPartUpload.<init>(PageCRCVerifyingS3ABlockOutputStream.java:153)
    at shaded.databricks.org.apache.hadoop.fs.s3a.PageCRCVerifyingS3ABlockOutputStream.initMultipartUpload(PageCRCVerifyingS3ABlockOutputStream.java:113)
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3ABlockOutputStream.uploadCurrentBlock(S3ABlockOutputStream.java:326)
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3ABlockOutputStream.write(S3ABlockOutputStream.java:306)
    at shaded.databricks.org.apache.hadoop.fs.s3a.PageCRCVerifyingS3ABlockOutputStream.write(PageCRCVerifyingS3ABlockOutputStream.java:18)
    at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
    at java.io.DataOutputStream.write(DataOutputStream.java:107)
    at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
    at org.apache.parquet.hadoop.util.HadoopPositionOutputStream.write(HadoopPositionOutputStream.java:45)
    at org.apache.parquet.bytes.ConcatenatingByteArrayCollector.writeAllTo(ConcatenatingByteArrayCollector.java:46)
    at org.apache.parquet.hadoop.ParquetFileWriter.writeDataPages(ParquetFileWriter.java:536)
    at org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writeToFileWriter(ColumnChunkPageWriteStore.java:246)
    at org.apache.parquet.hadoop.ColumnChunkPageWriteStore.flushToFileWriter(ColumnChunkPageWriteStore.java:316)
    at org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:202)
    at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:127)
    at org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:165)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:41)
    at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:58)
    at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:75)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$2(FileFormatWriter.scala:377)
    at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1654)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:383)
    ... 19 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Anonymous users cannot initiate multipart uploads.  Please authenticate.; request: POST https://prd-ifood-data-lake-transient-groceries.bucket.vpce-08b663a29475cd9f4-wcex1383.s3.us-east-1.vpce.amazonaws.com cdc_streaming/sitemercado/data/groceries-sm.dbo.produto_integracao/part-00007-43d2b092-ea37-43c3-908a-a987df7a9a88-c000.snappy.parquet {key=[null]} Hadoop 2.7.4, aws-sdk-java/1.11.655 Linux/5.4.0-1063-azure OpenJDK_64-Bit_Server_VM/25.302-b08 java/1.8.0_302 scala/2.12.10 vendor/Azul_Systems,_Inc. com.amazonaws.services.s3.model.InitiateMultipartUploadRequest; Request ID: WFX4E5H30RYT9SXJ, Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=, Cloud Provider: Azure, Instance ID: da1fea5b08ef43f1b09adb89a162772c (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: WFX4E5H30RYT9SXJ; S3 Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=), S3 Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4926)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4872)
    at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:3560)
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.initiateMultipartUpload(S3AFileSystem.java:3641)
    at shaded.databricks.org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$initiateMultiPartUpload$0(WriteOperationHelper.java:216)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
    ... 48 more
</code>```

Since the problem is with multipart upload, I've tried to disable it. I've already tried:

set spark.hadoop.fs.s3.multipart.uploads.enabled to false

set spark.hadoop.fs.s3a.multipart.uploads.enabled to false

set spark.hadoop.fs.s3n.multipart.uploads.enabled to false

set spark.hadoop.fs.s3.multipart.threshold to a very very big value

set spark.hadoop.fs.s3a.multipart.threshold to a very very big value

set spark.hadoop.fs.s3n.multipart.threshold to a very very big value

Everything on cluster startup and nothing seems to work, resulting in the same error.
It is worthwhile to say that:

It is an Azure Databricks instance.

I'm using Pyspark.

There's a Security restriction on creating users (access through accessKey/secretKey) thus, the access through anonymous user.

Does anyone had a similar issue and disabled with success the multipart upload?
Cheers!

答:
As a performance optimization for very small updates, by default the commit service sometimes pushes small updates directly from the control plane to S3. To disable this optimization, set the Spark parameter spark.hadoop.fs.s3a.databricks.s3commit.directPutFileSizeThreshold to 0. You can apply this setting in the cluster’s Spark config or set it in a global init script.

Link: docs.databricks.com

Title: Disable Multipart Upload to S3 On Spark

Body:
I'm trying to write on a bucket that access is granted anonymously (policy allows our VPC).
For a small workload, it works fine, but for a big one, I get the following exception:

22/02/08 19:25:40 WARN TaskSetManager: Lost task 7.0 in stage 2.0 (TID 227) (172.20.64.7 executor 1): org.apache.spark.SparkException: Task failed while writing rows.
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:396)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$15(FileFormatWriter.scala:284)
    at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$3(ResultTask.scala:75)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:75)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:55)
    at org.apache.spark.scheduler.Task.doRunTask(Task.scala:150)
    at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:119)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.Task.run(Task.scala:91)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$13(Executor.scala:813)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1620)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:816)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:672)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.AccessDeniedException: cdc_streaming/sitemercado/data/groceries-sm.dbo.produto_integracao/part-00007-43d2b092-ea37-43c3-908a-a987df7a9a88-c000.snappy.parquet: initiate MultiPartUpload on cdc_streaming/sitemercado/data/groceries-sm.dbo.produto_integracao/part-00007-43d2b092-ea37-43c3-908a-a987df7a9a88-c000.snappy.parquet: com.amazonaws.services.s3.model.AmazonS3Exception: Anonymous users cannot initiate multipart uploads.  Please authenticate.; request: POST https://prd-ifood-data-lake-transient-groceries.bucket.vpce-08b663a29475cd9f4-wcex1383.s3.us-east-1.vpce.amazonaws.com cdc_streaming/sitemercado/data/groceries-sm.dbo.produto_integracao/part-00007-43d2b092-ea37-43c3-908a-a987df7a9a88-c000.snappy.parquet {key=[null]} Hadoop 2.7.4, aws-sdk-java/1.11.655 Linux/5.4.0-1063-azure OpenJDK_64-Bit_Server_VM/25.302-b08 java/1.8.0_302 scala/2.12.10 vendor/Azul_Systems,_Inc. com.amazonaws.services.s3.model.InitiateMultipartUploadRequest; Request ID: WFX4E5H30RYT9SXJ, Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=, Cloud Provider: Azure, Instance ID: da1fea5b08ef43f1b09adb89a162772c (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: WFX4E5H30RYT9SXJ; S3 Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=), S3 Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=:AccessDenied
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:248)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:236)
    at shaded.databricks.org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:132)
    at shaded.databricks.org.apache.hadoop.fs.s3a.WriteOperationHelper.initiateMultiPartUpload(WriteOperationHelper.java:215)
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.<init>(S3ABlockOutputStream.java:579)
    at shaded.databricks.org.apache.hadoop.fs.s3a.PageCRCVerifyingS3ABlockOutputStream$PageCRCVerifyingMultiPartUpload.<init>(PageCRCVerifyingS3ABlockOutputStream.java:153)
    at shaded.databricks.org.apache.hadoop.fs.s3a.PageCRCVerifyingS3ABlockOutputStream.initMultipartUpload(PageCRCVerifyingS3ABlockOutputStream.java:113)
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3ABlockOutputStream.uploadCurrentBlock(S3ABlockOutputStream.java:326)
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3ABlockOutputStream.write(S3ABlockOutputStream.java:306)
    at shaded.databricks.org.apache.hadoop.fs.s3a.PageCRCVerifyingS3ABlockOutputStream.write(PageCRCVerifyingS3ABlockOutputStream.java:18)
    at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
    at java.io.DataOutputStream.write(DataOutputStream.java:107)
    at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
    at org.apache.parquet.hadoop.util.HadoopPositionOutputStream.write(HadoopPositionOutputStream.java:45)
    at org.apache.parquet.bytes.ConcatenatingByteArrayCollector.writeAllTo(ConcatenatingByteArrayCollector.java:46)
    at org.apache.parquet.hadoop.ParquetFileWriter.writeDataPages(ParquetFileWriter.java:536)
    at org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writeToFileWriter(ColumnChunkPageWriteStore.java:246)
    at org.apache.parquet.hadoop.ColumnChunkPageWriteStore.flushToFileWriter(ColumnChunkPageWriteStore.java:316)
    at org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:202)
    at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:127)
    at org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:165)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:41)
    at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:58)
    at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:75)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$2(FileFormatWriter.scala:377)
    at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1654)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:383)
    ... 19 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Anonymous users cannot initiate multipart uploads.  Please authenticate.; request: POST https://prd-ifood-data-lake-transient-groceries.bucket.vpce-08b663a29475cd9f4-wcex1383.s3.us-east-1.vpce.amazonaws.com cdc_streaming/sitemercado/data/groceries-sm.dbo.produto_integracao/part-00007-43d2b092-ea37-43c3-908a-a987df7a9a88-c000.snappy.parquet {key=[null]} Hadoop 2.7.4, aws-sdk-java/1.11.655 Linux/5.4.0-1063-azure OpenJDK_64-Bit_Server_VM/25.302-b08 java/1.8.0_302 scala/2.12.10 vendor/Azul_Systems,_Inc. com.amazonaws.services.s3.model.InitiateMultipartUploadRequest; Request ID: WFX4E5H30RYT9SXJ, Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=, Cloud Provider: Azure, Instance ID: da1fea5b08ef43f1b09adb89a162772c (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: WFX4E5H30RYT9SXJ; S3 Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=), S3 Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4926)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4872)
    at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:3560)
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.initiateMultipartUpload(S3AFileSystem.java:3641)
    at shaded.databricks.org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$initiateMultiPartUpload$0(WriteOperationHelper.java:216)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
    ... 48 more

Since the problem is with multipart upload, I've tried to disable it. I've already tried:

set spark.hadoop.fs.s3.multipart.uploads.enabled to false

set spark.hadoop.fs.s3a.multipart.uploads.enabled to false

set spark.hadoop.fs.s3n.multipart.uploads.enabled to false

set spark.hadoop.fs.s3.multipart.threshold to a very very big value

set spark.hadoop.fs.s3a.multipart.threshold to a very very big value

set spark.hadoop.fs.s3n.multipart.threshold to a very very big value

Everything on cluster startup and nothing seems to work, resulting in the same error.
It is worthwhile to say that:

It is an Azure Databricks instance.

I'm using Pyspark.

There's a Security restriction on creating users (access through accessKey/secretKey) thus, the access through anonymous user.

Does anyone had a similar issue and disabled with success the multipart upload?
Cheers!


问:
C# Forms. Changing the background color (Blink)
I am working on a program that allows you to control the OBS program. I receive json messages from OBS which I then process. These messages control the lighting on the Launchpad. However, I would like the buttons in the program itself to change color in addition to the lights on the Launchpad. The problem for me is the fact that the colors are not only static, but flickering too.
I need to change them depending on the state of some control in OBS. I don't know how to solve my problem. Create a new method or something?

Example json i recive

{"muted":false,"sourceName":"Audio Device","update-type":"SourceMuteStateChanged"}
{"muted":true,"sourceName":"Audio Device","update-type":"SourceMuteStateChanged"}
```
<p>In this example, the first message means that the audio input has been unmuted and then that it has been muted.</p>
<p>The part of c # that handles this</p>
<pre class="lang-cs prettyprint-override"><code>else if (upadte_type.Groups[0].Value == "SourceMuteStateChanged")
                    {
                        Match sourceName = Regex.Match(e.Data, "(?<=\"sourceName\":\").*?(?=\")");
                        if (sourceName.Success)
                        {
                            Match isMuted = Regex.Match(e.Data, "(?<=\"muted\":).*?(?=,)");
                            if (isMuted.Success)
                            {
                                if(isMuted.Groups[0].Value == "false")
                                {
                                    foreach(MuteListClass s in padsMute)
                                    {
                                        if(s.mute == sourceName.Groups[0].Value)
                                        {
                                            outputDevice.Open();
                                            byte[] stat = { 240, 0, 32, 41, 2, 13, 3, 0, (byte)s.id, (byte)padColorSettingsON[s.id, 0], 247 };
                                            outputDevice.SendSysEx(stat);
                                            outputDevice.Close();
                                        }
                                    }
                                }
                                else
                                {
                                    foreach (MuteListClass s in padsMute)
                                    {
                                        if (s.mute == sourceName.Groups[0].Value)
                                        {
                                            outputDevice.Open();
                                            byte[] stat = { 240, 0, 32, 41, 2, 13, 3, 0, (byte)s.id, (byte)padColorSettingsOFF[s.id, 0], 247 };
                                            outputDevice.SendSysEx(stat);
                                            outputDevice.Close();
                                        }
                                    }
                                }
                            }
                        }
                    }
</code>```

outputDevice = DeviceManager.OutputDevices[midiOutBox.SelectedIndex];
```
<pre class="lang-cs prettyprint-override"><code>List<MuteListClass> padsMute = new List<MuteListClass>();

public class MuteListClass
{
    public string mute { get; set; }
    public int id { set; get; }
}
</code>```

padColorSettingsON //int array which holds a number that corresponds to the color of the light in Launchpad
```
<pre class="lang-cs prettyprint-override"><code>byte[] stat = { 240, 0, 32, 41, 2, 13, 3, 0, (byte)s.id, (byte)padColorSettingsON[s.id, 0], 247 }; //byte array that will be sent to Launchpad
</code>```

I know Regex is not a right way for json, but this part of the code is quite old and I'm going to change it to serialization. (I am still learning all of this)

Screenshot of the program

I would like those gray buttons in the program to change color like the lights on the Launchpad. For example. When the device is unmuted the lower left corner is green, but when the device is muted it blinks red and green.

For Launchpad there is no big problem because this function is built in (I send a different byte array) but I don't know how to solve it in c #. I want to be able to control between static and flickering colors.

I've heard about async and await but I don't know how to apply them and if they will work for me in this case.

I was also thinking about making a separate function that would constantly change the color of the buttons based on boolean or something like that, but I don't know how to make a function that would still run in the background and check everything.

I think it is also worth mentioning that the function in c # that I showed earlier is only called when OBS sends a new json message, and all 81 buttons that you can see in the screenshot can be lit simultaneously. The buttons themselves are also modified and each button has an ID corresponding to the buttons on the Launchpad so they can be edited separately, for example.

foreach(LaunchpadButton b in GetAll(this, typeof(LaunchpadButton)))
            {
                if(b.ID == 11){b.BackColor = Color;}
            }
```

答:
This question is a little vague to be honest, so it's not quite clear what you're asking in full, maybe looking for guidance on how to approach it which isn't really what SO is for, but I'll take a stab.

So this kind of creates the effect you're looking for I think?

```<code>Button targetButton;
public Form1()
{
    InitializeComponent();
    targetButton = button1;
}

private void button2_Click(object sender, EventArgs e) => Task.Run(() => FlickerButton(targetButton, 1, Color.Red));
private void button3_Click(object sender, EventArgs e) => Task.Run(() => FlickerButton2(targetButton, 1, Color.Red, Color.Blue, Color.Green));

public void FlickerButton(Button button, int seconds, Color color)
{
    var currColor = button.BackColor;
    button.BackColor = color;
    Thread.Sleep(seconds * 1000);
    button.BackColor = currColor;
}

public void FlickerButton2(Button button, int seconds, params Color[] colors)
{
    var currColor = button.BackColor;
    foreach (var color in colors)
    {
        button.BackColor = color;
        Thread.Sleep(seconds * 1000);
    }
    button.BackColor = currColor;
}
</code>```

You can see when I click button 2, changes color for 1 second and then changes back

And I'm not going to screenshot this, but clicking on button 3 will do similar but with multiple colors.

I'm not quite sure how your byte array fits in if I'm honest, but I guess that's for you to figure out

byte[] stat = { 240, 0, 32, 41, 2, 13, 3, 0, (byte)s.id, (byte)padColorSettingsON[s.id, 0], 247 };

You can do this 100 times all at the same time, which based on the video I watched on Launchpad is what you're going for.

Body:
I am working on a program that allows you to control the OBS program. I receive json messages from OBS which I then process. These messages control the lighting on the Launchpad. However, I would like the buttons in the program itself to change color in addition to the lights on the Launchpad. The problem for me is the fact that the colors are not only static, but flickering too.
I need to change them depending on the state of some control in OBS. I don't know how to solve my problem. Create a new method or something?

Example json i recive

{"muted":false,"sourceName":"Audio Device","update-type":"SourceMuteStateChanged"}
{"muted":true,"sourceName":"Audio Device","update-type":"SourceMuteStateChanged"}

In this example, the first message means that the audio input has been unmuted and then that it has been muted.

The part of c # that handles this

else if (upadte_type.Groups[0].Value == "SourceMuteStateChanged")
                    {
                        Match sourceName = Regex.Match(e.Data, "(?<=\"sourceName\":\").*?(?=\")");
                        if (sourceName.Success)
                        {
                            Match isMuted = Regex.Match(e.Data, "(?<=\"muted\":).*?(?=,)");
                            if (isMuted.Success)
                            {
                                if(isMuted.Groups[0].Value == "false")
                                {
                                    foreach(MuteListClass s in padsMute)
                                    {
                                        if(s.mute == sourceName.Groups[0].Value)
                                        {
                                            outputDevice.Open();
                                            byte[] stat = { 240, 0, 32, 41, 2, 13, 3, 0, (byte)s.id, (byte)padColorSettingsON[s.id, 0], 247 };
                                            outputDevice.SendSysEx(stat);
                                            outputDevice.Close();
                                        }
                                    }
                                }
                                else
                                {
                                    foreach (MuteListClass s in padsMute)
                                    {
                                        if (s.mute == sourceName.Groups[0].Value)
                                        {
                                            outputDevice.Open();
                                            byte[] stat = { 240, 0, 32, 41, 2, 13, 3, 0, (byte)s.id, (byte)padColorSettingsOFF[s.id, 0], 247 };
                                            outputDevice.SendSysEx(stat);
                                            outputDevice.Close();
                                        }
                                    }
                                }
                            }
                        }
                    }

outputDevice = DeviceManager.OutputDevices[midiOutBox.SelectedIndex];

List<MuteListClass> padsMute = new List<MuteListClass>();

public class MuteListClass
{
    public string mute { get; set; }
    public int id { set; get; }
}

padColorSettingsON //int array which holds a number that corresponds to the color of the light in Launchpad

byte[] stat = { 240, 0, 32, 41, 2, 13, 3, 0, (byte)s.id, (byte)padColorSettingsON[s.id, 0], 247 }; //byte array that will be sent to Launchpad

I know Regex is not a right way for json, but this part of the code is quite old and I'm going to change it to serialization. (I am still learning all of this)

Screenshot of the program

I would like those gray buttons in the program to change color like the lights on the Launchpad. For example. When the device is unmuted the lower left corner is green, but when the device is muted it blinks red and green.

For Launchpad there is no big problem because this function is built in (I send a different byte array) but I don't know how to solve it in c #. I want to be able to control between static and flickering colors.

I've heard about async and await but I don't know how to apply them and if they will work for me in this case.

I was also thinking about making a separate function that would constantly change the color of the buttons based on boolean or something like that, but I don't know how to make a function that would still run in the background and check everything.

I think it is also worth mentioning that the function in c # that I showed earlier is only called when OBS sends a new json message, and all 81 buttons that you can see in the screenshot can be lit simultaneously. The buttons themselves are also modified and each button has an ID corresponding to the buttons on the Launchpad so they can be edited separately, for example.

foreach(LaunchpadButton b in GetAll(this, typeof(LaunchpadButton)))
            {
                if(b.ID == 11){b.BackColor = Color;}
            }

Body:
This question is a little vague to be honest, so it's not quite clear what you're asking in full, maybe looking for guidance on how to approach it which isn't really what SO is for, but I'll take a stab.

So this kind of creates the effect you're looking for I think?

Button targetButton;
public Form1()
{
    InitializeComponent();
    targetButton = button1;
}

private void button2_Click(object sender, EventArgs e) => Task.Run(() => FlickerButton(targetButton, 1, Color.Red));
private void button3_Click(object sender, EventArgs e) => Task.Run(() => FlickerButton2(targetButton, 1, Color.Red, Color.Blue, Color.Green));

public void FlickerButton(Button button, int seconds, Color color)
{
    var currColor = button.BackColor;
    button.BackColor = color;
    Thread.Sleep(seconds * 1000);
    button.BackColor = currColor;
}

public void FlickerButton2(Button button, int seconds, params Color[] colors)
{
    var currColor = button.BackColor;
    foreach (var color in colors)
    {
        button.BackColor = color;
        Thread.Sleep(seconds * 1000);
    }
    button.BackColor = currColor;
}

You can see when I click button 2, changes color for 1 second and then changes back

And I'm not going to screenshot this, but clicking on button 3 will do similar but with multiple colors.

I'm not quite sure how your byte array fits in if I'm honest, but I guess that's for you to figure out

byte[] stat = { 240, 0, 32, 41, 2, 13, 3, 0, (byte)s.id, (byte)padColorSettingsON[s.id, 0], 247 };

You can do this 100 times all at the same time, which based on the video I watched on Launchpad is what you're going for.


问:
Which project files are relevant still for Visual Studio 2022?
After upgrading from VS-2015 and even earlier versions, I still have some project files in my folder which I might no longer need.

I am wondering, are there any of these files that I can delete (* = project name)?

```<code>*.VC.db -> 100+ MB
*.sdf -> ~100 MB
*.vcxproj.user
*.aps
*.filters
*.sln.docstates.suo
*.suo
*.user
*.vcxproj.filters
msvcp140.dll
vcruntime140.dll
</code>```

答:
For reference, I deleted the following files, as I figured they were not used anymore in VS2022, based on the advice of @Adrian Mole:

*.VC.db (I believe it is moved to the .vs folder)

*.sdf

*.suo

*.sln.docstates.suo

*.aps (if you do not edit resource files much)

*.user (but keep vcxproj.user)

*.filters (but keep vcxproj.filters)

It saves a little bit of disk space and unnecessary files :) Have a great day!

Body:
After upgrading from VS-2015 and even earlier versions, I still have some project files in my folder which I might no longer need.

I am wondering, are there any of these files that I can delete (* = project name)?

*.VC.db -> 100+ MB
*.sdf -> ~100 MB
*.vcxproj.user
*.aps
*.filters
*.sln.docstates.suo
*.suo
*.user
*.vcxproj.filters
msvcp140.dll
vcruntime140.dll


问:
Flutter: TextField or TextFormField label inside border using ThemeData
How can i make my TextField label float to the top but stay inside the TextField border (not at the edge of it) when my TextField widget is active?

Example of what i'm going for:

Becomes

This

Is this possible using the inputDecorationTheme on ThemeData? or do i have to make a wrapper Widget to accomplish this?

Would very much prefer to control it from an app wide ThemeData

答:
Unfortunately i don't know how to do it with default tools, but i do it with another way, may be it will be helpful for you.

Create variable for FocusNode and border color inside of your widget:

```<code>// Use it to change color for border when textFiled in focus
FocusNode _focusNode = FocusNode();

// Color for border
Color _borderColor = Colors.grey;
</code>```

Inside of initState create listener for textField, if textField will be in focus, change border color to orange, otherwise change to grey:

```<code>@override
void initState() {
  super.initState();
  // Change color for border if focus was changed
  _focusNode.addListener(() {
    setState(() {
      _borderColor = _focusNode.hasFocus ? Colors.orange : Colors.grey;
    });
  });
}
</code>```

Create Container with border for textField, add focusNode and set decoration to textField:

```<code>Container(
  decoration: BoxDecoration(
    border: Border.all(color: _borderColor),
    borderRadius: BorderRadius.circular(4),
  ),
  child: TextField(
    focusNode: _focusNode,
    style: TextStyle(color: Colors.grey),
    keyboardType: TextInputType.number,
    decoration: InputDecoration(
      contentPadding: EdgeInsets.zero,
      border: InputBorder.none,
      labelText: "Amount",
      prefixIconConstraints: BoxConstraints(minWidth: 0, minHeight: 0),
      prefixIcon: Padding(
        padding: EdgeInsets.symmetric(vertical: 18, horizontal: 8),
        child: Text("₦", style: TextStyle(fontSize: 16, color: Colors.grey)),
      ),
    ),
  ),
),
</code>```

Don't forget call dispose for focusNode:

```<code>@override
void dispose() {
  _focusNode.dispose();
  super.dispose();
}
</code>```

Full code:

```<code>class TextFieldDesignPage extends StatefulWidget {
  TextFieldDesignPage({Key? key}) : super(key: key);

  @override
  _TextFieldDesignPageState createState() => _TextFieldDesignPageState();
}

class _TextFieldDesignPageState extends State<TextFieldDesignPage> {
  // Use it to change color for border when textFiled in focus
  FocusNode _focusNode = FocusNode();

  // Color for border
  Color _borderColor = Colors.grey;

  @override
  void initState() {
    super.initState();
    // Change color for border if focus was changed
    _focusNode.addListener(() {
      setState(() {
        _borderColor = _focusNode.hasFocus ? Colors.orange : Colors.grey;
      });
    });
  }

  @override
  void dispose() {
    _focusNode.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: Center(
        child: Container(
          margin: EdgeInsets.all(8),
          decoration: BoxDecoration(
            border: Border.all(color: _borderColor),
            borderRadius: BorderRadius.circular(4),
          ),
          child: TextField(
            focusNode: _focusNode,
            style: TextStyle(color: Colors.grey),
            keyboardType: TextInputType.number,
            decoration: InputDecoration(
              contentPadding: EdgeInsets.zero,
              border: InputBorder.none,
              labelText: "Amount",
              prefixIconConstraints: BoxConstraints(minWidth: 0, minHeight: 0),
              prefixIcon: Padding(
                padding: EdgeInsets.symmetric(vertical: 18, horizontal: 8),
                child: Text("₦", style: TextStyle(fontSize: 16, color: Colors.grey)),
              ),
            ),
          ),
        ),
      ),
    );
  }
}
</code>```

Result:

Body:
Unfortunately i don't know how to do it with default tools, but i do it with another way, may be it will be helpful for you.

Create variable for FocusNode and border color inside of your widget:

// Use it to change color for border when textFiled in focus
FocusNode _focusNode = FocusNode();

// Color for border
Color _borderColor = Colors.grey;

Inside of initState create listener for textField, if textField will be in focus, change border color to orange, otherwise change to grey:

@override
void initState() {
  super.initState();
  // Change color for border if focus was changed
  _focusNode.addListener(() {
    setState(() {
      _borderColor = _focusNode.hasFocus ? Colors.orange : Colors.grey;
    });
  });
}

Create Container with border for textField, add focusNode and set decoration to textField:

Container(
  decoration: BoxDecoration(
    border: Border.all(color: _borderColor),
    borderRadius: BorderRadius.circular(4),
  ),
  child: TextField(
    focusNode: _focusNode,
    style: TextStyle(color: Colors.grey),
    keyboardType: TextInputType.number,
    decoration: InputDecoration(
      contentPadding: EdgeInsets.zero,
      border: InputBorder.none,
      labelText: "Amount",
      prefixIconConstraints: BoxConstraints(minWidth: 0, minHeight: 0),
      prefixIcon: Padding(
        padding: EdgeInsets.symmetric(vertical: 18, horizontal: 8),
        child: Text("₦", style: TextStyle(fontSize: 16, color: Colors.grey)),
      ),
    ),
  ),
),

Don't forget call dispose for focusNode:

@override
void dispose() {
  _focusNode.dispose();
  super.dispose();
}

Full code:

class TextFieldDesignPage extends StatefulWidget {
  TextFieldDesignPage({Key? key}) : super(key: key);

  @override
  _TextFieldDesignPageState createState() => _TextFieldDesignPageState();
}

class _TextFieldDesignPageState extends State<TextFieldDesignPage> {
  // Use it to change color for border when textFiled in focus
  FocusNode _focusNode = FocusNode();

  // Color for border
  Color _borderColor = Colors.grey;

  @override
  void initState() {
    super.initState();
    // Change color for border if focus was changed
    _focusNode.addListener(() {
      setState(() {
        _borderColor = _focusNode.hasFocus ? Colors.orange : Colors.grey;
      });
    });
  }

  @override
  void dispose() {
    _focusNode.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: Center(
        child: Container(
          margin: EdgeInsets.all(8),
          decoration: BoxDecoration(
            border: Border.all(color: _borderColor),
            borderRadius: BorderRadius.circular(4),
          ),
          child: TextField(
            focusNode: _focusNode,
            style: TextStyle(color: Colors.grey),
            keyboardType: TextInputType.number,
            decoration: InputDecoration(
              contentPadding: EdgeInsets.zero,
              border: InputBorder.none,
              labelText: "Amount",
              prefixIconConstraints: BoxConstraints(minWidth: 0, minHeight: 0),
              prefixIcon: Padding(
                padding: EdgeInsets.symmetric(vertical: 18, horizontal: 8),
                child: Text("₦", style: TextStyle(fontSize: 16, color: Colors.grey)),
              ),
            ),
          ),
        ),
      ),
    );
  }
}

Result:


问:
How to open a text file with 'open with' option by the software , created by visual basic?
In Windows we can open a text file by right clicking on a it, and selecting 'open with' option by any software like notepad, notepad++ etc. I want to create a notepad type software using visual basic , Which can open (load) any text file using 'open with' option in Windows explorer by right clicking.
I tried by creating a simple notepad using textbox and button which can load text file by browsing file system and selecting the file . But not load by 'open with' option.</p>

答:
Here is an example. You need to understand that on my Windows 7 system the Default Program Setting for file type .txt is Sublime Text. I assume you understand that if you have .txt file type set to use Notepad. With my code on your system the text in the file will be displayed with Notepad using this line of code below.

```<code>Process.Start(openPath)
</code>```

To use this code create a RichTextBox on your form a good size is about 1150 by 850 Name it rtbViewCode

You will need two Buttons named btnViewFile and btnCopy

I have included some bonus code that lets you copy the files text to the clipboard

```<code> Private Sub btnViewFile_Click(sender As Object, e As EventArgs) Handles btnViewFile.Click
    Dim openFileDialog As OpenFileDialog = New OpenFileDialog()
    Dim dr As DialogResult = openFileDialog.ShowDialog()

    If dr = System.Windows.Forms.DialogResult.OK Then

        Dim openPath As String = openFileDialog.FileName
        MsgBox("openPath " & openPath)

        Process.Start(openPath)
        'Line of code above should be used behind it's own Button Click Event

        rtbViewCode.Text = File.ReadAllText(openPath)

    End If
End Sub

Private Sub btnCopy_Click(sender As Object, e As EventArgs) Handles btnCopy.Click
    rtbViewCode.SelectAll()
    rtbViewCode.Copy()
End Sub
</code>```

Body:
Here is an example. You need to understand that on my Windows 7 system the Default Program Setting for file type .txt is Sublime Text. I assume you understand that if you have .txt file type set to use Notepad. With my code on your system the text in the file will be displayed with Notepad using this line of code below.

Process.Start(openPath)

To use this code create a RichTextBox on your form a good size is about 1150 by 850 Name it rtbViewCode

You will need two Buttons named btnViewFile and btnCopy

I have included some bonus code that lets you copy the files text to the clipboard

 Private Sub btnViewFile_Click(sender As Object, e As EventArgs) Handles btnViewFile.Click
    Dim openFileDialog As OpenFileDialog = New OpenFileDialog()
    Dim dr As DialogResult = openFileDialog.ShowDialog()

    If dr = System.Windows.Forms.DialogResult.OK Then

        Dim openPath As String = openFileDialog.FileName
        MsgBox("openPath " & openPath)

        Process.Start(openPath)
        'Line of code above should be used behind it's own Button Click Event

        rtbViewCode.Text = File.ReadAllText(openPath)

    End If
End Sub

Private Sub btnCopy_Click(sender As Object, e As EventArgs) Handles btnCopy.Click
    rtbViewCode.SelectAll()
    rtbViewCode.Copy()
End Sub


问:
Llisten for multiple events in socket.io
I would like to display to the user the state the socket.io connection is in.  To do that I have a bunch of these to tell the user.

```<code>#I can't use `*` because I have some customized events
socket.on 'my_command', =>
    set_status "connecting"

socket.on 'connecting', =>
    set_status "connecting"

socket.on 'disconnect', =>
    set_status "disconnect"

socket.on 'connect_failed', =>
    set_status "connect_failed"
</code>```

Is there a way to compress all these down to something smaller?  Something like:

```<code>socket.on 'connecting, disconnect, connect_failed', (name_of_event)=>
    set_status name_of_event
</code>```

答:
As of Socket.IO v3, Catch-all listeners were introduced :

```<code>socket.onAny((eventName, ...args) => {
  // ...
});
</code>```

you can add conditional logic inside the listener for different cases.

check the official documentation for further information : catch-all listeners

Body:
I would like to display to the user the state the socket.io connection is in.  To do that I have a bunch of these to tell the user.

#I can't use `*` because I have some customized events
socket.on 'my_command', =>
    set_status "connecting"

socket.on 'connecting', =>
    set_status "connecting"

socket.on 'disconnect', =>
    set_status "disconnect"

socket.on 'connect_failed', =>
    set_status "connect_failed"

Is there a way to compress all these down to something smaller?  Something like:

socket.on 'connecting, disconnect, connect_failed', (name_of_event)=>
    set_status name_of_event

Body:
As of Socket.IO v3, Catch-all listeners were introduced :

socket.onAny((eventName, ...args) => {
  // ...
});

you can add conditional logic inside the listener for different cases.

check the official documentation for further information : catch-all listeners