问:
Simmer Get_attribute | there is no arrival running error
first of all, this simmer_vignette and this linkadvanced_simmer_usage seem to indicate that the error stems from the fact that "get_name, get_attribute and get_prioritization are meant to be used inside a trajectory; otherwise, there will be no arrival running and these functions will throw an error"
A minimal workable example:

```<code>  patient_traj <- trajectory(name = "patient_trajectory") %>%
  set_attribute("my_key", 123) %>%
  timeout(5) %>%
  set_attribute("my_key", function() get_attribute(env, "my_key") + 1) %>%
  timeout(5) %>%
  set_attribute("dependent_key", function() ifelse(get_attribute(env, "my_key")<=123, 1, 0)) %>%
  timeout(5) %>%
  set_attribute("independent_key", function() runif(1))

env<- simmer() %>%
  add_generator("patient", patient_traj, at(0), mon = 2)
env %>% run()
#> simmer environment: anonymous | now: 15 | next:
#> { Generator: patient | monitored: 2 | n_generated: 1 }

get_mon_attributes(env)
#>   time     name             key       value replication
#> 1    0 patient0          my_key 123.0000000           1
#> 2    5 patient0          my_key 124.0000000           1
#> 3   10 patient0   dependent_key   0.0000000           1
#> 4   15 patient0 independent_key   0.9234335           1
</code>```

Now this works  as it's supposed to work, the problem starts when I try to call get_attribute() in any other sense. Adding this line after set_attribute() at the very end of the trajectory definition:

```<code>log_(get_attribute(env, "independent_key"))
</code>```

throws the abovementioned error.
What I actually want to do is call the "leave" function and give it as a probability an attribute. I still do this in the trajectory.

```<code>leave(prob = get_attribute(env, "independent_key"))
</code>```

Needless to say, this also throws the error "Error in get_attribute_(private$sim_obj, key, global) :  there is no arrival running".

Does anyone know what might cause this? I feel like the only option is the above explanatio "get_attribute is meant to be used inside a trajectory" - but I feel like I am doing this.

Thanks already!

答:
Okay, I am embarrased to say this but the problem was rather easily fixed. It seems as if the problem was to access the attribute directly.

So log_(get_attribute(env, "independent_key")) does not work, but log_(function() get_attribute(env, "independent_key")) does.

That's all it takes.

If anyone has an explanation as to why that is all it takes, I would highly appreciate it.

Body:
first of all, this simmer_vignette and this linkadvanced_simmer_usage seem to indicate that the error stems from the fact that "get_name, get_attribute and get_prioritization are meant to be used inside a trajectory; otherwise, there will be no arrival running and these functions will throw an error"
A minimal workable example:

  patient_traj <- trajectory(name = "patient_trajectory") %>%
  set_attribute("my_key", 123) %>%
  timeout(5) %>%
  set_attribute("my_key", function() get_attribute(env, "my_key") + 1) %>%
  timeout(5) %>%
  set_attribute("dependent_key", function() ifelse(get_attribute(env, "my_key")<=123, 1, 0)) %>%
  timeout(5) %>%
  set_attribute("independent_key", function() runif(1))

env<- simmer() %>%
  add_generator("patient", patient_traj, at(0), mon = 2)
env %>% run()
#> simmer environment: anonymous | now: 15 | next:
#> { Generator: patient | monitored: 2 | n_generated: 1 }

get_mon_attributes(env)
#>   time     name             key       value replication
#> 1    0 patient0          my_key 123.0000000           1
#> 2    5 patient0          my_key 124.0000000           1
#> 3   10 patient0   dependent_key   0.0000000           1
#> 4   15 patient0 independent_key   0.9234335           1

Now this works  as it's supposed to work, the problem starts when I try to call get_attribute() in any other sense. Adding this line after set_attribute() at the very end of the trajectory definition:

log_(get_attribute(env, "independent_key"))

throws the abovementioned error.
What I actually want to do is call the "leave" function and give it as a probability an attribute. I still do this in the trajectory.

leave(prob = get_attribute(env, "independent_key"))

Needless to say, this also throws the error "Error in get_attribute_(private$sim_obj, key, global) :  there is no arrival running".

Does anyone know what might cause this? I feel like the only option is the above explanatio "get_attribute is meant to be used inside a trajectory" - but I feel like I am doing this.

Thanks already!


来源: stackexchange

问:
Need to detect whether jQuery is triggered in normal or multiselect combobox
I have some jQuery code that allows me to select multiple entries in a multiselect combobox without the need to press the Ctrl key.

This works, but interferes with a normal select box, which can now not be changed with the mouse.

The original jQuery code is:

```<code>(function ($) {
$(document).ready(function () {
    $("option").mousedown(function(e) {
      this.selected = ! this.selected;
      e.preventDefault();
      return false;
    });
});
})(jQuery);
</code>```

Of course, the $("option") selector applies to the normal select box as well, which is why it fails.

I tried to detect whether the event is triggered by a select box with the "multiple" attribute, but couldn't figure out how. The following doesn't work for various reasons, one of them being that the event is triggered by the option, not the select:

```<code>(function ($) {
$(document).ready(function () {
    $("option").mousedown(function(e) {
        if($(this).attr('multiple','multiple')) {
          this.selected = ! this.selected;
          e.preventDefault();
          return false;
        }
        else {
          return true;
        }
    });
});
})(jQuery);
</code>```

Any ideas of how this could be done?

Ah, I don't want to include the ID of the multiselect, because the same code is also used by another multiselect in my real-world case.

Here is a fiddle:
https://jsfiddle.net/mheumann/3zn76mex/

答:
You can target the options of every multi-select using this selector: $('select[multiple] option').

This selector targets every <option> that occurs within a <select> with the multiple attribute present.

ContentLicense: CC BY-SA 3.0

Body:
I have some jQuery code that allows me to select multiple entries in a multiselect combobox without the need to press the Ctrl key.

This works, but interferes with a normal select box, which can now not be changed with the mouse.

The original jQuery code is:

(function ($) {
$(document).ready(function () {
    $("option").mousedown(function(e) {
      this.selected = ! this.selected;
      e.preventDefault();
      return false;
    });
});
})(jQuery);

Of course, the $("option") selector applies to the normal select box as well, which is why it fails.

I tried to detect whether the event is triggered by a select box with the "multiple" attribute, but couldn't figure out how. The following doesn't work for various reasons, one of them being that the event is triggered by the option, not the select:

(function ($) {
$(document).ready(function () {
    $("option").mousedown(function(e) {
        if($(this).attr('multiple','multiple')) {
          this.selected = ! this.selected;
          e.preventDefault();
          return false;
        }
        else {
          return true;
        }
    });
});
})(jQuery);

Any ideas of how this could be done?

Ah, I don't want to include the ID of the multiselect, because the same code is also used by another multiselect in my real-world case.

Here is a fiddle:
https://jsfiddle.net/mheumann/3zn76mex/


问:
Microsoft.Office.Interop.Excel add new row without using index
I am trying to convert a list to an excel file. But I don't think it is going to work by using WorkSheet.Cells[i, y] = "value". I am wondering whether there is a general way to add rows at the end of the last row in Excel.

```<code>public void AttorneysToExcelFile(List<Scraper.StaggingAttorney> liveScraperAttorneys, string FileName)
{
    Excel.Workbook WorkBook = ExcelApp.Workbooks.Add();
    Excel.Worksheet xlWorkSheet = (Excel.Worksheet)WorkBook.Worksheets.get_Item(1);
    xlWorkSheet.Name = "No Category";
    liveScraperAttorneys = liveScraperAttorneys.OrderBy(x=>x.Department).ToList();

    xlWorkSheet.Cells[1, 1] = "caseNumber";
    xlWorkSheet.Cells[1, 2] = "Category";
    xlWorkSheet.Cells[1, 3] = "Names";
    xlWorkSheet.Cells[1, 4] = "Email";
    xlWorkSheet.Cells[1, 5] = "Address";

    for (int i=2; i<liveScraperAttorneys.Count; i++)
    {
        Scraper.StaggingAttorney attorney = liveScraperAttorneys[i];

        xlWorkSheet.Cells[i, 1] = attorney.caseNumber;
        xlWorkSheet.Cells[i, 2] = attorney.Category;
        xlWorkSheet.Cells[i, 3] = attorney.Names;
        xlWorkSheet.Cells[i, 4] = attorney.Email;
        xlWorkSheet.Cells[i, 5] = attorney.Address;
    }
    WorkBook.SaveAs(FileName);
    WorkBook.Close();
}
</code>```

As you can see I am relying on i. It would help also if I can do it at once. Also I am wondering whether the way I have added headers is the right way to do it.

答: Iterating Interop objects is brutally slow.  Convert your list to a object[,] and then assign it to the Excel range.Value.  Add the header row as the first element in the array, then your data as the rest of the elements.  range is a range of cells of equal dimensions to your array.  That is, the number of items in your list by the number of properties in your object.

Body:
I am trying to convert a list to an excel file. But I don't think it is going to work by using WorkSheet.Cells[i, y] = "value". I am wondering whether there is a general way to add rows at the end of the last row in Excel.

public void AttorneysToExcelFile(List<Scraper.StaggingAttorney> liveScraperAttorneys, string FileName)
{
    Excel.Workbook WorkBook = ExcelApp.Workbooks.Add();
    Excel.Worksheet xlWorkSheet = (Excel.Worksheet)WorkBook.Worksheets.get_Item(1);
    xlWorkSheet.Name = "No Category";
    liveScraperAttorneys = liveScraperAttorneys.OrderBy(x=>x.Department).ToList();

    xlWorkSheet.Cells[1, 1] = "caseNumber";
    xlWorkSheet.Cells[1, 2] = "Category";
    xlWorkSheet.Cells[1, 3] = "Names";
    xlWorkSheet.Cells[1, 4] = "Email";
    xlWorkSheet.Cells[1, 5] = "Address";

    for (int i=2; i<liveScraperAttorneys.Count; i++)
    {
        Scraper.StaggingAttorney attorney = liveScraperAttorneys[i];

        xlWorkSheet.Cells[i, 1] = attorney.caseNumber;
        xlWorkSheet.Cells[i, 2] = attorney.Category;
        xlWorkSheet.Cells[i, 3] = attorney.Names;
        xlWorkSheet.Cells[i, 4] = attorney.Email;
        xlWorkSheet.Cells[i, 5] = attorney.Address;
    }
    WorkBook.SaveAs(FileName);
    WorkBook.Close();
}

As you can see I am relying on i. It would help also if I can do it at once. Also I am wondering whether the way I have added headers is the right way to do it.


问:
Function for pivot tables and merge in R
I need a help in an easy task, I believe. I am not so used to functions in R.
I have two data frames (df1 and df2) and I would like to:

1) filter df1 by a column value (output = A);

2) generate a pivot table from A (output = B);

3) rename B's columns;

4) merge this pivot table B with the second data frame df2 (output = C);

5) divide the values of a column from df2 by the values of a column from C.

I want to perform this task 13 times, that's why I would like to do a function / loop / use lapply or any other way that would make my life easier.

As I see, in R usually it is better to use lapply. Intuitively, I would use some kind of a function or loop for this task.

Check below a reproducible example for two variables (MLB and D).

```<code>library(dplyr)
df<-   as.data.frame(cbind(c("NC_1","NC_1","NC_2","NC_2","NC_2","NC_1","NC_1","NC_1","NC_2","NC_1"),
              c("MLB","MLB","MLB","MLB","MLB","MLB","D","D","D","D"),
                 c("ATT","ATT","ATT","ATT1","ATT1","ATT1","ATT1","ATT1","ATT1","ATT1")))

names(df) <- c("TAG", "ST", "ATR")
general.table<-table(df$TAG)
general.table<-as.data.frame(general.table)
names(general.table) <- c("TAG", "OCC")

# MLB

MLB.df<-filter(df, grepl('MLB', ST)) # 1) Filter
MLB.df.table<-as.data.frame(table(MLB.df$TAG)) # 2) Pivot Table
names(MLB.df.table)<-c("TAG", "Partial_Score") # 3) Rename
MLB.df.final.table<-merge(general.table,MLB.df.table,by='TAG', all.x=T) # 4) Merge
MLB.df.final.table$Partial_Score<-MLB.df.final.table$Partial_Score / general.table$OCC # 5) Divide one column by other

# D - The same tasks to be performed just like MLB
D.df<-filter(df, grepl('D', ST))
D.df.table<-as.data.frame(table(D.df$TAG))
names(D.df.table)<-c("TAG", "Partial_Score")
D.df.final.table<-merge(general.table,D.df.table,by='TAG', all.x=T)
D.df.final.table$Partial_Score<-D.df.final.table$Partial_Score / general.table$OCC
</code>```

I also wonder if there is a way to performe those 5 tasks by only changing the strings. For instance, the only difference between the tasks 1) to 5) performed for MLB and D was only the strings "MLB" and "D". I tried to create a vector with this MLB, D and the other 11 strings and then use assign and paste functions, without success.

So, in short, what I would like is to have a solution using:

A) Function / loop;

B) lapply;

C) If it is possible, perform the tasks by only varying the strings MLB and D.

Of course, one solution is enough for me. The others are just to didactically better understand R.

Thank you very much in advance. Please let me know if you guys need any more information.

答:
If you don't need the intermediate solutions, only the ...final.table ones, I would suggest the following function:

```<code>calc.partial.score <- function(df, general.table, filter.expr) {
  df %>%
    filter(grepl(filter.expr, ST)) %>%
    count(TAG) %>%
    merge(general.table, by = 'TAG', all.x = T) %>%
    mutate(Partial_Score = n / OCC) %>%
    select(-n)
}
</code>```

Use it the following way:

```<code>> calc.partial.score(df, general.table, "MLB")
   TAG OCC Partial_Score
1 NC_1   6          0.50
2 NC_2   4          0.75

> calc.partial.score(df, general.table, "D")
   TAG OCC Partial_Score
1 NC_1   6          0.50
2 NC_2   4          0.25
</code>```

Body:
I need a help in an easy task, I believe. I am not so used to functions in R.
I have two data frames (df1 and df2) and I would like to:

1) filter df1 by a column value (output = A);

2) generate a pivot table from A (output = B);

3) rename B's columns;

4) merge this pivot table B with the second data frame df2 (output = C);

5) divide the values of a column from df2 by the values of a column from C.

I want to perform this task 13 times, that's why I would like to do a function / loop / use lapply or any other way that would make my life easier.

As I see, in R usually it is better to use lapply. Intuitively, I would use some kind of a function or loop for this task.

Check below a reproducible example for two variables (MLB and D).

library(dplyr)
df<-   as.data.frame(cbind(c("NC_1","NC_1","NC_2","NC_2","NC_2","NC_1","NC_1","NC_1","NC_2","NC_1"),
              c("MLB","MLB","MLB","MLB","MLB","MLB","D","D","D","D"),
                 c("ATT","ATT","ATT","ATT1","ATT1","ATT1","ATT1","ATT1","ATT1","ATT1")))

names(df) <- c("TAG", "ST", "ATR")
general.table<-table(df$TAG)
general.table<-as.data.frame(general.table)
names(general.table) <- c("TAG", "OCC")

# MLB

MLB.df<-filter(df, grepl('MLB', ST)) # 1) Filter
MLB.df.table<-as.data.frame(table(MLB.df$TAG)) # 2) Pivot Table
names(MLB.df.table)<-c("TAG", "Partial_Score") # 3) Rename
MLB.df.final.table<-merge(general.table,MLB.df.table,by='TAG', all.x=T) # 4) Merge
MLB.df.final.table$Partial_Score<-MLB.df.final.table$Partial_Score / general.table$OCC # 5) Divide one column by other

# D - The same tasks to be performed just like MLB
D.df<-filter(df, grepl('D', ST))
D.df.table<-as.data.frame(table(D.df$TAG))
names(D.df.table)<-c("TAG", "Partial_Score")
D.df.final.table<-merge(general.table,D.df.table,by='TAG', all.x=T)
D.df.final.table$Partial_Score<-D.df.final.table$Partial_Score / general.table$OCC

I also wonder if there is a way to performe those 5 tasks by only changing the strings. For instance, the only difference between the tasks 1) to 5) performed for MLB and D was only the strings "MLB" and "D". I tried to create a vector with this MLB, D and the other 11 strings and then use assign and paste functions, without success.

So, in short, what I would like is to have a solution using:

A) Function / loop;

B) lapply;

C) If it is possible, perform the tasks by only varying the strings MLB and D.

Of course, one solution is enough for me. The others are just to didactically better understand R.

Thank you very much in advance. Please let me know if you guys need any more information.

Body:
If you don't need the intermediate solutions, only the ...final.table ones, I would suggest the following function:

calc.partial.score <- function(df, general.table, filter.expr) {
  df %>%
    filter(grepl(filter.expr, ST)) %>%
    count(TAG) %>%
    merge(general.table, by = 'TAG', all.x = T) %>%
    mutate(Partial_Score = n / OCC) %>%
    select(-n)
}

Use it the following way:

> calc.partial.score(df, general.table, "MLB")
   TAG OCC Partial_Score
1 NC_1   6          0.50
2 NC_2   4          0.75

> calc.partial.score(df, general.table, "D")
   TAG OCC Partial_Score
1 NC_1   6          0.50
2 NC_2   4          0.25


问:
Jmap can't connect to make a dump
We have an open beta of an app which occasionally causes the heapspace to overflow. The JVM reacts by going on a permanent vacation.

To analyze this I would like to peek into the memory at the point where it failed. Java does not want me to do this. The process is still in memory but it doesn't seem to be recognized as a java process.

The server in question is a debian Lenny server, Java 6u14

```<code>/opt/jdk/bin# ./jmap -F -dump:format=b,file=/tmp/apidump.hprof 11175
Attaching to process ID 11175, please wait...
sun.jvm.hotspot.debugger.NoSuchSymbolException: Could not find symbol "gHotSpotVMTypeEntryTypeNameOffset" in any of the known library names (libjvm.so, libjvm_g.so, gamma_g)
at sun.jvm.hotspot.HotSpotTypeDataBase.lookupInProcess(HotSpotTypeDataBase.java:390)
at sun.jvm.hotspot.HotSpotTypeDataBase.getLongValueFromProcess(HotSpotTypeDataBase.java:371)
at sun.jvm.hotspot.HotSpotTypeDataBase.readVMTypes(HotSpotTypeDataBase.java:102)
at sun.jvm.hotspot.HotSpotTypeDataBase.<init>(HotSpotTypeDataBase.java:85)
at sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:568)
at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:494)
at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:332)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:163)
at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jmap.JMap.runTool(JMap.java:179)
at sun.tools.jmap.JMap.main(JMap.java:110)
Debugger attached successfully.
sun.jvm.hotspot.tools.HeapDumper requires a java VM process/core!
</code>```

答:
Follow the below steps to take the thread and Heap dumps from a docker container

Run the below command to bash into the container. Please change the CONTAINER_NAME appropriately

```<code>   docker exec -it CONTAINER_NAME bash
</code>```

Then type jps to find the all the Java application details and extract the PID for your application

```<code>jps
</code>```

Then run the below command to get the thread dump. Please change the PID appropriately

```<code>jstack PID > threadDump.tdump
</code>```

Then run the below command to get the Heap dump. Please change the PID appropriately

```<code>    jmap -dump:live,format=b,file=heapDump.hprof PID
</code>```

Then exit from the docker container and download the threadDump.tdump and heapDump.hprof from the docker container by running the below command. Please change the CONTAINER_NAME appropriately

```<code> sudo docker cp CONTAINER_NAME:threadDump.tdump .
 sudo docker cp CONTAINER_NAME:heapDump.hprof .
</code>```

Title: Jmap can't connect to make a dump

Body:
We have an open beta of an app which occasionally causes the heapspace to overflow. The JVM reacts by going on a permanent vacation.

To analyze this I would like to peek into the memory at the point where it failed. Java does not want me to do this. The process is still in memory but it doesn't seem to be recognized as a java process.

The server in question is a debian Lenny server, Java 6u14

/opt/jdk/bin# ./jmap -F -dump:format=b,file=/tmp/apidump.hprof 11175
Attaching to process ID 11175, please wait...
sun.jvm.hotspot.debugger.NoSuchSymbolException: Could not find symbol "gHotSpotVMTypeEntryTypeNameOffset" in any of the known library names (libjvm.so, libjvm_g.so, gamma_g)
at sun.jvm.hotspot.HotSpotTypeDataBase.lookupInProcess(HotSpotTypeDataBase.java:390)
at sun.jvm.hotspot.HotSpotTypeDataBase.getLongValueFromProcess(HotSpotTypeDataBase.java:371)
at sun.jvm.hotspot.HotSpotTypeDataBase.readVMTypes(HotSpotTypeDataBase.java:102)
at sun.jvm.hotspot.HotSpotTypeDataBase.<init>(HotSpotTypeDataBase.java:85)
at sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:568)
at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:494)
at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:332)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:163)
at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jmap.JMap.runTool(JMap.java:179)
at sun.tools.jmap.JMap.main(JMap.java:110)
Debugger attached successfully.
sun.jvm.hotspot.tools.HeapDumper requires a java VM process/core!

Body:
Follow the below steps to take the thread and Heap dumps from a docker container

Run the below command to bash into the container. Please change the CONTAINER_NAME appropriately

   docker exec -it CONTAINER_NAME bash

Then type jps to find the all the Java application details and extract the PID for your application

jps

Then run the below command to get the thread dump. Please change the PID appropriately

jstack PID > threadDump.tdump

Then run the below command to get the Heap dump. Please change the PID appropriately

    jmap -dump:live,format=b,file=heapDump.hprof PID

Then exit from the docker container and download the threadDump.tdump and heapDump.hprof from the docker container by running the below command. Please change the CONTAINER_NAME appropriately

 sudo docker cp CONTAINER_NAME:threadDump.tdump .
 sudo docker cp CONTAINER_NAME:heapDump.hprof .


问:
no jwt token Authorization in header when it is an external source with angular2
I am having this code (angular2) :

```<code>let headers = new Headers({'Content-Type': 'application/json'});
    headers.append('Authorization',  this.authService.currentUser.token)
    let options = new RequestOptions({ headers: headers });
    return this.http.get(this.url + 'quote/' , options)
</code>```

when this.url = '/' (local request), I have Authorization in header:

When this.url = 'http://212.227.201.82/', Authorization token disappear.

How Can I include headers Authorization for external request?
Thanks for your Help

答:
I have founded the issue!
It is not from Front end, but backend! CORS (Cross-origin resource sharing) must be enabled in the API

EG with ExpressJS:

```<code>$ npm install cors
</code>```

Simple Usage (Enable All CORS Requests)

```<code>var express = require('express')
var cors = require('cors')
var app = express()

app.use(cors())

app.get('/products/:id', function (req, res, next) {
  res.json({msg: 'This is CORS-enabled for all origins!'})
})
</code>```

Reference: Allow multiple CORS domain in express js / How to allow CORS? / https://github.com/expressjs/cors

Body:
I am having this code (angular2) :

let headers = new Headers({'Content-Type': 'application/json'});
    headers.append('Authorization',  this.authService.currentUser.token)
    let options = new RequestOptions({ headers: headers });
    return this.http.get(this.url + 'quote/' , options)

when this.url = '/' (local request), I have Authorization in header:

When this.url = 'http://212.227.201.82/', Authorization token disappear.

How Can I include headers Authorization for external request?
Thanks for your Help

Body:
I have founded the issue!
It is not from Front end, but backend! CORS (Cross-origin resource sharing) must be enabled in the API

EG with ExpressJS:

$ npm install cors

Simple Usage (Enable All CORS Requests)

var express = require('express')
var cors = require('cors')
var app = express()

app.use(cors())

app.get('/products/:id', function (req, res, next) {
  res.json({msg: 'This is CORS-enabled for all origins!'})
})

Reference: Allow multiple CORS domain in express js / How to allow CORS? / https://github.com/expressjs/cors


问:
Python Selenium Can't see data
OK, so I'm new to programming with Python and HTML/Javascript but I have managed to scrape a website or two so far.  However, I have come across this site and its driving me round the bend.

My Code is:

```<code>#import libraries
from urllib.request import urlopen as ureq
from bs4 import BeautifulSoup as soup
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

#set chromepath to include .exe
chrome_path = r"C:\Program Files\Python37\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)

#set variables
myurl='http://www.ctbar.org/search/newsearch.asp'
driver.get(myurl) # loads the 1st page into the browser

element = driver.find_elements(By.ID, 'SearchResultsGrid')

print (element)
print('Done')

driver.quit()
</code>```

The only output I get is:

[]
Done

What bit(s) am I getting wrong??

Please help before I give up in frustration:-)

答:
The element you are looking for is contained within an iframe, SearchResultsFrame.

Try this:

```<code>driver.switch_to_frame('SearchResultsFrame')
element = driver.find_elements(By.ID, 'SearchResultsGrid')
</code>```

Title: Python Selenium Can't see data

Body:
OK, so I'm new to programming with Python and HTML/Javascript but I have managed to scrape a website or two so far.  However, I have come across this site and its driving me round the bend.

My Code is:

#import libraries
from urllib.request import urlopen as ureq
from bs4 import BeautifulSoup as soup
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

#set chromepath to include .exe
chrome_path = r"C:\Program Files\Python37\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)

#set variables
myurl='http://www.ctbar.org/search/newsearch.asp'
driver.get(myurl) # loads the 1st page into the browser

element = driver.find_elements(By.ID, 'SearchResultsGrid')

print (element)
print('Done')

driver.quit()

The only output I get is:

[]
Done

What bit(s) am I getting wrong??

Please help before I give up in frustration:-)

Body:
The element you are looking for is contained within an iframe, SearchResultsFrame.

Try this:

driver.switch_to_frame('SearchResultsFrame')
element = driver.find_elements(By.ID, 'SearchResultsGrid')


问:
code to make dialog box or popup Menu in iphone
i need to open dialog box or popup Menu on the button click.But i didn't find the concept of dialog box in iphone.can anybody tell me how to make dialog box or popup Menu in iphone.</p>

答: You can customize the UIAlerView or you can use the UIActionSheet to provide popup menu features.

ContentLicense: CC BY-SA 2.5


问:
Validating by attribute
I would like to check by attribute that value passed to method is correct, because adding if condition to method body make code less readable (in my opinion)

For example,

```<code>    public object GetSomething(int id)
    {
        // there could be more conditions
        if (id<= 0)
            throw new ArgumentOutOfRangeException(nameof(id), $"Forbiden value: '{id}'");
        // real logic
        ...
</code>```

I want to check it is such way

```<code>    [PossitiveValueAttr(id)]
    public object GetSomething(int id)
    {
        // real logic
</code>```

or (I would prefer that)

```<code>    public object GetSomething([PossitiveValueAttr] int id)
    {
        // real logic
</code>```

1) Do you know any similar existing solution?

2) How can I pass method's argument to my attribute?

答:
Inspired by Manfred...

So if for any method defined as

```<code>void DoSomething([Attr1] Type1 param1, [Attr2] Type2 param2, ...)
{
}
</code>```

You're happy to add as the first statement of the method:

```<code>Validate(() => DoSomething(param1, param2, ...))
</code>```

Where Attr1/Attr2 etc. all implement ValidationAttribute:

```<code>[AttributeUsage(AttributeTargets.Parameter)]
public class ArgumentAttribute : Attribute
{
    public void Validate(string argumentName, object value)
    {
        // throw exception if not valid
    }
}
</code>```

Then you can implement Validate as:

```<code>static void Validate(Expression<Action> expr)
{
    var body = (MethodCallExpression)expr.Body;

    foreach (MemberExpression a in body.Arguments)
    {
        FieldInfo fi = a.Member as FieldInfo;
        ParameterInfo pi = body.Method.GetParameters().FirstOrDefault(p => p.Name == fi.Name);
        ValidationAttribute va = pi?.GetCustomAttribute<ValidationAttribute>();
        if (va != null)
            va.Validate(pi.Name, fi.GetValue(((ConstantExpression)a.Expression).Value));
    }
}
</code>```

And use it generically wherever you need to perform argument validation defined via custom attributes.

The main drawback is that ValidationAttribute can't be generic, so you always get a plain 'object' passed in that you'll have to downcast.

ContentLicense: CC BY-SA 4.0

Title: Validating by attribute

Body:
I would like to check by attribute that value passed to method is correct, because adding if condition to method body make code less readable (in my opinion)

For example,

    public object GetSomething(int id)
    {
        // there could be more conditions
        if (id<= 0)
            throw new ArgumentOutOfRangeException(nameof(id), $"Forbiden value: '{id}'");
        // real logic
        ...

I want to check it is such way

    [PossitiveValueAttr(id)]
    public object GetSomething(int id)
    {
        // real logic

or (I would prefer that)

    public object GetSomething([PossitiveValueAttr] int id)
    {
        // real logic

1) Do you know any similar existing solution?

2) How can I pass method's argument to my attribute?

Body:
Inspired by Manfred...

So if for any method defined as

void DoSomething([Attr1] Type1 param1, [Attr2] Type2 param2, ...)
{
}

You're happy to add as the first statement of the method:

Validate(() => DoSomething(param1, param2, ...))

Where Attr1/Attr2 etc. all implement ValidationAttribute:

[AttributeUsage(AttributeTargets.Parameter)]
public class ArgumentAttribute : Attribute
{
    public void Validate(string argumentName, object value)
    {
        // throw exception if not valid
    }
}

Then you can implement Validate as:

static void Validate(Expression<Action> expr)
{
    var body = (MethodCallExpression)expr.Body;

    foreach (MemberExpression a in body.Arguments)
    {
        FieldInfo fi = a.Member as FieldInfo;
        ParameterInfo pi = body.Method.GetParameters().FirstOrDefault(p => p.Name == fi.Name);
        ValidationAttribute va = pi?.GetCustomAttribute<ValidationAttribute>();
        if (va != null)
            va.Validate(pi.Name, fi.GetValue(((ConstantExpression)a.Expression).Value));
    }
}

And use it generically wherever you need to perform argument validation defined via custom attributes.

The main drawback is that ValidationAttribute can't be generic, so you always get a plain 'object' passed in that you'll have to downcast.


问:
Tic Tac Toe checking winner in boolean method
I'm playing a TicTacToe game where the user plays the computer. The user is always X and the computer is O. I have my code written to check every row, column, and diagonal to see if the user or computer wins. If neither then it just prints false. I would like the game to print out if the user win Human player won" or if the computer wins "Computer won". I just don't know how to add it to the boolean method.</p>

```<code>public static boolean checkWin()
{
    //checks for human win
    if ("X".equals(board[0][0]) && "X".equals(board[0][1]) && "X".equals(board[0][2]))
        return true;
    else if ("X".equals(board[1][0]) && "X".equals(board[1][1]) && "X".equals(board[1][2]))
        return true;
    else if ("X".equals(board[2][0]) && "X".equals(board[2][1]) && "X".equals(board[2][2]))
        return true;
    else if ("X".equals(board[0][0]) && "X".equals(board[1][0]) && "X".equals(board[2][0]))
        return true;
    else if ("X".equals(board[0][1]) && "X".equals(board[1][1]) && "X".equals(board[2][1]))
        return true;
    else if ("X".equals(board[0][2]) && "X".equals(board[1][2]) && "X".equals(board[2][2]))
        return true;
    else if ("X".equals(board[0][0]) && "X".equals(board[1][1]) && "X".equals(board[2][2]))
        return true;
    else if ("X".equals(board[0][2]) && "X".equals(board[1][1]) && "X".equals(board[2][0]))
        return true;
    //checks if computer has won
    else if ("O".equals(board[0][0]) && "O".equals(board[0][1]) && "O".equals(board[0][2]))
        return true;
    else if ("O".equals(board[1][0]) && "O".equals(board[1][1]) && "O".equals(board[1][2]))
        return true;
    else if ("O".equals(board[2][0]) && "O".equals(board[2][1]) && "O".equals(board[2][2]))
        return true;
    else if ("O".equals(board[0][0]) && "O".equals(board[1][0]) && "O".equals(board[2][0]))
        return true;
    else if ("O".equals(board[0][1]) && "O".equals(board[1][1]) && "O".equals(board[2][1]))
        return true;
    else if ("O".equals(board[0][2]) && "O".equals(board[1][2]) && "O".equals(board[2][2]))
        return true;
    else if ("O".equals(board[0][0]) && "O".equals(board[1][1]) && "O".equals(board[2][2]))
        return true;
    else if ("O".equals(board[0][2]) && "O".equals(board[1][1]) && "O".equals(board[2][0]))
        return true;
    else
        //it's a tie
        return false;
}
</code>```

答:
Although I agree with other answers that say a method should do one thing, if you want to do it that way you can create a class, Result

```<code>public class Result{
    boolean win;
    String message;
}
</code>```

and return Result instead of boolean

```<code>public static Result checkWin()
{
    Result result = new Result();
    if ("X".equals(board[0][0]) && "X".equals(board[0][1]) && "X".equals(board[0][2])){
        result.win = true;
        result.messsage = "Computer won";
    }
    ...

    return result;

}
</code>```

Body:
I'm playing a TicTacToe game where the user plays the computer. The user is always X and the computer is O. I have my code written to check every row, column, and diagonal to see if the user or computer wins. If neither then it just prints false. I would like the game to print out if the user win Human player won" or if the computer wins "Computer won". I just don't know how to add it to the boolean method.

public static boolean checkWin()
{
    //checks for human win
    if ("X".equals(board[0][0]) && "X".equals(board[0][1]) && "X".equals(board[0][2]))
        return true;
    else if ("X".equals(board[1][0]) && "X".equals(board[1][1]) && "X".equals(board[1][2]))
        return true;
    else if ("X".equals(board[2][0]) && "X".equals(board[2][1]) && "X".equals(board[2][2]))
        return true;
    else if ("X".equals(board[0][0]) && "X".equals(board[1][0]) && "X".equals(board[2][0]))
        return true;
    else if ("X".equals(board[0][1]) && "X".equals(board[1][1]) && "X".equals(board[2][1]))
        return true;
    else if ("X".equals(board[0][2]) && "X".equals(board[1][2]) && "X".equals(board[2][2]))
        return true;
    else if ("X".equals(board[0][0]) && "X".equals(board[1][1]) && "X".equals(board[2][2]))
        return true;
    else if ("X".equals(board[0][2]) && "X".equals(board[1][1]) && "X".equals(board[2][0]))
        return true;
    //checks if computer has won
    else if ("O".equals(board[0][0]) && "O".equals(board[0][1]) && "O".equals(board[0][2]))
        return true;
    else if ("O".equals(board[1][0]) && "O".equals(board[1][1]) && "O".equals(board[1][2]))
        return true;
    else if ("O".equals(board[2][0]) && "O".equals(board[2][1]) && "O".equals(board[2][2]))
        return true;
    else if ("O".equals(board[0][0]) && "O".equals(board[1][0]) && "O".equals(board[2][0]))
        return true;
    else if ("O".equals(board[0][1]) && "O".equals(board[1][1]) && "O".equals(board[2][1]))
        return true;
    else if ("O".equals(board[0][2]) && "O".equals(board[1][2]) && "O".equals(board[2][2]))
        return true;
    else if ("O".equals(board[0][0]) && "O".equals(board[1][1]) && "O".equals(board[2][2]))
        return true;
    else if ("O".equals(board[0][2]) && "O".equals(board[1][1]) && "O".equals(board[2][0]))
        return true;
    else
        //it's a tie
        return false;
}

Body:
Although I agree with other answers that say a method should do one thing, if you want to do it that way you can create a class, Result

public class Result{
    boolean win;
    String message;
}

and return Result instead of boolean

public static Result checkWin()
{
    Result result = new Result();
    if ("X".equals(board[0][0]) && "X".equals(board[0][1]) && "X".equals(board[0][2])){
        result.win = true;
        result.messsage = "Computer won";
    }
    ...

    return result;

}


问:
Using Tensorflow on GPU interrupts the kernel and yields error: "...TensorFlow binary was not compiled to use: AVX AVX2"
Real noob here - trying to build a CNN using tensorflow operating on the GPU but I am struggling (tested it on the CPU and the code runs fine). When I run the code on the GPU

```<code>device_name = "GPU:0"
with tf.device(device_name):
    # Code here
</code>```

things get started but after a few seconds then I get an error with the following title (this is not just a notice - it actually stops the code from continuing to run):

```<code>An error ocurred while starting the kernel
</code>```

and the following details (only included the first few lines since it is long and repetitive)

```<code>2017 11:37:20.918191: I C:\tf_jenkins\home\workspace\rel‑win\M\windows‑gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2017 11:37:21.959053: I C:\tf_jenkins\home\workspace\rel‑win\M\windows‑gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62
pciBusID: 0000:01:00.0
totalMemory: 4.00GiB freeMemory: 3.29GiB
2017 11:37:21.959079: I C:\tf_jenkins\home\workspace\rel‑win\M\windows‑gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) ‑> (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2017 11:37:22.119554: I C:\tf_jenkins\home\workspace\rel‑win\M\windows‑gpu\PY\35\tensorflow\core\common_runtime\direct_session.cc:299] Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 ‑> device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1
</code>```

I am pleased it found the device - the important bit seems to be:

```<code>Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
</code>```

I have tried to ignore the warning using

```<code>import os
os.environ['TF_CPP_MIN_LOG_LEVEL']='2'
import tensorflow as tf
</code>```

but that does not change anything.

I am still learning the basics so any and all guidance would be greatly appreciated. Thank you.

答:
I think there are 2 different things going on here. Have you tried not specifying the device name? TensorFlow will automatically use all GPUs it has access to.

The AVX issues are because you probably installed TensorFlow using pip, but if you were to compile it using bazel, those warnings would go away.

Without seeing the ‘repetitive’ stack trace, it becomes difficult to know exactly what is going on.

Body:
Real noob here - trying to build a CNN using tensorflow operating on the GPU but I am struggling (tested it on the CPU and the code runs fine). When I run the code on the GPU

device_name = "GPU:0"
with tf.device(device_name):
    # Code here

things get started but after a few seconds then I get an error with the following title (this is not just a notice - it actually stops the code from continuing to run):

An error ocurred while starting the kernel

and the following details (only included the first few lines since it is long and repetitive)

2017 11:37:20.918191: I C:\tf_jenkins\home\workspace\rel‑win\M\windows‑gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2017 11:37:21.959053: I C:\tf_jenkins\home\workspace\rel‑win\M\windows‑gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62
pciBusID: 0000:01:00.0
totalMemory: 4.00GiB freeMemory: 3.29GiB
2017 11:37:21.959079: I C:\tf_jenkins\home\workspace\rel‑win\M\windows‑gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) ‑> (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2017 11:37:22.119554: I C:\tf_jenkins\home\workspace\rel‑win\M\windows‑gpu\PY\35\tensorflow\core\common_runtime\direct_session.cc:299] Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 ‑> device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1

I am pleased it found the device - the important bit seems to be:

Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2

I have tried to ignore the warning using

import os
os.environ['TF_CPP_MIN_LOG_LEVEL']='2'
import tensorflow as tf

but that does not change anything.

I am still learning the basics so any and all guidance would be greatly appreciated. Thank you.


问:
BindError: Unable to find a consistent port for localhost
Sorry to be asking this again, I see it has been asked before, but there were no answers, and I still have this problem. I did not know how to ping the question. :)

I have periodic problems running the Google App Engine development server on localhost.
I want to stress that they are periodic problems, that I have a hard time figuring out where stems from.
I am running on Mac OS X Mavericks.

I am mostly working from two locations, and it has previously been working both places, now I only have the problem in one location. I have tested stupid moves as being on wired or wireless, or even without internet.

I run

```<code>dev_appserver.py ShoppingList/
</code>```

And get the following output

```<code>INFO     2014-10-29 20:19:40,423 devappserver2.py:733] Skipping SDK update check.
WARNING  2014-10-29 20:19:40,427 api_server.py:383] Could not initialize images API; you are likely missing the Python "PIL" module.
WARNING  2014-10-29 20:19:40,428 simple_search_stub.py:1098] Could not read search indexes from /var/folders/0x/6w6yzwhx28780kdnwpqdz2dc0000gn/T/appengine.husk-lige.mortenmedici/search_indexes
Traceback (most recent call last):
  File "/Users/mortenmedici/Directory to extract under (this will create a directory google-cloud-sdk) /Users/mortenmedici install_dst/google-cloud-sdk/platform/google_appengine/dev_appserver.py", line 82, in <module>
    _run_file(__file__, globals())
  File "/Users/mortenmedici/Directory to extract under (this will create a directory google-cloud-sdk) /Users/mortenmedici install_dst/google-cloud-sdk/platform/google_appengine/dev_appserver.py", line 78, in _run_file
    execfile(_PATHS.script_file(script_name), globals_)
  File "/Users/mortenmedici/Directory to extract under (this will create a directory google-cloud-sdk) /Users/mortenmedici install_dst/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 986, in <module>
    main()
  File "/Users/mortenmedici/Directory to extract under (this will create a directory google-cloud-sdk) /Users/mortenmedici install_dst/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 979, in main
    dev_server.start(options)
  File "/Users/mortenmedici/Directory to extract under (this will create a directory google-cloud-sdk) /Users/mortenmedici install_dst/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 778, in start
    apis.start()
  File "/Users/mortenmedici/Directory to extract under (this will create a directory google-cloud-sdk) /Users/mortenmedici install_dst/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 170, in start
    super(APIServer, self).start()
  File "/Users/mortenmedici/Directory to extract under (this will create a directory google-cloud-sdk) /Users/mortenmedici install_dst/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 314, in start
    raise BindError('Unable to find a consistent port for %s' % host)
google.appengine.tools.devappserver2.wsgi_server.BindError: Unable to find a consistent port for localhost
</code>```

I have tried the solutions on the BindError from the other answers:
Apps won't run on GAE - 'unable to bind to localhost:0'

Failed to start devlopment server -- BindError: Unable to find a consistent port localhost
I have no other instances of the server running, I have only one localhost in my /etc/hosts and it does not help to run on a different port.

Notice that the Error is only thrown to 'localhost' not a 'localhost:0' or some specific port as in other questions. I do not know if that provide any info.

All help will be appreciated.

答:
So I've recently had this problem. Let me start by saying I did the "lsof -i :8080" to search for a process already bound to that port, yet it still kept happening.

Turns out the issue was the /etc/hosts file itself. It was empty. After populating it..it still didn't work. Only after changing the permissions to 644 did it work correctly. so...

```<code>    sudo chmod 644 /etc/hosts
</code>```

let me know If that helps.

Body:
Sorry to be asking this again, I see it has been asked before, but there were no answers, and I still have this problem. I did not know how to ping the question. :)

I have periodic problems running the Google App Engine development server on localhost.
I want to stress that they are periodic problems, that I have a hard time figuring out where stems from.
I am running on Mac OS X Mavericks.

I am mostly working from two locations, and it has previously been working both places, now I only have the problem in one location. I have tested stupid moves as being on wired or wireless, or even without internet.

I run

dev_appserver.py ShoppingList/

And get the following output

INFO     2014-10-29 20:19:40,423 devappserver2.py:733] Skipping SDK update check.
WARNING  2014-10-29 20:19:40,427 api_server.py:383] Could not initialize images API; you are likely missing the Python "PIL" module.
WARNING  2014-10-29 20:19:40,428 simple_search_stub.py:1098] Could not read search indexes from /var/folders/0x/6w6yzwhx28780kdnwpqdz2dc0000gn/T/appengine.husk-lige.mortenmedici/search_indexes
Traceback (most recent call last):
  File "/Users/mortenmedici/Directory to extract under (this will create a directory google-cloud-sdk) /Users/mortenmedici install_dst/google-cloud-sdk/platform/google_appengine/dev_appserver.py", line 82, in <module>
    _run_file(__file__, globals())
  File "/Users/mortenmedici/Directory to extract under (this will create a directory google-cloud-sdk) /Users/mortenmedici install_dst/google-cloud-sdk/platform/google_appengine/dev_appserver.py", line 78, in _run_file
    execfile(_PATHS.script_file(script_name), globals_)
  File "/Users/mortenmedici/Directory to extract under (this will create a directory google-cloud-sdk) /Users/mortenmedici install_dst/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 986, in <module>
    main()
  File "/Users/mortenmedici/Directory to extract under (this will create a directory google-cloud-sdk) /Users/mortenmedici install_dst/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 979, in main
    dev_server.start(options)
  File "/Users/mortenmedici/Directory to extract under (this will create a directory google-cloud-sdk) /Users/mortenmedici install_dst/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 778, in start
    apis.start()
  File "/Users/mortenmedici/Directory to extract under (this will create a directory google-cloud-sdk) /Users/mortenmedici install_dst/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 170, in start
    super(APIServer, self).start()
  File "/Users/mortenmedici/Directory to extract under (this will create a directory google-cloud-sdk) /Users/mortenmedici install_dst/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 314, in start
    raise BindError('Unable to find a consistent port for %s' % host)
google.appengine.tools.devappserver2.wsgi_server.BindError: Unable to find a consistent port for localhost

I have tried the solutions on the BindError from the other answers:
Apps won't run on GAE - 'unable to bind to localhost:0'

Failed to start devlopment server -- BindError: Unable to find a consistent port localhost
I have no other instances of the server running, I have only one localhost in my /etc/hosts and it does not help to run on a different port.

Notice that the Error is only thrown to 'localhost' not a 'localhost:0' or some specific port as in other questions. I do not know if that provide any info.

All help will be appreciated.

Body:
So I've recently had this problem. Let me start by saying I did the "lsof -i :8080" to search for a process already bound to that port, yet it still kept happening.

Turns out the issue was the /etc/hosts file itself. It was empty. After populating it..it still didn't work. Only after changing the permissions to 644 did it work correctly. so...

    sudo chmod 644 /etc/hosts

let me know If that helps.


问:
how to alter primary key to set Default value DEFAULT SYS_GUID() in ORACLE
I create table in oracle and I want add auto increment for my primary key

```<code>CREATE TABLE "TEST_1"."PERSON"
   ("ID" NUMBER NOT NULL ENABLE,
    "FNAME" VARCHAR2(20 BYTE),
    "LNAME" VARCHAR2(20 BYTE),
     CONSTRAINT "PERSON_PK" PRIMARY KEY ("ID"));
</code>```

Using  Oracle sql develope when I want alter ID to get Auto Increment for primary key I get error  ORA-02262: ORA-932

I have two raw in table

```<code>ALTER TABLE PERSON
MODIFY (ID DEFAULT SYS_GUID() );
</code>```

答:
(Note: my answer and examples are using Oracle 11g)

Reason for the issue/error

The Oracle error ORA-02262 is thrown when there are inconsistent data types. In your case, when creating the table you specify that the ID column is of type NUMBER:

```<code>CREATE TABLE "TEST_1"."PERSON"
("ID" NUMBER NOT NULL ENABLE, ...
</code>```

The SYS_GUID() function in Oracle "generates and returns a globally unique identifier (RAW value) made up of 16 bytes). The documentation then states that the 16-byte RAW value can be represented by a 32-character hexadecimal representation, or VARCHAR2(32).

For this reason, when SYS_GUID() is used as the default value of a column in Oracle, the result is often stored as a 32-byte VARCHAR or VARCHAR2:

```<code>CREATE TABLE TEST_1.PERSON (
  ID VARCHAR2(32) NOT NULL ENABLE DEFAULT SYS_GUID(),
  ...
);
</code>```

Solution(s)

If altering the data type of the column is a suitable solution, the code below will successfully alter the table to store SYS_GUID() values as identifiers:

```<code>ALTER TABLE TEST_1.PERSON MODIFY(
  ID VARCHAR2(32) DEFAULT SYS_GUID()
);
</code>```

If you must have a numeric value as the ID in your table, there is an excellent answer here:

How to create id with AUTO_INCREMENT on Oracle?

Body:
I create table in oracle and I want add auto increment for my primary key

CREATE TABLE "TEST_1"."PERSON"
   ("ID" NUMBER NOT NULL ENABLE,
    "FNAME" VARCHAR2(20 BYTE),
    "LNAME" VARCHAR2(20 BYTE),
     CONSTRAINT "PERSON_PK" PRIMARY KEY ("ID"));

Using  Oracle sql develope when I want alter ID to get Auto Increment for primary key I get error  ORA-02262: ORA-932

I have two raw in table

ALTER TABLE PERSON
MODIFY (ID DEFAULT SYS_GUID() );

Body:
(Note: my answer and examples are using Oracle 11g)

Reason for the issue/error

The Oracle error ORA-02262 is thrown when there are inconsistent data types. In your case, when creating the table you specify that the ID column is of type NUMBER:

CREATE TABLE "TEST_1"."PERSON"
("ID" NUMBER NOT NULL ENABLE, ...

The SYS_GUID() function in Oracle "generates and returns a globally unique identifier (RAW value) made up of 16 bytes). The documentation then states that the 16-byte RAW value can be represented by a 32-character hexadecimal representation, or VARCHAR2(32).

For this reason, when SYS_GUID() is used as the default value of a column in Oracle, the result is often stored as a 32-byte VARCHAR or VARCHAR2:

CREATE TABLE TEST_1.PERSON (
  ID VARCHAR2(32) NOT NULL ENABLE DEFAULT SYS_GUID(),
  ...
);

Solution(s)

If altering the data type of the column is a suitable solution, the code below will successfully alter the table to store SYS_GUID() values as identifiers:

ALTER TABLE TEST_1.PERSON MODIFY(
  ID VARCHAR2(32) DEFAULT SYS_GUID()
);

If you must have a numeric value as the ID in your table, there is an excellent answer here:

How to create id with AUTO_INCREMENT on Oracle?


问:
Easy way to excecute method after a given delay?
Is there a easy way to perform a method after a given delay like in iOS out of the box?

On iPhone I would do this:

[self performSelector:@selector(connectSensor) withObject:nil afterDelay:2.5];

It will then schedule the method connectSensor on the main thread (UI thread) to be executed after 2,5 seconds. And because it is automatically scheduled on the main thread, you don't have to worry about cross thread issues. (There is also a performSelectorOnBackground version)

So how would I do this properly in WP7?

Currently I'm accomplishing this with a timer, but I'm not sure if this is a good solution.

```<code>    private Timer timer;
    private void DoSomethingAfterDaly()
    {
        // ... do something here

        timer = new Timer( (o) => Deployment.Current.Dispatcher.BeginInvoke(() => NavigationService.GoBack()), null, 2500, Timeout.Infinite);
    }
</code>```

How could this be encapsulated into an extension method so I can just call this.Perform(MyMethod, null, 2500); ?

答:
You can use a BackgroundWorker like so:

```<code>    private void Perform(Action myMethod, int delayInMilliseconds)
    {
        BackgroundWorker worker = new BackgroundWorker();

        worker.DoWork += (s, e) => Thread.Sleep(delayInMilliseconds);

        worker.RunWorkerCompleted += (s, e) => myMethod.Invoke();

        worker.RunWorkerAsync();
    }
</code>```

The call into this method would look like this:

```<code>this.Perform(() => MyMethod(), 2500);
</code>```

The background worker will run the sleep on a thread off of the UI thread so your application is free to do other things while the delay is occurring.

Body:
Is there a easy way to perform a method after a given delay like in iOS out of the box?

On iPhone I would do this:

[self performSelector:@selector(connectSensor) withObject:nil afterDelay:2.5];

It will then schedule the method connectSensor on the main thread (UI thread) to be executed after 2,5 seconds. And because it is automatically scheduled on the main thread, you don't have to worry about cross thread issues. (There is also a performSelectorOnBackground version)

So how would I do this properly in WP7?

Currently I'm accomplishing this with a timer, but I'm not sure if this is a good solution.

    private Timer timer;
    private void DoSomethingAfterDaly()
    {
        // ... do something here

        timer = new Timer( (o) => Deployment.Current.Dispatcher.BeginInvoke(() => NavigationService.GoBack()), null, 2500, Timeout.Infinite);
    }

How could this be encapsulated into an extension method so I can just call this.Perform(MyMethod, null, 2500); ?

Body:
You can use a BackgroundWorker like so:

    private void Perform(Action myMethod, int delayInMilliseconds)
    {
        BackgroundWorker worker = new BackgroundWorker();

        worker.DoWork += (s, e) => Thread.Sleep(delayInMilliseconds);

        worker.RunWorkerCompleted += (s, e) => myMethod.Invoke();

        worker.RunWorkerAsync();
    }

The call into this method would look like this:

this.Perform(() => MyMethod(), 2500);

The background worker will run the sleep on a thread off of the UI thread so your application is free to do other things while the delay is occurring.


问:
Insert records into SQL Server database from user defined table type using stored procedure
Scenario

I have this Datatable which I'm getting from when user assign role to other user:

In the above table, I assign a new role to user mushtaq and through below procedure I want to insert it in my database.

```<code>ALTER PROCEDURE [dbo].[SP_ROLE_ASSIGNMENT_ASSIGN_ROLES_TO_USER]
    @RoleAssignmentDT _ROLEASSIGNMENTSCHEMA ReadOnly
AS
BEGIN TRY
BEGIN TRANSACTION
    IF EXISTS (SELECT Role_ID
               FROM Role_Assignment RA
               INNER JOIN @RoleAssignmentDT t ON t._Role_ID = Ra.Role_ID
                                              AND RA.Username = t._Username)
    BEGIN
        UPDATE Role_Assignment
        SET Role_Assign = 0,
            Modified_By_ID = t._Modified_By_ID,
            Modified_Date = GETDATE()
        FROM @RoleAssignmentDT t
        WHERE Role_Assignment.Role_ID <> t._Role_ID
          AND Role_Assignment.Username = t._Username
          AND (t._Role_Assign = 1 AND Role_Assignment.Role_Assign = 1)

        UPDATE Role_Assignment
        SET Role_Assign = t._Role_Assign,
            Modified_By_ID = t._Modified_By_ID,
            Modified_Date = GETDATE()
        FROM @RoleAssignmentDT t
        WHERE Role_Assignment.Role_ID = t._Role_ID
          AND Role_Assignment.Username = t._Username
    END
    ELSE
    BEGIN
        INSERT INTO Role_Assignment (Role_ID, Username, Role_Assign, Prepared_By_ID, Prepared_Date)
            SELECT
                t._Role_ID, t._Username, t._Role_Assign, t._Prepared_By_ID, GETDATE()
            FROM
                @RoleAssignmentDT t
    END

    COMMIT TRAN -- Transaction Success!
END TRY
BEGIN CATCH
    IF @@TRANCOUNT > 0
        ROLLBACK TRAN --RollBack in case of Error

-- you can Raise ERROR with RAISEERROR() Statement including the details of the exception
--RAISERROR('No Record Inserted, Contact to administrator',16,1)
    DECLARE @Msg NVARCHAR(MAX)
    SELECT @Msg=ERROR_MESSAGE()
    RAISERROR('Error Occured: %s', 20, 101,@msg) WITH LOG
END CATCH
</code>```

Below is my SQL Role_Assignment table's schema

Now, when I assign new roles to other user above procedure only updates user ahmer. Because when my procedure gets the datatable my IF condition run and it finds user ahmer and updates it.

What I want

I want the stored procedure to check and update only those users who exist in the SQL Server table Role_Assignment; if they don't exist, then insert them first.

I think I provide brief information about my current problem. Please help me what modification I need to do in my store procedure so it first check weather user exists or not. Those who not exists before then insert them and update other those who exists.

答:
Try the following in place of your t-sql code in the stored procedure   </p>

```<code> MERGE Role_Assignment ra
    USING @RoleAssignmentDT t
    ON Role_Assignment.Username = t._Username
    WHEN MATCHED AND
       Role_Assignment.Role_ID <> t._Role_ID
    and (t._Role_Assign = 1 and Ra.Role_Assign = 1) THEN
       UPDATE
        Set Role_Assign = 0, Modified_By_ID = t._Modified_By_ID, Modified_Date = GETDATE()
        From @RoleAssignmentDT
    WHEN MATCHED AND
    Role_Assignment.Role_ID = t._Role_ID  THEN
      UPDATE
        Set Role_Assign = t._Role_Assign, Modified_By_ID = t._Modified_By_ID, Modified_Date = GETDATE()
        From @RoleAssignmentDT
    WHEN NOT MATCHED BY TARGET THEN
      Insert (Role_ID,Username,Role_Assign,Prepared_By_ID,Prepared_Date)
        VALUES (t._Role_ID, t._Username, t._Role_Assign, t._Prepared_By_ID, GETDATE());
</code>```

Body:
Scenario

I have this Datatable which I'm getting from when user assign role to other user:

In the above table, I assign a new role to user mushtaq and through below procedure I want to insert it in my database.

ALTER PROCEDURE [dbo].[SP_ROLE_ASSIGNMENT_ASSIGN_ROLES_TO_USER]
    @RoleAssignmentDT _ROLEASSIGNMENTSCHEMA ReadOnly
AS
BEGIN TRY
BEGIN TRANSACTION
    IF EXISTS (SELECT Role_ID
               FROM Role_Assignment RA
               INNER JOIN @RoleAssignmentDT t ON t._Role_ID = Ra.Role_ID
                                              AND RA.Username = t._Username)
    BEGIN
        UPDATE Role_Assignment
        SET Role_Assign = 0,
            Modified_By_ID = t._Modified_By_ID,
            Modified_Date = GETDATE()
        FROM @RoleAssignmentDT t
        WHERE Role_Assignment.Role_ID <> t._Role_ID
          AND Role_Assignment.Username = t._Username
          AND (t._Role_Assign = 1 AND Role_Assignment.Role_Assign = 1)

        UPDATE Role_Assignment
        SET Role_Assign = t._Role_Assign,
            Modified_By_ID = t._Modified_By_ID,
            Modified_Date = GETDATE()
        FROM @RoleAssignmentDT t
        WHERE Role_Assignment.Role_ID = t._Role_ID
          AND Role_Assignment.Username = t._Username
    END
    ELSE
    BEGIN
        INSERT INTO Role_Assignment (Role_ID, Username, Role_Assign, Prepared_By_ID, Prepared_Date)
            SELECT
                t._Role_ID, t._Username, t._Role_Assign, t._Prepared_By_ID, GETDATE()
            FROM
                @RoleAssignmentDT t
    END

    COMMIT TRAN -- Transaction Success!
END TRY
BEGIN CATCH
    IF @@TRANCOUNT > 0
        ROLLBACK TRAN --RollBack in case of Error

-- you can Raise ERROR with RAISEERROR() Statement including the details of the exception
--RAISERROR('No Record Inserted, Contact to administrator',16,1)
    DECLARE @Msg NVARCHAR(MAX)
    SELECT @Msg=ERROR_MESSAGE()
    RAISERROR('Error Occured: %s', 20, 101,@msg) WITH LOG
END CATCH

Below is my SQL Role_Assignment table's schema

Now, when I assign new roles to other user above procedure only updates user ahmer. Because when my procedure gets the datatable my IF condition run and it finds user ahmer and updates it.

What I want

I want the stored procedure to check and update only those users who exist in the SQL Server table Role_Assignment; if they don't exist, then insert them first.

I think I provide brief information about my current problem. Please help me what modification I need to do in my store procedure so it first check weather user exists or not. Those who not exists before then insert them and update other those who exists.

Body:
Try the following in place of your t-sql code in the stored procedure

 MERGE Role_Assignment ra
    USING @RoleAssignmentDT t
    ON Role_Assignment.Username = t._Username
    WHEN MATCHED AND
       Role_Assignment.Role_ID <> t._Role_ID
    and (t._Role_Assign = 1 and Ra.Role_Assign = 1) THEN
       UPDATE
        Set Role_Assign = 0, Modified_By_ID = t._Modified_By_ID, Modified_Date = GETDATE()
        From @RoleAssignmentDT
    WHEN MATCHED AND
    Role_Assignment.Role_ID = t._Role_ID  THEN
      UPDATE
        Set Role_Assign = t._Role_Assign, Modified_By_ID = t._Modified_By_ID, Modified_Date = GETDATE()
        From @RoleAssignmentDT
    WHEN NOT MATCHED BY TARGET THEN
      Insert (Role_ID,Username,Role_Assign,Prepared_By_ID,Prepared_Date)
        VALUES (t._Role_ID, t._Username, t._Role_Assign, t._Prepared_By_ID, GETDATE());


问:
Save params between script runs in iPython console
I want to find iPython console equivalent to the Spyder console command.

When I use Spyder app all my variables are saved between script runs.
By that I don't only mean I can inspect the values after script finished running but that those values will be preserved within next script run.

Spyder console command (doesn't work in iPython console):

runfile('some_file.py', wdir='/some/project/folder')

There is a similar command in iPython console:

%run -i "some_script.py"

The problem is that this command deletes old values when new script starts executing.

Why is this important?

Let's say my script among other things builds some model which takes long(er) then I'm willing to wait (every time). In  Spyder I can run it just the first time and then comment out this line of code and next time only rest of the code is run and model is pulled from working memory.

(yes I know I can save the model in pickle format etc. but that's totally beside the point)

EDIT:

This is awkward but for some reason I don't have any problems with -i flag anymore. So I'm able to get the desired functionality with it. Maybe it's because of newer version of Anaconda.

答:
You can save the entire session using dill (An extension of python pickle). So every end of your script you would save the current session.

```<code>import dill
dill.dump_session("temp.pkl")
</code>```

And every start of the script you would load the previous session.

```<code>import dill
dill.load_session("temp.pkl")
</code>```

This retains all variables in the session so you can just comment out variables you don't need to change.

Body:
You can save the entire session using dill (An extension of python pickle). So every end of your script you would save the current session.

import dill
dill.dump_session("temp.pkl")

And every start of the script you would load the previous session.

import dill
dill.load_session("temp.pkl")

This retains all variables in the session so you can just comment out variables you don't need to change.


问:
Arithmetic overflow when trying to take an average
I am getting the following error when trying to execute a query

Arithmetic overflow error converting nvarchar to data type numeric

I have narrowed the issue down to the included code. My problem is I am unsure of how to handle the overflow error. Any help would be appreciated. Thank you.

```<code>avg(cast((Substring([Ice].Entry(5, UD03.Character01,';'),0,
CharIndex('|',[Ice].Entry(5, UD03.Character01,';'))))
as decimal(38,38))) as [Calculated_Markup]
</code>```

答:
Use try_cast() or try_convert():

```<code>avg(try_convert(decimal(38, 38),
                Substring([Ice].Entry(5, UD03.Character01,';'), 1,
                          CharIndex('|', [Ice].Entry(5, UD03.Character01, ';')))) as [Calculated_Markup]
</code>```

Note:  I changed the 0 to a 1 because substring processing starts at 1.  You can write this more succinctly using LEFT():

```<code>avg(try_convert(decimal(38, 38),
                left([Ice].Entry(5, UD03.Character01,';'),
                     CharIndex('|', [Ice].Entry(5, UD03.Character01, ';')))) as [Calculated_Markup]
</code>```

Note:  This fixes the error by using NULL if the convert fails.

I will note that decimal(38, 38) is a trifle unusual -- that is a number that has only 38 digits after the decimal.  Often there are more digits before the decimal, so I would expect something more like decimal(38, 15).  That would be 15 digits after and 23 before.

Body:
I am getting the following error when trying to execute a query

Arithmetic overflow error converting nvarchar to data type numeric

I have narrowed the issue down to the included code. My problem is I am unsure of how to handle the overflow error. Any help would be appreciated. Thank you.

avg(cast((Substring([Ice].Entry(5, UD03.Character01,';'),0,
CharIndex('|',[Ice].Entry(5, UD03.Character01,';'))))
as decimal(38,38))) as [Calculated_Markup]

Body:
Use try_cast() or try_convert():

avg(try_convert(decimal(38, 38),
                Substring([Ice].Entry(5, UD03.Character01,';'), 1,
                          CharIndex('|', [Ice].Entry(5, UD03.Character01, ';')))) as [Calculated_Markup]

Note:  I changed the 0 to a 1 because substring processing starts at 1.  You can write this more succinctly using LEFT():

avg(try_convert(decimal(38, 38),
                left([Ice].Entry(5, UD03.Character01,';'),
                     CharIndex('|', [Ice].Entry(5, UD03.Character01, ';')))) as [Calculated_Markup]

Note:  This fixes the error by using NULL if the convert fails.

I will note that decimal(38, 38) is a trifle unusual -- that is a number that has only 38 digits after the decimal.  Often there are more digits before the decimal, so I would expect something more like decimal(38, 15).  That would be 15 digits after and 23 before.


问:
What is MASM ? which we generally use it for learning assembly language code?
In 8086 processor,what ic's we use are different...i think they are of 16 bit...but we practice the code in 64 bit computers..how is it possible?please expalin>>>>What is MaSm actually?? and can you explain me where I can get the software....</p>

答:
MASM stands for "microsoft macro assembler". In modern (32 and 64 bit) here's the so called "real mode" which supports also 16, and even 8, bit register, for backward compatibility. Basically a processor "emulates" an older one so programs designed for that processor will run even on newer ones. For MASM just ask google.

 I suggest you using Easy Code with MASM. It's an IDE that allows you to build win32 executables using assembly language.


问:
Powershell: Get-Item vs Get-ChildItem
I would like understand the difference between Get-Item and Get-ChildItem.
If is possible with one example.

答:
Get the current directory

```<code>Get-Item .
</code>```

Get all the items in the current directory

```<code># use Get-Item
Get-Item *
# or use Get-ChildItem
Get-ChildItem
</code>```

Get all txt files

```<code># use Get-Item
Get-Item *.txt
# or use Get-ChildItem
Get-ChildItem *.txt
</code>```

Reference:

https://learn.microsoft.com/en-us/powershell/module/Microsoft.PowerShell.Management/Get-Item?view=powershell-5.1

https://learn.microsoft.com/en-us/powershell/module/Microsoft.PowerShell.Management/Get-ChildItem?view=powershell-5.1

Title: Powershell: Get-Item vs Get-ChildItem

Body:
Get the current directory

Get-Item .

Get all the items in the current directory

# use Get-Item
Get-Item *
# or use Get-ChildItem
Get-ChildItem

Get all txt files

# use Get-Item
Get-Item *.txt
# or use Get-ChildItem
Get-ChildItem *.txt

Reference:

https://learn.microsoft.com/en-us/powershell/module/Microsoft.PowerShell.Management/Get-Item?view=powershell-5.1

https://learn.microsoft.com/en-us/powershell/module/Microsoft.PowerShell.Management/Get-ChildItem?view=powershell-5.1


问:
Update row, delete the same row if update fails
I have a table that aggregate items and their categories. An item can have multiple categories. Here is the simplified schema and example data:

Table name: xref, (itemID, catID) defined as primary key

```<code>itemID | catID
-------+-------
4059   | 159
4059   | 219
</code>```

I also have a category merge function. When doing this, I also move products that linked in source category to the destination (merged) category . For moving items from category 159 to 219, I use a SQL like this:

```<code>UPDATE `xref` SET `catID` = 219 WHERE `catID` = 159
</code>```

But the update fails because there is already a 4059-219 (PK constraint).

The desired result is

```<code>itemID | catID
-------+-------
4059   | 219
</code>```

I can't use stored procedures or functions on the database but can implement the required steps in my application.

Is there a way to delete row when an UPDATE query fails?

How can I write a query that "predicts" rows for to-be-failed UPDATE query? Something like "return itemID's that having catID's 219 AND 159 at the same time". Is SELECT * FROM xref WHERE catID IN (159, 219) GROUP BY itemID HAVING COUNT(*) >= 2 is a correct approach?

答:
This way temporarily removes your constraints to allow the update and then puts them back at the end.

If your database is a production database TEST ON A COPY!

```<code># set up tables for the purpose of this example
CREATE TABLE Item
(
    ItemId INT,
    ItemName VARCHAR(255),
    PRIMARY KEY (ItemId)
);

 CREATE Table Category
 (
     CatId INT,
     CatName VARCHAR(255),
      PRIMARY KEY (CatId)
 );

 CREATE TABLE xref
(
    ItemId INT,
    CatId INT,
    PRIMARY KEY (ItemId,CatId),
    FOREIGN KEY (ItemID) REFERENCES Item(ItemId),
    FOREIGN KEY (CatId) REFERENCES Category(CatId)
 );

 # insert data for the purpose of this example
 INSERT INTO Item
 VALUES
 (1,'Hammer'),
 (2,'Chisel'),
 (3,'Wrench');

 INSERT INTO Category
 VALUES
 (159,'Tool'),
 (219,'Toy'),
 (3,'foo');

 INSERT INTO xref
 VALUES
 (1,159),
 (1,219),
 (2,159),
 (3,3);

 START TRANSACTION;

# remove constraints so we can run the update
ALTER TABLE xref DROP FOREIGN KEY xref_ibfk_1;
ALTER TABLE xref DROP FOREIGN KEY xref_ibfk_2;
ALTER TABLE xref DROP PRIMARY KEY;

# Update the categories
UPDATE xref SET CatId = 219 WHERE CatId = 159;

# Copy the table into a tmep table but exclude duplicates we have created above
CREATE TEMPORARY TABLE IF NOT EXISTS xref_tmp AS (SELECT DISTINCT
                                                           ItemId,
                                                           CatId
                                                    FROM    xref);

# Empty the original table
DELETE FROM xref;

# Copy the clean results back to the orginal table
INSERT INTO xref
SELECT * FROM xref_tmp;

# put our constaints back
ALTER TABLE xref ADD PRIMARY KEY (ItemId,CatId);
ALTER TABLE xref ADD FOREIGN KEY (ItemID) REFERENCES Item(ItemID);
ALTER TABLE xref ADD FOREIGN KEY (CatID) REFERENCES Category(CatID);

COMMIT
</code>```

Body:
I have a table that aggregate items and their categories. An item can have multiple categories. Here is the simplified schema and example data:

Table name: xref, (itemID, catID) defined as primary key

itemID | catID
-------+-------
4059   | 159
4059   | 219

I also have a category merge function. When doing this, I also move products that linked in source category to the destination (merged) category . For moving items from category 159 to 219, I use a SQL like this:

UPDATE `xref` SET `catID` = 219 WHERE `catID` = 159

But the update fails because there is already a 4059-219 (PK constraint).

The desired result is

itemID | catID
-------+-------
4059   | 219

I can't use stored procedures or functions on the database but can implement the required steps in my application.

Is there a way to delete row when an UPDATE query fails?

How can I write a query that "predicts" rows for to-be-failed UPDATE query? Something like "return itemID's that having catID's 219 AND 159 at the same time". Is SELECT * FROM xref WHERE catID IN (159, 219) GROUP BY itemID HAVING COUNT(*) >= 2 is a correct approach?

Body:
This way temporarily removes your constraints to allow the update and then puts them back at the end.

If your database is a production database TEST ON A COPY!

# set up tables for the purpose of this example
CREATE TABLE Item
(
    ItemId INT,
    ItemName VARCHAR(255),
    PRIMARY KEY (ItemId)
);

 CREATE Table Category
 (
     CatId INT,
     CatName VARCHAR(255),
      PRIMARY KEY (CatId)
 );

 CREATE TABLE xref
(
    ItemId INT,
    CatId INT,
    PRIMARY KEY (ItemId,CatId),
    FOREIGN KEY (ItemID) REFERENCES Item(ItemId),
    FOREIGN KEY (CatId) REFERENCES Category(CatId)
 );

 # insert data for the purpose of this example
 INSERT INTO Item
 VALUES
 (1,'Hammer'),
 (2,'Chisel'),
 (3,'Wrench');

 INSERT INTO Category
 VALUES
 (159,'Tool'),
 (219,'Toy'),
 (3,'foo');

 INSERT INTO xref
 VALUES
 (1,159),
 (1,219),
 (2,159),
 (3,3);

 START TRANSACTION;

# remove constraints so we can run the update
ALTER TABLE xref DROP FOREIGN KEY xref_ibfk_1;
ALTER TABLE xref DROP FOREIGN KEY xref_ibfk_2;
ALTER TABLE xref DROP PRIMARY KEY;

# Update the categories
UPDATE xref SET CatId = 219 WHERE CatId = 159;

# Copy the table into a tmep table but exclude duplicates we have created above
CREATE TEMPORARY TABLE IF NOT EXISTS xref_tmp AS (SELECT DISTINCT
                                                           ItemId,
                                                           CatId
                                                    FROM    xref);

# Empty the original table
DELETE FROM xref;

# Copy the clean results back to the orginal table
INSERT INTO xref
SELECT * FROM xref_tmp;

# put our constaints back
ALTER TABLE xref ADD PRIMARY KEY (ItemId,CatId);
ALTER TABLE xref ADD FOREIGN KEY (ItemID) REFERENCES Item(ItemID);
ALTER TABLE xref ADD FOREIGN KEY (CatID) REFERENCES Category(CatID);

COMMIT


问:
AWS connection timeout when running Spark job on EMR
I'm trying to submit a simple spark job in an Amazon EMR cluster. My cluster has 5 M4.2xlarge instances (1 master, 4 slaves), each with 16 vCPU, and 32 gigs of memory.

This is my code:

```<code>def main(args : Array[String]): Unit = {
 val sparkConfig = new SparkConf()
  .set("hive.exec.dynamic.partition", "true")
  .set("hive.exec.dynamic.partition.mode", "nonstrict")
  .set("hive.s3.max-client-retries", "50")
  .set("hive.s3.max-error-retries", "50")
  .set("hive.s3.max-connections", "100")
  .set("hive.s3.connect-timeout", "5m")
  .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
  .set("spark.kryo.registrationRequired", "true")
  .set("spark.kryo.classesToRegister", "org.apache.spark.graphx.impl.VertexAttributeBlock")
  .set("spark.broadcast.compress", "true")

 val spark = SparkSession.builder()
    .appName("Spark Hive Example")
    .enableHiveSupport()
    .config(sparkConfig)
    .getOrCreate()

// Set Kryo for serializing
GraphXUtils.registerKryoClasses(sparkConfig)
val res = spark.sql("SELECT col1, col2, col3 FROM table1 limit 10000")
val edgesRDD = res.rdd.map(row => Edge(row.getString(0).hashCode, row.getString(1).hashCode, row(2).asInstanceOf[String]))

val res_two = spark.sql("SELECT col1 FROM table2 where col1 is not NULL and col1 != '' limit 100000")
val vertexRDD: RDD[(VertexId, String)] = res_two.rdd.map(row => (row.getString(0).hashCode, row(0).asInstanceOf[String]))

val graph = Graph(vertexRDD, edgesRDD)

val connectedComponents = graph.connectedComponents().vertices
</code>```

Both table1, and table2 are S3 backed external tables on hive. When I run this program, my job fails with the following error:

```<code>Job aborted due to stage failure: Task 827 in stage 0.0 failed 4 times, most recent failure: Lost task 827.3 in stage 0.0 (TID 921, xxx.internal, executor 3): com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1069)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1035)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4169)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4116)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1237)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.call.GetObjectMetadataCall.perform(GetObjectMetadataCall.java:24)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.call.GetObjectMetadataCall.perform(GetObjectMetadataCall.java:10)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor.execute(GlobalS3Executor.java:82)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:176)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.getObjectMetadata(AmazonS3LiteClient.java:94)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.AbstractAmazonS3Lite.getObjectMetadata(AbstractAmazonS3Lite.java:39)
    at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:211)
    at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy35.retrieveMetadata(Unknown Source)
    at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.getFileStatus(S3NativeFileSystem.java:768)
    at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.open(S3NativeFileSystem.java:1194)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:773)
    at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.open(EmrFileSystem.java:166)
    at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:355)
    at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:316)
    at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:237)
    at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1204)
    at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:246)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:245)
    at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:203)
    at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:94)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
    at com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:286)
    at com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
    at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.conn.$Proxy37.get(Unknown Source)
    at com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
    at com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
    at com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1190)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
    ... 59 more
</code>```

Not sure if it is coming from hadoop or when reading from hive, but I saw a similar issue here, so I added the following params in my spark-submit command:

```<code>--conf "spark.driver.extraJavaOptions=-Djavax.net.ssl.sessionCacheSize=1000 -Djavax.net.ssl.sessionCacheTimeout=60" --conf "spark.executor.extraJavaOptions=-Djavax.net.ssl.sessionCacheSize=1000 -Djavax.net.ssl.sessionCacheTimeout=60"
</code>```

Still doesn't work. Does anyone know what's going on?

答:
TLDR: The property you need to set is fs.s3.maxConnections in the emrfs-site.xml configuration file. It defaults to 50. We were getting exactly the same error/stack trace as you, so I set it to 5000, which fixed the problem and had no ill effects.

From what I can tell, the root cause is InputFormat implementations that do not properly use try...finally to ensure that connections get closed when an exceptions are thrown. Notably, older versions of Hive, including v1.2.1 that Spark is compiled against, exhibit this bug. Hive 2.x massively refactors OrcInputFormat, though I haven't verified that the bug is fixed, nor do I know if/when/how you can compile Spark against Hive 2.x.

The workaround increases the size of the connection pool, as suggested in another answer, but both the property and its location are quite different than in the "classic" S3 filesystems (s3/s3a/s3n). Of course, this isn't documented anywhere and required decompilation of the emrfs jar to tease out...

Body:
I'm trying to submit a simple spark job in an Amazon EMR cluster. My cluster has 5 M4.2xlarge instances (1 master, 4 slaves), each with 16 vCPU, and 32 gigs of memory.

This is my code:

def main(args : Array[String]): Unit = {
 val sparkConfig = new SparkConf()
  .set("hive.exec.dynamic.partition", "true")
  .set("hive.exec.dynamic.partition.mode", "nonstrict")
  .set("hive.s3.max-client-retries", "50")
  .set("hive.s3.max-error-retries", "50")
  .set("hive.s3.max-connections", "100")
  .set("hive.s3.connect-timeout", "5m")
  .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
  .set("spark.kryo.registrationRequired", "true")
  .set("spark.kryo.classesToRegister", "org.apache.spark.graphx.impl.VertexAttributeBlock")
  .set("spark.broadcast.compress", "true")

 val spark = SparkSession.builder()
    .appName("Spark Hive Example")
    .enableHiveSupport()
    .config(sparkConfig)
    .getOrCreate()

// Set Kryo for serializing
GraphXUtils.registerKryoClasses(sparkConfig)
val res = spark.sql("SELECT col1, col2, col3 FROM table1 limit 10000")
val edgesRDD = res.rdd.map(row => Edge(row.getString(0).hashCode, row.getString(1).hashCode, row(2).asInstanceOf[String]))

val res_two = spark.sql("SELECT col1 FROM table2 where col1 is not NULL and col1 != '' limit 100000")
val vertexRDD: RDD[(VertexId, String)] = res_two.rdd.map(row => (row.getString(0).hashCode, row(0).asInstanceOf[String]))

val graph = Graph(vertexRDD, edgesRDD)

val connectedComponents = graph.connectedComponents().vertices

Both table1, and table2 are S3 backed external tables on hive. When I run this program, my job fails with the following error:

Job aborted due to stage failure: Task 827 in stage 0.0 failed 4 times, most recent failure: Lost task 827.3 in stage 0.0 (TID 921, xxx.internal, executor 3): com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1069)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1035)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4169)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4116)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1237)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.call.GetObjectMetadataCall.perform(GetObjectMetadataCall.java:24)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.call.GetObjectMetadataCall.perform(GetObjectMetadataCall.java:10)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor.execute(GlobalS3Executor.java:82)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:176)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.getObjectMetadata(AmazonS3LiteClient.java:94)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.AbstractAmazonS3Lite.getObjectMetadata(AbstractAmazonS3Lite.java:39)
    at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:211)
    at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy35.retrieveMetadata(Unknown Source)
    at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.getFileStatus(S3NativeFileSystem.java:768)
    at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.open(S3NativeFileSystem.java:1194)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:773)
    at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.open(EmrFileSystem.java:166)
    at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:355)
    at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:316)
    at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:237)
    at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1204)
    at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:246)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:245)
    at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:203)
    at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:94)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
    at com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:286)
    at com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
    at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.conn.$Proxy37.get(Unknown Source)
    at com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
    at com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
    at com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1190)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
    ... 59 more

Not sure if it is coming from hadoop or when reading from hive, but I saw a similar issue here, so I added the following params in my spark-submit command:

--conf "spark.driver.extraJavaOptions=-Djavax.net.ssl.sessionCacheSize=1000 -Djavax.net.ssl.sessionCacheTimeout=60" --conf "spark.executor.extraJavaOptions=-Djavax.net.ssl.sessionCacheSize=1000 -Djavax.net.ssl.sessionCacheTimeout=60"

Still doesn't work. Does anyone know what's going on?