问:
Retain only unique/distinct columns for each row of an input table
I have a very large data frame (nrow=~273,000) which I've subset as an example below: Each row is a protein name(s) and has various numbers of columns that lists the subcellular structures in which they can be found in human cells. 1) I would like to remove duplicate entries for each row and am struggling with this (code below). 2) I would then like to be able to count how many columns (subcellular structures) each gene can be found in.

Background: I got this data from Uniprot and cleaned it up as best as I could using regex but there are still some cases where there are rows with duplicate entries (e.g. FMR1 lists Chromosome 2x, Cytoplasm 3x and Plasma Membrane 2x - furthermore there are some blank columns in between them)

```<code>dput(df1)
structure(list(FMR1 = structure(c(41L, 3L, 17L, 63L, 16L, 24L,
35L, 33L, 52L, 6L, 49L, 5L, 71L, 72L, 42L, 58L, 22L, 20L, 19L,
80L, 9L, 51L, 66L, 64L, 23L, 14L, 60L, 45L, 28L, 54L, 7L, 30L,
29L, 44L, 53L, 8L, 69L, 79L, 10L, 11L, 26L, 37L, 39L, 40L, 82L,
73L, 18L, 21L, 27L, 47L, 4L, 46L, 1L, 13L, 36L, 70L, 74L, 67L,
78L, 77L, 61L, 62L, 31L, 56L, 34L, 57L, 25L, 81L, 75L, 59L, 2L,
65L, 55L, 38L, 50L, 68L, 32L, 12L, 43L, 15L, 48L, 76L), .Label = c("AAMP",
"ADCY10 SAC", "AIMP1 EMAP2 SCYE1", "ANTXR2 CMG2", "APBB1 FE65 RIR",
"APC DP2", "APLP1", "ARHGAP26 GRAF KIAA0621 OPHN1L", "ARL4A ARL4",
"ATP6V0D1 ATP6D VPATPD", "ATP6V1D ATP6M VATD", "AZIN2 ADC KIAA1945 ODCP",
"CACNB2 CACNLB2 MYSB", "CAMK2D CAMKD", "CDCA8 PESCRG3", "CDK1 CDC2 CDC28A CDKN1 P34CDC2",
"CEMIP KIAA1199", "CIB1 CIB KIP PRKDCIP", "CLTA", "CLTB", "CMTM8 CKLFSF8",
"DMD", "DSP", "ECT2", "EHD2 PAST2", "ENTPD2 CD39L1", "ERBB2 HER2 MLN19 NEU NGL",
"EVPL", "FCHO1 KIAA0290", "FCHO2", "FGR SRC2", "GPER1 CEPR CMKRL2 DRY12 GPER GPR30",
"HDAC6 KIAA0901 JM21", "ITCH", "ITGB1BP1 ICAP1", "KCTD7", "KIFC3",
"MFN1", "MISP C19orf21", "MYOT TTID", "NGDN C14orf120", "NISCH IRAS KIAA0975",
"NR1D1 EAR1 HREV THRAL", "PGM5 PGMRP", "PKP4", "PLA2G6 PLPLA9",
"PNKD KIAA1184 MR1 TAHCCP2 FKSG19 UNQ2491/PRO5778", "POP7 RPP20",
"PPL KIAA0568", "PRDX3 AOP1", "PTOV1 ACID2 PP642 UNQ6127/PRO20092",
"PTPN23 KIAA1471", "PTPRE", "PTPRR ECPTP PTPRQ", "RAB13 GIG4",
"RAB23 HSPC137", "RAB29 RAB7L1", "RAB30", "RAB38", "RAB40AL RLGP",
"RAB8A MEL RAB8", "RAB9A RAB9", "RACGAP1 KIAA1478 MGCRACGAP",
"RAP1B OK/SW-cl", "RGS8", "RPSA LAMBR LAMR1", "SGIP1", "SHMT2",
"SHROOM3 KIAA1481 SHRML MSTP013", "SLC28A3 CNT3", "SNTA1 SNT1",
"SNTB1 SNT2B1", "SNX11", "SNX12", "STOM BND7 EPB72", "TEX10 L18 Nbla10363",
"TNFRSF8 CD30 D1S166E", "TNS4 CTEN PP14434", "TRIM72 MG53", "USP6 HRP1 TRE2",
"VCL", "YES1 YES"), class = "factor"), Nucleus = structure(c(3L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L,
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L,
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L,
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 5L, 5L, 5L, 5L, 5L,
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L,
2L), .Label = c("Mitochondrion  ", "Nucleus", "Nucleus  ", "Plasma membrane",
"Plasma membrane  "), class = "factor"), Chromosome = structure(c(1L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L,
3L), .Label = c("Chromosome", "Cytoplasm", "Cytoplasm  "), class = "factor"),
    Chromosome.1 = structure(c(4L, 5L, 7L, 5L, 14L, 12L, 20L,
    18L, 5L, 20L, 20L, 2L, 1L, 1L, 8L, 10L, 19L, 1L, 1L, 8L,
    16L, 16L, 17L, 19L, 20L, 21L, 15L, 13L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 9L, 10L,
    16L, 16L, 16L, 22L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 11L,
    7L, 14L, 9L, 17L, 11L, 9L, 2L, 6L, 6L, 17L, 18L, 10L, 1L,
    1L, 17L, 19L, 19L, 1L, 3L, 5L, 1L), .Label = c("", " ", "Chromosome",
    "Cytoplasm  ", "Cytoplasmic vesicle", "Cytoplasmic vesicle  ",
    "Endoplasmic reticulum", "Endosome", "Endosome  ", "Golgi apparatus",
    "Golgi apparatus  ", "Midbody", "Midbody  ", "Mitochondrion",
    "Mitochondrion  ", "Nucleus", "Nucleus  ", "Perikaryon  ",
    "Plasma membrane", "Plasma membrane  ", "Sarcoplasmic reticulum  ",
    "Secreted"), class = "factor"), Cytoplasm = structure(c(1L,
    15L, 13L, 10L, 1L, 13L, 1L, 1L, 5L, 2L, 11L, 1L, 1L, 1L,
    5L, 8L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 1L, 14L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 5L, 9L, 2L, 3L, 6L, 7L, 2L, 1L, 2L, 4L, 11L, 12L,
    5L, 1L, 1L, 1L, 7L, 3L, 1L, 2L, 2L, 2L), .Label = c("", " ",
    "Cytoplasmic vesicle", "Endoplasmic reticulum", "Endosome",
    "Endosome  ", "Golgi apparatus", "Golgi apparatus  ", "Golgi appartus",
    "Midbody", "Mitochondrion  ", "Nucleus  ", "Plasma membrane",
    "Plasma membrane  ", "Secreted  "), class = "factor"), Cytoplasm.1 = structure(c(1L,
    4L, 7L, 7L, 1L, 1L, 1L, 1L, 5L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    6L, 3L, 2L, 1L, 1L, 1L), .Label = c("", " ", "Endoplasmic reticulum",
    "Endoplasmic reticulum  ", "Endosome", "Mitochondrion", "Plasma membrane"
    ), class = "factor"), Cytoplasmic.vesicle = structure(c(1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 2L, 1L, 1L, 1L, 1L), .Label = c("", "Golgi apparatus"
    ), class = "factor"), Perikaryon = structure(c(2L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 3L,
    1L, 1L, 1L, 1L), .Label = c("", " ", "Golgi apparatus"), class = "factor"),
    X = structure(c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 2L, 1L, 1L, 1L, 1L), .Label = c("",
    "Cytoplasmic granule"), class = "factor"), X.1 = structure(c(1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 2L, 1L, 1L, 1L, 1L), .Label = c("", "Perikaryon"), class = "factor"),
    X.2 = c(NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA), X.3 = c(NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA), Plasma.membrane = c(NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA), Plasma.membrane.1 = c(NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
    )), .Names = c("FMR1", "Nucleus", "Chromosome", "Chromosome.1",
"Cytoplasm", "Cytoplasm.1", "Cytoplasmic.vesicle", "Perikaryon",
"X", "X.1", "X.2", "X.3", "Plasma.membrane", "Plasma.membrane.1"
), class = "data.frame", row.names = c(NA, -82L))
</code>```

I've tried getting only unique columns for each row with no luck, for example:

```<code>unique(df1) # Original data with repeats removed
dplyr::distinct(df1) # Retain only unique/distinct rows from an input tb
</code>```

I think the problem is that the above functions are looking for row names which are identical which is not what I want. I want distinct columns for each row. I was thinking of using the melt function but since there is an odd number of columns for each row this won't work.

I would expect the output to look like this newDF

```<code>structure(list(FMR1 = structure(c(7L, 1L, 3L, 9L, 2L, 4L, 6L,
5L, 8L), .Label = c("AIMP1 EMAP2 SCYE1", "CDK1 CDC2 CDC28A CDKN1 P34CDC2",
"CEMIP KIAA1199", "ECT2", "HDAC6 KIAA0901 JM21", "ITGB1BP1 ICAP1",
"NGDN C14orf120", "PTPN23 KIAA1471", "RACGAP1 KIAA1478 MGCRACGAP"
), class = "factor"), Nucleus = structure(c(2L, 1L, 1L, 1L, 1L,
1L, 1L, 1L, 1L), .Label = c("Nucleus", "Nucleus  "), class = "factor"),
    Chromosome = structure(c(1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
    2L), .Label = c("Chromosome", "Cytoplasm"), class = "factor"),
    Cytoplasmic.vesicle = structure(c(1L, 8L, 2L, 4L, 5L, 4L,
    7L, 6L, 3L), .Label = c("Cytoplasm  ", "Endoplasmic reticulum",
    "Endosome", "Midbody", "Mitochondrion", "Perikaryon  ", "Plasma membrane  ",
    "Secreted  "), class = "factor"), Perikaryon = structure(c(1L,
    2L, 3L, 3L, 1L, 3L, 1L, 1L, 1L), .Label = c("", "Endoplasmic reticulum  ",
    "Plasma membrane"), class = "factor"), Plasma.membrane = c(NA,
    NA, NA, NA, NA, NA, NA, NA, NA)), .Names = c("FMR1", "Nucleus",
"Chromosome", "Cytoplasmic.vesicle", "Perikaryon", "Plasma.membrane"
), class = "data.frame", row.names = c(NA, -9L))
</code>```

From here I would like to get a rowSums(df1) so I was thinking of coercing each term to a number (e.g. Cytoplasmic vesicle=1, Nucleus=1, Endoplasmic reticiulum=1, etc.) but run into a problem on this dummy-dataset.

```<code>df2 <- as.numeric(newDF)
Error: (list) object cannot be coerced to type 'double'
df2 <- as.numeric(newDF[,2:n])
Error in 2:n : NA/NaN argument
</code>```

Thank you for your help.

EDIT

I would like to get a count for each row of how many unique columns in the newDF like so:

```<code>FMR1 5
NGDN C14orf120 3
AIMP1 EMAP2 SCYE1 4
CEMIP KIAA1199 4
RACGAP1 KIAA1478 MGCRACGAP 4
CDK1 CDC2 CDC28A CDKN1 P34CDC2 3
ECT2 4
ITGB1BP1 ICAP1 3
HDAC6 KIAA0901 JM21 3
PTPN23 KIAA1471 3
</code>```

答:
This may be one way to go. Since your expected result is a character vector, I cannot visualize the final output. Yet, you said you want to check how many columns each protein appears in in the data. I hope the outcome I have is what you are after.

First, I converted all columns to character. Then, I converted the data to long format one using gather(). For each subcellular structure group (i.e., subcellular), I added row indices (e.g., 1 means the 1st row in your original data), and trim white space. Then, remove any rows with NA in protein. Remove any rows with "" and " ". Now tidying up is done. For each row (i.e., row.index), remove duplicated protein types.
Ungroup the data, and finally count how many columns each protein appears (i.e., sucellular structure). Basically, you want to count how many times each protein appear in the data set by this time.

With your sample data, I got the following result. But I am not sure if this is what you want. (I am off to bed now. So I cannot help you for some hours. If anybody can jump in, please do so.)

```<code>mutate_all(mydf, as.character) %>%
gather(key = subcellular, value = protein) %>%
group_by(subcellular) %>%
mutate(row.index = 1:n(),
       protein = trimws(protein)) %>%
filter(!is.na(protein)) %>%
filter(!protein %in% c("", " ")) %>%
group_by(row.index) %>%
filter(!duplicated(protein)) %>%
ungroup %>%
count(protein, sort = TRUE)

#                  protein     n
#                   <chr> <int>
# 1             Cytoplasm    82
# 2       Plasma membrane    70
# 3               Nucleus    25
# 4              Endosome     9
# 5         Mitochondrion     9
# 6   Cytoplasmic vesicle     8
# 7       Golgi apparatus     7
# 8 Endoplasmic reticulum     5
# 9               Midbody     3
#10            Perikaryon     3
# ... with 87 more rows
</code>```

Given jjl"s comment, I did the following. Instead of counting how many columns each protein appears in, I counted how many protein names exist for each row.

```<code>mutate_all(mydf, as.character) %>%
gather(key = subcellular, value = protein) %>%
group_by(subcellular) %>%
mutate(row.index = 1:n(),
       protein = trimws(protein)) %>%
filter(!is.na(protein)) %>%
filter(!protein %in% c("", " ")) %>%
group_by(row.index) %>%
filter(!duplicated(protein)) %>%
ungroup %>%
count(row.index)

#   row.index     n
#       <int> <int>
# 1         1     4
# 2         2     6
# 3         3     5
# 4         4     6
# 5         5     4
# 6         6     5
# 7         7     4
# 8         8     4
# 9         9     5
#10        10     3
# ... with 72 more rows
</code>```

EDIT

If you want to remove the 1st column (i.e, FMR1), you can do that by filtering that column. I added filter(subcellular != "FMR1") to my code before I used count() at the end.

```<code>mutate_all(mydf, as.character) %>%
gather(key = subcellular, value = protein) %>%
group_by(subcellular) %>%
mutate(row.index = 1:n(),
       protein = trimws(protein)) %>%
filter(!is.na(protein)) %>%
filter(!protein %in% c("", " ")) %>%
group_by(row.index) %>%
filter(!duplicated(protein)) %>%
ungroup %>%
filter(subcellular != "FMR1") %>%
count(row.index)

# A tibble: 9 x 2
#  row.index     n
#      <int> <int>
#1         1     3
#2         2     4
#3         3     4
#4         4     4
#5         5     3
#6         6     4
#7         7     3
#8         8     3
#9         9     3
</code>```

Body:
I have a very large data frame (nrow=~273,000) which I've subset as an example below: Each row is a protein name(s) and has various numbers of columns that lists the subcellular structures in which they can be found in human cells. 1) I would like to remove duplicate entries for each row and am struggling with this (code below). 2) I would then like to be able to count how many columns (subcellular structures) each gene can be found in.

Background: I got this data from Uniprot and cleaned it up as best as I could using regex but there are still some cases where there are rows with duplicate entries (e.g. FMR1 lists Chromosome 2x, Cytoplasm 3x and Plasma Membrane 2x - furthermore there are some blank columns in between them)

dput(df1)
structure(list(FMR1 = structure(c(41L, 3L, 17L, 63L, 16L, 24L,
35L, 33L, 52L, 6L, 49L, 5L, 71L, 72L, 42L, 58L, 22L, 20L, 19L,
80L, 9L, 51L, 66L, 64L, 23L, 14L, 60L, 45L, 28L, 54L, 7L, 30L,
29L, 44L, 53L, 8L, 69L, 79L, 10L, 11L, 26L, 37L, 39L, 40L, 82L,
73L, 18L, 21L, 27L, 47L, 4L, 46L, 1L, 13L, 36L, 70L, 74L, 67L,
78L, 77L, 61L, 62L, 31L, 56L, 34L, 57L, 25L, 81L, 75L, 59L, 2L,
65L, 55L, 38L, 50L, 68L, 32L, 12L, 43L, 15L, 48L, 76L), .Label = c("AAMP",
"ADCY10 SAC", "AIMP1 EMAP2 SCYE1", "ANTXR2 CMG2", "APBB1 FE65 RIR",
"APC DP2", "APLP1", "ARHGAP26 GRAF KIAA0621 OPHN1L", "ARL4A ARL4",
"ATP6V0D1 ATP6D VPATPD", "ATP6V1D ATP6M VATD", "AZIN2 ADC KIAA1945 ODCP",
"CACNB2 CACNLB2 MYSB", "CAMK2D CAMKD", "CDCA8 PESCRG3", "CDK1 CDC2 CDC28A CDKN1 P34CDC2",
"CEMIP KIAA1199", "CIB1 CIB KIP PRKDCIP", "CLTA", "CLTB", "CMTM8 CKLFSF8",
"DMD", "DSP", "ECT2", "EHD2 PAST2", "ENTPD2 CD39L1", "ERBB2 HER2 MLN19 NEU NGL",
"EVPL", "FCHO1 KIAA0290", "FCHO2", "FGR SRC2", "GPER1 CEPR CMKRL2 DRY12 GPER GPR30",
"HDAC6 KIAA0901 JM21", "ITCH", "ITGB1BP1 ICAP1", "KCTD7", "KIFC3",
"MFN1", "MISP C19orf21", "MYOT TTID", "NGDN C14orf120", "NISCH IRAS KIAA0975",
"NR1D1 EAR1 HREV THRAL", "PGM5 PGMRP", "PKP4", "PLA2G6 PLPLA9",
"PNKD KIAA1184 MR1 TAHCCP2 FKSG19 UNQ2491/PRO5778", "POP7 RPP20",
"PPL KIAA0568", "PRDX3 AOP1", "PTOV1 ACID2 PP642 UNQ6127/PRO20092",
"PTPN23 KIAA1471", "PTPRE", "PTPRR ECPTP PTPRQ", "RAB13 GIG4",
"RAB23 HSPC137", "RAB29 RAB7L1", "RAB30", "RAB38", "RAB40AL RLGP",
"RAB8A MEL RAB8", "RAB9A RAB9", "RACGAP1 KIAA1478 MGCRACGAP",
"RAP1B OK/SW-cl", "RGS8", "RPSA LAMBR LAMR1", "SGIP1", "SHMT2",
"SHROOM3 KIAA1481 SHRML MSTP013", "SLC28A3 CNT3", "SNTA1 SNT1",
"SNTB1 SNT2B1", "SNX11", "SNX12", "STOM BND7 EPB72", "TEX10 L18 Nbla10363",
"TNFRSF8 CD30 D1S166E", "TNS4 CTEN PP14434", "TRIM72 MG53", "USP6 HRP1 TRE2",
"VCL", "YES1 YES"), class = "factor"), Nucleus = structure(c(3L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L,
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L,
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L,
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 5L, 5L, 5L, 5L, 5L,
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L,
2L), .Label = c("Mitochondrion  ", "Nucleus", "Nucleus  ", "Plasma membrane",
"Plasma membrane  "), class = "factor"), Chromosome = structure(c(1L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L,
3L), .Label = c("Chromosome", "Cytoplasm", "Cytoplasm  "), class = "factor"),
    Chromosome.1 = structure(c(4L, 5L, 7L, 5L, 14L, 12L, 20L,
    18L, 5L, 20L, 20L, 2L, 1L, 1L, 8L, 10L, 19L, 1L, 1L, 8L,
    16L, 16L, 17L, 19L, 20L, 21L, 15L, 13L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 9L, 10L,
    16L, 16L, 16L, 22L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 11L,
    7L, 14L, 9L, 17L, 11L, 9L, 2L, 6L, 6L, 17L, 18L, 10L, 1L,
    1L, 17L, 19L, 19L, 1L, 3L, 5L, 1L), .Label = c("", " ", "Chromosome",
    "Cytoplasm  ", "Cytoplasmic vesicle", "Cytoplasmic vesicle  ",
    "Endoplasmic reticulum", "Endosome", "Endosome  ", "Golgi apparatus",
    "Golgi apparatus  ", "Midbody", "Midbody  ", "Mitochondrion",
    "Mitochondrion  ", "Nucleus", "Nucleus  ", "Perikaryon  ",
    "Plasma membrane", "Plasma membrane  ", "Sarcoplasmic reticulum  ",
    "Secreted"), class = "factor"), Cytoplasm = structure(c(1L,
    15L, 13L, 10L, 1L, 13L, 1L, 1L, 5L, 2L, 11L, 1L, 1L, 1L,
    5L, 8L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 1L, 14L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 5L, 9L, 2L, 3L, 6L, 7L, 2L, 1L, 2L, 4L, 11L, 12L,
    5L, 1L, 1L, 1L, 7L, 3L, 1L, 2L, 2L, 2L), .Label = c("", " ",
    "Cytoplasmic vesicle", "Endoplasmic reticulum", "Endosome",
    "Endosome  ", "Golgi apparatus", "Golgi apparatus  ", "Golgi appartus",
    "Midbody", "Mitochondrion  ", "Nucleus  ", "Plasma membrane",
    "Plasma membrane  ", "Secreted  "), class = "factor"), Cytoplasm.1 = structure(c(1L,
    4L, 7L, 7L, 1L, 1L, 1L, 1L, 5L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    6L, 3L, 2L, 1L, 1L, 1L), .Label = c("", " ", "Endoplasmic reticulum",
    "Endoplasmic reticulum  ", "Endosome", "Mitochondrion", "Plasma membrane"
    ), class = "factor"), Cytoplasmic.vesicle = structure(c(1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 2L, 1L, 1L, 1L, 1L), .Label = c("", "Golgi apparatus"
    ), class = "factor"), Perikaryon = structure(c(2L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 3L,
    1L, 1L, 1L, 1L), .Label = c("", " ", "Golgi apparatus"), class = "factor"),
    X = structure(c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 2L, 1L, 1L, 1L, 1L), .Label = c("",
    "Cytoplasmic granule"), class = "factor"), X.1 = structure(c(1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
    1L, 2L, 1L, 1L, 1L, 1L), .Label = c("", "Perikaryon"), class = "factor"),
    X.2 = c(NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA), X.3 = c(NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA), Plasma.membrane = c(NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA), Plasma.membrane.1 = c(NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
    )), .Names = c("FMR1", "Nucleus", "Chromosome", "Chromosome.1",
"Cytoplasm", "Cytoplasm.1", "Cytoplasmic.vesicle", "Perikaryon",
"X", "X.1", "X.2", "X.3", "Plasma.membrane", "Plasma.membrane.1"
), class = "data.frame", row.names = c(NA, -82L))

I've tried getting only unique columns for each row with no luck, for example:

unique(df1) # Original data with repeats removed
dplyr::distinct(df1) # Retain only unique/distinct rows from an input tb

I think the problem is that the above functions are looking for row names which are identical which is not what I want. I want distinct columns for each row. I was thinking of using the melt function but since there is an odd number of columns for each row this won't work.

I would expect the output to look like this newDF

structure(list(FMR1 = structure(c(7L, 1L, 3L, 9L, 2L, 4L, 6L,
5L, 8L), .Label = c("AIMP1 EMAP2 SCYE1", "CDK1 CDC2 CDC28A CDKN1 P34CDC2",
"CEMIP KIAA1199", "ECT2", "HDAC6 KIAA0901 JM21", "ITGB1BP1 ICAP1",
"NGDN C14orf120", "PTPN23 KIAA1471", "RACGAP1 KIAA1478 MGCRACGAP"
), class = "factor"), Nucleus = structure(c(2L, 1L, 1L, 1L, 1L,
1L, 1L, 1L, 1L), .Label = c("Nucleus", "Nucleus  "), class = "factor"),
    Chromosome = structure(c(1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
    2L), .Label = c("Chromosome", "Cytoplasm"), class = "factor"),
    Cytoplasmic.vesicle = structure(c(1L, 8L, 2L, 4L, 5L, 4L,
    7L, 6L, 3L), .Label = c("Cytoplasm  ", "Endoplasmic reticulum",
    "Endosome", "Midbody", "Mitochondrion", "Perikaryon  ", "Plasma membrane  ",
    "Secreted  "), class = "factor"), Perikaryon = structure(c(1L,
    2L, 3L, 3L, 1L, 3L, 1L, 1L, 1L), .Label = c("", "Endoplasmic reticulum  ",
    "Plasma membrane"), class = "factor"), Plasma.membrane = c(NA,
    NA, NA, NA, NA, NA, NA, NA, NA)), .Names = c("FMR1", "Nucleus",
"Chromosome", "Cytoplasmic.vesicle", "Perikaryon", "Plasma.membrane"
), class = "data.frame", row.names = c(NA, -9L))

From here I would like to get a rowSums(df1) so I was thinking of coercing each term to a number (e.g. Cytoplasmic vesicle=1, Nucleus=1, Endoplasmic reticiulum=1, etc.) but run into a problem on this dummy-dataset.

df2 <- as.numeric(newDF)
Error: (list) object cannot be coerced to type 'double'
df2 <- as.numeric(newDF[,2:n])
Error in 2:n : NA/NaN argument

Thank you for your help.

EDIT

I would like to get a count for each row of how many unique columns in the newDF like so:

FMR1 5
NGDN C14orf120 3
AIMP1 EMAP2 SCYE1 4
CEMIP KIAA1199 4
RACGAP1 KIAA1478 MGCRACGAP 4
CDK1 CDC2 CDC28A CDKN1 P34CDC2 3
ECT2 4
ITGB1BP1 ICAP1 3
HDAC6 KIAA0901 JM21 3
PTPN23 KIAA1471 3

Body:
This may be one way to go. Since your expected result is a character vector, I cannot visualize the final output. Yet, you said you want to check how many columns each protein appears in in the data. I hope the outcome I have is what you are after.

First, I converted all columns to character. Then, I converted the data to long format one using gather(). For each subcellular structure group (i.e., subcellular), I added row indices (e.g., 1 means the 1st row in your original data), and trim white space. Then, remove any rows with NA in protein. Remove any rows with "" and " ". Now tidying up is done. For each row (i.e., row.index), remove duplicated protein types.
Ungroup the data, and finally count how many columns each protein appears (i.e., sucellular structure). Basically, you want to count how many times each protein appear in the data set by this time.

With your sample data, I got the following result. But I am not sure if this is what you want. (I am off to bed now. So I cannot help you for some hours. If anybody can jump in, please do so.)

mutate_all(mydf, as.character) %>%
gather(key = subcellular, value = protein) %>%
group_by(subcellular) %>%
mutate(row.index = 1:n(),
       protein = trimws(protein)) %>%
filter(!is.na(protein)) %>%
filter(!protein %in% c("", " ")) %>%
group_by(row.index) %>%
filter(!duplicated(protein)) %>%
ungroup %>%
count(protein, sort = TRUE)

#                  protein     n
#                   <chr> <int>
# 1             Cytoplasm    82
# 2       Plasma membrane    70
# 3               Nucleus    25
# 4              Endosome     9
# 5         Mitochondrion     9
# 6   Cytoplasmic vesicle     8
# 7       Golgi apparatus     7
# 8 Endoplasmic reticulum     5
# 9               Midbody     3
#10            Perikaryon     3
# ... with 87 more rows

Given jjl"s comment, I did the following. Instead of counting how many columns each protein appears in, I counted how many protein names exist for each row.

mutate_all(mydf, as.character) %>%
gather(key = subcellular, value = protein) %>%
group_by(subcellular) %>%
mutate(row.index = 1:n(),
       protein = trimws(protein)) %>%
filter(!is.na(protein)) %>%
filter(!protein %in% c("", " ")) %>%
group_by(row.index) %>%
filter(!duplicated(protein)) %>%
ungroup %>%
count(row.index)

#   row.index     n
#       <int> <int>
# 1         1     4
# 2         2     6
# 3         3     5
# 4         4     6
# 5         5     4
# 6         6     5
# 7         7     4
# 8         8     4
# 9         9     5
#10        10     3
# ... with 72 more rows

EDIT

If you want to remove the 1st column (i.e, FMR1), you can do that by filtering that column. I added filter(subcellular != "FMR1") to my code before I used count() at the end.

mutate_all(mydf, as.character) %>%
gather(key = subcellular, value = protein) %>%
group_by(subcellular) %>%
mutate(row.index = 1:n(),
       protein = trimws(protein)) %>%
filter(!is.na(protein)) %>%
filter(!protein %in% c("", " ")) %>%
group_by(row.index) %>%
filter(!duplicated(protein)) %>%
ungroup %>%
filter(subcellular != "FMR1") %>%
count(row.index)

# A tibble: 9 x 2
#  row.index     n
#      <int> <int>
#1         1     3
#2         2     4
#3         3     4
#4         4     4
#5         5     3
#6         6     4
#7         7     3
#8         8     3
#9         9     3


来源: stackexchange

问:
PERL How to skip error in while loop
I have this script which automates a process of like on Instagram.
It does exactly what it should do except for one thing. Within the while loop, if a user does not exist the script dies.
In this case, the error should be ignored and the script should continue processing the other users of the list.

This is the error i get:

Error GETing https://www.instagram.com/usernamethatdoesntexist/?__
a=1: Not Found at instagram-post-liker.pl line 29

Thanks for any help

```<code>use WWW::Mechanize;
use JSON;

my $filename = 'data.txt';
open(my $fh, '<:encoding(UTF-8)', $filename)
  or die "Could not open file '$filename' $!";

my $mech_get = WWW::Mechanize->new(agent => 'Mozilla/5.0 (Windows NT 6.0; rv:2.0) Gecko/20100101 Firefox/4.0 Opera 12.14');

my $mecho = WWW::Mechanize->new(agent => 'Mozilla/5.0 (Linux; U; Android 2.2; de-de; HTC Desire HD 1.18.161.2 Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1');

$mecho->get('https://www.instagram.com/');

$mecho->add_header('Accept-Encoding' => 'gzip, deflate', 'X-Instagram-AJAX' => '1', 'Accept-Language' => 'de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4,pl;q=0.2,nl;q=0.2' ,'X-Requested-With' => 'XMLHttpRequest', 'X-CSRFToken' => getcsrf($mecho));

my $post_data = {
    username    =>  'username',
    password    =>  'password'
};

$mecho->post('https://www.instagram.com/accounts/login/ajax/', $post_data);

  while (my $row = <$fh>) {
  chomp $row;
  print "getting user $row\n";

    $mech_get->get('https://www.instagram.com/'.$row.'/?__a=1');

    my $decoded_json = decode_json( $mech_get->response()->decoded_content() );

     print "Found Post \n";

    $mech_get->get('https://api.instagram.com/oembed/?url=https://www.instagram.com/p/'.$decoded_json->{user}->{media}->{nodes}[0]->{code}.'/');

    my $decoded_json2 = decode_json( $mech_get->response()->decoded_content() );

    print "Linking Post by Id";
    my $post;

    if($decoded_json2->{media_id} =~ /^([^_]*);*)   {
        $post = $1;
    }

    $mecho->add_header('Accept-Encoding' => 'gzip, deflate', 'X-Instagram-AJAX' => '1', 'Accept-Language' => 'de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4,pl;q=0.2,nl;q=0.2' ,'X-Requested-With' => 'XMLHttpRequest', 'X-CSRFToken' => getcsrf($mecho));

    $mecho->post('https://www.instagram.com/web/likes/'.$post.'/like/');

}

sub getcsrf {

    my ($somemech) = @_;

    my $cookies2  = $somemech->response->header('Set-Cookie');

    my $csrf_token2;
    if ($cookies2 =~ /csrftoken=([^;]+);/i){
        return $1;
    }
    else {
        die "Unable to find csrftoken in cookie headers, did something change?";

        }

};
</code>```

答:
In general, any piece of code that may fail may be placed within an eval block like this:

```<code>eval {
  # Piece of code that may fail
};
if($@) {
  # In case you want to deal with the error; $@ contains the text describing the error
}
</code>```

The piece of code inside eval will be compiled, so you know that it contains no syntax errors. A run-time error will be ignored but you will be able to catch it within the if block.

ContentLicense: CC BY-SA 3.0

Title: PERL How to skip error in while loop

Body:
I have this script which automates a process of like on Instagram.
It does exactly what it should do except for one thing. Within the while loop, if a user does not exist the script dies.
In this case, the error should be ignored and the script should continue processing the other users of the list.

This is the error i get:

Error GETing https://www.instagram.com/usernamethatdoesntexist/?__
a=1: Not Found at instagram-post-liker.pl line 29

Thanks for any help

use WWW::Mechanize;
use JSON;

my $filename = 'data.txt';
open(my $fh, '<:encoding(UTF-8)', $filename)
  or die "Could not open file '$filename' $!";

my $mech_get = WWW::Mechanize->new(agent => 'Mozilla/5.0 (Windows NT 6.0; rv:2.0) Gecko/20100101 Firefox/4.0 Opera 12.14');

my $mecho = WWW::Mechanize->new(agent => 'Mozilla/5.0 (Linux; U; Android 2.2; de-de; HTC Desire HD 1.18.161.2 Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1');

$mecho->get('https://www.instagram.com/');

$mecho->add_header('Accept-Encoding' => 'gzip, deflate', 'X-Instagram-AJAX' => '1', 'Accept-Language' => 'de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4,pl;q=0.2,nl;q=0.2' ,'X-Requested-With' => 'XMLHttpRequest', 'X-CSRFToken' => getcsrf($mecho));

my $post_data = {
    username    =>  'username',
    password    =>  'password'
};

$mecho->post('https://www.instagram.com/accounts/login/ajax/', $post_data);

  while (my $row = <$fh>) {
  chomp $row;
  print "getting user $row\n";

    $mech_get->get('https://www.instagram.com/'.$row.'/?__a=1');

    my $decoded_json = decode_json( $mech_get->response()->decoded_content() );

     print "Found Post \n";

    $mech_get->get('https://api.instagram.com/oembed/?url=https://www.instagram.com/p/'.$decoded_json->{user}->{media}->{nodes}[0]->{code}.'/');

    my $decoded_json2 = decode_json( $mech_get->response()->decoded_content() );

    print "Linking Post by Id";
    my $post;

    if($decoded_json2->{media_id} =~ /^([^_]*);*)   {
        $post = $1;
    }

    $mecho->add_header('Accept-Encoding' => 'gzip, deflate', 'X-Instagram-AJAX' => '1', 'Accept-Language' => 'de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4,pl;q=0.2,nl;q=0.2' ,'X-Requested-With' => 'XMLHttpRequest', 'X-CSRFToken' => getcsrf($mecho));

    $mecho->post('https://www.instagram.com/web/likes/'.$post.'/like/');

}

sub getcsrf {

    my ($somemech) = @_;

    my $cookies2  = $somemech->response->header('Set-Cookie');

    my $csrf_token2;
    if ($cookies2 =~ /csrftoken=([^;]+);/i){
        return $1;
    }
    else {
        die "Unable to find csrftoken in cookie headers, did something change?";

        }

};

Body:
In general, any piece of code that may fail may be placed within an eval block like this:

eval {
  # Piece of code that may fail
};
if($@) {
  # In case you want to deal with the error; $@ contains the text describing the error
}

The piece of code inside eval will be compiled, so you know that it contains no syntax errors. A run-time error will be ignored but you will be able to catch it within the if block.


问:
tcl tablelist delete all rows with certain entry
Having a tablelist I want to delete all rows containing a string in the first column.

```<code>set Joints      [.dsm.nb.f11.jointData_f11 getcolumns cJoint  ]
set killrow 0
foreach Joint $Joints {
         if { $Joint eq "" } {
               .dsm.nb.f11.jointData_f11 delete $killrow $killrow
         }
              incr killrow
}
</code>```

The problem is that the row indexes of the table changes while rows get deleted, so that after the first delete command everything gets messd upp. What is the best way to deal with this Problem? Is there a build in feature i could use?

答:
The simplest way of making this work is to go through the list backwards. lreverse helps a lot.

```<code>set Joints [.dsm.nb.f11.jointData_f11 getcolumns cJoint]
set killrow 0
set rowsToKill {}
foreach Joint $Joints {
    if {$Joint eq ""} {
        lappend rowsToKill $killrow
    }
    incr killrow
}
foreach killrow [lreverse $rowsToKill] {
    .dsm.nb.f11.jointData_f11 delete $killrow $killrow
}
</code>```

Yes, you're iterating over the list a few times. It's still simpler to do it this way than to try to do it any other.

Body:
Having a tablelist I want to delete all rows containing a string in the first column.

set Joints      [.dsm.nb.f11.jointData_f11 getcolumns cJoint  ]
set killrow 0
foreach Joint $Joints {
         if { $Joint eq "" } {
               .dsm.nb.f11.jointData_f11 delete $killrow $killrow
         }
              incr killrow
}

The problem is that the row indexes of the table changes while rows get deleted, so that after the first delete command everything gets messd upp. What is the best way to deal with this Problem? Is there a build in feature i could use?

Body:
The simplest way of making this work is to go through the list backwards. lreverse helps a lot.

set Joints [.dsm.nb.f11.jointData_f11 getcolumns cJoint]
set killrow 0
set rowsToKill {}
foreach Joint $Joints {
    if {$Joint eq ""} {
        lappend rowsToKill $killrow
    }
    incr killrow
}
foreach killrow [lreverse $rowsToKill] {
    .dsm.nb.f11.jointData_f11 delete $killrow $killrow
}

Yes, you're iterating over the list a few times. It's still simpler to do it this way than to try to do it any other.


问:
Selecting specific part of an HTML text
I have a function that returns some HTML fragment that I store in a variable called data, with its whole structure. What I want is to extract from it some of those parts.

```<code><!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Title</title>
    <script id="hello">

    </script>
</head>
<body>

</body>
</html>
</code>```

For example, I want to get the body and save it in a new variable:

```<code>var body = data.split("<body")[1].split(">").slice(1).join(">").split("</body>")[0];
</code>```

Where data is the HTML text as a string that the original function is returning.

Is there any way I could save an specific script, from its ID (in this case with id = hello), and save it in another variable??

Thank you very much

答:
Let's suppose you have an HTML string in a variable, for example

```<code>var foo = '<body><span>bar</span></body>';
</code>```

Now, let's initialize a parser, to convert this into HTML:

```<code>var parser = new DOMParser();
var doc = parser.parseFromString(foo, "text/html");
</code>```

Now, you can read anything from foo, as it is converted into HTML:

```<code>document.getElementsByTagName("body")[0].innerHTML = doc.querySelectorAll("body")[0].innerHTML;
</code>```

Title: Selecting specific part of an HTML text

Body:
I have a function that returns some HTML fragment that I store in a variable called data, with its whole structure. What I want is to extract from it some of those parts.

<!DOCTYPE html>

    <meta charset="UTF-8">
    <title>Title</title>

For example, I want to get the body and save it in a new variable:

var body = data.split("").slice(1).join(">").split("")[0];

Where data is the HTML text as a string that the original function is returning.

Is there any way I could save an specific script, from its ID (in this case with id = hello), and save it in another variable??

Thank you very much

Body:
Let's suppose you have an HTML string in a variable, for example

var foo = '<body><span>bar</span></body>';

Now, let's initialize a parser, to convert this into HTML:

var parser = new DOMParser();
var doc = parser.parseFromString(foo, "text/html");

Now, you can read anything from foo, as it is converted into HTML:

document.getElementsByTagName("body")[0].innerHTML = doc.querySelectorAll("body")[0].innerHTML;


问:
How to read a .txt file for each int on a line & storing to array?

```<code>public class assignment2
{
    public static void main(String[] args) throws IOException
    {
        // TODO Auto-generated method stub
        FileInputStream fstream = new FileInputStream("assignment2.txt");
        BufferedReader br = new BufferedReader(new InputStreamReader(fstream));

        String strLine;

        //Read File Line By Line
        while ((strLine = br.readLine()) != null) {
            // Print the content on the console
            System.out.println (strLine);
            String[] numbers = strLine.split(" ");
            System.out.print(numbers[1]);
        }
        //Close the input stream
        br.close();
    }
}
</code>```

I expect that the code will print the String[] numbers as
1 0 10

I get this result 00371010

Please note that the input file is formatted as such:

1 0 10

2 0 9

3 3 5

4 7 4

5 10 6

6 10 7

答:
Just replace your line:

```<code>System.out.print(numbers[1]);
</code>```

With:

```<code>for (int i = 0; i < numbers.length; i++)
    System.out.print(numbers[i] + " ");

System.out.println();
</code>```

Let me know how it works.

Body:
public class assignment2
{
    public static void main(String[] args) throws IOException
    {
        // TODO Auto-generated method stub
        FileInputStream fstream = new FileInputStream("assignment2.txt");
        BufferedReader br = new BufferedReader(new InputStreamReader(fstream));

        String strLine;

        //Read File Line By Line
        while ((strLine = br.readLine()) != null) {
            // Print the content on the console
            System.out.println (strLine);
            String[] numbers = strLine.split(" ");
            System.out.print(numbers[1]);
        }
        //Close the input stream
        br.close();
    }
}

I expect that the code will print the String[] numbers as
1 0 10

I get this result 00371010

Please note that the input file is formatted as such:

1 0 10

2 0 9

3 3 5

4 7 4

5 10 6

6 10 7

Body:
Just replace your line:

System.out.print(numbers[1]);

With:

for (int i = 0; i < numbers.length; i++)
    System.out.print(numbers[i] + " ");

System.out.println();

Let me know how it works.


问:
Bash: how to automatically insert spacing characters according to indent?
I've got lots of debug statements making for unreadable stacktraces (not my call) like this:

```<code> 00:53:59,906  - j.util.indexing.FileBasedIndex - START INDEX SHUTDOWN
 00:53:09,192  - .impl.stores.XmlElementStorage - Document was not loaded for $APP_CONFIG$/macros.xml
 00:53:09,195  - s.impl.stores.FileBasedStorage - Document was not loaded for $APP_CONFIG$/quicklists.xml file is null
 00:53:09,195  - .impl.stores.XmlElementStorage - Document was not loaded for $APP_CONFIG$/quicklists.xml
 00:53:09,696  - ij.openapi.wm.impl.IdeRootPane - App initialization took 5584 ms
 00:53:11,677  -                  TestNG Runner - Create TestNG Template Configuration
 00:53:13,628  - indexing.UnindexedFilesUpdater - Unindexed files update started
 00:53:15,370  - indexing.UnindexedFilesUpdater - Unindexed files update done
 00:53:20,873  - tor.impl.FileEditorManagerImpl - Project opening took 10569 ms
 00:53:31,862  - s.impl.stores.FileBasedStorage - Document was not loaded for $APP_CONFIG$/intentionSettings.xml file is null
 00:53:31,862  - .impl.stores.XmlElementStorage - Document was not loaded for $APP_CONFIG$/intentionSettings.xml
 00:54:00,723  - j.util.indexing.FileBasedIndex - END INDEX SHUTDOWN
</code>```

And I want them to look like this:

```<code> 00:53:59,906  - j.util.indexing.FileBasedIndex - START INDEX SHUTDOWN
 00:53:09,192  - .impl.stores.XmlElementStorage -   Document was not loaded for $APP_CONFIG$/macros.xml
 00:53:09,195  - s.impl.stores.FileBasedStorage -   Document was not loaded for $APP_CONFIG$/quicklists.xml file is null
 00:53:09,195  - .impl.stores.XmlElementStorage -   Document was not loaded for $APP_CONFIG$/quicklists.xml
 00:53:09,696  - ij.openapi.wm.impl.IdeRootPane -     App initialization took 5584 ms
 00:53:11,677  -                  TestNG Runner -       Create TestNG Template Configuration
 00:53:13,628  - indexing.UnindexedFilesUpdater -        Unindexed files update started
 00:53:15,370  - indexing.UnindexedFilesUpdater -        Unindexed files update done
 00:53:20,873  - tor.impl.FileEditorManagerImpl -     Project opening took 10569 ms
 00:53:31,862  - s.impl.stores.FileBasedStorage -    Document was not loaded for $APP_CONFIG$/intentionSettings.xml file is null
 00:53:31,862  - .impl.stores.XmlElementStorage -   Document was not loaded for $APP_CONFIG$/intentionSettings.xml
 00:54:00,723  - j.util.indexing.FileBasedIndex - END INDEX SHUTDOWN
</code>```

Now my code is formatted automatically so I know exactly how many characters there are in my source code for indent level.

What I want to do is replace automatically every line in my code looking like:

```<code>log.debug("some silly debug line");
</code>```

with:

```<code>log.debug(" " + "some silly debug line");
</code>```

or:

```<code>log.debug("  " + "some silly debug line");
</code>```

depending on the number of indents before the log.debug line.

I'm pretty sure some Bash/shell magic can do this automatically.

Basically I'd start with a:

```<code>find . -iname "*java" -exec ...
</code>```

then I'd like to insert " " on lines starting with log.debug depending on the indent level of these lines.

How would I go about it?

EDIT

Damn: it needs to be able to run several times, without re-shifting the lines already containing log.debug(" " +).  It gets more complicated but now thanks to the good answer I've got something to start from :)

Note: before yelling and shooting as to why this ain't a good idea, read my comment(s)

答:
This should work when run multiple times on files:

```<code>sed -i.bak 's/^\([[:blank:]]*\)\(log.debug(\)\("[[:blank:]]*" + \)\?/\1\2"\1" + /; s/^log.debug("" + /log.debug(/'
</code>```

Combined with your find command:

```<code>find . -iname "*java" -exec sed -i.bak 's/^\([[:blank:]]*\)\(log.debug(\)\("[[:blank:]]*" + \)\?/\1\2"\1" + /; s/^log.debug("" + /log.debug(/' \;
</code>```

Test run:

```<code>$ sed -i.bak '...' test.java
$ diff test.java test.java.bak
(report of differences)
$ sed -i.bak '...' test.java
$ diff test.java test.java.bak
(no differences)
</code>```

Explanation of sed command:

```<code>s/^\([[:blank:]]*\)\(log.debug(\)\("[[:blank:]]*" + \)\?/\1\2"\1" + /
</code>```

Capture the tabs and spaces, if any, at the beginning of the line into group \1.

Capture the string "log.debug(" (only so we don't have to repeat it verbatim) into group \2.

Capture any existing indent string ("indentlevel" +) into so it can be overwritten in case the indentation has changed. The \? makes its existence optional. We don't care that it would be capture group \3 since we're throwing it away. The parentheses are necessary only for the \?.

Now the replacement is output. Capture groups \1 (the indent) and \2 (the "log.debug(") are put back like they were. And "indentlevel" + is added (where "indentlevel" is the same sequence of whitespace characters that indented the line). Note that this may result in nothing between the quotes.

The next part of the command removes the indentation stuff if it's null. You can omit that if you don't mind resulting lines of the form log.debug("" + "foo") when the line isn't indented.

```<code>s/^log.debug("" + /log.debug(/
</code>```

ContentLicense: CC BY-SA 2.5

Body:
I've got lots of debug statements making for unreadable stacktraces (not my call) like this:

 00:53:59,906  - j.util.indexing.FileBasedIndex - START INDEX SHUTDOWN
 00:53:09,192  - .impl.stores.XmlElementStorage - Document was not loaded for $APP_CONFIG$/macros.xml
 00:53:09,195  - s.impl.stores.FileBasedStorage - Document was not loaded for $APP_CONFIG$/quicklists.xml file is null
 00:53:09,195  - .impl.stores.XmlElementStorage - Document was not loaded for $APP_CONFIG$/quicklists.xml
 00:53:09,696  - ij.openapi.wm.impl.IdeRootPane - App initialization took 5584 ms
 00:53:11,677  -                  TestNG Runner - Create TestNG Template Configuration
 00:53:13,628  - indexing.UnindexedFilesUpdater - Unindexed files update started
 00:53:15,370  - indexing.UnindexedFilesUpdater - Unindexed files update done
 00:53:20,873  - tor.impl.FileEditorManagerImpl - Project opening took 10569 ms
 00:53:31,862  - s.impl.stores.FileBasedStorage - Document was not loaded for $APP_CONFIG$/intentionSettings.xml file is null
 00:53:31,862  - .impl.stores.XmlElementStorage - Document was not loaded for $APP_CONFIG$/intentionSettings.xml
 00:54:00,723  - j.util.indexing.FileBasedIndex - END INDEX SHUTDOWN

And I want them to look like this:

 00:53:59,906  - j.util.indexing.FileBasedIndex - START INDEX SHUTDOWN
 00:53:09,192  - .impl.stores.XmlElementStorage -   Document was not loaded for $APP_CONFIG$/macros.xml
 00:53:09,195  - s.impl.stores.FileBasedStorage -   Document was not loaded for $APP_CONFIG$/quicklists.xml file is null
 00:53:09,195  - .impl.stores.XmlElementStorage -   Document was not loaded for $APP_CONFIG$/quicklists.xml
 00:53:09,696  - ij.openapi.wm.impl.IdeRootPane -     App initialization took 5584 ms
 00:53:11,677  -                  TestNG Runner -       Create TestNG Template Configuration
 00:53:13,628  - indexing.UnindexedFilesUpdater -        Unindexed files update started
 00:53:15,370  - indexing.UnindexedFilesUpdater -        Unindexed files update done
 00:53:20,873  - tor.impl.FileEditorManagerImpl -     Project opening took 10569 ms
 00:53:31,862  - s.impl.stores.FileBasedStorage -    Document was not loaded for $APP_CONFIG$/intentionSettings.xml file is null
 00:53:31,862  - .impl.stores.XmlElementStorage -   Document was not loaded for $APP_CONFIG$/intentionSettings.xml
 00:54:00,723  - j.util.indexing.FileBasedIndex - END INDEX SHUTDOWN

Now my code is formatted automatically so I know exactly how many characters there are in my source code for indent level.

What I want to do is replace automatically every line in my code looking like:

log.debug("some silly debug line");

with:

log.debug(" " + "some silly debug line");

or:

log.debug("  " + "some silly debug line");

depending on the number of indents before the log.debug line.

I'm pretty sure some Bash/shell magic can do this automatically.

Basically I'd start with a:

find . -iname "*java" -exec ...

then I'd like to insert " " on lines starting with log.debug depending on the indent level of these lines.

How would I go about it?

EDIT

Damn: it needs to be able to run several times, without re-shifting the lines already containing log.debug(" " +).  It gets more complicated but now thanks to the good answer I've got something to start from :)

Note: before yelling and shooting as to why this ain't a good idea, read my comment(s)

Body:
This should work when run multiple times on files:

sed -i.bak 's/^\([[:blank:]]*\)\(log.debug(\)\("[[:blank:]]*" + \)\?/\1\2"\1" + /; s/^log.debug("" + /log.debug(/'

Combined with your find command:

find . -iname "*java" -exec sed -i.bak 's/^\([[:blank:]]*\)\(log.debug(\)\("[[:blank:]]*" + \)\?/\1\2"\1" + /; s/^log.debug("" + /log.debug(/' \;

Test run:

$ sed -i.bak '...' test.java
$ diff test.java test.java.bak
(report of differences)
$ sed -i.bak '...' test.java
$ diff test.java test.java.bak
(no differences)

Explanation of sed command:

s/^\([[:blank:]]*\)\(log.debug(\)\("[[:blank:]]*" + \)\?/\1\2"\1" + /

Capture the tabs and spaces, if any, at the beginning of the line into group \1.

Capture the string "log.debug(" (only so we don't have to repeat it verbatim) into group \2.

Capture any existing indent string ("indentlevel" +) into so it can be overwritten in case the indentation has changed. The \? makes its existence optional. We don't care that it would be capture group \3 since we're throwing it away. The parentheses are necessary only for the \?.

Now the replacement is output. Capture groups \1 (the indent) and \2 (the "log.debug(") are put back like they were. And "indentlevel" + is added (where "indentlevel" is the same sequence of whitespace characters that indented the line). Note that this may result in nothing between the quotes.

The next part of the command removes the indentation stuff if it's null. You can omit that if you don't mind resulting lines of the form log.debug("" + "foo") when the line isn't indented.

s/^log.debug("" + /log.debug(/


问:
Setting compiler options for intero in emacs
When I am developing I like to have -fdefer-type-errors and -XPartialTypeSignatures enabled. Currently I just add

```<code>{-# LANGUAGE PartialTypeSignatures #-}
{-# OPTIONS_GHC -fdefer-type-errors #-}
</code>```

to the top of the file I am working with. However this is not ideal as I often forget to remove these lines once I am done. Is there an option to pass these to intero in emacs?

答:
Updating your cabal file to specify the relevant extensions and ghc-options should work, i.e.:</p>

```<code>...
lib
  hs-source-dirs:      src
  ghc-options:         -fdefer-type-errors
  extensions:          PartialTypeSignatures
</code>```

Body:
When I am developing I like to have -fdefer-type-errors and -XPartialTypeSignatures enabled. Currently I just add

{-# LANGUAGE PartialTypeSignatures #-}
{-# OPTIONS_GHC -fdefer-type-errors #-}

to the top of the file I am working with. However this is not ideal as I often forget to remove these lines once I am done. Is there an option to pass these to intero in emacs?

Body:
Updating your cabal file to specify the relevant extensions and ghc-options should work, i.e.:

...
lib
  hs-source-dirs:      src
  ghc-options:         -fdefer-type-errors
  extensions:          PartialTypeSignatures


问:
Hide but not remove Image button CKEditor
In CKEditor, I added my custom image button, which directly trigger a file input. However, images can only be rendered when the Image plugin is in use.

I don't want to have 2 image buttons on the toolbar, is there a way to hide the Image button, but still use it (like display: none but in a more structural way?)

Thanks in advance.

答:
Since 'CKEditor 4.1' you have something which is called Advanced content filtering.
This allows you set enable or disable certain tags.

The easiest way to allow the images to be displayed is to add

```<code>config.allowedContent = true;
</code>```

in your config.js file. But this will allow everything.

To just add the enablement of the 'img' tag you can add it in the 'extraAllowedContent' element when you create the CKEditor

```<code>    var myEditor = CKEDITOR.replace(editorId, {
    extraAllowedContent : 'img(*){*}[*]'
    });
</code>```

Body:
Since 'CKEditor 4.1' you have something which is called Advanced content filtering.
This allows you set enable or disable certain tags.

The easiest way to allow the images to be displayed is to add

config.allowedContent = true;

in your config.js file. But this will allow everything.

To just add the enablement of the 'img' tag you can add it in the 'extraAllowedContent' element when you create the CKEditor

    var myEditor = CKEDITOR.replace(editorId, {
    extraAllowedContent : 'img(*){*}[*]'
    });


问:
.updateOne on MongoDB not working in Node.js
I have the following code:

```<code>connection((db) => {
            db.collection('orders')
                .updateOne(
                    { "_id": req.body._id}, // Filter
                    {"name": req.body.name} // Update
                )
                .then((obj) => {
                    console.log('Updated - ' + obj);
                    res.redirect('orders')
                })
                .catch((err) => {
                    console.log('Error: ' + err);
                })
        })
</code>```

I want to change the name in the order but it doesn't update it.
The result in the console is

Updated - {"n":0,"nModified":0,"ok":1}

I tried to read the documentation but it's horrific

EDIT: {$set: {"name": req.body.name}}, didn't work as well

EDIT 2: The passed ID matches the _id in the database. Could it be a problem that I'm querying for plain text ID while in the database it is referred to as "ObjectId('5a42ja...')"

答:
Use {$set: {"name": req.body.name}} (as Sparw mentioned) to update the the properties you want in the document. Also, if the id that you pass as a parameter does not exists in the collection (and you want to create one with the same id) you can pass as a third parameter {upsert: true} to create one.

In your example:

```<code>connection((db) => {
          db.collection('orders')
               .updateOne(
                  { "_id": req.body._id}, // Filter
                  {$set: {"name": req.body.name}}, // Update
                  {upsert: true} // add document with req.body._id if not exists

             )
            .then((obj) => {
               console.log('Updated - ' + obj);
              res.redirect('orders')
         })
        .catch((err) => {
           console.log('Error: ' + err);
      }) })
</code>```

Body:
I have the following code:

connection((db) => {
            db.collection('orders')
                .updateOne(
                    { "_id": req.body._id}, // Filter
                    {"name": req.body.name} // Update
                )
                .then((obj) => {
                    console.log('Updated - ' + obj);
                    res.redirect('orders')
                })
                .catch((err) => {
                    console.log('Error: ' + err);
                })
        })

I want to change the name in the order but it doesn't update it.
The result in the console is

Updated - {"n":0,"nModified":0,"ok":1}

I tried to read the documentation but it's horrific

EDIT: {$set: {"name": req.body.name}}, didn't work as well

EDIT 2: The passed ID matches the _id in the database. Could it be a problem that I'm querying for plain text ID while in the database it is referred to as "ObjectId('5a42ja...')"

Body:
Use {$set: {"name": req.body.name}} (as Sparw mentioned) to update the the properties you want in the document. Also, if the id that you pass as a parameter does not exists in the collection (and you want to create one with the same id) you can pass as a third parameter {upsert: true} to create one.

In your example:

connection((db) => {
          db.collection('orders')
               .updateOne(
                  { "_id": req.body._id}, // Filter
                  {$set: {"name": req.body.name}}, // Update
                  {upsert: true} // add document with req.body._id if not exists

             )
            .then((obj) => {
               console.log('Updated - ' + obj);
              res.redirect('orders')
         })
        .catch((err) => {
           console.log('Error: ' + err);
      }) })


问:
Coding unity shaders with unity visual studio
Recently I've been coding shaders for unity in visual studio and I've noticed that since unity shaders are written in a combination of unity's shader lab language and CG, visual studio 2015 doesn't seem to recognize the languages. Because of this, visual studio is not reconizing keywords, has no intellesense, and worse of all tabbing incorrectly whenever I press enter to go to a new line. I've tried this visual studio extension https://visualstudiogallery.msdn.microsoft.com/ed812631-a7d3-4ca3-9f84-7efb240c7bb5 and it doesn't seem to fully work. I was wondering if anyone on here has had experience working with shaders and has an extension they know about to fix this problem.</p>

答:
i bassicly used this for shaders:

http://wiki.unity3d.com/index.php/Silhouette-Outlined_Diffuse

the part that says "Outline only"

with the description: "The thing that does the trick here is "Blend Zero One" which is to completely forego rendering our object and use only the destination color (i.e. whatever is behind the object). In effect, the object itself is invisible, but we still let the outline render itself. So that's what we're left with: only the outline."

You first need to make a shader script and place it somewhere it suits you i always place this into the folder shaders.

the code is bassicly on the site but to make it easier for you i will paste it in here. be sure to read the code because you can edit this pretty easily from the code or inspector of Unity.
hereby the code for the created shaderscript:

```<code>Shader "Outlined/Silhouette Only" {
    Properties {
        _OutlineColor ("Outline Color", Color) = (0,0,0,1)
        _Outline ("Outline width", Range (0.0, 0.03)) = .005
    }

CGINCLUDE
#include "UnityCG.cginc"

struct appdata {
    float4 vertex : POSITION;
    float3 normal : NORMAL;
};

struct v2f {
    float4 pos : POSITION;
    float4 color : COLOR;
};

uniform float _Outline;
uniform float4 _OutlineColor;

v2f vert(appdata v) {
    // just make a copy of incoming vertex data but scaled according to normal direction
    v2f o;
    o.pos = mul(UNITY_MATRIX_MVP, v.vertex);

    float3 norm   = mul ((float3x3)UNITY_MATRIX_IT_MV, v.normal);
    float2 offset = TransformViewToProjection(norm.xy);

    o.pos.xy += offset * o.pos.z * _Outline;
    o.color = _OutlineColor;
    return o;
}
ENDCG

    SubShader {
        Tags { "Queue" = "Transparent" }

        Pass {
            Name "BASE"
            Cull Back
            Blend Zero One

            // uncomment this to hide inner details:
            //Offset -8, -8

            SetTexture [_OutlineColor] {
                ConstantColor (0,0,0,0)
                Combine constant
            }
        }

        // note that a vertex shader is specified here but its using the one above
        Pass {
            Name "OUTLINE"
            Tags { "LightMode" = "Always" }
            Cull Front

            // you can choose what kind of blending mode you want for the outline
            //Blend SrcAlpha OneMinusSrcAlpha // Normal
            //Blend One One // Additive
            Blend One OneMinusDstColor // Soft Additive
            //Blend DstColor Zero // Multiplicative
            //Blend DstColor SrcColor // 2x Multiplicative

CGPROGRAM
#pragma vertex vert
#pragma fragment frag

half4 frag(v2f i) :COLOR {
    return i.color;
}
ENDCG
        }

    }

    Fallback "Diffuse"
}
</code>```

P.S. can you tell me what kind of shaders you're aiming to make?

Body:
i bassicly used this for shaders:

http://wiki.unity3d.com/index.php/Silhouette-Outlined_Diffuse

the part that says "Outline only"

with the description: "The thing that does the trick here is "Blend Zero One" which is to completely forego rendering our object and use only the destination color (i.e. whatever is behind the object). In effect, the object itself is invisible, but we still let the outline render itself. So that's what we're left with: only the outline."

You first need to make a shader script and place it somewhere it suits you i always place this into the folder shaders.

the code is bassicly on the site but to make it easier for you i will paste it in here. be sure to read the code because you can edit this pretty easily from the code or inspector of Unity.
hereby the code for the created shaderscript:

Shader "Outlined/Silhouette Only" {
    Properties {
        _OutlineColor ("Outline Color", Color) = (0,0,0,1)
        _Outline ("Outline width", Range (0.0, 0.03)) = .005
    }

CGINCLUDE
#include "UnityCG.cginc"

struct appdata {
    float4 vertex : POSITION;
    float3 normal : NORMAL;
};

struct v2f {
    float4 pos : POSITION;
    float4 color : COLOR;
};

uniform float _Outline;
uniform float4 _OutlineColor;

v2f vert(appdata v) {
    // just make a copy of incoming vertex data but scaled according to normal direction
    v2f o;
    o.pos = mul(UNITY_MATRIX_MVP, v.vertex);

    float3 norm   = mul ((float3x3)UNITY_MATRIX_IT_MV, v.normal);
    float2 offset = TransformViewToProjection(norm.xy);

    o.pos.xy += offset * o.pos.z * _Outline;
    o.color = _OutlineColor;
    return o;
}
ENDCG

    SubShader {
        Tags { "Queue" = "Transparent" }

        Pass {
            Name "BASE"
            Cull Back
            Blend Zero One

            // uncomment this to hide inner details:
            //Offset -8, -8

            SetTexture [_OutlineColor] {
                ConstantColor (0,0,0,0)
                Combine constant
            }
        }

        // note that a vertex shader is specified here but its using the one above
        Pass {
            Name "OUTLINE"
            Tags { "LightMode" = "Always" }
            Cull Front

            // you can choose what kind of blending mode you want for the outline
            //Blend SrcAlpha OneMinusSrcAlpha // Normal
            //Blend One One // Additive
            Blend One OneMinusDstColor // Soft Additive
            //Blend DstColor Zero // Multiplicative
            //Blend DstColor SrcColor // 2x Multiplicative

CGPROGRAM
#pragma vertex vert
#pragma fragment frag

half4 frag(v2f i) :COLOR {
    return i.color;
}
ENDCG
        }

    }

    Fallback "Diffuse"
}

P.S. can you tell me what kind of shaders you're aiming to make?


问:
moving colorbar with gridspec
I am making a plot with multiple figures and corresponding colorbars. This page http://www.sc.eso.org/~bdias/pycoffee/codes/20160407/gridspec_demo.html claims to know the best way of doing so. I am inclined to believe them.

However, I have run into issues adressing the kwags of the colorbar when I do it their way:

```<code>from matplotlib.colorbar import Colorbar
import matplotlib.gridspec as gridspec
import matplotlib.pyplot as plt
import numpy as np
median = np.zeros((100,100))
map0 = np.zeros((100,100))
map1 =  map0
map2 =  map0

fig = plt.figure()
plt.tight_layout()
gs = gridspec.GridSpec(2,6)

ax  = plt.subplot(gs[0,0])
cbax = plt.subplot(gs[0,1])

ax1 = plt.subplot(gs[0,2])
cbax1 = plt.subplot(gs[0,3])

ax2 = plt.subplot(gs[0,4])
cbax2 = plt.subplot(gs[0,5])

ax3 = plt.subplot(gs[1,0])
cbax3 = plt.subplot(gs[1,1])

ax4 = plt.subplot(gs[1,2])
cbax4 = plt.subplot(gs[1,3])

ax5 = plt.subplot(gs[1,4])
cbax5 = plt.subplot(gs[1,5])

cax = ax.imshow(map0)
ax.contour(median)
cb = Colorbar(ax = cbax,mappable = cax,shrink=0.8)

cax1 = ax1.imshow(map1)
ax1.contour(median)
cb1 = Colorbar(ax = cbax1,mappable = cax1)

cax2 = ax2.imshow(map2)
ax2.contour(median)
cb2 = Colorbar(ax = cbax2,mappable = cax2)

cax3 = ax3.imshow(map0/median)
ax3.contour(median)
cb3 = Colorbar(ax = cbax3,mappable = cax3)

cax4 = ax4.imshow(map1/median)
ax4.contour(median)
cb4 = Colorbar(ax = cbax4,mappable = cax4)

cax5 = ax5.imshow(map2)
ax5.contour(median)
cb5 = Colorbar(ax = cbax5,mappable = cax5)
</code>```

When I now call the kwargs shrink and or pad I get the following message:

```<code>Traceback (most recent call last):
  File "plot_integratedMaps.py", line 173, in <module>
    main()
  File "plot_integratedMaps.py", line 171, in main
    plot_integratedMaps(map630,map869,mapTot,median)
  File "plot_integratedMaps.py", line 129, in plot_integratedMaps
    cb = Colorbar(ax = cbax,mappable = cax,shrink=0.8)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/colorbar.py", line 943, in __init__
    ColorbarBase.__init__(self, ax, **kw)
TypeError: __init__() got an unexpected keyword argument 'shrink'
</code>```

I guess it makes sense that I cant pad the colorbar in in the gs[0,1] and have to movegs[0,1] instead. But I don't get why shrink doesn't work?

am using Python 2.7.12

答:
I would not consider it useful to create Colorbar directly like in the link; instead one could use fig.colorbar(). However, this is only tangential to the problem.

First consider creating a colorbar next to a plot.

```<code>import matplotlib.pyplot as plt
import numpy as np

median = np.zeros((100,100))
map0 = np.zeros((100,100))

fig, ax = plt.subplots()

im = ax.imshow(map0)
ax.contour(median)
cb = fig.colorbar(im, ax=ax, shrink=0.8)

plt.show()
</code>```

Here, shrink works fine because you want the axes in which the colorbar resides to be a factor 0.8 smaller than the axes ax to which it belongs.

Now, if you specify the axes in which the colorbar should reside, shrink does not make any sense, because the axes does not need to be created inside the colorbar function, but you supply it externally.

```<code>import matplotlib.pyplot as plt
import numpy as np

median = np.zeros((100,100))
map0 = np.zeros((100,100))

fig, (ax,cax) = plt.subplots(ncols=2)

im = ax.imshow(map0)
ax.contour(median)

#using `shrink` here would produce an error,
# because the colorbar axes (cax) already exists
# instead of
# cb = fig.colorbar(im, cax=cax, shrink=0.8)
# you need
cb = fig.colorbar(im, cax=cax)

plt.show()
</code>```

Note that this is independend of gridspec. Whether or not you want to use gridspec is also a question of taste, but surely not needed for simple plots.

If you have more plots, it again depends what you want to show. The edited example from the question looks more like a regular grid. Here creating a colorbar axes for each subplot could be efficiently done via make_axes_locatable.

```<code>from mpl_toolkits.axes_grid1 import make_axes_locatable
import matplotlib.pyplot as plt
import numpy as np

data = np.random.rand(6,6)

fig, axes = plt.subplots(nrows=2, ncols=3)

for ax in axes.flatten():
    im = ax.imshow(data)
    div = make_axes_locatable(ax)
    cax = div.append_axes("right", size="5%", pad=0.1)
    cbar = fig.colorbar(im, cax=cax)

plt.tight_layout()
plt.show()
</code>```

Taking the above, you may shrink the colorbars by not using this axes divider, but as usual, create your colorbar and use the shrink argument.

```<code>import matplotlib.pyplot as plt
import numpy as np

data = np.random.rand(6,6)

fig, axes = plt.subplots(nrows=2, ncols=3)

for ax in axes.flatten():
    im = ax.imshow(data)
    cbar = fig.colorbar(im, ax=ax, shrink=0.4)

plt.tight_layout()
plt.show()
</code>```

Title: moving colorbar with gridspec

Body:
I am making a plot with multiple figures and corresponding colorbars. This page http://www.sc.eso.org/~bdias/pycoffee/codes/20160407/gridspec_demo.html claims to know the best way of doing so. I am inclined to believe them.

However, I have run into issues adressing the kwags of the colorbar when I do it their way:

from matplotlib.colorbar import Colorbar
import matplotlib.gridspec as gridspec
import matplotlib.pyplot as plt
import numpy as np
median = np.zeros((100,100))
map0 = np.zeros((100,100))
map1 =  map0
map2 =  map0

fig = plt.figure()
plt.tight_layout()
gs = gridspec.GridSpec(2,6)

ax  = plt.subplot(gs[0,0])
cbax = plt.subplot(gs[0,1])

ax1 = plt.subplot(gs[0,2])
cbax1 = plt.subplot(gs[0,3])

ax2 = plt.subplot(gs[0,4])
cbax2 = plt.subplot(gs[0,5])

ax3 = plt.subplot(gs[1,0])
cbax3 = plt.subplot(gs[1,1])

ax4 = plt.subplot(gs[1,2])
cbax4 = plt.subplot(gs[1,3])

ax5 = plt.subplot(gs[1,4])
cbax5 = plt.subplot(gs[1,5])

cax = ax.imshow(map0)
ax.contour(median)
cb = Colorbar(ax = cbax,mappable = cax,shrink=0.8)

cax1 = ax1.imshow(map1)
ax1.contour(median)
cb1 = Colorbar(ax = cbax1,mappable = cax1)

cax2 = ax2.imshow(map2)
ax2.contour(median)
cb2 = Colorbar(ax = cbax2,mappable = cax2)

cax3 = ax3.imshow(map0/median)
ax3.contour(median)
cb3 = Colorbar(ax = cbax3,mappable = cax3)

cax4 = ax4.imshow(map1/median)
ax4.contour(median)
cb4 = Colorbar(ax = cbax4,mappable = cax4)

cax5 = ax5.imshow(map2)
ax5.contour(median)
cb5 = Colorbar(ax = cbax5,mappable = cax5)

When I now call the kwargs shrink and or pad I get the following message:

Traceback (most recent call last):
  File "plot_integratedMaps.py", line 173, in <module>
    main()
  File "plot_integratedMaps.py", line 171, in main
    plot_integratedMaps(map630,map869,mapTot,median)
  File "plot_integratedMaps.py", line 129, in plot_integratedMaps
    cb = Colorbar(ax = cbax,mappable = cax,shrink=0.8)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/colorbar.py", line 943, in __init__
    ColorbarBase.__init__(self, ax, **kw)
TypeError: __init__() got an unexpected keyword argument 'shrink'

I guess it makes sense that I cant pad the colorbar in in the gs[0,1] and have to movegs[0,1] instead. But I don't get why shrink doesn't work?

am using Python 2.7.12

Body:
I would not consider it useful to create Colorbar directly like in the link; instead one could use fig.colorbar(). However, this is only tangential to the problem.

First consider creating a colorbar next to a plot.

import matplotlib.pyplot as plt
import numpy as np

median = np.zeros((100,100))
map0 = np.zeros((100,100))

fig, ax = plt.subplots()

im = ax.imshow(map0)
ax.contour(median)
cb = fig.colorbar(im, ax=ax, shrink=0.8)

plt.show()

Here, shrink works fine because you want the axes in which the colorbar resides to be a factor 0.8 smaller than the axes ax to which it belongs.

Now, if you specify the axes in which the colorbar should reside, shrink does not make any sense, because the axes does not need to be created inside the colorbar function, but you supply it externally.

import matplotlib.pyplot as plt
import numpy as np

median = np.zeros((100,100))
map0 = np.zeros((100,100))

fig, (ax,cax) = plt.subplots(ncols=2)

im = ax.imshow(map0)
ax.contour(median)

#using `shrink` here would produce an error,
# because the colorbar axes (cax) already exists
# instead of
# cb = fig.colorbar(im, cax=cax, shrink=0.8)
# you need
cb = fig.colorbar(im, cax=cax)

plt.show()

Note that this is independend of gridspec. Whether or not you want to use gridspec is also a question of taste, but surely not needed for simple plots.

If you have more plots, it again depends what you want to show. The edited example from the question looks more like a regular grid. Here creating a colorbar axes for each subplot could be efficiently done via make_axes_locatable.

from mpl_toolkits.axes_grid1 import make_axes_locatable
import matplotlib.pyplot as plt
import numpy as np

data = np.random.rand(6,6)

fig, axes = plt.subplots(nrows=2, ncols=3)

for ax in axes.flatten():
    im = ax.imshow(data)
    div = make_axes_locatable(ax)
    cax = div.append_axes("right", size="5%", pad=0.1)
    cbar = fig.colorbar(im, cax=cax)

plt.tight_layout()
plt.show()

Taking the above, you may shrink the colorbars by not using this axes divider, but as usual, create your colorbar and use the shrink argument.

import matplotlib.pyplot as plt
import numpy as np

data = np.random.rand(6,6)

fig, axes = plt.subplots(nrows=2, ncols=3)

for ax in axes.flatten():
    im = ax.imshow(data)
    cbar = fig.colorbar(im, ax=ax, shrink=0.4)

plt.tight_layout()
plt.show()


问:
How to Add Bitbucket as Source code repository to Bluemix tool chanin service
I want to build pipeline in bluemix with bitbucket as my source repository.

I am using Bluemix and by default it takes github as a source code repository.

答:
Actually, you can't. IBM Bluemix only connects to public github repositories in their Continous Integration pipeline.

I suggest to user another solution, like using a tool like Jenkins that can listen easily to changes to the bitbucket repository, and then do the deploy (using the Cloud Foundry CLI) to a Bluemix environment.


问:
SQL inner join with slow order by performance
We have an .NET application in Azure with one page that shows an overview of time entries. We use Entity Framework to build and execute the SQL queries. When we sort by a column of the time table (like date or hours) it's fast enough. But when we sort on a column thats from an other table (from an inner join, like the user diplayname) its really slow.

Also when we look in our Azure portal in the DTU metrics (after this query) its peaked to 100% DTU. We have P4 service plan with 500 DTU's, so don't must be an problem I think.

This is an example of an slow query thats get execute by Entity Framework:

```<code>exec sp_executesql N'SELECT
[Project1].[Id] AS [Id],
[Project1].[Date] AS [Date],
[Project1].[Hours] AS [Hours],
[Project1].[Notes] AS [Notes],
[Project1].[StartEnd] AS [StartEnd],
[Project1].[Status] AS [Status],
[Project1].[Timer] AS [Timer],
[Project1].[TimeRowId] AS [TimeRowId],
[Project1].[InvoiceId] AS [InvoiceId],
[Project1].[Pause] AS [Pause],
[Project1].[ClientStatus] AS [ClientStatus],
[Project1].[ExternalUrl] AS [ExternalUrl],
[Project1].[ExternalName] AS [ExternalName],
[Project1].[ApprovedBy] AS [ApprovedBy],
[Project1].[ApprovedDate] AS [ApprovedDate],
[Project1].[ClientApprovedBy] AS [ClientApprovedBy],
[Project1].[ClientApprovedDate] AS [ClientApprovedDate]
FROM ( SELECT
    [Extent1].[Id] AS [Id],
    [Extent1].[Date] AS [Date],
    [Extent1].[Hours] AS [Hours],
    [Extent1].[Notes] AS [Notes],
    [Extent1].[StartEnd] AS [StartEnd],
    [Extent1].[Status] AS [Status],
    [Extent1].[Timer] AS [Timer],
    [Extent1].[TimeRowId] AS [TimeRowId],
    [Extent1].[InvoiceId] AS [InvoiceId],
    [Extent1].[Pause] AS [Pause],
    [Extent1].[ClientStatus] AS [ClientStatus],
    [Extent1].[ExternalUrl] AS [ExternalUrl],
    [Extent1].[ExternalName] AS [ExternalName],
    [Extent1].[ApprovedBy] AS [ApprovedBy],
    [Extent1].[ApprovedDate] AS [ApprovedDate],
    [Extent1].[ClientApprovedBy] AS [ClientApprovedBy],
    [Extent1].[ClientApprovedDate] AS [ClientApprovedDate],
    [Extent6].[DisplayName] AS [DisplayName]
    FROM      [dbo].[Time] AS [Extent1]
    INNER JOIN [dbo].[TimeRow] AS [Extent2] ON [Extent1].[TimeRowId] = [Extent2].[Id]
    INNER JOIN [dbo].[ProjectUser] AS [Extent3] ON [Extent2].[ProjectUserId] = [Extent3].[Id]
    INNER JOIN [dbo].[Project] AS [Extent4] ON [Extent3].[ProjectId] = [Extent4].[Id]
    INNER JOIN [dbo].[Customer] AS [Extent5] ON [Extent4].[CustomerId] = [Extent5].[Id]
    INNER JOIN [dbo].[User] AS [Extent6] ON [Extent3].[UserId] = [Extent6].[Id]
    WHERE ([Extent5].[CompanyId] = @p__linq__0) AND (@p__linq__1 <= [Extent1].[Date]) AND (@p__linq__2 >= [Extent1].[Date])
)  AS [Project1]
ORDER BY [Project1].[DisplayName] DESC
OFFSET 0 ROWS FETCH NEXT 25 ROWS ONLY ',N'@p__linq__0 int,@p__linq__1 datetime2(7),@p__linq__2 datetime2(7)',@p__linq__0=19218,@p__linq__1='2017-01-01 00:00:00',@p__linq__2='2017-12-31 00:00:00'
</code>```

We tried to add an nonclustered index on displayname (of the user table), but that didn't make any differents. Also we have automatic tuning on in Azure on the database, so you must think Azure adds the necessary indexes it self, right?

How can we make this query perform?

COMMENT 05-12-2017
I think we fixed the problem. We execute the query in Database Tuning Adviser and got 9 recommended indexes. After we apply this recommendations the query is really fast!

答:
I can't answer your question on how to make the query faster, as Sean Lange states, we'd need the table structure and execution plan for that.

I'm not very experienced with Azure, but you should see any indexes being used.

This document: https://learn.microsoft.com/en-us/azure/sql-database/sql-database-advisor

Indexes created using recommendations are always flagged as
  auto_created indexes. You can see which indexes are auto_created by
  looking at sys.indexes view.

That document may help you find if any indexes are being created for you by Azure or if you need to accept any suggestions.

Body:
We have an .NET application in Azure with one page that shows an overview of time entries. We use Entity Framework to build and execute the SQL queries. When we sort by a column of the time table (like date or hours) it's fast enough. But when we sort on a column thats from an other table (from an inner join, like the user diplayname) its really slow.

Also when we look in our Azure portal in the DTU metrics (after this query) its peaked to 100% DTU. We have P4 service plan with 500 DTU's, so don't must be an problem I think.

This is an example of an slow query thats get execute by Entity Framework:

exec sp_executesql N'SELECT
[Project1].[Id] AS [Id],
[Project1].[Date] AS [Date],
[Project1].[Hours] AS [Hours],
[Project1].[Notes] AS [Notes],
[Project1].[StartEnd] AS [StartEnd],
[Project1].[Status] AS [Status],
[Project1].[Timer] AS [Timer],
[Project1].[TimeRowId] AS [TimeRowId],
[Project1].[InvoiceId] AS [InvoiceId],
[Project1].[Pause] AS [Pause],
[Project1].[ClientStatus] AS [ClientStatus],
[Project1].[ExternalUrl] AS [ExternalUrl],
[Project1].[ExternalName] AS [ExternalName],
[Project1].[ApprovedBy] AS [ApprovedBy],
[Project1].[ApprovedDate] AS [ApprovedDate],
[Project1].[ClientApprovedBy] AS [ClientApprovedBy],
[Project1].[ClientApprovedDate] AS [ClientApprovedDate]
FROM ( SELECT
    [Extent1].[Id] AS [Id],
    [Extent1].[Date] AS [Date],
    [Extent1].[Hours] AS [Hours],
    [Extent1].[Notes] AS [Notes],
    [Extent1].[StartEnd] AS [StartEnd],
    [Extent1].[Status] AS [Status],
    [Extent1].[Timer] AS [Timer],
    [Extent1].[TimeRowId] AS [TimeRowId],
    [Extent1].[InvoiceId] AS [InvoiceId],
    [Extent1].[Pause] AS [Pause],
    [Extent1].[ClientStatus] AS [ClientStatus],
    [Extent1].[ExternalUrl] AS [ExternalUrl],
    [Extent1].[ExternalName] AS [ExternalName],
    [Extent1].[ApprovedBy] AS [ApprovedBy],
    [Extent1].[ApprovedDate] AS [ApprovedDate],
    [Extent1].[ClientApprovedBy] AS [ClientApprovedBy],
    [Extent1].[ClientApprovedDate] AS [ClientApprovedDate],
    [Extent6].[DisplayName] AS [DisplayName]
    FROM      [dbo].[Time] AS [Extent1]
    INNER JOIN [dbo].[TimeRow] AS [Extent2] ON [Extent1].[TimeRowId] = [Extent2].[Id]
    INNER JOIN [dbo].[ProjectUser] AS [Extent3] ON [Extent2].[ProjectUserId] = [Extent3].[Id]
    INNER JOIN [dbo].[Project] AS [Extent4] ON [Extent3].[ProjectId] = [Extent4].[Id]
    INNER JOIN [dbo].[Customer] AS [Extent5] ON [Extent4].[CustomerId] = [Extent5].[Id]
    INNER JOIN [dbo].[User] AS [Extent6] ON [Extent3].[UserId] = [Extent6].[Id]
    WHERE ([Extent5].[CompanyId] = @p__linq__0) AND (@p__linq__1 <= [Extent1].[Date]) AND (@p__linq__2 >= [Extent1].[Date])
)  AS [Project1]
ORDER BY [Project1].[DisplayName] DESC
OFFSET 0 ROWS FETCH NEXT 25 ROWS ONLY ',N'@p__linq__0 int,@p__linq__1 datetime2(7),@p__linq__2 datetime2(7)',@p__linq__0=19218,@p__linq__1='2017-01-01 00:00:00',@p__linq__2='2017-12-31 00:00:00'

We tried to add an nonclustered index on displayname (of the user table), but that didn't make any differents. Also we have automatic tuning on in Azure on the database, so you must think Azure adds the necessary indexes it self, right?

How can we make this query perform?

COMMENT 05-12-2017
I think we fixed the problem. We execute the query in Database Tuning Adviser and got 9 recommended indexes. After we apply this recommendations the query is really fast!


问:
How to insert elements from arrayList to a map, sorting by their name?
I have a Worker class with grade and name for each worker.
I need to write a method that will return me a sorted map by the workers name..
The sort is by a Comparator I build that calld workersComparator..
And there is a limit that I cant write the word Worker(The class) more then once..I need to work generic way..

The comparator is like this:

```<code>import java.util.Comparator;

public class workersComparator implements Comparator<Worker<?>>{

    @Override
    public int compare(Worker<?> worker1, Worker<?> worker2) {
        return worker1.getName().compareTo(worker2.getName());
    }
}
</code>```

and the method is like this:

```<code>public static Map<Integer,?> createMap(){

    TreeSet<Worker<?>> treeSet = new TreeSet<>(new workersComparator());
    for (int i = 0; i < workers.size(); i++) {
        treeSet.add(workers.get(i));
    }
    Map<Integer,?> workersMap = new TreeMap<>();
    int index = 1;
    int pointer = 0;
    while( pointer < treeSet.size() ){
        workersMap.put(index,treeSet.iterator().next());
        index++;
        pointer++;
    }
    return workersMap;
}
</code>```

答:
If you want to convert a java.util.List to a java.util.TreeSet and have it sorted by name, you can use the constructor of java.util.TreeSet. But the Worker class needs to implement the Comparable interface.

In this case you could simply use:

```<code>TreeSet<Worker> worker = new TreeSet<>(workers);
</code>```

Here is a full code example you can try out:

```<code>public class ArrayListToSortedMap {

    public static void main(String[] args) {
        List<Worker> workers = Arrays.asList(new Worker("john"), new Worker("jane"),
            new Worker("james"), new Worker("maria"), new Worker("andre"));
        TreeSet<Worker> worker = new TreeSet<>(workers);
        worker.stream().map(Worker::getName).forEach(System.out::println);
    }

    private static class Worker implements Comparable<Worker> {

        private String name;

        public Worker(String name) {
            this.name = name;
        }

        public String getName() {
            return name;
        }

        @Override
        public int compareTo(Worker o) {
            return name.compareTo(o.name);
        }
    }
}
</code>```

This is the output of this code:

```<code>andre
james
jane
john
maria
</code>```

Alternatively you can use a comparator in a stream and convert the ArrayList to a map, but the code is more convoluted:

```<code>public class ArrayListToSortedMap2 {

    public static void main(String[] args) {
        List<Worker> workers = Arrays.asList(new Worker("john"), new Worker("jane"),
                new Worker("james"), new Worker("maria"), new Worker("andre"));
        Map<String, Worker> worker = workers.stream()
            .collect(Collectors.toMap(Worker::getName, Function.identity()));
        worker.entrySet().forEach(System.out::println);
    }

    private static class Worker {

        private String name;

        Worker(String name) {
            this.name = name;
        }

        public String getName() {
            return name;
        }

        @Override
        public String toString() {
            return String.format("worker -> %s", name);
        }
    }
}
</code>```

This is the output:

```<code>andre=worker -> andre
james=worker -> james
john=worker -> john
maria=worker -> maria
jane=worker -> jane
</code>```

Body:
I have a Worker class with grade and name for each worker.
I need to write a method that will return me a sorted map by the workers name..
The sort is by a Comparator I build that calld workersComparator..
And there is a limit that I cant write the word Worker(The class) more then once..I need to work generic way..

The comparator is like this:

import java.util.Comparator;

public class workersComparator implements Comparator<Worker<?>>{

    @Override
    public int compare(Worker<?> worker1, Worker<?> worker2) {
        return worker1.getName().compareTo(worker2.getName());
    }
}

and the method is like this:

public static Map<Integer,?> createMap(){

    TreeSet<Worker<?>> treeSet = new TreeSet<>(new workersComparator());
    for (int i = 0; i < workers.size(); i++) {
        treeSet.add(workers.get(i));
    }
    Map<Integer,?> workersMap = new TreeMap<>();
    int index = 1;
    int pointer = 0;
    while( pointer < treeSet.size() ){
        workersMap.put(index,treeSet.iterator().next());
        index++;
        pointer++;
    }
    return workersMap;
}

Body:
If you want to convert a java.util.List to a java.util.TreeSet and have it sorted by name, you can use the constructor of java.util.TreeSet. But the Worker class needs to implement the Comparable interface.

In this case you could simply use:

TreeSet<Worker> worker = new TreeSet<>(workers);

Here is a full code example you can try out:

public class ArrayListToSortedMap {

    public static void main(String[] args) {
        List<Worker> workers = Arrays.asList(new Worker("john"), new Worker("jane"),
            new Worker("james"), new Worker("maria"), new Worker("andre"));
        TreeSet<Worker> worker = new TreeSet<>(workers);
        worker.stream().map(Worker::getName).forEach(System.out::println);
    }

    private static class Worker implements Comparable<Worker> {

        private String name;

        public Worker(String name) {
            this.name = name;
        }

        public String getName() {
            return name;
        }

        @Override
        public int compareTo(Worker o) {
            return name.compareTo(o.name);
        }
    }
}

This is the output of this code:

andre
james
jane
john
maria

Alternatively you can use a comparator in a stream and convert the ArrayList to a map, but the code is more convoluted:

public class ArrayListToSortedMap2 {

    public static void main(String[] args) {
        List<Worker> workers = Arrays.asList(new Worker("john"), new Worker("jane"),
                new Worker("james"), new Worker("maria"), new Worker("andre"));
        Map<String, Worker> worker = workers.stream()
            .collect(Collectors.toMap(Worker::getName, Function.identity()));
        worker.entrySet().forEach(System.out::println);
    }

    private static class Worker {

        private String name;

        Worker(String name) {
            this.name = name;
        }

        public String getName() {
            return name;
        }

        @Override
        public String toString() {
            return String.format("worker -> %s", name);
        }
    }
}

This is the output:

andre=worker -> andre
james=worker -> james
john=worker -> john
maria=worker -> maria
jane=worker -> jane


问:
Declaring a variable as final for Firebase function. Checking if username and email exist
I am trying to make a register page. Before registering that user I will try to validate the username and email do not exist in the database. But I somehow cannot set the variable to be false when the username/email already exist so it will end up still registering the user.

How can I do this? Please guide me on this.

Checking if username and email already exist before register user:

```<code>            if (!checkIfUsernameExists(userName) && !checkIfEmailExists(userEmail)){
            registerUser();
        }
</code>```

Method for checking if the username exists or not:

```<code>    public boolean checkIfUsernameExists(String userName){
    boolean isExist = false;
    databaseReferenceUser.orderByChild("userName").equalTo(userName).addListenerForSingleValueEvent(new ValueEventListener() {
        @Override
        public void onDataChange(DataSnapshot dataSnapshot) {
            if (dataSnapshot.exists()){
                editTextUserNameRegister.setError("Username already exists");
                editTextUserNameRegister.requestFocus();
                final boolean isExist = true;
            }
        }

        @Override
        public void onCancelled(DatabaseError databaseError) {

        }
    });

    return isExist;
}
</code>```

Method for checking if the email already exists or not:

```<code>    public boolean checkIfEmailExists(String userEmail){
    boolean isExist = false;
    databaseReferenceUser.orderByChild("userEmail").equalTo(userEmail).addListenerForSingleValueEvent(new ValueEventListener() {
        @Override
        public void onDataChange(DataSnapshot dataSnapshot) {
            if (dataSnapshot.exists()){
                editTextEmailRegister.setError("Email already exists");
                editTextEmailRegister.requestFocus();
                final boolean isExist = true;
            }
        }

        @Override
        public void onCancelled(DatabaseError databaseError) {

        }
    });

    return isExist;
}
</code>```

答:
I see you are using Firebase, and it would be a much better idea to use FirebaseAuth, which lets you authenticate users on a secure, encrypted and versatile way. If you use the database you may be risking the privacy and security of the user accounts (I assume the passwords aren't encrypted and the accessing rules are not exactly private).

Add in your Gradle file:

```<code>compile 'com.google.firebase:firebase-auth:11.6.2'
</code>```

And then you only have to enable an Email/Password auth:

LoginRegisterActivity.java

1 - Get the instance.

```<code>FirebaseAuth mAuth = FirebaseAuth.getInstance();
</code>```

2 - Get the user. Once it's registered and/or signed in, it will be saved.

```<code>FirebaseUser user = mAuth.getCurrentUser();
</code>```

3 - So let's try to create a new user

```<code>mAuth.createUserWithEmailAndPassword(email,password)
            .addOnCompleteListener(this, new OnCompleteListener<AuthResult>() {
                @Override
                public void onComplete(@NonNull Task<AuthResult> task) {
                    if(task.isSuccessful()){
                       //User registered successfully
                    }else{
                       // WAIT! SOMETHING WENT WRONG, DESPITE IT IS COMPLETE
                    }
                }
            });
</code>```

4 - If the task is succesful, you can just leave everything like that and finish() the activity, FirebaseAuth will save it for you, and you can get the info from anywhere in your app.

5 - But the task may not be successful, and that is because there was already another user in the project with the same email. So inside the else in onComplete() you can try to login the user. This is pretty straight-forward as well.

```<code>mAuth.signInWithEmailAndPassword(email, password).addOnCompleteListener(activity, new OnCompleteListener<AuthResult>() {
                @Override public void onComplete(@NonNull Task<AuthResult> task) {
//And here you can do whatever you want: create a file on your DB, send a verification email...
});
</code>```

Body:
I am trying to make a register page. Before registering that user I will try to validate the username and email do not exist in the database. But I somehow cannot set the variable to be false when the username/email already exist so it will end up still registering the user.

How can I do this? Please guide me on this.

Checking if username and email already exist before register user:

            if (!checkIfUsernameExists(userName) && !checkIfEmailExists(userEmail)){
            registerUser();
        }

Method for checking if the username exists or not:

    public boolean checkIfUsernameExists(String userName){
    boolean isExist = false;
    databaseReferenceUser.orderByChild("userName").equalTo(userName).addListenerForSingleValueEvent(new ValueEventListener() {
        @Override
        public void onDataChange(DataSnapshot dataSnapshot) {
            if (dataSnapshot.exists()){
                editTextUserNameRegister.setError("Username already exists");
                editTextUserNameRegister.requestFocus();
                final boolean isExist = true;
            }
        }

        @Override
        public void onCancelled(DatabaseError databaseError) {

        }
    });

    return isExist;
}

Method for checking if the email already exists or not:

    public boolean checkIfEmailExists(String userEmail){
    boolean isExist = false;
    databaseReferenceUser.orderByChild("userEmail").equalTo(userEmail).addListenerForSingleValueEvent(new ValueEventListener() {
        @Override
        public void onDataChange(DataSnapshot dataSnapshot) {
            if (dataSnapshot.exists()){
                editTextEmailRegister.setError("Email already exists");
                editTextEmailRegister.requestFocus();
                final boolean isExist = true;
            }
        }

        @Override
        public void onCancelled(DatabaseError databaseError) {

        }
    });

    return isExist;
}

Body:
I see you are using Firebase, and it would be a much better idea to use FirebaseAuth, which lets you authenticate users on a secure, encrypted and versatile way. If you use the database you may be risking the privacy and security of the user accounts (I assume the passwords aren't encrypted and the accessing rules are not exactly private).

Add in your Gradle file:

compile 'com.google.firebase:firebase-auth:11.6.2'

And then you only have to enable an Email/Password auth:

LoginRegisterActivity.java

1 - Get the instance.

FirebaseAuth mAuth = FirebaseAuth.getInstance();

2 - Get the user. Once it's registered and/or signed in, it will be saved.

FirebaseUser user = mAuth.getCurrentUser();

3 - So let's try to create a new user

mAuth.createUserWithEmailAndPassword(email,password)
            .addOnCompleteListener(this, new OnCompleteListener<AuthResult>() {
                @Override
                public void onComplete(@NonNull Task<AuthResult> task) {
                    if(task.isSuccessful()){
                       //User registered successfully
                    }else{
                       // WAIT! SOMETHING WENT WRONG, DESPITE IT IS COMPLETE
                    }
                }
            });

4 - If the task is succesful, you can just leave everything like that and finish() the activity, FirebaseAuth will save it for you, and you can get the info from anywhere in your app.

5 - But the task may not be successful, and that is because there was already another user in the project with the same email. So inside the else in onComplete() you can try to login the user. This is pretty straight-forward as well.

mAuth.signInWithEmailAndPassword(email, password).addOnCompleteListener(activity, new OnCompleteListener<AuthResult>() {
                @Override public void onComplete(@NonNull Task<AuthResult> task) {
//And here you can do whatever you want: create a file on your DB, send a verification email...
});


问:
Sum integer list when next integer is the same value
So I need to have a code that checks one integer, and checks if the integer after it is the same value. If so, it will add the value to x.

```<code>input1 = [int(i) for i in str(1234441122)]
x= 0
</code>```

So my code currently gives the result [1, 2, 3, 4, 4, 4, 1, 1 ,2 ,2]. I want it to give the result of x = 0+4+4+1+2.
I do not know any way to do that.

答:
Here's a slightly more space efficient version of @schwobaseggl's answer.

```<code>>>> lst = [1, 2, 3, 4, 4, 4, 1, 1, 2, 2]
>>> it = iter(lst)
>>> next(it) # throw away first value
>>> sum(x for x,y in zip(lst, it) if x == y)
11
</code>```

Alernatively, using an islice from the itertools module is equivalent but looks a bit nicer.

```<code>>>> from itertools import islice
>>> sum(x for x,y in zip(lst, islice(lst, 1, None, 1)) if x == y)
11
</code>```

Body:
So I need to have a code that checks one integer, and checks if the integer after it is the same value. If so, it will add the value to x.

input1 = [int(i) for i in str(1234441122)]
x= 0

So my code currently gives the result [1, 2, 3, 4, 4, 4, 1, 1 ,2 ,2]. I want it to give the result of x = 0+4+4+1+2.
I do not know any way to do that.

Body:
Here's a slightly more space efficient version of @schwobaseggl's answer.

>>> lst = [1, 2, 3, 4, 4, 4, 1, 1, 2, 2]
>>> it = iter(lst)
>>> next(it) # throw away first value
>>> sum(x for x,y in zip(lst, it) if x == y)
11

Alernatively, using an islice from the itertools module is equivalent but looks a bit nicer.

>>> from itertools import islice
>>> sum(x for x,y in zip(lst, islice(lst, 1, None, 1)) if x == y)
11