问:
Rearranging multiple rows of a dataframe on big data in R
I am relatively new in R. I have a dataframe test that looks like this:

```<code>PMID    # id
LID
STAT
MH
RN
OT
PST     # cue
LID
STAT
MH
PMID    # id
OT
PST     # cue
LID
DEP
RN
PMID    # id
PST     # cue
</code>```

and I want it to look like this:

```<code>PMID    # id
LID
STAT
MH
RN
OT
PST     # cue
PMID    # id
LID
STAT
MH
OT
PST     # cue
PMID    # id
LID
DEP
RN
PST     # cue
</code>```

Basically, I want the entries that follow PMID to be for that particular PMID, which is the case for the first PMID. However, after the first PMID, the PMID is nestled randomly in between its entries. However, each PMID ends with a PST, so I want the subsequent PMIDs after the first one to be moved to the location after the previous PST's location. I have two dataframes that contain the index location of each PMID and of PST. So for instance, for PMID, df a_new contains

```<code>1
11
17
</code>```

and for PST, df b contains

```<code>7
13
18
</code>```

This is what I have tried, but because I have over 24 million rows, it didn't finish running after hours and when I stopped it, my dataframe had not changed:

```<code>for (i in 1:nrow(test))
{
  if (i %in% a_new$X1) # if it's a PMID
  {
    entry <- match(i, a_new$X1) # find entry index of PMID
    if (entry != 1) # as long as not first row from a_new (that's corrected)
    {
      r <- b[i, 1] # row of PST
      test <- rbind(test[1:r, ], test[entry, 1], test[-(1:r), ])
      test <- test[-c(i+1), ] # remove duplicate PMID
    }
  }
}
</code>```

As you can see, rbind would be extremely inefficient in this situation. Please advise.

答:
Here is an answer using data.table.

```<code>library(data.table)

dat <- fread("Origcol
             PMID
             LID
             STAT
             MH
             RN
             OT
             PST
             LID
             STAT
             MH
             PMID
             OT
             PST
             LID
             DEP
             RN
             PMID
             PST")

dat[, old_order := 1:.N]
pst_index <- c(0, which(dat$Origcol == "PST"))
dat[, grp := unlist(lapply(1:(length(pst_index)-1),
                           function(x) rep(x,
                                           times = (pst_index[x+1] - pst_index[x]))))]
dat[, Origcol := factor(Origcol, levels = c("PMID", "LID", "STAT",
                                            "MH", "RN", "OT",
                                            "DEP", "PST"))]
dat[order(grp, Origcol)]
</code>```

Results in:

```<code>    Origcol old_order grp
 1:    PMID         1   1
 2:     LID         2   1
 3:    STAT         3   1
 4:      MH         4   1
 5:      RN         5   1
 6:      OT         6   1
 7:     PST         7   1
 8:    PMID        11   2
 9:     LID         8   2
10:    STAT         9   2
11:      MH        10   2
12:      OT        12   2
13:     PST        13   2
14:    PMID        17   3
15:     LID        14   3
16:      RN        16   3
17:     DEP        15   3
18:     PST        18   3
</code>```

The benefit of this is data.table is doing a lot of operations by reference, and should be fast once you scale up the size. You said you have 14 million rows, lets try it out. Generate some synthetic data of that size:

```<code>dat_big <- data.table(Origcol = c("PMID", "LID", "STAT", "MH", "RN", "OT", "PST"))
dat_big_add <- rbindlist(lapply(1:10000,
                                function(x) data.table(Origcol = c(sample(c("PMID", "LID", "STAT",
                                                                            "MH", "RN", "OT")),
                                                                   "PST"))))
dat_big <- rbindlist(list(dat_big,
                          dat_big_add, dat_big_add, dat_big_add, dat_big_add, dat_big_add,
                          dat_big_add, dat_big_add, dat_big_add, dat_big_add, dat_big_add,
                          dat_big_add, dat_big_add, dat_big_add, dat_big_add, dat_big_add,
                          dat_big_add, dat_big_add, dat_big_add, dat_big_add, dat_big_add))

dat <- rbindlist(list(dat_big, dat_big, dat_big, dat_big, dat_big,
                      dat_big, dat_big, dat_big, dat_big, dat_big))
</code>```

We now have:

```<code>          Origcol
       1:    PMID
       2:     LID
       3:    STAT
       4:      MH
       5:      RN
      ---
14000066:    STAT
14000067:      MH
14000068:      OT
14000069:    PMID
14000070:     PST
</code>```

Apply the same code as above:

```<code>dat[, old_order := 1:.N]
pst_index <- c(0, which(dat$Origcol == "PST"))
dat[, grp := unlist(lapply(1:(length(pst_index)-1),
                           function(x) rep(x,
                                           times = (pst_index[x+1] - pst_index[x]))))]
dat[, Origcol := factor(Origcol, levels = c("PMID", "LID", "STAT",
                                            "MH", "RN", "OT",
                                            "DEP", "PST"))]
dat[order(grp, Origcol)]
</code>```

We now get:

```<code>          Origcol old_order     grp
       1:    PMID         1       1
       2:     LID         2       1
       3:    STAT         3       1
       4:      MH         4       1
       5:      RN         5       1
      ---
14000066:    STAT  14000066 2000010
14000067:      MH  14000067 2000010
14000068:      RN  14000064 2000010
14000069:      OT  14000068 2000010
14000070:     PST  14000070 2000010
</code>```

How long does it take?

```<code>library(microbenchmark)
microbenchmark(
  "data.table" = {
    dat[, old_order := 1:.N]
    pst_index <- c(0, which(dat$Origcol == "PST"))
    dat[, grp := unlist(lapply(1:(length(pst_index)-1),
                               function(x) rep(x,
                                               times = (pst_index[x+1] - pst_index[x]))))]
    dat[, Origcol := factor(Origcol, levels = c("PMID", "LID", "STAT",
                                                "MH", "RN", "OT",
                                                "DEP", "PST"))]
    dat[order(grp, Origcol)]
  },
  times = 10)
</code>```

And it takes:

```<code>Unit: seconds
       expr      min       lq     mean  median       uq      max neval
 data.table 5.755276 5.813267 6.059665 5.87151 6.034506 7.310169    10
</code>```

Under 10 seconds for 14 million rows. It took a lot longer to generate the test data.

Body:
I am relatively new in R. I have a dataframe test that looks like this:

PMID    # id
LID
STAT
MH
RN
OT
PST     # cue
LID
STAT
MH
PMID    # id
OT
PST     # cue
LID
DEP
RN
PMID    # id
PST     # cue

and I want it to look like this:

PMID    # id
LID
STAT
MH
RN
OT
PST     # cue
PMID    # id
LID
STAT
MH
OT
PST     # cue
PMID    # id
LID
DEP
RN
PST     # cue

Basically, I want the entries that follow PMID to be for that particular PMID, which is the case for the first PMID. However, after the first PMID, the PMID is nestled randomly in between its entries. However, each PMID ends with a PST, so I want the subsequent PMIDs after the first one to be moved to the location after the previous PST's location. I have two dataframes that contain the index location of each PMID and of PST. So for instance, for PMID, df a_new contains

1
11
17

and for PST, df b contains

7
13
18

This is what I have tried, but because I have over 24 million rows, it didn't finish running after hours and when I stopped it, my dataframe had not changed:

for (i in 1:nrow(test))
{
  if (i %in% a_new$X1) # if it's a PMID
  {
    entry <- match(i, a_new$X1) # find entry index of PMID
    if (entry != 1) # as long as not first row from a_new (that's corrected)
    {
      r <- b[i, 1] # row of PST
      test <- rbind(test[1:r, ], test[entry, 1], test[-(1:r), ])
      test <- test[-c(i+1), ] # remove duplicate PMID
    }
  }
}

As you can see, rbind would be extremely inefficient in this situation. Please advise.

Body:
Here is an answer using data.table.

library(data.table)

dat <- fread("Origcol
             PMID
             LID
             STAT
             MH
             RN
             OT
             PST
             LID
             STAT
             MH
             PMID
             OT
             PST
             LID
             DEP
             RN
             PMID
             PST")

dat[, old_order := 1:.N]
pst_index <- c(0, which(dat$Origcol == "PST"))
dat[, grp := unlist(lapply(1:(length(pst_index)-1),
                           function(x) rep(x,
                                           times = (pst_index[x+1] - pst_index[x]))))]
dat[, Origcol := factor(Origcol, levels = c("PMID", "LID", "STAT",
                                            "MH", "RN", "OT",
                                            "DEP", "PST"))]
dat[order(grp, Origcol)]

Results in:

    Origcol old_order grp
 1:    PMID         1   1
 2:     LID         2   1
 3:    STAT         3   1
 4:      MH         4   1
 5:      RN         5   1
 6:      OT         6   1
 7:     PST         7   1
 8:    PMID        11   2
 9:     LID         8   2
10:    STAT         9   2
11:      MH        10   2
12:      OT        12   2
13:     PST        13   2
14:    PMID        17   3
15:     LID        14   3
16:      RN        16   3
17:     DEP        15   3
18:     PST        18   3

The benefit of this is data.table is doing a lot of operations by reference, and should be fast once you scale up the size. You said you have 14 million rows, lets try it out. Generate some synthetic data of that size:

dat_big <- data.table(Origcol = c("PMID", "LID", "STAT", "MH", "RN", "OT", "PST"))
dat_big_add <- rbindlist(lapply(1:10000,
                                function(x) data.table(Origcol = c(sample(c("PMID", "LID", "STAT",
                                                                            "MH", "RN", "OT")),
                                                                   "PST"))))
dat_big <- rbindlist(list(dat_big,
                          dat_big_add, dat_big_add, dat_big_add, dat_big_add, dat_big_add,
                          dat_big_add, dat_big_add, dat_big_add, dat_big_add, dat_big_add,
                          dat_big_add, dat_big_add, dat_big_add, dat_big_add, dat_big_add,
                          dat_big_add, dat_big_add, dat_big_add, dat_big_add, dat_big_add))

dat <- rbindlist(list(dat_big, dat_big, dat_big, dat_big, dat_big,
                      dat_big, dat_big, dat_big, dat_big, dat_big))

We now have:

          Origcol
       1:    PMID
       2:     LID
       3:    STAT
       4:      MH
       5:      RN
      ---
14000066:    STAT
14000067:      MH
14000068:      OT
14000069:    PMID
14000070:     PST

Apply the same code as above:

dat[, old_order := 1:.N]
pst_index <- c(0, which(dat$Origcol == "PST"))
dat[, grp := unlist(lapply(1:(length(pst_index)-1),
                           function(x) rep(x,
                                           times = (pst_index[x+1] - pst_index[x]))))]
dat[, Origcol := factor(Origcol, levels = c("PMID", "LID", "STAT",
                                            "MH", "RN", "OT",
                                            "DEP", "PST"))]
dat[order(grp, Origcol)]

We now get:

          Origcol old_order     grp
       1:    PMID         1       1
       2:     LID         2       1
       3:    STAT         3       1
       4:      MH         4       1
       5:      RN         5       1
      ---
14000066:    STAT  14000066 2000010
14000067:      MH  14000067 2000010
14000068:      RN  14000064 2000010
14000069:      OT  14000068 2000010
14000070:     PST  14000070 2000010

How long does it take?

library(microbenchmark)
microbenchmark(
  "data.table" = {
    dat[, old_order := 1:.N]
    pst_index <- c(0, which(dat$Origcol == "PST"))
    dat[, grp := unlist(lapply(1:(length(pst_index)-1),
                               function(x) rep(x,
                                               times = (pst_index[x+1] - pst_index[x]))))]
    dat[, Origcol := factor(Origcol, levels = c("PMID", "LID", "STAT",
                                                "MH", "RN", "OT",
                                                "DEP", "PST"))]
    dat[order(grp, Origcol)]
  },
  times = 10)

And it takes:

Unit: seconds
       expr      min       lq     mean  median       uq      max neval
 data.table 5.755276 5.813267 6.059665 5.87151 6.034506 7.310169    10

Under 10 seconds for 14 million rows. It took a lot longer to generate the test data.


来源: stackexchange

问:
Strange convolution result
I want to understand how convolution works.
Here is some code:

```<code>import numpy
from scipy import misc

data = misc.imread("path_to_a_512x512_grayscale_image.png")
data = data/255.0

masque = numpy.array([[-1,0,1],
                      [-2,0,0],
                      [-1,0,1]],numpy.double)

def my_convolution(image, masque):
    hauteur,largeur = image.shape
    resultat = numpy.empty((hauteur,largeur))
    for y in range(1,hauteur-1):
        for x in range(1,largeur-1):
            pixel = 0.0
            for ym in range(3):
                for xm in range(3):
                    pixel += masque[ym,xm]*image[y-1+ym,x-1+ym]
            resultat[y,x]=pixel/9.0
    return resultat

my_result = my_convolution(data,masque)
plt.imshow(my_result, cmap='gray')
</code>```

Result is not exactly the same with this basic method bellow.
My previous method gives a a picture that seems to be darker

```<code>from scipy import signal
result2 = signal.convolve2d(data, masque)
result2 = result2[1:-1,1:-1]
plt.imshow(result2, cmap='gray')
</code>```

Anyone call explain me with those 2 codes does not give the same result ?
I do not want to know which method is fastest, i know the first method is very ugly, i just want to understand.

Thanks

答:
The convolution requires going backwards over one of the convolved functions, which means subtracting the inner indices, not adding them. Also, the second index in the image access expression has mismatched terms. So,

```<code>pixel += masque[ym,xm]*image[y-1+ym,x-1+ym]
</code>```

should instead be more like

```<code>pixel += masque[ym,xm]*image[y-1-ym,x-1-xm]
</code>```

For confirmation look at the code that runs when you call signal.convolve2d (specifically here and here). The inner indices match their respective outer indices, and they are subtracted during a convolution, not added.

ContentLicense: CC BY-SA 3.0

Title: Strange convolution result

Body:
I want to understand how convolution works.
Here is some code:

import numpy
from scipy import misc

data = misc.imread("path_to_a_512x512_grayscale_image.png")
data = data/255.0

masque = numpy.array([[-1,0,1],
                      [-2,0,0],
                      [-1,0,1]],numpy.double)

def my_convolution(image, masque):
    hauteur,largeur = image.shape
    resultat = numpy.empty((hauteur,largeur))
    for y in range(1,hauteur-1):
        for x in range(1,largeur-1):
            pixel = 0.0
            for ym in range(3):
                for xm in range(3):
                    pixel += masque[ym,xm]*image[y-1+ym,x-1+ym]
            resultat[y,x]=pixel/9.0
    return resultat

my_result = my_convolution(data,masque)
plt.imshow(my_result, cmap='gray')

Result is not exactly the same with this basic method bellow.
My previous method gives a a picture that seems to be darker

from scipy import signal
result2 = signal.convolve2d(data, masque)
result2 = result2[1:-1,1:-1]
plt.imshow(result2, cmap='gray')

Anyone call explain me with those 2 codes does not give the same result ?
I do not want to know which method is fastest, i know the first method is very ugly, i just want to understand.

Thanks

Body:
The convolution requires going backwards over one of the convolved functions, which means subtracting the inner indices, not adding them. Also, the second index in the image access expression has mismatched terms. So,

pixel += masque[ym,xm]*image[y-1+ym,x-1+ym]

should instead be more like

pixel += masque[ym,xm]*image[y-1-ym,x-1-xm]

For confirmation look at the code that runs when you call signal.convolve2d (specifically here and here). The inner indices match their respective outer indices, and they are subtracted during a convolution, not added.


问:
Batch file & folder renaming in Windows Powershell
I have hundred thousands files located in ~40 folders (these subfolders being located in a same super-folder which I am going to call 'test'). I have tried to use powershell to replace a syntax found in the folder name but also in the filenames.

Basically my folders and files contains the same string of characters "PXXXX_0" that I want to batch rename in "PXXXX_"

Here is the code I used to first map the files & folders of interest

```<code>cd C:\test

Get-ChildItem -Filter "*PXXXX_0*" -Recurse | Rename-Item -NewName {$_.name -replace 'PXXXX_0', 'PXXXX_'} -whatif
</code>```

This gives me the right files and previewed action (e.g.

```<code>What if: Performing operation "Rename File" on Target "Item: C:\test\PXXXX_053\PXXXX_053_558.jpg Destination C:\test\PXXXX_053\PXXXX_53_558.jpg
</code>```

So this seems to work, but only on the files inside the folders that I also want to rename.

Now if I launch the code without -whatif it appears that only the folder names are rename, not the files...

I pretty sure I have missed something but since I am not really familiar with this language, I am struggling to find the solution. I am using Powershell v1.0

Thanks for any help.

Best

答:
One Liner:

```<code> $items = get-childitem -filter "*PXXXX_0*" -Recurse; foreach ($item in $items) { $newName = $item.fullname -replace 'PXXXX_0', 'PXXXX_'; rename-item $item.fullname -NewName $newName; }
</code>```

Easy on the eyeballs:

```<code>$items = get-childitem -filter "*PXXXX_0*" -Recurse;
foreach ($item in $items) {
  $newName = $item.fullname -replace 'PXXXX_0', 'PXXXX_';
  rename-item $item.fullname -NewName $newName;
}
</code>```

In short, PowerShell 1 was trying to rename the file with just the file name, not the full path. Basically it was not able to find your files when they were in subfolders.

I tested your logic in v5 and it works like you would expect it to.

I'm going to go out on a limb here and assume you are trying to run this script via System Center Orchestrator since you are working with PowerShell v1. I've felt that pain. I recommend wrapping all your PS scripts headed for runbooks in invoke-command -scriptblock {} -Credential $myCreds blocks. This promotes the execution to the target system's version of PowerShell. It also makes it very easy to tack on a -ComputerName to run the script on the target instead of the runbook server.

You can then store a username and password pair as Orchestrator variables ( use the encryption option ) and insert them in your get-credential statement above. It feels very messy at first but you get used to it.

to test this, I ran PowerShell with -version 1. Here is my $PSVersionTable:

```<code>Name                           Value
----                           -----
CLRVersion                     2.0.50727.8689
BuildVersion                   6.1.7600.16385
PSVersion                      2.0
WSManStackVersion              2.0
PSCompatibleVersions           {1.0, 2.0}
SerializationVersion           1.1.0.1
PSRemotingProtocolVersion      2.1
</code>```

It had the same problem but it might not be a 1 for 1 match to what Orchestrator is going to run in. Post Output if it does not work and I'll do my best.

Body:
I have hundred thousands files located in ~40 folders (these subfolders being located in a same super-folder which I am going to call 'test'). I have tried to use powershell to replace a syntax found in the folder name but also in the filenames.

Basically my folders and files contains the same string of characters "PXXXX_0" that I want to batch rename in "PXXXX_"

Here is the code I used to first map the files & folders of interest

cd C:\test

Get-ChildItem -Filter "*PXXXX_0*" -Recurse | Rename-Item -NewName {$_.name -replace 'PXXXX_0', 'PXXXX_'} -whatif

This gives me the right files and previewed action (e.g.

What if: Performing operation "Rename File" on Target "Item: C:\test\PXXXX_053\PXXXX_053_558.jpg Destination C:\test\PXXXX_053\PXXXX_53_558.jpg

So this seems to work, but only on the files inside the folders that I also want to rename.

Now if I launch the code without -whatif it appears that only the folder names are rename, not the files...

I pretty sure I have missed something but since I am not really familiar with this language, I am struggling to find the solution. I am using Powershell v1.0

Thanks for any help.

Best

Body:
One Liner:

 $items = get-childitem -filter "*PXXXX_0*" -Recurse; foreach ($item in $items) { $newName = $item.fullname -replace 'PXXXX_0', 'PXXXX_'; rename-item $item.fullname -NewName $newName; }

Easy on the eyeballs:

$items = get-childitem -filter "*PXXXX_0*" -Recurse;
foreach ($item in $items) {
  $newName = $item.fullname -replace 'PXXXX_0', 'PXXXX_';
  rename-item $item.fullname -NewName $newName;
}

In short, PowerShell 1 was trying to rename the file with just the file name, not the full path. Basically it was not able to find your files when they were in subfolders.

I tested your logic in v5 and it works like you would expect it to.

I'm going to go out on a limb here and assume you are trying to run this script via System Center Orchestrator since you are working with PowerShell v1. I've felt that pain. I recommend wrapping all your PS scripts headed for runbooks in invoke-command -scriptblock {} -Credential $myCreds blocks. This promotes the execution to the target system's version of PowerShell. It also makes it very easy to tack on a -ComputerName to run the script on the target instead of the runbook server.

You can then store a username and password pair as Orchestrator variables ( use the encryption option ) and insert them in your get-credential statement above. It feels very messy at first but you get used to it.

to test this, I ran PowerShell with -version 1. Here is my $PSVersionTable:

Name                           Value
----                           -----
CLRVersion                     2.0.50727.8689
BuildVersion                   6.1.7600.16385
PSVersion                      2.0
WSManStackVersion              2.0
PSCompatibleVersions           {1.0, 2.0}
SerializationVersion           1.1.0.1
PSRemotingProtocolVersion      2.1

It had the same problem but it might not be a 1 for 1 match to what Orchestrator is going to run in. Post Output if it does not work and I'll do my best.


答:
I'd try to construct the formula and then use model.matrix and matrix multiplication, something like this:

```<code>betas$term = with(betas, ifelse(
    Form == "Li", CovGen,
    ifelse(Form == "Sq", sprintf("I(%s^2)", CovGen),
           ifelse(Form == "Ps", sprintf("log(%s)", CovGen), NA)
)))
betas
#   CovGen Form     Estimate   term
# 1      A   Li  0.029457318      A
# 2      B   Li  0.010031512      B
# 3      C   Li -0.015586419      C
# 4      D   Li -0.008713449      D
# 5      E   Li  0.036253899      E
# 6      F   Li -0.026307292      F
# 7      G   Li  0.086574212      G
# 8      G   Sq  0.061468915 I(G^2)
# 9      H   Ps  0.002297457 log(H)
(my_formula = as.formula(paste("~", paste(betas$term, collapse = " + "))))
#~A + B + C + D + E + F + G + I(G^2) + log(H)

X = model.matrix(my_formula, data = dat)
prediction = X %*% betas$Estimate
</code>```

As MrFlick says, this won't work with your current sample data.

Title: manually predict with different functional forms

Body:
I have a data frame with coefficients from a glm (betas below). The data frame contains the covariate label, covariate form, and the estimate. The forms are linear (Li), squared/quadratic (Sq), and log (Ps).

betas <- structure(list(CovGen = c("A", "B", "C", "D", "E", "F", "G",
                                   "G", "H"), Form = c("Li", "Li", "Li", "Li", "Li", "Li", "Li",
                                                       "Sq", "Ps"), Estimate = c(0.0294573176934061, 0.0100315121169383,
                                                                                -0.0155864186367343, -0.00871344935814372, 0.0362538988332902,
                                                                                -0.0263072916746069, 0.0865742118052235, 0.0614689145750204,
                                                                                0.00229745713752781)), .Names = c("CovGen", "Form", "Estimate"
                                                                                ), row.names = c(NA, 9L), class = "data.frame")

betas
  CovGen Form     Estimate
1      A   Li  0.029457318
2      B   Li  0.010031512
3      C   Li -0.015586419
4      D   Li -0.008713449
5      E   Li  0.036253899
6      F   Li -0.026307292
7      G   Li  0.086574212
8      G   Sq  0.061468915
9      H   Ps  0.002297457

I trying to apply the coefficient estimates to manually predict values for a new data frame (dat included here with dput).

dat <- structure(list(B = c(-1.47218074669544, -1.46929972689195, -1.46641870708846,
                            -1.46353768728497, -1.46065666748148, -1.45777564767799), C = c(-1.09847692593512,
                                                                                -1.09375316152745, -1.08902939711978, -1.08430563271211, -1.07958186830444,
                                                                                -1.07485810389677), D = c(-1.0109875688763, -1.00407851818141,
                                                                                -0.997169467486518, -0.990260416791627, -0.983351366096736, -0.976442315401845
                                                                                ), E = c(-3.19632050296668, -3.19041566990116, -3.18451083683563,
                                                                                -3.17860600377011, -3.17270117070458, -3.16679633763906), F = c(-2.81211918021003,
                                                                                -2.80673925496675, -2.80135932972346, -2.79597940448018, -2.7905994792369,
                                                                                -2.78521955399362), G = c(-2.32916817000267, -2.32368219245727,
                                                                                -2.31819621491187, -2.31271023736647, -2.30722425982107, -2.30173828227567
                                                                                ), H = c(0.442067970883549, 0.417909464459238, 0.393750958034926,
                                                                                0.369592451610615, 0.345433945186303, 0.321275438761992)), .Names = c("B",
                                                                                "C", "D", "E", "F", "G", "H"), row.names = c(NA, 6L), class = "data.frame")                                                                                "C", "D", "E", "F", "G", "H"), row.names = c(NA, 6L), class = "data.frame")

> dat
          B         C          D         E         F         G         H
1 -1.472181 -1.098477 -1.0109876 -3.196321 -2.812119 -2.329168 0.4420680
2 -1.469300 -1.093753 -1.0040785 -3.190416 -2.806739 -2.323682 0.4179095
3 -1.466419 -1.089029 -0.9971695 -3.184511 -2.801359 -2.318196 0.3937510
4 -1.463538 -1.084306 -0.9902604 -3.178606 -2.795979 -2.312710 0.3695925
5 -1.460657 -1.079582 -0.9833514 -3.172701 -2.790599 -2.307224 0.3454339
6 -1.457776 -1.074858 -0.9764423 -3.166796 -2.785220 -2.301738 0.3212754

I am trying to multiply the new data values in the dat df by the respective betas and account for functional forms. More specifically, in the example included here, I want to apply the Sq form of the G beta to dat$G^2 and the Ps H beta to log(dat$H). All other betas and values can simply be multiplied directly without accounting for functional forms. Note that the A beta is not applied to a new value in the dat df.

I may need to work through an award ifelse statement but am wondering if there are other ideas and/or suggestions.

I am working within a larger loop and will not have a consistent form for each covariate.

The desired result would be a matrix or df with a column containing the predicted values for each beta-form combination. For example, there would be a single column containing predicted values for all betas except for G which would have predicted values for G and G^2.

Thanks in advance.

Body:
I'd try to construct the formula and then use model.matrix and matrix multiplication, something like this:

betas$term = with(betas, ifelse(
    Form == "Li", CovGen,
    ifelse(Form == "Sq", sprintf("I(%s^2)", CovGen),
           ifelse(Form == "Ps", sprintf("log(%s)", CovGen), NA)
)))
betas
#   CovGen Form     Estimate   term
# 1      A   Li  0.029457318      A
# 2      B   Li  0.010031512      B
# 3      C   Li -0.015586419      C
# 4      D   Li -0.008713449      D
# 5      E   Li  0.036253899      E
# 6      F   Li -0.026307292      F
# 7      G   Li  0.086574212      G
# 8      G   Sq  0.061468915 I(G^2)
# 9      H   Ps  0.002297457 log(H)
(my_formula = as.formula(paste("~", paste(betas$term, collapse = " + "))))
#~A + B + C + D + E + F + G + I(G^2) + log(H)

X = model.matrix(my_formula, data = dat)
prediction = X %*% betas$Estimate

As MrFlick says, this won't work with your current sample data.


问:
jQuery toggle on Internet Explorer 7. Position and height Problem
i made a simple script that hide/show a hidden div. The page automatically adjusts its height on Firefox, Safari and Chrome.

However on Internet Explorer 7, the div overlaps the contents at the bottom
I'm currently using Joomla as CMS. Where have i gone wrong ? I have tried including "position:relative" for the bottom contents but it isn't helping.

```<code>$j(idName).css({"opacity": "0"});
$j(idName).slideToggle('fast', function() {
    $j(idName).animate({opacity:"1"}, function() {
        if(jQuery.browser.msie) {
            this.style.removeAttribute('filter');
            $j("#main-body").css("height","auto");
        }
    });

});
</code>```

Visit http://educationtechnologysummit.com/sponsors-a-partners.html to get a clearer view

答: Your CSS should work fine in IE7 if you remove the position: relative style from the .wrapper class in template.css (or, remove the .wrapper class entirely).

ContentLicense: CC BY-SA 2.5

Body:
i made a simple script that hide/show a hidden div. The page automatically adjusts its height on Firefox, Safari and Chrome.

However on Internet Explorer 7, the div overlaps the contents at the bottom
I'm currently using Joomla as CMS. Where have i gone wrong ? I have tried including "position:relative" for the bottom contents but it isn't helping.

$j(idName).css({"opacity": "0"});
$j(idName).slideToggle('fast', function() {
    $j(idName).animate({opacity:"1"}, function() {
        if(jQuery.browser.msie) {
            this.style.removeAttribute('filter');
            $j("#main-body").css("height","auto");
        }
    });

});

Visit http://educationtechnologysummit.com/sponsors-a-partners.html to get a clearer view


问:
Is there a way to read an input string with "ifstream"
Currently my programme takes a string as an input which I access using argc and argv

then I use

```<code>FILE *fp, *input = stdin;
fp = fopen("input.xml","w+b");
    while(fgets(mystring,100,input) != NULL)
    {
        fputs(mystring,fp);
    }
    fclose(fp);
</code>```

I did this part only to create a file input.xml which I then supply to

```<code>ifstream in("input.xml");
    string s((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>());
</code>```

to get s as a string(basic string).
Is there a way to feed my input directly to ifstream? (i.e feeding a string to ifstream).

答:
Let me get this straight:

You read a string from standard input

You write it to a file

You then read it from the file

And use the file stream object to create a string

That's crazy talk!

Drop the file streams and just instantiate the string from STDIN directly:

```<code>string s(
   (std::istreambuf_iterator<char>(std::cin)),
   std::istreambuf_iterator<char>()
);
</code>```

Remember, std::cin is a std::istream, and the IOStreams part of the standard library is designed for this sort of generic access to data.

Be aware, though, that your approach with std::istreambuf_iterator is not the most efficient, which may be a concern if you're reading a lot of data.

Body:
Currently my programme takes a string as an input which I access using argc and argv

then I use

FILE *fp, *input = stdin;
fp = fopen("input.xml","w+b");
    while(fgets(mystring,100,input) != NULL)
    {
        fputs(mystring,fp);
    }
    fclose(fp);

I did this part only to create a file input.xml which I then supply to

ifstream in("input.xml");
    string s((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>());

to get s as a string(basic string).
Is there a way to feed my input directly to ifstream? (i.e feeding a string to ifstream).

Body:
Let me get this straight:

You read a string from standard input

You write it to a file

You then read it from the file

And use the file stream object to create a string

That's crazy talk!

Drop the file streams and just instantiate the string from STDIN directly:

string s(
   (std::istreambuf_iterator<char>(std::cin)),
   std::istreambuf_iterator<char>()
);

Remember, std::cin is a std::istream, and the IOStreams part of the standard library is designed for this sort of generic access to data.

Be aware, though, that your approach with std::istreambuf_iterator is not the most efficient, which may be a concern if you're reading a lot of data.


问:
How to generate and export a big CSV file from my PHP server?
My server can be PHP or nodeJS, prefer staying in PHP but if it can't be done/better in node i'll appreciate the answer.

I want to generate and export a very big CSV file, the problem is that I can't load all the data from MySQL once, so currently I limit my data to some amount which doesn't crash the app, but it's slow and most data won't be exported.

I thought of this solutions to generate and export the CSV file:

Send 1..N calls to my server, each call will generate Part1...PartN CSV and the browser will append them [won't the browser crash?]

In one request, stream the data to the browser, but then how does the browser start downloading the file?

I don't mind the client will wait a lot, just want a good way to export ~200MB csv from the data I got in MySQL.

Thanks.

答:
This is a nice solution using Mysql notation:

```<code>SELECT *
INTO OUTFILE '/tmp/dbexport.csv'
FIELDS ENCLOSED BY '"'
TERMINATED BY ';'
ESCAPED BY '"'
LINES TERMINATED BY '\r\n'
FROM tables;
</code>```

Have a look at this answer too.

You could consider also use a .tsb (tab separated file) it does make no difference for you:

```<code>mysql your_database -e "select * from tables" -B > export.tsv
</code>```

Body:
This is a nice solution using Mysql notation:

SELECT *
INTO OUTFILE '/tmp/dbexport.csv'
FIELDS ENCLOSED BY '"'
TERMINATED BY ';'
ESCAPED BY '"'
LINES TERMINATED BY '\r\n'
FROM tables;

Have a look at this answer too.

You could consider also use a .tsb (tab separated file) it does make no difference for you:

mysql your_database -e "select * from tables" -B > export.tsv


问:
Javascript convert string to tab of float
how can I convert string variables to tab of float ?

I have a strange problem using a plugin : flot. I'm trying to draw a graph from data i've received from a server. The problem is that the plugin doesn't take values as string.

For exemple if I do this :

```<code>values=[[-955, 0.13325974],[-880, 0.23449805],[-795, 0.24322645],[-720, -0.008947457]];
$.plot(graphPersonnel, values);
</code>```

It work, but if I do this it doesnt :

```<code>values="[[-955, 0.13325974],[-880, 0.23449805],[-795, 0.24322645],[-720, -0.008947457]]";
$.plot(graphPersonnel, values);
</code>```

The problem is that I receive values from my server from a string and I extract them this way :

```<code>message = getGraph(); // = some_useless_string[[-955, 0.13325974],[-880, 0.23449805],[-795, 0.24322645],[-720, -0.008947457]]
values = message.replace("some_useless_string", "");
</code>```

So I would need to convert it from string to tab of arrays. How could i do that ?

Thx

答:
You're getting JSON data off the server as a string. simply parse the JSON using JSON.parse, and you'll be good to go:

var values="[[-955, 0.13325974],[-880, 0.23449805],[-795, 0.24322645],[-720, -0.008947457]]";

var valArray = JSON.parse(values);

console.log(valArray);```

But yeah, reading the javascript docs is always a good plan.

Body:
how can I convert string variables to tab of float ?

I have a strange problem using a plugin : flot. I'm trying to draw a graph from data i've received from a server. The problem is that the plugin doesn't take values as string.

For exemple if I do this :

values=[[-955, 0.13325974],[-880, 0.23449805],[-795, 0.24322645],[-720, -0.008947457]];
$.plot(graphPersonnel, values);

It work, but if I do this it doesnt :

values="[[-955, 0.13325974],[-880, 0.23449805],[-795, 0.24322645],[-720, -0.008947457]]";
$.plot(graphPersonnel, values);

The problem is that I receive values from my server from a string and I extract them this way :

message = getGraph(); // = some_useless_string[[-955, 0.13325974],[-880, 0.23449805],[-795, 0.24322645],[-720, -0.008947457]]
values = message.replace("some_useless_string", "");

So I would need to convert it from string to tab of arrays. How could i do that ?

Thx

Body:
You're getting JSON data off the server as a string. simply parse the JSON using JSON.parse, and you'll be good to go:

var values="[[-955, 0.13325974],[-880, 0.23449805],[-795, 0.24322645],[-720, -0.008947457]]";

var valArray = JSON.parse(values);

console.log(valArray);

But yeah, reading the javascript docs is always a good plan.


问:
center absolutely positioned  to absolutely positioned image
I try to make text that will be centered in the middle of the image, but the problem is that in my attempts I did not succeed to make it with resizing website. Any suggestions how to fix it and what I do wrong?

* {box-sizing:border-box}
body {font-family: Verdana,sans-serif;margin:0}

.imageSlide-container {
    position: relative;
    margin: auto;
}
.imageSlide {
    position: absolute;
    width: 100%;
    height: auto;
    opacity: 0;
    z-index: 1;

    -webkit-transition: opacity 2.5s;
    -moz-transition: opacity 2.5s;
    -o-transition: opacity 2.5s;
    transition: opacity 2.5s;
}
.showing {
    opacity: 1;
    z-index: 2;
}

.prev, .next {
    z-index: 3;
    position: absolute;
    top: 50%;
    width: auto;
    cursor: pointer;
    color: red;
    /*margin-top: -22px;*/
    padding: 15px;
    font-weight: bold;
    font-size: 18px;
    transition: 0.6s ease;
    border-radius: 0 3px 3px 0;

}
.next {
    right: 0;
    border-radius: 3px 0 0 3px;
}
.prev:hover, .next:hover {
    background-color: rgba(0,0,0,0.8)
}
@media only screen and (max-width: 300px) {
    .prev, .next,.text {font-size: 11px}```
<pre class="snippet-code-html lang-html prettyprint-override"><code><div class="imageSlide-container">
    <img class="imageSlide showing" src="https://www.w3schools.com/howto/img_nature_wide.jpg">
    <img class="imageSlide" src="https://www.w3schools.com/howto/img_fjords_wide.jpg">
    <img class="imageSlide" src="https://www.w3schools.com/howto/img_mountains_wide.jpg">

    ❮
    ❯
</div></code>```

答:
this is an updated version using your code from the comments below, i added top:0 to .imageSlide and solved the slider dropping the last image issue.

https://jsfiddle.net/k391zyn1/3/

html:

```<code><div class="imageSlide-container">
    <img class="imageSlide showing" src="https://www.w3schools.com/howto/img_nature_wide.jpg">
    <img class="imageSlide" src="https://www.w3schools.com/howto/img_fjords_wide.jpg">
    <img class="imageSlide" src="https://www.w3schools.com/howto/img_mountains_wide.jpg">

    <a class="prev" onclick="plusSlide-button(-1)">❮</a>
    <a class="next" onclick="plusSlide-button(1)">❯</a>
</div>
</code>```

css:

```<code>* {box-sizing:border-box}
body {font-family: Verdana,sans-serif;margin:0}

.imageSlide-container {
    position: relative;
    margin: auto;
}
.imageSlide {
    position: absolute;
    top:0;
    width: 100%;
    height: auto;
    opacity: 0;
    z-index: 1;

     -webkit-transition: opacity 2.5s;
    -moz-transition: opacity 2.5s;
    -o-t ransition: opacity 2.5s;
    transition: opacity 2.5s;
}
.showing {
    position: static;
    opacity: 1;
    z-index: 2;
}

.prev, .next {
    z-index: 3;
    position: absolute;
    top: 50%;
    width: auto;
    cursor: pointer;
    color: red;
    margin-top: -22px;
    padding: 15px;
    font-weight: bold;
    font-size: 18px;
    transition: 0.6s ease;
    border-radius: 0 3px 3px 0;

}
.next {
    right: 0;
    border-radius: 3px 0 0 3px;
}
.prev:hover, .next:hover {
    background-color: rgba(0,0,0,0.8)
}
@media only screen and (max-width: 300px) {
    .prev, .next,.text {font-size: 11px}
}
</code>```

javascript:

```<code>  var slides = document.querySelectorAll('.imageSlide');
   var currentSlide = 0;
   var slideInterval = setInterval(nextSlide,3000);

   function nextSlide(){
       slides[currentSlide].className = 'imageSlide';
       currentSlide = (currentSlide+1)%(slides.length);
       console.log('currentSlide',currentSlide);
       slides[currentSlide].className = 'imageSlide showing';
   }
</code>```

Title: center absolutely positioned <a> to absolutely positioned image

Body:
I try to make text that will be centered in the middle of the image, but the problem is that in my attempts I did not succeed to make it with resizing website. Any suggestions how to fix it and what I do wrong?

* {box-sizing:border-box}
body {font-family: Verdana,sans-serif;margin:0}

.imageSlide-container {
    position: relative;
    margin: auto;
}
.imageSlide {
    position: absolute;
    width: 100%;
    height: auto;
    opacity: 0;
    z-index: 1;

    -webkit-transition: opacity 2.5s;
    -moz-transition: opacity 2.5s;
    -o-transition: opacity 2.5s;
    transition: opacity 2.5s;
}
.showing {
    opacity: 1;
    z-index: 2;
}

.prev, .next {
    z-index: 3;
    position: absolute;
    top: 50%;
    width: auto;
    cursor: pointer;
    color: red;
    /*margin-top: -22px;*/
    padding: 15px;
    font-weight: bold;
    font-size: 18px;
    transition: 0.6s ease;
    border-radius: 0 3px 3px 0;

}
.next {
    right: 0;
    border-radius: 3px 0 0 3px;
}
.prev:hover, .next:hover {
    background-color: rgba(0,0,0,0.8)
}
@media only screen and (max-width: 300px) {
    .prev, .next,.text {font-size: 11px}

    ❮
    ❯

Body:
this is an updated version using your code from the comments below, i added top:0 to .imageSlide and solved the slider dropping the last image issue.

https://jsfiddle.net/k391zyn1/3/

html:

<div class="imageSlide-container">
    <img class="imageSlide showing" src="https://www.w3schools.com/howto/img_nature_wide.jpg">
    <img class="imageSlide" src="https://www.w3schools.com/howto/img_fjords_wide.jpg">
    <img class="imageSlide" src="https://www.w3schools.com/howto/img_mountains_wide.jpg">

    <a class="prev" onclick="plusSlide-button(-1)">&#10094;</a>
    <a class="next" onclick="plusSlide-button(1)">&#10095;</a>
</div>

css:

* {box-sizing:border-box}
body {font-family: Verdana,sans-serif;margin:0}

.imageSlide-container {
    position: relative;
    margin: auto;
}
.imageSlide {
    position: absolute;
    top:0;
    width: 100%;
    height: auto;
    opacity: 0;
    z-index: 1;

     -webkit-transition: opacity 2.5s;
    -moz-transition: opacity 2.5s;
    -o-t ransition: opacity 2.5s;
    transition: opacity 2.5s;
}
.showing {
    position: static;
    opacity: 1;
    z-index: 2;
}

.prev, .next {
    z-index: 3;
    position: absolute;
    top: 50%;
    width: auto;
    cursor: pointer;
    color: red;
    margin-top: -22px;
    padding: 15px;
    font-weight: bold;
    font-size: 18px;
    transition: 0.6s ease;
    border-radius: 0 3px 3px 0;

}
.next {
    right: 0;
    border-radius: 3px 0 0 3px;
}
.prev:hover, .next:hover {
    background-color: rgba(0,0,0,0.8)
}
@media only screen and (max-width: 300px) {
    .prev, .next,.text {font-size: 11px}
}

javascript:

  var slides = document.querySelectorAll('.imageSlide');
   var currentSlide = 0;
   var slideInterval = setInterval(nextSlide,3000);

   function nextSlide(){
       slides[currentSlide].className = 'imageSlide';
       currentSlide = (currentSlide+1)%(slides.length);
       console.log('currentSlide',currentSlide);
       slides[currentSlide].className = 'imageSlide showing';
   }


问:
ionic 2 infinite scroll error
I'm trying to do infinite scroll in ionic2 but whenever the page loads i get this error Cannot read property 'complete' of undefined

HTML

```<code><ion-content>

 <ion-list>
   <ion-item *ngFor="let item of posts" (click)="passurl($event,item)">
     <h2>{{item.title}}</h2>
     </ion-item>
 </ion-list>

 <ion-infinite-scroll (ionInfinite)="ionViewDidLoad($event)">
   <ion-infinite-scroll-content></ion-infinite-scroll-content>
 </ion-infinite-scroll>

</ion-content>
</code>```

JS

```<code>ionViewDidLoad(infiniteScroll){
    let loader = this.LoadingController.create({
    content: 'Please Wait'
    });
    loader.present().then(()=>{
      this.http.get('http://mynewweb/templates/messages/titles.php?church_id='+ this.church_id+'&msg_type='+this.NavParams.data.id+ "&page="+this.page).map(res => res.json()).subscribe(data =>{
            console.log(JSON.stringify(data));
            setTimeout(() => {
            for(var i=0;i<data.length;i++){
              this.numofposts = data.length;
            this.posts.push(data[i]);
             }
        infiniteScroll.complete();
          }, 500);
        });
     this.page=this.page+1;
    loader.dismiss();
    });

  }
</code>```

i followed the documentation on the ionic2 framework from their website. any help with this? thanks

答:
```<code>public play=true;
  public loadData(infiniteScroll?: any): void {
    let loader = this.LoadingController.create({
        content: 'Please Wait'
    });

    loader.present().then(()=>{

        this.http.get('http://mynewweb/templates/messages/titles.php?church_id='+ this.church_id+'&msg_type='+this.NavParams.data.id+ "&page="+this.page).map(res => res.json()).subscribe(
            data => {
              this.numofposts=this.numofposts+data.length;
              this.itemsdisplayed = this.itemsdisplayed + 10;

                if(this.itemsdisplayed > this.numofposts) {
                    // Hide the infiniteScroll if there's no more data
                    this.play = false;
                }

                setTimeout(() => {
                    for(var i=0;i<data.length;i++) {
                        this.numofposts = data.length;
                        this.posts.push(data[i]);
                    }

                    // Check if it's not null/undefined before calling the complete method
                    if(infiniteScroll) {
                        infiniteScroll.complete();
                    }

                }, 500);
            });

        this.page=this.page+1;
        loader.dismiss();
    });

}
</code>```

HTML

```<code><ion-infinite-scroll *ngIf="play" (ionInfinite)="loadData($event)">
  <ion-infinite-scroll-content></ion-infinite-scroll-content>
</ion-infinite-scroll>
</code>```

Title: ionic 2 infinite scroll error

Body:
I'm trying to do infinite scroll in ionic2 but whenever the page loads i get this error Cannot read property 'complete' of undefined

HTML

<ion-content>

 <ion-list>
   <ion-item *ngFor="let item of posts" (click)="passurl($event,item)">

{{item.title}}

     </ion-item>
 </ion-list>

 <ion-infinite-scroll (ionInfinite)="ionViewDidLoad($event)">
   <ion-infinite-scroll-content></ion-infinite-scroll-content>
 </ion-infinite-scroll>

</ion-content>

JS

ionViewDidLoad(infiniteScroll){
    let loader = this.LoadingController.create({
    content: 'Please Wait'
    });
    loader.present().then(()=>{
      this.http.get('http://mynewweb/templates/messages/titles.php?church_id='+ this.church_id+'&msg_type='+this.NavParams.data.id+ "&page="+this.page).map(res => res.json()).subscribe(data =>{
            console.log(JSON.stringify(data));
            setTimeout(() => {
            for(var i=0;i<data.length;i++){
              this.numofposts = data.length;
            this.posts.push(data[i]);
             }
        infiniteScroll.complete();
          }, 500);
        });
     this.page=this.page+1;
    loader.dismiss();
    });

  }

i followed the documentation on the ionic2 framework from their website. any help with this? thanks

Body:
public play=true;
  public loadData(infiniteScroll?: any): void {
    let loader = this.LoadingController.create({
        content: 'Please Wait'
    });

    loader.present().then(()=>{

        this.http.get('http://mynewweb/templates/messages/titles.php?church_id='+ this.church_id+'&msg_type='+this.NavParams.data.id+ "&page="+this.page).map(res => res.json()).subscribe(
            data => {
              this.numofposts=this.numofposts+data.length;
              this.itemsdisplayed = this.itemsdisplayed + 10;

                if(this.itemsdisplayed > this.numofposts) {
                    // Hide the infiniteScroll if there's no more data
                    this.play = false;
                }

                setTimeout(() => {
                    for(var i=0;i<data.length;i++) {
                        this.numofposts = data.length;
                        this.posts.push(data[i]);
                    }

                    // Check if it's not null/undefined before calling the complete method
                    if(infiniteScroll) {
                        infiniteScroll.complete();
                    }

                }, 500);
            });

        this.page=this.page+1;
        loader.dismiss();
    });

}

HTML

<ion-infinite-scroll *ngIf="play" (ionInfinite)="loadData($event)">
  <ion-infinite-scroll-content></ion-infinite-scroll-content>
</ion-infinite-scroll>


问:
How do you make scatterplot annotations appear on mouse click when scatterplot is based on dataframe and involves colour-coded data?
I have a dataset:

```<code>a   b   c   d
10-Apr-86   Jimmy   1   Silly.doc
11-Apr-86   Minnie  2   Lala.doc
12-Apr-86   Jimmy   3   Goofy.doc
13-Apr-86   Minnie  4   Hilarious.doc
14-Apr-86   Jimmy   5   Joyous.doc
15-Apr-86   Eliot   6   Crackingup.doc
16-Apr-86   Jimmy   7   Funny.doc
17-Apr-86   Eliot   8   Happy.doc
18-Apr-86   Minnie  9   Mirthful.doc
</code>```

Using the following code in python 2.7.12..

```<code>df = (pd.read_csv('python.csv'))
df_wanted = pd.pivot_table(
    df,
    index='a',
    columns='b',
    values='c')

df_wanted.index = pd.to_datetime(df_wanted.index)

fig = plt.figure()
ax1 = fig.add_subplot(111)

ax1.scatter(df_wanted.index, df_wanted['Jimmy'], s=50, c='b', marker="s")
ax1.scatter(df_wanted.index,df_wanted['Minnie'], s=50, c='r', marker="o")
ax1.scatter(df_wanted.index,df_wanted['Eliot'], s=50, c='g', marker="8")
plt.legend(loc='upper left');

for k, v in df.set_index('a').iterrows():
    plt.text(k, v['c'], v['d'])
plt.show()
</code>```

.. I can create the following visualization in matplotlib:

The problem: this is only a toy dataset.  When I apply this code to my real dataset, which has more than 3000 points, all the data labels blend together in a black illegible block.

I would like to avoid this problem by using the code here to make the data labels appear when they are clicked.

The issue I'm having is with this part of the above-mentioned code,

```<code>x=[1,2,3,4,5]
y=[6,7,8,9,10]

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
scat = ax.scatter(x, y)
DataCursor(scat, x, y)
plt.show()
</code>```

Obviously, I need to replace the "x" and "y" with my pivot table columns, but I don't know how to make scat = ax.scatter(x, y) or DataCursor(scat, x, y) work with my data.

I tried the following

```<code>scat = ax1.scatter(df_wanted.index, df_wanted['Minnie'], s=50, c='b', marker="s")
scat1 = ax1.scatter(df_wanted.index,df_wanted['Jimmy'], s=50, c='r', marker="o")
scat2 = ax1.scatter(df_wanted.index,df_wanted['Eliot'], s=50, c='g', marker="8")

DataCursor(scat,df_wanted.index,df_wanted['Minnie'])
DataCursor(scat1,df_wanted.index,df_wanted['Jimmy'])
DataCursor(scat2,df_wanted.index,df_wanted['Eliot'])
plt.show()
</code>```

But I get this error TypeError: Invalid Type Promotion

UPDATE: I used the code from here to get the doc name in the console:

```<code>from matplotlib.pyplot import figure, show
import numpy as npy
from numpy.random import rand
import pandas as pd

df = (pd.read_csv('python.csv'))
df_wanted = pd.pivot_table(
    df,
    index='a',
    columns='b',
    values='c')

df_wanted.index = pd.to_datetime(df_wanted.index)

if 1: # picking on a scatter plot     (matplotlib.collections.RegularPolyCollection)

    c = 'r'
    c1 = 'b'
    c2 = 'g'
    s = 85
    y = df_wanted['Minnie']
    z = df_wanted['Jimmy']
    f = df_wanted['Eliot']
    x = df_wanted.index
    def onpick3(event):
        ind = event.ind
        print npy.take(df['d'], ind)

    fig = figure()
    ax1 = fig.add_subplot(111)
    col = ax1.scatter(x, y, s, c, picker=True)
    ax2 = fig.add_subplot(111)
    col = ax1.scatter(x, z, s, c1, picker=True)
    ax3 = fig.add_subplot(111)
    col = ax1.scatter(x, f, s, c2, picker=True)
    plt.legend(loc='upper left')
    #fig.savefig('pscoll.eps')
    fig.canvas.mpl_connect('pick_event', onpick3)

show()
</code>```

The problem now is that the document name being returned is not accurate.  I think the problem is that the ind number is for each individual series.  I need a way to combine all the series, and assign an ind number to their total.

答:
I found a solution.  I realized I wanted to follow this example (Matplotlib scatterplot; colour as a function of a third variable), but needed to first make a single list of x values and a single list of y values, rather than individual lists of x and y values for each series.

```<code>import matplotlib.pyplot as plt
from matplotlib.pyplot import figure, show
import numpy as npy
from numpy.random import rand
import pandas as pd

df = (pd.read_csv('python.csv')) #upload dataset

df['a'] = pd.to_datetime(df['a']) #convert date column to useable format

x = list(df['a'].values.flatten()) #get dataframe column data in list format
y= list(df['c'].values.flatten()) #get dataframe column data in list format

var_names = list(df['b'].values.flatten()) #get dataframe column data in list format
var_names1 = list(set(var_names)) #get unique values from column b (names)

d = {var_names1[n]:n for n in range(len(var_names1))} #generate dictionary that assigns number to each unique name in col B

namesAsNumbers = [d[z] for z in var_names] #replace names with numbers in column B

c=  namesAsNumbers

if 1: # picking on a scatter plot (matplotlib.collections.RegularPolyCollection) # user picks point on scatter

    def onpick3(event):
        ind = event.ind
        print npy.take(df['d'], ind) #print the document name associated with the point that's been picked

    fig = figure()
    ax1 = fig.add_subplot(111)
    col = ax1.scatter(x, y, s= 100, c=c, picker=True)
    #fig.savefig('pscoll.eps')
    fig.canvas.mpl_connect('pick_event', onpick3)
    plt.legend()
show()
</code>```

Only problem I still have: can't seem to get a legend to appear.

Body:
I have a dataset:

a   b   c   d
10-Apr-86   Jimmy   1   Silly.doc
11-Apr-86   Minnie  2   Lala.doc
12-Apr-86   Jimmy   3   Goofy.doc
13-Apr-86   Minnie  4   Hilarious.doc
14-Apr-86   Jimmy   5   Joyous.doc
15-Apr-86   Eliot   6   Crackingup.doc
16-Apr-86   Jimmy   7   Funny.doc
17-Apr-86   Eliot   8   Happy.doc
18-Apr-86   Minnie  9   Mirthful.doc

Using the following code in python 2.7.12..

df = (pd.read_csv('python.csv'))
df_wanted = pd.pivot_table(
    df,
    index='a',
    columns='b',
    values='c')

df_wanted.index = pd.to_datetime(df_wanted.index)

fig = plt.figure()
ax1 = fig.add_subplot(111)

ax1.scatter(df_wanted.index, df_wanted['Jimmy'], s=50, c='b', marker="s")
ax1.scatter(df_wanted.index,df_wanted['Minnie'], s=50, c='r', marker="o")
ax1.scatter(df_wanted.index,df_wanted['Eliot'], s=50, c='g', marker="8")
plt.legend(loc='upper left');

for k, v in df.set_index('a').iterrows():
    plt.text(k, v['c'], v['d'])
plt.show()

.. I can create the following visualization in matplotlib:

The problem: this is only a toy dataset.  When I apply this code to my real dataset, which has more than 3000 points, all the data labels blend together in a black illegible block.

I would like to avoid this problem by using the code here to make the data labels appear when they are clicked.

The issue I'm having is with this part of the above-mentioned code,

x=[1,2,3,4,5]
y=[6,7,8,9,10]

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
scat = ax.scatter(x, y)
DataCursor(scat, x, y)
plt.show()

Obviously, I need to replace the "x" and "y" with my pivot table columns, but I don't know how to make scat = ax.scatter(x, y) or DataCursor(scat, x, y) work with my data.

I tried the following

scat = ax1.scatter(df_wanted.index, df_wanted['Minnie'], s=50, c='b', marker="s")
scat1 = ax1.scatter(df_wanted.index,df_wanted['Jimmy'], s=50, c='r', marker="o")
scat2 = ax1.scatter(df_wanted.index,df_wanted['Eliot'], s=50, c='g', marker="8")

DataCursor(scat,df_wanted.index,df_wanted['Minnie'])
DataCursor(scat1,df_wanted.index,df_wanted['Jimmy'])
DataCursor(scat2,df_wanted.index,df_wanted['Eliot'])
plt.show()

But I get this error TypeError: Invalid Type Promotion

UPDATE: I used the code from here to get the doc name in the console:

from matplotlib.pyplot import figure, show
import numpy as npy
from numpy.random import rand
import pandas as pd

df = (pd.read_csv('python.csv'))
df_wanted = pd.pivot_table(
    df,
    index='a',
    columns='b',
    values='c')

df_wanted.index = pd.to_datetime(df_wanted.index)

if 1: # picking on a scatter plot     (matplotlib.collections.RegularPolyCollection)

    c = 'r'
    c1 = 'b'
    c2 = 'g'
    s = 85
    y = df_wanted['Minnie']
    z = df_wanted['Jimmy']
    f = df_wanted['Eliot']
    x = df_wanted.index
    def onpick3(event):
        ind = event.ind
        print npy.take(df['d'], ind)

    fig = figure()
    ax1 = fig.add_subplot(111)
    col = ax1.scatter(x, y, s, c, picker=True)
    ax2 = fig.add_subplot(111)
    col = ax1.scatter(x, z, s, c1, picker=True)
    ax3 = fig.add_subplot(111)
    col = ax1.scatter(x, f, s, c2, picker=True)
    plt.legend(loc='upper left')
    #fig.savefig('pscoll.eps')
    fig.canvas.mpl_connect('pick_event', onpick3)

show()

The problem now is that the document name being returned is not accurate.  I think the problem is that the ind number is for each individual series.  I need a way to combine all the series, and assign an ind number to their total.

Body:
I found a solution.  I realized I wanted to follow this example (Matplotlib scatterplot; colour as a function of a third variable), but needed to first make a single list of x values and a single list of y values, rather than individual lists of x and y values for each series.

import matplotlib.pyplot as plt
from matplotlib.pyplot import figure, show
import numpy as npy
from numpy.random import rand
import pandas as pd

df = (pd.read_csv('python.csv')) #upload dataset

df['a'] = pd.to_datetime(df['a']) #convert date column to useable format

x = list(df['a'].values.flatten()) #get dataframe column data in list format
y= list(df['c'].values.flatten()) #get dataframe column data in list format

var_names = list(df['b'].values.flatten()) #get dataframe column data in list format
var_names1 = list(set(var_names)) #get unique values from column b (names)

d = {var_names1[n]:n for n in range(len(var_names1))} #generate dictionary that assigns number to each unique name in col B

namesAsNumbers = [d[z] for z in var_names] #replace names with numbers in column B

c=  namesAsNumbers

if 1: # picking on a scatter plot (matplotlib.collections.RegularPolyCollection) # user picks point on scatter

    def onpick3(event):
        ind = event.ind
        print npy.take(df['d'], ind) #print the document name associated with the point that's been picked

    fig = figure()
    ax1 = fig.add_subplot(111)
    col = ax1.scatter(x, y, s= 100, c=c, picker=True)
    #fig.savefig('pscoll.eps')
    fig.canvas.mpl_connect('pick_event', onpick3)
    plt.legend()
show()

Only problem I still have: can't seem to get a legend to appear.


问:
Package Microsoft.AspNet.WebApi.Owin 5.2.3 is not compatible with netcoreapp1.1
It's new WebApi project on .NET Core 1.1 on VS2017.

I'm going to install package DryIoc.WebApi.Owin from Nuget, but it requires a lot of dependent stuff under "Net45". One of them is Microsoft.AspNet.WebApi.Owin 5.2.3.

I had already added line in *.csproj file

```<code><PackageTargetFallback Condition="'$(TargetFramework)'=='Net45'">
  $(PackageTargetFallback);portable-net45+win8+wpa81+wp8
</PackageTargetFallback>
</code>```

but it didn't save me.

Any suggestions will be helpful.

答:
You can install that on .net core 2 + you've to configure Owin pipeline on top of asp.net core + you need visual studio 2017 preview 3

I'm gonna add a sample in 1 day

Body:
It's new WebApi project on .NET Core 1.1 on VS2017.

I'm going to install package DryIoc.WebApi.Owin from Nuget, but it requires a lot of dependent stuff under "Net45". One of them is Microsoft.AspNet.WebApi.Owin 5.2.3.

I had already added line in *.csproj file

<PackageTargetFallback Condition="'$(TargetFramework)'=='Net45'">
  $(PackageTargetFallback);portable-net45+win8+wpa81+wp8
</PackageTargetFallback>

but it didn't save me.

Any suggestions will be helpful.


问:
How show horizontal scrolling of UICollectionView in Objective c?
Currently i am using this code :

```<code>[_collectionVwForTopScroll registerNib:[UINib nibWithNibName:@"DrinksCollectionViewCell" bundle:nil] forCellWithReuseIdentifier:@"DrinksCollectionViewCell"];

_collectionVwForTopScroll.backgroundColor = [UIColor redColor];

UICollectionViewFlowLayout *collectionViewFlowLayout = [[UICollectionViewFlowLayout alloc] init];
[collectionViewFlowLayout setScrollDirection:UICollectionViewScrollDirectionHorizontal];
_collectionVwForTopScroll.collectionViewLayout = collectionViewFlowLayout;

_collectionVwForTopScroll.delegate = self;
_collectionVwForTopScroll.dataSource = self;
</code>```

But itemForIndexpath Notcalled .

答:
The method

```<code>- (__kindof UICollectionViewCell *)collectionView:(UICollectionView *)collectionView cellForItemAtIndexPath:(NSIndexPath *)indexPath;
</code>```

will be called if you have:

Added UICollectionViewDataSource protocol

Minimum amount (1) of sections and items in section

Check out those methods

Body:
Currently i am using this code :

[_collectionVwForTopScroll registerNib:[UINib nibWithNibName:@"DrinksCollectionViewCell" bundle:nil] forCellWithReuseIdentifier:@"DrinksCollectionViewCell"];

_collectionVwForTopScroll.backgroundColor = [UIColor redColor];

UICollectionViewFlowLayout *collectionViewFlowLayout = [[UICollectionViewFlowLayout alloc] init];
[collectionViewFlowLayout setScrollDirection:UICollectionViewScrollDirectionHorizontal];
_collectionVwForTopScroll.collectionViewLayout = collectionViewFlowLayout;

_collectionVwForTopScroll.delegate = self;
_collectionVwForTopScroll.dataSource = self;

But itemForIndexpath Notcalled .

Body:
The method

- (__kindof UICollectionViewCell *)collectionView:(UICollectionView *)collectionView cellForItemAtIndexPath:(NSIndexPath *)indexPath;

will be called if you have:

Added UICollectionViewDataSource protocol

Minimum amount (1) of sections and items in section

Check out those methods


问:
Background not covering full page
My background image is not covering the full page when I type in my nav bar. I have set the background width and height to 100%, which I thought would solve the problem.

https://jsfiddle.net/oefu0rmk/

CSS

```<code> #intro {
    background: #151515 url(../img/b1.jpg) no-repeat center bottom;
    -webkit-background-size: cover;
    -moz-background-size: cover;
    background-size: 650px;
    background-attachment: fixed;
    width: 100%;
    height: 100%;
    min-height: 720px;
    display: table;
    position: relative;
    text-align: center;
}

.intro-overlay{
    position:absolute;
    top: 0;
    left: 0;
    height: 100%;
    width: 100%;
    background: #111111;
    opacity: .85
}
</code>```

HTML

```<code><nav>
    <a href="#about">about me</a>
    <a href="#my parents">about my parents</a>
    <a href="#hobbies">My hobbies</a>
    </nav>

<body>

    <section id="intro">
        <div class="intro-overlay"></div>
            <div class="intro-content">
                <h5>Oh, hi there! My name is Tatsu</h5>
                <h1> (Tah-T-Su)</h1>
                <a class="button stroke smoothscroll" href="#about" title="about me button">find out more about me</a>
</div>
</code>```

答:
Your code doesn't match your fiddle. The problem is you are not adding the background to the whole page, rather to the #intro element. There is copy outside of this
.

Add this to your fiddle and it will extend to the full page:

```<code>body {
  background: #151515 url(../img/b1.jpg) no-repeat center bottom;
    -webkit-background-size: cover;
    -moz-background-size: cover;
    background-size: 650px;
    background-attachment: fixed;
}
</code>```

Title: Background not covering full page

Body:
My background image is not covering the full page when I type in my nav bar. I have set the background width and height to 100%, which I thought would solve the problem.

https://jsfiddle.net/oefu0rmk/

CSS

 #intro {
    background: #151515 url(../img/b1.jpg) no-repeat center bottom;
    -webkit-background-size: cover;
    -moz-background-size: cover;
    background-size: 650px;
    background-attachment: fixed;
    width: 100%;
    height: 100%;
    min-height: 720px;
    display: table;
    position: relative;
    text-align: center;
}

.intro-overlay{
    position:absolute;
    top: 0;
    left: 0;
    height: 100%;
    width: 100%;
    background: #111111;
    opacity: .85
}

HTML

    about me
    about my parents
    My hobbies

Oh, hi there! My name is Tatsu

 (Tah-T-Su)

                find out more about me

Body:
Your code doesn't match your fiddle. The problem is you are not adding the background to the whole page, rather to the #intro element. There is copy outside of this <section>.

Add this to your fiddle and it will extend to the full page:

body {
  background: #151515 url(../img/b1.jpg) no-repeat center bottom;
    -webkit-background-size: cover;
    -moz-background-size: cover;
    background-size: 650px;
    background-attachment: fixed;
}


问:
Pymongo bulk inserts
I am trying to insert documents in bulk. But it doesn't insert more than 84 documents during bulk insert.
Gives me this error:

```<code>in insert pymongo.errors.InvalidOperation: cannot do an empty bulk insert
</code>```

Is it possible to do batch inserts, like inserting 50 documents per insert?

答:
Just maintain a flag to track weather there is something to execute in bulk operation. So you can avoid the exception and making the db call.</p>

```<code>bulk_operation = collection.initialize_unordered_bulk_op()
execute_bulk = False
for _id in _ids:
    bulk_operation.find({'_id': _id}).update({'$set': {'key': 'value'}})
    execute_bulk = True

if execute_bulk:
    bulk_operation.execute()
</code>```

Title: Pymongo bulk inserts

Body:
I am trying to insert documents in bulk. But it doesn't insert more than 84 documents during bulk insert.
Gives me this error:

in insert pymongo.errors.InvalidOperation: cannot do an empty bulk insert

Is it possible to do batch inserts, like inserting 50 documents per insert?

Body:
Just maintain a flag to track weather there is something to execute in bulk operation. So you can avoid the exception and making the db call.

bulk_operation = collection.initialize_unordered_bulk_op()
execute_bulk = False
for _id in _ids:
    bulk_operation.find({'_id': _id}).update({'$set': {'key': 'value'}})
    execute_bulk = True

if execute_bulk:
    bulk_operation.execute()


问:
unexpected result in parsing nested json in qt (array does not exist)
I am new in qt and I also searched in stack overflow but I can't get my answer so this not a duplicated post because all similar post have array [] but in this code i have not any array
I want to parse this complex JSON file:

```<code>{
    "query": {

        "lang": "en-US",
        "results": {
            "channel": {
                "units": {
                    "distance": "mi",
                    "pressure": "in"
                },
                "ttl": "60",
                "location": {
                    "city": "city",
                    "country": "not important",
                    "region": " kkk"
                },
                "wind": {
                    "chill": "99",
                    "direction": "180",
                    "speed": "14"
                }
            }
       ...(more code)
</code>```

i want to get chill data but out put is " " , please help me to print chill data in qt

its a part of my code:

```<code>QNetworkAccessManager manager;

QNetworkReply *response = manager.get(QNetworkRequest(QUrl(url)));
QEventLoop event;
connect(response, SIGNAL(finished()), &event, SLOT(quit()));

event.exec();
json = response->readAll();

QJsonDocument doc = QJsonDocument::fromJson(json.toUtf8());

QJsonObject jsonObj = doc.object();

foreach (const QJsonValue &value, jsonObj) {

      QJsonObject jsonobj = value.toObject();

      qDebug() <<  jsonobj["chill"].toString();
}
</code>```

output of qDebug()<<doc.object();  is

```<code>D/libuntitled7.so(13258): : ** QJsonObject({"query":{"count":1,"created":"2017-07-06T21:21:16Z","lang":"en-US","results":{"channel":{"astronomy":{"sunrise":"5:48 am","sunset":"8:15 pm"},"atmosphere":{"humidity":"16","pressure":"875.0","rising":"0","visibility":"16.1"},"description":"Yahoo! Weather","image":{"height":"18","link":"http://weather.yahoo.com","title":"Yahoo! Weather","url":"http://l.yimg.com/a/i/brand/purplelogo//uh/us/news-wea.gif","width":"142"},"item":{"condition":{"code":"31","date":"Fri, 07 Jul 2017 12:30 AM IRDT","temp":"85","text":"Clear"},"description":"<![CDATA[<img src=\"http://l.yimg.com/a/i/us/we/52/31.gif\"/>\n<BR/>\n<b>Forecast:</b>\n<BR /> Fri - Sunny. High: 97Low: 78\n<BR /> Sat - Sunny. High: 100Low: 79\n<BR /> Sun - Sunny. High: 101Low: 81\n<BR /> Mon - Sunny. High: 100Low: 81\n<BR /> Tue - Mostly
D/libuntitled7.so(13258): ..\untitled7\dialog2.cpp:84 (void Dialog2::on_pushButton_clicked()):
</code>```

and next out put is 86

答:
You need to iterate in your channels array too. Easy solution:

```<code>const QJsonDocument doc = QJsonDocument::fromJson(response->readAll().toUtf8());
// Access to "query"
const QJsonObject queryObject = doc.object();
// Access to "results"
const QJsonObject resultsObject = queryObject.value("results").toObject();
// Access to "chanels"
const QJsonObject channelsObject = resultsObject.value("channels").toObject();
// Access to "wind"
const QJsonObject windObject = channelsObject.value("wind").toObject();
// And then access to "chill"
const QJsonValue chill = windObject.value("chill");
</code>```

Best practice its to create a recursive function to parse the JSON recursively.

Body:
I am new in qt and I also searched in stack overflow but I can't get my answer so this not a duplicated post because all similar post have array [] but in this code i have not any array
I want to parse this complex JSON file:

{
    "query": {

        "lang": "en-US",
        "results": {
            "channel": {
                "units": {
                    "distance": "mi",
                    "pressure": "in"
                },
                "ttl": "60",
                "location": {
                    "city": "city",
                    "country": "not important",
                    "region": " kkk"
                },
                "wind": {
                    "chill": "99",
                    "direction": "180",
                    "speed": "14"
                }
            }
       ...(more code)

i want to get chill data but out put is " " , please help me to print chill data in qt

its a part of my code:

QNetworkAccessManager manager;

QNetworkReply *response = manager.get(QNetworkRequest(QUrl(url)));
QEventLoop event;
connect(response, SIGNAL(finished()), &event, SLOT(quit()));

event.exec();
json = response->readAll();

QJsonDocument doc = QJsonDocument::fromJson(json.toUtf8());

QJsonObject jsonObj = doc.object();

foreach (const QJsonValue &value, jsonObj) {

      QJsonObject jsonobj = value.toObject();

      qDebug() <<  jsonobj["chill"].toString();
}

output of qDebug()<<doc.object();  is

D/libuntitled7.so(13258): : ** QJsonObject({"query":{"count":1,"created":"2017-07-06T21:21:16Z","lang":"en-US","results":{"channel":{"astronomy":{"sunrise":"5:48 am","sunset":"8:15 pm"},"atmosphere":{"humidity":"16","pressure":"875.0","rising":"0","visibility":"16.1"},"description":"Yahoo! Weather","image":{"height":"18","link":"http://weather.yahoo.com","title":"Yahoo! Weather","url":"http://l.yimg.com/a/i/brand/purplelogo//uh/us/news-wea.gif","width":"142"},"item":{"condition":{"code":"31","date":"Fri, 07 Jul 2017 12:30 AM IRDT","temp":"85","text":"Clear"},"description":"<![CDATA[\n
\nForecast:\n
 Fri - Sunny. High: 97Low: 78\n
 Sat - Sunny. High: 100Low: 79\n
 Sun - Sunny. High: 101Low: 81\n
 Mon - Sunny. High: 100Low: 81\n
 Tue - Mostly
D/libuntitled7.so(13258): ..\untitled7\dialog2.cpp:84 (void Dialog2::on_pushButton_clicked()):

and next out put is 86

Body:
You need to iterate in your channels array too. Easy solution:

const QJsonDocument doc = QJsonDocument::fromJson(response->readAll().toUtf8());
// Access to "query"
const QJsonObject queryObject = doc.object();
// Access to "results"
const QJsonObject resultsObject = queryObject.value("results").toObject();
// Access to "chanels"
const QJsonObject channelsObject = resultsObject.value("channels").toObject();
// Access to "wind"
const QJsonObject windObject = channelsObject.value("wind").toObject();
// And then access to "chill"
const QJsonValue chill = windObject.value("chill");

Best practice its to create a recursive function to parse the JSON recursively.


问:
Raspberry pi 3 cpuinfo ARMv7
I just installed last raspbian on my pi 3.
lscpu tells me that my cpu model is ARMv7 but it should be ARMv8 (according to rpi 3 specs).
Did I buy a fake card ?</p>

答:
It's running 32 bit so it reports v7.

The distro isn't 64bit yet (from what I read a week ago), when it is, it'll report v8.

Title: Raspberry pi 3 cpuinfo ARMv7


问:
How to export VARCHAR2 data that includes commas(!) to Excel from MSSQLMgmtStudio(2012)
First, a grumble: MS builds SQL Server Studio AND Excel, but can't make one save in the standard format of the other?

OK, I'm a data analyst, but not allowed to change/mod either the data or structures directly. So full READ, but no WRITE.

I'm trying to do a dump so I can do some of this analysis offline, as I have no remote access either.

So one VARCHAR2 column in this table is for comments on the purchase of the asset being described/tracked. Of course, there are commas. The only export types built into SQL Server Studio are .csv and .txt, and .csv just turns into a mess when 'comma' is included as a delimiter.

So after an hour or so of screwing around with this, (including reading a thread on methods for excluding the one column from a SELECT while still exporting the other 221 columns in the table, without having to write them all out manually (fun reading, impressive, but means I'd have to figure out which of them actually works, and then still export the one column separately and insert it in the Excel separately)) I am throwing this problem on the pile at StackOverflow.

Someone else must have worked around this frustration of the .csv format as export VS the commas embedded in 'comment' text.
Any help would be appreciated.

答:
So If you replace the ' to some special character you can export it.

```<code>Select
    Replace(columnName,'''','`')
from Table
</code>```

Other solution if you use the manager studio
https://learn.microsoft.com/en-us/sql/integration-services/import-export-data/start-the-sql-server-import-and-export-wizard

Body:
So If you replace the ' to some special character you can export it.

Select
    Replace(columnName,'''','`')
from Table

Other solution if you use the manager studio
https://learn.microsoft.com/en-us/sql/integration-services/import-export-data/start-the-sql-server-import-and-export-wizard


问:
Docker tar command doesnt work
I am trying to unzip samtools-0.1.18.tar.bz2 using the tar command

```<code>tar -xjvf samtools-0.1.18.tar.bz2
</code>```

I am able to run tar command without docker. but it gives me error when i try to untar it using docker. Can someone please help.

Dockerfile:

```<code>FROM amazonlinux
RUN yum -y install bzip2
WORKDIR  /sharedData
RUN echo "Installing SMTOOLS."
ADD https://sourceforge.net/projects/samtools/files/samtools/0.1.18/samtools-0.1.18.tar.bz2 /sharedData

RUN tar -xjvf samtools-0.1.18.tar.bz2
</code>```

Error:

```<code>Step 5/6 : ADD https://sourceforge.net/projects/samtools/files/samtools/0.1.18/samtools-0.1.18.tar.bz2 /sharedData
Downloading 130.2 kB
 ---> c524de5863ce
Removing intermediate container 941f57797c7b
Step 6/6 : RUN tar -xjvf samtools-0.1.18.tar.bz2
 ---> Running in 0391b2f5bf9c
tar (child): bzip2: Cannot exec: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
The command '/bin/sh -c tar -xjvf samtools-0.1.18.tar.bz2' returned a non-zero code: 2
</code>```

答:
This works:</p>

```<code>FROM amazonlinux
WORKDIR  /sharedData
RUN yum install -y bzip2
RUN yum install -y wget
RUN wget https://sourceforge.net/projects/samtools/files/samtools/0.1.18/samtools-0.1.18.tar.bz2
RUN tar -xjvf samtools-0.1.18.tar.bz2
</code>```

Title: Docker tar command doesnt work

Body:
I am trying to unzip samtools-0.1.18.tar.bz2 using the tar command

tar -xjvf samtools-0.1.18.tar.bz2

I am able to run tar command without docker. but it gives me error when i try to untar it using docker. Can someone please help.

Dockerfile:

FROM amazonlinux
RUN yum -y install bzip2
WORKDIR  /sharedData
RUN echo "Installing SMTOOLS."
ADD https://sourceforge.net/projects/samtools/files/samtools/0.1.18/samtools-0.1.18.tar.bz2 /sharedData

RUN tar -xjvf samtools-0.1.18.tar.bz2

Error:

Step 5/6 : ADD https://sourceforge.net/projects/samtools/files/samtools/0.1.18/samtools-0.1.18.tar.bz2 /sharedData
Downloading 130.2 kB
 ---> c524de5863ce
Removing intermediate container 941f57797c7b
Step 6/6 : RUN tar -xjvf samtools-0.1.18.tar.bz2
 ---> Running in 0391b2f5bf9c
tar (child): bzip2: Cannot exec: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
The command '/bin/sh -c tar -xjvf samtools-0.1.18.tar.bz2' returned a non-zero code: 2

Body:
This works:

FROM amazonlinux
WORKDIR  /sharedData
RUN yum install -y bzip2
RUN yum install -y wget
RUN wget https://sourceforge.net/projects/samtools/files/samtools/0.1.18/samtools-0.1.18.tar.bz2
RUN tar -xjvf samtools-0.1.18.tar.bz2


问:
MySQL query combining sums from multiple tables
I have the following query that is trying to generate a list of all clients, their projects, time spent on each project, total invoiced amount and total number of invoices.

```<code>SELECT
    c.name as client,
    p.name as project,
    SUM(t.hours) as hours,
    SUM(ci.amount) as invoice_amount,
    COUNT(ci.id) as number_invoices
FROM
    projects p
    INNER JOIN client_master c ON c.harvest_id = p.client_id
    INNER JOIN timesheet t ON t.project_id = p.id
    LEFT OUTER JOIN projects_contracts pc ON pc.project_id = p.id
    INNER JOIN contracts_invoices ci ON ci.contract_id = pc.contract_id
GROUP BY
    c.name,
    p.name
ORDER BY
    c.name,
    p.name,
    t.hours
</code>```

When I run the query, the "hours" are being calculated correctly but the sums/counts for invoice_amount and number_invoices aren't being joined contextually to the project but to the total number of invoices in the table. If I comment out one of the sum()'s then I get the proper value but not when they're both in.

Expected Results

Client A, Project 1, 1000, $10,000, 2

Client A, Project 2, 5000, $5,000, 1

Client B, Project 1, 100, $3000, 1

Actual Results

Client A, Project 1, 1000, $150,000, 20

Client A, Project 2, 5000, $50,000, 81

Client B, Project 1, 100, $83000, 12

I've tried messing w/ the grouping and join statements but haven't found success. Is there something simple I'm overlooking or do I need a total rewrite?

答:
This may not be the best way, but I bet it works</p>

```<code>SELECT
    c.name as client,
    p.name as project,
    t.hours as hours,
    ci.amount as invoice_amount,
    ci.count as number_invoices
FROM
    projects p
    INNER JOIN client_master c
        ON c.harvest_id = p.client_id
    INNER JOIN (SELECT project_id, SUM(hours) AS hours FROM timesheet GROUP BY project_id) t
        ON t.project_id = p.id
    LEFT OUTER JOIN projects_contracts pc
        ON pc.project_id = p.id
    LEFT OUTER JOIN (SELECT contract_id, SUM(amount) AS amount, COUNT(DISTINCT id) AS count FROM contracts_invoices GROUP BY contract_id) ci
        ON ci.contract_id = pc.contract_id
GROUP BY
    c.name,
    p.name
ORDER BY
    c.name,
    p.name,
    t.hours
</code>```

Body:
I have the following query that is trying to generate a list of all clients, their projects, time spent on each project, total invoiced amount and total number of invoices.

SELECT
    c.name as client,
    p.name as project,
    SUM(t.hours) as hours,
    SUM(ci.amount) as invoice_amount,
    COUNT(ci.id) as number_invoices
FROM
    projects p
    INNER JOIN client_master c ON c.harvest_id = p.client_id
    INNER JOIN timesheet t ON t.project_id = p.id
    LEFT OUTER JOIN projects_contracts pc ON pc.project_id = p.id
    INNER JOIN contracts_invoices ci ON ci.contract_id = pc.contract_id
GROUP BY
    c.name,
    p.name
ORDER BY
    c.name,
    p.name,
    t.hours

When I run the query, the "hours" are being calculated correctly but the sums/counts for invoice_amount and number_invoices aren't being joined contextually to the project but to the total number of invoices in the table. If I comment out one of the sum()'s then I get the proper value but not when they're both in.

Expected Results

Client A, Project 1, 1000, $10,000, 2

Client A, Project 2, 5000, $5,000, 1

Client B, Project 1, 100, $3000, 1

Actual Results

Client A, Project 1, 1000, $150,000, 20

Client A, Project 2, 5000, $50,000, 81

Client B, Project 1, 100, $83000, 12

I've tried messing w/ the grouping and join statements but haven't found success. Is there something simple I'm overlooking or do I need a total rewrite?

Body:
This may not be the best way, but I bet it works

SELECT
    c.name as client,
    p.name as project,
    t.hours as hours,
    ci.amount as invoice_amount,
    ci.count as number_invoices
FROM
    projects p
    INNER JOIN client_master c
        ON c.harvest_id = p.client_id
    INNER JOIN (SELECT project_id, SUM(hours) AS hours FROM timesheet GROUP BY project_id) t
        ON t.project_id = p.id
    LEFT OUTER JOIN projects_contracts pc
        ON pc.project_id = p.id
    LEFT OUTER JOIN (SELECT contract_id, SUM(amount) AS amount, COUNT(DISTINCT id) AS count FROM contracts_invoices GROUP BY contract_id) ci
        ON ci.contract_id = pc.contract_id
GROUP BY
    c.name,
    p.name
ORDER BY
    c.name,
    p.name,
    t.hours


问:
Can't make cross-domain requests without jQuery
I'm working on a web project where using jQuery (or other libraries/dependencies) is not an option, so I'm trying to replicate the code jQuery uses to make AJAX requests. The project previously used jQuery, so I've structured my replacement to the $.ajax() method to have the same behavior, however I cannot get mine to make cross-domain requests. When trying to load in scripts for example, I get this error in the console.

```<code>XMLHttpRequest cannot load <URL>. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin <URL> is therefore not allowed access.
</code>```

I'm familiar with CORS and the whole cross-origin-security policy and what that entails, but what I'm confused about is how jQuery can seem to circumvent that, while my code cannot. I'm thinking there must be something special jQuery is doing?

This is how my $.ajax() replacement function is currently scripted.

```<code>function ajax (options) {
    var request = new XMLHttpRequest();
    if ("withCredentials" in request)
        request.withCredentials = false;
    else if (typeof XDomainRequest !== "undefined")
        var request = new XDomainRequest();

    options.type = options.type || options.method || "GET";
    request.open(options.type.toUpperCase(), options.url, true);

    for (var i in options.headers) {
        request.setRequestHeader(i, options.headers[i]);
    }

    if (typeof options.beforeSend == "function")
        var go = options.beforeSend(request, options);

    if (go == false)
        return false;

    request.onreadystatechange = function () {
        if (this.readyState === XMLHttpRequest.DONE) {
            var resp = this.responseText;
            try {
                var body = JSON.parse(resp);
                this.responseJSON = body;
            }
            catch (err) {
                var body = resp;
            }

            if (this.status < 300 && typeof options.success == "function")
                options.success(body, this.status, this);
            else if (typeof options.error == "function")
                options.error(this, this.status, body);

            if (typeof options.complete == "function")
                options.complete(this, this.status);
        }
    };

    if (typeof options.data == "object" && options.data !== null)
        options.data = JSON.stringify(options.data);

    if (options.type.toUpperCase() != "GET")
        request.send(typeof options.data !== "undefined" ? options.data : null);
    else
        request.send();

    return request;
}
</code>```

Can someone point out if I'm missing something obvious? Do I need to manually also do the OPTIONS pre-flight or something?

答:
I've found a fix for this. The issue seemed to stem from trying to load in cross-domain scripts specifically. jQuery uses $.getScript() to do that, which actually just adds a script to the page instead of making a cross-domain HTTP request.

I used this code as a replacement for that function.

```<code>function getScript (url, callback) {
    var loadScript = function (url, callback) {
        var scrpt = document.createElement("script");
        scrpt.setAttribute("type", "text/javascript");
        scrpt.src = url;
        scrpt.onload = function () {
            scrpt.parentNode.removeChild(scrpt);
            console.log("Script is ready, firing callback!");
            if (callback)
                callback();
        };

        document.body.appendChild(scrpt);
    }

    var isReady = setInterval(function () {
        console.log(document.body);
        if (document.body) {
            clearInterval(isReady);
            loadScript(url, callback);
        }
    }, 25);
}
</code>```

Body:
I'm working on a web project where using jQuery (or other libraries/dependencies) is not an option, so I'm trying to replicate the code jQuery uses to make AJAX requests. The project previously used jQuery, so I've structured my replacement to the $.ajax() method to have the same behavior, however I cannot get mine to make cross-domain requests. When trying to load in scripts for example, I get this error in the console.

XMLHttpRequest cannot load <URL>. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin <URL> is therefore not allowed access.

I'm familiar with CORS and the whole cross-origin-security policy and what that entails, but what I'm confused about is how jQuery can seem to circumvent that, while my code cannot. I'm thinking there must be something special jQuery is doing?

This is how my $.ajax() replacement function is currently scripted.

function ajax (options) {
    var request = new XMLHttpRequest();
    if ("withCredentials" in request)
        request.withCredentials = false;
    else if (typeof XDomainRequest !== "undefined")
        var request = new XDomainRequest();

    options.type = options.type || options.method || "GET";
    request.open(options.type.toUpperCase(), options.url, true);

    for (var i in options.headers) {
        request.setRequestHeader(i, options.headers[i]);
    }

    if (typeof options.beforeSend == "function")
        var go = options.beforeSend(request, options);

    if (go == false)
        return false;

    request.onreadystatechange = function () {
        if (this.readyState === XMLHttpRequest.DONE) {
            var resp = this.responseText;
            try {
                var body = JSON.parse(resp);
                this.responseJSON = body;
            }
            catch (err) {
                var body = resp;
            }

            if (this.status < 300 && typeof options.success == "function")
                options.success(body, this.status, this);
            else if (typeof options.error == "function")
                options.error(this, this.status, body);

            if (typeof options.complete == "function")
                options.complete(this, this.status);
        }
    };

    if (typeof options.data == "object" && options.data !== null)
        options.data = JSON.stringify(options.data);

    if (options.type.toUpperCase() != "GET")
        request.send(typeof options.data !== "undefined" ? options.data : null);
    else
        request.send();

    return request;
}

Can someone point out if I'm missing something obvious? Do I need to manually also do the OPTIONS pre-flight or something?

Body:
I've found a fix for this. The issue seemed to stem from trying to load in cross-domain scripts specifically. jQuery uses $.getScript() to do that, which actually just adds a script to the page instead of making a cross-domain HTTP request.

I used this code as a replacement for that function.

function getScript (url, callback) {
    var loadScript = function (url, callback) {
        var scrpt = document.createElement("script");
        scrpt.setAttribute("type", "text/javascript");
        scrpt.src = url;
        scrpt.onload = function () {
            scrpt.parentNode.removeChild(scrpt);
            console.log("Script is ready, firing callback!");
            if (callback)
                callback();
        };

        document.body.appendChild(scrpt);
    }

    var isReady = setInterval(function () {
        console.log(document.body);
        if (document.body) {
            clearInterval(isReady);
            loadScript(url, callback);
        }
    }, 25);
}


问:
Why do statistics get out of date so quickly in SQL Server?
We have a somewhat complex SQL update query which gets run a few times a month.  Most of the time it seems to run really fast, but on some databases, it takes a really long time.  After running an "UPDATE STATISTICS" on the tables involved, the update immediately runs quickly again.  We finally set up a nightly task which calls UPDATE STATISTICS on all the tables in the database.  But that didn't seem to fix the problem.  We're still ending up having to run "UPDATE STATISTICS" manually each time.  Why would statistics go stale so quickly?

Here's approximately what the query looks like:

```<code>UPDATE DataTableA
SET DataTableA.IndexedColumn1 = 123456789, DataTableA.Flag1 = 1
FROM DataTableA WITH (INDEX(IX_DataTableA))
INNER JOIN GroupingTableA ON GroupingTableA.ForeignKey1 = GroupingTableA.PrimaryKey
INNER JOIN LookupTableA ON DataTableA.ForeignKey3 = LookupTableA.PrimaryKey
LEFT OUTER JOIN GroupingTableB ON DataTableA.IndexedColumn2 = GroupingTableB.IndexedColumn2
WHERE GroupingTableB.IndexedColumn1 = 123456789
AND DataTableA.IndexedColumn1 IS NULL
AND DataTableA.IndexedColumn2 IN ( ... 300 entries here ... )
AND DataTableA.Deleted = 0
AND GroupingTableA.Date <= GroupingTableB.EndDate
AND GroupingTableA.Date >= DATEADD(month, -1, GroupingTableB.StartDate)
AND LookupTableA.Column2 = 1
AND DataTableA.Status1 IN (1, 3)
AND DataTableA.Status2 NOT IN (1, 3, 9)
</code>```

DataTableA contains millions of rows.

GroupingTableA and GroupingTableB each contain tens of thousands of rows.

LookupTableA contains dozens of rows.

Index IX_DataTableA is an index on (IndexedColumn1 ASC, IndexedColumn2 ASC)

答:
When the auto stats are on, the engine would like to update the stats quite often, usually way more often than nightly:

```<code> Table Type | Empty Condition | Threshold When Empty |Threshold When Not Empty
_________________________________________________________________________________
 Permanent  | < 500 rows      | # of Changes >= 500  | # of Changes >= 500 + (20% of Cardinality)
___________________________________________________________________________
 Temporary  | < 6 rows        | # of Changes >= 6    | # of Changes >= 500 + (20% of Cardinality)
</code>```

But why not take out the guess and simply deploy a plan guide? See Understanding Plan Guides.

Body:
We have a somewhat complex SQL update query which gets run a few times a month.  Most of the time it seems to run really fast, but on some databases, it takes a really long time.  After running an "UPDATE STATISTICS" on the tables involved, the update immediately runs quickly again.  We finally set up a nightly task which calls UPDATE STATISTICS on all the tables in the database.  But that didn't seem to fix the problem.  We're still ending up having to run "UPDATE STATISTICS" manually each time.  Why would statistics go stale so quickly?

Here's approximately what the query looks like:

UPDATE DataTableA
SET DataTableA.IndexedColumn1 = 123456789, DataTableA.Flag1 = 1
FROM DataTableA WITH (INDEX(IX_DataTableA))
INNER JOIN GroupingTableA ON GroupingTableA.ForeignKey1 = GroupingTableA.PrimaryKey
INNER JOIN LookupTableA ON DataTableA.ForeignKey3 = LookupTableA.PrimaryKey
LEFT OUTER JOIN GroupingTableB ON DataTableA.IndexedColumn2 = GroupingTableB.IndexedColumn2
WHERE GroupingTableB.IndexedColumn1 = 123456789
AND DataTableA.IndexedColumn1 IS NULL
AND DataTableA.IndexedColumn2 IN ( ... 300 entries here ... )
AND DataTableA.Deleted = 0
AND GroupingTableA.Date <= GroupingTableB.EndDate
AND GroupingTableA.Date >= DATEADD(month, -1, GroupingTableB.StartDate)
AND LookupTableA.Column2 = 1
AND DataTableA.Status1 IN (1, 3)
AND DataTableA.Status2 NOT IN (1, 3, 9)

DataTableA contains millions of rows.

GroupingTableA and GroupingTableB each contain tens of thousands of rows.

LookupTableA contains dozens of rows.

Index IX_DataTableA is an index on (IndexedColumn1 ASC, IndexedColumn2 ASC)

Body:
When the auto stats are on, the engine would like to update the stats quite often, usually way more often than nightly:

 Table Type | Empty Condition | Threshold When Empty |Threshold When Not Empty
_________________________________________________________________________________
 Permanent  | < 500 rows      | # of Changes >= 500  | # of Changes >= 500 + (20% of Cardinality)
___________________________________________________________________________
 Temporary  | < 6 rows        | # of Changes >= 6    | # of Changes >= 500 + (20% of Cardinality)

But why not take out the guess and simply deploy a plan guide? See Understanding Plan Guides.


问:
How to prevent VLAN tag from being stripped from packets received by socket?
I'm writing a simple python script to parse the headers of raw data packets received via socket. The script runs on a VM and another VM is generating packets and sending them in. (Both VMs are running linux). This is part of a setup in which custom headers are made for the packets before sending them off, and the entire purpose of the script is to receive the packets completely unblemished via the socket, and verify their content.

The first header is a standard Ethernet header, with 6 bytes each for the DMAC and SMAC, four bytes for VLAN related fields (TPID, PCP and ID), and two bytes for Ethertype.

The problem I'm encountering is that the VLAN fields (four bytes) are stripped from the packets before I receive them (all the rest of the packet is present). I am not very familiar with the various elements of virtual HW along the network path between the two VMs, though I suspect that the stripping is being done there.

Here is the method I use to open the socket:

```<code>def openSocketToInterface(nicInterface):
    ETH_P_ALL = 3 # To receive all Ethernet protocols
    socketToInterface = socket.socket(socket.AF_PACKET, socket.SOCK_RAW,
        socket.htons(ETH_P_ALL))
    socketToInterface.bind((nicInterface, 0))
    subprocess.check_call('ifconfig %s promisc' % (nicInterface),
        shell=True)
    return socketToInterface
</code>```

Is there a way I can prevent the VLAN fields from being stripped?

答:
vSwitches and vNIC do not use tagging. The purpose of tags is to mark physical frames on the wire. Virtual frames don't need that, the vSwitch knows which port group they belong to.

Even when you set up a mirror port group on ESXi with all VLANs (promiscuous mode allowed, VID 4095), none of the frames are tagged.

A way to work around it is to use multiple vNICs. Alternatively, you can try mapping the the physical NIC into the VM (PCI passthrough or DirectPath).

Body:
I'm writing a simple python script to parse the headers of raw data packets received via socket. The script runs on a VM and another VM is generating packets and sending them in. (Both VMs are running linux). This is part of a setup in which custom headers are made for the packets before sending them off, and the entire purpose of the script is to receive the packets completely unblemished via the socket, and verify their content.

The first header is a standard Ethernet header, with 6 bytes each for the DMAC and SMAC, four bytes for VLAN related fields (TPID, PCP and ID), and two bytes for Ethertype.

The problem I'm encountering is that the VLAN fields (four bytes) are stripped from the packets before I receive them (all the rest of the packet is present). I am not very familiar with the various elements of virtual HW along the network path between the two VMs, though I suspect that the stripping is being done there.

Here is the method I use to open the socket:

def openSocketToInterface(nicInterface):
    ETH_P_ALL = 3 # To receive all Ethernet protocols
    socketToInterface = socket.socket(socket.AF_PACKET, socket.SOCK_RAW,
        socket.htons(ETH_P_ALL))
    socketToInterface.bind((nicInterface, 0))
    subprocess.check_call('ifconfig %s promisc' % (nicInterface),
        shell=True)
    return socketToInterface

Is there a way I can prevent the VLAN fields from being stripped?